This might be a stupid question but are you sure your pages are categorized?
My understanding of the permissions model is that it checks the global permissions first (user group permissions), if the user of a particular user group has sufficient privileges to access a given object then all well and good. If not, the you have to check the special permissions of the category, then of the object itself. If the user doesn't have any of those then it's game over.
So in your case, you should be able to have it so that anonymous users can't see any wiki pages except those that are in special categories, with the appropriate permissions designated. In general, those permissions need to be tiki_p_view_categories and tiki_p_view_categorized, in other words they need to be able to see the category and the object within the category.