Ok, found the issue. Your guest user was actually part of the Users group, not the Guest group. So anyone not logged in was getting user group permissions.