depending on how you setup what a 'public' group is, sure. i'd assume that the admin would decide.
right now i have a forum where none of the users are a part of the hardwired 'user' group at all.
they key from my vantage point is to not allow a user to remove themselves from 'non-public' group. the admin can decide if the 'user' group is either public or non-public.
the admin would probably also decide that the moderator and admin groups were off limits too!
bottom line though: use the 'user' group as you see fit