I agree here; but we'd have to have several group types...
1. Non-Public - Users cannot add or remove themselves.
2. Public - Users can request to join and can leave at any time.
3. Private - Users cannot request to join, but can leave the group (for paid user groups for example).