The person who starts a PM should have the rights not just to invite but also to kick out whatever user(s) they want.