The Person who started the pm in the first place
The concern would be, what if the other users in the PM don't approve. Example. Say you invite me to a topic and I start badmouthing Gizmo. You decide to invite Gizmo to the private topic. Now he can see all the horrible things I said about him, even though I thought the conversation was just going to be me and you.