Ahh, gotcha
Actually, that type of control is built in already on a per group basis. Access to forums are based on Groups. There are several built in groups, Admins, Moderators, Users and Guests. Then you can create your own custom groups that might contain 1 user, or 1000 users. At that point you can assign individual permissions to forums to those groups.
So, Group A, might be able to see and post to forums 1, 2 and 3. Group B might only be able to see and post to forums 2 and 3. Group C might be able to see forums 2 and 3 but only post to forum 3.