You'd use Groups and Forum Permissions. First you'd create a group and then add the user's you want to that group.
You'd then go in and edit the Forum's Permissions, making it so only that group and your Admins/Moderators have access to it, the "Can See Forum" Permission. Make sure it unset for the Users group and the Guest group.