How do you set an e-mail notify message's "reply to" address to be that of the user reporting a post?

We want to be able to reply to a user without having to go look up their profile by their user name to get their e-mail address.

"Did you click on it anyway?"