Recently we've taken and done away with the mail service on our production server, since we use Google Apps with all of our domains, there wasn't much point to keeping SMTP/IMAP/POP3 on the server.
Through trial and error we got it running by setting the "Use Specified SMTP Server" setting (CP -> Master Settings -> Primary Settings -> General Tab -> Advanced Options) to "smtp.googlemail.gom" and modifying the libs/phpmailer/class.phpmailer.php as follows (the UBB does not have fields available for these settings currently):
Find (these are not stored together, find them line by line):
var $Host = 'localhost';
var $Port = 25;
var $SMTPSecure = "";
var $SMTPAuth = false;
var $Username = '';
var $Password = '';
var $Host = 'smtp.googlemail.com';
var $Port = 465;
var $SMTPSecure = "ssl";
var $SMTPAuth = true;
var $Username = 'firstname.lastname@example.org';
var $Password = 'yourpassword';
This should work with any SMTP server, however your settings for these variables may need to be changed (depending on if you're using SMTP Authentication, SSL or not, and the available port).