I just leave them for a little while. The person could have left town or their mail server could be busy. Your mailer should continue to try and send if their busy.
Normally within a few days they go away on their own as the user shows up and finds it in their junk mail.
