If the referer check is causing major issues, go ahead and turn it off. The check itself isn't foolproof as it relies on the referer variable. It basically is a bit of an attemp to help with spammers, but most good spammers can spoof the referer variable anyways.