Are you really running your site with only 4gb ram? That's the same amount I have for my entire VPS, and my sites aren't nearly as active as yours...
At this point, since it's more than likely a server resource issue, and that's outside the scope of user to user support on the forums, I'd advise contacting your host/data center regarding server tuning and recommendations on their end.