If you are on a shared server, the issue might be related to other traffic on the server. Perhaps at a time when your host is running a backup etc.

Try and get a log of times to give to your host and find out the load at those times.