Every day my cache gets aggressively spidered - usually a return visit looking for files which no longer exist - and apache crashes as a result.
Robots.txt does not stop them - so every time apache crashes I check the logs and block the offending IP. A few hours later a new IP spiders the cache (all overseas IP numbers). I do this several times a day which is getting annoying.
Is there a way to stop the spidering of the cache files without affecting the operation of the forum? A permissions setting perhaps?
