UBB.threads PHP Forum Software Community Forums Resources and Customizing Admin Zone What to do about spiders stuck on one topic.

Somebody posted on this subject a while back I think it was Gizmo. But can't seem to track it down.
But any way I have one topic that is 20 pages long that the spiders seem to constantly crawl. For several months now.
Okay for three months.
The topic has pretty much died as far as activity goes, Last post was 03/07/07 but the same spiders seem to keep going back to the same topic. It is mostly google and yahoo.
Is there such a thing as a stuck spider? And how do you correct it?

In case this is pertinent. Prior to 12/30/2008 I was on a windows server so after changing to a Linux server 01/01/2009 I was able to enable se friendly urls.

Well you could try blocking it in robots.txt. Example of a thread on your forum would be something like:

Question, If I block a particular topic would the spider "give up" for lack of a better word and not index the rest of the new topics?

No because your only 'requesting' that the spider ignores an individual post, not the whole forum / category or list of new topics in your case.

Okay. Thanks I will give that a try.

Keep in mind, every url currently allows more than one url to a thread... so you'll want to be sure those you block are what YOUR forum feeds to bots... There are several URLs (especially for each page) so it could be quite annoying to block them all lol...

Okay I blocked the topic for a few days and then unblocked it using robots.txt.
It seems to have fixed itself. The spiders have moved on to other topics.
Go figure!