I don't mind them being there, I'm just assuming that all the thrashing going on indicates something bad as it's seems inefficient on its face.

Well, there are some "black holes", such as the calendar, where they can go up day by day into oblivion, but that's an easy fix with robots.txt:
User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday

I never in almost 10 years with a forum online have had one person who registered by having gone through one of those ISPs.
In any case what they are charging their customers as a product is our content we not only produce but pay for the domain and webhosting costs

You've never had a user who's come to your site through a search engine and registered? Somehow I find that hard to believe, unless theres some really un-searched for content on your site...

IF you want to block SE's all together, just add this to your robots.txt:
User-agent: *
Disallow: /

They'll never touch your site again (so long as they follow the robots.txt standard, which most major ones do)

Honestly though, the max BW i've ever seen wasted by SE's in a month is about a gig; and this was on a huge site with loads of content that bankrolls about 4k+ a month due to advertising and depends on search engines...

Yahoo is a pig. Googlebot and the others are nowhere near as bad as Yahoo is. I've been doing this since 1995, so I have some idea of what I'm talking about.

Right now I have 4 registered users on, 6 guests, and 135 spiders. Almost all of them are Yahoo from various IP addresses - as was pointed out that's not one spider stuck, that's a ton of them. There's no reason why they should be that friggin piggish.

Makes me wonder if I shouldn't do something like this to them.

However, Yahoo's own help has some ideas:

http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html

I will agree that the crawler delay option they mention would be a valid way to slow down pounding; thanks for the link Joe :).

There are some areas of the UBB that spiders will get stuck in (as mentioned in the faq), the calendar is one of them, as are member files (neither of which need to be crawled).

Look at this.

http://www.black-sabbath.com/forums/ubbthreads.php?ubb=online

Right now I have 3 users, 10 guests, and 162 friggin search spiders. The overhelming majority are Yahoo. I implemented the delay option, it didn't seem to make much of a difference.

My Texas Rangers site isn't nearly as bad. 1 user (me), 0 guests, and 9 spiders (all but one are Yahoo). Sigh.

http://www.rangerfans.com/forums/ubbthreads.php?ubb=online

Unless I did it wrong, but I don't think so.

Lol, 160 yahoo spiders is nothing; I had 500 the one night lol

I added a delay to robots.txt
its in the root oy my domain

User-agent: *
Disallow: /cgi-bin/

User-agent: Slurp
Crawl-delay: 5

Has not slowed down Yahoo slurp ( which must mean they are saying they want to suck up everyones content ) one bit

Well, they don't check robots.txt every time they request something; they request it at a set interval that can take up to a few weeks to pass.