Yahoo is a pig. Googlebot and the others are nowhere near as bad as Yahoo is. I've been doing this since 1995, so I have some idea of what I'm talking about.
Right now I have 4 registered users on, 6 guests, and 135 spiders. Almost all of them are Yahoo from various IP addresses - as was pointed out that's not one spider stuck, that's a ton of them. There's no reason why they should be that friggin piggish.
Makes me wonder if I shouldn't do something
like this to them.
However, Yahoo's own help has some ideas:
http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html
