Since installing 7x and the spider tracking, I've noticed that Yahoo is all over our boards all of the time.

Our Who's Online

What gives? I am assuming something is wrong or yahoo wouldn't be doing this 24 hours / day.

I have 4 communities, and I can view them at several other sites; it's normal; Google used to do this a lot as well, but it seems they streamlined how they crawl data

I dunno... it seems weird. I agree that it's a quirk in the yahoo spider, but it's probably interacting with something on the boards to loop continuously over the same junk?

That's a performance hit I could do without if we could track it down.

Has anyone analyzed their logs to see what the yahoo spiders are actually doing? Is it legit traffic or are they caught somehow?

They show up in the WOL data reading different topics, forums, etc. It doesn't seem like they're 'stuck', it does seem like yahoo just sends a plethora of them out daily tho, one right after another.

Hm.

Wonder if they're somehow getting an error on their end parsing what they've collected and just go back to retry the next day.

I see seperate IP's on differant pages; so it's not that they're getting "stuck" it's just that they're sending a lot of bots out... It shouldn't effect anything too much (50 bots don't take up as much resources as you'd think).

There are some robots.txt rules to keep bots out of "un-needed spots" (such as the calendar, where they'll incriment day by day into oblivian).

50 bots would be nice. Right now I have 171 bots on my site, by far most of them are yahoo. And this is ALL The time.

http://boards.collectors-society.com/ubbthreads.php?ubb=online

There's just got to be something wrong with that.

Well even here on UBB they were up to about 700 bots at one time.

MSN, Yahoo and Google are always on my forum and I do not like it.
They in fact should pay all of us for they are getting our content for free to users they are charging a fee to use their system and the money they make from vendors who pay them.

If they do not pay then there should be some methods we use to block them as I also get some koint out of japan that also sucks onto our content.

I have never made a penny from anyone coming via those ISPs

Actually, I feel the opposite, I feel we shoudl pay them... Think of it, they download our pages for their DB, their users search their database and they send their users to our site, which tend to register and click advertising links which in turn make us money... For those of us who advertise anyway...

Now, if you want to stop them from visiting your site at all, you can, it's what robots.txt is for, just stop them from visiting your forums and never worry about them again (though you'll soon notice traffic decreases, and some sites depend on search engines for new users)