Cuil was created by the Google creators; I've yet to really use it, but I'll dig around later

BTW, I banned them due to mass abuse/bw usage in the first place lol

Some of our customers have banned twiceler because it has brought their sites down. There is just no need to abuse a forum the way they do. It's almost like a DOS attack the way they hit you so hard.

My point exactly! it was crazy when I was hit by them, resources dropped to near nothing, and all I saw was them and yahoo on my forums lol

Stupid question... or maybe a good one for those of us who are not as skilled and knowledgeable as you Gents:

How do you know if your site is being abused by Twiceler?

A good question indeed

You only need to click on 'Who's Online' to be able to see a list of spiders etc

Great! But I am still running version 7.0.2 which would explain why my "who's on-line" does not show spiders.

WOL + Google Analyicts + AWStats led to noting the abuse; as they where hammering the hell out of the server...

Additionally, I have server information logged out to all of the ip's accessing the server and their hostname; twinceler was listed constantly.

It all added up to me ending up banning the hell out of them

AW stats, webalyzer, any log parser.

I hate for something like this to be my first post, but alas...

What are the chances of twiceler/cuill causing our site to basically come to it's knees?

Usually, everything is fine, but tonight, suddenly, mysqld is consuming a LOT of CPU time, and it's not going away. (In a dual proc setup, it's going from 98% to 160% cpu.) There were TONS of httpd procs/connections open. I initially thought it was an index problem, but once I saw this thread, I figured I'd look at the access logs, and sure enough... there were a LOT of twiceler requests in the timeframe of the slowdown.

For us, it's not a bandwidth issue, but a cpu issue. Has that been the case for anyone else or should I pursue my index theory?

Well, you could try adding a Crawl Delay to your robots.txt which should slow down the amount of visits the robot makes.

Example:

10 seconds:



2 minutes: