Well, I've gone the last 3 days with UGN down, the culprit? The constant indexing and re-indexing from my little supposed "friend" Twiceler...

Twiceler is a supposed "new search engine" which just doesn't exist... why they have to constantly reindex I have no clue, but it does nothing more than RAPE my bandwidth...

It was ok for a while, just a little here, and a little there... Then I started going in and looking at my weblogs... 5+gb of unknown bandwidth usage that had to be cut down... And what a better time than being held offline due to a flood of traffic...

I finally get back online, and check the resources in current use... There's my "friend" Twiceler... with 8 threads just chumming away...

You all know me, the guy who says "well, if they have the money for this many servers they HAVE to be making something good, just wait it out"... Tonight, I'm no longer the passive voice of reason, Twiceler is a waste of my time, resources, and money.

Ways of BANNING Twiceler.

Robots.txt is the quickest way to ban robots, BUT not all check back immediately, there can be a several day, week, or month passive check on their cache; so if it can wait, oh well, here you go...

Robots.txt


.htaccess (apache, you should also be able to define this in your httpd.conf)
here, we ban their IP's! These class C's are actually provided on their website, so we can easily ban them with 3 sections (note, you only need to turn the rewrite engine "on" ONCE in your .htaccess file)


Ban the U/A in your .htaccess
These lines will have your webserver ban the useragent before they can even access your site.


3 ways to show our "good friend" Twiceler some good old-fashioned "hard love"...

Can you send the bots back to cuill.com? Hopefully we can get it to fold in on itself.

'eh you can send a forward request, but I'd bet that their bots are configured to ignore their homepage

hmmm... we need to create a black hole web site to forward them to - a site that only uses the images and css sheets from cuill.com and just sends the bot back and forth over 2-3 pages, the page changes content just barely everytime it's accessed. The bot would think there were billions of pages to index.

'eh would waste your wn bw :/

I know you guy are trying to block it etc.
But in my post i just sent them an e-mail
and it never came back again

Click Me

This topic has been getting a lot of visits from various search engines. Topic is only 4 days old, and looking in the referer logs there are at least 100 hits from people coming from google or yahoo.

If you simply search for issues you'll find plenty of them... I have heard of plenty of users asking for it to be blacklisted and they just don't care, or will return after a while; so I decided to take a more direct approach and force it to never return...

And I'm glad to hear of the popularity! lol...

Update on this. Maybe you shouldn't block this any more as it is being used by a new Google rival - Cuil.

(Thread)

Google has a rival?
Bigger, better?

So was Betamax....

Cuil was created by the Google creators; I've yet to really use it, but I'll dig around later

BTW, I banned them due to mass abuse/bw usage in the first place lol

Some of our customers have banned twiceler because it has brought their sites down. There is just no need to abuse a forum the way they do. It's almost like a DOS attack the way they hit you so hard.

My point exactly! it was crazy when I was hit by them, resources dropped to near nothing, and all I saw was them and yahoo on my forums lol

Stupid question... or maybe a good one for those of us who are not as skilled and knowledgeable as you Gents:

How do you know if your site is being abused by Twiceler?

A good question indeed

You only need to click on 'Who's Online' to be able to see a list of spiders etc

Great! But I am still running version 7.0.2 which would explain why my "who's on-line" does not show spiders.

WOL + Google Analyicts + AWStats led to noting the abuse; as they where hammering the hell out of the server...

Additionally, I have server information logged out to all of the ip's accessing the server and their hostname; twinceler was listed constantly.

It all added up to me ending up banning the hell out of them

AW stats, webalyzer, any log parser.

I hate for something like this to be my first post, but alas...

What are the chances of twiceler/cuill causing our site to basically come to it's knees?

Usually, everything is fine, but tonight, suddenly, mysqld is consuming a LOT of CPU time, and it's not going away. (In a dual proc setup, it's going from 98% to 160% cpu.) There were TONS of httpd procs/connections open. I initially thought it was an index problem, but once I saw this thread, I figured I'd look at the access logs, and sure enough... there were a LOT of twiceler requests in the timeframe of the slowdown.

For us, it's not a bandwidth issue, but a cpu issue. Has that been the case for anyone else or should I pursue my index theory?

Well, you could try adding a Crawl Delay to your robots.txt which should slow down the amount of visits the robot makes.

Example:

10 seconds:



2 minutes:

I doubt they'll respect it... they aren't kind at ALL...

Well, I tried the IP banning approach. We'll see how it goes.

One thing though: We're on 7.2.2, and at one point I was seeing a bunch of twiceler requests in the accesslog, but I don't see anything other than google, msn, and yahoo in the spider list on the who's online page. Is this because of our version being slightly older or does cuil do something different to simply show up as an anonymous user?

Also, I noticed from our logs that they're using the 38.99.44.x subnet as well, so I added that.

Well, the UserAgent needs to be added to the CP, otherwise the search engine will show as anon...

There is a thread here somewhere showing different UA strings that several of us worked on... I really should sticky it, wherever it is...

Gizmo,

Allow me to make a very small contribution.

Here are two relevant threads. Perhaps one of these is the one you were thinking of.

UA Strings 1

UA Strings 2