Well, I've gone the last 3 days with UGN down, the culprit? The constant indexing and re-indexing from my little supposed "friend" Twiceler...
Twiceler is a supposed "new search engine" which just doesn't exist... why they have to constantly reindex I have no clue, but it does nothing more than RAPE my bandwidth...
It was ok for a while, just a little here, and a little there... Then I started going in and looking at my weblogs... 5+gb of unknown bandwidth usage that had to be cut down... And what a better time than being held offline due to a flood of traffic...
I finally get back online, and check the resources in current use... There's my "friend" Twiceler... with 8 threads just chumming away...
You all know me, the guy who says "well, if they have the money for this many servers they HAVE to be making something good, just wait it out"... Tonight, I'm no longer the passive voice of reason, Twiceler is a waste of my time, resources, and money.
Ways of BANNING Twiceler.Robots.txt is the quickest way to ban robots, BUT not all check back immediately, there can be a several day, week, or month passive check on their cache; so if it can wait, oh well, here you go...
Robots.txtUser-agent: twiceler
Disallow: /
here, we ban their IP's! These class C's are actually provided on their website, so we can easily ban them with 3 sections (note, you only need to turn the rewrite engine "on" ONCE in your .htaccess file)
RewriteEngine on
# Deny users IP's #
order allow,deny
deny from 38.99.13.
deny from 64.1.215.
deny from 208.36.144.
allow from all
These lines will have your webserver ban the useragent before they can even access your site.
RewriteEngine on
# Block Bad Bots #
RewriteCond %{HTTP_REFERER} cuill\.com [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteRule .* - [F,L]
3 ways to show our "good friend" Twiceler some good old-fashioned "hard love"...
