|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
Twiceler (leech?) - Found in Spider List. Hi, Ive posted this in the Admin section as i think its only the admin that can resolve this. I have been notified that Twiceler steels bandwidth, and to stop it you can send them an e-mail to leave your site alone. More Info .... So is it something to worry about or can i just add to my Robot.txt User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow:/ ---- Your thoughts / Solutions / Truth / Myth ?? This site thinks they lost 2GB per Month Linky Poo
Last edited by Mark S; 09/04/2007 7:08 PM.
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
9949 Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robotHITS 9948 FILES 9920 KBytes 100790 IP 208.36.144.9 And thats only after 4 days September !
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
well, they do a lot of pounding i will say; i hear it's a startup webspider so it's only understandable that they aren't as kind as google/yahoo...
I'd just monitor it, if it worries you, ban it in robots.txt, else i wouldn't worry about it...
Keep in mind that NO robot automatically reads your robots.txt as soon as you update it, they instead cache the file for up to 2 weeks, so even if you robots.txt it there won't be an immediate change.
|
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
I've tried the e-mail route as in the link i posted.
Will let you know what / if i get a reply.
costello@cuill.com
What i think is happening is since this one has come on
the scene Google and Yahoo have slowed (Lower Numbers)
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
'eh I don't like dealing with humans when it comes to SE's... It respects the robots.txt standard, so it's all that matters to me; likely all the "human" will do is tell it to refresh the robots.txt file immediately 
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
What i think is happening is since this one has come on the scene Google and Yahoo have slowed (Lower Numbers) Well, not really... Think of it, there are trillions of sites on the internet; why throw all of your bandwidth at one site when you can distribute it and cover more ground... Plus they know that by throwing too much bandwidth AT your site, theres a good chance to crash your server (mmm, flooding)
|
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
Ahhh Thanks for the feedback 
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
My Leechy friend has now gone No reply to my e-mail but i think its gone now I hadn't amended my robot.txt
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
Well, they own quite a large netblock, comparable to some of what google and yahoo send at me; as such they're obviously not there to be a nuscense; so as they're sending about the same ammount of bots at me as yahoo, I think I'll let them stay in the hopes that they may be something good in the future ...
|
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
This one would just sit there Twiceler (leech?) All on its own, just the one, no others. He's still missing and a new one has arrived which i'm taking to be Friendly EntireWeb.com
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
'eh yahoo rapes just as much bw as twincler does (at least on my forums), which is about what google "used" to...
|
|
|
|
|
Joined: May 2006
Posts: 579
addict
|
|
addict
Joined: May 2006
Posts: 579 |
Yes, we've just had to ban Yahoo with our robots.txt. It was sucking up a HUGE amount of our bandwidth, and for the first time we were in danger of going over our bandwidth limit for the month.
I've counted as many as 70 Yahoo bots on my site at once. What is that??
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
Yahoo wants to grab as much as possible in as little time; google on the otherhand knows all sites don't have unlimited bw...
|
|
|
|
|
Joined: Dec 2003
Posts: 1,796
Pooh-Bah
|
|
Pooh-Bah
Joined: Dec 2003
Posts: 1,796 |
ugh, twiceler is the top 2 bandwidth stealers on every site I run - 46% of all bandwidth for the month  I'm going to try the robots.txt file for now and see how it goes.
|
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
What are your bandwidth limits?
How much do you get charged after the limit?
|
|
|
|
|
Joined: Dec 2003
Posts: 1,796
Pooh-Bah
|
|
Pooh-Bah
Joined: Dec 2003
Posts: 1,796 |
My bandwidth limit is something like 350gb/month - I don't run over, but the problem is some un-used bot (you can't even use their search service, it's not operational yet/ever) is constantly indexing and re-indexing my site over and over. It's more of a leech than anything else and they need to fix it or shut it off :smash:
|
|
|
|
|
Joined: Dec 2006
Posts: 1,235
veteran
|
|
veteran
Joined: Dec 2006
Posts: 1,235 |
I blocked Twiceler when the Bots section first appeared in UBBt. Here's a current list of blocked bots in my .htaccess file that have visited and leached: RewriteEngine On
RewriteCond %{HTTP_REFERER} iaea\.org [OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeBot [OR]
RewriteCond %{HTTP_USER_AGENT} BecomeJPBot [OR]
RewriteCond %{HTTP_USER_AGENT} BilgiBot [OR]
RewriteCond %{HTTP_USER_AGENT} Bot [OR]
RewriteCond %{HTTP_USER_AGENT} ContactBot [OR]
RewriteCond %{HTTP_USER_AGENT} EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} Gaisbot [OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [OR]
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR]
RewriteCond %{HTTP_USER_AGENT} IRLbot [OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} my-heritrix-crawler [OR]
RewriteCond %{HTTP_USER_AGENT} Psbot [OR]
RewriteCond %{HTTP_USER_AGENT} PlantyNet_WebRobot [OR]
RewriteCond %{HTTP_USER_AGENT} RobSoft [OR]
RewriteCond %{HTTP_USER_AGENT} SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} shelob [OR]
RewriteCond %{HTTP_USER_AGENT} sohu-search [OR]
RewriteCond %{HTTP_USER_AGENT} sogou [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-spider [OR]
RewriteCond %{HTTP_USER_AGENT} sogou-web-spider [OR]
RewriteCond %{HTTP_USER_AGENT} Twiceler [OR]
RewriteCond %{HTTP_USER_AGENT} wwwster [OR]
RewriteCond %{HTTP_USER_AGENT} Y!J-SRD [OR]
RewriteCond %{HTTP_USER_AGENT} "Yahoo! Slurp China" [OR]
RewriteCond %{HTTP_USER_AGENT} YANDEX
RewriteRule .* - [F,L]
|
|
|
|
|
Joined: Nov 2006
Posts: 3,095 Likes: 1
Carpal Tunnel
|
|
Carpal Tunnel
Joined: Nov 2006
Posts: 3,095 Likes: 1 |
What are your bandwidth limits? I get 3TB a month - but since my site is down right now except for a few downloads it doesn't use much
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
My dedi is something like 3tb transfer X 40gb disk.
My VPS which is now retired was something like 350gb transfer X 30gb space.
|
|
|
Bots
by Outdoorking - 04/13/2024 5:08 PM
|
|
|
|
|
|
|
|
|
|
|
1 members (Nightcrawler),
445
guests, and
142
robots. |
|
Key:
Admin,
Global Mod,
Mod
|
|
|
|
