Previous Thread
Next Thread
Print Thread
Hop To
Joined: Jun 2007
Posts: 286
T
enthusiast
enthusiast
T Offline
Joined: Jun 2007
Posts: 286
This business http://cuill.com/
hammers our forum with some robot called
Twiceler is an experimental robot as they state on their website.

I have sent them 2 emails telling them to cease hammering my forum with no reply from them

What and why are they doing this ?


JR
Team ZR-1 Corvette Racer's
Joined: Jun 2007
Posts: 286
T
enthusiast
enthusiast
T Offline
Joined: Jun 2007
Posts: 286
I just finally got a response from them and they are using our bandwidth to test their product so they can sell their services
------------

Twiceler is an experimental crawler that we are developing for our new search engine.
It is important to us that it obey robots.txt, and that it not crawl sites that do not
wish to be crawled. I will add *www.teamzr1.com <http://teamzr1.com> *to our list of sites to exclude
and I apologize for any inconvenience this has caused you.


Sincerely,


James Akers
Operations Engineer
Cuill, Inc.


JR
Team ZR-1 Corvette Racer's
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
robots.txt is a way to stop most robots from visiting your site, most will respect your choice to NOT allow them to visit your site.

Simply create a file named robots.txt and enter:
Code
User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Disallow:/

Keep in mind that most bots have a cache of your robots.txt which they update on a regular basis into their database, i believe Twinceler uses a 7 day cache.

If you wanted to block them by say a .htaccess file (if you're on an apache webserver and have applicable permissions), you could do so as:
Code
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Twiceler
RewriteRule ^.* - [F]


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
1 members (Ruben), 476 guests, and 111 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)