Previous Thread
Next Thread
Print Thread
Hop To
#227677 07/15/2009 6:32 PM
Joined: Feb 2007
Posts: 1,294
Likes: 2
Veteran
Veteran
Joined: Feb 2007
Posts: 1,294
Likes: 2
3 registered (JAISP, Gizmo, GEN), 13 Guests and 67 Spiders online.

Really

I have never seen a message board other then this one that always has more spiders in it then users and guest's. Is this for real? I have been watching this for a very long time. I'm sure there is some monkey business here some where.

JAISP #227679 07/15/2009 6:54 PM
Joined: Jun 2006
Posts: 16,299
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,299
Likes: 116
Any well crawled site will have a load of spiders, my main site:
3 registered (Gizmo, kanemartin, jonconley), 354 Guests and 92 Spiders online.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #227680 07/15/2009 8:43 PM
Joined: Jul 2006
Posts: 2,143
Pooh-Bah
Pooh-Bah
Joined: Jul 2006
Posts: 2,143
It's normal nowadays. I've got a few sites that have more (by FAR) traffic from search engines than people. Since bing started binging MSNbot has been hitting hard and fast. Google is not far behind.

Wis these guys would find a way to realize when they already have content and it is unchanged. They spider the same sites over and over and over.....


This thread for sale. Click here! [Linked Image from navaho.infopop.cc]
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
And if you look closely you'll see many of them are from the same search spider. Like currently MSN has about 30 some bots online. Each of them coming from a unique IP and hits a single page, and then they go idle and a new one comes in.

Rick #227682 07/16/2009 4:15 AM
Joined: Jan 2004
Posts: 2,474
Likes: 3
D
Pooh-Bah
Pooh-Bah
D Offline
Joined: Jan 2004
Posts: 2,474
Likes: 3
When I was testing the backup of my forum I created a UBBthreads installation on a local PC, then promptly - forgot about it.

Went back a few days later to find the who's online box inhabited only by spiders. Damned things go everywhere. tongue

driv #227690 07/16/2009 5:07 PM
Joined: Jun 2006
Posts: 16,299
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,299
Likes: 116
Well, they more get stuck in infinite voids tongue... Immagine, you can navigate to a future date in the calendar, so what's to stop them from going to year 4098+... You really need to block them from certain threads locations...

robots.txt:
Code
User-agent: *
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #227691 07/16/2009 6:05 PM
Joined: Feb 2007
Posts: 1,294
Likes: 2
Veteran
Veteran
Joined: Feb 2007
Posts: 1,294
Likes: 2
Originally Posted by Gizmo
You really need to block them from certain threads locations...

That should be built into the software and not need to be added by the user.

JAISP #227692 07/16/2009 8:38 PM
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
I think most are smart enough not to get stuck in the calendar, but we can certainly look at some type of SE block on that area if necessary. We do some pretty extensive spider tracking here, and while I have seen them look at the current calendar day, I've never seen one navigate around through several days.

Rick #227693 07/16/2009 9:41 PM
Joined: Jun 2006
Posts: 16,299
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,299
Likes: 116
I have a couple of entries in my robots.txt, I'll embed it here later on this evening (on a seperate dev machine right now)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Gizmo #227694 07/17/2009 3:26 AM
Joined: Jun 2006
Posts: 16,299
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,299
Likes: 116
Sorry for delay; heres my robots.txt:
Code
User-agent: *
Crawl-Delay: 3
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser

I'm looking at a client site right now, Yahoo has been "Viewing events for a day" for several hours on the Who's Online page; this is on a 7.5.3 page.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
Bots
by Outdoorking - 04/13/2024 5:08 PM
Can you add html to language files?
by Baldeagle - 04/07/2024 2:41 PM
Do I need to rebuild my database?
by Baldeagle - 04/07/2024 2:58 AM
This is not a bug, but a suggestion
by Baldeagle - 04/05/2024 11:25 PM
Is UBB.threads still going?
by Aaron101 - 04/01/2022 8:18 AM
Who's Online Now
1 members (Nightcrawler), 607 guests, and 159 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)