Previous Thread
Next Thread
Print Thread
Hop To
Who's Online #227677 07/15/2009 6:32 PM
Joined: Feb 2007
Posts: 1,273
JAISP Online Tapedshut OP
Veteran
OP Online Tapedshut
Veteran
Joined: Feb 2007
Posts: 1,273
3 registered (JAISP, Gizmo, GEN), 13 Guests and 67 Spiders online.

Really

I have never seen a message board other then this one that always has more spiders in it then users and guest's. Is this for real? I have been watching this for a very long time. I'm sure there is some monkey business here some where.

Re: Who's Online [Re: JAISP] #227679 07/15/2009 6:54 PM
Joined: Jun 2006
Posts: 15,844
Gizmo Offline
UBB.threads Developer
Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,844
Any well crawled site will have a load of spiders, my main site:
3 registered (Gizmo, kanemartin, jonconley), 354 Guests and 92 Spiders online.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Who's Online [Re: Gizmo] #227680 07/15/2009 8:43 PM
Joined: Jul 2006
Posts: 2,143
David Dreezer Offline
Pooh-Bah
Offline
Pooh-Bah
Joined: Jul 2006
Posts: 2,143
It's normal nowadays. I've got a few sites that have more (by FAR) traffic from search engines than people. Since bing started binging MSNbot has been hitting hard and fast. Google is not far behind.

Wis these guys would find a way to realize when they already have content and it is unchanged. They spider the same sites over and over and over.....


This thread for sale. Click here! [Linked Image from navaho.infopop.cc]
Re: Who's Online [Re: David Dreezer] #227681 07/15/2009 9:46 PM
Joined: Jun 2006
Posts: 9,243
Rick Offline
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,243
And if you look closely you'll see many of them are from the same search spider. Like currently MSN has about 30 some bots online. Each of them coming from a unique IP and hits a single page, and then they go idle and a new one comes in.

Re: Who's Online [Re: Rick] #227682 07/16/2009 4:15 AM
Joined: Jan 2004
Posts: 2,468
driv Offline
Offline
Joined: Jan 2004
Posts: 2,468
When I was testing the backup of my forum I created a UBBthreads installation on a local PC, then promptly - forgot about it.

Went back a few days later to find the who's online box inhabited only by spiders. Damned things go everywhere. tongue


Using version :: 7.7.2
Re: Who's Online [Re: driv] #227690 07/16/2009 5:07 PM
Joined: Jun 2006
Posts: 15,844
Gizmo Offline
UBB.threads Developer
Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,844
Well, they more get stuck in infinite voids tongue... Immagine, you can navigate to a future date in the calendar, so what's to stop them from going to year 4098+... You really need to block them from certain threads locations...

robots.txt:
Code
User-agent: *
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Who's Online [Re: Gizmo] #227691 07/16/2009 6:05 PM
Joined: Feb 2007
Posts: 1,273
JAISP Online Tapedshut OP
Veteran
OP Online Tapedshut
Veteran
Joined: Feb 2007
Posts: 1,273
Originally Posted by Gizmo
You really need to block them from certain threads locations...


That should be built into the software and not need to be added by the user.

Re: Who's Online [Re: JAISP] #227692 07/16/2009 8:38 PM
Joined: Jun 2006
Posts: 9,243
Rick Offline
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,243
I think most are smart enough not to get stuck in the calendar, but we can certainly look at some type of SE block on that area if necessary. We do some pretty extensive spider tracking here, and while I have seen them look at the current calendar day, I've never seen one navigate around through several days.

Re: Who's Online [Re: Rick] #227693 07/16/2009 9:41 PM
Joined: Jun 2006
Posts: 15,844
Gizmo Offline
UBB.threads Developer
Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,844
I have a couple of entries in my robots.txt, I'll embed it here later on this evening (on a seperate dev machine right now)


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Who's Online [Re: Gizmo] #227694 07/17/2009 3:26 AM
Joined: Jun 2006
Posts: 15,844
Gizmo Offline
UBB.threads Developer
Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,844
Sorry for delay; heres my robots.txt:
Code
User-agent: *
Crawl-Delay: 3
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser


I'm looking at a client site right now, Yahoo has been "Viewing events for a day" for several hours on the Who's Online page; this is on a 7.5.3 page.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Forum Search
ShoutChat Box
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
Having issue with redirects to UBB classic URLs
by amciotola - 10/16/2019 1:11 AM
Unable to save General settings
by amciotola - 10/13/2019 4:23 AM
I see signature line images, but no avatars
by Morgan - 10/12/2019 2:21 PM
Show PDF
by Geoff - 10/09/2019 1:33 AM
Subscription’s Auto Return URL for PayPal
by Geoff - 09/27/2019 12:10 AM
Who's Online Now
1 registered members (JAISP), 98 guests, and 427 spiders.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Amusing Terain Scenics
Amusing Terain Scenics
by isaac, August 19
Sky places
Sky places
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 7.7.4
(Snapshot build 20190930)