|
|
Joined: Feb 2007
Posts: 1,294 Likes: 2
Veteran
|
|
Veteran
Joined: Feb 2007
Posts: 1,294 Likes: 2 |
3 registered (JAISP, Gizmo, GEN), 13 Guests and 67 Spiders online.
Really
I have never seen a message board other then this one that always has more spiders in it then users and guest's. Is this for real? I have been watching this for a very long time. I'm sure there is some monkey business here some where.
|
|
|
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129 |
Any well crawled site will have a load of spiders, my main site:
3 registered (Gizmo, kanemartin, jonconley), 354 Guests and 92 Spiders online.
|
|
|
|
|
Joined: Jul 2006
Posts: 2,143
Pooh-Bah
|
|
Pooh-Bah
Joined: Jul 2006
Posts: 2,143 |
It's normal nowadays. I've got a few sites that have more (by FAR) traffic from search engines than people. Since bing started binging MSNbot has been hitting hard and fast. Google is not far behind.
Wis these guys would find a way to realize when they already have content and it is unchanged. They spider the same sites over and over and over.....
|
|
|
|
|
Joined: Jun 2006
Posts: 9,242 Likes: 1
Former Developer
|
|
Former Developer
Joined: Jun 2006
Posts: 9,242 Likes: 1 |
And if you look closely you'll see many of them are from the same search spider. Like currently MSN has about 30 some bots online. Each of them coming from a unique IP and hits a single page, and then they go idle and a new one comes in.
|
|
|
|
|
Joined: Jan 2004
Posts: 2,474 Likes: 3
Pooh-Bah
|
|
Pooh-Bah
Joined: Jan 2004
Posts: 2,474 Likes: 3 |
When I was testing the backup of my forum I created a UBBthreads installation on a local PC, then promptly - forgot about it. Went back a few days later to find the who's online box inhabited only by spiders. Damned things go everywhere. 
|
|
|
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129 |
Well, they more get stuck in infinite voids ... Immagine, you can navigate to a future date in the calendar, so what's to stop them from going to year 4098+... You really need to block them from certain threads locations... robots.txt: User-agent: *
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
|
|
|
|
|
Joined: Feb 2007
Posts: 1,294 Likes: 2
Veteran
|
|
Veteran
Joined: Feb 2007
Posts: 1,294 Likes: 2 |
You really need to block them from certain threads locations... That should be built into the software and not need to be added by the user.
|
|
|
|
|
Joined: Jun 2006
Posts: 9,242 Likes: 1
Former Developer
|
|
Former Developer
Joined: Jun 2006
Posts: 9,242 Likes: 1 |
I think most are smart enough not to get stuck in the calendar, but we can certainly look at some type of SE block on that area if necessary. We do some pretty extensive spider tracking here, and while I have seen them look at the current calendar day, I've never seen one navigate around through several days.
|
|
|
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129 |
I have a couple of entries in my robots.txt, I'll embed it here later on this evening (on a seperate dev machine right now)
|
|
|
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129
|
|
Joined: Jun 2006
Posts: 16,379 Likes: 129 |
Sorry for delay; heres my robots.txt: User-agent: *
Crawl-Delay: 3
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser I'm looking at a client site right now, Yahoo has been "Viewing events for a day" for several hours on the Who's Online page; this is on a 7.5.3 page.
|
|
|
|
2 members (Mender, Gizmo),
101
guests, and
149
robots. |
|
Key:
Admin,
Global Mod,
Mod
|
|
|
|
![[Linked Image from navaho.infopop.cc]](http://navaho.infopop.cc/progress.gif)
