Spiders - UBB.threads PHP Forum Software Community

You are not logged in. [Log In] UBB.threads PHP Forum Software Community » Forums » General and Testing Forums » General Chat » Spiders

Site Links

Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com

Top Posters (30 Days)

Ruben	27
Gizmo	23
Bert	18
sb	5
hema0359	4
After the Rose	4
gladiator	3
skicomau	3
Willzter	2
mona said	2

Latest Photos

Topic Options

#249629 - 07/06/12 08:50 PM

Spiders

SteveS Offline

addict

Registered: 03/22/07
Posts: 466
Loc: Massachusetts

Any idea why the Google spiders would spend an inordinate amount of time doing their thing in the calendar on my new site? I welcome any spider activity at this point, but I have yet to see Google anywhere but the calendar. Other spiders have searched the good stuff, but not Google. What's up with that?

I definitely need more spider activity...

_________________________
Steve

UBB.classic from 2000-2003
UBB.threads from 2003-present!

Top

Express Hosting

Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."

#249630 - 07/07/12 12:42 AM

Re: Spiders [Re: SteveS]

Gizmo

Registered: 06/05/06
Posts: 15476
Loc: Portland, OR; USA

It gets stuck, because the URLs never cease (think, it clicks next page into oblivion).

Block it in your robots.txt with:

Code:

User-agent: *
Crawl-Delay: 3
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser

_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime Supporter, Beta Tester & Resident Post-A-Holic.
Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!

Top

#249644 - 07/08/12 09:11 AM

Re: Spiders [Re: SteveS]

SteveS Offline

addict

Registered: 03/22/07
Posts: 466
Loc: Massachusetts

Sadly, it seems to be ignoring the robots.txt file.

_________________________
Steve

UBB.classic from 2000-2003
UBB.threads from 2003-present!

Top

#249649 - 07/08/12 02:39 PM

Re: Spiders [Re: SteveS]

Gizmo

Registered: 06/05/06
Posts: 15476
Loc: Portland, OR; USA

It's not an immediate change, they check it every so often to eliminate mass hits on the file; give it a week and you should see things go back to where they should be...