|
|
|
|
Joined: Mar 2007
Posts: 522
Addict
|
|
Addict
Joined: Mar 2007
Posts: 522 |
Any idea why the Google spiders would spend an inordinate amount of time doing their thing in the calendar on my new site? I welcome any spider activity at this point, but I have yet to see Google anywhere but the calendar. Other spiders have searched the good stuff, but not Google. What's up with that?
I definitely need more spider activity...
Steve
UBB.classic from 2000-2003
UBB.threads from 2003-present!
|
|
|
|
|
Joined: Jun 2006
Posts: 16,532 Likes: 150
|
|
Joined: Jun 2006
Posts: 16,532 Likes: 150 |
It gets stuck, because the URLs never cease (think, it clicks next page into oblivion). Block it in your robots.txt with: User-agent: *
Crawl-Delay: 3
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
|
|
|
|
|
Joined: Mar 2007
Posts: 522
Addict
|
|
Addict
Joined: Mar 2007
Posts: 522 |
Sadly, it seems to be ignoring the robots.txt file.
Steve
UBB.classic from 2000-2003
UBB.threads from 2003-present!
|
|
|
|
|
Joined: Jun 2006
Posts: 16,532 Likes: 150
|
|
Joined: Jun 2006
Posts: 16,532 Likes: 150 |
It's not an immediate change, they check it every so often to eliminate mass hits on the file; give it a week and you should see things go back to where they should be...
|
|
|
|
|
Joined: Mar 2007
Posts: 522
Addict
|
|
Addict
Joined: Mar 2007
Posts: 522 |
Mr. Google is obeying now.
Steve
UBB.classic from 2000-2003
UBB.threads from 2003-present!
|
|
|
|
|
Joined: Mar 2007
Posts: 522
Addict
|
|
Addict
Joined: Mar 2007
Posts: 522 |
...and now he isn't. He has been stuck in the calendar all day.
Steve
UBB.classic from 2000-2003
UBB.threads from 2003-present!
|
|
|
|
1 members (Geoff),
494
guests, and
94
robots. |
|
Key:
Admin,
Global Mod,
Mod
|
|
|
|
|
|
