You are not logged in. [Log In] UBB.threads PHP Forum Software Community » Forums » Managing & Customizing your Board » Admin Zone » How to STOP Google from Looking at the calendar 100 times a day
Register User    Portal Page     Forum List        Calendar     Active Topics     Search     FAQ
Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online
2 registered (Gizmo, SteveS), 38 Guests and 14 Spiders online.
Key: Admin, Global Mod, Mod
Featured Member
Registered: 06/07/07
Posts: 4
Top Posters (30 Days)
Ruben 50
DennyP 24
Gizmo 23
Dunny 15
SteveS 13
AllenAyres 12
dbremer 10
SD 10
drkknght00 9
doug 8
Latest Photos
OK Corral Shoot Out
Testing
Basildon Train Station
Basildon Town Centre looking from the rounderbout
Basildon Town Square
Page 1 of 2 1 2 >
Topic Options
#246404 - 10/20/11 06:19 PM How to STOP Google from Looking at the calendar 100 times a day
teamzr1 Offline
enthusiast 		Registered: 06/19/07
Posts: 249
Version 7.5.6
How can I stop the Ahole Google from sucking up CPU in Looking at the calendar hundreds of times a day when that function is never really used so it is blank in content ?

Thanks
_________________________
JR
Team ZR-1 Corvette Racer's
Top
Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
#246406 - 10/20/11 09:18 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
KuKuKaChu Offline
member 		Registered: 12/24/05
Posts: 122
Loc: Jakarta, Indonesia
put something like this in your robots.txt file:

Code:
User-agent: *
Allow: /
Sitemap: http://jakchat.com/sitemap.xml
Disallow: /files/texts
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php?ubb=showprofile
Disallow: /forums/ubbthreads.php/ubb/showprofile
Disallow: /forums/ubbthreads.php?ubb=showmembers
Disallow: /forums/ubbthreads.php/ubb/showmembers
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser


this will stop search engines from indexing unnecessary pages.
_________________________
JakChat.com -- Forums for Indonesia's English-speaking community
Ubuntu-Indonesia.com -- Forums for Indonesia's Ubuntu Users
Top
#246407 - 10/20/11 09:30 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
teamzr1 Offline
enthusiast 		Registered: 06/19/07
Posts: 249
I have those disallows in the robots.txt for some months and has not stopped this
It is in the root of the domain.
Does this txt file have to be located somewhere else ?
_________________________
JR
Team ZR-1 Corvette Racer's
Top
#246409 - 10/20/11 09:59 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
Gizmo Online cat

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
The version i share actually throttles google down with "Crawl-Delay":

Code:
User-agent: *
Crawl-Delay: 3
Disallow: /files/texts
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#246414 - 10/21/11 05:21 AM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
KuKuKaChu Offline
member 		Registered: 12/24/05
Posts: 122
Loc: Jakarta, Indonesia
Originally Posted By: teamzr1
I have those disallows in the robots.txt for some months and has not stopped this
It is in the root of the domain.
Does this txt file have to be located somewhere else ?

yes, it must be located in the web root directory. to test if it's in the right place, open it in a web browser like this: http://mydomain.com/robots.txt . if it doesn't appear, then it's in the wrong place.
_________________________
JakChat.com -- Forums for Indonesia's English-speaking community
Ubuntu-Indonesia.com -- Forums for Indonesia's Ubuntu Users
Top
#246416 - 10/21/11 12:43 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
teamzr1 Offline
enthusiast 		Registered: 06/19/07
Posts: 249
I was able to stop Google flooding my forum 24/7 by added this

User-Agent: Googlebot
Disallow: /

As the examples above on it's own would not
_________________________
JR
Team ZR-1 Corvette Racer's
Top
#246418 - 10/21/11 02:18 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
Gizmo Online cat

Registered: 06/05/06
Posts: 14995
Loc: Portland, OR; USA
Well, that would block Google entirely; a lot of us live off of our Search Engine traffic...
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#246419 - 10/21/11 02:47 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
teamzr1 Offline
enthusiast 		Registered: 06/19/07
Posts: 249
Well for whatever reason when the robots.txt content was as what others shown above that google bot was ignoring those disallows and constantly flooding the server my forum is on and when there is couple hundred of domains on a web server the forum functioned very slowly

I think attracta.com and Google Crawler have something to do with my issues.
_________________________
JR
Team ZR-1 Corvette Racer's
Top
#246437 - 10/22/11 11:56 AM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
teamzr1 Offline
enthusiast 		Registered: 06/19/07
Posts: 249
What is wrong with the content of this robots.txt content as if I take out the disallow for Google bot then it again ignores the disallows for like events, calendar,etc ?

User-agent: *
Disallow: /cgi-bin/
User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Disallow:/

User-agent: Slurp
Crawl-delay: 5

User-agent: dotbot
Disallow: /

User-agent: *
Crawl-Delay: 120
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser

Using Google Webmaster tools website there is a throttle adjustment for crawler but I find it does not do much and if you do reduce the requests and increase a delay if you do not renew that function after 90 days then they go right back to flooding your domain with crawler requests sick

Also via this tools webpage and reviewing the robots.txt they say they ignore the Crawl-Delay: and it's value
_________________________
JR
Team ZR-1 Corvette Racer's
Top
#246439 - 10/22/11 02:48 PM Re: How to STOP Google from Looking at the calendar 100 times a day [Re: teamzr1]
Ruben Offline

Registered: 12/20/03
Posts: 4424
Loc: Lutz,FL
Originally Posted By: teamzr1
Version 7.5.6
How can I stop the Ahole Google from sucking up CPU in Looking at the calendar hundreds of times a day when that function is never really used so it is blank in content ?

Thanks

Then turn off the calendar feature.
You can start with,
Control Panel ยป Feature Settings
Un-select enable the calendar.
I think you are getting to convoluted in your robot file.
Besides many crawlers do not honor them anyway.
And I can't figure out why the exclude for cgi since threads does not use that folder anyway.
Also if you disable the calendar google will still look for the url for some time. You might want to update your google settings and exclude the url.
_________________________
Blue Man Group
Top
Page 1 of 2 1 2 >



Shout Box

Today's Birthdays
No Birthdays
Recent Topics
Temporary Password email not being received
by
05/24/12 10:02 PM
Ability to "like" individual posts (not Facebook "likes)
by doug
05/23/12 09:03 AM
Island Permissions
by ThreadsUser
05/22/12 03:03 PM
streaming video
by prkrgrp
05/20/12 07:02 PM
New Posts Corrupted? Can someone help?
by PianoWorld
05/19/12 09:41 AM
Forum Stats
10492 Members
36 Forums
33842 Topics
181709 Posts

Max Online: 978 @ 06/24/07 11:19 PM
Random Image
Board Rules · Mark all read
Contact Us · UBBCentral · Top