Previous Thread
Next Thread
Print Thread
Hop To
#242384 03/30/2011 9:41 PM
Joined: Dec 2003
Posts: 16
C
stranger
stranger
C Offline
Joined: Dec 2003
Posts: 16
I have found numerous postings about seeing who the spiders are and how to display them...how do I stop them outright? I don't want google crawling my message boards, or any part of the system really. I have already set the forums to -1 for guests. I show a google spider in the calendar right now. Anyone have bugspray? a shoe?


If at First you do not succeed then Skydiving is not for you..
Joined: Jun 2006
Posts: 1,344
G
veteran
veteran
G Offline
Joined: Jun 2006
Posts: 1,344
You will need to do a robot.txt file, but not all spiders play nice

http://www.robotstxt.org/robotstxt.html

Last edited by gliderdad; 03/30/2011 9:44 PM.
Joined: Dec 2003
Posts: 16
C
stranger
stranger
C Offline
Joined: Dec 2003
Posts: 16
Excellent. I'll make that change and watch it for a bit.
Thanks for your assistance


If at First you do not succeed then Skydiving is not for you..
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
If perms are set to -1 for guests then it'll be for SE's as well as they're treated as guests. If they show in the access list it's them hitting the main page (and getting an error, as they're a guest).


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2008
Posts: 65
A
journeyman
journeyman
A Offline
Joined: Jun 2008
Posts: 65
I have made a private members forum within our large forum (which is open) and want to stop Google from indexing it. Can I use a robot.txt file for this as well?


Auckland, New Zealand

http://www2.everybody.co.nz/forum/
Gizmo #242435 04/04/2011 12:15 AM
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Originally Posted by Gizmo
If perms are set to -1 for guests then it'll be for SE's as well as they're treated as guests. If they show in the access list it's them hitting the main page (and getting an error, as they're a guest).
Again, if a guest can't access it, neither can a search engine...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2008
Posts: 65
A
journeyman
journeyman
A Offline
Joined: Jun 2008
Posts: 65
Thanks, sorry, got it now, have set guests to -1.


Auckland, New Zealand

http://www2.everybody.co.nz/forum/

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
1 members (Havenofsobriety), 458 guests, and 91 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)