|
Joined: Dec 2003
Posts: 16
stranger
|
stranger
Joined: Dec 2003
Posts: 16 |
I have found numerous postings about seeing who the spiders are and how to display them...how do I stop them outright? I don't want google crawling my message boards, or any part of the system really. I have already set the forums to -1 for guests. I show a google spider in the calendar right now. Anyone have bugspray? a shoe?
If at First you do not succeed then Skydiving is not for you..
|
|
|
|
Joined: Jun 2006
Posts: 1,344
veteran
|
veteran
Joined: Jun 2006
Posts: 1,344 |
You will need to do a robot.txt file, but not all spiders play nice http://www.robotstxt.org/robotstxt.html
Last edited by gliderdad; 03/30/2011 9:44 PM.
|
|
|
|
Joined: Dec 2003
Posts: 16
stranger
|
stranger
Joined: Dec 2003
Posts: 16 |
Excellent. I'll make that change and watch it for a bit. Thanks for your assistance
If at First you do not succeed then Skydiving is not for you..
|
|
|
|
Joined: Jun 2006
Posts: 16,301 Likes: 116
|
Joined: Jun 2006
Posts: 16,301 Likes: 116 |
If perms are set to -1 for guests then it'll be for SE's as well as they're treated as guests. If they show in the access list it's them hitting the main page (and getting an error, as they're a guest).
|
|
|
|
Joined: Jun 2008
Posts: 65
journeyman
|
journeyman
Joined: Jun 2008
Posts: 65 |
I have made a private members forum within our large forum (which is open) and want to stop Google from indexing it. Can I use a robot.txt file for this as well?
|
|
|
|
Joined: Jun 2006
Posts: 16,301 Likes: 116
|
Joined: Jun 2006
Posts: 16,301 Likes: 116 |
If perms are set to -1 for guests then it'll be for SE's as well as they're treated as guests. If they show in the access list it's them hitting the main page (and getting an error, as they're a guest). Again, if a guest can't access it, neither can a search engine...
|
|
|
|
Joined: Jun 2008
Posts: 65
journeyman
|
journeyman
Joined: Jun 2008
Posts: 65 |
Thanks, sorry, got it now, have set guests to -1.
|
|
|
Test
by Phun - 05/28/2024 7:31 PM
|
|
|
|
|
|