Previous Thread
Next Thread
Print Thread
Hop To
#260150 05/06/2017 5:19 PM
Joined: Mar 2017
Posts: 4
S
stranger
stranger
S Offline
Joined: Mar 2017
Posts: 4
Hi, I hope you can help. We have our forum as access via subscription only so members must log in before they can see the various forums within.

I can see Google Spiders looking at things like our calendar and generally public accessible areas. So my question is can Google Spiders crawl the whole site or are they limited to the public areas only. If so, will this have a negative effect on how the site scores with Google and again if so, is there a way to approve their spiders to see the whole site.

ShaneHenderson #260151 05/06/2017 5:52 PM
Joined: Apr 2004
Posts: 1,945
Likes: 145
UBB.threads Developer
UBB.threads Developer
Joined: Apr 2004
Posts: 1,945
Likes: 145
Spiders can only see items that a non-logged-in user can see. So if you log out and then start browsing your site, thats what a spider will see.

If a page that is linked in another section of your forum (or on another website) is linked to a "restricted area," the item displayed to them is "Sorry, you are not logged in or you don't have permission for that." for forums, or "You do not have access to the [AREA NAME]" for sections such as Search, User List, Calendar, etc...They are offered a log in prompt, but for anything other than a stock forum feature, there is no hint given to them about what the page is about or what content is on the page. Except obviously, whatever subject name is in the URL address if you are using "Spider-friendly URLs."

You can test this for yourself. Log in to your website with your account. Open two browser windows and load the same "Restricted" page on both. Then log out in window-1, and press refresh in window-2. What you see in window-2, is exactly what the spider sees.

Last edited by isaac; 05/06/2017 5:59 PM.

Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com
ShaneHenderson #260152 05/06/2017 6:08 PM
Joined: Apr 2004
Posts: 1,945
Likes: 145
UBB.threads Developer
UBB.threads Developer
Joined: Apr 2004
Posts: 1,945
Likes: 145
Add pages to your robots.txt file. Though, not all spiders obey the rules. Those rouge spiders do whatever they want.

Code
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/templates/
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php/ubb/calendar.html
Disallow: /forums/ubbthreads.php?ubb=printthread
Disallow: /forums/ubbthreads.php/ubb/printthread
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php/ubb/showday.html
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php/ubb/online.html
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php/ubb/search.html
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php/ubb/faq.html
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies.html
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php/ubb/markallread.html
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser
Disallow: /forums/ubbthreads.php/ubb/newuser.html
Disallow: /forums/ubbthreads.php?ubb=contact
Disallow: /forums/ubbthreads.php/ubb/contact
Disallow: /forums/ubbthreads.php/ubb/contact.html

The robots.txt has zero effect on users linking to items or manually typing entries in to their browser.

The robots.txt file should be placed in your domain folder:
https://www.example.com/robots.txt

Further reading at:
http://www.robotstxt.org/robotstxt.html


Current developer of UBB.threads PHP Forum Software
Current Release: UBBT 7.7.5 // Preview: UBBT 8.0.0
isaac @ id242.com // my forum @ CelicaHobby.com

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
1 members (Ruben), 476 guests, and 111 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)