Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
2 registered members (Gizmo, fuzzy47), 72 guests, and 215 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
TGCsanderson
TGCsanderson
Hackleton, U.K.
Posts: 78
Joined: September 2007
Show All Member Profiles 
Top Posters(30 Days)
Ruben 28
fuzzy47 26
isaac 24
Gizmo 22
Morgan 12
Latest Photos
Testing to drag photos
Comfortable Cats
Test
BSA photos
Chinese Buddhist temple.
Previous Thread
Next Thread
Print Thread
Spiders #260150
05/06/17 05:19 PM
05/06/17 05:19 PM
Joined: Mar 2017
Posts: 4
S
ShaneHenderson Offline OP
stranger
ShaneHenderson  Offline OP
stranger
S
Joined: Mar 2017
Posts: 4
Hi, I hope you can help. We have our forum as access via subscription only so members must log in before they can see the various forums within.

I can see Google Spiders looking at things like our calendar and generally public accessible areas. So my question is can Google Spiders crawl the whole site or are they limited to the public areas only. If so, will this have a negative effect on how the site scores with Google and again if so, is there a way to approve their spiders to see the whole site.

Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
Re: Spiders [Re: ShaneHenderson] #260151
05/06/17 05:52 PM
05/06/17 05:52 PM
Joined: Apr 2004
Posts: 1,146
California
isaac Online splat
UBB.threads Developer
isaac  Online Splat
UBB.threads Developer
Joined: Apr 2004
Posts: 1,146
California
Spiders can only see items that a non-logged-in user can see. So if you log out and then start browsing your site, thats what a spider will see.

If a page that is linked in another section of your forum (or on another website) is linked to a "restricted area," the item displayed to them is "Sorry, you are not logged in or you don't have permission for that." for forums, or "You do not have access to the [AREA NAME]" for sections such as Search, User List, Calendar, etc...They are offered a log in prompt, but for anything other than a stock forum feature, there is no hint given to them about what the page is about or what content is on the page. Except obviously, whatever subject name is in the URL address if you are using "Spider-friendly URLs."

You can test this for yourself. Log in to your website with your account. Open two browser windows and load the same "Restricted" page on both. Then log out in window-1, and press refresh in window-2. What you see in window-2, is exactly what the spider sees.

Last edited by isaac; 05/06/17 05:59 PM.
Re: Spiders [Re: ShaneHenderson] #260152
05/06/17 06:08 PM
05/06/17 06:08 PM
Joined: Apr 2004
Posts: 1,146
California
isaac Online splat
UBB.threads Developer
isaac  Online Splat
UBB.threads Developer
Joined: Apr 2004
Posts: 1,146
California
Add pages to your robots.txt file. Though, not all spiders obey the rules. Those rouge spiders do whatever they want.

Code
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/templates/
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php/ubb/calendar.html
Disallow: /forums/ubbthreads.php?ubb=printthread
Disallow: /forums/ubbthreads.php/ubb/printthread
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php/ubb/showday.html
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php/ubb/online.html
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php/ubb/search.html
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php/ubb/faq.html
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies.html
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php/ubb/markallread.html
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser
Disallow: /forums/ubbthreads.php/ubb/newuser.html
Disallow: /forums/ubbthreads.php?ubb=contact
Disallow: /forums/ubbthreads.php/ubb/contact
Disallow: /forums/ubbthreads.php/ubb/contact.html


The robots.txt has zero effect on users linking to items or manually typing entries in to their browser.

The robots.txt file should be placed in your domain folder:
https://www.example.com/robots.txt

Further reading at:
http://www.robotstxt.org/robotstxt.html


Shout Box
Today's Birthdays
No Birthdays
Recent Topics
Games
by fuzzy47. 04/24/18 09:05 PM
No share/like buttons
by fuzzy47. 04/24/18 08:53 PM
Backgroud colors in posts
by fuzzy47. 04/16/18 08:27 PM
Avatar Upload Issue
by fuzzy47. 04/15/18 11:58 AM
Forum database - Table prefix question
by Morgan. 04/15/18 06:45 AM
Forum Statistics
Forums36
Topics35,106
Posts191,186
Members12,068
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.2
(Preview build 20180410)