Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
1 registered members (JAISP), 52 guests, and 218 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
Morgan
Morgan
Sweden
Posts: 591
Joined: June 2006
Show All Member Profiles 
Top Posters(30 Days)
Gizmo 11
Ruben 5
M4TT 4
Morgan 3
isaac 1
Latest Photos
Previous Thread
Next Thread
Print Thread
Spiders #260150
05/06/17 05:19 PM
05/06/17 05:19 PM
Joined: Mar 2017
Posts: 4
S
ShaneHenderson Offline OP
stranger
ShaneHenderson  Offline OP
stranger
S
Joined: Mar 2017
Posts: 4
Hi, I hope you can help. We have our forum as access via subscription only so members must log in before they can see the various forums within.

I can see Google Spiders looking at things like our calendar and generally public accessible areas. So my question is can Google Spiders crawl the whole site or are they limited to the public areas only. If so, will this have a negative effect on how the site scores with Google and again if so, is there a way to approve their spiders to see the whole site.

Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
Re: Spiders [Re: ShaneHenderson] #260151
05/06/17 05:52 PM
05/06/17 05:52 PM
Joined: Apr 2004
Posts: 1,172
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,172
California
Spiders can only see items that a non-logged-in user can see. So if you log out and then start browsing your site, thats what a spider will see.

If a page that is linked in another section of your forum (or on another website) is linked to a "restricted area," the item displayed to them is "Sorry, you are not logged in or you don't have permission for that." for forums, or "You do not have access to the [AREA NAME]" for sections such as Search, User List, Calendar, etc...They are offered a log in prompt, but for anything other than a stock forum feature, there is no hint given to them about what the page is about or what content is on the page. Except obviously, whatever subject name is in the URL address if you are using "Spider-friendly URLs."

You can test this for yourself. Log in to your website with your account. Open two browser windows and load the same "Restricted" page on both. Then log out in window-1, and press refresh in window-2. What you see in window-2, is exactly what the spider sees.

Last edited by isaac; 05/06/17 05:59 PM.

isaac @ id242.com // forum @ CelicaHobby.com
a current developer of UBB.threads php forum software // 7.6.2 Changelog
Re: Spiders [Re: ShaneHenderson] #260152
05/06/17 06:08 PM
05/06/17 06:08 PM
Joined: Apr 2004
Posts: 1,172
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,172
California
Add pages to your robots.txt file. Though, not all spiders obey the rules. Those rouge spiders do whatever they want.

Code
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/templates/
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php/ubb/calendar.html
Disallow: /forums/ubbthreads.php?ubb=printthread
Disallow: /forums/ubbthreads.php/ubb/printthread
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php/ubb/showday.html
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php/ubb/online.html
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php/ubb/search.html
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php/ubb/faq.html
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies.html
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php/ubb/markallread.html
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser
Disallow: /forums/ubbthreads.php/ubb/newuser.html
Disallow: /forums/ubbthreads.php?ubb=contact
Disallow: /forums/ubbthreads.php/ubb/contact
Disallow: /forums/ubbthreads.php/ubb/contact.html


The robots.txt has zero effect on users linking to items or manually typing entries in to their browser.

The robots.txt file should be placed in your domain folder:
https://www.example.com/robots.txt

Further reading at:
http://www.robotstxt.org/robotstxt.html


isaac @ id242.com // forum @ CelicaHobby.com
a current developer of UBB.threads php forum software // 7.6.2 Changelog

Shout Box
Today's Birthdays
No Birthdays
Recent Topics
Member being directed to "stopforumspam?"
by Lisanne. 07/14/18 10:27 AM
7.6.1.1 - Broken ReCaptcha (Not A Bug)
by M4TT. 07/01/18 06:02 PM
Control Panel / Transactions Username
by Morgan. 06/20/18 05:38 AM
Custom Insert won't save
by Baldeagle. 06/18/18 07:44 PM
random 500 server errors on post
by Bad Frog. 05/31/18 09:34 AM
Forum Statistics
Forums36
Topics35,125
Posts191,350
Members12,091
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.2
(Preview build 20180524)