Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
0 registered members (), 72 guests, and 392 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
whk
whk
Portsmouth, RI
Posts: 86
Joined: January 2009
Show All Member Profiles 
Top Posters(30 Days)
Gizmo 17
M4TT 12
FREAK 8
Ruben 8
isaac 4
mmkk 4
Latest Photos
Chinese Buddhist temple.
My buddha beads.
Rendered Walls
Multi-Screen wallpaper
Stockholm Metro
Previous Thread
Next Thread
Print Thread
Spiders #260150
05/06/17 04:19 PM
05/06/17 04:19 PM
Joined: Mar 2017
Posts: 4
S
ShaneHenderson Offline OP
stranger
ShaneHenderson  Offline OP
stranger
S
Joined: Mar 2017
Posts: 4
Hi, I hope you can help. We have our forum as access via subscription only so members must log in before they can see the various forums within.

I can see Google Spiders looking at things like our calendar and generally public accessible areas. So my question is can Google Spiders crawl the whole site or are they limited to the public areas only. If so, will this have a negative effect on how the site scores with Google and again if so, is there a way to approve their spiders to see the whole site.

Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
Re: Spiders [Re: ShaneHenderson] #260151
05/06/17 04:52 PM
05/06/17 04:52 PM
Joined: Apr 2004
Posts: 1,004
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,004
California
Spiders can only see items that a non-logged-in user can see. So if you log out and then start browsing your site, thats what a spider will see.

If a page that is linked in another section of your forum (or on another website) is linked to a "restricted area," the item displayed to them is "Sorry, you are not logged in or you don't have permission for that." for forums, or "You do not have access to the [AREA NAME]" for sections such as Search, User List, Calendar, etc...They are offered a log in prompt, but for anything other than a stock forum feature, there is no hint given to them about what the page is about or what content is on the page. Except obviously, whatever subject name is in the URL address if you are using "Spider-friendly URLs."

You can test this for yourself. Log in to your website with your account. Open two browser windows and load the same "Restricted" page on both. Then log out in window-1, and press refresh in window-2. What you see in window-2, is exactly what the spider sees.

Last edited by isaac; 05/06/17 04:59 PM.
Re: Spiders [Re: ShaneHenderson] #260152
05/06/17 05:08 PM
05/06/17 05:08 PM
Joined: Apr 2004
Posts: 1,004
California
isaac Offline
UBB.threads Developer
isaac  Offline
UBB.threads Developer
Joined: Apr 2004
Posts: 1,004
California
You can also add pages to your robots.txt file. But not all spiders obey the rules, and they do whatever they want.

Code
User-agent: *
Disallow: /forums/admin/
Disallow: /forums/templates/
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php/ubb/calendar.html
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php/ubb/showday.html
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php/ubb/online.html
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php/ubb/search.html
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php/ubb/faq.html
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies.html
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php/ubb/markallread.html
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser
Disallow: /forums/ubbthreads.php/ubb/newuser.html
Disallow: /forums/ubbthreads.php?ubb=contact
Disallow: /forums/ubbthreads.php/ubb/contact
Disallow: /forums/ubbthreads.php/ubb/contact.html


http://www.robotstxt.org/robotstxt.html

The robots.txt has zero effect on users linking to items or manually typing entries in to their browser.

Re: Spiders [Re: ShaneHenderson] #260268
06/05/17 11:47 AM
06/05/17 11:47 AM
Joined: Mar 2017
Posts: 1
R
Richie01 Offline
stranger
Richie01  Offline
stranger
R
Joined: Mar 2017
Posts: 1
Hi Isaac
New to this Shane and myself are struggling a bit with this end of it
Any help on this would be great

Cheers Richie


Shout Box
Today's Birthdays
No Birthdays
Recent Topics
Users Unable to Upload Avatar [Not a Bug]
by M4TT. 12/13/17 08:51 AM
Shout Box Sound Effect
by M4TT. 11/29/17 08:28 PM
Ad island
by TGCsanderson. 11/25/17 06:41 PM
Taking to long to connect to DB
by AstroCat. 11/24/17 12:34 PM
Forum Statistics
Forums36
Topics35,015
Posts190,544
Members12,045
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.1
(Snapshot build 20171106)