|
|
Joined: Dec 2006
Posts: 1,235
veteran
|
|
veteran
Joined: Dec 2006
Posts: 1,235 |
I've noticed that spiders are now crawling pages on my forum in the list below now that the .html has been enabled. With regards to Search Engine's seem to get "suck" on my site... does the following need to be updated and if so would I be correct in thinking that it would be an addition to every 3rd line with .html added? Disallow: /forum/ubbthreads.php?ubb=calendar Disallow: /forum/ubbthreads.php/ubb/calendar Disallow: /forum/ubbthreads.php/ubb/calendar.htmlUser-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
|
|
|
|
|
Joined: Jul 2006
Posts: 4,057
|
|
Joined: Jul 2006
Posts: 4,057 |
< Thinks > What's the official line on turning on the .html Is it better or worse, or what's the pro's cons. WE see it but what's its purpose. Sorry not hijacking but think its relevant to your post too 
BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
|
|
|
|
|
Joined: Aug 2006
Posts: 1,649 Likes: 1
Pooh-Bah
|
|
Pooh-Bah
Joined: Aug 2006
Posts: 1,649 Likes: 1 |
Search engines assume content is more important and relevant when the title reflects its keywords. They also find it easier to follow ".html" pages than "dynamic" content.
As for the original question, I would assume that it wouldn't hurt. But whether that's the best way to do it, I'm really not sure.
GangsterBB.NET (Ver. 7.6.1.1) PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54 2007 Content Rulez Contest - Hon Mention UBB.classic 6.7.2 - RIP
|
|
|
|
|
Joined: Dec 2006
Posts: 1,235
veteran
|
|
veteran
Joined: Dec 2006
Posts: 1,235 |
The other good thing is that the page URLs now end in part or all of the subject, not just a number.
Old: Number/206879
New: Number/206879/Update_robots_txt_for_new_7_3_.html
Makes it much easier to see what your visitors are looking at from your logs and search engine pages.
|
|
|
|
|
Joined: Aug 2006
Posts: 1,649 Likes: 1
Pooh-Bah
|
|
Pooh-Bah
Joined: Aug 2006
Posts: 1,649 Likes: 1 |
Is there much server overhead enabling that?
And do the old links still work as well?
GangsterBB.NET (Ver. 7.6.1.1) PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54 2007 Content Rulez Contest - Hon Mention UBB.classic 6.7.2 - RIP
|
|
|
|
|
Joined: Dec 2006
Posts: 1,235
veteran
|
|
veteran
Joined: Dec 2006
Posts: 1,235 |
Is there much server overhead enabling that? I don't know. I'm not techy enough to know how to find that info out.  And do the old links still work as well? Yes but I don't know if it's backwards compatible. i.e. If the site is crawled with .html extensions and you disable them, what happens to hits from search engines displaying your .html links?
|
|
|
|
|
Joined: Aug 2006
Posts: 1,649 Likes: 1
Pooh-Bah
|
|
Pooh-Bah
Joined: Aug 2006
Posts: 1,649 Likes: 1 |
Good point, thanks. I better ask my web host if the already-laggy shared server I'm on can handle that... 
GangsterBB.NET (Ver. 7.6.1.1) PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54 2007 Content Rulez Contest - Hon Mention UBB.classic 6.7.2 - RIP
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
Yes, needs updated; could you find me the old thread link and i'll update there when i get a bit of time tonight.
|
|
|
|
|
Joined: Aug 2006
Posts: 1,358
Veteran
|
|
Veteran
Joined: Aug 2006
Posts: 1,358 |
Scriptkeeper already included a link the original topic in the FAQ forum.
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
there, found, updated, and found a bug in the process ... lol...
|
|
|
|
|
Joined: Aug 2006
Posts: 1,649 Likes: 1
Pooh-Bah
|
|
Pooh-Bah
Joined: Aug 2006
Posts: 1,649 Likes: 1 |
I suppose there's no way to exclude SE's from specific forums? I mean, I could add this: Disallow: /threads/ubbthreads.php?ubb=postlist&Board=14 But that won't stop them from getting there via CFRM's "Last Post" links like /threads/ubbthreads.php?ubb=showflat&Number=480326&Main=13885#Post480326 (Forum #14 is devoted to silly postcount games that I despise, and don't think SE's should spend the time crawling them )
GangsterBB.NET (Ver. 7.6.1.1) PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54 2007 Content Rulez Contest - Hon Mention UBB.classic 6.7.2 - RIP
|
|
|
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116
|
|
Joined: Jun 2006
Posts: 16,299 Likes: 116 |
You are correct, so many unique urls...
I'd leave it though, sure it may waste a little smidge of BW, but who knows, some random user may still stumble in for it...
|
|
|
|
|
Joined: Dec 2006
Posts: 1,235
veteran
|
|
veteran
Joined: Dec 2006
Posts: 1,235 |
I noticed Yahoo! checking the Who's Online page and found an anomaly in the URLs. Some links are: ...threads.php/ubb/page.htmland others miss out the ubb directory: ...threads.php/page.htmlYou'll need to update your list again and change the following: - Disallow: /forum/ubbthreads.php/ubb/online.html - Remove ubb/
- Disallow: /forum/ubbthreads.php/ubb/showprofile.html - I think this should now be ubbthreads.php/users
- Disallow: /forum/ubbthreads.php/ubb/search.html - Remove ubb/
|
|
|
Bots
by Outdoorking - 04/13/2024 5:08 PM
|
|
|
|
|
|
|
|
|
|
|
1 members (Nightcrawler),
1,165
guests, and
234
robots. |
|
Key:
Admin,
Global Mod,
Mod
|
|
|
|
![[Linked Image from siemons.org]](http://www.siemons.org/foto/yarp.gif)
