Previous Thread
Next Thread
Print Thread
Hop To
Page 2 of 4 1 2 3 4
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Basil, you definitely have something interesting going on. You still have all of the old links in google from your 6.5 forum. You have a bunch of links to the new forum as far as to the general forum listing and the individual postlist pages. But you don't appear to have many new links to the new showflat script.

What we usually see is the old links eventually go away and google reindexes your new ones. For some reason, they are keeping all of your old links in the system. You have thousands and thousands pointing to the old showflat script but barely any to the new.

For example, do this:

site:britishcarforum.com +ubb/postlist

You'll see it has a bunch of links to individual postlist pages. But if you do this:

site:britishcarforum.com +ubb/showflat

You'll see that there aren't many listed at all. Your site appears to be the only one with this problem. Everyone else that has posted has many many links to the new showflat. I can search for distinct subjects from their site and they are listed in Google. Yours I cannot however. I'm not sure if your site is being penalized for something, or what. I know you had made the change to/from search engine friendly not too long ago, so not sure if it's related.

Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Originally Posted by Rick
Basil, you definitely have something interesting going on.
Interesting and troubling. I see a single Google bot all the time and it "says" it is reading posts, but they don't seem to actually be indexing. Well, now we know ther eis "something" going on, now the trick will be to figure out exactly what!

Originally Posted by Rick
You still have all of the old links in google from your 6.5 forum. You have a bunch of links to the new forum as far as to the general forum listing and the individual postlist pages. But you don't appear to have many new links to the new showflat script. What we usually see is the old links eventually go away and google reindexes your new ones. For some reason, they are keeping all of your old links in the system.

Right, that's what I've been saying. It is indexing "part" of my site, but not the main parts. I am wondering if having the re-director to the old threads could be part of the problem? Should I get rid of that at some point on the theory that Google will then evantually drop the old links and re-index the new? Just a thought.

Originally Posted by Rick
For example, do this:

site:britishcarforum.com +ubb/postlist

You'll see it has a bunch of links to individual postlist pages. But if you do this:

site:britishcarforum.com +ubb/showflat

You'll see that there aren't many listed at all.

Even using the site:britishcarforum.com +ubb/postlist there were only 153 that cam eup for me (only 20 the other search for showflat). Clearly I have a problem.

Originally Posted by Rick
Your site appears to be the only one with this problem. Everyone else that has posted has many many links to the new showflat.

Lucky me. When I do YOUR site with the showflat seach above, I get just over 12,000 results (but you have over 26,000 topics!!) But that is clearly better than the 20 results that I get for my site.

Originally Posted by Rick
I can search for distinct subjects from their site and they are listed in Google. Yours I cannot however. I'm not sure if your site is being penalized for something, or what. I know you had made the change to/from search engine friendly not too long ago, so not sure if it's related.

That change was only temporary and only because I was desperate for Google to index my site. Ok, so not all are having this issue, so now I have to figure out what is different from my site than everyone elses. One thing that is also different for me besides the new Threads 7, is that I moved to a dedicated server and am using a server management service to monitor things, install security, etc. Now I am wondering is maybe something about my server set up could be preventing google from crawling my site? For example, when I recently installed a wiki on my site, I found that I could not save any edited pages - I would get "Forbidden You don't have access..." Well, I finally figured out that if I disabled ssomething called mod_secure - an Apachie security thingy, then my wiki worked right. Now I only disabled it for the wiki directory, not my entire root directory, so I'm wondering if maybe something like that might be at issue with Google? I know I'm grasping at straws, but I don't have anything else to grasp at right now.

Or could there be a difference in my PhP setup?

Basil

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
It's hard to say for sure what's keeping them from indexing the newer content. On your redirect scripts did you have a 301 or a 302 for the redirect? It shouldn't matter because we have people using both without a problem, but figured I'd ask.

If you use the google webmaster tools and you look at the Crawl Errors is it giving anything specific?

Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
It says its temporary. Should I change to perm?

I have about 2600+ 404 Page not found errors and almost all of them are llokoing for things in my old 6.5 install directory.

Also, while I try to figure this out I am trying a program I happened to have (I forgot about it) called GSiteCrawl which is a free site map generator that will generate a site map for the entire site for both Goggle, Yahoo and others. This will be the first time I've tried it, so I'm hoping it will help.

Basil

Last edited by Basil; 03/20/2007 5:09 PM.
Joined: Jun 2006
Posts: 136
O
member
member
O Offline
Joined: Jun 2006
Posts: 136
Originally Posted by Ian
I do not pretend to know google - I am not saying it is the fault of threads at all.

My only concern is that I get as many pages as possible listed on google and compete with other sites.

My apologies if I added to any hysteria - not my intention smile

ditto for me too. as far as i can see, there is no major issue with the product in relation to google.

however, the the fact that some people -- rightly or wrongly -- perceive that they have a problem should be taken seriously and talked about to see what if anything can be done about it.

getting content listed in search engines is critically important to most forums. many forums provide invaluable information on a huge variety of different subjects -- often using language and example understandable to non-experts -- and as such deserve their place in the search engines.

thus we should not be shy about helping the search engines find what they need.



Okusi Associates -- Indonesian business and management services
Joined: Aug 2006
Posts: 111
member
member
Joined: Aug 2006
Posts: 111
This morning I was looking at my "Who's Online" and noticed a Google spider.

I figured out something, maybe my "thoughts" on this will help out.

First, some background on my site, it is almost completely private. The only 2 forums viewable to the public are the rules and signup instructions and both of them are locked down so all you can do is read them, not reply or anything.

So, how long does it take for Google to scan your site? I mean, with out any posts there it should go quick, right? Wrong.

Google was indexing our site for about 1.5 hours. What was it doing? THE CALENDAR

This is perfectly fine by us, nothing is in the calendar that can't be public and helps us with Google rankings, so cool.

How this might help others and what I am guessing at.

Someone mentioned above that Google is no longer using multiple agents to fish/spider a site.

I can verify this by confirming that only one spider was on the site the entire time.

Here is where it might be of importance. When do they split off and spider other things? Do they ever make it past the calendar? In other words, if you have a large calendar, will the spider get confused and "stuck" in it with links back to pages it just visited? (ie Previous or next months OR going to multiple day listings with links to the other days).

The calendar is a major change (I love it BTW) from the previous version and maybe the Google spider is getting "stuck" in it. I know it was in our calendar for 1.5 hours and we hardly have any events in there beyond birthdays for 800 or so users.

Again, not that this is UBB's fault or anything, just an observation of what MAY be happening.

Also, we do not have the Google/spider friendly URLS turned on, but Google was in there for 1.5 hours.

Fred

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
A little description of what google does and several suggestions...

When google "hits" your site, it grabs new links and grabs content from the pages it was sent to do so; basically it does crawling and discovery at once.

Adding a SiteMap aids highly in discovery, since you're telling google what is there, and it knows what links it needs to crawl by your sitemap (it still does its own discovery, but it's hit and grab vs your feeding it valid links).

You can block search engines from crawling your calendar with a robots.txt entry:
robots.txt:
Code
User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday

As for "why you'd need a sitemap", I can only recommend you read the FAQ at google here (or read about my UBB.Sitemap generator here) as nothing i say can say it better.

Also, by using a sitemap, your links will be crawled regardless of SE links being on, as the links to your threads that are fed to the search engines (most major SE's now support sitemaps for note) will not "break" links, as you won't have to worry about most of the discovery (as you're telling them what links are there and are valid)

As for Ian saying he wants a sitemap for both his site and his forums, the UBB Sitemap generator grabs feeds live from your forum, and you can submit numerous sitemaps to google, so you could use alternate "link building" sitemap generators for miscellaneous site data.



I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 136
O
member
member
O Offline
Joined: Jun 2006
Posts: 136
that's useful to know. i've already added that to my robots.txt.

to prevent search engines from indexing user profiles, would this entry be correct and/or sufficient?

Code
Disallow: /forums/ubbthreads.php?ubb=showprofile
Disallow: /forums/ubbthreads.php/ubb/showmembers





Okusi Associates -- Indonesian business and management services
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Theres actually a setting in the cp to disallow guests viewing profiles; that should be a sufficient way of doing that in general.

But, if you dont' want that on, then:
Code
Disallow: /forums/ubbthreads.php?ubb=showprofile
Disallow: /forums/ubbthreads.php/ubb/showprofile

If you didn't want them spidering the userlist, then this as well:
Code
Disallow: /forums/ubbthreads.php?ubb=showmembers
Disallow: /forums/ubbthreads.php/ubb/showmembers


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 136
O
member
member
O Offline
Joined: Jun 2006
Posts: 136
this is the robots.txt i am now using to (hopefully) stop search engines from wasting their time and my bandwidth:

Code
User-agent: *
Disallow: /forums/ubbthreads.php?ubb=calendar
Disallow: /forums/ubbthreads.php/ubb/calendar
Disallow: /forums/ubbthreads.php?ubb=showday
Disallow: /forums/ubbthreads.php/ubb/showday
Disallow: /forums/ubbthreads.php?ubb=showprofile
Disallow: /forums/ubbthreads.php/ubb/showprofile
Disallow: /forums/ubbthreads.php?ubb=showmembers
Disallow: /forums/ubbthreads.php/ubb/showmembers
Disallow: /forums/ubbthreads.php?ubb=online
Disallow: /forums/ubbthreads.php/ubb/online
Disallow: /forums/ubbthreads.php?ubb=search
Disallow: /forums/ubbthreads.php/ubb/search
Disallow: /forums/ubbthreads.php?ubb=faq
Disallow: /forums/ubbthreads.php/ubb/faq
Disallow: /forums/ubbthreads.php?ubb=viewprivacy
Disallow: /forums/ubbthreads.php/ubb/viewprivacy
Disallow: /forums/ubbthreads.php?ubb=mycookies
Disallow: /forums/ubbthreads.php/ubb/mycookies
Disallow: /forums/ubbthreads.php?ubb=markallread
Disallow: /forums/ubbthreads.php/ubb/markallread
Disallow: /forums/ubbthreads.php?ubb=newuser
Disallow: /forums/ubbthreads.php/ubb/newuser


Okusi Associates -- Indonesian business and management services
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Looks interesting enough, though several of those will just error out from a guest viewing them anyway (mark all read, my cookies, online, showprofile should you have guest viewing off)

It's amazing how often something so easy as robots.txt crafting goes un-noticed to people, yet it's such a powerful tool and easy to work with and requiring minimal reading to understand.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 136
O
member
member
O Offline
Joined: Jun 2006
Posts: 136
installing that robots.txt certainly made an immediate difference with the search engines.

besides reducing the total number of active spiders (esp. yahoo), the spiders left are now all engaged in indexing forums, rather than peripheral and unimportant sections of my site.



Okusi Associates -- Indonesian business and management services
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Told you that i'm of some use wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
You are always of use wink

It is always better to ban spiders from going to places where guests are banned, as they will still spider and file these pages - which means you will have several 'you do not have permission to view this page' pages within the search engines smile

Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
I also have...


Disallow: /tmp
Disallow: /logs
Disallow: /forum/
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /generic/
Disallow: /photos/

which includes my old forums in case any spiders are still lurking there smile

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
my old forums just have a redirector; any requests get zoomed to threads wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
so do mine - but google was still in there LOL

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
yeh, getting bounced to the new locations; keep in mind mr google has to see that old links don't exist to update their databases to your new links wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
I'm all up for giving it a go.

Using Gizmo's example :

User-agent: *
Disallow; /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday


Note line 2 has a ; and should have been a :
This can cause google to error on the syntax


Right lets see who's looking.....

Yahoo = 28 Viewing list of forums
Yahoo Reading = 32 Reading a post in flat mode
MSN = 5 Reading a post in flat mode
Google = 1 Reading a post in X

And that's a normal afternoon for me,
so i will update you next week if the robot.txt has worked
for me ??

I cant any sooner as i'm away frown

Last edited by Mark S; 03/22/2007 2:36 PM. Reason: code error highlighted

BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
Just checked these forums "Whos on Line"
and currently 209 anonymous.

But only 2 or 3 max from Yahoo "Viewing list of forums"
Where as mine is almost 50-50 reading and viewing.


Its an observation thats all,
as i can use as a possible reference upon my return.
As Yahoo is reading 99% for this forum.
And mine at the moment before robot.txt is 50%

Maybe its normal to be 50-50 but if no one say's
no one would know??


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
Originally Posted by Gizmo
yeh, getting bounced to the new locations; keep in mind mr google has to see that old links don't exist to update their databases to your new links wink


But then if you delete the old forums they would not exist anyway.

Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
Originally Posted by Mark S
I'm all up for giving it a go.

Using Gizmo's example :

Right lets see who's looking.....

Yahoo = 28 Viewing list of forums
Yahoo Reading = 32 Reading a post in flat mode
MSN = 5 Reading a post in flat mode
Google = 1 Reading a post in X

And that's a normal afternoon for me,
so i will update you next week if the robot.txt has worked
for me ??

I cant any sooner as i'm away frown


Just FYI - I have had this for a while - at the moment I have 1 google (nothing recorded as to where they are looking), 1 MSN (nothing recorded as to where they are looking), and 168 from Yahoo all eithin the last 30 minutes.

Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
what's yahoo doing in the last 30 mins ian?


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
183 (at the moment) of the blighters all over the forum LOL

Always only 1 MSN & 1 Google

Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
cheers smile Ian


User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday

Note line 2 had a ; and should have been a :
This can cause google to error on the syntax


Check your code if you have used the earlier examples.
You can test what you have at google tools.


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
I believe that some bots check every hit to a new page whereas some cache it for a day or so.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
I updated the earlier example, the reason it had a ; was because had to manually enter the one line cas i deleted some other rules and deleted a smidge too much lol


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 136
O
member
member
O Offline
Joined: Jun 2006
Posts: 136
Originally Posted by Mark S
User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday

should that be forum, or forums??


Okusi Associates -- Indonesian business and management services
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
It should be the directory you installed threads to; so it could be either, or neither...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Originally Posted by Rick
Basil, you definitely have something interesting going on. You still have all of the old links in google from your 6.5 forum. You have a bunch of links to the new forum as far as to the general forum listing and the individual postlist pages. But you don't appear to have many new links to the new showflat script.

What we usually see is the old links eventually go away and google reindexes your new ones. For some reason, they are keeping all of your old links in the system. You have thousands and thousands pointing to the old showflat script but barely any to the new.

For example, do this:

site:britishcarforum.com +ubb/postlist

You'll see it has a bunch of links to individual postlist pages. But if you do this:

site:britishcarforum.com +ubb/showflat

You'll see that there aren't many listed at all. Your site appears to be the only one with this problem. Everyone else that has posted has many many links to the new showflat. I can search for distinct subjects from their site and they are listed in Google. Yours I cannot however. I'm not sure if your site is being penalized for something, or what. I know you had made the change to/from search engine friendly not too long ago, so not sure if it's related.

Well, now that some time has passed, and I have included a site map, I tried this again: site:britishcarforum.com +ubb/showflat
and I STILL only have 5 threads show up and none of them - NONE -- are recent. If anyone can figure out WHY Google is not crawling my site I would sure be greatful! I used to regularly get 10-15 new members per day from Google, not it has been two days and ZERO new members. Could it be something in my security settings for my new server?

Getting desparate

Basil

Basil

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Basil, if you don't mind, could I possibly get the info to your googles webmaster tools that includes all of your crawl information? I'd like to take a look if I could. If you don't want to give out this info, I understand.

Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Rick you have a PM

Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
Google + Gizmo's Site Map smile

I Have recently moved my forums to a new Domain.
So it was important to me to get my site under its new Domain name back into google as soon as i could.

Gizmo has been raving about the site map program he made,
so it thought to my self (He seems like a nice chap)
I think i will give it a go and see if it really does work.

The answer is YES smile

21 days after i installed the site map Google has listed my
site under its new domain name with 6,000 + links.
Not all my forums are public

It is fair to say what would have happened if i didn't have
the site map installed? Well i cant answer that to be honest
as I'm now listed. I like to think that it has helped
in my situation where a Domain move for my forums.

The Site Map has worked for me, but if you have any questions
about it functionality then please contact Gizmo via PM
as i can only report back the results i have.

Gizmo's site map isn't free, but for my needs worth the money.
You will also need a Google web masters account.
It also works with Yahoo.

My Results Click Me After 21 days.


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Lol the discovery time of a new domain with 0 existing data in google, would be 6+ months to crawl 6k links; based on experiance wink...

Glad to know it worked out for you Mark wink.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
BTW, it works for any search engine that supports sitemaps; and comes in numerous formats to accustom just about any type needed wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
I have installed a site map (not Gizmos but a third party I already had) and STILL Google is NOT indexing any of my threads - I am pulling my hair out! I see Google bots (one at a time) that say they are "Reading a Post in Flat Mode" all teh time, yet NONE of the posts are showing up. If I do this search:

site:www.britishcarforum.com +ubb/showflat I get 2 results of threads that are very old. (Back when I first installed threads7). I have spider-friendly URLs enabled. WHat on earth might be preventing Google from indexing any of me threads? Could it be something in the basic server set up I have that prevents Google from crawling certain threads? Any thought would be appreciated. I installed Threads7 back in January and my membership growth has been declining ever since. I MUST fix this. Any help or ideas would be most welcome.

Basil

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Basil, I see quite a few for your site (Results 1 - 10 of about 20,700 from www.britishcarforum.com/bcforum. (0.17 seconds)) site:www.britishcarforum.com/bcforum

I think your query was trying to get all occourances of www.britishcarforum.com/ubb vs www.britishcarforum.com/bcforum; since www.britishcarforum.com/ubb doesn't exist it'd explain why you wouldn't be recieving anything.


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
When I do the querry for site:www.britishcarforum.com/bcforum, I also see 20,700 but NONE are threads - they are all from other parts of my site like classifieds, photogallery, etc. But NONE are from any posts in my forums.

If you do this query for this site you will get over 1000 hits (still very small as far as I'm concerned):

site:https://www.ubbcentral.com +ubb/showflat

That is looking for strings in the site ubbcentral.com that include the string ubb/showflat, which all threads have. But if youdo the same for my site:

site:http://www.britishcarforum.com +ubb/showflat you only get 2 results from tow very early posts on my site when I moved to Threadss 7. This SHOULD return any thread or post that has been indexed from my site.

One of the two threads that does get returned is:

http://www.britishcarforum.com/bcforum/ubbthreads.php/ubb/showflat/Number/257432

But that is only one of two - yet ALLL of my threads look like this yet none NONE are being indexed. What sorts of things might prevent Google from indexing my site? I am wondering if it is something in my setup or maybe the fact that I have a link back to my "home" page www.britishcarforum.com in my main page? I dont know I'm grasping at straws. If I can't solve this pretty soon I may be forced to move to a different software package - I can't afford not to have people finding my site and like I've said, my new users from Google have dropped to near zero whereas I used to get many a day before I moved to T7. There has to be something about either my threads set up or my server set up, or something similar, that Google does not like. But WHAT?

Last edited by Basil; 04/15/2007 1:35 AM.
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
If you do that on mine you get 1 item:
site:www.undergroundnews.com +ubb/showflat

However doing the one i do I see plenty from my forum from my original:
site:www.undergroundnews.com/forum

When using google and searching:
site:www.britishcarforum.com/bcforum

I see pages of results... All from your forum...

I think it helps to note that it looks like most of the stuff that is indexed on your site isn't using the SE friendly extensions, it's simply using the standard script parsing:
www.britishcarforum.com/bcforum/ubbthreads.php?ubb=showday&day=20&month=5&year=2007

I would however recommend that if you're going to use a sitemap program that you tell it not to crawl profiles, or member only content, as there is a ton of links to "ubb error" pages that are restricted/member only.

You may also want to filter out the calendar, as your sitemap creator and google will go off the wall TRYING to index it into infinity lol...

FYI, both of the sugguestions i've made (about blocking private data and the calendar) are both stock in my sitemaps script (literally all you need fed to google is the forum indexes and the thread links, it doens't really need much else...)

What I am wondering however is if your sitemap is possibly creating more links than what Google can handle, have you gotten any errors such as the "Sitemaps cannot be submitted which contain more than 50,000 links" message?


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Originally Posted by Gizmo
Mark, the change won't be noticeable immedaitely, likely after a month you'll see things start to get larger (keep in mind the sitemap is to ultimately shove their discovery bot to the side, and tell their crawling bot where exactly to look, so once it gets started it'll work well).

BTW, my rankings are only from 2 months of the sitemap (as I made it, dun dun dun, 2 months ago lol).

For me I have been running the new Threads since January and Google simply is NOT crawling ANY of my threads. I do have a site map and robots.txt file and Google is crawling my site, but just NOT the threads - zero, zip, nada since January. If I do this:

site:http://www.britishcarforum.com I get about 38,400 hits, but ZERO none are of my current threads set up. The threads that do show are for my OLD threads 6.5 install - I can tell by the directory that shows up. If I do this, to see just threads in my new directory/threads install:

site:http://www.britishcarforum +bcforum/ubbthreads.php/ubb/showflat/

I get exactly ONE thread returned. This is just weird. There must be something about the layout of pages under Threads 7 that Google just does not like - and I think we as a community owe it to ourselves to figure out WHAT is the problem so it can be fixed. One thing that is different with Threads 7 from previous version is that that main site index page is not the portal page or a forum page, but only a re-director to the forums. Also, when guests view teh forums, ther see the side boxes and I'm wondering if that "portal-style" configuration might be what is messing up Google? Just brainstorming here - any other thoughts?

Basil

Page 2 of 4 1 2 3 4

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
1 members (Havenofsobriety), 522 guests, and 99 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)