UBB.threads PHP Forum Software Community Forums General Suggestions and Feedback Google still not there

Basil, you definitely have something interesting going on. You still have all of the old links in google from your 6.5 forum. You have a bunch of links to the new forum as far as to the general forum listing and the individual postlist pages. But you don't appear to have many new links to the new showflat script.

What we usually see is the old links eventually go away and google reindexes your new ones. For some reason, they are keeping all of your old links in the system. You have thousands and thousands pointing to the old showflat script but barely any to the new.

For example, do this:

site:britishcarforum.com +ubb/postlist

You'll see it has a bunch of links to individual postlist pages. But if you do this:

site:britishcarforum.com +ubb/showflat

You'll see that there aren't many listed at all. Your site appears to be the only one with this problem. Everyone else that has posted has many many links to the new showflat. I can search for distinct subjects from their site and they are listed in Google. Yours I cannot however. I'm not sure if your site is being penalized for something, or what. I know you had made the change to/from search engine friendly not too long ago, so not sure if it's related.

Interesting and troubling. I see a single Google bot all the time and it "says" it is reading posts, but they don't seem to actually be indexing. Well, now we know ther eis "something" going on, now the trick will be to figure out exactly what!


Right, that's what I've been saying. It is indexing "part" of my site, but not the main parts. I am wondering if having the re-director to the old threads could be part of the problem? Should I get rid of that at some point on the theory that Google will then evantually drop the old links and re-index the new? Just a thought.


Even using the site:britishcarforum.com +ubb/postlist there were only 153 that cam eup for me (only 20 the other search for showflat). Clearly I have a problem.


Lucky me. When I do YOUR site with the showflat seach above, I get just over 12,000 results (but you have over 26,000 topics!!) But that is clearly better than the 20 results that I get for my site.


That change was only temporary and only because I was desperate for Google to index my site. Ok, so not all are having this issue, so now I have to figure out what is different from my site than everyone elses. One thing that is also different for me besides the new Threads 7, is that I moved to a dedicated server and am using a server management service to monitor things, install security, etc. Now I am wondering is maybe something about my server set up could be preventing google from crawling my site? For example, when I recently installed a wiki on my site, I found that I could not save any edited pages - I would get "Forbidden You don't have access..." Well, I finally figured out that if I disabled ssomething called mod_secure - an Apachie security thingy, then my wiki worked right. Now I only disabled it for the wiki directory, not my entire root directory, so I'm wondering if maybe something like that might be at issue with Google? I know I'm grasping at straws, but I don't have anything else to grasp at right now.

Or could there be a difference in my PhP setup?

Basil

It's hard to say for sure what's keeping them from indexing the newer content. On your redirect scripts did you have a 301 or a 302 for the redirect? It shouldn't matter because we have people using both without a problem, but figured I'd ask.

If you use the google webmaster tools and you look at the Crawl Errors is it giving anything specific?

It says its temporary. Should I change to perm?

I have about 2600+ 404 Page not found errors and almost all of them are llokoing for things in my old 6.5 install directory.

Also, while I try to figure this out I am trying a program I happened to have (I forgot about it) called GSiteCrawl which is a free site map generator that will generate a site map for the entire site for both Goggle, Yahoo and others. This will be the first time I've tried it, so I'm hoping it will help.

Basil

ditto for me too. as far as i can see, there is no major issue with the product in relation to google.

however, the the fact that some people -- rightly or wrongly -- perceive that they have a problem should be taken seriously and talked about to see what if anything can be done about it.

getting content listed in search engines is critically important to most forums. many forums provide invaluable information on a huge variety of different subjects -- often using language and example understandable to non-experts -- and as such deserve their place in the search engines.

thus we should not be shy about helping the search engines find what they need.

This morning I was looking at my "Who's Online" and noticed a Google spider.

I figured out something, maybe my "thoughts" on this will help out.

First, some background on my site, it is almost completely private. The only 2 forums viewable to the public are the rules and signup instructions and both of them are locked down so all you can do is read them, not reply or anything.

So, how long does it take for Google to scan your site? I mean, with out any posts there it should go quick, right? Wrong.

Google was indexing our site for about 1.5 hours. What was it doing? THE CALENDAR

This is perfectly fine by us, nothing is in the calendar that can't be public and helps us with Google rankings, so cool.

How this might help others and what I am guessing at.

Someone mentioned above that Google is no longer using multiple agents to fish/spider a site.

I can verify this by confirming that only one spider was on the site the entire time.

Here is where it might be of importance. When do they split off and spider other things? Do they ever make it past the calendar? In other words, if you have a large calendar, will the spider get confused and "stuck" in it with links back to pages it just visited? (ie Previous or next months OR going to multiple day listings with links to the other days).

The calendar is a major change (I love it BTW) from the previous version and maybe the Google spider is getting "stuck" in it. I know it was in our calendar for 1.5 hours and we hardly have any events in there beyond birthdays for 800 or so users.

Again, not that this is UBB's fault or anything, just an observation of what MAY be happening.

Also, we do not have the Google/spider friendly URLS turned on, but Google was in there for 1.5 hours.

Fred

A little description of what google does and several suggestions...

When google "hits" your site, it grabs new links and grabs content from the pages it was sent to do so; basically it does crawling and discovery at once.

Adding a SiteMap aids highly in discovery, since you're telling google what is there, and it knows what links it needs to crawl by your sitemap (it still does its own discovery, but it's hit and grab vs your feeding it valid links).

You can block search engines from crawling your calendar with a robots.txt entry:
robots.txt:

As for "why you'd need a sitemap", I can only recommend you read the FAQ at google here (or read about my UBB.Sitemap generator here) as nothing i say can say it better.

Also, by using a sitemap, your links will be crawled regardless of SE links being on, as the links to your threads that are fed to the search engines (most major SE's now support sitemaps for note) will not "break" links, as you won't have to worry about most of the discovery (as you're telling them what links are there and are valid)

As for Ian saying he wants a sitemap for both his site and his forums, the UBB Sitemap generator grabs feeds live from your forum, and you can submit numerous sitemaps to google, so you could use alternate "link building" sitemap generators for miscellaneous site data.

that's useful to know. i've already added that to my robots.txt.

to prevent search engines from indexing user profiles, would this entry be correct and/or sufficient?

Theres actually a setting in the cp to disallow guests viewing profiles; that should be a sufficient way of doing that in general.

But, if you dont' want that on, then:

If you didn't want them spidering the userlist, then this as well:

this is the robots.txt i am now using to (hopefully) stop search engines from wasting their time and my bandwidth:

Looks interesting enough, though several of those will just error out from a guest viewing them anyway (mark all read, my cookies, online, showprofile should you have guest viewing off)

It's amazing how often something so easy as robots.txt crafting goes un-noticed to people, yet it's such a powerful tool and easy to work with and requiring minimal reading to understand.

installing that robots.txt certainly made an immediate difference with the search engines.

besides reducing the total number of active spiders (esp. yahoo), the spiders left are now all engaged in indexing forums, rather than peripheral and unimportant sections of my site.

Told you that i'm of some use

You are always of use

It is always better to ban spiders from going to places where guests are banned, as they will still spider and file these pages - which means you will have several 'you do not have permission to view this page' pages within the search engines

I also have...


Disallow: /tmp
Disallow: /logs
Disallow: /forum/
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /generic/
Disallow: /photos/

which includes my old forums in case any spiders are still lurking there

my old forums just have a redirector; any requests get zoomed to threads

so do mine - but google was still in there LOL

yeh, getting bounced to the new locations; keep in mind mr google has to see that old links don't exist to update their databases to your new links

I'm all up for giving it a go.

Using Gizmo's example :

User-agent: *
Disallow; /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday

Note line 2 has a ; and should have been a :
This can cause google to error on the syntax

Right lets see who's looking.....

Yahoo = 28 Viewing list of forums
Yahoo Reading = 32 Reading a post in flat mode
MSN = 5 Reading a post in flat mode
Google = 1 Reading a post in X

And that's a normal afternoon for me,
so i will update you next week if the robot.txt has worked
for me ??

I cant any sooner as i'm away

Just checked these forums "Whos on Line"
and currently 209 anonymous.

But only 2 or 3 max from Yahoo "Viewing list of forums"
Where as mine is almost 50-50 reading and viewing.


Its an observation thats all,
as i can use as a possible reference upon my return.
As Yahoo is reading 99% for this forum.
And mine at the moment before robot.txt is 50%

Maybe its normal to be 50-50 but if no one say's
no one would know??

But then if you delete the old forums they would not exist anyway.

Just FYI - I have had this for a while - at the moment I have 1 google (nothing recorded as to where they are looking), 1 MSN (nothing recorded as to where they are looking), and 168 from Yahoo all eithin the last 30 minutes.

what's yahoo doing in the last 30 mins ian?

183 (at the moment) of the blighters all over the forum LOL

Always only 1 MSN & 1 Google

cheers Ian


User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday

Note line 2 had a ; and should have been a :
This can cause google to error on the syntax

Check your code if you have used the earlier examples.
You can test what you have at google tools.

I believe that some bots check every hit to a new page whereas some cache it for a day or so.

I updated the earlier example, the reason it had a ; was because had to manually enter the one line cas i deleted some other rules and deleted a smidge too much lol

should that be forum, or forums??

It should be the directory you installed threads to; so it could be either, or neither...

Well, now that some time has passed, and I have included a site map, I tried this again: site:britishcarforum.com +ubb/showflat
and I STILL only have 5 threads show up and none of them - NONE -- are recent. If anyone can figure out WHY Google is not crawling my site I would sure be greatful! I used to regularly get 10-15 new members per day from Google, not it has been two days and ZERO new members. Could it be something in my security settings for my new server?

Getting desparate

Basil

Basil

Basil, if you don't mind, could I possibly get the info to your googles webmaster tools that includes all of your crawl information? I'd like to take a look if I could. If you don't want to give out this info, I understand.

Rick you have a PM

Google + Gizmo's Site Map

I Have recently moved my forums to a new Domain.
So it was important to me to get my site under its new Domain name back into google as soon as i could.

Gizmo has been raving about the site map program he made,
so it thought to my self (He seems like a nice chap)
I think i will give it a go and see if it really does work.

The answer is YES

21 days after i installed the site map Google has listed my
site under its new domain name with 6,000 + links.
Not all my forums are public

It is fair to say what would have happened if i didn't have
the site map installed? Well i cant answer that to be honest
as I'm now listed. I like to think that it has helped
in my situation where a Domain move for my forums.

The Site Map has worked for me, but if you have any questions
about it functionality then please contact Gizmo via PM
as i can only report back the results i have.

Gizmo's site map isn't free, but for my needs worth the money.
You will also need a Google web masters account.
It also works with Yahoo.

My Results Click Me After 21 days.

Lol the discovery time of a new domain with 0 existing data in google, would be 6+ months to crawl 6k links; based on experiance ...

Glad to know it worked out for you Mark .

BTW, it works for any search engine that supports sitemaps; and comes in numerous formats to accustom just about any type needed

I have installed a site map (not Gizmos but a third party I already had) and STILL Google is NOT indexing any of my threads - I am pulling my hair out! I see Google bots (one at a time) that say they are "Reading a Post in Flat Mode" all teh time, yet NONE of the posts are showing up. If I do this search:

site:www.britishcarforum.com +ubb/showflat I get 2 results of threads that are very old. (Back when I first installed threads7). I have spider-friendly URLs enabled. WHat on earth might be preventing Google from indexing any of me threads? Could it be something in the basic server set up I have that prevents Google from crawling certain threads? Any thought would be appreciated. I installed Threads7 back in January and my membership growth has been declining ever since. I MUST fix this. Any help or ideas would be most welcome.

Basil

Basil, I see quite a few for your site (Results 1 - 10 of about 20,700 from www.britishcarforum.com/bcforum. (0.17 seconds)) site:www.britishcarforum.com/bcforum

I think your query was trying to get all occourances of www.britishcarforum.com/ubb vs www.britishcarforum.com/bcforum; since www.britishcarforum.com/ubb doesn't exist it'd explain why you wouldn't be recieving anything.

When I do the querry for site:www.britishcarforum.com/bcforum, I also see 20,700 but NONE are threads - they are all from other parts of my site like classifieds, photogallery, etc. But NONE are from any posts in my forums.

If you do this query for this site you will get over 1000 hits (still very small as far as I'm concerned):

site:https://www.ubbcentral.com +ubb/showflat

That is looking for strings in the site ubbcentral.com that include the string ubb/showflat, which all threads have. But if youdo the same for my site:

site:http://www.britishcarforum.com +ubb/showflat you only get 2 results from tow very early posts on my site when I moved to Threadss 7. This SHOULD return any thread or post that has been indexed from my site.

One of the two threads that does get returned is:

http://www.britishcarforum.com/bcforum/ubbthreads.php/ubb/showflat/Number/257432

But that is only one of two - yet ALLL of my threads look like this yet none NONE are being indexed. What sorts of things might prevent Google from indexing my site? I am wondering if it is something in my setup or maybe the fact that I have a link back to my "home" page www.britishcarforum.com in my main page? I dont know I'm grasping at straws. If I can't solve this pretty soon I may be forced to move to a different software package - I can't afford not to have people finding my site and like I've said, my new users from Google have dropped to near zero whereas I used to get many a day before I moved to T7. There has to be something about either my threads set up or my server set up, or something similar, that Google does not like. But WHAT?

If you do that on mine you get 1 item:
site:www.undergroundnews.com +ubb/showflat

However doing the one i do I see plenty from my forum from my original:
site:www.undergroundnews.com/forum

When using google and searching:
site:www.britishcarforum.com/bcforum

I see pages of results... All from your forum...

I think it helps to note that it looks like most of the stuff that is indexed on your site isn't using the SE friendly extensions, it's simply using the standard script parsing:
www.britishcarforum.com/bcforum/ubbthreads.php?ubb=showday&day=20&month=5&year=2007

I would however recommend that if you're going to use a sitemap program that you tell it not to crawl profiles, or member only content, as there is a ton of links to "ubb error" pages that are restricted/member only.

You may also want to filter out the calendar, as your sitemap creator and google will go off the wall TRYING to index it into infinity lol...

FYI, both of the sugguestions i've made (about blocking private data and the calendar) are both stock in my sitemaps script (literally all you need fed to google is the forum indexes and the thread links, it doens't really need much else...)

What I am wondering however is if your sitemap is possibly creating more links than what Google can handle, have you gotten any errors such as the "Sitemaps cannot be submitted which contain more than 50,000 links" message?

For me I have been running the new Threads since January and Google simply is NOT crawling ANY of my threads. I do have a site map and robots.txt file and Google is crawling my site, but just NOT the threads - zero, zip, nada since January. If I do this:

site:http://www.britishcarforum.com I get about 38,400 hits, but ZERO none are of my current threads set up. The threads that do show are for my OLD threads 6.5 install - I can tell by the directory that shows up. If I do this, to see just threads in my new directory/threads install:

site:http://www.britishcarforum +bcforum/ubbthreads.php/ubb/showflat/

I get exactly ONE thread returned. This is just weird. There must be something about the layout of pages under Threads 7 that Google just does not like - and I think we as a community owe it to ourselves to figure out WHAT is the problem so it can be fixed. One thing that is different with Threads 7 from previous version is that that main site index page is not the portal page or a forum page, but only a re-director to the forums. Also, when guests view teh forums, ther see the side boxes and I'm wondering if that "portal-style" configuration might be what is messing up Google? Just brainstorming here - any other thoughts?

Basil