UBB.threads PHP Forum Software Community Forums General Suggestions and Feedback Google not crawling my site

With my old 6.5.X threads, when I'd go to "Who's Online" I would see tons on Google IPs crawling my site. Now, with this version, I usually only see one or two Google IPs. Something seems to be broken with respect to Google. I do see lots of Yahoo and a few MSN, etc., but I am no longer seeing lots of Google IPs like I used to. This can't be good!

In fact, I just looked at the "Who's Online" on this site and saw the same thing - lots of Yahoo and only ONE Google IP.

Anyone have any idea why Google doesn't seem to want to crawl Threads 7 as it did the old version? (I've also notice that the number of new users I get based off Google searches has dropped dramatically. In fact, my overall new user rate has dropped. The past two days I['ve not had a single new user and THAT has not happened in at least 3 years).

Basil

I think this is quite typical- not sure why or how we can improve it.

It needs to be fixed! If I were the developer I'd consider this an absolute priority...above all other issues.

Basil

I've noticed the same thing with a lower number of visible google bots, but even on non-threads sites. Here we only see a couple online, but using their tools it's showing that they are crawling the site regularly, and not encountering any errors during their crawls.

The best thing I can recommend is to use their site overview tools. You can get information here as to how recently they've crawled your site, how many pages they are indexing on an average day, what pages they have encountered errors during their crawls, etc.

My resutls show peaks once a month then pretty steady
for the rest of the month

Had forgotten about that.

That told me that Google has not succesfully visited my site for a week - and all the pages googled before that were on the old version of threads - I have been switched for about 6 weeks.

So not good for me

For this site the latest index was yesterday, and they've been averaging around 2500 pages indexed per day.

I can't see any evidence that the new forums have been googled on my site

Humm
What settings do you have in the cp Rick ?

As ive looked closer and mins not as high as it should be ?

Number of pages crawled per day

Max 444
Average 63
Min 13

I'm still running 7.02

O hang on i have only 3 forums visible to joe public lol..

Intresting what you do have selected in the control pannel?

could help alot of us.

My Settings.

Enable Spider-friendly URLs? NO

But ive never enabled that ever lol..

Not too sure about the extra load?

I have spider friendly urls enabled here, but that's about it beyond the fact that all of our forums are available to the public.

How bad is the extra load in general?
In the warning about friendly Spider URL's ?

The only load would be from any increased spidering, so it's more of a bandwidth thing really. Even with all of the spidering that's google has done, I don't see them on very often, it's more Yahoo. But, Google definitely is visiting as it has a variety of posts already indexed that have been tagged as fixed for 7.1b5.

Enabled Now

Will keep an eye on things, but my sites not going
to give the results you have. As i have 3 or 4 forums
visible to all. The others are members only.

maybe one day they will start mine This is despite me linking directly to the forums from my signature here.

Unless I am looking at the wrong info - but I do not think I am...

2,500,000 pages/day??

Mine's being crawled regularly:

Googlebot last successfully accessed your home page on Jan 27, 2007.
Number of pages crawled per day: 4559 (max) 2657 (avg) 1004 (min)

Have you verified/authenticated your domain(s) and created/indicated sitemap.xml files at Google (and Yahoo)?

Ok, I'm a moron, surely not 2500k per day, I don't even think we have that many pages. Here's the stats for this site:

6137 (max) 3329 (avg) 564 (min)

You could always do a sitemap and include it in Google and Yahoo; i coded a custom tool to push my latest threads into the database..

can't say I have done a sitemap - however I have just created a basic google one.

Looking at the site linked above they can generate a more comprehensive one for $15 - I guess this is the best option?

Sitemaps are new to me LOL

lol, why pay someone to make one when you can pay me for mine and to install it on your server :x...

Mine won't have any duplicates (as it only pulls the threadid from the database) whereas anyone else publicly crawling your site will count as bandwidth used for their crawler to hit all of your pages, plus you'll have a ton of duplicate content

Is yours $15, too?

Mine ist $50, but mine also won't give you any waste of bandwidth, it will generate rss/asp/xml/text/html link feeds (which covers just about any search engine) plus i'll install it for you ...

will yours cover all other pages on the site as well?

ie non-forum pages?

Mine's simply a forum integration designed to relay only threads; it's not designed to allow other items as it builds links on the fly ...

I currently don't have any plans to allow users to insert other links, and if i did, it'd be manual entry...

no worries then - I am building a lot of pages using the ubb themes, and these are as important as the threads - as they will contain content

Thanks anyway.

lol you could always create your own xml document to display non-threads content

I just signed up for Google Tools. What do you make of the attached graphs? I swicthed to Threads 7 just before Christmas. Look what happened to the crawl rate in early January - it just drops to nothing (I'm not sure but I think this was around the time I upgraded to 7.0.2).
Basil

You might want to go through some of the other sections of their tools and see if there is any more info. The errors section, robots.txt, and a few others might be useful so you can see if there are any reasons why it's not or unable to index. All of the redirects in place from the old version to the new one?

Yes, I have the redirects in place. But question - should I keep my entire old site/database on the server, or can I just have the re-direct scripts? Also, just wondering what you graphs look like?

I actually fluxuate a lot on mine, and it varies so much over the displayed period that it looks fine :scratches head:

You only need the redirects left behind, once you're done with the old forum, you can do away with everything else.

Ours fluctuated a bit in the beginning but I think that might have been because we were cleaning up a lot of old content. For the past month and a half or so it's been very steady it looks like:

Any issues in Diagnostic > Crawl errors > Web Crawl?

Hopefully it says the same as mine: "We have no errors to report. We crawl regularly, so check back later to see updates."

Here're my Crawl Stats:

It really bothers me that mine took a nose dive in early Jan and has not recovered. I have not done anything with robots.txt or any such things, so any ideas what might cause my stats to drop off suddenly?

Like jgeoff suggested, try checking under:

Any issues in Diagnostic > Crawl errors > Web Crawl

See if there are any errors listed.

Looks like I have about 240 404 "Page not found" errors, a large number of them are looking for threads in my "old" threads install. Funny thing is, when I click on many of the links it says it could not find, I go to the page in question? Could the fact that I have my old install still there be causing the issue? Rick I think you said it would be ok now for me to remove the entire OLD install except for the re-direct scripts?

Basil

Please let us know if you see improvement. I'll turn mine on too if you do

I was going to ask you about that...I might be interested in this too. Can you PM me pricing and details?

Never mind about pricing I just read in a previous post. I'll give it some thought. My site is very geographically targeted so I'm not as concerned about search engines right now.

I just enabled mine - but don't use the Google tools so don't have the statistics. Will watch my resources and see if it has issue though.

Marty at www.AmbergrisCaye.com has been running it for a while now, he's fairly happy with the results