UBB.threads PHP Forum Software Community Forums General Suggestions and Feedback Google not crawling my site

Now when I look at the Google tool Crawl Rate, it says the last time it crawled my site was 4 Feb - that 3 days ago? But yet, I have been seeing a single bot on my site pretty much every day. Anyone know how often they update the stats?

Basil

I don't know the details, but I only see "last time crawled" about once a month -- usually like clockwork. Perhaps that means the last time it *completed* crawling, I don't know -- but I wouldn't worry about it...

Our crawling at ubbdev was pretty big during the fall but dropped a lot in December:

We've been using the index page at ubbdev (pulling the portal page into it).I don't think that's the issue.

Maybe google is making a concerted effort to not index forums so heavily?

how can i labell may vallage in google earth

umm, thats well beyond the scope of this support forum...

Had you made any changes to forum software around that time?

We installed the 7.1 beta December 3rd or 4th

hmm.. google really has slowed down their bot assault, there's only 1 here and maybe 30 other yahoo/ msn bots.

I've noticed my incoming traffic from google has slowed; and thus the monetizing done through adsense has been cut in half (and it already was slow)

Even my AdSense income is WAY down the past month or two

'eh I used to get about $200 every 2 months, now I'm lucky to get $100 every 3 months... My hits are about half of what they used to be according to adsense's page impressions.

This fall off in Goggle presence really has me worried. I used to get half a dozen new members every day - now I'm lucky if I get one or two new members. Sometimes I go for days without a new member and that hasn't happened in years! The question is, is this due to something about the 7.X threads layout, or did Google just change how they crawl?

Does anyone know of anyone still running 6.5.X threads so we can see how they are being crawled by Google? Until I switched, Google had numerous bots all over my forums all the time, now there is never more than ONE! I really think we need to figure this out - it's not a trivial problem in my humble opinion.

Basil

I'd say they changed the way they crawl; Imean look at yahoo, puttering around... then look at google, one bot online at a time usually...

I agree, I had 100 spiders crawling before and there was one or two that were google. Has anyone looked into seeing if they changed how the crawl?

what Basil is saying is that we need to compare 6.5x and 7.x to see if it is google or threads.

Bingo!

We're up to about 20k links to individual showflat pages now. Considering we have around 26k topics and these are all links to version 7 and not old ones, we're just about fully indexed again here.

And we're just running the stock version. No robots.txt, no sitemap.

How can I determine how many links to showflat pages Google has indexed on my site/

Search google for:
site:www.yoursite.tld/forumlocation/

You may ahve to go through a couple pages, the initial one says i have 800 and some odd, then the 3rd page says i have 8k lol...


Results 31 - 40 of about 8,070 from www.undergroundnews.com/forum. (0.09 seconds)

What is .tld?

top level domain:
.com
.net
.org
.info
.us
.co.uk

etc

We noticed a change in the way google spiders eve not so long ago. It's a welcome change really. Rather than having a bunch of little spiders DOSing your site they've figured out that so many of the links on the board all go to the same place. Somehow they've adapted to that and now they use fewer spiders to do the same amount of actual content additions. Lower load, same content. Nice.

I have to imagine that the same thing is happening with .threads.

Now if only those ....... over at MSN would figure it out.

and yahoo

Yahoo rapes my bandwidth, constantly lol

The problem is, when I do this in Google: site:www.mysite.com/forums/ubbthreads.php/ubb/showflat/ I see only 253 resluts and only TWO of them are displayed, the others are not included because they are "similar" to the two being shown. I have over 27 throusand threads and 253 thousand posts on my forum and only 253 are indexed? Houston, we have a problem! Also, my new member registrations had dropped off significantly since I switched to Threads 7 from 6.5.X.

I would like to find someone still running a Threads 6.5.X board and see what kind of crawling they are seeing! Something is terribly wrong with this picture.

Basil

PS: I'm doing an experiment and turning OFF Spider-friendly URLs for a while to see if I get more than one Google bot at a time. At this point it couldn't hurt since essentially nothing is getting indexed now.

So, 27 thousand threads, only 253 are indexed. In other words you're saying then that NONE from the 6.5.5 threads are indexed? Or they were removed? Where did they go if you only have 253 threads indexed?

Your math isn't working, or more likely, your explanation isn't. Please slow down and try again. Whatever your situation is, you're not communicating it very well. Rather than posting how bad this is in large RED letters, why not calm down some and do a better job helping people to help you.

Don't do that, do this:
site:www.mysite.com/forums/

By doing that you can have it searching just for the script ubbthreads.php which can bork things in the engine.

Also, it won't index your posts invidicually, it'll do threads, so you should have so many threads shown.

Also, what's your domain name so we can look at it and try to figure something out?

I still have access to a 6.5 site at threadsdev.net. As of right now there are about 30 guests online. Of those IP's, most are either Yahoo or MSN, and there is only 1 IP that is identified as a googlebot.

IP would start with 66.249.XX.XX, right?

Yep, that's the one.

216.239.46.
64.68.8
64.68.9
164.71.1.
192.51.44.
66.249.71.
66.249.66.
66.249.65.
66.249.64.
66.249.72.
64.233.166.136

hmm.. my personal site has 4059 topics and I get this at google:

Results 1 - 20 of about 9,730 from www.praisecafe.org/forum. (0.28 seconds)

At ubbdev we have:
Total Topics: 36582

Google has:
Results 1 - 20 of about 26,900 from www.ubbdev.com/forums. (0.21 seconds)

Kinda all over the place. I updated ubbdev to 7.1 2.5 months ago and my personal site this week.

We have 24K out of 84K listed - so looks like we are getting there

Just looked at the old location and that lists 149,000 pages - so still a long way to go....

Of course the number of pages is far greater than the number of topics - you need to add in all links including every member - many of my links have a title in google of 'Fans Focus: Fans Focus Error' - so google is clearly going where it shouldn't - need to improve my robots.txt file - any suggestions?.

Hopefully if it is adding 12,000 posts a month, we should be fully listed within a year

Ian, google's webmaster tools will tell you what their crawl rate is for your site ...

Maximum Average Minimum
Number of pages crawled per day
5361 1734 451
Number of kilobytes downloaded per day
76838 26097 4712
Time spent downloading a page (in milliseconds)
2996 856 445

When I do site:www.praisecafe.org, I only get 1-20 of 7810

When I do site:www.ubbdev.com I get 1-20 of about 22,300

I wonder why we get different results?

I just noticed something which may mean nothing, but thought it worth mentioning. When I look at whose online here (and my site) and I highlight the link it says the Googlebot is crawling, it shows a link of the non-spider-friendly style like this: www.ubbcentral.com/forums/ubbthreads.php?ubb= etc., rather than www.ubbcentral/forums/ubbthreads.php/ubb/showflat/...etc.