UBB.threads PHP Forum Software Community Forums General Suggestions and Feedback Google not crawling my site

I did that and I got six results for my site...and NONE of them were threads or posts, they were just stuff like the calendar ect. And frankly, if your site is anything like mine, you should have a boatload MORE than 2,190 results. (I have 246,415 posts, none of which are indexed in Google). This is very troubleing and probably explains why my new member rate has dropped from 6-10 per day to almost nothing since "upgrading" to Threads 7".

:ahem: a sitemap would do wonders :ahem: lol

Oops

Google crawled all over my old 6.5 install without that. Something is broken and needs to be fixed.

I just checked and noticed just under the check box for the "Enable spider friendly URLs" there is a statement that says something like the PATH_INFO invironment variable must be available. I checked and my PATH_INFO says:
/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin

Is that what it is supposed to say? I do have teh Spider Friendly box checked and I have disabled teh HTTP Referer check.

Basil

Your linux path doesn't mean your path_info; path_info is a php setting; try enabling shorturls and see if it works (easiest way to see if it's enabled).

If you're on a server which allows you to define variables via .htaccess and path info is off you could try adding this to your .htaccess:
AcceptPathInfo On

I am on my own dedicated server, so I can do pretty much whatever I want, if I can figure out how. When you say "enable short URLs" what do you mean? How do I do that?

Basil

Control Panel -> Master Settings -> Primary Settings -> Advanced Options -> Enable Spider-friendly URLs?

Oh, ok, well I have had that set (enabled) from the beginning. So How can I tell if its working?

Never min, I figured it out...yes, it is working. When I enable spider friendly URLs it changes the URLs in my post to get ride of variables like ubb=showflat=177103&page=0Post12345 and changes it to a regular URL as site.com/dirq/dir2/blah/blah

So it seems to be working fine, yet Google is NOT indexing any of my threads or posts.

well, then it's not that at least lol

Well, I'm definitely not a search engine expert. I do see that they've at least indexed your cfrm (main forum page), but nothing beyond that. What little I know is only what I've read to make sure that threads is compliant and can be indexed. I've heard of stories where it can take several months up to a year for a site to be reindexed after a big change. All of your old links are still in Google so people are still getting to your forums from there.

A big thing I've read about is making sure you have back links that link to your forums. Could be as simple as adding it to your signature here as the more links you have on external sites pointing back to your site the more you'll get indexed.

Beyond that, I don't know a heck of a lot. Gizmo knows more than I do as I wouldn't even know what to put into a robots.txt file. I do see Google indexing this site and we're not doing anything other than the fact we do have a lot of back links to the site from everyone that runs threads. It still hasn't completely reindexed this site as that will take a while, but it's able to navigate around so all I could do is guess as to why it hasn't started respidering your forums yet.

'eh i'm quite anal retentive about things lol...

One thing i recommend after a move to UBB.T7 is a sitemap (no i'm not just saying it because the sitemap software that I created is a possiblity); but inputting all of your threads into google at once does have positive aspects.

SE Friendly links is another good thing.

Also adding this to your header may help:

Then Linkbacks from other sites help too.

Also, make sure you're not blocking SE bots from crawling your forums via your robots.txt

Did a few more queries on your site and it is indexing your site, just appears to be taking awhile. You have roughly around 500 or so showflat links indexed in the new format. You also have a variety of profiles indexed, but they all result in an access denied since public viewing of profiles is turned off.

So, it's definitely indexing the new site. I've seen cached pages ranging from Dec 21 to Jan 26th.

It should be prudent to state that it will take a while to completely reindex your site; are you using a 301 or 302 redirect in the redirectors?

Looks like a 302

FYI - Since my import to version 7 I have 11,000 pages indexed by Google. I moved the board to a subforum and deleted the old board with no redirects being used. I seeded a few links in a couple other boards via my sig and a few posts to get the interest of the bots and they showed up right away.

I don't think it's a big deal if the number of bots is low. I usually only have one, but it is very active and can be found at the top of the online list most of the time.

Switch the php redirector to a 301 (permanent) redirect for starters (a temporary redirect is good for the first few months of crawling, but may not properly insert new data into SE's)

I am using...

<meta http-equiv="Pragma" content="no-cache">
<title>Football and Cricket Forums - Fans Focus - Focusing on Fans of Sport</title>
<meta name="robots" content="all,index,follow" />
<meta name="description" content="my description." />
<meta name="keywords" content="my keywords" />
<meta name="Publisher" content="isdoo Media 0871 236 0 250" />
<meta name="Author" content="isdoo Media 0871 236 0 250" />
<meta http-equiv="content-type" content="text/html; charset=iso-8859-1" />


I will add

<meta name="rating" content="General" />
<meta name="distribution" content="Global" />

Not sure about the revisit - had mine set to 5 days - but really I want google every day LOL

Also my robots is different.

you're setting a title? why? the ubb sets its own title...

I have description and keywords, but they're not really too huge on the se path anymore (imo) so i left them out.

As for publisher and author, you have some interesting values, what do they come from? or is that a funky foreign telephone number?

'eh I woudln't set revisit to 1 day; firstly some engines see 1 day and others 1 days, and most not both; 2 days is good! besides, thats a lot of reindexing for one day lol

hmm... I can't see a way to control the title - I certainly do not want it just saying Fans Focus for the site name - it needs to be a bit more than that.

then rename your community

Er no.... I do not want a long title in every breadcrumbs

so you want every page's title in a search engine to be the same?

no - I want the best of both worlds LOL

tisk tisk tisk...

Perhaps the CP could allow us to designate a "long" and "short" version of the Community Name...

Seems this site is the same as mine. No matter when you look at who's online, the most you will see is ONE Googlebot (and dozens of Yahoo bots). My old Threads used to gave Google bots crawling all over it.

Idont have 7.1 on my main site, but I'd estimate at least 3 on there; but i'm not concerned as I have a sitemap that automatically feeds my forum urls to google and yahoo.

Not for nothin', but even Google says the sitemaps are just guidelines/recommendations to help their crawl -- it's not a be-all, end-all index.

never said it was, but if you're feeding them URLs to crawl, you're giving them a basis to work with, vs letting them auto discover

Well their auto-discover is a monster as long as you don't take steps on purpose to stop it or confuse it.

They also have reasons why they will quit crawling your site too.

indeed so, they generally do an efficient job; helping it along though is just plain fun

sadly none of this really explains why on most sites google has gone from many bots at once to one or two at the most - whereas yahoo is ending many. I have checked several forums and all I am seeing is one google and say 48 yahoo bots.

My crawl rate has continued to be a wimper, so no pickup, dspite having a sitemap of some description.

I'll give it another week or so.....

Ian, no offense, but you can't expect a drastic change within a few days... A couple weeks, several months, perhaps, but a handful of days?

I think you are missing the point - why has Google suddenly reduced it's activity since 7.x was installed.

I did not have a sitemap installed under 6.x and was using SE friendly URL's.

Nothing has changed apart from threads software.

Either Google has changed their policy or something has altered on most if not all threads sites.

If Google has changed their policy and we need to adapt fine - but I do not think this is the case.

I was averaging 80,000 pages a day, with a max of 175,055 - I am now on around 4,000 a day!

But if you feel this is nothing to worry about then great I am happy - it just seems strange

But at least my redirects are partially working

Well, I'm just saying that any "fix" will not be an overnight fix.

Search engines also dont' update their databases instantly, it takes time for information to be updated.

Also, if you've been using a temporary redirect for your forums for a while, you should move to a permanent one.

Also leaving your old forum up as a placeholder don't help your content get crawled faster, in fact it can penalize you for having duplicate content (not to mention theres already duplicate links to the same stuff in 7.x).

Didn't mean to sound mean, but was trying to let you know that things take time to get re-crawled.

no worries - I do not profess to being an expert on search engines - but I find it strange that mine and Basil's graphs are very similar.

Have you guys noticed if the Googlebot stays up toward the top of the online page most of the time? I think I have the same bot doing the work of several now.

Yes, I have noticed that too...and ther eis never more than ONE bot showing at any time. This is a definite change from the old threads. (For those of you still running 7.0.X you can tell google by looking for IP that starts with 66.249.X.X)

Basil

It's possible; they've donea lot of updates lately; it's completely probable that one of the updates was to have one bot crawl vs a "get links then hit then visit later" strategy...

If you use a stats logger like awstats you can see how much data (in mb) they actually transverse (which would actually be a good way to deterime if they're truely crawling less than they used to, as you could compare bw usage from them one month to a prior month)