With my old 6.5.X threads, when I'd go to "Who's Online" I would see tons on Google IPs crawling my site. Now, with this version, I usually only see one or two Google IPs. Something seems to be broken with respect to Google. I do see lots of Yahoo and a few MSN, etc., but I am no longer seeing lots of Google IPs like I used to. This can't be good!
In fact, I just looked at the "Who's Online" on this site and saw the same thing - lots of Yahoo and only ONEGoogle IP.
Anyone have any idea why Google doesn't seem to want to crawl Threads 7 as it did the old version? (I've also notice that the number of new users I get based off Google searches has dropped dramatically. In fact, my overall new user rate has dropped. The past two days I['ve not had a single new user and THAT has not happened in at least 3 years).
I've noticed the same thing with a lower number of visible google bots, but even on non-threads sites. Here we only see a couple online, but using their tools it's showing that they are crawling the site regularly, and not encountering any errors during their crawls.
The best thing I can recommend is to use their site overview tools. You can get information here as to how recently they've crawled your site, how many pages they are indexing on an average day, what pages they have encountered errors during their crawls, etc.
That told me that Google has not succesfully visited my site for a week - and all the pages googled before that were on the old version of threads - I have been switched for about 6 weeks.
The only load would be from any increased spidering, so it's more of a bandwidth thing really. Even with all of the spidering that's google has done, I don't see them on very often, it's more Yahoo. But, Google definitely is visiting as it has a variety of posts already indexed that have been tagged as fixed for 7.1b5.
lol, why pay someone to make one when you can pay me for mine and to install it on your server :x...
Mine won't have any duplicates (as it only pulls the threadid from the database) whereas anyone else publicly crawling your site will count as bandwidth used for their crawler to hit all of your pages, plus you'll have a ton of duplicate content
Mine ist $50, but mine also won't give you any waste of bandwidth, it will generate rss/asp/xml/text/html link feeds (which covers just about any search engine) plus i'll install it for you ...
I just signed up for Google Tools. What do you make of the attached graphs? I swicthed to Threads 7 just before Christmas. Look what happened to the crawl rate in early January - it just drops to nothing (I'm not sure but I think this was around the time I upgraded to 7.0.2). Basil
You might want to go through some of the other sections of their tools and see if there is any more info. The errors section, robots.txt, and a few others might be useful so you can see if there are any reasons why it's not or unable to index. All of the redirects in place from the old version to the new one?
Yes, I have the redirects in place. But question - should I keep my entire old site/database on the server, or can I just have the re-direct scripts? Also, just wondering what you graphs look like?
You only need the redirects left behind, once you're done with the old forum, you can do away with everything else.
Ours fluctuated a bit in the beginning but I think that might have been because we were cleaning up a lot of old content. For the past month and a half or so it's been very steady it looks like:
It really bothers me that mine took a nose dive in early Jan and has not recovered. I have not done anything with robots.txt or any such things, so any ideas what might cause my stats to drop off suddenly?
Looks like I have about 240 404 "Page not found" errors, a large number of them are looking for threads in my "old" threads install. Funny thing is, when I click on many of the links it says it could not find, I go to the page in question? Could the fact that I have my old install still there be causing the issue? Rick I think you said it would be ok now for me to remove the entire OLD install except for the re-direct scripts?
lol, why pay someone to make one when you can pay me for mine and to install it on your server :x...
Mine won't have any duplicates (as it only pulls the threadid from the database) whereas anyone else publicly crawling your site will count as bandwidth used for their crawler to hit all of your pages, plus you'll have a ton of duplicate content
I was going to ask you about that...I might be interested in this too. Can you PM me pricing and details?
Never mind about pricing I just read in a previous post. I'll give it some thought. My site is very geographically targeted so I'm not as concerned about search engines right now.
Looks like mine has tailed right off after installing 7.x.
Of course the redirect script does not work for my site (as previously mentioned) so that doesn't help (just get taken to the forums home page) - it looks like google is searching the old forums.
This is not good news - it is bad enough everyone who does click on existing links just goes to my home page - making all those links worthless
Maybe the sitemap will help - but as it was not required in Novmeber, I can't see why it should be required now.
As seen in previous post/graph, mine droped off dramatically also as soon as I installed 7.0 (in early January). I am wondering if the problem isn't the use use the re-direct in the index? With my old Threads 6.5 the main index page was the portal page of my site, with all of the links necessary to crawl the site. I notice that this site does not seem to have a redirect, but rather a "real" index page that in turns provides a real link to the forums/portal page. I think as a test, I might try eliminating the index re-direct script and just put a page similar to the index page here and see if that helps the Google crawl rate. Seems to be much better on this site. Thoughts?
Ian, what product did you come from? Theres an issue with *most* hosts i've worked on and the cgi importer for classic; basically apache sends the redirect message already so trying to send a new one nerfs the redirector...
But i think you came form threads so that shoudln't be a problem there.
Ian, is your old threads board still up? I'd like to take a look if possible to see what's going on with the redirects. Mind you, google should still begin crawling the new site as long as they can get to it.
Ian, is your old threads board still up? I'd like to take a look if possible to see what's going on with the redirects. Mind you, google should still begin crawling the new site as long as they can get to it.
Rick - To restate my previous question: Is it possible that the re-direct script for the index file is messing up Google? It seems you don't have a re-director here and your site seems to be getting crawled by Google just fine. If you go back and look at my graphs, you will notice a sharp drop off on the day I installed Threads 7 (and the redirector). I'm thinking I should just have a normal index page like you have here?
We do have the redirector in place for our old threads board so I don't think that would be it. Since google has pretty much stopped indexing, you might try it and see if it makes a difference. If google can find it, it definitely should start indexing it again.
This is interesting (sort of). LAst night I finally deleted ever file on my old Threads install and teh old database, but only left the redirect scripts. Tonight, when I look at the Google crawls graph, it looks like a slight up-tic. Could be coincidence. Tonight I replaced the re-director index file with a "normal" index similar to the one used on this site so it will be interesting to see what google does in the coming days. Here is tonights graph. If youcompare to the previos graph you can see slight improvement:
Honestly, give it a month to see if it's really anything other than a conicidence; but i've been running the redirectors since i first moved over, so it could account for our continuous traffic
Ian, is your old threads board still up? I'd like to take a look if possible to see what's going on with the redirects. Mind you, google should still begin crawling the new site as long as they can get to it.
Ian, it seems that your redirects are working ok, or at least some of them. For example I found a couple in google such as this one, that redirects properly:
I see it has a couple thousand over your topics indexed from threads 7, and it's definitely indexing now as the top one if you view the "Cached page" it shows that it indexed it on the 30th of January. It might be a bit before it reindexes the site however.
Certainly the directories were not forwarding - must admit didn't try too many individual posts - as I was more concerned with the directories / categories as people have these as links.
Unforuntately, we never did get that part of the mapping done on either classic or threads. The main one that was concentrated on during the initial creation was just getting all of the topics to redirect properly.
Unforuntately, we never did get that part of the mapping done on either classic or threads. The main one that was concentrated on during the initial creation was just getting all of the topics to redirect properly.
no biggie, think about it, the redirectors are for search engines, and their last crawl before your migration has all the data it will ever have for your forum...
So why does it need to crawl forums, it has your posts, and thats all that really matters... a cat/forum redirector would simply forward to a new location, which would be crawled the second that google (or any other SE) sees your new forum index (as they're the first links it sees at the new location)
I see it has a couple thousand over your topics indexed from threads 7, and it's definitely indexing now as the top one if you view the "Cached page" it shows that it indexed it on the 30th of January. It might be a bit before it reindexes the site however.
Rick when I do that same search with my site I get exactly 6 results and none of them are threads! HELP!!!! (You've got a PM)
I did that and I got six results for my site...and NONE of them were threads or posts, they were just stuff like the calendar ect. And frankly, if your site is anything like mine, you should have a boatload MORE than 2,190 results. (I have 246,415 posts, none of which are indexed in Google). This is very troubleing and probably explains why my new member rate has dropped from 6-10 per day to almost nothing since "upgrading" to Threads 7".
Google crawled all over my old 6.5 install without that. Something is broken and needs to be fixed.
I just checked and noticed just under the check box for the "Enable spider friendly URLs" there is a statement that says something like the PATH_INFO invironment variable must be available. I checked and my PATH_INFO says: /bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
Is that what it is supposed to say? I do have teh Spider Friendly box checked and I have disabled teh HTTP Referer check.
Your linux path doesn't mean your path_info; path_info is a php setting; try enabling shorturls and see if it works (easiest way to see if it's enabled).
If you're on a server which allows you to define variables via .htaccess and path info is off you could try adding this to your .htaccess: AcceptPathInfo On
I am on my own dedicated server, so I can do pretty much whatever I want, if I can figure out how. When you say "enable short URLs" what do you mean? How do I do that?
Never min, I figured it out...yes, it is working. When I enable spider friendly URLs it changes the URLs in my post to get ride of variables like ubb=showflat=177103&page=0Post12345 and changes it to a regular URL as site.com/dirq/dir2/blah/blah
So it seems to be working fine, yet Google is NOT indexing any of my threads or posts.
Rick when I do that same search with my site I get exactly 6 results and none of them are threads! HELP!!!! (You've got a PM)
Well, I'm definitely not a search engine expert. I do see that they've at least indexed your cfrm (main forum page), but nothing beyond that. What little I know is only what I've read to make sure that threads is compliant and can be indexed. I've heard of stories where it can take several months up to a year for a site to be reindexed after a big change. All of your old links are still in Google so people are still getting to your forums from there.
A big thing I've read about is making sure you have back links that link to your forums. Could be as simple as adding it to your signature here as the more links you have on external sites pointing back to your site the more you'll get indexed.
Beyond that, I don't know a heck of a lot. Gizmo knows more than I do as I wouldn't even know what to put into a robots.txt file. I do see Google indexing this site and we're not doing anything other than the fact we do have a lot of back links to the site from everyone that runs threads. It still hasn't completely reindexed this site as that will take a while, but it's able to navigate around so all I could do is guess as to why it hasn't started respidering your forums yet.
One thing i recommend after a move to UBB.T7 is a sitemap (no i'm not just saying it because the sitemap software that I created is a possiblity); but inputting all of your threads into google at once does have positive aspects.
Did a few more queries on your site and it is indexing your site, just appears to be taking awhile. You have roughly around 500 or so showflat links indexed in the new format. You also have a variety of profiles indexed, but they all result in an access denied since public viewing of profiles is turned off.
So, it's definitely indexing the new site. I've seen cached pages ranging from Dec 21 to Jan 26th.
FYI - Since my import to version 7 I have 11,000 pages indexed by Google. I moved the board to a subforum and deleted the old board with no redirects being used. I seeded a few links in a couple other boards via my sig and a few posts to get the interest of the bots and they showed up right away.
I don't think it's a big deal if the number of bots is low. I usually only have one, but it is very active and can be found at the top of the online list most of the time.
Switch the php redirector to a 301 (permanent) redirect for starters (a temporary redirect is good for the first few months of crawling, but may not properly insert new data into SE's)
One thing i recommend after a move to UBB.T7 is a sitemap (no i'm not just saying it because the sitemap software that I created is a possiblity); but inputting all of your threads into google at once does have positive aspects.
you're setting a title? why? the ubb sets its own title...
I have description and keywords, but they're not really too huge on the se path anymore (imo) so i left them out.
As for publisher and author, you have some interesting values, what do they come from? or is that a funky foreign telephone number?
'eh I woudln't set revisit to 1 day; firstly some engines see 1 day and others 1 days, and most not both; 2 days is good! besides, thats a lot of reindexing for one day lol
hmm... I can't see a way to control the title - I certainly do not want it just saying Fans Focus for the site name - it needs to be a bit more than that.
hmm... I can't see a way to control the title - I certainly do not want it just saying Fans Focus for the site name - it needs to be a bit more than that.
Seems this site is the same as mine. No matter when you look at who's online, the most you will see is ONE Googlebot (and dozens of Yahoo bots). My old Threads used to gave Google bots crawling all over it.
Idont have 7.1 on my main site, but I'd estimate at least 3 on there; but i'm not concerned as I have a sitemap that automatically feeds my forum urls to google and yahoo.
Idont have 7.1 on my main site, but I'd estimate at least 3 on there; but i'm not concerned as I have a sitemap that automatically feeds my forum urls to google and yahoo.
Not for nothin', but even Google says the sitemaps are just guidelines/recommendations to help their crawl -- it's not a be-all, end-all index.
GangsterBB.NET (Ver. 7.6.1.1) PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54 2007 Content Rulez Contest - Hon Mention UBB.classic 6.7.2 - RIP
Idont have 7.1 on my main site, but I'd estimate at least 3 on there; but i'm not concerned as I have a sitemap that automatically feeds my forum urls to google and yahoo.
Not for nothin', but even Google says the sitemaps are just guidelines/recommendations to help their crawl -- it's not a be-all, end-all index.
never said it was, but if you're feeding them URLs to crawl, you're giving them a basis to work with, vs letting them auto discover
sadly none of this really explains why on most sites google has gone from many bots at once to one or two at the most - whereas yahoo is ending many. I have checked several forums and all I am seeing is one google and say 48 yahoo bots.
My crawl rate has continued to be a wimper, so no pickup, dspite having a sitemap of some description.
Well, I'm just saying that any "fix" will not be an overnight fix.
Search engines also dont' update their databases instantly, it takes time for information to be updated.
Also, if you've been using a temporary redirect for your forums for a while, you should move to a permanent one.
Also leaving your old forum up as a placeholder don't help your content get crawled faster, in fact it can penalize you for having duplicate content (not to mention theres already duplicate links to the same stuff in 7.x).
Didn't mean to sound mean, but was trying to let you know that things take time to get re-crawled.
Have you guys noticed if the Googlebot stays up toward the top of the online page most of the time? I think I have the same bot doing the work of several now.
Have you guys noticed if the Googlebot stays up toward the top of the online page most of the time? I think I have the same bot doing the work of several now.
Yes, I have noticed that too...and ther eis never more than ONE bot showing at any time. This is a definite change from the old threads. (For those of you still running 7.0.X you can tell google by looking for IP that starts with 66.249.X.X)
Have you guys noticed if the Googlebot stays up toward the top of the online page most of the time? I think I have the same bot doing the work of several now.
It's possible; they've donea lot of updates lately; it's completely probable that one of the updates was to have one bot crawl vs a "get links then hit then visit later" strategy...
If you use a stats logger like awstats you can see how much data (in mb) they actually transverse (which would actually be a good way to deterime if they're truely crawling less than they used to, as you could compare bw usage from them one month to a prior month)
Now when I look at the Google tool Crawl Rate, it says the last time it crawled my site was 4 Feb - that 3 days ago? But yet, I have been seeing a single bot on my site pretty much every day. Anyone know how often they update the stats?
I don't know the details, but I only see "last time crawled" about once a month -- usually like clockwork. Perhaps that means the last time it *completed* crawling, I don't know -- but I wouldn't worry about it...
GangsterBB.NET (Ver. 7.6.1.1) PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54 2007 Content Rulez Contest - Hon Mention UBB.classic 6.7.2 - RIP
'eh I used to get about $200 every 2 months, now I'm lucky to get $100 every 3 months... My hits are about half of what they used to be according to adsense's page impressions.
'eh I used to get about $200 every 2 months, now I'm lucky to get $100 every 3 months... My hits are about half of what they used to be according to adsense's page impressions.
This fall off in Goggle presence really has me worried. I used to get half a dozen new members every day - now I'm lucky if I get one or two new members. Sometimes I go for days without a new member and that hasn't happened in years! The question is, is this due to something about the 7.X threads layout, or did Google just change how they crawl?
Does anyone know of anyone still running 6.5.X threads so we can see how they are being crawled by Google? Until I switched, Google had numerous bots all over my forums all the time, now there is never more than ONE! I really think we need to figure this out - it's not a trivial problem in my humble opinion.
We're up to about 20k links to individual showflat pages now. Considering we have around 26k topics and these are all links to version 7 and not old ones, we're just about fully indexed again here.
And we're just running the stock version. No robots.txt, no sitemap.
We noticed a change in the way google spiders eve not so long ago. It's a welcome change really. Rather than having a bunch of little spiders DOSing your site they've figured out that so many of the links on the board all go to the same place. Somehow they've adapted to that and now they use fewer spiders to do the same amount of actual content additions. Lower load, same content. Nice.
I have to imagine that the same thing is happening with .threads.
Now if only those ....... over at MSN would figure it out.
Somehow they've adapted to that and now they use fewer spiders to do the same amount of actual content additions.
The problem is, when I do this in Google: site:www.mysite.com/forums/ubbthreads.php/ubb/showflat/ I see only 253 resluts and only TWO of them are displayed, the others are not included because they are "similar" to the two being shown. I have over 27 throusand threads and 253 thousand posts on my forum and only 253 are indexed? Houston, we have a problem! Also, my new member registrations had dropped off significantly since I switched to Threads 7 from 6.5.X.
I would like to find someone still running a Threads 6.5.X board and see what kind of crawling they are seeing! Something is terribly wrong with this picture.
PS: I'm doing an experiment and turning OFF Spider-friendly URLs for a while to see if I get more than one Google bot at a time. At this point it couldn't hurt since essentially nothing is getting indexed now.
So, 27 thousand threads, only 253 are indexed. In other words you're saying then that NONE from the 6.5.5 threads are indexed? Or they were removed? Where did they go if you only have 253 threads indexed?
Your math isn't working, or more likely, your explanation isn't. Please slow down and try again. Whatever your situation is, you're not communicating it very well. Rather than posting how bad this is in large RED letters, why not calm down some and do a better job helping people to help you.
I still have access to a 6.5 site at threadsdev.net. As of right now there are about 30 guests online. Of those IP's, most are either Yahoo or MSN, and there is only 1 IP that is identified as a googlebot.
Just looked at the old location and that lists 149,000 pages - so still a long way to go....
Of course the number of pages is far greater than the number of topics - you need to add in all links including every member - many of my links have a title in google of 'Fans Focus: Fans Focus Error' - so google is clearly going where it shouldn't - need to improve my robots.txt file - any suggestions?.
Hopefully if it is adding 12,000 posts a month, we should be fully listed within a year
Ian, google's webmaster tools will tell you what their crawl rate is for your site ...
Maximum Average Minimum Number of pages crawled per day 5361 1734 451 Number of kilobytes downloaded per day 76838 26097 4712 Time spent downloading a page (in milliseconds) 2996 856 445
It only returns 3 results and the ONLY post that is actually there is one that is accessable from out on my portal page. NONE of my other "threads" seem to be indexed by Google. What the heck is going on? I see a single Googlebot on my site, but It seems that none on my threads are getting indexed - ZERO, ZIP, NADA. I must figure this out! Help!
I notice that google crawl my site onmly with one indexer (showed in 7.1 as spider) and several other bots from google come as images.google... the google images spider.
I've checked on all the forums I have admin access to and don't see them in the calendar currently. You might just try adding a robots.txt file to exclude the calendar section.
i just blocked the calendar on mine, i wonder if it'll speed up results lol... then again, i use my sitemap addon so that could attribute to why all of my threads are there.
Basically it grabs all of the id's from your database and populates links for google of what content is where; it assists in the discovery of new urls for google.
Basically it grabs all of the id's from your database and populates links for google of what content is where; it assists in the discovery of new urls for google.
Then you tell Google that that is your site map? If you have tens of thousands of posts, wouldn;t that file get very big? Or does it not actually list every single thread?
The file is generated when it's called each week by google; the files size is a constant of under like 8k. The links are built when it's called each week by google's sitemap crawler.
But you have the rest, basically google hits your map and grabs the links, and lists them in its discovery database
Google crawls a site, link by link, to discover new links.
A SiteMap feeds them those links directly.
The benefit: You don't have to wait for their bot to discover new threads, you're cutting out 90% of the wait to discover new threads; the mod listing at ubbdev has google's "why should i use a sitemap" article from their help site quoted on it for more information.
![[Linked Image from gangsterbb.net]](http://www.gangsterbb.net/threads/images/graemlins/default/shocked.gif)
![[Linked Image from rb69.com]](http://www.rb69.com/stuff/google_graph.gif)
Still runs - can log in vis the CP etc....
how can i labell may vallage in google earth
![[Linked Image from navaho.infopop.cc]](http://navaho.infopop.cc/progress.gif)
