Previous Thread
Next Thread
Print Thread
Hop To
Page 1 of 5 1 2 3 4 5
#176233 01/31/2007 10:24 AM
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
With my old 6.5.X threads, when I'd go to "Who's Online" I would see tons on Google IPs crawling my site. Now, with this version, I usually only see one or two Google IPs. Something seems to be broken with respect to Google. I do see lots of Yahoo and a few MSN, etc., but I am no longer seeing lots of Google IPs like I used to. This can't be good!

In fact, I just looked at the "Who's Online" on this site and saw the same thing - lots of Yahoo and only ONE Google IP.

Anyone have any idea why Google doesn't seem to want to crawl Threads 7 as it did the old version? (I've also notice that the number of new users I get based off Google searches has dropped dramatically. In fact, my overall new user rate has dropped. The past two days I['ve not had a single new user and THAT has not happened in at least 3 years).

Basil

Last edited by Basil; 01/31/2007 10:25 AM.
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
I think this is quite typical- not sure why or how we can improve it.

Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Originally Posted by Ian
I think this is quite typical- not sure why or how we can improve it.

It needs to be fixed! If I were the developer I'd consider this an absolute priority...above all other issues.

Basil

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
I've noticed the same thing with a lower number of visible google bots, but even on non-threads sites. Here we only see a couple online, but using their tools it's showing that they are crawling the site regularly, and not encountering any errors during their crawls.

The best thing I can recommend is to use their site overview tools. You can get information here as to how recently they've crawled your site, how many pages they are indexing on an average day, what pages they have encountered errors during their crawls, etc.


Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
My resutls show peaks once a month then pretty steady
for the rest of the month smile


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
Had forgotten about that.

That told me that Google has not succesfully visited my site for a week - and all the pages googled before that were on the old version of threads - I have been switched for about 6 weeks.

So not good for me frown

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
For this site the latest index was yesterday, and they've been averaging around 2500 pages indexed per day.

Last edited by Rick; 01/31/2007 2:38 PM. Reason: Edited again, because I'm a goober
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
I can't see any evidence that the new forums have been googled on my site frown

Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
Humm
What settings do you have in the cp Rick ?

As ive looked closer and mins not as high as it should be ?

Number of pages crawled per day

Max 444
Average 63
Min 13

I'm still running 7.02

O hang on i have only 3 forums visible to joe public lol..

Intresting what you do have selected in the control pannel?

could help alot of us.


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
My Settings.

Enable Spider-friendly URLs? NO

But ive never enabled that ever lol..

Not too sure about the extra load?


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
I have spider friendly urls enabled here, but that's about it beyond the fact that all of our forums are available to the public.

Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
How bad is the extra load in general?
In the warning about friendly Spider URL's ?


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
The only load would be from any increased spidering, so it's more of a bandwidth thing really. Even with all of the spidering that's google has done, I don't see them on very often, it's more Yahoo. But, Google definitely is visiting as it has a variety of posts already indexed that have been tagged as fixed for 7.1b5.

Joined: Jul 2006
Posts: 4,057
Joined: Jul 2006
Posts: 4,057
Enabled Now smile

Will keep an eye on things, but my sites not going
to give the results you have. As i have 3 or 4 forums
visible to all. The others are members only.


BOOM !! Version v7.6.1.1
People who inspire me Isaac ME Gizmo
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
maybe one day they will start mine frown This is despite me linking directly to the forums from my signature here.

Unless I am looking at the wrong info - but I do not think I am...

Joined: Aug 2006
Posts: 1,649
Likes: 1
Pooh-Bah
Pooh-Bah
Joined: Aug 2006
Posts: 1,649
Likes: 1

Originally Posted by Rick
For this site the latest index was yesterday, and they've been averaging around 2500k pages indexed per day.


2,500,000 pages/day?? [Linked Image from gangsterbb.net] whistle

Mine's being crawled regularly:

Googlebot last successfully accessed your home page on Jan 27, 2007.
Number of pages crawled per day: 4559 (max) 2657 (avg) 1004 (min)


GangsterBB.NET (Ver. 7.6.1.1)
PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54
2007 Content Rulez Contest - Hon Mention
UBB.classic 6.7.2 - RIP
Joined: Aug 2006
Posts: 1,649
Likes: 1
Pooh-Bah
Pooh-Bah
Joined: Aug 2006
Posts: 1,649
Likes: 1

Originally Posted by Ian
maybe one day they will start mine frown This is despite me linking directly to the forums from my signature here.


Have you verified/authenticated your domain(s) and created/indicated sitemap.xml files at Google (and Yahoo)?



GangsterBB.NET (Ver. 7.6.1.1)
PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54
2007 Content Rulez Contest - Hon Mention
UBB.classic 6.7.2 - RIP
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Ok, I'm a moron, surely not 2500k per day, I don't even think we have that many pages. Here's the stats for this site:

6137 (max) 3329 (avg) 564 (min)

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
You could always do a sitemap and include it in Google and Yahoo; i coded a custom tool to push my latest threads into the database..


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
can't say I have done a sitemap - however I have just created a basic google one.

Looking at the site linked above they can generate a more comprehensive one for $15 - I guess this is the best option?

Sitemaps are new to me LOL

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
lol, why pay someone to make one when you can pay me for mine and to install it on your server :x...

Mine won't have any duplicates (as it only pulls the threadid from the database) whereas anyone else publicly crawling your site will count as bandwidth used for their crawler to hit all of your pages, plus you'll have a ton of duplicate content wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Aug 2006
Posts: 1,649
Likes: 1
Pooh-Bah
Pooh-Bah
Joined: Aug 2006
Posts: 1,649
Likes: 1

Is yours $15, too? wink


GangsterBB.NET (Ver. 7.6.1.1)
PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54
2007 Content Rulez Contest - Hon Mention
UBB.classic 6.7.2 - RIP
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Mine ist $50, but mine also won't give you any waste of bandwidth, it will generate rss/asp/xml/text/html link feeds (which covers just about any search engine) plus i'll install it for you wink...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
will yours cover all other pages on the site as well?

ie non-forum pages?

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Mine's simply a forum integration designed to relay only threads; it's not designed to allow other items as it builds links on the fly wink...

I currently don't have any plans to allow users to insert other links, and if i did, it'd be manual entry...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 3,837
I
Ian Offline
Carpal Tunnel
Carpal Tunnel
I Offline
Joined: Jun 2006
Posts: 3,837
no worries then - I am building a lot of pages using the ubb themes, and these are as important as the threads - as they will contain content smile

Thanks anyway.

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
lol you could always create your own xml document to display non-threads content wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
I just signed up for Google Tools. What do you make of the attached graphs? I swicthed to Threads 7 just before Christmas. Look what happened to the crawl rate in early January - it just drops to nothing (I'm not sure but I think this was around the time I upgraded to 7.0.2).
Basil
Attachments
345-chart1.png 346-chart2.png

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
You might want to go through some of the other sections of their tools and see if there is any more info. The errors section, robots.txt, and a few others might be useful so you can see if there are any reasons why it's not or unable to index. All of the redirects in place from the old version to the new one?

Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Yes, I have the redirects in place. But question - should I keep my entire old site/database on the server, or can I just have the re-direct scripts? Also, just wondering what you graphs look like?

Last edited by Basil; 01/31/2007 9:03 PM.
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
I actually fluxuate a lot on mine, and it varies so much over the displayed period that it looks fine :scratches head:
Attachments
347-chart.png 348-chart1.png 349-chart2.png


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
You only need the redirects left behind, once you're done with the old forum, you can do away with everything else.

Ours fluctuated a bit in the beginning but I think that might have been because we were cleaning up a lot of old content. For the past month and a half or so it's been very steady it looks like:

[Linked Image from rb69.com]

Joined: Aug 2006
Posts: 1,649
Likes: 1
Pooh-Bah
Pooh-Bah
Joined: Aug 2006
Posts: 1,649
Likes: 1

Any issues in Diagnostic > Crawl errors > Web Crawl?

Hopefully it says the same as mine: "We have no errors to report. We crawl regularly, so check back later to see updates."

Here're my Crawl Stats:
Attachments
350-GoogleStats.gif


GangsterBB.NET (Ver. 7.6.1.1)
PHP Version 5.6.40 / MySQL 5.7.23-23 (was 5.6.41-84.1) / Apache 2.4.54
2007 Content Rulez Contest - Hon Mention
UBB.classic 6.7.2 - RIP
Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
It really bothers me that mine took a nose dive in early Jan and has not recovered. I have not done anything with robots.txt or any such things, so any ideas what might cause my stats to drop off suddenly?

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Like jgeoff suggested, try checking under:

Any issues in Diagnostic > Crawl errors > Web Crawl

See if there are any errors listed.

Joined: Aug 2006
Posts: 583
old hand
old hand
Joined: Aug 2006
Posts: 583
Looks like I have about 240 404 "Page not found" errors, a large number of them are looking for threads in my "old" threads install. Funny thing is, when I click on many of the links it says it could not find, I go to the page in question? Could the fact that I have my old install still there be causing the issue? Rick I think you said it would be ok now for me to remove the entire OLD install except for the re-direct scripts?

Basil

Joined: Oct 2006
Posts: 128
member
member
Joined: Oct 2006
Posts: 128
Originally Posted by Mark S
Enabled Now smile

Will keep an eye on things, but my sites not going
to give the results you have. As i have 3 or 4 forums
visible to all. The others are members only.

Please let us know if you see improvement. I'll turn mine on too if you do grin

Joined: Oct 2006
Posts: 128
member
member
Joined: Oct 2006
Posts: 128
Originally Posted by Gizmo
lol, why pay someone to make one when you can pay me for mine and to install it on your server :x...

Mine won't have any duplicates (as it only pulls the threadid from the database) whereas anyone else publicly crawling your site will count as bandwidth used for their crawler to hit all of your pages, plus you'll have a ton of duplicate content wink

I was going to ask you about that...I might be interested in this too. Can you PM me pricing and details?

Never mind about pricing I just read in a previous post. I'll give it some thought. My site is very geographically targeted so I'm not as concerned about search engines right now.

Last edited by tonto69; 02/01/2007 1:31 AM.
Joined: Nov 2006
Posts: 3,095
Likes: 1
Carpal Tunnel
Carpal Tunnel
Joined: Nov 2006
Posts: 3,095
Likes: 1
I just enabled mine - but don't use the Google tools so don't have the statistics. Will watch my resources and see if it has issue though.

Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Marty at www.AmbergrisCaye.com has been running it for a while now, he's fairly happy with the results wink


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Page 1 of 5 1 2 3 4 5

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
0 members (), 744 guests, and 147 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)