This morning I was looking at my "Who's Online" and noticed a Google spider.
I figured out something, maybe my "thoughts" on this will help out.
First, some background on my site, it is almost completely private. The only 2 forums viewable to the public are the rules and signup instructions and both of them are locked down so all you can do is read them, not reply or anything.
So, how long does it take for Google to scan your site? I mean, with out any posts there it should go quick, right? Wrong.
Google was indexing our site for about 1.5 hours. What was it doing? THE CALENDAR
This is perfectly fine by us, nothing is in the calendar that can't be public and helps us with Google rankings, so cool.
How this might help others and what I am guessing at.
Someone mentioned above that Google is no longer using multiple agents to fish/spider a site.
I can verify this by confirming that only one spider was on the site the entire time.
Here is where it might be of importance. When do they split off and spider other things? Do they ever make it past the calendar? In other words, if you have a large calendar, will the spider get confused and "stuck" in it with links back to pages it just visited? (ie Previous or next months OR going to multiple day listings with links to the other days).
The calendar is a major change (I love it BTW) from the previous version and maybe the Google spider is getting "stuck" in it. I know it was in our calendar for 1.5 hours and we hardly have any events in there beyond birthdays for 800 or so users.
Again, not that this is UBB's fault or anything, just an observation of what MAY be happening.
Also, we do not have the Google/spider friendly URLS turned on, but Google was in there for 1.5 hours.
