I have a large forum that has more then 70K pages so far.
Many are quite old, but good for research, but bad for search engines index.
Google index only the first 40.000 pages and it's taking couple days to search my whole site.
Well, I was thinking if I could break the forum, putting the messages 2 years old or more in a different folder/forum (not sure), that I will block from Google to visit, but would keep open for the users to search.
Is that possible? any one else had/have this problem?
What would be solutions?

I thought about duplicating the database, installing a new sub-directory with a second second of the forum programs (not sure if it would break the license ), then delete the messages older then 2 years and create entry points to the second forum.
BUT: then what to do next time? I don't know how to 'merge' the 2005 messages next year, when that will be 2 years old, on the old original database.

Any ideas would be appreciate.

I dont know if this will help,
but send a pm to Gizmo you can see his name as the top poster
he has made a site map thing for UBB Threads and could
sort youre problem.

I use it with mine,
but your best asking him about the details.
Its not free by the way.
But works for me

If google can not SEE the content, the content is not there; they will remove it from their index.

As for the 40k limit, where is this coming in to play? It doesn't make sense that they'd have a 40k limit when their sitemap index allows up to 50k links per sitemap...

and Mark, they don't need to pm me; I don't provide support for issues via PM as it limits content that others may come looking for.

The 40.000 files are a limit that I read some where in a SEO web site. It was a while a go and I had it on my mind.
Now I'm running a sitemap program and then it came to my mind again.
This is why I decided to reduce the size of my forum, just to have a 'fresh' topics on google, but if one user want, he/she can go back many years.

Per the first part of your response: I could not get it.
I want to move part of my forum to a folder that will be behind a "robots.txt" file that will not allow google to see it, but normal users would be able to search it.

I'm not sure how could I split the forum in that way: old posts on a specific folder, new posts on the regular folder.

If google cannot see the content, it assumes it doesn't exist; by setting your robots.txt to not allow them to crawl the data, you're telling them it does not exist there.

As for the limit they allow, again I don't believe this is the case, especially since they support multiple sitemaps, sitemap indexes, and have a 50k link limit.

As for sitemaps; a sitemap generator like the one I sell for the UBB should populate the links from threads automatically, including an index for going over a limit.

I recommend Giz's site map, it works great

I know that if google cannot see the content, it will not exist to google...
Now, lets try to make a simple question here:
How do I split the forum?
All posts before 2005 will be on a different folder that google will not see.
All posts 2005 and on will be on the current forum.

I have no clue how to do the split.


Now, for the sitemap: where can I find it?
I went to those 3 sites at his signature, but could not find it.
Thank you

The thing is, if you SPLIT the content, and split off everything before a specific date, google won't keep it in its database; they'll purge it... Just making sure you know if you DO do this step it will delete a lot of content for your forum in google (whihc I don't see why you'd ever want to do this so I'm trying to be sure I'm clear here).

There is information on my sitemap script here; the SiteMap script will generate a sitemap index at a specified offset value (the default in-script is hardcoded to 25k links per sitemap and the indexer will generate links/offset values automatically based on whatever the hardcoded offset value is).