Hey Gizmo-

You seem to be at the leading edge of search engine promotion with respects to the UBB Threads board and generating a sitemap for it.

I am an ASP programmer and had already written code a few years ago that used the File Scripting Object to iterate through all the flat files generated by the old UBB and generate URL's to all the posts and topics for inclusion in my sitemap.

We switched to Threads 7.2 several months ago, and I just now got around to modifying my sitemap generator to include the Threads-based URLs.

I basically just wrote code that uses ADO to open the MySQL database and iterate through the ubbt_topics, ubbt_posts, and ubbt_users tables to generate all the possible URLs.

I'm sure the resulting XML that is written is correctly formatted and valid, but much to my dismay, after submitting the newly generated sitemap to Google, they aren't indexing ANY of the URLs!

Did you have any problems with Google not indexing the URLs your sitemap generator generates? Did it just take a while for them to get around to indexing them?

The sitemap in question doesn't just include generated URLs for the Threads board, but also from our inventory database and other sources.

If you want to see what my URL's look like, it is located at http://www.myfootshop.com/myfootshopcom-sitemap.xml
It includes over 33,000 URLs, but the first ones in the XML file are the Threads URLs.

Well, theres a couple of hints for you... You should limit your sitemap to 25k urls per page, use a sitemap index if you need to go over.

Theres no need to have your users or posts in a sitemap, all you need is the topic link as the posts will all go here, you'll just be feeding it more data than it needs.

A sitemap won't get all of your stuff crawled overnight; you're simply aiding the SE's discovery bot in finding new content, it can still take months for all of that yummy content to get indexed.

A huge sitemap will cause strain on your server, the sitemap index will definately help ease requests, especially if the sitemap itself isn't cached and builds live, you'll be putting strain on the sql server for having it push out all possible urls.

Also, try avoiding duplicate URLs, once it has a url it'll crawl all possible urls on that page, feeding it say 5 possible urls to the same page is simply going to bog down the request time...

Okay....I have an issue.

I am running 7.5.3 on a Linux server hosted @ godaddy. I am having a problem. When I enable "Spider Friendly search engines" then I get an "access denied" error. I deselect the spider friendly box, and all goes back to normal.

I pulled up my php info box, and I found this reference to path_info:

PATH_INFO /forums/admin/phpinfo_iframe.php

I have looked through the forums to see what I can do to fix this issue, and I have not found anything. I assume that this is on godaddy's end and not through UBB, but I don't know what to ask their tech team to do to fix the issue. Any help you could offer would be appreciated!!

Well, there isn't much you can do if your host doesn't support Path_Info; they can simply add this to your apache user:
AcceptPathInfo On

Or, if for whatever reason they've allowed it, you can add that to a .htaccess file, but this only works if they've allowed you privileges to override their settings (which most hosts won't allow).

I talked to the host, and they indicated that it is ON, and I do have access....now to figure out the .htaccess thing.

Do you have any recommendations on a good place for a beginner to learn how to do this? I will be searching this site and the web also, but if this has been answered before, I would appreciate a nod in the right direction.

I have created a .htaccess file and placed it in the /forums directory. I tried it with only "AcceptPathInfo On" in it, I got a 500 internal server error. I then tried it with only "AllowOverride All" in it, same error. I then moved that file to the admin directory, same result, and also tried it with only "AcceptPathInfo On" in the admin directory, getting the error again....am I doing this wrong??

You want to remove the quotes. Only put this:

AcceptPathInfo On

That is what I tried....without the quotes....any other ideas??

Hmm, do you have access to your webserver error logs? If so, then right after you get the error, I'd check the logs and it should have some info on the error.

If a .htaccess file in your web root throws a 500 error, you likely don't have AllowOverride All set in your account and it'd throwing the error.

Literally, all you need in your .htaccess is:
AcceptPathInfo On

Any apache webserver, with the AllowOverride option on should be able to work witht hat setting...