Sorry for the late reply, I've been out of town.
That file is an index of the topics in your forum - you really don't want to delete it
Rebuilding data should delete the indexes before rebuilding them, I don't think it adds on to the old ones or else what would be the purpose for rebuilding?
What's your cache hit percentage? Even in your archive forums I am getting a new page vs one pulled from the cache (the page is an ultimatebb.cgi file vs an ultimatebb.php file).
Your forum is pretty good sized - there's a few things you can do to improve speed while staying on .classic:
- reduce the who's online time from 30 min to 10-15 min.
- set everyone to view all topics (not the last 10 days, 30 days, 90 days, etc) - the idea is to get everyone on the same page pulling from the cached file as much as possible. Members and Search Engine bots will go back and look at the old stuff - making it available to be cached will keep the cache hits up.
- don't allow searches of more than one forum - people should refine their search prior to tagging your server for their own laziness
- your search index would probably work better for one for each forum rather than one big one for the whole site - lots less info to search through at one time.
- I'm sure there are other things I can't think of right now
