I've successfully (I think) imported a forum with 4,000,000+ posts in just a tad over 3 hours. I did this with a modified version of the importer script that requires shell access to the server. There were a number of changes made but the key one, as far as I can tell, was to change the definition of the empty ubbt_POSTS table to not include any of the indexes except the primary key, do the import, and then create the indexes.

Things to watch out for:

1. There is a limit to the size of MyISAM tables. There is a parameter in my.cnf that may need to be increased for tables of this size. I used: myisam_data_pointer_size = 8. (I was burned when the last 9,000 or so posts were not added because of this limitation.)

2. Adding the FULLTEXT INDEX will take a *lot* of time. For large tables if you don't have enough space for temporary files the technique used for building the index can take, literally, days. The parameters of interest are myisam_sort_buffer_size and myisam_max_extra_sort_file_size. I set both of these to something like 10 gigs. With enough space the days were reduced to 30+ minutes.

I hope this helps someone.

Chuck

Thanks for sharing Chuck. Quite the speedup.

I did get your ticket and will reply as soon as possible.

Hi Chuck Weinstock , Rick :
Can you explain more ? I need more detail information...
My importer script has ran more than 3 days , and it is still processing 81xxxth post (total 87xxx). There are still PMs to import , and it may take more time ...

Hi , I grabbed the latest importer yesterday , and ran the importer again . It finished the import in hours , great !!!

Glad to hear it, though I'm not sure if the importers have been updated lately; possibly less resource usage on the server when it ran.

When I did mine it went quick the first couple of tests, but the live import took days... And i'm sure it abandoned a handful of data as well due to timeouts lol

Two years after the last post, I'm curious if there is any way to speed up this import process.

I am trying to import a board that was running an old version of Classic for years and years, so I upgraded it over the weekend to 6.7.3 - which went reasonably smoothly except for the very long downtime to rebuild member post histories, search index....all those things we love about Classic.

Then I installed Threads 7.5.3, downloaded the Classic upgrader from Classic and started an import. Since this is just testing, I left the board running, even after reading the posts warning about that.

The import started just over 24 hours ago and has imported 517,000 posts out of 1,524,939. So we have about 2 days more to go.

I've tried increasing the MAX_POSTS_PER_PAGE, etc; thinking that would get more done on each refresh and avoid the overhead of script startup, but it does not seem to make much difference.

If I shut down the board I suppose it would go somewhat faster with less server load, but even overnight when nobody was online it wasn't much faster. I am on a dedicated server with only one other very low-resource site hosted.

Is there a unix shell version available as mentioned at the top of this thread?

Or, is there any way / mod to only import threads with posts after a given date/time?

Could I:

• shut down the Classic board and back it up (tar...),
• delete all posts on the Classic board made subsequent to the time the test import began
• then re-import again, just picking up the updates? I would have to manually delete those topics in the Threads board before the update, but that would not be so hard.

We really don't want to be down 3 or 4 days for a software upgrade.....

Thanks!

Dave Kandz (rbdigital)

Unfortunately there isn't a really good option for this. We currently don't have a command line version available for this.

One option would be to make a copy of your current classic forum files (the non-cgi files). You'd need to make an anouncement that any new users or posts made after that time until the import is finished will not be carried over.

Then you could run the importer and when it asks for your non-ubb path, you can put in the path to the copy that you to be assured there is no corruption. (One of the possible problems when running the importer on a live forum). It's ok to use the live copy of everything else, just the forum ones are updated quite often so possible problems with those.

Not the best option obviously since A) You'd need to make a copy of the files, so that's quite a bit of space needed and B) you'd have a couple of days where things wouldn't be carried over to the new forum.

Didn't really answer your question about importing items after a current date. That's another "no" at this time, since it requires the users to be imported along with the posts so it maps them correctly.

We do expect to address this once version 8 is finished and we roll out our new importer system, but that's still quite a ways off, since version 8 is still in development.

That si so great

Reading the above, how will the UBB8 importer handle things? I've got a site with 1.5 million posts to convert. Would I be looking at days to convert or hours? (my own dedicated server).