I'm running the migration script to copy all the topics, users, etc. from our old ultimatebb board over to the new ubbthreads board. We have a *lot* of data to move. The script has been running for almost three hours now and it's probably just over half done.

Meanwhile our old board is still up and running, with new users registering and new posts and replies being created.

What happens if I run the migration script a second time? Is it smart enough to skip data already migrated? Will it create tons of duplicates? Will it refuse to run?

if you run it again, it'll re-migrate everything from the beginning, you'll have duplicates.

It is recommended to close up shop when migrating.

Yikes! That's not an option. We have a huge, very busy board with 24/7 activity. The script has been running since 5:30PM CDT, and it's little more than half done. You're saying I have to shut down for half a day at a minimum? Ouch!

Is there no other option?

I have no clue how you could edit the import script to do anything more than what it does; we've all had to do the same thing to import up, so sadly it's the only path I can recommend for you.

there is an option. hire an expert.

someone who'd probably let your script run it's course, somehow getting a timestamp for time critical stuff (posts, topics, pms -- essentially everything that you MUST preserve)

then after the import, close the board for a short period to grab all the stuff that wasn't gotten after the import.

it's more than that though, because you wouldn't only be searching for NEW things that have been added.. but also things that might have been deleted in that time period as well.

in short, it's messy, but with care the transitional process could be mitigated to an acceptable data transformation.

bottom line: if you want your board to be imported 'as is' , then close it for half the day and just let it rip. otherwise hire someone to get you as close as is reasonable, while you let the board be 'live'. i'd lean toward board closure and save yourself what it costs to hire it done.. but that's just me.

below botton line: also keep in mind that importing while the board is live can cause problems too.. ie: consider that you might import a topic ID, but not the posts, since they are asyncronous.. or vice versa.. not good

Well, I'm somewhat of an expert myself, but I think you're right. The most logical way to do it is to close the board. I *could* edit the import script or write my own to import everything past a certain timestamp, but why bother?

Just means I'll have to drop and recreate the db when I ready to go live and rerun everything, but first I'll have to announce the downtime and prepare everyone for it. I'll play around with the present setup to discover any other gotchas and prep the owners for the inevitable downtime.

Thanks for the input.

one thing i learned from doing a very LARGE db, was to trial run the import first.

there are pitfalls that come up with very HUGE post tables for example. things like mysql running out of temporary storage space or the like.

now your hosting environment might be such as to not have issues, but mine did. so i just did a DB backup.. zipped it up.. and loaded it locally on my windows box (running wamp).. and tried the import/upgrade.

it was good to do that, just to get a feel for what might befall me on things like an altertable on a POSTS table that approached 1gb in size..

So you might at least give that a try to see how it comes out, then prepare your members for the REAL deal and let it rip..

oh and i actually found that it was faster for me to do the following:

1. close the real board
2. do a backup (using cmd line) to a .gz file
3. grab it to my local system
4. do the upgrade
5. backup the file locally (.gz)
6. send to live board
7. unzip and import fresh
8. open board

reason being, my local computer is a beast in terms of CPU, Ram and the like (QX6850, Raid 5 Raptors @ 10K RPM, bla bla).. it churned thru the process in nothing flat.

in the end, the process took less time than running the upgrade script on the real host on a closed board.

your mileage may vary..

Good luck to you,
SD

I think UBB should provide like a separate SQL script for the update. The problem is that some of us have REALLY large Databases with UBB and the script is
1. Crashing the server allocating too many resources to MySQL
or
2. Taking infinitely long.

I did an import this past week that took about 8 hours, it really depends on your forum size. If you have a nice server it's possible to bump up the number of topics pulled in at one time, but you'll have to take responsibility if something happens and you have to re-import. I've doubled the number on a generous server without issue tho.

Another way you can do this would be to close registration the day of so your member data won't get corrupted, warn people to not edit their profiles once import starts. Open a temp forum for them to post in and close the rest. Do the import and as it gets toward the end close the forum completely until it completes. A bit risky for data corruption, but if you keep tight watch on the process it's not too bad.

Myself - I tell my users to go get some fresh air, I'll be finished when they get back... after all, it's still just a forum - pay attention to the family for a few hours and everyone will benefit

Hello Allen,
The problem is not in the social part of the process.
The problem is that a script like the upgrade of the message board, which runs over the apache and updates a large mysqld database,will block for 8 hours all the websites!
Last time I had to do my upgrade at night and it took 4 hours. During that time, the mysql of any other site on the host did not respond, and the server had high loads of over 15.
All this because of the Upgrade process!

Server? Xeon 1 Gb Ram

There's some issues with the server then... really, the script is dumbed down quite a bit to run on shared host platforms. As long as the forum is turned off, the amount of server load should be less than the forum running at other times normally with users posting and etc.

Now, you mention the upgrade tool, are we still discussing the importer or the upgrade? Upgrading is a bit different and for some versions being upgraded it's possible to tank a large forum when indexes are being added/ updated/ deleted. The original topic is regarding the importer tho.

In such a situation you could use the method that Sirdude described here, make a backup, run your importer locally, and transfer that data to the actual server when done.