Previous Thread
Next Thread
Print Thread
Hop To
Speeding Up Imports #181013 03/19/2007 8:17 PM
Joined: Apr 2004
Posts: 52
Chuck Weinstock Offline OP
journeyman
OP Offline
journeyman
Joined: Apr 2004
Posts: 52
I've successfully (I think) imported a forum with 4,000,000+ posts in just a tad over 3 hours. I did this with a modified version of the importer script that requires shell access to the server. There were a number of changes made but the key one, as far as I can tell, was to change the definition of the empty ubbt_POSTS table to not include any of the indexes except the primary key, do the import, and then create the indexes.

Things to watch out for:

1. There is a limit to the size of MyISAM tables. There is a parameter in my.cnf that may need to be increased for tables of this size. I used: myisam_data_pointer_size = 8. (I was burned when the last 9,000 or so posts were not added because of this limitation.)

2. Adding the FULLTEXT INDEX will take a *lot* of time. For large tables if you don't have enough space for temporary files the technique used for building the index can take, literally, days. The parameters of interest are myisam_sort_buffer_size and myisam_max_extra_sort_file_size. I set both of these to something like 10 gigs. With enough space the days were reduced to 30+ minutes.

I hope this helps someone.

Chuck

Express Hosting
Re: Speeding Up Imports [Re: Chuck Weinstock] #181025 03/19/2007 9:58 PM
Joined: Jun 2006
Posts: 9,246
Rick Offline
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,246
Thanks for sharing Chuck. Quite the speedup. wink

I did get your ticket and will reply as soon as possible.

Re: Speeding Up Imports [Re: Rick] #181281 03/22/2007 5:04 AM
Joined: Jun 2006
Posts: 217
smallufo Offline
enthusiast
Offline
enthusiast
Joined: Jun 2006
Posts: 217
Hi Chuck Weinstock , Rick :
Can you explain more ? I need more detail information...
My importer script has ran more than 3 days , and it is still processing 81xxxth post (total 87xxx). There are still PMs to import , and it may take more time ...



English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Re: Speeding Up Imports [Re: smallufo] #183904 05/06/2007 12:20 PM
Joined: Jun 2006
Posts: 217
smallufo Offline
enthusiast
Offline
enthusiast
Joined: Jun 2006
Posts: 217
Hi , I grabbed the latest importer yesterday , and ran the importer again . It finished the import in hours , great !!!



English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Re: Speeding Up Imports [Re: smallufo] #183910 05/06/2007 8:12 PM
Joined: Jun 2006
Posts: 15,850
Gizmo Offline
UBB.threads Developer
Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 15,850
Glad to hear it, though I'm not sure if the importers have been updated lately; possibly less resource usage on the server when it ran.

When I did mine it went quick the first couple of tests, but the live import took days... And i'm sure it abandoned a handful of data as well due to timeouts lol


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Need to Upgrade?
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Speeding Up Imports [Re: Gizmo] #229070 09/07/2009 6:37 PM
Joined: Sep 2009
Posts: 1
RBDigital Offline
stranger
Offline
stranger
Joined: Sep 2009
Posts: 1
Two years after the last post, I'm curious if there is any way to speed up this import process.

I am trying to import a board that was running an old version of Classic for years and years, so I upgraded it over the weekend to 6.7.3 - which went reasonably smoothly except for the very long downtime to rebuild member post histories, search index....all those things we love about Classic. wink

Then I installed Threads 7.5.3, downloaded the Classic upgrader from Classic and started an import. Since this is just testing, I left the board running, even after reading the posts warning about that.

The import started just over 24 hours ago and has imported 517,000 posts out of 1,524,939. So we have about 2 days more to go. cry

I've tried increasing the MAX_POSTS_PER_PAGE, etc; thinking that would get more done on each refresh and avoid the overhead of script startup, but it does not seem to make much difference.

If I shut down the board I suppose it would go somewhat faster with less server load, but even overnight when nobody was online it wasn't much faster. I am on a dedicated server with only one other very low-resource site hosted.

Is there a unix shell version available as mentioned at the top of this thread?

Or, is there any way / mod to only import threads with posts after a given date/time?

Could I:
  • shut down the Classic board and back it up (tar...),
  • delete all posts on the Classic board made subsequent to the time the test import began
  • then re-import again, just picking up the updates? I would have to manually delete those topics in the Threads board before the update, but that would not be so hard.


We really don't want to be down 3 or 4 days for a software upgrade.....

Thanks!

Dave Kandz (rbdigital)


Re: Speeding Up Imports [Re: RBDigital] #229071 09/07/2009 7:09 PM
Joined: Jun 2006
Posts: 9,246
Rick Offline
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,246
Unfortunately there isn't a really good option for this. We currently don't have a command line version available for this.

One option would be to make a copy of your current classic forum files (the non-cgi files). You'd need to make an anouncement that any new users or posts made after that time until the import is finished will not be carried over.

Then you could run the importer and when it asks for your non-ubb path, you can put in the path to the copy that you to be assured there is no corruption. (One of the possible problems when running the importer on a live forum). It's ok to use the live copy of everything else, just the forum ones are updated quite often so possible problems with those.

Not the best option obviously since A) You'd need to make a copy of the files, so that's quite a bit of space needed and B) you'd have a couple of days where things wouldn't be carried over to the new forum.

Re: Speeding Up Imports [Re: Rick] #229072 09/07/2009 7:11 PM
Joined: Jun 2006
Posts: 9,246
Rick Offline
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,246
Didn't really answer your question about importing items after a current date. That's another "no" at this time, since it requires the users to be imported along with the posts so it maps them correctly.

We do expect to address this once version 8 is finished and we roll out our new importer system, but that's still quite a ways off, since version 8 is still in development.

Re: Speeding Up Imports [Re: Chuck Weinstock] #232606 12/23/2009 8:01 AM
Joined: Dec 2009
Posts: 1
karodal Offline
stranger
Offline
stranger
Joined: Dec 2009
Posts: 1
That si so great

Re: Speeding Up Imports [Re: karodal] #232749 12/29/2009 9:59 AM
Joined: Jun 2006
Posts: 33
Monkeyra Offline
newbie
Offline
newbie
Joined: Jun 2006
Posts: 33
Reading the above, how will the UBB8 importer handle things? I've got a site with 1.5 million posts to convert. Would I be looking at days to convert or hours? (my own dedicated server).

Re: Speeding Up Imports [Re: Monkeyra] #232751 12/29/2009 10:24 AM
Joined: Jun 2006
Posts: 9,246
Rick Offline
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,246
The only real slow importer at this point is the one for UBB.classic when there are a large number of posts since it has to chug through so many flat files.

Any importer that we have that grabs it's data from a database is much faster, so you'd be looking at hours, not days wink Plus, since you're on a dedicated, you can adjust the number of posts/users/topics it handles per refresh. The default is only around 20, since we have to play nicely with shared servers.

Re: Speeding Up Imports [Re: Rick] #232754 12/29/2009 1:42 PM
Joined: Jun 2006
Posts: 33
Monkeyra Offline
newbie
Offline
newbie
Joined: Jun 2006
Posts: 33
Sweet smile

Re: Speeding Up Imports [Re: Monkeyra] #232757 12/29/2009 3:50 PM
Joined: Jul 2006
Posts: 2,144
David Dreezer Offline
Pooh-Bah
Offline
Pooh-Bah
Joined: Jul 2006
Posts: 2,144
Remind me to tell you about the time I imported a .classic with over a million posts into a .threads board..........


This thread for sale. Click here! [Linked Image from navaho.infopop.cc]
Re: Speeding Up Imports [Re: David Dreezer] #233165 01/03/2010 3:32 PM
Joined: Jun 2006
Posts: 33
Monkeyra Offline
newbie
Offline
newbie
Joined: Jun 2006
Posts: 33
Take a while? lol


Forum Search
ShoutChat Box
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
File Manager and Image Uploads
by Ty Griffin - 08/19/2019 4:48 PM
Minimum Post Character Count
by M4TT - 08/19/2019 1:58 PM
[FIXED for 7.7.4] Strange Merge Issue
by Baldeagle - 08/16/2019 3:37 PM
Formatting Links
by Ty Griffin - 08/13/2019 4:39 PM
Who's Online Now
2 registered members (JAISP, isaac), 44 guests, and 211 spiders.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Photos
Stones
Amusing Terain Scenics
Sky places
Work spaces
Powered by UBB.threads™ PHP Forum Software 7.7.3