Previous Thread
Next Thread
Print Thread
Hop To
Speeding Up Imports
#181013 03/19/2007 8:17 PM
Joined: Apr 2004
Posts: 52
journeyman
OP Offline
journeyman
Joined: Apr 2004
Posts: 52
I've successfully (I think) imported a forum with 4,000,000+ posts in just a tad over 3 hours. I did this with a modified version of the importer script that requires shell access to the server. There were a number of changes made but the key one, as far as I can tell, was to change the definition of the empty ubbt_POSTS table to not include any of the indexes except the primary key, do the import, and then create the indexes.

Things to watch out for:

1. There is a limit to the size of MyISAM tables. There is a parameter in my.cnf that may need to be increased for tables of this size. I used: myisam_data_pointer_size = 8. (I was burned when the last 9,000 or so posts were not added because of this limitation.)

2. Adding the FULLTEXT INDEX will take a *lot* of time. For large tables if you don't have enough space for temporary files the technique used for building the index can take, literally, days. The parameters of interest are myisam_sort_buffer_size and myisam_max_extra_sort_file_size. I set both of these to something like 10 gigs. With enough space the days were reduced to 30+ minutes.

I hope this helps someone.

Chuck

Re: Speeding Up Imports
Chuck Weinstock #181025 03/19/2007 9:58 PM
Joined: Jun 2006
Posts: 9,243
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,243
Thanks for sharing Chuck. Quite the speedup. wink

I did get your ticket and will reply as soon as possible.

Re: Speeding Up Imports
Rick #181281 03/22/2007 5:04 AM
Joined: Jun 2006
Posts: 215
enthusiast
Offline
enthusiast
Joined: Jun 2006
Posts: 215
Hi Chuck Weinstock , Rick :
Can you explain more ? I need more detail information...
My importer script has ran more than 3 days , and it is still processing 81xxxth post (total 87xxx). There are still PMs to import , and it may take more time ...



English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Re: Speeding Up Imports
smallufo #183904 05/06/2007 12:20 PM
Joined: Jun 2006
Posts: 215
enthusiast
Offline
enthusiast
Joined: Jun 2006
Posts: 215
Hi , I grabbed the latest importer yesterday , and ran the importer again . It finished the import in hours , great !!!



English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Re: Speeding Up Imports
smallufo #183910 05/06/2007 8:12 PM
Joined: Jun 2006
Posts: 15,940
Likes: 29
UBB.threads Developer
Online Tapedshut
UBB.threads Developer
Joined: Jun 2006
Posts: 15,940
Likes: 29
Glad to hear it, though I'm not sure if the importers have been updated lately; possibly less resource usage on the server when it ran.

When I did mine it went quick the first couple of tests, but the live import took days... And i'm sure it abandoned a handful of data as well due to timeouts lol


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, I Find Rocks, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Speeding Up Imports
Gizmo #229070 09/07/2009 6:37 PM
Joined: Sep 2009
Posts: 1
stranger
Offline
stranger
Joined: Sep 2009
Posts: 1
Two years after the last post, I'm curious if there is any way to speed up this import process.

I am trying to import a board that was running an old version of Classic for years and years, so I upgraded it over the weekend to 6.7.3 - which went reasonably smoothly except for the very long downtime to rebuild member post histories, search index....all those things we love about Classic. wink

Then I installed Threads 7.5.3, downloaded the Classic upgrader from Classic and started an import. Since this is just testing, I left the board running, even after reading the posts warning about that.

The import started just over 24 hours ago and has imported 517,000 posts out of 1,524,939. So we have about 2 days more to go. cry

I've tried increasing the MAX_POSTS_PER_PAGE, etc; thinking that would get more done on each refresh and avoid the overhead of script startup, but it does not seem to make much difference.

If I shut down the board I suppose it would go somewhat faster with less server load, but even overnight when nobody was online it wasn't much faster. I am on a dedicated server with only one other very low-resource site hosted.

Is there a unix shell version available as mentioned at the top of this thread?

Or, is there any way / mod to only import threads with posts after a given date/time?

Could I:
  • shut down the Classic board and back it up (tar...),
  • delete all posts on the Classic board made subsequent to the time the test import began
  • then re-import again, just picking up the updates? I would have to manually delete those topics in the Threads board before the update, but that would not be so hard.


We really don't want to be down 3 or 4 days for a software upgrade.....

Thanks!

Dave Kandz (rbdigital)


Re: Speeding Up Imports
RBDigital #229071 09/07/2009 7:09 PM
Joined: Jun 2006
Posts: 9,243
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,243
Unfortunately there isn't a really good option for this. We currently don't have a command line version available for this.

One option would be to make a copy of your current classic forum files (the non-cgi files). You'd need to make an anouncement that any new users or posts made after that time until the import is finished will not be carried over.

Then you could run the importer and when it asks for your non-ubb path, you can put in the path to the copy that you to be assured there is no corruption. (One of the possible problems when running the importer on a live forum). It's ok to use the live copy of everything else, just the forum ones are updated quite often so possible problems with those.

Not the best option obviously since A) You'd need to make a copy of the files, so that's quite a bit of space needed and B) you'd have a couple of days where things wouldn't be carried over to the new forum.

Re: Speeding Up Imports
Rick #229072 09/07/2009 7:11 PM
Joined: Jun 2006
Posts: 9,243
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,243
Didn't really answer your question about importing items after a current date. That's another "no" at this time, since it requires the users to be imported along with the posts so it maps them correctly.

We do expect to address this once version 8 is finished and we roll out our new importer system, but that's still quite a ways off, since version 8 is still in development.

Re: Speeding Up Imports
Chuck Weinstock #232606 12/23/2009 8:01 AM
Joined: Dec 2009
Posts: 1
stranger
Offline
stranger
Joined: Dec 2009
Posts: 1
That si so great

Re: Speeding Up Imports
karodal #232749 12/29/2009 9:59 AM
Joined: Jun 2006
Posts: 33
newbie
Offline
newbie
Joined: Jun 2006
Posts: 33
Reading the above, how will the UBB8 importer handle things? I've got a site with 1.5 million posts to convert. Would I be looking at days to convert or hours? (my own dedicated server).

Re: Speeding Up Imports
Monkeyra #232751 12/29/2009 10:24 AM
Joined: Jun 2006
Posts: 9,243
Former Developer
Offline
Former Developer
Joined: Jun 2006
Posts: 9,243
The only real slow importer at this point is the one for UBB.classic when there are a large number of posts since it has to chug through so many flat files.

Any importer that we have that grabs it's data from a database is much faster, so you'd be looking at hours, not days wink Plus, since you're on a dedicated, you can adjust the number of posts/users/topics it handles per refresh. The default is only around 20, since we have to play nicely with shared servers.

Re: Speeding Up Imports
Rick #232754 12/29/2009 1:42 PM
Joined: Jun 2006
Posts: 33
newbie
Offline
newbie
Joined: Jun 2006
Posts: 33
Sweet smile

Re: Speeding Up Imports
Monkeyra #232757 12/29/2009 3:50 PM
Joined: Jul 2006
Posts: 2,143
Pooh-Bah
Offline
Pooh-Bah
Joined: Jul 2006
Posts: 2,143
Remind me to tell you about the time I imported a .classic with over a million posts into a .threads board..........


This thread for sale. Click here! [Linked Image from navaho.infopop.cc]
Re: Speeding Up Imports
David Dreezer #233165 01/03/2010 3:32 PM
Joined: Jun 2006
Posts: 33
newbie
Offline
newbie
Joined: Jun 2006
Posts: 33
Take a while? lol


Link Copied to Clipboard
ShoutChat Box
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
Followed Threads
by FREAK - 07/13/2020 9:13 PM
Can't Delete or Add New Forum
by FREAK - 07/13/2020 8:25 PM
Non-Admin users can't see attached pictures
by jjjjj - 07/11/2020 8:09 PM
[NOTABUG] Report Post bug..
by Morgan - 07/05/2020 1:48 PM
Trial Subscription question.
by Ruben - 07/03/2020 2:34 PM
Who's Online Now
3 members (Gizmo, FREAK, 1 invisible), 83 guests, and 43 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Amusing Terain Scenics
Amusing Terain Scenics
by isaac, August 19
Sky places
Sky places
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 7.7.4