Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online Now
0 registered members (), 63 guests, and 388 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
Morgan
Morgan
Sweden
Posts: 518
Joined: June 2006
Show All Member Profiles 
Top Posters(30 Days)
Gizmo 17
M4TT 12
FREAK 8
Ruben 8
mmkk 4
isaac 4
Latest Photos
Chinese Buddhist temple.
My buddha beads.
Rendered Walls
Multi-Screen wallpaper
Stockholm Metro
Previous Thread
Next Thread
Print Thread
Speeding Up Imports #181013
03/19/07 07:17 PM
03/19/07 07:17 PM
Joined: Apr 2004
Posts: 52
C
Chuck Weinstock Offline OP
journeyman
Chuck Weinstock  Offline OP
journeyman
C
Joined: Apr 2004
Posts: 52
I've successfully (I think) imported a forum with 4,000,000+ posts in just a tad over 3 hours. I did this with a modified version of the importer script that requires shell access to the server. There were a number of changes made but the key one, as far as I can tell, was to change the definition of the empty ubbt_POSTS table to not include any of the indexes except the primary key, do the import, and then create the indexes.

Things to watch out for:

1. There is a limit to the size of MyISAM tables. There is a parameter in my.cnf that may need to be increased for tables of this size. I used: myisam_data_pointer_size = 8. (I was burned when the last 9,000 or so posts were not added because of this limitation.)

2. Adding the FULLTEXT INDEX will take a *lot* of time. For large tables if you don't have enough space for temporary files the technique used for building the index can take, literally, days. The parameters of interest are myisam_sort_buffer_size and myisam_max_extra_sort_file_size. I set both of these to something like 10 gigs. With enough space the days were reduced to 30+ minutes.

I hope this helps someone.

Chuck

Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
Re: Speeding Up Imports [Re: Chuck Weinstock] #181025
03/19/07 08:58 PM
03/19/07 08:58 PM
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
R
Rick Offline
Former Developer
Rick  Offline
Former Developer
R
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
Thanks for sharing Chuck. Quite the speedup. wink

I did get your ticket and will reply as soon as possible.

Re: Speeding Up Imports [Re: Rick] #181281
03/22/07 04:04 AM
03/22/07 04:04 AM
Joined: Jun 2006
Posts: 266
Taiwan
smallufo Offline
enthusiast
smallufo  Offline
enthusiast
Joined: Jun 2006
Posts: 266
Taiwan
Hi Chuck Weinstock , Rick :
Can you explain more ? I need more detail information...
My importer script has ran more than 3 days , and it is still processing 81xxxth post (total 87xxx). There are still PMs to import , and it may take more time ...



English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Re: Speeding Up Imports [Re: smallufo] #183904
05/06/07 11:20 AM
05/06/07 11:20 AM
Joined: Jun 2006
Posts: 266
Taiwan
smallufo Offline
enthusiast
smallufo  Offline
enthusiast
Joined: Jun 2006
Posts: 266
Taiwan
Hi , I grabbed the latest importer yesterday , and ran the importer again . It finished the import in hours , great !!!



English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Re: Speeding Up Imports [Re: smallufo] #183910
05/06/07 07:12 PM
05/06/07 07:12 PM
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Gizmo Offline
UBB.threads Developer
Gizmo  Offline
UBB.threads Developer
Joined: Jun 2006
Posts: 16,788
Portland, OR; USA
Glad to hear it, though I'm not sure if the importers have been updated lately; possibly less resource usage on the server when it ran.

When I did mine it went quick the first couple of tests, but the live import took days... And i'm sure it abandoned a handful of data as well due to timeouts lol


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Forums: A Gardeners Forum Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Re: Speeding Up Imports [Re: Gizmo] #229070
09/07/09 05:37 PM
09/07/09 05:37 PM
Joined: Sep 2009
Posts: 3
St. Petersburg, FL USA
RBDigital Offline
stranger
RBDigital  Offline
stranger
Joined: Sep 2009
Posts: 3
St. Petersburg, FL USA
Two years after the last post, I'm curious if there is any way to speed up this import process.

I am trying to import a board that was running an old version of Classic for years and years, so I upgraded it over the weekend to 6.7.3 - which went reasonably smoothly except for the very long downtime to rebuild member post histories, search index....all those things we love about Classic. wink

Then I installed Threads 7.5.3, downloaded the Classic upgrader from Classic and started an import. Since this is just testing, I left the board running, even after reading the posts warning about that.

The import started just over 24 hours ago and has imported 517,000 posts out of 1,524,939. So we have about 2 days more to go. cry

I've tried increasing the MAX_POSTS_PER_PAGE, etc; thinking that would get more done on each refresh and avoid the overhead of script startup, but it does not seem to make much difference.

If I shut down the board I suppose it would go somewhat faster with less server load, but even overnight when nobody was online it wasn't much faster. I am on a dedicated server with only one other very low-resource site hosted.

Is there a unix shell version available as mentioned at the top of this thread?

Or, is there any way / mod to only import threads with posts after a given date/time?

Could I:
  • shut down the Classic board and back it up (tar...),
  • delete all posts on the Classic board made subsequent to the time the test import began
  • then re-import again, just picking up the updates? I would have to manually delete those topics in the Threads board before the update, but that would not be so hard.


We really don't want to be down 3 or 4 days for a software upgrade.....

Thanks!

Dave Kandz (rbdigital)


Re: Speeding Up Imports [Re: RBDigital] #229071
09/07/09 06:09 PM
09/07/09 06:09 PM
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
R
Rick Offline
Former Developer
Rick  Offline
Former Developer
R
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
Unfortunately there isn't a really good option for this. We currently don't have a command line version available for this.

One option would be to make a copy of your current classic forum files (the non-cgi files). You'd need to make an anouncement that any new users or posts made after that time until the import is finished will not be carried over.

Then you could run the importer and when it asks for your non-ubb path, you can put in the path to the copy that you to be assured there is no corruption. (One of the possible problems when running the importer on a live forum). It's ok to use the live copy of everything else, just the forum ones are updated quite often so possible problems with those.

Not the best option obviously since A) You'd need to make a copy of the files, so that's quite a bit of space needed and B) you'd have a couple of days where things wouldn't be carried over to the new forum.

Re: Speeding Up Imports [Re: Rick] #229072
09/07/09 06:11 PM
09/07/09 06:11 PM
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
R
Rick Offline
Former Developer
Rick  Offline
Former Developer
R
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
Didn't really answer your question about importing items after a current date. That's another "no" at this time, since it requires the users to be imported along with the posts so it maps them correctly.

We do expect to address this once version 8 is finished and we roll out our new importer system, but that's still quite a ways off, since version 8 is still in development.

Re: Speeding Up Imports [Re: Chuck Weinstock] #232606
12/23/09 07:01 AM
12/23/09 07:01 AM
Joined: Dec 2009
Posts: 5
K
karodal Offline
stranger
karodal  Offline
stranger
K
Joined: Dec 2009
Posts: 5
That si so great

Re: Speeding Up Imports [Re: karodal] #232749
12/29/09 08:59 AM
12/29/09 08:59 AM
Joined: Jun 2006
Posts: 34
London
Monkeyra Offline
newbie
Monkeyra  Offline
newbie
Joined: Jun 2006
Posts: 34
London
Reading the above, how will the UBB8 importer handle things? I've got a site with 1.5 million posts to convert. Would I be looking at days to convert or hours? (my own dedicated server).

Re: Speeding Up Imports [Re: Monkeyra] #232751
12/29/09 09:24 AM
12/29/09 09:24 AM
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
R
Rick Offline
Former Developer
Rick  Offline
Former Developer
R
Joined: Jun 2006
Posts: 10,177
Aberdeen, WA
The only real slow importer at this point is the one for UBB.classic when there are a large number of posts since it has to chug through so many flat files.

Any importer that we have that grabs it's data from a database is much faster, so you'd be looking at hours, not days wink Plus, since you're on a dedicated, you can adjust the number of posts/users/topics it handles per refresh. The default is only around 20, since we have to play nicely with shared servers.

Re: Speeding Up Imports [Re: Rick] #232754
12/29/09 12:42 PM
12/29/09 12:42 PM
Joined: Jun 2006
Posts: 34
London
Monkeyra Offline
newbie
Monkeyra  Offline
newbie
Joined: Jun 2006
Posts: 34
London
Sweet smile

Re: Speeding Up Imports [Re: Monkeyra] #232757
12/29/09 02:50 PM
12/29/09 02:50 PM
Joined: Jul 2006
Posts: 2,199
David Dreezer Offline

Pooh-Bah
David Dreezer  Offline

Pooh-Bah
Joined: Jul 2006
Posts: 2,199
Remind me to tell you about the time I imported a .classic with over a million posts into a .threads board..........


This thread for sale. Click here! [Linked Image]
Re: Speeding Up Imports [Re: David Dreezer] #233165
01/03/10 02:32 PM
01/03/10 02:32 PM
Joined: Jun 2006
Posts: 34
London
Monkeyra Offline
newbie
Monkeyra  Offline
newbie
Joined: Jun 2006
Posts: 34
London
Take a while? lol


Shout Box
Today's Birthdays
No Birthdays
Recent Topics
Users Unable to Upload Avatar [Not a Bug]
by M4TT. 12/13/17 08:51 AM
Shout Box Sound Effect
by M4TT. 11/29/17 08:28 PM
Ad island
by TGCsanderson. 11/25/17 06:41 PM
Taking to long to connect to DB
by AstroCat. 11/24/17 12:34 PM
Forum Statistics
Forums36
Topics35,015
Posts190,544
Members12,045
Most Online978
Jun 24th, 2007
Random Image
Powered by UBB.threads™ PHP Forum Software 7.6.1
(Snapshot build 20171106)