Previous Thread
Next Thread
Print Thread
Hop To
#181013 03/19/2007 8:17 PM
Joined: Apr 2004
Posts: 52
C
journeyman
journeyman
C Offline
Joined: Apr 2004
Posts: 52
I've successfully (I think) imported a forum with 4,000,000+ posts in just a tad over 3 hours. I did this with a modified version of the importer script that requires shell access to the server. There were a number of changes made but the key one, as far as I can tell, was to change the definition of the empty ubbt_POSTS table to not include any of the indexes except the primary key, do the import, and then create the indexes.

Things to watch out for:

1. There is a limit to the size of MyISAM tables. There is a parameter in my.cnf that may need to be increased for tables of this size. I used: myisam_data_pointer_size = 8. (I was burned when the last 9,000 or so posts were not added because of this limitation.)

2. Adding the FULLTEXT INDEX will take a *lot* of time. For large tables if you don't have enough space for temporary files the technique used for building the index can take, literally, days. The parameters of interest are myisam_sort_buffer_size and myisam_max_extra_sort_file_size. I set both of these to something like 10 gigs. With enough space the days were reduced to 30+ minutes.

I hope this helps someone.

Chuck

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Thanks for sharing Chuck. Quite the speedup. wink

I did get your ticket and will reply as soon as possible.

Rick #181281 03/22/2007 5:04 AM
Joined: Jun 2006
Posts: 215
enthusiast
enthusiast
Joined: Jun 2006
Posts: 215
Hi Chuck Weinstock , Rick :
Can you explain more ? I need more detail information...
My importer script has ran more than 3 days , and it is still processing 81xxxth post (total 87xxx). There are still PMs to import , and it may take more time ...


English is not my native language. I try my best to express my thought precisely. I hope you understand what I mean. If any misunderstanding results from culture gaps, I apologize first.
Joined: Jun 2006
Posts: 215
enthusiast
enthusiast
Joined: Jun 2006
Posts: 215
Hi , I grabbed the latest importer yesterday , and ran the importer again . It finished the import in hours , great !!!


English is not my native language. I try my best to express my thought precisely. I hope you understand what I mean. If any misunderstanding results from culture gaps, I apologize first.
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Glad to hear it, though I'm not sure if the importers have been updated lately; possibly less resource usage on the server when it ran.

When I did mine it went quick the first couple of tests, but the live import took days... And i'm sure it abandoned a handful of data as well due to timeouts lol


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Sep 2009
Posts: 1
stranger
stranger
Joined: Sep 2009
Posts: 1
Two years after the last post, I'm curious if there is any way to speed up this import process.

I am trying to import a board that was running an old version of Classic for years and years, so I upgraded it over the weekend to 6.7.3 - which went reasonably smoothly except for the very long downtime to rebuild member post histories, search index....all those things we love about Classic. wink

Then I installed Threads 7.5.3, downloaded the Classic upgrader from Classic and started an import. Since this is just testing, I left the board running, even after reading the posts warning about that.

The import started just over 24 hours ago and has imported 517,000 posts out of 1,524,939. So we have about 2 days more to go. cry

I've tried increasing the MAX_POSTS_PER_PAGE, etc; thinking that would get more done on each refresh and avoid the overhead of script startup, but it does not seem to make much difference.

If I shut down the board I suppose it would go somewhat faster with less server load, but even overnight when nobody was online it wasn't much faster. I am on a dedicated server with only one other very low-resource site hosted.

Is there a unix shell version available as mentioned at the top of this thread?

Or, is there any way / mod to only import threads with posts after a given date/time?

Could I:
  • shut down the Classic board and back it up (tar...),
  • delete all posts on the Classic board made subsequent to the time the test import began
  • then re-import again, just picking up the updates? I would have to manually delete those topics in the Threads board before the update, but that would not be so hard.


We really don't want to be down 3 or 4 days for a software upgrade.....

Thanks!

Dave Kandz (rbdigital)


Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Unfortunately there isn't a really good option for this. We currently don't have a command line version available for this.

One option would be to make a copy of your current classic forum files (the non-cgi files). You'd need to make an anouncement that any new users or posts made after that time until the import is finished will not be carried over.

Then you could run the importer and when it asks for your non-ubb path, you can put in the path to the copy that you to be assured there is no corruption. (One of the possible problems when running the importer on a live forum). It's ok to use the live copy of everything else, just the forum ones are updated quite often so possible problems with those.

Not the best option obviously since A) You'd need to make a copy of the files, so that's quite a bit of space needed and B) you'd have a couple of days where things wouldn't be carried over to the new forum.

Rick #229072 09/07/2009 7:11 PM
Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
Didn't really answer your question about importing items after a current date. That's another "no" at this time, since it requires the users to be imported along with the posts so it maps them correctly.

We do expect to address this once version 8 is finished and we roll out our new importer system, but that's still quite a ways off, since version 8 is still in development.

Joined: Dec 2009
Posts: 1
K
stranger
stranger
K Offline
Joined: Dec 2009
Posts: 1
That si so great

Joined: Jun 2006
Posts: 33
newbie
newbie
Joined: Jun 2006
Posts: 33
Reading the above, how will the UBB8 importer handle things? I've got a site with 1.5 million posts to convert. Would I be looking at days to convert or hours? (my own dedicated server).

Joined: Jun 2006
Posts: 9,242
Likes: 1
R
Former Developer
Former Developer
R Offline
Joined: Jun 2006
Posts: 9,242
Likes: 1
The only real slow importer at this point is the one for UBB.classic when there are a large number of posts since it has to chug through so many flat files.

Any importer that we have that grabs it's data from a database is much faster, so you'd be looking at hours, not days wink Plus, since you're on a dedicated, you can adjust the number of posts/users/topics it handles per refresh. The default is only around 20, since we have to play nicely with shared servers.

Rick #232754 12/29/2009 1:42 PM
Joined: Jun 2006
Posts: 33
newbie
newbie
Joined: Jun 2006
Posts: 33
Sweet smile

Joined: Jul 2006
Posts: 2,143
Pooh-Bah
Pooh-Bah
Joined: Jul 2006
Posts: 2,143
Remind me to tell you about the time I imported a .classic with over a million posts into a .threads board..........


This thread for sale. Click here! [Linked Image from navaho.infopop.cc]
Joined: Jun 2006
Posts: 33
newbie
newbie
Joined: Jun 2006
Posts: 33
Take a while? lol


Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
1 members (Havenofsobriety), 522 guests, and 99 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)