Site Links
Home
Features
Documentation
Pricing & Order
Members Area
Support Options
UBBDev.com
UBBWiki.com
Who's Online
4 registered (capnbob, id242, johnH, Mike L), 41 Guests and 19 Spiders online.
Key: Admin, Global Mod, Mod
Featured Member
Registered: 04/12/04
Posts: 154
Top Posters (30 Days)
Ruben 49
DennyP 24
Gizmo 23
Dunny 18
SteveS 14
AllenAyres 12
dbremer 10
drkknght00 9
SD 9
driv 8
Latest Photos
OK Corral Shoot Out
Testing
Basildon Train Station
Basildon Town Centre looking from the rounderbout
Basildon Town Square
Page 1 of 2 1 2 >
Topic Options
#180782 - 03/16/07 04:56 PM Rick - Importer Speed for Large Site?
JLester Offline
stranger
Registered: 08/06/06
Posts: 14
Rick,

We have been trying to get everything laid out to do the upgrade from 6.5 to 7.1. My 2 production servers (1 for HTTP and 1 for MySQL) are both dual-Xeon 2.4s with 2 gigabytes of memory and fast hard drives. My test servers are also dual-processor machines, except with 933s and 1 gigabyte of memory each. On my test servers, the importer script took almost exactly 24 hours to complete. That is a LOT of downtime for a site as active as ours. What can we do to allow for a faster conversion?

I thought about doing the import with the forums online, but that would lose any posts that were made during the import. I also tried the modification that did more rows per refresh, but it did not seem to work for me.

We archive posts ever so often, so our active database is not extremely large, but still pretty sizable. The stats are:

Total members: 74330
Total forums: 36
Total topics: 141011
Total posts: 1180436

Any help is appreciated with making the upgrade as smooth as possible for our users.

Thanks,
Jason
_________________________
TheDieselStop.Com
UBB User Since 1999!
Top
Express Hosting
Express Hosting "We are the official hosting company of UBB.threads. Ask us about our free migration services to migrate your UBB.threads installation."
#180812 - 03/16/07 08:54 PM Re: Rick - Importer Speed for Large Site? [Re: JLester]
Ian Offline
Registered: 06/05/06
Posts: 4337
Loc: Essex, UK
24 hours does seem a little long - I think it took around 12 hours to import with just over 1 million for us.

I simply did it overnight - but kept the old forums up with a notice that said any posts made after say 22:00 would not appear on the new upgraded forums.
Top
#180813 - 03/16/07 10:44 PM Re: Rick - Importer Speed for Large Site? [Re: Ian]
Chuck Weinstock Offline
journeyman
Registered: 04/27/04
Posts: 52
I'm working this problem right now. I have an importer from 6.5.5 to 7.1 that did 4,000,000 posts in something like 19 hours. Unfortunately it had a minor bug which I'm fixing now (and then I'll have to test, etc.) My plan is to make it available to the community.

When I tried to import this same table using the normal importer I gave up after 3 days of it running on an empty server.


Edited by Chuck Weinstock (03/16/07 11:22 PM)
Top
#180815 - 03/17/07 02:09 AM Re: Rick - Importer Speed for Large Site? [Re: Chuck Weinstock]
Ian Spence Offline
Registered: 06/04/06
Posts: 255
 Originally Posted By: Chuck Weinstock
I'm working this problem right now. I have an importer from 6.5.5 to 7.1 that did 4,000,000 posts in something like 19 hours. Unfortunately it had a minor bug which I'm fixing now (and then I'll have to test, etc.) My plan is to make it available to the community.

When I tried to import this same table using the normal importer I gave up after 3 days of it running on an empty server.


So was the database on a different server than the importer? If so, I think that could cause the slow import times since the importer is pretty much 30mill queries in your case
Top
#180821 - 03/17/07 09:49 AM Re: Rick - Importer Speed for Large Site? [Re: Ian Spence]
Chuck Weinstock Offline
journeyman
Registered: 04/27/04
Posts: 52
 Originally Posted By: Ian Spence

So was the database on a different server than the importer? If so, I think that could cause the slow import times since the importer is pretty much 30mill queries in your case


Nope. Same server. Just a lot of inserts and updates.
Top
#180826 - 03/17/07 11:53 AM Re: Rick - Importer Speed for Large Site? [Re: Ian Spence]
JLester Offline
stranger
Registered: 08/06/06
Posts: 14
Mine is on a different server.

Jason
_________________________
TheDieselStop.Com
UBB User Since 1999!
Top
#180844 - 03/17/07 05:20 PM Re: Rick - Importer Speed for Large Site? [Re: JLester]
smallufo Offline
enthusiast
Registered: 06/05/06
Posts: 265
Loc: Taiwan
I have the same problem.
That's why I still cannot decide when to upgrade from 6.5.4 to 7.0.2.

I have :
Total topics: 84897
Total posts: 851749

On my test server , the importer script has already run 20+ hours.
And the importer script is running more and more slowly. It is processing 64???th topic now.
I estimate that it'll take the whole two days to finish "topic" import. Not including PM's import , that may be another two or three days ...

My question : Why is the script running more and more slowly ?
At first , it processes 20 topics like fly , but later , it's like a snail... and my HD read/write more and more constantly.

I can't shut down my website for 4 or 5 days to upgrade . Is there any better solution ?

How about the ubb6's db and ubb7's db on different servers , will it help ?
_________________________

English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Top
#180854 - 03/17/07 07:15 PM Re: Rick - Importer Speed for Large Site? [Re: smallufo]
JLester Offline
stranger
Registered: 08/06/06
Posts: 14
Mine slows down as it goes too. I suppose because the tables and indexes keep growing.

Jason
_________________________
TheDieselStop.Com
UBB User Since 1999!
Top
#180859 - 03/18/07 12:58 AM Re: Rick - Importer Speed for Large Site? [Re: JLester]
Rick Offline
Post-a-holic
Registered: 06/04/06
Posts: 10164
Loc: Aberdeen, WA
Hmm, there was an issue with the importer for 7.0.x where it would slow down as it goes, that was due to a bad query. Are you running the very latest importer and importing into 7.1?
Top
#180863 - 03/18/07 05:59 AM Re: Rick - Importer Speed for Large Site? [Re: Rick]
smallufo Offline
enthusiast
Registered: 06/05/06
Posts: 265
Loc: Taiwan
 Originally Posted By: Rick
Are you running the very latest importer and importing into 7.1?

No, I am running 7.0.2 importer. I think I have to upgrade from 6.5.4 -> 7.0.2 -> 7.1
Is it possible to upgrade direct from 6.5.x to 7.1 ? (skip 7.0.x)
_________________________

English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Top
#180868 - 03/18/07 10:50 AM Re: Rick - Importer Speed for Large Site? [Re: smallufo]
Rick Offline
Post-a-holic
Registered: 06/04/06
Posts: 10164
Loc: Aberdeen, WA
Yes, you can upgrade directly from 6.5.x to 7.1. Just grab the latest importer from the members area as it has some changes that should speed things up during the posts stage as well.
Top
#180869 - 03/18/07 10:54 AM Re: Rick - Importer Speed for Large Site? [Re: Rick]
JLester Offline
stranger
Registered: 08/06/06
Posts: 14
I was running the earlier one as well.

Jason
_________________________
TheDieselStop.Com
UBB User Since 1999!
Top
#180885 - 03/18/07 05:00 PM Re: Rick - Importer Speed for Large Site? [Re: smallufo]
Gizmo Offline

Registered: 06/05/06
Posts: 14994
Loc: Portland, OR; USA
You can upgrade from 6.5 to ANY release in the 7 series; upgrading to 7.0.2 is not required at all.
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#180886 - 03/18/07 05:01 PM Re: Rick - Importer Speed for Large Site? [Re: Gizmo]
Gizmo Offline

Registered: 06/05/06
Posts: 14994
Loc: Portland, OR; USA
Again, i really need to make sure theres not more pages after I decide to post on something...
_________________________
Forums: UGN Security & VNC Web Design & Development
UBB.Threads: UBB.Wiki, My UBBSkins, UBB.Sitemaps
Longtime UBB Supporter, UBB Beta Tester & Resident Post-A-Holic.
UBB Modifications, Styling, Coding Services, Disaster Recovery, and more!
Top
#180941 - 03/19/07 01:29 PM Re: Rick - Importer Speed for Large Site? [Re: Rick]
smallufo Offline
enthusiast
Registered: 06/05/06
Posts: 265
Loc: Taiwan
 Originally Posted By: Rick
Yes, you can upgrade directly from 6.5.x to 7.1. Just grab the latest importer from the members area as it has some changes that should speed things up during the posts stage as well.


I grabbed the importer script from members area last night , and ran the importer again.
But I didn't notice any speed improvement till now. It seems the same slow as before ... \:\(
_________________________

English is not my native language.
I try my best to express my thought precisely.
I hope you understand what I mean.
If any misunderstanding results from culture gaps , I apologize first.
Top
Page 1 of 2 1 2 >



Moderator:  AllenAyres, Harold, Ian, Ron M 
Shout Box

Today's Birthdays
No Birthdays
Recent Topics
Ability to "like" individual posts (not Facebook "likes)
by doug
05/23/12 09:03 AM
Island Permissions
by ThreadsUser
05/22/12 03:03 PM
streaming video
by prkrgrp
05/20/12 07:02 PM
New Posts Corrupted? Can someone help?
by PianoWorld
05/19/12 09:41 AM
Custom forum permissions
by ntdoc
05/18/12 02:07 PM
Forum Stats
10489 Members
36 Forums
33841 Topics
181698 Posts

Max Online: 978 @ 06/24/07 11:19 PM
Random Image