I am having a general intermittent problem where my UBB becomes very very slow at times during the day. Load avg go from below 1 to upwards of 25-30. New posts can take up to 3 mins before they get posted.

I am running UBB 6.7.0. The accelerator and cache are in use. I have be running UBB since 1998 and have never seen my machine become so bogged down.

Do you think this is all load dependent or might I have something not configured properly?

http://www.ibsgroup.org/cgi-local/ubbcgi/ultimatebb.cgi


'ps' and 'top' commands do show an enormous amount of perl activity.

This is the kind of thing I am seeing in 'top' when the slow down occurs:

15:01:50 up 5:03, 1 user, load average: 7.19, 6.37, 5.22
64 processes: 53 sleeping, 11 running, 0 zombie, 0 stopped
CPU states: 95.8% user 4.1% system 0.0% nice 0.0% iowait 0.0% idle
Mem: 512588k av, 412252k used, 100336k free, 0k shrd, 45052k buff
335608k actv, 48400k in_d, 4788k in_c
Swap: 1028148k av, 0k used, 1028148k free 234280k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
10097 nobody 23 0 11520 11M 1720 R 11.9 2.2 0:02 0 ultimatebb.cg
10100 nobody 23 0 10080 9.8M 1716 R 11.9 1.9 0:02 0 ultimatebb.cg
10107 nobody 21 0 8952 8952 1712 R 11.9 1.7 0:00 0 ultimatebb.cg
10094 nobody 25 0 17936 17M 1720 R 11.5 3.4 0:04 0 ultimatebb.cg
10103 nobody 22 0 9976 9976 1720 R 9.9 1.9 0:01 0 ultimatebb.cg
10104 nobody 22 0 10336 10M 1724 R 9.9 2.0 0:01 0 ultimatebb.cg
10105 nobody 22 0 10116 9.9M 1724 R 9.9 1.9 0:01 0 ultimatebb.cg
10106 nobody 21 0 8576 8576 1708 R 9.9 1.6 0:00 0 ultimatebb.cg
10108 nobody 21 0 3424 3424 1612 R 4.3 0.6 0:00 0 ultimatebb.cg

A 'ps -ax':

PID TTY STAT TIME COMMAND
1 ? S 0:05 init [3]
2 ? SW 0:00 [migration/0]
3 ? SW 0:00 [keventd]
4 ? SW 0:00 [kapmd]
5 ? SWN 0:00 [ksoftirqd_CPU0]
10 ? SW 0:00 [bdflush]
6 ? SW 0:00 [kswapd]
7 ? SW 0:00 [kscand/DMA]
8 ? SW 0:14 [kscand/Normal]
9 ? SW 0:00 [kscand/HighMem]
11 ? SW 0:00 [kupdated]
12 ? SW 0:00 [mdrecoveryd]
16 ? SW 0:01 [kjournald]
618 ? SW 0:00 [kjournald]
885 ? S 0:00 syslogd -m 0
889 ? S 0:00 klogd -x
926 ? S 0:03 /usr/sbin/named -u named -c /etc/named.conf -u named
940 ? S 0:00 /usr/sbin/sshd
951 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
970 ? S 0:00 sendmail: accepting connections
982 ? S 0:00 /usr/sbin/httpd
991 ? S 0:00 proftpd: (accepting connections)
998 ? S 0:01 /usr/local/jre1.3.1/bin/i386/native_threads/java -Dus
1013 ? S 0:00 crond
1053 tty1 S 0:00 /sbin/mingetty tty1
1054 tty2 S 0:00 /sbin/mingetty tty2
1055 tty3 S 0:00 /sbin/mingetty tty3
1056 tty4 S 0:00 /sbin/mingetty tty4
1057 tty5 S 0:00 /sbin/mingetty tty5
1058 tty6 S 0:00 /sbin/mingetty tty6
8290 ? S 0:02 /usr/sbin/httpd
8784 ? S 0:01 /usr/sbin/httpd
8866 ? S 0:01 /usr/sbin/httpd
8884 ? R 0:02 /usr/sbin/httpd
8885 ? S 0:01 /usr/sbin/httpd
8894 ? S 0:01 /usr/sbin/httpd
8895 ? R 0:01 /usr/sbin/httpd
8906 ? S 0:01 /usr/sbin/httpd
8909 ? S 0:02 /usr/sbin/httpd
8924 ? S 0:01 /usr/sbin/httpd
8927 ? S 0:01 /usr/sbin/httpd
9314 ? S 0:01 /usr/sbin/httpd
9659 ? S 0:00 /usr/sbin/httpd
9661 ? S 0:00 /usr/sbin/httpd
9911 ? S 0:00 /usr/sbin/httpd
9914 ? S 0:00 /usr/sbin/httpd
9915 ? S 0:00 /usr/sbin/httpd
9917 ? S 0:00 /usr/sbin/httpd
9919 ? S 0:00 /usr/sbin/httpd
9927 ? S 0:00 /usr/sbin/httpd
10044 ? S 0:00 /usr/sbin/sshd
10047 ? S 0:00 /usr/sbin/httpd
10048 ? S 0:00 /usr/sbin/sshd
10049 pts/0 S 0:00 -bash
10115 ? S 0:01 /usr/local/bin/perl /home/ibsgroup/public_html/cgi-lo
10127 ? R 0:03 /usr/local/bin/perl /home/ibsgroup/public_html/cgi-lo
10132 ? R 0:00 /usr/local/bin/perl /home/ibsgroup/public_html/cgi-lo
10133 ? R 0:00 /usr/local/bin/perl /home/ibsgroup/public_html/cgi-lo
10134 ? R 0:00 /usr/local/bin/perl /home/ibsgroup/public_html/cgi-lo
10135 pts/0 R 0:00 ps -ax

Some questions:

How many user are active at any given time?
Do you have any code hacks installed?
About how many page views are you getting in a month on your forums?
What is your cache percentage?

I'm sure Charles will have a few more questions/answers than I have, but that's a good start

I average 50-100 users active at once. My max is 240.

No code hacks. All have now been incorporated into the base product. I have only added to the navigation bar in public_common.pl

550,000 page view per month on average

Cache hit is roughly 48%

What are the stats on the server? What type of connection is the server on? Who is the host of the server? Who is on the server becides just yourself (if you know)?

Mainly, when it comes to who is on the server, are you are on a shared or a dedicated?

After a second look, I notice that you have over 23,000 registered users on your boards. If I'm not mistaken, once you get over 20,000 users you risk some performance issues due to the number of directory entries in the Members directory.

It is a dedicated server running RedHat 9.0.

CPU: Intel Celeron 1.7 Ghz
RAM: 512 MB RAM
Hard Drive: 80 GB IDE

Hosting with Interland out of the Dialtone data center in Florida.

10Mb. connection to switch

Virtually one host running on the server. Some single page websites with static ip's.

What you're seeing is "normal"... though that sounds too positive.

Basically, it's a side-effect of the file locking scheme UBB uses. When one instance needs to lock a file, it locks both the file in question and a global file lock. While the global file lock is in place, no other currently running instances of the UBB can perform any file I/O operation... and on a system that uses files for everything, that brings things to a grinding halt if a single UBB process needs to take forever and a day to do something.

That's just about enough server for that board... with high cache hits like that, you shouldn't be putting too much of a dent on that server.

I suggest turning off Recent Visitors, then pruning away about half of your members. Performance starts to drop around the 10,000 mark, and we generally advise not going over 20,000.

Those two steps will both reduce the frequency and duration of the file locks, and also reduce the time it takes to open member profiles, resulting in what will hopefully be a noticable speedup (and thereby also reducing the duration of file locks, which in itself will speed things up more...)

CC, can large archives slow down a UBB? ie 308,000 posts?

Thanks for all the info.

Do you think it is time for me to consider UBB.threads?

Large read-only archives are rarely touched. They would be a performance hit, but not nearly as much as a live, active forum.

You have nearly a half million messages... you could consider UBB.threads, but only if the advise I gave above fails to make performance acceptable. Otherwise, there's no need to move yet. Now, if your traffic were to double, then I'd advise moving.

However, with such a large message base, you would almost certainly need beefier hardware so MySQL can perform well.

Thanks again for the advise. I've turned off Recent Visitors... the easy task. I'll look into pruning over the next few days and see what the performance is like afterwards.

Jeff

An Update:

I pruned 18,000 members today. Some users are reporting better performance as a result of this. I also moved ubbcgi/Members to a new directory so that the directory size of "Members" wasn't so large. I figured that it might also speed up file system searches through the directory.

Moving the directory contents shouldn't have any real effect...

Your board does seem a little more responsive. How has server load been?

Server load was down yesterday. I'll give it a little more time. The BB does seem slightly more snappy. Starting to think moving to a Celeron box from a PIII wasn't the best plan last fall, but my cost went down from $229/mth to $79/mth.

Many of the inexpensive dedicated providers now offer good Pentium 4 boxes for around $120-$150 a month, depending on configuration.

If you see UBB.threads in your future, be sure to grab one with an obscene amount of memory, say around 2 gigs... that will allow for maximum growth once MySQL has been properly tweaked.

I've heard some good things about www.webreseller.net and their dedicated plans, their not that expensive. www.serverbeach.com didn't sound half bad either.

I can convincingly say now that my server and UBB are performing much better with the purging of members, turning off of recent visitors and purging of old posts.

Haven't seen any high load averages in a week.

Jeff

My host just notified me of a similar problem with my board... I've opened a support ticket, but anything extra that can be suggested would be appreciated.

Small board, on a shared unix server, 2800 members, no code hacks, cp says 190K pageviews in March and 99.82% cache. Typical online folks range from 20 to 60.

Host is freaking out as server load is reaching 15% or higher and has disabled ultimatebb.cgi on me.

They want the problem "fixed" before reactivating... so? disable who's online, prune unactive members, what else?

Oh, and here's the load report:

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
10239 cca 15 0 7712 7640 1896 R 15.5 0.3 1:57 ultimatebb.cgi
6253 cca 15 0 7716 7428 1896 R 15.3 0.3 3:52 ultimatebb.cgi
8674 cca 15 0 7672 7672 1896 R 15.3 0.3 2:27 ultimatebb.cgi
10351 cca 15 0 7724 7128 1896 R 15.3 0.3 1:54 ultimatebb.cgi
5414 cca 16 0 7692 7680 1896 R 15.1 0.3 4:23 ultimatebb.cgi
10004 cca 15 0 7708 7484 1896 R 15.1 0.3 2:00 ultimatebb.cgi
6358 cca 15 0 7720 6448 1896 R 14.9 0.3 3:45 ultimatebb.cgi
17103 cca 15 0 7760 7624 1896 R 14.7 0.3 1:32 ultimatebb.cgi
22523 cca 14 0 7784 7772 1896 R 14.3 0.3 1:19 ultimatebb.cgi
23968 cca 15 0 7744 7712 1896 R 14.3 0.3 1:12 ultimatebb.cgi

brushie, was this taken care of via the ticket? Honestly, your board sounds far too small to even put a dent in a server...

Charles, I opened the ticket before finding this thread - search turned up nothing useful, then I happened upon this one.

Anyhow, after reading the above, and posting my stuff, I implemented the above - pruned old inactive members and shut down who's online. Host was happy, board's back up and about a day or so later someone looked at the ticket for the first time and said "let us know if you have more problems"

As of yet, there are none, but I haven't put who's online back up either.

I think your hosts's request is rediculous personally; your board doesn't seem that large to add a meaningful dent; most likely they have issues against UBB.Classic as in a large enviroment it can give issues, but with a small board they just fear what it can become...