UBB.threads PHP Forum Software Community Forums UBB.threads Version 7/8 Support Server Crashing

I have a dedicated server which only does two things - runs the UBB forum and runs MailEnable mail software. MailEnable is running very smoothly and in the MailEnable world we're a relatively small install for them, only 400 or so boxes. On the UBB side we have 25,000 members and 400 forums, which I believe makes us one of the largest Windows installs.

http://forum.bellaonline.com/ubbthreads.php

Our server kept crashing in 2006 until we had a total system failure in the fall. We rebuilt the server and hoped we could hobble through until this new version came out. It has been crashing every few days in the past few weeks. We did the upgrade last week even though I had concerns about the content island issues, in hopes of getting the crashing to stop. Unfortunately the server's still crashing, it was down for 3 hours yesterday morning.

We have a gig of RAM and oodles of disk space. The server is usually only 20% busy so it's not maxing out the processor.

Any thoughts on any tuning we can try, to help this to run more smoothly? I do need the forums to all have content islands, but I have them set to generate only every 30 minutes to try to keep things relatively stress free ...

The server event logs never show anything amiss before the crashes.

When you say crashes do you mean UBBT stops responding or hangs? Or do you mean Windows hangs or quits responding or is it blue screening?

What version of Windows are you running?
What version of PHP and MySQL are you running?

It could also be hardware issues if you're that popular things may not be able to be written to disk properly; are you using a standard 7200rpm ide disk?

I remote terminal into the server at its hosting facility, so what I see is that the machine no longer responds to pings, no longer displays any web pages and no longer answers remote terminal requests. The techs have to reboot it to get it responsive again.

The server's running Windows 2000, sp4, 5.00.2195

PHP is 5.1.1

MySQL is 4.1.16

When the server completely went down last fall (which happened when I was on a trip to Ukraine, I had to rebuild the server dialing in from a hotel in Lviv) they replaced the hard drive, in case that was the issue. The server has two hard drives - one for the OS, and one for the websites. I'm asking them to put in a third hard drive to handle the log files. I found on my other servers that providing a hard drive just for log file writing helped things go more smoothly.

Are they IDE or SATA or SCSI? If you can afford it I'd go with faster SATA or SCSI disks (7200rpm, 10k rpm, 15k rpm); if the disks are turning out to be the issues it would likely be a resources issue on the machine with writing to disk (it's also prudent to let you know that even writing to MySQL writes to your hard disk, as mysql files are stored as flat files in the end).

Well yes writing to any database causes disk access I know that. I will chat with my hosting company to get the details on the disk types.

Sorry, I've gotten in the habbit of providing way too much information in my inqueries; you never know when someone will stumble in with the same issues in the future ...

Well you probably get no choice but Server 2003 implements a completely different IO scheme and is much better at handling memory and disk IO. As for disk speed 15K SCSI drives on a GOOD RAID controller is going to beat out the SATA drives.

That does not explain though what is causing it to hang and not respond. Slow writes should show up in event viewer as an error.

Enable event logging and it may give you more details.

And this is a stand alone server, not a VPS correct ?

When I'm on your site and it's running fine it seems to be okay speed wise so that leads me to think something else is going on and hanging the system. Maybe it is a hardware issue that locks it up and nothing to do with software.

What type of access do you have to the system when it's running normally? Can you run all the normal Windows tools and get to all the diagnostic tools?
.

I agree on a good raid controller and a 15k scsi beating the hell out of ide or sata, I'm just afraid that her host is running her on a standard ide/ata versus something more "modern" for the "ideal" setup...

Oh I know the host provider isn't running a SCSI system for them as it's TOOOO expensive to build AND maintain. They all put in cheap stuff to get the most bang for the buck.

But if you co-locate or are allowed to build your own then it would be better to have the SCSI setup.
For SATA something like this is much better than what comes on the motherboards:
AMCC 3Ware 9650SE
http://www.3ware.com/products/serial_ata2-9650.asp


Hey I speced out a server for work to handle 5,000 transactions a day and it was not cheap. hp was about $37K and dell was about $25K not including the fail over system which is smaller and cheaper. Awesome system though.

.

Man I looked up that card, $230-$910 on NewEgg; it looks sweet but man, it's not cheap hehe...

No it's not cheap but has better specs/speed than Promise and others. I would still rather have an hp DL380 G5 with their RAID controller

I think i've met my equivelant geek on a server area lol...

A single gig of RAM for such a huge site . If you are sure the reason for crashing is not a hardware failure or such, the first thing I would do is to add more RAM. Configure MySQL to hold all indexes into RAM to relieve hd load.

Well something is wrong with your forums, your ubb=online page is doing 41 queries every single time it's viewed, which leads me to believe it's rebuilding the cache every time that page is viewed. That could easily bring down a server. I don't know exactly what could cause that, but will let Rick work with my observation

Offhand I'd say to check the portal page module cache and ensure things are set to 15minutes or so (at least 5); I'd highly advise against using a low number on such a large forum...

The disks are 7200RPM SATA drives. I will ask about faster disks.

This is a dedicated machine that is solely mine. The three standard event logs do not show any problems before the crash. I remote terminal into the machine so I can do whatever I want to it. I manage it and install the software myself. Normally it runs at very low processor usage and reasonable memory usage. When it crashes, I can't log in of course Then when it reboots everything seems fine and the log files don't report any problems before the crash. So it's been very hard to troubleshoot this. We had hoped it was a hardware issue with the previous disk, but apparently that was not the case.

Actually I just looked at the task manager to check, it's barely using half the RAM. That has never seemed an issue to me. If I go the route of tossing all indices into RAM, would that really take up the other 500mb?

Here are some links to help you diagnose problems.

You may have to start and run a long term performance log to see if it can track down an issue.

Hopefully something here helps you to track down the issue.

Microsoft ® Windows Server ™ 2003 Performance Advisor
http://www.microsoft.com/downloads/...6b9-a9a5-9bffcd237da2&DisplayLang=en

Windows Server 2003 Performance Counters Reference
http://technet2.microsoft.com/Windo...4f52-a9f8-09d5ebeb9ef21033.mspx?mfr=true

Introduction to Windows Support Tools
http://technet2.microsoft.com/Windo...4623-98ab-dc26f1faa6eb1033.mspx?mfr=true

Windows Server 2003 Service Pack 1 Administration Tools Pack
http://www.microsoft.com/downloads/...36A-A392-25793A25BAD7&displaylang=en

Windows Server 2003 Resource Kit Tools
http://www.microsoft.com/downloads/...AE7-96EE-B18C4790CFFD&displaylang=en

Windows Server 2003 Service Pack 1 32-bit Support Tools
http://www.microsoft.com/downloads/...E81-B3BE-4E7AC4F0912D&displaylang=en

Internet Information Services (IIS) 6.0 Resource Kit Tools
http://www.microsoft.com/downloads/...C73-B628-ADE629C89499&displaylang=en

With my server running Windows 2000, I feel uncomfortable downloading Windows 2003 resources onto it ...

Okay my mistake. Similar tools but I'll look them up for you tomorrow.

Does it have all the up to date recent Critical updates installed ?

.

I've always found that using windows update leads to an unstable system ... Think of it, all these new fangled utilities being listed as critical is worthles... I say you decide on your own what you want to install

Gizmo that is not true - Yes it was very problematic years ago, but now days it is pretty rock solid (imho) Yes on occasion there are issues but often only due to some specific setting or other software you have installed. I've managed very large sites and smaller sites as well and all of them work fine with Microsoft critical updates. Microsoft is also available for free to repair your issue if the critical update did break it.

Granted many of the updates are not quite as "critical" as some would want to believe but still 99% of the time you will not have an issue with an update (imho). Which is still better than most companies out there that provide updates.

.

You can easily check that by looking into the directory where your MySQL db-files reside. Add up the size of all the *.MYI files. If you have unused RAM, you really should make use of it.

NT, do a fresh install of XP Pro and do no updates, let it run for 2 weeks and monitor your resource usage.

Now, install windows updates, all of them like any standard sheep would do... And now monitor your resource usage.

Your resource usage (on the bare system, running no extra utilities) will run like crap compared to the fresh system, with only Windows Updates installed.

So yes, it is true; the windows updates "new fangled utilties" that come as standard updates are a complete drain on your machine; it's quite discusting and makes baby jesus want to cry...

I haven't done this in several years, but it's been the same for several versions of Windows; it's sad when you can't even trust the provider to do something as simple as provide reliable updates to their software lol...

In general only 1 gig of RAM is pretty light for as large an installation as you have. If you're only using but half of that gig you're not doing your server or it's discs a favor.

All of that extra unused RAM could and should be used for database caching. More memory cache less disc access.

As for why your machine becomes unresponsive we likely cannot tell you unless we're there to see it, but experience can be helpful. Here's the most common scenario that I've seen -

Machine is either low on resources or isn't using them well but hums along just fine until something is done that brings it close to the edge of it's limits. Maybe it's a bunch of searches all at the same time, maybe it's a huge cache rebuild that occurs while a search is being run. Whatever the exact cause, it was a combination of things that brought a machine close to it's max out put.

Now the machine is slowed and requests start to back up. Perhaps it's disc IO that is the culprit. Memory use while the machine is backing up still looks fine. Load, while rising isn't huge. That's because the discs are so backed up nothing is being processed. Load is climbing though because of the number of processes that are waiting for disc.

Doesn't take but a few seconds in that state to get a locked up server. Nothing is going to fix it now except a reboot.

Do I know if that is what happened to your machine? No. But I wouldn't be surprised at all. I see it every day. You've got a gig of RAM, which is light for your volume, and only half of that is in use. Just think how much better the thing would run if it was making efficient use of the entire gig of RAM?

Go here and read the tuning tutorials. A gig likely isn't enough for your traffic by why have a gig of RAM if you're only using half? Get that machine making more efficient use of your RAM and it will help.

Of course, I'm taking your word that it is only using half the gig. That in itself sounds kinda fishy to me. Do yourself a favor and verify that again before you make any changes to your MySQL.

I was hoping someone with more windows hosting experience would happen over this threads; and in comes the mac user ...

"You can easily check that by looking into the directory where your MySQL db-files reside. Add up the size of all the *.MYI files. If you have unused RAM, you really should make use of it. "

I most definitely know I have half the RAM unused.

We do not have MYI files. We are running InnoDB.

An update, the server crashed Friday night. They rebooted it Saturday morning. It crashed again Saturday night, and they rebooted it Sunday morning. On Monday they are going to add in a third hard drive to handle all the log files, to see if that helps.

Intriguingly, MailEnable (the only other application on this server) reported errors during the crash time period. It says -

-----------

21 Jan 2007 00:07:05 MailEnable MailEnable Database Provider error: 0,FAILURE: (SELECT ID,Mode,AddressMask,Status,Account,AccessRight FROM SMTP_ACCESS WHERE ((Mode = 1) AND ((AddressMask = '64.*.*.*') OR (AddressMask = '64.74.*.*') OR (AddressMask = '64.74.173.*') OR (AddressMask = '64.74.173.203')) AND (AccessRight = 'CONNECT') AND ID > 0) ORDER BY ID LIMIT 1;), Module: MESMTPC.EXE; Error: [MySQL][ODBC 3.51 Driver][mysqld-4.1.16-nt]MySQL server has gone away

21 Jan 2007 08:33:55 MailEnable MailEnable Database Provider error: 0,FAILURE: (SELECT ID,Name,Status,Account FROM POSTOFFICE ORDER BY Name;), Module: MERADMS.exe; Error: [MySQL][ODBC 3.51 Driver][mysqld-4.1.16-nt]MySQL server has gone away

-----------

So where did MySQL go?

silly question - but as it is at night - is it crashing during a backup procedure?

Dumb question; how big are the log files? If they're over 2gb they could be causing crashing...

OK I take back my other innodb comment - it looks like the UBB install overrode my defaults and made all the MYI files and such.

I do still have an ib_logfile0 but it's only 51mb.

I don't have any automated backups running on this server, I am trying to keep activity to a minimum so I can track down what the issue is.

I did just download the MySQL admin tools and will be turning on logging.

If you meant the webserver log files, no those aren't over a gig.

I meant any server logfiles that could be associated with apache/php/any web scripts; but it's good none are that big, so that tosses that idea out of the window.

"MySQL server has gone away"

"MySQL server has gone away"


MySQL is crashing. Again, you have a very very large installation, only one gig of RAM, only half of which is in use by the entire server, including operating system, web server and MySQL. It really is no surprise that MySQL and the server are crashing. There are not adequate resources.

Take the time to read the topics that I pointed to, tune your MySQL so that it can be at least somewhat efficient.

Start from there. Your posts only reinforce what we already suspect to be true. It's not going to fix itself. Throwing more RAM into your machine will do not a thing if the server doesn't use it either. You need to tune your MySQL and at least make use of the ram you have.

But again, there is plenty of RAM free. I'm running the SQL Administrator and it says the server is only being 5% used. All the cache hits seem great. Here are my settings - what do you think is set incorrectly?

I'm a database designer and have done a ton of work with MS-SQL, Oracle etc - this forum system really isn't that big at all ...

---------------

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=800

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=33M

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_cache=1520

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=16M


# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before. This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=38

#*** MyISAM Specific options

# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method. This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_max_extra_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method. This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=64M

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=256M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=1M
read_rnd_buffer_size=4M

# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=4M

key_buffer = 256M
max_allowed_packet = 1M

And also for example during this week the server ran flawlessly Mon - Thu, then crashed twice on Thu and twice on Fri with no increase in traffic. It really doesn't point to some sort of SQL setting being wrong. It points to something in particular that the forum is trying to do choking it.

Well, what is strange that it was crashing before the upgrade as well. The difference in the design of your prior version and version 7 are completely different, and the fact that it was crashing then and crashing now even with all the changes leads me to believe there is some underlying factor.

Are you running the latest MySQL version? If not then you could look at upgrading MySQL.

Bella you probably need to do some long term logged performance analysis on the system. This should leave a trace of the cause even if it were to blue screen as it would still keep the log up to that time.

I'd have to go back through my notes but I thought there was a reason I was on 4.1. Wasn't it UBB that didn't support the new version, 5.0? Or is that limitation gone with this new release of software? If so I'll gladly update to 5.0 and see if that helps ...

The old version had *all* sorts of problems - content islands got stuck daily, the forums ran really slowly, etc. So that was just sick in general. The new version is generating content islands fine, it runs quickly, so we're just down to it crashing occasionally Since crashing is such a generic "something has failed" thing, I'm not sure we can really compare them against each other yet.

Yes, with the latest version you can run on version 4 or version 5, so I'd definitely try a MySQL upgrade if possible.

MySQL5/PHP5 ftw!

ftw ?? What's that mean?

FTW == For the Win

LOL, I have a different meaning

Let me guess, "f___ the world" (like on the 1970's prison tattoo's)?

Yup

Free the Win ??

Sorry this is about servers crashing and pretty serious too.
Hope you get it sorted with the upgrades.

For the win, see the gameshow "hollywood squares"

I would upgrade to MySQL5 - but I am not 100% confident that all software will run on it - some can be a little funny as to the version.

i've had no real compat problems with MySQL5; it's PHP5 that can be the stickler; mainly becasue of people not coding for register globals being off...