FWIW, I JUST (like a week ago) upgraded the servers (CentOS6, Linux (De-branded RHEL)) for two clients who ultimately use them for one site each:
Client Alpha
52,715 Registered Members
72 Forums
674,845 Topics
9,734,647 Posts
99 GB Ram, 4 CPU 2.4 GHz with 6 Cores per CPU
Client Bravo
33,947 Registered Members
72 Forums
190,344 Topics
2,783,777 Posts
74 GB Ram, 4 CPU 2.8 Ghz with 6 Cores per CPU
Heck, even my tiny VPS is 4 GB Ram, 2 GHz CPU with 4 Cores.
The goal is to have as little swap/page file usage as possible, which is achieved by adding additional memory, as when the page/swap space is used it can (and will) slow the system until resources are freed; on a Unix/Linux system you'd check this by simply looking at your iowait and swap usage.