I tried turning the numbers off, and it made no noticeable difference in speed. Everyone noticed that the numbers were gone, however, and I got a ton of complaints!
Here are my server specs (dedicated):
dell PowerEdge R200
Dual-core Xeon 3060
4096 MB Memory
160 GB x2 SATA RAID 1
I have never optimized anything, but the host may have.