Are you using persistant connections or not?
If so, try to turn them off.

You also might want to log the slow queries. It helps to determine the bottleneck.

Are you running a dedicated server or is it a shared one?