It really relies on a properly tuned and configured MySQL installation. Are you on a dedicated or shared host? If you're just running a stock version of MySQL without any tweaks, then yeah the number of posts you have will definitely be a bit slow.
Configuring and tweaking MySQL takes a while. I'd recommend visiting
UBBDev.com There are 2 topics you really should read. Let's tune MySQL I and II.
I'd also turn on debugging down so you can get a good idea what the actual page generation times are, and what portion of those are just CPU time as opposed to MySQL time.
