I tried clicking on the Control Panel's Reset Visitor Record and Reset all Visitor Data. This got rid of the list of bogus recent visitors and has sped up the board back to normal. I'm going to monitor this to see if it slowly creeps back to being slow after time since the problem happened about 2-3 days after starting the board on my new server.
