GD has to have TrueType support, and ImageMagick has to be installed; paths are in the "Paths" section of the UBB control panel for IM.
As for troubleshooting, only .ttf files (and the index document) should be in the images/captcha/fonts directory; i've seen some systems add files that can't be parsed and it breaks.
Also make sure the session, tmp, templates/compile directories are all chmodded 777.