GD and ImageMagick are the image processors, the Captcha uses one of the to make the image. GD comes with PHP (usually) and ImageMagick must be downloaded and configured on the server; most users use GD as it's usually just *there*.
I didn't see captcha configured for registrations, are you using it for guest posting? If you enable it, just make sure that you have it setup to use GD and it should work.
