We found what was the issue with the captcha.
Installing Imagemagick techically didn't fix the issue but it did give out a nice error message, unlike GD2. It was trying to gererate an image using the Thumbs.db
file that was in the backgrounds folder. It was hidden in the windows explorer so I didn't see it the first time. Deleting that file fixed the errors.
Thanks for the help.