First thing would be to make sure that it is ImageMagick6 that you have installed and not 5. You don't need to do any PHP changes for ImageMagick to work, but you do need to make sure that PHP has the permission to use the exec() function.

A quick way to test if it's setup properly, is to call it directly. When you're logged in as an admin, bring up this page:

ubbthreads.php?ubb=captcha&t=im&test=1