Vast majority of the Russian language sites use 'windows-1251' encoding. It is standard. Changing it to unicode is likely to cause problems for the users. Currently I use Ubbthreads 6.5.2 and it works perfectly in Russian with 'windows-1251' encoding.
As for the error, did you just translate the button text, or the value text as well? (I'm not sure if the value text for the search items is included in the language strings)
I used built-in language editor, which is the 'official' way to translate the board and this is how I translated version 6.5.2. It allows you to change the string. I replaced 'Advanced' with 'Сложный' - this is exactly what I did.