Previous Thread
Next Thread
Print Thread
Hop To
Char coding utf-8 and older forums #261912 12/15/2018 4:59 PM
Joined: Jun 2006
Posts: 955
Zarzal Offline OP
Old Hand
OP Offline
Old Hand
Joined: Jun 2006
Posts: 955
I use ubb.threads for a very long time starting with a v6 version centuries ago. I life in Germany and this includes to use german special chars like öäüß (umlaute). This can you drive crazy and I like to share my experience with this and how to handle it.

If you install a brand new version 7.6.2 from stock you create a utf-8 databse, the install script create utf-8 tables and the language files (generic.php) contains the utf-8 string. All language files are in utf-8 coding. So you dont have any trouble and can use any king of char you like. It display always correct. The only thing to watch out for: while editing a language file offline use a editor (I use notepad ++) who can keep the file coding. Be carefull, check if utf-8 ist selected and not ansi or something else.

The second precaution is coding the special chars with html. Example
Code
Ä -> Ä

I alwas use the coding in my language files to avoid any surprise. But watch out for the mailer text strings! Some of them sent out in plain text and the coding will not parsed. Theses strings must contain the char itself.

Back to my old installation and the problems I encounter after updating to 7.6.2

My database runs in utf-8 but the tables was created in swedish-latin-1 long time ago. To use german chars the coding in generic.php was set to iso-8859-1. This work but if you interact with rss feeds to a utf-8 site you run in trouble. You will see ? for a special char. Sad.

Now we have to change everything. Setting the language string in generic.php to utf-8 dont fix our problem. We must digg little bit deeper. Close your forum and do a backup of the database. I use mysqldumper for this. We need it later for some steps because the tables are to big for normal importing.
After doing your backup go to phpmyadmin and watch your database tables. We have to chance the collate to utf-8. This can be done with this SQL-statement
Code
ALTER TABLE table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
or from the options menu inside the table.
If you have done this to all tables go to your generic.php and set the language code to utf-8
Now you encounter a lot of display errors in your forum. The special chars look strange and need to convert to display properly. I go this way to do it:
Create a test post with all you chars you need to convert. View this post with phpmyadmin in uub_Post table at the end of the table. You will see the correct coding and can use it für copy/paste for the next step.
Next step: export with mysqldumper the table for posts, download it, edit it with notepad++ and do a search&replace for any special char in your test posting. Re-Upload it and do a restore of this table with mysqldumper. Repeat this with Post-Topic and both private posts tables. This will convert 99% of all chars.

After this operation your ubb runs in utf-8 and is prepared for V7.7
Please note that is my way to do it without any warrenty. You should know to handle phpmyadmin and mysqldumper

Greetings
Zarzal


my board: http://www.dragonclan-forum.de
my hobby: http://www.biker-reise.de
Ich kann bei Fragen zu UBBthreads in Deutsch weiterhelfen oder es zumindest versuchen
Express Hosting
Re: Char coding utf-8 and older forums [Re: Zarzal] #261924 12/16/2018 6:36 PM
Joined: Jun 2006
Posts: 955
Zarzal Offline OP
Old Hand
OP Offline
Old Hand
Joined: Jun 2006
Posts: 955
After doing all the changes I found a new problem. If I try to send out something with email I recive all mails with coding iso-8859-1 and my german chars are gone. I look in the mail header, yes it is iso-8859-1 and not utf-8.
So I search for this string and found /libs/phpmailer/class.phpmailer.php
I see there a hardcode iso-8859-1 string and change it to utf-8 too. Now my emails still show up the correct way. I have no idea why the phpmailer use the old iso format and not utf-8 in stock installation.


my board: http://www.dragonclan-forum.de
my hobby: http://www.biker-reise.de
Ich kann bei Fragen zu UBBthreads in Deutsch weiterhelfen oder es zumindest versuchen
Re: Char coding utf-8 and older forums [Re: Zarzal] #261925 12/17/2018 4:21 AM
Joined: Feb 2007
Posts: 1,274
JAISP Offline
Veteran
Offline
Veteran
Joined: Feb 2007
Posts: 1,274
Hard coding like that I was told is NOT permitted when I was doing beta testing.

Re: Char coding utf-8 and older forums [Re: JAISP] #261926 12/17/2018 4:26 AM
Joined: Apr 2004
Posts: 1,405
isaac Online Splat
UBB.threads Developer
Online Splat
UBB.threads Developer
Joined: Apr 2004
Posts: 1,405
Originally Posted by JAISP
Hard coding like that I was told is NOT permitted when I was doing beta testing.


if you open the stock phpMailer script ( https://github.com/PHPMailer/PHPMailer/blob/5.2-stable/class.phpmailer.php ) it looks like phpMailer defaults to iso-8859-1. A strange choice in 2018. Needless to say, it's in there as a default setting which can be modified as desired.

notes: phpMailer has been used as the default SMTP mailer for all of UBBT.threads forum software's existence -- long before I jumped on as the developer in 2015. I've recently been looking to upgrade past phpMailer, so we can get php 7.2+ compatibility while keeping the legacy php 5.6 guys still functional.


isaac @ id242.com // my forum @ CelicaHobby.com
a current developer of UBB.threads php forum software // 7.7.2 Progress Notes
Re: Char coding utf-8 and older forums [Re: Zarzal] #261927 12/17/2018 7:15 AM
Joined: Jun 2006
Posts: 955
Zarzal Offline OP
Old Hand
OP Offline
Old Hand
Joined: Jun 2006
Posts: 955
It is just a note where to look into if you encounter problems with e-mail charsets. It is no criticism.
Everything was fine in the past if ubb.thread runs default on iso-8859-1. This litte point is hard to find if you code default in english and dont use country specific chars.
I knew that the phpmailer script comes from external sources.
Maybe it is a good Idea to set the page charset coding as an option in the admin panel and use vars inside all scripts. So you can change the charset easy at one place and dont need modify language and script files.

I knew the problems thats comes up with php 7.x and database connects very well. We have to keep an eye on it and how to handle the charsets while connecting the db and do some write/read to it.


my board: http://www.dragonclan-forum.de
my hobby: http://www.biker-reise.de
Ich kann bei Fragen zu UBBthreads in Deutsch weiterhelfen oder es zumindest versuchen
Re: Char coding utf-8 and older forums [Re: Zarzal] #261928 12/17/2018 7:27 AM
Joined: Apr 2004
Posts: 1,405
isaac Online Splat
UBB.threads Developer
Online Splat
UBB.threads Developer
Joined: Apr 2004
Posts: 1,405
Thanks Zarzal thumbsup you have PM


isaac @ id242.com // my forum @ CelicaHobby.com
a current developer of UBB.threads php forum software // 7.7.2 Progress Notes
Re: Char coding utf-8 and older forums [Re: Zarzal] #261930 12/17/2018 10:58 AM
Joined: Jun 2006
Posts: 955
Zarzal Offline OP
Old Hand
OP Offline
Old Hand
Joined: Jun 2006
Posts: 955
Quick side note: I'm not a coder. ubb.threads handle mails a very easy way and use only very few functions of phpmailer. The iso setting is only a default value if no parameter given while assamble the mail. It is possible to overwrite the default value while generating the mail. The mail functions in ubb.threads up to V7.6.2 are very basic and based on very old code (20 years?). At this point is changing the default in phpmailer the easy way to reach to goal. Lets see what happen in future releases of ubb.threads smile Step by Step


my board: http://www.dragonclan-forum.de
my hobby: http://www.biker-reise.de
Ich kann bei Fragen zu UBBthreads in Deutsch weiterhelfen oder es zumindest versuchen

Additional Community Resources
UBB Developers
UBB Wiki
ShoutChat Box
Recent Topics
Forum - Left Alignment / Margin?
by keysolutns - 06/19/2019 9:58 AM
Forum Not Working after Cpanel upgrade.
by doug - 06/14/2019 2:35 PM
YouTube does not display?
by BlackMale - 06/09/2019 6:34 AM
Removing a redirect?
by Baldeagle - 06/01/2019 4:50 PM
Who's Online Now
5 registered members (Gizmo, maghreb, keysolutns, Ruben, isaac), 38 guests, and 262 spiders.
Key: Admin, Global Mod, Mod
Member Spotlight
mcasado
mcasado
oregon and belize
Posts: 173
Joined: November 2006
Show All Member Profiles 
Forum Statistics
Forums35
Topics35,303
Posts192,520
Members12,170
Most Online978
Jun 24th, 2007
Random Gallery Image
Latest Photos
Land Rocks
Cozy Places
Trees
Retreats
Powered by UBB.threads™ PHP Forum Software 7.7.2
(Snapshot build 20190617)