Previous Thread
Next Thread
Print Thread
Hop To
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
I ran a test in this forum here and it seems to work fine here. So it must be something about the settings in my forums.

Here's a standard post that one of my BellaOnline editors would make in her forum area. It's a promo for one of her articles. Note the curly quotes in the first word I'm.

~~~~~~~~~~~

I’m writing this during December so maybe that has something to do with all the tired and stressed business owners I’ve been meeting lately. Somehow though, it feels deeper than just the end of the year, it almost feels like many people are experiencing business burn out.

Is Your Business Burning You Out?

~~~~~~~~~~~~~~~

That will NOT work on my forum for some reason. If I manually go in and change the curly quotes to straight quotes it will work. The curly quotes worked fine until we upgraded to this latest version.

Any ideas?


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
Oh the behavior it gives is that the entire post is simply empty. There's no errors or anything else, but the resulting post is an empty box.


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
Do your error logs show anything? Perhaps a Mod_Security notice?


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
OK it's not just brand new posts. I'm finding blank on-screen boxes for old postings too.

Here's an example. Look at this link -

http://www.wineintro.com/forum/ubbthreads.php/topics/125879/9

There are several posts from 2006 which show up blank. So I looked at the database for post 125964.

The database says it should be:

Code

[quote]Originally posted by Believer:\n[b]....\"Evil flourishes when good people do nothing”. If the majority of the world can see this and make a famous saying of it then I\'m sure you can too.[/b]\n\nI am not arguing against that.\n\n[b] .. as for God, yes he is all powerful, and can stop the evil that men do, but if he did then our free will would not be so free. [/b]\n\nSo we have evil flourishing when God does nothing. And God is not a mere mortal - he is all-powerful.\n\nIf I were to see a child being abducted, a woman being raped or a man being murdered, I hope that I would do something about it. \n\nHowever, if I did not, could I use the argument that I was allowing both the criminals and their victims the opportunity to use free will, in accordance with God\'s wishes?\n[/quote]


There's an end curly quote in the initial quote. I bet that is what's choking it. So I'd have to edit all old posts with curly quotes to get them to work.

Last edited by BellaOnline; 12/25/2012 3:53 PM.

Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
The MySQL error log doesn't show anything at all for today except some notes when I rebooted the server.


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
My charset was originally iso-8859-1. I tried changing it to utf-8 which is what this forum here has, to see if that helped. It didn't. The content-type now shows with charset=utf-8 but the post still won't show.


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 16,292
Likes: 116
UBB.threads Developer
UBB.threads Developer
Joined: Jun 2006
Posts: 16,292
Likes: 116
It'd be the web server log; though you're on Windows so I'm not sure if you would have mod security installed...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!
Joined: Apr 2007
Posts: 3,940
Likes: 1
SD Offline
Former Developer
Former Developer
Joined: Apr 2007
Posts: 3,940
Likes: 1
it's htmlspecialchars in php 5.4x

that pukes

temp fix... run php 5.3.19 or edit bbcode.inc.php

no time to post a patch now wink

Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
I've got bbcode.inc.php open now - what am I editing in here?

I did try backing up the database and then changing the posts table database character set from latin to utf-8. That didn't help, although it didn't hurt the data either. I imagine it's good to have everything consistent with utf-8 so it was probably good to do that, as I move towards an actual solution smile.


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
Hmmm I've tried all sorts of combinations on the 7 instances of htmlspecialchars in bbcode.inc.php. None seem to change the end results. I thought this would work -

//$old_smiley = htmlspecialchars($code);
$old_smiley = htmlspecialchars($code, ENT_IGNORE, 'UTF-8');

but no luck. I tried it with the old ISO code, with ENT_SUBSTITUTE, and other options. I would have thought if the database is now UTF-8 and the page's charset is UTF-8 that the default setting of htmlspecialchars to UTF-8 would match that.

Totally commenting out all the htmlspecialchars calls didn't help either, so I'm less certain it's those causing the problem.

I'm seeing suggestions to force the client into the same mode -

mysql_query ("set character_set_client='utf8'");
mysql_query ("set character_set_results='utf8'");
mysql_query ("set collation_connection='utf8_general_ci'");

I'll see if I can do that to test.

Last edited by BellaOnline; 12/25/2012 5:33 PM.

Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Apr 2007
Posts: 3,940
Likes: 1
SD Offline
Former Developer
Former Developer
Joined: Apr 2007
Posts: 3,940
Likes: 1
PHP Code
	 // SD:Mod -- newer versions of php will choke
		if ($this->convert == "markup") {
			$body = str_replace(array('>', '<'), array('&gt;', '&lt;'), $body);
			//$body = htmlspecialchars($body);
		} 

but this is only one place where there is an issue, albeit the most often called

i just tell peeps to run php 5.3.19 for now, but i'm perty sure you won't go that route laugh

so that'll tide you over with your latest php with a patch that doesn't really fix the problem in __every__ place it needs for a final solution wink

a real wrapper solution should be employed.. maybe in next version 7.5.7.1 or whatever, since it's in multiple places

Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
Hmmm but if that was the case then it should have fixed when I tested out commenting out all of the htmlspecialchars lines in the file.

Indeed, I just made that recommended change and the problem is still occurring. I even rebooted the server in case something was stuck somewhere.

I then tried ensuring the client was using UTF-8 by using in the my.ini file -

Code

[client]
default-character-set=utf8

[mysql]
default-character-set=utf8


[mysqld]
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8


When I rebooted, the main forum listing claimed there were no forums. But oddly I could go into my user profile and see everything there, and the sidebars then filled in properly with the forum names and latest post names.

I undid it and rebooted. Everything is back as before.

I would think I want everything utf8 to be in sync. Right now I have it in my HTML header and as the posts table definition. So I wonder if I should take the plunge, make this my.ini file to have those utf8 entries, know it will temporarily hose the forums, and then go page by page through the forum to see where it's causing problems and fixing those issues. I could then report back to you guys where I made the changes, in case other people wanted to move to utf8.


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
OK I'm now testing in my WineIntro forum which is still using the default iso-8859-1 character set in the HTML header, and the posts table is still latin1 and MyISAM. So all the same as before.

I'm working in showflat.inc.php to test. On line 590 it does the query to get the post_body.

I made a post with the text "xyzzy" in it to test. I then added this code at line 684, right when the first loop is being done through the posts.

Code
$lisatest = $results[$i]['POST_BODY'];
$lisapos = strrpos($lisatest, "xyzzy");
if ($lisapos === false) {
	// not found...
}
else {
	$lisatest = "replacedthexyzzyitworked";
$results[$i]['POST_BODY'] = $lisatest;
}

That works perfectly. The post in question turns into the string. So the code does see the post and works with it.

Then I have a separate test post which begins with the text "test post with unicode" and then goes on to have some curly quotes in it. It currently does not display.

When I change the above code to read:

Code
$lisatest = $results[$i]['POST_BODY'];
$lisapos = strrpos($lisatest, "test post with unicode");
if ($lisapos === false) {
	// not found...
}
else {
	$lisatest = "replacedthexyzzyitworked";
$results[$i]['POST_BODY'] = $lisatest;
}

This does NOT work. I still have a blank post.

If this did work, there would be zero unicode characters in the post from this point forward and everything else should work as usual.

I am getting the sense that even here, so early in the process, the post body value is simply empty. The content of that post is not being put into the variable. And to me that indicates it's something about the SELECT statement, that it is unable to receive the row because of the UNICODE characters in it. There is pretty much nothing at all between when the data is selected and when I try to act on it.

Am I missing something that is happening between the SELECT and my code?


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
Aha I think I've proven it. I made a test script with just this code.

Code
<?php
error_reporting(E_ALL & ~E_NOTICE);
ini_set("display_errors", true);

// We always want to work from the directory this script is in
chdir(dirname(__FILE__));

// We'll want to buffer everything...
$config = array();
ob_start();

// Setup the smarty class
require('libs/smarty/Smarty.class.php');
$smarty = new Smarty();

$smarty->template_dir = 'templates/default';
$smarty->compile_dir = 'templates/compile';

// include all of the required libraries
require_once("libs/mysql.inc.php");
?>

<html>
 <head>
  <title>PHP Test</title>
 </head>
 <body>

<?php

$con = mysql_connect("XXXXX","XXXXX","XXXXX");
if (!$con)
  {
  die('Could not connect: ' . mysql_error());
  }

mysql_select_db("XXXXX", $con);

$result = mysql_query("SELECT post_body from ubbt_posts where post_id = 358825");

while($row = mysql_fetch_array($result))
  {
  echo "Post Body: " . $row['post_body'];
  echo "<br />";
  }

mysql_close($con);

?>

 </body>
</html>

When I run that code fetching the post_subject, that works perfectly. It gets the subject and displays it.

However, when I change subject to body, it does NOT get the body. I get blankness. So this has nothing at all to do with HTMLSpecialChars. It is a base level issue with bringing UniCode from the database and into PHP to work with it.

Any thoughts are appreciated. I will keep working on it. I found that some people had a similar issue with the combo of IIS / PHP / FastCGI and I tried to deinstall FastCGI but even though the files are gone and the IIS mappings are pointing directly to php-cgi.exe again, PHP reports that it's still using FastCGI. Could that be a registry entry somewhere? I don't think it could actually BE using FastCGI since those files no longer exist smile

UPDATE: I'll also note just to be clear that when I run that same pair of tests on my ID number that does not have UniCode in it, both work perfectly. So it definitely does show the body of a post that is UniCode-free.

Last edited by BellaOnline; 12/26/2012 10:40 PM.

Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
More testing notes. I have now run tests both from PHP and ASP. In *both* cases I see identical results:

Post Subject without unicode: test post without unicode

Post Body without unicode: test post with just normal characters in it, as well as xyzzy for testing.

length of post body: 74

Post Subject WITH unicode: test post with unicode

Post Body WITH unicode:

length of post body: 0


So it's definitely something about connecting to the database that is independent of PHP or ASP. Any thoughts would be greatly appreciated!


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
I think I've got it fixed! I made a bunch of changes and it still wasns't working. Then I realized that somewhere along the way - probably when I explicitly cast the post_body field as utf8 - that it wiped out any field that had unicode in it. It left the non-unicode fields fine and emptied the ones with unicode. There were only 164 rows so I copied the data over from post_body_default for those rows. Voila! I can now see those in the database!

I'll do it more step by step on my other forums to see which step was the one that helped.

What *is* that post_body_default field for? Is it really a duplicate copy of post_body? Or did I lose some data by doing that?


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
OK even more bizarre. A forum I hadn't touched at all yet had 126 rows in it with an empty post_body. Looking at post_body_default it appears those had unicode characters in them. The posts table is still at its default latin1-default collation and MyISAM setting. So it can't have anything to do with touching the table. So maybe this happened during the MySQL upgrade? When I try to add a post to it with curly quotes in this state the post_body stays blank but the post_default_body fills in properly.

I have verified this. On the other forum all I did was change the ubbt_posts table to be UTF8 default for both the whole table and the body row in particular. Then I copied over the post_body_default field into the post_body field. Those rows now work fine.

However, I still have an issue. When I make a *new* post with unicode characters, the post_body_default fills in fine - but the post_body stays blank. It won't set with the values.

Thoughts?


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
OK, now with everything in place, making SD's edit to bbcode.inc.php allows new posts to go in properly. So I think that is now set.

So it's a multi-step process. Get the table into UTF8. Copy over the rows to fill in the missing data. Then alter the code so data going forward is inserted OK.


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
Here is my remaining issue. I now have the whole posts table as UTF8 and also have explicitly set the subject, post, and default_post fields to be UTF8. So the data is all stable. New posts go in and edit fine.

My lingering problem is with old unicode-holding posts.

1) the unicode characters show up as black question marks. I do have a charset=utf-8" line in there. In my test PHP, the old post shows up quite fine. So there's something different about my test SELECT and the UBB SELECT. It appears it's because my test PHP has the line

$result1 = mysql_set_charset('utf8', $con);

How do I add that line into the UBB code?

2) When I try to edit one of these posts, the edit box comes up empty. So I can't edit the post to even manually remove those characters. Thoughts on why the edit box is empty?

If I can get these two issues resolved with historic posts I'll be all set. You can see this issue here -

http://www.wineintro.com/forum/ubbthreads.php/topics/425917/1


Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts
Joined: Jun 2006
Posts: 693
Addict
Addict
Joined: Jun 2006
Posts: 693
I'm still stuck on the remaining two issues. I tried editing the my.ini and adding

init_connect='SET NAMES utf8'

but that didn't do the trick. It looks like I do need to add that charset command in the code itself, after the connect. So far when I try to add it into mysql.inc.php I get access denied.

UPDATE: It appears the reason this won't work is that if you log in with SUPER access it ignores this command. Their theory is that you might otherwise lock yourself out of your own database by accident. So now I'm trying to get a user without SUPER access to work with the database properly, to at least fix this part.

Last edited by BellaOnline; 12/27/2012 11:04 PM.

Lisa Shea, owner, BellaOnline.com
BellaOnline Website
BellaOnline Forums - UBB since Apr 2002 - 58,000 members / 850,000 posts

Link Copied to Clipboard
ShoutChat
Comment Guidelines: Do post respectful and insightful comments. Don't flame, hate, spam.
Recent Topics
spam issues
by ECNet - 03/19/2024 11:45 PM
Looking for a forum
by azr - 03/15/2024 11:26 PM
Editing Links in Post
by Outdoorking - 03/15/2024 9:31 AM
Question on barkrowler and the like
by Mors - 02/29/2024 6:51 PM
Member Permissions Help
by domspeak - 02/27/2024 6:31 PM
Who's Online Now
1 members (Ruben), 476 guests, and 111 robots.
Key: Admin, Global Mod, Mod
Random Gallery Image
Latest Gallery Images
Los Angeles
Los Angeles
by isaac, August 6
3D Creations
3D Creations
by JAISP, December 30
Artistic structures
Artistic structures
by isaac, August 29
Stones
Stones
by isaac, August 19
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20230217)