Encoding(?) problem after upgrading

In this board you can talk about general questions about phpMyFAQ

Moderator: Thorsten

Post Reply
jazcyk
Posts: 385
Joined: Wed Sep 07, 2005 1:32 pm

Encoding(?) problem after upgrading

Post by jazcyk » Sun Jul 12, 2009 12:24 pm

This is a follow-up on viewtopic.php?f=6&t=9866

I ran into one more problem: a unicode/encoding problem maybe.

This faq: http://www.webyog.com/faq/content/33/7/ ... story.html reads like in the public webpage
"SQLyog can be started with a ‘-dir’ switch like “SQLyogENT -dir somefolder”. This dir switch specifies where SQLyog will look for the sqlyog.ini file and where all writeable files will be saved. This was mainly implemented for users that want to have all SQLyog-related files on a removeable drive or some kind of encrypted storage. Note that if you use the setting in ‘preferences’ to store TAGS file some specific place the ‘preferences’ setting will still have effect, also if the -dir switch is used."

but in the 'staging' installation (from both PFM 2.0.15 and 1.5.0 RC that are now installed in parallel) it reads like:
"SQLyog can be started with a ‘-dir’ switch like “SQLyogENT -dir somefolder�. This dir switch specifies where SQLyog will look for the sqlyog.ini file and where all writeable files will be saved. This was mainly implemented for users that want to have all SQLyog-related files on a removeable drive or some kind of encrypted storage. Note that if you use the setting in ‘preferences’ to store TAGS file some specific place the ‘preferences’ setting will still have effect, also if the -dir switch is used."

In SQLyog and Query Browser (that both use utf8 internally and SET NAMES UTF8) data display correct in both installations.

The special quotes “...” and ‘...’ occur because the text was copined from a (WordPress) Blog. When I write release notes I will normally create a Blogpost and after we have discussed in the team I will update, publish and copy to the FAQ 'version history' document and the help file shipped with the program. So whatever formatting Wordpress does has effect everywhere. I have not checked yet is other characters are affected (backtick, national letters etc.)

The default charset for the database imported to was utf8 (CREATE DATABASE `phpmyfaq` /*!40100 DEFAULT CHARACTER SET utf8 */), but the data table table definition reads like:

CREATE TABLE `faq_faqdata` (
`id` int(11) NOT NULL DEFAULT '0',
`lang` varchar(5) NOT NULL DEFAULT '',
`solution_id` int(11) NOT NULL DEFAULT '0',
`revision_id` int(11) NOT NULL DEFAULT '0',
`active` char(3) NOT NULL DEFAULT '',
`sticky` int(11) NOT NULL,
`keywords` text,
`thema` text NOT NULL,
`content` longtext,
`author` varchar(255) NOT NULL DEFAULT '',
`email` varchar(255) NOT NULL DEFAULT '',
`comment` enum('y','n') NOT NULL DEFAULT 'y',
`datum` varchar(15) NOT NULL DEFAULT '',
`links_state` varchar(7) DEFAULT NULL,
`links_check_date` int(11) NOT NULL DEFAULT '0',
`date_start` varchar(14) NOT NULL DEFAULT '00000000000000',
`date_end` varchar(14) NOT NULL DEFAULT '99991231235959',
PRIMARY KEY (`id`,`lang`),
FULLTEXT KEY `keywords` (`keywords`,`thema`,`content`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

Our FAQ was originally created on MySQL 4.1.10 having latin1 as default charset (we have upgraded MySQL on ther server from 4.1.10 > 4.1.20 > 5.1.34 > 5.1.36 since then). So latin1 is 'sticky' for existing columns - changing database and table *default* charset does not change existing data/columns.


Encoding 'discrepancies' between a true Unicode client and PHP is a common problem (sigh!). We have this FAQ:
http://www.webyog.com/faq/content/34/15 ... qlyog.html

But this looks not quite the same. The 'staging' installation runs PHP 5.3.0 .. so maybe an issue with PHP 5.3? Can you please comment on this. I can provide a dump of the database, try to replace PHP 5.3 with 5.2 on the 'stagin installation' or whatever you suggest. I am willing to spend time with this, but I think we'd better coordinate, so I'd take your advice first.


2)
Also this time there is a bonus question:

I understand that PMF is now fully unicode compliant. Thanks! But then UTF8 would be optimal for storing PMF data on all platforms, right? Should not the update script perform ALTER statement to existing installation like
* ALTER TABLE ... DEFAULT CHARSET UTF8 (for every table)
* ALTER TABLE ... CHANGE {whatever charset to} UTF8 (for every column)
? Or alternatively provide a script for users that want to change the charset of the datastore?
(fulltext indexes are used quite a lot - and they will need to be dropped and recreated after charset has been changed - so doing this manually is non-trivial)

jazcyk
Posts: 385
Joined: Wed Sep 07, 2005 1:32 pm

Re: Encoding(?) problem after upgrading

Post by jazcyk » Sun Jul 12, 2009 12:27 pm

Going back h PHP 5.2 solved the problem!

I think you have good contact to PHP developers? Could you please report this?

jazcyk
Posts: 385
Joined: Wed Sep 07, 2005 1:32 pm

Re: Encoding(?) problem after upgrading

Post by jazcyk » Sun Jul 12, 2009 12:30 pm

Going back *to* PHP 5.2 solved the problem!

jazcyk
Posts: 385
Joined: Wed Sep 07, 2005 1:32 pm

Re: Encoding(?) problem after upgrading

Post by jazcyk » Sun Jul 12, 2009 1:57 pm

One more serious typo in the frist post :(
but in the 'staging' installation (from both PFM 2.0.15 and 1.5.0 RC that are now installed in parallel) it reads like:
should have been
but in the 'staging' installation (from both PFM 2.0.15 and 2.5.0 RC3 that are now installed in parallel) it reads like:
So there is an issue with PMF+PHP 5.3.0. In which of those the problem is, I don't know!

Thorsten
Posts: 15107
Joined: Tue Sep 25, 2001 11:14 am
Location: #phpmyfaq
Contact:

Re: Encoding(?) problem after upgrading

Post by Thorsten » Sun Jul 12, 2009 2:22 pm

Hi,

I tested phpMyFAQ 2.5 and PHP 5.3 about 2 months ago, but I will do that as soon as with the final 5.3 release.

phpMyFAQ 2.5 is not fully UTF-8 compatible, but phpMyFAQ 2.6 will be and it's already implemented (branch: PMF_UTF8 in SVN.

bye
Thorsten
phpMyFAQ Maintainer and Lead Developer
amazon.de Wishlist

jazcyk
Posts: 385
Joined: Wed Sep 07, 2005 1:32 pm

Re: Encoding(?) problem after upgrading

Post by jazcyk » Sun Jul 12, 2009 3:15 pm

I now rather think that it is PHP 5.3 having problems with specific characters outside the ASCII-range. Both PMF 2.0 and 2.5 have this issue with those 'beautiful quotes' by WordPress (and what are also used by several word processors) when using PHP 5.3.0 (and not when using 5.2.x). It is not a PMF upgrade 2.0>2.5 issue as I first thought.

I don't have other PHP applications (also not phpMyAdmin) that can read the data. It is no big issue as we don't plan an upgrade to PHP 5.3 in near future. But additional information/analysis would be nice!

Post Reply