Features / Usability

Hi hrsms,

If this issue happened already in your previous upgade, it might be related then to a character encoding issue in your database. That was known in the upgrade from before Tiki5 to Tiki6 onwards. You might want to try with a test db clone of your site, to perform the appropriate upgrade steps to fix the enconding issue in your database, to see whether this fixes your issue once and for all.

See this:
https://doc.tiki.org/Upgrade#If_you_need_to_fix_the_encoding_issue

and

https://doc.tiki.org/Understanding+Encoding

I read https://doc.tiki.org/Upgrade#If_you_need_to_fix_the_encoding_issue. They seem familiar. I think I did this in my upgrade from 4 to 6.

I created a copy of my 9.2 instance and performed all of these steps again. I believe I followed all the instructions correctly. I did not get an "encoding issue" warning is step 5 of tiki-install.php (evidence I did do this once before). I proceeded as if it had said latin1, including the "fix double encoding" step. I know it was latin1 when it was created. I also tested to confirm latin1 is still the default used when I create a new database via cPanel on my shared host.

The only difference between this test and my first migration to 11.2 is in the "Set the database connection" step in the "Character set" section. Here it says for the enforce UTF-8 setting "Highly recommended for new installations. However, if you are upgrading or migrating a previous tiki database, you are recommended to uncheck this box". This contradicts the instructions in https://doc.tiki.org/Upgrade#If_you_need_to_fix_the_encoding_issue. I unchecked this box in my first migration. In the this most recent test I left it checked.

The results are nearly the same. Instead of a blank page when I edit, everything after the first special character is gone. Again, if I save it in this state and rollback to the prior state, everything is restored.

Very odd.

I've followed (correctly, I hope) the instructions in https://doc.tiki.org/Upgrade#If_you_need_to_fix_the_encoding_issue, yet I am still having very odd behaviors.

If this is indeed an encoding issue, is there another approach I can take to finding and fixing the issues in the database? Some manual SQL queries and hand-editing?

If it didn't work for you, then I have no more clues (it did work for me and other people yers ago when we had encoding issues).

Maybe just check with phpmyadmin or similar that you don't have any table or db encoding still in non-utf-8 encodind. If so, you should be able to change the encoding of the remaining table or db directly in phpmyadmin.

If this is not enough, then I would suggest that you look for hiring some consultant highly skilled in mysql encoding (not me, sorry; I just knew the recipies that worked for most of us).

I browsed the tiki_pages table and looked at pages I am having problems with. I can see that the characters are not displayed as viewed online. For instance, the character 'é' is shown as "é" and '»' is shown as ». I don't know if this is "double encoding", some other problem, or is actually expected.

Does the above indicate I did not correctly perform the steps to force utf-8, or the steps to avoid double encoding?

The good news is that I replaced the text directly in phpMyAdmin and I have seen improvement in the behavior (testing not yet complete). So I am cautiously optimistic I can fix this manually in the tiki tables. Fortunately I am not looking at a mountain of work. I'll report back when I'm done.

If I paste characters from the Windows character map into the tiki_pages table data field, do I run the risk of some incompatibility? Does the Windows character map include items that are not UTF-8 compliant? It is unlikely I would need any such characters (all I plan to use are visible in the tiki character map viewed in the editor), but the quesion is worth asking all the same.

I did not do what I described in my previous post. Instead, I tried one more time to upgrade from my original 9.2 install. This time everything seems to be fine. This is the third attempt. Only two things have changed between each attempt.

a) Character set - Always force connection to use UTF-8
https://doc.tiki.org/dl343?display
"Highly recommended for new installations. However, if you are upgrading or migrating a previous tiki database, you are recommended to uncheck this box"

b) Fix double encoding
https://doc.tiki.org/dl347?display

First attempt:
a)Since I am upgrading, I followed the instructions and unchecked this box
b)Not run
Result: as described in the first message of this thread and in https://tiki.org/tiki-view_forum_thread.php?forumId=6&comments_parentId=50424#threadId6

Second attempt:
a) after reading elsewhere that this should be left checked (contradicts the tiki-install message!!), I left it checked
b) since someone had suggested I may have a double encoding issue, I ran this
Result: As described above at Tue 03 Dec 2013 03:13. Edits sometimes produced more odd characters, however.

Third attempt:
a) Left checked
b) Not run
Result: so far, everything I've looked at after the migration looks correct, and edits behave as expected!!

Hi hrsms

Glad to hear that.

Would this message have been clearer for you then (to guide you to your successful path), according to your experience?:

"..AND you know what you are doing..."

I had to read that a couple of times to convince myself you weren't trying to be rude to me. It's clear now. Perhaps I should have finished my morning coffee first 😉

Yes, your statement is better, presuming my situation is typical. I think I went through steps to fix the encoding when I went from 6 to 9 (not using the link you gave above, but with help from user 'fabricius'). Perhaps this is the difference, and if someone is migrating an older db had not done already fixed the db encoding, they should leave it unchecked.

Was the encoding changed at some specific point, like 5.x? Maybe it should read:

"Highly recommended for new installations and upgrades. However, if you are upgrading or migrating a database created 4.x or earlier that has NOT been converted to use UTF-8 encoding, you are recommended to uncheck this box"