Unicode chars incorrectly shown in native apps with Ext.device.Notification.show()

Ext version tested:

• Sencha Touch 2.0.1.1 (commercial)
• Sencha SDK Tools 2.0.0-beta3-windows

Mobile platform tested against:

• Apple iOS 5.1.1

Description::

When using UTF8-encoded JS strings for title, message, or buttons parameters of Ext.device.Notification.show(): Characters with Unicode codes beyond U+007F are not displayed properly in the dialog window appearing on the device.

Steps to reproduce the problem:

• Create a Sencha Touch 2 application which uses Ext.device.Notification.show() to display a native iOS dialog with some text in it
• Pass UTF8-encoded JS strings to title, message, and buttons parameters of Ext.device.Notification.show()
• Ensure that the strings contain Unicode characters with codes beyond U+007F (e.g. use some Cyrillic characters in the U+0400 - U+04FF range)
• Build the native app for iOS with Sencha SDK Tools and install it on iPhone or iPad
• Run the application and trigger calling the native dialog on the device screen

The result that was expected:

• The dialog is show with all the UTF8 characters properly displayed

The result that occurs instead:

• The dialog is shown where each UTF8 character beyond U+007F is replaced with a pair of completely different UTF8 characters

Additional comments:

It seems like NSString is not properly created (encoded) from the original JavaScript string in stbuild, in the current implementation of Notification.show().

Debugging already done:

• Please, see my further analysis and suggestions on the likely root-cause in my next post below, inside this thread.

Possible fix:

• not known for this reported case with native iOS dialogs called from Sencha Touch 2 apps and with the usage of stbuild (using non-native dialogs and using PhoneGap is not suitable for my purposes at the moment)

Any comments or suggestions on this issue from the Sencha team will be appreciated.

Thanks for the report

@mitchellsimoens Okay. After some extra investigation, I can tell more specifically where the bug is likely hides (though, again, I cannot see the original code of stbuild, thus I am guessing to some extend):

When calling methods of Ext.device.Notification.Sencha or any other Ext.device Sencha-interfaces to natives iOS APIs, those are sent as HTTP requests to the built-in HTTP server (running inside the native iOS application built with stbuild). Preparing the URI for that GET request to the built-in server is done using Ext.Object.toQueryString(), which, in turn, calls JavaScript native encodeURIComponent(). Thus all the JS strings passed as parameters to the native API calls inside stbuild eventually become URI params. And the Unicode characters beyond U+007F inside them are properly percent-encoded as UTF8 characters.

What happens then is that, for some reason, the embedded HTTP server decodes that URI string so that each percent-encoded character is decoded as a stand-alone code between 0x00 and 0xFF, not as a UTF8-encoded character (which may occupy several encoded bytes). And that is causing the problem that we see.

For example, the Russian character Б (U+0411) in a JS-string will be encoded by encodeURIComponent() as Б. Upon receiving on the HTTP server side, it will be decoded as two separate characters with codes U+00D0 and U+0091 respectively. And thus in the dialog shown we will see character Ð (for U+00D0) and a "space" (for U+0091, which is a control character in Unicode).

Based on that, I think the issue is not specific to Notification.show() and must likely affect any API calls accessible from JS inside stbuild when those accept string parameters. I believe that it would also be wise for developers to check if there maybe a similar issue with the string contents of replies coming to the JS part from the build-in HTTP server in stbuild.

I hope the notes above will be of some help and will let the Sencha developers to find and fix the code in stbuild. And I hope we can see an improved version of Sencha SDK Tools with that fix rather soon...

we fixed this problem some time ago, there is a patch, please try it here: https://dl.dropbox.com/u/8793605/stbuild.pkg or here for windows: https://dl.dropbox.com/u/8793605/stbuild.msi

@voloshyn @mitchellsimoens I have tried the suggested patched version of stbuild available at the link above. Alas, it did not work for me. See all details below.

1) After installing the updated stbuild with stbuild.msi: I cannot build my native iOS apps anymore using command "sencha app build native". It all goes well till the last stage when stbuild is invoked and then (right after the info message that "The application was successfully packaged") the sencha command is aborted with the following message:

Bundle Seed ID is missing from config file
Failed to package application

Based on that, it looks to me much like it could possibly be fixed quickly, by manually editing an appropriate config file inside stbuild installation dir (C:\Windows\stbuild), but I was not yet able to dig in and figure out what specific file and change that should be. Indeed, if you can advise me on that, I can edit myself and then hopefully test the fix in the way you'd expected me to.


2) As I was not able to build with the new stbuild, I have re-installed Sencha SDK Tools (2.0.0-beta3): After that the native iOS apps can be built successfully again. And, of course, the issue with Unicode chars passed to native wrappers is also back again.

Then I tried to leave the old installation of stbuild in place, but manually replace the stbuild_temporary file in C:\Windows\stbuild\st-res\templates\stbuild_template_dev.app with the updated stbuild_temporary file taken from stbuild.pkg that you pointed in your message above. With that change I could build the native app. And checking the contents of my app directory under ./build/native/ I can see that the stbuild_temporary file in it is the new one.

That native application is successfully loaded on my iPhone (with iOS 5.1.1) and it runs, but the issue with Unicode is still there, no differences from what I described originally.

3) Just in case, I also tried replacing both stbuild_temporary and Info.plist inside C:\Windows\stbuild\st-res\templates\stbuild_template_dev.app. using the appropriate files from stbuild.pkg. In that case I can successfully build the native iOS app with "sencha app build native", it is reported to be signed successfully. But when I drag and drop it to iTunes (version 10.6.3.25 for Windows) it does not appear in the list of applications -- I tried several times and it is the same. That is unlike the case 2 described above, where I was not replacing Info.plist.


Certainly the cases 2 and 3 may be not too indicative as I don't have enough knowledge about how stbuild functions: I can easily suppose that I need to use both the updated stbuild_temporary and the updated Info.plist and the new stbuild to be able to correctly create the native iOS app. Those two cases only reflect my attempts to see if there could be a quick fix for non-working "sencha app build native" after installing the updated stbuild.

Anyway, your comments and suggestions on the above will be appreciated.

@voloshyn @mitchellsimoens Okay, I have eventually found the info on how to make the updated stbuild successfully building native iOS apps. I am sorry, I was simply not aware of the bundleSeedId parameter in the config file. (And, somehow, in the original installation of SDK Tools [2.0.0-beta3] that specific parameter does not appear at all in the packager.json file generated with "sencha generate app". Yet the stbuild coming with that original version of SDK Tools does create the native apps successfully without that config parameter being specified. Anyways, I am aware now and will remember.)

Now, with the seed ID param in the config and the updated stbuild (installed from the stbuild.msi package suggested above), I have quickly re-built my app and installed it on my iPhone. Alas, I can just confirm that the Unicode issue that I have reported is still here. No difference at all. Thus I assume that some more investigation would need to be done by Sencha developers. And I will be more than glad to help you, guys, with the specific information, if you need any, or to test more of preliminary builds of stbuild, if any. Feel free to give me a shout on that, if you want.

@voloshyn @mitchellsimoens Also, just in case, I am attaching an archive with a simple test app to reproduce and check the issue with Unicode. It contains the sources (note that the standard-generated ./resources and ./sdk directories are not included). Update packager.json (I have removed developer-specific params that I use.)

After starting the app, simply enter the hex code of a Unicode character (just the hex digits of it, without prefixing "0x" or alike.) Then press the button to see the expected and actual results (using Sencha-based and native dialogs respectively). I have also added a screenshot, which should be self-explaining.

Hopefully this can be of help to you.

After testing it, I can now confirm that the issue is fixed when using Sencha Cmd 3.0.0.122 to build native iOS apps. Thanks for that, guys!