Callback in iso-8859-1 ?

[Moonix]

hi all,

I use prado 3.0 from SVN with ActiveControls support.

With the autocompleter the callback is encode in utf-8 so I can't see french character (replace by ?)

how can I change the encoding in iso-8859-1 ? perhaps if I can change the header of the callback with
header ('Content-Type: text/html; charset=iso-8859-1');

[wei]

Try to use UTF-8 if you can, because at some point the JSON encode in PHP might need to be used, and it only supports UTF-8.

Wei.

[wastiee]

i've had the same problem today, fixed by some hacks that use iconv to update if the page is a callback, something like:

Code:

$txt1 = 'Some text in ISO-8859-x';
$txt2 = 'Some more text in ISO-8859-x';
if ($this->Page->IsCallback)
{
    $txt1 = iconv('ISO-8859-x','UTF-8',$txt1);
    $txt2 = iconv('ISO-8859-x','UTF-8',$txt2);
}
// ... update text inside controls within TActivePanel with $txt1 and $txt2

wei, can't this be handled by putting an iconv/mb_convert call just before/after the json encode/decode steps? Javascript always uses UTF-8 without caring the page's own encoding. If communication between javascript/prado is only made using json, this may work.

Maybe an application configuration for both the converter function to use and the application encoding; or better, a class like TCharsetConverter, which can have two methods, application_encoding_to_utf8($s) and utf8_to_application_encoding($s), which may be called (if configured within application configuration) each time a JSON encoding/decoding is made, and the application developer can just subclass it and override the methods and configure the xml file, and all works together

-kerem

[wastiee]

I've examined the sources of prado and javascript stuff for the last few hours. This seems to me possible.

First, prado output for callbacks:

There are 5 json encoded data fields and 1 html content field (not json encoded). As follows:

Code:

const CALLBACK_DATA_HEADER = 'X-PRADO-DATA';
const CALLBACK_ACTION_HEADER = 'X-PRADO-ACTIONS';
const CALLBACK_ERROR_HEADER = 'X-PRADO-ERROR';
const CALLBACK_PAGESTATE_HEADER = 'X-PRADO-PAGESTATE';
const CALLBACK_REDIRECT = 'X-PRADO-REDIRECT';
[i]These are from framework/Web/UI/ActiveControls/TActivePageAdapter.php[/i]

CALLBACK_ERROR_HEADER is sent using HTTP HEADER, this is line 320 from TActivePageAdapter.php

Code:

$response->appendHeader(TActivePageAdapter::CALLBACK_ERROR_HEADER.': '.$trace);

All other headers are sent using the HTTP content body.

Prado divides the HTTP content body into subfields, called "content part"s. Each part is of the form

Code:

<!--DelimiterString-->Part's content<!--DelimiterString-->

Delimiter strings are unique for each content part, and they are different among other content part delimiters.

Normally, nearly every content part holds some HTML fragment to be replaced on the page. The association between delimiter strings and the associated controls' client-side identifiers are passed in the CALLBACK_xxxx_HEADER values.

These CALLBACK_xxxx_HEADER values are also passed as part contents (except for CALLBAC_ERROR_HEADER, that's passed thru http header). Their delimiter strings are the 'X-PRADO-xxxx' strings.


This is some example on how a callback reply might be:

Code:

HTTP/1.1 200 OK
Content-Type: [i don't know this, but probably text/html. It is possible to put a charset=something here, using Globalization.charset]
X-PRADO-ERROR: //json encoded utf8 string//

<!--X-PRADO-DATA-->//json encoded utf8 string//<!--X-PRADO-DATA-->
<!--X-PRADO-PAGESTATE-->//json encoded utf8 string//<!--X-PRADO-PAGESTATE-->
<!--X-PRADO-ACTIONS-->//another json encoded utf8 string//<!--X-PRADO-ACTIONS-->
<!--vergverg-->//some control's html content//<!--vergverg-->
<!--cnfy834-->//another control's html content//<!--cnfy834-->

(I've put the line ends myself for readability, the only separator is the comment-looking delimiter strings)

Now, the problem.
1. Browser doesn't find a html head inside the content body, thus, is unable to guess the correct encoding if globalization.charset is not used. Browser thinks this is UTF-8 (in fact it may guess anything, maybe local computer's charset)
2. Controls' html contents are sent using their own encoding, when the application is not UTF-8, this means there is wrong content in the html. Browser thinks this is utf-8, so renders it wrong
3. The problem seems to be solvable with just sending a correct charset= parameter, this will fix the problem for html updates. But we have a another problem, the contents of X-PRADO-xxxx fields. These (probably?) contain information like, the contents of a textbox. These 'data' is not given as html but as data to be parsed by javascript (I guess only tactivepanel is sent as html in practice). If i set a non UTF-8 text to an active control, my head will probably ache

4. A note on "json encoded utf8 string": although json.org says any utf8 character may be passed as is, but may also be encoded in a form like \uXXXX, the implementation prado uses always encodes non-ASCII characters as \uXXXX. This makes the "json encoded utf8 string"s valid for any US-ASCII compatible charset, i mean, whatever charset= is given in the HTTP header.


Enough reverse engineering, now what i think may be feasible is something like:

The json encoding is done only for these PRADO_HEADER_xxxx fields, in TActivePageAdapter.php. It is possible to change encoding from "application's encoding" to utf8 just before TJavascript::jsonEncode is called in this page.

For the "content part"s, no charset conversion is necessary, as long as the charset is specified in http header. Specifying a US-ASCII compatible charset will not break the json-encoded values either, as JSON will encode any character that is not in range 0..127 to some characters in that range.


This is for sending. For receiving data from active controls, we will still have encoding problems, i didn't examined those, but i think it sends data as JSON-encoded too. As javascript is the one that created the data, strings in it are UTF8. After the JSON-decoding, if these strings are converted to application's encoding, well, it seems to me that everything will work perfectly in AJAX for non UTF8 charsets



BTW, Prado rocks with only a change of 1-2 lines in 5-6 places, all of the above is possible

-kerem

[wastiee]

hmm, here are a little more thoughts after more examination.

As Globalization.charset denotes the content-encoding of the pages sent in TPageService, i think, it can be assumed as "charset of the application", not some value just for TPageService. Anyway, the components will render themselves and they will render themselved using their values in viewstate etc, that means not only HTML codes, but any string in memory will possibly in Globalization.charset encoding, if specified.

There are 2 points where JSON conversion is made, the main one is TJSON, and there is a wrapper for it, TJavaScript, the latter tries to bring all javascript-related stuff together in one component. TActivePageAdapter uses TJavascript::jsonEncode() (::jsonDecode()), and TJSONService uses TJSON::encode() (::decode()) directly.

What i think is, if Globalization.charset denotes the content-encoding of the strings in components, then it may be wise to add "if (Globalization.charset is set) convert string from Globalization.charset to UTF-8" to TJSON::encode(), and add "if (Globalization.charset is set) convert string from UTF-8 to Globalization.charset" to TJSON::decode()

For example in this way, if i deploy a TPageService using active controls, or just a TJSONService using another charset than UTF-8, i won't hassle with converting my strings to UTF-8 every time, i will just specify my charset in Globalization.charset (which i already do now), and PRADO will take care of it even if i'm dealing with JSON.


There is a slight problem, not a problem with TJSON::encode(), but ::decode(), what if client sends in unrepresentable characters in my specified encoding. Then this simple assumption will take precedence: if i'm creating a JSON service with specifying my internal encoding different from UNICODE, then that means i don't really care about non-representable strings in my own charset, and if a user sends me such data, there is no problem with throwing an exception, and leaving the request unanswered

any thoughts? especially from prado developers? i'm really very very new to prado, it's been just a few weeks

-kerem

PS: sorry for making this 1 year old post so active, if i had known that i would write so much about this in advance, i would have created a new topic at the beginning

[wastiee]

Hi again, the solution was more simpler than i thought at first

here's the patch for prado/framework/Web/Javascripts/TJSON.php (the source was in 3.1.0RC, i don't know if it's changed in 3.1.0 final)

Code:

148a149,152
>                 $globalization = Prado::getApplication()->getGlobalization(false);
>                 if ($globalization !== null && strtoupper($globalization->getCharset()) != 'UTF-8')
>                     $var = iconv($globalization->getCharset(), 'UTF-8', $var);
>
488a493,496
>                     $globalization = Prado::getApplication()->getGlobalization(false);
>                     if ($globalization !== null && strtoupper($globalization->getCharset()) != 'UTF-8')
>                         $utf8 = iconv('UTF-8', $globalization->getCharset().'//IGNORE', $utf8);
>
758c766
< ?>
\ No newline at end of file
---
> ?>

Well, how this works is as follows;

Globalization.charset is specified in application.xml if you're using something other than UTF-8. This causes Content-type: text/html; charset=iso-8859-xxx to be sent in HTTP headers.

As this already causes setting of html charset encoding, i thought it would be good to use this one for JSON too, anyway, if you're sending your content in something other than UTF-8, you won't expect postbacks in UTF-8, neither you would expect JSON calls (or callbacks).

This patch just converts the strings in the input from Globalization.charset to UTF-8 while encoding JSON, and converts input strings from UTF-8 to Globalization.charset when decoding JSON. It uses //IGNORE parameter of iconv, so if a user enters a character representable in UTF-8 but not in Globalization.charset, it just skips the character without giving problems (if you're using some other encoding, anyway you wouldn't handle non representable characters )

I've been using this solution for a while, i didn't have any problems in 1) Updating TActivePanel contents 2) Updating TActiveDropDownList contents. (i was having utf-8 problems in these two cases before this patch)

-kerem