Subject: [Json] That encoding thing

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Dec 2013 10:54:26 -0800

<hat on>

The following is technically correct and should at least partially appease the pedants among us. 

s/JSON text SHALL be encoded in Unicode./JSON text SHALL be encoded in one of the character encoding schemes defined by Unicode./

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 20:01:44 +0100

> s/JSON text SHALL be encoded in Unicode./JSON text SHALL be encoded in one of the character encoding schemes defined by Unicode./

Both statements are technically correct (unless read with —pedantic).  The second statement says less than the first.  I don’t think that change is intended.  We are not only importing the CES from Unicode.  (Now that attention to detail may be pedantic in its own right, sorry about that, but I’m again trying to anticipate how this will be read.)

Grüße, Carsten

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 05 Dec 2013 20:54:02 +0100

* Carsten Bormann wrote:
>Both statements are technically correct (unless read with —pedantic).  
>The second statement says less than the first.  I don’t think that 
>change is intended.  We are not only importing the CES from Unicode.  
>(Now that attention to detail may be pedantic in its own right, sorry 
>about that, but I’m again trying to anticipate how this will be read.)

It is necessary to enumerate the encodings because "Unicode" defines
character encoding schemes that cannot be used for application/json
under the rules of RFC 4627. An example is CESU-8. In effect the text
should say that an application/json entity is malformed unless it is a
properly encoded string in UTF-8/UTF-16LE/UTF-16BE/UTF-32LE/UTF-32BE
as far as the rules in RFC 4627 go.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 6 Dec 2013 07:14:47 +0100

On 05 Dec 2013, at 20:54, Bjoern Hoehrmann <[email-masked]> wrote:

> under the rules of RFC 4627. An example is CESU-8.

Well, the Unicode standard doesn’t define them, but there are UTRs that do.
Certainly, neither UTF-EBCDIC nor CESU-8 are valid CES for JSON interchange, and I agree this needs to be more explicit.

> In effect the text

Indeed.  RFC 4627 says so in section 3, and we seem to have lost this information.  (The important observation is that only five out of the seven CES of Unicode are explicitly supported, even though the text elsewhere sloppily talks about the other two, UTF-16 and UTF-32, apparently meaning “UTF-16BE or UTF-16LE” and “UTF-32BE or UTF-32LE”, respectively.  This is all moot in practice where only UTF-8 is used, but important to the language lawyers.)

Grüße, Carsten

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 06 Dec 2013 15:20:01 +0900

On 2013/12/06 4:54, Bjoern Hoehrmann wrote:
> * Carsten Bormann wrote:
>> On 05 Dec 2013, at 19:54, Paul Hoffman<[email-masked]>  wrote:
>>

I think this change is a significant and necessary improvement. But it 
seems both Carsten and Björn disagree. I think this may be because 
"character encoding schemes defined by Unicode" is still slightly 
ambiguous (because it's circumscriptive, rather than using an 
established term). I propose to change this to:

JSON text SHALL be encoded in one of the Unicode Encoding Schemes.
(see Section 2.6 of Unicode 6.3 
(http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf))

This will clearly exclude CESU-8 and UTF-7, as desired.

Regards,   Martin.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 6 Dec 2013 08:39:40 -0800

On Dec 5, 2013, at 10:20 PM, Martin J. Dürst <[email-masked]> wrote:

> I think this change is a significant and necessary improvement. But it seems both Carsten and Björn disagree. I think this may be because "character encoding schemes defined by Unicode" is still slightly ambiguous (because it's circumscriptive, rather than using an established term). I propose to change this to:
> 
> (see Section 2.6 of Unicode 6.3 (http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf))
> 

The sentence that follows the changed one says:

   The default encoding is
   UTF-8, and JSON texts which are encoded in UTF-8 are interoperable in
   the sense that they will be read successfully by the maximum number
   of implementations; there are many implementations which cannot
   successfully read texts in other encodings (such as UTF-16 and
   UTF-32).

If someone wants to encode in UTF-7, they are going to do it regardless of the wording here. Us spending effort to make it harder for them to do something stupid (and some people here felt that someone using UTF-16 was also stupid) is wasted effort.

--Paul Hoffman