Subject: [Json] Encoding Schemes

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Jun 2013 18:27:22 +0000

Background: the word "encoding" is somewhat imprecise.  From the Unicode
Glossary (http://www.unicode.org/glossary/):

"""
Character Encoding Scheme. A character encoding form plus byte
serialization. There are seven character encoding schemes in Unicode:
UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE.

Unicode Encoding Form. A character encoding form that assigns each Unicode
scalar value to a unique code unit sequence. The Unicode Standard defines
three Unicode encoding forms: UTF-8, UTF-16, and UTF-32.

Unicode Encoding Scheme. A specified byte serialization for a Unicode
encoding form, including the specification of the handling of a byte order
mark (BOM), if allowed.
"""

In UTF-16 and UTF-32 (note: *not* BE or LE), you SHOULD include a Byte
Order Mark, but if it's not there, you assume BE.  From Unicode 6.2,
section 3.10, D98:

"""
The BOM is not considered part of the content of the text.

The UTF-16 encoding scheme may or may not begin with a BOM. However,
when there is no BOM, and in the absence of a higher-level protocol, the
byte
order of the UTF-16 encoding scheme is big-endian.
"""

and from D101:

"""
The BOM is not considered part of the content of the text.

The UTF-32 encoding scheme may or may not begin with a BOM. However,
when there is no BOM, and in the absence of a higher-level protocol, the
byte
order of the UTF-32 encoding scheme is big-endian.
"""

One could argue that the current table in section 3 of 4627 constitutes
such a higher-level protocol, but since we haven't specified either UTF-16
or UTF-32 in the table, nor have we specified what to do with BOMs, I
don't think that 4627 currently acts as that higher-level protocol.

As such, I believe that the phrase from section 3 of 4627 that says:

"JSON text SHALL be encoded in Unicode.  The default encoding is UTF-8."

needs a little help.  We probably want to include only the 5 encoding
schemes currently mentioned in the draft, not all 7.  Some suggest text:

"When serialized to an octet stream, JSON text SHALL be encoded in one of
the following Unicode encoding schemes: UTF-8,  UTF-16BE, UTF-16LE,
UTF-32BE, and UTF-32LE.  The default and RECOMMENDED encoding is UTF-8.
Byte Order Marks MUST NOT be used to signal byte order in any encoding."

and then on with the changed version of the table that we've discussed.
This text intentionally allows U+FEFF in strings (where they've been
allowed to date as zero width no-break space), but NOT as the first
codepoint in the stream, where, since U+FEFF isn't in the ws production,
they can't be used.

As well, if Unicode adds another encoding scheme one day, I don't want
4627bis to automatically require support for it unintentionally.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Tue, 18 Jun 2013 14:39:26 -0400

> UTF-32BE, and UTF-32LE.  The default and RECOMMENDED encoding is UTF-8.

Oh no, that SHALL will not fly.  Any encoding (which means "encoding
scheme" in a media-type context) can be used to represent JSON.
Including an EBCDIC variant.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Half the lies they tell about me are true.
        --Tallulah Bankhead, American actress

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Jun 2013 18:54:35 +0000

>Including an EBCDIC variant.

I assume you're talking about UTF-EBCDIC from tr16?  I don't see how you
could auto-determine the encoding for that.  How about this:

"""
Without an external mechanism that specifies encoding, when serialized to
an octet stream, JSON text SHALL be encoded in one of the following
Unicode encoding schemes: UTF-8,  UTF-16BE, UTF-16LE, UTF-32BE, and
UTF-32LE.  The default and RECOMMENDED encoding is UTF-8.

Note: the MIME type registered in section 6 does not specify a mechanism
to specify the encoding scheme, so when used in a MIME context, one of the
above encoding schemes MUST be used.

Other Unicode encoding schemes MAY be used, but such octet streams cannot
have their encoding scheme automatically detected and SHOULD NOT be
assumed to interoperate with existing software.
"""

-- 
Joe Hildebrand

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 18 Jun 2013 20:11:48 +0100

> assumed to interoperate with existing software.
I like this wording, because it is somewhat compatible with the current
state of affairs.
Unlike XML, where most encodings can be detected from content (including
even EBCDIC, given xml declaration), JSON decoders have to rely on small
subset of encodings for auto-detection, and external encoding information
for others.

-+ Tatu +-

---

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 18 Jun 2013 12:18:51 -0700

> Including an EBCDIC variant.

Do you mean JSON embedded in some other document format, or application/json? How do you align this statement with the current "JSON text SHALL be encoded in Unicode" in section 3 and with the encoding considerations and the lack of a charset parameter in section 6?

Norbert

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Jun 2013 21:20:56 +0200

> Any encoding (which means "encoding

We are confusing JSON with JSON again.

JSON the interchange format, which specifies a sequence of characters, certainly can be encoded in any character encoding scheme that is able to represent the characters that are needed in a specific case.

JSON the media type (application/json) is specifically limited to UTF-8 (and theoretically the two or possibly four other character encoding schemes listed in RFC 4627; the RFC isn't quite consistent here).
There is no way to use another encoding in that media type as there is no charset parameter in the media type.
(This is intentional and certainly is not a candidate for change.)

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Jun 2013 12:34:12 -0700

<no hat>

> JSON the interchange format, which specifies a sequence of characters, certainly can be encoded in any character encoding scheme that is able to represent the characters that are needed in a specif…

...as long as the receiver can determine the encoding scheme. Otherwise, there is no chance for meaningful interop.

> JSON the media type (application/json) is specifically limited to UTF-8 (and theoretically the two or possibly four other character encoding schemes listed in RFC 4627; the RFC isn't quite consiste…

Can you point to the text in the draft that supports that statement? I see the opposite:
   Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

     JSON may be represented using UTF-8, UTF-16, or UTF-32.  When JSON
     is written in UTF-8, JSON is 8bit compatible.  When JSON is
     written in UTF-16 or UTF-32, the binary content-transfer-encoding
     must be used.

> (This is intentional and certainly is not a candidate for change.)

It would certainly be my preference to not add a charset parameter in the -bis document; that would seem like a major change.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Tue, 18 Jun 2013 15:56:00 -0400

Norbert Lindenberg scripsit:

> Do you mean JSON embedded in some other document format, or
> application/json?

Either.

> How do you align this statement with the current "JSON
> text SHALL be encoded in Unicode" in section 3

Like Alice, I don't believe there's an atom of meaning in it.  "Unicode"
is not the name of an encoding.

> and with the encoding
> considerations and the lack of a charset parameter in section 6?

If there's no charset parameter, there isn't one.  Otherwise, section 6
only explains what Content-Transfer-Encoding to use in various cases,
without saying that no other cases are supported.

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
Pour moi, les villes du Silmarillion ont plus de réalité que Babylone.
		--Christopher Tolkien, as interviewed by Le Monde

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Jun 2013 21:57:28 +0200

>     written in UTF-16 or UTF-32, the binary content-transfer-encoding

You listed two of the three places, the third is in section 3, which doesn't really list UTF-16 but UTF-16 "BE or LE" (same for UTF-32).  Note that these are three different character encoding schemes (CESs), so it is not clear which ones are actually meant.

   Since the first two characters of a JSON text will always be ASCII
   characters [RFC0020], it is possible to determine whether an octet
   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
   at the pattern of nulls in the first four octets.

           00 00 00 xx  UTF-32BE
           00 xx 00 xx  UTF-16BE
           xx 00 00 00  UTF-32LE
           xx 00 xx 00  UTF-16LE
           xx xx xx xx  UTF-8

[Obligatory Unicode bashing: Giving one of the three encoding schemes (CESs) for the encoding form (CEF) UTF-16 the same name, i.e., "UTF-16", must have been decided by a very cruel person.]

(Strictly speaking, the other mentions of UTF-16/-32 might be the encoding form, not the scheme, but the RFC simply isn't completely specified here.  I think the only reasonable way to read this has the same result as what Joe is proposing, but some text interpretation is required.  Clearly, the text in section 3 does not work with the BOM-based CESs.  But you might also read the text in 6 as asking for UTF-16 CES and the text in 3 then excluding BOMs so the UTF-16 CES is implicitly big-endian.  So much effort for something so theoretical.)

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Tue, 18 Jun 2013 16:05:41 -0400

Carsten Bormann scripsit:

> Clearly, the text in section 3 does not work with the
> BOM-based CESs.  But you might also read the text in 6 as asking for
> UTF-16 CES and the text in 3 then excluding BOMs so the UTF-16 CES is
> implicitly big-endian.  So much effort for something so theoretical.)

Unfortunately, it isn't even clear that it means that.  It could simply
mean "UTF-16, either big-endian or little-endian".

I am always astonished when people hold up RFC 4627 as some kind of
masterpiece of standards writing.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
Original line from The Warrior's Apprentice by Lois McMaster Bujold:
"Only on Barrayar would pulling a loaded needler start a stampede toward one."
English-to-Russian-to-English mangling thereof: "Only on Barrayar you risk to
lose support instead of finding it when you threat with the charged weapon."

---

From: Nico Williams <[email-masked]>
Date: Tue, 18 Jun 2013 15:32:31 -0500

> mean "UTF-16, either big-endian or little-endian".

I read it as "UTF-16, big-endian or little-endian, without BOMs".  I
read the RFC as specifying that JSON encodings are (not normative)
UTF-8, UTF-16 (either endianness, no BOM), or UTF-32 (either
endianness).  It may well be that JSON is only ever sent as UTF-8...

...But then, following the thread on what a JSON string is, I now
understand that JSON strings generally contain Unicode character data
but really they may contain any codepoints from the BMP, and, really,
any 16-bit values, with some values having to be escaped.  Of course,
given the whole discussion re: surrogates, JSON uses neither UTF-8 nor
CESU-8, but an amalgam of sorts, so, to wrap it up, JSON is intended
to be encoded in UTF-8 (and maybe -16 and -32), with all the caveats
about surrogates getting smuggled in because bad Unicode character
data in JSON strings bleeds into the encoding.

Nico
--