Subject: [Json] Complete section 3 proposal

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Jun 2013 20:40:22 +0000

Combining the threads

"""
Without an external mechanism that specifies encoding, when serialized to
an octet stream, JSON text SHALL be encoded in one of the following
Unicode encoding schemes: UTF-8, UTF-16BE, UTF-16LE, UTF-32BE, or
UTF-32LE.  The default and RECOMMENDED encoding is UTF-8.

    Note: the MIME type registered in [Section 6] does not specify such an
external mechanism, so when used in a MIME context, one of the above
encoding schemes MUST be used.

    Since the first code point of JSON text will always be an ASCII
character [RFC0020], it is possible to determine whether an octet stream
is UTF-8, UTF-16BE, UTF-16LE, UTF-32BE, or UTF-32LE by looking at the
pattern of nulls in the first four octets of a stream.  In the following
table "00" corresponds to an octet with value zero, "xx" corresponds to an
octet known to be non-zero, and "??" corresponds to an octet that is not
checked.

    00 00 00 xx  UTF-32BE
    00 xx ?? xx  UTF-16BE
    xx 00 00 00  UTF-32LE
    xx 00 xx ?? UTF-16LE
    xx xx ?? ?? UTF-8

Note: streams less than four octets long are not UTF-32BE or UTF-32LE, and
streams less than two octets long are UTF-8.

    Other Unicode encoding schemes MAY be used, but such octet streams
cannot have their encoding scheme automatically detected as required by
the MIME type in [Section 6], and SHOULD NOT be assumed to interoperate
with existing software.
"""

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Jun 2013 22:49:48 +0200

Looks great, except that the phrase

> in a MIME context

may be not quite what is meant here.

(This paragraph is specifically about application/json; if I want to go ahead and register an application/json-in-zos-ebcdic tomorrow, there should be no problem with me using that once I've been released from the asylum.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Tue, 18 Jun 2013 15:52:28 -0500

> UTF-32LE.  The default and RECOMMENDED encoding is UTF-8.

Note that if a JSON string in JSON data contains unescaped naked
surrogates then the encoding of that data will not be valid UTF-8,
UTF-16, nor, for that matter, CESU-8.  And some implementations
probably produce CESU-8-encoded data.  I'm not sure whether that's
worth stating here or elsewhere, but the fact that there's
not-quite-UTF-8 JSON out there means this SHALL is either
interop-breaking or the matter must be mentioned nearby.

With this caveat, +1.

> encoding schemes MUST be used.

Agreed.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Jun 2013 22:00:56 +0000

On 6/18/13 2:49 PM, "Carsten Bormann" <[email-masked]> wrote:

>(This paragraph is specifically about application/json; if I want to go
>ahead and register an application/json-in-zos-ebcdic tomorrow, there
>should be no problem with me using that once I've been released from the
>asylum.)

Hm.  Maybe the bulk of that graf should be moved to section 6?

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Jun 2013 22:05:23 +0000

On 6/18/13 2:52 PM, "Nico Williams" <[email-masked]> wrote:

>probably produce CESU-8-encoded data.

I think the spirit of 4627 was that it literally be UTF-8, and that all of
those other odd encodings are already non-conformant.  We could always add
a note that says that there has been a history of encodings not being
quite adequately specified, so old software may produce octet streams that
this document doesn't describe.

>I'm not sure whether that's

I agree it might be interop-breaking, but I don't think that's necessarily
the spec's fault.  People will write bad software, particularly when they
don't have test vectors easily at hand for them to probe what they
originally thought were edge cases.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Jun 2013 15:16:36 -0700

On Jun 18, 2013, at 1:40 PM, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> character [RFC0020],

You were so good with using "encoding schemes", then you fell into this. :-) While the reference to RFC 20 is cute, it actually doesn't help here. A more direct statement would be:

=====
Because the first code point of a JSON text will always be a character in the range U+0022 to U+007B,
=====

> (This paragraph is specifically about application/json; if I want to go ahead and register an application/json-in-zos-ebcdic tomorrow, there should be no problem with me using that once I've been r…

=====
so when used in the application/json MIME context, 
=====

---

From: Nico Williams <[email-masked]>
Date: Tue, 18 Jun 2013 17:17:29 -0500

> this document doesn't describe.

We've been over this.  The spirit was that strings were of Unicode
characters, but really they're of code points.  And so on.  And
there's no consensus to break existing encoders.  And so I don't think
we can get consensus for that MUST w/o that note.

> originally thought were edge cases.

Well, if you can get consensus for it...

It's not that people write bad code.  It's that bad *data* exists and
why should encoders look for it (particularly naked surrogates, why
look for them)?  I think in practice not much can be done about this
but note the problem and encourage encoders not to add to this (i.e.,
encoders should escape naked surrogates).

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Jun 2013 09:36:05 +0200

> Because the first code point of a JSON text will always be a character in the range U+0022 to U+007B,

The following six characters can occur at the start of a JSON text:

           %x20                ; Space
           %x09                ; Horizontal tab
           %x0A                ; Line feed or New line
           %x0D                ; Carriage return

           %x5B                ; [ left square bracket
           %x7B                ; { left curly bracket

Four of those six characters are not in that range.
They are all in the ASCII subset of Unicode.

(Of course, that set might increase if we redefine what a JSON text is, but it is likely to stay ASCII.)

Grüße, Carsten

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 19 Jun 2013 14:26:08 +0100

Original Message From: "Joe Hildebrand (jhildebr)"
> streams less than two octets long are UTF-8.

To nit-pic, I think in the UTF-16BE case, by the time you've seen 00 xx you 
don't need anymore characters, so it's line should be:

    00 xx ?? ??  UTF-16BE

Or is the idea to include some redundancy in the detection?

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 19 Jun 2013 06:39:44 -0700

> Four of those six characters are not in that range.

Very good point. s/U+0022/U+0009/.

> They are all in the ASCII subset of Unicode.

The "ASCII subset of Unicode" is not well-defined, and for bonus confusion, includes U+0000.

> (Of course, that set might increase if we redefine what a JSON text is, but it is likely to stay ASCII.)

We are trying to deal with one discussion topic at a time. We are not succeeding, but we are trying.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Wed, 19 Jun 2013 09:56:53 -0400

> confusion, includes U+0000.

Of course it does.  ASCII = ECMA-6 = ISO 646:1991 is, as it proclaims, a
set of 128 characters.  Where's the lack of well-definition?

Or as Irving Berlin didn't quite write:

    There's no ASCII but US-ASCII,
    The only ASCII, I know;
    Everything about it is US-based,
    Everything about it's seven-bit.
    You do that X-three-dot-four wheeler-dealing,
    When you're feeling
    Retro-fit.

    There's no ASCII but US-ASCII,
    It's Unicode's first half-row;
    Though it is a turkey that we know must die,
    While systems chop off the bits that are high,
    It is the only charset that will always fly,
    ASCII, backward we go,
    ASCII, on with the show!

-- 
Verbogeny is one of the pleasurettes    John Cowan <[email-masked]>
of a creatific thinkerizer.             http://www.ccil.org/~cowan
   --Peter da Silva

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 19 Jun 2013 14:47:47 +0000

On 6/18/13 4:16 PM, "Paul Hoffman" <[email-masked]> wrote:

>You were so good with using "encoding schemes", then you fell into this.
>:-) While the reference to RFC 20 is cute, it actually doesn't help here.

I wasn't trying to be cute, I was trying to reduce the size of the diff
from 4627.  I'm fine with changing the proposed text to be more precise.

>A more direct statement would be:
>
>=====

With the suggestion of tab, etc, needing to be included, my preference
would be to say U+0001 to U+007F, rather than calling out all of the
possibilities; this allows us to decouple the "what values can occur at
the top level" conversation from this one.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 19 Jun 2013 14:48:17 +0000

>Or is the idea to include some redundancy in the detection?

+1

-- 
Joe Hildebrand