Subject: [Json] Proposed minimal change for strings

From: Paul Hoffman <[email-masked]>
Date: Tue, 2 Jul 2013 16:27:15 -0700

<chair hats on>

Do either or both of these proposals work for people in the WG?

--Matt Miller and Paul Hoffman

The following are proposed minimal changes to make the JSON spec interoperable with respect to
which unescaped code units are legal in strings.

Proposal 1 (allow all code units in their unescaped form):

In section 1 (Introduction):
  Change the sentence about Unicode characters to:
    A string is a sequence of zero or more Unicode code units [UNICODE].
In section 2.2 (Strings):
  Leave the production for "unescaped" as-is.
In section 3 (Encoding):
  Add "Some strings, notably those that have unescaped surrogate code units
  (value 0xD800 to 0xDFFF), cannot be encoded in UTF-8."

Proposal 2 (prohibit unescaped surrogates):

In section 1 (Introduction):
  A string is a sequence of zero or more Unicode scalar values [UNICODE].
In section 2.2 (Strings):
  Change the production for "unescaped" to be:
    unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

---

From: John Cowan <[email-masked]>
Date: Tue, 2 Jul 2013 21:26:14 -0400

>     A string is a sequence of zero or more Unicode code units [UNICODE].

I object to this, because "code unit" is not well defined without
mentioning its size.  I suppose it is 16-bit code units that are meant,
but if so, that must be said.  "If Parliament does not mean what it says,
it must say so."

>   (value 0xD800 to 0xDFFF), cannot be encoded in UTF-8."

While this statement is true, it is not the whole truth, which is that
unescaped surrogate code units cannot be encoded in *any* encoding.
Indeed, it is useless to permit unescaped surrogate code units at all.
Therefore, I reject this proposal in toto.

>     unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

I approve of this proposal.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
"The exception proves the rule."  Dimbulbs think: "Your counterexample proves
my theory."  Latin students think "'Probat' means 'tests': the exception puts
the rule to the proof."  But legal historians know it means "Evidence for an
exception is evidence of the existence of a rule in cases not excepted from."

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 2 Jul 2013 20:07:30 -0700

<hats on>

> but if so, that must be said.  "If Parliament does not mean what it says,

It does not appear that the Unicode Standard requires mentioning its size, at least in the definition of "code unit" in D77.

> unescaped surrogate code units cannot be encoded in *any* encoding.

John is correct. After further review, we should have worded the above as "...cannot be encoded in any Unicode encoding form".

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Tue, 2 Jul 2013 22:44:14 -0500

> Proposal 1 (allow all code units in their unescaped form):

Huh?  Do you mean that any code unit may be allowed if escaped?  Or
did you really mean that any code unit may be sent unescaped?
Certainly the latter would be bad (e.g., double-quotes and newline
must be escaped at the very, very least).

>     A string is a sequence of zero or more Unicode code units [UNICODE].

These aren't Unicode things though.  They are 16-bit values that
usually can be interpreted as part of UTF-16-encoded Unicode text.

>   (value 0xD800 to 0xDFFF), cannot be encoded in UTF-8."

Unescaped and *unpaired*.

Note that parsers can unescape escaped characters/code points/code
units, and many do.  Therefore banning unescaped unpaired surrogate
halves is not sufficient.  We must say something about what parsers
SHOULD do when they see these, whether escaped or unescaped.

>     unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

I don't think we need to make this change: parsers can be expected to
do something with unpaired surrogate halves, like either throw them
away or escape them (if it makes sense in whatever context the parser
is executing in) or even output a binary string instead of a text
string.  Certainly, if we don't make this change, then we should say
something about what parsers SHOULD do in the face of such things.
Anyways, I'd +1 this, but we need to discuss what to do with these
when they are escaped.

Nico
--

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 3 Jul 2013 13:56:37 +1000

> The following are proposed minimal changes to make the JSON spec
> interoperable with respect to which unescaped code units are legal in
> strings.
> 

Yes, proposal 2 works for me.

--
James Manger

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 00:41:50 -0400

Paul Hoffman scripsit:

> It does not appear that the Unicode Standard requires mentioning its
> size, at least in the definition of "code unit" in D77.

That is so.  But are we to understand, then, that a JSON string is a
sequence of code units of some unspecified size?  For example,
is the 8-bit code unit sequence <FF, FE, FF, FE> a JSON string?
How about the 32-bit code unit sequence <ABCDABCD, 10101010>?
Might 3-bit or 17-bit or 99-bit code unit sequences be allowed also?
Surely not.

> John is correct. After further review, we should have worded the above
> as "...cannot be encoded in any Unicode encoding form".

As noted, that should have been "unpaired unescaped surrogate code units".

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have seen farther than others, it is because I was standing on
the shoulders of giants.
        --Isaac Newton

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 00:53:35 -0400

> SHOULD do when they see these, whether escaped or unescaped.

Parsers may safely report 16-bit code units as 16-bit unsigned integers.

-- 
Verbogeny is one of the pleasurettes    John Cowan <[email-masked]>
of a creatific thinkerizer.             http://www.ccil.org/~cowan
   --Peter da Silva

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 3 Jul 2013 08:28:45 -0700

<no hat>

On Jul 2, 2013, at 8:44 PM, Nico Williams <[email-masked]> wrote:

> Huh?  Do you mean that any code unit may be allowed if escaped?  

That is exactly what the current document says, I believe. Do you see anything in the grammar that says differently?

> These aren't Unicode things though.  

Yes, they are. See the Unicode Standard, definition D77. Do you see anything in the Unicode Standard that disagrees with D77?

> Unescaped and *unpaired*.

No, any surrogate code point. RFC 3629, the IETF's definition of UTF-8, says:
   The definition of UTF-8 prohibits encoding character numbers between
   U+D800 and U+DFFF, which are reserved for use with the UTF-16
   encoding form (as surrogate pairs) and do not directly represent
   characters.
Similar language is used in The Unicode Standard's definition of UTF-8 (D92), and of UTF-32 (D90).

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 3 Jul 2013 08:31:24 -0700

<no hat>

On Jul 2, 2013, at 9:41 PM, John Cowan <[email-masked]> wrote:

> That is so.  

Good.

> But are we to understand, then, that a JSON string is a
> sequence of code units of some unspecified size?  

That appears to be the case for the current document. If you have a preferred size, by all means propose it to the list. If we can get rough consensus on that, it would help this discussion.

> As noted, that should have been "unpaired unescaped surrogate code units".

That is only true for UTF-16. The definitions of UTF-8 and UTF-32 say that you cannot encode any surrogate code points, paired or not.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 12:02:28 -0400

>    encoding form (as surrogate pairs) and do not directly represent

You are conflating code points (integers in the range 0-0x10FFFF,
definition D10) with code units (bit-strings of a specified length,
definition D77).  The code *points* corresponding to surrogates
(0xD800-0xDFFF) cannot be encoded with any encoding.  The 16-bit code
*units* corresponding to surrogates are used in pairs to represent
characters from U+10000 to U+10FFFF.  There are, obviously, no such
8-bit code units, as the numbers involved will not fit in 8 bits, and
the equivalent 32-bit code units are not used.

Now you make me wonder if your proposal 1 was supposed to be about code
points rather than code units.

(from another email in this thread)

> That appears to be the case for the current document. If you have a
> preferred [code unit] size, by all means propose it to the list. If
> we can get rough consensus on that, it would help this discussion.

In the JSON context, the only code unit size that makes sense is 16-bit,
because JavaScript (from which JSON comes) deals in 16-bit code units,
sequences of which are called "strings".  But if you mean to talk of code
points rather than units, then of course there is no size to specify,
as integers don't have a size.

> That is only true for UTF-16. The definitions of UTF-8 and UTF-32 say
> that you cannot encode any surrogate code points, paired or not.

This is the same conflation.

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
Sir, I quite agree with you, but what are we two against so many?
    --George Bernard Shaw,
         to a man booing at the opening of _Arms and the Man_

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 11:41:45 -0500

> That is exactly what the current document says, I believe. Do you see anything in the grammar that says differently?

No, but I couldn't understand what you wrote, which was

| Proposal 1 (allow all code units in their unescaped form):

surely you did not mean that all 16-bit code units could be sent
unescaped.  But that is how I read that.  I figured you'd typoed.

> Similar language is used in The Unicode Standard's definition of UTF-8 (D92), and of UTF-32 (D90).

Yet there are ECMAScript applications that send these.  In their
escaped form there's no conflict with UTF-8 (nor UTF-32, nor even in
UTF-16 if unpaired).  In their unescaped form there is most definitely
a conflict.

Which brings up: what should a parser do when it sees escaped code
units?  Should it attempt to unescape them in the parsed result?  What
if the escaped code unit cannot be unescaped because it would result
in invalid UTF-8/16/32?  RFC4627 says nothing about this.

My proposal is to allow any code units as long as they are escaped if
they are unpaired surrogate code points or as long as they are
unescaped and properly represented in the UTF of the JSON document
otherwise (i.e., if a pair of surragates appear in UTF-16 then when
re-encoding to UTF-8 the result must be UTF-8, not CESU-8, and the
pair must be decoded into a code point then re-encoded in UTF-8).

Nico
--

---

From: Norbert Lindenberg <[email-masked]>
Date: Wed, 3 Jul 2013 22:06:36 -0700

>   A string is a sequence of zero or more Unicode code units [UNICODE].

This should be "code points", not "code units".

If you allow a sequence of code units, you might get UTF-8 code units in random sequence, or a mix of UTF-8 code units and UTF-16 code units, which wouldn't make sense at all.

Also, JSON documents have to be understood at the level of code points, because their code unit sequences are often changed in transit - e.g., from UTF-16 code units in JavaScript to UTF-8 code units over the network back to UTF-16 code units in Java - with no intended change in content.

See also
http://www.unicode.org/glossary/#code_point
http://www.unicode.org/glossary/#code_unit

> (value 0xD800 to 0xDFFF), cannot be encoded in UTF-8."

As above, "surrogate code units" should be "surrogate code points", therefore U+D800 to U+DFFF. Also, it's exactly the strings containing unescaped surrogate code points that cannot be encoded in well-formed UTF-8, so we can get rid of "some" and "notably those".

There are a few additional references to "character" that would have to be changed to "Unicode code point" to make this proposal complete. See
http://www.ietf.org/mail-archive/web/json/current/msg00870.html

Norbert

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 5 Jul 2013 03:09:15 +0000

>Proposal 1 (allow all code units in their unescaped form):

vehemently -1

>     unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

That works for me.  I think that the current text implies this by limiting
the valid encodings to UTF-8, UTF-16BE, UTF-16LE, UTF-32BE, and UTF-32LE,
none of which can legally encode unescaped surrogates.  The ABNF is
therefore a spec bug that needs to be cleaned up, and any existing code
that emits unescaped values in the range %xD800-%xDFFF when not producing
UTF-16, or invalid combinations of those values when producing UTF-16 has
a bug that needs to be fixed.  Note: I don't care if parsers choose to do
something heroic in the face of these buggy implementations instead of
rejecting the input (viva Postel!).

-- 
Joe Hildebrand

---

From: Jorge <[email-masked]>
Date: Fri, 5 Jul 2013 12:01:03 +0200

>> Proposal 1 (allow all code units in their unescaped form):

But, don't the JavaScripts in ˜ every browser in the world permit it?
-- 
( Jorge )();

---

From: Stephen Dolan <[email-masked]>
Date: Fri, 5 Jul 2013 16:50:32 +0100

> But, don't the JavaScripts in ˜ every browser in the world permit it?

They also permit numbers to be NaN or Inf, objects to contain keys
which are not strings, array syntax to include a trailing comma, and
so on. "Javascript permits it" does not imply that JSON does or
should.

Regarding the proposals: +1 for proposal 2, and very much -1 for
proposal 1 (both because I disagree that unpaired surrogates should be
part of JSON, and because it is badly worded, using the term "code
unit" in strange ways).

Stephen

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 05 Jul 2013 19:15:53 +0200

>  A string is a sequence of zero or more Unicode scalar values [UNICODE].

Section 1 describes the JSON data model and the proposed change would
prohibit escaped unpaired surrogates, implying that something like

  JSON.stringify([ JSON.parse('["..."]')[0].substring(0, 140) ]);

occasionally returns a string that is not JSON text, even if the input
to `JSON.parse` is proper JSON text, when there is a non-BMP character
in just the wrong place (`substring` counts UCS-2 code units).

With JSON.stringify this can be avoided using a replace function like

  JSON.stringify(obj, function(key, value) {
    if (typeof value == 'string') {
      return value.replace(/.../g, "\uFFFD");
    }
    return value;
  })

but then you have dataloss, unnecessarily so because you might just be
putting an object into a datastore for later use and ecmascript doesn't
mind unpaired surrogates, and the code is slower and harder to read.

It seems pretty clear to me that decoders are a better place to deal
with unpaired surrogate code points. In environments where they pose no
problem you probably want them to roundtrip, in others substituting is
a better option, and in yet others you might want to fail hard on them.

Prohibiting unpaired surrogate escapes would place the burden to avoid
them on senders, and I have not seen in argument why that is a better
option, all things considered.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Jorge <[email-masked]>
Date: Fri, 5 Jul 2013 22:36:29 +0200

On 05/07/2013, at 17:50, Stephen Dolan wrote:
> "Javascript permits it" does not imply that JSON does or
> should.

But the built-in *JSON* in every current browser allows it:

JSON.parse(JSON.stringify({a:'_\ud800_'})).a.charCodeAt(1).toString(16)
"d800"
-- 
( Jorge )();

---

From: "Manger, James H" <[email-masked]>
Date: Sat, 6 Jul 2013 23:43:28 +1000

> mind unpaired surrogates, and the code is slower and harder to read.

It is only "unnecessary dataloss" when you are certain a value will only be handled by ECMAScript -- in which case relying on ECMAScript's (reasonable) extension beyond JSON is ok.
On the other hand, what happens if the datastore understands JSON and tries to store the non-dataloss values as UTF-8? That is likely to fail. In this case "dataloss" that the app controls (and can avoid by not splitting a non-BMP character) is much better that unpredictable behaviour (works or fails) from different JSON-compliant datastores.

> option, all things considered.

I think your previous paragraph provides the argument. If different decoders can legitimately roundtrip, substitute, or fail on parsing an unpaired surrogate escape than senders need to be warned (in big flashing lights) of this non-interoperable behaviour. The best way to do that is to exclude unpaired surrogate escapes from the specification of JSON.

--
James Manger

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 06 Jul 2013 16:46:55 +0200

* Manger, James H wrote:
>It is only "unnecessary dataloss" when you are certain a value will only 
>be handled by ECMAScript -- in which case relying on ECMAScript's 
>(reasonable) extension beyond JSON is ok.

This is not an ecmascript-specific problem. Environments that can encode
unpaired surrogates in the native string type are the norm; Perl can

  % perl -MJSON -e "print JSON->new->ascii->encode([chr(0xd800)])"
  ["\ud800"]

and as far as I am aware so can Java, C#, C++, Python, and many others,
and for many of them it is easy to create lone surrogates by accident.

>I think your previous paragraph provides the argument. If different 
>decoders can legitimately roundtrip, substitute, or fail on parsing an 
>unpaired surrogate escape than senders need to be warned (in big 
>flashing lights) of this non-interoperable behaviour. The best way to do 
>that is to exclude unpaired surrogate escapes from the specification of 
>JSON.

But receivers need to be warned that they will likely have to deal with
JSON documents with unpaired surrogate escapes in strings, and the best
way to do that is to include them in the specification...
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/