Subject: [Json] A possible summary of the discussion so far on code points and characters

From: Paul Hoffman <[email-masked]>
Date: Sat, 8 Jun 2013 13:15:42 -0700

No hat at all, just trying to get some focus on the current facts before trying to reach conclusions.

1) Some people have read the following statement from the RFC to mean "only Unicode characters are allowed in strings":
   A string is a sequence of zero or more Unicode characters [UNICODE].

2) The ABNF is more liberal about what can be in a string than that statement:
      char = unescaped /
          escape ( ...
              %x75 4HEXDIG )  ; uXXXX                U+XXXX
      ...
      unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

3) Some JSON parsers enforce (1), rejecting JSON texts that have strings that have some unallowed code points.

4) Some JSON parsers accept strings with all code points.

5) The definition of "Unicode character" has been surprising to some people on this list, and thus might be surprising to some developers and users of JSON.

6) Some people on the list consider some code points that are Unicode non-characters to be more objectionable than other code points.

--Paul Hoffman

---

From: R S <[email-masked]>
Date: Sat, 8 Jun 2013 13:52:25 -0700

A seventh point of view, which I happen to agree with: JSON strings are a
sequence of code units.

This is similar to the definition of 'source text' in ECMAScript:

"ECMAScript source text is assumed to be a sequence of 16-bit code units
for the purposes of this specification. Such a source text may include
sequences of 16-bit code units that are not valid UTF-16 character
encodings."

http://es5.github.io/x6.html

- Rob

On Sat, Jun 8, 2013 at 1:15 PM, Paul Hoffman <[email-masked]> wrote:

> 1) Some people have read the following statement from the RFC to mean
> "only Unicode characters are allowed in strings":
> 5) The definition of "Unicode character" has been surprising to some
> people on this list, and thus might be surprising to some developers and
> users of JSON.
>
> 6) Some people on the list consider some code points that are Unicode
> non-characters to be more objectionable than other code points.
>
> --Paul Hoffman
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 8 Jun 2013 14:09:34 -0700

On Jun 8, 2013, at 1:52 PM, R S <[email-masked]> wrote:

> A seventh point of view, which I happen to agree with: JSON strings are a sequence of code units.

That aligns with (2).

--Paul Hoffman

---

From: Stephen Dolan <[email-masked]>
Date: Sat, 8 Jun 2013 22:11:26 +0100

On Sat, Jun 8, 2013 at 9:52 PM, R S <[email-masked]> wrote:
> "ECMAScript source text is assumed to be a sequence of 16-bit code units for
> the purposes of this specification. Such a source text may include sequences
> of 16-bit code units that are not valid UTF-16 character encodings."

That's a very out-of-context quote. The linked document states:

"ECMAScript source text is represented as a sequence of characters in
the Unicode character encoding, version 3.0 or later."

It then gives your quote, and states "If an actual source text is
encoded in a form other than 16-bit code units it must be processed as
if it was first convert [sic] to UTF-16". It seems like UTF-16 is a
convenient way to frame the document, rather than a requirement of the
specification.

Stephen

---

From: R S <[email-masked]>
Date: Sat, 8 Jun 2013 17:23:57 -0700

On Sat, Jun 8, 2013 at 2:11 PM, Stephen Dolan <[email-masked]>wrote:

> for

It's a requirement. Here are some additional references:

<
http://wiki.ecmascript.org/doku.php?id=strawman:support_full_unicode_in_strings&rev=1305822947
<https://mail.mozilla.org/pipermail/es-discuss/2011-May/014337.html>

The paragraph following the one I cited:

'Throughout the rest of this document, the phrase “code unit” and the word
“character” will be used to refer to a 16-bit unsigned value used to
represent a single 16-bit unit of text. The phrase “Unicode character” will
be used to refer to the abstract linguistic or typographical unit
represented by a single Unicode scalar value (which may be longer than 16
bits and thus may be represented by more than one code unit). The phrase
“code point” refers to such a Unicode scalar value. “Unicode character”
only refers to entities represented by single Unicode scalar values: the
components of a combining character sequence are still individual “Unicode
characters,” even though a user might think of the whole sequence as a
single character.' <http://es5.github.io/x6.html>

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 9 Jun 2013 02:39:19 +0200

>> A seventh point of view, which I happen to agree with: JSON strings are a sequence of code units.

That is not a very precise or useful statement if it refers to Unicode "code units", because code units can be bytes ("UTF-8 code units"), 16-bit values (UTF-16), or 32-bit values (UTF-32).
In actual JSON interchange, the code units are bytes (unless you are using the hypothetical UTF-16 or UTF-32 form of JSON).
Rob might have been trying to say "UTF-16 code units".

This view is most natural to those who think JSON should be useful as an interchange format for the "use a JavaScript string as a vector of unconstrained 16-bit values" hack.
(It is not aligned with JSON's main purpose.)

> That aligns with (2).

Not at all: It is different, and it is at a different level.

>>      unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

RFC 4234 is about characters and this excerpt from RFC 4627 makes it clear that its usage of ABNF is based on Unicode characters/Unicode "code points", not about UTF-16 "code units" (which would stop at 0xFFFF).

Now the ABNF is about the representation, not about the data model, so this has no bearing on whether at the data model level the "characters" in a string are Unicode characters, Unicode code points, or UTF-16 code units.

(For the latter case, if you happen to have a sequence of a high-surrogate (0xD800..0xDBFF) and a low-surrogate (0xDC00..0xDFFF), that sequence will look like a single 0x10000 to 0x10FFFF in the Unicode character/"code point" view and can be interchanged natively.  Any other usage of the surrogate "code units" will need to use a \uXXXX escape, because they can't be represented in Unicode.)

Grüße, Carsten

---

From: R S <[email-masked]>
Date: Sat, 8 Jun 2013 18:24:52 -0700

On Sat, Jun 8, 2013 at 5:39 PM, Carsten Bormann <[email-masked]> wrote:

>
> This view is most natural to those who think JSON should be useful as an
> interchange format for the "use a JavaScript string as a vector of
> unconstrained 16-bit values" hack.
>

We should document what currently works. It's not about expressing a
preference.

> (It is not aligned with JSON's main purpose.)

I am not sure what the rationale for that statement is.

The RFC says:

"An implementation may set limits on the length and character contents of
strings."

So maybe there's once again nothing to do here.

- Rob

sayrer$ python
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.loads('{"End of data marker": "\uFFFF" }')
{u'End of data marker': u'\uffff'}
>>> quit()

sayrer$ node
(^C again to quit)

sayrer$ node --version
v0.8.14

sayrer$ node
> JSON.parse('{"End of data marker": "\uFFFF" }')
{ 'End of data marker': '' }
> process.exit()

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 9 Jun 2013 04:29:58 +0200

On Jun 9, 2013, at 03:24, R S <[email-masked]> wrote:

> We should document what currently works.

A survey of which implementations react to what input in which way would be nice, indeed, but is not the purpose of this spec update.

Changing the JSON spec retroactively to put in a requirement for handling strings in UTF-16 code units so that unpaired surrogates might work more uniformly is something different.

(BTW, your examples show that two JSON implementations handle Unicode non-characters nicely, which is great and probably something to be recommended, but doesn't have anything to do with switching to UTF-16 code units.  Now let's put in a couple of (paired!) surrogates to show how well the code units work:

>>> print(json.loads('{"a": "\ud800\udd51" }')["a"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 0: surrogates not allowed
>>> print(json.loads('{"a": "\\ud800\\udd51" }')["a"])
𐅑

... which demonstrates nicely what I have been saying: Don't put unescaped surrogates into your JSON texts, because there is no equivalence at the UTF-16 code unit level.)

There is only a single place where the UTF-16 legacy of JavaScript shines through in JSON today:

   To escape an extended character that is not in the Basic Multilingual
   Plane, the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair.  So, for example, a string
   containing only the G clef character (U+1D11E) may be represented as
   "\uD834\uDD1E".

And that is OK because it is just a slightly weird representation of the character.

> I am not sure what the rationale for that statement is.

The first sentence of RFC 4627:

Abstract

   JavaScript Object Notation (JSON) is a lightweight, text-based,
   language-independent data interchange format.

Grüße, Carsten

---

From: R S <[email-masked]>
Date: Sat, 8 Jun 2013 21:48:04 -0700

On Sat, Jun 8, 2013 at 7:29 PM, Carsten Bormann <[email-masked]> wrote:
>
>
> Changing the JSON spec retroactively to put in a requirement for handling
> strings in UTF-16 code units so that unpaired surrogates might work more
> uniformly is something different.
>

I haven't proposed a change to the spec--have you? I'm fine with the status
quo: vaguely referring to Unicode characters with the full knowledge that
JSON is intended to produce identical results to JavaScript's eval function
for the subset of JavaScript syntax that JSON supports.

>
> (BTW, your examples show that two JSON implementations handle Unicode
> non-characters nicely, which is great and probably something to be
> recommended, but doesn't have anything to do with switching to UTF-16 code
> units.  Now let's put in a couple of (paired!) surrogates to show how well
> the code units work:
>
> ... which demonstrates nicely what I have been saying: Don't put unescaped
> surrogates into your JSON texts, because there is no equivalence at the
> UTF-16 code unit level.)
>

That must be Python3. That shell session is a little misleading, because
it's the print function throwing the exception. The Python3 JSON parser
accepts both, and the Python3 JSON encoder produces identical output from
those inputs. Although they do in Python 2.7, the two inputs don't compare
as equal in Python 3. However, both JSON messages seem unambiguous to
Python's JSON encoder.

~$ python
Python 2.7.3 (default, Sep 26 2012, 21:53:58)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.loads('{"a": "\\ud800\\udd51"}')["a"] ==  json.loads('{"a":
"\ud800\udd51"}')["a"]
True

~$ python3
Python 3.2.3 (default, Sep 30 2012, 16:43:30)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.loads('{"a": "\\ud800\\udd51"}')["a"] ==  json.loads('{"a":
"\ud800\udd51"}')["a"]
False
>>> json.dumps(json.loads('{"a": "\ud800\udd51"}')["a"])
'"\\ud800\\udd51"'
>>> json.dumps(json.loads('{"a": "\\ud800\\udd51"}')["a"])
'"\\ud800\\udd51"'

Here's Node.js / V8 for comparison:

~$ node --version
v0.8.9
~$ node
> JSON.parse('{"a": "\ud800\udd51"}')["a"] ==  JSON.parse('{"a":
"\\ud800\\udd51"}')["a"]
true
> JSON.stringify(JSON.parse('{"a": "\ud800\udd51"}')["a"])
'"𐅑"'
> JSON.stringify(JSON.parse('{"a": "\\ud800\\udd51"}')["a"])
'"𐅑"'

> The first sentence of RFC 4627:

I don't find that convincing at all. It is obvious that strings in JSON are
meant to encode all JavaScript strings, as if they were passed to
JavaScript's eval function (there is one unrelated bug here). RFC 4627 even
refers to JSON as a JavaScript subset, and references the eval function in
the security considerations.

If we must "improve" the current text, I have a suggested addition which
borrows from your emails. I'm not sure where to add it, because it doesn't
fit well with the current structure of the document.

"At their most basic level, JSON strings represent a vector of
unconstrained 16-bit values which largely map to UCS-2. Implementations MAY
apply more stringent Unicode validation."

- Rob

---

From: Tim Bray <[email-masked]>
Date: Sun, 9 Jun 2013 00:08:20 -0700

It seems clear that the intent of JSON, judging by the language in 4627,
and the observed usage in a zillion RESTful protocols currently in
production, is that JSON strings be used to interchange Unicode character
sequences.

It seems clear that (at least partly as a side-effect of the JavaScript
“character” model) there is no normative requirement to avoid Unicode abuse
such as the use of non-character codepoints and naked surrogates, which
will predictably lead to consequences such as Carsten’s exploding-python
example.

So maybe just leave the spec more or less the way it is. Say in the
introduction that strings are for interchanging Unicode characters, observe
in the fine print that the specification does not forbid the use of things
that cannot be useful in the Unicode context and will quite likely cause
software breakage. And in the best-practices doc, say “Encode only Unicode
codepoints, and use only UTF-8 to do it.”

Also, I think that any significant change to the sentence in the
Introduction: “A string is a sequence of zero or more Unicode characters
[UNICODE].” would represent a dramatic change in the specification of JSON,
and thus be out of scope for this WG.

-T

On Sat, Jun 8, 2013 at 9:48 PM, R S <[email-masked]> wrote:

> I haven't proposed a change to the spec--have you? I'm fine with the
> status quo: vaguely referring to Unicode characters with the full knowledge
> that JSON is intended to produce identical results to JavaScript's eval
> function for the subset of JavaScript syntax that JSON supports.
>
>
>
>>
>> ... which demonstrates nicely what I have been saying: Don't put
>> unescaped surrogates into your JSON texts, because there is no equivalence
>> at the UTF-16 code unit level.)
>>
>
>
> I don't find that convincing at all. It is obvious that strings in JSON
> are meant to encode all JavaScript strings, as if they were passed to

---

From: Stephen Dolan <[email-masked]>
Date: Mon, 10 Jun 2013 18:02:30 +0100

On Sun, Jun 9, 2013 at 1:23 AM, R S <[email-masked]> wrote:
> <http://wiki.ecmascript.org/doku.php?id=strawman:support_full_unicode_in_strings&rev=1305822947>
> “code point” refers to such a Unicode scalar value. “Unicode character” only
> refers to entities represented by single Unicode scalar values: the

Sorry, I misread.

Stephen

---

From: Norbert Lindenberg <[email-masked]>
Date: Wed, 12 Jun 2013 09:51:22 -0700

On Jun 8, 2013, at 21:48 , R S wrote:

> If we must "improve" the current text, I have a suggested addition which borrows from your emails. I'm not sure where to add it, because it doesn't fit well with the current structure of the document.
> 
> "At their most basic level, JSON strings represent a vector of unconstrained 16-bit values which largely map to UCS-2. Implementations MAY apply more stringent Unicode validation."

JSON is in no way constrained to the Basic Multilingual Plane, so if we discuss 16-bit values in JSON, they're UTF-16 code units, not UCS-2.

On Jun 9, 2013, at 0:08 , Tim Bray wrote:

> It seems clear that the intent of JSON, judging by the language in 4627, and the observed usage in a zillion RESTful protocols currently in production, is that JSON strings be used to interchange Unicode character sequences.

I agree.

> It seems clear that (at least partly as a side-effect of the JavaScript “character” model) there is no normative requirement to avoid Unicode abuse such as the use of non-character codepoints and naked surrogates, which will predictably lead to consequences such as Carsten’s exploding-python example.
> 
> So maybe just leave the spec more or less the way it is. Say in the introduction that strings are for interchanging Unicode characters, observe in the fine print that the specification does not forbid the use of things that cannot be useful in the Unicode context and will quite likely cause software breakage.

It might be better to say "Unicode code points" rather than "Unicode characters":

- This makes the spec independent of Unicode versions - the set of Unicode code points is fixed (U+0000 to U+10FFFF), while the set of assigned characters in Unicode keeps growing, and different systems communicating via JSON may not be based on the same Unicode version.

- It makes clear that noncharacters, unassigned code points, and surrogate code points are all allowed in JSON (although subject to the limitations imposed by parsers, communication channels, security systems, or the character encoding used).

> And in the best-practices doc, say “Encode only Unicode codepoints, and use only UTF-8 to do it.”

UTF-8 is the right encoding to use over the wire or in files, but at runtime many systems (including all that implement the ECMAScript or DOM specifications) have to use UTF-16 semantics.

Norbert