Subject: [Json] Proposal: Code points and surrogates

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 18 Jun 2013 00:43:53 -0700

This proposal attempts to clarify that JSON allows all Unicode code points, but also which issues exist with surrogate code points. It proposes no normative changes so as to remain compatible with the ECMAScript specification, but recommends that clients carefully consider whether to allow surrogate code points in their uses of JSON.

Details:
- changed from "character" to "code point", unless specific characters are referenced,
- added paragraph with information about surrogate code points,
- rearranged paragraphs so that escape sequences for BMP and non-BMP code points are discussed side by side.

Compared to Tim's proposal [1], this proposal
- says what the specification actually allows (code points) rather than what may be intended (characters),
- gets rid of the term "16-bit quantities",
- provides more concrete information about surrogate code points than just "breakage",
- doesn't discuss the need, or lack thereof, to update the RFC in the future.

Section 1, Introduction:

Before:
    A string is a sequence of zero or more Unicode characters [UNICODE].

After:
    A string is a sequence of zero or more Unicode code points [UNICODE].

Section 2.5, Strings:

A string begins and ends with quotation marks. All Unicode code points may be placed within the quotation marks except for the characters that must be escaped: quotation mark, reverse solidus, and the control characters (U+0000 through U+001F).

Any code point may be escaped. If the code point is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence: a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the code point. The hexadecimal letters A though F can be upper or lowercase. So, for example, a string containing only a single reverse solidus character may be represented as "\u005C".

To escape a code point that is not in the Basic Multilingual Plane, the code point is represented as a twelve-character sequence, encoding the UTF-16 surrogate pair. So, for example, a string containing only the G clef character (U+1D11E) may be represented as "\uD834\uDD1E".

Alternatively, there are two-character sequence escape representations of some popular characters. So, for example, a string containing only a single reverse solidus character may be represented more compactly as "\\".

Note that this specification allows the inclusion of surrogate code points (U+D800 through U+DFFF) in JSON text, both directly and through escape sequences. However, Unicode code unit sequences containing surrogate code points are not well-formed, are prohibited by standards such as [RFC 3629], and may be rejected or modified by software such as character encoding converters. Developers and specification authors should carefully consider whether to allow surrogate code points in their uses of JSON.

[continue with grammar]

Section 4, Parsers:

Before:
   An implementation may set limits on the length and character contents of strings.

After:
   An implementation may set limits on the length and code point contents of strings.

[1] http://www.ietf.org/mail-archive/web/json/current/msg00814.html

Norbert

---

From: Stephen Dolan <[email-masked]>
Date: Sat, 22 Jun 2013 15:58:09 +0100

>     A string is a sequence of zero or more Unicode code points [UNICODE].

Unicode's three character encoding forms (UTF-8, UTF-16, and UTF-32)
define means of encoding sequences of unicode scalar values. A unicode
scalar value is a code point which is not a surrogate. Noncharacter
codepoints like U+FFFF are unicode scalar values.

Since JSON is invariably represented in some unicode encoding, it
seems odd that JSON would include things which are not representable
in any standard unicode encoding.

Should we instead say "a string is a sequence of zero or more Unicode
scalar values"? Even if we use "code point", it still seems odd to
claim "this specification allows the inclusion of surrogate code
points (U+D800 through U+DFFF) in JSON text, both directly and through
escape sequences" - it's not clear that the inclusion of surrogate
code points directly is actually possible.

Stephen

---

From: Norbert Lindenberg <[email-masked]>
Date: Sat, 22 Jun 2013 09:29:21 -0700

> code points directly is actually possible.

They can be included in Unicode strings, as defined in the Unicode Standard, section 2.7 [1]. The strings used in JavaScript, Java, and some other languages are Unicode strings, with no requirement of well-formedness.

The specification of the JSON object in the ECMAScript Language Specification [2] requires that unpaired surrogates in strings are passed through by both parse() and stringify(), and that parse() unescapes escape sequences for all surrogate values, paired or not. Testing I've done indicates that the implementations in the major browsers and in Node.js conform to this requirement.

[1] http://www.unicode.org/versions/Unicode6.2.0/ch02.pdf
[2] http://ecma-international.org/ecma-262/5.1/#sec-15.12

Norbert

---

From: John Cowan <[email-masked]>
Date: Sat, 22 Jun 2013 13:23:15 -0400

Norbert Lindenberg scripsit:

> The specification of the JSON object in the ECMAScript Language
> Specification [2] requires that unpaired surrogates in strings are
> passed through by both parse() and stringify(), and that parse()
> unescapes escape sequences for all surrogate values, paired or
> not. Testing I've done indicates that the implementations in the major
> browsers and in Node.js conform to this requirement.

Indeed.  But it does not say AFAIK how to map ECMAScript strings onto
byte-oriented external representations, which a standard that specifies
the meaning of "application/json" must necessarily do.  Since there
is no standard way to represent unpaired surrogates in any character
encoding, we have the choice of creating a new one de facto or de jure,
or banning them from the external representation (though they can still
appear escaped).

-- 
Long-short-short, long-short-short / Dactyls in dimeter,     John Cowan
Verse form with choriambs / (Masculine rhyme):           [email-masked]
One sentence (two stanzas) / Hexasyllabically
Challenges poets who / Don't have the time.     --robison who's at texas dot net

---

From: Nico Williams <[email-masked]>
Date: Sat, 22 Jun 2013 15:32:18 -0500

I don't think it's any more reasonable to require that parsers pass through
any old non-Unicode string contents than it would be to require a specific
range of real numbers to be passed by all parsers.  Or at least it would be
surprising to find JSON to be so.  Also, I can agree that parsers should
pass through any odd escaped non-character data, but not that they should
unescape such data.

---

From: John Cowan <[email-masked]>
Date: Sat, 22 Jun 2013 16:42:00 -0400

Nico Williams scripsit:

> I don't think it's any more reasonable to require that parsers pass
> through any old non-Unicode string contents than it would be to require
> a specific range of real numbers to be passed by all parsers.

Right, that's another thing for the ECMAScript Stupid Tricks section:
ECMAscript JSON limits the range of numbers to the finite values of
64-bit IEEE floats, whereas RFC JSON has no fixed limitation, though it
licenses parsers to specify one.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Objective consideration of contemporary phenomena compel the conclusion
that optimum or inadequate performance in the trend of competitive
activities exhibits no tendency to be commensurate with innate capacity,
but that a considerable element of the unpredictable must invariably be
taken into account. --Ecclesiastes 9:11, Orwell/Brown version

---

From: R S <[email-masked]>
Date: Sat, 22 Jun 2013 13:50:34 -0700

> licenses parsers to specify one.

I think ECMAScript JSON.parse implementations conform to RFC 4627 in this
case, so a note is not strictly necessary. I wouldn't oppose including it,
though. Similar interoperability issues could occur between Python and
Java, for example.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 23 Jun 2013 00:10:28 +0200

On Jun 18, 2013, at 09:43, Norbert Lindenberg <[email-masked]> wrote:

> JSON allows

are you trying to say

(a) a conforming parser is required to accept

or

(b) a conforming parser is not required to reject

Grüße, Carsten

---

From: Norbert Lindenberg <[email-masked]>
Date: Sat, 22 Jun 2013 22:42:48 -0700

> (b) a conforming parser is not required to reject

I mean "the JSON grammar allows" (the proposed text would go into a subsection of the Grammar section).

According to the last sentence of section 4, parsers are not required to accept any characters or code points at all.

Norbert

---

From: John Cowan <[email-masked]>
Date: Sun, 23 Jun 2013 01:47:33 -0400

Norbert Lindenberg scripsit:

> According to the last sentence of section 4, parsers are not required
> to accept any characters or code points at all.

According to my reading of Section 4 as a whole, they are required to accept
all JSON texts at first, but then are free to reject them afterwards.

-- 
We do, doodley do, doodley do, doodley do,        John Cowan <[email-masked]>
What we must, muddily must, muddily must, muddily must; 
Muddily do, muddily do, muddily do, muddily do,    http://www.ccil.org/~cowan
Until we bust, bodily bust, bodily bust, bodily bust.  --Bokonon