Subject: [Json] surrogate code points in JSON text

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 11 Dec 2013 11:08:55 -0800

IETF RFC4627bis-09 states that strings are sequences of Unicode characters.
The ABNF for unescaped, however, permits any Unicode code point except for
double quote, reverse solidus, and ASCII control characters.  This may be
read to indicate that JSON text can contain surrogate code points (both
paired and unpaired).

Surrogate code points cannot be encoded in UTF-8, UTF-16, or UTF-32 as
Unicode encodings are only defined on Unicode scalar values.

I suggest that RFC4627bis be clarified to explicitly state that JSON texts
do not contain surrogate code points in strings.  This could be done by
modifying the unescaped production to exclude these code points.

Peter F. Patel-Schneider

---

From: Julian Reschke <[email-masked]>
Date: Wed, 11 Dec 2013 20:24:39 +0100

> modifying the unescaped production to exclude these code points.

You realize that the situation is different for unpaired surrogates that 
appear as \u sequences?

Best regards, Julian

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 11 Dec 2013 11:30:15 -0800

Yes, this comment is only about surrogate code points that appear directly in 
JSON texts.

peter

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 11 Dec 2013 20:36:37 +0100

On 11 Dec 2013, at 20:30, Peter F. Patel-Schneider <[email-masked]> wrote:

> Yes, this comment is only about surrogate code points that appear directly in JSON texts.

They can’t, because JSON is encoded in one of the Unicode CES, and those won’t provide those code points.
(See also section 8.2.)

One could argue that this point should be made more explicit than leave it as an exercise to the reader.

When this was last discussed, we were still trying not to change JSON from RFC4627.
Since that objective has been abandoned, we might as well fix the grammar:

OLD:
      unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
NEW:
      unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %E000-10FFFF

Is that what you mean?

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 11 Dec 2013 19:49:37 +0000

>      unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %E000-10FFFF

Yes, please.  Strong +1.  If you really want to include an unpaired
surrogate, you MUST escape it.  "\uDEAD" is valid, <0022 DEAD 0022> is
not, because it can't occur in a valid transmitted document.

-- 
Joe Hildebrand

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 11 Dec 2013 11:55:34 -0800

Yes, I agree that the best way forward is to make the change to unescaped that 
you suggest (and add some discussion of the change).

However, JSON can exist outside a Unicode encoding, so it is arguable that 
JSON texts can contain surrogate code points.  I say arguable because 
RFC4627bis talks about sequences of Unicode characters, but that would also 
exclude unassigned code points which I don't think is intended.  Of course 
JSON texts containing surrogate characters are not very useful in the IETF 
context as they can only be transferred using some agreement outside RFC4627.

peter

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 11 Dec 2013 20:06:28 +0000

On 12/11/13 11:55 AM, "Peter F. Patel-Schneider" <[email-masked]>
wrote:

>JSON texts can contain surrogate code points.

Not in 4627bis land.  But I agree that in the internal programming
language representation, there are times when this can happen.

I think we need to make it crystal clear that if you want to transmit one
of those internal representations and call it JSON, the only way it's
going to interop is if you first \u-encode the brokenness.

>I say arguable because

That's not currently correct.  They can be transmitted in escaped form.

-- 
Joe Hildebrand

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 11 Dec 2013 12:48:18 -0800

On 12/11/2013 12:06 PM, Joe Hildebrand (jhildebr) wrote:
> [...]
>> JSON texts containing surrogate characters [code points!] are not very useful in the
>> IETF
Where is the process of escaping a JSON text that contains a surrogate code 
point described?

A JSON text that represents the same sequence of (string) code points can 
sometimes be constructed by using the \u mechanism, but that's not an escaped 
form of the original JSON text.

peter

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 11 Dec 2013 22:00:43 +0100

On 11 Dec 2013, at 21:48, Peter F. Patel-Schneider <[email-masked]> wrote:

> Where is the process of escaping a JSON text that contains a surrogate code point described?

Can’t answer that question, as JSON texts don’t contain surrogate code points.

You mean, how to I encode a string with a surrogate code point?
Well, strings can’t contain surrogate code points either as they are sequences of characters.

So 8.2 may strike the right balance here by describing the phenomenon of unpaired surrogate escapes but not really explaining how they are created except accidentally:

"Instances of this have been observed, for example when a
   library truncates a UTF-16 string without checking whether the
   truncation split a surrogate pair."

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 11 Dec 2013 13:34:25 -0800

This was discussed at length earlier, and failed for lack of WG consensus. Different people strongly felt that their view of What We Are Talking About must be the only right one; the strength of those views clearly remain, given the threads from this past week.

The current wording is the best that we could get consensus on.

Seeing this flare up yet again here gives me a bit more compassion for TC39...

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 12 Dec 2013 00:15:53 +0100

On 11 Dec 2013, at 22:34, Paul Hoffman <[email-masked]> wrote:

> This

This thread has already talked about so many different things that I’m not able to find the referent here.

Grüße, Carsten

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 11 Dec 2013 16:02:02 -0800

When reading RFC4627bis-09, one gets the following information:

The sequence %x22.DEAD.22 matches JSON-text.
A JSON parser must accept this sequence.
There is no JSON encoding scheme that can encode this sequence.

 From the discussion on this thread, and the discussion back in June
and July, which I just waded through, I take it that none of these three can
be changed.

peter