Subject: [Json] problem with using code units to define comparison of Unicode code points

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Mon, 16 Dec 2013 16:46:03 -0800

I do not think that the following wording in rfc4627bis-09 is correct:

8.3. String Comparison

Software implementations are typically required to test names of object members for equality. Implementations which transform the textual representation into sequences of Unicode code units, and then perform the comparison numerically, code unit by code unit, are interoperable in the sense that implementations will agree in all cases on equality or inequality of two strings. For example, implementations which compare strings with escaped characters unconverted may incorrectly find that "a\\b" and "a\u005Cb" are not equal.

A naive implementation of this process will result in different answers for UTF-16 code units from UTF-8 and UTF-32 code units because UTF-16 must encode extended characters as two code units so the string containing U+1D11E will compare equal to the string containing U+D834,U+DD1E but in UTF-8 and UTF-32 the naive implementation would result in inequality.

This is separate from the issue that no UTF encoding scheme can encode surrogate code points.

peter

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 Dec 2013 07:25:58 +0100

I believe this is simply a bug.  s/code unit/code point/g and the text makes sense.

Grüße, Carsten

On 17 Dec 2013, at 01:46, Peter F. Patel-Schneider <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Mon, 16 Dec 2013 22:37:16 -0800

That would certainly be better, but then the problem that the unescaping 
process is not fully specified would still arise.

peter

PS:  This problem is that the encoding process produces the same escaped 
sequence for multiple sequences of code points.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 Dec 2013 08:02:02 +0100

On 17 Dec 2013, at 07:37, Peter F. Patel-Schneider <[email-masked]> wrote:

> That would certainly be better, but then the problem that the unescaping process is not fully specified would still arise.

While it is not fully specified, combining the definition of a string, the definition of escaping, and the (implicit but obvious) requirement for round-tripping leaves you with exactly one way to do the unescaping.
It may not be nice to leave this for the implementer to figure out, but it doesn’t seem to me the spec has an actual hole.

> PS:  This problem is that the encoding process produces the same escaped sequence for multiple sequences of code points.

That is just a result of the fact that UTF-16 cannot represent all sequences of code points, and JavaScript is stuck on UTF-16, which bleeds through to JSON through the escape convention.
So I don’t know how this could be fixed in a JSON spec.  (It is not much of a problem in practice, where we encode sequences of characters [scalar values], which do decode the same.)

Grüße, Carsten