Subject: [Json] Code units

From: Tim Bray <[email-masked]>
Date: Thu, 19 Sep 2013 17:12:36 -0700

On Thu, Sep 19, 2013 at 4:47 PM, John Cowan <[email-masked]> wrote:

> There's a bug.  The use of "code unit" in 8.3 is undefined unless it's
> specified whether 8-bit, 16-bit, or 32-bit code units are intended
> (see the Unicode glossary).  I think changing the first instance to
> "16-bit code units" will suffice.
>

Hm, section 2.5 of Unicode repeatedly uses the term “code unit” without a
bit-size prefix.  And an implementation might well parse the text into
wchar_t arrays, which are usually 32 bits per.  So I’m not convinced that
the code units are *necessarily* 16-bit.  And I think the text is perfectly
clear.

Also, the official definition isn’t terribly helpful:
http://www.unicode.org/glossary/#code_unit

>
> --
> John Cowan      http://www.ccil.org/~cowan      [email-masked]
> Be yourself.  Especially do not feign a working knowledge of RDF where
> no such knowledge exists.  Neither be cynical about RELAX NG; for in
> the face of all aridity and disenchantment in the world of markup,
> James Clark is as perennial as the grass.  --DeXiderata, Sean McGrath
>

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 19 Sep 2013 17:16:23 -0700

On Sep 19, 2013, at 5:12 PM, Tim Bray <[email-masked]> wrote:

> Hm, section 2.5 of Unicode repeatedly uses the term “code unit” without a bit-size prefix.  And an implementation might well parse the text into wchar_t arrays, which are usually 32 bits per.  So I’m not convinced that the code units are *necessarily* 16-bit.  And I think the text is perfectly clear.

The definition of "code unit" in the Unicode Standard, D77, specifically ties the size of the unit to the encoding form. Section 8.3 is not about a specific encoding form, and therefore I would say that it would be an error to specify any size.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Sep 2013 20:42:15 -0400

Paul Hoffman scripsit:

> > Hm, section 2.5 of Unicode repeatedly uses the term “code unit”
> > without a bit-size prefix.  And an implementation might well
> > parse the text into wchar_t arrays, which are usually 32 bits per.
> > So I’m not convinced that the code units are *necessarily* 16-bit.
> > And I think the text is perfectly clear.

Either 16-bit or 32-bit code units would be fine, but 8-bit code units
would not, because there is simply no representation of unpaired-surrogate
code points in terms of 8-bit code units, which would render the whole
section useless.  I wrote "16-bit" solely for concreteness.

> The definition of "code unit" in the Unicode Standard, D77, specifically
> ties the size of the unit to the encoding form. Section 8.3 is not
> about a specific encoding form, and therefore I would say that it
> would be an error to specify any size.

In that case, why not speak of code points, particularly as that term
is already used in 4627 and 4627bis?

-- 
Her he asked if O'Hare Doctor tidings sent from far     John Cowan
coast and she with grameful sigh him answered that      http://ccil.org/~cowan
O'Hare Doctor in heaven was. Sad was the man that word  [email-masked]
to hear that him so heavied in bowels ruthful.  All
she there told him, ruing death for friend so young,    James Joyce, Ulysses
algate sore unwilling God's rightwiseness to withsay.   "Oxen of the Sun"

---

From: Tim Bray <[email-masked]>
Date: Thu, 19 Sep 2013 17:48:56 -0700

Because \uDEAD isn’t a code point but it is a code unit.  You actually have
a point because if it weren’t for \uDEAD we could could write the
discussion in terms of code points, which would be much more humane, so
it’s only because JSON allows some particular icky non-code-point 16-bit
code units that we have to write about code units. But I don’t think that
the correctness of the language in the spec depends on the code units being
of any particular bit-ness.

---

From: John Cowan <[email-masked]>
Date: Thu, 19 Sep 2013 22:10:07 -0400

Tim Bray scripsit:

> Because \uDEAD isn’t a code point but it is a code unit.  

You have a hold of the right spade, but you are calling it a shovel.
The integer 0xDEAD = 57005:

0) does not represent a character;

1) is not a Unicode scalar value (USV) (integer in the range 0-0xD7FF
or 0xE000-0x10FFFF);

2) is a code point (integer in the range 0-0x10FFFF);

3) is a 32-bit code unit (integer in the range 0-0x10FFFF);

4) is a 16-bit code unit (integer in the range 0-0xFFFF);

5) is not an 8-bit code unit (integer in the range 0-0xFF) and cannot
be represented as a valid sequence of 8-bit code units either.

> [I]f it weren’t for \uDEAD we could could write the discussion in
> terms of code points, which would be much more humane,

If it weren't for \uDEAD we could write the discussion in terms of
Unicode scalar values or characters, which *would* be much more humane.
Since we do have to account for \uDEAD, we have to use code points,
16-bit code units, or 32-bit code units.

> code units that we have to write about code units. 

It's only because JSON allows some particular icky non-USV code points
that we have to write about code points or 16-bit code units or 32-bit
code units rather than USVs or characters.

-- 
They tried to pierce your heart                 John Cowan
with a Morgul-knife that remains in the         http://www.ccil.org/~cowan
wound.  If they had succeeded, you would
become a wraith under the domination of the Dark Lord.         --Gandalf

---

From: R S <[email-masked]>
Date: Thu, 19 Sep 2013 19:16:37 -0700

What if we left Unicode specifics out of the spec? The current RFC is
actually OK, otherwise JSON wouldn't work.

This thread is veering into Unicode conformance, which is not controlled by
JSON implementors, in general.

- Rob
 On Sep 19, 2013 7:10 PM, "John Cowan" <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 20 Sep 2013 12:49:34 +1000

> Because \uDEAD isn’t a code point but it is a code unit.

But section 8.3. "String Comparison" talks about implementations that "are interoperable". Comparison is not interoperable for "\uDEAD" so it does not have to be accommodated in this sentence.

                                       Implementations which transform the	
 	   textual representation into sequences of Unicode code units, and then	
 	   perform the comparison numerically, code unit by code unit, are	
 	   interoperable in the sense that implementations will agree in all	
 	   cases on equality or inequality of two strings.

I think the correct fix is to change "code unit" to "scalar value" in this sentence.

--
James Manger

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 20 Sep 2013 11:54:49 +0900

There has just be an extensive discussion on the Unicode list about 
terms such as "code unit", "code point" and "Unicode scalar value" 
including experts and references to the standard that confirms John's 
points below.

Regards,   Martin.

---

From: Tim Bray <[email-masked]>
Date: Thu, 19 Sep 2013 20:35:30 -0700

> I think the correct fix is to change "code unit" to "scalar value" in this

I am wide open to arguments as to what words we should use to refer to the
integers that we parse out of JSON strings and then use in comparison, but
they should be backed up by specific arguments from
scripture^H^H^H^H^H^H^H^H^H the Unicode standard.  I think “code unit” is
plausible on that basis but we’re prepared to be convinced that there’s a
better alternative.

---

From: Tim Bray <[email-masked]>
Date: Thu, 19 Sep 2013 20:54:07 -0700

Martin, you are closer to Unicode than the rest of us.  How would you write
that sentence in a way that best combines Unicode correctness and human
readability?

On Thu, Sep 19, 2013 at 7:54 PM, "Martin J. Dürst"
<[email-masked]>wrote:

> There has just be an extensive discussion on the Unicode list about terms
> such as "code unit", "code point" and "Unicode scalar value" including
> experts and references to the standard that confirms John's points below.
>
> Regards,   Martin.
>
>
>>  ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: John Cowan <[email-masked]>
Date: Fri, 20 Sep 2013 00:00:46 -0400

> actually OK, otherwise JSON wouldn't work.

If you don't believe the RFC needs any revision, what are you doing here?
(This is not intended to be insulting; it's a genuine question.)

-- 
John Cowan
        [email-masked]
                I am a member of a civilization. --David Brin

---

From: R S <[email-masked]>
Date: Thu, 19 Sep 2013 21:25:27 -0700

> (This is not intended to be insulting; it's a genuine question.)

The current RFC's vague Unicode text seems to work in practice.

Further, the WG could do harm. In fact, that seems likely.

For example, any JSON library that one might actually want to use handles
duplicate keys in objects and primitives at the root level.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 20 Sep 2013 07:53:55 +0200

On Sep 20, 2013, at 05:35, Tim Bray <[email-masked]> wrote:

> On Thu, Sep 19, 2013 at 7:49 PM, Manger, James H <[email-masked]> wrote:
> 
> 
> I am wide open to arguments as to what words we should use to refer to the integers that we parse out of JSON strings and then use in comparison, but they should be backed up by specific arguments from scripture^H^H^H^H^H^H^H^H^H the Unicode standard.  I think “code unit” is plausible on that basis but we’re prepared to be convinced that there’s a better alternative.

JSON is not concerned with code units (which are something that occurs in a transformation format aka coding scheme), so any occurrence of "code unit" in 4627bis is bound to be an error.

Grüße, Carsten

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 20 Sep 2013 17:29:28 +1000

Even if two implementations both compare strings code unit by code unit, they are not going to “agree in all cases”. One can throws a “fatal runtime exception” on "\uDEAD" (as allowed by the previous section), for instance. By suggesting a switch to the term “scalar value” I hoped to take those problematic cases out-of-scope for the sentence on interoperable string comparison.

Perhaps that isn’t the right approach though.

I assume the point of section 8.3. “String Comparison” is to tell implementations that they must undo escapes, but they don’t have to go to the effort of Unicode normalization when they compare object element names. After undoing escapes on parsing, it is sufficient to compare code unit by code unit, with whatever code unit size the implementation uses.

Not having to normalize is not the same as not being allowed to normalize, however.

Are we sure no JSON libraries (or no apps that use JSON libraries) ever do normalization?

I would be quite happy if a JSON parser rejected {"\u00c4":0, "\u0041\u0308":1} as having duplicates (Ä vs Ä), though obviously I cannot rely on this behaviour. I wonder if any non-malicious systems need JSON like that to work?

The new text in rfc4627bis on Unicode chars (8.2), numbers (6), and unique names in objects (4) is aimed at people using JSON. It doesn’t add new MUSTs; it adds sentences saying if you write JSON within this subset you will get interop. The string comparison text feels different. It is aimed at library implementers so I think the interop language needs a different form.

Suggestion:

8.3.  String Comparison

   Software implementations are typically required to test names of
   object members for equality.  Escapes MUST be undone before any such
   test. For example, the following four strings are equal:
   "/", "\/", "\u002F", "\u002f". No other normalization is required.
   After undoing escapes, an implementation can compare the resulting code units
   Numerically (whatever size of code unit it uses).

I doubt we want (or need) to say which subset of JSON you need to use to ensure your object names are not considered equal, interoperably.

--
James Manger

From: Tim Bray [mailto:[email-masked]]
Sent: Friday, 20 September 2013 1:36 PM
To: Manger, James H
Cc: John Cowan; Paul Hoffman; [email-masked]
Subject: Re: [Json] Code units

On Thu, Sep 19, 2013 at 7:49 PM, Manger, James H <[email-masked]<mailto:[email-masked]>> wrote:

--
James Manger

---

From: John Cowan <[email-masked]>
Date: Fri, 20 Sep 2013 09:40:13 -0400

> Are we sure no JSON libraries (or no apps that use JSON libraries)

No.  Nor are we sure that no JSON libraries or apps do case folding or
accent stripping or ISO 9 transliteration, nor that their authors are
not secret masters of international terrorism.  Absence of evidence may
not be evidence of absence, but sometimes it's all you're gonna get.

> The new text in rfc4627bis on Unicode chars (8.2), numbers (6),
> and unique names in objects (4) is aimed at people using JSON. It
> doesn’t add new MUSTs; it adds sentences saying if you write JSON
> within this subset you will get interop. 

More accurately, that if you exceed this subset you might not get interop.
Interop is *never* guaranteed.

-- 
I now introduce Professor Smullyan,             John Cowan
who will prove to you that either               [email-masked]
he doesn't exist or you don't exist,            http://www.ccil.org/~cowan
but you won't know which.                               --Melvin Fitting