Subject: [Json] Proposed change: update the Unicode version

From: Paul Hoffman <[email-masked]>
Date: Tue, 4 Jun 2013 09:58:21 -0700

<no hat>

The current spec refers to the then-current Unicode spec:

   [UNICODE] The Unicode Consortium, "The Unicode Standard Version 4.0",
             2003, <http://www.unicode.org/versions/Unicode4.1.0/>.

I propose to update the reference to:

   [UNICODE] The Unicode Consortium, "The Unicode Standard Version 6.2",
             2012, <http://www.unicode.org/versions/Unicode6.2.0/>.

--Paul Hoffman

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 04 Jun 2013 11:59:31 -0600

>              2012, <http://www.unicode.org/versions/Unicode6.2.0/>.

There are other i18n issues that we might want to clarify, but updating
the Unicode reference seems uncontroversial.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 4 Jun 2013 18:56:50 +0000

>the Unicode reference seems uncontroversial.

Suggested change to section 1:

WAS:

A string is a sequence of zero or more Unicode characters [UNICODE].

SUGGESTED:

A string is a sequence of encoded Unicode codepoints defined in
[UNICODE6.2] or later versions of the Unicode specification.

Goals:

- we're defining a protocol, not an infoset
- make it clear that codepoints that become valid in the future are valid
JSON

Section 3 also needs to be tweaked:

WAS:

JSON text SHALL be encoded in Unicode.  The default encoding is
   UTF-8.

SUGGESTED:

JSON text SHALL be transmitted as encoded Unicode codepoints.  The default
encoding is
UTF-8.

Note: the rest of section 3 may need to change if we allow strings as root
JSON objects later.

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Tue, 4 Jun 2013 13:31:55 -0700

> [UNICODE6.2] or later versions of the Unicode specification.

Joe, I get what you’re trying to do, and I suspect this is technically
correct, but the language is kind of klunky and I don’t think it helps
comprehension.  I think that when you say “A string is a series of Unicode
characters" and have a reference to chapter 2, especially section 2.7, of
Unicode, it’s really perfectly clear what you mean.  I think we can
probably avoid mentioning “code points”, which is a good thing.

So how about “A string is a sequence of zero or more Unicode characters
defined in version 6.2 (or any subsequent version) of [UNICODE].”

> JSON text SHALL be transmitted as encoded Unicode codepoints.  The default

Whether it’s transmitted or read out of a file is irrelevant.

JSON text takes the form of a Unicode string [UNICODE, section 2.7]; the
default encoding is UTF-8.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 4 Jun 2013 22:45:26 +0000

On 6/4/13 2:31 PM, "Tim Bray" <[email-masked]> wrote:

>So how about ³A string is a sequence of zero or more Unicode characters
>defined in version 6.2 (or any subsequent version) of [UNICODE].²

Much better.

>Whether it¹s transmitted or read out of a file is irrelevant.

My point was that "Unicode" is not an encoding.

>default encoding is UTF-8.

That's fine.

-- 
Joe Hildebrand

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 04 Jun 2013 16:47:49 -0600

> My point was that "Unicode" is not an encoding.

Indeed. RFC 6365 is your friend. :-)

>> default encoding is UTF-8.

Are people using UTF-16 or UTF-32? Is there a good reason not to say
"MUST encode using UTF-8"?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Tim Bray <[email-masked]>
Date: Tue, 4 Jun 2013 15:51:30 -0700

> "MUST encode using UTF-8"?

I would *love* to, but we’re <sob> not allowed to change anything.  This is
another candidate for that best-practices doc. -T

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 04 Jun 2013 17:08:29 -0600

On 6/4/13 4:51 PM, Tim Bray wrote:
> On Tue, Jun 4, 2013 at 3:47 PM, Peter Saint-Andre <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> I would *love* to, but we’re <sob> not allowed to change anything.  This
> is another candidate for that best-practices doc. -T

You are right.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Wed, 5 Jun 2013 01:50:14 +0000

To gather this together, the proposed changes look to be:

1) In 1. Introduction --

#### OLD TEXT:

#### NEW TEXT:

A string is a sequence of zero or more Unicode characters
defined in version 6.2 (or any subsequent version) of [UNICODE]

####

2) In 3. Encoding --

#### OLD TEXT:

#### NEW TEXT:

####

Correct?

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 5 Jun 2013 16:15:47 +0000

+1

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 09:26:45 -0700

Looks good to me. -T

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Jun 2013 18:51:24 +0200

> default encoding is UTF-8.

It would help me if you could briefly explain what the reference to 2.7 specifically adds here.
(To me that is a bit confusing, as 2.7 is about internal programming language representation, not about representation for interchange.  But maybe I just don't understand.)

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 12:55:34 -0400

Carsten Bormann scripsit:

> It would help me if you could briefly explain what the reference to
> 2.7 specifically adds here.  (To me that is a bit confusing, as 2.7
> is about internal programming language representation, not about
> representation for interchange.  But maybe I just don't understand.)

+1

I also do not understand why JSON strings can't contain characters that
are unassigned as yet.  Checking this restriction is onerous if you don't
have 6.2 or later Unicode tables, or even if you do.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have not seen as far as others, it is because giants were standing
on my shoulders.
        --Hal Abelson

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 09:56:50 -0700

On Wed, Jun 5, 2013 at 9:51 AM, Carsten Bormann <[email-masked]> wrote:

> (To me that is a bit confusing, as 2.7 is about internal programming
> language representation, not about representation for interchange.  But
> maybe I just don't understand.)
>

Hm? The first paragraph says “A Unicode string data type is simply an
ordered sequence of code units. Thus a Unicode 8-bit string is an ordered
sequence of 8-bit code units, a Unicode 16-bit string is an ordered
sequence of 16-bit code units, and a Unicode 32-bit string is an  ordered
sequence of 32-bit code units.”

>
> Grüße, Carsten
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 13:00:29 -0400

> sequence of 32-bit code units.”

Yes, but is that what you actually want?  If so, you probably want the
"sequence of 16-bit code units" language.  This matches JavaScript and
at least some implementations of JSON.

However, I want a JSON string to be an ordered sequence of codepoints/
characters, not of code units.

-- 
Only do what only you can do.               John Cowan <[email-masked]>
  --Edsger W. Dijkstra's advice
    to a student in search of a thesis

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Wed, 5 Jun 2013 17:55:23 +0000

> characters, not of code units.

/me doffs hat

When it's actually inside the JavaScript interpreter, a "sequence of 16-bit code units" is most likely correct.  Over the wire or on the filesystem, that isn't necessarily true; often, it is a "sequence of 8-bit code units".

I think here we need to concentrate on what's over the wire/on the filesystem, not what's in the interpreter/processor/etc.

- m&m

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Jun 2013 20:43:26 +0200

On Jun 5, 2013, at 18:56, Tim Bray <[email-masked]> wrote:

> Hm? The first paragraph says “A Unicode string data type is simply an ordered sequence of code units. Thus a Unicode 8-bit string is an ordered sequence of 8-bit code units, a Unicode 16-bit string is an ordered sequence of 16-bit code units, and a Unicode 32-bit string is an  ordered sequence of 32-bit code units.”

If you were trying to answer my question I must admit that I'm not any further along in understanding why the reference is to 2.7.

4627 Section 3 is about encoding JSON on the wire (correct me if that impression is wrong).  (Actually, it doesn't say that much, the meat is then section 6, which with the title change raises the question whether application/json and the JSON format are the same thing or not.  But I digress.)  Unicode section 2.7 is most emphatically NOT about Unicode on the wire, it is about data types in programming languages.  Much of it is about the problems of processing incomplete UTF-16 strings (unpaired surrogates) in programming languages and their libraries.

The section in the Unicode standard about encoding is 2.6.  This defines seven encoding schemes, some of which mainly differ in their treatment of BOMs.  BOMs are not allowed by the JSON grammar, but it is not entirely obvious that that applies to the sequence of characters obtained after decoding the Unicode encoding scheme.

Now Douglas says:

> I think the section on encoding is not saying anything useful and should be completely removed.

Works for me (and makes this discussion moot).

Grüße, Carsten