Subject: Re: [Json] Proposal: the minimal edit

From: John Cowan <[email-masked]>
Date: Mon, 24 Jun 2013 23:20:14 -0400

Paul Hoffman scripsit:

> Add Section 1.2, "Changes from RFC 4627"
> 
>    This section lists all changes between this document and the text in
>    RFC 4627.
> 
>    - Applied errata #607 from RFC 4627 to correctly align the artwork
>      for the definition of "object".
> 
>    - Applied errata #3607 from RFC 4627 by removing the security
>      consideration that begins "A JSON text can be safely passed"
>      and the JavaScript code that went with that consideration.
> 
>    - Added Section 1.3, "Differences from the JSON Definition in ECMAScript". 
> 
>    - Changed the [ECMA] and [UNICODE] references to be non-version-specific.

Given these changes, and *only* these changes, then I do not believe that

>    - ECMAScript implementations can generate and consume code points
>      in JSON strings that are not Unicode characters.

belongs in Section 1.3, for it is equally true of ECMA-JSON and RFC 4627
JSON.  Both of them allow code points that are not Unicode characters,
e.g. unassigned codepoints.  So this point is true but not relevant.

-- 
I Hope, Sir, that we are not                    John Cowan
mutually Un-friended by this                    [email-masked]
Difference which hath happened                  http://www.ccil.org/~cowan
betwixt us.     --Thomas Fuller, Appeal of Injured Innocence (1659)

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 24 Jun 2013 20:31:48 -0700

> e.g. unassigned codepoints.  So this point is true but not relevant.

Hrm. Either you are focused only on the grammar section and not the whole document, or you missed the Introduction to RFC 4627, which say unequivocally:

   A string is a sequence of zero or more Unicode characters [UNICODE].

That statement disagrees with ABNF in the grammar, so the RFC is ambiguous. Some people in the WG believe that this sentence is right and the ABNF is wrong; others believe the other.

Does that help make the sentence about the differences more believable?

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Mon, 24 Jun 2013 23:48:27 -0400

Paul Hoffman scripsit:

> Hrm. Either you are focused only on the grammar section and not the
> whole document, or you missed the Introduction to RFC 4627, which
> say unequivocally:
> 

It's actually quite equivocal: it neither states that any Unicode character
may appear in a string, nor that everything that appears in a string is
a Unicode character.  But let that go for a moment.

> That statement disagrees with ABNF in the grammar, so the RFC is
> ambiguous. Some people in the WG believe that this sentence is right
> and the ABNF is wrong; others believe the other.

Since your proposal leaves the ambiguity intact, we don't know whether
the meaning of the RFC as revised by your proposal is or is not the same
as what ECMA-JSON prescribes.

> Does that help make the sentence about the differences more believable?

I don't disbelieve it; I don't think it belongs in a list of differences
until we have decided whether or not there is any difference.

-- 
"Well, I'm back."  --Sam        John Cowan <[email-masked]>

---

From: R S <[email-masked]>
Date: Mon, 24 Jun 2013 20:53:28 -0700

On Mon, Jun 24, 2013 at 8:48 PM, John Cowan <[email-masked]> wrote:

> a Unicode character.

I think that is a pretty strained interpretation.

> as what ECMA-JSON prescribes.

The RFC is ambiguous and ECMA-JSON is not. That's the difference, and we
don't have to fix the RFC.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jun 2013 07:25:55 +0200

On Jun 25, 2013, at 05:31, Paul Hoffman <[email-masked]> wrote:

> That statement disagrees with ABNF in the grammar, so the RFC is ambiguous.

That statement *combines* with the ABNF.  
The ABNF limits the representations, the first limits the strings themselves.

There is no ambiguity caused by having multiple restrictions in a specification.

Similarly, {"a": 1, "a": 2} isn't ambiguously valid JSON or not JSON by being allowed by the ABNF.
(Here, the other statement is a SHOULD, but that doesn't detract from my point.)

Of course, the statement about strings being sequences of Unicode characters is in the introduction, so the level of normative detail it imposes can be discussed, in particular in the light of the fact that the specification isn't quite clear on Unicode terms in other places.  But the intent is very clear.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Mon, 24 Jun 2013 22:45:48 -0700

On Mon, Jun 24, 2013 at 10:25 PM, Carsten Bormann <[email-masked]> wrote:

> ambiguous.

I just can’t bring myself to see it that way.  Any reasonable person would
read this statement in the introduction and conclude that a string is, you
know, a sequence of Unicode characters.  The fact that the ABNF later on
contradicts this does not “supplement” the introduction, it’s just a
spec-drafting goof.  This is reflected in the empirical reality that in
JSON's sweet spot, chiefly in the kind of network protocols the IETF cares
about, JSON strings are de facto used to interchange sequences of Unicode
characters, and reasonable developers would regard a failure to do so, even
though blessed by the spec, as a bug.

 -T

> Of course, the statement about strings being sequences of Unicode
> characters is in the introduction, so the level of normative detail it
> imposes can be discussed, in particular in the light of the fact that the
> specification isn't quite clear on Unicode terms in other places.  But the
> intent is very clear.
>
> Grüße, Carsten
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jun 2013 08:07:12 +0200

On Jun 25, 2013, at 07:45, Tim Bray <[email-masked]> wrote:

> I just can’t bring myself to see it that way.  Any reasonable person would read this statement in the introduction and conclude that a string is, you know, a sequence of Unicode characters.  The fact that the ABNF later on contradicts this does not “supplement” the introduction, it’s just a spec-drafting goof.  

I really have no idea where the "contradict" meme comes from.
(Well, actually, I do, see below.)

I'm much more interested in getting interoperability right than in
language lawyering, but where we are arguing from the existing specs,
we do have to read them right first.

Not trying to express every constraint in the ABNF is reasonable spec economy.  
Why try to express clumsily in the ABNF what can already be said referencing a well-defined concept?

Try writing the ABNF for surrogate pair escapes and you'll see what I mean.
(Sure, it *can* be done.  Would you want to?)
The specific objective of the ABNF here was to explain what must be escaped for *JSON* reasons.
Re-encoding the Unicode spec in the ABNF apparently wasn't the objective.

I'm writing this mainly because I have had to make the same kind of decision before:
Encode all possible restrictions in the ABNF or keep it readable?

Again, not everything allowed by the ABNF is a valid instance of the format.
I'm not aware of any usage of ABNF where that would be the case.
You always have to read the rest of the specification, too.

> This is reflected in the empirical reality that in JSON's sweet spot, chiefly in the kind of network protocols the IETF cares about, JSON strings are de facto used to interchange sequences of Unicode characters, and reasonable developers would regard a failure to do so, even though blessed by the spec, as a bug.

That is an important reality.
(It is hardly derived from a careful reading of RFC 4627, however.)

Our problem in this WG really is that this is only part of the reality.

There is another community that believes JSON is a format proprietary to JavaScript, so whatever is common (even if bad) practice in the JavaScript world needs to be within the JSON envelope.  Of course, if you believe that, you will grasp for straws like this "contradiction".
That clash of communities is why we even have the discussion here.  

Grüße, Carsten

---

From: R S <[email-masked]>
Date: Mon, 24 Jun 2013 23:36:41 -0700

On Mon, Jun 24, 2013 at 11:07 PM, Carsten Bormann <[email-masked]> wrote:

>
> There is another community that believes JSON is a format proprietary to
> JavaScript, so whatever is common (even if bad) practice in the JavaScript
> world needs to be within the JSON envelope.  Of course, if you believe
> that, you will grasp for straws like this "contradiction".

First, I'll note that you seem to be saying something quite different from
John and Norbert (I think).

Second, I think the disagreement you've highlighted is real, although think
it's phrased very helpfully. The crux of the matter is that the JavaScript
implementations will not change, and they are material to the success of
JSON and this Working Group, so they need to be documented.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jun 2013 12:19:20 +0200

On Jun 25, 2013, at 08:36, R S <[email-masked]> wrote:

>  The crux of the matter is that the JavaScript implementations will not change, and they are material to the success of JSON and this Working Group, so they need to be documented.

I completely agree with the need for documenting the JSON usage that has developed in that community.
We seem to have painted ourselves into a corner a bit by distinguishing the "standard" from a "best practices document", which is likely to cause endless debate which community usage gets documented where.

(The context of my message, however, was the interpretation that there is an ambiguity in RFC 4627 with respect to its Unicode usage, and I was trying to imagine how that interpretation might have been coming to life.)

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Tue, 25 Jun 2013 08:48:23 -0400

> The ABNF limits the representations, the first limits the strings themselves.

That would fly if the RFC didn't REQUIRE parsers to "accept all texts
that conform to the JSON grammar".  That means that

	["X"]

where X is an unpaired surrogate code point, is acceptable.

-- 
What has four pairs of pants, lives             John Cowan
in Philadelphia, and it never rains             http://www.ccil.org/~cowan
but it pours?                                   [email-masked]
        --Rufus T. Firefly

---

From: John Cowan <[email-masked]>
Date: Tue, 25 Jun 2013 08:41:55 -0400

Tim Bray scripsit:

> spec-drafting goof.  

Except that we know, from postings to this list and other lists, that
the author's intent was to be able to exchange arbitrary code points.
The goof, if any, was in the introduction, and consisted in the failure
to define "Unicode character" in the ECMAScript rather than the Unicode
sense.  If read in connection with the ECMAScript 3 definition of
"string value", all is clear:

# 4.3.16 String Value
# 
# A string value is a member of the type String and is a finite ordered
# sequence of zero or more 16-bit unsigned integer values.
# 
# NOTE
# 
# Although each value usually represents a single 16-bit unit of UTF-16
# text, the language does not place any restrictions or requirements on
# the values except that they be 16-bit unsigned integers.

Now it's true that authorial intent cannot dictate the meaning of a spec,
and it's also true that allowing the interchange of unescaped surrogate
code points is bad, because there is no reliable way to encode them.
Those are among the reasons why we are here, I hope.

-- 
John Cowan                                <[email-masked]>
Yakka foob mog.  Grug pubbawup zink wattoom gazork.  Chumble spuzz.
    --Calvin, giving Newton's First Law "in his own words"

---

From: John Cowan <[email-masked]>
Date: Tue, 25 Jun 2013 08:51:11 -0400

> (Sure, it *can* be done.  Would you want to?)

Sure, why not?  You just change

    unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

to

     unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

Trivial.

-- 
Values of beeta will give rise to dom!          John Cowan
(5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan
to rename '.' or '..' entries; see              [email-masked]
http://cm.bell-labs.com/cm/cs/who/dmr/odd.html)

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jun 2013 14:57:17 +0200

>     unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF

You also have to fix the "\u" 4HEXDIG part (if you don't, it seems that ill-formed Unicode is allowed in escape sequences and not in native coding).
This gets ugly quickly.
Even uglier if you want to exclude the non-character code points as well.

Remember that this was written at a time when JSON was still vying for its space.
The appearance of complexity needed to be avoided.
So why encode things in the ABNF that already are handled by Unicode?

(Again, this is all about spec exegesis, not about reality.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Jun 2013 14:57:24 +0200

> where X is an unpaired surrogate code point, is acceptable.

The grammar is in terms of characters.
You can't ship an unpaired surrogate code point in Unicode (neither in UTF-8 nor in one of the UTF-16s and UTF-32s), so the grammar never gets to see them.

(Again, this is about spec language lawyering, not about reality.)

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Tue, 25 Jun 2013 11:18:41 -0400

Carsten Bormann scripsit:

> You also have to fix the "\u" 4HEXDIG part (if you don't, it seems
> that ill-formed Unicode is allowed in escape sequences and not in
> native coding).

I don't consider that a problem.

-- 
John Cowan  [email-masked]  http://www.ccil.org/~cowan
Thor Heyerdahl recounts his attempt to prove Rudyard Kipling's theory
that the mongoose first came to India on a raft from Polynesia.
        --blurb for Rikki-Kon-Tiki-Tavi