Subject: Re: [Json] Last Call: <draft-ietf-json-rfc4627bis-07.txt> (The JSON Data Interchange Format) to Proposed Standard

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 10:57:14 +0100

Hi there,

I see only some of my LC comments have been addressed in -08; please 
have another look at those I list below.

Hi there,

below are my editorial comments:

Abstract

    JavaScript Object Notation (JSON) is a lightweight, text-based,
    language-independent data interchange format.  It was derived from
    the ECMAScript Programming Language Standard.  JSON defines a small
    set of formatting rules for the portable representation of structured
    data.

    This document makes no changes to the definition of JSON; it repairs
    specification errors and offers experience-based interoperability
    guidance.

I believe historical considerations do not belong into the abstract (but 
into the Introduction)

Status of This Memo

    This Internet-Draft is submitted in full conformance with the
    provisions of BCP 78 and BCP 79.

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF).  Note that other groups may also distribute
    working documents as Internet-Drafts.  The list of current Internet-
    Drafts is at http://datatracker.ietf.org/drafts/current/.

    Internet-Drafts are draft documents valid for a maximum of six months
    and may be updated, replaced, or obsoleted by other documents at any
    time.  It is inappropriate to use Internet-Drafts as reference
    material or to cite them other than as "work in progress."

    This Internet-Draft will expire on May 10, 2014.

Are we sure that we do not need the "pre-5378 escape clause" here? 
(Section 4 of <http://trustee.ietf.org/docs/IETF-Copyright-FAQ.pdf>)

    The grammatical rules in this document are to be interpreted as
    described in [RFC5234].

Maybe note which productions are imported as well (HEXDIG and DIGIT it 
seems).

    This revision does not change any of the rules of the specification;
    all texts which were legal JSON remain so, and none which were not
    JSON become JSON.  The revision's goal is to fix the errata and
    highlight practices which can lead to interoperability problems.

s/fix errata/apply errata/ ?

    Insignificant whitespace is allowed before or after any of the six
    structural characters.

    ws = *(
            %x20 /              ; Space
            %x09 /              ; Horizontal tab
            %x0A /              ; Line feed or New line
            %x0D )              ; Carriage return

We *could* use SP, HTAB, LF, and CR here.

    JSON text SHALL be encoded in Unicode...

That's a bit misleading. How do I "encode in Unicode"? I think what it 
tries to say is that one of the Unicode-compatible character encoding 
schemes needs to be used.

    An implementation may set limits on the size of texts that it
    accepts.  An implementation may set limits on the maximum depth of
    nesting.  An implementation may set limits on the range and precision
    of numbers.  An implementation may set limits on the length and
    character contents of strings.

Maybe this should be a bullet list?

10.  Generators

    A JSON generator produces JSON text.  The resulting text MUST
    strictly conform to the JSON grammar.

"strictly"?

    o  Changed Working Group attribution to JSON Working Group.

...this is a change that will not be visible in the RFC.

Best regards, Julian

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Dec 2013 07:34:44 -0800

<no hat>

On Dec 5, 2013, at 1:57 AM, Julian Reschke <[email-masked]> wrote:

> I believe historical considerations do not belong into the abstract (but into the Introduction)

This seems like bikeshedding, and the text has been where it is for a long time.

> Are we sure that we do not need the "pre-5378 escape clause" here? (Section 4 of <http://trustee.ietf.org/docs/IETF-Copyright-FAQ.pdf>)

Good catch, yes.

> Maybe note which productions are imported as well (HEXDIG and DIGIT it seems).

Sure.

> s/fix errata/apply errata/ ?

Sure.

> We *could* use SP, HTAB, LF, and CR here.

...but we don't need to.

> That's a bit misleading. How do I "encode in Unicode"? I think what it tries to say is that one of the Unicode-compatible character encoding schemes needs to be used.

This was a swamp where consensus seemed impossible to reach.

> Maybe this should be a bullet list?

Bikeshed

>   strictly conform to the JSON grammar.

You would prefer "squishily"?

> ...this is a change that will not be visible in the RFC.

Good point; that can be removed.

--Paul Hoffman

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 16:53:31 +0100

On 2013-12-05 16:34, Paul Hoffman wrote:
> ...

Nope. We discussed lots of other things but not this one.

This is about the sentence "JSON text SHALL be encoded in Unicode" which 
IMHO is meaningless. There is no "Unicode encoding" on the wire.

Alternatively, can you explain what it actually means?

> ...

I'd prefer not to say "strictly". Either you conform or you don't.

> ...

Best regards, Julian

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 08:05:01 -0800

> This seems like bikeshedding, and the text has been where it is for a long

Let me introduce an argument whiich I’ll label LNS.

LNS: This is not obviously incorrect, the text is reasonably idiomatic and
clear, and we’ve had excellent interoperability. Let’s not screw around
with it.

> (Section 4 of <http://trustee.ietf.org/docs/IETF-Copyright-FAQ.pdf>)

Uh, I looked at that doc and I’m not sure I get it. What do you want to
change?

> > Maybe note which productions are imported as well (HEXDIG and DIGIT it

LNS

> > s/fix errata/apply errata/ ?

An erratum is a description of an error.  We want to fix the error.
Also, LNS.

> This was a swamp where consensus seemed impossible to reach.

I thought about this one quite a bit.  Yes, this is flawed, but the actual
state of affairs is that per 4627, JSON can be encoded in any encoding of
Unicode and what’s being encoded aren’t code points but code units, and I
could write a 3-and-a-half sentence introduction which would make people
say WTF, but in fact this sentence fragment makes it reasonably obvious
what’s going on. Let’s not change it.

> > Maybe this should be a bullet list?

LNS.

> You would prefer "squishily"?

LNS.

> OK. Oh, and drat, I totally forgot to update this section per -08.
>
> --Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 17:11:54 +0100

On 05 Dec 2013, at 16:53, Julian Reschke <[email-masked]> wrote:

> This is about the sentence "JSON text SHALL be encoded in Unicode" which IMHO is meaningless. There is no "Unicode encoding" on the wire.
> 

The current text may not the most unambiguous way to say this, but I read it as “encoded in one of the ways defined by the Unicode standard”.  (Which defines seven of these ways, and not all are actually supported by JSON.  Ouch.)  I agree with Tim’s “LNS” here.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 17:13:59 +0100

On 2013-12-05 17:05, Tim Bray wrote:
> On Thu, Dec 5, 2013 at 7:34 AM, Paul Hoffman <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     <no hat>
>
> LNS: This is not obviously incorrect, the text is reasonably idiomatic
> and clear, and we’ve had excellent interoperability. Let’s not screw
> around with it.

;-)

>      > Are we sure that we do not need the "pre-5378 escape clause"
>     here? (Section 4 of
>     <http://trustee.ietf.org/docs/IETF-Copyright-FAQ.pdf>)
>
>     Good catch, yes.
>
>

See 
<http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#attribute-ipr-pre5378Trust200902>.

> An erratum is a description of an error.  We want to fix the error.

Um, no. This is new text so the claim you have excellent interop doesn't 
work ;-). We are fixing the errors, not the descriptions of the errors.

>      > That's a bit misleading. How do I "encode in Unicode"? I think
>     what it tries to say is that one of the Unicode-compatible character
>     encoding schemes needs to be used.
>
> I thought about this one quite a bit.  Yes, this is flawed, but the
> actual state of affairs is that per 4627, JSON can be encoded in any
> encoding of Unicode and what’s being encoded aren’t code points but code
> units, and I could write a 3-and-a-half sentence introduction which
> would make people say WTF, but in fact this sentence fragment makes it
> reasonably obvious what’s going on. Let’s not change it.

I find it totally non-obvious to the point that I really have no idea 
what it means.

What we are concerned with is the wire format. If there is a requirement 
here (and it seems so because it says "SHALL"), can we rephrase it in 
terms of character encoding schemes?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 17:16:16 +0100

> The current text may not the most unambiguous way to say this, but I read it as “encoded in one of the ways defined by the Unicode standard”.  (Which defines seven of these ways, and not all are ac…

I still don't get it.

Unicode is about code points.

*This* document is about octets on the wire.

Just because the text "always has been link that" doesn't make it correct.

As far as I can tell we have a BCP14 keyword in a meaningless statement, 
and that is a problem.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 17:21:41 +0100

> *This* document is about octets on the wire.

The Unicode standard is also very much about about bytes on the medium, or have section 2.5/2.6 recently been excised?

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 17:39:14 +0100

> The Unicode standard is also very much about about bytes on the medium, or have section 2.5/2.6 recently been excised?

You are right.

OK then, if this is about the 7 encoding schemes mentioned in chapter 
2.6, can we then clarify what it exactly means?

2.6 mentions 7 character encoding schemes.

draft 08 says:

>    JSON text SHALL be encoded in Unicode.  The default encoding is
>    UTF-8, and JSON texts which are encoded in UTF-8 are interoperable in
>    the sense that they will be read successfully by the maximum number
>    of implementations; there are many implementations which cannot
>    successfully read texts in other encodings (such as UTF-16 and
>    UTF-32).
>
>    Implementations have been observed to generate JSON texts prefixed
>    with a Byte Order Mark character.  While this is not allowed by the
>    grammar in this specification, in the interests of interoperability
>    it is RECOMMENDED that implementations which parse JSON texts ignore
>    the presence of a byte order mark rather than treating it as an
>    error.

Can we create a table that says for each of the encoding schemes, both 
with and without BOM, what senders and receivers are supposed to support?

Best regards, Julian

PS: I may look stubborn here, but if the text manages to confuse me 
about encodings, it's likely to confuse most of the audience.

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 08:57:57 -0800

> with and without BOM, what senders and receivers are supposed to support?

Please, please, no.

PS: I may look stubborn here, but if the text manages to confuse me about
> encodings, it's likely to confuse most of the audience.
>

Weirdly, you’re in a minority; more or less everyone surveyed the
landscape, decided to just use UTF-8, and interoperability is great.  I
think the draft reflects that reality very closely.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 18:08:27 +0100

On 2013-12-05 17:57, Tim Bray wrote:
> On Thu, Dec 5, 2013 at 8:39 AM, Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Can we create a table that says for each of the encoding schemes,
>     both with and without BOM, what senders and receivers are supposed
>     to support?
>
>
> Please, please, no.
>

I totally agree what almost (*) everyone use UTF-8, and that that is a 
good thing.

What I'm complaining about is that the spec is unclear. I hear you 
saying "people use UTF-8 despite the spec being unclear, and that's why 
we don't need to fix it". However, the same argument could be used for 
almost every other change we made. Why is this different?

Can we at least

s/JSON text SHALL be encoded in Unicode./JSON text SHALL be encoded in 
one of the character encoding schemes defined by Unicode./

?

Best regards, Julian

(*) for some value of "almost"

---

From: John Cowan <[email-masked]>
Date: Thu, 5 Dec 2013 12:12:20 -0500

Julian Reschke scripsit:

> s/JSON text SHALL be encoded in Unicode./JSON text SHALL be encoded
> in one of the character encoding schemes defined by Unicode./

+1 to that wording.

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
"Make a case, man; you're full of naked assertions, just like Nietzsche."
"Oh, i suffer from that, too.  But you know, naked assertions or GTFO."
                        --heard on #scheme, sorta

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 18:18:13 +0100

>> in one of the character encoding schemes defined by Unicode./

That sounds like we are disowning the rest of Unicode.

If we really must change this to mention the CES, how about

> JSON text SHALL be encoded in Unicode, which defines seven character encoding schemes.

Which leads nicely into the existing text:

> The default encoding is
>   UTF-8, and JSON texts which are encoded in UTF-8 are interoperable […]

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 09:33:34 -0800

On Thu, Dec 5, 2013 at 9:08 AM, Julian Reschke <[email-masked]>wrote:

>
> s/JSON text SHALL be encoded in Unicode./JSON text SHALL be encoded in one
> of the character encoding schemes defined by Unicode./
>

I’m not even sure that’s true, because JSON has this weird thing where
they’re doing code units not code points and and then there’s this other
layer of escaping with \u and astral planes via \u-encoded surrogates and
blecch blecch blecch.

I haven’t seen anything shakes my LNS sentiment.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 18:45:06 +0100

On 2013-12-05 18:33, Tim Bray wrote:
> On Thu, Dec 5, 2013 at 9:08 AM, Julian Reschke <[email-masked]
> layer of escaping with \u and astral planes via \u-encoded surrogates
> and blecch blecch blecch.
>

Tim,

can *you* explain what you think the text means (if it's not what 
Carsten said)?

If we collectively don't known what it means then it really really needs 
to be fixed.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Dec 2013 11:52:52 -0600

OLD:

   UTF-8.

NEW:

   JSON texts are composed of Unicode characters.  The default
encoding is UTF-8.  Note that other Unicode encodings may not
interoperate as well as UTF-8.

But I agree with Tim that while the old text is broken but its meaning
understood by all anyways, therefore it's best left alone,
particularly if fixing it leads to a large sub-thread.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 09:59:19 -0800

Anyhow, JSON texts are *not* necessarily composed of Unicode characters:
"\uDEAD".

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Dec 2013 19:10:45 +0100

> Anyhow, JSON texts are *not* necessarily composed of Unicode characters:

Yuck ;-) Are you saying that this is why it's a SHALL, not a MUST (or 
just a statement of fact, which I would prefer...)?

Best regards, Julian

---

From: John Cowan <[email-masked]>
Date: Thu, 5 Dec 2013 13:12:14 -0500

> Anyhow, JSON texts are *not* necessarily composed of Unicode characters:

That text is composed of eight Unicode characters.  What goes on inside
strings has nothing to do with the encoding of JSON documents.  What is
forbidden is to represent them in EBCDIC or Windows-1252 or whatever.

-- 
The Unicode Standard does not encode            John Cowan
idiosyncratic, personal, novel, or private      http://www.ccil.org/~cowan
use characters, nor does it encode logos
or graphics.                                    [email-masked]

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 5 Dec 2013 18:13:18 +0000

>Anyhow, JSON texts are *not* necessarily composed of Unicode characters:

I thought we decided that this is legal:

"\ud800"

but this is not (in hex):

22 ed a0 80 22

(that's the U+d800 improperly encoded as UTF-8 between quotes)

In which case, the syntax layer is always in a valid Unicode encoding.
It's at the semantic layer that all of the weirdness happens.

So, to tweak Nico's proposed text:

OLD:

NEW:

JSON is transmitted in a valid Unicode encoding.  The default
encoding is UTF-8.  Note that other Unicode encodings may not
interoperate as well as UTF-8.

We could also add some sort of statement about the semantics of strings
not being quite Unicode, but I think that's too much detail for the
encoding section.

-- 
Joe Hildebrand