Subject: [Json] Fwd: New Version Notification - draft-ietf-json-rfc4627bis-08.txt

From: Tim Bray <[email-masked]>
Date: Wed, 4 Dec 2013 18:10:12 -0800

---------- Forwarded message ----------
From: <[email-masked]>
Date: Wed, Dec 4, 2013 at 6:09 PM
Subject: New Version Notification - draft-ietf-json-rfc4627bis-08.txt
To: [email-masked], [email-masked],
[email-masked]

A new version (-08) has been submitted for draft-ietf-json-rfc4627bis:
http://www.ietf.org/internet-drafts/draft-ietf-json-rfc4627bis-08.txt

The IETF datatracker page for this Internet-Draft is:
https://datatracker.ietf.org/doc/draft-ietf-json-rfc4627bis/

Diff from previous version:
http://www.ietf.org/rfcdiff?url2=draft-ietf-json-rfc4627bis-08

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

IETF Secretariat.

---

From: Tim Bray <[email-masked]>
Date: Wed, 4 Dec 2013 18:15:31 -0800

Oh, and there’s real HTML at
http://www.tbray.org/tmp/draft-ietf-json-rfc4627bis-08.html

---

From: R S <[email-masked]>
Date: Wed, 4 Dec 2013 19:33:34 -0800

Some of the text in this version looks contradictory:

"This revision does not change any of the rules of the specification"

vs.

"Note that certain previous specifications of JSON constrained a JSON text
to be an object or an array."

Wouldn't one of those be RFC4627? It's right there in the diff.

- Rob

On Wed, Dec 4, 2013 at 6:10 PM, Tim Bray <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 09:20:10 +0100

Major new bug (apart from the one Rob found):

— A SHOULD-level requirement for BOM tolerance is a gratuitous change.  (I thought we were shooting for a MAY.)

— Even if the WG decides that change is warranted, the text on -08 will be read as “JSON now allows BOMs” by anyone who is not a language lawyer.  The “SHOULD accept” needs to be accompanied by a (redundant, but contextually necessary) “MUST NOT send”.

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Dec 2013 07:22:28 -0800

<no hat>

On Dec 4, 2013, at 7:33 PM, R S <[email-masked]> wrote:

> "Note that certain previous specifications of JSON constrained a JSON text to be an object or an array."
> 

Good catch, Rob. Given that we had consensus to make a significant change, I propose just removing that whole paragraph.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Dec 2013 07:27:30 -0800

<hat on>

> — A SHOULD-level requirement for BOM tolerance is a gratuitous change.  (I thought we were shooting for a MAY.)

There was a reasonable number of people who were very concerned that JSON documents that were being sent around with a BOM would be rejected.

Given the current text:
   Implementations have been observed to generate JSON texts prefixed
   with a Byte Order Mark character.  While this is not allowed by the
   grammar in this specification, in the interests of interoperability
   it is RECOMMENDED that implementations which parse JSON texts ignore
   the presence of a byte order mark rather than treating it as an
   error.
what change would you make?

> — Even if the WG decides that change is warranted, the text on -08 will be read as “JSON now allows BOMs” by anyone who is not a language lawyer.  The “SHOULD accept” needs to be accompanied by a (…

If it is redundant (and it is, given the carefully worded intro to that sentence), why do you feel it is "contextually necessary"?

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 17:00:59 +0100

> There was a reasonable number of people who were very concerned that JSON documents that were being sent around with a BOM would be rejected.

As they should be.

One way to deal with a common mistake is to legitimize it.

Doing this too often leads to soup.

(The alternative is to point out the mistake is a mistake.)

However, I’m not at all convinced that people adding BOMs to their JSON senders even is a common mistake.

I have seen way more HT characters (TABs) in strings, trailing commas etc., than BOMs in JSON documents.  Should we legitimize those, too?

it is RECOMMENDED that implementations which parse JSON texts ignore
->
implementations that parse JSON texts MAY WISH TO ignore

(OK, without the RFC 6919ism, as apt as that may be here, that becomes:)

implementations that parse JSON texts MAY ignore

(Maybe add that this is mostly relevant for parsers that are used to interpret JSON files entered by users, not so much for those used for data interchange.)

> If it is redundant (and it is, given the carefully worded intro to that sentence), why do you feel it is "contextually necessary”?

Because a number of decades of watching people interpret specifications teaches me that people will read this as “JSON now allows BOMs”, all careful wording aside.  So they will add BOMs to all JSON documents they send and then complain to the other JSON implementers that haven’t changed their implementations for BOM tolerance that “this is now valid JSON”, as you "SHOULD ignore the BOM".

Adding the (theoretically redundant) “MUST NOT send” makes it more likely that this obvious misunderstanding is nipped in the bud.

If the ensuing clarity creates too much cognitive dissonance, at least s/the grammar in//.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 08:08:31 -0800

Yup.

On Thu, Dec 5, 2013 at 7:22 AM, Paul Hoffman <[email-masked]> wrote:

> Good catch, Rob. Given that we had consensus to make a significant change,
> I propose just removing that whole paragraph.
>
> --Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 09:40:33 -0800

How about changing it from

 in the interests of interoperability it is RECOMMENDED that
implementations which parse JSON texts ignore the presence of a byte order
mark rather than treating it as an error.

to

 in the interests of interoperability it is RECOMMENDED that
implementations which generate JSON texts not prepend a byte order mark,
and implementations which parse JSON texts ignore the presence of a byte
order mark rather than treating it as an error.

On Thu, Dec 5, 2013 at 8:00 AM, Carsten Bormann <[email-masked]> wrote:

> > There was a reasonable number of people who were very concerned that
> JSON documents that were being sent around with a BOM would be rejected.
>
> As they should be.
>
> I have seen way more HT characters (TABs) in strings, trailing commas
> etc., than BOMs in JSON documents.  Should we legitimize those, too?
>
> > Given the current text:
> (Maybe add that this is mostly relevant for parsers that are used to
> interpret JSON files entered by users, not so much for those used for data
> interchange.)
>
> >> — Even if the WG decides that change is warranted, the text on -08 will
> be read as “JSON now allows BOMs” by anyone who is not a language lawyer.
>  The “SHOULD accept” needs to be accompanied by a (redundant, but
> contextually necessary) “MUST NOT send”.
> >
> > If it is redundant (and it is, given the carefully worded intro to that
> sentence), why do you feel it is "contextually necessary”?
>
> Because a number of decades of watching people interpret specifications
> teaches me that people will read this as “JSON now allows BOMs”, all
> careful wording aside.  So they will add BOMs to all JSON documents they
> send and then complain to the other JSON implementers that haven’t changed
> their implementations for BOM tolerance that “this is now valid JSON”, as
> you "SHOULD ignore the BOM".
>
> Adding the (theoretically redundant) “MUST NOT send” makes it more likely
> that this obvious misunderstanding is nipped in the bud.
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 5 Dec 2013 18:45:44 +0100

On 05 Dec 2013, at 18:40, Tim Bray <[email-masked]> wrote:

>  in the interests of interoperability it is RECOMMENDED that implementations which parse JSON texts ignore the presence of a byte order mark rather than treating it as an error.
> 
> to 
> 
>  in the interests of interoperability it is RECOMMENDED that implementations which generate JSON texts not prepend a byte order mark,

Ouch, that changes the MUST NOT into a SHOULD.  No go.

> and implementations which parse JSON texts ignore the presence of a byte order mark rather than treating it as an error.

RECOMMENDED = SHOULD.  I think I have argued that turning BOM tolerance into a SHOULD is not a smart move.  (I have also said that I can live with that mistake.)

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Dec 2013 10:35:17 -0800

<no hat>

On Dec 5, 2013, at 9:40 AM, Tim Bray <[email-masked]> wrote:

>  in the interests of interoperability it is RECOMMENDED that implementations which generate JSON texts not prepend a byte order mark, and implementations which parse JSON texts ignore the presence of a byte order mark rather than treating it as an error.

I agree with Carsten: that first part is wrong. For interoperability with systems that had never heard of BOMs because we never mentioned them until now, you MUST NOT add one.

Proposed:

... in the interests of interoperability, implementations which parse JSON texts MAY ignore the presence of a byte order mark rather than treating it as an error. Implementations MUST NOT purposely add a byte order mark to the beginning of a JSON text.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 10:37:46 -0800

Works for me.

On Thu, Dec 5, 2013 at 10:35 AM, Paul Hoffman <[email-masked]> wrote:

> I agree with Carsten: that first part is wrong. For interoperability with
> systems that had never heard of BOMs because we never mentioned them until
> now, you MUST NOT add one.
>
> Proposed:
>
> ... in the interests of interoperability, implementations which parse JSON
> texts MAY ignore the presence of a byte order mark rather than treating it
> as an error. Implementations MUST NOT purposely add a byte order mark to
> the beginning of a JSON text.
>
> --Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 5 Dec 2013 18:39:16 +0000

On 12/5/13 11:35 AM, "Paul Hoffman" <[email-masked]> wrote:

>Proposed:
>
>... in the interests of interoperability, implementations which parse
>JSON texts MAY ignore the presence of a byte order mark rather than
>treating it as an error. Implementations MUST NOT purposely add a byte
>order mark to the beginning of a JSON text.

Just remove "purposely", or change MUST NOT to "REALLY SHOULD NOT" from
RFC 6919.

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 10:40:28 -0800

Yeah, “purposely” isn’t actually a word. Purposefully is, but just lose it.

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 10:47:25 -0800

OK, I’m wrong, it is so a word, but let’s drop it anyhow.

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 11:02:29 -0800

Rob is right. I propose we replace that text with:

<t>This document removes inconsistencies with other specifications of JSON,
repairs specification errors, and offers experience-based interoperability
guidance.</t>

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Dec 2013 11:08:18 -0800

<hat on>

On Dec 5, 2013, at 11:02 AM, Tim Bray <[email-masked]> wrote:

> <t>This document removes inconsistencies with other specifications of JSON, repairs specification errors, and offers experience-based interoperability guidance.</t>

That seems fine.

--Paul Hoffman

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 05 Dec 2013 20:28:22 +0100

>what change would you make?

I wanted to make an example to demonstrate how the text above is non-
sensical and checked the latest draft to make sure I get it right, only
to find that the encoding detection requirements have been removed. I
believe my concerns in this area are not receiving due consideration.

I am happy to take suggestions about available means of appeal.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Jacob Davies <[email-masked]>
Date: Thu, 5 Dec 2013 11:35:17 -0800

Is your concern that the language now says "ignore the BOM" but has no
other method of determining the encoding of the text? Because yeah.

It seems to me that if you claim to continue to support UTF-16 &
UTF-32 (which I think is pointless, but anyway) you need some way of
indicating that. In the absence of the hacky first-two-bytes table
from the original spec (which doesn't work if you allow strings at the
top level) then don't you need to support BOMs, rather than ignore
them?

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Dec 2013 11:39:19 -0800

There’s a lot of history in this list on how we got to where we are.  Since
we dropped the top-level restriction to objects/arrays, the assertion that
the first two characters must be ASCII no longer holds, so the table would
have to be reconstructed to handle things like "ڃ". And that’s not
cost-effective, since nobody ever sends anything but UTF-8 over the wire,
so the whole thing is a red herring.  The spec is more useful the way it is.

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Dec 2013 13:46:01 -0600

> top level) then don't you need to support BOMs, rather than ignore

Starting with at least one ASCII character (which JSON texts will
continue to do) is sufficient to determine which of UTF-8/16/32 is used
and byte order.  The table is unnecessary: programmers can figure it
out.

BOMs are therefore unnecessary.  I get that some implementations send
it.  I *feel* that the best thing to do would be to be silent as to
BOMs, but if we must say something we should say that parsers MUST
ignore BOMs and encoders SHOULD NOT include BOMs.

Nico
--

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 06 Dec 2013 01:27:40 +0100

>top level) then don't you need to support BOMs, rather than ignore

Say you have two application/json processors and pass them the same byte
sequence and one says "Oh, it's an array with no members" and the other
says "No, it's an object with no members". Some argue that there should
be no mandatory-to-implement mechanism to tell arrays and objects in
application/json entities apart. Without such a mechanism, it would not
be possible to consult the application/json specification and determine
whether the byte sequence represents `{}` or `[]`, and in fact both pro-
cessors may be fully conforming. I think that's bad for interoperability
and may open up attack vectors in systems when they process maliciously
crafted input. As an example,

  data:application/json,+AFsAXQ-

If an implementation treats that as `[]` then it fully conforms with all
requirements in draft-ietf-json-rfc4627bis-08 and so does the entity, it
is "encoded in Unicode", specifically, in UTF-7. Another example:

  data:application/json,+/v8-...

This is a UTF-7 encoded document that starts with a U+FEFF character. Is
that a "Byte Order Mark character" that should be ignored? It might seem
silly to assume UTF-7 but web browsers and mail clients used to do silly
things like that. Another funny Unicode encoding is CESU-8. Is it okay
to generate CESU-8 encoded application/json entities? To parse them as
if they were CESU-8 encoded? That might seem crazy, but

  https://code.google.com/p/v8/issues/detail?id=2875

such is the world of character encodings. With a mandatory-to-implement
encoding detection mechanism others might not repeat Google's mistakes.

As for Unicode signatures, I am not aware anything has changed since

  http://www.ietf.org/mail-archive/web/json/current/msg02044.html

I also do not understand what is being proposed. As an example, I can't
tell whether under the rules in draft-ietf-json-rfc4627bis-08

  data:application/json,%FE%FF...

implementations "SHOULD" use the initial byte sequence to determine the
encoding, and if so, how that is consistent with the idea to remove the
mandatory-to-implement encoding detection mechanism. As I pointed out in

  http://www.ietf.org/mail-archive/web/json/current/msg01983.html

implementations do not actually do this. Why require something that is
not needed and not useful and not actually done in practise?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/