Subject: [Json] [Technical Errata Reported] RFC8259 (6208)

From: RFC Errata System <[email-masked]>
Date: Wed, 10 Jun 2020 06:32:58 -0700 (PDT)

The following errata report has been submitted for RFC8259,
"The JavaScript Object Notation (JSON) Data Interchange Format".

--------------------------------------
You may review the report below and at:
https://www.rfc-editor.org/errata/eid6208

--------------------------------------
Type: Technical
Reported by: David Golden <[email-masked]>

Section: 8.1

Original Text
-------------
In the interests of interoperability, implementations that parse JSON texts MAY ignore the presence of a byte order mark rather than treating it as an error.

Corrected Text
--------------
In the interests of interoperability, implementations that parse JSON texts MAY ignore the presence of a byte order mark or MAY interpret a byte order mark to indicate an alternate encoding rather than treating it as an error.

Notes
-----
The original line is copied from previous RFCs that specifically allowed alternate encodings.  In the context of a new, UTF-8 only restriction, interoperability provisions should also address interpreting legacy formats that predate the restriction.  By omission, readers may conclude that the *only* option for a BOM is to ignore or error.

Instructions:
-------------
This erratum is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party  
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC8259 (draft-ietf-jsonbis-rfc7159bis-04)
--------------------------------------
Title               : The JavaScript Object Notation (JSON) Data Interchange Format
Publication Date    : December 2017
Author(s)           : T. Bray, Ed.
Category            : INTERNET STANDARD
Source              : Javascript Object Notation Update
Area                : Applications and Real-Time
Stream              : IETF
Verifying Party     : IESG

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 10 Jun 2020 15:43:57 +0200

The WG had pretty strong consensus that BOMs don’t belong on JSON.
This “MAY” prepares implementations for the presence of other implementations that do not heed that consensus.  No functionality is assigned by this MAY.

Since Errata are not intended to unravel WG decisions: Reject.

Apart from that, I seriously don’t understand what the "indicate an alternate encoding” would be — the JSON text already is UTF-8?

Grüße, Carsten

> On 2020-06-10, at 15:32, RFC Errata System <[email-masked]> wrote:
> 
> ——————
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: Barry Leiba <[email-masked]>
Date: Wed, 10 Jun 2020 10:37:25 -0400

Yes, I agree: this is asking to revisit what we had consensus on, not
reporting an error in the RFC.  Off to reject it now.

Barry

---

From: David Golden <[email-masked]>
Date: Wed, 10 Jun 2020 10:37:46 -0400

I agree that BOMs don't belong on JSON, but three prior RFC allowed them.
(The first one even specified how to do so.)

Given a JSON text generated under a previous specification with a UTF-16/32
BOM, ignoring the BOM is incorrect behavior.  As currently written --
copied verbatim from the prior two RFCs when BOMs were legal -- the only
interoperability suggestion is to ignore a BOM.

My proposal does not encourage the use of BOMs or unravel the WG consensus
(which I fully support).  It merely prepares implementations for the
presence of other implementations that predate the current, more
restrictive specification.

Respectfully yours,
David

On Wed, Jun 10, 2020 at 9:43 AM Carsten Bormann <[email-masked]> wrote:

> This “MAY” prepares implementations for the presence of other
> implementations that do not heed that consensus.  No functionality is
> assigned by this MAY.
>
> Apart from that, I seriously don’t understand what the "indicate an
> alternate encoding” would be — the JSON text already is UTF-8?
>
> Grüße, Carsten
>
>
> > In the interests of interoperability, implementations that parse JSON
> texts MAY ignore the presence of a byte order mark rather than treating it
> as an error.
> >
> > Corrected Text
> > --------------
> > In the interests of interoperability, implementations that parse JSON
> texts MAY ignore the presence of a byte order mark or MAY interpret a byte
> order mark to indicate an alternate encoding rather than treating it as an
> error.
> >
> > Notes
> > -----
> > The original line is copied from previous RFCs that specifically allowed
> alternate encodings.  In the context of a new, UTF-8 only restriction,
> interoperability provisions should also address interpreting legacy formats
> that predate the restriction.  By omission, readers may conclude that the
> *only* option for a BOM is to ignore or error.
> >
> > Instructions:
> > -------------

-- 
David Golden <[email-masked]> • https://xdg.me/ • Twitter/GitHub: @xdg

---

From: Barry Leiba <[email-masked]>
Date: Wed, 10 Jun 2020 10:42:11 -0400

David, I understand that it might be worth having further discussion
about the issue.  The problem *here* is that there is not an error in
the RFC: the RFC accurately reflects the consensus of the working
group and the text that was reviewed and agreed on at the time.  So
it's not in scope for an errata report.  Discussion about whether or
not to revisit this and to publish an RFC that says something
different is... a separate discussion.

Barry, ART AD

On Wed, Jun 10, 2020 at 10:38 AM David Golden <[email-masked]> wrote:
>
> I agree that BOMs don't belong on JSON, but three prior RFC allowed them.  (The first one even specified how to do so.)
>
> Given a JSON text generated under a previous specification with a UTF-16/32 BOM, ignoring the BOM is incorrect behavior.  As currently written -- copied verbatim from the prior two RFCs when BOMs were legal -- the only interoperability suggestion is to ignore a BOM.
>
> My proposal does not encourage the use of BOMs or unravel the WG consensus (which I fully support).  It merely prepares implementations for the presence of other implementations that predate the current, more restrictive specification.
>
> Respectfully yours,
> David
>

---

From: David Golden <[email-masked]>
Date: Wed, 10 Jun 2020 10:49:47 -0400

I understand.  Would it be more appropriate then to limit the "may ignore a
BOM" to "may ignore a UTF-8 BOM" so that the interop suggestion does not
inadvertently encourage incorrect decoding?  That would be consistent with
the UTF-8 only restriction and leave legacy encoding/BOM considerations out
of scope.

I realize this may seem picky, but this stems from discussions at work
about whether/how to support legacy BOMs and people have interpreted the
RFC differently.  I was hoping to create clarity consistent with the intent
of the working group, whatever that turns out to be.

David

On Wed, Jun 10, 2020 at 10:42 AM Barry Leiba <[email-masked]>
wrote:

> > I agree that BOMs don't belong on JSON, but three prior RFC allowed
> them.  (The first one even specified how to do so.)
> >
> > Given a JSON text generated under a previous specification with a
> UTF-16/32 BOM, ignoring the BOM is incorrect behavior.  As currently
> written -- copied verbatim from the prior two RFCs when BOMs were legal --
> the only interoperability suggestion is to ignore a BOM.
> >
> > My proposal does not encourage the use of BOMs or unravel the WG
> consensus (which I fully support).  It merely prepares implementations for
> the presence of other implementations that predate the current, more
> >> > The original line is copied from previous RFCs that specifically
> allowed alternate encodings.  In the context of a new, UTF-8 only
> restriction, interoperability provisions should also address interpreting
> legacy formats that predate the restriction.  By omission, readers may
> conclude that the *only* option for a BOM is to ignore or error.
> >> >
> >> > Instructions:
> >> > -------------

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 10 Jun 2020 17:00:03 +0200

Hi David,

In the spirit of Barry’s point that this is no longer about the Errata Report, I’m changing the subject.

The question at hand really is whether we can ever fix mistakes in a protocol specification.

The 2006 RFC (4627) didn’t take benefit of the observation that JSON is used with UTF-8 in the wild.  With 7159 (2014) and 8259 (2017) we progressively did.

The question really is how much of a backward compatibility onus we impose on implementers.
The reality is that JSON works well in the Internet Standard (RFC 8259, 2017) form, and there is no point for implementers to bow to what wasn’t really the reality in 2006 anyway.

So adding any verbiage that could be misused to pressure implementers to implement the deprecated stuff would be counter-productive.  Yes, it might still be a good idea for a JSON implementers to accept BOMs (and "JSON comments", and trailing commas, and all the other cruft that we do to "increase interoperability" and that then hurts interoperability because other people come to rely on that lenience (*)), but the JSON Standard doesn’t have to (even mildly) encourage that.

There are no standard JSON supersets (at least not ones that can be called “JSON” (**)).

Grüße, Carsten

(*) a.k.a. “soup”
(**) (CBOR diagnostic notation clearly is a JSON superset)

> On 2020-06-10, at 16:37, David Golden <[email-masked]> wrote:
>

---

From: Tim Bray <[email-masked]>
Date: Wed, 10 Jun 2020 08:00:51 -0700

The erratum would have been more convincing had it described any plausible
course of action other than rejecting or silently accepting the BOM.  I
can't think of anything useful that software could do. That doesn't mean
there isn't anything, but a suggestion would be helpful.

On Wed, Jun 10, 2020 at 7:50 AM David Golden <[email-masked]> wrote:

> I understand.  Would it be more appropriate then to limit the "may ignore
> a BOM" to "may ignore a UTF-8 BOM" so that the interop suggestion does not
>> >> > In the interests of interoperability, implementations that parse
>> JSON texts MAY ignore the presence of a byte order mark rather than
>> treating it as an error.
>> >> >
>> >> > Corrected Text
>> >> > --------------
>> >> > In the interests of interoperability, implementations that parse
>> JSON texts MAY ignore the presence of a byte order mark or MAY interpret a
>> byte order mark to indicate an alternate encoding rather than treating it
>> as an error.
>> >> >
>> >> > Notes
>> >> > -----

---

From: "John R. Levine" <[email-masked]>
Date: 10 Jun 2020 12:03:00 -0400

> Apart from that, I seriously don’t understand what the "indicate an alternate encoding” would be — the JSON text already is UTF-8?

if it's a UTF-16BE BOM, quite possibly the text is encoded as UTF-16BE. 
But I agree we're deep into "don't do that" territory.

R's,
John

>> "The JavaScript Object Notation (JSON) Data Interchange Format".

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Nico Williams <[email-masked]>
Date: Wed, 10 Jun 2020 11:08:40 -0500

> This “MAY” prepares implementations for the presence of other implementations that do not heed that consensus.  No functionality is assigned by this MAY.

And the proposed edit merely says what the existing MAY implies.

> Since Errata are not intended to unravel WG decisions: Reject.

+1

> alternate encoding” would be — the JSON text already is UTF-8?

UTF-16, presumably.

No BOM is needed for that anyways.  One can unambiguously detect both,
the use of UTF-16 (and even UTF-32) and the byte endianness used.  At
any rate, for Internet purposes, we don't care one iota for any UTF
other than UTF-8, and we don't care about non-Unicode.  So, reject.

Nico
--