Subject: [Json] First two characters

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 22 Nov 2013 19:59:36 +0100

Hi,

  Now that string literals can occur at the top level, the statement in
8.1 http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-07 "the first
two characters of a JSON text will always be ASCII characters" is
incorrect and needs to be changed alongside changing the JSON-text
grammar.

regards,
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Nico Williams <[email-masked]>
Date: Fri, 22 Nov 2013 16:19:41 -0600

> incorrect and needs to be changed alongside changing the JSON-text

Good point.  Using 'xx' to mean 0x01..0x7F, XX to mean 0x01..0xFF, YY to
mean 0x80..0xFF, ZZ to mean 0x00..0xFF, and 00 to mean 0x00, we have
these possible BOM-less patterns:

   xx 00 00 00  UTF-32LE
   00 00 00 xx  UTF-32BE
   xx 00 ZZ ZZ  UTF-16LE
         but ZZ bytes cannot both be 00
   00 xx ZZ ZZ  UTF-16BE
         but ZZ bytes cannot both be 00
   xx XX        UTF-8

Simplified to assume sequential checking:

   00 00 00 xx  UTF-32BE
   00 xx        UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00        UTF-16LE
   xx           UTF-8

Optimized for the common cases:

   xx XX        UTF-8    (XX > 0)
   xx 00 ZZ ZZ  UTF-16LE (ZZ can be any value, but both ZZs can't be 0)
   00 xx        UTF-16BE
   00 00 00 xx  UTF-32BE
   xx 00 00 00  UTF-32LE

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 22 Nov 2013 14:36:41 -0800

<hat on>

Please note that the chairs tried to find some consensus in the BOM discussion and found none. Given that, and given that the current table is now wrong, our proposal is to remove it, not try to doctor it.

Current Section 8.1:

   JSON text SHALL be encoded in Unicode.  The default encoding is
   UTF-8.

   Since the first two characters of a JSON text will always be ASCII
   characters [RFC0020], it is possible to determine whether an octet
   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
   at the pattern of nulls in the first four octets.

   00 00 00 xx  UTF-32BE
   00 xx 00 xx  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 xx 00  UTF-16LE
   xx xx xx xx  UTF-8

Proposed replacement:

   The default encoding for JSON transmitted over the Internet is UTF-8.
   Transmitting JSON using other encodings may not be interoperable
   unless the receiving system definitively knows the encoding.

Does anyone have a technical objection to the proposed replacement? If so, please state the error and (hopefully) a correction.

--Matt Miller and Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Fri, 22 Nov 2013 16:47:18 -0600

On Fri, Nov 22, 2013 at 02:36:41PM -0800, Paul Hoffman wrote:
> Please note that the chairs tried to find some consensus in the BOM
> discussion and found none. Given that, and given that the current
> table is now wrong, our proposal is to remove it, not try to doctor
> it.

That's fair.

> Does anyone have a technical objection to the proposed replacement? If
> so, please state the error and (hopefully) a correction.

Might it be worthwhile to note that the previous RFC had implied (but
not stated) that UTF-16 and UTF-32 had been intended to be supported?

Also, what does this mean for the MIME media type encoding
considerations text?

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 22 Nov 2013 14:56:31 -0800

> not stated) that UTF-16 and UTF-32 had been intended to be supported?

Sorry, yes, we will need to add some words to that effect to Appendix A as well.

> Also, what does this mean for the MIME media type encoding

Nothing. The text above says that UTF-8 is the default, not the only possible encoding.

--Paul Hoffman

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 23 Nov 2013 00:02:07 +0100

>so, please state the error and (hopefully) a correction.

It is necessary for reasons of security and interoperability that all
application/json processors agree on how to get from the sequence of
bytes that make up the application/json entity to a sequence of integers
that are used in the ABNF definition of the JSON syntax. For example,

  data:application/json,%5B%22Bj+APY-rn%22%5D

Under the rules you propose, this can be interpreted as if it were

  ["Björn"]

An implementation that interprets it thus is fully conforming because
the bytes look like they are UTF-7 encoded text and the specification
does not unambiguously say that, no, the implementation must not take
it as UTF-7 encoded document, instead it must use UTF-8 to decode.

Under the rules of RFC 4627 there can be no UTF-7 encoded application/
json entities at all and processors are required to decode the example
using UTF-8.

Some web browsers had exactly this problem for a long time with other
formats, it is entirely unacceptable to go back to "just do whatever"
specifications for character encoding determination.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Pete Cordell" <[email-masked]>
Date: Sat, 23 Nov 2013 09:45:02 -0000

I believe we must have consensus that this is a contentious issue, and there 
is a lot of confusion around it.  Therefore, in the interests of 
interoperability I believe it is inappropriate to decide to be silent on all 
of these issues.  Therefore, I propose text along the following lines:

    JSON text is a sequence of Unicode codepoints.  The transfer encoding
    used to represent the characters on-the-wire is beyond the scope
    of this document.  It is therefore up to the specifications that
    reference this document to specify whether JSON messages will be
    transferred using UTF-8 (recommended), UTF-16 and/or UTF-32
    (discouraged), and whether preceding BOMs must be present,
    must not be present or are optional.

    If multiple encodings are permitted, implementers may choose to
    auto-detect a message's encoding by exploiting the fact that the
    first character of a JSON text must be in the ASCII character
    range and use the following table to deduce the active encoding:

           xx xx -- --  UTF-8
           xx 00 xx --  UTF-16LE
           xx 00 00 xx  UTF-16LE
           xx 00 00 00  UTF-32LE
           00 xx -- --  UTF-16BE
           00 00 -- --  UTF-32BE

I don't think that's a lot of text with which to describe the issues here, 
and I'm sure Tim (or someone else) can make it even snappier and more 
accurate.

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
----- Original Message ----- 
From: "Paul Hoffman" <[email-masked]>
To: "JSON WG" <[email-masked]>
Sent: Friday, November 22, 2013 10:36 PM
Subject: [Json] Wording on encoding; removing the table

> discussion and found none. Given that, and given that the current table is
> now wrong, our proposal is to remove it, not try to doctor it.
>
> Current Section 8.1:
>
> Does anyone have a technical objection to the proposed replacement? If so,
> please state the error and (hopefully) a correction.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: "Pete Cordell" <[email-masked]>
Date: Sat, 23 Nov 2013 09:21:18 -0000

----- Original Message From: "Nico Williams"
>...

I don't see why that offers any benefit over the following table presented 
earlier:

       xx xx -- --  UTF-8
       xx 00 xx --  UTF-16LE
       xx 00 00 xx  UTF-16LE
       xx 00 00 00  UTF-32LE
       00 xx -- --  UTF-16BE
       00 00 -- --  UTF-32BE

What it does demonstrate is that if we are not restricting JSON to UTF-8 and 
only UTF-8 then a table of some description would be very useful in the 
interests of interoperability.

Pete Cordell
Codalogic Ltd

---

From: "Markus Lanthaler" <[email-masked]>
Date: Sat, 23 Nov 2013 14:54:01 +0100

On Fri, Nov 22, 2013 at 07:59:36PM +0100, Bjoern Hoehrmann wrote:
>   Now that string literals can occur at the top level, ...

Can they? When did we decide that?

--
Markus Lanthaler
@markuslanthaler

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 23 Nov 2013 15:00:56 +0100

>Can they? When did we decide that?

http://www.ietf.org/mail-archive/web/json/current/msg02040.html
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Larry Masinter <[email-masked]>
Date: Mon, 25 Nov 2013 05:59:13 +0000

" The transfer encoding
    of this document."

On the contrary, it seems to me that this document is (or should be)
MAINLY ABOUT normatively establishing how JSON text is encoded
in an application/json message body "on the wire" -- something
ECMA 404 doesn't do.

JSON in ECMA-404 is a sequence of Unicode codepoints.
application/json is a sequence of octets. Those octets are used
to represent ECMA 404 JSON text, in the manner prescribed
in this document.

Insofar as ECMA 404 is updated to include the substance from this
draft (guidelines for encoding JSON text and registration of
the media type), then anything that remains (implementation
advice, non-normative ABNF) could remain in IETF, and
obsolete 4627 but normatively point to 404.

This might help resolve some of the liaison  issues as well
as clarifying the relationship of the two documents.

>From a "first principles" point of view:

* MIME types for a format are best defined in the same document 
  which defines the format. That sometimes means enhancing
  the format definition to talk about encodings in bytes in a way
  that it wouldn't otherwise be defined.

*  Having two overlapping normative documents with different
   change control is bad for multiple reasons. Even if non-optimal
   for one side, it's better to avoid overlap; one way to do that
   is by making one document    normatively reference the other,
   even if it (for information) it restates some of the information.

Larry
--
http://larry.masinter.net

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 25 Nov 2013 18:44:02 +0900

I agree with Björn here. Removing UTF-16 and UTF-32 from the spec 
completely opens the door for any other odd encoding, which would be a 
net decrease in interoperability.

Regards,   Martin.

---

From: "Pete Cordell" <[email-masked]>
Date: Tue, 26 Nov 2013 10:56:58 -0000

----- Original Message From: "Larry Masinter"
> To: "Pete Cordell"

Maybe the MIME type description could say only UTF-8 with or without a BOM, 
but the RFC is now the JSON Data Interchange Format, not just the MIME type. 
I think the JSON Data Interchange Format suggests wider applicability and 
should allow wider choice of UTFs.

Pete Cordell
Codalogic Ltd

---

From: "Markus Lanthaler" <[email-masked]>
Date: Tue, 26 Nov 2013 15:23:05 +0100

> http://www.ietf.org/mail-archive/web/json/current/msg02040.html

Thanks, I missed that. Could someone please explain how this doesn't break
existing implementations relying on the fact that valid application/json
documents are always objects or arrays at the top-level?

--
Markus Lanthaler
@markuslanthaler

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Nov 2013 15:37:25 +0100

On 26 Nov 2013, at 15:23, Markus Lanthaler <[email-masked]> wrote:

> Could someone please explain how this doesn't break

No, you can’t.

But given that this was unilaterally changed in the ECMA world already, this breaking change may still be the right thing to do.  (I don’t have a strong opinion either way, but the chairs have found WG consensus to do this.)

Grüße, Carsten