Subject: Re: [Json] JSON: remove gap between Ecma-404 and IETF draft

From: Paul Hoffman <[email-masked]>
Date: Tue, 12 Nov 2013 07:29:00 -0800

[[ Adding the JSON WG to this thread ]]

On Nov 11, 2013, at 11:01 PM, Anne van Kesteren <[email-masked]> wrote:

> To improve JSON interoperability the IETF should not define a more
> restricted version of JSON than defined by Ecma-404.
> 
> Parsers exist that can parse "42" today and parsers that cannot parse
> "42" today can be meaningfully upgraded to do so too. This would not
> break those parsers, unless they depend on parsing 42 as an error,
> which is a far more unlikely scenario than parsing it as 42 given
> precedence.
> 
> (Worth pondering about: what to do about a leading BOM, which
> XMLHttpRequest and browsers allow, but neither IETF nor Ecma-404
> allow.)
> 
> 
> -- 
> http://annevankesteren.nl/
>

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 20:24:20 +0000

>> which is a far more unlikely scenario than parsing it as 42 given

I see Anne's input on the top-level grammar as interesting and useful.  I
believe that we could choose to be reasonable here by changing the ABNF:

JSON-text = value

and then adding text about interoperability in the same way that we have
approached numbers, strings, and duplicate keys.  Protocols that come
after this draft is published as an RFC can safely decide to allow any top
level value.  Parsers that claim compatibility with the new doc will allow
any value.  For widest interoperability, however, you can't assume that an
unknown receiver will correctly process anything but an object or an array
at the top level.

We would also need to change section 8.1 according to the mechanism that
was previously proposed:

00 00 00 xx  UTF-32BE
    00 xx ?? xx  UTF-16BE
    xx 00 00 00  UTF-32LE
    xx 00 xx ?? UTF-16LE
    xx xx ?? ?? UTF-8

in order to account for strings at the top level whose first character has
a codepoint greater than 127.

It would be awful for there to remain two mutually-incompatible versions
of JSON going forward, and I think we should take this opportunity to
address the biggest incompatibility.

>>XMLHttpRequest and browsers allow, but neither IETF nor Ecma-404

If 404 doesn't allow it, I don't see a strong need to add it.  Parsers can
always be more forgiving of what they will parse than what the spec says,
particularly since section 9 says "A JSON parser MAY accept non-JSON forms
or extensions".

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Nov 2013 12:32:30 -0800

On 13 Nov 2013, at 12:24, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> changing the ABNF:

(would need to add the whitespace — that is not visible in the ECMA 404 racetracks because they use an implicit whitespace rule.)

There has been a long discussion in the WG about this potential change, I don’t want to repeat it here.
It may be worthwhile to dig up pointers to how that resulted in not making the change, for observers outside the WG.

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 20:43:10 +0000

On 11/13/13 1:32 PM, "Carsten Bormann" <[email-masked]> wrote:

>(would need to add the whitespace ‹ that is not visible in the ECMA 404
>racetracks because they use an implicit whitespace rule.)

Good catch.

>There has been a long discussion in the WG about this potential change, I
>don¹t want to repeat it here.
>It may be worthwhile to dig up pointers to how that resulted in not
>making the change, for observers outside the WG.

I would argue that we didn't have adequate representation from TC39 in
that discussion, and that their getting involved is as good a reason as
any to double-check that we've still got the same consensus.  I know that
I for one decided to stop pushing for this when it looked like there were
a few more participants that wanted the status quo.

-- 
Joe Hildebrand

---

From: Tony Hansen <[email-masked]>
Date: Wed, 13 Nov 2013 16:04:54 -0500

On 11/13/13, 3:43 PM, Joe Hildebrand (jhildebr) wrote:
> that discussion, and that their getting involved is as good a reason 
> as any to double-check that we've still got the same consensus. I know 
> that I for one decided to stop pushing for this when it looked like 
> there were a few more participants that wanted the status quo. 

I personally am on the side of making this change. I think it'd be an 
extreme shame if this incompatibility remained in the document.

     Tony Hansen

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Nov 2013 13:27:58 -0800

<no hat>

> a codepoint greater than 127.

A string at the top level of a JSON text still needs to start with an ASCII " character, so the logic is still fine, I believe.

Carsten's point about whitespace is more problematic. Does the ECMA-404 definition of a JSON text allow it to start with one (or more) whitespace characters? The text in that document says:
. . .
A JSON text is a sequence of tokens formed from Unicode code points that conforms to the JSON value grammar. The set of tokens includes six structural tokens, strings, numbers, and three literal name tokens.
. . .
Insignificant whitespace is allowed before or after any token.
. . .

It would be nice if ECMA-404 was clearer on this, given that the racetrack illustrations show everything other than the whitespace. In specific, it would be good to know whether or not the racetrack for "value" in Section 5 is meant to have optional whitespace at the left and right to match the above text. If TC39 could say for certain on that, it would be useful to the community.

--Paul Hoffman

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Wed, 13 Nov 2013 13:39:16 -0800

> It would be nice if ECMA-404 was clearer on this, given that the racetrack illustrations show everything other than the whitespace. In specific, it would be good to know whether or not the racetrac…

Yes, leading white space is allowed:

"The set of tokens includes the six structural tokens, *strings*,  *numbers*, ..."  (emphasis added)

"Insignificant whitespace is allowed before or after any token"

The elements matched by the value production are all tokens (or productions that begin and end with a token) so whitespace can occur to the left or right of any value

Allen

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Nov 2013 13:40:01 -0800

On 13 Nov 2013, at 13:27, Paul Hoffman <[email-masked]> wrote:

> It would be nice if ECMA-404 was clearer on this

I think the only way to read the English prose in ECMA-404 is to always allow insignificant whitespace around every token, including the single token that makes up an out-of-container JSON value.
That is also consistent with the most widely deployed extensions of JSON for out-of-container values, including the one in recent JavaScript.

(Yes, it is easier to be unambiguous with ABNF.)

Grüße, Carsten

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 13 Nov 2013 14:36:37 -0700

On 11/13/13 2:27 PM, "Paul Hoffman" <[email-masked]> wrote:

>A string at the top level of a JSON text still needs to start with an
>ASCII " character, so the logic is still fine, I believe.

Without top level strings, the first *two* characters of any JSON text are
always ASCII.  This:

"?"  (that's U+0022 U+0100 U+0022)

would encode the first two characters in UTF-16BE as:

00 22 01 00

8.1 currently says:

00 00 00 xx UTF-32BE
00 xx 00 xx UTF-16BE
xx 00 00 00 UTF-32LE
xx 00 xx 00 UTF-16LE
xx xx xx xx UTF-8

So the JSON text above would not match any of the table entries, causing
an error.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 21:54:29 +0000

On 11/13/13 2:40 PM, "Carsten Bormann" <[email-masked]> wrote:

>I think the only way to read the English prose in ECMA-404 is to always
>allow insignificant whitespace around every token, including the single
>token that makes up an out-of-container JSON value.
>That is also consistent with the most widely deployed extensions of JSON
>for out-of-container values, including the one in recent JavaScript.
>

Note further that since ECMA-404 says:

"The whitespace characters are: character tabulation (U+0009), line feed
(U+000A), carriage return (U+000D), and space (U+0020)"

that the encoding detection does not need to be modified again with this
change.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Nov 2013 14:18:42 -0800

> would encode the first two characters in UTF-16BE as:

Yep, you're right, and ignore my previous message. I can't count to two these days....

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Wed, 13 Nov 2013 17:47:38 -0500

Joe Hildebrand (jhildebr) scripsit:

> I see Anne's input on the top-level grammar as interesting and useful.
> I believe that we could choose to be reasonable here by changing
> the ABNF:
>
> JSON-text = value
>
> and then adding text about interoperability in the same way that we
> have approached numbers, strings, and duplicate keys.

+1 for choosing to be reasonable.

> If 404 doesn't allow [a BOM], I don't see a strong need to add it.
> Parsers can always be more forgiving of what they will parse than what
> the spec says, particularly since section 9 says "A JSON parser MAY
> accept non-JSON forms or extensions".

It's not clear that 404 disallows it, since 404 is defined in terms of
characters, and a BOM is not a character but an out-of-band signal.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
This great college [Trinity], of this ancient university [Cambridge],
has seen some strange sights. It has seen Wordsworth drunk and Porson
sober. And here am I, a better poet than Porson, and a better scholar
than Wordsworth, somewhere betwixt and between.  --A.E. Housman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 23:51:44 +0000

>characters, and a BOM is not a character but an out-of-band signal.

Agree.  However, that signal would be a part of the 4627bis octet stream,
so a little interop guidance would likely be useful.  Something like:

"Some producers of JSON produce JSON-text that starts with a redundant
U+FEFF (ZERO WIDTH NO-BREAK SPACE, previously known as BYTE ORDER MARK)
with the ostensible purpose of signaling the encoding of the text to
follow.  Since JSON has other mechanisms to determine encoding, this is
not required.  Receiving applications MAY safely ignore this initial
character without generating an error.  Implementations that do not send
U+FEFF are interoperable in the sense that all software implementations
which receive the un-prefixed text will not generate parse errors."

-- 
Joe Hildebrand

---

From: Mark Davis ☕ <[email-masked]>
Date: Wed, 13 Nov 2013 16:50:44 -0800

On Wed, Nov 13, 2013 at 3:51 PM, Joe Hildebrand (jhildebr) <
[email-masked]> wrote:

> that all software implementations

perhaps:

...​all conformant software ...​

Mark <https://google.com/+MarkDavis>

*— Il meglio è l’inimico del bene —*

---

From: John Cowan <[email-masked]>
Date: Wed, 13 Nov 2013 20:49:33 -0500

> U+FEFF (ZERO WIDTH NO-BREAK SPACE, previously known as BYTE ORDER MARK)

For "previously" read "also".  Indeed, the use of U+FEFF as a
zero-width no-break space is deprecated in favor of U+2600 WORD JOINER.
See <http://www.unicode.org/faq/utf_bom.html#BOM>.

-- 
John Cowan                                <[email-masked]>
Yakka foob mog.  Grug pubbawup zink wattoom gazork.  Chumble spuzz.
    --Calvin, giving Newton's First Law "in his own words"

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Wed, 13 Nov 2013 18:35:06 -0800

>> characters, and a BOM is not a character but an out-of-band signal.

However, for example, a conforming implementation of the ECMAScript JSON.parse function would reject any string passed to it that starts with a U+FFEF code point because the unquoted occurrence of that code point does not conform to the ECMA-252, 5th Edition or Ecma-404 JSON grammar.

In order to be successfully processed, that code point would have to be stripped from the string prior to calling JSON.parse.

Allen

> which receive the un-prefixed text will not generate parse errors."

Isn't it an interooperbility issue that many receiving applications do not ignore it.

Allen

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Nov 2013 19:15:28 -0800

<no hat>

> In order to be successfully processed, that code point would have to be stripped from the string prior to calling JSON.parse.

The question was specifically about ECMA-404, not ECMA-252. It would be great to hear from TC39 whether or not ECMA-404 allows or disallows it.

--Paul Hoffman

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 14 Nov 2013 20:14:29 +0900

Hello Henry, others,

On 2013/11/14 18:44, Henry S. Thompson wrote:
> John Cowan writes:
> I think this is a crucial observation.

Yes, and I think it's based on the experience with XML. But while this 
experience may be applicable to JSON, Anne's original comment about the 
BOM and XMLHttpRequest suggests that 404 actually currently does not 
tolerate a BOM, and that implementations (except for XMLHttpRequest) 
also don't.

To give some historic background, the BOM for UTF-8 wasn't in the first 
edition of XML 
(http://www.w3.org/TR/1998/REC-xml-19980210#sec-guessing). It only later 
came in because Microsoft used it for notepad to be able to quickly 
distinguish between UTF-8 and the legacy system encoding. Because many 
people were writing some XML by hand, and some of them were using 
notepad, the pressure on XML to accept a BOM at the start of an UTF-8 
file mounted, and it was included in the second edition of the XML 
Recommendation (http://www.w3.org/TR/2000/REC-xml-20001006#sec-guessing).

Compared to XML, JSON may be much less edited by hand, or much less 
edited on notepad, or otherwise just have a different history from XML, 
but we have to make sure.

Regards,   Martin.

> I note that XML approaches
> this problem in what might be a useful way.  The XML ABNF makes no
> mention of BOM, it's not part of any XML document as such.  But it
> _is_ allowed.  The relevant wording [1] is:
>
>    Entities ... may begin with the Byte Order Mark described by Annex H
>    of [ISO/IEC 10646:2000], section 16.8 of [Unicode] (the ZERO WIDTH
>    NO-BREAK SPACE character, #xFEFF). _This is an encoding signature,_
>    _not part of either the markup or the character data of the XML_
>    _document._ XML processors must be able to use this character to
>    differentiate between UTF-8 and UTF-16 encoded documents. [emphasis
>    added]
>
> ht
>
> [1] http://www.w3.org/TR/REC-xml/#charencoding

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 14 Nov 2013 12:04:16 -0000

Original Message From: "Joe Hildebrand" <[email-masked]>

> ...

 In http://www.ietf.org/mail-archive/web/json/current/msg00565.html I 
mentioned that we also need to allow for characters such as U+2c00 to be the 
first character in a quoted string.

This requires a pattern like:

    xx 00 00 xx  UTF-16LE

giving:

   00 00 00 xx  UTF-32BE
   00 xx 00 xx  UTF-16BE
   00 xx xx xx  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 00 xx  UTF-16LE
   xx 00 xx 00  UTF-16LE
   xx 00 xx xx  UTF-16LE
   xx xx xx xx  UTF-8

That can be reduced a bit if we use "--" to indicate "not-tested":

   00 00 -- --  UTF-32BE
   00 xx -- --  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 00 xx  UTF-16LE
   xx 00 xx --  UTF-16LE
   xx xx -- --  UTF-8

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 14 Nov 2013 14:59:57 +0000

>first character in a quoted string.

Ah, yes.  Sorry, I quoted from the wrong part of the conversation.  I
completely agree.

>That can be reduced a bit if we use "--" to indicate "not-tested":

+1 to this table.  It's clear, correct, and implementable.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Nov 2013 10:33:41 -0500

Paul Hoffman scripsit:

> The question was specifically about ECMA-404, not ECMA-252. It would be
> great to hear from TC39 whether or not ECMA-404 allows or disallows it.

The point of issuing standards is so that all may use them, so there is
no need to ask anyone.  A quick examination of ECMA-404 makes it clear
that there are no references to BOMs, whether under that name, or "byte
order mark", or "U+FEFF".

But that is not determinative for our purposes, because of this statement
from the Unicode Standard (I cite section 16.8 of Unicode 6.2, but
substantially equivalent statements can be found back to Unicode 3.2):

    Systems that use the byte order mark must recognize when an
    initial U+FEFF signals the byte order. In those cases, it is
    not part of the textual content and should be removed before
    processing, because otherwise it may be mistaken for a legitimate
    zero width no-break space.

Per contra, ECMA-404 refers only to text(ual content).  The BOM is
meaningful when transforming byte sequences into code point sequences,
but ECMA-404 deals in the latter only.  So it is the furthest thing
from surprising that it makes no mention of BOMs, and has nothing to
say about their use outside text.

-- 
Overhead, without any fuss, the stars were going out.
        --Arthur C. Clarke, "The Nine Billion Names of God"
                John Cowan <[email-masked]>

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 14 Nov 2013 09:44:00 +0000

> characters, and a BOM is not a character but an out-of-band signal.

I think this is a crucial observation.  I note that XML approaches
this problem in what might be a useful way.  The XML ABNF makes no
mention of BOM, it's not part of any XML document as such.  But it
_is_ allowed.  The relevant wording [1] is:

ht

[1] http://www.w3.org/TR/REC-xml/#charencoding
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 14 Nov 2013 07:48:30 -0800

> The question was specifically about ECMA-404, not ECMA-252. It would be great to hear from TC39 whether or not ECMA-404 allows or disallows it.

ECMA=nnn is the correct designation for Ecma standards.  When speaking about the organization "Ecma" is the usage.

Allen

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 14 Nov 2013 07:54:56 -0800

> say about their use outside text.

Exactly ECMA-404 only defines a textual content format.  That was intentional.

Allen

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 14 Nov 2013 17:17:23 +0000

> +1 to this table.  It's clear, correct, and implementable.

Doesn't work if you want to allow a pre-document BOM.  But I think the
following enlarged grid does work, where "--" now means "not any
value(s) explicitly tested for with an 'earlier' shared prefix":

00 00 FE FF  UTF-32 (BE) w. BOM
[00 00 FF FE  UCS-4 w. BOM, unusual octet order (2143)]
00 00 -- --  UTF-32BE
00 xx -- --  UTF-16BE
FF FE 00 00  UTF-32 (LE) w. BOM
[FE FF 00 00  UCS-4 w. BOM, unusual octet order (3412)]
FE FF -- --  UTF-16 (BE) w. BOM
FF FE -- --  UTF-16 (LE) w. BOM
xx 00 00 00  UTF-32LE
xx 00 00 xx  UTF-16LE [Note that the following algorithm disagrees here]
xx 00 xx --  UTF-16LE
EF BB BF --  UTF-8 w. BOM
xx xx -- --  UTF-8

A possibly simpler algorithm, which has the same outcome I think,
minus the unusual octet cases and the exception noted above, is used
by the Python requests package [1] for JSON charset detection.
Schematically, this works as follows:

FF FE 00 00  UTF-32 (LE) w. BOM
00 00 FE FF  UTF-32 (BE) w. BOM
EF BB BF     UTF-8 w. BOM
FE FF        UTF-16 (BE) w. BOM
FF FE        UTF-16 (LE) w. BOM
Now count 00 bytes in first 4:
 0           UTF-8
 2
  00 -- 00 --   UTF-16 (BE)
  -- 00 -- 00   UTF-16 (LE)
 3
  00 00 00 --   UTF-32 (BE)
  -- 00 00 00   UTF-32 (LE)
 error

I _think_ requests is only correct because it assumes "JSON always
starts with two ASCII characters", depending on 4627, i.e.  continuing
to rule out e.g.

  "Ѐкземпияр"

To accommodate this case, we would need to add

 1
  -- 00 -- --   UTF-16 (LE)
 2
  -- 00 00 --   UTF-16 (LE)

to the requests algorithm.

(There are, it has to be said, few Unicode characters whose UTF-16-L
form is 00xx, i.e. U+xx00, the first code point on a code page -- I
had to hunt pretty hard to find the above specimen, which is in fact a
slight cheat :-) Many code pages have a gap at the 00 point.  Not sure
about the status of U+4E00, one variant of the ideograph for the
numeral 1).

ht

[1] http://docs.python-requests.org/en/latest/
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND    (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this    mail without it is forged spam]

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Nov 2013 13:29:52 -0500

Henry S. Thompson scripsit:

> form is 00xx, i.e. U+xx00, the first code point on a code page --
> I had to hunt pretty hard to find the above specimen, which is in
> fact a slight cheat :-) Many code pages have a gap at the 00 point.

There are 68 of them on the Basic Multilingual Plane.  But many
characters in other planes involve such 16-bit code units.  For example,
all of U+10000 to U+103FF are encoded as D800 DC00 through D800 DFFF.
Currently there are 622 characters in this range alone, and the number
will probably grow.

> Not sure about the status of U+4E00, one variant of the ideograph for
> the numeral 1).

Google reports over 3 gigahits for this character.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
To say that Bilbo's breath was taken away is no description at all.  There are
no words left to express his staggerment, since Men changed the language that
they learned of elves in the days when all the world was wonderful. --The Hobbit

---

From: David Booth <[email-masked]>
Date: Thu, 14 Nov 2013 15:24:26 -0500

> Henry S. Thompson scripsit:
[ . . . ]
> Google reports over 3 gigahits for this character.

Curious: How did you do that query?

David

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Nov 2013 17:43:55 -0500

> Curious: How did you do that query?

Using <http://www.sceneonthe.net/unicode.htm>, I put 4E00 into the
"Character numbers" box, and pasted the entry in the "Characters" box
into a Google window.

I could equally well have typed 4300 Alt+X into Microsoft Word, or used
<http://www.fileformat.info/info/unicode/char/4E00>.

-- 
On the Semantic Web, it's too hard to prove     John Cowan    [email-masked]
you're not a dog.  --Bill de hOra               http://www.ccil.org/~cowan

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 15 Nov 2013 16:10:37 +0900

> Google reports over 3 gigahits for this character.

Yes indeed. U+4E00 is not just one variant of the ideograph for the 
numeral 1, except for special occasions such as bank checks, it's the 
only ideograph used for the numeral 1, as well as the word 'one', and 
one of the most used ideographs overall.

Regards,   Martin.

---

From: [email-masked] (Henry S. Thompson)
Date: Fri, 15 Nov 2013 08:45:00 +0000

> Google reports over 3 gigahits for this character.

Thanks for the facts, much better than my suppositions.

ht

---

From: "Pete Cordell" <[email-masked]>
Date: Sun, 17 Nov 2013 10:02:34 -0000

There's some debate about whether BOMs are allowed (at least officially.  I
think I would accommodate them if I were to implement a JSON parser.)

If I did handle BOMs I think I would adopt the "if it starts with a BOM..."
approach similar to the Python method, and only resort to deduction based on
assumption about ASCII characters if no BOM was found.  (I think BOM
presence can be surmised by only looking at the first byte.  If it's greater
than 0x80 - specifically if it's 0xef, 0xfe, 0xff - then it's an error if
you don't go on to find a BOM.)

While I'm here, Joe mentioned "implementable".  From an implementer's
perspective the table I presented earlier might be better presented in the
following order:

   00 00 -- --  UTF-32BE
   00 xx -- --  UTF-16BE
   xx xx -- --  UTF-8
   xx 00 xx --  UTF-16LE
   xx 00 00 xx  UTF-16LE
   xx 00 00 00  UTF-32LE

Or is that taking the fun out of it for the implementer?!

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: "Henry S. Thompson" <[email-masked]>
To: "Joe Hildebrand (jhildebr)" <[email-masked]>
Cc: "Pete Cordell" <[email-masked]>; "Paul Hoffman" 
<[email-masked]>; <[email-masked]>; "JSON WG" <[email-masked]>
Sent: Thursday, November 14, 2013 5:17 PM
Subject: Re: Encoding detection

  "Ѐкземпияр"

 1
  -- 00 -- --   UTF-16 (LE)
 2
  -- 00 00 --   UTF-16 (LE)

ht

 [mail from me _always_ has a .sig like this    mail without it is forged 
spam]

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 17 Nov 2013 08:34:24 -0800

> Or is that taking the fun out of it for the implementer?!

I'm not sure that is the right order. Wouldn't that make UTF-16LE be mistaken for UTF-8? It seems to me that the UTF-8 rule has to be last because it is more general than any of the ones above it.

--Paul Hoffman

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 17 Nov 2013 21:00:48 +0100

* Paul Hoffman wrote:
>I'm not sure that is the right order. Wouldn't that make UTF-16LE be 
>mistaken for UTF-8? It seems to me that the UTF-8 rule has to be last 
>because it is more general than any of the ones above it.

No, `xx` stands for a non-zero byte.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 17 Nov 2013 12:19:30 -0800

> No, `xx` stands for a non-zero byte.

Sorry, I thought "xx" meant "unknown". Then I think Pete's formulation is correct.

--Paul Hoffman

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 10:05:07 -0000

Given the history below, would it be sensible to accept BOMs for UTF-8
encoding, but not for UTF-16 and UTF-32?  In other words, are BOMs needed
and/or used in the wild for UTF-16 and UTF-32?

Maybe the text can say something like "SHOULD accept BOMs for UTF-8, and MAY 
accept BOMs for UTF-16 and / or UTF-32"?

Thanks,

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: ""Martin J. Dürst"" <[email-masked]>
To: "Henry S. Thompson" <[email-masked]>
Cc: "John Cowan" <[email-masked]>; "IETF Discussion"
<[email-masked]>; "Paul Hoffman" <[email-masked]>; "JSON WG"
<[email-masked]>; "Joe Hildebrand (jhildebr)" <[email-masked]>; "Anne van
Kesteren" <[email-masked]>; <[email-masked]>; "es-discuss"
<[email-masked]>
Sent: Thursday, November 14, 2013 11:14 AM
Subject: Re: [Json] JSON: remove gap between Ecma-404 and IETF draft

> tolerate a BOM, and that implementations (except for XMLHttpRequest) also
> don't.
>
> edition of XML (http://www.w3.org/TR/1998/REC-xml-19980210#sec-guessing).
> It only later came in because Microsoft used it for notepad to be able to
> quickly distinguish between UTF-8 and the legacy system encoding. Because
> many people were writing some XML by hand, and some of them were using
> notepad, the pressure on XML to accept a BOM at the start of an UTF-8 file
> mounted, and it was included in the second edition of the XML
> Compared to XML, JSON may be much less edited by hand, or much less edited
> on notepad, or otherwise just have a different history from XML, but we
> have to make sure.
>
> Regards,   Martin.
>
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 10:02:15 -0000

----- Original Message ----- 
From: "Paul Hoffman" <[email-masked]>
To: "Bjoern Hoehrmann" <[email-masked]>
Cc: "Pete Cordell" <[email-masked]>; "JSON WG" <[email-masked]>
Sent: Sunday, November 17, 2013 8:19 PM
Subject: Re: [Json] Encoding detection

> Sorry, I thought "xx" meant "unknown". Then I think Pete's formulation is 

Although my note about quick detection of BOMs (i.e.: I think BOM
presence can be surmised by only looking at the first byte.  If it's greater
than 0x80 - specifically if it's 0xef, 0xfe, 0xff - then it's an error if
you don't go on to find a BOM.) missed the case for 00 00 FE FF  UTF-32 (BE) 
w. BOM.  Argh!!!

Pete Cordell
Codalogic Ltd

---

From: [email-masked] (Henry S. Thompson)
Date: Mon, 18 Nov 2013 11:11:01 +0000

Pete Cordell writes:

> Maybe the text can say something like "SHOULD accept BOMs for UTF-8,
> and MAY accept BOMs for UTF-16 and / or UTF-32"?

My sense is that you'll see more UTF-16 BOMs than anything else.
UTF-32 support seems to be waning (at least in the browsers), but
UTF-16 is in pretty widespread use.  John, do you think you can fool
google into counting BOMs for us?

ht

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 18 Nov 2013 20:26:44 +0900

> My sense is that you'll see more UTF-16 BOMs than anything else.

Yes indeed. BOM means Byte Order Mark. It's crucial for over-the-wire 
UTF-16. (It's irrelevant for in-memory UTF-16, but that's not what we 
are discussing.) To bring up the XML example again, XML actually 
strictly requires a BOM for UTF-16. The IETF definition of UTF-16 does 
not require a BOM for UTF-16. See http://tools.ietf.org/html/rfc2781, in 
particular http://tools.ietf.org/html/rfc2781#section-3.2, 
http://tools.ietf.org/html/rfc2781#section-3.3, and 
http://tools.ietf.org/html/rfc2781#section-4.

For UTF-8, the BOM is not a Byte Order Mark, because such a mark isn't 
necessary at all. It may serve as a signature, but is not necessary, and 
in some circumstances counterproductive.

As for what to say about whether to accept BOMs or not, I'd really want 
to know what the various existing parsers do. If they accept BOMs, then 
we can say they should accept BOMs. If they don't accept BOMs, then we 
should say that they don't.

Regards,   Martin.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 18 Nov 2013 13:35:00 +0100

>should say that they don't.

Unicode signatures are not useful for application/json resources and are
likely to break exisiting and future code, it is not at all uncommon to
construct JSON text by concatenating, say, string literals with some web
service response without passing the data through a JSON parser. And as
RFC 4627 makes no mention of them, there is little reason to think that
implementations tolerate them.

Perl's JSON module gives me

  malformed JSON string, neither array, object, number, string
  or atom, at character offset 0 (before "\x{ef}\x{bb}\x{bf}[]")

Python's json module gives me

  ValueError: No JSON object could be decoded

Go's "encoding/json" module gives me

  invalid character 'ï' looking for beginning of value

http://shadowregistry.org/js/misc/#t2ea25a961255bb1202da9497a1942e09 is
another example of what kinds of bugs await us if we were to specify the
use of Unicode signatures for JSON, essentially

  new DOMParser().parseFromString("\uBBEF\u3CBF\u7979\u3E2F","text/xml")

Now U+BBEF U+3CBF U+7979 U+3E2F is not an XML document but Firefox and
Internet Explorer treat it as if it were equivalent to "<yy/>".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: [email-masked] (Henry S. Thompson)
Date: Mon, 18 Nov 2013 12:59:18 +0000

>   invalid character 'ï' looking for beginning of value

I'm curious to know what level you're invoking the parser at.  As
implied by my previous post about the Python 'requests' package, it
handles application/json resources by stripping any initial BOM it
finds -- you can try this with

>>> import requests
>>> r=requests.get("http://www.ltg.ed.ac.uk/ov-test/b16le.json")
>>> r.json()

Signatures are not part of the text of a document, as the UNICODE spec
makes clear, so asking what happens when you pass a string beginning
with a BOM to a parser is not really the right question in this
context, is it?

As I tried to say in an earlier post, there's a distinction which
needs to be carefully insisted on between, on the one hand, languages
and their parsers, where I agree signatures/BOMs have no place, and,
on the other hand, (media-typed) resources/entities/payloads and _their_
processing, where a discussion of BOMs/signatures _is_ appropriate
and, often, necessary.

BTW I agree that the status of the UTF-8 BOM as signature is slightly
hazy, but again the UNICODE spec itself [1] says

  "this sequence can serve as signature for UTF-8 encoded text where
   the character set is unmarked"

ht

[1] http://www.unicode.org/versions/Unicode6.2.0/ch16.pdf

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 13:36:13 -0000

----- Original Message ----- 
> UTF-16. (It's irrelevant for in-memory UTF-16, but that's not what we are 
> discussing.)

The in-memory case is not entirely irrelevant because a number of JSON 
messages will be constructed in memory and then squirted to line.

I did a little experiment with Visual Studio.  It will allow me to save in 
UTF-8 with or without a BOM (like thing).  Saving in UTF-16 (Or was it 
UCS2?) is always with a BOM.  There didn't seem to be a UTF-32 option.

JSON doesn't need BOMs.  However, there are cases where people might hand 
edit messages, and if they choose to save in UTF-16 they will likely have a 
BOM.

Is it acceptable to tell people not to save hand editted files in UTF-16, 
suggesting UTF-8 (possibly with an encoded BOM) as an alternative?

I would imagine that if someone did have a hand editted UTF-8 file on 
Windows then the allowance of a BOM would help their sanity immeasurably, 
but it's not something I have firsthand knowledge of.

I believe Unix/Linux works with UTF-8 without BOMs.  Is this the case?

Pete Cordell
Codalogic Ltd

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 18 Nov 2013 14:48:19 +0100

>>>> r=requests.get("http://www.ltg.ed.ac.uk/ov-test/b16le.json")

The Perl code was

  perl -MJSON -MEncode -e
    "my $s = encode_utf8(chr 0xFEFF) . '[]'; JSON->new->decode($s)"

The Python code was

  import json
  json.loads(u"\uFEFF[]".encode('utf-8'))

The Go code was

  package main
  
  import "encoding/json"
  import "fmt"
  
  func main() {
    r := "\uFEFF[]"
  
    var f interface{}
    err := json.Unmarshal([]byte(r), &f)
    
    fmt.Println(err)
  }

In other words, always passing a UTF-8 encoded byte string to the byte
string parsing part of the JSON implementation. RFC 4627 is the only
specification for the application/json on-the-wire format and it does
not mention anything about Unicode signatures. Looking for certain byte
sequences at the beginning and treating them as a Unicode signature is
the same as looking for `/* ... */` and treating it as a comment.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Mon, 18 Nov 2013 07:54:58 -0800

This feels backward, because BOMs are actually useful for UTF-16 and
UTF-32, but essentially useless for UTF-8.

On Mon, Nov 18, 2013 at 2:05 AM, Pete Cordell <[email-masked]>wrote:

> Maybe the text can say something like "SHOULD accept BOMs for UTF-8, and
> MAY accept BOMs for UTF-16 and / or UTF-32"?
>
> Thanks,
>
> Pete Cordell
> Codalogic Ltd
> ----- Original Message ----- From: ""Martin J. Dürst"" <
> [email-masked]>

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 16:08:53 -0000

----- Original Message From: "Tim Bray" <[email-masked]>
> UTF-32, but essentially useless for UTF-8.

Not useless if you're trying to tell the difference between a hand editted 
Windows cp-1252 (or whatever it's called) encoded text file and a UTF-8 
encoded text file.

I don't think we need them for any other reason, but I think some 
international Windows users would be thankful if you allowed them for that 
case.

On Mon, Nov 18, 2013 at 2:05 AM, Pete Cordell <[email-masked]>wrote:

>> file
>> edited

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Nov 2013 11:19:54 -0500

> My sense is that you'll see more UTF-16 BOMs than anything else.

I agree.

> google into counting BOMs for us?

No, because Google transcodes everything into UTF-8 as soon as it starts
to process it.  What I can say (auct. Mark Davis) is that UTF-16 documents
in all formats represent much less than 0.1% of the searchable Web.
By contrast, UTF-8 (including ASCII) amounts to 80% of it.  This reflects
actual rather than declared encodings, and is as of January 2012.

-- 
So they play that [tune] on                     John Cowan
their fascist banjos, eh?                       [email-masked]
        --Great-Souled Sam                      http://www.ccil.org/~cowan

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Nov 2013 17:22:42 +0100

On 2013-11-18 17:08, Pete Cordell wrote:
> Not useless if you're trying to tell the difference between a hand
> editted Windows cp-1252 (or whatever it's called) encoded text file and
> a UTF-8 encoded text file.

Ye.

> international Windows users would be thankful if you allowed them for
> that case.

So I gather "non-international Windows" users never need non-ASCII 
characters? :-) (Yes, that explains lots of I18N brokenness originating 
from certain countries...)

Best regards, Julian

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Nov 2013 12:07:09 -0500

Pete Cordell scripsit:

> editted Windows cp-1252 (or whatever it's called) encoded text file
> and a UTF-8 encoded text file.

In principle you cannot tell, but in practice it's possible to discriminate
betwen the two with extremely high reliability.

-- 
"Well, I'm back."  --Sam        John Cowan <[email-masked]>

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 18:26:15 -0000

----- Original Message From: "Julian Reschke" 
> from certain countries...)

Only as ISO-8859-1 :-)

(Quickly hides!!!)

Pete Cordell
Codalogic Ltd

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 18:19:46 -0000

----- Original Message From: "John Cowan"

> betwen the two with extremely high reliability.

John,

Do you mean that the presence of a UTF-8 BOF sequence doesn't prove that 
it's not Windows cp-1252 or do you mean you can tell apart a UTF-8 and 
cp-1252 file without BOMs?

If the latter, do the relevant tools take the time to distinguish the 2 
without BOMs?

Thanks,

Pete Cordell
Codalogic Ltd

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Nov 2013 13:41:53 -0500

Pete Cordell scripsit:

> Do you mean that the presence of a UTF-8 BOF sequence doesn't prove
> that it's not Windows cp-1252 or do you mean you can tell apart a
> UTF-8 and cp-1252 file without BOMs?

I meant the latter, but the former is true, too.  A plain text document
beginning "ï»¿" in Windows-1252 will appear to begin with an 8-BOM
in the absence of out of band information.

> If the latter, do the relevant tools take the time to distinguish
> the 2 without BOMs?

Some tools do, some don't.  The IRC client I use, XChat, attempts to
convert input as UTF-8, and if that fails, converts it as Latin-1.
I have not yet seen it produce mojibake.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is
dramatically overdescribed.  Still other languages are simultaneously
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King

---

From: Tatu Saloranta <[email-masked]>
Date: Mon, 18 Nov 2013 10:59:48 -0800

Dominant Java implementations support UTF-16 with BOM; either directly or
through Java's Reader implementations that handle BOMs.
String concatenation case seems irrelevant, since BOMs are not included in
in-memory representation anyway, as opposed to byte stream serialization.

-+ Tatu +-

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 18 Nov 2013 15:56:11 -0500

> but it's not something I have firsthand knowledge of.

I believe the opposite is true.

The failure of Windows to correctly process documents without BOM markers
is a constant pain trying to use .NET to parse XML.

The ability to compose a JSON message by wrapping another JSON message is
essential. That is, it has to be possible to write something like

printf ("{\"Object\", %s}", Text);

I use the .NET platform heavily. Please do not let Microsoft off the hook
here. The cost of doing so is having to write code to kick out spurious BOM
sequences occurring at any random point in the text. Which becomes really
painful when having to deal with strings where there might actually be a
reason to put the BOM in.

The benefit of not doing so is that it might encourage Microsoft to fix
their tools so that they don't insert spurious BOM sequences in documents
where doing so breaks them.

-- 
Website: http://hallambaker.com/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 Nov 2013 13:32:37 +0900

Okay, here are some more tests.

http://www.sw.it.aoyama.ac.jp/2013/pub/json_tests/test1_utf8_nobom.json
http://www.sw.it.aoyama.ac.jp/2013/pub/json_tests/test2_utf8_bom.json

They are self-describing JSON files served with application/json, the 
first without a BOM, and the second with a BOM.

They contain some Japanese, and a tiny bit of Spanish.

[see more below]

>>>> r=requests.get("http://www.ltg.ed.ac.uk/ov-test/b16le.json")

I get a 404 on this example. I can put up UTF-16 examples, too.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 Nov 2013 20:09:30 +0900

On 2013/11/19 19:10, t.p. wrote:
> From: "Martin J. Dürst"<[email-masked]>

> Martin
> We had a similar discussion with syslog back in 2005, the issue being
> that UTF-8 was new and different and how to tell whether it was being
> used or not, and what made it into RFC5424 was
> "  If a syslog application encodes MSG in UTF-8, the string MUST start
>     with the Unicode byte order mask (BOM), which for UTF-8 is ABNF
>     %xEF.BB.BF.  "
> which remains a MUST to this day.  There are no relevant Errata.
>
> Tom Petch

This is something that seems to have made quite a lot of sense for 
syslog. I can understand that if before 2005, syslog was used with 
legacy encodings (iso-8859-1, Shift_JIS and similar), and there was 
otherwise no easy way to label the UTF-8 strings.

But another solution (for syslog, that is) would also have been 
possible. As John already pointed out, UTF-8 is very easy to detect 
heuristically: If a byte sequence follows the UTF-8 byte pattern, it's 
most definitely UTF-8 and not something else. For more background, 
please see http://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf, where 
that idea came up first.

As for JSON, it doesn't have the problem of legacy encodings. JSON by 
definition is encoded in an Unicode encoding form, and it's easy to 
distinguish these because of the restrictions on character sequences in 
JSON. And this can be done without a BOM (or with a BOM).

What's most important now is to know what receivers actually accept. We 
are not in a design phase, we are just updating the definition of JSON 
and making sure we fix problems if there are problems, but we have to 
use the installed base for the main guidance, not other protocols or 
formats.

Regards,   Martin.

---

From: "Pete Cordell" <[email-masked]>
Date: Tue, 19 Nov 2013 11:26:26 -0000

Hi Philip,

Could you explain further how these spurious BOMs are getting added in by MS 
in .Net?  If you could come up with a small piece of C# code to demonstrate 
that would be great to help my understanding.

Thanks,

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: "Phillip Hallam-Baker" <[email-masked]>
To: "Pete Cordell" <[email-masked]>
Cc: "Martin J. Dürst" <[email-masked]>; "Henry S. Thompson" 
<[email-masked]>; "John Cowan" <[email-masked]>; "IETF Discussion" 
<[email-masked]>; "JSON WG" <[email-masked]>; "Anne van Kesteren" 
<[email-masked]>; <[email-masked]>; "es-discuss" <[email-masked]>
Sent: Monday, November 18, 2013 8:56 PM
Subject: Re: BOMs

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Nov 2013 13:31:02 +0100

>in-memory representation anyway, as opposed to byte stream serialization.

HTTP implementations cannot correctly determine whether an entity body
is text in a single character encoding and if so what that encoding is,
accordingly the dominant API deals in byte[] arrays, not text Strings;
furthermore, many programming languages default to byte[] arrays for
string literals. That often combines into forms of

  byte[] json = sprintf('{"x": %s, "y": %s}', GET(...), GET(...));

which works fine if all three byte[] arrays are UTF-8 encoded and use
no Unicode signature, which is the case 99% of the time.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: t.p. <[email-masked]>
Date: Tue, 19 Nov 2013 10:10:47 +0000

----- Original Message -----
From: "Martin J. Dürst" <[email-masked]>
To: "Henry S. Thompson" <[email-masked]>
Cc: "John Cowan" <[email-masked]>; "IETF Discussion"
<[email-masked]>; "Pete Cordell" <[email-masked]>; "JSON WG"
<[email-masked]>; "Anne van Kesteren" <[email-masked]>;
<[email-masked]>; "es-discuss" <[email-masked]>
Sent: Monday, November 18, 2013 11:26 AM

Martin

Tom Petch

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Tue, 19 Nov 2013 08:59:36 -0800

On Nov 19, 2013, at 3:09 AM, Martin J. Dürst wrote:
> ...
> As for JSON, it doesn't have the problem of legacy encodings. JSON by definition is encoded in an Unicode encoding form, and it's easy to distinguish these because of the restrictions on character sequences in JSON. And this can be done without a BOM (or with a BOM).
> 
> What's most important now is to know what receivers actually accept. We are not in a design phase, we are just updating the definition of JSON and making sure we fix problems if there are problems, but we have to use the installed base for the main guidance, not other protocols or formats.

There can be no doubt that the most widely deployed JSON parsers are those that are built intp the browser javascript implementations.  The ECMAScript 5 specification for JSON.parse that they implement says BOM is an illegal character.  But what do the browser actually implement?  This:

//FireFox 25 scratchpad execution:
JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/

JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/

JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/
JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/
JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/

//Safari 5.1.9 JS console
JSON.parse('\ufeff {"abc": 0} ')
message: "JSON Parse error: Unrecognized token '?'"

//Chrome 31 JS console
JSON.parse('\ufeff {"abc": 0} ')
SyntaxError: Unexpected token 
message: "Unexpected token "

Unfortunately, I don't have access to IE right now,  but the trend is clear

Allen

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 19 Nov 2013 10:30:47 -0800

> no Unicode signature, which is the case 99% of the time.

My point was just that although it appears that many scripting languages
may not deal with BOM properly, same is not true on all platforms. Proper
JSON APIs on JVM do accept both String and byte[] based input; byte[] being
preferred since it is more efficient, and reliably with auto-detection,
assuming that -- as per JSON specification -- the only single-byte encoding
used is UTF-8.

-+ Tatu +-

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 19 Nov 2013 18:55:29 +0000

Allen Wirfs-Brock writes:

> There can be no doubt that the most widely deployed JSON parsers are
> those that are built intp the browser javascript implementations.
> The ECMAScript 5 specification for JSON.parse that they implement
> says BOM is an illegal character.  But what do the browser actually
> implement?  This:

No, try e.g. jsonviewer.stack.hu [1] (works in Chrome, Safari, Opera,
not in IE or Firefox) or feed [2] to www.jsoneditoronline.org (Use
Open/Url) (works in Chrome, IE, Firefox, ran out of time to test more).

As previously discussed, _no-one_ is arguing that BOMs are in the JSON
language as such.  JSON parsers shouldn't accept BOMs.

BOMs are, to quote the UNICODE spec, "not part of the text".  It is
appropriate that specs concerned with JSON-on-the-wire, for example
the media type registration for 'application/json', _should_ discuss
the BOM, and it's open to them, _without changing the language at
all_, to say that BOMs are acceptable but, again, are not part of the
text which the parser has to accept.

ht

[1] http://jsonviewer.stack.hu/#http://www.ltg.ed.ac.uk/~ht/ov-test/b16le.json

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Nov 2013 20:24:36 +0000

I've been back and forth on this several times, however I think I've come
to a conclusion for myself unless new arguments or data comes in.

I don't see any value in a BOM ever being included intentionally.  RFC4627
does not specify anything about BOMs.  RFC4627 explicitly has a mechanism
for determining encoding that does not use BOMs.  Although ECMA-262
supports more whitespace than RFC4627, <BOM> is not one of the supported
whitespace characters in my reading.  Same with ECMA-404.  Many existing
implementations do not parse BOM-prepended text in practice (thanks
Allen). 

I understand the use case of manually creating a document in notepad and
getting a BOM you didn't expect, but I don't think that's a compelling
enough reason to break backward-compatibility by adding effectively-new
blessings for BOMs.

Section 9 provides complete air-cover for any parser writer that wants to
support BOMs anyway:

"A JSON parser MAY accept non-JSON forms or extensions."

What they do with them, particularly when they don't match the encoding
detection of section 8.1 would of course be unspecified, but those
documents would have been rejected by a strictly-conforming parser anyway.

As such, I think our best course of action is to change nothing with
respect to BOMs, leaving them as non-interoperable syntax extensions.

On 11/19/13 5:59 PM, "Allen Wirfs-Brock" <[email-masked]> wrote:

>those that are built intp the browser javascript implementations.  The
>ECMAScript 5 specification for JSON.parse that they implement says BOM is
>an illegal character.  But what do the
> browser actually implement?  This:
>
>
>*/JSON.parse('\ufeff {"abc": 0} ')
>/*
>*/JSON.parse('\ufeff {"abc": 0} ')
>/*
>1. message: "JSON
> Parse error: Unrecognized token '?'"
>
>
>
>//Chrome 31 JS console
>JSON.parse('\ufeff
> {"abc": 0} ')
>1. 
>
>
>1. message: "Unexpected
> token "
>Unfortunately,
> I don't have access to IE right now,  but the trend is clear
>
>
>Allen
>
>
>
>
>
>
>
>

-- 
Joe Hildebrand

---

From: Chris Lilley <[email-masked]>
Date: Tue, 19 Nov 2013 08:13:37 +0100

Hello Bjoern,

Monday, November 18, 2013, 2:48:19 PM, you wrote:

> string parsing part of the JSON implementation.

Yes, a byte stream will contain a BOM if one is present.

> RFC 4627 is the only

No, because /* */ are characters and are found in a character stream.
And a character stream does not contain a BOM as a BOM is not
character data.

RFC 4627 doesn't need to say this, because Unicode says it.

If JSON mixes up characters and bytes there will of course be
confusion. But hopefully it doesn't, as this is not 1990 anymore.

-- 
Best regards,
 Chris                            mailto:[email-masked]

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 20 Nov 2013 11:45:35 -0000

----- Original Message From: "Henry S. Thompson"

> I agree that XML is a useful point of comparison, in particular
> because it too does not allow a BOM as part of an XML document, but
> rather treats it as an aspect of packaging/transport external to the
> XML document, which seems to me to be the kind of approach to BOMs the
> JSON WG might consider.

If I remember rightly, XML demotes notes about encoding detection and BOMs 
to a rather lowly (informational?) appendix.  Maybe that's something JSON 
should do in the interests of interoperability (AKA avoidance of confusion - 
which there seems to be a lot of).

Pete Cordell
Codalogic Ltd

---

From: [email-masked] (Henry S. Thompson)
Date: Wed, 20 Nov 2013 12:03:49 +0000

Pete Cordell writes:

> If I remember rightly, XML demotes notes about encoding detection and
> BOMs to a rather lowly (informational?) appendix.  Maybe that's
> something JSON should do in the interests of interoperability (AKA
> avoidance of confusion -

There is indeed a non-normative appendix [1] which gives
implementation advice.  But there is also a normative section [2]
which, _inter alia_ makes a BOM a requirement in certain
circumstances.

ht

[1] http://www.w3.org/TR/REC-xml/#sec-guessing
[2] http://www.w3.org/TR/REC-xml/#charencoding
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 20 Nov 2013 15:18:45 +0900

Hello Henry, others,

On 2013/11/20 3:55, Henry S. Thompson wrote:
> not in IE or Firefox)

In Firefox, I got some garbled characters, in particular some question 
marks for each of the two bytes of the BOM and one question mark for the 
e-acute. Because of the type of the errors, I strongly suspect it is 
related to what we are trying to investigate, and so I don't think this 
can be taken as evidence one way or another.

or feed [2] to www.jsoneditoronline.org (Use
> Open/Url) (works in Chrome, IE, Firefox, ran out of time to test more).

The fact that some libraries or Web sites accept a BOM for JSON isn't a 
proof that all (well, let's say the majority) accept a BOM.

> text which the parser has to accept.

I agree that *from a theoretical viewpoint*, this is correct. But theory 
isn't everything. As I have written before (and you have cited in 
another thread, for another spec):

   What's most important now is to know what receivers actually
   accept. We are not in a design phase, we are just updating the
   definition ... and making sure we fix problems if there are
   problems, but we have to use the installed base for the main
   guidance

For our update from RFC 4627, the null hypothesis is that there are no 
BOMs (neither for UTF-8 nor for UTF-16). The patterns given in 
http://tools.ietf.org/html/rfc4627#section-3 cannot apply to characters, 
they can only apply to bytes. If we want to allow a spec in 
application/json, then we have to have strong evidence that almost all 
parsers can deal with BOMs, not just fragmentary evidence that some 
parsers don't choke on a BOM.

Please note that there's some parallel to XML, in that neither Unicode 
(for the encoding form) nor the IETF (for the 'charset') require a BOM 
for "UTF-16", but XML nevertheless strictly requires it.

Regards,   Martin.

---

From: [email-masked] (Henry S. Thompson)
Date: Wed, 20 Nov 2013 10:29:35 +0000

Martin J. Dürst writes:

> Hello Henry, others,
>
> The fact that some libraries or Web sites accept a BOM for JSON isn't
> a proof that all (well, let's say the majority) accept a BOM.

I wasn't suggesting that it did, rather that the kind of testing that
needed to be done was testing that included the interface between
transport and parser, not parser alone, since, as we are all agreed,
the BOM is not allowed by the language itself.

> I agree that *from a theoretical viewpoint*, this is correct. But
> theory isn't everything. As I have written before (and you have cited
> in another thread, for another spec):
>
> http://tools.ietf.org/html/rfc4627#section-3 cannot apply to
> characters, they can only apply to bytes. If we want to allow a spec
> in application/json, then we have to have strong evidence that almost
> all parsers can deal with BOMs, not just fragmentary evidence that
> some parsers don't choke on a BOM.

I'm not suggesting parsers should allow BOMs, if by parser you mean
what implements the grammar of the language.  And I'm certainly not
suggesting that two examples pulled at random from a Google search
constitute strong evidence about practice at large.  All I was trying
to suggest was that showing that the language parser as such in
Firefox etc.didn't allow BOMs, as the OP had done, was the right kind
of evidence _either_.

ht

---

From: Henri Sivonen <[email-masked]>
Date: Wed, 20 Nov 2013 15:34:24 +0200

> +1 to this table.  It's clear, correct, and implementable.

-1 ಠ_ಠ

As a person who has actually ended up (re)implementing many of the
cases where Firefox sets up the conversion bytes into characters, I
find this kind of format-specific making stuff up in order to enable
the use of BOMless UTF-16 or any sort of UTF-32 reprehensible.

There is no legitimate reason to use UTF-32 for interchange. UTF-32 as
an interchange encoding only serves to increase development and QA
cost and to potentially add security holes (all multibyte decoders
implemented in C or C++ are fine opportunities for security holes).
The W3C or the IETF should not act like UTF-32 was a legitimate
interchanging coding for any purpose for any format. Producers should
be prohibited from emitting UTF-32. Consumers should be prohibited
from supporting UTF-32.

As for UTF-16, there's no legitimate reason* for any new producer to
use it for interchange, but consumers might (depending on format)
need to implement it for compatibility with legacy producers, even
though in retrospect UTF-16 is a bad idea in general and a terrible
idea for interchange. I don't know if JSON is a format where it's
necessary to support the consumption of UTF-16 as an interchange
encoding, but if it is, the way detection happens for in-band
indications should happen in a way consistent with the "decode"
algorithm in the Encoding Standard
(http://encoding.spec.whatwg.org/#decode) for consistency with how the
Web Platform handles text/html, text/plain and text/css (and, shocker,
in practice XML). That is, there should be no special rules about
looking for patterns of zeros. However, the three BOMs—and only three
(UTF-8, big-endian UTF-16, little-endian UTF-16) BOMs—should take
precedence over everything else, including out-of-band metadata.

* gzip takes away the advantage UTF-16 might have over UTF-8 for East
Asian-heavy text.
-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: John Cowan <[email-masked]>
Date: Wed, 20 Nov 2013 17:33:05 -0500

Henri Sivonen scripsit:

> There is no legitimate reason to use UTF-32 for interchange.

[snip]

> use it for interchange, 

This is all very well, but we are not at present in the business of
banning previously permitted forms of JSON.  If you have evidence
that the specific use of these encodings harms JSON interchange,
bring it forward.

-- 
You are a child of the universe no less         John Cowan
than the trees and all other acyclic            http://www.ccil.org/~cowan
graphs; you have a right to be here.            [email-masked]
  --DeXiderata by Sean McGrath

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Wed, 20 Nov 2013 21:53:22 -0800

On Nov 19, 2013, at 10:18 PM, Martin J. Dürst wrote:

> In Firefox, I got some garbled characters, in particular some question marks for each of the two bytes of the BOM and one question mark for the e-acute. Because of the type of the errors, I strongly suspect it is related to what we are trying to investigate, and so I don't think this can be taken as evidence one way or another.
> 
> The fact that some libraries or Web sites accept a BOM for JSON isn't a proof that all (well, let's say the majority) accept a BOM.

Just to be clear about this.  My tests directly tested JavaScript built-in JSON parsers WRT to BOM support in three major browsers.  The tests directly invoked the built-in JSON.parse functions and directly passed to them a source strings that was explicitly constructed to contain a BOM code point .  This was done to ensure that the all transport layers  (and any transcodings they might perform) were bypassed and that we were actually testing the real built-in JSON parse functions.

Neither of the sites referenced about perform a comparable test.  They take user inputed text when is then pass through whose knows what layers of browser and application preprocessing and then they present something derived from that original user input to a JSON parser.  In both bases the actual parser does not appear to be the the built-in JavaScript JSON.parse function that I was testing.

json.view.stack.hu uses Ext.util.JSON.decode whose document describe it as "Modified version of Douglas Crockford"s json.js".   In other words not the built-in JSON.parse function

www.jsoneditoronlineorg uses a library called JSONLint in preference to the built-in JSON.parse function.  It does not conform to the ECMAScript 5 JSON.parse specification.

So testing using either of these sites say nothing relevant to about my observation concern BOM handling by the most widely deployed JSON parsers (the ones that are built into browser JavaScript implementations) 

Allen

---

From: Henri Sivonen <[email-masked]>
Date: Thu, 21 Nov 2013 15:35:43 +0200

On Thu, Nov 21, 2013 at 12:33 AM, John Cowan <[email-masked]> wrote:
> banning previously permitted forms of JSON.

Why not? Surely existing still deployed producers should be what
matters when deciding what needs to be ingested--not previous specs.
That is, compatibility should be considered in terms of what's out
there--not in terms of what unreasonable things were written down in a
previous RFC.

> If you have evidence

UTF-32 harms JSON interchange, because Gecko removed all UTF-32
support throughout the engine (other engines probably did, too, but
I'm too busy to check) and, therefore, XHR responseType = "json"
doesn't support UTF-32.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: Anne van Kesteren <[email-masked]>
Date: Thu, 21 Nov 2013 13:39:09 +0000

> I'm too busy to check) and, therefore, XHR responseType = "json"

XHR's responseType = "json" only supports UTF-8 (optionally with a
leading BOM), across the board.

---

From: Henri Sivonen <[email-masked]>
Date: Thu, 21 Nov 2013 15:28:43 +0200

On Thu, Nov 21, 2013 at 7:53 AM, Allen Wirfs-Brock
<[email-masked]> wrote:
> Just to be clear about this.  My tests directly tested JavaScript built-in
> JSON parsers WRT to BOM support in three major browsers.  The tests directly
> invoked the built-in JSON.parse functions and directly passed to them a
> source strings that was explicitly constructed to contain a BOM code point .
> This was done to ensure that the all transport layers  (and any transcodings
> they might perform) were bypassed and that we were actually testing the real
> built-in JSON parse functions.

It would be surprising if JSON.parse() accepted a BOM, since it
doesn't take bytes as input.

However, XHR's responseType = "json" exercises browsers in a way where
the input is bytes from the network. From the perspective of JSON
support in XHR,
http://lists.w3.org/Archives/Public/www-tag/2013Nov/0149.html (which
didn't reach the es-discuss part of this thread previously) applies.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: John Cowan <[email-masked]>
Date: Thu, 21 Nov 2013 11:56:15 -0500

> there--not in terms of what unreasonable things were written down in a

In principle, maybe.  But testing a dozen browsers isn't enough here.
We simply don't know how much non-browser traffic involves JSON (though
we know there are many such interactions), or what representations they
are using.  We have therefore decided in the 4627bis effort not to say
that anything that was previously valid is now invalid.  At least, that
is what I understand us to be doing.

> I'm too busy to check) and, therefore, XHR responseType = "json"

That has about as much weight as "XYZ implementation only supports
ASCII, so the use of non-ASCII characters harms JSON interchange" or "ABC
implementation only supports 32-bit integers, so the use of decimal points
harms JSON interchange."  An implementation's self-imposed limitations
don't affect the standard.

-- 
John Cowan            http://www.ccil.org/~cowan     [email-masked]
                if if = then then then = else else else = if;

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 21 Nov 2013 17:11:43 +0000

On 11/21/13 5:56 PM, "John Cowan" <[email-masked]> wrote:

>We have therefore decided in the 4627bis effort not to say

Specifically, the charter (http://datatracker.ietf.org/wg/json/charter/)
says:

"Any changes that break compatibility with existing implementations of
either RFC 4627 or the ECMAScript specification will need to have very
strong justification and broad support."

-- 
Joe Hildebrand

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 21 Nov 2013 09:01:01 -0800

> doesn't take bytes as input.

ECMAScript's JSON.parse accepts an ECMAScript string value as its input.  ECMAScript strings are sequences of 16-bit values.  JSON.parse (and most other ECMAScript functions) interpret those values  as Unicode code units.  The value U+FEFF can appear at any position within a string. When defining a string as an ECMAScript literal, a sequence like \ufeff is an escape sequence that means place the code unit value 0xefff into the string at this position in the sequence. Also note that the actual strings passed below to JSON.parse contain the actual code point value U+FEFF not the escape sequence that was used to express it.  To include the actual escape sequence characters in the string it would have to be expressed as '\\feff'.

JSON.parse('\ufeff ["XYZ"]');  //note outer quotes delimit an ECMAScript string, the inner quotes are a JSON string.  

throws a runtime SyntaxError exception because the JSON grammar does not allow U+FEFF to appear that position

JSON.parse('["\ufeffXYZ"]');

operates without error and returns a Array containing a four element ECMAScript string.   This works because the JSON grammar allows any code unit except for " and \ and the ASCII control characters to appear literally in a JSON string. 

> didn't reach the es-discuss part of this thread previously) applies.

Right, JSON use via XHR is a different usage scenario and that probably involves encoding and decoding steps. It has very little to do with the JSON syntax, as defined in ECMA-404. It's all about how the bits that represent a string are interchanged, not the eventual semantic processing of the string (ie, processing by JSON.parse or some other JSON parser)

Allen

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 21 Nov 2013 18:41:08 +0100

>ECMAScript's JSON.parse accepts an ECMAScript string value as its input.
>ECMAScript strings are sequences of 16-bit values.  JSON.parse (and most
>other ECMAScript functions) interpret those values  as Unicode code 
>units.  The value U+FEFF can appear at any position within a string. 
>When defining a string as an ECMAScript literal, a sequence like \ufeff 
>is an escape sequence that means place the code unit value 0xefff into 
>the string at this position in the sequence. Also note that the actual 
>strings passed below to JSON.parse contain the actual code point value 
>U+FEFF not the escape sequence that was used to express it.  To include 
>the actual escape sequence characters in the string it would have to be 
>expressed as '\\feff'.

A byte order mark indicates the order of bytes in a sequence of bytes.
An ecmascript string is not a sequence of bytes and therefore it cannot
have a byte order mark inside it. Your test is not for BOM support but
for an egregious semantic error in the implementation of JSON.parse.

  http://shadowregistry.org/js/misc/#t2ea25a961255bb1202da9497a1942e09

That is a similar test. It makes Firefox see UTF-8 BOMs in ecmascript
strings. Firefox is not supposed to look for UTF-8 BOMs in ecmascript
strings because ecmascript strings are not sequences of bytes at that
level of reasoning.

Is there any chance, by the way, to change `JSON.stringify` so it does
not output strings that cannot be encoded using UTF-8? Specifically,

  JSON.stringify(JSON.parse("\"\uD800\""))

would need to escape the surrogate instead of emitting it literally.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mathias Bynens <[email-masked]>
Date: Thu, 21 Nov 2013 10:52:47 -0800

> would need to escape the surrogate instead of emitting it literally.

Previous discussion: http://esdiscuss.org/topic/code-points-vs-unicode-scalar-values#content-14

---

From: John Cowan <[email-masked]>
Date: Thu, 21 Nov 2013 14:15:33 -0500

> would need to escape the surrogate instead of emitting it literally.

No, there isn't.  We've been down this road repeatedly.  People can and
do use JSON strings to encode arbitrary sequences of unsigned 16-bit integers.

-- 
You let them out again, Old Man Willow!                 John Cowan
What you be a-thinking of?  You should not be waking!   [email-masked]
Eat earth!  Dig deep!  Drink water!  Go to sleep!
Bombadil is talking.                                    http://ccil.org/~cowan

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 21 Nov 2013 20:37:49 +0100

>do use JSON strings to encode arbitrary sequences of unsigned 16-bit integers.

The output of JSON.stringify("\uD800") contains no backslash character,
if you call `utf8_encode(JSON.stringify("\uD800"))` you get an exception
because UTF-8 cannot encode the lone surrogate and `utf8_encode` does
not know it could encode it as `\uD800` without loss of information. If
`JSON.stringify` produced an escape sequence instead, there would be no
problem passing the output to `utf8_encode`.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Nov 2013 13:58:13 -0600

> problem passing the output to `utf8_encode`.

That's just one implementation.  We had hundreds of e-mails in this
list about this.  Well over a thousand to cover several issues like
this.  I think the only area where we have [roughly] consensus to
revisit the previous consensus is the top-level value restriction,
which has led to the whole UTF and byte-order detection sub-thread
(which we had, also, had before).  We're on much stronger ground to
revisit this one matter than the whole unpaired surrogates matter, and
we're much much less likely to change our consensus on that because
one proposal is about relaxing JSON to match ECMAScript's definition,
while yours is to do the opposite.

Nico
--

---

From: Henri Sivonen <[email-masked]>
Date: Fri, 22 Nov 2013 11:22:35 +0200

> leading BOM), across the board.

Good point. I wrote the code that enforces that constraint, but I forgot.

Well, there's an interoperability reason against UTF-16, too, then.

> is what I understand us to be doing.

Even if no one or approximately no one (outside test cases) actually
emits JSON in UTF-32?

> don't affect the standard.

Well, what (broadly deployed) running code does affects
interoperability regardless of who imposed the behavior. (In this
case, the behavior is imposed by a spec: the XHR spec requires that
JSON be always treated as UTF-8.)

> strong justification and broad support."

"existing implementations" ≠ "existing specs"

I think you should have to show an existing implementation with
substantial deployment that in its substantially deployed
configuration emits JSON in UTF-32 to have a justification for keeping
UTF-32 in the spec.

(I have to wonder what kind of theorizing was the cause of putting
UTF-32 in the spec in the first place. I also have to wonder if the
IETF JSON spec would have supported UTF-64 for completeness if someone
had written an April 1st RFC for UTF-64.)

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: "Pete Cordell" <[email-masked]>
Date: Fri, 22 Nov 2013 11:33:14 -0000

----- Original Message From: "Henri Sivonen"
> Well, there's an interoperability reason against UTF-16, too, then.

Personally I think we have to be careful not to fall into the trap of 
assuming that the only use-case for JSON will be in "to browser" 
communications.  I'm hoping that for the IETFs purposes we'll be looking at 
JSONs wider utility into broader areas, which may even include logging to 
files and interprocess communication where there might be sensible reasons 
to choose something other than UTF-8.

Pete Cordell
Codalogic Ltd

---

From: Julian Reschke <[email-masked]>
Date: Fri, 22 Nov 2013 13:26:28 +0100

On 2013-11-22 12:47, Robin Berjon wrote:
> On 22/11/2013 12:33 , Pete Cordell wrote:
>> communications.  I'm hoping that for the IETFs purposes we'll be looking
>> at JSONs wider utility into broader areas, which may even include
>> logging to files and interprocess communication where there might be
>> sensible reasons to choose something other than UTF-8.
>
> Sure, but given the prominence of JSON communicated to the browser, it
> would be a pretty bad idea to have JSON variants that don't interoperate
> there.

For some value of "interoperate". Native JSON support in XHR is a 
relatively new feature (right?), and JSON has been used long before in 
the browser.

Best regards, Julian

---

From: Robin Berjon <[email-masked]>
Date: Fri, 22 Nov 2013 12:47:39 +0100

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

---

From: John Cowan <[email-masked]>
Date: Fri, 22 Nov 2013 11:39:24 -0500

> Even if no one or approximately no one (outside test cases) actually

How on earth would you know that?

> configuration emits JSON in UTF-32 to have a justification for keeping

As things now stand, there is zero support for removing anything from
4627bis.  If you want to argue for an interoperability warning like
the ones we already have, go ahead.  Given the evidence you've shown,
that's probably a good idea.  But the watchword of 4627bis (as opposed
to future I-JSON) is "No breaking changes.  Anywhere.  Ever."

> had written an April 1st RFC for UTF-64.)

You'll have to ask Mr. Crockford.

-- 
Real FORTRAN programmers can program FORTRAN    John Cowan
in any language.  --Ed Post                     [email-masked]

---

From: Tim Bray <[email-masked]>
Date: Fri, 22 Nov 2013 08:39:49 -0800

I’ve been using JSON for quite a few years, but hardly ever in either a
to-browser or from-browser role; what I care about is mostly its use in
RESTful APIs generally and identity APIs specifically.  In those scenarios,
it would be seen as wildly inappropriate to use anything but UTF-8; I’ve
never actually seen anything else.  In practice, it would be very unlikely
for anyone to deploy UTF-16 or any other non-UTF-8 flavor in a non-browser
scenario.

Having said that, I’m still, hundreds of messages later, not 100% sure what
our draft should say about BOMs :(

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Fri, 22 Nov 2013 08:54:46 -0800

On Nov 22, 2013, at 8:39 AM, Tim Bray wrote:

> I’ve been using JSON for quite a few years, but hardly ever in either a to-browser or from-browser role; what I care about is mostly its use in RESTful APIs generally and identity APIs specifically.  In those scenarios, it would be seen as wildly inappropriate to use anything but UTF-8; I’ve never actually seen anything else.  In practice, it would be very unlikely for anyone to deploy UTF-16 or any other non-UTF-8 flavor in a non-browser scenario.
> 
> Having said that, I’m still, hundreds of messages later, not 100% sure what our draft should say about BOMs :(

You should say it that it is not an actual issue of the JSON format whose grammar clearly defines the handling of the 0xfeff code point.  Rather it is an upstream data interchange issue that should be dealt with in exactly the same way as with any other data interchange on a similar channel.  Say whatever you think is appropriate about BOMs in the transmission of data conforming to the "application/json" MIME type.  Just be clear that whatever you decide has nothing to do with the abstract, grammar-based interpretation of the actual JSON payload.

Allen

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 22 Nov 2013 18:21:21 +0000

There appears to be consensus to change JSON-text to allow for any JSON value -- not just object / array -- while noting that object or array as the top-level is the most interoperable.

We will ask the Document Editor to make this change to draft-ietf-json-rfc4627bis.

- Paul Hoffman and Matt Miller

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 22 Nov 2013 18:29:01 +0000

There does not appear to be any consensus on explicitly allowing or disallowing of a Byte Order Mark (BOM).  Neither RFC4627 nor the current draft mention BOM anywhere, and the modus operandi of the JSON Working Group has been to leave text unchanged unless there was wide support.

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 22 Nov 2013 18:33:28 +0000

There does seem to be rough consensus that using an encoding other than UTF-8 can have interoperability issues.  The also seems to be rough consensus that the current text and table in section 8.1 for detecting the encoding will be inaccurate (and potentially harmful).

That appears to mean the approach with the most consensus is to remove the encoding detection entirely, leaving only:

""""
   JSON text SHALL be encoded in Unicode.  The default encoding is
   UTF-8.
""""

---

From: Julian Reschke <[email-masked]>
Date: Fri, 22 Nov 2013 19:41:35 +0100

> - Paul Hoffman and Matt Miller

Again:

   "JSON text SHALL be encoded in Unicode."

doesn't make any sense. "Unicode" is not a character encoding.

Can we please use proper terminology here?

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 22 Nov 2013 19:43:05 +0100

* Matt Miller (mamille2) wrote:
>There does not appear to be any consensus on explicitly allowing or 
>disallowing of a Byte Order Mark (BOM).  Neither RFC4627 nor the current 
>draft mention BOM anywhere, and the modus operandi of the JSON Working 
>Group has been to leave text unchanged unless there was wide support.

To be clear, that means application/json entities that start with a byte
sequence that matches U+FEFF encoded in UTF-8/16/32 is malformed because
the ABNF does not allow a U+FEFF at that position (and interpreting such
a sequence as anything other than ordinary character data requires
explicit specification). I do think an informational note saying as much
could be useful.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 22 Nov 2013 19:52:26 +0100

* Matt Miller (mamille2) wrote:
>There does seem to be rough consensus that using an encoding other than 
>UTF-8 can have interoperability issues.  The also seems to be rough 
>consensus that the current text and table in section 8.1 for detecting 
>the encoding will be inaccurate (and potentially harmful).
>
>That appears to mean the approach with the most consensus is to remove
>the encoding detection entirely, leaving only:
>
>""""

Neither of the quoted statements mean anything as far as I can tell.
The encoding detection rules are a vital part of the specification and
cannot be removed without replacement. I am not aware of any argument
that the text "will be" inaccurate or harmful.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: John Cowan <[email-masked]>
Date: Fri, 22 Nov 2013 13:57:59 -0500

Matt Miller (mamille2) scripsit:

> There does seem to be rough consensus that using an encoding other
> than UTF-8 can have interoperability issues.

I agree.

> The also seems to be rough consensus that the current text and table
> in section 8.1 for detecting the encoding will be inaccurate (and
> potentially harmful).

If it is inaccurate, it should be fixed, but as it is a statement of
fact and not normative, I can see no grounds for removing it altogether.

>    JSON text SHALL be encoded in Unicode.  The default encoding is

I do not believe this is strong enough.  I think an interoperability
problems warning similar to those we already have would be better.
Proposed text:

   The use of other encodings is known to cause interoperability problems.

-- 
One Word to write them all,             John Cowan <[email-masked]>
  One Access to find them,              http://www.ccil.org/~cowan
One Excel to count them all,
  And thus to Windows bind them.                --Mike Champion

---

From: John Cowan <[email-masked]>
Date: Fri, 22 Nov 2013 13:58:52 -0500

> that the text "will be" inaccurate or harmful.

If we are going to allow interchange of bare strings, the text will be
inaccurate.

-- 
And they pack their lyrics till they're so damn dense           John Cowan
You could put 'em in your yard and you could use 'em for a fence.
  --Alan Chapman, "Everybody Wants to Be Sondheim"     www.ccil.org/~cowan

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 22 Nov 2013 19:05:57 +0000

> Can we please use proper terminology here?

This was merely quoting the existing text in section 8.1, minus the encoding detection text.

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: "Pete Cordell" <[email-masked]>
Date: Fri, 22 Nov 2013 19:28:14 -0000

----- Original Message From: "Matt Miller (mamille2)"

> There does seem to be rough consensus that using an encoding
> other than UTF-8 can have interoperability issues.  The also
> seems to be rough consensus that the current text and table
> in section 8.1 for detecting the encoding will be inaccurate
> (and potentially harmful).
>
> That appears to mean the approach with the most consensus is
> to remove the encoding detection entirely, leaving only:
>
> """"

I think we can be a little more helpful here.  For example, something along 
the lines of:

    JSON text is a sequence of Unicode codepoints.  The transfer encoding 
used to
    represent those characters on-the-wire is beyond the scope of this
    document.  It is therefore up to the specifications that reference this 
document to
    specify whether JSON messages will be transferred using UTF-8 
(recommended),
    UTF-16 and/or UTF-32 (discouraged), and whether preceding BOMs must be
    present, must not be present or are optional.

    If multiple encodings are permitted, implementers may choose to 
auto-detect a
    message's encoding by exploiting the fact that the first character of a 
JSON text
    must be in the ASCII character range and use the following table to 
deduce the
    active encoding:

       00 00 -- --  UTF-32BE
       00 xx -- --  UTF-16BE
       xx 00 00 00  UTF-32LE
       xx 00 00 xx  UTF-16LE
       xx 00 xx --  UTF-16LE
       xx xx -- --  UTF-8

Pete Cordell
Codalogic Ltd

---

From: "Pete Cordell" <[email-masked]>
Date: Fri, 22 Nov 2013 19:32:46 -0000

Further to my earlier comment, I also wondered about taking a leaf out of 
cipher suites and allow specifications that use JSON to encode their 
encoding requirements along the lines of:

    JSON-8OB-16MB-32NB

where OB = Optional BOM, MB = Mandatory BOM and NB = No BOM.  So the above 
would mean UTF-8 is supported with or without BOMs, UTF-16 is supported, but 
must have a BOM and UTF-32 is supported with NO BOM.

Another example would be:

    JSON-8OB

i.e. UTF-16 and UTF-32 are not supported.

Maybe that's going too far though!

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: "Pete Cordell" <[email-masked]>
To: "Matt Miller (mamille2)" <[email-masked]>; "JSON WG" <[email-masked]>
Cc: <[email-masked]>; "es-discuss" <[email-masked]>
Sent: Friday, November 22, 2013 7:28 PM
Subject: Re: [Json] Encoding detection (Was: Re: JSON: removegapbetween 
Ecma-404 and IETF draft)

> I think we can be a little more helpful here.  For example, something 
> along the lines of:
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 25 Nov 2013 18:37:43 +0900

> You should say it that it is not an actual issue of the JSON format whose grammar clearly defines the handling of the 0xfeff code point.  Rather it is an upstream data interchange issue that should…

That works for ECMA-404. It does not work for the IETF draft, because it 
is extremely relevant for application/json, which is part of that draft.

Regards,    Martin.

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Mon, 25 Nov 2013 16:29:12 -0800

On Nov 25, 2013, at 1:37 AM, Martin J. Dürst wrote:

> That works for ECMA-404. It does not work for the IETF draft, because it is extremely relevant for application/json, which is part of that draft.
> 
> Regards,    Martin.

It still seems pretty clear.  Anything feed as input into an actual parser for the JSON grammar as standardized by ECMA-404 must not contain any U+feff code points other than as part of JSON string values.  If the application/json wire format chooses to to use BOMs, then they must be removed before processing by a standard JSON parser.   From a JSON parser prospect, I think that's pretty much all you need to say. 

allen

---

From: Henri Sivonen <[email-masked]>
Date: Tue, 26 Nov 2013 16:10:14 +0200

On Fri, Nov 22, 2013 at 1:33 PM, Pete Cordell <[email-masked]> wrote:
> communications.

I don't expect it to be the only use.

> I'm hoping that for the IETFs purposes we'll be looking at

What sensible reasons could there possibly be?

The one reason for using UTF-16 is contrived. (Your JSON consists
almost entirely of East Asian string literals with next to no JSON
syntax itself, you are bandwidth-constrained and, magically,
simultaneously so CPU-constrained that you can't use gzip.) For UTF-32
not even contrived reasons exist.

(If you use shared-memory IPC between processes that use non-UTF-8 for
representing Unicode strings, you shouldn't treat the exchange as
happening via char* plus the encoding layer and the JSON MIME type but
using char16_t* or char32_t* without the encoding layer involved. For
example, if you JSONify data to communicate from Web Workers to the
main thread, conceptually, the JSONification happens to Unicode
strings--not to bytes, so the JSON RFC doesn't get involved.)

> How on earth would you know that?

There exists no situation where using UTF-32 for interchange makes
sense. I think proponents of craziness of the level of using UTF-32
for interchange should show evidence of existing crazy deployments
instead of asking future implementers to support UTF-32 just because
it wasn't possible to prove non-existence.

>       xx xx -- --  UTF-8

I continue to strongly disapprove of non-BOM-based sniffing rules
unless there's compelling evidence that such rules are needed in order
to interoperate with bogus existing serializers.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Nov 2013 09:27:05 -0600

> What sensible reasons could there possibly be?

I can't think of any either.  UTF-32 is superficially appealing (O(1)
indexing!) but it's only O(1) indexing by codepoint counts, not
character counts so it's still lame and you pay for longer strings.
It's possible that on some architectures / for some use cases it
performs fabulously better than the alternatives, and though I doubt it,
that would be a reason not to *forbid* the use of UTF-32.  What we
clearly don't have consensus for is requiring support of UTF-32.

> it wasn't possible to prove non-existence.

No, I think this is too much.  If someone wants to use UTF-32 because
they have numbers showing that for IPC and local processing it's faster,
that might be compelling; let them.

Anyways, I think we're focusing too hard on details that aren't terribly
important.  The "non-BOM-based sniffing rules" work and can be derived
by any capable implementor whether stated or not by the RFC.

> to interoperate with bogus existing serializers.

I think it's fair to object to requiring sniffing, and I support not
requiring it.  I don't see anything wrong with leaving those in for
those who want to include support for it.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Tue, 26 Nov 2013 11:01:27 -0500

> What sensible reasons could there possibly be?

The fact that you (or even I) can't think of them doesn't mean they
don't exist.  There would be no sensible reason for me to write to you in
Finnish: your command of written English is near-native, and my command
of Finnish is zero.  But it would be absurd of me to say that people
should not communicate in Finnish because it harms interoperability.
It so happens that I know that there are five million people cheerfully
writing to each other in Finnish, under the impression that it is allowed.
But even if I didn't happen to know that, the point would be the same.

Now Finnish is a natural language, and JSON is not: it exists only by
virtue of its definition.  But that definition explains how to communicate
in JSON, and anyone who adheres to it is communicating correctly.  For us
to chop their feet out from under them by saying that what they are doing
does not count as JSON would be just as arbitrary as banning Finnish
because almost nobody speaks it.  We can say that we think it's a bad idea
to use non-UTF-8 encodings in JSON, and that's as far as we can justly go.

---

From: Jacob Davies <[email-masked]>
Date: Tue, 26 Nov 2013 09:18:48 -0800

The Finns can be verified to exist (or at least are a very convincing
hoax), but I think there is a real question whether anyone actually
uses anything other than UTF-8 with JSON. If nobody does, then support
for UTF-16 and UTF-32 autodetection is pointless and at least
potentially dangerous or buggy in the way that such features tend to
be.

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Nov 2013 11:24:14 -0600

> potentially dangerous or buggy in the way that such features tend to

There are human cultures with which we don't have enough contact to
learn their languages.  This is silly.

We must not forbid the use of any other encodings than UTF-8.  We must
encourage the use of UTF-8 for interop by, for example, stating that
only UTF-8 is known to interop well.

We must not require encoding detection functionality in parsers.  We
must not forbid it either.  We might need to say that encodings other
than UTF-8/16/32 may not be reliably detected, therefore they are highly
discouraged, even forbidden except where protocols specifically call for
them.

But really, we can't forbid the use of UTF-16/32, we can only require
parser support for UTF-8.

Nico
--

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 26 Nov 2013 21:15:38 +0100

>discouraged, even forbidden except where protocols specifically call for

When I pass a fully conforming UTF-8 encoded application/json entity to
a fully conforming JSON parser I do not want the parser to do something
funny like interpreting the document as if it were Windows-1252 encoded.
I am amazed how many people here think a parser that does that should
not be considered broken.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Nov 2013 21:22:28 +0100

> not be considered broken.

+1

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Nov 2013 16:00:41 -0600

> not be considered broken.

You missed the point.  I'm outlining what we can and should do.  We
should strongly encourage UTF-8 (require it, even, for parsers).  We
should not forbid other encodings -- at least not UTF-16 nor UTF-32 --
though we might agree to say nothing about them.

As to non-UTF encodings, well, think of something like the Korn Shell,
with it's... very strange "compound variables", and consider something
more like the Windows Power Shell.

It might be awesome to have a Unix shell that uses JSON as a [far]
superior alternative to the Korn Shell's compound variable disaster.
But you see, if you have any non-Unicode locales, how would such a shell
encode its JSON values?  Obviously: not in any UTF (except, maybe,
UTF-7).  It'd not be hard for such a shell to handle non-Unicode locales
just fine.  Not that such a shell's JSON parser should auto-detect
encodings (no way), but you know well enough that there's text documents
lying around in all sorts of encodings without the encoding metadata
being recorded anywhere.

If you wanted to forbid non-Unicode, non-UTF encodings, then you'd be
preventing such a shell, and for what reason?  If you only mean that
auto-detection of encoding should not even be mentioned, I'm fine with
that, and I've already said so earlier.

(Of course I'd love to see non-Unicode locales disappear, but I don't
think that's in the cards.  And yes, I had a Unix Power-like shell in
mind when I wrote the text you quoted.)

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Nov 2013 00:00:33 +0100

On 26 Nov 2013, at 23:00, Nico Williams <[email-masked]> wrote:

> encode its JSON values?  

In UTF-8?

> Obviously: not in any UTF (except, maybe,
> UTF-7).  

I don’t see the obviousness here.
The locale is irrelevant for the encoding of JSON values.

> preventing such a shell, 

This is a very strange argument.

If your application needs a JSON-like interchange format that is encoded in Windows-1252, just go ahead and create that, it’s not hard.  Just don’t call it JSON.  (I think KSON is still free except as a radio station call sign.:-)  Nothing here is prevented by keeping JSON useful as an interchange format.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Nov 2013 17:07:34 -0600

On Wed, Nov 27, 2013 at 12:00:33AM +0100, Carsten Bormann wrote:
> [...]

Did you actually disagree with any of my recommendations?  Please point
out which.  I'll repeat them here to make it easy:

    We should strongly encourage UTF-8 (require it, even, for parsers).
    We should not forbid other encodings -- at least not UTF-16 nor
    UTF-32 -- though we might agree to say nothing about them.

Do you want to say anything about other encodings?  What would that be?
And why?

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Nov 2013 00:20:25 +0100

> Do you want to say anything about other encodings?  What would that be?

JSON is encoded in UTF-8.

There is no need to discuss JSON in other encodings, because it wouldn’t be JSON.

(And no, I see no need to handle UTF-16LE, UTF-16BE, UTF-32LE or UTF-32BE in any special way, even if RFC 4627 was written at a time when it still seemed useful to pay them lip service.  But I recognize that there appears to be WG consensus to keep these corpses on life support, maybe because UTF-16 is the internal encoding of the programming language that gave JSON its name.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Nov 2013 18:11:31 -0600

> There is no need to discuss JSON in other encodings, because it

Thanks.

My opinion as to MIME contexts:

    I'm not opposed to saying that the application/json media type
    requires UTF-8.  Others have objected, and I believe the WG
    consensus to be that the application/json media type allows all of
    UTF-8/16/32.

    I believe we should settle for an interop note noting that UTF-8 has
    the best interoperability, and a recommendation that UTF-8 be used.

My opinion as to non-MIME contexts:

    I'm not opposed to recommending that JSON texts for interchange in
    non-MIME contexts be encoded in UTF-8, and I'm not opposed to
    requiring that use of any other encoding be expressed as metadata.

    I do object to requiring that under all circumstances -even in
    non-MIME contexts- UTF-8 must be used.

> (And no, I see no need to handle UTF-16LE, UTF-16BE, UTF-32LE or
> UTF-32BE in any special way, even if RFC 4627 was written at a time
> when it still seemed useful to pay them lip service.  But I recognize
> that there appears to be WG consensus to keep these corpses on life
> support, maybe because UTF-16 is the internal encoding of the
> programming language that gave JSON its name.)

Right, that appears to be the consensus, and more than that, it seems
extremely unlikely to change.

Assuming *that*, what are you willing to settle for?

Nico

PS: Back to my hypo...

    If my hypothetical JSON-using shell were to escape all non-ASCII
    characters in JSON string values, then encode the JSON text in
    UTF-8, then convert the result to the current locale's codeset
    (doing the reverse to parse), and the resulting texts either never
    leak to other locales, why should anyone care?

    Most (but not all) non-Unicode locales use ASCII-compatible codesets,
    thus the result would be "proper" JSON texts in most cases anyways...

    As to why one might want to do that: because JSON texts are...
    *text*, i.e., editable in your favorite $EDITOR, readable with your
    favorite $PAGER, and so on.  It might be a problem if such texts
    leaked outside that locale, but we already have that problem in
    spades, and no JSON parser would be called upon to try to
    auto-detect any encodings other than UTF-8/16/32.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 27 Nov 2013 01:14:00 +0100

* Nico Williams wrote:
>You missed the point.

"We must require encoding detection functionality in parsers. We must
forbid encoding detection functionality beyond that. We must say that
encodings other than UTF-8/16/32 are forbidden in any and all cases."
is how I would modify what you said above (with some caveats).

Note that I am talking about labeled sequences of octets, application/
json entities, not paintings on a cave wall that look similar to JSON
text in a strange font. In a labeled sequence of octets I can tell for
sure whether there are invisible characters in it if I know the en-
coding.

There are two forms to consider. One is the labeled sequence of octets
that we call "application/json entity". The other is a sequence of Uni-
code scalar values. That is the alphabet of the ABNF grammar in the
specification. If you have anything else, then the specification does
not apply to your situation.

>that, and I've already said so earlier.

Above I said that there are two forms to consider. Encoding detection
is what allows us to convert the "application/json entity" form into
the "sequence of Unicode scalar values" form. We need the latter form
in order to apply the ABNF grammar. Imagine you receive this:

  HTTP/1.1 200 OK
  Content-Type: application/json
  ...

  ABCD...

There would be at least two specifications that apply here, the HTTP
and the application/json specification. Would you like them to say
that you are on your own, "ABCD..." could mean anything? I would like
them to say "ABCD..." is an array with three times the integer zero,
like `[0,0,0]`. I can build robust software based on that.

I cannot build robust software based on "well, maybe it's EBCDIC?
Have you tried GB 18030? UTF-7 might be worth a try otherwise. Are
you sure this matters at all?"
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Nov 2013 18:39:49 -0600

On Wed, Nov 27, 2013 at 01:14:00AM +0100, Bjoern Hoehrmann wrote:
> that you are on your own, "ABCD..." could mean anything? [...]

Absolutely not, and I didn't say that either.  The application/json
media type absolutely has to be in a UTF, and preferably UTF-8.

> you sure this matters at all?"

That is not even remotely what I want.  Either you misunderstood me or I
wasn't specific or clear enough.  Either way, I think it should be clear
as of my last message (though you might find things there to disagree
with).

Nico
--

---

From: Henri Sivonen <[email-masked]>
Date: Wed, 27 Nov 2013 15:14:11 +0200

> character counts so it's still lame and you pay for longer strings.

It doesn't matter for the interchange encoding, which is a different
issue from in-RAM string representation (where indeed O(1) indexing by
code point is overrated but appeals to people who have heard about
astral characters but haven't yet considered considered that due to
combining characters, they can't treat treat code units independently
of the adjacent code units anyway).

> that might be compelling; let them.

If someone wants to use UTF-32  for local shared-memory IPC, the
communicating parties are tightly-coupled  and don't need a standard
to authorize whatever they are doing. Standards are for
loosely-coupled communication where there aren't bilateral
arrangements between the communicating parties. (Also, UTF-32  doesn't
make sense even for local IPC  if it happens over a socket rather than
over shared memory.)

> by any capable implementor whether stated or not by the RFC.

Has anyone tested  whether a substantial proportion of the existing
implementations already support the non-BOM-based sniffing rules? That
is,  can the rules be relied on with existing implementations?

> those who want to include support for it.

Optional features are a trap. They are used to get the appearance of
consensus when people who are oppose to misfeatures are told that they
don't need to implement them. However,  when some implementations
start emitting syntax that exercises optional features (or someone
writes test cases just to smugly point out the lack of support for
optional features), everyone ends up having to implement optional
features in order to be compatible.

On Tue, Nov 26, 2013 at 6:01 PM, John Cowan <[email-masked]> wrote:
> don't exist.

You and I have a pretty good idea of Unicode stuff. If we can't think
of sensible reasons and others in the discussion haven't seen UTF-32
(or UTF-16) JSON in the wild, either, we should have enough confidence
to say that UTF-8 is enough and not suggest that future implementors
waste implementation and QA effort on UTFs that don't make sense for
interchange.

The notion that in theory, maybe, someone might use a non-UTF-8
encoding for something, even if there's no data to support such
conjecture, is the source of a lot of harmful inertia around character
encodings.

> There would be no sensible reason for me to write to you in

This is the bad analogy for several reasons:

 * Banning a particular layout of bits among equally expressive
alternatives is not the same thing as banning someone's native natural
language.

 * Finnish has uses other than test cases for the sake of test cases
or XSS exploits.

 * Communication in Finnish can actually be shown to happen in practice.

 * It would be unreasonable to suggest that everyone ought to support
the receipt of Finnish communication in addition to supporting the
receipt of English communication.

 * Accepting your analogy leads not only to supporting UTF-32 but also
to supporting various non-UTF encodings, which would be even worse
than suggesting that UTF-16 and UTF-32 be supported in addition to
UTF-8.

> for anyone to deploy UTF-16 or any other non-UTF-8 flavor in a non-browser

Right.

> our draft should say about BOMs :(

When not considering JSON specifically, it is unusual to use BOMless
UTF-16 (unusual even if the use of UTF-16 is assumed; it is itself
unusual) and in the context of other textual formats, BOMless UTF-16
is a pain. It seems to me if support for UTF-16 (or UTF-32) is
retained, it would be good to test a bunch of existing implementations
(that section 11 of the draft mentions) to see how they behave with
BOMful and BOMless input.

Of course, such testing takes work, so banning UTF-16 and UTF-32 would
avoid such work. :-)

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: John Cowan <[email-masked]>
Date: Wed, 27 Nov 2013 12:22:51 -0500

Henri Sivonen scripsit:

> Optional features are a trap. 

No doubt.  But what's stated in the RFC is given as a matter of fact,
with no MUST, SHOULD, or even MAY.  Since we have adopted a change
which makes it no longer factual, we should IMO revise it so it is
again factual, and leave it at that.

> encoding for something, 

Not "might use", but "may already be using".

> than suggesting that UTF-16 and UTF-32 be supported in addition to

They are already banned by the SHALL in section 3.

> Of course, such testing takes work, so banning UTF-16 and UTF-32 would

Banning unpaired surrogates, non-integers, etc. etc. would have saved
us work too.

-- 
John Cowan  <[email-masked]>  http://ccil.org/~cowan
Micropayment advocates mistakenly believe that efficient allocation of
resources is the purpose of markets.  Efficiency is a byproduct of market
systems, not their goal.  The reasons markets work are not because users
have embraced efficiency but because markets are the best place to allow
users to maximize their preferences, and very often their preferences are
not for conservation of cheap resources.  --Clay Shirky

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 Nov 2013 16:13:40 -0800

To do this, I think the draft requires these changes:

- Remove the trailing section of section 1.2, starting with “ECMAscript 5.1
enumerates...” [because the difference no longer exists]

- In section 2:

-- remove “A JSON text is a serialized object or array.”

-- Insert: “A JSON text is a serialized value.  Note that certain previous
specifications of JSON constrained a JSON text to be an object or an array.
 Implementations which generate only objects or arrays where a JSON text is
called for will be interoperable in the sense that all implementations will
accept these as conforming JSON texts.”

-- Change the JSON-text production to read:

JSON-text  = value

On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <[email-masked]
> wrote:

> There appears to be consensus to change JSON-text to allow for any JSON
> value -- not just object / array -- while noting that object or array as
> the top-level is the most interoperable.
>

---

From: Alex Russell <[email-masked]>
Date: Wed, 27 Nov 2013 17:00:30 -0800

Will you also be citing ECMA-404 normatively to avoid this sort of
divergence in the future?

On Wed, Nov 27, 2013 at 4:13 PM, Tim Bray <[email-masked]> wrote:

> - Remove the trailing section of section 1.2, starting with “ECMAscript
> 5.1 enumerates...” [because the difference no longer exists]
>
> - In section 2:
>
> On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <
> [email-masked]> wrote:
>

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 27 Nov 2013 19:29:18 -0800

<no hat>

On Nov 27, 2013, at 5:00 PM, Alex Russell <[email-masked]> wrote:

> Will you also be citing ECMA-404 normatively to avoid this sort of divergence in the future?

If you believe that ECMA-404 will change in the future, that would indicate that ECMA might break interoperability with current implementations, even for what they perceive as "good reasons". In general, the IETF tries not to have its long-lived standards normatively latch on to moving targets for this very reason. Even when other SDOs have assured us that they will not make backwards-incompatible changes, they have done so anyway (cue the Klensin-esqe theme music...), and that has caused serious interoperability problems for the IETF specs.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Wed, 27 Nov 2013 22:49:41 -0500

Paul Hoffman scripsit:

> If you believe that ECMA-404 will change in the future, that
> would indicate that ECMA might break interoperability with current
> implementations, even for what they perceive as "good reasons". In
> general, the IETF tries not to have its long-lived standards normatively
> latch on to moving targets for this very reason. Even when other SDOs
> have assured us that they will not make backwards-incompatible changes,
> they have done so anyway (cue the Klensin-esqe theme music...), and
> that has caused serious interoperability problems for the IETF specs.

Binding specifically to the first edition of ECMA-404 would resolve
that problem.  Binding to a specific edition doesn't work so well with
Unicode, but that is because Unicode is a special case: it expands its
coverage in successive editions to ever-larger portions of the world
of writing systems.  It would have been impossible to do the whole
thing at once, unlike even the largest ordinary technical standard.
ECMA-404 is not in that league.

-- 
Not to perambulate                 John Cowan <[email-masked]>
    the corridors                  http://www.ccil.org/~cowan
during the hours of repose
    in the boots of ascension.       --Sign in Austrian ski-resort hotel

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 04:51:01 +0100

> JSON-text  = value

Add optional whitespace around that.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 Nov 2013 20:00:12 -0800

I listed some arguments against this in
http://lists.w3.org/Archives/Public/www-tag/2013Oct/0041.html and at the
moment I still believe them. Is there new information?

On top of that, I have no fear of anyone trying to change JSON in the
future; they would be resoundingly ignored by the community of
implementers.  I speak as one who would love to add built-in date/time
literals but know that it won’t happen.  -T

On Wed, Nov 27, 2013 at 5:00 PM, Alex Russell <[email-masked]>wrote:

>> -- Insert: “A JSON text is a serialized value.  Note that certain
>> previous specifications of JSON constrained a JSON text to be an object or
>> an array.  Implementations which generate only objects or arrays where a
>> JSON text is called for will be interoperable in the sense that all
>> implementations will accept these as conforming JSON texts.”
>>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 27 Nov 2013 23:55:01 -0500

> literals but know that it won’t happen.  -T

The JavaScript variant may stay the same but people will add features if
they find they are necessary to meet their requirements.

As was previously established, many implementations already support a list
of values so as to enable use in append only logs. I have added length
encoded blocks so as to avoid repeatedly base64 encoding binary blobs.

The Javascript world can ignore me, I try my best to ignore the javascript
world. But if people are implementing one of my protocols they will find
that while it will work perfectly OK with JSON, implementations will be
more efficient if they support JSON-A or JSON-B.

Suggesting that people will never change a spec is silly. Of course specs
will be changed.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 28 Nov 2013 07:44:23 +0000

>-- Change the JSON-text production to read:

We also need to make an explicit decision about whitespace-tolerance.

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 09:41:04 +0100

> We also need to make an explicit decision about whitespace-tolerance.

If the point is to track the changes made by ECMA-404 that decision has been made for us (see end of section 4 in 404).
(Note also that JSON is whitespace-tolerant today, so not having that for the new non-containers at top-levels but everywhere else would be rather surprising.*)

Minimal change appears to be:

JSON-text = ws value ws

Grüße, Carsten

*) The other interesting practical usage here is removing newlines from the allowed whitespace for a single JSON-text and joining sequences of these by newlines to form streams of JSON values.
But that is outside the scope of application/json and application/__+json, an instance of which is exactly one JSON-text.
(truenull3true is a valid stream of JSON texts if you don’t require adding whitespace as delimiters.
Implementations that employ a typical form of tokenization without knowing what those tokens can be in JSON won’t like that.  In any case, whitespace is needed to separate numbers.)

---

From: Tim Berners-Lee <[email-masked]>
Date: Wed, 27 Nov 2013 23:51:29 -0500

On 2013-11 -27, at 23:00, Tim Bray wrote:

> I listed some arguments against this in http://lists.w3.org/Archives/Public/www-tag/2013Oct/0041.html and at the moment I still believe them. Is there new information?
> 
> On top of that, I have no fear of anyone trying to change JSON in the future; they would be resoundingly ignored by the community of implementers.  I speak as one who would love to add built-in date/time literals but know that it won’t happen.  -T
> 

JSON is interesting in being a subset of ECMAscript.  That is a big dependency -- will it be preserved?
However as it is unwise to feed JSON into an ECMAscript processor for security reasons, that dependency may not affect code, just mean that JSON and ECMAscript parsers can share parts at  the moment.

One could imagine that the arc of ECMAscript's evolution could end up 
having all kinds of impact on the data structure syntax and semantics.
(unordered sets as alternative to lists? who knows).  So in that case one
could imagine pressure to make a new version of JSON to match.

Yes, literal ISO dates and dateTimes -- I added them to my own N3/turtle parsers without much fanfare, wish they had been put in the Turtle language too.  Maybe they will. 

-timbl

> On Wed, Nov 27, 2013 at 5:00 PM, Alex Russell <[email-masked]> wrote:
> - Remove the trailing section of section 1.2, starting with “ECMAscript 5.1 enumerates...” [because the difference no longer exists]
> 
> - In section 2:
> 
> -- Insert: “A JSON text is a serialized value.  Note that certain previous specifications of JSON constrained a JSON text to be an object or an array.  Implementations which generate only objects or arrays where a JSON text is called for will be interoperable in the sense that all implementations will accept these as conforming JSON texts.”
> 
> On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <[email-masked]> wrote:

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 17:16:31 +0100

On 28 Nov 2013, at 05:51, Tim Berners-Lee <[email-masked]> wrote:

> JSON is interesting in being a subset of ECMAscript.  

That was a clever device used by Douglas Crockford to minimize the bike shedding that was inevitable when he designed the data format.  That device is no longer necessary, indeed, it has become a liability as ECMAscript moved forward.

> That is a big dependency -- will it be preserved?

There is no dependency any more, and the subset relationship has already been destroyed by the introduction of Line/Paragraph separators (U+2028/U+2029) in recent versions of ECMAscript (these are valid in JSON strings but not in ECMAscript 5 strings).

It is best to acknowledge the role of JavaScript as the inspiration for the JSON data interchange format (and one of its biggest customers), but otherwise consider the two specifications independent of each other.

Grüße, Carsten

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 28 Nov 2013 09:37:32 -0800

> If you believe that ECMA-404 will change in the future, that would indicate that ECMA might break interoperability with current implementations, even for what they perceive as "good reasons". In ge…

Stability of ECMA-404 should not be a concern. As far as I can observe TC39 has absolutely no interest in every changing or extending the JSON grammar. I'm confident TC39 would record that as a statement of policy if asked. In fact, the reason TC39 chose to issue ECMA-404 was because there was concern that the JSON WG was on a path to modify and possibly extend JSON syntax. The only sort of changes to ECAM-404 I ever expect to see would be technical corrections to errors found in the current specification language or the addition of informative material to help clarify understanding of the specification.  For example, if there was sufficient public interest we might consider adding an informative Annex that restates the grammar using ABNF notation.

The JSON syntax has its roots in the ECMAScript syntax but is not at all linked to the ECMAScript syntax.  The ECMAScript object and array literal constructs were the inspiration for JSON. But even in the beginning, the JSON syntax was only a subset set of the full syntax of those ECMAScript language features.  ECMAScript object literal syntax has been significantly extended in both the ECMA-262 5th Edition(2009) and the forthcoming 6th Edition.  But those extensions have and will not ever be added to the JSON syntax defined in ECMA-404.

Allen Wirfs-Brock
ECMA-262 Project Editor

---

From: Noah Mendelsohn <[email-masked]>
Date: Sat, 30 Nov 2013 13:57:14 -0500

> However as it is unwise to feed JSON into an ECMAscript processor for
> security reasons, that dependency may not affect code, just mean that
> JSON and ECMAscript parsers can share parts at  the moment.

There may be many other situations in which there is benefit to having JSON 
be a proper subset of ECMAscript. For example, one can imagine tool chains 
used for development or testing that would depend on the ability to copy 
text fragments between JSON and ECMAscript source documents, and one might 
foresee more "vertical" standards that would depend on the commonality as 
well.

Noah