Subject: Re: [Json] ***SPAM*** 5.294 (5) UTF-8 for application/json contexts, UTF-8/16/32 for other contexts

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Nov 2013 21:20:59 +0100

On 27 Nov 2013, at 21:07, Pete Cordell <[email-masked]> wrote:

> UTF-8 with BOM

Why would we add tolerance for BOMs after a decade of living very well without them?

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Wed, 27 Nov 2013 14:54:10 -0600

There's no need for BOMs when using UTF-8 -- there's no case where a
multi-byte UTF-8 codepoint encoding would have more than one ordering
of its bytes.

BOMs are only of interest for UTF-16 and UTF-32.  We don't need BOMs
for UTF-16-/UTF-32-encoded JSON texts either because having the first
character of a JSON text be an ASCII character is sufficient to permit
detection of the text's byte order.

Nico
--

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 27 Nov 2013 20:55:41 -0000

----- Original Message From: "Carsten Bormann"

> Why would we add tolerance for BOMs after a decade of living very well 

Microsoft's editor development team's opinion may differ on that.

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 27 Nov 2013 21:21:03 -0000

----- Original Message From: "Nico Williams"
> detection of the text's byte order.

A number of editors on Windows want to insert a UTF-8 BOM at the start of a 
file to differentiate it from Windows cp1252 etc.  It's important to be 
pragmatic, not just theoretically correct.  A little leeway here could make 
some developers' lives much easier and avoid a few pointless gotchas.

Pete Cordell
Codalogic Ltd

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Nov 2013 22:38:55 +0100

On 27 Nov 2013, at 22:21, Pete Cordell <[email-masked]> wrote:

> A number of editors on Windows want to insert a UTF-8 BOM at the start of a file to differentiate it from Windows cp1252 etc.  It's important to be pragmatic, not just theoretically correct.  A little leeway here could make some developers' lives much easier and avoid a few pointless gotchas.

You can be pragmatic all you want in your code.  Don’t forget to skip comments, accept tabs and newlines in strings, skip trailing commas,  automatically close unbalanced delimiters, and support lots of other non-JSON things people will type up in these editors.

Retrofitting hundreds of production JSON decoder implementations with BOM tolerance is a non-starter.
If JSON is to be useful as an interchange format for small devices, we have to work towards removing fluff, not adding it.  
The only reasonable move towards reality here would be getting rid of the illusion of UTF-16/UTF-32 support.

Grüße, Carsten

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 27 Nov 2013 15:40:50 -0800

On Wed, Nov 27, 2013 at 1:38 PM, Carsten Bormann <[email-masked]> wrote:

> > A number of editors on Windows want to insert a UTF-8 BOM at the start
> of a file to differentiate it from Windows cp1252 etc.  It's important to
> be pragmatic, not just theoretically correct.  A little leeway here could
> make some developers' lives much easier and avoid a few pointless gotchas.
>
> You can be pragmatic all you want in your code.  Don’t forget to skip
> comments, accept tabs and newlines in strings, skip trailing commas,
>  automatically close unbalanced delimiters, and support lots of other
> non-JSON things people will type up in these editors.
>

Perhaps implementors of JSON decoders should have done better job to begin
with and support BOMs. XML parsers do this, good JSON libs do as well.
I feel very little sympathy towards dozens of toy implementations that have
been hacked together with little care towards actual interoperability. And
this knowing exactly how much effort goes into supporting above-mentioned
features; done that.

Same goes for UTF-16 and even UTF-32 support; those were explicitly spelled
out in JSON specification, and at this point I find it little bit
ridiculous that it is possible to both claim "no one should do BOMs since
they were not discussed in the spec" AND "no one ever supported UTF-16
anyway".

-+ Tatu +-

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 05:00:39 +0100

On 28 Nov 2013, at 00:40, Tatu Saloranta <[email-masked]> wrote:

> Same goes for UTF-16 

No, that’s completely different.

BOM tolerance was never in JSON, and arguing that you should add it is like arguing that you should accept tabs in strings or trailing commas.

UTF-16 *was* in the spec.  We have just learned in the intervening time that there is no point.  Completely different situations.  I’m still arguing that now is the time to simplify and recognize UTF-16 as a legacy part of the spec that is going away, but that is a completely different, more tenuous argument.

Yes, I know that “serious” implementations can be made to support all this (including trailing commas), and you have all reasons to be proud that yours is a serious one.  But that doesn’t detract one iota from my point.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 Nov 2013 20:12:02 -0800

FWIW, in the years that I’ve been using JSON as a nice compact expressive
wire format, I have never even once encountered an instance of JSON being
transmitted over the wire in a format other than UTF-8.  Even those
languages which unfortunately use UTF-16 internally have nice JSON
libraries that serialize as UTF-8.   I’m pretty sure that if you started
sending JSON in UTF-16 or (gasp) UTF-32 to well-known public REST-flavored
APIs out there, a substantial proportion of them would just blow up,
because NOBODY DOES THAT.

I still haven’t made up my mind as to what the spec should say, but I sure
wouldn’t go out of my way to serve the interests of wire formats that are
not in practice used.  -T

On Wed, Nov 27, 2013 at 8:00 PM, Carsten Bormann <[email-masked]> wrote:

> BOM tolerance was never in JSON, and arguing that you should add it is
> like arguing that you should accept tabs in strings or trailing commas.
>
> UTF-16 *was* in the spec.  We have just learned in the intervening time
> that there is no point.  Completely different situations.  I’m still
> arguing that now is the time to simplify and recognize UTF-16 as a legacy
> part of the spec that is going away, but that is a completely different,
> more tenuous argument.
>
> Yes, I know that “serious” implementations can be made to support all this
> (including trailing commas), and you have all reasons to be proud that
> yours is a serious one.  But that doesn’t detract one iota from my point.
>
> Grüße, Carsten
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: John Cowan <[email-masked]>
Date: Wed, 27 Nov 2013 23:47:45 -0500

> wouldn’t go out of my way to serve the interests of wire formats that are

So let it say (right after UTF-8 being the default) "The use of other
encodings is known to produce interoperability problems." (Which it is;
we know at least some JSON decoders that won't accept them).  Finis.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is
dramatically overdescribed.  Still other languages are simultaneously
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 28 Nov 2013 07:42:11 +0000

On 11/27/13 6:47 PM, "John Cowan" <[email-masked]> wrote:

>Tim Bray scripsit:
>
>> I still haven¹t made up my mind as to what the spec should say, but I
>>sure
>> wouldn¹t go out of my way to serve the interests of wire formats that
>>are
>> not in practice used.  -T
>

+0.9.  We *could* also add language that allows you to claim to be fully
JSON-compliant while only supporting UTF-8, which is just a small step
further.  That way, if you want to send UTF-[16,32], you MAY, but you
can't really expect it to interop, which is more-or-less the status quo.

-- 
Joe Hildebrand

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 28 Nov 2013 09:11:58 -0000

----- Original Message From: "Carsten Bormann"
On 27 Nov 2013, at 22:21, Pete Cordell <[email-masked]> wrote:

>> A number of editors on Windows want to insert a UTF-8 BOM
>> at the start of a file to differentiate it from Windows
>> cp1252 etc.  It's important to be pragmatic, not just
>> theoretically correct.  A little leeway here could make
> You can be pragmatic all you want in your code.  Don’t forget to
> skip comments, accept tabs and newlines in strings, skip trailing
> commas,  automatically close unbalanced delimiters, and support
> lots of other non-JSON things people will type up in these editors.

I don't see those things as in any way equivalent.

> with BOM tolerance is a non-starter.

The reports received so far seem to suggest that the JSON stack (as opposed 
to just the JSON grammer parser) of most parsers are quite capable of 
handling BOMs.

> If JSON is to be useful as an interchange format for small
> devices, we have to work towards removing fluff, not adding it.
> The only reasonable move towards reality here would be getting
> rid of the illusion of UTF-16/UTF-32 support.

I don't find the 'small device' argument a compelling reason to disallow 
BOMs.  But I agree that if you are using JSON is a application/json / HTTP / 
MIME context then we should limit it to UTF-8.

I think other applications should be allowed a little more latitude to 
decide which of the Unicode encodings they want to use without being told 
that they a not using valid JSON.

Pete Cordell
Codalogic Ltd

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 28 Nov 2013 12:00:07 -0800

> can't really expect it to interop, which is more-or-less the status quo.

This is a good example where your wording creates concerns for TC39 members.  If you were to say that "to be fully application/json compliant..." then TC39 wouldn't have any issues with it. But saying "JSON-compliant" raises many concerns.

Allen

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 29 Nov 2013 23:01:17 +0000

>This is a good example where your wording creates concerns for TC39
>members.  If you were to say that "to be fully application/json
>compliant..." then TC39 wouldn't have any issues with it. But saying
>"JSON-compliant" raises many concerns.

Sorry, I was imprecise.  I meant compliant with the doc at hand, 4627bis,
which is the scope of this discussion.

-- 
Joe Hildebrand