Subject: [Json] What is a JSON-text? [REVISITED]

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Thu, 3 Oct 2013 16:33:31 +0000

Hello all,

There appears to be rough consensus to reference ECMA-262 edition 5.1.  As James pointed out[1], something will need to be said about what is a JSON-text; either 4627bis states a difference from ECMA-262 edition 5.1, or 4627bis states a difference from RFC 4627.

There was previously much discussion on this topic[2].  Some have restated their support.  What do others think about changing JSON-text to allow any JSON value versus just objects or arrays?

-  Paul Hoffman and Matt Miller

[1] < http://www.ietf.org/mail-archive/web/json/current/msg01721.html >
[2] < http://www.ietf.org/mail-archive/web/json/current/msg00553.html >

---

From: Tim Bray <[email-masked]>
Date: Thu, 3 Oct 2013 10:10:25 -0700

I am very strongly against changing the -bis to allow top-level values.
- It’s a bad practice
- support for it in popular libraries is very patchy.

On Thu, Oct 3, 2013 at 9:33 AM, Matt Miller (mamille2)
<[email-masked]>wrote:

> Hello all,
>
> There appears to be rough consensus to reference ECMA-262 edition 5.1.  As
> James pointed out[1], something will need to be said about what is a
> JSON-text; either 4627bis states a difference from ECMA-262 edition 5.1, or
> 4627bis states a difference from RFC 4627.
>
> There was previously much discussion on this topic[2].  Some have restated
> their support.  What do others think about changing JSON-text to allow any
> JSON value versus just objects or arrays?
>
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

---

From: Peter Patel-Schneider <[email-masked]>
Date: Thu, 3 Oct 2013 10:34:08 -0700

The proposal is to *change* JSON in a way that would break some current
JSON parsers and, worse, break some current consumers of JSON, to get
closer syntactic conformance with the ECMAScript JSON implementation.

If there is a goal to align JSON with the ECMAScript JSON implementation,
then I suggest that requiring EMCAScript numbers is much more important and
that even officially requiring support for unpaired UTF-16 surrogate code
units is more important.

I think that this proposal should not be adopted.

Peter F. Patel-Schneider
Nuance Communications

On Thu, Oct 3, 2013 at 9:33 AM, Matt Miller (mamille2)
<[email-masked]>wrote:

> [....]

---

From: Stefan Drees <[email-masked]>
Date: Thu, 03 Oct 2013 19:51:59 +0200

> - support for it in popular libraries is very patchy.

Me also very strongly against "non-container" top-level entities.

Additional rationale:
- Add two bytes to wrap the serialized/transported value into array or
   say six bytes and use eg. {"d":my_top_level_value} should be cheap
   for the benefit of producing a JSON-text.

Even if value is king, values like strings, numbers, true, false or null 
just wrapped into "no clothes" make me wonder why one would serialize 
(or forward) them as advertised JSON-text in the first place.

>> ...

["Stefan"]

---

From: Nico Williams <[email-masked]>
Date: Thu, 3 Oct 2013 13:18:41 -0500

On Thu, Oct 3, 2013 at 12:34 PM, Peter Patel-Schneider
<[email-masked]> wrote:
> The proposal is to *change* JSON in a way that would break some current JSON
> parsers and, worse, break some current consumers of JSON, to get closer
> syntactic conformance with the ECMAScript JSON implementation.

Not really: you can have two definitions of top-level and describe
their interoperability characteristics (the original array-or-object
interops best).

In practice many, many JSON parsers already support at least an option
for top-level value types other than array/object.

> that even officially requiring support for unpaired UTF-16 surrogate code

I think the OP wanted one cheap ECMAScript compat improvement.  I
agree that this is the "cheapest" such improvement.  All the others
you propose have significantly more impact on existing implementations
for them to be able to support the ECMAScript variant.

I'd settle for noting the differences between IETF JSON and ECMAScript.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 3 Oct 2013 13:24:15 -0500

>   for the benefit of producing a JSON-text.

That's very lame; see below for rationale.

> just wrapped into "no clothes" make me wonder why one would serialize (or
> forward) them as advertised JSON-text in the first place.

Because JSON isn't always used to transport values across networks
(where it'd be silly to send scalar values with all the overheade of
HTTP and TCP/IP, say).  JSON is often used for interop between local
components, where the overhead that would make laughable the use of
scalar values at the top-level just doesn't apply.  E.g., the Python
bindings to libjq.  Indeed, here your suggestion that I call "lame"
*adds* undesirable overhead.

"But JSON is an interchange format for networking" is not a good
answer to that.  It's a good interchange format, full stop.  The train
with JSON used for local interchange purposes left the station.  This
is why parsers commonly have an option to accept scalar values at the
top-level.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 3 Oct 2013 13:33:02 -0500

> *adds* undesirable overhead.

I should add that the overhead is also developer overhead: having to
create the array/object to contain the solitary scalar value, then
destructure it to get it back.

I don't care if you have two JSON text definitions (see previous post)
or just allow all value types at the top-level.  It's the same to me,
as long as parser implementors know to provide the option for it (if
they don't just bleeping allow all value types at the top-level
outright, no option necessary, especially so for those implementations
that use run-time typing, which includes several C implementations,
FYI).

Nico
--

---

From: "Markus Lanthaler" <[email-masked]>
Date: Thu, 3 Oct 2013 20:43:55 +0200

On Thursday, October 03, 2013 6:34 PM, Matt Miller (mamille2) wrote:
> There appears to be rough consensus to reference ECMA-262 edition 5.1.
> As James pointed out[1], something will need to be said about what is a
> JSON-text; either 4627bis states a difference from ECMA-262 edition
> 5.1, or 4627bis states a difference from RFC 4627.
> 
> There was previously much discussion on this topic[2].  Some have
> restated their support.  What do others think about changing JSON-text
> to allow any JSON value versus just objects or arrays?

I'm strongly against this. I think it is a bad idea in general - for a
serialization format, not JavaScript - and it would break a lot of existing
systems.

--
Markus Lanthaler
@markuslanthaler

---

From: Jacob Davies <[email-masked]>
Date: Thu, 3 Oct 2013 11:56:08 -0700

> is why parsers commonly have an option to accept scalar values at the

As I said in the last discussion: the asymmetry of a JSON document
having to be an object or an array while a JSON value can be an object
or an array or a string or a number or a boolean or null is
unintuitive and has no principled rationale. The arguments against
allowing values at top level are taste-based, not based on principle
or practice.

In the systems I work on we treat "JSON serializability" as meaning
"transformable into a JSON value". Not a JSON text. The natural
serialization for many types is a string or a number or a boolean. So,
why do I suddenly have to act differently when I am sending something
on the wire? What is the principled distinction between a JSON value
in memory and a JSON text sent over the network? Why am I supposed to
put this in an envelope just because network transport is involved?

Allowing values at the top level is not a breaking change since the
JSON spec says that implementations may reject texts they don't like.
In practice, it is widely interoperable already.

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 3 Oct 2013 11:59:15 -0700

<no hat>

> There was previously much discussion on this topic[2].  Some have restated their support.  What do others think about changing JSON-text to allow any JSON value versus just objects or arrays?

I'm +.5 on the proposal. It would make 4627bis more aligned with ECMAScript, and there is no rationale given for "a JSON-text can be less that what you think it can be". This change will make some parsers that are 4627-but-not-ECMAScript compliant become not compliant. To me, the latter is not as important as the former.

--Paul Hoffman

---

From: Stefan Drees <[email-masked]>
Date: Thu, 03 Oct 2013 21:21:57 +0200

> is why parsers commonly have an option to accept scalar values at the

  ... and no one stops implementers nor users from parametrizing a 
parser to adapt to special situations.

Say there is a document named pi.json that contains the text /3.14/ (but 
without the slashes) that may be the (best effort) serialized 
representation of the "number" pi or just the first four bytes, as the 
file became truncated (or the local message content if you are so 
inclined). As a JSON text wraped inside array or object, the situation 
would be clear, in the above "unwrapped" case it is not anymore.

So I think, the above '[...]values like strings, numbers, true, false or 
null just wrapped into "no clothes" make me wonder why one would 
serialize (or forward) them as advertised JSON-text in the first place.' 
remains as it stands, doesn't it?

Whatever un-"lame" and un-"laughable" ;-) "train has left the station", 
I guess that without making "wrapping" mandatory for JSON-text, one 
might consider JSON-izing every binary protocol on earth into such an 
unwrapped JSON+* channel (by combining the unlimited "number" feature of 
the JSON rfc and "interpreting" some representation bilaterally 
negotiated as the underlying bit pattern for the stored/transmitted 
binary protocol message).

The set of relations, where one uses JSON as a means to communicate 
between otherwise unrelated components, that are on one hand so closely 
coupled, that no wrapping is needed, but on the other hand so far apart, 
that they cannot accept non JSON-text (as it stands now) looks similar 
to the empty set to me, but I am sure, people will let me soon know of 
exotic situations ... :-)

["Stefan"]

---

From: R S <[email-masked]>
Date: Thu, 3 Oct 2013 13:57:29 -0700

On Thu, Oct 3, 2013 at 11:59 AM, Paul Hoffman <[email-masked]> wrote:

> restated their support.  What do others think about changing JSON-text to
> allow any JSON value versus just objects or arrays?
>
> I'm +.5 on the proposal. It would make 4627bis more aligned with
> ECMAScript, and there is no rationale given for "a JSON-text can be less
> that what you think it can be". This change will make some parsers that are
> 4627-but-not-ECMAScript compliant become not compliant. To me, the latter
> is not as important as the former.

In the abstract, I think matching ECMAScript is a good idea. If I were
choosing a parser, I would definitely pick one that supported any JSON
value at the root.

As a sender, I wouldn't want the RFC to lead me to believe that every JSON
parser could handle the ECMAScript root-level primitive JSON values at this
time. Having a note that points out this variation exists is a good idea.

Some other group of people can come back in 10 years and write down what
happened (maybe the note will tempt everyone to support primitive root
values, maybe not).

- Rob

---

From: Larry Masinter <[email-masked]>
Date: Thu, 3 Oct 2013 14:09:40 -0700

I think you might make more headway if you think about the primary role of 4627 and 4627bis
as registering Internet Media Types.  Consider, for example, having two media types

text/json and text/json+

the first of which is 4627 compatible, and the second (text/json+) allowing "non-container"
top level.

Everything that is valid text/json would also be valid text/json+ with the same interpretation.

Or, you could keep the same internet media type and just declare it an extension,
like image/gif changed for GIF89a. Or Postscript changed and still kept
application/postscript. Yes, it's a change, no it fits within the convention of Internet
Media Types to accept extensions that are lexically distinguishable.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Stefan Drees
> Sent: Thursday, October 03, 2013 12:22 PM
> To: Nico Williams
> Cc: Matt Miller (mamille2); Tim Bray; JSON WG
> Subject: Re: [Json] What is a JSON-text? [REVISITED]
>

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Oct 2013 08:58:06 +1000

What Larry said.

The RFC is very clearly defined to not allow this. The reasons for allowing other top-level values make total sense, but they're not JSON. One language that uses JSON allows more, but that's not JSON, else Crock would have written something different down.

Allowing more top-level values in text/json does *not* help interop; it hurts it.

OTOH doing it in another, separate media type (along with all of the other things we want to fix up) is perfectly workable; it improves interop for those who choose to use it.

Regards,

P.S. What we need to realise here is that just because we have the pen for bis, doesn't mean we can change the reality of deployed implementations, except with *very* gentle nudges over long periods of time. Or, put another way: <http://www.youtube.com/watch?v=409Pjtq7jzY>

--
Mark Nottingham   http://www.mnot.net/

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 4 Oct 2013 11:43:59 +1000

I certainly agree with this part of what Larry said:
>> it fits within the convention of Internet Media Types to
>> accept extensions that are lexically distinguishable.

> The RFC is very clearly defined to not allow this. The reasons for
> allowing other top-level values make total sense, but they're not JSON.
> One language that uses JSON allows more, but that's not JSON, else
> Crock would have written something different down.

I thought Crock did write something different down -- at the next opportunity which was when specifying JSON in ECMAScript 5.1.

> OTOH doing it in another, separate media type (along with all of the
> other things we want to fix up) is perfectly workable; it improves
> interop for those who choose to use it.
> 
> Regards,
> 
> P.S. What we need to realise here is that just because we have the pen
> for bis, doesn't mean we can change the reality of deployed
> implementations, except with *very* gentle nudges over long periods of
> time. Or, put another way: <http://www.youtube.com/watch?v=409Pjtq7jzY>

I thought this would be a gentle nudge over a long period.

--
James Manger

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 04 Oct 2013 16:33:21 +0200

* Matt Miller (mamille2) wrote:
>What do others think about changing JSON-text to allow
>any JSON value versus just objects or arrays?

As far as I am concerned, the Working Group is supposed to standardise
JSON and right now we have a chaotic mix of implementations and usages
where a parser might default to "lax" parsing permitting `null` at the
top-level and a "strict RFC 4627" mode disallowing that, others might
default to the strict mode and offer a lax one, others might offer only
one mode; there are probably even cases where `json_encode` is con-
servative in what it sends, while the corresponding `json_decode` func-
tion is liberal in what it accepts, with the unfortunate property that

  json_encode(json_decode(...))

might fail, even though nobody would suspect that reading the code. In
some cases the "lax" parser mode might also support other features not
normally allowed, say trailing commas, comments, or the empty string as
JSON text. A consequence might be that turning on the "lax" mode adds
security vulnerabilities or other hazards to a system. It seems pretty
clear to me that we would be better off now if RFC 4627 had made it
clear that all values can be used at the top level.

I assume some people see JSON primarily in the sense of "JSON file" or
"complete message of a JSON-based protocol" and dislike scalar values
at the top level because you would not encounter them in such settings.

In practise, JSON is frequently used in many other settings, to encode
values for other protocol elements or APIs for instance. In web browser
settings for instance people might put JSON-serialised data into cookies
or the text-only "local storage", in `application/x-www-form-urlencoded`
parameters, or communication across windows using `.postMessage` and its
ilk, or as message in low-overhead protocols like the WebSocket.

With such usage it is fairly common that values might be either an array
of numbers, or just the value null, or an object or null, just as it may
be a string or null. Sometimes it's either an object, or a error string.

So, if one were to design a JSON parsing and serialisation API, it seems
clear that one would allow serialisation of scalar values and would make
the parser accept whatever the serialiser generates; especially this

  var json_text = "...";
  var obj = json_decode(json_text);

  assert( obj.fragment eq json_decode(json_encode(obj.fragment)) );

should succeed without failure if `json_text` is properly formed JSON.
If that is not the case, users of the parsing and serialisation API are
likely to have plenty of rarely surfacing hazards in their code. As a
simple example, consider

  var json_text = "...";
  var obj = json_decode(json_text);
  ...
  var response = send_network_request(..., json_encode(obj.fragment));

  if (!response.is_success) {
    ...
  }

If `json_encode` serialises `obj.fragment` as requested, even when it
is `null`, then send_network_request() would be passed a normal string
just like it would in all other cases, and assuming the other endpoint
in the network connection regards a top level `null` as error, the error
would likely be caught alongside other network errors. If instead the
`json_encode` method threw an exception, who knows what would happen; if
it returned a special value, say an `undef` value, again, who knows how
`send_network_request` will react.

Now, if the JSON library happily generates scalar values as the top then
it becomes the responsibility of the caller to ensure scalar values at
the top level are not generated when it's considered "wrong". That is
very hard: there seems to be a large grey area where it's not clear if
scalar top level values are a bad practise, and if you take the example
above, telling whether `obj.fragment` might ever be a scalar can take a
lot of effort, especially if the code around json_encode can change, and
the `json_encode` caller might not even know because the `json_text` is
produced by a third party network service where the documentation does
not say conclusively whether it may ever be a scalar; and guarding each
and every `json_encode` call with checks for top level scalars is hard
aswell, especially considering the costs of the resulting code bloat.
And the same goes for stuffing everything into an extra array or object.

Worse, even in cases like "JSON in HTTP request body" there appears to
be no consensus that sending top level scalar values is bad practise; I
remind once more of the proposed JSON support in XMLHttpRequest that has
no problems generating them, and the lack of complaints against that.

The "arguments" against allowing scalar values at the top level so far,
well, I have seen no attempt to argue that, all things considered, those
in charge of running code that supports JSON-based protocols are best
off with top level scalars disallowed, and it seems a rather unlikely
proposition to me. So I am very much against allowing people to say
strings like `null`, `""`, `1.0` are not JSON on their own.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 4 Oct 2013 09:58:46 -0700

<hat on>

There seems to be deep disagreement, with sound technical arguments on both sides. The chairs take this as "don't change the document".

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Fri, 4 Oct 2013 13:58:32 -0500

On Oct 4, 2013 11:39 AM, "Bjoern Hoehrmann" <[email-masked]> wrote:
> [...]

Exactly.  To add to this, just as we'll have text about interop for
numbers, we should have text about top-level value interop.  There's no
reason to exclude a very commonly implemented aspect of ECMAScript (one
that is much simpler to manage than matching ECMAScript's number system).

That, is my proposal is to:

   allow any value type at the top-level in the ABNF, just as the ABNF
allows bignums, and then note that many implementations by default (or
always) only allow arrays and objects at the top-level.

I don't see how this creates an incompatibility given what's already out
there.

Nico
--

---

From: R S <[email-masked]>
Date: Fri, 4 Oct 2013 18:12:38 -0700

Just to be clear, I think this means: the chairs feel there is no consensus
to change conformant top-level values in the new RFC, but there is
consensus to reference ES 5.1 for the purposes of pointing out that there
are differences.

Is that correct?

- Rob

On Friday, October 4, 2013, Paul Hoffman wrote:

> <hat on>
>
> There seems to be deep disagreement, with sound technical arguments on
> both sides. The chairs take this as "don't change the document".
>
> --Paul Hoffman
> [email-masked] <javascript:;>

---

From: Tim Bray <[email-masked]>
Date: Fri, 4 Oct 2013 19:01:30 -0700

Dunno what the chairs think, but that seems sensible to me.

On Fri, Oct 4, 2013 at 6:12 PM, R S <[email-masked]> wrote:

> Just to be clear, I think this means: the chairs feel there is no
> consensus to change conformant top-level values in the new RFC, but there
> is consensus to reference ES 5.1 for the purposes of pointing out that
> there are differences.
>
> Is that correct?
>
> - Rob
>
>

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 4 Oct 2013 19:16:18 -0700

On Oct 4, 2013, at 6:12 PM, R S <[email-masked]> wrote:

> Just to be clear, I think this means: the chairs feel there is no consensus to change conformant top-level values in the new RFC, but there is consensus to reference ES 5.1 for the purposes of pointing out that there are differences.
> 
> Is that correct?

Yes, definitely. This question was just about the technical change to the definition of JSON-text.

--Paul Hoffman