Subject: Re: [Json] I-JSON Tpic #2: Top-Level

From: Matt Miller <[email-masked]>
Date: Tue, 20 May 2014 14:18:38 -0600

It has been several days since the last discussion on this topic.  As
it currently stands, it appears there is no consensus to restrict the
top-level to only object (or object / array).

Does anyone have more to say with regards to top-level restrictions in
I-JSON?

- -- 
- - m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: Tim Bray <[email-masked]>
Date: Tue, 20 May 2014 13:27:22 -0700

For what it’s worth, this particular consensus excludes me.

1. The goal for I-JSOn is to maximize interoperability, and there is
software out there coded to 4627 which will fail when presented with 42 or
true; so interoperability suffers when you do that.
2. For the type of work the IETF is engaged in, Must-Ignore is almost
always a win in protocol designs, and thus we would do well to encourage
the use of JSON that makes this straightforward.

On Tue, May 20, 2014 at 1:18 PM, Matt Miller <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: GPGTools - https://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBCgAGBQJTe7ieAAoJEDWi+S0W7cO1/TAH/0l9Um416SDS1RNYyrVyVlip
> Bc/u5zGGaxjASEw1BVt0cckKMcp6syEMRFHmZ8XaUFMFJFFUMn8G6Ob4Lq7Za1K+
> untUCjYkmavzR8frGGJPfmH9EdlUov8Xjm3ByZ+TRswb20cIgrRqw+baI8TbTNR1
> eNZKJ2Y4vW0nFkPvIWdvYwf4Sht8DakM7bYjC4iImrU0t58ESwbFw1uLKGVgrVfU
> y62Jxmp1Yz6WWPIf1Q8MQupDXZNosBS/hLDUoF4FjOUXaNR3CCr8wpQBMm4uBlmZ
> tE6Kwddutq5Kwvvwq7YjpJGUS4uhYNcIVDWwkoBz7ZDOGtctva5UjLjIZ8OnQWg=
> =3Bkb
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 20 May 2014 13:50:53 -0700

On May 20, 2014, at 1:27 PM, Tim Bray <[email-masked]> wrote:

> 1. The goal for I-JSOn is to maximize interoperability, and there is software out there coded to 4627 which will fail when presented with 42 or true; so interoperability suffers when you do that. 
> 2. For the type of work the IETF is engaged in, Must-Ignore is almost always a win in protocol designs, and thus we would do well to encourage the use of JSON that makes this straightforward.

<no hat>

The goal of this document is to maximize interoperability. RFC 4627 and other specs had the rule that JSON texts had to be objects or arrays. Restricting JSON texts *in this profile* maximizes interoperability better than RFC 7159 did.

This profile is not meant to be correcting RFC 7159; it is meant to be giving a profile that maximizes interoperability.

--Paul Hoffman

---

From: Stefan Drees <[email-masked]>
Date: Wed, 21 May 2014 07:22:49 +0200

On 2014-05-20 22:50 +02:00, Paul Hoffman wrote:
>> 1. The goal for I-JSOn is to maximize interoperability, and there
>> is software out there coded to 4627 which will fail when presented with 42
>> or true; so interoperability suffers when you do that.
> The goal of this document is to maximize interoperability. RFC 4627 and
> other specs had the rule that JSON texts had to be objects or arrays.
> Restricting JSON texts *in this profile* maximizes interoperability
> better than RFC 7159 did.
>
> This profile is not meant to be correcting RFC 7159; it is meant to be
> giving a profile that maximizes interoperability.

+1 from here

{"Stefan": "Greetings"}

---

From: Jacob Davies <[email-masked]>
Date: Tue, 20 May 2014 23:02:44 -0700

> The goal of this document is to maximize interoperability. RFC 4627 and other specs had the rule that JSON texts had to be objects or arrays. Restricting JSON texts *in this profile* maximizes inte…

Restricting top-level elements to objects or objects and arrays
repeats the mistake of the original JSON specification. But almost
every JSON parser ignored that advice, most importantly including the
Javascript JSON.parse() method.

Unlike the other concerns being addressed, the issue of non-object
top-level elements is not a subtle gotcha. If an application says it
emits or receives JSON but fails to specify the numeric range they
support or fails to specify the behavior when duplicate keys are given
or fails to specify that non-character code points are disallowed,
there is a real risk of subtle interoperability problems. But if an
application says it emits or receives JSON strings as JSON documents,
the interoperability error with some client that does not support
top-level strings is gross. There is no chance of a subtle failure.

Mostly I think including this constraint is a mistake because it
overshadows the excellent advice given in the rest of the I-JSON spec
on the limits of numeric precision, non-character code points, and
duplicate keys. It makes I-JSON an unsuitable drop-in replacement for
JSON by making certain unambiguous and nearly-universally-supported
use cases illegal. This is throwing the baby out with the bathwater;
the rationale that it presents a real interoperability concern is not
really supportable, as the general indifference to dropping this
constraint in recent JSON specifications makes obvious.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 21 May 2014 08:16:31 +0200

Restricting I-JSON to top-level objects only, would immediately make JCard (RFC 7095) and JCal (draft-ietf-jcardcal-jcal-10.txt, in RFC-EDITOR state) non-I-JSON.

Since these are significant IETF efforts, that would make me unhappy.

(I don’t have an opinion yet whether I-JSON should restrict itself to RFC 4627 JSON, i.e. objects and arrays, or should embrace the ECMA-404/RFC 7159 extension.  Both sides have quite good arguments for them.)

Grüße, Carsten

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 21 May 2014 17:32:51 +0900

> Restricting I-JSON to top-level objects only, would immediately make JCard (RFC 7095) and JCal (draft-ietf-jcardcal-jcal-10.txt, in RFC-EDITOR state) non-I-JSON.

I wanted to write that I didn't see that much of a need for single 
strings, numbers,... in any serious format for data interchange.

As the above seems to offer good counterexamples to my understanding, I 
went to look at the specs. I only skimmed them for the examples, but I 
didn't find a single one that would have consisted of just a string or 
just a number. If there are such examples in these specs, or if there 
are no examples but the spec allows them nonetheless, could you please 
point out where (and more interestingly, for what) they are allowed?

Regards,   Martin.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 21 May 2014 10:39:15 +0200

JCard and JCal use top-level arrays, not JSON “objects” (maps).
My message was about the proposal to recommend not using arrays as top level data items.

AFAICS, JCard and JCal are RFC 4627 compliant.

Grüße, Carsten

PS.: In my experience, calling JSON maps “objects” is the single most confusing thing in talking to people about JSON.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 21 May 2014 15:08:34 +0000

On 5/20/14, 2:27 PM, "Tim Bray" <[email-masked]> wrote:

>software out there coded to 4627 which will fail when presented with 42

This is a compelling argument to me.  Limiting I-JSON to objects+arrays as
top level would be my preference because of this.

>the use of JSON that makes this straightforward.

Although I agree with this, I think it is protocol design guidance not
interoperability requirements.  We can add a non-normative section to
I-JSON for this (or start a new document) but I don't think we should use
it to limit I-JSON to only object top-levels.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 21 May 2014 08:09:08 -0700

> Restricting I-JSON to top-level objects only, would immediately make JCard (RFC 7095) and JCal (draft-ietf-jcardcal-jcal-10.txt, in RFC-EDITOR state) non-I-JSON.

This is a good reason to change I-JSON's current requirement to be object+array. I think Tim's original paring-down was to allow an identifier, but we scrapped that, so making I-JSON no more restrictive than 4627 and friends would be fine.

--Paul Hoffman

---

From: Jacob Davies <[email-masked]>
Date: Wed, 21 May 2014 08:12:54 -0700

On Wed, May 21, 2014 at 1:32 AM, "Martin J. Dürst"
<[email-masked]> wrote:
> I wanted to write that I didn't see that much of a need for single strings,
> numbers,... in any serious format for data interchange.

Some examples from the earlier discussion:

1. The retrieval of part of a resource by path. Requiring a JSON
object envelope is pointless and asymmetric.
2. Patching just part of a resource by path. Again, requiring an
object envelope is unnecessary.

The most common transport for JSON is HTTP, which already provides for
an out-of-band key-value space for extra data. It probably is the case
that JSON objects are the most suitable choice for the messages
exchanged in most protocols, but there are exceptions as described
above, and in any case I doubt anyone is really confused over whether
an object or a string is the most suitable message format.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 21 May 2014 15:39:47 +0000

>an object or a string is the most suitable message format.

For these cases, why send JSON at all?  Why not just send a string?  That
way you also don't need the unnecessary double-quote envelope?

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 May 2014 10:48:10 -0500

> The goal of this document is to maximize interoperability. RFC 4627 and other specs had the rule that JSON texts had to be objects or arrays. Restricting JSON texts *in this profile* maximizes inte…

Is that so?  Why?  Because the earlier RFC also so restricted them?
But why should that matter now for a new profile of RFC7159?  Perhaps
because interop with RFC4627 implementations is desired?  But every
one of those I've looked at handles non-object/array values at the
top-level... (granted, I've not looked at all of them).

Anyways, IIUC Tim's proposed restriction is about his preference is
for an ignore-unknown/unexpected policy for extensibility reasons.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 May 2014 10:51:13 -0500

On Wed, May 21, 2014 at 3:32 AM, "Martin J. Dürst"
<[email-masked]> wrote:
>> Restricting I-JSON to top-level objects only, would immediately make JCard
>> (RFC 7095) and JCal (draft-ietf-jcardcal-jcal-10.txt, in RFC-EDITOR state)
>> non-I-JSON.
>
> [...]

Well, one could make use of JSON text sequences where each text is not
an object or an array.  The canonical example for me would be database
query results when the result set is of indeterminate length at first.

Suppose you have an I-JSON application that you want to extend to use
JSON text sequences...

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 May 2014 10:54:16 -0500

> way you also don't need the unnecessary double-quote envelope?

Because one might want to distinguish numeric, string, boolean, and
null values from each other.  Or because one might want to apply JSON
text sequencing.

If you use JSON, you can make use of JSON text sequences.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 21 May 2014 09:05:24 -0700

On May 21, 2014, at 8:48 AM, Nico Williams <[email-masked]> wrote:

> Is that so?

Yes. The goal is stated in the abstract and introduction.

>  Why?  

Because that's what the WG wanted.

> Because the earlier RFC also so restricted them?

Only partially. The more important part is that implementations followed the earlier RFC and fall over when they see something different. That's a lack of interop.

> But why should that matter now for a new profile of RFC7159?  

This is feeling circular. To interop with existing implementations, this profile needs to pay attention to what they do. If you don't care about interop, then (a) don't implement I-JSON and (b) propose N-JSON.

> Perhaps
> because interop with RFC4627 implementations is desired?  

Not perhaps: definitely.

> But every

There were reports that some fell over or processed the other top-level items wrong.

> for an ignore-unknown/unexpected policy for extensibility reasons.

YUI.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 May 2014 11:21:52 -0500

>> because interop with RFC4627 implementations is desired?

Thanks.

> There were reports that some fell over or processed the other top-level items wrong.

I missed them.  Where can I find those reports?  (what search terms
might I use?)

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 21 May 2014 09:43:36 -0700

> I missed them.  Where can I find those reports?  (what search terms

<sigh> Maybe I made them up. I just used all the search terms I could think of and failed to find the (multiple!) reports I remember.

Having said that, a few people (including you...) felt that requiring containers at the top level was beneficial for streaming parsers to detect broken streams. There could be streaming parsers that rely on that.

--Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 21 May 2014 16:46:56 +0000

>I missed them.  Where can I find those reports?  (what search terms

Ruby:
http://www.ietf.org/mail-archive/web/json/current/msg00670.html

I just tried it with the stock Ruby on an up-to-date Mac, and it still
fails.  We might have stopped looking when we found a mainstream
implementation that broke.

Anyone who wants to try other languages, there are implementations listed
at http://www.json.org/.

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 21 May 2014 19:08:53 +0200

On 21 May 2014, at 18:46, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> fails.

... as it should — an RFC 4627 implementation that just accepts non-4627 JSON without some form of special flag is broken.

There is a tendency of JSON implementers to accept soup, and that also includes accepting JSON extensions such as this one without appropriate permission.
(Unfortunately, Ruby’s default JSON parser currently has only a single not very well documented toggle called “quirks_mode” that switches on the 7159 extensions together with a whole lot more questionable stuff.
Can also be activated by using JSON.load() instead of JSON.parse().)

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 21 May 2014 17:47:26 +0000

On 5/21/14, 11:08 AM, "Carsten Bormann" <[email-masked]> wrote:

>... as it should — an RFC 4627 implementation that just accepts non-4627
>JSON without some form of special flag is broken.

Oh, agree.  My point was that there is at least one implementation that
breaks when you pass it non-4627, that there are likely others, and that
therefore the interop guidance for object-or-array is reasonable in I-JSON.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 May 2014 13:43:35 -0500

> http://www.ietf.org/mail-archive/web/json/current/msg00670.html

Thanks.  And it has no options to accept any type at the top-level
either.  Bummer.

I'm not sure that I care much about this though.  The choice to be
interoperable with such implementations is arbitrary.  If the WG wants
that, then restricting the top-level in I-JSON is the right thing to
do.  Paul asserts that that is what the WG wanted, but Matt's reading
of the consensus on this issue implies the opposite.  We're not done
establishing consensus though.  It might help to cast the interop
issue in starker terms: interop with RFC4627 implementations with no
option to parse all types at the top-level.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 21 May 2014 18:44:40 +0000

>option to parse all types at the top-level.

That's what I want.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 May 2014 17:01:25 -0500

>>option to parse all types at the top-level.

I have no position on that issue.  IF that was not a WG goal, then I'd
much prefer to not impose any top-level type restrictions.  My
position is therefore neutral on this.

---

From: Jacob Davies <[email-masked]>
Date: Wed, 21 May 2014 15:47:02 -0700

>>option to parse all types at the top-level.

I have to ask again: why?

The other issues mentioned in I-JSON are subtle interoperability
problems that could trip people up unless they take care to use a
compliant parser/encoder. In fact, they certainly will trip people up
if they do not carefully choose a good library, which means not
picking any old RFC4267 library, because there are lots of extant
libraries that allow the precise constructs that I-JSON mentions.

For instance, there are multiple JSON libraries - including the Java
json.org library - that encode Java longs as JSON numbers, which is
exactly the kind of problem I-JSON is concerned with, since Javascript
cannot correctly deserialize these values, a problem that has led to
numerous prominent bugs whose fix (replacing numbers with strings) was
not backwards-compatible. The point about subtle problems is that they
can go unnoticed for a long time and the cost of fixing them is large.

These kinds of problems are what I-JSON intends to address, but to use
I-JSON effectively means either:

1. Using an pre-existing JSON library that is already pretty-well
compatible with the I-JSON requirements (for instance, Javascript's
JSON.parse()). This does NOT include many - perhaps even most -
existing JSON libraries. In my experience with the Java libraries,
most of them fail on at least one of number range, duplicate keys, and
non-characters. Yet almost all of them handle top-level values of any
type just fine (I believe the json.org code does not).

or 2. Writing new libraries that are fully compliant with I-JSON, in
which case the point about existing libraries is irrelevant.

I have never heard of any interoperability problems caused by
top-level values of non-object types and I find it hard to imagine it
would be a real concern, since agreement on the types to be sent and
received is hard to avoid upfront and grossly obvious when it occurs.
In the cases where a top-level primitive or array is desirable, it
will be pretty clear why that is the case (most importantly, selecting
part of a composite object).

This restriction was widely ignored since it never existed in
Javascript in the first place; eval() and JSON.parse() have always
accepted all Javascript literal values anyway. Most people just did
what JS did and accepted them too. That there exist some libraries
that enforce it is not actually evidence that it matters to I-JSON,
since most of those libraries probably fail the other I-JSON tests and
will need rewriting to comply with its requirements.

The restriction to objects and arrays at the top level was a clever
hack to avoid using any out-of-band information to distinguish between
UTF variants at a time when UTF-8 had not taken over the world. It was
obsolete a long time ago. Replicating it in recommended guidance for
using JSON in modern applications just seems very odd to me (and the
original rationale for distinguishing UTF variant doesn't even exist
in I-JSON since UTF-8 is required).

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 23 May 2014 04:05:28 +0000

>I have to ask again: why?

...

>Yet almost all of them handle top-level values of any

That's why.

-- 
Joe Hildebrand

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 26 May 2014 16:06:23 +0900

> implementation that broke.

I just submitted a bug report, please see 
https://github.com/flori/json/issues/206.

I don't have a clue how quickly, and with how much of a need for 
options, this will be fixed.

Regards,   Martin.

---

From: Matt Miller <[email-masked]>
Date: Wed, 28 May 2014 11:33:57 -0600

Hello All,

After another round of discussion, it appears there is rough consensus
for restricting the top-level in I-JSON to object or array.  The
Chairs note consensus is rough here.

Does anyone have additional arguments to make for or against what
constitutes an I-JSON top-level?

- -- 
- - m&m

---

From: Matthew Morley <[email-masked]>
Date: Wed, 28 May 2014 15:57:03 -0400

> constitutes an I-JSON top-level?

No additional arguments from me, though I would prefer to see all types
supported at the top level... but instead I'll post an observation from the
RPC problem space.

Given that I-JSON limits the top level to be an object or an array, then
systems that generate results which are not those structures, therefore
require encapsulation to be communicated at all.

Meaning I can cannot respond with 42 and consider it a valid I-JSON (or
JSON).

Effectively this means all response data for such systems should be
encapsulated, including objects or arrays, into a known response structure
of some shape. The JSON-RPC specification does this with 'request' and
'response' objects, exactly for this reason and the desire to convey errors
and other data.

Thinking about RPC style protocols where local methods may be executed,
with the method and params then turned into I-JSON, then shipped off to a
processing end point, decoded, processed, re-encoded, sent back, decoded,
and finally received to be handled and understood.

In practice this means:
  - If the actual result of calling sum(40,2) is 42.
    - at the I-JSON layer it must look something like: {"result":42}.
  - If the result of calling object() is {};
    - at the I-JSON layer it must look something like: {"result":{}}.

With the top-level restriction in place, but without a standard structure:
  - non-structured data must be encapsulated: {"result":42}
  - structured data would be ambiguous: {} -or-  {"result":{}}

If you are in a system that uses function composition, it means you need to
bake this into your expected types/categories, or have a design that
handles and hides the encapsulation/decapsulation steps from the
composition reasoning portion. I don't have (or plan) such a system, so
this is outside my expertise, but imagine this being a limitation for those
that might have/want such a system. (?)

Though again, a limitation that can be overcome with standardized
encapsulation.

[...]

-- 
Matthew P. C. Morley

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 May 2014 16:54:00 -0500

[...]

> If you are in a system that uses function composition, it means you need
to bake this into your expected types/categories, or have a design that

Thanks for capturing this issue so clearly.  Yes, superfluous encapsulation
is obnoxious, particularly in systems with computable units not all of
which produce the blessed output types.

Remind me, what do we gain from this?  Interop with old libraries that
probably don't conform to I-JSON anyways and which stand a good chance of
being fixed soon enough that this top-level restriction ends up buying us
nothing a year hence.  Please re-think this!

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 29 May 2014 12:09:42 -0500

s/computable/composable/

autocorrect kills me.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 29 May 2014 13:45:22 -0400

For protocols, I find the most convenient approach to be that the
message type be a tag and the message data be the corresponding value.

So if I have a protocol 'Fred' the HTTP binding is going to look like this:

GET /.well-known/Fred/operation1
...

{ "operation1" : {
   "tag1" : "value1" ...
   }}

While it is possible to rely on the command value in the URI, this
makes authentication a lot harder as the URI part of a request is
likely to be 'damaged' by proxies and the like. So I much prefer that
the command be a part of the JSON payload.

This enables the use of message layer authentication. It also makes
schemes of the form 'take command from Alice and lob it over to Carol
along with the authorization from Bob'. It also removes the assumption
that HTTP is the only legitimate binding for Web Services.

I can see cases where it would be useful to have a scheme where the
binding allows for an array of Web Service messages per binding
message rather than just one.

When I look at what my applications actually use from HTTP, it is basically:

1) The URL part
2) The session authentication header I defined as an extension
3) The Content-Length header for framing
4) Possibly the Content-Type in the future

Everything else is just noise that is needed to get through proxies,
firewalls and other nonsense that will never ever add value because we
know the message type and the data is not static so it can't be
cached.

Rather than looking at HTTP/2 as the next platform for Web Services,
it seems to me that we could do rather better with no headers at all
rather than some whacky compression scheme.