Subject: Re: [Json] The names within an object SHOULD be unique.

From: Jorge Chamorro <[email-masked]>
Date: Sun, 28 Jul 2013 00:10:20 +0200

On 27/07/2013, at 22:22, Matt Miller (mamille2) wrote:

> Hello all,
> 
> Here is an attempt to summarize responses to the duplicate names questions.
> 
> The consensus from this thread seems to be that duplicate names within the same JSON object are not generally desirable.
> 
> There appears to be rough consensus that a generator (for some definition of "generator") ought to take steps to avoid emitting duplicate names, but that there exists a non-trivial amount of software where the prohibition of duplicates cannot be completely enforced.
> 
> There appears to be consensus that parsers (for some definition of "parser") need to be able to deal with duplicate names in some manner.  There is rough consensus that 4627bis ought to provide guidance on how parsers deal with duplicate names.  However, there is no one option that has any consensus; instead, there appears to be acquiescence that a given parser ought to support one option from a limited set of documented choices.  The most common suggestions -- which appear to reflect the majority of existing implementations -- far so:
> 
> * report an error in some manner
> * report the last value that appears in the JSON text
> * report all the values that appear in the JSON text

If duplicate names are generally considered a no-no, and given that they're mostly used for comments (usually with the key "//"), and given the security risks discussed in the "command":"launch-missiles" thread, perhaps the best thing to do with the duplicate keys and their values might be to simply drop them altogether, not throw any errors, not nothing, just drop them, all of them, including the last one.

So missiles won't be launched and comments will vaporize, which is good.
-- 
( Jorge )();

---

From: John Cowan <[email-masked]>
Date: Sat, 27 Jul 2013 18:46:13 -0400

Jorge Chamorro scripsit:

> If duplicate names are generally considered a no-no, and given that
> they're mostly used for comments (usually with the key "//"), and
> given the security risks discussed in the "command":"launch-missiles"
> thread, perhaps the best thing to do with the duplicate keys and their
> values might be to simply drop them altogether, not throw any errors,
> not nothing, just drop them, all of them, including the last one.

This ignores a very important reason for not defining what to do with them,
which is so that streaming generators and parsers don't have to keep track
of what keys they have already processed.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have seen farther than others, it is because I am surrounded by dwarves.
        --Murray Gell-Mann

---

From: Jorge Chamorro <[email-masked]>
Date: Sun, 28 Jul 2013 03:28:12 +0200

> of what keys they have already processed.

Yes, but it's problematic only for the streaming parsers that emit [key,value] pairs, not for those which emit entire objects.

In parsers that emit [key,value] pairs perhaps it doesn't even make sense to expect this behaviour, probably where it belongs is in the next layer.
-- 
( Jorge )();

---

From: John Cowan <[email-masked]>
Date: Sat, 27 Jul 2013 23:46:01 -0400

Jorge Chamorro scripsit:

> Yes, but it's problematic only for the streaming parsers that emit
> [key,value] pairs, not for those which emit entire objects.

In what sense are they streaming parsers, then?  The whole idea of a streaming
parser is that it passes along the elements of JSON objects, arrays, and
perhaps even strings and numbers bit by bit to the invoking application.

-- 
Andrew Watt on Microsoft:                       John Cowan
Never in the field of human computing           [email-masked]
has so much been paid by so many                http://www.ccil.org/~cowan
to so few! (pace Winston Churchill)

---

From: Jorge Chamorro <[email-masked]>
Date: Tue, 30 Jul 2013 13:11:19 +0200

> perhaps even strings and numbers bit by bit to the invoking application.

Say you've got a stream (a pipe or a network connection) and a stream of json-objects arriving through it, you may very well want the parser to emit entire objects instead of its parts, no?
-- 
( Jorge )();

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 30 Jul 2013 14:11:42 +0000

On 7/30/13 1:11 PM, "Jorge Chamorro" <[email-masked]> wrote:

>Say you've got a stream (a pipe or a network connection) and a stream of
>json-objects arriving through it, you may very well want the parser to
>emit entire objects instead of its parts, no?

I've never seen an implementation that is both streaming and not streaming
at the same time.  Can you say more about how you'd write the API for
that?  Better, can you give an example of such an implementation?

If you mean an application that takes a stream of complete JSON objects
without wrapping them in an array, that problem is likely out of scope,
and the processor would act identically to a non-streaming implementation
in our existing worldview.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Tue, 30 Jul 2013 10:26:23 -0400

Joe Hildebrand (jhildebr) scripsit:

> I've never seen an implementation that is both streaming and
> not streaming at the same time.  Can you say more about how you'd
> write the API for that?  Better, can you give an example of such an
> implementation?

I haven't either, but I could see the uses of an implementation that
streamed arrays but not objects (of course, it could hardly stream an
array that was within an object).

> If you mean an application that takes a stream of complete JSON
> objects without wrapping them in an array, that problem is likely out
> of scope, and the processor would act identically to a non-streaming
> implementation in our existing worldview.

+1.  I was trying to think how to say that ....

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
If a soldier is asked why he kills people who have done him no harm, or a
terrorist why he kills innocent people with his bombs, they can always
reply that war has been declared, and there are no innocent people in an
enemy country in wartime.  The answer is psychotic, but it is the answer
that humanity has given to every act of aggression in history.  --Northrop Frye

---

From: "John Levine" <[email-masked]>
Date: 30 Jul 2013 16:07:19 -0000

>array that was within an object).

It has been my impression that the reasons that people use streaming
implementations are either that it's a tiny system that doesn't have
room to buffer full arrays and full objects, or else it's a realtime
application that needs to see the first item in the array or object
before the rest of it arrives.

I understand that there are plenty of implementations that do not have
those constraints, but since there are apparently also plenty of
implementations that do have one or both, Paul's short list of
possible treatments of duplicate names is the best we can do.

---

From: Tim Bray <[email-masked]>
Date: Tue, 30 Jul 2013 09:39:32 -0700

On Tue, Jul 30, 2013 at 9:07 AM, John Levine <[email-masked]> wrote:

> >> I've never seen an implementation ...  Better, can you give an example
> of such an
> >> implementation?
> >
> >I haven't either, but I could see the uses of an implementation

I don’t think “I could see the uses of” is a good reason for writing an
Internet Standard.  Once again as with naked surrogates, there’s an
invisible class of alleged implementors of JSON sending/receiving code for
whom dupe-checking is intractable; except for we haven’t heard from anyone
whose application actually depends on this behavior.

Has anyone any personal knowledge of an app whose correct function depends
on the use of duplicate keys?  -T

> that
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Nico Williams <[email-masked]>
Date: Tue, 30 Jul 2013 11:54:54 -0500

> on the use of duplicate keys?  -T

On was linked earlier, one whose docs explicitly said "JSON supports
this, so we use it".  I don't mind breaking them.  What I do mind is
language that leaves streaming implementations in the cold.

---

From: Stephan Beal <[email-masked]>
Date: Tue, 30 Jul 2013 18:55:25 +0200

On Tue, Jul 30, 2013 at 6:39 PM, Tim Bray <[email-masked]> wrote:

> Standard.  Once again as with naked surrogates, there’s an invisible class
> of alleged implementors of JSON sending/receiving code for whom
> dupe-checking is intractable; except for we haven’t heard from anyone whose
> application actually depends on this behavior.
>

i've mentioned before that it's a cost i would be very upset about having
to pay. For me it's mostly a matter of principal - i feel strongly that
this is a corner case which falls under the "worst practice" category. Use
arrays of key/value pairs when dupes are required. Incompatibilities in
impls == less reliability, and the duplicate keys scenario is _easily_
worked around _within_ a unique-keyed JSON world view, so no changes to
JSON are necessary in order to get the effect for those who really need it
(which is apparently a minority, if my memory of the list traffic serves me
correctly).

> on the use of duplicate keys?  -T

Not i.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 30 Jul 2013 10:56:36 -0700

> on the use of duplicate keys?  -T
You are conflating the issues here.

If you ask whether someone depends on use of duplicate keys, I don't, and
have never seen use I consider valid. One mentioned use case for "comments"
does not seem valid to me.

But your comment on "alleged" intractability is different (and tone
unnecessary).

This is a common use case for processing large JSON files; either output as
JSON arrays, or just sequences of space-separate objects. Typical data sets
are log output, processing from map/reduce style jobs and batch jobs.

In those cases, object mapper or tree builder can build representation for
single object, using streaming parser, only reading that subset of content.
Reading the whole data set would be prohibitive and unnecessary.
Duplicate name detection at higher level is doable. Doing it at streaming
parser level is unnecessary and expensive relative to necessary parts of
decoding.
Separation between streaming part and higher layers is for good separation
of concern, as well as practical matter for reusing components.

I use such processing pipelines regularly, and based on number of questions
on various forums (Jackson user/dev mailing lists, StackOverflow) it is a
common use case.

-+ Tatu +-

---

From: Jorge Chamorro <[email-masked]>
Date: Tue, 30 Jul 2013 20:13:22 +0200

On 30/07/2013, at 16:11, Joe Hildebrand (jhildebr) wrote:

> that?

Easy:

parser= CreateStreamingParser(stream, listener);

:-)

Whenever the parser's got a complete json-object, it simply calls listener(JSON.parse(json-object));

> Better, can you give an example of such an implementation?

There's a bunch, most use a delimiter (say \0) as a chunks boundary marker so that they can be stateless.

But some aren't stateless:
<http://www.chrisdew.com/blog/2013/06/11/json-stream-rfc-a-standard-for-delimiting-in-a-stream-protocol-tcp/>

There's also node.js' fork(), where communication between parents and children happens via "messages" that a are json.strigify()ed objects, via a pipe or a network connection.

The thing is, a streaming parser is one whose *input* is a stream, isn't it?

> in our existing worldview.

That's why I said that duplicate keys complicate only those parsers that emit parts as soon as they arrive.
-- 
( Jorge )();

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 30 Jul 2013 11:13:58 -0700

On Tue, Jul 30, 2013 at 7:11 AM, Joe Hildebrand (jhildebr) <
[email-masked]> wrote:

> that?  Better, can you give an example of such an implementation
>

I do not know if this is what was meant, but there is an obvious ways to
combine streaming parser with object binding (and by object binding I mean
both binding to host language types as well as tree representations).
It is what many Java JSON parsers do; Jackson, Gson, and Genson at least.

Data binding component takes a streaming parser through which it can
iterate over JSON tokens. Data binder simply reads enough tokens to build
specific object indicate by the first token: if it is START_OBJECT, it will
consume up to until matching END_OBJECT. Similarly for other types (for
scalars single token is value).

But application can also iterate over tokens directly, instead of passing
it to data binder. If content is, for example, long array of objects, it
would read START_ARRAY and first START_OBJECT, then start calling data
binder.

Here's an example (not optimal, but functional) of using streaming and
data-binding modes, in complementary way

http://stackoverflow.com/questions/15688951/combining-jackson-objectmapper-and-stream-parsing

Apologies if this is obvious -- I assumed this was common knowledge but
perhaps it is not.

-+ Tatu +-

---

From: Jorge Chamorro <[email-masked]>
Date: Tue, 30 Jul 2013 20:53:57 +0200

> before the rest of it arrives.

Or because the *input* is a *stream* not a string.
-- 
( Jorge )();

---

From: "John R Levine" <[email-masked]>
Date: 30 Jul 2013 23:10:40 +0200

> language that leaves streaming implementations in the cold.

Seems like we're talking past each other again.

We all agree that duplicate keys suck, but since they have been observed 
in the wild, we ask a parser to document what it does with them, from the 
list of options mentioned before.  I think that's both all we can ask, and 
it's plenty.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.