Subject: [Json] Proposed minimal change for duplicate names in objects

From: Paul Hoffman <[email-masked]>
Date: Tue, 2 Jul 2013 16:25:48 -0700

<chair hats on>

Do either or both of these proposals work for people in the WG?

--Matt Miller and Paul Hoffman

The following are proposed minimal changes to make the JSON spec interoperable with respect to
objects that have duplicate names.

Proposal 1:

In section 2.2 (Grammar -> Objects):
  No change
In section 4 (Parsers):
  Add: "If a parser encounters an object with duplicate names, the parser MUST use only the last
  name/value pair that has the duplicate name.
In section 5 (Generators):
  Add: "The names within an object SHOULD be unique."

Proposal 2:

Same as Proposal 1 *except* that a second sentence is added for section 4: "A parser that streams
  its outputs MUST fail to finish parsing if it encounters more than one name/value pair with the
  same name."

---

From: Vinny A <[email-masked]>
Date: Tue, 2 Jul 2013 16:33:44 -0700 (PDT)

I prefer proposal 1, as proposal 2 may cause undue burden on parsers (they must keep track of all past keys, then we also have to discuss string key comparison which is decidedly not a minimal topic).
 
----------------
-Vinny
www.jsontest.com

________________________________
 From: Paul Hoffman <[email-masked]>
To: "[email-masked] WG" <[email-masked]> 
Sent: Tuesday, July 2, 2013 6:25 PM
Subject: [Json] Proposed minimal change for duplicate names in objects

<chair hats on>

Proposal 1:

Proposal 2:

_______________________________________________
json mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/json

---

From: John Cowan <[email-masked]>
Date: Tue, 2 Jul 2013 21:11:28 -0400

>   name/value pair that has the duplicate name.

I object to this, because it is unclear what "use" means.  If it means
"report to the application", then a streaming parser cannot do this
without ceasing to be streaming: that is, it must buffer up the whole
object in order to be sure not to report any inappropriate pairs.
If "use" means something else, it must be explained more clearly.

>   Add: "The names within an object SHOULD be unique."

I approve of this.  However, an additional sentence is required to explain
which names are distinct and which are not.

>   its outputs MUST fail to finish parsing if it encounters more than one name/value pair with the

This puts only a slightly lower burden on streaming parsers: they now
do not need to buffer up the whole object, but may report it on the fly,
keeping track of the names already seen in order to fail if one of them
is a duplicate.  I still think this burden is excessive.

I therefore propose that nothing be said about parsers, but that generators
SHOULD NOT generate duplicate names, where "duplicate" is defined to mean
consisting of the same code points or scalar values, as the case may be,
in the same order.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is
dramatically overdescribed.  Still other languages are simultaneously
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King

---

From: Nico Williams <[email-masked]>
Date: Tue, 2 Jul 2013 22:30:46 -0500

> Do either or both of these proposals work for people in the WG?

No way.  The threads on this were long, I know, but it should be clear
that streaming parsers cannot even detect that there are duplicates,
much less do something about them.

In short, for streaming parsers (and generators) there's nothing we can do.

What we can do is RECOMMEND that a) generators not produce duplicates
(and explain how streaming ones cannot prevent dups), and b) that
parsers use the last name (and explain how streaming ones will produce
all dups).  (We might as well point out that some parsers produce the
first name, but I don't mind saying those are on the wrong side of
this SHOULD.)

Nico
--

---

From: Stefan Drees <[email-masked]>
Date: Wed, 03 Jul 2013 07:22:47 +0200

I fully second the below points that John makes.

{"Stefan":true}

---

From: Eliot Lear <[email-masked]>
Date: Wed, 03 Jul 2013 08:35:40 +0200

Nico,
> all dups).

To me this isn't strong enough.  I have some sympathy for both the
existing base and for parsers, but generators as an architectural
component should generate unambiguous output, streaming or otherwise. 
And that follow's Postel's Law:
   
"In general, an implementation must be conservative in its sending
behavior, and liberal in its receiving behavior."

-- RFC 791 Section 3.2

Eliot

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 01:45:17 -0500

> And that follow's Postel's Law:

How can a streaming generator do that?  The API for one might look like:

ObjectStart(context)
ObjectAdd(context, name, value)
...

There's no way a minimal-state streaming generator (but I repeat
myself; the whole point of a streaming interface is to keep minimal
state :) can detect duplicates with O(1) state (probabilistic
structures with false positives won't be acceptable either).

> behavior, and liberal in its receiving behavior."

I've seen this maligned too.  How many times have we heard this as a
root cause of security vulnerabilities on receiver sides?  We should
be much more exacting, but in *this* case we can't be.

It's not a question of what is ideal...  If we were starting from
scratch we might forbid O(1) state generators and even parsers.  But
it's too late for that and we have consensus to achieve.

BTW, we're retreading.  We should try not to, but it's difficult to
avoid as no one can have read *every* post since the WG's inception.
I think it's fair to say that on this issue there's no consensus to
ban the bad behavior (duplicate names).  Please don't blame me for
that lack of consensus.

Nico
--

---

From: Eliot Lear <[email-masked]>
Date: Wed, 03 Jul 2013 08:57:22 +0200

Hi Nico,

> structures with false positives won't be acceptable either).

You've got to retain state.  You don't have to retain the value, but you
do have to retain the name.  It's O(k), and knowing you I know you're
smart enough to approach that (because I am and so I'm using inductive
theory ;-).  And I'll bet you a nickel that this is not going to be
anyone's high order bit.
> be much more exacting, but in *this* case we can't be.

That's a fair point about security, but even in THAT context one should
be conservative in what one sends.  And yes we CAN be more exacting, but
at a cost.  What we're debating is the cost of ambiguity.

> it's too late for that and we have consensus to achieve.

Nothing to be done about installed base, as I wrote.  Let's write down
what the right thing to do is.  These are RFCs we're generating, now laws.
> ban the bad behavior (duplicate names).  Please don't blame me for

WGs often retread.  Sometimes we find ways to get around our blocks.  We
both have no time for blame.

Eliot

---

From: Stefan Drees <[email-masked]>
Date: Wed, 03 Jul 2013 10:09:37 +0200

> ban the bad behavior (duplicate names). ...

JSON in the role of data interchange format allows at that point the 
multiset use case. Bad behaviour IMO is when misusing this.

The expectation, that the unordered list of names in a single object 
must form a set and not a multiset is questionable.

Just because everyone and her dog "blindly" feeds this into a keyed map 
structure does not mean, the format is to blame I think :-)

{"Stefan":"male","Stefan":"native German","could not resist":true}

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 3 Jul 2013 10:31:09 +0200

> Do either or both of these proposals work for people in the WG?

I would be fine with proposal 1.

I think we should really ask ourselves what we would do if we could start
with a clean slate and try "approximate" that by tweaking the RFC in that
direction. There may be *some* situations where we have to fall back to say
how it should be done (no duplicates names in objects) but acknowledging
that, due to historic reasons, there are implementations which do it
differently. If we do that too often, however, I don't see any value in this
exercise and we should stop it.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 3 Jul 2013 08:15:14 -0700

<chair hats on>

> much less do something about them.

The earlier threads indicated that some people felt streaming parsers could not retain the names they had emitted, while others didn't. 

> In short, for streaming parsers (and generators) there's nothing we can do.

The more recent threads indicated that predictable interoperability was important to many people on the list.

The purpose of this thread, and its companion, is to propose changes to the spec that would be to see if there is rough consensus for proposals that bring predictable interoperability. If such rough consensus is not possible, that is valuable input for the Working Group.

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 11:04:50 -0500

> theory ;-).  And I'll bet you a nickel that this is not going to be

You'd only have to retain state to satisfy this up-till-now
non-existent requirement.

> at a cost.  What we're debating is the cost of ambiguity.

It's a psychological cost too.  No one wants their code rendered
non-compliant by a new RFC.  That's part of what's making it hard to
get consensus.

To be clear I have no implementations that would be broken (because I
own no implementations, but I've worked on two that also wouldn't be
broken) by any of the proposed changes.  But I have written a
streaming XML encoder, for example, and I wouldn't want to have to
write O(N) state streaming encoders for JSON.

> what the right thing to do is.  These are RFCs we're generating, now laws.

There's no IETF police, that's for sure, but these are law-like
documents.  We should write ones that people can strive to meet.

*My* opinion here is that in the case of streaming generators and
parsers the burden of avoiding/dealing with duplicate names should be
one layer higher: at the applications using them.  But it's also my
opinion that some such apps may not be able to do anything about this
either, which is why my proposal is we write SHOULD and explain when
it's OK to not stick to the SHOULD.

> both have no time for blame.

Understood.  I was voicing what I thought the consensus was.  I wanted
to make sure that was clear.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 12:13:12 -0400

Paul Hoffman scripsit:

> The earlier threads indicated that some people felt streaming parsers
> could not retain the names they had emitted, while others didn't.

It seems to me that a little practical work is in order.  Someone needs
to try out the eight streaming parsers I mentioned, and any others that
they know of, to find out which, if any, (a) report errors on input
containing duplicate names or (b) report only the first value named by a
duplicate name.  A parser that guarantees to return only the last value
cannot, per definitionem, be a streaming parser.

Unfortunately, I have my hands full with Scheme (45 implementations,
anybody?)  and can't take this on.

I will iterate, like Cato the Censor, that we cannot avoid talking
about name equality even if we make no changes in this part of 4627,
since it already says (2.2) that the names within an object SHOULD
be unique.  A unique name is one that is not equal to any other name,
and if this is to mean anything, we have to know what name equality is.
In practice it is clear that name equality is defined as "the same code
points [not units!] in the same order", but this isn't said anywhere.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
The present impossibility of giving a scientific explanation is no proof
that there is no scientific explanation. The unexplained is not to be
identified with the unexplainable, and the strange and extraordinary
nature of a fact is not a justification for attributing it to powers
above nature.  --The Catholic Encyclopedia, s.v. "telepathy" (1913)

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 12:27:29 -0400

> > theory ;-).  And I'll bet you a nickel that this is not going to be

I do not understand the third sentence.  If k is a constant, then O(k) is
the same as O(1), and it is palpable that you cannot pack all the names
of a JSON object into O(1) space, since the amount you need varies as the
product of the (mean) length of the names and the number of names.  As
such, it is not even O(n).  You can get an O(n) probabilistic implementation
by retaining fixed-length hashes of the names already seen, though.

> non-existent requirement.

Quite so.

> non-compliant by a new RFC.  That's part of what's making it hard to

True, but we gave way to that feeling, we'd never have any new standards,
as the first thing a standard does is to break existing implementations.
("The first thing a principle does is kill somebody."  --Peter Wimsey)
We must simply take the view of the ANSI C Rationale, which wisely said
"Existing code [meaning user code, the equivalent of JSON data] is
important; existing implementations are not."

> either,

+1

> which is why my proposal is we write SHOULD and explain when

In that case, we should write a conditional MUST.  We use SHOULD when
we want to provide scope for implementer judgement.  But that is an
editorial detail.

-- 
John Cowan              http://www.ccil.org/~cowan      [email-masked]
"After all, would you consider a man without honor wealthy, even if his
Dinar laid end to end would reach from here to the Temple of Toplat?"
"No, I wouldn't", the beggar replied.  "Why is that?" the Master asked.
"A Dinar doesn't go very far these days, Master.        --Kehlog Albran
Besides, the Temple of Toplat is across the street."      The Profit

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 12:58:10 -0400

> must form a set and not a multiset is questionable.

Nevertheless, anyone who generates JSON with duplicate names in order to
represent a multimap deserves to lose, because many, many JSON parsers
will simply throw away all but the last as an unintended consequence of
their implementation strategies.

I assume you mean a multimap rather than a multiset: a multiset would be
most naturally represented as a JSON object whose values are integers
representing the number of occurrences of a particular object in the
multiset.  By the same token, the natural representation of a multiset
is a JSON object whose values are arrays. whose order is semantically
irrelevant.

> {"Stefan":"male","Stefan":"native German","could not resist":true}

{
  "given name" : "John",
  "gender" : "male",
  "citizenship" : {"country" : "US", "source" : "jus soli"},
  "ethnicity" : ["German", "Irish"],
  "annotations" : {
    "verbose" : true,
    "machine-processable" : true
  }
}

-- 
John Cowan                              [email-masked]
            http://www.ccil.org/~cowan
Humpty Dump Dublin squeaks through his norse
                Humpty Dump Dublin hath a horrible vorse
But for all his kinks English / And his irismanx brogues
                Humpty Dump Dublin's grandada of all rogues.  --Cousin James

---

From: Eliot Lear <[email-masked]>
Date: Wed, 03 Jul 2013 19:00:24 +0200

On 7/3/13 6:27 PM, John Cowan wrote:
> product of the (mean) length of the names and the number of names.  

I was thinking computational complexity not storage.

Eliot

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 13:03:05 -0400

> I was thinking computational complexity not storage.

Oh.  But surely the whole point of streaming parsers is to reduce
the amount of storage required?

-- 
I could dance with you till the cows            John Cowan
come home.  On second thought, I'd              http://www.ccil.org/~cowan
rather dance with the cows when you             [email-masked]
come home.  --Rufus T. Firefly

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 3 Jul 2013 10:19:15 -0700

<no hat>

> the amount of storage required?

In the work I have been doing on CBOR, I have heard both as reasons for different types of streaming parsers and encoders.

It would be unwise for this WG to assume we know the design considerations for any JSON parser or encoder. There are, as we now see, a zillion of them with overlapping design goals and constraints.

--Paul Hoffman

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 03 Jul 2013 19:26:49 +0200

>cannot, per definitionem, be a streaming parser.

http://www.ietf.org/mail-archive/web/json/current/msg00480.html has code
testing Microsoft's .NET Framework's System.Runtime.Serialization.Json.
It reports duplicate instances and does not report any errors for them.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 3 Jul 2013 12:45:48 -0700

> theory ;-).  And I'll bet you a nickel that this is not going to be

There is difference between retaining parent path (names of "open" Object
properties), and retaining names of all preceding siblings. Latter is
expensive and/or complicated.
Former is indeed done by parsers that try to give useful error information
with respect to nesting problems and such.

I also think that probabilistic approaches would be problematic from user
POV: one has to allocate lots of memory -- not just for hashes for names
seen (or derivative thereof like Bloom filter(s)), but also locations for
error messages.

If spec did mandate requirement to check for dups, then both generators and
parsers would have to do it; and at least parsers would need additional
metadata to give proper error messages to indicate nature of collision
(what, where).

-+ Tatu +-

---

From: Pete Resnick <[email-masked]>
Date: Wed, 3 Jul 2013 14:48:51 -0500

Speaking strictly as a participant, and just to give guidance, not 
insist on an outcome.

> component should generate unambiguous output, streaming or otherwise.

Let's remember the meaning of these capitalized words in IETF parlance. 
We use them "where it is actually required for interoperation or to 
limit behavior which has potential for causing harm." So a MUST would 
mean that not doing so would prevent interoperation or cause harm. 
RECOMMENDED (like SHOULD) would also mean that not doing so would 
prevent interoperation or cause harm, but "that there may exist valid 
reasons in particular circumstances to ignore a particular item, but the 
full implications must be understood and carefully weighed before 
choosing a different course."

In this instance, saying that generators SHOULD not produce duplicates 
is to say that if you do produce them, you might not interoperate or you 
might cause harm to some parsers, but there might be valid reasons 
(i.e., because your storage or compute footprint is too small to retain 
state) to produce duplicates. And documenting that there may be reasons 
that generators will produce duplicates (and explicitly stating those 
reasons) certainly puts parsers on notice that they will need to deal 
with it. Giving parsers some guidance on what to do in that case is all 
the better.

So I don't think saying RECOMMENDED or SHOULD regarding not producing 
duplicates should be seen as a non-starter. That's a defensible position 
for the WG to take.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 16:11:43 -0400

Pete Resnick scripsit:

> So I don't think saying RECOMMENDED or SHOULD regarding not
> producing duplicates should be seen as a non-starter. That's a
> defensible position for the WG to take.

+1.  The only person saying otherwise right now is Tim, and he grants
that he is only concerned with a single use case.

-- 
Overhead, without any fuss, the stars were going out.
        --Arthur C. Clarke, "The Nine Billion Names of God"
                John Cowan <[email-masked]>

---

From: Eliot Lear <[email-masked]>
Date: Wed, 03 Jul 2013 22:29:04 +0200

> Speaking strictly as a participant, and just to give guidance, not

Same.

>
> Let's remember the meaning of these capitalized words in IETF
> parlance. We use them "where it is actually required for
> interoperation or to limit behavior which has potential for causing
> harm." So a MUST would mean that not doing so would prevent
> interoperation or cause harm. RECOMMENDED (like SHOULD) would also
> mean that not doing so would prevent interoperation or cause harm, but
> "that there may exist valid reasons in particular circumstances to
> ignore a particular item, but the full implications must be understood
> and carefully weighed before choosing a different course."

I have to say that ambiguity of a language implies an introduction to
interoperability problems.  But MUST we use MUST in those cases?   I
would prefer the catch the problem when we have the opportunity.  If the
WG feels differently, Meh.  As Nico said, deal with it at a higher layer.

Eliot

---

From: Tim Bray <[email-masked]>
Date: Wed, 3 Jul 2013 13:44:37 -0700

Hm?  I’m perfectly OK with a minimal edit leaving most things as they are
and just pointing out potential interop gotchas.   JSON is what JSON is.
For a BCP doc or “Internet JSON” profile, it’s a different story.  -T

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 16:37:37 -0500

> WG feels differently, Meh.  As Nico said, deal with it at a higher layer.

I think we're very likely to get consensus for SHOULD language with an
out for streaming generators/parsers pushing the MUST to the
application.

We could force MUST with an out for streaming generators/parsers
pushing the MUST to the application, but it'd leave some implementors
on the rough side of consensus and unhappy.

I'd be very happy either way.

In the case of the application I mentioned object well-formedness is a
feature of the underlying DB so there's never a way to end up with
duplicate names, so using non-conformant generators/parsers is not a
problem: the result would be conformant.  As long as that's OK I'll be
happy.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 16:39:52 -0500

> For a BCP doc or “Internet JSON” profile, it’s a different story.  -T

I think John thought that you wanted a strict "MUST NOT have dup
names" requirement, but if you'd just settle for an out for streaming
generators/parsers, pushing the non-dup name responsibility up a layer
(or three) then maybe we could just all agree.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 3 Jul 2013 14:43:18 -0700

There’s no need to mention streaming; to start with, we’d have to define a
bunch of terms.

Say, as the RFC does, that you SHOULD NOT emit duplicates because it leads
to variable results, and also note that there is software in the world that
nonetheless does, possibly note that ECMA allows it, and say that receiving
software SHOULD do last-dupe-wins.  JSON is what JSON is.

For “Internet JSON”, it’s a different argument. -T

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 23:27:01 -0400

Tatu Saloranta scripsit:

> properties), and retaining names of all preceding siblings. 

Indeed, you need to hold the names not only of preceding siblings, but of
preceding siblings of all ancestral objects as well, though only the former
are searched for conflicts.

> with respect to nesting problems and such.

No doubt, but nobody is going to *mandate* good error messages: that is QOI,
and in some implementations (space constrained) you don't even want them.

-- 
But the next day there came no dawn,            John Cowan
and the Grey Company passed on into the         [email-masked]
darkness of the Storm of Mordor and were        http://www.ccil.org/~cowan
lost to mortal sight; but the Dead
followed them.          --"The Passing of the Grey Company"

---

From: Stefan Drees <[email-masked]>
Date: Thu, 04 Jul 2013 07:38:14 +0200

> their implementation strategies.

"lose" what? If this "anyone" (it) wants a self-describing robust form 
to interchange data, it will annotate it with metadata accordingly, 
publish the dictionary, or choose microxml or the like.

We might share the view, that many applications exchanging data do also 
share information that is not inside the data. It may well be a one to 
one communication, not necessarily a one to many.

A say sensor automat observing some features, serializing the extracted 
info into JSON and pushing this to another collecting automat first 
finds some name for the subject ("Stefan") than estimates from 
processing further, that this should be tagged with attribute A 
("male"), starts a json object (to wrap it), enters the first member 
"Stefan":"male", eventualy pushes the json text part upstream, processes 
next observation B "native German", notes it linked to the same subject 
as member "Stefan": "native German" and pushes ...

Why not leave this possibility open, more human like bots might act this 
way very conveniently, mixing some serial processing somewhere along the 
way.

Even I sometimes overwrite observations on other people in my personal 
storage ...

> is a JSON object whose values are arrays. whose order is semantically

hm, no, I wanted to focus on the "no-keyspace" not on what is associated 
with the names. Often a boundary condition enforces wrapping inside an 
object. It is often natural for me (as a person) to express my thoughts 
and feelings, but in some situation I better do not do it naturally.

>      "machine-processable" : true

That's very nice, John, thanks. But you gave it some thought, 
backtracked mayb, edited and it is very deep nested for a record missing 
the family name ;-) That could be some digested output of the receiving 
end in my "HumBotTalk" scenario above.

My "signature" was adhoc, expecting the JSON "reader" to strip the 
"last" member. The use case (described in more detail above) is more of 
the kind: Tag with a variable whatever (sha256 of an image?), associate 
predefined attributes with this "whatever" and push upstream as the 
analyzer detects.

{"Stefan":{"Greetings":true}}

---

From: Matthew Morley <[email-masked]>
Date: Thu, 4 Jul 2013 15:47:43 -0400

>   its outputs MUST fail to finish parsing if it encounters more than one

#1 I guess. Duplicate keys *SHOULD* fail, IMHO.

I feel like I am being disruptive by posting this, but I really think we
are going down a path of splitting hairs and missing the broader point of
this issue. All the same, I'm posting as a reply to the initial thread
post, instead of all along the chain of replies.

Perhaps I'm in the minority, but I don't want to see the validation around
this split depending on the style of the decoding code. "Use the last
value, except if you stream then fail" seems a strange position to hold.
Failure should be a correct behavior in general.

The meaning of *SHOULD* for unique keys within the specification provides
enough of an expectation about the behavior, as well as a warning for those
that "weighed" the choice to not follow it. *MUST* would have been better,
but the complications we have been discussing are reasonable to expect and
predict.

Relying on ordered behavior in addition to duplicate keys, within a
knowingly unordered structure that *SHOULD* have unique keys is clearly
grounds for reasonable interop. errors and issues. "Clever" uses for this
small gap in behavior changes do not warrant protection.

Are there really legitimate uses of duplicate keys that should be honored,
warrant this protection, and the level of effort going into this discussion?

As an aside, it seems to me that none of this removes the need for
streaming parsers to be aware of duplicate keys and to probably fail upon
encountering them within the scope of a not-yet-closed object. If valid
json should unique keys, parsers should fail on non-unique keys, stream or
otherwise.

A similar topic comes up with streaming json-rpc servers writers who are
trying this SAX style approach (usually with batch requests), and the
reality has always been that it must be valid json first. This means you
cannot fully process the data with confidence until you confirm the payload
is actually valid json. Doing otherwise is simply betting
against fulfillment of the JSON format. If you can isolate the impact of
the changes, process away until you can confirm, then merge those changes
into the world. Otherwise, you need to be prepared to strike the actions
from having an impact.

Duplicate keys also prevent the use any meaningful path style strings, that
are to be used as an either an reference point or as an identity/location
token within and across objects (JSON Pointer for example). If your json
object produces more than one value for an identity, that is concerning.

Perhaps I'm just in shock that `json` is pondering new wording to handle
something that clearly comes with such complications and disagreement in
practice. Rather than promoting the behavior that is less complicated
conceptually and clearly intended.

-- 
Matthew P. C. Morley

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 4 Jul 2013 15:17:21 -0700

> on the rough side of consensus and unhappy.
One problem I have had is assumption that we only have 2 layers
(application, parser/generator). While it simplifies language, it makes it
difficult to define whether combination of parser and deserializer (as well
as serializer and generator) should be considered "parser" (and
"generator") in specifications or not.
Perhaps "application" here would then refer to serialization library and
application code?

I would be happy with "SHOULD" at parser/generator level, and even "MUST"
for serializer/deserializer level, since those are cases that are easy to
support efficiently.

At the same time, given such separation of processing layers, nothing
prevents applications from doing custom serialization/deserialization, and
by-passing strict checks. I don't personally mind that -- as long as
ramifications are made clear, it's their responsibility -- but I don't
think is universally agreed view.

-+ Tatu +-

---

From: Nico Williams <[email-masked]>
Date: Thu, 4 Jul 2013 21:52:02 -0500

On Thu, Jul 4, 2013 at 5:17 PM, Tatu Saloranta <[email-masked]> wrote:
> as serializer and generator) should be considered "parser" (and "generator")
> in specifications or not.

There can be many layers.

One application I work with is a Ruby on Rails app with a DB
underneath.  In one client I have to read a very large subset of the
DB and the run with that and an audit feed from the same DB.  A JSON
streaming generator and parser that ignored duplicate names would work
just fine because schema is enforced on the *write* side and so there
can be no duplicate names, and *nothing* but the write side need ever
concern itself with duplicate names.  There's quite a few moving
pieces and layers in this one app, but only one piece that ensures
properly-formed data -- outside that piece there is no relevance to
duplicate name handling for there can't be duplicate names.

I don't think it'd be critical to wordsmith the RFC so that this app
can work with streaming generators/parsers that are oblivious to
duplicate names, but it'd be nice to.  One way to do it is to use the
passive voice.  Otherwise there might have to be a [what do we call
this?] consideration for application deverlopers using streaming
encoders/parsers that are duplicate name oblivious.  There's a
trade-off between writing very generic text (but in a passive voice)
and having to cover a number [possibly lots] of specific cases.

> for serializer/deserializer level, since those are cases that are easy to

That's not really enough for my use case.

> ramifications are made clear, it's their responsibility -- but I don't think
> is universally agreed view.

Right, "strict checks" somewhere.  Just not necessarily in the
generator nor parser.

Nico
--

---

From: Jorge <[email-masked]>
Date: Fri, 5 Jul 2013 11:47:50 +0200

On 05/07/2013, at 00:17, Tatu Saloranta wrote:
> 
> At the same time, given such separation of processing layers, nothing prevents applications from doing custom serialization/deserialization, and by-passing strict checks. I don't personally mind that -- as long as ramifications are made clear, it's their responsibility -- but I don't think is universally agreed view.

I've been using JSON in a server in a language that doesn't have objects... so for the outgoing JSON texts I've had to write a custom generator for every kind of payload, manually grabbing the data from a predefined set of vars, and idem for the incoming JSON texts: .parse()ing had to fill a different predefined set of vars for each kind of payload.

This is one of those scenarios in which JSON text parsers and generators have to be ad hoc.
-- 
( Jorge )();

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 05 Jul 2013 16:11:31 +0200

>post, instead of all along the chain of replies.

The broader point here is that the Working Group is supposed to document
the JSON format based on running code and rough consensus and changes to
what is in RFC 4627 require very strong justification and support. That
means duplicate names in objects will be permitted and parsers will not
be required to detect and report duplicate names.

Relying on duplicated names is a bad practise, and for parsers that
report only one of the names, it is a good practise to pick the last in-
stance, and of course parsers are always free to reject any content for
security reasons as they see fit, this's all largely agreed in the group
and people do not care terribly much how the specification conveys this.

I for instance would prefer telling people that picking the last value
is the logical and typical choice and the behavior of standard APIs like
JSON.parse and picking the first one or a random one instead is sure to
cause interoperability, security, migration, and other headaches -- over
fiddling with RFC 2119 keywords, but in the end it does not really have
an effect on how people are going to use and implement JSON in the wild.

We know all the open questions, as far as I am concerned we could just
write them all down and survey the Working Group; one implied question
above, as an example, is

  JSON parsers that treat objects with duplicate keys as malformed
  for security reasons should be made non-conforming in JSONbis.

I would strongly disagree with that and am quite sure other participants
would do the same; in studying the survey results the chairs might find
there is indeed a rough consensus JSON parsers should be allowed to re-
ject such JSON documents for security reasons, and then task editors or
whoever volunteers with implementing that in the specification prose.
Same for all the other issues, say

  Allow Unicode Signature in UTF-8 encoded application/json entities.

  Allow Strings at the top level.

  ...

Then we would be getting somewhere...
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Jim Schaad" <[email-masked]>
Date: Sat, 6 Jul 2013 16:18:17 -0700

And the JOSE working group.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> John Cowan
> Sent: Wednesday, July 03, 2013 1:12 PM
> To: Pete Resnick
> Cc: Nico Williams; Paul Hoffman; Eliot Lear; [email-masked] WG
> Subject: Re: [Json] Proposed minimal change for duplicate names in objects
> 
> Pete Resnick scripsit:
> 
> producing
> > duplicates should be seen as a non-starter. That's a defensible
> > position for the WG to take.
>

---

From: "Jim Schaad" <[email-masked]>
Date: Sat, 6 Jul 2013 17:13:22 -0700

Further clarification (on request)

Tim is not the only person saying this.  The JOSE working group has this as
a requirement to be enforced in the documents it is producing.
Specification that JOSE objects MUST fail validation if there are duplicate
names in an object.

> Jim Schaad
> Sent: Saturday, July 06, 2013 4:18 PM
> To: 'John Cowan'; 'Pete Resnick'
> Cc: 'Nico Williams'; 'Paul Hoffman'; 'Eliot Lear'; [email-masked]
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of John Cowan

---

From: Nico Williams <[email-masked]>
Date: Sat, 6 Jul 2013 20:20:20 -0500

> Specification that JOSE objects MUST fail validation if there are duplicate

But does it follow (did you even mean to say) that since JOSE wants
requirement X (no dup names here) JSON should have it itself?

One might argue that otherwise JOSE implementations would have to use
JSON parsers that implement JOSE's requirement even though JSON
doesn't have the same requirement.  But then. JOSE implementations
also could use streaming JSON parsers and check for dup names
themselves.  So I don't think JOSE's requirement adds anything new to
the discussion.

We still need to decide (directly or indirectly) whether to impose dup
name checking on all JSON parsers, even minimal-state streaming
parsers, whether we want to impose a requirement on the parser and the
application (so we need not mention streaming), or on the parser if
it's not a streaming parser *and* the application if the parser is
streaming.

My view (for whatever it counts) is that we don't have consensus [yet]
for imposing a requirement on parsers to reject objects with dup
names.  We've had at least a number of examples of streaming parsers
that cannot implement such a requirement -- the whole point of
streaming being nullified by state-keeping requirements like this one.
 So we'd have to explicitly decide that we don't want to allow minimal
state streaming parsers.  Might as well call for consensus on that.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Sat, 6 Jul 2013 18:44:07 -0700

This feels like a no-brainer to me, but that’s probably because (as I’ve
said before) I’m an API guy, and the only use for JSON objects in my world
is to transfer a hash table or database record or whatever from here to
there, and there, and in such a situation dupes can never be useful or
intended and can only be a symptom of breakage (or, in the JOSE case, a
symptom of a malicious attack on my crypto).

-Tim

---

From: Nico Williams <[email-masked]>
Date: Sat, 6 Jul 2013 21:57:52 -0500

> symptom of a malicious attack on my crypto).

I agree.  As a security guy I would prefer if one way or another we
end up with no dup names, but as an "API guy" myself I think of the
streaming parsers (they offer an API after all).  Just say the magic
words: "to hell with minimal state streaming parsers" or perhaps
something to the effect that *some* component of a layered application
MUST reject objects with dup names.  It's either or.  Let's choose.

I'm happy with "some component of a layered application MUST reject
objects with duplicate names" -- I prefer this to the "no minimal
state streaming parsers" alternative.

I will assume that in general objects rarely have lots of names, so
that parsers need not keep that much state in order to check for dups.
 Requiring parsers to reject objects with dup names is my second
choice.

Nico
--

---

From: Tatu Saloranta <[email-masked]>
Date: Sat, 6 Jul 2013 21:37:19 -0700

>  Requiring parsers to reject objects with dup names is my second

Just to make sure: I also do not have any use for duplicates, and consider
them flaws in processing. I have never used duplicates for anything, nor
find that interesting approach.
The only concern really is that of mandating (or not) detection and/or
prevention at lowest level components of commonly used processing stacks
(low-level push/pull parser, higher-level library or app code that builds
full representations), since this is significant cost, based on extensive
profiling I have done at this level.

Case of application code directly using streaming parser/generators is not
nearly as common as that of frameworks using them to produce higher level
abstractions.
These higher level abstraction (JSON tree representation, binding to native
objects) do either report errors such as duplicates, and at very least can
detect it and use consistent handling. They can do it much more efficiently
than low-level components since they have to build representations that
then serve as data structures for detecting/preventing duplicates.

Specific concern for me is this: if specification mandates detection and/or
prevention for parser, generators, without any mention that 'parser' and
'generator' are logical concepts (thing that reads JSON, thing that writes
JSON), I will have lots of users who promptly demand low-level components
to force checks.
And then I get to spend lots of time discussing why such checks can (and
IMO should) still be pushed to higher level of processing. It is amazing
how much FUD can be generated based on cursory reading of specifications.

This concern extends to suggested "Internet JSON messages" specification.

I would like simple suggestion that some component of the processing
should/must detect and report duplicates; and prevent producing of same;
or, lengthier explanation of what "parser" and "generator" mean (parser is
such a horrible misnomer -- there is very very little parsing involved,
it's just a lexer, and optional object builder -- but I digress).

-+ Tatu +-

---

From: Tim Bray <[email-masked]>
Date: Sat, 6 Jul 2013 21:58:24 -0700

I’ll assume you’re right when you say dupe detection has been measured as
expensive at run time, but I’m thinking that if I were writing a reader I'd
implement a hash table with a test-and-set method, so I admit I'm surprised
by the finding.  I think I’d need to see a little more research before I’d
accept that as a given.  -T

On Sat, Jul 6, 2013 at 9:37 PM, Tatu Saloranta <[email-masked]> wrote:

> On Sat, Jul 6, 2013 at 7:57 PM, Nico Williams <[email-masked]>wrote:
>
> These higher level abstraction (JSON tree representation, binding to
> native objects) do either report errors such as duplicates, and at very
> least can detect it and use consistent handling. They can do it much more
> efficiently than low-level components since they have to build
> representations that then serve as data structures for detecting/preventing
> duplicates.
>
> Specific concern for me is this: if specification mandates detection
> and/or prevention for parser, generators, without any mention that 'parser'
> and 'generator' are logical concepts (thing that reads JSON, thing that
> writes JSON), I will have lots of users who promptly demand low-level
> components to force checks.

---

From: "Jim Schaad" <[email-masked]>
Date: Sat, 6 Jul 2013 22:44:08 -0700

To be perfectly honest,  I would personally argue that this is a requirement that parsers need to enforce.  The JOSE group is going to need some interesting text about what happens if it is using a parser that does not enforce this mechanism.  And an even more interesting question of how this would ever be determined if the parser is not part of the application itself.  That is how does a JOSE application know that the built in (assuming there is one) parser for the FOO language/environment it is operating in will correctly do the detection for it and not use the rule of grab the last value.

I would also be open to the argument that a streaming parser is not really an entire parser and the thing that consumes the output of a streaming parser could be the entity that enforces the uniqueness requirement.  

Jim

> From: Nico Williams [mailto:[email-masked]]
> Sent: Saturday, July 06, 2013 6:20 PM
> To: [email-masked]
> Cc: Jim Schaad
> > Tim is not the only person saying this.  The JOSE working group has
> > this as a requirement to be enforced in the documents it is producing.
> > Specification that JOSE objects MUST fail validation if there are
> > duplicate names in an object.
> 
> JSON parsers that implement JOSE's requirement even though JSON doesn't
> have the same requirement.  But then. JOSE implementations also could use
> streaming JSON parsers and check for dup names themselves.  So I don't
> think JOSE's requirement adds anything new to the discussion.
> 
> We still need to decide (directly or indirectly) whether to impose dup name
> checking on all JSON parsers, even minimal-state streaming parsers,
> whether we want to impose a requirement on the parser and the application
> (so we need not mention streaming), or on the parser if it's not a streaming
> parser *and* the application if the parser is streaming.
> 
> My view (for whatever it counts) is that we don't have consensus [yet] for
> imposing a requirement on parsers to reject objects with dup names.  We've
> had at least a number of examples of streaming parsers that cannot
> implement such a requirement -- the whole point of streaming being
> nullified by state-keeping requirements like this one.
>  So we'd have to explicitly decide that we don't want to allow minimal state
> streaming parsers.  Might as well call for consensus on that.
> 
> Nico
> --

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Jul 2013 01:53:38 -0400

What I wish to avoid is a specification that says that having two
fields of the same name in an object is a perfectly okay thing to do.
In other words, such documents for my purposes have to be erroneous,
rather than a legitimate way of encoding multimaps.  Is there anyone
who can't live with that restriction?

If there isn't anyone, then we can talk about whether this is an error
that generators and/or parsers MUST or SHOULD detect.  But if we can't
even get consensus on that much, it's futile to discuss parser or
generator behavior.

-- 
Do I contradict myself?                         John Cowan
Very well then, I contradict myself.            [email-masked]
I am large, I contain multitudes.               http://www.ccil.org/~cowan
        --Walt Whitman, Leaves of Grass

---

From: Nico Williams <[email-masked]>
Date: Sun, 7 Jul 2013 01:25:30 -0500

On Sun, Jul 7, 2013 at 12:44 AM, Jim Schaad <[email-masked]> wrote:
> To be perfectly honest,  I would personally argue that this is a
> requirement that parsers need to enforce.  The JOSE group is going to
> need some interesting text about what happens if it is using a parser
> that does not enforce this mechanism.  And an even more interesting

JOSE was going to need that anyways.  If we manage to reach the
consensus you want then JOSE won't need to, but will JOSE wait for
JSON?

> question of how this would ever be determined if the parser is not
> part of the application itself.  That is how does a JOSE application
> know that the built in (assuming there is one) parser for the FOO
> language/environment it is operating in will correctly do the
> detection for it and not use the rule of grab the last value.

I'd say... "read the docs" and "test the parser" :)

(you'd have to no matter what; what if you got a YAML-based parser
that had remote code execution vulnerabilities?  surely you'd want to
avoid that...)

> I would also be open to the argument that a streaming parser is not
> really an entire parser and the thing that consumes the output of a
> streaming parser could be the entity that enforces the uniqueness
> requirement.

Right, that was my proposal.  There's little appetite for describing
parser types in the RFC though.

I'm ambivalent, but willing to go with requiring rejection of objects
with dup names.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sun, 7 Jul 2013 01:25:44 -0500

> accept that as a given.  -T

Dunno about Tatu, but I think the vast majority of JSON usage must be
coupled either with generic hash tables (objects/dicts/maps/whatever)
or structs (with associated schema).  For the latter dup detection may
well be more expensive than not, but probably not that much more
expensive.  For the former dup detection is essentially free.

For other use cases, like Stephen Dolan's "launch missiles" example,
dup detection might well be expensive, but maybe JSON is the wrong
tool for those use cases.

The CPU cost of dup detection doesn't concern me.  It's the
architectural implication that does: minimal state streaming parsers
are out.

I have to ask myself: are there use cases where I would want a minimal
state streaming parser?  and do I mind losing the ability to have such
a thing?  My personal, tentative answers: "yes" and "probably not,
maybe in such a case I should not use JSON".  Others will no doubt
have different answers, which brings us to...

...more retreading.

I still think the best thing to do is note the divergent
interpretations and publish Internet JSON.

Nico
--

---

From: Stefan Drees <[email-masked]>
Date: Sun, 07 Jul 2013 12:22:58 +0200

> who can't live with that restriction?

If the above question is equivalent to:

Does someone speak up, that MUST encode multimaps through members in an 
object, where the names are not pairwise distinct?

I for myself will **not** speeak up. But that is neither necessary nor 
sufficient for this important pre condition of going further as 
described i your second paragraph below.

Maybe it might good to request, that the chairs at this point - or not 
too far in the future - start such a consensus preparing call?

> even get consensus on that much, it's futile to discuss parser or

I need not consider any mandate - like the WG - only the rules for 
participating in these discussions, and I am willing to evolve the JSON 
into a versatile future proof data interchange format, as long as it 
doesn't mutate into the serialization format of just another specific 
language culture.

{"Scene":[{"Me":"All"},{"Dog":"(wags his tail)"},{"Me":"the Best."}]}

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 07 Jul 2013 15:34:46 +0200

>who can't live with that restriction?

The current text is "The names within an object SHOULD be unique". That
makes it clear that you might encounter duplicate names in the wild and
that your parser might not report it when it finds repeated names. That
is the running code situation and so I would oppose making it a "MUST".

>even get consensus on that much, it's futile to discuss parser or

People who would like JSON parsers to detect repeated names in objects
are welcome to implement and deploy and evangelise them; and when other
parsers have become the exception, then it might be useful to capture
that reality in an updated specification. Right now parsers usually do
not detect repeated names and so it would be incorrect to tell people
they can expect parsers to detect repeated names.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 07 Jul 2013 15:38:04 +0200

>symptom of a malicious attack on my crypto).

Designs like http://wiki.apache.org/solr/UpdateJSON#Solr_3.1_Example are
not uncommon.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Jul 2013 16:02:10 +0200

On Jul 7, 2013, at 15:38, Bjoern Hoehrmann <[email-masked]> wrote:

> http://wiki.apache.org/solr/UpdateJSON#Solr_3.1_Example

Well, fixing that misconception is one of the points of this discussion.

After JSON get standardized, people should no longer be able to claim "It is legal in JSON to have duplicate names." with a straight face.  

(It never was, but the requirement was stated with sufficiently soft language that people have simply re-interpreted what was said as something else that happened to be convenient at the time.)

Grüße, Carsten

---

From: Tatu Saloranta <[email-masked]>
Date: Sun, 7 Jul 2013 10:57:47 -0700

> accept that as a given.  -T
Point is that no hash table need be created at all. All one has is simple
tokenizer (lexer). No tree is built, just parent stack with information to
match close markers, report error locations. And bit of state information
to validate well-formedness wrt separators.
I agree that if a per-document symbol table/map is maintained, extra cost
would probably be negligible and not worth worrying too much about.

Hash table will typically be used at higher level, if building an object
representation, such as tree model, or native objects: either as structure
to build (tree), or mapping to object fields (name to method/field). At
this level catching duplicates is easier and more efficient to handle, and
I would not be worried about mandating such handling.

So why separation to layers like this? While writing a naive tokenizer is
an easy task, it takes much more effort to create an efficient one. But
there are not many variations there, so once an efficient one exists, it is
easy to use that as a building block, save time and have some confidence
that simple encoding/decoding works correctly and efficiently.
Conversely, higher level components vary a lot regarding functionality.
This is similar to division between low-level SAX or Stax APIs, and
higher-level DOM (tree) and JAXB (data binding) for XML on Java. I think
C#/dotnet has similar separation between pull-parsing and higher-level
abstractions, but it seems absent from most scripting languages.

As to research, I don't know if this --
https://github.com/eishay/jvm-serializers/wiki -- helps, but contrasting
these two entries:

                           create     ser   deser   total   size  +dfl

  json/jackson/manual           62    1137    1519  2656    468   253
  json/jackson/tree             63    2045    2650  4695    485   259

gives difference between manually written data-binding (streaming parser,
hand-written code to match names to fields), and building a HashMap-based
simple tree model (built on streaming parser) as 2-to-1 performance
difference (2656 nsec vs 4695 nsec).
Part of this is with overhead of Maps vs. plain java objects, but not all.

I could prototype with actual detection too (in fact, there is an RFE for
streaming parser to do just that -- and I consider it a legitimate thing to
do as optional settings) and see how much actual overhead a highly-tuned
minimal-state parser will have on Java platform.

-+ Tatu +-

>
> On Sat, Jul 6, 2013 at 9:37 PM, Tatu Saloranta <[email-masked]>wrote:
>
>> Just to make sure: I also do not have any use for duplicates, and
>> consider them flaws in processing. I have never used duplicates for
>> anything, nor find that interesting approach.
>> Case of application code directly using streaming parser/generators is
>> not nearly as common as that of frameworks using them to produce higher
>> level abstractions.

---

From: Tatu Saloranta <[email-masked]>
Date: Sun, 7 Jul 2013 11:07:53 -0700

> expensive.  For the former dup detection is essentially free.

Yes (for second case, schema typically coming from Java class definitions,
implicitly).
I did not imply that overhead here would be significant.

> have different answers, which brings us to...
I would just note that at this point JSON is very competive,
performance-wise, exceeding performance of other textual formats, and
competing well with binary formats as well. In fact, I rarely see
compelling reason to even consider binary formats, unless payload size is
significant factor.
If this aspect was lost due to clarification for solving a problem that has
more to do with concerns for _possible_ security issues, that would be sad.

End users rarely have real need for minimal-state parsing. It is
framework-builders -- such as, say Solr, Elastic Search, Hadoop, JAX-RS
implementations (these for Java, similar for other platforms) -- that care
as performance implications there have more effect. And they are the ones
that have legitimate use for minimal-state components.

I assume all of above is understood by now, so apologies if I sound like a
broken record here.

> interpretations and publish Internet JSON.
I agree.

-+ Tatu +-

---

From: Stephan Beal <[email-masked]>
Date: Sun, 7 Jul 2013 20:09:36 +0200

> it's not a streaming parser *and* the application if the parser is

FWIW, as a JSON library implementor, i can say that a MUST on the whole
duplicate keys issue is not going to be enforced in any of my impls. Having
to check for an existing key on every insert turns O(N) operations in to
catastrophically poor operation if the property store is not fast (and JSON
has no business specifying performance characteristics). e.g. my JSON
libraries tend to use simple key/value pair _lists_, not maps, because the
nature of JSON is (for most apps) such that it's spit out and slurped in,
but not manipulated as the basic data structure (at least not in C, where
i'm working). Streaming or not, enforcing the implied level of
infrastructure and potential performance hits which MUST  implies is not
something i plan on doing, even if it means not being fully compliant.
"Duplicate keys lead to unspecified behaviour," end of story.

> state streaming parsers.  Might as well call for consensus on that.

i'm for not splitting them into categories, as this document really has no
business (IMO) specifying that level of implementation detail. Specify the
duplicate keys as poor practice leading to
unspecified/implementation-defined behaviour, and be done it.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Jul 2013 14:09:55 -0400

> Part of this is with overhead of Maps vs. plain java objects, but not all.

I'm wondering if the parser interns object names.  That would be a big win
for a manually built system (you can use == comparison to figure out
which field to set, rather than String.equals), but no advantage at all
to a Map-based tree (unless indeed it used IdentityMaps).

-- 
Income tax, if I may be pardoned for saying so,         John Cowan
is a tax on income.  --Lord Macnaghten (1901)           [email-masked]

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Jul 2013 14:11:21 -0400

> unspecified/implementation-defined behaviour, and be done it.

+1 to that.  "Unspecified behavior" is another way of saying "is an error".

-- 
John Cowan            http://www.ccil.org/~cowan     [email-masked]
Uneasy lies the head that wears the Editor's hat! --Eddie Foirbeis Climo

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Jul 2013 20:39:16 +0200

On Jul 7, 2013, at 20:09, John Cowan <[email-masked]> wrote:

> I'm wondering if the parser interns object names. 

The problem with that may be that it leads to an easy DoS (interned strings are never released).

In many JSON designs, the receiver knows the full set of keys ahead of time.  In this case, you are better off with conditionally interning it (i.e., if I already have the symbol, return it, otherwise keep the string around, now we know that it didn't map to a known key).

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Jul 2013 14:43:22 -0400

Carsten Bormann scripsit:

> In many JSON designs, the receiver knows the full set of keys ahead
> of time.  In this case, you are better off with conditionally interning
> it (i.e., if I already have the symbol, return it, otherwise keep the
> string around, now we know that it didn't map to a known key).

I don't know any way to do that in Java, because there seems to be no
way to ask if a string is interned except by interning it.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
I come from under the hill, and under the hills and over the hills my paths
led. And through the air. I am he that walks unseen.  I am the clue-finder,
the web-cutter, the stinging fly. I was chosen for the lucky number.  --Bilbo

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 07 Jul 2013 20:57:34 +0200

>that have legitimate use for minimal-state components.

Note that the performance implications can easily become security impli-
cations. A naive implementation of checking for duplicates may well be
vulnerable to algorithmic complexity attacks causing denial of service
(by using maliciously crafted input that exhibits worst-case behavior).

http://events.ccc.de/congress/2011/Fahrplan/events/4680.en.html for in-
stance surprisingly showed that many common environments had failed to
pay attention when the problem was addressed in Perl eight years prior.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 7 Jul 2013 21:03:12 +0200

> way to ask if a string is interned except by interning it.

No idea about Java, but same problem at the language level in Ruby.
However, at the C API level, there is a conditional intern (rb_check_id()).
Too bad that this isn't exported to the Ruby level.

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 8 Jul 2013 06:35:06 +0000

On 7/7/13 12:09 PM, "Stephan Beal" <[email-masked]> wrote:

>Specify the duplicate keys as poor practice leading to

I went in to this conversation really wanting to move to MUST. I've now
been convinced that strong language about how awful duplicates are is the
best we can do, so +1.

I don't want people to be able to read the text and argue with a straight
face that their protocol is "legal" when it's just poorly designed.

-- 
Joe Hildebrand

---

From: Vinny A <[email-masked]>
Date: Mon, 8 Jul 2013 02:16:39 -0500

On Jul 7, 2013, at 1:09 PM, Stephan Beal <[email-masked]> wrote:

>  Streaming or not, enforcing the implied level of infrastructure and potential performance hits which MUST  implies is not something i plan on doing, even if it means not being fully compliant. "Duplicate keys lead to unspecified behaviour," end of story.
> 

The TL;DR of this discussion seems to be that MUST is too restrictive for some (streaming parsers, overhead, etc) yet SHOULD leaves too much space for ambiguity (as it does in the current  RFC 4627).

I'd like to point this list's attention to a post Eliot made a while ago, where he suggested wording that 1) expresses heavy disapproval on an implementation, 2) still allows for an alternative implementation if (1) is not acceptable, 3) is relatively minimal wording (which seems to be a priority given last week's discussions), and 4) has been used in previous RFCs. Here's the relevant comment:

On Jun 5, 2013, at 3:08 PM, Eliot Lear <[email-masked]> wrote:
> Demonstrating my age, an old trick we've done that goes back to RFC-1123
> is to say MUST NOT send, and then implementations MAY NOT process, and
> if they do MUST process thusly...

Could people come to a consensus on this if the proposed wording was similar to the above? I.e "Implementations MUST NOT send duplicate names; if received, parsers MAY NOT process the JSON, instead issuing an error. If implementations insist on supporting duplicated names, they MUST process in the following manner (insert boilerplate warnings about undesirable behavior)." 

-----------------
Vinny
www.jsontest.com

---

From: Eliot Lear <[email-masked]>
Date: Mon, 08 Jul 2013 09:32:02 +0200

And this is where we simply can't code ourselves out of a paper bag. 
The reason one generally doesn't hardcode rules at the lower layer is
that you don't have visibility as to what the poor practice is.  Here we
do.  ALSO, I'll go one farther.  The generator itself needn't always
keep state.  It simply needs to be told to keep state or not.  If it is
told not to, then the upper layer is assumed to be doing so, but at that
point it is not a matter for the wire protocol, which is where we are.

Eliot

---

From: "Markus Lanthaler" <[email-masked]>
Date: Mon, 8 Jul 2013 10:29:27 +0200

On Monday, July 08, 2013 9:17 AM, Vinny A wrote:
> Could people come to a consensus on this if the proposed wording was
> similar to the above? I.e "Implementations MUST NOT send duplicate names;
> if received, parsers MAY NOT process the JSON, instead issuing an error.
> If implementations insist on supporting duplicated names, they MUST
> process in the following manner (insert boilerplate warnings about
>  undesirable behavior)." 

I personally could certainly live with something like this but the "MUST NOT send" is problematic as such data exists and we shouldn't invalidate it. Changing that to SHOULD NOT would probably be something more people would be equally (un)happy with.

--
Markus Lanthaler
@markuslanthaler

---

From: Stephan Beal <[email-masked]>
Date: Mon, 8 Jul 2013 10:46:44 +0200

On Mon, Jul 8, 2013 at 10:29 AM, Markus Lanthaler
<[email-masked]>wrote:

> I personally could certainly live with something like this but the "MUST
> NOT send" is problematic as such data exists and we shouldn't invalidate
> it. Changing that to SHOULD NOT would probably be something more people
> would be equally (un)happy with.
>

>From me, certainly. i don't see "MUST NOT SEND" as being too helpful
because the receivers need to be able to recognize the existence of dupes
in that case. i.e. it implies a MUST on the consumers, and the "must" on
the consumers is what i find most prickly in this particular topic.

---

From: Eliot Lear <[email-masked]>
Date: Mon, 08 Jul 2013 11:05:02 +0200

On 7/8/13 10:46 AM, Stephan Beal wrote:

> because the receivers need to be able to recognize the existence of
> dupes in that case. i.e. it implies a MUST on the consumers, and the
> "must" on the consumers is what i find most prickly in this particular
> topic.
>

No such implication exists.  Often we say "MUST NOT" to the sender and
then prescribe  "MAY or SHOULD" some behavior (like an error) the
receiver.  There are many reasons for this, but a good one is security
or resource processing.

---

From: John Cowan <[email-masked]>
Date: Mon, 8 Jul 2013 07:55:01 -0400

Vinny A scripsit:

> similar to the above? I.e "Implementations MUST NOT send duplicate
> names; if received, parsers MAY NOT process the JSON, instead issuing
> an error. If implementations insist on supporting duplicated names,
> they MUST process in the following manner (insert boilerplate warnings
> about undesirable behavior)."

"MAY NOT" is not RFC 2119 language, and for good reason: it's ambiguous
between "must not" (as in "You may not leave the room") and "may" ("We
may not go, I don't know yet", which is synonymous with "We may go,
I don't know yet").

-- 
Some people open all the Windows;       John Cowan
wise wives welcome the spring           [email-masked]
by moving the Unix.                     http://www.ccil.org/~cowan
  --ad for Unix Book Units (U.K.)
        (see http://cm.bell-labs.com/cm/cs/who/dmr/unix3image.gif)

---

From: Tatu Saloranta <[email-masked]>
Date: Mon, 8 Jul 2013 16:18:09 -0700

> way to ask if a string is interned except by interning it.

This can be done if caller has a way to indicate set of expected possible
names, which may be the case for data-binding. This also means that actual
intern()ing can be done ahead of time, and potentially decode straight from
byte sequences to intern()ed Strings. Jackson does something similar,
although it goes via general symbol table as reuse coupled with
multi-level, varying vocabularies becomes rather hairy thing to manage
(multi-level symbol tables).

Anyway... this works for subset of cases; and many users use dynamically
generated names, open-ended domain.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Mon, 8 Jul 2013 16:24:48 -0700

> pay attention when the problem was addressed in Perl eight years prior.
Yes, good point.

-+ Tatu +-

---

From: John Cowan <[email-masked]>
Date: Fri, 12 Jul 2013 11:35:17 -0400

> publish the dictionary, or choose microxml or the like.

"Deserves to lose" is an idiom meaning "deserves the unfortunate
consequences of one's actions."

> as member "Stefan": "native German" and pushes ...

My view is that this application design (as opposed, say, to using an
array of single-element objects) is an inappropriate way to use JSON,
depending as it does on a feature (support for duplicate names) that is
not widely provided.

> Why not leave this possibility open, more human like bots might act
> this way very conveniently, mixing some serial processing somewhere
> along the way.

It's not much more convenient than [{"Stefan" : "male}, {"Stefan" :
"native German", ...], which is entirely safe.

> backtracked mayb, edited and it is very deep nested for a record
> missing the family name ;-) That could be some digested output of
> the receiving end in my "HumBotTalk" scenario above.

True.  I wanted it to represent in a machine-processable way all the
information in your version (which is not very machine-processable).