Subject: [Json] Questions regarding duplicate names

From: Pete Resnick <[email-masked]>
Date: Mon, 15 Jul 2013 21:57:50 -0500

There are folks who want the spec to forbid generators from emitting 
objects that contain duplicate names. There are folks who don't. To the 
folks that don't:

     a) Do you actually want a normative statement saying that 
generators MAY produce duplicate names?

     b) Do you want/are you OK with text that would give instructions 
about what parsers should do with duplicate names? If so:

     c) What should parsers do with duplicate names?

Yes/No answers aren't terribly interesting to a) or b). Give some 
explanation.

I'll likely have more questions to follow up on this topic, but these 
seem like a good starting point.

I look forward to your answers.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Tatu Saloranta <[email-masked]>
Date: Mon, 15 Jul 2013 20:22:53 -0700

On Mon, Jul 15, 2013 at 7:57 PM, Pete Resnick <[email-masked]>wrote:

>     a) Do you actually want a normative statement saying that generators
> MAY produce duplicate names?
>
>     b) Do you want/are you OK with text that would give instructions about
> what parsers should do with duplicate names? If so:
>
> I'll likely have more questions to follow up on this topic, but these seem
> like a good starting point.
>
My point is that I do not see any need to produce duplicate names, but that
requirement to detect and prevent such output can be unnecessarily
expensive.

So:

(a) I would prefer SHOULD NOT produce, with a note that this may be either
impractical, or acceptable to be left at discretion of caller (be it
application or higher-level library/framework). Same goes for parsing.

(b) I think it makes sense to suggest acceptable behaviors. But there are
multiple ones, depending on processing style. I would even be ok with
"implementation-dependant" choice.

(c) Two common choices suggested have been "only use last one" and "fail".
But I would add "expose any and all", as that is approach taken my many
libraries, deferring actual handling to caller.

Problem here, again, is that different processing models suggest different
approaches. Full in-memory, whole-document handling processing can
guarantee no duplicates behavior. Open-ended minimal-state parsing,
generation work better with more laissez-faire approach.

-+ Tatu +-

> pr
>
> --
> Pete Resnick<http://www.qualcomm.**com/~presnick/<http://www.qualcomm.com/~presnick/>
> >
> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: Nico Williams <[email-masked]>
Date: Mon, 15 Jul 2013 23:22:31 -0500

On Mon, Jul 15, 2013 at 9:57 PM, Pete Resnick <[email-masked]> wrote:
> There are folks who want the spec to forbid generators from emitting objects
> that contain duplicate names. There are folks who don't. To the folks that
> don't:
>
>     a) Do you actually want a normative statement saying that generators MAY
> produce duplicate names?

No, it should be "SHOULD NOT produce duplicate names", with text
explaining that for some implementation designs there may be no
reasonable way to guarantee this.

If we say "applications SHOULD NOT produce duplicate names" we cover
more cases than if we say "generators SHOULD NOT ..." unless we take
the generator to be or include what I'd call the 'application' in the
streaming generator case.

"MUST NOT" would work for me if the text allows it to be one, the
other, or both, of application or generator in any given use case,
that guarantees no dup names.  Others may not want to go that far.

E.g., that would cover an app I use that can a JSON array with a large
number of objects, possibly via a streaming generator.  The app makes
sure that the data is well-formed, so there is never a case where
duplicate names can arise.  In the general case, however, it may be
that neither the app nor the generator can guarantee a lack of dup
names -- especially the generator.

> what parsers should do with duplicate names? If so:

Not necessarily SHOULD nor MUST, for the reasons that Tatu gave (DoS
attacks, minimal state streaming parsers).

(As an aside, it's worth adding text regarding collision attacks on
hash tables, to the security considerations section.)

>     c) What should parsers do with duplicate names?

Undefined behavior, possibly configurable, with loud security
considerations text warning applications about this.  "Applications
where authorization is separate from interpretation the authorization
module SHOULD/MUST reject inputs with duplicate names (at least if
they are relevant to authorization decisions)."  Or even,
"applications that use JSON values for making authorization decisions
MUST reject inputs with duplicate names".

Nico
--

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 16 Jul 2013 17:15:51 +0200

>     c) What should parsers do with duplicate names?

I think the Working Group has developed a rough consensus to say that
JSON documents with objects with duplicate names are a terrible idea,
but they are not unconditionally in violation of the specification and
no behavior is unconditionally required of recepients if they encounter
such documents, and nobody would mind if the specification points out
things like that many implementations always take the last value. The
RFC2119 voodoo, if any, expressing the above is uninteresting.

JSON is a little bit rough around the edges, as is common with widely
used worse-is-better, good-enough, 80/20 style formats, and lacking
in some areas, as is common with baseline interoperability formats,
and it would be fair to assume that that is part of the reason why it
has been successful; and even if Douglas Crockford had vowed to
eternally curse anyone who ever dared to make documents with duplicate
names or a parser that does not crash and burn when encountering such
documents in a huge blinking box in all JSON documentation ever, the
reality would not be very different from what it is now with respect
to this issue.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Tue, 16 Jul 2013 08:51:22 -0700

> MAY produce duplicate names?

I speak from the constituency that uses JSON in APIs, and the only reason
objects exist is to represent hash tables, database records, and the like.
In these scenarios, dupes can never be useful and are always evidence of
breakage, so please, a big “no”.

> what parsers should do with duplicate names? If so:

I suspect it will be hard to achieve consensus on this, and I’m not sure
how useful the effort is, since it will likely have little effect on
existing implementations.

>     c) What should parsers do with duplicate names?

My preferred course of action would be for the receiver of dupe keys to
treat this as an error and signal the breakage back to the sender.

---

From: Pete Resnick <[email-masked]>
Date: Tue, 16 Jul 2013 10:58:07 -0500

Wow. I must say that I'm very disappointed. We've had a few replying, 
but not answering the questions I asked. Let me start here, and I will 
answer some individual posts in a bit. *Please* try to answer the 
questions. They are designed to *change* the discussion, not repeat more 
of the circles this group has been going around. Please read carefully.

On 7/15/13 9:57 PM, Pete Resnick wrote:
> objects that contain duplicate names. There are folks who don't. To 
> the folks that don't:

Note that I asked for the folks who *don't* object to generators 
emitting duplicate names to answer.

> generators MAY produce duplicate names?

Tatu and Nico say that they want the document to say SHOULD NOT. But 
that's not what I asked. I asked if you thought that it should say MAY. 
I take it from their answers that they meant to say, "No, it should not 
say 'MAY produce duplicate names'". Tim said specifically "No". But I 
suspect that this means that they are folks who *want* the spec to 
forbid generators from emitting duplicate names. Do I have this correct?

(Bjoern made some inappropriate conclusions about the consensus of the 
WG, ones that I suspect are actually false, but didn't answer the 
question at all.)

> about what parsers should do with duplicate names?

Here I did not ask about 2119 language. I asked whether we should be 
giving instructions at all to parsers. Only Tatu answered that we 
should. Tim made some comment about not thinking we will achieve 
consensus to do so, which again doesn't answer the question.

> If so:

Tatu gave a few choices here, but expressed no preference. I'd like to 
hear what you think *should* be done.
Nico gave no indication of what parsers should do.
Tim wants such things rejected as an error.
Bjoern didn't answer the question.

Folks, I'm trying to get the WG to unwedge the conversation. If you 
don't want to participate in the conversations that I'm trying to start, 
that's entirely your choice. But please don't drift onto tangents.

pr

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Jul 2013 11:26:48 -0500

On Tue, Jul 16, 2013 at 10:58 AM, Pete Resnick
<[email-masked]> wrote:
> Tatu and Nico say that they want the document to say SHOULD NOT. But that's
> not what I asked. I asked if you thought that it should say MAY. I take it
> from their answers that they meant to say, "No, it should not say 'MAY
> produce duplicate names'". Tim said specifically "No". But I suspect that
> this means that they are folks who *want* the spec to forbid generators from
> emitting duplicate names. Do I have this correct?

Well, OK, "MAY" is not the same as "SHOULD NOT", but it seemed close
enough because "SHOULD NOT" with text explaining *why* one might
nonetheless go ahead, in this case, seems to cover the reasons for the
MAY.

In any case, you got your answer: no one seems to want that MAY.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Jul 2013 11:46:29 -0500

> In any case, you got your answer: no one seems to want that MAY.

It's been pointed out that no one has spoken up in favor of that MAY *yet*.

I don't think anyone will if there's a suitable construction that says
"SHOULD NOT" that allows whatever it is they do.

However, if we can't reach consensus on such a thing, then I suppose
MAY is better than silence: it will drive the point home.  So perhaps
you can even score me as in support of that MAY, mildly anyways.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 16 Jul 2013 16:50:26 +0000

I'm intentionally not reading the other replies yet, so I apologize if I'm
being redundant.

>generators MAY produce duplicate names?

No.  As far as I can tell, the intended semantic was never that duplicate
names were a goal, notwithstanding the claims of those who don't
understand what "SHOULD" means in a spec that references RFC 2119.

The intended semantic is what needs to be adequately described.

>about what parsers should do with duplicate names? If so:

Yes.  I think that is important for interoperability.

>     c) What should parsers do with duplicate names?

I think they should have some choices:

1) throw an error.  This is important for certain security use cases.
2) use the last one.  This is important for ECMA-262 interoperability.
3) report all of them.  This MUST only be done by streaming parsers that
can't maintain state.  If parser events are reified into a concrete
object, one of the first two choices MUST be made.

Similar choices for generators.  If you can, you MUST prevent duplicates.
There MUST NOT be special APIs for sending duplicates (e.g. an API input
that takes a multimap).

Regardless of the choice made above, protocol designers MUST NOT rely on a
particular implementation choice, so intentional duplication of keys is
prohibited.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 16 Jul 2013 16:52:40 +0000

>MUST reject inputs with duplicate names".

I like using strong security language as a way of making this more clear.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 16 Jul 2013 16:59:21 +0000

>you can even score me as in support of that MAY, mildly anyways.

Those people should speak up then.  I'm pretty strongly opposed to MAY
without a valid technical argument.

However, arguments that start with "people who can't read 4627 correctly
do it today" haven't yet been compelling to me.

-- 
Joe Hildebrand

---

From: "John Levine" <[email-masked]>
Date: 16 Jul 2013 17:20:33 -0000

>generators MAY produce duplicate names?

No.  MAY means it's OK either way, and dups are clearly bad.

I'd prefer SHOULD NOT, noting that some generators may not have the
necessary buffering to detect or delete dups.

>about what parsers should do with duplicate names? If so:

No.  I'd prefer not to standardize bad behavior.  Perhaps note that
parsers should at least avoid crashing on dups and observe that many
but not all parsers have historically used the last instance so
generators can't count on it.

R's,
John

---

From: John Cowan <[email-masked]>
Date: Tue, 16 Jul 2013 14:42:52 -0400

Pete Resnick scripsit:

> >There are folks who want the spec to forbid generators from
> >emitting objects that contain duplicate names. There are folks who
> >don't. To the folks that don't:
> 

Well, I fall into both camps, because you are equivocating again, not
on the noun "object" this time, but on the verb "object".  I do not
object to generators emitting duplicate names, in the sense that I do
not believe that RFC 4627bis should forbid them to do so.  I do object
to generators emitting duplicate names, in the sense that I believe it
is a very bad idea for them to do so.

(I've been told privately that I am being excessively snarky and
legalistic.  But I believe that anyone who insists on narrow
interpretations to their questions had better ask carefully
stated questions.)

> >generators MAY produce duplicate names?

No, I don't.  That would mean that producing duplicate names is
a worthy thing to do, and I believe it isn't.

> that's not what I asked. I asked if you thought that it should say
> MAY. I take it from their answers that they meant to say, "No, it
> should not say 'MAY produce duplicate names'". Tim said specifically
> "No". But I suspect that this means that they are folks who *want*
> the spec to forbid generators from emitting duplicate names. Do I
> have this correct?

I don't see how you could possibly believe that anyone who says SHOULD
NOT really meant "forbidden", i.e. "MUST NOT".  Unless you believe there
is some shade of difference between saying that something is forbidden
and saying that it MUST NOT be done, in which case I invite you to
explain further.

> >    b) Do you want/are you OK with text that would give
> >instructions about what parsers should do with duplicate names?
> 
> Here I did not ask about 2119 language. 

So here you are distinguishing between "should" and "SHOULD".  Alas, I
do not know how to say what parsers should do (since they are not moral
agents); I only know how to say what they MAY, SHOULD, MUST, SHOULD NOT,
or MUST NOT do.  I believe that (the authors of) RFC 4627bis should not
say that parsers SHOULD do anything in particular in this circumstance.

I believe that (the authors of) RFC 4627bis should say that parsers MAY
report an error, ignore all dupes except the last, report all dupes,
or anything else.

-- 
John Cowan          http://www.ccil.org/~cowan         [email-masked]
The native charset of SMS messages supports English, French, mainland
Scandinavian languages, German, Italian, Spanish with no accents, and
GREEK SHOUTING.  Everything else has to be Unicode, which means you get
only 70 16-bit characters in a text instead of 160 7-bit characters.

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Sat, 27 Jul 2013 20:22:53 +0000

Hello all,

Here is an attempt to summarize responses to the duplicate names questions.

The consensus from this thread seems to be that duplicate names within the same JSON object are not generally desirable.

There appears to be rough consensus that a generator (for some definition of "generator") ought to take steps to avoid emitting duplicate names, but that there exists a non-trivial amount of software where the prohibition of duplicates cannot be completely enforced.

There appears to be consensus that parsers (for some definition of "parser") need to be able to deal with duplicate names in some manner.  There is rough consensus that 4627bis ought to provide guidance on how parsers deal with duplicate names.  However, there is no one option that has any consensus; instead, there appears to be acquiescence that a given parser ought to support one option from a limited set of documented choices.  The most common suggestions -- which appear to reflect the majority of existing implementations -- far so:

* report an error in some manner
* report the last value that appears in the JSON text
* report all the values that appear in the JSON text

- Paul Hoffman and Matt Miller

---

From: Tim Bray <[email-masked]>
Date: Sat, 27 Jul 2013 14:18:51 -0700

Agree with chairs’ outline of the rough consensus. -T

On Sat, Jul 27, 2013 at 1:22 PM, Matt Miller (mamille2)
<[email-masked]>wrote:

> The consensus from this thread seems to be that duplicate names within the
> same JSON object are not generally desirable.
>
> There appears to be rough consensus that a generator (for some definition
> of "generator") ought to take steps to avoid emitting duplicate names, but
> that there exists a non-trivial amount of software where the prohibition of
> duplicates cannot be completely enforced.
>
> There appears to be consensus that parsers (for some definition of
> "parser") need to be able to deal with duplicate names in some manner.
>  There is rough consensus that 4627bis ought to provide guidance on how
> parsers deal with duplicate names.  However, there is no one option that
> has any consensus; instead, there appears to be acquiescence that a given
> parser ought to support one option from a limited set of documented
> choices.  The most common suggestions -- which appear to reflect the
> majority of existing implementations -- far so:
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

---

From: Jorge Chamorro <[email-masked]>
Date: Sun, 28 Jul 2013 00:18:54 +0200

(re-sending, sorry, attached to the wrong thread)

> * report all the values that appear in the JSON text

If duplicate names are generally considered a no-no, and given that they're mostly used for comments (usually with the key "//"), and given the security risks discussed in the "command":"launch-missiles" thread, perhaps the best thing to do with the duplicate keys and their values might be to simply drop them altogether, not throw any errors, not nothing, just drop them, all of them, including the last one.

So missiles won't be launched and comments will vaporize, which is good.
-- 
( Jorge )();

---

From: Jorge Chamorro <[email-masked]>
Date: Sun, 28 Jul 2013 00:28:08 +0200

> So missiles won't be launched and comments will vaporize, which is good.

On the other hand, if this were so, an attacker could *delete* other keys.
-- 
( Jorge )();

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Sat, 27 Jul 2013 23:22:23 +0000

The point is, you can't control what the receiver is going to do, so you
REALLY SHOULD NOT (see: RFC 6919) send them, or you will have
interoperability issues.

Nobody should be able to claim that their comments approach is a valid
protocol; that's not what "SHOULD" means in RFC 4627 or any other spec
that references 2119.

On 7/28/13 12:28 AM, "Jorge Chamorro" <[email-masked]> wrote:

>> If duplicate names are generally considered a no-no, and given that
>>they're mostly used for comments (usually with the key "//"), and given
>>the security risks discussed in the "command":"launch-missiles" thread,
>>perhaps the best thing to do with the duplicate keys and their values
>>might be to simply drop them altogether, not throw any errors, not
>>nothing, just drop them, all of them, including the last one.
>> 

-- 
Joe Hildebrand

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 28 Jul 2013 17:36:05 +0900

Works for me!   Regards,   Martin.

---

From: Stefan Drees <[email-masked]>
Date: Mon, 29 Jul 2013 07:24:17 +0200

On 2013-07-27 22:22 +02:00, Matt Miller (mamille2) wrote (on the behalf 
of Paul Hoffman and Matt Miller):
> The consensus from this thread seems to be that duplicate names within
> the same JSON object are not generally desirable.
>
> There appears to be rough consensus that a generator (for some
> definition of "generator") ought to take steps to avoid emitting
> duplicate names, but that there exists a non-trivial amount of software
> where the prohibition of duplicates cannot be completely enforced.
>
> has any consensus; instead, there appears to be acquiescence that a
> given parser ought to support one option from a limited set of
> documented choices. The most common suggestions -- which appear to
> reflect the majority of existing implementations -- far so:
>

Works for me. Thanks for summarizing, Stefan.

> ...

---

From: "John Levine" <[email-masked]>
Date: 29 Jul 2013 14:28:36 -0000

>Works for me. Thanks for summarizing, Stefan.

Looks right to me, too.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly