Subject: [Json] The names within an object SHOULD be unique.

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 11:33:29 -0700

This was the biggest blunder in the RFC. SHOULD should have been MUST.

It is, sadly, too late to repair this. Instead, we must specify what 
happens when you do the thing you SHOULD NOT do. We need to provide 
implementations some slack here because some implementations do the 
right thing and reject. Some implementations do the lazy thing and take 
that last use of the name.

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 11:38:06 -0700

Or, leave it as is and address it in the best practices doc.
-T
On Jun 5, 2013 11:35 AM, "Douglas Crockford" <[email-masked]> wrote:

> implementations some slack here because some implementations do the right
> thing and reject. Some implementations do the lazy thing and take that last
> use of the name.
> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 11:59:39 -0700

On 6/5/2013 11:38 AM, Tim Bray wrote:
> On Jun 5, 2013 11:35 AM, "Douglas Crockford" <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     It is, sadly, too late to repair this. Instead, we must specify
>     what happens when you do the thing you SHOULD NOT do. We need to
>     provide implementations some slack here because some
>     implementations do the right thing and reject. Some
>     implementations do the lazy thing and take that last use of the name.
>

I like that, except that I am concerned about security exploits that are 
possible if some systems keep the first use and others keep the last. 
HTTP, for example, suffered from this. So I would like to be more 
specific. An implementation SHOULD reject, but if it accepts, it MUST 
take the last use.

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 15:00:26 -0400

Douglas Crockford scripsit:

> It is, sadly, too late to repair this.

I agree that it's too late to repair this on the acceptance side.
But why can't we say that parsers MUST accept such documents but
authors/producers/ generators MUST NOT create them?

> Instead, we must specify what happens when you do the thing you SHOULD
> NOT do.

+1 to that.

> We need to provide implementations some slack here because some
> implementations do the right thing and reject. Some implementations
> do the lazy thing and take that last use of the name.

Or, indeed, the first use.

-- 
They tried to pierce your heart                 John Cowan
with a Morgul-knife that remains in the         http://www.ccil.org/~cowan
wound.  If they had succeeded, you would
become a wraith under the domination of the Dark Lord.         --Gandalf

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 5 Jun 2013 21:17:02 +0200

On Wednesday, June 05, 2013 9:00 PM, Douglas Crockford wrote:
> I like that, except that I am concerned about security exploits that
> are possible if some systems keep the first use and others keep the
> last. HTTP, for example, suffered from this. So I would like to be more

This seems like the best approach to me to address this unfortunate
situation. This is so fundamental (and often discussed) that I think it
really belongs in the specification. It would also make sense to add a clear
interoperability warning and perhaps explain that this "feature" is there
only due to historic reasons.

--
Markus Lanthaler
@markuslanthaler

---

From: Stephan Beal <[email-masked]>
Date: Wed, 5 Jun 2013 21:30:39 +0200

On Wed, Jun 5, 2013 at 9:17 PM, Markus Lanthaler <[email-masked]>
 wrote:

> situation.

@Markus: sorry for the accidental off-list reply. gmail is sending to the
sender by default.

+1

> This is so fundamental (and often discussed) that I think it

i wouldn't quite go so far as to say it's strictly historical in nature. It
gives implementations leeway on what is essentially a corner case (who
actively generates non-unique keys?). Forcing a specific behaviour here
could potentially leave a lot of current 100% JSON-compliant parsers
suddenly non-compliant because of [what i perceive as] an unusual corner
case. Pointing it out as a potential pitfall, sure, but i wouldn't claim
that it's "historical baggage."

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Nico Williams <[email-masked]>
Date: Wed, 5 Jun 2013 14:56:07 -0500

> actively generates non-unique keys?). Forcing a specific behaviour here

If no one does that then we can make a "breaking" change here that
breaks no one, no?  (I'm being the devil's advocate here.)

> that it's "historical baggage."

There's no compliance police here.  If some change merely leaves us
with non-compliant implementations then maybe I don't care, after all
they'd still compliant with the old RFC and could be updated to be
compliant with the new one.  But we mustn't break interoperability any
more than it already was.  If indeed there are senders with duplicate
keys (and different values) that alternately get to talk to receivers
that take the first or last key, then there's already broken interop,
so picking one of the two should be fair game.  But I bet we have
little real data on this, so we'll end up assuming that such a change
can break interop.

Nico
--

---

From: Eliot Lear <[email-masked]>
Date: Wed, 05 Jun 2013 22:08:47 +0200

Hi,

On 6/5/13 8:33 PM, Douglas Crockford wrote:
> right thing and reject. Some implementations do the lazy thing and
> take that last use of the name.

Demonstrating my age, an old trick we've done that goes back to RFC-1123
is to say MUST NOT send, and then implementations MAY NOT process, and
if they do MUST process thusly...

That gives you the possibility of potentially getting rid of something
at a later date.  I don't know if you want to use it here.  It only
sometimes works.

Eliot

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 5 Jun 2013 22:52:08 +0200

On Wednesday, June 05, 2013 9:56 PM, Nico Williams wrote:
> On Wed, Jun 5, 2013 at 2:30 PM, Stephan Beal wrote:

I've seen people using this for comments. People relied on the fact that serializers ignored all but the last occurrence of a key.

> > suddenly non-compliant because of [what i perceive as] an unusual
> > corner case. Pointing it out as a potential pitfall, sure, but i
> > wouldn't claim that it's "historical baggage."
> 

I think we have to choose one to get interoperability. I've seen some systems that merge the values of multiple keys in an array but most do what JavaScript does, i.e., they ignore all but the last key.

I really like the proposal Eliot Lear made in another thread:

> Demonstrating my age, an old trick we've done that goes back to RFC-
> 1123 is to say MUST NOT send, and then implementations MAY NOT process,
> and if they do MUST process thusly...
> 

--
Markus Lanthaler
@markuslanthaler

---

From: Vinny A <[email-masked]>
Date: Wed, 5 Jun 2013 15:59:45 -0500

On Jun 5, 2013, at 2:30 PM, Stephan Beal <[email-masked]> wrote:
> It gives implementations leeway on what is essentially a corner case (who actively generates non-unique keys?). Forcing a specific behaviour here could potentially leave a lot of current 100% JSON-compliant parsers suddenly non-compliant because of [what i perceive as] an unusual corner case.

If parsers are accepting duplicate keys, they're already non-compliant. RFC 4627 states that keys should be unique, and the org.json reference parser throws an exception when it hits a duplicate key (line 1186 in JSONObject).

If any implementation is relying on using duplicate keys, they should also understand that they are already violating best practice.

> if they do MUST process thusly...

While I'd prefer to simply state that implementations MUST generate unique keys, Mr. Lear's suggestion gets my +1.

-----------------
Vinny
www.jsontest.com

---

From: Stephan Beal <[email-masked]>
Date: Wed, 5 Jun 2013 23:14:33 +0200

On Wed, Jun 5, 2013 at 10:59 PM, Vinny A <[email-masked]> wrote:

> If parsers are accepting duplicate keys, they're already non-compliant.
> RFC 4627 states that keys should be unique, and the org.json reference
> parser throws an exception when it hits a duplicate key (line 1186 in
> JSONObject).
>

(Sorry, Vinny - gmail is giving me all kinds of grief with the list
replies.)

"should" is not "must" :/.

> If any implementation is relying on using duplicate keys, they should also
> understand that they are already violating best practice.
>

Absolutely.

> keys, Mr. Lear's suggestion gets my +1.

But that only solves the generator side of the equation, and that solution
would imply that the parser also MUST NOT parse them (because they "can't"
be generated, so why bother?), regardless of best/worse-practice. i.e. it
would force changes onto existing parsers if they want to be officially
compliant. Maybe i'm nit-picking, but that point could certainly disappoint
some people who would have to re-label their implementations as "JSON
comp^H^H^H^Halmost-compliant".

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 14:56:08 -0700

On 6/5/2013 1:59 PM, Vinny A wrote:
>
> On Jun 5, 2013, at 2:30 PM, Stephan Beal <[email-masked] 
> <mailto:[email-masked]>> wrote:
>> It gives implementations leeway on what is essentially a corner case 
>> (who actively generates non-unique keys?). Forcing a specific 
>> behaviour here could potentially leave a lot of current 100% 
>> JSON-compliant parsers suddenly non-compliant because of [what i 
>> perceive as] an unusual corner case.
>
> If parsers are accepting duplicate keys, they're already 
> non-compliant. RFC 4627 states that keys should be unique, and the 
> org.json reference parser throws an exception when it hits a duplicate 
> key (line 1186 in JSONObject).
>
> If any implementation is relying on using duplicate keys, they should 
> also understand that they are already violating best practice.

There are people who interpreted the SHOULD as DON'T HAVE TO and 
intentionally duplicated the keys. My personal feeling is that we should 
change the SHOULD to MUST and let those applications break. But the 
consensus of ECMA TC39 is that the SHOULD must stand.

So regrettably, that means we should be explicit about what happens when 
this particular bad practice is practiced.

---

From: Vinny A <[email-masked]>
Date: Wed, 5 Jun 2013 16:59:22 -0500

On Jun 5, 2013, at 4:14 PM, Stephan Beal <[email-masked]> wrote:
> Maybe i'm nit-picking, but that point could certainly disappoint some people who would have to re-label their implementations as "JSON comp^H^H^H^Halmost-compliant".

I don't think you're nit picking. You bring up a very valid concern. With that said, any choice made is going to disappoint someone out there.

JSON implementations vary wildly in how they deal with duplicate keys (rejecting them, first key only, last key only, transforming duplicated keys into an array) and it gets rather tiresome after a while. Any choice that this mailing list makes is going to break something, but it'll be an improvement for future interoperability. Frankly the actual choice is irrelevant - there are pros/cons to all 4 choices - interop is what matters.

Delaying this discussion until bis document writing is also a good idea.

On Jun 5, 2013, at 4:14 PM, Stephan Beal <[email-masked]> wrote:
> (Sorry, Vinny - gmail is giving me all kinds of grief with the list replies.)
> 

OT: No worries. Yahoo mail is also having difficulty with this list - some emails are getting marked as spam by Yahoo's filters.

-----------------
Vinny
www.jsontest.com

---

From: Vinny A <[email-masked]>
Date: Wed, 5 Jun 2013 17:46:42 -0500

On Jun 5, 2013, at 4:56 PM, Douglas Crockford <[email-masked]> wrote:
> There are people who interpreted the SHOULD as DON'T HAVE TO and intentionally duplicated the keys. My personal feeling is that we should change the SHOULD to MUST and let those applications break. But the consensus of ECMA TC39 is that the SHOULD must stand.

My personal opinion echoes yours. I understand ECMA's consensus though.

On Jun 5, 2013, at 4:56 PM, Douglas Crockford <[email-masked]> wrote:
> So regrettably, that means we should be explicit about what happens when this particular bad practice is practiced.

Do we need to add this to the draft? As in (not proposing this, but as fodder for discussion): "The names within an object SHOULD be unique. If duplicate names are encountered, the last key:value pair MUST be used."

-----------------
Vinny
www.jsontest.com

---

From: Nico Williams <[email-masked]>
Date: Wed, 5 Jun 2013 18:16:55 -0500

On Wed, Jun 5, 2013 at 3:59 PM, Vinny A <[email-masked]> wrote:
> If parsers are accepting duplicate keys, they're already non-compliant. RFC
> 4627 states that keys should be unique, and the org.json reference parser
> throws an exception when it hits a duplicate key (line 1186 in JSONObject).

Not clear.  The RFC uses a passive voice here, not placing a
requirement (or, rather, recommendation) on either the encoder nor the
parser.  It seems more reasonable to read it as a recommendation for
encoders than one for parsers.  I note that the words "reject",
"fail", and "error" don't appear in the RFC, lending even more weight,
I think to the interpretation that this SHOULD applies to encoders.

Perhaps that should be an action item here: change from passive voice
to active as much as possible, with discussion of why some ambiguity
(like this one) is left in, if it is.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 5 Jun 2013 18:19:41 -0500

> I've seen people using this for comments. People relied on the fact that serializers ignored all but the last occurrence of a key.

Oh, this might actually be important.  Perhaps the best way to handle
comments is to have the "schema" (if there is one, stated or
otherwise) specify some keys as being comments, but that doesn't help
for free-form data.

If I were doing this duplicate-keys-for-comments thing and wanted to
ensure interop I'd triplicate them instead, with the value I actually
want decoded sent first and last, and the comment sandwiched in
between.  Painful.

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 16:26:28 -0700

On 6/5/2013 3:46 PM, Vinny A wrote:
>
> On Jun 5, 2013, at 4:56 PM, Douglas Crockford <[email-masked] 
> intentionally duplicated the keys. My personal feeling is that we 
> should change the SHOULD to MUST and let those applications break. But 
> the consensus of ECMA TC39 is that the SHOULD must stand.
>
> On Jun 5, 2013, at 4:56 PM, Douglas Crockford <[email-masked] 
> > So regrettably, that means we should be explicit about what happens 
> when this particular bad practice is practiced.
>
> Do we need to add this to the draft? As in (not proposing this, but as 
> fodder for discussion): "The names within an object SHOULD be unique. 
> If duplicate names are encountered, the last key:value pair MUST be used."
>
It think it should be added to the draft. If duplicate names are 
encountered, the parse MAY fail (because some implementations correctly 
do that), or it MAY succeed by accepting the last duplicated key:value pair.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 5 Jun 2013 23:40:29 +0000

+1

do we have a preference for new software?

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 16:45:12 -0700

Sure, but let’s put that in the best-practices doc.  We’re just defining a
format here. -T

On Wed, Jun 5, 2013 at 4:40 PM, Joe Hildebrand (jhildebr) <
[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Matthew Morley <[email-masked]>
Date: Wed, 5 Jun 2013 21:56:02 -0400

On Wed, Jun 5, 2013 at 7:19 PM, Nico Williams <[email-masked]> wrote:

> > I've seen people using this for comments. People relied on the fact that
> serializers ignored all but the last occurrence of a key.
>

The best way to handle 'comments' is to have them be first class citizens
in the data. It is a topic that goes back a long way, and since they were
removed with intention then, using ambiguity in the specification to re-add
them seems like a poor approach now.

> want decoded sent first and last, and the comment sandwiched in

Like a poorly structure compliment sandwich... best avoided. ;)

-- 
Matthew P. C. Morley

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 5 Jun 2013 19:07:04 -0700

On Wed, Jun 5, 2013 at 1:59 PM, Vinny A <[email-masked]> wrote:

> It gives implementations leeway on what is essentially a corner case (who
> case.
Doing this is fine, but requiring verification is quite another thing. It
opens a new possibility for DoS attacks, if parser (tokenizer) has to
verify this invariant, as it must keep all seen names for all open objects
in memory. From user perspective this also seems like pure overhead; and
non-trivial amount as well, for fast data-binding maintaining hash set may
take as much time as data-binding.

-+ Tatu +-

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 03:44:53 +0000

On 6/5/13 7:56 PM, "Matthew Morley" <[email-masked]> wrote:

>removed with intention then, using ambiguity in the specification to
>re-add them seems like a poor approach
> now.

Do you have a backward-compatible proposal?  I can't think of one, but
that doesn't make it impossible.

-- 
Joe Hildebrand

---

From: Stefan Drees <[email-masked]>
Date: Thu, 06 Jun 2013 07:21:17 +0200

>> do that), or it MAY succeed by accepting the last duplicated key:value

+1 also from me.

> do we have a preference for new software?

(somehow side stepping the "overridden keys as comment usage" fork)

all know that the sequence of keys should not matter when talking JSON, 
but then if you know who you are talking to ... I guess we have 
streaming "interpretations" in the wild, where the producer and consumer 
relation is based on the (out-of-band) assumption, that regardless of 
JSON type the interchange medium behaves like a queue (and not like a 
stack or a shuffle) and also that the ordering of keys at best is 
optimally (top level) application defined, or at least is dependably 
ordered.

So I think gently pushing new implementations to not duplicate names as 
keys in an object would also help these JSON "extenders" to base their 
"private" orderings upon (As these are formulated in terms of names).

Thus I propose append a

"""Implementations are strongly urged to not duplicate names as keys in 
an object."""

(or the like) after the above proposed statement with the two MAYs.

Stefan.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 06:47:31 +0000

>(or the like) after the above proposed statement with the two MAYs.

RFC 6919 has "REALLY SHOULD NOT" for this.

-- 
Joe Hildebrand

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 06 Jun 2013 12:09:25 +0200

>that last use of the name.

I have seen no evidence that the world is going to standardise around
rejecting JSON content with repeated object keys, and consequently do
not support any suggestion in the specification that taking the last
value is worse than rejecting the whole content. Saying that parsers
"MUST either or" is fine though.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 04:18:03 -0700

> "MUST either or" is fine though.
Taking the last is clearly worse, as will be explained in the new 
Security Considerations.

---

From: Tony Hansen <[email-masked]>
Date: Thu, 06 Jun 2013 08:35:37 -0400

Of all the wordings that have been proposed so far, I like Eliot's best.

    Tony Hansen

---

From: Stephan Beal <[email-masked]>
Date: Thu, 6 Jun 2013 16:55:34 +0200

> otherwise) specify some keys as being comments, but that doesn't help

This brings with it a performance penalty because all keys being slurped up
now have to be checked against a whitelist/blacklist of keys.

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 08:24:14 -0700

On Jun 5, 2013, at 4:26 PM, Douglas Crockford <[email-masked]> wrote:

> If duplicate names are encountered, the parse MAY fail (because some implementations correctly do that), or it MAY succeed by accepting the last duplicated key:value pair.

The new text should use language that is already in RFC 4627; "key" is not such a word. To be fair to implementers, the new document also needs to deal with both emitters and parsers.

Proposal:

In Section 2.2:
Current:
   The names within an object SHOULD be unique.
Proposed:
   If the names within an object are not unique, the result of parsing the 
   object is unpredictable, and the parse may even fail completely. Thus,
   the names within an object SHOULD be unique.

In Section 4, add a new paragraph:
   If a parser encounters an object with duplicate names, the parser MAY
   fail to parse the JSON text; if the parser accepts objects with duplicate
   names, it SHOULD accept only the last name/value pair that has the
   duplicate name. 

--Paul Hoffman

---

From: Stephan Beal <[email-masked]>
Date: Thu, 6 Jun 2013 18:03:21 +0200

>    the names within an object SHOULD be unique.

+1 for anything along those lines.

>    names, it SHOULD accept only the last name/value pair that has the

+1. That keeps (i think) any current implementation behaviours compliant
while making the intended/best-practice behaviour clear.

---

From: Vinny A <[email-masked]>
Date: Thu, 6 Jun 2013 11:40:40 -0500

>   names, it SHOULD accept only the last name/value pair that has the

If we have to recommend to both parsers and emitters, I'd like to make a slight change to your wording:

In Section 2.2:
Current:
    The names within an object SHOULD be unique.
Proposed:
   The names within an object SHOULD be unique. Non-unique keys have unpredictable effects (refer to sections 4 & 5).

In Section 4 (Parsers) add a new paragraph:
  If a parser encounters an object with duplicate names, the parser MAY fail to parse the JSON text; if the parser accepts objects with duplicate names, it MUST accept only the last name/value pair that has the duplicate name. 

In Section 5 (Generators) add a new paragraph:
   A JSON generator SHOULD not duplicate names. If duplicate names are generated, the authoritative name/value pair MUST be listed last.

-----------------
Vinny
www.jsontest.com

---

From: Matthew Morley <[email-masked]>
Date: Thu, 6 Jun 2013 12:56:17 -0400

> that doesn't make it impossible.

Nope :)

I was meaning that the comment should be in data structure itself, existing
as a pattern of usage in JSON, rather than part of the JSON specification.

There are examples of adding "_comment_{foo}":"comment on pair foo" to
objects and insertion of {"_comment_0":"comment on index 0"} into arrays.
With the expectation that those comment keys and comment objects be
stripped if their existence will break schema expectations. But it goes a
bit further than just that, in that comments on a top level array or object
would require the payload being wrapped in an array with the 0 index being
the comment object, and the 1 index being the real payload.

It is the unordered nature of keys, that makes the use of duplicate keys
for comments seem like a very strange choice. Every time I look at a JSON
object, I try to imagine the keys (usually on a single line with the value)
constantly shifting their order and cycling around. That is the reality of
what can happen as it encodes/decodes and gets passed around.

-- 
Matthew P. C. Morley

---

From: "Markus Lanthaler" <[email-masked]>
Date: Thu, 6 Jun 2013 19:10:44 +0200

>    names, it SHOULD accept only the last name/value pair that has the

+1 if the last SHOULD is replaced with a MUST

--
Markus Lanthaler
@markuslanthaler

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 10:20:00 -0700

FWIW, I like Vinny's construct better than my proposal. Less description in Section 2, and pointers to Sections 4 and 5. And I agree that the MUST for the parser is more appropriate than the SHOULD.

--Paul Hoffman

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 6 Jun 2013 10:32:03 -0700

On Thu, Jun 6, 2013 at 7:55 AM, Stephan Beal <[email-masked]> wrote:

>
> On Thu, Jun 6, 2013 at 1:19 AM, Nico Williams <[email-masked]>wrote:
>
> This brings with it a performance penalty because all keys being slurped
> up now have to be checked against a whitelist/blacklist of keys.
>
>
And if generators were to be prohibited from outputting duplicates, they
too must keep track of all keys for all open object contexts

I think it is reasonable to point out that use of duplicate keys is not
supported, nor should systems rely on such usage working. In fact, NOT
specifying proper handling for this case may be better for
interoperability; if specific "use first" or "use last" is sanctioned, it
will be more likely to support usage of multiple instances. Furthermore,
mandating one or the other will hurt some use cases; for "use first",
decoder/parser must keep information around to suppress (or error on)
multiples.
And "use last" case would be impractical for purely streaming
implementation since it is impossible to force such usage without buffering
the whole content to determine value to use.

For these reasons, I would prefer simple discouraging use of duplicate
names, without trying to specify "but if you must, then use X" processing.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 6 Jun 2013 10:36:07 -0700

On Thu, Jun 6, 2013 at 9:40 AM, Vinny A <[email-masked]> wrote:

> The new text should use language that is already in RFC 4627; "key" is not
> such a word. To be fair to implementers, the new document also needs to
> deal with both emitters and parsers.
>
> Proposal:
>
> In Section 2.2:
> Current:
>   * The names within an object SHOULD be unique. Non-unique keys have
> unpredictable effects (refer to sections 4 & 5).*
>
>
This sounds good.

>   *If a parser encounters an object with duplicate names, the parser
> MAY fail to parse the JSON text; if the parser accepts objects with
> duplicate names, it MUST accept only the last name/value pair that has the
> duplicate name. *
>
> *   A JSON generator SHOULD not duplicate names. If duplicate names are
> generated, the authoritative name/value pair MUST be listed last.*
>

I find this problematic from streaming parsing perspective: if JSON content
is to be exposed as a stream of tokens, then this behavior is impractical
to implement. Even exposing just the first such key (or entry) is
problematic due to need to keep track of all seen keys. But at least it is
more practical than filtering out all but last.

Streaming parsing is the natural way to level processing, similar to XML
division of labor (SAX/Stax in Java, DOM/databind/XSLT on top), and
although more commonly used by frameworks and not end users, is an
important and common use case.

-+ Tatu +-

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 6 Jun 2013 18:50:05 +0100

Original Message From: "Tatu Saloranta" .

> names, without trying to specify "but if you must, then use X" processing.

+1.

(editing text suggested elsewhere) Can you scare the heck out of 
implementers by saying something like:

In Section 4 (Parsers) add a new paragraph:
   If a parser encounters an object with duplicate names, the behavior is 
undefined.  The parser MAY fail to parse the JSON text, it MAY only accept 
the first name/value pair that has the duplicate name, it MAY only accept 
the last name/value pair that has the duplicate name, it MAY ignore all 
name/value pair that have the duplicate name, or it MAY perform some other 
action.

In Section 5 (Generators) add a new paragraph:
    A JSON generator SHOULD not duplicate names.

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 17:21:34 -0500

>From a security point of view disparate handling of ambiguities makes
me very uncomfortable.

Thinking of encoders that do not have full knowledge of an object as
it is encoded, I'm uncomfortable with saying "MUST NOT send duplicate
keys", since determining whether a given key has already been sent
might require large amounts of state.  But note that this concern also
applies to parsers!

Consider a parser that invokes an application-provided
callback/closure with a {path, value} for each value parsed.  How can
such a parser do anything at all about duplicate keys?  At best it
could detect (having kept a fair bit of state) duplicate keys.  In
this case the responsibility for detecting and handling duplicates
moves onto the application.  The application might be a jq-like
filter, in which case there's nothing to do there either and we move
the responsibility for handling dups onto the last parser in the pipe.
 Or there might not be a pipe at all and actions might be taken on the
fly.

Regarding comments, I thought we'd have to accept that JSON does not
have comments, but since there's talk of extensions, and since we
could (and many, many parsers do) have options that can default to off
(unavailable), we might indeed want to consider a comments option.

[*] The thought occurs that when we speak of duplicate keys we mean in
the memcmp() (byte-wise) sense.  We're specifically not speaking of
Unicode equivalence.  But depending on whether a *human* must
understand object keys (doubtful in production environments) we might
have to.

Having thought this far, I propose this text:

   Encoders SHOULD NOT send duplicate keys.  Some encoders might not
be able to prevent duplicate keys.  Therefore parsers MUST be prepared
to handle duplicate keys.

   Stateful parsers MUST accept [use?] only the last of any set of
duplicate keys.

   Some parsers might not be able to detect duplicate keys, much less
pick only the last of them.  Here a "stateful parser" is one that
keeps on hand all of the values it decodes, as it decodes them.  Note
that accepting duplicate keys presents potential security risks.  Note
that sending duplicate keys risks data loss (that is, the loss of all
but the last of a duplicated key's values).

   For the purposes of the above discussion, two or more keys in the
same object are duplicates of each other if, when expressed as a
sequence of code units, they are byte-wise equivalent.

Nico
--

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Thu, 6 Jun 2013 22:33:44 +0000

/me doffs hat inline ...

> Having thought this far, I propose this text:

Note that so far, the document uses the term "name", not "key".  I think the following needs to substitute "key" with "name" to be consistent.

>   Stateful parsers MUST accept [use?] only the last of any set of

I think this still needs to allow for stateful parsers that reject duplicate keys (for which there are some).

Maybe:

####

Stateful parsers MAY reject duplicate names. However, if duplicate names are accepted, it MUST accept only the last value of any set of duplicate names.

####

> but the last of a duplicated key's values).

Can we describe a couple of specific security risks that are incurred?  I think one would be something like overwriting of the original value by an attacker intercepting the exchange.

> sequence of code units, they are byte-wise equivalent.

Other than the suggestions above, I think it's rather good.

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 18:37:11 -0400

> might require large amounts of state.  But note that this concern also

The amount of state can be reduced probabilistically by storing hashes
of the values rather than the values themselves.

-- 
Kill Gorgun!  Kill orc-folk!            John Cowan
No other words please Wild Men.         [email-masked]
Drive away bad air and darkness         http://www.ccil.org/~cowan
with bright iron!   --Ghan-buri-Ghan    http://www.ccil.org/~cowan

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 17:44:04 -0500

> of the values rather than the values themselves.

Or a refinement: Bloom filters.  But I don't think that's a good idea.
 For one the parser would still have to grow the filter (with
adaptable Bloom filters) as it gets too big.  For another, we have the
risk of false positives.  Finally, such a parser could at best produce
an error on [suspected] duplicate key (excuse me, name) usage -- it
could not prevent the *first* value from being used, and if the
consumer has already taken some action based on the first value then
we suddenly have a conflict vis-a-vis parsers that accept only the
last value.  I'd rather accept that streaming parsers can do nothing
about duplicate names.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 17:39:55 -0500

On Thu, Jun 6, 2013 at 5:33 PM, Matt Miller (mamille2)
<[email-masked]> wrote:
>> [...]

Ah, sure.

> I think this still needs to allow for stateful parsers that reject duplicate keys (for which there are some).

That's fair.

> Stateful parsers MAY reject duplicate names. However, if duplicate names are accepted, it MUST accept only the last value of any set of duplicate names.

Sure.  I think we should define terms like "stateful parser" and
"streaming parser".

> Can we describe a couple of specific security risks that are incurred?  I think one would be something like overwriting of the original value by an attacker intercepting the exchange.

I'm not concerned about MITMs and such.  I'm concerned about attacks
where we have a validator of some sorts as a filter and then a final
consumer.  The sender might send JSON that the validator accepts as
valid, that will be passed on to the final consumer, and where the
consumer will receive a different document (from it's p.o.v.) than the
validator saw.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 15:58:43 -0700

<co-chair hat on>

Some of the recent comments on this thread have gone down the path of naming different types of parsers and saying they might have different rules because of their capabilities.

Do we really want to do that for what is supposed to be a minimal update of the spec? (That is a leading question, but "yes" is an acceptable answer if it justified.)

Proposal: we simply list what we want encoders and parsers to do without saying what type of encoder, or what type of parser, gets to do what.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 18:03:15 -0500

> Do we really want to do that for what is supposed to be a minimal update of the spec? (That is a leading question, but "yes" is an acceptable answer if it justified.)

So far I think there's only two types of encoders and decoders:
streaming and non-streaming.  I don't think we'll see more types.  If
the list stays short, then "yes".

> Proposal: we simply list what we want encoders and parsers to do without saying what type of encoder, or what type of parser, gets to do what.

I would like to say that those parsers than can MUST accept only the
last value.  I would also like to not preclude parsers that can't even
detect duplicates keys (streaming parsers).

I would like to say that encoders that can MUST NOT send duplicate
keys.  I would also like to not preclude encoders that can't even
detect that they are sending duplicate keys.

Nico
--

---

From: "Markus Lanthaler" <[email-masked]>
Date: Fri, 7 Jun 2013 01:08:45 +0200

> to handle duplicate keys.

-1, it is very reasonable to reject such JSON.

--
Markus Lanthaler
@markuslanthaler

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 6 Jun 2013 16:17:04 -0700

On Thu, Jun 6, 2013 at 3:37 PM, John Cowan <[email-masked]> wrote:

> of the values rather than the values themselves
>

But hashes would only allow finding potential duplicates, and one still has
to verify with real Strings. I don't see a way around false positives.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 6 Jun 2013 16:19:20 -0700

On Thu, Jun 6, 2013 at 3:58 PM, Paul Hoffman <[email-masked]> wrote:

> <co-chair hat on>
>
> Some of the recent comments on this thread have gone down the path of
> naming different types of parsers and saying they might have different
> rules because of their capabilities.
>
> Do we really want to do that for what is supposed to be a minimal update
> of the spec? (That is a leading question, but "yes" is an acceptable answer
> if it justified.)
>
> Proposal: we simply list what we want encoders and parsers to do without
> saying what type of encoder, or what type of parser, gets to do what.

I think mention of different modes of operation would only be relevant to
explain why certain limits (MUST not write duplicates, MUST report error
for duplicates) can not be imposed. So they could be mentioned as
rationale; explain why SHOULD instead of MUST.

-+ Tatu +-

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 19:27:14 -0400

> to verify with real Strings. I don't see a way around false positives.

That's the point of probabilistic algorithms: you can reduce the probability
of a false positive below any value you are willing to accept.

-- 
Values of beeta will give rise to dom!          John Cowan
(5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan
to rename '.' or '..' entries; see              [email-masked]
http://cm.bell-labs.com/cm/cs/who/dmr/odd.html)

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 19:13:48 -0500

> -1, it is very reasonable to reject such JSON.

I wrote "handle", not "accept".

---

From: R S <[email-masked]>
Date: Thu, 6 Jun 2013 19:59:38 -0700

JSON is sometimes produced by naive string concatenation, which will
probably never change. It is consumed by browsers, which will probably
never change their duplicate key handling.

The text from RFC 4627 should stay unchanged. The suggestions for
requirements on encoders are prescriptive and therefore inaccurate at this
stage in JSON's adoption.

- Rob

---

From: Tim Bray <[email-masked]>
Date: Thu, 6 Jun 2013 20:00:45 -0700

+1. I am highly unconvinced that this -bis effort really has anything
useful to add in this area.  -T

On Thu, Jun 6, 2013 at 7:59 PM, R S <[email-masked]> wrote:

> On Thu, Jun 6, 2013 at 5:13 PM, Nico Williams <[email-masked]>wrote:
>
>>

---

From: Vinny A <[email-masked]>
Date: Thu, 6 Jun 2013 22:15:07 -0500

> Proposal: we simply list what we want encoders and parsers to do without saying what type of encoder, or what type of parser, gets to do what.

+1, while also saying that I would like to revisit this discussion (and also the discussion on comments) when writing the best practices doc.

-----------------
Vinny
www.jsontest.com

---

From: "Markus Lanthaler" <[email-masked]>
Date: Fri, 7 Jun 2013 11:04:40 +0200

>I wrote "handle", not "accept".

If your "handle" implies that an error might be raised then it would be OK but I believe most people wouldn't understand it that way.

--
Markus Lanthaler
@markuslanthaler

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 12:22:52 +0200

On 2013-06-07 11:04, Markus Lanthaler wrote:
> If your "handle" implies that an error might be raised then it would
> be OK but I believe most people wouldn't understand it that way.

weighting the part following the "but" I read (and second) this as -1

In case TL;DR please goto "TL;DR"

Three further remarks ignited in me by the above text (that was proposed 
by Nico Williams):

"""
"""

Which b.t.w. I would certainly rewrite as:

NEW
"""
Generators SHOULD NOT duplicate names in objects.  Parsers MUST be 
prepared to either accept duplicate names in objects or reject the 
complete JSON text containing these, as a generator might not avoid nor 
detect such duplication.
"""

"TL;DR":
Above is my proposal if we really want to change the current wording and 
if we settle on the terms generator and parser. Otherwise replace with 
encoder and decoder ;-)

Now for something completely different (the promised remarks):

A) encoders relate to parsers?

There seem to be some projections of the spec we really need to check 
periodically, so not "hypernyms" slip in ...

Let us please not cross-mix members of dimensional pairs:

* ("encoder","decoder")
* ("accept","reject")
* ("interpret","ignore")
* ("generator","parser")
* ("producer","consumer")
* ("sender","receiver")
* ("service","client")
* ...

B) We can "handle" anything :-)

The verb "handle" IMO is related to far too many possible actions:
accept, reject, interpret, ignore all jump to my mind, when I'm 
requested to handle something.

On a more higher level the robustness principle (a.k.a. Postel’s law) 
cf. [RFC1122]: "Be liberal in what you accept, and conservative in what 
you send" (*) might at first glance help advice the participants in the 
lax historically grown state we're, **but**

1. we now have all kind of JSON consumers ranging from the brittle 
("picky") to the robust ("close to blind").

2. we define a format, and not a "JSON service" or the like

As I understood, we do not change practical usage of the spec, but on 
occasion of augmenting the RFC to STD and avoiding a cyclic normative 
referencing between JSON RFC and ECMAScript spec we perform "some non 
interupting hygienic actions".

So I think there will be only microscopic changes (compared to the old 
RFC) on this topic left over after consensus calls.

C) "names" are not "keys"

Again in above text (and at the heart of the discussion IMO) we take the 
"name"s of the object members as "key" but do not say so in the RFC, and 
did not in the original.

This semantic difference and the historical fact that it was not spelled 
out in the original RFC gently directs us to just leave the "names" 
condition as is, and not start expressing our whishful thinking, we 
could simply rename these as "key" and have all the uniqueness readily 
at hand.

References:

[RFC1122]: Braden, R., Ed., "Requirements for Internet Hosts -
            Communication Layers", STD 3, RFC 1122, October 1989.
            http://www.ietf.org/rfc/rfc1122.txt.

*): I know RFC791 is historically the first similar mention, but it is 
obsoleted as RFC, so ...

All the best,
Stefan.

---

From: Vinny A <[email-masked]>
Date: Fri, 7 Jun 2013 08:38:31 -0500

On Jun 7, 2013, at 5:22 AM, Stefan Drees <[email-masked]> wrote:
> Generators SHOULD NOT duplicate names in objects.  Parsers MUST be prepared to either accept duplicate names in objects or reject the complete JSON text containing these, as a generator might not avoid nor detect such duplication.
> """

For the sake of readability, can you split this proposed text into separate prescriptions for parsers and emitters? I.e. section 4 of the RFC is for parsers, section 5 is for generators, put their respective advice into the appropriate sections.

-----------------
Vinny
www.jsontest.com

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 16:07:05 +0200

On 2013-06-07 15:38, "Vinny A" wrote:
>
> On Jun 7, 2013, at 5:22 AM, Stefan Drees ... wrote:
> For the sake of readability, can you split this proposed text into
> separate prescriptions for parsers and emitters? I.e. section 4 of the
> RFC is for parsers, section 5 is for generators, put their respective
> advice into the appropriate sections.
>

sure thing. Here comes my proposed rewrite of sections 4 and 5 based on 
the above, but possibly ignoring ideas in other messages already sent to 
this list (the subject of a mail thread MAY relate to the message body 
content ;-)

OLD:
"""
4.  Parsers

    A JSON parser transforms a JSON text into another representation.  A
    JSON parser MUST accept all texts that conform to the JSON grammar.
    A JSON parser MAY accept non-JSON forms or extensions.

    An implementation may set limits on the size of texts that it
    accepts.  An implementation may set limits on the maximum depth of
    nesting.  An implementation may set limits on the range of numbers.
    An implementation may set limits on the length and character contents
    of strings.

5.  Generators

    A JSON generator produces JSON text.  The resulting text MUST
    strictly conform to the JSON grammar.

"""

NEW:
"""
4.  Parsers

    A JSON parser transforms JSON text into another representation,
    MUST accept all texts that conform to the JSON grammar and MUST be
    prepared to either accept duplicate names in objects or reject the
    complete JSON text containing these.
    A JSON parser MAY accept non-JSON forms or extensions.

    An implementation may set limits on any of the following: the size
    of texts that it accepts, the maximum depth of nesting, the range of
    numbers, and the length and character contents of strings.

5.  Generators

    A JSON generator produces JSON text.  The resulting text MUST
    strictly conform to the JSON grammar.

    Generators SHOULD NOT duplicate names in objects if they can avoid
    or detect such duplication.

"""

What do you think?

All the best,
Stefan.

---

From: Vinny A <[email-masked]>
Date: Fri, 7 Jun 2013 10:19:13 -0500

> or detect such duplication.

+1, with the understanding that some parsers (in particular streaming parsers) will not be able to reject the "complete JSON text" because they've already sent earlier parts of the JSON text for processing by other app components.

But that's a bit nitpicky, so feel free to keep the current text as-is.

-----------------
Vinny
www.jsontest.com

---

From: Stephen Dolan <[email-masked]>
Date: Fri, 7 Jun 2013 16:19:25 +0100

> complete JSON text containing these, as a generator might not avoid nor

Regarding duplicate keys, the issue was raised before that having
multiple valid interpretations of an input leads to security problems.
Consider this document:

    {"command": "no-op", "user": "untrustworthy-person", "command":
"launch-missiles"}

if submitted to a system which separates authorization from execution,
where both use JSON implementations which differ in their
interpretation of duplicate keys, this could end badly.

There are some JSON implementations which preserve duplicate keys, and
many which don't. However, I know of no implementation which
interprets {"a":2, "a":1} as {"a":2} - the implementations I've seen
keep either all or just the last entry. Codifying this behaviour in
the RFC would help to prevent the security issue described above, so
how about some text along the lines of:

    If an object contains several key-value entries with the same key,
a JSON parser MAY ignore all but the last of these entries. A JSON
parser MUST NOT ignore the last such entry.

Stephen Dolan

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 17:38:18 +0200

>> complete JSON text containing these, as a generator might not avoid nor

Please note: The above proposal has already been split and updated upon 
request of "Vinny A" as the spec separates Parser and Generator sections.

> parser MUST NOT ignore the last such entry. ...

I do think, that forcing the consumer to "keep" the last parsed 
(whatever this means from the perspective of the source of the message) 
name as "the one" is **not** "good", nor "more secure" nor "doesn't 
break anything in the wild".

A "source of the message" shall indicate, that this is not a format 
concern, but one of the sender and receivers rules.
Say e.g. a program running on node A has a list of names and these have 
values attached. The names are not unique (thus a list ;-) if this 
program decides to send this information through a JSON data 
interchange, when selecting the JSON type object, it does so knowingly, 
that the order of members in a JSON object is undefined, so there is IMO 
no principal security dilemma in ordering, but in ambiguity of a message 
that would have better been build with a unique set of names from the start.

As has been noted in many differnt mails already here on this list, we 
have no time-machine and somehow have to deal with that :-)

All the best,
Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 Jun 2013 08:46:02 -0700

On Jun 6, 2013, at 7:59 PM, R S <[email-masked]> wrote:

> JSON is sometimes produced by naive string concatenation, which will probably never change. It is consumed by browsers, which will probably never change their duplicate key handling.
> 
> The text from RFC 4627 should stay unchanged. The suggestions for requirements on encoders are prescriptive and therefore inaccurate at this stage in JSON's adoption.

Your first paragraph does not match your second. Many of the proposals here keep the gist of RFC 4267 for encoders, "The names within an object SHOULD be unique.".  For parsers in browsers, so far all the examples we have seen have followed the proposed wording of "MUST choose the last one".

--Paul Hoffman

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 7 Jun 2013 15:46:38 +0000

Nico, would you clarify your proposal to indicate place or places in 4627bis-02 your text replace or append to?

With regards to parsers types elsewhere in this, while I also think there might only be two, Paul does raise an interesting point about keeping the extra terms to a minimum.  I think we can further refine the language as long as we're clear where and how this proposal fits.

- m&m

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 6 Jun 2013 08:42:31 -0700

Just got signed on to this list...

There is also a discussion of this topic going on at [email-masked] which I'm trying to redirect here.

My recommendation from the es-disuss thread is below.

Allen Wirfs-Brock
ECMAScript Language Spec. Project Editor

>   names, it SHOULD accept only the last name/value pair that has the

Some that should be pointed out is that one reason JSON needs to accept duplicate field names is that some people currently use them to add  comments to JSON datasets:

{
"//": "This is a comment about my data",
"//":  "that takes more than one line",
"field1" : 1,
"field2" : 2
}

The above is valid according to the existing RFC and is accepted as valid JSON by the existing ECMAScript  JSON.parse spec. We don't want to invalid existing JSON datasets that use this technique so duplicate keys MUST be accepted.

What I would say, as a replacement for the current text  <section 2.2> is:

         The names within an object SHOULD be unique.  If a key is duplicated, a parser MUST take  <<use?? interpret??>> only the last of the duplicated key pairs.

I would be willing to loose the first sentence (containing the SHOULD) entirely as it doesn't add any real normative value.

Given that duplicate names have historically been valid, that some people use them, and that the standard ECMAScript JSON parser accepts them I don't see why allowing a parser to reject duplicate names is helpful.

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 6 Jun 2013 10:45:01 -0700

Just got signed on to this list...
(and this message didn't seem to make it there the first time, so sorry if this is duplicated)

Allen Wirfs-Brock

{

---

From: Douglas Crockford <[email-masked]>
Date: Fri, 07 Jun 2013 09:46:00 -0700

On 6/6/2013 8:42 AM, Allen Wirfs-Brock wrote:
> Given that duplicate names have historically been valid, that some 
> people use them, and that the standard ECMAScript JSON parser accepts 
> them I don't see why allowing a parser to reject duplicate names is 
> helpful.
>
>
>
Because some parsers do so, and have for years. JSON.parse is not the 
only JSON parser. Our purpose here is not to attempt to fix this thing, 
because fixing it will break it. There were unfortunately ambiguities in 
the RFC. If we can, we should remove the ambiguities, but we must do so 
without changing the meaning of what JSON is.

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 11:47:02 -0500

> If your "handle" implies that an error might be raised then it would be OK but I believe most people wouldn't understand it that way.

I meant "not crash" :)

However, the note by Allen Wirfs-Brock convinces me that parsers
SHOULD NOT reject dups.

Nico
--

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Fri, 7 Jun 2013 09:59:29 -0700

On Jun 7, 2013, at 9:46 AM, Douglas Crockford wrote:

> Because some parsers do so, and have for years. JSON.parse is not the only JSON parser. Our purpose here is not to attempt to fix this thing, because fixing it will break it. There were unfortunately ambiguities in the RFC. If we can, we should remove the ambiguities, but we must do so without changing the meaning of what JSON is.

I think it is more important to preserve the validity of existing archival datasets then it is to preserve the validity of existing parsers that throw on duplicate keys. 

Allen

---

From: Douglas Crockford <[email-masked]>
Date: Fri, 07 Jun 2013 10:02:19 -0700

> I think it is more important to preserve the validity of existing archival datasets then it is to preserve the validity of existing parsers that throw on duplicate keys.
We should not be invalidating data or programs. We should only be 
explaining better. No breakage is acceptable. No breakage.

---

From: Jacob Davies <[email-masked]>
Date: Fri, 7 Jun 2013 10:30:44 -0700

+1

I don't think there is anything you can do about this in the
specification except give the doctor's advice: "Don't do that." It is
better to leave the behavior unspecified and advise against using it
than to try to specify some one way of handling it that can't be
relied on in the real world.

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 10:34:08 -0700

+1

---

From: Vinny A <[email-masked]>
Date: Fri, 7 Jun 2013 12:43:15 -0500

>         The names within an object SHOULD be unique.  If a key is duplicated, a parser MUST take  <<use?? interpret??>> only the last of the duplicated key pairs.

Just to be clear here, ECMA is OK with using MUST in regards to the last key pair?  

Because we already have proposed language that says that, but one of the points under contention on this list is what to do with duplicated key pairs.

Is the following language acceptable to ECMA (disclaimer: original text by Paul,  modifications by me):

In Section 2.2:
Current:
    The names within an object SHOULD be unique.
Proposed:
   The names within an object SHOULD be unique. Non-unique names have unpredictable effects (refer to sections 4 & 5).

-----------------
Vinny
www.jsontest.com

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Fri, 7 Jun 2013 15:33:00 -0700

> Just to be clear here, ECMA is OK with using MUST in regards to the last key pair?  

The ECMAScript standard specifies that the JSON parse built-in to a conforming ECMAScript implementation must accept duplicate key pairs and that it provides as its output the value of the last such pair.  Changing that behavior would be a breaking change that  we are very unlikely to make.

In general, the ECMAScript specification is hyper prescriptive about many behaviors, such as this, that traditionally would have been considered reasonable areas to allow implementation variability. The reason is that we really are a community of multiple independently developed implementations whose users (web developers) expect and demand identical behavior from our implementations. Experience has shown that when implementation variation is permitted that a de facto standard rapidly emerges around whatever implementation choices were made by the currently dominant implementation (historically MS Internet Explore on the desktop, more recently Webkit on mobile devices) and that all other implementations are forced to conform in order to remain viable. The preference within TC39 is to pro actively avoid such situations and reach early consensus on a single required behavior.

>>   the names within an object SHOULD be unique.

First, a terminology issue. Section 2 is defining a grammar.  There are no syntactic ambiguities associated with multiple key-value pairs.  There is no reason they should cause parsing (take literally) to fail. The issue is what semantic interpretation is applied to such occurrences. JSON actually specifies very little with regard to the semantics of a JSON text but a plausible interpretation of duplicate key-value pairs might be that the entire JSON text is invalid because it is semantically ambiguous.  Personally I wouldn't call that a parsing issue.

Back to your question, Ecma TC39 is a consensus driven organization, and I don't think we would find consensus on the above proposed language.  More below

>    The names within an object SHOULD be unique. Non-unique names have unpredictable effects (refer to sections 4 & 5).

This is better and I think we could get consensus (it's a variation of what I suggested above):

In Section 2.2:
Current:
    The names within an object SHOULD be unique.
Proposed:
   The names within an object SHOULD be unique because it is ambiguous as to which value should be associated with a duplicated key.

>   If a parser encounters an object with duplicate names, the parser MAY fail to parse the JSON text; if the parser accepts objects with duplicate names, it MUST accept only the last name/value pair…

I think this is too restrictive and gives license to reject existing valid datasets that use commenting conventions that introduce duplicate names.

Proposal:
   A parser MUST accept a JSON text that includes objects with duplicate names. A parser SHOULD associate with such a name the value of the last name/value pair that has the    duplicate name. 

TC39 would leave the specification of the ECMAScript JSON.parse function unchanged as it conforms to this proposal.

Note that a streaming parser might reasonably and validly present all of the duplicate name/value pairs to its clients.

>    A JSON generator SHOULD not duplicate names. If duplicate names are generated, the authoritative name/value pair MUST be listed last.

I think this is unnecessary if we specify that JSON text's with duplicated names must be accepted by parsers.

Allen

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 17:52:00 -0500

On Fri, Jun 7, 2013 at 5:33 PM, Allen Wirfs-Brock <[email-masked]> wrote:
> Proposal:
>    A parser MUST accept a JSON text that includes objects with duplicate
> names. A parser SHOULD associate with such a name the value of the last
> name/value pair that has the    duplicate name.

+1.

In conclusion:

We can't say "MUST use only the last pair" because of streaming
parsers (and their applications).  We can't require encoders to not
send duplicates for the same reason.  And we can't allow rejection of
objects with duplicate names because that would break interop with
streaming encoders.

> TC39 would leave the specification of the ECMAScript JSON.parse function
> unchanged as it conforms to this proposal.

When we have a SHOULD (or SHOULD NOT) we should specify some reasons
that one might not adhere to the given recommendation.  Streaming
parsers would be one reason.  I propose this text from you be added to
the RFC as well:

> Note that a streaming parser might reasonably and validly present all of the
> duplicate name/value pairs to its clients.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 23:37:28 -0400

Allen Wirfs-Brock scripsit:

> In general, the ECMAScript specification is hyper prescriptive about
> many behaviors, such as this, that traditionally would have been
> considered reasonable areas to allow implementation variability. 

I've often wondered why ECMAScript remains silent about the order
of property enumeration.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
The whole of Gaul is quartered into three halves.
        --Julius Caesar

---

From: R S <[email-masked]>
Date: Fri, 7 Jun 2013 22:37:47 -0700

> I've often wondered why ECMAScript remains silent about the order

It is largely uniform, but iirc there are differences in implementations
when enumerating properties that are array indices, and no consensus on
resolving the differences there.

- Rob

---

From: Stephen Dolan <[email-masked]>
Date: Sat, 8 Jun 2013 22:07:59 +0100

On Fri, Jun 7, 2013 at 4:38 PM, Stefan Drees <[email-masked]> wrote:
> I do think, that forcing the consumer to "keep" the last parsed (whatever
> this means from the perspective of the source of the message) name as "the
> one" is **not** "good", nor "more secure" nor "doesn't break anything in the
> wild".

It is more secure, and I'm not aware of anything in the wild that it
breaks. Perhaps my previous example of a security hole caused by this
was overly terse, so here's a longer description.

Consider a system that takes requests from the outside world. Each
request is represented as a JSON object, and must include a "password"
field and a "command" field.

Some commands (like "call-reception") may be performed by anyone by
passing a null password. Some commands (like "launch-missiles")
require a password.

The command-processing system, upon receiving a request, forwards the
request to the authorization system, which determines whether the
command is authorized. The attacker sends:

    {"command": "launch-missiles",
     "password":null,
     "command": "call-reception"}

The authorization system, using the standard JSON parser from Python
(or Ruby or PHP or JS or whatever), parses this as having a "command"
field which says "phone-reception", and authorizes the command.

The command-processing system, using the hypothetical parser jsDumb,
parses this as having a "command" field which says "launch-missiles".
Since the authorization system has OKed the command, this ends badly.

Luckily for us, I do not believe jsDumb exists. A reasonably extensive
survey in another thread has failed to find such a parser. There are
of course streaming parsers such as Java's Jackson, which return all
of the keys and values - this is fine, as if the command-processing
system were to use such a parser it could simply error on requests
having multiple "command" fields.

Unfortunately, the hypothetical jsDumb is considered a compliant
parser according to the current RFC, leading to the above security
hole where two compliant parsers can return entirely contradictory
results on the same document.

The issue is not two parsers returning different amounts of
information: Jackson returns more information about the document than
JSON.parse, for instance. The issue is two parsers returning *entirely
contradictory* information about the same document.

It may be that jsDumb already exists and is deployed on millions of
machines, and JSON is doomed to insecurity forever. I don't think this
has happened yet - I don't know of any parser with jsDumb's behaviour.
In that case, the updated RFC should make clear that this behaviour is
wrong, to prevent future implementations having this hole.

Various phrases have been proposed, here's a couple:

[Douglas Crockford]
If duplicate names are encountered, the parse MAY fail (because some
implementations correctly do that), or it MAY succeed by accepting the
last duplicated key:value pair.

[Alan Wirfs-Brock]
The names within an object SHOULD be unique.  If a key is duplicated,
a parser MUST take  <<use?? interpret??>> only the last of the
duplicated key pairs.

[myself]

None of these disallow streaming parsers or other parsers that keep
all of the key-value pairs of an object. None of them disallow objects
with duplicate keys. The only thing disallowed is a parser which
interprets `{a:1, a:2}` as equivalent to `{a:1}`.

Regards,
Stephen Dolan

---

From: Stefan Drees <[email-masked]>
Date: Sun, 09 Jun 2013 12:57:46 +0200

>> one" is **not** "good", nor "more secure" nor "doesn't break anything in the

If TL;DR **please** at least goto "TL;DR", as there is an updated 
proposal. Thanks

> field which says "phone-reception", and authorizes the command.

I think you mean "call-reception", right?

> Since the authorization system has OKed the command, this ends badly.

Really having a hard time trying to imagine a use case, where someone 
came up with such an implementation of "separation of concerns".
But I'd have a good old name for it: "Broken phone" :-)

> results on the same document.

I bet, if such a hypothetical system existed, it would be better of,
when the authorization part yielded a transform as result of the check, 
as that would contain (besides additional transaction relevant data) as 
the only remaining user input:

        {"command":"call-reception"}

so the command-processing system of your plot, would only communicate 
through the authorization component.

If the "channel" would be secured only, anything goes ...

> wrong, to prevent future implementations having this hole.

Ahem, the "hole" appears to be two system components playing broken 
phone. The rest of that scenario is quite exchangeable, isn't it?

B.t.w: I have for sure invented far more dumb parsers, than that 
imaginary "jsDumb", trust me I just can't remember which ... ;-)

The RFC should IMO stay within the naturally fuzzy, but nevertheless 
distinguishable realm of being a very general format in use for long and 
by many, thus I would opt for REALLY SHOULD but not for MUST.

For detailed reasons for this please see my updated proposal and the 
relevant notes below (goto "TL;DR").

> interprets `{a:1, a:2}` as equivalent to `{a:1}`.

updating my proposal for both the parser and generator sections yields:

TL;DR:

NEW:
"""
4.  Parsers

    A JSON parser transforms JSON text into another representation,
    MUST accept all texts that conform to the JSON grammar and MUST be
    prepared to either accept duplicate names in objects or reject the
    complete JSON text containing these.
    If the JSON parser transforms name/value pairs of JSON objects
    into maps inside the target representation by using the received
    names as keys, then it REALLY SHOULD always yield the values of the
    last encountered occurences of the name/value pairs as to satisfy
    the principle of least surprise.
    A JSON parser MAY accept non-JSON forms or extensions.

    An implementation may set limits on any of the following: the size
    of texts that it accepts, the maximum depth of nesting, the range
    and precision of numbers, and the length and character contents of
    strings.

5.  Generators

    Generators REALLY SHOULD NOT duplicate names in objects if they can
    avoid or detect such duplication.

"""

Notes:

1. Section 4 proposal part builds upon [1], where I introduced the
    possible number precision constraint as suggested by John Levine.

2. Inserted REALLY to form REALLY SHOULD NOT in generator section last
    paragraph as suggested by Joe Hildebrand in [2]

3. The new inserted text accomodating the "if not keep all, please keep
    last" proposal takes into account, that of course there are nice
    JSON parsers, that separate the transform of JSON to target storage
    reps from the parsing steps of the JSON message itself, by eg.
    providing callbacks (For instance YAJL c.f. [3]) others might offer
    method overrides / decorations or the like.
    Also a parser ignoring a token (or not) is not fully describing
    the problem we want to address, as we mix the scanning, parsing
    (where it would be applicable) with the target storage
    representation selected.
    I used yield, as a parser yields something (storage agnostically
    this remains true).

4. The formulation (as in 3.) also tries to spell out, that of course
    we do not have duplicate key:value pairs or the like (name/value
    pairs would be the correct intra JSON lingo) because we then had no
    problem at all, right :?)
    We have multiple pairs sharing the same name but not necessarily
    a common value.

5. Also (refering to the text as in 3.) we do not need a special recipe
    for the parser based on multiplicity of names, as keeping all or
    keeping the last occurence in the JSON text work for singleton cases
    alike.

6. (ibid) But we need to constrain on a single object (as my proposal
    does)! We really do not want to keep the last in this situation:

	{"foo":{"foo":{"bar":42}},"bar":"baz"}        sample 6.a

    But of course in this:

         {"foo":{"foo":{"bar":42}},"foo":"baz"}        sample 6.b

    where the result (keep last branch) would be undistinguisable from:

         {"foo":"baz"}                                 sample 6.c

7. Hopefully we do not forget, that besides the serialized ordering
    of names in a JSON object, this depends on the ordering the
    generator used. In the current RFC this is completely implementation
    dependent. All hope for at least somehow "dependable ordering" i.e.
    reusing sample 6.a after the insertion a generator input side
    deletion of say the "inner" foo's value setting it to null:

	{"foo":{"foo":null},"bar":"baz"}              sample 7.a

    And not (what would be completely ok):

	{"bar":"baz","foo":{"foo":null}}              sample 7.b

    where inner changes have impact on the outer ordering.

References:

[1]: http://www.ietf.org/mail-archive/web/json/current/msg00688.html
[2]: http://www.ietf.org/mail-archive/web/json/current/msg00428.html
[3]: https://github.com/lloyd/yajl

All the best,
Stefan.

---

From: Nico Williams <[email-masked]>
Date: Sun, 9 Jun 2013 12:22:09 -0500

>
> FYI, "REALLY" is not an RFC2119 word :)

---

From: Stefan Drees <[email-masked]>
Date: Sun, 09 Jun 2013 20:33:06 +0200

>     FYI, "REALLY" is not an RFC2119 word :)

that WOULD PROBABLY well be, but as suggested in the reference [2] in 
the proposing mail, I thereby disclose my hidden agenda, that the JSON 
RFC additionally references [RFC6919] empowering us "en passant" with a 
really deeper felt SHOULD NOT while discussing and secretly removing the 
reference and lowercasing the word "REALLY" just before the last 
consensus call :-)

References:

[RFC6919]: R. Barnes, S. Kent, E. Rescorla "Further Key Words for Use
      in RFCs to Indicate Requirement Levels", 1. April 2013,
      http://www.ietf.org/rfc/rfc6919.txt
[2]: http://www.ietf.org/mail-archive/web/json/current/msg00428.html

Please excuse the occurence of the terms "hidden" and "secretly" in a 
mail being part of an open exchange of thoughts ;-)

Stefan.

---

From: Nico Williams <[email-masked]>
Date: Sun, 9 Jun 2013 14:41:24 -0500

The problem with normatively referencing RFC6919 is that we'd be limited to
publication on just one day in any given year, otherwise I tthink it'd be a
great idea!

---

From: Dean Landolt <[email-masked]>
Date: Mon, 10 Jun 2013 08:55:25 -0400

On Fri, Jun 7, 2013 at 1:02 PM, Douglas Crockford <[email-masked]>wrote:

>>>> people use them, and that the standard ECMAScript JSON parser accepts them
>>>> I don't see why allowing a parser to reject duplicate names is helpful.
>>>>
>>>>
>>>>
>> I think it is more important to preserve the validity of existing
>> archival datasets then it is to preserve the validity of existing parsers
>> that throw on duplicate keys.
>>

In this case "no breakage" effectively means no hope of interoperability
going forward. What, then, is the point of this exercise? To echo AWB's
point, I've seen the duplicate-keys-for-comments technique used (and
encouraged) by many in the js community for years -- I'm sure there's a
large volume of otherwise-valid json out there. This is handled in a
variety of ways in the wild -- shouldn't this be a high priority to
correct? And all things being equal, programs rot a lot faster than data --
there's no correcting for the mass of historical duplicate-key json.

Another possible solution: specify that parsers MUST NOT throw on duplicate
keys and recommend omitting duplicate keys entirely. Given history I
believe this is the only sane way to handle duplicate keys -- as was noted
earlier in this thread they're completely useless after a
parse/serialization cycle anyway. This may put some noses out of joint but
web reality suggests this is the only robust solution.

---

From: John Cowan <[email-masked]>
Date: Mon, 10 Jun 2013 09:47:08 -0400

Dean Landolt scripsit:

> going forward. What, then, is the point of this exercise? 

Indeed.  For if the spec is ambiguous (and it is) and has been interpreted
differently by different users (and it has), "no breakage" would mean
carefully preserving all those ambiguities exactly as-is.  How is that
"explaining better"?  Better to call off the whole exercise if we are
constrained to bug-for-bug compatibility.

Even HTML5, which is all about compatibility, doesn't insist that each
and every pre-HTML5 parser is already fully conformant.

> Another possible solution: specify that parsers MUST NOT throw on
> duplicate keys and recommend omitting duplicate keys entirely.

"Recommend" is a synonym for SHOULD, so it already says that.

-- 
You're a brave man! Go and break through the            John Cowan
lines, and remember while you're out there              [email-masked]
risking life and limb through shot and shell,           http://ccil.org/~cowan
we'll be in here thinking what a sucker you are!
        --Rufus T. Firefly

---

From: Marcos Caceres <[email-masked]>
Date: Mon, 10 Jun 2013 14:57:19 +0100

> and every pre-HTML5 parser is already fully conformant.

Agree. Maybe we can get agreement of what is intolerable parsing behavior first (i.e., what is currently considered harmful and why in existing implementations - if anything)?  

-- 
Marcos Caceres

---

From: Stephen Dolan <[email-masked]>
Date: Mon, 10 Jun 2013 17:55:19 +0100

> I think you mean "call-reception", right?

You're right, apologies.

> Really having a hard time trying to imagine a use case, where someone came
> up with such an implementation of "separation of concerns".
> Ahem, the "hole" appears to be two system components playing broken phone.

The issue is, that currently there exists a JSON document which one
RFC-conforming correct JSON parser can say is indistinguishable from
{a: true}, and another equally RFC-conforming correct JSON parser can
say is indistinguishable from {a: false}.

While my example might not have been great, this still seems easily
exploitable: two parsers parsing a supposedly standard format can be
tricked into seeing completely different results from the same input.
Surely it would be better if we could say that one of those parsers
was not RFC-conforming?

Stephen

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 10:01:37 -0700

<no hat>

> Surely it would be better if we could say that one of those parsers

Stated that narrowly, yes. But given your first paragraph, the comparison might be more fully stated as "would it be better if we could says that one of these parsers was not RFC-conforming if making such a statement caused what was a stored valid JSON text now non-valid".

--Paul Hoffman

---

From: Stephen Dolan <[email-masked]>
Date: Mon, 10 Jun 2013 18:08:25 +0100

On Mon, Jun 10, 2013 at 6:01 PM, Paul Hoffman <[email-masked]> wrote:
> Stated that narrowly, yes. But given your first paragraph, the comparison might be
> more fully stated as "would it be better if we could says that one of these parsers was
> not RFC-conforming if making such a statement caused what was a stored valid
> JSON text now non-valid".

I don't propose making any stored valid JSON text become non-valid.
{a: true, a: false} is currently a valid JSON text, and it should
remain so or things would break.

Some JSON parsers consider that document distinguishable from {a:
false} (e.g. streaming parsers that return all key/value pairs). Most
of them don't.

I propose that it be invalid to consider that document
indistinguishable from {a: true}. Parsers are free to accept or reject
it, and if they accept it they are free to return either one or two
"a" entries, but if they only return one it must be {a: false}.

Stephen

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Jun 2013 12:43:33 -0500

> "a" entries, but if they only return one it must be {a: false}.

More briefly: {"a": true, "a": false} MAY be indistinguishable from
either {"a": true, "a": false} or from {"a": false}, but MUST NOT be
indistinguishable from {"a": true}.

That would allow a streaming parser application like this:

while ((name, value) = parser.nextPair()) {
    doCommand(name, value);
}

But not one like this, which I doubt anyone will argue should be allowed:

while ((name, value) = parser.nextPair()) {
    if (hash[name])
        continue;
    hash[name] = value;
    doCommand(name, value);
}

But what about this:

while ((name, value) = parser.nextPair()) {
    if (hash[name])
        throw "duplicate names!";
    hash[name] = value;
    doCommand(name, value);
}

Here {"a": true, "a": false} would be distinguished from {"a": true}
only by the error thrown, but the doCommand("a", true) action will
have run.  Is this invalidated by your construction?  I think it's
not, and one could make a reasonable argument that it should not be.
(In particular {"a": true, "a": false, "b": null} would be equivalent
to {"a": true} + an exception in this last case, but it would be
distinct from {"a": true, "b": null}.)

+1 from me.

---

From: John Cowan <[email-masked]>
Date: Mon, 10 Jun 2013 13:50:36 -0400

> while ((name, value) = parser.nextPair()) {

I do argue that it should be allowed.  Both "assume last" and "assume
first", and for that matter "assume middle", are valid interpretations
of RFC 4627, and ought to continue to be allowed.

-- 
Ambassador Trentino: I've said enough. I'm a man of few words.
Rufus T. Firefly: I'm a man of one word: scram!
        --Duck Soup                     John Cowan <[email-masked]>

---

From: Stefan Drees <[email-masked]>
Date: Mon, 10 Jun 2013 19:51:55 +0200

On 2013-06-10 19:08 CEST, Stephen Dolan wrote:
> On Mon, Jun 10, 2013 at 6:01 PM, Paul Hoffman ... wrote:
>> <no hat>
>>
>> On Jun 10, 2013, at 9:55 AM, Stephen Dolan ... wrote:
>>
> "a" entries, but if they only return one it must be {a: false}. ...

I presume, that if round-tripping - that is taking the "JSONing" as 
reversible operation for whatever is deemed important to keep intact 
over multiple encode-decode cycles - would have been a concern, this 
somehow would have been noted in the JSON RFC right from the start ;-)

In the current discussions I see a lot of focus on the transformation 
from JSON text into some storage structure of another representation 
motivated by consequences thereof.

If I take eg. a conforming JSON parser that keeps even the duplicated 
entries inside a map in the parsers implementation "private 
representation" and than let it act - say as a proxy - and generate 
"matching" JSON text from that again, it is perfectly valid and ok to 
receive:

     {a: true, a: false}

parse this (i.e. store) and emit as (re-)generated JSON text (forward):

     {a: false, a: true}

Right :-?)

There is some traditional asymmetry of "interest" comparing:

A) Private Storage Representation of Generator

and

B) Private Storage Representation of Parser

Isn't it? It looks more like JSON text is "simply found" as the 
interchange of the JSON represented data is the emphasis not the initial 
transformation of whatever that lead to the JSON text in the first place.

So I think what generators MAY introduce as indeterminism into the 
conversion (since years), the parsers will have to deal with (wishes 
aside).
Setting all "hope" on this weak ordering assumption leading to the 
concept of a "last" entry, which breaks down in the proxy sample given 
above, might be misleading.

Please note, that I do believe, that the discussions transported through 
all these - I guess already - 300000 words exchanged back and forth in 
the endeavour to apply some cosmetics to the 2000 words RFC are of 
sufficient value for everyone participating :-)

Stefan.

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Jun 2013 13:04:26 -0500

> of RFC 4627, and ought to continue to be allowed.

I'm OK with disallowing the above, but if we can't get consensus even
on that then what we should do is leave the original text as it was
and move this discussion to the Security Considerations section (to
point out the risks) and the best practices document (to say "best
practice is not to do this").

---

From: Stephan Beal <[email-masked]>
Date: Mon, 10 Jun 2013 22:18:19 +0200

> practice is not to do this").

That also has the strong advantage of not upsetting anyone (==implementors)
who isn't already upset by the status quo (and i haven't seen any posts
suggesting that the status quo is fundamentally broken).

---

From: "John Levine" <[email-masked]>
Date: 10 Jun 2013 23:41:01 -0000

>"a" entries, but if they only return one it must be {a: false}.

That would rule out streaming parsers, or at least require that a
parser have access to arbitrarily large storage to buffer up large
JSON objects, or store all of the names or hash them or otherwise add
signficant processing that existing streaming parsers don't do.  Given
that their current behavior is presumably adequate for their current
applications, it seems unlikely that the programmers will add all that
extra code just because we say to.

This strikes me as an issue that we need to document, not one that we
can legislate away, e.g.

"Parsers that do not create a single in-memory structure for a parsed
object probably will not detect duplicate names."

---

From: Stephen Dolan <[email-masked]>
Date: Tue, 11 Jun 2013 11:51:54 +0100

> extra code just because we say to.

No, it wouldn't. Streaming parsers return both entries.

Stephen