Subject: [Json] Call for real-world examples of how parsers deal with duplicate keys

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 10:41:16 -0700

Greetings again. Knowing what current parsers do with duplicate keys might be useful to this discussion. Of course, some people will exclaim "but that's wrong!" to some of what we see, but seeing it may be useful nonetheless.

I propose the following JSON text for the tests:

{"a":1,"a":2}

--Paul Hoffman

==============================

And I'll start with Python.

from __future__ import print_function
import json
a = '{"a":1,"a":2}'
print(json.loads(a))

# python2.6 keytest.py
{u'a': 2}
# python2.7 keytest.py
{u'a': 2}
# python3.2 keytest.py
{'a': 2}

---

From: Stefan Drees <[email-masked]>
Date: Thu, 06 Jun 2013 19:49:35 +0200

> I propose the following JSON text for the tests:

don't tell anyone, but I'll continue with php (5.3.25):
$> cat test_json.php
<?php
$d = json_decode('{"a":1,"a":2}');
print_r($d);
"""

$> php test_json.php
stdClass Object
(
     [a] => 2
)

All the best,
Stefan.

---

From: Stefan Drees <[email-masked]>
Date: Thu, 06 Jun 2013 20:04:08 +0200

> I propose the following JSON text for the tests:

and also ruby(1.8.7) and the JSON parser as implemented in the json gem:

$> cat test_json.rb
require 'rubygems'
require 'json'
require 'pp'
pp JSON.parse('{"a":1,"a":2}')

$> ruby ./test_json.rb
{"a"=>2}

All the best,
Stefan.

---

From: Riccardo Bernardini <[email-masked]>
Date: Thu, 6 Jun 2013 20:05:21 +0200

Ada language, parser in the package GNATCOLL.JSON (used, I think, also
by the Ada Web Server)

-------- Source code -------
with GNATCOLL.JSON;
use GNATCOLL.JSON;
with Ada.Text_IO; use Ada.Text_IO;

procedure my_test is
   Parsed : constant JSON_Value := Read ("{""a"":1,""a"":2}", "");
   Val : Integer;
begin
   Val := Get (Parsed, "a");
   Put_Line ("A=" & Integer'Image (Val));
end my_test;

----  Output ----

$ ./my_test
A= 2

On Thu, Jun 6, 2013 at 7:49 PM, Stefan Drees <[email-masked]> wrote:
>> Greetings again. Knowing what current parsers do with duplicate keys might
>> be useful to this discussion. Of course, some people will exclaim "but
>> that's wrong!" to some of what we see, but seeing it may be useful
>> nonetheless.
>>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Thu, 6 Jun 2013 18:05:36 +0000

> I propose the following JSON text for the tests:

Experiments should always have a control!  I think we can consider this the control:

-----BEGIN JAVASCRIPT-----
var input = '{"a":1,"a":2}';

var output;
output = JSON.parse(input);
output = JSON.stringify(output);

console.log(output);
-----END JAVASCRIPT-----

# node.js 0.10.8
{"a":2}

# Firefox 22
{"a":2}

# Safari 6.5
{"a":2}

# Chrome 27
{"a":2}

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 06 Jun 2013 20:18:23 +0200

* Paul Hoffman wrote:
>Greetings again. Knowing what current parsers do with duplicate keys 
>might be useful to this discussion. Of course, some people will exclaim 
>"but that's wrong!" to some of what we see, but seeing it may be useful 
>nonetheless.
>

I note that this test is insufficient for a conclusion like "takes the
last value", but it should be okay for concluding "does not reject the
input under default settings". With Perl, the JSON module on CPAN has
a pure-perl module JSON::PP, but would generally prefer a a C-based
module like JSON::XS when available (and `use JSON;` hides the choice).

  #!perl -w
  use JSON::PP;
  use Data::Dumper;
  print Dumper([ JSON::PP->new->decode('{"a":1,"a":2}') ]);

and the same with JSON::XS in place of JSON::PP will print

  $VAR1 = [
            {
              'a' => 2
            }
          ];

In Microsoft's .NET Framework the System.Runtime.Serialization.Json
reader abstracts JSON into XML...

  using System;
  using System.Text;
  using System.Runtime.Serialization.Json;
  
  class Program {
    static void Main(string[] args) {
      var json = "{\"a\":1,\"a\":2}";
      var bytes = UTF8Encoding.UTF8.GetBytes(json);
      var p = JsonReaderWriterFactory.CreateJsonReader(bytes,
        new System.Xml.XmlDictionaryReaderQuotas());
  
      while (p.Read()) {
        Console.WriteLine("{0}", p.ReadOuterXml());
      }
    }
  }

which results in

  <root type="object">
    <a type="number">1</a>
    <a type="number">2</a>
  </root>

This being a streaming parser it reports both instances as they are in
the document.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 6 Jun 2013 12:37:53 -0700

> I propose the following JSON text for the tests:

On Java, Jackson library:

- Exposes both entries (key/value pairs) at streaming parsing level
- Exposes last value via tree model and data-binding (which use streaming
parser for tokenization) -- may optionally report errors during building of
tree model, if error reporting not disabled.

other Java libraries I am familiar with will also do one of three
possibilities, I think GSON for example behaves same as Jackson as it also
uses streaming/builder layering.

-+ Tatu +-

---

From: Vinny A <[email-masked]>
Date: Thu, 6 Jun 2013 14:39:33 -0500

The org.json reference parser causes an exception when it hits a duplicated key. See https://github.com/douglascrockford/JSON-java/blob/master/JSONObject.java , line 1186, in putOnce.

-----------------
Vinny
www.jsontest.com

---

From: Stephan Beal <[email-masked]>
Date: Thu, 6 Jun 2013 21:50:06 +0200

> I propose the following JSON text for the tests:

My C and C++ implementations would result in {"a":2}. That was the only
implementation which made intuitive sense to me (and is faster than
erroring on dupes in the C++ code, which of course uses std::map as the
basis, and avoids me having to search that map to check for a dupe). While
library-level coding effort would not be notably more to error on dupes, it
does add another error case/code for clients to check for, and it just
didn't make sense to me to deal with this (what i consider to be a) corner
case. This approach also, IMO, conforms to the "principal of least
surprise," which for me means "last one wins.'

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Vinny A <[email-masked]>
Date: Thu, 6 Jun 2013 14:33:11 -0700 (PDT)

The Go JSON parser ( http://golang.org/pkg/encoding/json/ ) uses the last name/value entry. The code for my test is below and there's also a runnable version at http://play.golang.org/p/Df1ahCHivB .
 
package main

import "fmt"
import "encoding/json"

func main() {
var jsonBlob = []byte(`
{"A": 1, "A": 2}
`)
type JSONExample struct {
A int
}
var json_example JSONExample
err := json.Unmarshal(jsonBlob, &json_example)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", json_example)
}

Output: {A:2}

----------------
-Vinny
www.jsontest.com

________________________________
 From: Paul Hoffman <[email-masked]>
To: "[email-masked]" <[email-masked]> 
Sent: Thursday, June 6, 2013 12:41 PM
Subject: [Json] Call for real-world examples of how parsers deal with duplicate keys

{"a":1,"a":2}

--Paul Hoffman

# python2.6 keytest.py
{u'a': 2}
# python2.7 keytest.py
{u'a': 2}
# python3.2 keytest.py
{'a': 2}

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 17:32:08 -0500

> {"a":1,"a":2}

jq 1.3 returns 2:

%  echo '{"a":1,"a":2}'|./jq '.a'
2
%

Heimdal's parser takes the last value as well (from code inspection).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 17:35:02 -0500

> - Exposes both entries (key/value pairs) at streaming parsing level

I don't think we should disqualify that sort of streaming parser implementation.

This leads me to conclude that we should distinguish between streaming
and stateful parsers (my terms; please suggest better ones).  Stateful
parsers MUST accept only the last value, while streaming parsers MAY
(and probably always will) accept all duplicate keys' values.

Nico
--

---

From: Tatu Saloranta <[email-masked]>
Date: Thu, 6 Jun 2013 16:12:12 -0700

> (and probably always will) accept all duplicate keys' values.

I agree with this.

It also goes to terminology: term "parser" is being used quite liberally,
meaning anything from tokenizer, to the thing that builds higher level
object representation (JSON-centrics trees, or host language objects), or
combination of the whole thing.
In this respect, related thread that tries to divide specification into
different sections makes sense; physical structure and serialization are
most related to low-level tokenization/generation, and then optional
logical model(s) more to builders/serializers.

Same is true for encoding aspects: at logical model level, underlying
character encoding is irrelevant. But for low-level tokenization it
matters: question like how encoding is obtained; or if no encoding
information is available, what are possible encodings (only UTF-8? UTF-8
and UTF-16 since two can be auto-detected?).

To me separation of physical and logical layers makes sense, even if most
users are not aware of separation of the two: implementors can not ignore
this.

-+ Tatu +-

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 6 Jun 2013 12:51:41 -0700

> output = JSON.stringify(output);

The ECMAScript standard requires the behavior shown above.

Allen