Subject: [Json] Allow any JSON value at the top level

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 13:51:09 +1000

The current spec only allows a JSON text to be an object or array. Section 2. "JSON Grammar" says:

   A JSON text is a serialized object or array.

   JSON-text = object / array

I propose allowing a JSON text to be any JSON value.

   A JSON text is a serialization of any JSON value.

   JSON-text = value

   value = false / null / true / object / array / number / string

ECMAScript already allows this. http://www.ecma-international.org/ecma-262/5.1/#sec-15.12

   * The top level JSONText production of the ECMAScript JSON grammar
     may consist of any JSONValue rather than being restricted to being
     a JSONObject or a JSONArray as specified by RFC 4627.

--
James Manger

---

From: R S <[email-masked]>
Date: Thu, 6 Jun 2013 20:54:19 -0700

I support this change. It is required in interoperable implementations.

On Thu, Jun 6, 2013 at 8:51 PM, Manger, James H <
[email-masked]> wrote:

> ECMAScript already allows this.
> http://www.ecma-international.org/ecma-262/5.1/#sec-15.12
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 08:45:30 +0200

On 07.06.13 05:54, "R S" wrote:
> On Thu, Jun 6, 2013 at 8:51 PM, Manger, James H ... wrote:
>> ...

> I support this change. It is required in interoperable implementations.

+0 from me, but I expect a lot of discussion on this one, due to 
encoding detection tricks depending on the first character (or whatever 
you may call that) c.f. section 3. Encoding second paragraph in current 
revision:
"""
    Since the first two characters of a JSON text will always be ASCII
    characters [RFC0020], it is possible to determine whether an octet
    stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
    at the pattern of nulls in the first four octets.
"""

So proposing '{"Äther":null}' could be transmitted as '"Äther"' is nice, 
but not compatible with the above "trick/hack", is it? (The single 
quotes are of course not part of the JSON samples, but shall only 
delimit the latter inside this text ;-)

Stefan.

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 08:58:43 +0200

consistency correction below, sorry for the inconvenience.
<data:application/json,"Stefan">
> encoding detection tricks depending on the first character (or whatever

sorry, of course "first two characters" instead of "first character".

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 17:04:17 +1000

> So proposing '{"Äther":null}' could be transmitted as '"Äther"' is
> nice, but not compatible with the above "trick/hack", is it? (The
> single quotes are of course not part of the JSON samples, but shall
> only delimit the latter inside this text ;-)

Encoding detection is still possible, even though the first 2 characters will not always by ASCII. Since U+0000 MUST be escaped in a JSON string that cannot introduce 0x00 bytes to confuse things.

Adjust the sentence to start:

    Since the first character of a JSON text will always be ASCII [RFC0020]
    and a U+0000 character is not allowed in unescaped form, it is possible...

Adjust the UTF-16 patterns to:

    00 xx xx xx  UTF-16BE
    xx 00 xx xx  UTF-16LE

--
James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 17:12:22 +1000

> Encoding detection is still possible, even though the first 2
> characters will not always by ASCII. Since U+0000 MUST be escaped in a
> JSON string that cannot introduce 0x00 bytes to confuse things.
> 

Add those patterns, don't replace the existing ones. The table becomes:

   00 00 00 xx  UTF-32BE
   00 xx 00 xx  UTF-16BE
   00 xx xx xx  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 xx 00  UTF-16LE
   xx 00 xx xx  UTF-16LE
   xx xx xx xx  UTF-8

"old" parsers might reject "new" JSON matching one of the 2 new patterns, but the "old" parser would have rejected the new string-at-top-level anyway.

--
James Manger

---

From: "Pete Cordell" <[email-masked]>
Date: Fri, 7 Jun 2013 09:22:37 +0100

Original Message From: "Manger, James H"
> Add those patterns, don't replace the existing ones. The table becomes:

If xx means non-zero, then I think we also have to include the following for 
characters like U+2c00:

    xx 00 00 xx  UTF-16LE

giving:

   00 00 00 xx  UTF-32BE
   00 xx 00 xx  UTF-16BE
   00 xx xx xx  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 00 xx  UTF-16LE
   xx 00 xx 00  UTF-16LE
   xx 00 xx xx  UTF-16LE
   xx xx xx xx  UTF-8

That can be reduced a bit if we use "--" to indicate "not-tested":

   00 00 -- --  UTF-32BE
   00 xx -- --  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 00 xx  UTF-16LE
   xx 00 xx --  UTF-16LE
   xx xx -- --  UTF-8

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 10:48:42 +0200

On 2013-06-07 10:22, Pete Cordell wrote:
>> ... {"content":"snipped"}
>
> If xx means non-zero, then I think we also have to include the following
> for characters like U+2c00: ...
> ... {"content":"snipped"}
>

+1 for this reduced display.

Stefan.

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 7 Jun 2013 10:59:42 +0200

>   A JSON text is a serialization of any JSON value.

Apart from the (intellectually nicely challenging, but practically irrelevant) auto-detection thing, one other result is that JSON texts no longer stream.  RFC 4627 texts are self-delimiting, the new ones wouldn't always be.  Of course, if you want to stream sequences of JSON texts, you could always limit those back to arrays or maps ("objects").

Grüße, Carsten

---

From: "Markus Lanthaler" <[email-masked]>
Date: Fri, 7 Jun 2013 11:20:58 +0200

>    value = false / null / true / object / array / number / string

I'm strictly against this. Not only would it break many implementations but
also specs build on top of JSON that rely on JSON-text = object / array.

--
Markus Lanthaler
@markuslanthaler

---

From: Stephan Beal <[email-masked]>
Date: Fri, 7 Jun 2013 15:06:51 +0200

> also specs build on top of JSON that rely on JSON-text = object / array.

+Inf

(even though JSON doesn't support it ;)

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Vinny A <[email-masked]>
Date: Fri, 7 Jun 2013 08:38:15 -0500

On Jun 7, 2013, at 3:59 AM, Carsten Bormann <[email-masked]> wrote:
> Apart from auto-detection, one other result is that JSON texts no longer stream.  RFC 4627 texts are self-delimiting, the new ones wouldn't always be.  Of course, if you want to stream sequences of JSON texts, you could always limit those back to arrays or maps

I'm +0 on this topic, partly for Carsten's reason, but also because this looks to be a breaking change.

Do commonly used parsers support top level JSON values?

-----------------
Vinny
www.jsontest.com

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 Jun 2013 08:40:08 -0700

<no hat>

>   value = false / null / true / object / array / number / string

+1 to this, and to removing the "how to tell if you are looking at a JSON text" from the parser section. That rule is fragile and not useful.

I am interested to hear examples of where this change would cause breakage, particularly given that ECMAScript already has this rule.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 08:57:16 -0700

...

> I propose allowing a JSON text to be any JSON value.

-1.

1. Our mandate is not to enhance or improve JSON.
2. This is a bad idea in general and a terrible idea for network protocols,
which is where JSON is most valuable.

-T

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 18:07:05 +0200

On 07.06.13 17:40, Paul Hoffman wrote:
> +1 to this, and to removing the "how to tell if you are looking at a
> JSON text" from the parser section. That rule is fragile and not useful.
>

+0 as long as the future encoding detection tricks in section 3 are 
coherent with it.

> I am interested to hear examples of where this change would cause
> breakage, particularly given that ECMAScript already has this rule.

I expect that all the php, python whatever (de-)serializers will deliver 
no surprises. Sample PHP:

$> cat ./json_test_null.php
<?php
foreach(array('false','null','true','{}','[]',42,'"Äther"') as $value) {
     $d = json_decode($value);
     echo strval($value) . ' transforms to: "'
         . print_r($d, TRUE)
         . '"" of type: ' . gettype($d)
         . "\n";
}

$> php ./json_test_null.php
false transforms to: """ of type: boolean
null transforms to: """ of type: NULL
true transforms to: "1"" of type: boolean
{} transforms to: "stdClass Object
(
)
"" of type: object
[] transforms to: "Array
(
)
"" of type: array
42 transforms to: "42"" of type: integer
"Äther" transforms to: "Äther"" of type: string

So all well with php (5.3.25).

All the ebst,
Stefan.

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 7 Jun 2013 16:22:17 +0000

> which is where JSON is most valuable.

/me doffs hat

This is one area where I think we need experimentation before we can consider such as change.  It very well could be that "everyone" (for some definition of everyone) has ignored this particular restriction in order to maximize compatibility with ECMAScript, in which case maintaining this condition means everyone is technically non-compliant.

However, I don't know that with any degree of certainty.

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 16:43:42 +0000

I like this change, but it has impact on section three.  These JSON docs:

0
""
"?"

means that the table should probably look like:

   00 00 00 xx  UTF-32BE
   00 xx ?? xx  UTF-16BE
   xx 00 00 00  UTF-32LE
   xx 00 xx ?? UTF-16LE
   xx xx ?? ?? UTF-8

algorithmically:

if length<2
  return UTF-8
if buf[0]
  if buf[1]
    return UTF-8
  if length<4 or buf[2]
    return UTF-16LE
  return UTF-32LE
if buf[1]
  return UTF-16BE
return UTF-32BE

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 12:48:03 -0400

> breakage, particularly given that ECMAScript already has this rule.

I think the self-bounding nature of current JSON is important to preserve.

-- 
In politics, obedience and support      John Cowan <[email-masked]>
are the same thing.  --Hannah Arendt    http://www.ccil.org/~cowan

---

From: R S <[email-masked]>
Date: Fri, 7 Jun 2013 09:50:59 -0700

On Fri, Jun 7, 2013 at 9:22 AM, Matt Miller (mamille2)
<[email-masked]>wrote:

>
> This is one area where I think we need experimentation before we can
> consider such as change.  It very well could be that "everyone" (for some
> definition of everyone) has ignored this particular restriction in order to
> maximize compatibility with ECMAScript, in which case maintaining this
> condition means everyone is technically non-compliant.
>

RFC 4627 says "A JSON parser MAY accept non-JSON forms or extensions." and
implementations that are compatible with ECMAScript are common enough that
it seems this extension to RFC 4627 actually doesn't cause problems with
applications expecting only arrays or objects at the root.

Does anyone have examples of this behavior creating problems in the wild?
They must be out there, since support for this extension is so common.

- Rob

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 11:52:44 -0500

On Fri, Jun 7, 2013 at 11:07 AM, Stefan Drees <[email-masked]> wrote:
> +0 as long as the future encoding detection tricks in section 3 are coherent
> with it.

If the top-level may be any value then we still have the first
character being ASCII and the section 3 encoding detection algorithm
still works:

           00 00 00 xx  UTF-32BE
           00 xx  UTF-16BE
           xx 00  UTF-32LE
           xx 00  UTF-16LE
           xx xx  UTF-8

I'm in favor of this change, but I'd be happy to have the new RFC
allow for parsers that can't handle this.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 17:08:40 +0000

>also specs build on top of JSON that rely on JSON-text = object / array.

What about a new production in parallel to JSON-text, like 'JSON-document'?

-- 
Joe Hildebrand

---

From: Jacob Davies <[email-masked]>
Date: Fri, 7 Jun 2013 10:33:01 -0700

> I am interested to hear examples of where this change would cause breakage, particularly given that ECMAScript already has this rule.

+1

So do many parsers. Since parsers can reject values they don't like, I
don't think this will cause breakage (parsers that reject
non-object/non-array values are still compliant, they just aren't
interested in your part of the value space).

This would just allow what lots of people do already with no apparent harm.

---

From: Tatu Saloranta <[email-masked]>
Date: Fri, 7 Jun 2013 10:54:52 -0700

On Fri, Jun 7, 2013 at 1:59 AM, Carsten Bormann <[email-masked]> wrote:

> Apart from the (intellectually nicely challenging, but practically
> irrelevant) auto-detection thing,

Irrelevant based on what? I thought this is what any half-decent JSON
parser did; unless platform does not expose byte sequence as input, case
for some scripting languages.
I have written multiple parsers (json, xml) that do just this, and know
that others exist as well.

I don't know what leads to assumption of irrelevancy here, and it should be
fully spelled out.

one other result is that JSON texts no longer stream.  RFC 4627 texts are
> self-delimiting, the new ones wouldn't always be.  Of course, if you want
> to stream sequences of JSON texts, you could always limit those back to
> arrays or maps ("objects").
>

I don't see this as any less self-delimiting, although best-practices would
probably dictate suggestion to add white space between root-level values.

Which is what existing tools that allow this do, as expected.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Fri, 7 Jun 2013 10:56:18 -0700

On Fri, Jun 7, 2013 at 6:38 AM, Vinny A <[email-masked]> wrote:

> > Apart from auto-detection, one other result is that JSON texts no longer
> stream.  RFC 4627 texts are self-delimiting, the new ones wouldn't always
> be.  Of course, if you want to stream sequences of JSON texts, you could
> always limit those back to arrays or maps
>
> Do commonly used parsers support top level JSON values
>

Yes, at least on Java platform.

Follow-up question would be whether use cases exist for sending root-level
non-structured values: that I do not know. I do know that some JSON
validators struggle with this question (should it be allowed but flagged or
marked as error); and that suggests that usage exists.

-+ Tatu +-

---

From: "Markus Lanthaler" <[email-masked]>
Date: Fri, 7 Jun 2013 22:11:06 +0200

On Friday, June 07, 2013 7:09 PM, Joe Hildebrand (jhildebr) wrote:
> >I'm strictly against this. Not only would it break many
> implementations but also specs build on top of JSON that rely on
> JSON-text = object / array.
> 

And how is that production used? A "JSON document", i.e., a representation
labeled as application/json must be an array or an object at the top-level.
If you want something else, then it's clearly out of scope for this WG IMO.
You can of course write another RFC standardizing JSON+ :-P

--
Markus Lanthaler
@markuslanthaler

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 15:19:48 -0500

> character being ASCII and the section 3 encoding detection algorithm

Ah, no, what a braino.  /me is embarrassed.

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 15:53:15 -0500

> Ah, no, what a braino.  /me is embarrassed.

No, not a braino after all.  Since we'd not have two NUL bytes in a
row in any encoding other than UTF-32.  There'd be no ambiguity, even
with single-character values (numbers).

---

From: Jorge <[email-masked]>
Date: Fri, 7 Jun 2013 22:02:43 +0200

> Do commonly used parsers support top level JSON values?

Yes:

JSON.parse('"string"')
"string"
JSON.parse('27')
27
JSON.parse('true')
true
JSON.parse('false')
false
JSON.parse('null')
null

-- 
( Jorge )();

---

From: R S <[email-masked]>
Date: Fri, 7 Jun 2013 14:37:49 -0700

I believe this proposal is in scope--it's one of the inconsistencies with
ECMAScript this group has been tasked with resolving, and it was mentioned
during the chartering process.

Are there interoperability problems we should be aware of? In my
experience, this change doesn't matter to applications that can only handle
objects or arrays--they simply encounter an error immediately after parsing
rather than during it.

- Rob

On Friday, June 7, 2013, Markus Lanthaler wrote:

> > On 6/7/13 3:20 AM, "Markus Lanthaler" <[email-masked]<javascript:;>>
> wrote:
> >
> [email-masked] <javascript:;>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 8 Jun 2013 00:04:11 +0200

> What about a new production in parallel to JSON-text, like 'JSON-document'?

I'd call it JSON-value, and write a separate document that defines a new media type

	application/json-value

No gratuitous change, no interoperability problems.

(Actually, we have the production "value" already, so anyone can go ahead and register that media type.)

Grüße, Carsten

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sat, 08 Jun 2013 12:12:15 +0900

What programming language/library?

Regards,    Martin.

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 23:28:15 -0400

> ahead and register that media type.)

If there's demand for this, we can register it in this document: no separate
document needed.

-- 
If you have ever wondered if you are in hell,         John Cowan
it has been said, then you are on a well-traveled     http://www.ccil.org/~cowan
road of spiritual inquiry.  If you are absolutely     [email-masked]
sure you are in hell, however, then you must be
on the Cross Bronx Expressway.          --Alan Feuer, NYTimes, 2002-09-20

---

From: Jorge <[email-masked]>
Date: Sat, 8 Jun 2013 15:12:03 +0200

> What programming language/library?

JavaScript's built-in parser (ES5).

I tried it with ruby's JSON gem but it won't swallow it:

>> JSON.parse("\"string\"")
JSON::ParserError: 757: unexpected token at '"string"'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from (irb):14
>> JSON.parse("27")
JSON::ParserError: 757: unexpected token at '27'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from (irb):15
>> JSON.parse("true")
JSON::ParserError: 757: unexpected token at 'true'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from (irb):16
>> JSON.parse("false")
JSON::ParserError: 757: unexpected token at 'false'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from (irb):17
>> JSON.parse("null")
JSON::ParserError: 757: unexpected token at 'null'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from /Library/Ruby/Gems/1.8/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
	from (irb):18
>> JSON.parse("[27]")
=> [27]

-- 
( Jorge )();

---

From: "Manger, James H" <[email-masked]>
Date: Sat, 8 Jun 2013 23:25:03 +1000

Tim Bray said:
> 2. This is a bad idea in general and a terrible idea for network
> protocols, which is where JSON is most valuable.

Tim, could you expand on why this is bad and terrible?

I would have thought an HTTP request like this would be a reasonable design, and shouldn't require a new media type to be minted:

-> GET /path/to/resource.json?part=json-pointer

<- Content-Type: application/json
   <JSON value selected by JSON pointer in request>

--
James Manger

---

From: "Markus Lanthaler" <[email-masked]>
Date: Sun, 9 Jun 2013 21:46:40 +0200

On Friday, June 07, 2013 11:38 PM, R S wrote:
> I believe this proposal is in scope--it's one of the inconsistencies
> with ECMAScript this group has been tasked with resolving, and it was
> mentioned during the chartering process.

I disagree. Allowing something else than an object or an array at the
top-level is clearly out of scope IMO.

> experience, this change doesn't matter to applications that can only
> handle objects or arrays--they simply encounter an error immediately
> after parsing rather than during it.

Well, changing the {}s to <>s would also result in an error but that doesn't
mean that it won't cause problems. The most important thing for this WG is
to preserve backwards-compatibility. Turning JSON parsers that are currently
completely conformant obsolete and arguing that they would error clearly
breaks backwards compatibility. As someone else showed in a different
thread, this would e.g. break Rail's default JSON parser.

--
Markus Lanthaler
@markuslanthaler

---

From: R S <[email-masked]>
Date: Sun, 9 Jun 2013 13:24:30 -0700

On Sun, Jun 9, 2013 at 12:46 PM, Markus Lanthaler
<[email-masked]>wrote:

> mean that it won't cause problems.
>

That would not be backward compatible.

> The most important thing for this WG is
> breaks backwards compatibility.
>

"In telecommunications and computing, a product or technology is backward
or downward compatible if it can work with input generated by an older
product or technology." <
https://en.wikipedia.org/wiki/Backward_compatibility>

Arrays and Objects at the root will still work, obviously, so this proposal
can be said to be backward compatible.

> As someone else showed in a different

How would it break that parser? My employer happens to operate a very large
Rails site with heavy JSON traffic. We use a popular open-source JSON gem.
It supports the primitive root values proposed here.

- Rob

---

From: "Markus Lanthaler" <[email-masked]>
Date: Sun, 9 Jun 2013 22:57:12 +0200

On Sunday, June 09, 2013 10:25 PM, R S wrote:
>> to preserve backwards-compatibility. Turning JSON parsers
>> that are currently completely conformant obsolete and arguing
>> that they would error clearly breaks backwards compatibility.
>
> "In telecommunications and computing, a product or technology
> is backward or downward compatible if it can work with input
> generated by an older product or technology."
> <https://en.wikipedia.org/wiki/Backward_compatibility>
>
> Arrays and Objects at the root will still work, obviously,
> so this proposal can be said to be backward compatible.

You are of course completely right, backward compatibility was the wrong
term here. Sorry.

> How would it break that parser? My employer happens to operate
> a very large Rails site with heavy JSON traffic. We use a popular
> open-source JSON gem. It supports the primitive root values proposed
> here.

See http://www.ietf.org/mail-archive/web/json/current/msg00670.html

Why can't a separate media type, e.g., application/json-value, be defined
allowing this? The fact that that hasn't been done in the last decade should
tell us something. Why do we have to change JSON in such a fundamental way?

--
Markus Lanthaler
@markuslanthaler

---

From: Nico Williams <[email-masked]>
Date: Sun, 9 Jun 2013 16:08:18 -0500

At least several encoders and parsers allow any value type at the
top-level, either as an option or no matter what.

---

From: Jorge <[email-masked]>
Date: Sun, 9 Jun 2013 23:39:19 +0200

On 09/06/2013, at 23:08, Nico Williams wrote:

> At least several encoders and parsers allow any value type at the top-level, either as an option or no matter what.

All the current browsers can both .parse() and .stringify() strings, numbers, null and booleans at the top-level, per the ES5 specs (*):

JSON.parse(JSON.stringify("string"))
"string"
JSON.parse(JSON.stringify(27))
27
JSON.parse(JSON.stringify(true))
true
JSON.parse(JSON.stringify(false))
false
JSON.parse(JSON.stringify(null))
null
JSON.parse(JSON.stringify([27]))
[27]
JSON.parse(JSON.stringify({a:1}))
{a:1}

(*) <http://ecma-international.org/ecma-262/5.1/#sec-15.12.1.2>

<http://ecma-international.org/ecma-262/5.1/#sec-15.12.2>
"The parse function parses a JSON text (a JSON-formatted String) and produces an ECMAScript value. The JSON format is a restricted form of ECMAScript literal. JSON objects are realized as ECMAScript objects. JSON arrays are realized as ECMAScript arrays. JSON strings, numbers, booleans, and null are realized as ECMAScript Strings, Numbers, Booleans, and null."

<http://ecma-international.org/ecma-262/5.1/#sec-15.12.3>
"The stringify function returns a String in JSON format representing an ECMAScript value. It can take three parameters. The first parameter is required. The value parameter is an ECMAScript value, which is usually an object or array, although it can also be a String, Boolean, Number or null."
-- 
( Jorge )();

---

From: "Markus Lanthaler" <[email-masked]>
Date: Mon, 10 Jun 2013 00:54:18 +0200

On Sunday, June 09, 2013 11:06 PM, R S wrote:
>> Why can't a separate media type, e.g., application/json-value,
>> be defined allowing this? 
>
> We could do that, but it might be a bit pointless. It won't change
> the fact that primitive values are already sent as application/json.

Who's doing that?

--
Markus Lanthaler
@markuslanthaler

---

From: Jacob Davies <[email-masked]>
Date: Mon, 10 Jun 2013 12:32:55 -0700

Lots of people do this, in my experience, including me pretty often.

The original specification has an asymmetry in what values may be sent
at the top level motivated by an encoding-detection scheme that is now
obsolete. Some people seem to like it because composite values are a
good idea for top-level payloads, but if that were the case one ought
to object on similar principle to any value that does not permit
further arbitrary sub-structure (like a number). Sometimes a number is
just a number. And arrays are a poor choice for arbitrarily complex
metadata anyway.

The big problem with the current requirement is that arbitrary
sub-values of a JSON value cannot be sent as application/json. So for
instance, if you wish to implement a form of PATCH on a JSON document,
you must demand that clients wrap their new value in a JSON object or
array instead of sending raw primitive values. If you implement a
service returning an inherently primitive value (e.g. a plain-text
document server) you must wrap your responses in a JSON object or
array even though there is nothing else that could reasonably be
added. A system designer might be justified in saying that every
response is an object, but the spec shouldn't dictate what is really a
matter of taste (use of in-band rather than out-of-band metadata).

---

From: John Cowan <[email-masked]>
Date: Mon, 10 Jun 2013 16:17:46 -0400

Jacob Davies scripsit:

> obsolete. 

How do we know what the motivation is?  I think the self-delimiting nature
is more important.

In any case, even if numbers, strings, booleans, and null were allowed at
the top level, all application/json entity bodies would still begin with
a non-null ASCII character, which is sufficient for encoding detection.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
Any sufficiently-complicated C or Fortran program contains an ad-hoc,
informally-specified bug-ridden slow implementation of half of Common Lisp.
        --Greenspun's Tenth Rule of Programming (rules 1-9 are unknown)

---

From: Jacob Davies <[email-masked]>
Date: Mon, 10 Jun 2013 14:36:24 -0700

> How do we know what the motivation is?  I think the self-delimiting nature

The encoding-detection is the only motivation mentioned in the RFC,
and then only implicitly: "Since the first two characters of a JSON
text will always be ASCII characters [RFC0020], it is possible to
determine whether an octet stream is UTF-8, UTF-16 (BE or LE), or UTF
-32 (BE or LE) by looking at the pattern of nulls in the first four
octets."

Of course there can be other notable benefits or drawbacks regardless
of the stated motivation. Self-delimiting is a benefit, it appears.
The fact that a sub-part of a valid JSON text is not also a valid JSON
text is a drawback in an otherwise symmetric data model.

> a non-null ASCII character, which is sufficient for encoding detection.

Good to know.

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Jun 2013 17:18:17 -0500

> text is a drawback in an otherwise symmetric data model.

All JSON values are self-delimited regardless of whether at the
top-level or not.  That is, we always know how to find the end of the
value (if it is in the message).  Lengths of values are always
indefinite (i.e., the parser has to scan forward to find the end of
the value).

>> a non-null ASCII character, which is sufficient for encoding detection.

Right.  We can have one-character values (must be numeric) or
two-or-more-character values (strings, boolean, null).  The second
character in the string case might not be ASCII though, but it still
works out as long as \u0000 is always escaped, and this is what we end
up with

Two or more characters:

00 00 00 xx  UTF-32BE
00 xx xx xx  UTF-16BE
xx 00 00 00  UTF-32LE
xx 00 xx xx  UTF-16LE
xx xx xx xx  UTF-8

One-character numeric top-level values:

00 00 00 xx  UTF-32BE
00 xx <end>  UTF-16BE
xx 00 00 00  UTF-32LE
xx 00 <end>  UTF-16LE
xx <end>     UTF-8

So the section 3 text would have to be updated to include the above
patterns, which can be further compacted to:

00 00 00 xx  UTF-32BE
xx 00 00 00  UTF-32LE
00 xx .. ..  UTF-16BE
xx 00 .. ..  UTF-16LE
xx .. .. ..  UTF-8

If the parser checks for these patterns in order (other orders are
also possible).  Here '..' means "any or no byte values".

Nico
--

---

From: John Cowan <[email-masked]>
Date: Mon, 10 Jun 2013 18:29:13 -0400

> indefinite (i.e., the parser has to scan forward to find the end of

What I meant was that if the message is truncated within an object, array,
boolean, or null, it is detectable: not so if it is truncated within a
top-level number.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Nobody expects the RESTifarian Inquisition!  Our chief weapon is
surprise ... surprise and tedium  ... tedium and surprise ....
Our two weapons are tedium and surprise ... and ruthless disregard
for unpleasant facts....  Our three weapons are tedium, surprise, and
ruthless disregard ... and an almost fanatical devotion to Roy Fielding....

---

From: Jorge <[email-masked]>
Date: Tue, 11 Jun 2013 00:53:10 +0200

It seems to me that top-level JSON values are a problem *only* for *streaming* parsers.

A streaming parser can't tell whether the stream 333 was 3 separate threes or three-hundred-thirty-three, or even part of an incomplete 333445, or (chop it as you like).

Parsing a stream with the other data types (strings, null, booleans): e.g. say, this stream: "string"truenullfalse"anotherstring"null"string"true"etc. etc." doesn't seem to be so much a problem (unless it gets out of sync!).

But non-streaming parsers (such as the ones in EcmaScript 5) haven't any of these problems because they expect to receive a complete JSON payload containing a single JSON value (structured or not) and that's easy to parse unambiguously.
-- 
( Jorge )();

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Jun 2013 18:26:06 -0500

> A streaming parser can't tell whether the stream 333 was 3 separate threes or three-hundred-thirty-three, or even part of an incomplete 333445, or (chop it as you like).

Ahhh.

> But non-streaming parsers (such as the ones in EcmaScript 5) haven't any of these problems because they expect to receive a complete JSON payload containing a single JSON value (structured or not)…

We could require that numeric top-level values be followed by one or
more space characters.

---

From: Tatu Saloranta <[email-masked]>
Date: Mon, 10 Jun 2013 16:49:05 -0700

On Mon, Jun 10, 2013 at 3:53 PM, Jorge <[email-masked]> wrote:

> A streaming parser can't tell whether the stream 333 was 3 separate threes
> or three-hundred-thirty-three, or even part of an incomplete 333445, or
> (chop it as you like).
>
> Parsing a stream with the other data types (strings, null, booleans): e.g.
> say, this stream: "string"truenullfalse"anotherstring"null"string"true"etc.
> etc." doesn't seem to be so much a problem (unless it gets out of sync!).
>
> But non-streaming parsers (such as the ones in EcmaScript 5) haven't any
> of these problems because they expect to receive a complete JSON payload
> containing a single JSON value (structured or not) and that's easy to parse
> unambiguously.
>

FWIW, streaming parsers I have been involved with expect separating
white-space and have no problem telling these cases apart; although it does
require a lookup to detect a white-space or end-of-stream.
Although there is no support from specification (since non-structured root
values are not specified as acceptable), this seems intuitive from user
perspective based on feedback I have gotten on this. This despite there
being theoretical problems (like: should commas be allowed or required, for
"implicit array" approach).

But what John C pointed out -- truncation of '333' from full '333445' -- is
actually an interesting and valid potential issue.

-+ Tatu +-

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 16:56:35 -0700

<no hat>

Although I earlier supported the idea of any JSON value at the top level, a bunch of recent posts show that this is probably a significant revision to the RFC because it would take a lot of extra requirements as well. I now support leaving the requirement in the -bis document as it is now.

If we want to add a new MIME type for "any JSON item", that is clearly a significant addition to the RFC. It can wait until the WG recharters.

--Paul Hoffman

---

From: R S <[email-masked]>
Date: Mon, 10 Jun 2013 17:18:04 -0700

On Mon, Jun 10, 2013 at 4:56 PM, Paul Hoffman <[email-masked]> wrote:

> <no hat>
>
> Although I earlier supported the idea of any JSON value at the top level,
> a bunch of recent posts show that this is probably a significant revision
> to the RFC because it would take a lot of extra requirements as well. I now
> support leaving the requirement in the -bis document as it is now.
>

Leaving the document as-is does not seem to be an option per the charter:

"It is acknowledged that there are differences between RFC 4627 and the
ECMAScript specification in the rules for parsing JSON. Any changes that
break compatibility with existing implementations of either RFC 4627 or
the ECMAScript specification will need to have very strong justification
and broad support. All differences between RFC 4627 or the current
ECMAScript specification will be documented in the new RFC."

> If we want to add a new MIME type for "any JSON item", that is clearly a
> significant addition to the RFC. It can wait until the WG recharters.

A new media type would solve any problem that I'm aware of.

- Rob

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Jun 2013 19:30:15 -0500

> If we want to add a new MIME type for "any JSON item", that is clearly a significant addition to the RFC. It can wait until the WG recharters.

I see no reason [yet... the way these threads have been going...] why
we couldn't recommend that parsers be able to parse any top-level
value, with the only interesting note being that end-of-message
termination of numeric top-level values must not be accidentally
confused with premature EOF.  Also, we could just allow any
non-numeric top-level value.  With a similar recommendation for
encoders.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 19:05:51 -0700

> non-numeric top-level value.  With a similar recommendation for

Why add any recommendation for parsers for that case? The current doc already has:

   A JSON parser MAY accept non-JSON forms or extensions.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Mon, 10 Jun 2013 22:11:08 -0400

>    A JSON parser MAY accept non-JSON forms or extensions.

That combined with the lack of a data model, or any requirements on
parsers to report anything, means that `cat` counts as a JSON parser
currently: it will not reject any legal JSON, it accepts non-JSON
forms (to wit, anything), and it reports its input exactly.

Is it in scope to fix that?  That would mean some kind of data model
and reporting requirements in section 4.

-- 
Babies are born as a result of the              John Cowan
mating between men and women, and most          http://www.ccil.org/~cowan
men and women enjoy mating.                     [email-masked]
    --Isaac Asimov in Earth: Our Crowded Spaceship

---

From: Tim Bray <[email-masked]>
Date: Mon, 10 Jun 2013 19:43:59 -0700

Rob, just to make sure I understand, you’re saying that because ECMA
doesn’t impose the 4627 restriction that a top-level object has to be a
composite, 4627bis has to be changed to acknowledge that?  -T

On Mon, Jun 10, 2013 at 5:18 PM, R S <[email-masked]> wrote:

> On Mon, Jun 10, 2013 at 4:56 PM, Paul Hoffman <[email-masked]>wrote:
>
>> <no hat>
>>

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 19:57:40 -0700

On Jun 10, 2013, at 7:38 PM, R S <[email-masked]> wrote:

> On Mon, Jun 10, 2013 at 7:04 PM, Paul Hoffman <[email-masked]> wrote:
> 
> Good catch. When I said "as-is", I meant "don't change the current text". We need to add a point to the differences section for 4627-ECMAScript, and would need to duplicate that in the section for the current-doc-ECMAScript section.
> 
> Wouldn't that be a section that amounts to "here's what you need to do to interoperate with browsers and most other popular implementations"?

Oh gopod no. I certainly hope the WG find that defining "browser" and "most" and "popular" is out of scope.

> Many endpoints only accept specific root elements. For example, json-patch only accepts arrays. In other words, no one will notice if your JSON parser only accepts objects and arrays. 
> 
> What is the benefit of preserving RFC 4627 requirements? I don't understand the rationale accompanying your opinion. The charter mandates that the WG document top-level primitive parsing, no matter what.

Fully agree on the last part. But "document" does not mean "change the requirements". We can say "the requirement is X" and "X differs from 4627 and/or ECMAScript", at least in my reading of the charter.

--Paul Hoffman

---

From: R S <[email-masked]>
Date: Mon, 10 Jun 2013 20:06:18 -0700

> composite, 4627bis has to be changed to acknowledge that?  -T

I think the charter obligates the WG to document the difference. Whether
that takes the form of a normative requirement is another matter.

On a practical level, I'm not sure what the value of an informational
section rather than a requirement would be. It would be great if someone
could give a rationale for that.

I think there's value in having the Standards-Track RFC be an accurate
description of requirements for interoperability. What good does it do us
to have a non-normative section that is actually required to interoperate
with browsers and most other implementations?

Speaking as someone that updated a widely-used implementation to support
primitive root values,

- Rob

>>> Although I earlier supported the idea of any JSON value at the top
>>> level, a bunch of recent posts show that this is probably a significant
>>> revision to the RFC because it would take a lot of extra requirements as
>>> well. I now support leaving the requirement in the -bis document as it is
>>> now.
>>>
>>

---

From: "Manger, James H" <[email-masked]>
Date: Tue, 11 Jun 2013 14:19:23 +1000

> A new media type would solve any problem that I'm aware of.

A new media type for any JSON value (not just object or array) could work in theory, but I doubt it can work in practice.

It would significantly complicate conversations about JSON. Mentions of JSON in APIs, protocols, specs etc would have to qualify whether or not they support both, or just one, of the media types. Two media types probably need to be advertised in HTTP Accept headers. There would be two choices for media type when sending a JSON object or array, which would produce different behaviour in some cases (eg "old" systems that don't know about the new type).

Would media types with a +json suffix allow any JSON values? Or would we need a second suffix as well?

It would be better to allow any JSON value with the existing media type and suffix.

A top-level JSON value would no longer necessarily be self-delimiting. I doubt this is that crucial. Many media types are not self-delimiting. Even JSON objects and arrays are not strictly self-delimiting as they allow leading and trailing whitespace. Appending whitespace to a JSON value would make as self-delimiting again – restoring this property where needed (allowing whitespace around non-object-non-array top-level values would be a sensible part of the change).

--
James Manger

---

From: R S <[email-masked]>
Date: Mon, 10 Jun 2013 21:20:28 -0700

On Mon, Jun 10, 2013 at 9:19 PM, Manger, James H <
[email-masked]> wrote:

> A new media type for any JSON value (not just object or array) could work
> in theory, but I doubt it can work in practice.

Oh, yikes, I meant "would /not/ solve any problem that I'm aware of."

- Rob

---

From: R S <[email-masked]>
Date: Mon, 10 Jun 2013 21:27:42 -0700

On Mon, Jun 10, 2013 at 7:57 PM, Paul Hoffman <[email-masked]> wrote:

>
> ...I certainly hope the WG find that defining "browser" and "most" and
> "popular" is out of scope.

That is not a very charitable interpretation of what I wrote. For my part,
I certainly hope the WG doesn't stick its head in the sand and say "only
objects and arrays are JSON, because we say so".

> > Many endpoints only accept specific root elements. For example,
> json-patch only accepts arrays. In other words, no one will notice if your
> JSON parser only accepts objects and arrays.
> >
> > What is the benefit of preserving RFC 4627 requirements? I don't
> understand the rationale accompanying your opinion. The charter mandates
> that the WG document top-level primitive parsing, no matter what.
>
> Fully agree on the last part. But "document" does not mean "change the
> requirements". We can say "the requirement is X" and "X differs from 4627
> and/or ECMAScript", at least in my reading of the charter.

I agree with that. But ECMAScript is the better JSON parsing document, and
RFC 4627 is a now-obsolete Informational RFC.

- Rob

---

From: "Markus Lanthaler" <[email-masked]>
Date: Tue, 11 Jun 2013 10:09:31 +0200

> Lots of people do this, in my experience, including me pretty often.

I was hoping for something more concrete. Something like "Facebook's,
Google's and Bing's APIs are doing that. Here are the links."

> sub-values of a JSON value cannot be sent as application/json.

Of course not. How would you path a JSON object or array by just sending a
number? How should that number be interpreted?

> So for
> array instead of sending raw primitive values.

Are you really talking about PATCH? Looks more like a PUT to me.. PATCH
requires a different media type anyway.

> If you implement a
> added.

... or you just return text/plain. That would even enable browsers to render
the responses.

Cheers,
Markus

--
Markus Lanthaler
@markuslanthaler

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Jun 2013 11:25:35 +0200

On Jun 7, 2013, at 19:54, Tatu Saloranta <[email-masked]> wrote:

>> Apart from the (intellectually nicely challenging, but practically irrelevant) auto-detection thing,
> 
> Irrelevant based on what? I thought this is what any half-decent JSON parser did; unless platform does not expose byte sequence as input, case for some scripting languages.
> I have written multiple parsers (json, xml) that do just this, and know that others exist as well.

It is true that there are parsers that implement autodetection.
Lots of code is written that is then never exercised.

> I don't know what leads to assumption of irrelevancy here, and it should be fully spelled out.

This is irrelevant in practice as JSON is used with UTF-8 in practice.

The main job that the text in section 3 seems to do is to convince people not to send a BOM in front of their UTF-8 JSON.  Now that is a good thing...  Unfortunately, some JSON parsers have been "fixed" to accept BOMs.

Grüße, Carsten

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 11 Jun 2013 11:34:36 -0700

+1 for this: although I can see why additional media type is suggested, I
just do not see it as a better alternative from practical viewpoint, due to
these reasons.

-+ Tatu +-

On Mon, Jun 10, 2013 at 9:19 PM, Manger, James H <
[email-masked]> wrote:

> It would significantly complicate conversations about JSON. Mentions of
> JSON in APIs, protocols, specs etc would have to qualify whether or not
> they support both, or just one, of the media types. Two media types
> probably need to be advertised in HTTP Accept headers. There would be two
> choices for media type when sending a JSON object or array, which would
> produce different behaviour in some cases (eg "old" systems that don't know
> about the new type).
>
> A top-level JSON value would no longer necessarily be self-delimiting. I
> doubt this is that crucial. Many media types are not self-delimiting. Even
> JSON objects and arrays are not strictly self-delimiting as they allow
> leading and trailing whitespace. Appending whitespace to a JSON value would
> make as self-delimiting again – restoring this property where needed
> (allowing whitespace around non-object-non-array top-level values would be
> a sensible part of the change).
>
> --
> James Manger

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 11 Jun 2013 11:41:15 -0700

> Lots of code is written that is then never exercised.

If by lots of code you mean "less than 100 lines", yes.

> This is irrelevant in practice as JSON is used with UTF-8 in practice.

My main concern is with UTF-16. My understanding is that for "Big 5"
languages its use make sense, from efficiency perspective. I do not have
data on this; in XML space document test sets had non-trivial amount of
content in various encodings.

If UTF-16 is not widely used then I can see why this would be considered of
little significance.

> The main job that the text in section 3 seems to do is to convince people
> not to send a BOM in front of their UTF-8 JSON.  Now that is a good
> thing...  Unfortunately, some JSON parsers have been "fixed" to accept BOMs.
>
>
Yes. That takes about 50 more lines of code to add, to work around oddities
of generators by a certain big Redmond-based software company.

-+ Tatu +-

---

From: John Cowan <[email-masked]>
Date: Tue, 11 Jun 2013 15:09:33 -0400

Tatu Saloranta scripsit:

> languages its use make sense, from efficiency perspective. I do not
> have data on this; in XML space document test sets had non-trivial
> amount of content in various encodings.

See <http://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html>
which indicates that about 80% of Google-findable web content is in
Unicode, when you count in pure ASCII documents.  I asked Mark Davis,
and he said that UTF-16 was much less than 0.1%.  Of course, in absolute
terms that's a lot of documents.

-- 
Man has no body distinct from his soul,              John Cowan
for that called body is a portion of the soul        [email-masked]
discerned by the five senses,                        http://www.ccil.org/~cowan
the chief inlets of the soul in this age.  --William Blake

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Jun 2013 21:43:50 +0200

On Jun 11, 2013, at 06:27, R S <[email-masked]> wrote:

> ECMAScript is the better JSON parsing document

What makes you say that?

Grüße, Carsten

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 12 Jun 2013 10:25:35 +0900

On 2013/06/12 3:41, Tatu Saloranta wrote:
> On Tue, Jun 11, 2013 at 2:25 AM, Carsten Bormann<[email-masked]>  wrote:

> If UTF-16 is not widely used then I can see why this would be considered of

There are a lot of scripts (and therefore languages) where a character 
takes 3 bytes in UTF-8 but only 2 bytes in UTF-16. In particular, this 
includes all the languages of East/South East/South Asia, a huge area 
with a huge population. It's the reason why UTF-16 was made mandatory 
for XML.

However, as predicted by some, and widely confirmed in practice, most 
actual data (including XML and of course JSON) contains a significant 
amount of characters from the ASCII repertoire (syntax such as []{}"", 
and space for JSON, plus many if not most names and many values). These 
characters take only one byte in UTF-8, but two bytes in UTF-16. As a 
result, content is very often shorter in UTF-8, and when it happens to 
be longer, it's usually not much longer.

Overall, the advantage of occasionally shorter data is clearly 
outweighted by the simplicity of a single encoding (even if not on the 
receiver side, then on the generating side).

As a result, virtually the whole Web ecosystem is moving towards using 
UTF-8 only for public interchange, at a surprising speed. (Of course, 
many other encodings will still remain for years, but in lower and lower 
numbers).

Regards,   Martin.