Subject: Re: [Json] Some thoughts on the "text model" discussion

From: Paul Hoffman <[email-masked]>
Date: Mon, 5 Aug 2013 13:02:55 -0700

A general nudge to the WG to comment on Pete's message from last week.

---

From: Tim Bray <[email-masked]>
Date: Mon, 5 Aug 2013 13:33:59 -0700

On Wed, Jul 31, 2013 at 2:11 AM, Pete Resnick <[email-masked]>wrote:

- The discussion at the beginning regarding "a JSON object" vs. "a JSON
> text" was very revealing to me because it leads me to believe that nowhere
> do we have a general concept of a "JSON entity" aside from it encoding or
> representation in memory. I think that has made some of the discussion more
> difficult, but my sense is that we now share a more general view of what
> such a thing might be. This leads to another point:
>

Since I look at the world through HTTP-colored lenses, I think of JSON in
the context of an HTTP message body.  So at the top level, a JSON message
is the bytes between the CR-NL/CR-NL and the end of message; or, what comes
out of the generic JSON parser when I feed it bytes purported to contain
JSON.

> - The use of the terms "JSON parser" and "JSON generator" is often
> conflated with things that I would refer to as a "JSON lexer" on the
> parsing end of things and an "JSON encoder" or maybe a "JSON token
> producer" on the generator end of things. That is, when I think of a "JSON
> parser", I think of something that understands the semantics of the JSON
> entity it is creating out of the JSON text it is parsing, and conversely a
> "JSON generator" knows about the semantics of the JSON entity it is turning
> into a JSON text from some in-memory representation. Some of the difficulty
> in the conversation is that we know of pieces of code that don't know about
> the semantics and therefore don't "know" whether they are dealing with
> Unicode characters or just a series of Unicode scalar values. (This caused
> similar confusions in the duplicate names discussion.) Requirements on
> things that have to know the semantics are different than the requirements
> on things that are simply lexers or encoders. How we explain that (and what
> language we use for it) in the documents is independent of the point that
> they are different requirements.
>

tl;dr: I agree about the source of difficulty: Generic JSON library-ware is
almost always used and almost never aware of application-level semantics.

Long version: From where I sit, the situation you’re describing, with
application-sensitive encoders/decoders, is extremely unusual.  The
application programmer takes some data structure that it understands,
passes it to a 100%-generic JSON encoder, and gets some bytes back, which
it usually then interchanges over the network.  At the other end, the
programmer gets some bytes, passes them to a 100%-generic JSON decoder, and
maps what it gets back onto its semantic view of the world.  The details of
what sort of interfaces the encoders and decoders use, and how you map from
the JSON parse result to your own semantics, vary strongly from language to
language - I have direct experience with Java, Ruby, and Go.  In
strongly-typed languages like Java and Go this means application programers
have to do quite a bit of extra work to pull their view of the data out of
a org.json.JSONObject (Java) or interface{} (Go).  In dynamic languages
such as Ruby the mapping between JSON and native data structures is pretty
transparent and effortless.

Hence my motivation: Since application programmers universally depend on
application-oblivious libraries, I want to stamp out anything that makes
these libraries function less well.  One glaring instance of this is broken
Unicode, which has the potential to cause breakage in libraries that think
they’re dealing with strings of Unicode characters.  For those programmers
who are encoding/decoding their JSON in an application-sensitive way, those
issues would be much less severe.

> For each of these three items, I think if we end up with some explanatory
> text in the documents that helps make the appropriate distinctions (and
> note that I am being purposely agnostic on what the WG should say about
> each of them), I think that would help the clarity and implementability of
> the spec.
>
> I'll leave it to the chairs to decide where to take the discussion from
> here.
>
> pr
>
> --
> Pete Resnick<http://www.qualcomm.**com/~presnick/<http://www.qualcomm.com/~presnick/>
> >
> Qualcomm Technologies, Inc. - +1 (858)651-4478
>
> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: Nico Williams <[email-masked]>
Date: Mon, 5 Aug 2013 16:05:49 -0500

On Mon, Aug 5, 2013 at 3:33 PM, Tim Bray <[email-masked]> wrote:
> the context of an HTTP message body.  So at the top level, a JSON message is
> the bytes between the CR-NL/CR-NL and the end of message; or, what comes out
> of the generic JSON parser when I feed it bytes purported to contain JSON.

If you're going to look at it that way then you kinda have to mention
content-encoding...

That's a total nitpick on my part, but it may be the sort of thing Pete's after.

>> ...
> application programmer takes some data structure that it understands, passes
> it to a 100%-generic JSON encoder, and gets some bytes back, which it
> usually then interchanges over the network.  At the other end, the
> have to do quite a bit of extra work to pull their view of the data out of a
> org.json.JSONObject (Java) or interface{} (Go).  In dynamic languages such
> as Ruby the mapping between JSON and native data structures is pretty

There's a bit more to it.  Often the data has a schema that it needs
to conform to -- whether the schema is baked into the app or actually
expressed as a schema is irrelevant.  Then again, often the data is
free-form-ish.

If I have a... "JSON text" and I parse it I might actually want to
inspect it manually in a python REPL, or using jq.  Or by eye (and
people do write JSON texts by hand too).  In practice though, any JSON
API I care about has a schema, and the data is not really free-form.
But then, an app could easily allow arbitrary user inputs to be
represented as schema-free JSON, and then I'd have to care.

> issues would be much less severe.

But how often does the naked surrogate thing actually cause problems?

Are we merely trying to up the spec or are we really solving a problem
that needs solving?  Another possibility: we're trying to prevent a
problem from coming up.

Personally I think we should do something about naked surrogates and
such: provide recommendations, and possibly also requirements for
I-JSON.

Aside: If we pursue I-JSON, how do you envision implementations
actually handling it?  As more parser/encoder options?  As distinct
interfaces/classes/whatever?

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 8 Aug 2013 11:27:36 -0700

Another general nudge to the WG to comment on Pete's message from last week. The topic is pretty fundamental to the -bis document.

---

From: Tim Bray <[email-masked]>
Date: Thu, 8 Aug 2013 11:45:37 -0700

OK, since general discussion has not broken out, let’s be very specific in
answer to Pete’s question:

> each of them), I think that would help the clarity and implementability of

I propose this as the explanatory text for the -bis:

<<<<<<<<<<<<<<<<
A JSON text which is composed entirely of Unicode characters (however
encoded) will be interoperable in the sense that all Unicode-capable
software which parses it will agree on the contents of names and values of
object members.  However, the ABNF in this specification allows member
names and string values to contain bit sequences which cannot encode
Unicode characters, for example "\udddd" (an unpaired UTF-16 surrogate).
Instances of this have been observed, for example when a library truncates
a UTF-16 string to a maximum allowable length without checking whether the
truncation split a surrogate pair.  The behavior of software which receives
JSON texts containing such values is unpredictable; for example, two
different implementations might return different values for the length of a
string value, or even suffer a fatal runtime exception.
<<<<<<<<<<<<<<<<<

That’s all. Point out the problem as exactly and clearly as possible, then
shut up.

---

From: Jorge Chamorro <[email-masked]>
Date: Fri, 9 Aug 2013 19:26:38 +0200

On 08/08/2013, at 20:45, Tim Bray wrote:
> A JSON text which is composed entirely of Unicode characters (however encoded) will be interoperable in the sense that all Unicode-capable software which parses it will agree on the contents of names and values of object members.  However, the ABNF in this specification allows member names and string values to contain bit sequences which cannot encode Unicode characters, for example "\udddd" (an unpaired UTF-16 surrogate).  Instances of this have been observed, for example when a library truncates a UTF-16 string to a maximum allowable length without checking whether the truncation split a surrogate pair.  The behavior of software which receives JSON texts containing such values is unpredictable; for example, two different implementations might return different values for the length of a string value, or even suffer a fatal runtime exception.
> <<<<<<<<<<<<<<<<<
> 
> That’s all. Point out the problem as exactly and clearly as possible, then shut up.
>  

+1

"extraneous" illegal bit sequences might also happen to be converted to 0xfffd (the all too often seen �)
-- 
( Jorge )();

---

From: Larry Masinter <[email-masked]>
Date: Sun, 11 Aug 2013 05:34:37 -0700

In general, a “media type” identifies a means of serializing an abstract model of something; “image/jpeg” is a byte sequence means of expressing an image, “text/html” a means of expressing a hypertext document. XML is a serialization “application/json” is a means of expressing an object in a programming language.

As with XML and other , the serialization of the JSON model is conceptually handled on multiple levels:

1)      a data structure in a specific programming language (JavaScript, ECMAScript, Python, Lisp)
2)      an abstract object data structure, common between all JSON implementations
3)      as a sequence of abstract Unicode characters (identified by their code points)
4)      as a sequence of bytes, where the byte sequence is an encoding of the Unicode characters, using one of the Unicode transfer functions, UTF8, UTF16.

The mapping between (3) to (4) is defined by Unicode.
The mapping between (1) and (2) is defined by the binding of JSON to a particular programming language; ECMA XXXX defines the mapping for ECMAScript.
The syntax allowed for (3) and the mapping between (2) and (3) are defined by ECMA YYYY.

The abstract model for (2) is limited so as to support many programming languages and a simple mapping – there are many (level 1) programming language data structures and types which cannot be represented in JSON. For example, JSON (2) does not distinguish between arrays and lists. JSON (2) does not allow repeated attributes.

However, a JSON parser (mapping from 3 to 2) may encounter repeated attributes, and MUST ignore all but the last it encounters.  A JSON emitter MAY produce repeated attributes.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tim Bray
Sent: Thursday, August 08, 2013 8:46 PM
To: Pete Resnick
Cc: [email-masked]
Subject: Re: [Json] Some thoughts on the "text model" discussion

OK, since general discussion has not broken out, let’s be very specific in answer to Pete’s question:

On Wed, Jul 31, 2013 at 2:11 AM, Pete Resnick <[email-masked]<mailto:[email-masked]>> wrote:
For each of these three items, I think if we end up with some explanatory text in the documents that helps make the appropriate distinctions (and note that I am being purposely agnostic on what the WG should say about each of them), I think that would help the clarity and implementability of the spec.

<<<<<<<<<<<<<<<<

I'll leave it to the chairs to decide where to take the discussion from here.

pr

--
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478<tel:%2B1%20%28858%29651-4478>

_______________________________________________
json mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/json

---

From: "John Levine" <[email-masked]>
Date: 11 Aug 2013 15:07:59 -0000

>As with XML and other , the serialization of the JSON model is conceptually handled on
>multiple levels::

 ... looks great up to

>However, a JSON parser (mapping from 3 to 2) may encounter repeated attributes, and MUST
>ignore all but the last it encounters.

Uh, no.  If there were any hope of agreeing to that, we wouldn't be
having this argument.

---

From: Larry Masinter <[email-masked]>
Date: Sun, 11 Aug 2013 19:24:04 -0700

My hope was that by being clear about the representation layers
(1 programming-language / 2 abstract JSON / 3 sequence-of-characters / 4 sequence-of-bytes)

the design choices for "repeated attributes" might find some resolution.

Some 1 programming languages which have JSON support might support duplicate
elements (e.g., using an  'a-list' in Lisp) while others will not (an object in JavaScript).

Requiring the 1 programming language to support duplicate attribute names
would be an unreasonable burden.  For this reason, the 2 abstract JSON model
should (IMO) avoid supporting duplicate names.

However, it's pretty clear that at the 4 sequence-of-bytes and 3 sequence-of-character
level that duplicate attribute names are possible, and the behavior of 3->2  parsers
when encountering such might be specified, although the choice between
"signal error" and "take last" are both consistent with "might occur in 3 but
not supported in 2".

With non-Unicode code points in strings, there is a similar layer shift, although
there is a double-encoding  in string values -- if 4 -> 3 is actually UTF8
or UTF16 only, then  "\u005C" is still a sequence of 6 Unicode characters, 
and 3->2 needs to also handle or error on non-Unicode code points 
which (IMO) shouldn't be allowed at 2-abstract-JSON.

The goal is to acknowledge liberal acceptance of 3-sequence-of-character
diversions even if 2-abstract-JSON is limited as to not impose undue burdens
at 1-programming-language-model.

Larry

> -----Original Message-----
> From: John Levine [mailto:[email-masked]]
> Sent: Sunday, August 11, 2013 5:08 PM
> To: [email-masked]
> Cc: Larry Masinter

---

From: Tim Bray <[email-masked]>
Date: Sun, 11 Aug 2013 19:57:06 -0700

>From where I sit there are only two levels that matter:

1. Application-level: A developer needs to ship some data over the network.
2. Everything else: turn the data into a byte sequence, arrange to get them
received as transmitted, turn them back from a byte sequence into something
useful at the application layer.

>From the level-1 point of view (which is shared by everyone who is using
JSON for their HTTP-based APIs), duplicate keys are always bugs. Anything
that’s not a Unicode character is always a bug. But we can’t fix that in
the -bis, so we should just describe what is interoperable and what isn’t.

 -T

On Sun, Aug 11, 2013 at 7:24 PM, Larry Masinter <[email-masked]> wrote:

> > >As with XML and other , the serialization of the JSON model is
> conceptually
> > handled on
> > >multiple levels::
> >
> >  ... looks great up to
> >

---

From: "John R Levine" <[email-masked]>
Date: 11 Aug 2013 23:03:19 -0400

> at 1-programming-language-model.

If you'll review the previous thousand messages or so on this topic, I 
don't think you'll find any that argue that duplicate keys are a good 
idea, but you'll find plenty that say for whatever reason there is real 
json that has dup keys.  It's a decade too late to wave our hands and make 
it go away.

Javascript, which is still probably the dominant json environment, has a 
concrete way to deal with dup keys, so the undue burden argument isn't 
very persusasive.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Tim Bray <[email-masked]>
Date: Sun, 11 Aug 2013 20:56:46 -0700

On Sun, Aug 11, 2013 at 8:03 PM, John R Levine <[email-masked]> wrote:

> concrete way to deal with dup keys, so the undue burden argument isn't very
> persusasive.
>

I’ve used a whole lot of JSON over the years in Ruby, Java, and Go, but
never JavaScript. Your claim of JavaScript’s “dominance” is entirely
without supporting evidence and can safely be ignored.

Having said that; yeah, I don’t think there’s consensus either for saying
dupe keys etc are a good thing, or for trying to abolish them in the -bis.
That leaves documenting what’s interoperable and what’s not.  -T

---

From: R S <[email-masked]>
Date: Mon, 12 Aug 2013 19:14:51 -0700

> without supporting evidence and can safely be ignored.

Well, "dominance" is probably the wrong word. I'd say "definitiveness"
would be more appropriate--it is JavaScript Object Notation. Why would you
want a JSON library that blew up on JSON successfully handled by JavaScript
implementations, which all interoperate basically perfectly?

Even your proposed text doesn't have much proof yet: "or even suffer a
fatal runtime exception.". The only example we saw on this list was a
Python 3 exception. But its JSON implementation handled unpaired surrogates
just fine, the exception was in the print function.

Further, JSON library authors generally don't get involved with unicode
encoding issues. That's handled by libraries like Mozilla's
nsIUnicodeDecoder <
http://dxr.mozilla.org/mozilla-central/source/intl/uconv/src> and V8's
Unibrow <http://v8reference.com/namespaceunibrow.html>.

- Rob