Subject: [Json] JSON for Internet messages

From: Tim Bray <[email-masked]>
Date: Wed, 3 Jul 2013 09:34:00 -0700

I think it will be helpful to put a stick in the ground:  Most of my
professional activity centers around designing and using message-passing
application-level APIs.  In basically all of them, the payload is JSON.

So I care a lot about JSON, but I don’t care in the slightest about usages
where the JSON isn’t being used for application-level message protocol
payloads (are there such usages?  I'm curious).  Also, I’ve never
encountered a scenario where the messages were of sufficient size that
anyone gave a rat’s ass about streaming.

>From that point of view, the following seem obvious:

- All JSON messages MUST be encoded in valid UTF-8.
- All numbers MUST be of precision < 2**53 (use strings for your big crypto
numbers)
- All JSON objects MUST have unique keys
- All JSON messages MUST be either arrays or objects
- Software receiving something violating any of these MUSTs has encountered
conclusive evidence of serious upstream breakage and MUST NOT trust the
contents nor act upon them.

That’s all. It leaves lots of things to argue about (BOMs? Non-characters?
Control-characters? Maybe just objects at top-level? Should this flavor of
JSON self-identify? If so, how? etc etc)

I want there to be an RFC that says these things that protocol designers
can point to and just stop thinking about markup syntax.

I don’t think our mandate lets us write that RFC.  I think we should write
a -bis doc that documents the actual realities in the field (results of
using bad Unicode and dupe keys are unpredictable), recharter, and then
write that RFC.  -T

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 11:47:25 -0500

On Wed, Jul 3, 2013 at 11:34 AM, Tim Bray <[email-masked]> wrote:
> I don’t think our mandate lets us write that RFC.  I think we should write a
> -bis doc that documents the actual realities in the field (results of using
> bad Unicode and dupe keys are unpredictable), recharter, and then write that
> RFC.  -T

We could argue about your list of things to mandate.  Are you counting
escaped code units in their unescaped forms towards well-formed UTF-8?
 Would we need a new media type, or could we say it doesn't matter,
this is the Internet JSON?  That aside, +1, even to the non-duplicate
names requirement.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 3 Jul 2013 09:56:53 -0700

On Wed, Jul 3, 2013 at 9:47 AM, Nico Williams <[email-masked]> wrote:

> this is the Internet JSON?

Well-formed UTF-8 isn’t allowed to encode surrogates. Which meets the
minimal need; Frankly, I think string-handling code can, in practice, suck
it up and deal with all sorts of Unicode inappropriateness... just not
surrogates.  Also, I explicitly said the question of self-identification
(e.g. with a media type) is open. -T

> That aside, +1, even to the non-duplicate
> names requirement.
>
> Nico
> --
>

---

From: Eliot Lear <[email-masked]>
Date: Wed, 03 Jul 2013 19:03:56 +0200

On 7/3/13 6:34 PM, Tim Bray wrote:
> - All numbers MUST be of precision < 2**53 (use strings for your big
> crypto numbers)
> - Software receiving something violating any of these MUSTs has
> encountered conclusive evidence of serious upstream breakage and MUST
> NOT trust the contents nor act upon them.
>
> That’s all. It leaves lots of things to argue about (BOMs?
> Non-characters? Control-characters? Maybe just objects at top-level?
> Should this flavor of JSON self-identify? If so, how? etc etc)
>
> I want there to be an RFC that says these things that protocol
> designers can point to and just stop thinking about markup syntax.
>
> I don’t think our mandate lets us write that RFC.

I think it does, if you believe these are the minimal changes necessary
for interoperability.  The standard for standards goes UP as you go
through maturity.  Deal with this now or someone else will end up having to.

Eliot

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 13:15:47 -0400

Tim Bray scripsit:

> payloads (are there such usages?  I'm curious).

I suppose it depends on what you mean by "application-level".  There are
several NoSQL databases that speak JSON, and I wouldn't call them
applications per se.

> Also, I’ve never encountered a scenario where the messages were of
> sufficient size that anyone gave a rat’s ass about streaming.

Well, at least eight programmers have taken the trouble to write
such parsers, and I don't think we can assume that they are
all works of supererogation.  Rob Gonzalez says that he wrote
his PHP streaming parser because he needed to be able to react
to JSON input on the fly.  I quote the first few paragraphs of
<http://www.salsify.com/blog/json-streaming-parser-for-php>:

    Recently I looked around for a JSON streaming parser written in
    PHP and couldn’t find one. So I wrote my own and am making it
    available to anyone who wants to use it.

    For those unfamiliar with streaming parsers I’ll give a brief
    intro. In short, they’re useful if you have a very large
    document and don’t want to have to load the whole thing in
    memory at once.

    Most (non-streaming) parsers—including, for example, the stock
    JSON parsers that come with PHP—will load an entire document in
    memory and then give you access to the whole thing at once. The
    advantage to this whole-document-at-once approach is that you
    get random access to every object in the document. Also the
    programmatic interface they provide can be really convenient. In
    PHP’s standard library you can get a handle on a native PHP
    array for the entire JSON document, which is really easy to
    work with.

    The downside is, as mentioned, the whole document must be
    loaded into memory. For most web servers or web services this
    is a serious constraint. Furthermore, this means that your
    application can’t start actually doing anything useful with
    the data until the whole thing is loaded in memory. A streaming
    parser, in contrast, gives your application access to data almost
    as soon as the data is read by PHP so can be much faster.

    As I was writing a plugin for Magento, I needed a JSON streaming
    parser written in PHP. I looked at StackOverflow, PHP.net, and
    finally Google. It’s pretty crazy to me that no one seems to
    have had this need before given the popularity of both JSON and
    PHP, but there you go!

> - All numbers MUST be of precision < 2**53 (use strings for your big crypto

Better: All numbers MUST be representable as IEEE 754:2008 binary
64-bit floats.

> contents nor act upon them.

The trouble with all this is that They Break Data.  All of a sudden, what
was valid JSON before is not valid JSON now.  That's serious, especially
as people *do* keep around their JSON rather than using it only transiently
See above remarks about databases: it doesn't matter if the JSON is stores
as JSON or regenerated on the fly, because if I put in JSON today I expet
to get back JSON tomorrow, and not something that no longer counts as JSON.

-- 
All Gaul is divided into three parts: the part          John Cowan
that cooks with lard and goose fat, the part            http://ccil.org/~cowan
that cooks with olive oil, and the part that            [email-masked]
cooks with butter. --David Chessler

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 12:28:42 -0500

> anyone gave a rat’s ass about streaming.

Er, I work with an application where I need to atomically read *all*
the data (it's ok if it's slow) once, then poll for / subscribe to an
incremental update feed.  The first requirement has devolved into
doing a single GET with chunked data results -- pagination failed
because we couldn't get the application in question to provide
atomicity with pagination.

That data comes to me as JSON.  It can be huge.  It happens to be an
array of relatively small objects, so it's possible to implement the
generation (and parsing) in a streaming way for the top-level value
and still prevent / deal with duplicate names one level down.

But you can see how one can end up needing streaming generators and parsers.

I could write a parser that streams the array and doesn't stream the
objects.  But I'm going to use an off-the-shelf parser as much as
possible.  The pressure to use a streaming parser (and generator) will
increase as the dataset increases in size.  My fear is that the cost
of dealing with this dataset will increase superlinearly with dataset
size if I use a non-streaming parser; I fear that eventually I won't
be able to use a non-streaming parser at all.

For all other purposes the application in question uses small enough
JSON documents that I wouldn't need streaming anything.

Now, in this application it just so happens that there are other ways
to guarantee non-duplicate names that are already in place, so the use
of a streaming generator/parser that does nothing about duplicate
names would be just fine.

I don't mind saying "MUST NOT" have dups.  I just want the text
written in such a way that this application can use JSON after all.
For me that would mean that it's OK to have streaming generators and
parsers, as long as the application makes sure there's no dups in that
case.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 3 Jul 2013 12:31:10 -0500

> to get back JSON tomorrow, and not something that no longer counts as JSON.

Tim is proposing a different JSON, "Internet JSON", to be used where
appropriate.  Consider this like the BCP, only as a Standards-Track
RFC, and NOT updating/obsoleting RFC4627, but competing with it.

I could live with that.

---

From: Tim Bray <[email-masked]>
Date: Wed, 3 Jul 2013 10:31:22 -0700

> such parsers

Fair enough. If they can’t write them in such a way as to avoid duplicates,
they are NOT USABLE for application-level message passing, because the only
use for JSON objects is sending a hash-table (or equivalent) from my
computer to a hash-table (or equivalent) on yours, and if there are dupes,
that's either a serious bug or a malicious attack on, for example, a JOSE
header where they’re trying to fake out the algorithm selection. So parsers
that potentially emit dupes can’t comply with the RFC that I want to
exist.  Acceptable cost.

> crypto

I think my expression of the constraint is more readable, but I agree that
yours is isomorphic.

> encountered
> The trouble with all this is that They Break Data.

I care -Inf about this.  In my application domain, we cook these up in
memory, we send them, we unpack them at the other end.   I agree, that if
if we impose a new set of rules in a new RFC, previously-existing JSON
might not pass muster as “Internet JSON” or whatever we choose to call it.
It’s still JSON Classic.  Fair trade-off.  -T

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 15:12:12 -0400

Tim Bray scripsit:

> > > - All numbers MUST be of precision < 2**53 
> I think my expression of the constraint is more readable, but I agree
> that yours is isomorphic.

Well, no.  1E400 requires only one bit of precision, but it's not
representable as a double, because doubles only go up to 1E308 or so.

> I care -Inf about this.

That reminds me how *annoying* it is that there's no way to represent
Inf, -Inf, or NaN.  Too late now, I suppose.

> In my application domain, we cook these up in memory, we send them,
> we unpack them at the other end.   I agree, that if if we impose a
> new set of rules in a new RFC, previously-existing JSON might not pass
> muster as “Internet JSON” or whatever we choose to call it.

How about "Ephemeral Mini-JSON"?  Can't be too big, not expected to last.

-- 
With techies, I've generally found              John Cowan
If your arguments lose the first round          http://www.ccil.org/~cowan
    Make it rhyme, make it scan                 [email-masked]
    Then you generally can
Make the same stupid point seem profound!           --Jonathan Robie

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 3 Jul 2013 12:21:41 -0700

> anyone gave a rat’s ass about streaming.
Large payload are commonly used for things like map/reduce jobs and search
indexing. And databases of sorts (NoSQL or traditional) are similarly
exposing large datasets where one certainly does not want to or even be
able to handle all data in memory.

There are ways to avoid problems with large Objects for many cases within
this domain, for example, by sending sequences of root-level objects.
Parsers then need to be able to handle streams of multiple Objects.
Streaming is equally important (or perhaps more so) when generating
results; for latency (as well as memory usage) reasons large result sets
are best sent in streaming manner.

Same is true for XML as well, so I don't think this is JSON-specific
question.

I might as well claim that I have rarely worked on JSON use case where
Javascript matters at all. This is a true statement, but I would not draw
conclusions from this to allege that Javascript would be largely irrelevant
for JSON data exchange.

I think part of the challenge is that for many practicioners JSON "Just
Works" well enough that they could not care less about further
standardization: and in fact the whole notion that duplicate keys are a
major problem seems like a red herring.
It has never been a problem for me, nor has there been user requests beyond
schema (and schema validator) implementors who care more. But it is
possible that perhaps this has been a significant problem for others.

-+ Tatu +-

---

From: Jacob Davies <[email-masked]>
Date: Wed, 3 Jul 2013 12:29:55 -0700

> payloads (are there such usages?  I'm curious).

There are a number of practical use cases for JSON that don't fall
into the neat "externally delimited stream of bytes described by an
application/json media-type" category. That is a separate question to
whether the RFC needs to do much to accomodate them.

1. Files named ".json" and containing a single JSON object or array.
2. Text documents containing embedded JSON - informally, as in
documentation or books, or formally, as in Javascript in HTML or .js
files.
3. Database records where text or binary columns store embedded JSON
representing complex nested objects.

Case 1 is handled the same as a message described as application/json,
no big deal. Cases 2 & 3 may require accommodating non-Unicode
encodings, and may have some special security concerns (e.g.
additional escaping for HTML or Javascript embedding). It would be
nice if the spec separated format from encoding, but it's unlikely to
be the source of any significant problems since in most cases the
handling of encoding will have happened by the time you parse the
text, and the security concerns can be addressed in best-practices.

> Also, I’ve never encountered a scenario where the messages were of sufficient size that

Me either, and even if there were such cases I find the concerns about
maintaining state unconvincing - maintaining a seen-keys dictionary
for each level of nesting you're in is not all that much of an
imposition or incompatible with a streaming API.

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 3 Jul 2013 13:07:02 -0700

> imposition or incompatible with a streaming API.
Believe it or not, but this actually would become major overhead for
streaming level; as well as for data-binding tools of statically typed
languages (Java, C#).
I am sure that there are clever ways to limit the overhead, but a naive
implementation in Java would give us 30-50% processing time boost, which
generally adds no value (assuming "last value wins" is adhered to and/or if
generator can avoid producing dups).

Common use case for me is as follows:

1. Input comes as a typed object
2. Serializer accesses properties and guarantees uniqueness of keys, calls
generator
3. Generator simply outputs tokens as requested (maintaining basic stack to
ensure syntactic output)
4. Receiving end uses parser to expose JSON as tokens, without dup checks
5. Deserializer maps properties to local objects created: if duplicates
were encountered, last value would stick due overwrites

in such a case, duplicates are not problematic; and this without explicit
checks.
Having to check duplicates on both sending and receiving end (at
parser/generator level) would most likely add +50% overhead for processing
time, because overhead on generator is relatively higher (JSON can be
written about 2x as fast as decoded).
Dup detection could work more efficiently on deserializer level, but would
require separate null markers to distinguish between JSON nulls and lack of
property.

Alternate case of using a Tree representation ("JsonNode" etc) would have
similar solution to duplicate problem: tree serializer would not produce
dups (since tree can not contain them); and tree deserializer would either
use last, or (if preferred) signal an error.

-+ Tatu +-

---

From: John Cowan <[email-masked]>
Date: Wed, 3 Jul 2013 16:15:22 -0400

> possible that perhaps this has been a significant problem for others.

The trouble is that with sufficiently bad implementations, or chains
of implementations, such permissiveness can lead to a security breach,
and security being a Yang Worship Word these days, everyone cowers in
fear which such things are mentioned.  My most recent use case for
JSON (to represent the MicroXML data model) doesn't even require parsing it,
but I never suppose that for that reason JSON might as well be limited
to nothing but arrays in which the first element is always a string and
the second element is always an object with string values.

-- 
John Cowan  <[email-masked]>  http://ccil.org/~cowan
Micropayment advocates mistakenly believe that efficient allocation of
resources is the purpose of markets.  Efficiency is a byproduct of market
systems, not their goal.  The reasons markets work are not because users
have embraced efficiency but because markets are the best place to allow
users to maximize their preferences, and very often their preferences are
not for conservation of cheap resources.  --Clay Shirky

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 3 Jul 2013 13:21:50 -0700

> the second element is always an object with string values.

Ok. This makes more sense to me. I hope that security aspect can be
emphasized in context of duplicates, if it is the leading reason (which is
consistent with earlier discussions I think).
I also do not think there is anything wrong in having multiple documents
that add layers of requirements: strict handling of duplicates seems best
handled at higher level, and not at basic format level.

-+ Tatu +-