Subject: [Json] On flat vs nested JSON encoding style

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 4 Feb 2016 09:59:40 -0500

I think it would be helpful to have a style guide to encourage styles
of JSON use that impose the fewest constraints on
serialization/deserialization design.

For example, lets say that we have a protocol that has two types of
request, A and B. There are two common ways that this can be
represented in JSON:

Nested style:

{ "A" :
   { "field1" : "value1", "field2" : "value2", "field3" : "value3" } }

Flat style:

{ "Type" :"A" ,  "field1" : "value1", "field2" : "value2", "field3" : "value3" }

They both look similar, the only difference is that there is an extra
layer of nesting in the first case which means that the label "A" will
always precede the data it describes. This isn't the case for the
second which can also be written :

{ "field1" : "value1", "field2" : "value2",  "Type" : "A" , "field3" :
"value3" }

While the issue comes up with requests, it isn't limited to requests,
it can happen anywhere that you have a need to tag an object to
describe the type.

If you are using a scripting language, you parse the data to a DOM
tree and which is the in-memory representation of the data.

But those of us who use strongly typed languages like C# and Java
don't work of DOM trees. We map the data to objects in our
implementation language. and we type check and validate all our inputs
before we begin processing. Because that is best practice for
minimizing attack surface from an invalid input.

Nested style allows us to build fast, single pass deserialization
tools that require no additional buffering. A constrained device can
build the parse tree on the fly without buffering a whole message
before it starts processing.

Flat style requires us to buffer the data and do a two pass approach.
It is very much less efficient as a result.

Comments?

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 4 Feb 2016 16:24:05 +0100

> serialization/deserialization design.

By pure coincidence I wrote this (somewhat related) paper today:
http://webpki.org/papers/AppNote-XML2JSON-Migration.pdf

Using your nested design, the outermost property could be something like
"https://json.standards.org/payments/PaymentRequest"
Not ultra-pretty but certainly workable.
Using JCS an embedded signature would preferably be applied to the outermost object.

Anyway, isn't CBOR what people are targeting for constrained devices?

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 04 Feb 2016 16:39:13 +0100

> Anyway, isn't CBOR what people are targeting for constrained devices?

Yes, but the desire to be able to do sequential (as opposed to
DOM-based) processing goes beyond constrained devices.

But talking about those:

The original design of SenML <draft-jennings-core-senml-04.txt> had a
top level map (what is called "object" in JavaScript, but then arrays
are also objects in JavaScript) of the form:

{ "metadata1": "value",
  "metadata2": 4711,
  "contents": [
     ... potentially many measurements
     ... where the interpretation of those
     ... depends on the metadata.
  ]
}

This looks deceptively sequential on a napkin, and it creates no
problems for a DOM-based receiver implementation (until the measurements
don't fit into memory any more).  For an implementation that tries to do
sequential processing, this structure is a big problem:  The metadata
may occur before or after the measurement array, depending on the whims
of the encoder, so you may not have what you need to process the
measurements sequentially at the place they occur.

Requiring the encoder to send data in a particular sequence is
theoretically possible, but in practice just means you lose that part of
the ecosystem that happens not to do that, and often you don't even get
a guarantee (order may be dictated by hash values that may be randomized
in a good implementation).

So we are moving SenML to something like:

[ { ... metadata blob ... },
  ... measurements ...
]

(the details are still being discussed).

Of course, in CBOR we could do:

{ { ... metadata blob ... }:
  [ measurements ]
}

i.e., use the metadata as the key for the map.  Which is essentially
what PHB is proposing, but that works in JSON only if the metadata
happens to be a single string (which is the case in the example given by
PHB).

Back to the discussion about representation style:

Preferring

{ "foo":
  { ... }
}

over

{ "type": "foo",
  ...
}

may or may not be compatible with the "struct"-style (see section 2 of
<draft-greevenbosch-appsawg-cbor-cddl-07.txt> for a definition of
various styles of using JSON-like data models), depending on whether
"foo" happens to be a constant in your implementation or you expect to
encounter new values as you go along.

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 4 Feb 2016 16:52:24 +0100

On 2016-02-04 16:39, Carsten Bormann wrote:

> the ecosystem that happens not to do that

This ecosystem is rapidly diminishing since ECMAScript nowadays mandates
strict insertion/declaration order for serialization.

In addition, if you deal with signed data like specified by JWS, don't you
anyway lose the ability to do smart sequential processing?

That is, there are undoubtedly a bunch of applications where a nested scheme
like proposed by PHB is the wat to go, but claiming universality would IMO
be a bit of a stretch.

Anders

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 4 Feb 2016 10:58:11 -0500

> DOM-based) processing goes beyond constrained devices.

Exactly. Every device is constrained when you send it a big enough problem.

I did think of mentioning CBOR but for another reason. A lot of times
you are going to want to apply a protocol that has already been
designed for JSON to a constrained device. This is one of the reasons
I argued that there should have been a WG to design an efficient
binary encoding. My objective was to be able to indicate the 'binary'
version by specifying application/json-b without having to make any
other change.

I think it likely that people will want to do the same with CBOR. But
that is a lot harder if doing so requires the structure of the parse
tree to be 'messed about'.

I would much rather that we have a style guide that guides people
towards an approach that is flexible.

Carsten's example does demonstrate a more general version of the same
problem. But I don't think it is quite as critical. Because what I am
talking about here is mapping JSON data to objects in strongly typed
languages. And all the strongly typed declarative languages I am
familiar with (and I know a very large number) have a type system that
can be reduced to text label.

[Yes, I did functional programing, we did Orwell. But I have yet to
unpack a widely used open source tool to find that it was written in
F#, Oberon, or any other from that family]

---

From: John Cowan <[email-masked]>
Date: Thu, 4 Feb 2016 19:17:17 -0500

> strict insertion/declaration order for serialization.

That affects only ECMAscript implementations, though, so there are no such
guarantees on the server side, or server-to-server, unless Node.js or
something similar is being used.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
If you have ever wondered if you are in hell, it has been said, then
you are on a well-traveled road of spiritual inquiry.  If you are
absolutely sure you are in hell, however, then you must be on the Cross
Bronx Expressway.  --Alan Feuer, New York Times, 2002-09-20

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 4 Feb 2016 20:09:55 -0500

> something similar is being used.

+1

We are doing RFC7159 JSON. Order is not guaranteed. A sender can emit
the request in any order they like. Code that depends on the order of
elements within an object is broken.

---

From: Anders Rundgren <[email-masked]>
Date: Fri, 5 Feb 2016 07:23:10 +0100

> elements within an object is broken.

Well, the W3C Web Payment IG/WG folks who claim they are working with
a new (and presumable secure) system for the Web Payments have not yet
begun dealing with how they are going to sign their JSON messages.

This will be a litmus test, not for RFC7159, but for RFC7515:
https://github.com/golang/go/issues/14135#issuecomment-177265555

In the meantime, I'm helping JSON tool-vendors providing support for
"Predictable Serialization" a la ES6/V8.  With the sole exception of
"Number", this is not rocket science and that particular rocket has already
been built by Google and the blueprints are readily available as well.

Maybe we can come back to this issue in 18 months and see where it went?

Anders

---

From: Tim Bray <[email-masked]>
Date: Thu, 4 Feb 2016 23:35:12 -0800

Well, over at AWS, I’ve just helped launch an experiment in doing things at
a large scale exactly the way PHB recommends against :)  Check out our
“CloudWatch Events” JSON at
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CloudWatchEventsandEventPatterns.html

We have millions & millions & millions of these flowing through the pipes
already, with many more to come.

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 05 Feb 2016 08:41:20 +0100

Anders Rundgren wrote:
> 18 months

JSON has been around for more than a decade.  It would take another
decade for a change that goes so deep into the fundamentals to become
widespread (and sufficiently debugged* on enough platforms to become
reliable).

Wake me up when, say, ArduinoJson supports your form of canonicalization.

Grüße, Carsten

*) Decimal number representation code is hard.  (I'm too lazy to look up
the CVEs to provde this.)  Even sorting text is not exactly easy.

---

From: John Cowan <[email-masked]>
Date: Fri, 5 Feb 2016 03:53:48 -0500

> the CVEs to provde this.)  Even sorting text is not exactly easy.

Decimal number representations are a well-solved problem in the Scheme
community.  See <http://people.csail.mit.edu/jaffer/III/EZFPRW> for the
code.  The references provide the background.

For canonicalization, sorting text based on lexicographical code point order
is the Right Thing, and that's well-understood too.  The only trick is to
make sure not to sort by 16-bit code units, as it gives a different answer
when comparing characters above U+FFFF with those in the range E000-FFFF.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
But you, Wormtongue, you have done what you could for your true master.  Some
reward you have earned at least.  Yet Saruman is apt to overlook his bargains.
I should advise you to go quickly and remind him, lest he forget your faithful
service.  --Gandalf

---

From: Anders Rundgren <[email-masked]>
Date: Fri, 5 Feb 2016 10:06:10 +0100

> widespread (and sufficiently debugged* on enough platforms to become

I have successfully tested Chrome, Firefox, Safari, My Java library, and a Python
patch against a file with 100M of random and selected IEEE 64-bit values.

> Wake me up when, say, ArduinoJson supports your form of canonicalization.

There are different markets with different needs and traditions.  Must they
all use the same standard(s)?

An  acquaintance of mine working for a major IoT vendor (who is very active
in the IETF/JSON space), claims that the IoT market rather will go for CBOR.

FWIW, I'm claiming that the Payment industry won't buy into JOSE, they will
probably build something of their own and ES6/V8 serialization makes that a
viable option.

PHB also seems to be experimenting with ways to get around the Base64URL dogma:
https://mailarchive.ietf.org/arch/msg/acme/hLmlH1lEEQ2GgkhgDnl5bbhhNXQ

Standards are fun...the more the merrier! :-)
https://xkcd.com/927/

Regards
Anders

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 5 Feb 2016 09:01:55 -0500

> already, with many more to come.

If you use a particular tool and that is the only one you understand,
you won't see the problem.

This is a standards organization. So the architecture should impose
the fewest constraints on implementation as possible.

Very small changes in implementation can make a huge impact on
implementation. The reason ASN.1 is hated with a passion by pretty
much all crypto folk who have implemented it is one line in the
definition of DER encoding. The use of definite length encoding over
indefinite length had virtually no impact on X.509v1. Either could
have been chosen as canonical. The one they chose has a dramatic
impact on complexity of the implementation in X.509v3.

What you are arguing for is that you want to write your structures like this

struct fred { int a, int b, int c}

{ "a" : 1, "b" : 2, "type" : "fred", "c" : 3}

Now you might think that is a reasonable way to present the data in a
spec but most of us would say that it is hard to follow. The important
information of the type of the structure is buried in the
serialization stream.

What is hard for a human to work out imposes unnecessary constraints
on computer implementations. It makes JSON a special case for no good
reason.

I have a tool that can serialize/deserialize to JSON, ASN.1 or XML
from a schema that is essentially just a description of the data types
to be passed on the wire. It doesn't support unrestricted XML and
implementing PKIX requires the schema to be decorated with additional
information to define encoding details.

The reason I like JSON is precisely the fact that right now, it does
not require any decoration and the only constraint it imposes is that
the tag to describe the data structure precede the data being
described.

---

From: Anders Rundgren <[email-masked]>
Date: Fri, 5 Feb 2016 16:10:16 +0100

On 2016-02-05 15:01, Phillip Hallam-Baker wrote:
<snip>
> information of the type of the structure is buried in the

I fully buy into the efficiency aspect of the nested encoding style (which I
honestly had never thought of), but when it comes to "hard to follow", predictable
serialization a la ES6 seems like a more universal solution since this also sucks:
   {
      "city": "Paris",
      "name": "John Doe",
      "country": "FR",
      "zip": 245333
   }

ES6 serialization and the nested encoding style scheme can easily be combined,
hopefully bringing out the best of both worlds!

Anders

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 5 Feb 2016 11:27:51 -0500

> hopefully bringing out the best of both worlds!

There are probably valid use cases for a canonical serialization
format. And that is probably more viable in JSON than in XML.

However, I am very very skeptical of the benefit of canonicalization
for digital signatures. I have been in countless meetings where this
has been asserted as a requirement and I have never once heard a
plausible argument. Emit the bits then sign them.

BTW, yes I am working on a way to avoid the wrapped Base64-url encoding:

https://tools.ietf.org/html/draft-hallambaker-json-web-service-01

This is part of The Mathematical Mesh which is described at
http://prismproof.org/

I have a scheme to get rid of the Base64-url encoding in JSON as well.
But the result isn't JSON, it is JSON-B. Which is simply JSON with
some additional code points defined for sending binary data as
length-data chunks.

The reason I want this is that I use cryptography at multiple levels
in the Mesh and I want to use the JSON data model as my only encoding
regardless of the layer I am at in my protocol stack. You can't meet
requirements for meta-data protection and end-to-end security with a
single layer of crypto. A 33% increase in data volume for each crypto
pass is unacceptable.

So I have these schemes, but they are not JSON encoding. I call them
JSON-B, JSON-C and JSON-D. The simplest, JSON-B just adds binary
encoding for numbers and length delimited strings and binary data.
JSON-C adds tag compression and JSON-D adds in more floating point
formats such as you might need for scientific data.

I suggest that you write up the canonical serialization as a separate
draft and get it an assigned content-type. We could call it JSON-F
(for fixed) or something like that.

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 6 Feb 2016 03:41:32 +0100

On 2016-02-05 17:27, Phillip Hallam-Baker wrote:
<snip>
>> ES6 serialization and the nested encoding style scheme can easily be
>> combined, hopefully bringing out the best of both worlds!
>

I have gradually lost faith in standardization as the best way forward
for stuff that is close to applications.  It takes too long time, and
there are usually multiple roads to (virtually) the same goal.  The IMO
worst thing is that WGs do not generally care about changes in the market,
or the availability of new technology although this could motivate charter
updates and sometimes even abandoning the work-item altogether.

And if you are Google you can do set standards for the rest of the world
which also makes standardization anything but a level playing field.

By building on a Google standard (V8) for JSON normalization (not canonicalization),
I feel quite safe that this concept (rather than my signature definition verbatim),
will become a de-facto standard.  That's perfectly satisfactory for my work which
simply is helping the payment industry converting to JSON.

Anders
https://cyberphone.github.io/openkeystore/resources/docs/jcs.html#ECMAScript_Compatibility_Mode