Subject: [Json] Bridging the gap between ASN.1 and JSON

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 24 May 2018 09:53:15 -0400

JSON is a really good encoding for many applications. For some it is not
optional and neither are the existing alternatives on offer.

* Base 64 encoding binary data increases data size by 33% per pass.

* Encoding of floating point binary in decimal notation entails a loss of
precision.

* Tags are large

* Tags are required

* Cannot direct the order in which elements are presented.

What I want is an encoding that preserves the JSON data model as closely as
possible while providing support for these features. I do not want to
switch from JSON to ASN.1/CBOR/XML because they require me to change my
data model.

I have addressed all but the last two in the following:

http://mathmesh.com/Documents/draft-hallambaker-jsonbcd.html

It recently occurred to me that the JSON approach could be extended to
allow tags to be omitted when the sender and receiver share a schema
definition.

Consider the following template:

{ "A": "One", B: 2, C: { D: 4 } }

The template provides an implicit (partial) specification of the data types
for the object fields. A is a string, B a number and so on. Since it is
JSON, the order of the tags does not matter which means that we are
required to provide them to allow the object to be decoded:

{  C: { D: 4 } , "A": "One", B: 2 }
{ "A": "One", C: { D: 4 }, B: 2,}

Now let us imagine that the first template is known to both sides and that
if a tag is omitted, it is implied. We can now encode the above as follows.

{ "One", 2, { D: 4 } }

Or if we wanted (thanks Steve Prior) we could distinguish this production
by use of different braces:

< "One", 2, < 4 > >

For large data sets, this approach leads to a significant decrease in size,
a decrease that is not achieved by compression unless the underlying data
is also repetitive.

This approach is already used in many modern computing languages. C#
classes used to be a mess with often a dozen constructors for essentially
the same purposes. These days the constructors and methods can look like
this:

        public static byte[] Enhance (
                    byte[] MasterSecret,
                    byte[] Plaintext,
                    byte[] Salt = null,
                    CryptoProviderEncryption ProviderEncrypt = null
                    ) {

Default parameters are optional and will take the specified default value.
Tags are only required when a parameter is omitted entirely.

---

From: Nico Williams <[email-masked]>
Date: Thu, 24 May 2018 11:24:40 -0500

> optional and neither are the existing alternatives on offer.

Sounds a lot like you want something like FastInfoSet, which is
basically PER (Packed Encoding Rules) applied to XML.

> * Base 64 encoding binary data increases data size by 33% per pass.

Yep.

> * Encoding of floating point binary in decimal notation entails a loss of

Yep.

> * Tags are required

You can always use arrays...  Then missing "fields" only take up four or
five bytes ('null', ',null', or 'null,').  See below.

> * Cannot direct the order in which elements are presented.

I'd add:

 * Requires escaping strings (ugh)

Escaping/unescaping sucks.

> switch from JSON to ASN.1/CBOR/XML because they require me to change my

I thought JSON round-trips through CBOR.  What's wrong with that?

> allow tags to be omitted when the sender and receiver share a schema

This is the *same* idea as in ASN.1's PER and OER, which use no tags,
and minimize the use of lengths, so there's no TLV in PER/OER.

(There are some lengths in PER/OER, especially for fields added after
extensibility markers.)

The encoding of an object here is a lot like an array of field value
encodings (see above comment about arrays).

There's nothing new in this field, it seems :)

> a decrease that is not achieved by compression unless the underlying data

Yes.  That's why PER/OER exists.  It's also why FastInfoSet exists.

Now, I suspect that for JSON starting from scratch -- but using PER/OER
as inspiration -- will be better than the FastInfoSet approach.

Cheers,

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 24 May 2018 18:57:56 +0200

>> * Tags are required

(I think this is about what is called “names” in RFC 8259.)

Yep.

> I thought JSON round-trips through CBOR.  What’s wrong with that?

Absolutely nothing.

That is the approach that, e.g., OCF is using.
While they sometimes seem to wish they had all of CBOR available to them, their tools tie them to the JSON subset, and CBOR is just fine with that (no I-JSON jitters, either).
(And they can always liberate themselves once the tools catch up.)

> and minimize the use of lengths, so there’s no TLV in PER/OER.
> 

Exactly: the evolving parts of protocols don’t get to use this feature.
So after a while of living with a protocol, the utility of coding optimizations that only know about the ancient base spec diminishes.

> encodings (see above comment about arrays).

Well, you can deserialize (decode) CBOR/JSON arrays without a schema, which you can’t with PER.
(You may not know what the array elements *mean*, but you don’t know that with just a member name, either.)

XML(*) and JSON got this one right: The deserializer is schema-independent.
I don’t think we want to regress behind that for most applications — it turns out that plain data compression tends to work much better than what you can cook up in complex schema-dependent encoding rules.

Grüße, Carsten

(*) The original, character-based XML — binary XML a.k.a. EXI has a schema-dependent mode that is reminiscent of PER in this aspect.

---

From: Erik Wilde <[email-masked]>
Date: Thu, 24 May 2018 19:58:15 +0200

maybe have a brief look at http://bsonspec.org/ ? cheers, dret.

On 2018-05-24 15:53, Phillip Hallam-Baker wrote:
> * Encoding of floating point binary in decimal notation entails a loss 
> of precision.
> 
> * Tags are large
> 
> * Tags are required
> 
> What I want is an encoding that preserves the JSON data model as closely 
> as possible while providing support for these features. I do not want to 
> The template provides an implicit (partial) specification of the data 
> types for the object fields. A is a string, B a number and so on. Since 
> it is JSON, the order of the tags does not matter which means that we 
> are required to provide them to allow the object to be decoded:
> 
> Now let us imagine that the first template is known to both sides and 
> that if a tag is omitted, it is implied. We can now encode the above as 
> follows.
> 
> { "One", 2, { D: 4 } }
> 
> Or if we wanted (thanks Steve Prior) we could distinguish this 
> production by use of different braces:
> 
> < "One", 2, < 4 > >
> 
> For large data sets, this approach leads to a significant decrease in 
> size, a decrease that is not achieved by compression unless the 
> underlying data is also repetitive.
> 
> classes used to be a mess with often a dozen constructors for 
> essentially the same purposes. These days the constructors and methods 
> can look like this:
> 
> Default parameters are optional and will take the specified default 
> value. Tags are only required when a parameter is omitted entirely.
> 
> 
> 
> 
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
> 

-- 
erik wilde | mailto:[email-masked] |
            | http://dret.net/netdret    |
            | http://twitter.com/dret    |

---

From: Nico Williams <[email-masked]>
Date: Thu, 24 May 2018 13:24:49 -0500

> > I thought JSON round-trips through CBOR.  What’s wrong with that?

I thought so.  Thanks.

> >> It recently occurred to me that the JSON approach could be extended
> >> to allow tags to be omitted when the sender and receiver share a
> >> schema definition.
> > 
> > This is the *same* idea as in ASN.1's PER and OER, which use no
> > tags, and minimize the use of lengths, so there’s no TLV in PER/OER.
> > 
> > (There are some lengths in PER/OER, especially for fields added
> > after extensibility markers.)
> 
> Exactly: the evolving parts of protocols don’t get to use this
> feature.  So after a while of living with a protocol, the utility of
> coding optimizations that only know about the ancient base spec
> diminishes.

Eh?  You don't end up with TLV all the way down for extensions.  E.g.,
if you add a field of some SEQUENCE type to another, the encoding of the
new field will not be BER/DER/CER or in any way TLV-ish.  The length of
the new value will be included in the containing SEQUENCE, but that's
it.  So you still get the benfit of PER/OER.

Also, you can always use IOS or OCTET STRINGs to embed extensions
without having to resort to extensibility markers.  The net effect is
similar: each extension will have to include a TL, but not the inside of
the extension.

> Well, you can deserialize (decode) CBOR/JSON arrays without a schema,
> which you can’t with PER.  (You may not know what the array elements
> *mean*, but you don’t know that with just a member name, either.)

Well, sure, but that's not a win.  The whole point is that you have a
schema because you kinda have to have a schema, so just use the schema.

Yes, asn1dump-type programs can be useful without a schema for data
encoded in BER/DER/CER, but so what, you almost always want to dump data
of a known type (schema).

> XML(*) and JSON got this one right: The deserializer is
> schema-independent.  I don’t think we want to regress behind that for

This is good for text-based encodings because half the value of such
encodings is that humans can read them.

> most applications — it turns out that plain data compression tends to
> work much better than what you can cook up in complex schema-dependent
> encoding rules.

Well, if you've got lots of base64-encoded binary data in JSON, does it
still compress as well as just using a PER-like encoding?

Text-based encodings tend to end up getting used in contexts where the
ability to read them by naked eye is not that useful (e.g.,
cryptographic protocols), and then all the associated pain (verbosity,
escaping, ...) becomes a net negative.

This is not the first time this has happened.  XML has FastInfoSet
precisely because of this.  I'm sure XML/FastInfoSet was not the first
instance either, though I don't know of any others -- I'm sure just
because it seems like everything in this space is a reinvention of
something else.  Plus ça change...

I don't think Phillip is barking up the wrong tree, but using CBOR seems
like the right answer -- just stick to a subset that is semantically
compatible with JSON so your JSON round-trips.  If need be, extend your
CBOR tools so they know when to base64-decode binary data from input
JSON and when to base64-encode it for output.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 24 May 2018 15:56:05 -0400

> maybe have a brief look at http://bsonspec.org/ ? cheers, dret.

​Yes, I have seen it. As with CBOR, my objection is that I do not want a
new encoding, I want an ​encoding that has JSON as a subset. That allows me
to have one decoder that is capable of decoding any message regardless of
whether it is in basic JSON or an extension.

That in turn greatly simplifies content negotiation because we are only
negotiating the version of the encoding system, not the type. If a receiver
says it is capable of accepting JSON, JSON-B and JSON-C, the sender can
send any one of those encodings without having to specify which one because
there is only a JSON-C decoder.

I am also unhappy with binary encodings that extend the JSON data model.
Not only is every valid JSON sequence a valid JSON-B sequence (so round
tripping is supported trivially), it is possible to round trip JSON-B to
JSON and back without any changes other than the inevitable loss of
precision on floating point numbers.

​
On Thu, May 24, 2018 at 12:57 PM, Carsten Bormann <[email-masked]> wrote:

> ​
> ​​
> So after a while of living with a protocol, the utility of coding
> optimizations that only know about the ancient base spec diminishes.

> This is the *same* idea as in ASN.1's PER and OER, which use no tags,

​Not quite. ASN.1 is a mess because there is a half-baked version of the
same idea in BER which means that there is an excruciating version in DER.

We agree that it should be possible to decode data without a separate
schema. But if we are talking about really repetitive data such as log
files, ​having to specify every tag every time is a real drag.

W3C-Log file format has something of this idea, the first line of the log
specifies which identifiers are being logged. We could adopt the same sort
of thing to add implicit tagging to JSON without moving away from text
based encoding.

< "a", "b", "c" : < "d", "e"> >
{ 1, 2, { 3, 4}}
{ 5, 6, { 7, 8}, "f": 9}

​Can be expanded to:​

{ "a" :1, "b":2, "c":{ "d":3, "e":4}}
{ "a" :5, "b":6, "c":{ "d":7, "e":8} , "f": 9 }
​
​​Having been playing about with the idea of encrypting log files for GDPR
purposes, I am convinced that even an encrypted log needs to be essentially
a plaintext construct underneath.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 24 May 2018 23:30:10 +0200

On May 24, 2018, at 21:56, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> As with CBOR, my objection is that I do not want a new encoding, I want an ​encoding that has JSON as a subset. That allows me to have one decoder that is capable of decoding any message regardless of whether it is in basic JSON or an extension.

I know that this was a design goal of your various proposals; I just think it is not a particularly useful one.

But anyway, if it is important to you, it is easy to achieve with CBOR.  It’s easiest if your data model can get by with top-level containers (the way JSON was up to RFC 4627, before RFC 7159 embraced top-level atoms).

If your data model is limited to this at least for the CBOR case, then just send either JSON or CBOR.  The receiver can look at the first byte: If the high bit is set, decode as CBOR, else decode as JSON.

Mission accomplished: the JSON∪CBOR format I just described is a superset of JSON.

If you need the complete JSON data model on the CBOR side as well, too just add the “this is CBOR” tag 55799 — encoded as 0xd9d9f7 — to indicate the non-container CBOR case; see section 2.4.5 of RFC 7049.  This allows the high-bit discriminator to work for top-level integers and strings, too.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Thu, 24 May 2018 17:22:56 -0500

On Thu, May 24, 2018 at 03:56:05PM -0400, Phillip Hallam-Baker wrote:
> Yes, I have seen it. As with CBOR, my objection is that I do not want a
> new encoding, I want an encoding that has JSON as a subset. That allows me

I don't see the value of JSON as a subset of this new thing.  You can
always write two decoders, or one decoder that does both, provided you
can unambiguously tell which encoding the input is in.

> I am also unhappy with binary encodings that extend the JSON data model.

If it ain't a binary extension of JSON then you'll still have to use
base64 and you'll still have to escape string.  I thought you wanted to
avoid base64.  What gives.