Subject: [apps-discuss] Describing JSON document formats

From: Cyrus Daboo <[email-masked]>
Date: Fri, 30 Nov 2012 11:34:56 -0500

Hi folks,
<https://datatracker.ietf.org/doc/draft-douglass-timezone-service/> and 
<https://datatracker.ietf.org/doc/draft-kewisch-et-al-icalendar-in-json/> 
both define JSON document formats.

In the former case, the spec defines the semantics of the response - we 
choose to use 
<https://datatracker.ietf.org/doc/draft-newton-json-content-rules/> as the 
way to describe that.

In the later case, the spec defines a mapping of syntax only, with the 
actual semantics primarily determined by the underlying iCalendar 
specification. In that case we choose to use ABNF to represent the document 
syntax.

My question is, is there any right way to describe a JSON document format? 
I know there are various JSON schema proposals out there, of which 
draft-newton-json-content-rules was the one I liked the most, and obviously 
I would be interested in seeing that document proceed. Any thoughts?

-- 
Cyrus Daboo

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 30 Nov 2012 12:13:22 -0500

On Fri, Nov 30, 2012 at 11:34 AM, Cyrus Daboo <[email-masked]> wrote:

> Hi folks,
> <https://datatracker.ietf.org/**doc/draft-douglass-timezone-**service/<https://datatracker.ietf.org/doc/draft-douglass-timezone-service/>>
> and <https://datatracker.ietf.org/**doc/draft-kewisch-et-al-**
> icalendar-in-json/<https://datatracker.ietf.org/doc/draft-kewisch-et-al-icalendar-in-json/>>
> choose to use <https://datatracker.ietf.org/**
> doc/draft-newton-json-content-**rules/<https://datatracker.ietf.org/doc/draft-newton-json-content-rules/>>
> as the way to describe that.
>
> draft-newton-json-content-**rules was the one I liked the most, and
> obviously I would be interested in seeing that document proceed. Any
> thoughts?

I think it would be very useful to have a mechanism that allowed JSON
formats to be mapped onto data structures and back.

I think that a mechanism that looked anything like XML Schema with the
maximum and minimum values for integers or with regular expressions and
such should be left at the door.

I also think that there is no need to use JSON syntax for the purpose of
defining the data structures.

Good:

Structure Account
    String Username
    String Realm
    DateTime Created

Bad:

Structure Widget
    Integer Height { min=1, max=142}
    Integer Width {min=1, max=23}

I have written quite a few XML Schema specs and I have never ever found a
need for more than about 10% of the features. It is not necessary to
specify the minimum and maximum number of occurrences. The lower bound is
always 0 or 1, the upper bound is always 1 or infinity.

Schema validation is bunk.

-- 
Website: http://hallambaker.com/

---

From: Nico Williams <[email-masked]>
Date: Fri, 30 Nov 2012 12:45:31 -0600

> formats to be mapped onto data structures and back.

Agreed.

> maximum and minimum values for integers or with regular expressions and such
> should be left at the door.

Well, type constraints are a good thing.  If you know that some number
will be 0..2^32 - 1 then you can use a uint32_t in a C implementation
-- at least you'll want to distinguish between integer sizes, bignums,
and so on.  Yeah, you don't need min/max to convey integer size.

The key is that the JSON syntax/schema/whatever map easily onto your
implementation language, and given the programming languages we all
typically deal with (from C to Python, JS, Ruby) I think we need
something ASN.1-ish, just not ASN.1 itself, and very light-weight.

But, really, something ASN.1-ish, minus the awful syntax, minus the
tags, and all that.  We might call it JASN, say (heh).  All we need is
to be able to specify messages as being arrays or dicts, and if arrays
of what type (including "any"), and if dicts what keys are allowed,
which are required, which are optional, and their value types, and
whether unknown keys are allowed.  Even if that's too much info, the
default should be that everything is optional, value types are "any",
and let your implementation impose any actual constraints however you
like.  And if you have a protocol that could really use constraints in
the schema, use them.

> defining the data structures.

But since you can parse it you can easily turn it into what you like:

{ 'type' : 'struct', 'name' : 'Account', 'fields' : [ [ 'Username',
'String' ], [ 'Realm', 'String' ], [ 'Created', 'DateTime' ] ] }

Done right it can be made real easy to add implementation-specific
directives ("represent this number as uint32_t, that one as uint64_t,
this dict as a struct, that dict as a hash table, ...").  To me this
is important: the lack of a decent way to add implementation
directives to ASN.1 is one of the many problems with ASN.1.

>     Integer Width {min=1, max=23}

If you don't care about the constraints, don't implement them, no?
And if nobody cares, for some protocol, then don't declare them.

> Schema validation is bunk.

Oh, well, if you've got complex config file languages then you kinda
want schema validation as it becomes a configuration linter.  I'm
bored of writing config file parsers, validators, ...

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 30 Nov 2012 14:46:30 -0500

> and so on.  Yeah, you don't need min/max to convey integer size.

Representation constraints are useful, agreed.

But these can be 8, 16, 32, 64 or BIgNum, signed or unsigned.

> something ASN.1-ish, just not ASN.1 itself, and very light-weight.

Agree on the languages and willing to contribute a compiler that can
generate code for any of them.

Just got to get my ass into gear and work out how to use GIT in visual
studio.

> tags, and all that.  We might call it JASN, say (heh).

Why limit it to JSON, can generate a subset of XML and ASN.1 as well.

XML Scema was sufficiently bjorked that when I was writing the SAML spec I
had to use a tool to generate the XML schema from something sensible.

>  All we need is
> the schema, use them

Ah now here we get to a pet peeve of mine. Extensibility. I don't think any
of the protocols I have worked on has really had a good model.

The extensibility mechanisms in XML work fine for documents. For protocols
they are a #$()WQFE&QWE #$$# !!!! pain in the ass. They just don't work and
xxx:namespace yyy:prefixes are stupid.

> bored of writing config file parsers, validators, ...

My problem is that none of the schema validation attributes has been
sufficient for input validation. If I have a data item representing a
cheque then the important thing is that the payment is less than the
balance on that account, not that it is less than some static value in a
schema.

I agree that data input validation is important, just don't see that static
schema constraints help.

---

From: Nico Williams <[email-masked]>
Date: Fri, 30 Nov 2012 15:25:54 -0600

> But these can be 8, 16, 32, 64 or BIgNum, signed or unsigned.

Sure.  Note that this can be done in ASN.1:

UInt32 ::= Integer (0..4294967295)

but that sort of thing kinda needs to be standard in order to function
properly as compiler hints.

> generate code for any of them.

I'd be willing to contribute to an open source compiler.  Since I like
ASN.1 and have contributed (minor stuff) to the Heimdal ASN.1
compiler, one idea that comes to mind is to have a JASN->ASN.1
transformation and then use ASN.1 compilers where one has them.

> Just got to get my ass into gear and work out how to use GIT in visual

http://stackoverflow.com/questions/507343/using-git-with-visual-studio

> Why limit it to JSON, can generate a subset of XML and ASN.1 as well.

Well, I didn't want to go there (given the animosity to ASN.1 'round
these parts), but IMO there's been nothing really new under the sun in
data encoding since S-expressions.  There have been new things
regarding data handling (XPath and XSLT, though I also like to think
that these are [very, very awesome] generalizations of LISP features
like destructuring-bind).  But abstract syntax notations are all
roughly equivalent.  Existence proof: XER and FastInfoSet.  (Even XDR
is really a subset of ASN.1 paired with a form of PER that is 4-octet
aligned and with some minor differences in the encoding of
absent/defaulted fields.)

So, yes: this JASN thing should be able to generate proper ASN.1.

One of the nice things about using JSON for the syntax is that it
should be naturally extensible.  If we start with a syntax geared for
just JSON today, then later want to be able to generate arbitrary
ASN.1, we'll need to be able to specify tags and such, and... that's
just new fields that old JASN parsers can just ignore.

> had to use a tool to generate the XML schema from something sensible.

I know, but really, XML Schema can be mapped onto what you want
anyways.  It's just more useful to have a syntax that more simply maps
onto programming language types to represent the same data.

> > [...]

No?  Kerberos has been extended extensively over the years -- it's not
been perfect, but it has worked out.

> xxx:namespace yyy:prefixes are stupid.

Agreed.  For protocols we almost never need namespaces the way we do
for documents.

> [Validation stuff elided.]
>
> sufficient for input validation. If I have a data item representing a cheque
> then the important thing is that the payment is less than the balance on
> that account, not that it is less than some static value in a schema.

That's semantic validation.  Yeah, it can't be done with an abstract
syntax like ASN.1 or XML Schema.  (One might here mention ITU-T's SDL,
but that's never been used in the IETF, there's no toolkits, and SDL
is borked anyways by being case-insensitive, which renders it
incompatible with ASN.1.)

> I agree that data input validation is important, just don't see that static

Well, for config files it's a lot better than nothing, but yes, it
doesn't go far enough.

Nico
--