Subject: [Json] Schemas & so on

From: Tim Bray <[email-masked]>
Date: Sun, 1 May 2016 13:55:31 -0700

I find myself tasked with specifying a JSON-based DSL and preparing it for
public release, with a validator and so on.

I had never really concerned myself much with options for JSON language
definition, but have discovered they’re not very good.  The JSON Schema
project is not terribly appealing - opaque spec, poor documentation and
tools - and smells of neglect (last I-D expired in 2013).  It's been
suggested that a good approach would be just to write a jq program that
emits true or false.

Is there good conventional wisdom about formally specifying a JSON dialect?

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 2 May 2016 08:30:23 +0900

Hello Tim,

> Is there good conventional wisdom about formally specifying a JSON dialect?

Sorry for playing the devil's advocate, but asking the same for XML 
would give a lot of different opinions. I wouldn't expect the schema 
landscape (if it developed) to be much different, because there's a 
large span between simplicity and expressiveness that can be covered, 
and a lot of notational choices.

In addition to that, many people use JSON because they don't want to use 
XML (if they don't have any actual experience, they at least heard that 
it's somehow "problematic", and that often extends to schemata). So the 
JSON's users tendency seems to be to try and avoid schemata.

Regards,   Martin.

---

From: Rob Sayre <[email-masked]>
Date: Sun, 1 May 2016 16:41:01 -0700

It's rough terrain because anything that requires a schema could also avoid
parsing and use Protobufs, Thrift, Avro, etc. I've had good luck with
Thrift and Protobufs/gRPC. ymmv.

- Rob

On Sun, May 1, 2016 at 4:30 PM, Martin J. Dürst <[email-masked]>
wrote:

> Sorry for playing the devil's advocate, but asking the same for XML would
> give a lot of different opinions. I wouldn't expect the schema landscape
> (if it developed) to be much different, because there's a large span
> between simplicity and expressiveness that can be covered, and a lot of
> notational choices.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Erik Wilde <[email-masked]>
Date: Sun, 1 May 2016 16:40:55 -0700

> suggested that a good approach would be just to write a jq program that

FYI, at the WWW2016 conference three weeks ago there was a paper looking 
at JSON schema which came away with the same conclusions: 
implementations behave differently, but sadly that can (at least in 
part) be blamed on the spec itself, which simply is not completely 
specifying what a JSON schema really means:

http://www2016.net/proceedings/proceedings/p263.pdf

personally, i think that a better schema language might have some 
chances in the wild, something that's similar to what RELAX NG is in XML 
land: focused, clean, and simple. JSON schema is a rather odd grab bag 
of features, and as long as the semantics are not completely defined, i 
wouldn't even really call it a language.

so currently, there really isn't a language that's stable and popular 
enough so that some subset of readers/developers/tools would be happy to 
see and use a schema. which i think is not great.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked] |
            | http://dret.net/netdret    |
            | http://twitter.com/dret    |

---

From: Erik Wilde <[email-masked]>
Date: Sun, 1 May 2016 16:47:26 -0700

hello martin.

> and a lot of notational choices.

yes. and XML land was/is different because often, the document models 
are complex enough that some schema really is needed, and DTDs are just 
too primitive to be useful.

but seeing the popularity of JSON in APIs and how often these also 
specify non-trivial and non-rigid "schemas", part of me thinks that 
given a language that allows to better "document" the grammar part of an 
API, some not-so-small subset of API devs/users might find that useful.

> JSON's users tendency seems to be to try and avoid schemata.

to some extent, that's certainly what's happening. but part of that may 
just be that so far, there simply isn't a good language to use.

so they add a bunch of examples. which of course are not painting the 
full picture. and consumers have to look at all of them and have to 
figure out the white spots. i think there is some sweet spot to make 
that a bit less tedious, without making it so complicated/complex that 
people will get XSD shock syndrome.

cheers,

dret.

---

From: Tim Bray <[email-masked]>
Date: Sun, 1 May 2016 18:13:13 -0700

> between simplicity and expressiveness that can be covered, and a lot of

​Hey Martin, I think the evidence is against you on this.  Everyone I know
working in XML uses RelaxNG when they think they need a formal
specification - it’s very polished, and the fact that there’s an excellent
free validator written by James Clark is another really good argument.   On
the other hand, if you want to run a bunch of different tests ad-hoc tests
against some XML and get useful error messages, Schematron is super handy.
I don’t know anyone who seriously uses XSD for any new work.  ​
​

> ​​
> ​​
> ​​
​
I agree.  ​I also think you probably should’t use XML unless you are doing
something really document-flavored, and most people aren’t.  ​I also find
that a lot of apps, perhaps most, don’t benefit that much from schemas.

I’m currently trying to specify a DSL rather than an API, and I think
having a good schema language/toolset would make my life easier :(

​​

​​

---

From: John Cowan <[email-masked]>
Date: Sun, 1 May 2016 21:58:25 -0400

> I don’t know anyone who seriously uses XSD for any new work.  ​

It was definitely being used for new work inside my then employer as of
2014, although we were generating the XSDs by conversion from RELAX NG,
because it was the only schema language that was suitable for object
binding to Java.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
May the hair on your toes never fall out!  --Thorin Oakenshield (to Bilbo)

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 1 May 2016 23:54:47 -0400

https://github.com/hallambaker/PHB-Build-Tools

Protogen may do what you want. It isn't anything like XML Schema but
starting from an abstract definition of the protocol, it will generate
implementation code in C or C# and Internet Draft suitable documentation in
Markdown, XML2RFC or HTML.

I used it to develop this spec from start to finish in three days including
a reference implementation:

https://tools.ietf.org/html/draft-hallambaker-lurk-02

The big problem with XML schema is that it is trying to provide all the
functionality of DTDs and in a data description language you need none of
that functionality.

At this point the only distinction made in the schema is between lists of
zero, one and many items.

The schema language could be cleaned up. But if I was going to do that, I
would want to develop one input language that works for my ASN.1, RFC822
header, TLS Schema and JSON backends.

TLS schema requires things like integer sizes to be specified. JSON
obviously does not. Except of course for cryptographic bignums where we
tend to use base64 strings.

---

From: Anders Rundgren <[email-masked]>
Date: Mon, 2 May 2016 07:57:52 +0200

On 2016-05-01 22:55, Tim Bray wrote:
>
> I find myself tasked with specifying a JSON-based DSL and preparing it for public release, with a validator and so on.
>
> I had never really concerned myself much with options for JSON language definition, but have discovered they’re not very good.  The JSON Schema project is not terribly appealing - opaque spec, poor documentation and tools - and smells of neglect (last I-D expired in 2013).  It's been suggested that a good approach would be just to write a jq program that emits true or false.
>

I had used XML Schema for 10Y+ before I became a JSON "convert".

I addressed the problem you mentioned in another and less formal way:

1. Creating a strongly typed JSON parser with access methods like getString(), getInt() etc.  This also supports some derived types that you most likely need such as dateTime, BigInteger and BigDecimal (for the financial industry).

2. The I added a method for verifying that all elements have been read.

This works quite well but of course do not address the documentation issue or the ability creating JSON from static object declaration.

https://github.com/cyberphone/saturn/blob/master/resources/common/org/webpki/saturn/common/PayerAuthorization.java

Anders

>
> -- 
> - Tim Bray (If you’d like to send me a private message, see https://keybase.io/timbray)
>
>

---

From: Peter Cordell <[email-masked]>
Date: Mon, 2 May 2016 11:51:16 +0100

On 01/05/2016 21:55, Tim Bray wrote:
>
> I find myself tasked with specifying a JSON-based DSL and preparing it
> for public release, with a validator and so on.
>

Andy Newton and I are working on JSON Content Rules (JCR) [1] which is a 
basic JSON grammar schema language that mirrors the functionality of 
Relax NG and XSD.  We also have put together some thoughts on JSON 
Content Rules Co-Constraints (JCRCC) [2] which adds Schematron like 
functionality to JCR.  There's some links to pilot implementations at [3].

Our goal has been to make it as simple as we can without losing 
expressiveness with a view to not requiring users to be experts in the 
technology before they can make use of it.  We're also hoping that a 
simple solution will also readily enable multiple implementations.

We'd obviously be delighted to hear from anybody who's interested in the 
work.

Thanks,

Pete.

[1] https://datatracker.ietf.org/doc/draft-newton-json-content-rules/
[2] https://datatracker.ietf.org/doc/draft-cordell-jcr-co-constraints/
[3] http://codalogic.github.io/jcr/
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 2 May 2016 16:42:00 +0000

I've been using CDDL recently:

https://tools.ietf.org/html/draft-greevenbosch-appsawg-cbor-cddl-08

Even though it says "CBOR" in the name, it works *quite* nicely for JSON.  Install Ruby tooling with `gem install cddl`.  Perhaps Carsten can send us an updated pointer to the source for that tool.

Example CDDL:

person = {
  name: text,
  age: int
}

And a conforming person JSON would be: {"name": "Aardvark", "age": 1}

-- 
Joe Hildebrand

On 5/1/16, 2:55 PM, "json on behalf of Tim Bray" <[email-masked] on behalf of [email-masked]> wrote:

>I had never really concerned myself much with options for JSON language definition, but have discovered they’re not very good.  The JSON Schema project is not terribly appealing - opaque spec, poor documentation
> and tools - and smells of neglect (last I-D expired in 2013).  It's been suggested that a good approach would be just to write a jq program that emits true or false.
>
>
>https://keybase.io/timbray <https://keybase.io/timbray>)
>
>
>

---

From: Austin William Wright <[email-masked]>
Date: Mon, 2 May 2016 11:11:27 -0700

> Is there good conventional wisdom about formally specifying a JSON dialect?

JSON Schema _is_ an active and ongoing effort. But as is usually the case
with anything powerful and self-descriptive, there's a lot of problems that
will crop up, a tiny few of which we've identified for the next release,
but still certainly plenty more. This is compounded by the fact JSON Schema
is hypermedia.

We're actively soliciting feedback for a release at the GitHub repo:
<https://github.com/json-schema-org/json-schema-spec>

We'd like to hear what relative weaknesses you've encountered, and
comparisons to other technologies. As far as tooling goes, there's a
standard test suite, and as far as I'm aware JSON Schema has more
implementations than anything else.

Thanks,

Austin Wright.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 May 2016 10:10:00 +1000

FWIW -

This general area of discussion came up in the IETF maybe two years ago; can dig around in archives if people are interested, but IIRC it was mostly hallway-based.

I'd characterise the general feeling then as "run away screaming"; the wounds of XSD / WS-* were too fresh.

It came up again more recently (again, the hallway track), and the feeling I got was that people were more amenable to it, but still concerned about the numerous pitfalls.

It strikes me that it might be worth writing down what we think is good in such a beast, and what would be not-good. Getting agreement on that might be... interesting, but if we had such a document (or maybe just a wiki page), we'd be able to evaluate the current contenders, at least.

Cheers,

> On 2 May 2016, at 6:55 AM, Tim Bray <[email-masked]> wrote:
> 
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: John Cowan <[email-masked]>
Date: Mon, 2 May 2016 21:01:11 -0400

Mark Nottingham scripsit:

> It strikes me that it might be worth writing down what we think is good
> in such a beast, and what would be not-good. Getting agreement on that
> might be... interesting, but if we had such a document (or maybe just
> a wiki page), we'd be able to evaluate the current contenders, at least.

Here's what I think the bare necessities are for a schema language.
This is loosely based on FtanGram, the schema language for FtanML,
a markup language similar to XML and JSON.

1) A language for specifying the types of JSON items.  A type is a set of
JSON items.  If the root item matches the root type and each sub-item
recursively matches the subtype, then the JSON document is valid,
otherwise not.

2) The following primitive types:
2a) The set containing only JSON null.
2b) The set containing both JSON booleans.
2c) The set containing only JSON true.
2d) The set containing only JSON false.
2e) The set containing all JSON numbers.
2f) The set containing all JSON integers.
2g) The set containing all JSON strings.

3) The type of JSON numbers with a specified upper and/or lower bound
(inclusive or exclusive).

4) The type of JSON integers with a specified upper and/or lower bound
(inclusive or exclusive).

5) The type of JSON strings that match a specified regular expression.

6) The type of all JSON arrays.

7) The type of JSON arrays that match a specified regular expression
over types which match the elements of the array.  The regular expression
operators are sequence, choice, and bounded and unbounded repetition.

8) The type of all JSON objects.

9) The type of JSON objects with specified keys, with the types of the
values corresponding to those keys.  Keys may be required or optional.
In addition, a way to specify whether other keys are allowed or forbidden.

10) The type of all JSON items.

11) The ability to name a type and refer to it by name.

12) The ability to define type libraries and import types from them.

Comments?

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Awk!" sed Grep. "A fscking python is perloining my Ruby; let me bash
    him with a Cshell!  Vi didn't I mount it on a troff?" --Francis Turner

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 May 2016 11:13:46 +1000

> On 3 May 2016, at 11:01 AM, John Cowan <[email-masked]> wrote:
> 

Forbidding unrecognised keys isn't very JSONic, and arguably gets us into the same quagmire that XSD became.

Cheers,

---

From: Erik Wilde <[email-masked]>
Date: Mon, 2 May 2016 19:17:27 -0700

> It strikes me that it might be worth writing down what we think is good in such a beast, and what would be not-good. Getting agreement on that might be... interesting, but if we had such a document…

fwiw, after the discussion started on this list, it got me thinking 
because as mark noted, the discussion tends to start and then stop 
without a truly successful candidate emerging (so far). i think that 
there will be one, eventually, and over the weekend wrote down my 
thoughts about it here:

http://dret.typepad.com/dretblog/2016/05/json-schema-why-and-how.html

cheers,

dret.

---

From: Anders Rundgren <[email-masked]>
Date: Tue, 3 May 2016 04:53:56 +0200

> http://dret.typepad.com/dretblog/2016/05/json-schema-why-and-how.html

Thanx!

IMO, the #1 problem is that there are at least three potential consumers of a "schema":
- Validators
- Code generators
- System documentation

I don't think a static declarative system would be very useful.   OTOH, scripting
would work particularly well for documentation so I will personally stick to
the programmatic approach.

I'm mainly concerned about the external part (documentation).  Here is an example:
https://cyberphone.github.io/openkeystore/resources/docs/keygen2.html#InvocationRequest

Anders

---

From: Tim Bray <[email-masked]>
Date: Mon, 2 May 2016 20:03:05 -0700

On Mon, May 2, 2016 at 5:10 PM, Mark Nottingham <[email-masked]> wrote:

It strikes me that it might be worth writing down what we think is good in
> such a beast, and what would be not-good. Getting agreement on that might
> be... interesting, but if we had such a document (or maybe just a wiki
> page), we'd be able to evaluate the current contenders, at least.
>

[Not facetiously] RelaxNG, only for JSON.

1. based on a carefully-worked-through formalism (hedge automata in the XML
case)
2. Expressive and readable (didn't insist on schema & instance in same
syntax)
3. Came with a really-silck validator that Just Worked and was easy to
embed in other languages (who ya gonna call? James Clark!)

Unfortunately, if my dim recollection of hedge automata is correct, they
are not appropriate for JSON.
​​

---

From: Tim Bray <[email-masked]>
Date: Mon, 2 May 2016 20:04:27 -0700

> the same quagmire that XSD became.

​Haha, I like that, we’ll MUST-IGNORE-IFY ALL THE APIs!  Simply make​
MUST-UNDERSTAND inexpressible.  Effective if a bit Orwellian.

---

From: Rob Sayre <[email-masked]>
Date: Mon, 2 May 2016 20:15:07 -0700

> MUST-UNDERSTAND inexpressible.  Effective if a bit Orwellian.

This discussion has not adequately addressed the fact that there are much
more efficient ways to encode JSON data if you have a schema. It can be
protobufs, thrift, cbor, avro, or whatever. They do must-ignorify
everything, btw.

The IETF is far behind the industry standard here.

- Rob

---

From: Tim Bray <[email-masked]>
Date: Mon, 2 May 2016 20:27:56 -0700

> The IETF is far behind the industry standard here.

I suppose, but

1. It is rarely the case that markup serialization/deserialization
efficiency is that significant to the performance of an application, and
2. being binary sucks.

Anyhow, in my day job we want to have a JSON DSL and we want to have a
validator for it, and we think a decent schema language would facilitate
construction of the validator.

​​

​​

---

From: Tim Bray <[email-masked]>
Date: Mon, 2 May 2016 20:49:43 -0700

Somone linked to this paper:
http://www2016.net/proceedings/proceedings/p263.pdf [Foundations of JSON
Schema].  It’s really quite encouraging.

I don’t actually hate JSON schema, was just disappointed with the
specification and the tooling.

---

From: Erik Wilde <[email-masked]>
Date: Mon, 2 May 2016 20:58:34 -0700

On 2016-05-02 20:49, Tim Bray wrote:
> Somone linked to this
> paper: http://www2016.net/proceedings/proceedings/p263.pdf [Foundations
> of JSON Schema].  It’s really quite encouraging.

the talk they gave was good. what's needed would be to make the spec 
more formally sound (that's their specific focus, and apparently that's 
feasible without major changes to the general mechanics), and then, as a 
separate issue, maybe moving the syntax to something more palatable.

other possible improvements to the spec (personally, for example, i find 
the hypermedia aspect of it to be a bit strange at that level) then 
mostly depend on preference and perspective.

cheers,

dret.

---

From: Tim Bray <[email-masked]>
Date: Mon, 2 May 2016 21:17:22 -0700

Hrmf.  I have a big moderately-complex day-job DSL in front of me that
actually has a useful purpose and I expect tooling to be built around.  So
I need a validator and a schema would be helpful.  It’d be fun to build
schemas in JSON Schema, its subset described in that paper, CDDL, JCR, and
anything else that sounded plausible.  Then I’d have a useful opinion about
the issues, which I don’t have now.

Unfortunately our DSL isn’t decloaking till sometime in the second half of
the year.  If the matter is still relevant, I promise to do that then.

On Mon, May 2, 2016 at 8:58 PM, Erik Wilde <[email-masked]> wrote:

> the talk they gave was good. what's needed would be to make the spec more
> formally sound (that's their specific focus, and apparently that's feasible
> without major changes to the general mechanics), and then, as a separate
> issue, maybe moving the syntax to something more palatable.
>
> the hypermedia aspect of it to be a bit strange at that level) then mostly
> depend on preference and perspective.
>
>
> cheers,
>
> dret.
>
> --

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 09:12:49 -0400

> http://dret.typepad.com/dretblog/2016/05/json-schema-why-and-how.html

I'll respond here rather than as a blog comment.

I agree with most of your points and recently mentioned your XSD compact
syntax on xml-dev.  (I regret that this remained stand-alone and was never
incorporated into Jing and Trang, where it would have done the most good.)

However, I think that there is a counterexample to the "no JSON syntax"
claim, which is Examplotron.  When describing a grammar using a fixed
metaformat, using instance syntax is too much of a pain.  But when using
examples (which people have anyway) and annotating them to specify a few
critical generalities, instance syntax is a clear win.  I think that a
JSON schema language of this form, without too many bells and whistles,
might actually be usable.

I had hoped, when I started to read the JCR document, that JCR was a
language of this type.  But while valid JSON instances are for the most
part also valid JCR, an instance does not match other structurally similar
instances: instead, it matches only itself, making it entirely useless.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Any sufficiently-complicated C or Fortran program contains an ad-hoc,
informally-specified bug-ridden slow implementation of half of Common Lisp.
        --Greenspun's Tenth Rule of Programming (rules 1-9 are unknown)

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 09:17:45 -0400

> into the same quagmire that XSD became.

Well, it's not a very important point to me.  But I had thought that
fitting JSON into statically typed records (classes, structs, whatever)
was an important use case, and for that, unrecognized keys are nothing
but a nuisance.

I'll mention that I became convinced of the need for some kind of JSON
validation when I saw a tool that consumed externally specified JSON fail
because the producer had switched the capitalization conventions of a
single key, probably by accident.  The result was that all output from the
tool suddenly went to 'undefined', because there was no validation step
to ensure that the key on which the tool depended was actually present.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
A rabbi whose congregation doesn't want to drive him out of town isn't
a rabbi, and a rabbi who lets them do it isn't a man.    --Jewish saying

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 03 May 2016 09:38:20 -0400

Hi,

--On May 3, 2016 at 10:10:00 AM +1000 Mark Nottingham <[email-masked]> wrote:

> This general area of discussion came up in the IETF maybe two years ago;
> can dig around in archives if people are interested, but IIRC it was
> mostly hallway-based.
>
> It came up again more recently (again, the hallway track), and the
> feeling I got was that people were more amenable to it, but still
> concerned about the numerous pitfalls.
>
> might be... interesting, but if we had such a document (or maybe just a
> wiki page), we'd be able to evaluate the current contenders, at least.
>

At the time my personal reason for wanting something like this was not so 
much to have a formal "schema" or "rules", but rather something that could 
be used in protocol specs to "describe" what JSON an implementor would 
expect to see in a request or response. That is a very real problem that we 
faced when developing RFC7808 - which is a JSON-based HTTP protocol. 
Initially I wrote that using an early version of 
draft-newton-json-content-rules, but since that did not progress to 
publication, we were asked to switch to using the format developed in 
RFC7071 (<https://tools.ietf.org/html/rfc7071#section-6.2>).

So, rather than a formal "schema", I would rather see this group focus on a 
standard way to "describe" JSON structures in protocol specs, with the goal 
of having something easy to understand or base implementations off. Perhaps 
a purely informational document is all that is needed.

-- 
Cyrus Daboo

---

From: Victor Kareh <[email-masked]>
Date: Tue, 3 May 2016 09:57:50 -0400

Hello,

What you're describing sounds similar enough to the node.js Mongoose ORM 
schema definition [http://mongoosejs.com/docs/guide.html]. We actually 
use that style at my day job for our API documentation (not actually 
using mongoose, just the schema spec) and it works really well to 
communicate across teams providing/consuming the API. It supports a 
high-level description (no validation, just expected types and 
structure) or a more thorough version with validation rules and property 
constraints.

--
Victor Kareh

On 2016-05-03 9:38 AM, Cyrus Daboo wrote:
> At the time my personal reason for wanting something like this was not 
> so much to have a formal "schema" or "rules", but rather something 
> that could be used in protocol specs to "describe" what JSON an 
> implementor would expect to see in a request or response. That is a 
> very real problem that we faced when developing RFC7808 - which is a 
> JSON-based HTTP protocol. Initially I wrote that using an early 
> version of draft-newton-json-content-rules, but since that did not 
> progress to publication, we were asked to switch to using the format 
> developed in RFC7071 (<https://tools.ietf.org/html/rfc7071#section-6.2>).
>
> So, rather than a formal "schema", I would rather see this group focus 
> on a standard way to "describe" JSON structures in protocol specs, 
> with the goal of having something easy to understand or base 
> implementations off. Perhaps a purely informational document is all 
> that is needed.
>
>

---

From: Peter Cordell <[email-masked]>
Date: Tue, 3 May 2016 15:26:43 +0100

On 03/05/2016 14:12, John Cowan wrote:
> claim, which is Examplotron.  ...
>

I disagree that JCR is "entirely useless".  I think it's a great 
strength that you can start with an actual JSON example, but then 
generalise it via annotation to be more embracing.  I think Examplotron 
is neat, but it makes lots of assumptions about what you're trying to 
express.  It essentially guesses what you mean by a construct.  This may 
work for a little while, but soon runs out of steam.  There's a reason 
why people say you should be explicit about what you want.

Plus, there is the issue of practical syntax.  XML offers namespaces so 
you can do things like <bar eg:occurs="*"> to differentiate between one 
or more and zero or more (or optional etc).  Lacking even comments it's 
hard to see how a JSON based syntax could do this is an elegant way.  (I 
can imagine some ugly hacks involving unlikely member names, but I'd 
really rather not.)

And the ability to explicitly state that you want a particular string 
(for example), rather than having that being inferred as any string, 
opens up the use of JCR as part of a test vector suite, which could be 
very useful as part of implementation validation.

So, very useful indeed IMO.

Pete.

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 11:08:44 -0400

Peter Cordell scripsit:

> I disagree that JCR is "entirely useless".  

Sorry, I meant that using a plain JSON document as a JCR document is
entirely useless, not that JCR as a whole is useless.

> Examplotron is neat, but it makes lots of assumptions about what
> you're trying to express.  It essentially guesses what you mean by a
> construct.  

Well, I'd rather say that it has some well-documented conventions about
instance values.

> This may work for a little while, but soon runs out of
> steam.  There's a reason why people say you should be explicit about
> what you want.

>From the language designer's (and implenter's) viewpoint, yes.  From the
user's view, where the non-iconic features may be unnecessary for many
applications, not so much.  Examplotron has a gentler learning curve
than any explicit grammar-based system.

> Plus, there is the issue of practical syntax.  XML offers namespaces
> so you can do things like <bar eg:occurs="*"> to differentiate
> between one or more and zero or more (or optional etc).  Lacking
> even comments it's hard to see how a JSON based syntax could do this
> is an elegant way.  (I can imagine some ugly hacks involving
> unlikely member names, but I'd really rather not.)

And name quotation, yes.  I agree it's a problem.  Still, even if JSON
syntax should be or must be extended, I still think structural iconicity
where possible is an important principle.

> And the ability to explicitly state that you want a particular
> string (for example), rather than having that being inferred as any
> string, opens up the use of JCR as part of a test vector suite,
> which could be very useful as part of implementation validation.

Good point.

> So, very useful indeed IMO.

I have a lot of specific comments on JCR.  What's the best forum for them?

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
At times of peril or dubitation,
Perform swift circular ambulation,
With loud and high-pitched ululation.

---

From: Peter Cordell <[email-masked]>
Date: Tue, 3 May 2016 16:32:57 +0100

> I have a lot of specific comments on JCR.  What's the best forum for them?

I'd be very interested to get your feedback.  Now would be a good time 
as we've been quite fluid up until now, but are starting to feel things 
are firming up.

For the time being [email-masked] is the best option. 
It's not a mailing as such but will fan out to Andy and I.

Longer term, I believe that the IETF can set up (or used to be able to 
setup) a mailing list for topics even if they are not associated with a 
working group.  If that's still an option, and there's enough interest, 
that might be something worth pursuing.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 13:08:47 -0400

I don't see any need to specify upper or lower bounds on integers. We don't
do this in C, C# or any major programming language.

We do specify bounds on representation, int32, uint32 etc. But that is more
a matter of convenience and space management. It isn't something we need in
JSON serialization.

What we do need is a model for how to extend JSON based protocols. When is
it allowed to add in new elements to structures? what structures are
backwards compatible with the old? My answers to those being 'always' and
'when the tag is the same'.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 13:16:54 -0400

I don't see any value to 'validation' for data exchange.

The reason that it is useful in XML is that XML is a document format and it
is useful to tell the user if the document is well formed or not as they
type.

We have the following questions:

1) Is the input stream well formed JSON syntax?
2) Does the parsed JSON data contain all the necessary data fields required
for the message?
3) Are the fields of the correct type?
4) Can we extend messages and preserve backwards compatibility?
5) Can we define new messages that will be rejected by legacy systems that
are unable to process them?

If we go beyond that, we are into programming. Yes, it is occasionally
useful to state that the number of flanges to be delivered is less than 12.
But very very rarely in an IETF application protocol.

The type of constraints that are required in our types of protocol tend to
be of the sort, 'if there is a reference to an item named X, there must be
a corresponding definition elsewhere in the message'. And even those tend
to be rather complex as the widget being deleted or modified might have
been defined in a different message.

On Mon, May 2, 2016 at 11:03 PM, Tim Bray <[email-masked]> wrote:

> 1. based on a carefully-worked-through formalism (hedge automata in the
> XML case)

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 13:25:38 -0400

Phillip Hallam-Baker scripsit:

> a matter of convenience and space management. 

That seems to me to be a distinction without a difference.  We need to
specify the bounds on specific JSON numbers so that we know that we can
safely deserialize them to language-specific types.  I agree that we don't
need arbitrary bounds like -1..52 very much, but 0..255 is another matter.
It would probably be enough to allow a fixed set of bounds like u8, s8, ...
u64, s64, f32, f64.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Your worships will perhaps be thinking that it is an easy thing
to blow up a dog? [Or] to write a book?
    --Don Quixote, Introduction

---

From: Erik Wilde <[email-masked]>
Date: Tue, 3 May 2016 10:40:32 -0700

> It would probably be enough to allow a fixed set of bounds like u8, s8, ...

when it comes to datatypes and how values are being used, things get 
complicated relatively quickly. for example, even if you allow 0..255 as 
the datatype for a member, would it still be permissible to use 
"0000000000000042" as its value? it depends, right?

this is the part where XSD took a deep dive (making it complex but 
powerful) and separated the lexical space and the value space:

https://www.w3.org/TR/xmlschema-2/#typesystem

i am not saying that this complexity is what we want, but just pointing 
out that if we want some datatype facility, this is a slippery slope, 
and finding the sweet spot is not a trivial exercise.

cheers,

dret.

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 14:10:05 -0400

Erik Wilde scripsit:

> complicated relatively quickly. for example, even if you allow
> 0..255 as the datatype for a member, would it still be permissible
> to use "0000000000000042" as its value? it depends, right?

Of course it would be permissible.  42 is 00000000042 is 42.00000.

> powerful) and separated the lexical space and the value space:

I agree that this has to be done, but I think that JSON is simple enough
that the mapping is self-evident.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
It's the old, old story.  Droid meets droid.  Droid becomes chameleon.
Droid loses chameleon, chameleon becomes blob, droid gets blob back
again.  It's a classic tale.  --Kryten, Red Dwarf

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 14:12:07 -0400

> It would probably be enough to allow a fixed set of bounds like u8, s8, ...

Right now, my tool allows those hints to be specified when it is used to
generate TLS schema serializers/deserializers but I haven't seen the need
to use them for a JSON protocol so far.

I think syntactic sugar does actually matter. That is I see a big
difference between something like

int MyField;

and something like this

{"MyField" : { "type" : "Integer", "max" : 42, "min" : -32 } }

I think the chances of spotting a blooper in the second is a lot better.

If I ever did another revision to my tools to put all of the parsers into
one system, it would probably have an input language that was something
like the following:

MyStruct Class
    MyIntField Integer
    MyStringField String

If I then wanted to specify representation it would probably be by
subtyping 'Integer' to allow int8, uint8, ... int64, uint64 like you
suggest.

Right now, the only case where I see a need to call something out as a
special case is for handling BigNums. We use those a lot in Crypto but it
is a real pain to use a generator where every Integer is a bignum by
default. It is also inefficient as you have to convert your bignum to
decimal and back.

The reason for having the 'label before declaration' rule is that it
provides consistency. Every definition has the label first. C has a nasty
habit of doing this differently for classes and fields/variables. Doing it
the same way throughout is much cleaner. The label should go first because
it is always one token while the definition may be a structure.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 14:13:44 -0400

Not in JSON. No leading zeros. The only integer that starts with a zero is
zero.

(unless json.org has it wrong).

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 15:09:08 -0400

> {"MyField" : { "type" : "Integer", "max" : 42, "min" : -32 } }

I agree, but {"MyField": -254} (under a convention by which negative
numbers express signed values, if you need to distinguish signed from
unsigned) is not too bad.

> I think the chances of spotting a blooper in the second is a lot better.

The first, I suppose you mean.

> it is always one token while the definition may be a structure.

Every language not heavily under the thumb of C has always used name
before type.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
We want more school houses and less jails; more books and less arsenals;
more learning and less vice; more constant work and less crime; more
leisure and less greed; more justice and less revenge; in fact, more of
the opportunities to cultivate our better natures.  --Samuel Gompers

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 15:09:57 -0400

> 3) Are the fields of the correct type?

These points are just the kind of validation I am talking about.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
He made the Legislature meet at one-horse tank-towns out in the alfalfa
belt, so that hardly nobody could get there and most of the leaders
would stay home and let him go to work and do things as he pleased.
    --H.L. Mencken's translation of the Declaration of Independence

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 15:34:57 -0400

On Tue, May 3, 2016 at 3:09 PM, John Cowan <[email-masked]> wrote:

> Agreed. What we do not need is the type of validation XSD gives us:

1) Order in which the elements appear.
2) alternatives
3) model groups
4) the bizaro dual type system in which elments type data and types type
elements.
5) Distinguishing attributes and elements
6) etc.

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 15:50:37 -0400

> 1) Order in which the elements appear.

My understanding is that people do use JSON arrays this way: the first
element means X, the second means Y.  If arrays are always or almost
always used in a uniform way, then we don't have to worry about order,
just specify a single type type for all the individual elements.

> 2) alternatives

No cases of number-or-string, or object-or-null?

> 5) Distinguishing attributes and elements

Agreed on those.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
"But I am the real Strider, fortunately," he said, looking down at them
with his face softened by a sudden smile.  "I am Aragorn son of Arathorn,
and if by life or death I can save you, I will."

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 16:03:00 -0400

> No cases of number-or-string, or object-or-null?

I don't allow number or string or see the need.

For all fields I allow the following

Minimum Occurrences = 0 or 1
Maximum Occurrences = 1 or unlimited.

Null is only allowed if the Minimum Occurrences is 0. If Max is unlimited
then the field maps to a list rather than a simple variable and the
corresponding serialization is an array.

That is possibly too strict as right now going from max occurs 1 to
unlimited breaks backwards compatibility as the serialization changes from
item to array[item]. I could probably be loser there.

There might be an argument to be made for any 'Any' field type which can
contain anything. I don't support that right now. But I might have to
rejigger my parser if I am supporting ACME and they don't go to the nested
approach.

ACME has the following type system, lets say you are sending a Delete
request, an acceptable message serialization would be:

{ "field1" : "data",
 "field2" : "data",
 "field3" : "data",
 "type" : "Delete",
 "field4" : "data",
 "field5" : "data"}

The only way this can be supported is to parse the entire JSON tree to an
intermediate format and then perform schema processing on the tree. Which
is fine if that is the way you want to do things and you are not writing a
transaction where field1 might be several Gb and so you want to check if
you understand the transaction before accepting the data or otherwise
concerned about being practical.

If I have to support the above, I will probably do so by pre-parsing the
tree and so it would be fairly straight forward to support a variant type.

But there would have to be a good reason because it is going to have to be
mapped to a C / C# / Java type sometime.

---

From: Peter Cordell <[email-masked]>
Date: Tue, 3 May 2016 21:14:40 +0100

On 03/05/2016 19:12, Phillip Hallam-Baker wrote:
> On Tue, May 3, 2016 at 1:25 PM, John Cowan <[email-masked]
> <mailto:[email-masked]>> wrote:
>     I agree that we
>     don't
> ...

JCR allows:

    { "MyField" : -32..42 }

That's easy and very clear IMO.  While I agree that such constraints are 
rare, there is a big difference between "rare" and "never".  Parsing the 
above is no biggy and if it conveys important information to a developer 
I'd gladly pay the price for a slightly more complex bit of parsing code 
than having to do something like:

    int MyField; // Really it's restricted to -32 to 42

I think HTTP (and similar) response codes are effectively 100..999 for 
example.

Currently for numbers JCR just has integer and float.  I don't think 
there would be any resistance to adding types such as int8, uint8, 
int16, uint16, int32, uint32, int64, uint64, int128, uint128, bigint, 
float, double and decimal (or similar names).  Tedious to document, but 
if it makes a user's life easier to express what they want that's the 
correct balance IMO.

Pete.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 3 May 2016 20:34:40 +0000

>if it makes a user's life easier to express what they want that's the 

Note: int53 and uint53 are probably more important than 64 and 128, which will need special encoding anyway.

-- 
Joe Hildebrand

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 May 2016 16:44:09 -0400

On Tue, May 3, 2016 at 4:34 PM, Joe Hildebrand (jhildebr) <
[email-masked]> wrote:

> On 5/3/16, 2:14 PM, "json on behalf of Peter Cordell" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>
>

Why?

JSON does not specify limits on integer sizes. 128 would require special
handling but I do INT64s all the time.

---

From: Peter Cordell <[email-masked]>
Date: Tue, 3 May 2016 22:03:31 +0100

> Note: int53 and uint53 are probably more important than 64 and 128, which will need special encoding anyway.

Thanks.  I hadn't heard about Int53, so that's great feedback. 
Uncertainty and wanting community feedback about which values to include 
(are 128 bit values useful?) and what to call them (should they by byte, 
short, int, long for example) is the main reason why we haven't included 
such types already.

Pete.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 3 May 2016 21:24:48 +0000

On 5/3/16, 2:44 PM, "[email-masked] on behalf of Phillip Hallam-Baker" <[email-masked] on behalf of [email-masked]> wrote:

>JSON does not specify limits on integer sizes. 128 would require special handling but I do INT64s all the time.

You need to test integers larger than 2^53 with the JSON implementation in ECMAscript then.  RFC 7159, section 6 says 

"""
Note that when such software is used, numbers that are integers and
   are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
   sense that implementations will agree exactly on their numeric
   values.
"""

In particular, ECMAscript will corrupt your integers silently, unless they are encoded as (for example) strings. 

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Tue, 3 May 2016 17:45:29 -0400

Joe Hildebrand (jhildebr) scripsit:

> You need to test integers larger than 2^53 with the JSON implementation
> in ECMAscript then.  RFC 7159, section 6 says

Indeed, and browsers are surely not the only JSON implementations to
treat all numbers as IEEE 64-bit binary floats.

If we are going to have a type for integers representable as floats,
then it should be called int54 according to the usual conventions,
with the caveat that -(2**53) cannot be represented, because floats
are not two's-complement.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
On the Semantic Web, it's too hard to prove you're not a dog.
        --Bill de hOra

---

From: Peter Cordell <[email-masked]>
Date: Wed, 4 May 2016 11:43:24 +0100

> are not two's-complement.

If your protocol needed an unsigned value bigger than 32 bits, but not 
as big as 64-bits, to mirror int54, would it be appropriate to define a 
uint53 type (i.e. the unsigned version of int(N) stored in a floating 
point value is uint(N-1))?  Or would you do something else?

Thanks,

Pete.

---

From: John Cowan <[email-masked]>
Date: Wed, 4 May 2016 08:30:56 -0400

Peter Cordell scripsit:

> If your protocol needed an unsigned value bigger than 32 bits, but
> not as big as 64-bits, to mirror int54, would it be appropriate to
> define a uint53 type (i.e. the unsigned version of int(N) stored in
> a floating point value is uint(N-1))?  Or would you do something
> else?

I suppose so.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Sir, I quite agree with you, but what are we two against so many?
    --George Bernard Shaw,
         to a man booing at the opening of _Arms and the Man_

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 May 2016 10:02:41 -0400

> are not two's-complement.

I agree. I use -MaxInt as a Not a Number value in my code anyway...

Right now however, JSON isn't really suited to use as a data exchange
format for scientific use. Which is one of the things I tried to address in
JSON-B (Binary) and JSON-D (Data).

The JSON has unallocated code points that can be used to useful extend the
serialization to add additional encodings. For length delimited strings,
binary data etc. CBOR introduced a completely different data model with the
result that the JOSE working group had to be followed by a CBOR specific
one.

Binary encoding of floats is the simplest and most reliable method of
ensuring that they round trip. And that is essential if an encoding is
going to be viable for data exchange. JSON-B has binary encodings for the
IEEE 32 and 64 bit floats for that reason and JSON-D adds in the higher
precision Intel formats that use the whole 64 bits for the mantissa.

That said, binary encoding means losing the 'text file' property of JSON.
If I was going to use JSON for scientific purposes, I might well add in a
base16 or base64 encoding for floats.

Using the unused code points for extension has the advantage that even
though each serialization option requires a different encoder, a single
decoder can recognize all four variations (JSON, JSON-B, JSON-C, JSON-D).
That simplifies protocol implementation as it is only necessary for a
service to say what it will accept, it is not necessary for a client to tag
what it actually sends. It also allows encapsulation to work as you would
expect.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 May 2016 10:29:57 -0400

If we did get into extended schema discussions, one of my main issues that
I don't see well supported in existing schema systems is support for
encryption and signature. These are features that I really would like to be
able to have supported in the 'validation' layer.

In the Mesh I have nested structures where portions of the inner structure
are signed. A user profile contains a set of application profiles, both of
which are signed. One side effect is that every profile structure ends up
having two flavors - signed and unsigned. And that is messy.

So for example, say this is my schema without signatures

UserProfile Class
    Applications     List <ApplicationProfile>
    Devices      List <DeviceProfile>

ApplicationProfile Class <Abstract>
    ... fields

And a JSON encoded object would be:

"UserProfile" : { "Applications" : [ {"MailApplicationProfile" : {...} } ] }

Since Application Profile is an abstract class, the deserializer needs to
know which particular subtype is being decoded. So the instances of the
class need to be tagged.

Easy enough, right? Well if we add signatures (or encryption) they are in
effect just wrappers round a class. But they have the effect of burying
that tag I need to decode the object.

So I would like to have something like:

ApplicationProfile Class <Signed Abstract>

"UserProfile" : { "Applications" : [
     ["Jose-Header" ,  $$ {"MailApplicationProfile" : {...} } ] } $$,
"Jose-trailer" ]
      ] ...] }

Where $$...$$ is the base 64 signed data blob as a string. Same for
encrypted data.

This type of approach would let me move all the signature validation parts
into code. I would still need to hand code the path math for validation of
the signing keys. When rendered in C#, I would get a class like this:

class UserProfile : JSON.Object {
   List <ApplicationProfile> Applications;
   List <DeviceProfile> Devices;
   }

abstract class ApplicationProfile : JSON.SignedObject {
   ... fields
   }

class MailApplicationProfile : ApplicationProfile {
   ...
   }

The JSON.SignedObject class would inherit from JSON.Object and provide
mechanisms for verifying signatures, etc. Using the array allows the order
of the JOSE data items to be forced which helps a lot in writing
verification code into the deserializer which is of course essential if you
want to do a one pass deserialization on a constrained device.

Encryption would be handled in a similar way by JSON.EncryptedObject.

---

From: John Cowan <[email-masked]>
Date: Wed, 4 May 2016 11:00:50 -0400

> I don't allow number or string or see the need.

It comes up precisely when you have a bignum and want to send it out
as a JSON number if you can, a JSON string if you must.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
First known example of political correctness: After Nurhachi had united
all the other Jurchen tribes under the leadership of the Manchus, his
successor Abahai (1592-1643) issued an order that the name Jurchen should
be banned, and from then on, they were all to be called Manchus.
                --S. Robert Ramsey, The Languages of China

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 May 2016 11:10:04 -0400

> as a JSON number if you can, a JSON string if you must.

As I said before, I would add BigNum to the schema as a separate type, just
like binary and DateTime.

Could do the same for real32 and real64 of course.

What I think we are converging on is the idea of a schema that extends the
type system of the data model but not the JSON encoding.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 May 2016 15:58:54 -0400

While we are at it. Thinking of encoding a signed object as an array gives
me an idea.

If we take the approach of
  [ <signature header>, <signed Data>, <signature value>]

And also we are allowing a Bignum to be encoded as either a number or a
string that contains a Base63 encoded value:

MyStruct Class
   one BigNum
   two BigNum

  {"one" : 1, "two" : "AAAAAAAA2" }

The one thing I feel missing from XSD that is incredibly useful is the
ability to put in links between objects as Identifiers. Now I use this to a
vast extent in Goedel, my code synthesis system.

So right now in my JSON structures I have strings that are identifiers for
the enclosing object and I have other strings that are references to those
identifiers. But there is no information in the JSON schema to say which is
which or to impose type constraints, etc.

So lets say we have a definition as follows

MyStruct1 Class
   Inner MyStruct2

MyStruct2 Class
   Identifier ID

I can then encode in my current system as follows:

{ Inner :  { ID : "Fred"} }

But now lets say I want to refer to the same structure in another part of
the same document and I know it is already serialized. This could be
written as:

{ Inner :  "Fred" }

Now lets say that we have a child class defined

MyStruct3 Class <Inherit MyStruct2>

Encoding this requires us to tag the structure so we know what the correct
subtype is. We could use the nested approach I showed earlier. Or we could
use an array:

{ Inner :  ["MyStruct3 ", { ID : "Fred"}] }

This does seem very natural and flexible. And adding identifiers is
sufficient to make the encoding capable of being usefully self describing.