Subject: [Json] JSON Schema Language

From: Ulysse Carion <[email-masked]>
Date: Fri, 3 May 2019 16:59:12 -0700

Hello all,

I have recently published an I-D for what I'm calling JSON Schema
Language, a way to validate JSON data's shape, and generate code
(i.e., types, classes, docs, etc.) from that shape:

https://tools.ietf.org/html/draft-json-schema-language-00

In previous correspondence with this list, Carsten Bormann has noted
that this WG has hesitated to prematurely adopt any given proposal,
preferring to let them develop until their distinct applications
become clear.

It is my belief that JSON Schema Language does have a clear use-case,
one which Tim Bray has articulated clearly here:

https://www.tbray.org/ongoing/When/201x/2018/09/22/JSON-scheming

I believe JSON Schema Language solves precisely what Tim describes,
while eschewing his concerns with the json-schema.org project.

For those inclined toward the concrete, the bottom of this page
contains a live demo of JSON Schema Language using the TypeScript
implementation:

https://json-schema-language.github.io/

The project's GitHub organization also contains the source for a Rust
and TypeScript implementation, as well as a full test suite:

https://github.com/json-schema-language

With Carsten's note in mind, does there nevertheless exist a path for
JSON Schema Language, or something like it, to become an RFC? I
appreciate the desire to avoid premature adoption by this WG. I assume
this partially stems from caution around throwing the weight of
[email-masked] behind any given specification. Perhaps some other path
to specification exists?

With kind regards,
Ulysse Carion

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 4 May 2019 06:47:35 +0200

> Hello all,

Hello Ulysse,

I'm a former user of XML Schema.   I was therefore looking for similarities
like data types and facets but didn't find them.

Example: although 10.0 is a valid JSON Number, in system where you expect
an integer, this should be flagged as a syntax error.

thanx,
Anders

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 4 May 2019 09:58:32 +0200

> an integer, this should be flagged as a syntax error.

10.0 is an integer number.

“Schema Languages” operate at the data model level.  In the JSON data model, there is only one kind of number.  
Of course, the JSON data model is not actually defined in a standard, which is one of the major shortcomings of JSON.

See appendix E of draft-ietf-cbor-cddl-08.txt for the way that is handled in CDDL.

In an alternate JSON universe(*), you would start by applying ISO 6093 (ECMA-63 [1]).
NR1 becomes an integer, NR2 and NR3 become floating point values.  
You then still have to decide what the JSON constructions not covered by ISO 6093 become, e.g., what 1e5 is; in C that would be a float while in Ada that is an integer, IIRC.
If you define that int/float distinction properly, and make it available at the data model level, you can have the “schema language” act on that.

Grüße, Carsten

(*) which is indeed well-supported by implementations outside JavaScript, just not by specifications.
[1] https://www.ecma-international.org/publications/files/ECMA-ST-WITHDRAWN/ECMA-63,%201st%20Edition,%20September%201980.pdf

---

From: Austin Wright <[email-masked]>
Date: Sat, 4 May 2019 02:36:46 -0700

> On May 4, 2019, at 00:58, Carsten Bormann <[email-masked]> wrote:
> 

JSON Schema handles this exactly the same way, defining a data model [1]. The lexical representation is surjective onto the data model: as you point out, 10.0 is the same as 10, which is an integer.

The one case where this might fall apart is if significant digits are important, such that 10.0 is different than 10.00 (i.e., 10.00 is more precise by an order of magnitude). However, I’m not aware of any JSON parsers that keep track of the precision of numbers, even ones that support arbitrary precision. I imagine scientific applications would want to store an explicit precision, since they’re not always powers-of-10 (e.g. {value: 32.0, precision: 0.5}).

[1] http://json-schema.org/latest/json-schema-core.html#rfc.section.4.2.1 <http://json-schema.org/latest/json-schema-core.html#rfc.section.4.2.1> "4.2.1. Instance Data Model"

Austin.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 4 May 2019 11:42:28 +0200

Curious:

On that web page, it also says “For consistency, integer JSON numbers SHOULD NOT be encoded with a fractional part.”

What does that mean?

Grüße, Carsten

> On May 4, 2019, at 11:36, Austin Wright <[email-masked]> wrote:
> 
> 
> 
> [1] http://json-schema.org/latest/json-schema-core.html#rfc.section.4.2.1 "4.2.1. Instance Data Model"
> 
> Austin.

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 4 May 2019 13:44:06 +0200

Test run with a popular JSON Parser nowadays adopted by Microsoft as well:

Newtonsoft.Json.JsonReaderException: 'Input string '7800000.0' is not a valid integer. Path 'counter', line 1, position 39.'

However, in addition to supporting common data types, there should (IMO) also be a way to specify JSON serialization format.

One camp headed by the Open API folks maintain that incompatibility with JavaScript is of uttermost importance and therefore mandate that "long" integers should be serialized as JSON Numbers.

Another, and (if may say it) slightly more practical bunch of people, rather put "long" integers within quotes and thus maintain interoperability with essentially every platform.  To make things yet a bit more fuzzy there exists at least 4 different takes on the latter: Decimal, Hexadecimal, Base64, and Base64Url.

thanx,
Anders

---

From: Austin Wright <[email-masked]>
Date: Sat, 4 May 2019 19:51:50 -0700

> On May 4, 2019, at 02:42, Carsten Bormann <[email-masked]> wrote:
> 
> Curious:
> 

It’s a non-normative suggestion with the aim of enhancing performance: Many programming languages distinguish between integers and IEEE floats by the presence of a decimal point. While JSON makes no such distinction (all numbers are arbitrary precision decimal), some parsers do make that distinction, and it’s slightly easier to determine if an int32_t is an integer than if a double is an integer.

Cheers,

Austin.

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 5 May 2019 07:07:26 +0200

> It’s a non-normative suggestion with the aim of enhancing performance: Many programming languages distinguish between integers and IEEE floats by the presence of a decimal point. While JSON makes n…

in the C# example I provided before this is (by default) not non-normative, since it threw an exception.   Here is a little bit more detail:

class MyObject {
   int Counter;
    .
    .
}

Deserializing JSON into this type (which BTW works as as "schema"), REQUIRES "Counter" data to adhere to normal integer notation, including value span.

A scheme language should align with the actual use and interpretation of JSON.  This involves alternative serialization formats as well.  As an example monetary values are (probably without exceptions) expressed as JSON Strings since floating point is unsuited for decimal arithmetic.

Based on the RFC, one might come to the conclusion that JSON is an inferior information transfer format, but aided by external mapping it actually works extremely well, albeit being slightly verbose :)

Cheers,
Anders

---

From: Nico Williams <[email-masked]>
Date: Sun, 5 May 2019 00:39:22 -0500

> an integer, this should be flagged as a syntax error.

As a maintainer of one popular implementation, this idea that having a
zero fractional part makes a number non-integer annoys me a great deal.

I would insist that any parser that distinguishes real and integer
representations should parse numeric values with a zero fractional part
(and non-negative exponent) as integers.

10.0 is an integer, even if a parser only supports, e.g., IEEE754
representations of numeric values.

Nico
--

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 5 May 2019 07:52:18 +0200

> zero fractional part makes a number non-integer annoys me a great deal.

As I described, this is incompatible with other popular implementations like Jackson for C#.

A schema language should provide the flexibility you want but users should be aware of that certain combinations may not work so well in practice.

A schema may for example specify an BigInteger "huge" and suggest serializing this as JSON Number which will give surprises when processed by ECMAScript's JSON.parse().

Anders

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 5 May 2019 07:56:40 +0200

On May 5, 2019, at 04:51, Austin Wright <[email-masked]> wrote:
> It’s a non-normative suggestion

RFC 2119:

4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
   there may exist valid reasons in particular circumstances when the
   particular behavior is acceptable or even useful, but the full
   implications should be understood and the case carefully weighed
   before implementing any behavior described with this label.

Very much normative.  This is an RFC 2119 interoperability keyword because it creates an expectation in peers that they can mostly rely on this behavior, except for exceptional circumstances (which, by the way, should be spelled out with the SHOULD NOT).

I’m asking because we have a lot of experience with specifications that purport to use a base standard but instead attempt to create their own fork of that base standard by making mandates that meddle with the functioning of that base standard.  Not good.

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 5 May 2019 07:57:49 +0200

> I’m asking because we have a lot of experience with specifications that purport to use a base standard but instead attempt to create their own fork of that base standard by making mandates that med…

+100

---

From: Austin Wright <[email-masked]>
Date: Sat, 4 May 2019 23:16:42 -0700

> On May 4, 2019, at 22:07, Anders Rundgren <[email-masked]> wrote:
> 

The Newtonsoft.Json parser behavior would be incompatible with JSON Schema, according to what you provided. But I can’t imagine it’s much of an issue: If the encoder knows the value will always be an integer, why add a fractional part?

That behavior still seems arbitrary to me, though. Instead of erroring on the first period, all you have to do is error on the first nonzero after the period. Scientific notation is another matter, but presumably it allows 1.9e4 as a valid integer? (I’m not sure, off-hand.)

Probably a better way to approach JSON parsers is to raise an error if it can’t preserve all the information in the source. For example, you can preserve 1e10, 0.5, and even 0.1 (if you permit a small error, such that if you converted the float back to decimal, 0.1 would still be the closest decimal representation). But you can’t preserve 9007199254740993.5 as a double: it’s too many significant figures, and above that value, the precision is below 1. In this case, you would throw.

Parsers that ensure the preservation of the encoded data like this might encourage better use of the JSON types, like for monetary values, even if you are reading into an IEEE floating point.

Still, now that we bring it up, that SHOULD NOT seems suspicious. It’s stating the obvious.

Austin.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 5 May 2019 08:36:41 +0200

On May 5, 2019, at 08:16, Austin Wright <[email-masked]> wrote:
> 
> If the encoder knows the value will always be an integer, why add a fractional part?

(1) The encoder may not be privy to that information (it is a generic encoder).
(2) The encoder may try to fulfill a conflicting set of rules, set up by a different fork of the base standard, e.g.:

If the data comes as a floating point typed value, encode it as NR2 or NR3.
If the data comes as an integer typed value, encode it as NR1.
(This rule set is intended to preserve the distinction in implementation types, which may be important to some other application.
It is common in programming language notation, and since JSON was also derived from a programming language notation [albeit one without this rule set!], it is a common mistake to assume that this rule set can be applied to JSON without cost or damage.)

These conflicting sets of rules are exactly why it's so detrimental to fork a base standard with your own set of rules.

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 5 May 2019 08:49:43 +0200

> The Newtonsoft.Json parser behavior would be incompatible with JSON Schema, according to what you provided. But I can’t imagine it’s much of an issue: If the encoder knows the value will always be…

Right, my guess is that not a single mainstream serializer adds a fractional part to a number that can be exactly represented as an integer regardless if they underlying number type is an integer or not.

> That behavior still seems arbitrary to me, though. Instead of erroring on the first period, all you have to do is error on the first nonzero after the period. Scientific notation is another matter,…

To me the whole idea of having a scheme is to create a *clean* description of an object.  Predecessors like XML Schema supported this in (IMO) a pretty good way.  However, this was straightforward since XML doesn't come with a built-in data model.

JSON was derived from JavaScript (anno 2003 or so) which didn't have an explicit integer type, only an unconstrained Number type.  This obviously creates certain issues when communicating with platforms having a completely different handling of numeric data.  The new proposal doesn't take this is consideration.

> Probably a better way to approach JSON parsers is to raise an error if it can’t preserve all the information in the source. For example, you can preserve 1e10, 0.5, and even 0.1 (if you permit a sm…

A schema should not try to change how parsers work.

> Parsers that ensure the preservation of the encoded data like this might encourage better use of the JSON types, like for monetary values, even if you are reading into an IEEE floating point.

See previous point.

---

From: Nico Williams <[email-masked]>
Date: Sun, 5 May 2019 02:00:24 -0500

> implementations like Jackson for C#.

That sounds like a bug in Jackson.

Nico
--

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 5 May 2019 09:12:05 +0200

> That sounds like a bug in Jackson.

It also flags numbers that doesn't fit into the declared type.

Anders

---

From: Nico Williams <[email-masked]>
Date: Sun, 5 May 2019 11:36:27 -0500

> It also flags numbers that doesn't fit into the declared type.

That's a different story.

An application may not have control over what its encoder does here, but
it will have control over the semantic values.  So if a value is
expected to be an integer between 0 and 9, inclusive, then 10.0 is not
valid.  But 9.0 must be because there can be and almost certainly are
encoders that will output integers with a zero fractional part.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Sun, 5 May 2019 13:05:30 -0400

The Scheme language makes very clear distinctions here.  Integers are a
subset of rational numbers, which are a subset of real numbers, which are a
subset of complex numbers, and there are predicates for detecting any of
them (normally `complex?` is the same as `number?`, but if you want to add
quaternions you can).

Orthogonal to this, there are exact and inexact numbers: a numeric literal
can be explicitly marked as either one, but the default is that a number
with a decimal point or exponent is inexact, and without either it is
exact.  Mixed exactness operations produce inexact results unless the
implementation can prove that the inexactness can't matter: the product of
5.0 (inexact) and 0 (exact) can be either exact or inexact.

---

From: Nico Williams <[email-masked]>
Date: Sun, 5 May 2019 16:11:14 -0500

> them (normally `complex?` is the same as `number?`, but if you want to add

We're not talking about Scheme.  We're talking about JSON.

JSON allows encoders to output 10.0, therefore parsers should handle it.

If a schema indicates that some value should be an integer, and the
parser sees 10.0, then the parser should accept that as the integer
value 10.

Nico
--

---

From: Anders Rundgren <[email-masked]>
Date: Mon, 6 May 2019 08:33:06 +0200

On 2019-05-05 19:05, John Cowan wrote:
> The Scheme language makes very clear distinctions here.  Integers are a subset of rational numbers, which are a subset of real numbers, which are a subset of complex numbers, and there are predicates for detecting any of them (normally `complex?` is the same as `number?`, but if you want to add quaternions you can).
> 
> Orthogonal to this, there are exact and inexact numbers: a numeric literal can be explicitly marked as either one, but the default is that a number with a decimal point or exponent is inexact, and without either it is exact.  Mixed exactness operations produce inexact results unless the implementation can prove that the inexactness can't matter: the product of 5.0 (inexact) and 0 (exact) can be either exact or inexact.
>

That's the theory.  In practice, using the most widely deployed JSON parser there is (ECMAScript's JSON.parse()), an exact JSON Number like 555555555555555555555555555555 is truncated to 5.555555555555555e+29.  To cope with this, lots of JSON users (as well as IETF standards like RFC7515) have since years back rather turned to a JSON subset sometimes referred to as I-JSON (RFC 7493).

Using I-JSON, numbers that cannot be reliably represented by IEEE-754 double precision are serialized as JSON Strings [1]. This permits hassle-free, cross-platform JSON messaging.

https://tools.ietf.org/html/draft-handrews-json-schema-01 and "its partner in crime" the Open API does not take this in consideration.

Thanx,
Anders

1] There's actually quite a bunch of established String-serialized formats for BigIntegers out there including Decimal, Base64, Base64Url, and Hexadecimal.

---

From: Austin Wright <[email-masked]>
Date: Sun, 5 May 2019 23:56:56 -0700

> On May 4, 2019, at 22:56, Carsten Bormann <[email-masked]> wrote:
> 

You’re right. I should add, it’s not _supposed_ to be normative: it’s not our intention to modify how JSON works or impose restrictions on encoders. I don’t think we can do that even if we wanted to.

Austin.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 May 2019 09:00:33 +0200

On May 6, 2019, at 08:33, Anders Rundgren <[email-masked]> wrote:
> 
> IETF standards like RFC7515) have since years back rather turned to a JSON subset sometimes referred to as I-JSON (RFC 7493).

I cannot find a reference to RFC 7493 in RFC 7515 (part of JOSE).
JOSE does not care, because it signs representations (byte strings), not data model level information.

(JOSE does need ways to represent cryptographic information that may contain numbers that might exceed implementation limits of JSON decoders.  So it represents them as text strings containing BASE64URL(*) encoded byte strings representing big-endian unsigned integers.  That has nothing to do with I-JSON, or actually only with its [incomplete(**)] suggestion to use base64url encoding for byte strings.  Section 2.2 of RFC 7493 does NOT say how to “encode [numbers] in JSON string values”.)

Grüße, Carsten

(*) Where BASE64URL is defined as base64url without padding.
(**) RFC 7493 fails to specify encoding without padding, which, when read at face value, would mean WITH padding according to Section 5 of RFC 4648.
Of course, we know that this isn’t what is meant.

---

From: Anders Rundgren <[email-masked]>
Date: Mon, 6 May 2019 09:17:04 +0200

All I'm saying is that

    https://tools.ietf.org/html/draft-handrews-json-schema-01

doesn't support alternative number serialization schemes like featured in:

    https://tools.ietf.org/html/rfc7518#section-6.3.1

It also doesn't deal with common numeric data types like ubyte, int32, uint64, float, double, BigInt etc. etc.

XML Schema shows that there is no problem having such definitions in a schema.

I leave it at there: https://github.com/json-schema-org/json-schema-spec/issues/736

Anders

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 May 2019 09:22:13 +0200

On May 6, 2019, at 08:56, Austin Wright <[email-masked]> wrote:
> 
> You’re right. I should add, it’s not _supposed_ to be normative: it’s not our intention to modify how JSON works or impose restrictions on encoders. 

Good.  So consider my comment a comment on the specific language that is on that website.

> I don’t think we can do that even if we wanted to.

Well, there are several definitions of “can”.

By creating the expectation that 10.0 will be encoded as 10 in the vicinity of your schema data objects, you “are”, even if you “should not be able to”.  If there is an interoperability problem between an encoder that encodes as 10.0 and a decoder that only can handle 10, the implementer of the latter might point to your website and say “works as intended, because you are not supposed to”.  There is your fork.

Grüße, Carsten

---

From: Austin Wright <[email-masked]>
Date: Mon, 6 May 2019 00:22:45 -0700

> On May 4, 2019, at 23:49, Anders Rundgren <[email-masked]> wrote:
> 

Can you elaborate on this? My point is primarily concerned with JSON parsers, not JSON the media type.

Suppose a string were too large to store in memory, most ECMAScript engines would bail with an uncatchable error. The suggestion is to implement this more robustly, like throw an error if a string would be too long, or if a number has too many significant figures; instead of either crashing or silently approximating the number.

But also, why not build a parser that uses a schema to influence how values are parsed? Suppose you have a dynamically typed language, and want to parse some value, e.g. sometimes as an int, sometimes as a bignum, depending on the property. You might provide a JSON Schema to specify this behavior.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 May 2019 09:32:13 +0200

On May 6, 2019, at 09:22, Austin Wright <[email-masked]> wrote:
> 
> But also, why not build a parser that uses a schema to influence how values are parsed?

That is exactly what most of the members of this WG are afraid of when the talk comes to “schemas”.

XML (and later JSON) learned one thing from the problems of SGML: schema-oblivious decoders are more robust than schema-depending ones.
(They are, in the first place, and protocol evolution only amplifies this.)

> Suppose you have a dynamically typed language, and want to parse some value, e.g. sometimes as an int, sometimes as a bignum, depending on the property. You might provide a JSON Schema to specify this behavior.

There goes the generic decoder.

(There is nothing wrong with using a “schema” to specify upconversion from the JSON data model to your application data model.  What I’m objecting to is getting rid of JSON and replacing it with a gazillion dialects that happen to be syntactically compatible with JSON but the semantics of which are controlled by a “schema” mechanism.)

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Mon, 6 May 2019 10:55:56 +0200

On 2019-05-06 09:22, Austin Wright wrote:
<snip>
> Suppose a string were too large to store in memory, most ECMAScript engines would bail with an uncatchable error. The suggestion is to implement this more robustly, like throw an error if a string…

I don't understand this.  A JSON schema is a description of a JSON document.  If the description is incompatible with implementations that are supposed to parse compliant documents all bets are off.

> But also, why not build a parser that uses a schema to influence how values are parsed? Suppose you have a dynamically typed language, and want to parse some value, e.g. sometimes as an int, someti…

This sounds like an edge-case but it could be supported by an "any" type (although don't particularly see how this could work in a cross-platform scenario).

I would rather consider looking into more down-to-earth requirements such as supporting common numeric types and dealing with alternative already widely deployed serialization options for those.

https://github.com/json-schema-org/json-schema-spec/issues/736

Anders

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 6 May 2019 10:09:58 -0400

Lets back up here a moment.

JSON is actually more than one thing. It is:

* A data model
* A lexical token specification
* A serialization

For the Mesh I take the JSON data model and extend it to distinguish
integers from floats and binary encoded blobs from strings via base64url
encoding. I also extend the lexical token specification so that I do not
have to base64 encode binary data blobs. Which is the chief barrier to use
of JSON for crypto.

I am writing my own serializer/deserializer and my own lexical token
analyzer. But that part is really not well grounded in JSON tooling as it
exists in the wild (yet). The aim of JSON-B/C is to be plug-compatible with
JSON so that it is possible to take an existing encoding and just drop in a
JSON-B handler.

Looking at the JSON tools out there, I see that most of them simply take
the input stream and turn it into a parse tree that supports search path
type accessors. That is not the approach I like but it is what folk working
in scripting languages usually want.

>From an interop point of view, any application that relies on being able to
round trip integers that exceed the IEEE float mantissa capacity is going
to break. That is just a fact of life. Another fact is that I have never
found a use for a float in any protocol specification.

Serializations and encodings are important. But there is no point in making
a fetish of finding the one true encoding. The Mesh uses JSON/JSON-B for
all its native encodings. But the Mesh tools have handling for ASN.1,
RFC821, RFC822, TLS and will soon have to add iCal/vCard because they are
all needed in the wild.

I think a JSON schema is a mistake at this point. We should look at writing
a general schema language that is a good schema language and has the
ability to specify ASN.1 or XML schemas or anything else that might need to
be specified.

The world is focusing on JSON for a reason but I think it is the JSON data
model we are actually focusing on rather than the JSON serialization. And I
don't think that insisting that nobody ever touch the original perfection
of JSON encoding is helpful either. All that is doing is ensuring that
instead of there being one JSON/2.0 there are dozens and more are being
added all the time.

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 May 2019 12:01:57 -0500

On Mon, May 06, 2019 at 10:55:56AM +0200, Anders Rundgren wrote:
> > Suppose a string were too large to store in memory, most ECMAScript
> > engines would bail with an uncatchable error. The suggestion is to
> > implement this more robustly, like throw an error if a string would
> > be too long, or if a number has too many significant figures;
> > instead of either crashing or silently approximating the number.

What's that got to do with schemas?

> I don't understand this.  A JSON schema is a description of a JSON
> document.  If the description is incompatible with implementations
> that are supposed to parse compliant documents all bets are off.

A schema is not a description of a document but a description of a class
of documents.

I suspect that you think the point of a schema is to allow you to have a
parser that parses straight into an internal representation specifically
derived from the schema, but that is not the only possibility!

It should absolutely be possible to first parse a JSON text with a
*generic* JSON parser, *then* check if the parsed thing meets the
requirements of the schema.  This is a very reasonable thing to do,
especially with general-purpose languages like ECMAScript, or jq that
deal with JSON very naturally.  I object strongly to any JSON schema
that makes this approach not workable.

> > But also, why not build a parser that uses a schema to influence how
> > values are parsed? Suppose you have a dynamically typed language,
> > and want to parse some value, e.g. sometimes as an int, sometimes as
> > a bignum, depending on the property. You might provide a JSON Schema
> > to specify this behavior.
> 
> This sounds like an edge-case but it could be supported by an "any"
> type (although don't particularly see how this could work in a
> cross-platform scenario).

Or a CHOICE.

(Here, again, I have to say that ASN.1 already has all these things, so
we should at least be informed by it before we go off and reinvent that
wheel badly.)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 May 2019 14:24:54 -0500

On Mon, May 06, 2019 at 09:32:13AM +0200, Carsten Bormann wrote:
> > values are parsed?

Because some of us have generic parsers with associated DSLs and we
don't want to be left out.

Specifically, in my case, jq https://stedolan.github.io/jq -- something
of an XSLT/XPath for JSON.

It's perfectly reasonable to build a jq application that adheres to a
JSON schema, but it's not reasonable to expect the jq encoder to know to
do encoding in context-specific ways -- the jq encoder is too generic
and divorced from the rest of the jq machinery to possibly be able to do
that.

By pursuing a schema design that requires context-specific parsing or
encoding behaviors, though you could say that the result looks like
JSON, it precludes use of existing JSON tooling such as jq.

Substitute ECMAScript for jq if you like -- the result is the same.

I strenuously object to this sort of schema.

As a maintainer of jq, I wouldn't know what to say to users who then
complain about non-interop with such a schema other than to point to
this thread and blame the people who allowed it to happen.  But the
damage to jq would be real and unavoidable.  To avoid that damage, if I
cannot convince you, then I assure you that I would appeal any finding
of consensus for such a schema design.

I'm sorry, but 10.0 is perfectly acceptable as an integer, and MUST be
accepted as an integer when you expect an integer even if the JSON RFC
says encoders SHOULD NOT include zero fractional parts.

I would suggest that if you have a schema that says "and this is an
integer", the schema MUST accept zero fractional parts, and SHOULD let
you specify what to do if a non-integer real number is parsed for that
field (e.g., truncate the fractional part, floor, round).

> the talk comes to “schemas”.

+1

> XML (and later JSON) learned one thing from the problems of SGML:
> schema-oblivious decoders are more robust than schema-depending ones.

Not only that!

Can you imagine using XSLT/XPath if some XML schema requires different
parsing behavior for different elements?!  It wouldn't be possible to do
that *and* still be able to use XSLT/XPath because XSLT/XPath processors
generally have a built-in generic XML parser.  It would destroy the
utility of XSLT/XPath!

And that proves that an XML schema of that sort... would yield something
that is NOT XML, but merely looks like XML.

The same exact reasoning applies in the case of JSON.  The only
difference being that JSON doesn't [yet] have a standard equivalent to
XSLT/XPath, but since generic parsers and encoders exist, and since
node.jq, JavaScript, ECMAScript, jq, and such exist, the point stands.

> > Suppose you have a dynamically typed language, and want to parse
> > some value, e.g. sometimes as an int, sometimes as a bignum,
> > depending on the property. You might provide a JSON Schema to
> > specify this behavior.
> 
> (There is nothing wrong with using a “schema” to specify upconversion
> from the JSON data model to your application data model.  What I’m
> objecting to is getting rid of JSON and replacing it with a gazillion
> dialects that happen to be syntactically compatible with JSON but the
> semantics of which are controlled by a “schema” mechanism.)

+1.

Nico
--

---

From: Austin Wright <[email-masked]>
Date: Mon, 6 May 2019 14:00:54 -0700

> On May 6, 2019, at 12:24, Nico Williams <[email-masked]> wrote:
> 

Left out of what, exactly? See below.

> field (e.g., truncate the fractional part, floor, round).

I’m not suggesting a variance from the JSON semantics.

My suggestion is an alternative to parsers that lose data when they parse JSON documents, for example, ECMAScript's JSON.parse number parsing. You can, of course, always parse a JSON document according to the generic semantics.

The issue here is how does a program parse a JSON document that with a wider range of values than what the program has room for? Either a string that’s too long, an array with too many items, or a number with too many significant figures?

Normally you would have to write your own parser, or use a tokenizer that preserves the lexical values without casting them to native types. Then you perform your own validation in code, and decide how to convert a lexical JSON number into a native number, depending on context.

Right now, programs usually do this in code. My suggestion is to write these rules as a document. The idea of JSON Schema is you make assertions about the document—“this field must be an RFC3339 date string”—and let the parser or application make additional optimizations (in this case, an RFC3339 string will never be more than 25 characters).

>> semantics of which are controlled by a “schema” mechanism.)

There’s scarcely a completely generic decoder in existence. They should be, but they’ve typically decided it’s easier to make assumptions of one sort or another, especially around numbers.

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 May 2019 16:15:11 -0500

> Left out of what, exactly? See below.

Out of interoperating with other applications using this schema thing.

> > [...]

If you would have your parser reject 10.0 because the schema says to
expect an integer, then you are forking JSON as you are precluding
interoperability with some JSON encoders.  Especially too if you then
also insisted on a fractional part when a schema element expects a real
number so that you'd have the parser reject 10 -- then encoders would be
damned if they do and damned if they don't.

I explained how this would preclude use of jq or any other XSLT/XPath-
alike for JSON from interoperating with applications using such a schema
language.

> My suggestion is an alternative to parsers that lose data when they
> parse JSON documents, for example, ECMAScript's JSON.parse number
> parsing. You can, of course, always parse a JSON document according to
> the generic semantics.

The key word there is "alternative".  You're forking JSON.

> The issue here is how does a program parse a JSON document that with a
> wider range of values than what the program has room for? Either a
> string that’s too long, an array with too many items, or a number with
> too many significant figures?

I explained this earlier.

First, as to fractional parts of numeric values, a) zero fractional
parts MUST be tolerated, b) non-zero fractional parts can be truncated,
rounded, or rejected according to the schema authors' choice.

This is a case where being liberal in what you accept is a good thing.

> Normally you would have to write your own parser, or use a tokenizer

Why not use an off-the-shelf one?

> that preserves the lexical values without casting them to native
> types. Then you perform your own validation in code, and decide how to
> convert a lexical JSON number into a native number, depending on
> context.

That's one way, and it has to be possible, because that's essentially
what you'd do with jq or ECMAScript.

You could also use a streaming parser (which jq also has) to immediately
handle each value as it is parsed.

You could also write a combined parser and validator.

And, of course, you could generate a parser&validator from the schema.

I'm not rejecting any of those choices.

I'm insisting ONLY on the first choice (first parse, then validate)
needing to remain valid and workable.  If you make this choice no longer
feasible, then you've forked JSON.  Please don't.

Nico
--

---

From: Austin Wright <[email-masked]>
Date: Mon, 6 May 2019 14:26:43 -0700

> On May 6, 2019, at 14:15, Nico Williams <[email-masked]> wrote:
> 

You’ll need to provide a specific example, I think there’s a misunderstanding.

> damned if they do and damned if they don’t.

> alike for JSON from interoperating with applications using such a schema

I never suggested this. While JSON doesn’t specify what an “integer” is (i.e. we could define it to be any subset of number that we want), JSON Schema tries to be liberal and specifies that excessive precision still qualifies as an integer. (I’m the one who added that language.)

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 May 2019 16:32:26 -0500

> You’ll need to provide a specific example, I think there’s a misunderstanding.

I did.  Any application that uses an off-the-shelf generic parser and
encoder and does schema validation in a different step.

> I never suggested this. While JSON doesn’t specify what an “integer”
> is (i.e. we could define it to be any subset of number that we want),
> JSON Schema tries to be liberal and specifies that excessive precision
> still qualifies as an integer. (I’m the one who added that language.)

Then accept 10.0 as an integer.

Nico
--

---

From: Austin Wright <[email-masked]>
Date: Mon, 6 May 2019 14:40:08 -0700

> On May 6, 2019, at 14:32, Nico Williams <[email-masked]> wrote:
> 

The only JSON parsers/JSON Schema validators I maintain are in ECMAScript, and both validate `10.0` as an integer. [1] [2]

If you’re talking about the Newtonsoft JSON parser, or anything else, you should ask them directly. By all means, feel free to copy me on that email or GitHub issue.

Thanks,

Austin.

[1] https://github.com/awwright/jsonschemaparse <https://github.com/awwright/jsonschemaparse>
[2] https://github.com/tdegrunt/jsonschema <https://github.com/tdegrunt/jsonschema>

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 May 2019 16:46:08 -0500

On Mon, May 06, 2019 at 02:40:08PM -0700, Austin Wright wrote:
> The only JSON parsers/JSON Schema validators I maintain are in
> ECMAScript, and both validate `10.0` as an integer. [1] [2]
> 
> If you’re talking about the Newtonsoft JSON parser, or anything else,
> you should ask them directly. By all means, feel free to copy me on
> that email or GitHub issue.

Oy, I think I confused who wanted 10.0 not to parse as an integer.
Sorry about that.

---

From: Tim Bray <[email-masked]>
Date: Mon, 6 May 2019 15:38:19 -0700

Hi all.  I haven't got around to reading the draft (but will soon).  Prior
to that, a few notes:

1. I'm pretty sure that we need something better than what we have in the
area of JSON schemas.  At least, I'm 100% sure that my job at Amazon Web
Services would be easier, and our customer experiences would be more
pleasant, if we had something.

2. One thing schemas are useful for is to syntax-check JSON texts that
claim to conform to some language specification or another. Obviously no
schema can ever completely satisfy this requirement - there are always
things in specifications which are semantic and not addressable by schemas
- but they can still be super useful.

3. Another thing they are useful for is for providing help to developers
working in strongly typed programming languages. With a well-built schema
it is reasonably straightforward to auto-generate nice idiomatic class
declarations in modern programming languages, and also to build
serializers/deserializers that will move data back and forth between JSON
blobs and programming-language constructs, or fail in a clean deterministic
way if the JSON fails to match the schema.

I mostly fail to understand the debate about jq and integers and so on.
Clearly, the following is a valid JSON text and will be parsed successfully
by any JSON parser.

{
  "foo": 3.0
}

I imagine that most schema-driven software would first deserialize it into
a tree, probably something like Jackson ObjectMapper's JsonNode, and then
apply schema constructs to the tree.   I would hope that a sane schema
would accept this whether a top-level "foo" was required to be an integer
or double or most other flavors of number, and reject it if "foo" was
required to be a string or boolean.

Put another way, no JSON schema spec can change the definition of what JSON
is, or make the built-in type system anything but what it is.

---

From: Anders Rundgren <[email-masked]>
Date: Tue, 7 May 2019 06:44:49 +0200

On 2019-05-07 00:38, Tim Bray wrote:
> Hi all.  I haven't got around to reading the draft (but will soon). Prior to that, a few notes:
>
> 1. I'm pretty sure that we need something better than what we have in the area of JSON schemas.  At least, I'm 100% sure that my job at Amazon Web Services would be easier, and our customer experiences would be more pleasant, if we had something.
+1
>
> 2. One thing schemas are useful for is to syntax-check JSON texts that claim to conform to some language specification or another. Obviously no schema can ever completely satisfy this requirement - there are always things in specifications which are semantic and not addressable by schemas - but they can still be super useful.
+1
>
> 3. Another thing they are useful for is for providing help to developers working in strongly typed programming languages. With a well-built schema it is reasonably straightforward to auto-generate nice idiomatic class declarations in modern programming languages, and also to build serializers/deserializers that will move data back and forth between JSON blobs and programming-language constructs, or fail in a clean deterministic way if the JSON fails to match the schema.
+1
>
> I mostly fail to understand the debate about jq and integers and so on. Clearly, the following is a valid JSON text and will be parsed successfully by any JSON parser.
>
> {
>   "foo": 3.0
> }
>
> I imagine that most schema-driven software would first deserialize it into a tree, probably something like Jackson ObjectMapper's JsonNode, and then apply schema constructs to the tree.   I would hope that a sane schema would accept this whether a top-level "foo" was required to be an integer or double or most other flavors of number, and reject it if "foo" was required to be a string or boolean.

Here we have a little problem.  There are already quite popular parsers out there flagging 3.0 as an invalid integer when explicitly parsing for an integer[*].  Personally, I consider this as right thing to do in a schema as well.  Letting one or two rotten eggs (=JSON serializers that output Numbers that also represent exact integers with fractions) set the bar for the other 99.9% doesn't seem right to me.

>
> Put another way, no JSON schema spec can change the definition of what JSON is, or make the built-in type system anything but what it is.

I don't think anybody is actually proposing that; only mapping things that doesn't have a specific representation in JSON like integers.

Anders

*] When for example deserializing JSON into a class definition ("unmarshalling") containing an integer property.

>
>
>
>
>
> On Fri, May 3, 2019 at 4:59 PM Ulysse Carion <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hello all,
>
>     while eschewing his concerns with the json-schema.org <http://json-schema.org> project.
>
>     [email-masked] <mailto:[email-masked]> behind any given specification. Perhaps some other path
>     to specification exists?
>
>     With kind regards,
>     Ulysse Carion
>
>     [email-masked] <mailto:[email-masked]>

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 May 2019 10:42:02 -0500

On Tue, May 07, 2019 at 06:44:49AM +0200, Anders Rundgren wrote:
> > I mostly fail to understand the debate about jq and integers and so
> > on. Clearly, the following is a valid JSON text and will be parsed
> > successfully by any JSON parser.
> > 
> > {
> >   "foo": 3.0
> > }
> > 
> > I imagine that most schema-driven software would first deserialize
> > it into a tree, probably something like Jackson ObjectMapper's
> > JsonNode, and then apply schema constructs to the tree.   I would
> > hope that a sane schema would accept this whether a top-level "foo"
> > was required to be an integer or double or most other flavors of
> > number, and reject it if "foo" was required to be a string or
> > boolean.
> 
> Here we have a little problem.  There are already quite popular
> parsers out there flagging 3.0 as an invalid integer when explicitly
> parsing for an integer[*].  Personally, I consider this as right thing
> to do in a schema as well.  Letting one or two rotten eggs (=JSON
> serializers that output Numbers that also represent exact integers
> with fractions) set the bar for the other 99.9% doesn't seem right to
> me.

Those parsers are broken.

> > Put another way, no JSON schema spec can change the definition of
> > what JSON is, or make the built-in type system anything but what it
> > is.
> 
> I don't think anybody is actually proposing that; only mapping things
> that doesn't have a specific representation in JSON like integers.

Integers do have a specific representation in JSON, and anything that
precludes generic encoders that might emit 10.0 where a peer parser will
reject that as not-an-integer *is* a change to JSON.

Nico
--

---

From: Peter Patel-Schneider <[email-masked]>
Date: Tue, 7 May 2019 09:05:33 -0700

I was looking through the JSON specification and nowhere could I find a
reference to integers.  That indicates to me that any conversion from JSON
numbers to integers has to be done outside of JSON.  Conformant
implementations of JSON are thus free to do conversion from JSON numbers to
integers in whatever way they want.  There is a cost in doing the
conversion differently than common JSON implementations do, but that is a
different discussion.

I could, if I wanted, use JSON numbers to represent inexact numbers where
the precision of is +-1 in the last digit, so 4 is different from 4.0 is
different from 4.000000.  I have to be very careful that the only JSON
texts I allow are ones that are created with this understanding of numbers,
but I haven't done anything that violates the JSON specification.

Peter F. Patel-Schneider

---

From: Pete Cordell <[email-masked]>
Date: Tue, 7 May 2019 17:15:18 +0100

On 06/05/2019 23:38, Tim Bray wrote:
> 1. I'm pretty sure that we need something better than what we have in 
> the area of JSON schemas.  At least, I'm 100% sure that my job at Amazon 
> Web Services would be easier, and our customer experiences would be more 
> things in specifications which are semantic and not addressable by 
> schemas - but they can still be super useful.
> 
> working in strongly typed programming languages. With a well-built 
> schema it is reasonably straightforward to auto-generate nice idiomatic 
> class declarations in modern programming languages, and also to build 
> serializers/deserializers that will move data back and forth between 
> JSON blobs and programming-language constructs, or fail in a clean 
> deterministic way if the JSON fails to match the schema.

For me:

4. Person-to-person communication is an important use-case for schemas 
as a way of describing valid formats, either in design documents or 
during brainstorming sessions on whiteboards or personal notes on the 
back of envelopes.

When you do that you want to be discussing JSON rather than also working 
out a convention for defining JSON. Hence the need for something already 
defined that is readily and widely understood.

The human need puts me off a JSON format for this.  It is too long 
winded and, like W3C XML Schema, unnecessarily difficult for humans to 
use. It's why XML Relax NG Compact Syntax was developed.  And why I 
prefer JSON Content Rules JSON-like (but not JSON) syntax.

You can get an idea of a comparison of the two types of specification at:

http://json-content-rules.org/drafts/draft-newton-json-content-rules-10-pjc.html#rfc.section.2.3

I my mind, JCR is a clear winner in this situation.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 7 May 2019 18:41:04 +0200

On May 7, 2019, at 18:15, Pete Cordell <[email-masked]> wrote:
> long winded

Here are a few examples from the JSL draft with the CDDL equivalents (behind the “===“; note that CDDL supports both text and byte strings, so uses “text” for text strings):

   { “type”: “number” }     
   ===
   number

   {                        
     “elements”: {
       “type”: “number”
     }
   }
   ===
   [* number]

   ([] signifies an array, * signifies zero or more)

   {                           
     "properties": {
       "a": { "type": "string" },
       "b": { "type": "string" }
     },
     "optionalProperties": {
       "c": { "type": "string" },
       "d": { "type": "string" }
     }
   }
   ===
   { a: text, b: text, ? c: text, ? d: text }

   ({} signifies an object, ? signifies zero or one)

   {
     "values": {
       "type": "number"
     }
   }
   ===
   { * text => number }

   (=> can be used to separate type names for key and value)

   {
     "discriminator": {
       "tag": "version",
       "mapping": {
         "v1": {
           "properties": {
             "a": { "type": "number" }
           }
         },
         "v2": {
           "properties": {
             "a": { "type": "string" }
           }
         }
       }
     }
   }
   ===
   { tag: “v1”, a: number } / { tag: “v2”, a: text }

   (/ signifies a choice)

Noise in specifications hurts:
https://www.iab.org/wp-content/IAB-uploads/2016/03/Noise-in-specifications-hurts.pdf

Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 7 May 2019 12:46:03 -0400

> pleasant, if we had something.

I agree. And I think most others here agree. The problem is that when we go
down the 'schema' road, we tend to end up with schema languages that become
baroque and more hassle than they are worth.

> way if the JSON fails to match the schema.

That is one reason I need a schema language. The other is that I want to
document the protocol design so that I can generate the reference manual
and the reference code from the same source.

This is NOT something I have seen in any proposal other than mine to date.
But it is the one I think most relevant to IETF purposes. For example, this
is a fragment of a schema I am working with right now:

Section 1 "Shared Classes"
Description
|The following classes are used as common elements in
|Mesh profile specifications.a

Structure HostEntry
Description
|Describes a current or pending connection to a Mesh account
String ID
Description
|Unique object instance identifier.

The description sections flow straight into my Internet Drafts.

This is not how I would do the same thing now. That would be more like:

Shared: Section 1 "Shared Classes"
// The following classes are used as common elements in
// Mesh profile specifications.

HostEntry: Class
// Describes a current or pending connection to a Mesh account
ID: String
//Unique object instance identifier.

In fact, I would probably get rid of the colons as they aren't needed by
the parser and are therefore clutter.

The point is that documentation and code should be integrated.

> Clearly, the following is a valid JSON text and will be parsed successfully

It will be parsed successfully but the problem that comes up are that a
lexical analyzer may legitimately interpret 1.0 as a float and 1 as an
integer and so when the parse tree is traversed end up rejecting the data
as invalid. But that is only an issue if your schema validator doesn't know
how the parse tree handles numbers.

> a tree,

That isn't what I do. I parse directly to the memory data structures. I
don't need the tree structure.

That said, I am thinking of rewriting the code so that it does both at the
same time

probably something like Jackson ObjectMapper's JsonNode, and then apply
> schema constructs to the tree.   I would hope that a sane schema would
> accept this whether a top-level "foo" was required to be an integer or
> double or most other flavors of number, and reject it if "foo" was required
> to be a string or boolean.
>
> Put another way, no JSON schema spec can change the definition of what
> JSON is, or make the built-in type system anything but what it is.
>

But do we actually want a JSON schema spec or a general data schema spec
that supports JSON?

JSON meets pretty much all the requirements I have for writing protocols
except for representing binary data. The application that JSON does not
currently support is data representation because as things stand, floats do
not round trip.

One option would be to write a profile of JSON which ensures floats round
trip. But I can't see that being adopted consistently enough to get
traction. A better solution would be to introduce new float encodings which
do round trip. It wouldn't be JSON but is could use 95% of JSON, be 100%
backwards compatible reading old data and not corrupt data when writing the
new.

Point is that any spec that attempts to solve interop issues by declaring
it can never ever change will fail. Instead of there being one extension,
there will be many and we will end up in the Markdown situation where it
has taken ten years for the market to converge on a common set of tags. And
that mainly because GitHub chose one particular flavor and so
iPython/Jypityr did, etc.

---

From: Austin Wright <[email-masked]>
Date: Tue, 7 May 2019 14:13:33 -0700

> On May 7, 2019, at 08:42, Nico Williams <[email-masked]> wrote:
> 

I would put it slightly differently: While JSON doesn’t define any concept of an “integer”, applications that encode integers (e.g. uint32_t or similar) might conceivably emit it with the "frac” production (using a trailing decimal and zero); this is not ambiguous, and in the interest of being liberal in what you accept, parsers shouldn’t produce any error.

---

From: Erik Wilde <[email-masked]>
Date: Tue, 7 May 2019 14:29:13 -0700

> I would put it slightly differently: While JSON doesn’t define any concept of an “integer”, applications that encode integers (e.g. uint32_t or similar) might conceivably emit it with the "frac” pr…

at the risk of referring to XSD too much, but i think they got it right:

- constraining facets for constraining the value space.
- lexical facets for constraining the representation of values.

maybe JSON schema languages do not want to copy that exact model, but 
the general design approach might be helpful to look at.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked] |
            | http://dret.net/netdret    |
            | http://twitter.com/dret    |

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 7 May 2019 17:29:59 -0400

On Tue, May 7, 2019 at 5:14 PM Austin Wright <[email-masked]> wrote:

>
> I would put it slightly differently: While JSON doesn’t define any concept
> of an “integer”, applications that encode integers (e.g. uint32_t or
> similar) might conceivably emit it with the "frac” production (using a
> trailing decimal and zero); this is not ambiguous, and in the interest of
> being liberal in what you accept, parsers shouldn’t produce any error.
>

The problem that arise are:

1) The lexical analyzer may distinguish integers and floats and this may
cause a schema driven parser to reject legitimate number because it is
encoded as a double. (e.g. 1.0)

2) There may be precision loss because the number is presented in
exponential form (1.0E20)

3) There may be precision loss because the integer is larger than the
floating point mantissa can hold.

We should probably stop talking about 'float' values because I don't know
of any JSON tech likely to use them at this point. Pretty much every
programming language out there today will cause all calculations to be
performed in Intel 80 bit floating point until the data has to be stored in
a program data value when the mantissa will be truncated at 52 bits and
stored in a double.

Fact is that a 64 bit processor can do 64 bit mantissa ops more easily than
52 bit. But we are stuck in the choices of the 1980s when memory was scarce
and nobody had 64 bit CPUs yet.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 7 May 2019 17:48:46 -0400

On Tue, May 7, 2019 at 5:29 PM Erik Wilde <[email-masked]> wrote:

> > I would put it slightly differently: While JSON doesn’t define any
> concept of an “integer”, applications that encode integers (e.g. uint32_t
> or similar) might conceivably emit it with the "frac” production (using a

I disagree on the constraining facets part. The only values I have found
useful to put in a schema are 0 or 1 minimum occurrences and 1 or infinity
maximum occurrences.

I don't have these constraints in any programming language I have used
since I did formal methods 25 years ago. The only ones that did were Z and
VDM.

---

From: Austin Wright <[email-masked]>
Date: Tue, 7 May 2019 14:50:38 -0700

> On May 7, 2019, at 09:15, Pete Cordell <[email-masked]> wrote:
> 
> 4. Person-to-person communication is an important use-case for schemas as a way of describing valid formats, either in design documents or during brainstorming sessions on whiteboards or personal notes on the back of envelopes.
> 
> When you do that you want to be discussing JSON rather than also working out a convention for defining JSON. Hence the need for something already defined that is readily and widely understood.
> 
> The human need puts me off a JSON format for this.  It is too long winded and, like W3C XML Schema, unnecessarily difficult for humans to use. It's why XML Relax NG Compact Syntax was developed.  And why I prefer JSON Content Rules JSON-like (but not JSON) syntax.
> 

Yes, JSON is really not well suited for markup, I consider it as bulk data language, closer to ASN.1 than XML.

The upside of JSON Schema is that it’s primarily defined as a vocabulary; and although it defines a JSON-based media type to encode schemas as, you can invent any media type you want that’s capable of encoding the same assertions.

I still have to spend more time with JCR, but it appears to me JSON Schema implementations could import JCR documents as a JSON Schema with few problems, and it could very well be a superior method of authoring a JSON Schema.

Would you consider elaborating on this possibility in JCR?

Cheers,

Austin.

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 May 2019 16:54:31 -0500

> useful to put in a schema are 0 or 1 minimum occurrences and 1 or infinity

+1e3

I've seen the option to have arbitrary Ns of relations in a database,
and the only min/max value pairs ever used are {0,1}, {1,1}, {0,max},
and {1,max}, where max is usually just a very large number (2^31 - 1 in
the case I have in mind).

> since I did formal methods 25 years ago. The only ones that did were Z and

+1e3

Nico
--

---

From: Erik Wilde <[email-masked]>
Date: Tue, 7 May 2019 14:56:10 -0700

On 2019-05-07 14:48, Phillip Hallam-Baker wrote:
> useful to put in a schema are 0 or 1 minimum occurrences and 1 or 
> infinity maximum occurrences.

there may be a misunderstanding here. constraining facets are limiting 
the value space of datatypes, for example by defining minimum and 
maximum values.

https://www.w3.org/TR/xmlschema-2/#rf-facets

cheers,

dret.

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 May 2019 17:17:27 -0500

On Tue, May 07, 2019 at 02:56:10PM -0700, Erik Wilde wrote:
> there may be a misunderstanding here. constraining facets are limiting the
> value space of datatypes, for example by defining minimum and maximum
> values.

As long as they don't constrain the encoding when multiple
representations are available.

---

From: Ulysse Carion <[email-masked]>
Date: Tue, 7 May 2019 16:57:01 -0700

Hi all,

I'd like to offer a summary of what has been stated thus far, so as to
make a bit more sense of the conversation.

1. Considerable ink has been spilled on the question of integers in a
schema language. The main question has been how such an integral
constraint should relate to serialization.

2. Some have raised questions around the terseness of JSON Schema and
JSON Schema Language. Proposed alternatives are JCR and CDDL, by their
respective editors.

On the first question, I would agree with Messrs. Bormann and Bray,
who both suggest that a generic JSON parser runs first, followed by a
second pass over the data to convert the parsed data in the context of
a schema. On such a view, "3" and "3.0" are typically
indistinguishable by the time the schema is taken into account.
Bormann aptly notes that the alternative runs the risk of de facto
forking JSON itself. I don't think there is much more to be said on
the matter.

The second question is perhaps more interesting, although it has
provoked less discussion. I went with JSON as the representation
because it makes building tooling atop of JSON Schema Language much
easier, since it doesn't require that the implementor write a parser.
Such ease of extension is valuable: JSON is widespread largely because
of how easy it is to build atop of, and we should not readily
sacrifice this. JSON as the representation also unlocks syntax
highlighting, formatting, and other such niceties for free.
Furthermore, and most importantly, using JSON as a representation does
not preclude later creating a RELAX NG-style compact representation
later on.

I would also like to echo Bray's point about the real-world utility of
such a system. My day-job involves APIs, event-sourcing, and analytics
messages. In all three, JSON is the lingua franca, and in all three
considerable tedium is involved in writing and maintaining validators
and language-idiomatic containers for JSON messages. No
language-portable solution to this problem exists today, short of
moving away from JSON toward something like Protobufs or CBOR.

I'd like to return to the initial question of this thread, this time rephrased:

Are we in agreement that the question of describing JSON, and from
that description generating validators and idiomatic
classes/types/etc, is a problem worth solving?

Are we in agreement that something along the lines of the proposed I-D
may be a way we address this problem?

Best,
Ulysse

---

From: John Cowan <[email-masked]>
Date: Tue, 7 May 2019 22:15:27 -0400

On Tue, May 7, 2019 at 5:30 PM Phillip Hallam-Baker <[email-masked]>
wrote:

> floating point mantissa can hold.

JSON encodes absolutely no information about whether its numbers are to be
interpreted as exact or inexact.  1.23 may be an exact number equivalent to
123 divided by 100, and 1.0E20 may by the same token be the exact integer
100,000,000,000,000,000,000.   Per contra, 12 could be an inexact number
that happens to have been rounded to the nearest integer.  We are so used
to the idea that dots and Es *mean* inexact, and their absence *means*
exact, and that all our numbers are *inherently* bounded in range and (if
non-integral) precision, that we think these things must be requirements of
our data notation.  But they aren't.

I agree.. And I think most others here agree. The problem is that when we
> go down the 'schema' road, we tend to end up with schema languages that
> become baroque and more hassle than they are worth.

The conclusion I drew from the XML Schema debacle is that schema languages
get wrong-footed when they try to do two things at once: validating purely
syntactically (as Relax NG does) and validating for data binding,
especially to rigid static languages.

Pretty much every programming language out there today will cause all
> calculations to be performed in Intel 80 bit floating point until the data
> has to be stored in a program data value when the mantissa will be
> truncated at 52 bits and stored in a double.

Practically every programming language can handle arbitrary size, aribtrary
precision exact numbers, either as built-ins or through a library.

John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
De plichten van een docent zijn divers, die van het gehoor ook.
      --Edsger Dijkstra

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 8 May 2019 06:40:14 +0200

I believe I have stated my opinion but here it is again:

The idea is not changing JSON in any way but to add *strictness* to its interpretation.

This has ZERO impact on JSON parsers but could in some cases affect JSON serialization.

There are no problems defining a set of numeric types having a range and syntax that conforms to what is the norm for most programming languages since all of that fits nicely into JSON.

However, assuming that any JSON parser also could be used for *implementing* JSL (including verifying arbitrary input data) will set the bar way too low.

Regarding the integer discussion I believe Open API/Swagger which has quite an active and big user community already flags 3.0 as an invalid integer.  A bug report on Open API (or on Jackson) would most likely return "works as intended".

There's more, but I leave it there.

Anders

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 May 2019 06:43:52 +0200

On May 8, 2019, at 01:57, Ulysse Carion <[email-masked]> wrote:
> 
> Are we in agreement that the question of describing JSON,

Certainly.
Describing protocols that are based on JSON benefits from a notation for describing the JSON data items to be interchanged.
So like we have ABNF for protocols based on text (there are only very few people who would argue that ABNF is not useful — yes, I have met them, but I don’t agree with them), we have CDDL for protocols based on the JSON data model (as realized in CBOR or JSON).  See

https://datatracker.ietf.org/doc/draft-ietf-cbor-cddl/referencedby/
and
https://datatracker.ietf.org/doc/draft-greevenbosch-appsawg-cbor-cddl/referencedby/

for some 30 IETF documents referencing the approved CDDL specification or a previous version of it (CDDL has been essentially stable since March 2015 and was finally approved in March 2019); CDDL is also being used in other SDOs that define JSON-/CBOR-encoded protocols.

> and from

Well, we probably have different ideas on the specific contexts in which “validation” and code generators become useful.  (With ABNF, there is a whole spectrum of “specification only” usage up to parser generators that can read ABNF.)  But having a notation for the structures that the protocol provides is a first step that is needed whenever validators and code generators are useful, which they sometimes are.  Other useful tools that benefit from a description notation include example instance generators and pretty printers.

Comparing JSON-based tools with XML-based ones, CDDL is probably closest to Relax-NG (ISO/IEC 19757-2).  There are other approaches for describing data, e.g., Schematron (ISO/IEC 19757-3) for XML; I would expect we will grow those approaches in conjunction with  what we already have.

Grüße, Carsten

---

From: Pete Cordell <[email-masked]>
Date: Wed, 8 May 2019 14:38:06 +0100

> Would you consider elaborating on this possibility in JCR?

At a guess, the best approach might be to have JCR to JSON Schema (and 
vice versa) translation tools, possibly as a library that can be 
included into other programs rather than as standalone tools in their 
own right.

There will likely be some features that can't be translated. Identifying 
those features could be a useful exercise in converging the technologies.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Pete Cordell <[email-masked]>
Date: Wed, 8 May 2019 14:48:08 +0100

On 08/05/2019 00:57, Ulysse Carion wrote:
> ...  I went with JSON as the representation

This makes sense if the only things reading and writing JSON Schema are 
machines. Once you have humans involved, my experience (e.g. XML Schema) 
is that it is a bad choice.

Writing XML Schema becomes more bearable if you have XML Schema aware 
editors.  Being XML aware alone isn't sufficient.

So in addition to having a JSON aware editor, you would also need a JSON 
Schema aware editor that included auto-suggestion and more advanced 
editing features beyond syntax highlighting etc like that.

IMO, in that case you would be better off putting effort into a JSON 
specification format that didn't need as much help to edit it in the 
first place.

Pete.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 May 2019 17:46:19 +0200

On May 8, 2019, at 15:48, Pete Cordell <[email-masked]> wrote:
> 
> Writing XML Schema becomes more bearable if you have XML Schema aware editors.  Being XML aware alone isn't sufficient.

Writing is not the main activity.  Reading is.  That is nearly unbearable without tools.

If I have to work on a W3C Schema(*), I convert it to Relax-NG compact first.  True, I’ll lose some information, but as a net result I'll digest much more information out of the W3C Schema document than if I had directly looked at that.

At the time when I was closer to the XML community, the hallway consensus was that W3C Schema was *designed* to sell more tools.

(Hence my short note yesterday giving the CDDL translations for JSL; that would probably be the way I would be reading — or writing — JSL.)

Grüße, Carsten

(*) The data description spec that W3C called XML Schema

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 May 2019 10:51:11 -0500

On Tue, May 07, 2019 at 02:13:33PM -0700, Austin Wright wrote:
> concept of an “integer”, applications that encode integers (e.g.
> uint32_t or similar) might conceivably emit it with the "frac”
> production (using a trailing decimal and zero); this is not ambiguous,
> and in the interest of being liberal in what you accept, parsers
> shouldn’t produce any error.

+1

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 May 2019 11:03:11 -0500

> who both suggest that a generic JSON parser runs first, followed by a

It's not that "a generic JSON parser runs first, followed by ...", but
that that must be a valid implementation choice.

> indistinguishable by the time the schema is taken into account.

Correct.

> forking JSON itself. I don't think there is much more to be said on

Excellent.  I think that's been put to bed now.

+1

I've been meaning for some time now to create an ASN.1 -> JSON schema
converter.  The idea would be to then use JS or jq to perform additional
analysis / modifications, and generate code.  In particular I'm looking
to express code generation controls using path-like expressions and thus
avoid having to express such controls as commentary or other
modifications to an ASN.1 module.

A similar approach using XML would also work, but then the right tool
would be XSLT/XPath, and I'm not too keen on that, having written a
not-too-trivial XSLT app.

> highlighting, formatting, and other such niceties for free.

JSON is a bit awful, honestly, but yes, it has these properties: that is
widely used, somewhat readable and editable, and has a lot of tooling
available.

> moving away from JSON toward something like Protobufs or CBOR.

CBOR and other binary JSON formats have the benefit of keeping the data
model, which means that existing tooling can easily be made to interface
with them.

> classes/types/etc, is a problem worth solving?

Of course it is.

> may be a way we address this problem?

I shall have to read it.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 May 2019 11:07:41 -0500

> The idea is not changing JSON in any way but to add *strictness* to

That's not an innocent change.  It breaks interoperability.

> This has ZERO impact on JSON parsers but could in some cases affect

Not true: it makes them reject more inputs, which makes them less
interoperable.

> JSON serialization.

Exactly, so not OK.

> There are no problems defining a set of numeric types having a range
> and syntax that conforms to what is the norm for most programming
> languages since all of that fits nicely into JSON.

This is insufficient justification for making JSON parsing more strict.

> However, assuming that any JSON parser also could be used for
> *implementing* JSL (including verifying arbitrary input data) will set
> the bar way too low.
> 
> Regarding the integer discussion I believe Open API/Swagger which has
> quite an active and big user community already flags 3.0 as an invalid
> integer.  A bug report on Open API (or on Jackson) would most likely
> return "works as intended".

That's an interop bug.  They should fix it.

Nico
--

---

From: Erik Wilde <[email-masked]>
Date: Wed, 8 May 2019 09:42:52 -0700

> At the time when I was closer to the XML community, the hallway consensus was that W3C Schema was *designed* to sell more tools.

that's one way to look at it. given the complexity of XSD's model (which 
is a different discussion to have), it's hard to imagine how you could 
easily work with it just using a generic underlying model (of any kind: 
even the RNG compact syntax isn't easily revealing all relationships).

it's probably fair to say that any sufficiently complex domain model 
will require you to use tooling supporting it, instead of being able to 
just work with a source file of any format.

to me, the first discussion to have is how complex/powerful you want the 
domain model to be (XSD went completely overboard there). *after* that 
has been decided, and if it is of modest complexity, then it may make 
sense to consider a syntax based on a generic model so that people can 
directly read (and possibly even write) the source.

for old times sake, at https://www.xml.com/pub/a/2003/08/27/xscs.html is 
a specific XSD syntax we worked on to have a good "source file UX" for 
XSD. of course it never caught on because people used tools anyways, and 
this approach required specific code for reading and writing XSCS.

cheers,

dret.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 May 2019 18:59:54 +0200

> That's an interop bug.  They should fix it.

It also nicely demonstrates the law of inevitable extensibility.

JSON was designed not to provide extensibility.  Ever.
Radical, but a well motivated idea at the time.

Now people did not want to accept that.
They wanted to extend the JSON data model with the distinction between two number types.

The law of inevitable extensibility says:

# If you don’t provide an extension point, somebody will.
# They will shove their extension into any crevice available.
# (And your up to now rock-solid design breaks.)

The crevice here was that there are many ways to express the number 3: 3, 3.0, 3e0, 0.03e2, 300e-2 etc.  We can give some of these expressions different semantics than the other.  Voila.  Extensibility achieved.  Brittleness added.

I have seen the law of inevitable extensibility applied to:

— duplicate map keys for adding comments to JSON.  Some people believe:

  { “postcode”: “This is a comment about the post code 4711”,
    “postcode”: 4711 }

  is a valid way to extend JSON with comments.  (Because their implementation happens to ignore map keys that are invalidly used again later, which is not a given.)

- unnecessary escaping for ascribing some special semantics.  Say, if a string starts with “\u0073” and not with the equivalent “s”, this is supposed to mean the string has some different semantics (e.g., the rest is a base64url encoded byte string as opposed to what would normally be a text string, or a type tag, or…).

— millions of ways of interpreting whitespace or the lack thereof.  I’ve lost count.

I do not think any longer that anything that is meant to see significant usage can be designed without designing in extensibility points from the start.  At least if it is supposed to last.

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 8 May 2019 19:01:20 +0200

Since I have what appears to be the diametrically opposed opinion,
I'd better leave it there [*].  Flame-wars are kinda boring.

That a stricter interpretation (=subset) could break parsers
is beyond my understanding.  Probably it is because you seem
to plot with a different order of execution.

Or maybe the problem is rather than you want to apply schemas to
existing systems, [almost] irrespective how broken these may be?

In my book you start by validating data (strict) and only if that
works out you dispatch it to your good ol' JSON tools and applications
to digest.  For verifying serializations you would do the opposite.

Anyway, I wish you good luck with your project!

cheers,
Anders

*] I won't respond....

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 May 2019 13:31:31 -0500

> Radical, but a well motivated idea at the time.

The spec allows extensions, actually, but indeed, it doesn't provide for
extensibility.

(For example, jq will parse Inf and Infinity, but it will not output
such words, nor will it output NaN.)

> # (And your up to now rock-solid design breaks.)

The particular "extension" we're talking about isn't an extension.

> The crevice here was that there are many ways to express the number 3:
> 3, 3.0, 3e0, 0.03e2, 300e-2 etc.  We can give some of these
> expressions different semantics than the other.  Voila.  Extensibility
> achieved.  Brittleness added.

If anything one would expect this to bring up canonicalization...
(oh no, I've mentioned that.  please no one take that as a serious
invitation to discuss canonical JSON!)

>   is a valid way to extend JSON with comments.  (Because their
>   implementation happens to ignore map keys that are invalidly used
>   again later, which is not a given.)

Yes.

> - unnecessary escaping for ascribing some special semantics.  Say, if
> a string starts with “\u0073” and not with the equivalent “s”, this is
> supposed to mean the string has some different semantics (e.g., the
> rest is a base64url encoded byte string as opposed to what would
> normally be a text string, or a type tag, or…).

Oof.

> — millions of ways of interpreting whitespace or the lack thereof.

Play stupid games...

> I do not think any longer that anything that is meant to see
> significant usage can be designed without designing in extensibility
> points from the start.  At least if it is supposed to last.

Maybe.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 May 2019 21:30:36 +0200

Hi Nico,

> On May 8, 2019, at 20:31, Nico Williams <[email-masked]> wrote:
> 

Huh?  RFC 8259 is very outspoken here:

   Numeric values that cannot be represented in the grammar below (such
   as Infinity and NaN) are not permitted.

So you have a superset of JSON as your jq input grammar, which is fine for me (it just isn’t JSON anymore).  Maybe you mean the sentence in the section about Parsers:

   A JSON parser MAY accept non-JSON forms or extensions.

Yep, non-JSON.  JSON has no extensibility points.

> The particular “extension" we're talking about isn't an extension.

Extending the data model to enable expressing two different kinds of numbers, which are identical in numeric value (i.e., JSON semantics), but have different semantics in the data model defined by the extension (distinguishing integer 3 from floating point 3), definitely is an extension.  The fact that only instances are communicated that are also valid under (by definition unextended) JSON semantics doesn’t change this.

> invitation to discuss canonical JSON!)

C14n is a different thing: It reduces the number of different representations allowed for a specific instance in the data model.  There is no new semantics.  A generic decoder will have no problem with inputting a c14nized serialization.  A generic encoder won’t always spit out a c14nized instance (and we don’t expect it to), but a step can be added to convert the JSON instance into the equivalent c14nized instance without breaking that generic JSON encoder.  This is all fairly innocuous.  (It is also hard, and the specific proposal pushed here neither has fully demonstrated that it has nailed the hard parts nor does it use reasonable taste — principle of least surprise — in some other places.  Convert to UTF-16-BE before sorting?  Oof.  But sure, sorting Hittite characters between hangul Jamo and private use can be made interoperable, if it rocks your boat.)  More importantly, C14n ensures that there is no appetite for interpreting two lexically different instances that have the same semantics in JSON as different — at least one of them couldn’t be C14nal!

> […]

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 May 2019 14:58:35 -0500

On Wed, May 08, 2019 at 09:30:36PM +0200, Carsten Bormann wrote:
> So you have a superset of JSON as your jq input grammar, which is fine
> for me (it just isn’t JSON anymore).  Maybe you mean the sentence in
> the section about Parsers:
> 

Yes.

> Yep, non-JSON.  JSON has no extensibility points.

A JSON-like text with 'Infinity' in it is not-JSON.

> Extending the data model to enable expressing two different kinds of
> numbers, which are identical in numeric value (i.e., JSON semantics),
> but have different semantics in the data model defined by the
> extension (distinguishing integer 3 from floating point 3), definitely
> is an extension.  The fact that only instances are communicated that
> are also valid under (by definition unextended) JSON semantics doesn’t
> change this.

Yes, but note that accepting 'Infinity' doesn't cause an
interoperability problem, while rejecting '3.0' does.

Nico
--

---

From: Austin Wright <[email-masked]>
Date: Wed, 8 May 2019 15:49:20 -0700

> On May 8, 2019, at 09:59, Carsten Bormann <[email-masked]> wrote:
> 

Who’s “they”? There’s only two uses of “integer” in relation to JSON that I’m aware of:

1. Implementations that parse JSON documents into a pre-defined structure, and will raise an error if the struct cannot store the lexical data (for example, putting a null into a int64_t, or an object into a char*).

2. JSON Schema, which defines integer as a strict subset of “number”. (It’s not even a data model type, it’s a special case for the “type” keyword that means the same as `{type:”number”, multipleOf:1}`)

Neither of these “extend" JSON, because JSON does not impose any requirements on how implementations parse numbers. My reading of RFC8259 allows implementations to differentiate between `3` vs `3.0` vs `3.00`.

Nonetheless, JSON Schema specifies that numbers are mathematical, so `10.0` would still be considered an integer. Likewise, I think parsers should ignore excess precision if it doesn’t change the mathematical value. But this is entirely a choice based on how most implementations work; rather than what JSON seems to allow. For example, scientific software might store significant figures such that `10.0` is considered less precise than `10.000`. In my reading of RFC8259 and ECMA 404, this is perfectly legal, but JSON Schema makes no distinction for validation purposes (i.e. there would be no way to assert “number must be at least this precise”).

>  is a valid way to extend JSON with comments.  (Because their implementation happens to ignore map keys that are invalidly used again later, which is not a given.)

Certainly, this is not interoperable. But it is legal: RFC8259 merely says documents “SHOULD NOT” use multiple keys, and allows different implementations to handle it differently. ECMA-404 specifically says there’s no limits on string names, even repeated names (ECMA 404 is the document referenced by ECMAScript, though that specifies an algorithm that ignores repeated keys.)

For this reason, JSON Schema does not have any features dealing with repeated keys or their order (specifically, the data model specifies that objects are treated as an unordered string to value mapping), this is another artistic license that favors uniformity over supporting the complete lexical space of JSON.

> - unnecessary escaping for ascribing some special semantics.  Say, if a string starts with “\u0073” and not with the equivalent “s”, this is supposed to mean the string has some different semantics…

While JSON strongly implies these two forms must be considered the same thing, note this isn’t true for all formats. Namely, %2F is not the same as a forward slash in a URI, they are to be treated differently. (There’s been many security holes caused because people treat `%2F..%2F` the same as `/../` and it’s not.)

> — millions of ways of interpreting whitespace or the lack thereof.  I’ve lost count.

Could you provide one or two examples please?

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 9 May 2019 08:50:27 +0200

On May 9, 2019, at 00:49, Austin Wright <[email-masked]> wrote:
> Who’s “they”?

Those people who want 3 and 3.0 to mean different things in a JSON document.

> Neither of these “extend" JSON, because JSON does not impose any requirements on how implementations parse numbers. My reading of RFC8259 allows implementations to differentiate between `3` vs `3.0…

The language that Section 6 of RFC 8259 has inherited is not very good.
I cannot achieve a reading, though, that would support the above.

> Certainly, this is not interoperable. But it is legal: RFC8259 merely says documents “SHOULD NOT” use multiple keys,

Please re-read RFC 2119.  SHOULD is a normative statement.  It is not a MUST mainly because streaming implementations would have a hard time properly implementing that prohibition, so they are allowed to occasionally malfunction.  But a malfunction it is.

> and allows different implementations to handle it differently. ECMA-404 specifically says there’s no limits on string names, even repeated names (ECMA 404 is the document referenced by ECMAScript, though that specifies an algorithm that ignores repeated keys.)

Well, yeah.  There is a reason we needed to maintain an IETF specification here.

> While JSON strongly implies these two forms must be considered the same thing, note this isn’t true for all formats. Namely, %2F is not the same as a forward slash in a URI, they are to be treated…

I was talking about JSON, not about URIs, where percent notation indeed has different semantics (essentially creating a double of the encoded character without its syntactic properties).

The myths and misunderstandings cited in this message are actually a good corollary to the discussion of the Postel principle and soupification that is happening over at [email-masked]…  The JSON WG historically has shied away from making some of its statements very clear, and the resultant weasel wording will come back forever biting us.  I hope we can be better in the future.

Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 9 May 2019 13:15:11 -0400

> Radical, but a well motivated idea at the time.

My understanding is that you can extend JSON if you like. Just call it
something else. And that makes perfect sense for a data encoding.

My extensions to JSON are called JSON-B, J-SON-C and JSON-D. Each extension
adds functionality that is critical for certain applications (verbatim
encoding of binary and text blobs, compression, data types).

Since they are strict supersets, an implementation needs separate encoders
for each one but a JSON-B decoder will serve JSON and JSON-B extensions. So
an implementation only needs one decoder.

If we were going to do a text encoding for scientific data, the changes I
would propose would be

* Hex encoding for floats so as to preserve precision of Real64 and Real32
data.
* Allow compact encoding of table data without having to specify tags for
every data item.
* Allow for comments
* Ensure that the new format can be loss-lessly converted to and from JSON.

But it would not be JSON. It would be something else. And that is OK
because if I am trying to read a data file, I need to know the encoding.

I would really like to see a JSON-like format emerge as an alternative to
CSV because, I have spent too much time trying to analyze files that
contain data separated from the context. If I am doing science, I want
files that look like this

{ "TemperatureData" : {
   "StartDate" : "2010-1-1",
   "EndDate" : "2011-2-2",
   "Location" : "Rome",
   "Units" : "Centigrade",
   "Channels" : [ "C1", "C2", "C3", "C4",
   "Readings" : [[ 20.2, 23.3, 44.3, 20.1 ... ] [34.3, 3.4 ...] ... ]

Not this
20.2, 23.3, 44.3, 20.1 ...
34.3, 3.4...
...

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 9 May 2019 19:36:58 +0200

> I would really like to see a JSON-like format emerge as an alternative to CSV because, I have spent too much time trying to analyze files that contain data separated from the context.

SenML, RFC 8428, was made for this.  Can be serialized in XML, EXI, JSON, CBOR.

> If I am doing science, I want files that look like this
> 
> { "TemperatureData" : {
> …

SenML looks a bit different, but has the same kind of information content.
If you really want the arrays that are in your example, it would be an interesting exercise to extend SenML for that — up to now, the large organizations that use it have been happy with what we have.

Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 9 May 2019 16:04:26 -0400

On Thu, May 9, 2019 at 1:37 PM Carsten Bormann <[email-masked]> wrote:

>
> > I would really like to see a JSON-like format emerge as an alternative
> to CSV because, I have spent too much time trying to analyze files that
> contain data separated from the context.
>

I was using temperatures as an example to illustrate the fact that the
general case for science data is that I have a bunch of information
describing the data set and a series of tables.

If I am using SciKit-Learn to analyze data, I am typically importing a
bunch of data files containing different tables and doing joins and merges
on them. A lot of the context for the process is lost because a CSV file
can only contain the table. So this morning I was looking at a ZIP file
that had the following file layout:

descrip.txt  # this file
dataset.json # the metadata
twitter.csv # Twitter persona data
facebook.csv # Facebook persona data
tweetlog.csv # Posts to twitter
fblog.csv # Posts to Facebook

Oh and they each have different layouts and syntax. I have to pull them
into Pandas, construct data frames, drop out features to SciKit and then
identify the fake news sources.

I think it would be actually quite easy to just dump out one file that has
everything in JSON and for this to be engineered so that the relevant
Pandas frames are constructed automagically.. But that is not where the
field is right now.

---

From: John Cowan <[email-masked]>
Date: Thu, 9 May 2019 19:16:32 -0400

> * Allow compact encoding of table data without having to specify tags for

If data is stored columnwise instead of row-wise, that's not a problem with
as-is JSON.  You could represent a table of observations like this:

{"elapsedTime":  [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50],
 "waterTemperature": [25, 25, 33, 47, 60, 72, 88, 91, 100, 100, 100]}

---

From: Ulysse Carion <[email-masked]>
Date: Thu, 9 May 2019 20:55:21 -0700

Carsten,

Do you think that our difference in opinion on CDDL vs JSON Schema
Language may be attributable to a difference in requirements?

It seems to me that your use-case is centered around defining
standards and other complex data requirements. CDDL is, in my view,
unquestionably a better choice for this use-case. In my mind, CDDL is
ABNF for CBOR, and that is undeniably what standards dealing with CBOR
or its near-equivalents require. The existing references, from RFCs,
to CDDL are testament to this.

But I (and I suspect Tim) am more preoccupied solely with defining the
mundane sorts of messages that come out of JSON event processing and
repetitive JSON APIs. Tim has blogged (see link in my original email)
about dealing with AWS's CloudWatch events. That's messages that look
like this:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html

Tons of messages, and frequently being added and updated. But none of
these messages are particularly exciting from a schema perspective.

CDDL can do much more than is necessary for merely representing
CloudWatch events. This may seem like a good thing, but such excess
capability reduces the suitability of the solution. JSON Schema
Language is intentionally small and scuttled in scope, so as to
simplify code and UI generation. By being so limited in scope, JSON
Schema Language fits more easily into the architecture of a system
that would like to integrate it.

Do you think this may be an explanation for our disagreement? If not,
I may need to re-read the discussion, because I must have
misunderstood you.

---

From: Austin Wright <[email-masked]>
Date: Fri, 10 May 2019 00:40:21 -0600

> On May 8, 2019, at 23:50, Carsten Bormann <[email-masked]> wrote:
> 

Well, JSON says that any number that fits the syntax is legal, and it doesn’t proscribe a way to normalize the numbers. And some software does keep track of significant figures. Why should those applications interpret JSON any differently?

> Please re-read RFC 2119.  SHOULD is a normative statement.  It is not a MUST mainly because streaming implementations would have a hard time properly implementing that prohibition, so they are allo…

“SHOULD" allows implementations to vary from the suggested behavior if there’s a reason. RFC8259 specifically says the reason is interoperability, and also says implementations might report all of the key-value pairs, even repeated keys. So if two implementations agree that repeated keys are meaningful, that seems legal to me.

I don’t see how the fact JSON can be streamed changes this. HTTP says servers must produce errors on some headers if they’re repeated; and HTTP is also parsed by streaming parsers.

Now, if the vast majority of JSON implementations don’t recognize excess precision, and don’t allow (or ignore) repeated keys, then maybe JSON should be updated to reflect that. Look at HTTP as an example, which is far more specific about how to handle these cases (e.g. repeated Host header, repeated Content-Length, media type q-values).

Cheers,

Austin.

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 10 May 2019 09:16:22 +0200

> Language may be attributable to a difference in requirements?

Hi Ulysse,

I’m not even sure we disagree.  That was why I became interested in converting between JSL and CDDL.  As I was trying to show, CDDL might make a fine presentation language for JSL, and JSL might be a nice “profile” for CDDL that is very simple to process.

> to CDDL are testament to this.

Yes, you are describing the intention correctly.  I would add that CDDL has proven as useful for describing pure JSON protocols as for CBOR.  

Not all JSON/CBOR protocols need the full capabilities of CDDL.  For instance, the example in the CDDL spec for RFC 7071 could easily be expressed in JSL, except for two details: reputation-object is not meant to be extensible (reputon is), and there are some value constraints (some values are integers).

> these messages are particularly exciting from a schema perspective.

Well, I just had a look at (randomly selected)
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#health-event-types
You’d need to add enumerations to JSL.  There are also timestamps in the object (ironically in two different formats).  There is a table that maps language tags to messages in that language.  (And the second and third example have a missing bracket.)  But I can’t really say, because the description by example only just begins to expose the actual intention.

> that would like to integrate it.

I’ve seen my share of developments that start simple.  How much functionality will be added to JSL before it becomes a standard?  Also, the law of extensibility tells us it will be extended even after becoming a standard.

Of course, in its domain, CDDL is incredibly simple.  Compare to JCR: JCR is about three times as complex as CDDL.  This is because CDDL was built from a few very simple building blocks, which combine nicely to provide its functionality.  JCR is more of an accretion of features, which in sum can do most of what CDDL can do.

But back to JSL and CDDL:
What I’m interested in is what are the sweet spots on this functionality vs. complexity continuum.  I think we have found two of these sweet spots (at least maybe after a little more calibration).  Now how do we handle the onslaught of applications that don’t quite fit the sweet spots?  

The question that intrigues me: Is it possible to define something that is as simple as JSL when you need just that, but allows dipping into the capabilities of something like CDDL where needed?

By the way: You may not be aware of the WISHI activity we have in the T2TRG (thing-to-thing research group) of the IRTF.  Here we look at modeling (not just for data) and at translating between different modeling approaches.  http://wishi.space if you want to have a look.

Grüße, Carsten

---

From: Rob Sayre <[email-masked]>
Date: Fri, 10 May 2019 00:32:51 -0700

> make a bit more sense of the conversation.

I think there's no need to do this work. I know, because I replaced a large
JSON-based analytics provider with a protobuf-based system that was cheaper
with better correctness, and was an implementation detail of a product
doing other things.

- Rob

---

From: "Manger, James" <[email-masked]>
Date: Fri, 10 May 2019 07:56:47 +0000

> “SHOULD" allows implementations to vary from the suggested behavior if there’s a reason. RFC8259 specifically says the reason is interoperability, and also says implementations might report all of…

NO WAY!
RFC8259 JSON explicitly says when keys in an object are not unique the behaviour of parsers is unpredictable. So it you specify a system that gives meaning to repeated keys then lots of JSON parsers will be unusable in this case. Hence, it is harmful to call that JSON, call it "ECMA 404 syntax without normal JSON semantics".

> I don’t see how the fact JSON can be streamed changes this. HTTP says servers must produce errors on some headers if they’re repeated; and HTTP is also parsed by streaming parsers.

The number of elements in a JSON object and the number of headers in an HTTP message are both theoretically unbounded. But there is a strong assumption that the latter (#headers) is limited: typically 10s, perhaps 100, never 1000s. Web servers limit total header size to, say, 8KB. That is small enough to always handle repeats. While many JSON objects will also be small, JSON is such a generic format that you cannot make a global assumption that JSON objects will always be small enough to hold in memory. Hence some apps will need streaming JSON processors that cannot reasonably detect duplicate keys (so the "SHOULD" is there to accommodate these).

> Now, if the vast majority of JSON implementations don’t recognize excess precision, and don’t allow (or ignore) repeated keys, then maybe JSON should be updated to reflect that.

RFC8259 JSON does have that text. For example, "good interoperability can be achieved by implementations that expect no more precision or range than [64-bit floats] provide"; "when the names within an object are not unique, the behavior [...] is unpredictable".

For more concrete text, use RFC7493 I-JSON (Internet JSON is a restricted profile of JSON).

--
James Manger

---

From: Anders Rundgren <[email-masked]>
Date: Fri, 10 May 2019 10:34:27 +0200

> For more concrete text, use RFC7493 I-JSON (Internet JSON is a restricted profile of JSON).

Right on!  I have "hammered" a bit on Oracle who interpreted I-JSON "literally" making small BigIntegers serialize as Number and big such as String which probably breaks most schemas.

They now seem to have come to their senses:
https://github.com/eclipse-ee4j/jsonp/issues/160

Oracle do not support duplicate keys either.  If you want to use such you are probably not really a candidate for using a schema.

JSON is slowly but surely getting better :)

thanx
Anders

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 10 May 2019 09:31:44 -0400

On Wed, May 8, 2019 at 12:43 PM Erik Wilde <[email-masked]> wrote:

> > If I have to work on a W3C Schema(*), I convert it to Relax-NG compact
> first.  True, I’ll lose some information, but as a net result I'll digest
> much more information out of the W3C Schema document than if I had directly
> looked at that.
> > At the time when I was closer to the XML community, the hallway
> consensus was that W3C Schema was *designed* to sell more tools.
>

XSD was designed to be a replacement for SGML DDLs. It is a document
markup, not a data markup.

Document markup cares about the order in which fields are entered. Data
markup really doesn't.

XML has many features that are bogus. The distinction between elements and
attributes for example. But they were in SGML and XML couldn't drop them.

The decision to use SGML in the Web was purely driven by
deployment/adoption concerns. We all hated it. There were many technical
issues, the tooling was awful, the specification was broken. The only
reason HTML used SGML was to get the publishing industry onside.

XML was an attempt to get rid of as much of the stupid from SGML as
possible. XSD was a necessary part of enabling XML.

---

From: John Cowan <[email-masked]>
Date: Fri, 10 May 2019 20:40:49 -0400

> attributes for example. But they were in SGML and XML couldn't drop them.

There were a *huge* number of things in SGML that XML dropped.  Not enough,
I agree.  But the difference between elements and attributes is the
difference between ordered content (which documents need) and
not-so-ordered properties (which documents also need).

Here are two approaches to simplifying XML that I like very much.

MicroXML is a subset of XML, as XML is a subset of SGML.  See <
https://dvcs.w3.org/hg/microxml/raw-file/tip/spec/microxml.html>.  The spec
is 10 pages long including a data model.

FtanML is syntactically different and semantically more flexible: it would
make a good replacement for both XML and JSON.  See <
https://www.balisage.net/Proceedings/vol10/html/Kay01/BalisageVol10-Kay01.html>.
("Ftan" is a village in Switzerland where the language was developed, not
an acronym.)   The spec is 27 pages long, including a data model, a schema
language, and a transformation language.

> possible. XSD was a necessary part of enabling XML.

I can't agree with you there.  RELAX NG would have been just as usable for
the purpose.

John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
At times of peril or dubitation,
Perform swift circular ambulation,
With loud and high-pitched ululation.

---

From: Tim Bray <[email-masked]>
Date: Tue, 28 May 2019 19:40:44 -0700

So Carsten, I am a person who would benefit from a super-simple JSON schema
langauge; in effect, something like JSL + enums + timestamps.  Furthermore,
as I've said, I'm willing to invest some IETF-work cycles to get such a
thing, in the event we could interest the community.  Do you think that
doing such a JSL and defining it using established CDDL semantics might be
a good idea?

Alternately, might we retain CDDL syntax and define a profile/subset which
would be appropriate for us JSON-only simpletons?

I'm not terribly fussy about mechanisms.

On Fri, May 10, 2019 at 12:16 AM Carsten Bormann <[email-masked]> wrote:

> I’m not even sure we disagree.  That was why I became interested in
> converting between JSL and CDDL.  As I was trying to show, CDDL might make
> a fine presentation language for JSL, and JSL might be a nice “profile” for
> CDDL that is very simple to process.
>
> Yes, you are describing the intention correctly.  I would add that CDDL
> has proven as useful for describing pure JSON protocols as for CBOR.
>
> Not all JSON/CBOR protocols need the full capabilities of CDDL.  For
> instance, the example in the CDDL spec for RFC 7071 could easily be
> expressed in JSL, except for two details: reputation-object is not meant to
> be extensible (reputon is), and there are some value constraints (some
> values are integers).
>
> You’d need to add enumerations to JSL.  There are also timestamps in the
> object (ironically in two different formats).  There is a table that maps
> language tags to messages in that language.  (And the second and third
> example have a missing bracket.)  But I can’t really say, because the
> description by example only just begins to expose the actual intention.
>
> I’ve seen my share of developments that start simple.  How much
> functionality will be added to JSL before it becomes a standard?  Also, the
> law of extensibility tells us it will be extended even after becoming a
> standard.
>
> Of course, in its domain, CDDL is incredibly simple.  Compare to JCR: JCR
> is about three times as complex as CDDL.  This is because CDDL was built
> from a few very simple building blocks, which combine nicely to provide its
> functionality.  JCR is more of an accretion of features, which in sum can
> do most of what CDDL can do.
>
> What I’m interested in is what are the sweet spots on this functionality
> vs. complexity continuum.  I think we have found two of these sweet spots
> (at least maybe after a little more calibration).  Now how do we handle the
> onslaught of applications that don’t quite fit the sweet spots?
>
> The question that intrigues me: Is it possible to define something that is
> as simple as JSL when you need just that, but allows dipping into the
> capabilities of something like CDDL where needed?
>
> By the way: You may not be aware of the WISHI activity we have in the
> T2TRG (thing-to-thing research group) of the IRTF.  Here we look at
> modeling (not just for data) and at translating between different modeling
> approaches.  http://wishi.space if you want to have a look.
>
> Grüße, Carsten
>

---

From: Rob Sayre <[email-masked]>
Date: Tue, 28 May 2019 20:00:52 -0700

I'm not sure this schema language will be productive as a source format in
the long term, but I don't think it will be harmful. So, it's not worth an
objection. The worst case would be that servers claim to conform to this
schema language, but fail to serve schema-conformant JSON in practice. This
seems like the likely outcome, in the absence of an output specification.

I have to say, I think I am uniquely qualified to raise this point. I
objected to the original JSON Stringify specification, which was something
pretty close to string concatenation, and got it corrected to something
close to its current form.

If a schema language doesn't provide an output specification, it's probably
not going to work in practice. Best case, it's a standards checkbox.

- Rob

On Tue, May 28, 2019 at 7:41 PM Tim Bray <[email-masked]> wrote:

> So Carsten, I am a person who would benefit from a super-simple JSON
> schema langauge; in effect, something like JSL + enums + timestamps.
> Furthermore, as I've said, I'm willing to invest some IETF-work cycles to
> get such a thing, in the event we could interest the community.  Do you
> think that doing such a JSL and defining it using established CDDL
> semantics might be a good idea?
>
>> The question that intrigues me: Is it possible to define something that
>> is as simple as JSL when you need just that, but allows dipping into the

---

From: Tim Bray <[email-masked]>
Date: Tue, 28 May 2019 20:23:29 -0700

What do you mean by “Output specification”?

On Tue, May 28, 2019 at 8:01 PM Rob Sayre <[email-masked]> wrote:

> If a schema language doesn't provide an output specification, it's
> probably not going to work in practice. Best case, it's a standards
> checkbox.
>
> - Rob
>
>
>> Alternately, might we retain CDDL syntax and define a profile/subset
>> which would be appropriate for us JSON-only simpletons?
>>
>>> Of course, in its domain, CDDL is incredibly simple.  Compare to JCR:
>>> JCR is about three times as complex as CDDL.  This is because CDDL was
>>> built from a few very simple building blocks, which combine nicely to
>>> provide its functionality.  JCR is more of an accretion of features, which
>>> in sum can do most of what CDDL can do.
>>>

---

From: Rob Sayre <[email-masked]>
Date: Tue, 28 May 2019 20:40:30 -0700

> What do you mean by “Output specification”?

Fair question. I'd say a good example might be the objects transferable
across postMessage() in browser JavaScript.

https://html.spec.whatwg.org/multipage/structured-data.html#transferable-objects

It seems like a deep problem at first, but it isn't. Those message formats
add Date objects and a few other things, but not that much.

- Rob

>> I'm not sure this schema language will be productive as a source format
>> in the long term, but I don't think it will be harmful. So, it's not worth
>> an objection. The worst case would be that servers claim to conform to this
>>>> You’d need to add enumerations to JSL.  There are also timestamps in
>>>> the object (ironically in two different formats).  There is a table that
>>>> maps language tags to messages in that language.  (And the second and third
>>>> What I’m interested in is what are the sweet spots on this
>>>> functionality vs. complexity continuum.  I think we have found two of these
>>>> sweet spots (at least maybe after a little more calibration).  Now how do
>>>> we handle the onslaught of applications that don’t quite fit the sweet
>>>> spots?
>>>>

---

From: Tim Bray <[email-masked]>
Date: Tue, 28 May 2019 20:47:16 -0700

TBH, what I want from a schema system is (a) useful error messages and (b)
ability to drive code generation, classes and serializer/deserializers and
so on.

On Tue, May 28, 2019 at 8:40 PM Rob Sayre <[email-masked]> wrote:

>>>>> Yes, you are describing the intention correctly.  I would add that
>>>>> CDDL has proven as useful for describing pure JSON protocols as for CBOR.
>>>>>
>>>>> The question that intrigues me: Is it possible to define something
>>>>> that is as simple as JSL when you need just that, but allows dipping into
>>>>> the capabilities of something like CDDL where needed?
>>>>>

---

From: Rob Sayre <[email-masked]>
Date: Tue, 28 May 2019 20:54:30 -0700

> ability to drive code generation, classes and serializer/deserializers and

No problem. If it can drive serializer/deserializers, let's spec out some
conformance tests.

- Rob

>> It seems like a deep problem at first, but it isn't. Those message
>> formats add Date objects and a few other things, but not that much.
>>
>> - Rob
>>
>>
>>
>>
>>

---

From: Pete Cordell <[email-masked]>
Date: Wed, 29 May 2019 10:40:41 +0100

On 29/05/2019 03:40, Tim Bray wrote:
> Furthermore, as I've said, I'm willing to invest some IETF-work cycles 
> to get such a thing, in the event we could interest the community.  Do 
> you think that doing such a JSL and defining it using established CDDL 

Hi Tim,

Could you show some simple examples of:

- what you're thinking a JSL with established CDDL semantics would look like

- what you're hoping for in terms of useful error messages?

Thanks,

Pete.

---

From: Nico Williams <[email-masked]>
Date: Wed, 29 May 2019 09:40:07 -0500

> ability to drive code generation, classes and serializer/deserializers and

This is all I want as well.  Validation and codegen.

---

From: John Cowan <[email-masked]>
Date: Wed, 29 May 2019 15:14:43 -0400

The trouble is that "validation" is a very vague term.  I would be happy
with a language that can only express simple type validation, minimally as
follows:

0) JSON values are divided into types, namely null, numbers, strings,
booleans, object types, and arrays of any of these.

1) An object type specifies what attributes the corresponding object has
and what the types of the corresponding values are, or else specifies that
it may have arbitrary keys and values.   It may have a name in the schema
language

2) When an array is expected, the validator will check that all its
contents have the same specified type.

3) There is a way to specify for a given object type whether it must
contain exactly the specified keys, may contain a superset of the specified
keys where the additional keys are not checked, or may contain a subset of
the specified keys.

This language leaves validation of the values of strings and numbers to
lpost-validation code.  It just makes sure that for a statically typed
language a proper combination of records/structs/classes that can be
generated in advance can represent any JSON value that is valid against the
schema.

Here is a very simple examplotron language:

Sample document from json.org:

{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}
    ]
  }
}}

Corresponding schema (constructed by hand, may have errors):

{
  "root": {"menu", "menu"},
  "menu": {
    "id": "",
    "value": "",
    "popup": "popup",
  },
  "popup":  {"menuitem": ["menuitem"]},
  "menuitem": {
    "value": "",
    "onclick": ""
  }
  "__whole__": {"menu": "root"},
  "__strictness__": {"menu": "superset"}
}

A schema is a JSON object whose keys are the names of object types, and
whose values are the corresponding definitions.  Object type names are
meaningful only within the schema.  In addition, the key "__whole__"
(mandatory) is the type of the whole document. If the key "__strictness__"
is present, its keys are object types and its values are one of the strings
"exact", "subset", or "superset".  Object types without names, or those not
mentioned in "__strictness__", are always exact.

An object type definition is an object whose keys are the keys of objects
that are valid against it and whose values are the corresponding types.
Alternatively, an empty object designates an object with arbitrary keys (in
effect, a dictionary).

An object type's designator can be its name (as a string), or an in-place
object type definition.  A null type's designator is the null value.
Designators for a number, string, or boolean type are specified by any
number, the empty string "", or any boolean respectively. An array type's
designator is an array with one value, a designator for the type of the
elements of the array.

The schema for schemas is not very useful: it is simply {"__whole__": {}},
because the top-level keys have unpredictable names.

---

From: Nico Williams <[email-masked]>
Date: Wed, 29 May 2019 15:17:17 -0500

> with a language that can only express simple type validation, minimally as

Validation, to me, means making sure that the "shape" of a value adheres
to the "shape" given by the schema.

Examples:

 - if, where you expect an array you see a boolean, that's an error

 - where you expect a numeric value you might have some constraints
   (e.g., range constraints) that, again, should raise an error if not
   met

 - ditto strings ('must be bas64', 'must be one of these (enum)', 'must
   match some regexp', etc.).

That's it for validation.

Now, validation mostly should only happen at decode time.

In a language like JS (or jq) you might first parse the whole JSON text
and then validate its "shape".

> keys where the additional keys are not checked, or may contain a subset of

So far it sounds a lot like ASN.1.

> Corresponding schema (constructed by hand, may have errors):

I would prefer a schema language built around defining types, where
user-defined types have to be a) JSON types with b) optional
constraints.

Regarding (a), we're going to need a discriminated union, something JSON
doesn't have as such, but obviously is trivial to build by convention.

Also, using JSON itself as the language for the schema is nice because
you don't need to build a partser for the schema.  But it's not very
nice for users, so I'd prefer to have some non-JSON syntax for the
schema.

My counter:

  top_level : choice(file_menu, edit_menu);
  file_menu : object{"menu"->file_menu_contents};
  edit_menu : object{"edit_menu"->edit_menu_contents};
  file_menu_contents := object{
    "id":"file", /* this means this must be present with this value */
    "value":"File",
    "popup": object { ... }
  };

You can see the above has a hint of ASN.1 flavor, greatly simplified and
somewhat transformed to be sure.

I don't care one bit about the specifics of the syntax other than: it
should be easy to build parsers for it.

But I do very much like the idea of a schema language built around
defining types.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 29 May 2019 16:54:11 -0400

> This is all I want as well.  Validation and codegen.

I use mine for code generation and for document generation. The drafts are
produced by the same source that generates the code used to create the
examples.

Validation is another issue entirely. I don't think it is actually useful
beyond 'can the data be deserialized'. It is problematic if someone sends a
string in a field where I really need an integer, but additional fields I
will simply ignore unless they are in some SAML Conditions like structure
which MUST be understood. But the only reason to use that approach is when
you require a means of signalling a breaking change.

The tools I am currently using are on Github and they work on Windows, OSX
and Linux.

This is not the approach I would use if I was going to propose a standard.
There are a half dozen different schema systems that vary by small degrees
for different purposes that I think could be merged.

https://github.com/hallambaker/PHB-Build-Tools

---

From: Pete Cordell <[email-masked]>
Date: Wed, 29 May 2019 22:14:17 +0100

>      "id":"file", /* this means this must be present with this value */

For a quick taste of JCR, see:

    http://json-content-rules.org/tutorial

For a more examplotron feel, JCR could define a #inferred-types 
directive so, for example, string values are treated as generic string 
rather than a specific string. e.g.:

#inferred-types
{
     "Image": {
         "Width":  800,
         "Height": 600,
         "Title":  "View from 15th Floor",
         "Thumbnail": {
             "Url":    "http://www.example.com/image/481989943",
             "Height": 125,
             "Width":  100
         },
     "Animated" : false,
     "IDs": [116, 943, 234, 38793]
     }
}

gets treated as:

{
     "Image": {
         "Width":  integer,
         "Height": integer,
         "Title":  string,
         "Thumbnail": {
             "Url":    string,
             "Height": integer,
             "Width":  integer
         },
     "Animated" : boolean,
     "IDs": [integer *]
     }
}

Pete.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 29 May 2019 23:15:09 +0200

>    "id":"file", /* this means this must be present with this value */

While it is certainly enjoyable to design another language, I’m not sure why this should not be done in an ABNF-like syntax such as CDDL (which, by the way, is in AUTH48 and will soon be RFC 8610).  
Keeping your names (even the snake case, where I’d be using kebab case):

top_level = file_menu / edit_menu
file_menu = { menu: file_menu_contents }
edit_menu = { edit_menu: edit_menu_contents }
file_menu_contents = {
  id: text    ; assuming here this is your examplotron content
  value: text
  popup: { }
}
edit_menu_contents = { }

I didn’t put in the ignore-unknown stuff; that would be 12 characters in CDDL unless you define something to make this shorter.

Put the above into a menus.cddl and try

  cddl menus.cddl g 10

you get 10 examples:

{"menu": {"id": "gray", "value": "commixt", "popup": {}}}
{"menu": {"id": "proctoplasty", "value": "compunctious", "popup": {}}}
{"menu": {"id": "taglike", "value": "valvulate", "popup": {}}}
{"edit_menu": {}}
{"menu": {"id": "uninundated", "value": "unworthily", "popup": {}}}
{"edit_menu": {}}
{"edit_menu": {}}
{"menu": {"id": "Orangeist", "value": "imperialin", "popup": {}}}
{"edit_menu": {}}
{“edit_menu": {}}

This is a bit boring as there is only one choice and two non-singleton types.
Sorry if I didn’t get the intent of your example right.

Grüße, Carsten

PS.: OK, with ignore-unknown:

top_level = file_menu / edit_menu
file_menu = { menu: file_menu_contents _..._ }
edit_menu = { edit_menu: edit_menu_contents _..._ }
file_menu_contents = {
  id: text    ; I'm assuming here this is your examplotron content
  value: text
  popup: { _..._ }
  _..._
}
edit_menu_contents = { _..._ }

_..._ = ( * text => any )

➔ 

{"menu": {"id": "fahlerz", "value": "unshocked", "popup": {"astrophotometry": "perdurance"}, "foolship": "pentacosane", "larrikin": "myoglobin"}}
{"edit_menu": {"priggery": "cordaitean", "jackpudding": "righten", "unconfidence": "photolitho", "reviling": "ewe"}, "allophylic": "compound", "teacupful": "allergen"}
{"menu": {"id": "amazement", "value": "condylomatous", "popup": {"cosounding": "share", "contribute": "gabblement", "gunpaper": "dividualism", "undersociety": "noration"}}}
{"edit_menu": {"glareole": "clicket", "madling": "dizain"}, "scrimpily": "gypsywort", "neurofibrilla": "discolor"}
{"edit_menu": {"unpermissive": "cockneity"}, "caudillism": "visa", "interloop": "anthracnose"}
{"menu": {"id": "blankite", "value": "puffery", "popup": {"katuka": "Gnathostomata", "effeteness": "unspelt"}}, "Chaldaic": "extrafoliaceous", "trimesic": "dwindlement", "successful": "fluctuant", "nevadite": "aggradational"}
{"menu": {"id": "acrostolion", "value": "devocalize", "popup": {}}, "cymation": "premanufacturer", "mysticete": "analyst", "ethnocentric": "Myrtales"}
{"edit_menu": {"glycerolize": "blastoid", "undisputedness": "propheticly"}, "orchiotomy": "illiquidity"}
{"menu": {"id": "tenebriously", "value": "guardo", "popup": {"abeigh": "fibrillar", "Sminthurus": "disnosed", "accessibility": "autem"}}}
{"edit_menu": {"stableman": "erosional"}}

---

From: Ulysse Carion <[email-masked]>
Date: Wed, 29 May 2019 17:07:31 -0700

Hi Carsten,

Any thoughts on Tim Bray's suggestion? Regarding something akin to JSL (+
enums + timestamps), but perhaps built atop the foundations of CDDL?

- Ulysse

On Wed, May 29, 2019 at 2:15 PM Carsten Bormann <[email-masked]> wrote:

> While it is certainly enjoyable to design another language, I’m not sure
> why this should not be done in an ABNF-like syntax such as CDDL (which, by
> the way, is in AUTH48 and will soon be RFC 8610).
> I didn’t put in the ignore-unknown stuff; that would be 12 characters in
> CDDL unless you define something to make this shorter.
>
> {"menu": {"id": "fahlerz", "value": "unshocked", "popup":
> {"astrophotometry": "perdurance"}, "foolship": "pentacosane", "larrikin":
> "myoglobin"}}
> {"edit_menu": {"priggery": "cordaitean", "jackpudding": "righten",
> "unconfidence": "photolitho", "reviling": "ewe"}, "allophylic": "compound",
> "teacupful": "allergen"}
> {"menu": {"id": "amazement", "value": "condylomatous", "popup":
> {"cosounding": "share", "contribute": "gabblement", "gunpaper":
> "dividualism", "undersociety": "noration"}}}
> {"edit_menu": {"glareole": "clicket", "madling": "dizain"}, "scrimpily":
> "gypsywort", "neurofibrilla": "discolor"}
> {"menu": {"id": "blankite", "value": "puffery", "popup": {"katuka":
> "Gnathostomata", "effeteness": "unspelt"}}, "Chaldaic": "extrafoliaceous",
> "trimesic": "dwindlement", "successful": "fluctuant", "nevadite":
> "aggradational"}
> {"menu": {"id": "acrostolion", "value": "devocalize", "popup": {}},
> "cymation": "premanufacturer", "mysticete": "analyst", "ethnocentric":
> "Myrtales"}
> {"edit_menu": {"glycerolize": "blastoid", "undisputedness":
> "propheticly"}, "orchiotomy": "illiquidity"}
> {"menu": {"id": "tenebriously", "value": "guardo", "popup": {"abeigh":
> "fibrillar", "Sminthurus": "disnosed", "accessibility": "autem"}}}

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 30 May 2019 11:36:10 +0200

Hi Ulysse,

your message catches me at the start of a long weekend (which seamlessly turns into an IAB workshop and more travel next week).  So I won’t have much time preparing anything in the next 10 days.

But most of the job of translating JSL into CDDL is already done in example form in

https://mailarchive.ietf.org/arch/msg/json/1WpdpfsXr3gEIdXb8OV0R_J-O4E

Adding enums is easy once there is a JSL side (one would have to decide between two conventions of doing them in CDDL) — can you propose that?
CDDL does have ways of specifying timestamps (both POSIX epoch-based numbers of seconds and RFC 3339 style; the independent tag 1001 is addressing further types of time such as TAI or GPS); which ones do we want?

Having a JSON data definition language like JSL that is both homoiconic (represented in JSON itself) and unable to fully describe its own structure might be considered ironic by some, but it is also a powerful statement of direction:  Not even starting to boil the ocean.

Grüße, Carsten

> On May 30, 2019, at 02:07, Ulysse Carion <[email-masked]> wrote:
> 
> Hi Carsten,
> 
> Any thoughts on Tim Bray's suggestion? Regarding something akin to JSL (+ enums + timestamps), but perhaps built atop the foundations of CDDL?
> 
> - Ulysse
>

---

From: John Cowan <[email-masked]>
Date: Thu, 30 May 2019 08:53:50 -0400

On Wed, May 29, 2019 at 4:24 PM Nico Williams <[email-masked]> wrote:

Agreed,, because statically typed programming languages can't cope.

>    match some regexp', etc.).

The question is, where do you stop?  Range constraints are all very well,
but suppose you want to constrain a number value to be a U.S shoe size,
which is either an integer (in a certain range) or an integer plus 0.5?
What is more, the valid values of one field often depend on the actual
value of one or more other fields, and so you end up designing a whole
(functional) programming language.

But in general we expect JSON to be consumed by a program written
in some such language anyway.  So we might as well decode all numbers
as double floats, since you can only count on that much precision and
range anyway, and leave subtypes of number to general-purpose validation.
By the same token, I don't see that it makes sense to put a whole
regular expression subsystem into the validator when it can often
be expressed much more clearly in code (especially in languages
that have a combinator rather than a string representation of
regular expressions).

Actually, I'll amend my proposal to inclde integers, given how
important they are.  This is easily accomplished by using, say,
0 (equivalently 0.0) to designate an integer and 0.5 to designate a float.)

Using JSON itself as the language for the schema is nice because
> nice for users, so I'd prefer to have some non-JSON syntax for the

I don't care about the syntax, but examplotron-style (in which a
type designation is an instance of what it designates) is a strong
design constraint, which is why I adopted it.  I have no problem
with a BNF-style syntax for practical use, like RNG compact syntax.

John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
Assent may be registered by a signature, a handshake, or a click of a
computer
mouse transmitted across the invisible ether of the Internet. Formality
is not a requisite; any sign, symbol or action, or even willful inaction,
as long as it is unequivocally referable to the promise, may create a
contract.
       --Specht v. Netscape

---

From: John Cowan <[email-masked]>
Date: Thu, 30 May 2019 09:08:44 -0400

On Wed, May 29, 2019 at 5:15 PM Carsten Bormann <[email-masked]> wrote:

> While it is certainly enjoyable to design another language,

I cheerfully admit to that motive.

>  I’m not sure why this should not be done in an ABNF-like syntax

such as CDDL (which, by the way, is in AUTH48 and will soon be RFC 8610).

Keeping your names (even the snake case, where I’d be using kebab case):

My effort with this schema language is not to show that it can do something
that other proposals cannot, but that it is the simplest thing that can
possibly
work for both dynamically and statically typed programming languages,
and that everything else can and should be left to higher (inner) layers
that do general-purpose validation.  (Another example: a field that
must be a "dictionary word" in the sense that it appears in a particular
dictionary, whose contents may change over time.  Implementing this
in the schema language will require a database lookup capability oe
perhaps the ability to call a web-services API.)

 Antoine St.-Exupery said: "A designer knows he has achieved perfection
not when there is nothing left to add, but when there is nothing left to
take away."
("Il semble que la perfection soit atteinte non quand il n'y a plus rien à
ajouter,
mais quand il n'y a plus rien à retrancher.")  That's what I'm striving for.
My design has the advantage of being easy to implement in any programming
language, even Fortran.  It is also "obvious", with nothing to debate
except the syntax,
because the answer to "Shall we include this or that feature?" is almost
always "No."

Another amendment:  [] is the type definition of a polymorphic array, as {}
is the
type signature of a polymorphic object.

---

From: Pete Cordell <[email-masked]>
Date: Thu, 30 May 2019 15:39:38 +0100

On 30/05/2019 13:53, John Cowan wrote:
> On Wed, May 29, 2019 at 4:24 PM Nico Williams <[email-masked] 

The JCR solution to this type of problem is to be able to have an 
annotation that associates a URI with a primitive type to indicate it's 
internal format. e.g.

    "size" : @{format org.iso.shoe-size} string,

(http://json-content-rules.org/drafts/json-content-rules-10.html#rfc.section.6.11.6)

As far as JCR is concerned, this merely captures the format of the data. 
Whether it's the low-level validators job to validate the type or 
higher-layer functionality is a decision for the individual 
implementation's developer(s).

Same with strings specified with regular expressions. The validator can 
just treat it as a string and leave it to a higher layer, or it can be a 
deluxe validator and check the pattern matches.  (The latter might be 
more useful for a generic on-the-wire monitoring type scenario.)

> (functional) programming language.

We had a play with that sort of thing. Mainly as a proof-of-concept that 
the extensibility enabled by JCR's feature had sufficient mileage in it.

http://json-content-rules.org/drafts/draft-cordell-jcr-co-constraints-00.txt

> that have a combinator rather than a string representation of

In addition to code, I think a key 'parser' of schemas is human 
developers. That's why we wanted JCR compact and concise. If the human 
developer is able to immediately see the constraints in the schema, and 
not have to wade through pages of prose, I think things are likely to 
work much better.

Pete.

---

From: Nico Williams <[email-masked]>
Date: Thu, 30 May 2019 13:16:51 -0500

> Agreed,, because statically typed programming languages can't cope.

And dynamic typing sucks.

> which is either an integer (in a certain range) or an integer plus 0.5?

You start with what you need and add new constraints as you discover
they would be useful.

> (functional) programming language.

ASN.1 has SDL, but no one uses it.  It turns out that SEQUENCE OF/
SEQUENCE/CHOICE is enough to avoid having to add an actual programming
language to do much more validation, with "business logic" getting
written as natural language text in a spec.

Mind you, I wouldn't mind a language for this, but it's a lot of work to
expect the IETF to do, so I wouldn't bother.  If there were enough
interest (there won't be), I suppose we could take inspiration from jq
(which is a functional programming language built around JSON) and write
an RFC for it or a language inspired by it.

> in some such language anyway.  So we might as well decode all numbers

Exactly.

> range anyway, and leave subtypes of number to general-purpose validation.

I wasn't proposing that numerics get decoded as whatever the schema
says.  Rather, that if you have a stack that uses a generic JSON decoder
then a validation pass might validate and coerce numeric values to
integers/whatever as needed.

> that have a combinator rather than a string representation of

It's perfectly fine for us to decide we don't need that feature.

> 0 (equivalently 0.0) to designate an integer and 0.5 to designate a float.)

Again, if we have consensus for such a contraint, we should include it,
and if not, not.  I was not making a detailed proposal, just proposing
that a schema language should be built around types and user-defined
types -- just like ASN.1.

> with a BNF-style syntax for practical use, like RNG compact syntax.

The point about defining types is that you can express recursive
nesting, which is harder to do with by-example schemas.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Thu, 30 May 2019 23:53:48 -0400

> And dynamic typing sucks.

I certainly don't think so.  Not that I think static typing sucks either.
Dynamically typed languages are statically typed languages with only one
static type.

> You start with what you need and add new constraints as you discover

Okay, I'm proposing starting with *no* scalar constraints, only constraints
on the types of array elements and object elements.

> and if not, not.

It's needed because essentially all statically typed languages treat ints
and floats as disjoint types.  That's not a good thing, but we are stuck
with it de facto.

> nesting, which is harder to do with by-example schemas.

Without conditional types, there can be no finite recursion.

---

From: Nico Williams <[email-masked]>
Date: Thu, 30 May 2019 22:53:03 -0500

> on the types of array elements and object elements.

I can live with that.

> and floats as disjoint types.  That's not a good thing, but we are stuck

JSON doesn't distinguish between "ints" and "floats", so if you want
that distinction then you're asking for constraints on the numeric
scalar type.

> Without conditional types, there can be no finite recursion.

Conditional here means "may be present" vs "must be present".  That's
obviously necessary.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Fri, 31 May 2019 12:02:54 -0400

On Thu, May 30, 2019 at 11:57 PM Nico Williams <[email-masked]>
wrote:

> that distinction then you're asking for constraints on the numeric

Okay, just one constraint.  It's a concession to the behavior of statically
typed
languages right back to Fortran ("God is real unless explicitly declared
integer").
Leaving that to inner layers means that everything has to be deserialized
into floats
and then deep-copied into objects whose fields are integers.

> obviously necessary.

I don't mean that kind of conditional. I mean alternative, where an object
value or array element can be of type "foo" or type "bar".  If that's not
allowed
(as it is not in my proposal), then recursive schemas can only be satisfied
by infinitely large JSON values.

John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
weirdo:    When is R7RS coming out?
Riastradh: As soon as the top is a beautiful golden brown and if you
stick a toothpick in it, the toothpick comes out dry.

---

From: Nico Williams <[email-masked]>
Date: Fri, 31 May 2019 13:15:05 -0500

On Fri, May 31, 2019 at 12:02:54PM -0400, John Cowan wrote:
> Okay, just one constraint.  [...]

:)

> allowed (as it is not in my proposal), then recursive schemas can only
> be satisfied by infinitely large JSON values.

That's a union / discriminated union.  (CHOICE, in ASN.1.)

There's been nothing really new in this field since ASN.1...

XDR [RFC4506] is akin to a greatly simplified ASN.1, so it's much more
accessible and relevant to JSON.

Unfortunately XDR conflates "syntax" for specifying schema and "encoding
rules", and for JSON we only need "syntax" for specifying schema.  But
we could read RFC4506 for just the syntax and I think that might prove
useful here.

Nico
--