Subject: [Json] JSON Schema Language: extensibility and unspecified properties

From: Ulysse Carion <[email-masked]>
Date: Fri, 9 Aug 2019 18:43:37 -0700

Hi folks,

Thanks again for all the wonderful suggestions for JSON Schema
Language. I've just published a new draft incorporating suggestions
from the last update. You can read it here:

https://tools.ietf.org/html/draft-ucarion-json-schema-language-02

I would like advice on the two following changes I've made to the spec.

First, I added explicit discussion of how JSL can be extended. The
language is at the end of section 2, but I'll reproduce it here:

> This document does not describe any extension mechanisms for JSL schema
> validation. However, schemas (through the `non-keyword` CDDL rule in this
> section) are defined to allow members whose names are not equal to any of the
> specially-defined keywords (i.e. `definitions`, `elements`, etc.) described in
> this section. Call these members "non-keyword members".
>
> Users MAY add additional, non-keyword members to JSL schemas to convey
> information that is not pertinent to validation. For example, such non-keyword
> members could provide hints to code generators, or trigger some special behavior
> for a library that generates user interfaces from schemas.
>
> Users SHOULD NOT expect non-keyword members to be understood by other parties.
> As a result, if consistent validation with other parties is a requirement, users
> SHOULD NOT use non-keyword members to affect how schema validation, as described
> in {{semantics}}, works.

Basically, my stance is: you can add custom stuff to schemas, but you
can't change validation. I expect that people will change validation
anyway, but they're not going to expect portability. I don't feel that
adding some sort of registry of keywords / forms is valuable yet.
Perhaps something like that can be in whatever replaces JSL?

How do folks feel about this? Does anyone have suggestions on better
ways I can approach extensibility, or how I should phrase this
language?

The other thing I want advice on is "strict" mode, called "strict
instance semantics" in the spec. Basically, strict instance semantics
about whether it's ok to send "extra"/"unknown"/"unspecified"
properties in an object.

Right now, validation behavior is specified as a single, top-level
"strict": true/false (default is "true") property. You can't mix and
match strict and non-strict. But Carsten off-list noted that might be
a bit "blunt". Do others agree?

The context for my decision is that two of major typed languages used
in industry, Java and Golang, don't typically support mixing and
matching. Go just as a blunt DisallowUnknownFields:

https://golang.org/pkg/encoding/json/#Decoder.DisallowUnknownFields

And Java's Jackson has a FAIL_ON_UNKNOWN_PROPERTIES:

http://fasterxml.github.io/jackson-databind/javadoc/2.9/com/fasterxml/jackson/databind/DeserializationFeature.html#FAIL_ON_UNKNOWN_PROPERTIES

Both are all or nothing.

Also, there is the question of whether "strict" is a good name for
this property in schemas. Is there a better name for it? "strict" can
be understood in many ways. Open to ideas here as well.

Other changes:

- Removed int64/uint64 (and removed discussion related to I-JSON)
- Clarified that 1.0e1 is an integer

Finally, I've moved JSL to a desired status of "Informative", and I'll
be going through the independent stream once the discussion here
appears to have settled.

Thanks again for all the help,
Ulysse

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 10 Aug 2019 08:23:33 +0200

Examples are good:

   This schema […] limits “generated” and
   "expires" to timestamps less than 2^32 seconds after January 1, 1970
   00:00 UTC. 

This is a nice demonstration that it was a mistake to remove uint53.

(Note that on 32-bit systems time_t is an int32.  But you definitely don’t want that restriction; 2038 is coming too close now.  Uint32 is completely arbitrary and wrong in many ways.)

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 10 Aug 2019 21:00:46 +0200

If we stick to unspecified properties I'm currently working with a
payment authorization scheme where some components will be defined
by the payment networks.  The components start with a defined property
holding a JSON Object but the contents of the object is not known by
the "main" schema.

In my programmatic solution the "main schema" ignores such components
and only validates that they conform JSON.  The components are
then dealt with by customer supplied decoders which indeed may use
a schema.

How can you express this in a main schema using JCL?

Anders

   paymentRequest = new PaymentRequest(rd.getObject(PAYMENT_REQUEST_JSON));
   undecodedPaymentMethodSpecific = rd.getObject(BACKEND_METHOD_SPECIFIC_JSON);
   rd.scanAway(BACKEND_METHOD_SPECIFIC_JSON);
   referenceId = rd.getString(REFERENCE_ID_JSON);

On 2019-08-10 03:43, Ulysse Carion wrote:
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Henry Andrews <[email-masked]>
Date: Sat, 10 Aug 2019 19:55:37 +0000 (UTC)

I know folks here are not interested in JSON Schema itself, but i will point out that we have spent most of the past year on exactly this problem.  JSON Schema vocabularies are identified by URIs, which implementations can use to load extension libraries.  The keyword taxonomy we have developed classifies keywords into several sorts of high-level behaviors (primarily assertions, which produce boolean validation outcomes, annotations, which attach information for applications to consume, and applicators, which apply subschemas and potentially combine or modify their results), allowing extensible JSON Schema implementations to provide hooks for handling those basic types of keywords.
This work is the primary focus of the new draft, which will be published by the end of August.  Here are some links to the vocabulary- and taxonomy-related sections in the work-in-progress review copies:
http://json-schema.org/work-in-progress/WIP-jsonschema-core.html#rfc.section.6.5
http://json-schema.org/work-in-progress/WIP-jsonschema-core.html#rfc.section.7
http://json-schema.org/work-in-progress/WIP-jsonschema-core.html#rfc.section.8
   
thanks,-henry
 On Saturday, August 10, 2019, 12:01:20 PM PDT, Anders Rundgren <[email-masked]> wrote:  

Anders

---

From: Ulysse Carion <[email-masked]>
Date: Sun, 11 Aug 2019 10:33:50 -0700

> This is a nice demonstration that it was a mistake to remove uint53.

Perhaps. But perhaps instead I should just use {"type": "number"}, and
accept that JSL isn't going to be able to express certain things about
numbers?

Re-reading https://tools.ietf.org/html/rfc7071, it seems it constrains
the timestamps to not have a fractional part at the JSON syntax level.
That's another thing JSL can't deal with. But I'm not sure if in
practice that's such a big deal.

---

From: Ulysse Carion <[email-masked]>
Date: Sun, 11 Aug 2019 10:37:43 -0700

> How can you express this in a main schema using JCL?

I'm not entirely sure I follow your description. Could you illustrate
it with some example JSON values? I'm also a bit unclear on your
requirements for the "main schema", and how it fits with requirements
around "unspecified properties".

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 11 Aug 2019 22:06:26 +0200

> accept that JSL isn't going to be able to express certain things about

Sure, that is one way of handling it.
However, it seems bizarre to support int8, int16, and int32, but not JSON’s generic interoperable integers.

> the timestamps to not have a fractional part at the JSON syntax level.

Oops.
(I’d consider this an erratum, though, as Section 2.4 of RFC 4627 does not define “integer”.)

> That’s another thing JSL can't deal with.

Actually, that’s something JSON can’t deal with.

> But I'm not sure if in
> practice that’s such a big deal.

MS-Excel has repeatedly taught me that my university phone number is 2.1863921e7, but people still manage to call me :-)

Grüße, Carsten

---

From: Ulysse Carion <[email-masked]>
Date: Mon, 12 Aug 2019 22:03:55 -0700

> However, it seems bizarre to support int8, int16, and int32, but not JSON’s generic interoperable integers.

Supposing we add int53 to JSL, do you picture code generators
producing int64_t/long for {"type": "int53" }? Does that mean that,
before serializing an int64_t to something marked as int53 in JSL, the
application must first do an extra bounds check? Today, it's fairly
easy to generate code from JSL where serializing is an infallible
affair. But with int53, that property would be lost, because most type
systems cannot express int53.

I'm inclined to think this is something better handled by extensions.
Perhaps someone can define a "intt53" property to do something like:

{ "type": "number", "int53": true }

The person writing the extension would document that the "int53"
property indicates whether a number is meant to represent a number in
the I-JSON range. Applications which don't understand this keyword
will still do something reasonable -- validate for some sort of
number, and code generate a "double" -- but applications which care
about this can handle this case specially. It also signals an intent
about how the number will be used.

I expect this sort of approach is how JSL may need to handle
big-number encoding schemes. There are so many different ways approach
that problem, and I think JSL is most useful if it establishes an
uncontroversial foundation, and then lets additional, out-of-spec
keywords tighten the schema further in a way which most folks don't
care about, and hence don't want to have to implement.

Does that seem like a reasonable approach?

> Actually, that’s something JSON can’t deal with.

By this you mean that JSON prescribes "all numbers are doubles", and
so integers aren't really a good thing to try to foist onto JSON's
syntax?

> MS-Excel has repeatedly taught me that my university phone number is 2.1863921e7, but people still manage to call me :-)

Could you expand what you mean here? I realize no joke is funny enough
to survive explanation, but I'm afraid I'm perhaps missing your point.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 13 Aug 2019 07:31:47 +0200

> systems cannot express int53.

Many type systems don’t have a uint8, either.

> about how the number will be used.

Again, the same is true for uint8.

(This opens the whole “types vs. subtypes” discussion.)

> so integers aren't really a good thing to try to foist onto JSON's

Integer is a mathematical concept.  JSON does not have a problem with that.

The problem comes in when applications arbitrarily restrict the syntax.

Not allowing a fractional or exponent part is akin to requiring two spaces of indentation before or a newline after a number.

It gets worse when syntax variations are assigned different application semantics.
(E.g., I’m aware of at least one “JSON-based” syntax where a string that starts with “\u0073” has semantics different from a string that starts with “s”.)

You can’t do that to a format like JSON and still be part of the JSON ecosystem, because generic JSON decoders discard these syntax features, and generic JSON encoders generally can’t produce them.

> to survive explanation, but I’m afraid I'm perhaps missing your point.

Well, my phone number is 21863921, but some spreadsheets consider this a large number and turn it into an NR3 number…

Grüße, Carsten

---

From: Ulysse Carion <[email-masked]>
Date: Sat, 17 Aug 2019 18:34:58 -0700

Carsten, I think you are bringing up an important point. I should
clarify what is and isn't a type in the spec, and what the principle
is for deciding that.

I'd like to add a brief discussion of that reasoning in the
introduction of the next draft. Does this principle make sense to
folks here?

"Only include in the spec things which are (1) commonly (and portably)
used in the JSON ecosystem, and (2) which have a clear mapping to
programming languages in widespread use today."

By that standard, int64/uint64 fails (1), as it's unportable (i.e.,
not I-JSON) or implemented in a bunch mutually incompatible methods.
int53 fails (2) because no mainstream language I know of has support
for integers strictly up to 2^53.

I disagree with you about uint8. I think we agree it satisfies (1); it
seems we disagree on whether it passes (2)? Perhaps you were thinking
of Java's strictly-signed "byte". If so: by convention, Java folks
ignore the signed-ness of byte when uint8 is needed. This is perhaps a
bit sketchy (since you can so easily mix and match signed and unsigned
8-bit numbers), but it has gotten the job done in many applications.
So the spec's uint8 and int8 both correspond to Java's byte, in my
view.

On the principle above, I think I should remove {"type": "number"}
from the spec. It's unclear what "number" means. Does it mean
BigDecimal? If so, it's not portable and so fails (1). Does it mean
float64? Then just use {"type": "float64"} instead.

---

From: Ulysse Carion <[email-masked]>
Date: Sat, 17 Aug 2019 18:55:46 -0700

Also: from off-list discussion, and from talking with folks in
industry about their experiences taking legacy systems and
retrofitting schemas onto them, I think the spec's current approach to
"unexpected/unspecified/additional" properties is inadequate.

In the current draft, whether to tolerate properties not mentioned in
a schema is a global option on the schema. This is not good enough,
because oftentimes folks only progressively lock down parts of their
schema -- some parts are better documented and can be locked down, but
other parts may be inscrutable or just too risky to lock down.

So I'm thinking, in the next draft, for schemas -- be they root
schemas or sub-schemas -- to be able to specify whether they'll allow
properties outside those mentioned in properties/optionalProperties.
Also, since "strict" is super vague, I'll rename it
"additionalProperties". It'll still take true/false, but now the
default is "false" and the value's meaning is flipped.

---

From: Richard Gibson <[email-masked]>
Date: Sun, 18 Aug 2019 21:32:13 -0400

On Sat, Aug 17, 2019 at 9:35 PM Ulysse Carion <[email-masked]> wrote:

> no mainstream language I know of has support for integers strictly up to
> 2^53
>

>From a certain point of view, nearly all mainstream languages do so,
because that is the maximum value in the range of contiguous IEEE 754
binary64 integers and thus the upper bound on portable integers (which is
why RFC 7493 section 2.2 limits I-JSON numbers to being strictly less than
it). But you're right that languages have no int53-specific type, because
those with specific integer types have at least the larger range of int64
and those without them have the larger reduced-precision range of binary64.
But is the portability argument not sufficient for including such a
constraint, as it was for I-JSON?

---

From: Ulysse Carion <[email-masked]>
Date: Mon, 19 Aug 2019 13:46:40 -0700

> But is the portability argument not sufficient for including such a constraint, as it was for I-JSON?

What constraint are you referring to here?
On Sun, Aug 18, 2019 at 6:32 PM Richard Gibson <[email-masked]> wrote:
>> no mainstream language I know of has support for integers strictly up to 2^53
>
>
> From a certain point of view, nearly all mainstream languages do so, because that is the maximum value in the range of contiguous IEEE 754 binary64 integers and thus the upper bound on portable integers (which is why RFC 7493 section 2.2 limits I-JSON numbers to being strictly less than it). But you're right that languages have no int53-specific type, because those with specific integer types have at least the larger range of int64 and those without them have the larger reduced-precision range of binary64. But is the portability argument not sufficient for including such a constraint, as it was for I-JSON?

---

From: Richard Gibson <[email-masked]>
Date: Mon, 19 Aug 2019 20:51:54 -0400

An "int53" type or other means of requiring that a JSON number have zero
fractional part and magnitude no greater than 2^53.

---

From: Ulysse Carion <[email-masked]>
Date: Mon, 19 Aug 2019 20:26:15 -0700

> An "int53" type or other means of requiring that a JSON number have zero fractional part and magnitude no greater than 2^53.

Ah -- in that case, I think it fails the other part of the criterion I
put forward: having "a clear mapping to programming languages in
widespread use today." In case the language is unclear, my thinking
was that both (1) and (2) need to be satisfied for something to be
included in the spec.

At least on my view, int53's correspondence is unclear -- should it be
a double/f64, or a long/int64_t, with a runtime check to make sure
it's in-bounds? That's not very idiomatic. Better, in my view, to
stick to uncontroversial things?

On Mon, Aug 19, 2019 at 5:52 PM Richard Gibson <[email-masked]> wrote:
>
> An "int53" type or other means of requiring that a JSON number have zero fractional part and magnitude no greater than 2^53.
>

---

From: Pete Cordell <[email-masked]>
Date: Tue, 20 Aug 2019 12:54:02 +0100

> stick to uncontroversial things?

IMO, the schema should be about representing what the protocol needs 
rather than how it is represented on a target machine.

An implementation may choose to store an int8 in a 32-bit integer. As 
long as it does suitable bounds checking at the appropriate points 
nothing outside the machine need know that it has chosen to do that.

There may be cases where there is a need for something that is larger 
than int32 but more inter-operable than int64. int53 satisfies that need 
in many cases.

Mapping it to either double/f64 or long/int64_t is a local 
implementation choice.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Richard Gibson <[email-masked]>
Date: Tue, 20 Aug 2019 10:34:42 -0400

Exactly. binary64 is an obvious option since it is the source of the
bounds, but int64 is equally capable of fulfilling the role. Just like int8
imposes constraints on a value without requiring any particular
machine-local treatment or representation (which should remain irrelevant
to the schema language itself), so too would int53.

On Tue, Aug 20, 2019 at 7:54 AM Pete Cordell <[email-masked]> wrote:

> >> An "int53" type or other means of requiring that a JSON number have
> zero fractional part and magnitude no greater than 2^53.
> >

---

From: Ulysse Carion <[email-masked]>
Date: Tue, 20 Aug 2019 12:05:34 -0700

> rather than how it is represented on a target machine.

This is a reasonable stance. But I don't think it's what I'm trying to
solve for. My focus has continually been on code generation from
schemas; the spec's expressive power is intentionally limited
specifically to enable code generation. Optimizing for code generation
is why I *am* concerning myself with how things are represented on
target machines.

There are lots of good options for schemas languages that are focused
on being as expressive as protocols are complex -- you've written one
of these good options, Pete -- but I've been concerned with making a
schema language as inexpressive as mainstream type systems are simple.

I'm not debating whether there are ways of representing int53 in many
languages. My objection is that I would be making up a type that
doesn't exist in existing programming languages natively or
idiomatically. If someone wants to do int53 bounds checks, they can
always use the float64 type, and then do bounds checking after schema
validation. Or they can write an extension.

On Tue, Aug 20, 2019 at 7:34 AM Richard Gibson <[email-masked]> wrote:
>
> Exactly. binary64 is an obvious option since it is the source of the bounds, but int64 is equally capable of fulfilling the role. Just like int8 imposes constraints on a value without requiring any particular machine-local treatment or representation (which should remain irrelevant to the schema language itself), so too would int53.
>