Subject: [Json] JSON "Number" interoperability issues

From: Anders Rundgren <[email-masked]>
Date: Sat, 21 Apr 2018 17:14:16 +0200

One might believe that JSON is a done deal but if you scratch a bit on the surface it seems that this is not the case.

The root of the problem is that JSON offers a very limited set of data types, poorly matching applications.
The most problematic data type is the Number type which currently is {ab}used in various ways.

The Open API/JSON Scheme folks claim that Number should be used for everything that smells like a number including BigInteger, while Microsoft in their .NET implementation serializes a BigInteger as: "myBigInteger":{"_bits":[1156841472,1164153218],"_sign":1}.  Other implementations use everything from Base64-encoding to simply putting such numbers within quotes.

The same issue is valid for the monetary (decimal) data type which in payment-related standards like Open Banking in the UK and the W3C PaymentRequest is expressed as a JSON String "nnnnn.nnn", but in Open API/JSON Schema as a JSON Number.

That is, for anything beyond what you can do natively in JavaScript, JSON offers close to ZERO interoperability.

I-JSON [https://tools.ietf.org/html/rfc7493] was a good start but it seems that there is more to do or is the general feeling that it is better to slowly deprecate JSON in favor of CBOR [https://tools.ietf.org/html/rfc7049] and/or Protocol Buffers [https://developers.google.com/protocol-buffers/] ?

Personally, I don't see any fundamental issue using JSON since you can map any data type to the String type.  This is way more flexible than defining hordes of discrete data types (aka "Fixing JSON"). Mapping can be performed programmatically like obj.getBigInteger("myBigInteger") or through declarative schemes. However, without defined mapping schemes you don't get that far.

WDYT?

//anders

---

From: Henry Andrews <[email-masked]>
Date: Sat, 21 Apr 2018 09:14:19 -0700

The JSON Schema project says no such thing.  JSON Schema says that if you
want to define a sort of number that is *allowed by the JSON specification*
but not interoperable by whatever definition of "interoperable" for
whatever reason, then that is an ambiguity inherited from JSON and we will
not exclude it from JSON Schema.

We do, however, provide many tools for constraining JSON numbers or
numbers-as-strings within an interoperable subset.  I have been asking
(including in conversations on issues you filed) for someone to contribute
PRs on numeric / numbers-as-strings formats, but neither you nor anoyone
else has stepped up.  Here is an example where I last asked for
contributions in September 2017:
https://github.com/json-schema-org/json-schema-spec/issues/361

I am aware that you are unhappy with our response to your issue, but please
do not blame JSON Schema for JSON's ambiguities, or for other people
happening to use ambiguous values while they also happen to use JSON
Schema.  We help people describe how they want to use JSON.  We don't force
them to use it in a particular way.  If want to avoid such values, it is
entirely possible to write a JSON Schema to filter them out, and would be
more possible if anyone understanding the concerns here were willing to
submit a PR to define better values for "format".

thanks,
-henry

On Sat, Apr 21, 2018 at 8:14 AM, Anders Rundgren <
[email-masked]> wrote:

> One might believe that JSON is a done deal but if you scratch a bit on the
> surface it seems that this is not the case.
>
> The Open API/JSON Scheme folks claim that Number should be used for
> everything that smells like a number including BigInteger, while Microsoft
> in their .NET implementation serializes a BigInteger as:
> "myBigInteger":{"_bits":[1156841472,1164153218],"_sign":1}.  Other
> implementations use everything from Base64-encoding to simply putting such
> numbers within quotes.
>
> The same issue is valid for the monetary (decimal) data type which in
> payment-related standards like Open Banking in the UK and the W3C
> PaymentRequest is expressed as a JSON String "nnnnn.nnn", but in Open
> API/JSON Schema as a JSON Number.
>
> I-JSON [https://tools.ietf.org/html/rfc7493] was a good start but it
> seems that there is more to do or is the general feeling that it is better
> to slowly deprecate JSON in favor of CBOR [https://tools.ietf.org/html/r
> fc7049] and/or Protocol Buffers [https://developers.google.com
> /protocol-buffers/] ?
>
> Personally, I don't see any fundamental issue using JSON since you can map
> any data type to the String type.  This is way more flexible than defining
> hordes of discrete data types (aka "Fixing JSON"). Mapping can be performed
> programmatically like obj.getBigInteger("myBigInteger") or through
> declarative schemes. However, without defined mapping schemes you don't get
> that far.
>
> WDYT?
>
> //anders
>
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 

   -

   *Henry Andrews*  |  Systems Engineer
   [email-masked]
   <https://www.cloudflare.com/>

   1 888 99 FLARE  |  www.cloudflare.com
   -

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 21 Apr 2018 18:35:22 +0200

Dear Henry,

As seen by the .NET example I'm pointing to a rather generic problem.
I don't know who "owns" it.

[email-masked] may be an appropriate venue for discussing how this could/should be addressed.

Ideally that should also involve people designing declarative schemes based on "decorating" data objects like in .NET.

For the monetary (decimal) data type there are already some standards out there that on their own seem have come to the same conclusion including the W3C: https://www.w3.org/TR/payment-request/#paymentcurrencyamount-dictionary

Thanx,
Anders

On 2018-04-21 18:14, Henry Andrews wrote:
> The JSON Schema project says no such thing.  JSON Schema says that if you want to define a sort of number that is *allowed by the JSON specification* but not interoperable by whatever definition of "interoperable" for whatever reason, then that is an ambiguity inherited from JSON and we will not exclude it from JSON Schema.
> 
> We do, however, provide many tools for constraining JSON numbers or numbers-as-strings within an interoperable subset.  I have been asking (including in conversations on issues you filed) for someone to contribute PRs on numeric / numbers-as-strings formats, but neither you nor anoyone else has stepped up.  Here is an example where I last asked for contributions in September 2017: https://github.com/json-schema-org/json-schema-spec/issues/361
> 
> I am aware that you are unhappy with our response to your issue, but please do not blame JSON Schema for JSON's ambiguities, or for other people happening to use ambiguous values while they also happen to use JSON Schema.  We help people describe how they want to use JSON.  We don't force them to use it in a particular way.  If want to avoid such values, it is entirely possible to write a JSON Schema to filter them out, and would be more possible if anyone understanding the concerns here were willing to submit a PR to define better values for "format".
> 
> thanks,
> -henry
> 
> 
> 
> 
> On Sat, Apr 21, 2018 at 8:14 AM, Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     I-JSON [https://tools.ietf.org/html/rfc7493 <https://tools.ietf.org/html/rfc7493>] was a good start but it seems that there is more to do or is the general feeling that it is better to slowly deprecate JSON in favor of CBOR [https://tools.ietf.org/html/rfc7049 <https://tools.ietf.org/html/rfc7049>] and/or Protocol Buffers [https://developers.google.com/protocol-buffers/ <https://developers.google.com/protocol-buffers/>] ?
> 
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/json <https://www.ietf.org/mailman/listinfo/json>
> 
> 
> 
> 
> -- 
> 
>   *
>     |
> 
>     [email-masked] <mailto:[email-masked]>
> 
>     1 888 99 FLARE  | www.cloudflare.com <https://www.cloudflare.com/>
> 
>     |
> *
>

---

From: Tim Bray <[email-masked]>
Date: Sat, 21 Apr 2018 11:21:46 -0700

Well, the story, per RFC8259, is pretty clear. JSON numbers are safely
usable inside the numeric universe that's covered by IEEE 754
double-precision.  Fortunately, that covers probably a large majority of
all use-cases for numbers in computer programming.  If you need to go
outside that universe, JSON numbers don't help you.

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Henry Andrews <[email-masked]>
Date: Sat, 21 Apr 2018 16:47:41 -0700

Indeed.  I'm not sure where the "JSON Schema and OpenAPI claim that..."
comes from.  OpenAPI has an int64 format, presumably because someone has a
use for it.  I assume it's interoperable enough for that use case, but it
does not magically extend JSON's interoperable numeric range.  No one is
being forced to use that format, though.

The outcome of those discussions was that neither JSON Schema (which does
not define an int64 format, but does not forbid adding one) nor OpenAPI
thought it necessary to forbid such a construct since clearly someone was
getting some use out of it.  That's all.  There's nothing preventing anyone
who is using OpenAPI or JSON Schema from simply... not using that format.
Or defining a format that encodes very large numbers in JSON Strings.  As
noted, we're open to improving support for string-encoded numbers on the
JSON Schema side if someone would like to submit a PR.

This is not even the only weird interoperability edge in JSON.  Similar
concerns would arise if someone came up with an extension keyword for JSON
Schema to describe objects with multiple properties of the same name. This
is allowed by RFC 8259, which also notes that behavior of software which
encounters such objects is unpredictable.  I would personally not put such
a keyword into JSON Schema as it would be confusing for many, but if a
project (OpenAPI or otherwise) wanted to add one because it works for
them... it's allowed by RFC 8259 and it's not the responsibility of the
schema system to prevent it.

thanks,
-henry

On Sat, Apr 21, 2018 at 11:21 AM, Tim Bray <[email-masked]> wrote:

> double-precision..  Fortunately, that covers probably a large majority of
>> One might believe that JSON is a done deal but if you scratch a bit on
>> the surface it seems that this is not the case.
>>
>> Personally, I don't see any fundamental issue using JSON since you can
>> map any data type to the String type.  This is way more flexible than
>> defining hordes of discrete data types (aka "Fixing JSON"). Mapping can be
>> performed programmatically like obj.getBigInteger("myBigInteger") or
>> through declarative schemes. However, without defined mapping schemes you
>> don't get that far.
>>
>> WDYT?
>>
>> //anders
>>
>>

-- 

   -

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 22 Apr 2018 07:07:47 +0200

On 2018-04-21 20:21, Tim Bray wrote:
> Well, the story, per RFC8259, is pretty clear. JSON numbers are safely usable inside the numeric universe that's covered by IEEE 754 double-precision.  Fortunately, that covers probably a large majority of all use-cases for numbers in computer programming.  If you need to go outside that universe, JSON numbers don't help you.

Right, but if you need to go outside of that universe including using a "long" integer, you'll find that the most implementations by default indeed ignore this limitation which BTW is neither a SHOULD nor a MUST.

That is, a (for practical purposes) pretty major portion of the JSON specification have effectivaly moved into the application space.

Anders

> - Tim Bray (If you’d like to send me a private message, see https://keybase.io/timbray)

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 22 Apr 2018 08:09:29 +0200

On Apr 22, 2018, at 01:47, Henry Andrews <[email-masked]> wrote:
> 
> objects with multiple properties of the same name. This is allowed by RFC 8259, which also notes that behavior of software which encounters such objects is unpredictable.

It is “allowed” by RFC 8259 approximately as much as my parents “allowed” me to jump off a bridge by telling me that would be bad for my health.

Grüße, Carsten

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 22 Apr 2018 12:27:11 +0200

Since the JSON Number universe outside of IEEE-754 double precision has been left to application/platform developers to deal with [*], there are currently multiple incompatible (or incomplete) "standards" out there:
https://github.com/OAI/OpenAPI-Specification/issues/845#issuecomment-378139730
https://github.com/javaee/jsonb-spec/blob/master/spec/spec.pdf

For .NET I haven't been able obtaining any public information on this topic, so I wrote some code in order to find out what it actually does.

Maybe JSON Schema could serve a suitable place for defining a [potential] "real/de-facto" standard?

To make such a process even smother, I could imagine moving http://json-schema.org/latest/json-schema-validation.html#rfc.section.7.3 into a separate document but I haven't looked into the consequences of that.

Anders

*] https://www.ietf.org/mail-archive/web/json/current/msg04294.html

---

From: Henry Andrews <[email-masked]>
Date: Sun, 22 Apr 2018 11:21:27 -0700

JSON Schema provides tools to describe how people use JSON documents,
including imposing structural constraints.  While you could write a schema
describing the subset of JSON that you wish to use as your "standard", that
would be a *use* of JSON Schema, not *part* of JSON Schema.

Anders, you seem to want JSON Schema to prevent people from using things
that are technically allowed by the spec (as Carsten noted, "allowed" in
the sense that no one will stop you, not in the sense that any reasonable
person would expect a positive outcome).  That is not what JSON Schema, *as
a project*, is for.  You are more than welcome to use JSON Schema in your
own project to advocate for a more interoperable subset of JSON.

The section of JSON Schema's validation specification that you reference,
about the values for "format", has gotten some discussion around how to
manage larger extensibility question such as defining a registry.  The
issue for that discussion is
https://github.com/json-schema-org/json-schema-spec/issues/563

Furthermore, this email list is not the right forum for discussing changes
to JSON Schema.  This group has previously made it pretty clear that they
are at most interested in JSON Schema as one of many possible starting
points for a new effort, rather than developing JSON Schema itself into a
standards track RFC.  I have no objection to that- the IETF is welcome to
use or not use our work in whatever way makes sense.  I've never gotten
anything to RFC myself, and wouldn't presume to know all of the
considerations involved.  I'll be happy to support anyone who wants to use
any JSON Schema ideas in a new effort, but if we're talking about JSON
Schema as it is, that's better done through our repo or slack channel.

thanks,
-henry

[email-masked]
[email-masked]

On Sun, Apr 22, 2018 at 3:27 AM, Anders Rundgren <
[email-masked]> wrote:

> Since the JSON Number universe outside of IEEE-754 double precision has
> been left to application/platform developers to deal with [*], there are
> currently multiple incompatible (or incomplete) "standards" out there:
> For .NET I haven't been able obtaining any public information on this
> topic, so I wrote some code in order to find out what it actually does.
>
> To make such a process even smother, I could imagine moving
> http://json-schema.org/latest/json-schema-validation.html#rfc.section.7.3
> into a separate document but I haven't looked into the consequences of that.
>
> Anders
>

-- 

   -

---

From: Anders Rundgren <[email-masked]>
Date: Mon, 23 Apr 2018 08:25:25 +0200

I'm just saying that the JSON community has interoperability issues:
https://github.com/javaee/jsonb-spec/issues/80#issuecomment-383465639

I wouldn't characterize interoperable solutions as JSON "subsets".
 From an application writer's point of view OpenAPI is more of a subset since it out-of-the-box doesn't support BigInteger which Java and .NET do.

Anders

On 2018-04-22 20:21, Henry Andrews wrote:
> JSON Schema provides tools to describe how people use JSON documents, including imposing structural constraints.  While you could write a schema describing the subset of JSON that you wish to use as your "standard", that would be a *use* of JSON Schema, not *part* of JSON Schema.
> 
> Anders, you seem to want JSON Schema to prevent people from using things that are technically allowed by the spec (as Carsten noted, "allowed" in the sense that no one will stop you, not in the sense that any reasonable person would expect a positive outcome).  That is not what JSON Schema, *as a project*, is for.  You are more than welcome to use JSON Schema in your own project to advocate for a more interoperable subset of JSON.
> 
> The section of JSON Schema's validation specification that you reference, about the values for "format", has gotten some discussion around how to manage larger extensibility question such as defining a registry.  The issue for that discussion is https://github.com/json-schema-org/json-schema-spec/issues/563
> 
> Furthermore, this email list is not the right forum for discussing changes to JSON Schema.  This group has previously made it pretty clear that they are at most interested in JSON Schema as one of many possible starting points for a new effort, rather than developing JSON Schema itself into a standards track RFC.  I have no objection to that- the IETF is welcome to use or not use our work in whatever way makes sense.  I've never gotten anything to RFC myself, and wouldn't presume to know all of the considerations involved.  I'll be happy to support anyone who wants to use any JSON Schema ideas in a new effort, but if we're talking about JSON Schema as it is, that's better done through our repo or slack channel.
> 
> thanks,
> -henry
> 
> [email-masked] <mailto:[email-masked]>
> 
> 
> On Sun, Apr 22, 2018 at 3:27 AM, Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     https://github.com/OAI/OpenAPI-Specification/issues/845#issuecomment-378139730 <https://github.com/OAI/OpenAPI-Specification/issues/845#issuecomment-378139730>
>     https://github.com/javaee/jsonb-spec/blob/master/spec/spec.pdf <https://github.com/javaee/jsonb-spec/blob/master/spec/spec.pdf>
> 
>     To make such a process even smother, I could imagine moving http://json-schema.org/latest/json-schema-validation.html#rfc.section.7.3 <http://json-schema.org/latest/json-schema-validation.html#rfc.section.7.3> into a separate document but I haven't looked into the consequences of that.
> 
>     Anders
> 
>     *] https://www.ietf.org/mail-archive/web/json/current/msg04294.html <https://www.ietf.org/mail-archive/web/json/current/msg04294.html>
> 
> 
> 
> 
> -- 
> 
>   *
>     |
>