Subject: Re: [Json] JSON Schema Language is nearly done: int53

From: "Manger, James" <[email-masked]>
Date: Mon, 29 Jul 2019 01:02:36 +0000

> Given that introducing int64 as a JSON number type would be a footgun for many sorts of applications, I agree it should be changed.
>
> How would you feel about simply removing int64/uint64, but otherwise keeping everything else as-is? I don't know if adding "native" support for string-represented bigints is a great idea, as there are so many different options. Smaller specs are the ones most frequently implemented correctly and securely.

Just removing int64/uint64 is ok. I’d prefer to have int53. It covers all integer cases that work sensibly. Other options (int8, unit32, etc) are merely nice-to-have hints that could be omitted for a smaller spec.

--
James Manger

---

From: Ulysse Carion <[email-masked]>
Date: Mon, 29 Jul 2019 21:09:02 -0700

Perhaps int53 is a footgun for reasons similar to int64? Folks in practice
might pretend int53 is int64 for all intents and purposes, incurring the
same problems.

In the interest of keeping the spec small, safe, and unsurprising, I think
simply removing int64/uint64, but preserving the other types (including
less common ones like int8) may be the best option. I'd rather not
arbitrarily leave out powers of two or particular signed/unsigned variants,
nor leave in features which make it likelier that people make faulty
assumptions about interoperable JSON.

On Sun, Jul 28, 2019 at 6:02 PM Manger, James <
[email-masked]> wrote:

> > Given that introducing int64 as a JSON number type would be a footgun
> for many sorts of applications, I agree it should be changed.
>
> >
>
> > How would you feel about simply removing int64/uint64, but otherwise
> keeping everything else as-is? I don't know if adding "native" support for
> string-represented bigints is a great idea, as there are so many different
> options. Smaller specs are the ones most frequently implemented correctly
> and securely.
>
>
>
> Just removing int64/uint64 is ok. I’d prefer to have int53. It covers all
> integer cases that work sensibly. Other options (int8, unit32, etc) are
> merely nice-to-have hints that could be omitted for a smaller spec.
>
>
>
> --
>
> James Manger
>

---

From: "Manger, James" <[email-masked]>
Date: Tue, 30 Jul 2019 04:34:06 +0000

> Perhaps int53 is a footgun for reasons similar to int64? Folks in practice might pretend int53 is int64 for all intents and purposes, incurring the same problems.

You have to be fairly wilful to ignore that “int53” isn’t designed for all 64-bit integers.
And if 1 party ignored that, it is immediately obvious who is “at fault” when interop fails.

How crucial is it that code-gen can create uint8, int32 fields etc? As JSON is the exchange format you aren’t saving any bytes for transmission. I guess you might save a few bytes in memory. I’m not sure how important that is. If it is, those cases don’t have to use code-gen.

Include int8, int16, & int32 and people will want int64 (or extrapolate some implementations to int64).
Include int53 and they’ll know that’s all they get.

--
James Manger

---

From: Pete Cordell <[email-masked]>
Date: Tue, 30 Jul 2019 11:24:38 +0100

On 30/07/2019 05:34, Manger, James wrote:
>> Perhaps int53 is a footgun for reasons similar to int64? Folks in 
> practice might pretend int53 is int64 for all intents and purposes, 
> incurring the same problems.
> 
> How crucial is it that code-gen can create uint8, int32 fields etc? As 
> JSON is the exchange format you aren’t saving any bytes for 
> transmission. I guess you might save a few bytes in memory. I’m not sure 
> how important that is. If it is, those cases don’t have to use code-gen.

int8 etc let's you know that 8 bits is sufficient. It doesn't mean it 
has to be stored in an 8 bit integer.  But it can be if that helps your 
system.

> Include int53 and they’ll know that’s all they get.

IoT is supposedly still on the increase. Some smaller systems do not 
have float types so the one true int53 type doesn't work there.

Allow the specification to capture the information the designers need to 
build their system and let them work out how to meet the requirements.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 31 Jul 2019 07:02:47 +0200

Dropping int64 is hardly an option, neither is BigInteger.   int53 is a special that for example fits UNIX "epoch".

The purpose of fixed sized integers in data formats is not to save bytes, it is about valid ranges and compatibility with languages like Java and C++.

Arbitrarily sized integers are used by tons of applications based on JSON messaging and is available for most platforms including JS.

The monetary type is an item of its own.

That there are multiple ways of formatting integers in JSON is unfortunately a reality all JSON tool maker must (in some way) address.

I would reverse the integer syntax and claim that integers SHOULD conform to standard integer (not JSON) syntax since a standard should not break the multitude of JSON tools that already implement this.
A scheme author who thinks this is "wrong" should use "number" giving them the whole JSON number syntax.

Anders

---

From: "Manger, James" <[email-masked]>
Date: Wed, 31 Jul 2019 06:56:13 +0000

> Dropping int64 is hardly an option, neither is BigInteger.   int53 is a special that for example fits UNIX "epoch".

Dropping int64 or BigInteger from *programming languages* is not an option. But dropping the ability for a JSON schema to suggest that arbitrary values of those types should be serialized to JSON numbers would be sensible.

> Arbitrarily sized integers are used by tons of applications based on JSON messaging and is available for most platforms including JS.

Sure, but they are not serialized to a JSON number. They are serialized to a JSON string of some form (eg "65537.00" or "AQAB").

> That there are multiple ways of formatting integers in JSON is unfortunately a reality all JSON tool maker must (in some way) address.

Are you referring to 123, 123.0, 12.3E+1 and 1.23e2 as the multiple ways? Or 123, "123", "123.00", and "ew==" as the multiple ways?

> I would reverse the integer syntax and claim that integers SHOULD conform to standard integer (not JSON) syntax since a standard should not break the multitude of JSON tools that already implement…

It sounds like you want JSON serializers to never use the fraction or exponent notations when a schema says a field is an integer. That's adding a distinction that doesn't exist in JSON so it will break things. A tool parsing JSON *without knowing the schema* may well use a double for a number, which it may then serialize in exponent notation even if it is an integer. For instance, in Java Double.toString(123000000) returns "1.23E8".
Mind you, I suspect 1.23E8 will break many implementations that expect an integer so perhaps we should require the no-fraction-no-exponent form.

This is a separate issue to this thread, however, which was whether a JSON schema should offer "int53" as a type, and if it should offer "{u}int{8|16|32}" as types.

--
James Manger

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 31 Jul 2019 09:14:22 +0200

> Sure, but they are not serialized to a JSON number. They are serialized to a JSON string of some form (eg "65537.00" or "AQAB").

This is where we (as a community of users) have a problem.  Jackson (now adopted by MSFT) indeed serializes BigInteger as JSON Number.  Other parties like Oracle have taken another (presumable unique) approach by using JSON number for int53-compliant BigIntegers and string notation for BigIntegers that doesn't fits int53.  I believe Oracle are considering a redesign though.

The following lines in Chrome shows this division in clear:

var big = 5n;
JSON.stringify(big)

 > VM887:1 Uncaught TypeError: Do not know how to serialize a BigInt
     at Object.stringify (<anonymous>)
     at <anonymous>:1:6

Anders

---

From: Ulysse Carion <[email-masked]>
Date: Wed, 31 Jul 2019 20:55:26 -0700

Replying to Anders and James at once:

> The purpose of fixed sized integers in data formats is not to save bytes,
it is about valid ranges and compatibility with languages like Java and C++.

Agreed. The numerical types are about writing correct software more easily
in modern programming languages, not about optimizing bytes.

> Dropping int64 or BigInteger from *programming languages* is not an
option. But dropping the ability for a JSON schema to suggest that
arbitrary values of those types should be serialized to JSON numbers would
be sensible.

This is my view as well.

> This is where we (as a community of users) have a problem.  Jackson (now
adopted by MSFT) indeed serializes BigInteger as JSON Number.  Other
parties like Oracle have taken another (presumable unique) approach by
using JSON number for int53-compliant BigIntegers and string notation for
BigIntegers that doesn't fits int53.  I believe Oracle are considering a
redesign though.

It's precisely this sort of complication that motivates dropping numbers
above int32/uint32 from the JSL spec. The spec remains useful without them.

The same goes for monetary data types. There are too many options, and it's
better for the spec to stick to relatively uncontroversial things.

> Mind you, I suspect 1.23E8 will break many implementations that expect an
integer so perhaps we should require the no-fraction-no-exponent form.

This has been discussed in this thread before. Rough consensus suggests
that it's better to simply require a zero fractional part in the number
value, rather than impose constraints at the JSON parser level.

On Wed, Jul 31, 2019 at 12:14 AM Anders Rundgren <
[email-masked]> wrote:

> >> Dropping int64 is hardly an option, neither is BigInteger.   int53 is a
> special that for example fits UNIX "epoch".
> >
> > Dropping int64 or BigInteger from *programming languages* is not an
> option. But dropping the ability for a JSON schema to suggest that
> arbitrary values of those types should be serialized to JSON numbers would
> be sensible.
> >
> >> Arbitrarily sized integers are used by tons of applications based on
> JSON messaging and is available for most platforms including JS.
> >
> > Sure, but they are not serialized to a JSON number. They are serialized
> to a JSON string of some form (eg "65537.00" or "AQAB").
>
> This is where we (as a community of users) have a problem.  Jackson (now
> adopted by MSFT) indeed serializes BigInteger as JSON Number.  Other
> parties like Oracle have taken another (presumable unique) approach by
> using JSON number for int53-compliant BigIntegers and string notation for
> BigIntegers that doesn't fits int53.  I believe Oracle are considering a
> redesign though.
>
> >> That there are multiple ways of formatting integers in JSON is
> unfortunately a reality all JSON tool maker must (in some way) address.
> >
> > Are you referring to 123, 123.0, 12.3E+1 and 1.23e2 as the multiple
> ways? Or 123, "123", "123.00", and "ew==" as the multiple ways?
> >
> >> I would reverse the integer syntax and claim that integers SHOULD
> conform to standard integer (not JSON) syntax since a standard should not
> break the multitude of JSON tools that already implement this.
> >
> > It sounds like you want JSON serializers to never use the fraction or
> exponent notations when a schema says a field is an integer. That's adding
> a distinction that doesn't exist in JSON so it will break things. A tool
> parsing JSON *without knowing the schema* may well use a double for a
> number, which it may then serialize in exponent notation even if it is an
> integer. For instance, in Java Double.toString(123000000) returns "1.23E8".
> > Mind you, I suspect 1.23E8 will break many implementations that expect
> an integer so perhaps we should require the no-fraction-no-exponent form.
> >
> > This is a separate issue to this thread, however, which was whether a
> JSON schema should offer "int53" as a type, and if it should offer
> "{u}int{8|16|32}" as types.
> >
> > --
> > James Manger
> >
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 1 Aug 2019 07:11:36 +0200

On 2019-08-01 05:55, Ulysse Carion wrote:
> Agreed. The numerical types are about writing correct software more easily in modern programming languages, not about optimizing bytes.
> 

I don't understand what exactly would be dropped here...

> It's precisely this sort of complication that motivates dropping numbers above int32/uint32 from the JSL spec. The spec remains useful without them.
> 
> The same goes for monetary data types. There are too many options, and it's better for the spec to stick to relatively uncontroversial things.

I have only found two open standards using monetary types and they used the same notion.  There is always a possibility creating a unique solution using JSON Number or JSON String.

> This has been discussed in this thread before. Rough consensus suggests that it's better to simply require a zero fractional part in the number value,

> rather than impose constraints at the JSON parser level.

Right, JSL *seems* to presume that you build it on top of a standard JSON parser.  By doing that you inherit platform specific quirks that do not have a natural place in a universal JSON schema processor.

For documentation and code-gen purposes this idea should work just fine, but for run-time data validation it does not and that is the only case where the integer syntax issue would show up.

AFAICT you cannot even do range checking in a reliable manner using "any" JSON parser. Chrome:

JSON.parse('{"g":2e400}')
 > {g: Infinity}

I don't think mathematicians in general consider 2e400 as infinity :)

Before I became a JSON convert, I used XSD (XML Schema Definitions) extensively.  XSD didn't have the limitations of JSL (with respect to numbers), although it was designed 20 years ago. Unfortunately numbers in JSON are more difficult than in XML since there are two entirely different ways to textually represent numbers.

-- Anders

> 
> On Wed, Jul 31, 2019 at 12:14 AM Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
> 
>       > VM887:1 Uncaught TypeError: Do not know how to serialize a BigInt
>     [email-masked] <mailto:[email-masked]>