Subject: [Json] Limitations on number size?

From: Paul Hoffman <[email-masked]>
Date: Mon, 3 Jun 2013 15:10:36 -0700

Greetings. I'm forwarding this fragment with permission from the AppsAWG mailing list. It might warrant clarification in -bis document.

Begin forwarded message:

> From: Nico Williams <[email-masked]>
> On Mon, Jun 3, 2013 at 10:21 AM, Manger, James H
> <[email-masked]> wrote:
>>> Imagine JSON had bignums (doesn't, but imagine).
>> 
>> JavaScript might not, but JSON actually does have indefinite length bignums. A JSON number can have any number of decimal digits.
> 
> Yeah, but it is constrained to IEEE 754 64-bit floating point values, no?

I don't see any limitation in RFC 4627, but it would be interesting if other folks here believe there is a limitation as Nico does.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Jun 2013 19:42:53 -0500

> I don't see any limitation in RFC 4627, but it would be interesting if other folks here believe there is a limitation as Nico does.

Nor does the draft ECMAScript 6 say that.  But a) ECMAScript 6 does
restrict ECMAScript numeric values to IEEE 754 64-bit values
(including NaN, positive infinity, and negative infinity).

I have been told by the maintainer of at least one JSON parser/encoder
that they will not add support for anything other than IEEE 754 64-bit
values.  I'm not sure if I should name the project, but I'll say this:
it's damn near the best C JSON library I've yet seen.  The point is
that the perception that JSON is a subset of JavaScript (ECMAScript)
is out there, and that because the latter is constrained as described,
so should the former.

I think that JSON should be taken as so constrained by ECMAScript, but
more importantly, any constraint, or even no constraint, should be
stated explicitly.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 4 Jun 2013 07:25:37 +0200

> I don't see any limitation in RFC 4627, but it would be interesting if other folks here believe there is a limitation as Nico does.

I think this is a nice example of the distinction between JSON the representation format, a specific JSON application, and JSON the ecosystem.

JSON the format has no trouble in representing high-precision numbers (and big integers).  
So if somebody wanted to send ECDSA signature r/s values as JSON numbers, the format would be fine.

Many JSON applications have at least one end that is implemented in JavaScript.  JavaScript implementations of JSON (including the one now built into the language) typically represent JSON numbers as JavaScript numbers locally (in theory, they could implement their own high-precision numbers/bignums).  So in these applications, whatever information may have been in the JSON format is reduced to what can be represented in JavaScript's IEEE 754 double precision number format (i.e., 53 bits of precision).

Implementations in other languages may even be shaped by the expectation that they will be used in such applications, making the choice of IEEE 754 double as a number format for unencoded/decoded JSON numbers attractive.
This is particularly true for languages such as C that do not have a native way to work with bignums.

As a result, large parts of JSON the ecosystem may be biased to IEEE 754 numbers.  People designing JSON applications that need to work with a large part of the ecosystem (including the common JavaScript implementation and other implementations influenced by the capabilities of that) are better off designing to IEEE 754 double precision numbers.  
People who use JSON the format between implementations that do support bignums, however, may not care much.

So the answer is:
No, JSON the representation format doesn't have such a limitation.
JavaScript's common implementation does, and, as a result, some parts of the JSON ecosystem do as well.
Your JSON application may be well advised to live within these limitations.
Or not.
It depends on the application.

Now if I were designing a new standard for a JSON-based protocol that is going to be used in wider parts of the JSON ecosystem, I'd probably make use of the knowledge that many JSON implementations are limited to IEEE 754 doubles.  So I would choose to encode my ECDSA signatures in base64url instead of sending them as bignums.

It may be useful for RFC4627bis to document all this.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Tue, 4 Jun 2013 02:59:29 -0500

On Tuesday, June 4, 2013, Carsten Bormann wrote:

> >>> JavaScript might not, but JSON actually does have indefinite length
> bignums. A JSON number can have any number of decimal digits.
> >>
> > I don't see any limitation in RFC 4627, but it would be interesting if
> other folks here believe there is a limitation as Nico does.
>
> I think this is a nice example of the distinction between JSON the
> representation format, a specific JSON application, and JSON the ecosystem.
>

This is a plausible answer, sure.  I'm not sure I'd want to say JSON can
only represent IEEE754 64-bit numbers either, so if this is the consensus
then that'll be fine by me.

Do note that JSON's number encoding is indefinite-length...  ;)

Nico
--

---

From: Stefan Drees <[email-masked]>
Date: Tue, 04 Jun 2013 11:39:11 +0200

On 04.06.13 09:59, Nico Williams wrote:
>      >>> JavaScript might not, but JSON actually does have indefinite
>     length bignums. A JSON number can have any number of decimal digits.
>      >>
>      > I don't see any limitation in RFC 4627, but it would be
>     interesting if other folks here believe there is a limitation as
>     Nico does.
>
> only represent IEEE754 64-bit numbers either, so if this is the
> consensus then that'll be fine by me.
>
> Do note that JSON's number encoding is indefinite-length...  ;) ...

In the wild - and in my little eco system at least - JSON is used more 
like ONS (Object Notation Serialization), isn't it?

I regard the "indefinite-length stream of digits to be interpreted as a 
number" as a small price for the unversal usability and acceptance of 
JSON as a format.

In conclusion such a change should IMO a) not qualify as a change being 
covered by the charter of the WG and would b) break several interesting 
use cases, where implementers would then be forced to put potentially(!) 
non IEEE754 64-bit number literals always into quotes instead of as now 
happily producing and consuming "streams" of say:

{/*...*/ "foo": [1.1, 0.2, 1234567890123456789012345678.9, 0.0 //...

All the best,

Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 4 Jun 2013 07:07:01 -0700

And this is the point where one of the chairs says "please suggest specific wording changes to RFC 4627". (FWIW, Doug just got the XML for the bis draft yesterday, and we'll probably have the -00 soon, but that should not delay people making specific suggestions for wording changes.)

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Tue, 4 Jun 2013 13:09:13 -0500

On Tue, Jun 4, 2013 at 4:39 AM, Stefan Drees <[email-masked]> wrote:
> In the wild - and in my little eco system at least - JSON is used more like
> ONS (Object Notation Serialization), isn't it?

Often, yes.

> number" as a small price for the unversal usability and acceptance of JSON
> as a format.

Off-topic: Every value type in JSON (including null and booleans) has
indefinite-length encoding.  I think this is a highly valuable aspect
of JSON, except for the need to escape strings.

> covered by the charter of the WG and would b) break several interesting use
> cases, where implementers would then be forced to put potentially(!) non
> IEEE754 64-bit number literals always into quotes instead of as now happily
> producing and consuming "streams" of say:

I agree.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 4 Jun 2013 13:05:18 -0500

> And this is the point where one of the chairs says "please suggest specific wording changes to RFC 4627". (FWIW, Doug just got the XML for the bis draft yesterday, and we'll probably have the -00 s…

I think we need at least a note on interoperability, stating that
ECMAScript can only handle IEEE 754 64-bit numbers, other applications
might handle only 64-bit two's complement, others might handle bignum
integers, and yet others might handle arbitrary size reals.

Applications have to agree a priori on some schema (out of scope), and
this is part of that.

Should there be any advice to decoder implementors as to how to handle
numbers they cannot natively represent?

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Tue, 4 Jun 2013 12:16:09 -0700

I think that once we’re finished with 4627-bis, it would be a good idea to
do a JSON-best-practices RFC.  I have lots of ideas for things that could
go in there, most probably uncontroversial.

But for now, let’s just document the syntax & semantics.  JSON allows you
to transmit numbers of a magnitude that are likely to cause grief in
certain classes of recipient.  The industry seems to deal with this pretty
well.  -T

On Tue, Jun 4, 2013 at 11:05 AM, Nico Williams <[email-masked]>wrote:

> > And this is the point where one of the chairs says "please suggest
> specific wording changes to RFC 4627". (FWIW, Doug just got the XML for the
> bis draft yesterday, and we'll probably have the -00 soon, but that should
> not delay people making specific suggestions for wording changes.)
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Douglas Crockford <[email-masked]>
Date: Tue, 04 Jun 2013 15:01:21 -0700

> numbers they cannot natively represent?
I strongly advise against fixating too much on JavaScript. JSON is 
successful because it popularized the best of JavaScript. We should not 
burden other uses of JSON with the worst of JavaScript.

Keep in mind that this format has been working successfully in the wild 
for over a decade. The goal here should be to do the least that is 
necessary to formally upgrade its status from Informational, not to 
attempt to fix something that is already working well enough. It is only 
because it has been working well enough that we are considering the upgrade.

---

From: R S <[email-masked]>
Date: Tue, 4 Jun 2013 18:40:05 -0700

On Tue, Jun 4, 2013 at 3:01 PM, Douglas Crockford <[email-masked]>wrote:

> necessary to formally upgrade its status from Informational, not to attempt
> to fix something that is already working well enough. It is only because it
> has been working well enough that we are considering the upgrade.

Strongly agree. We should only be changing things that are incorrect or
obsolete. You can't write an interoperable JSON implementation by reading
RFC 4627 at the moment, but number precision is not one of the
deal-breakers.

- Rob

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Wed, 5 Jun 2013 01:46:14 +0000

> RFC 4627 at the moment, but number precision is not one of the

Would you be willing to contribute specific wording proposals to address those deal-breakers?

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: R S <[email-masked]>
Date: Tue, 4 Jun 2013 19:06:27 -0700

On Tue, Jun 4, 2013 at 6:46 PM, Matt Miller (mamille2)
<[email-masked]>wrote:

> > On Tue, Jun 4, 2013 at 3:01 PM, Douglas Crockford <[email-masked]
> >wrote:
> >
> >>
> attempt

Sure. I wrote a reasonably good JSON implementation after RFC 4627 but
before ES5, and then had to update it to work on the Internet after ES5. We
also have code at my current employer that will break if certain numbers
are not transmitted as strings. It's workable, although inconvenient.

That said, I am also pretty sure that Douglas knows which changes from ES5
need to go in. The main concerns for me are changes to the grammar that
allow more than just objects and arrays as the root value, and the
allowance in RFC 4627 for syntactic extensions (that is not realistic given
ES5 rules).

On the issue at hand, it might be good to rewrite the beginning of the
"Number" section along these lines:

- The representation of numbers is similar to that used in most programming
languages.
+ JSON numbers are arbitrary precision, and represented in a manner similar
to that used in most programming languages.

The current RFC is a little under-specified.

- Rob

---

From: Tim Bray <[email-masked]>
Date: Tue, 4 Jun 2013 19:48:00 -0700

Yeah, the number specification is a little flabby and omits the fact that
the base is 10
<old>
   The representation of numbers is similar to that used in most
   programming languages.  A number contains an integer component that
   may be prefixed with an optional minus sign, which may be followed by
   a fraction part and/or an exponent part.

   Octal and hex forms are not allowed.  Leading zeros are not allowed.

   A fraction part is a decimal point followed by one or more digits.

   An exponent part begins with the letter E in upper or lowercase,
   which may be followed by a plus or minus sign.  The E and optional
   sign are followed by one or more digits.
</old>

<proposal>
A number is represented in base 10 with no leading zeroes or punctuation
such as  commas or spaces.  It may have a preceding minus sign.  It may
have a "."-prefixed fractional part. It may have an exponent, prefixed by
"e" or "E" and optionally "+" or "-".
</proposal>

On Tue, Jun 4, 2013 at 7:06 PM, R S <[email-masked]> wrote:

> On Tue, Jun 4, 2013 at 6:46 PM, Matt Miller (mamille2) <[email-masked]
> > wrote:
>
>>
> - The representation of numbers is similar to that used in most
> programming languages.
> + JSON numbers are arbitrary precision, and represented in a manner
> similar to that used in most programming languages.
>

---

From: Nico Williams <[email-masked]>
Date: Tue, 4 Jun 2013 23:05:40 -0500

> to that used in most programming languages.

It might be nice to be able to specify some ranges that MUST be
supported by decoders, but in practice I think we can't (and probably
shouldn't try to) even do that.

Nico
--

---

From: R S <[email-masked]>
Date: Tue, 4 Jun 2013 22:23:13 -0700

On Tue, Jun 4, 2013 at 9:05 PM, Nico Williams <[email-masked]> wrote:

> programming
> similar

Agreed that we shouldn't try. That's one of the most toxic standards group
activities: Moving Blame Around.

The real problem is that number ranges differ between systems--something
that has nothing to do with JSON.

The ship has sailed. Besides, most JSON numbers fit in 32 bits, and most
JSON values are strings.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Jun 2013 17:51:39 +0200

On Jun 4, 2013, at 16:07, Paul Hoffman <[email-masked]> wrote:

> And this is the point where one of the chairs says "please suggest specific wording changes to RFC 4627".

I like Tim's idea of a separate "best practices" document and look
forward to the process of creating it (Popcorn!).  However, some
things need to be said in the specification.  Here's my take:

<NEW>
Relationship between JSON and JavaScript

As the name implies, JSON was defined using the literal notation the
JavaScript language uses for its data objects; some restrictions were
made to make it more robust to potential changes in the JavaScript
language.  With that derivation made, JSON is now a specification
separate from JavaScript.

Properties of JavaScript (and changes to the JavaScript specification
[ECMAscript]) do not automatically transfer to JSON.  For example,
considerations about the source character set for JavaScript programs
do not influence what is admissible in a JSON document.  Another
example is the number format: JSON can represent numbers in arbitrary
precision by just supplying the necessary number of decimal digits.
JavaScript uses a number model derived from IEEE 754 binary double
precision numbers; the precision and range it can represent natively
therefore is a subset of the precision and range possible in JSON.

This clear separation of the two specifications does not mean that
users of JSON can always ignore JavaScript properties: A sizable part
of the JSON ecosystem is either using JavaScript implementations or
implementations that have been optimized for interaction with
JavaScript implementations.  Depending on the area of application, a
JSON-based protocol may do well to consider JavaScript's specific
properties.  Discussion of best practices for JSON-based protocols is
outside the scope of this specification.  Here, we just want to point
out that, for example, exclusively using numbers natively
representable in JavaScript is not a requirement of the JSON format.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 09:05:52 -0700

-1

Superfluous. The spec is perfectly clear on number representations.

Once again, in scope for a best-practices doc.
-T
On Jun 5, 2013 8:52 AM, "Carsten Bormann" <[email-masked]> wrote:

> specific wording changes to RFC 4627".
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Jun 2013 18:08:43 +0200

> Superfluous. The spec is perfectly clear on number representations.

This text is about number representations only on the surface (and that part can be dialed down).

It may be completely superfluous to you and me, but I have run into enough people who simply don't know what is being said there and make the wrong conclusions.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Wed, 5 Jun 2013 11:20:04 -0500

> It may be completely superfluous to you and me, but I have run into enough people who simply don't know what is being said there and make the wrong conclusions.

The only thing RFC4627 says about this is that "[a]n implementation
may set limits on the range of numbers."  That is very clear, on the
one hand, but it's been missed (as I described earlier, where an
implementor told a contributor that JSON supports only IEEE 754 64-bit
numbers).  It's easy to miss because anyone who knows JavaScript knows
JSON, right?  So a section with a provocative title calling out
differences between JS and JSON would probably help at least those who
would read the RFC.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Jun 2013 19:53:42 +0200

On Jun 5, 2013, at 18:20, Nico Williams <[email-masked]> wrote:

> provocative title calling out
> differences between JS and JSON

Yes.  The more important statement of direction to be made very very explicit is that the (pretty much arrested) protocol evolution of JSON and the (now accelerating) language evolution of JavaScript/ECMAscript are decoupled, so people stop thinking that every new ES version changes JSON.

Grüße, Carsten

---

From: Jacob Davies <[email-masked]>
Date: Wed, 5 Jun 2013 10:51:58 -0700

> shouldn't try to) even do that.

My past reading of the number section and the many references to Javascript
in the spec led me to believe that the number type was Javascript's number
type, and that compliance required you to accept & emit numbers only in
that range (i.e. IEEE 754 64-bit numbers minus NaN and infinities).

Implementation limits on the length of strings or size of arrays or objects
were not specified either, but those are large enough in most languages
that you're unlikely to run into them unexpectedly; if you are thinking of
sending a terabyte of text in string or an object with a trillion keys it
has probably occurred to you that some special interoperability concerns
will apply. But just sending large 64-bit integers as numbers in a JSON
string will cause you interoperability problems with Javascript and a lot
of other JSON parsers, and my experience & some quick searches indicate
this has been a common bug (or misunderstanding, perhaps) in
implementations.

I wouldn't say a size restriction should be specified either. But I think
it might be worth noting this particular interoperability concern in the
specification rather than just as a best practice.