Subject: Re: [Json] Limitations on number size?

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 06:26:45 -0700

[I'm picking up this discussion rather late, but the issue has come up in 
JSON-LD, so I'm putting my oar in.]

My informed understanding of JSON, from reading all the relevant documents, 
was (and again is) also that JSON numbers are ECMAScript numbers are IEEE 
floating point doubles (minus some odd bits).  I was astonished to find out 
that some people disagree, apparently to the point that they believe that 0 is 
different from .0

Not being clear on the primitive data types in JSON is a very bad thing, in my 
opinion.

Peter F. Patel-Schneider

From: Jacob Davies <jacob at well.com>
To: "json at ietf.org" <json at ietf.org>
Date: Wed, 5 Jun 2013 10:51:58 -0700

On Tue, Jun 4, 2013 at 9:05 PM, Nico Williams <nico at cryptonector.com> wrote:

     It might be nice to be able to specify some ranges that MUST be
     supported by decoders, but in practice I think we can't (and probably
     shouldn't try to) even do that.

My past reading of the number section and the many references to _javascript_ 
in the spec led me to believe that the number type was _javascript_'s number 
type, and that compliance required you to accept & emit numbers only in that 
range (i.e. IEEE 754 64-bit numbers minus NaN and infinities).

Implementation limits on the length of strings or size of arrays or objects 
were not specified either, but those are large enough in most languages that 
you're unlikely to run into them unexpectedly; if you are thinking of sending 
a terabyte of text in string or an object with a trillion keys it has probably 
occurred to you that some special interoperability concerns will apply. But 
just sending large 64-bit integers as numbers in a JSON string will cause you 
interoperability problems with _javascript_ and a lot of other JSON parsers, 
and my experience & some quick searches indicate this has been a common bug 
(or misunderstanding, perhaps) in implementations.

I wouldn't say a size restriction should be specified either. But I think it 
might be worth noting this particular interoperability concern in the 
specification rather than just as a best practice.

---

From: Stephan Beal <[email-masked]>
Date: Tue, 9 Jul 2013 16:33:49 +0200

On Tue, Jul 9, 2013 at 3:26 PM, Peter F. Patel-Schneider <
[email-masked]> wrote:

> My informed understanding of JSON, from reading all the relevant
> documents, was (and again is) also that JSON numbers are ECMAScript numbers
> are IEEE floating point doubles (minus some odd bits).  I was astonished to
> find out that some people disagree, apparently to the point that they
> believe that 0 is different from .0
>

My (also informed) interpretation differs significantly ;). My last reading
of the RFC didn't reveal any mention of numeric precision. i.e. an
implementation which supports 5-bit precision is, from my reading of the
RFC, completely legal (it only has to be able to read a number with the
digits 0-9).

Not being clear on the primitive data types in JSON is a very bad thing, in
> my opinion.

It is impossible to require a specific precision because not all
platforms/environments can guaranty a specific precision.

> .,,Implementation limits on the length of strings or size of arrays or
> objects were not specified either
>

Which implies (to me) that a 5-bit-precision implementation is perfectly
legal (though obviously not very useful).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 9 Jul 2013 16:41:48 +0200

>  JSON numbers are ECMAScript numbers

That's a common misconception.

JSON numbers are what they are (see the production "number" in RFC 4627).
JSON numbers obviously are decimal, ECMAScript numbers are binary.

The text then goes on to say "The representation of numbers is similar to that used in most programming languages."; most programming languages distinguish 0 and 0.0 very heavily, and almost all modern ones have more than 53 bits of precision in numbers, but most programming languages also use binary numbers for the semantics of that decimal representation.

When generating JSON for high levels of interoperability, you wouldn't want to rely on the recipient distinguishing 0 from 0.0, or more generally on distinguishing two numbers that differ but map to the same IEEE 754 double precision number.  Some user communities have done the latter*) and I know a lot of parsers that do the former.

> ECMAScript numbers are IEEE floating point doubles (minus some odd bits).  

Indeed.

> I was astonished to find out that some people disagree, apparently to the point that they believe that 0 is different from .0

.0 is not a JSON number, but 0.0 is.
Whether that is different from 0 is up to the data model (RFC 4627 is mute about that, and I have reason to believe that is intentional); in interoperable JSON, it would be foolish to rely on them being different (but it is less clear to me how foolish it would be to rely on them not being different), in JSON syntax, they clearly are different.

Grüße, Carsten

*) https://dev.twitter.com/docs/twitter-ids-json-and-snowflake

---

From: Tim Bray <[email-masked]>
Date: Tue, 9 Jul 2013 08:27:59 -0700

Yeah, we can’t retroactively change the 4627 production for number, nor, in
the -bis, say very much useful aside from a caution that implementations
are all over the map.  When we move on to the BCP or Internet JSON or
whatever, I think the single most interesting discussion will be how to
constrain numeric values for maximum interoperability.

-T

On Tue, Jul 9, 2013 at 7:41 AM, Carsten Bormann <[email-masked]> wrote:

> The text then goes on to say "The representation of numbers is similar to
> that used in most programming languages."; most programming languages
> distinguish 0 and 0.0 very heavily, and almost all modern ones have more
> than 53 bits of precision in numbers, but most programming languages also
> use binary numbers for the semantics of that decimal representation.
>
> When generating JSON for high levels of interoperability, you wouldn't
> want to rely on the recipient distinguishing 0 from 0.0, or more generally
> on distinguishing two numbers that differ but map to the same IEEE 754
> double precision number.  Some user communities have done the latter*) and
> I know a lot of parsers that do the former.
>
> > I was astonished to find out that some people disagree, apparently to
> the point that they believe that 0 is different from .0
>
> Whether that is different from 0 is up to the data model (RFC 4627 is mute
> about that, and I have reason to believe that is intentional); in
> interoperable JSON, it would be foolish to rely on them being different
> (but it is less clear to me how foolish it would be to rely on them not
> being different), in JSON syntax, they clearly are different.
>
> Grüße, Carsten
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 09 Jul 2013 18:23:28 +0200

>that some people disagree, apparently to the point that they believe that 0 is 

RFC 4627 does not define this, but it notes among other things:

  Numeric values that cannot be represented as sequences of digits
  (such as Infinity and NaN) are not permitted.

If JSON used ecmascript semantics then `1e309` would be Infinity,
`-1E-324` would be another way to write `-0.0`.

Certain values like `0.12345678901234567` do have interoperability
problems in practise, consider for instance that

  0.12345678901234568 - 0.12345678901234566 != 0 (ecmascript and Perl)
  0.12345678901234567 - 0.12345678901234566 == 0 (ecmascript)
  0.12345678901234567 - 0.12345678901234568 == 0 (Perl)

and libraries are known round on serialisation; while in ecmascript

  % es -e "JSON.stringify(JSON.parse('[0.12345678901234567]')))"
  "[0.12345678901234566]"

in Perl

  % perl -MJSON -e "print JSON->new->encode(JSON->new
      ->decode('[0.12345678901234567]'))"
  [0.123456789012346]

I think it is well-understood that if you need particularily small,
big, precise or otherwise unusual numbers in JSON then it's best to
encode them as strings so you can do the string-to-number conversion
at a higher level than whatever JSON library you might be using today.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 11:37:49 -0500

> at a higher level than whatever JSON library you might be using today.

<semifacetious>
If that's what we should always do then why can't parsers do it automatically?
</semifacetious>

---

From: "Markus Lanthaler" <[email-masked]>
Date: Tue, 9 Jul 2013 20:11:49 +0200

> On Tuesday, July 09, 2013 6:38 PM, Nico Williams wrote:

Some parsers have options to support that. PHP, e.g. has a
JSON_BIGINT_AS_STRING flag.

--
Markus Lanthaler
@markuslanthaler

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 10:38:02 -0500

On Tue, Jul 9, 2013 at 10:27 AM, Tim Bray <[email-masked]> wrote:
> the -bis, say very much useful aside from a caution that implementations are
> all over the map.  When we move on to the BCP or Internet JSON or whatever,
> I think the single most interesting discussion will be how to constrain
> numeric values for maximum interoperability.

I would like to do two things here though:

1) give advice as to what numeric ranges and precisions are generally
supported (e.g., 32-bit signed integers);

2) what a parser SHOULD do when it parsers a number that it cannot represent.

Nico
--

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 9 Jul 2013 13:12:24 -0700

On Tue, Jul 9, 2013 at 11:11 AM, Markus Lanthaler
<[email-masked]>wrote:

> JSON_BIGINT_AS_STRING flag
>

Ditto for Java: usually coercion from JSON String to number is supported;
and ability to force output of all or some numeric types as Strings as well.
Additional settings to enable (or not) use of unlimited-length numeric
types.

Things are actually bit easier with data-binding, because expected target
type is the metadata that informs whether to truncate this (or error out),
or to keep full fidelity.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 9 Jul 2013 13:13:47 -0700

On Tue, Jul 9, 2013 at 8:38 AM, Nico Williams <[email-masked]> wrote:

> are

This would be useful. One challenge is that of precision for floating-point
types: figuring out whether loss occurs is rather costly. For integer types
this is easier.

-+ Tatu +-

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 14:24:23 -0700

[Somewhat less facetious.]

Where does this end?   Do I have to worry about whether, for example, 0.0 is 
different from 0.00?   (Some people, e.g., numerical analysts, would argue 
that they are different - the first is 0+-0.05 and the second is 0+-0.005.)  
Do I have to worry about whether 0.1 is different from 1E-1?  (Some people, 
e.g., XML-philes, might argue that the first is a decimal and the second is a 
float or double.) Do I have to worry about whether 0 is different from 0.?  
(Some people, e.g., C programmers, might argue that 0 is an int and 0. is a 
float.)

Searching for answers to these questions lead me various bits of RFC 4627.

I looked at "The representation of numbers is similar to that used in most 
programming languages."  However, the only programming language that I was 
acquainted with that had a single number type encompassing scientific notation 
was LISP.   Most other languages that I could recall, e.g., C, C++, and ML, 
had at least a distinction between integers and floats.   Java sort of splits 
the difference.   So this really didn't provide me with any guidance.

This lead me to the beginning of the document and the "is derived from the 
ECMAScript Programming Language Standard" and the "JSON defines a small set of 
formatting rules for the portable representation of structured data."

Finally, some guidance!  Hopefully ECMAScript will tell me how numbers in JSON 
can be portably interpreted as data.   So I read the ECMAScript document.  
Success!  The ECMAScript document tells me, in gory detail, how to interpret 
something that looks like JSON numbers and, moreover, provides a datatype for 
these numbers, namely IEEE floating point double.

So that is how I came up with JSON numbers being IEEE floating point double.

Imagine my surprise when I was told that my reasoning was not correct.

Peter F. Patel-Schneider

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 9 Jul 2013 14:34:12 -0700

I am surprised you came to this conclusion, since I assumed we were finally
getting rid of misconception that JSON is closely tied to Javascript.

This is not to say that the way JSON (under-)defines numbers is optimal;
but at this point forcing castrated version of numbers -- which would lead
to practical problems like preventing use of 64-bit longs for timestamps --
would be counter-productive and to me a non-starter.

-+ Tatu +-

On Tue, Jul 9, 2013 at 2:24 PM, Peter F. Patel-Schneider <
[email-masked]> wrote:

> Where does this end?   Do I have to worry about whether, for example, 0.0
> is different from 0.00?   (Some people, e.g., numerical analysts, would
> argue that they are different - the first is 0+-0.05 and the second is
> 0+-0.005.)  Do I have to worry about whether 0.1 is different from 1E-1?
>  (Some people, e.g., XML-philes, might argue that the first is a decimal
> and the second is a float or double.) Do I have to worry about whether 0 is
> different from 0.?  (Some people, e.g., C programmers, might argue that 0
> is an int and 0. is a float.)
>
> acquainted with that had a single number type encompassing scientific
> notation was LISP.   Most other languages that I could recall, e.g., C,
> C++, and ML, had at least a distinction between integers and floats.   Java
> sort of splits the difference.   So this really didn't provide me with any
> guidance.
>
> ECMAScript Programming Language Standard" and the "JSON defines a small set
> of formatting rules for the portable representation of structured data."
>
> Finally, some guidance!  Hopefully ECMAScript will tell me how numbers in
> JSON can be portably interpreted as data.   So I read the ECMAScript
> document.  Success!  The ECMAScript document tells me, in gory detail, how
> to interpret something that looks like JSON numbers and, moreover, provides
> a datatype for these numbers, namely IEEE floating point double.
>
>> ______________________________**_________________
>> json mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>>
> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: Jorge Chamorro <[email-masked]>
Date: Wed, 10 Jul 2013 00:19:58 +0200

> Imagine my surprise when I was told that my reasoning was not correct.

A "JSON number" is a *text* not a number:

number = [ minus ] int [ frac ] [ exp ]

         decimal-point = %x2E       ; .

         digit1-9 = %x31-39         ; 1-9

         e = %x65 / %x45            ; e E

         exp = e [ minus / plus ] 1*DIGIT

         frac = decimal-point 1*DIGIT

         int = zero / ( digit1-9 *DIGIT )

         minus = %x2D               ; -

         plus = %x2B                ; +

         zero = %x30                ; 0

A character representation of a number that follows the rules in the RFC 4627 that -unlike ECMAScript- doesn't say a thing about its precision, integer-ness, floating-point-ness, allowable ranges or the likes.

To say that a "JSON number" (which is a text) is === IEEE-754 double doesn't make too much sense.

JSON.stringify(number:number) -> text:JSON_number
JSON.parse(text:JSON_number)  -> number:number
-- 
( Jorge )();

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 17:22:15 -0500

> Imagine my surprise when I was told that my reasoning was not correct.

A great many participants on this list (since its very recent
inception) have been surprised, myself included, by many oddities.  It
should no longer surprise any of us that others are surprised by these
things :)

Nico
--

---

From: "John Levine" <[email-masked]>
Date: 9 Jul 2013 22:26:01 -0000

>supported (e.g., 32-bit signed integers);

Do we actually know?  Or is it just whatever number format a library's
underlying language happens to provide?

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 17:31:02 -0500

> underlying language happens to provide?

I've contributed to one JSON implementation that doesn't do better
than 32-bit signed integers.  I've contributed to one that uses C
doubles.

I doubt there's any that support only 0-bit numbers (heh).  And I
doubt there are any that only support 16-bit integers (or that we'd
care about them, but who knows).

I *suspect* 32-bit signed, or perhaps 31-bit unsigned integers is the
minimum that all implementations we'd care about support.

We might want to do a survey.

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 17:45:26 -0500

> [Somewhat less facetious.]

I was surprised that needn't have been at all facetious.  JSON
surprises me all the time.  Since so many parsers have optional
behaviours, let's explore this path.

> e.g., XML-philes, might argue that the first is a decimal and the second is
> a float or double.) Do I have to worry about whether 0 is different from 0.?

I think unnecessary leading or trailing zeros, or even zero decimal
fraction, are fair game to lose.  Such things bear no information if
all we care about is numbers.  (.0 in some languages implies floating
point, as opposed to integer, but that's not relevant here.)  Ditto
negative zero.

Otherwise the tolerance level for information loss in numbers should
be specified by the application.  We can probably agree on some common
options that implementations *might* provide.  Here's all the options
that I think we should consider, but I'm probably missing some:

 - IEEE 754 {32, 64}-bit double with unspecified loss tolerance-- any
value that can be represented as such no matter what lossage, will be
accepted

 - IEEE 754 {32, 64}-bit double with no loss in exponent and with N
digits of precision, with some specified rounding

 - IEEE 754 {32, 64}-bit double with no loss (e.g., 0.1 can't be
parsed as a double exactly)

 - take the floor of the number as an integer of {32, 64, 128} bits

 - take the floor of the number as bigint (whatever that means for the
parser / host language) (subject to some arbitrary max size
constraint)

 - bignum with no loss (subject to some arbitrary max size constraint)

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 15:46:09 -0700

> A "JSON number" is a *text* not a number:

OK, OK.  A JSON number represents a number.   I'm guilty of not being 
appropriately pedantic.

But everything that I've read about JSON indicates that JSON is supposed to be 
used for (portably) interchanging data.  If all there is to JSON is the 
grammar then what's the point of JSON?

> A character representation of a number that follows the rules in the RFC 4627 that -unlike ECMAScript- doesn't say a thing about its precision, integer-ness, floating-point-ness, allowable ranges o…

And neither does the BNF for numeric types in most programming languages, 
including ECMAScript.   The BNF syntax for numbers in ECMAScript permits 
arbitrarily long sequences of digits.   As far as I can recall C, C++, and ML 
all have the same characteristic.

What is (surprisingly) missing in JSON is a firm notion of the data being 
represented by the syntax.   A pale shadow of this issue for strings appears 
to have been consuming this working for quite some time.
> To say that a "JSON number" (which is a text) is === IEEE-754 double doesn't make too much sense.

OK, then, what I inferred was that a JSON number represents an IEEE floating 
point double and I was very surprised to find out that this was not the case.

Peter F. Patel-Schneider

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 15:58:58 -0700

Where then am I supposed to go to find out what a JSON number represents?   
There are many possibilities (float only, rational only, separate integer and 
float, separate rational and float, variable-precision decimal only, separate 
integer and variable-precision decimal, variable precision float only, ...). 
And then there are the various range possibilities.

The only suitable guidance provided in RFC4627 is via ECMAScript and 
ECMAScript is firmly IEEE floating point double only.

So why are you surprised that I came up with this conclusion?

peter

On 07/09/2013 02:34 PM, Tatu Saloranta wrote:
> This is not to say that the way JSON (under-)defines numbers is optimal; but 
> at this point forcing castrated version of numbers -- which would lead to 
> practical problems like preventing use of 64-bit longs for timestamps --

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 18:00:53 -0500

On Tue, Jul 9, 2013 at 5:46 PM, Peter F. Patel-Schneider
<[email-masked]> wrote:
> But everything that I've read about JSON indicates that JSON is supposed to
> be used for (portably) interchanging data.  If all there is to JSON is the

+1.  In practice agreeing on a subset of typing results in interop, at
least for implementations of specific applications.

Sometimes, when straying past that subset, there's loss of
information, and that mostly works out.  Sometimes parsers reject
inputs, this results in implementors figuring out what happened and
how to improve interop, or else the user doesn't bother reporting the
problem.

We can definitely identify a subset of JSON that is extremely likely
to interop successfully:

 - strings being Unicode character strings, possibly with paired
surrogates (there are still some implementations, no doubt, that can't
handle code points outside the BMP)

 - top-level is an array or object (though many parsers have options
to allow other top-level types, so I don't think this is too
important)

 - no dup names (parsers might or might not reject, or deconflict)

 - numbers as 32-bit signed integers, or maybe even numbers as IEEE
754 64-bit doubles -- TBD from a survey we might never do

Note that the proposed I-JSON need not be that subset of JSON most
likely to interop, just the subset we want for Internet protocols.
And we might be happy enough with I-JSON that we never define any
other subsets of JSON.

RFC4627's underspecification, coupled with host languages with widely
varying attributes led to JSON being a superset than most
implementations of it.

ECMAScript JSON is also a subset of RFC4627, and is neither a subset
nor superset of the proposed I-JSON.

> to have been consuming this working for quite some time.

Yes.

Nico
--

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 16:09:53 -0700

Each of your options provides a single type for what is represented by JSON 
numbers.  However, this doesn't include multiple types, such as
- 64 bit integers plus IEEE 754 64-bit double (for some loss tolerance)

I think that the biggest surprise that I have had with respect to JSON numbers 
was that this multiple typing for JSON numbers was allowed (and maybe even 
common).

This lead to my comments on whether the JSON numbers 1, 1.0, and 1.0E0 
represent the same thing or not.

Peter F. Patel-Schneider

---

From: Jorge Chamorro <[email-masked]>
Date: Wed, 10 Jul 2013 01:24:03 +0200

On 10/07/2013, at 00:46, Peter F. Patel-Schneider wrote:
> 
> 
> OK, then, what I inferred was that a JSON number represents an IEEE floating point double and I was very surprised to find out that this was not the case.

And given the super-simple rules in the RFC *any* number can be properly represented which is -it seems to me- exactly what Crockford had in mind: that JSON (the data interchange format) does not impose the limits.

That the platforms, languages, libraries, and stringify()er and parse()r implementations may have numeric data type limitations, isn't JSON's (the data interchange format) business.
-- 
( Jorge )();

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 16:44:02 -0700

On 07/09/2013 04:11 PM, Nico Williams wrote:
> On Tue, Jul 09, 2013 at 03:58:58PM -0700, Peter F. Patel-Schneider wrote:
>> Where then am I supposed to go to find out what a JSON number
>> represents?   There are many possibilities (float only, rational
>> [...]
> Today there is no specification to tell you that.  We're working on it.
> Here.  Right now.  With your input :)
>
> We seem to be concluding that there's no way to unify JSON.  So it may
> be that in the end you'll find two or more sets of guidance and you'll
> have to select one (or all, deferring the choice to run-time).

That's a very unhappy situation.   My interest in JSON is to consume data in 
JSON documents (mostly to use as input into representation systems that also 
use the W3C semantic web languages RDF and OWL). If JSON is ambiguous (e.g., 
as to whether 0.0 and 0 encode/stand for/represent the same thing) then JSON 
isn't very suitable for transmitting data, at least for me.
> It's not the only possible guidance.  You read RFC4627, you realize it's
> underspecified and look for guidance.  There's none from the IETF.
> There's search engines and if you look at enough JSON implementations
> (probably just two, at least of which must not be part of an ECMAScript
> implementation) you should note the divergent interpretations of the
> spec.
>
> Your [reasonable] mistake is to assume that because RFC4627 was inspired
> by ECMAScript notation then when ECMAScript adds a specification for
> JSON the latter must be authoritative.

Aah, but that's not what I did.  I didn't look at (the new-ish) section 15.12 
of ECMAScription version 5.1 at all.  Instead I looked at sections 7.8.3 
(numeric literals) and 8.5 (numbers) of ECMAScript version 3, which is 
referenced from, and pre-dates, RFC 4627.  I found indications that JSON 
numbers were derived from ECMAScript numeric literals, so I made the easy 
conclusion that the values represented by JSON numbers were supposed to be 
ECMAScript numbers, i.e., IEEE floating point double.

>   But standards development
> organizations don't usually recognize invasions into each others' turfs
> without prior agreements, and so... here we are.

I had no need, or even desire, to look at a subsequent non-IETF work.
>
> Now, why should anyone trying to implement JSON need to know about SDO
> politics?  Well, because there's no way to reconcile all the different
> JSONs now -- the ships have sailed.  Sooner or later anyone dealing with
> JSON will notice this, and then the question of who "owns" JSON comes
> up, and then one discovers this whole mess.  (Then one weeps.  Lather,
> rinse, repeat.)
>
> I'm not now, but once would have been.
>
> Nico
Peter F. Patel-Schneider

---

From: Jacob Davies <[email-masked]>
Date: Tue, 9 Jul 2013 16:49:25 -0700

> So why are you surprised that I came up with this conclusion?

What I found surprising was that there appeared to be lots of people
willing to say that the strings in the JSON data model (not the
format) are Unicode, or that the keys in the objects must be unique
according to one particular method of string normalization, but not
that the numbers in JSON are derived from Javascript and that
interoperability in the most important use of JSON (sending data to
browsers) absolutely requires you to stay within the size limit of
Javascript numbers (and failure to do so has led to some notable bugs
that require backwards-incompatible API changes, i.e. going from
number -> string for some field).

My take prior to this discussion was that the mild annoyance of not
being able to send 9223372036854775807 as a number in JSON was the
reasonable price you paid for having an interoperable number range in
the format. It seemed like most people here disagreed with my
impression, which is fair enough, and it sounded like Douglas
Crockford's intent was not to restrict them in such a way, but I would
be surprised if many people actually - intentionally at least! - use
numbers from outside the Javascript range in JSON, given the problems
it causes for Javascript.

As for numbers smaller than that range, the RFC says implementations
may reject numbers they don't like, so an implementation without
floating point numbers is OK, just not capable of general
interoperability.

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 17:16:29 -0700

*Any* number?  Not so!  Assuming sane rules, JSON numbers are inadequate for 
representing all rationals, and the same holds, of course, for any superset of 
the rationals, including reals and complex.  (Yes, yes, no usable data format 
can represent all the reals or complex numbers.)

JSON numbers are unlike the numeric type system of any programming language 
that I had been familiar with, even LISP.  That JSON numbers are so close to 
ECMAScript numeric literals is, to me, a very strong indication that JSON 
numbers are supposed to represent ECMAScript numbers.

I do agree that implementations may be range-restricted, and that a data 
interchange format shouldn't always nail the ranges down exactly.   However, 
floats (being rather unusual numbers) are one place where it can be fruitful 
to nail things completely down.

Peter F. Patel-Schneider

---

From: Jorge Chamorro <[email-masked]>
Date: Wed, 10 Jul 2013 02:19:39 +0200

> it causes for Javascript.

Right, the implementations set the limits, and JavaScript implementations in particular have to watch out for interoperability problems in the presence of 64 bit ints.
-- 
( Jorge )();

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 9 Jul 2013 18:09:02 -0700

On Tue, Jul 9, 2013 at 3:46 PM, Peter F. Patel-Schneider <
[email-masked]> wrote:

> But everything that I've read about JSON indicates that JSON is supposed
> to be used for (portably) interchanging data.  If all there is to JSON is
> the grammar then what's the point of JSON?
>
>
Proof is in pudding. JSON has been used successfully for years now for
interchanging data.
I can't recall a case where number precision had actually been reported as
a problem; even though it is obvious that something could go wrong
somewhere.
So while JSON is under-specified, it turns out that actual systems have
been successfully built and Just Work.

All alarmism aside it is easy to get tangled in the web of _potential_
issues. Same is true with many discussed aspects, including mortal danger
of duplicate names. For some reasons it has not registered as a reported
issue for library implementors (as far as I know).

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 9 Jul 2013 18:12:07 -0700

On Tue, Jul 9, 2013 at 3:58 PM, Peter F. Patel-Schneider <
[email-masked]> wrote:

>   There are many possibilities (float only, rational only, separate integer
> and float, separate rational and float, variable-precision decimal only,
> separate integer and variable-precision decimal, variable precision float
> only, ...). And then there are the various range possibilities.
>
>
So your proof is of the form "I was trying to find answer to X; and the
only answer I found was Y; thereby it MUST BE the right answer"? That is a
logical fallacy of some kind.

But you are right in that perhaps I should not be surprised. Different
people reach different conclusions all the time, even if given same
information.

-+ Tatu +-

>> I am surprised you came to this conclusion, since I assumed we were
>> finally getting rid of misconception that JSON is closely tied to
>> Javascript.
>>

---

From: John Cowan <[email-masked]>
Date: Tue, 9 Jul 2013 21:26:07 -0400

> underlying language happens to provide?

I would be very surprised if existing libraries did *not* handle the
entire range of IEEE doubles (other than Inf, -Inf, NaN, and -0.0)
at the very least.

-- 
weirdo:    When is R7RS coming out?                  John Cowan
Riastradh: As soon as the top is a beautiful         [email-masked]
           golden brown and if you stick a toothpick
          in it, the toothpick comes out dry.        www.ccil.org/~cowan

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 18:36:47 -0700

My reasoning is more like:

I was trying to find the answer to Y about X in the defining document for X 
and the only thing related to Y there (or related to Y in anything pointed to 
from there) was Z so Z should be the answer.

Yes, this isn't quite the situation, but it's close.

Yes, this isn't the way things should work in a defining document, but that's 
the situation with respect to this defining document.

Peter F. Patel-Schneider

On 07/09/2013 06:12 PM, Tatu Saloranta wrote:
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     represents?   There are many possibilities (float only, rational only,
>     separate integer and float, separate rational and float,
>     variable-precision decimal only, separate integer and variable-precision
>     decimal, variable precision float only, ...). And then there are the
>     various range possibilities.
>
> So your proof is of the form "I was trying to find answer to X; and the only 
> answer I found was Y; thereby it MUST BE the right answer"? That is a 
> people reach different conclusions all the time, even if given same information.
>
> -+ Tatu +-
>
>     peter
>
>
>
>
>         This is not to say that the way JSON (under-)defines numbers is
>         optimal; but at this point forcing castrated version of numbers --
>         which would lead to practical problems like preventing use of 64-bit
>         longs for timestamps -- would be counter-productive and to me a
>         non-starter.
>
>         -+ Tatu +-
>
>
>
>

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Tue, 09 Jul 2013 18:47:14 -0700

On 07/09/2013 06:09 PM, Tatu Saloranta wrote:
> On Tue, Jul 9, 2013 at 3:46 PM, Peter F. Patel-Schneider 
>     to be used for (portably) interchanging data.  If all there is to JSON
>     is the grammar then what's the point of JSON?
>
>
> I can't recall a case where number precision had actually been reported as a 
> problem; even though it is obvious that something could go wrong somewhere.
> So while JSON is under-specified, it turns out that actual systems have been 
> successfully built and Just Work.
>
> issues. Same is true with many discussed aspects, including mortal danger of 
> duplicate names. For some reasons it has not registered as a reported issue 
> for library implementors (as far as I know).
>
> -+ Tatu +-
>

So there is more to JSON than the grammar.

But what is this more?   I've heard quite divergent views on what this more 
is, both in the working group and, even more, outside it.

If you want to sent certain kinds of data (non-C0 ASCII strings, nothing that 
could be considered to be duplicate names in objects, 32-bit signed integers, 
...) around using JSON, then things work well, but what are the limits?  What 
happens when these limits are exceeded?

For example, the systems that I build need to know whether 0 and 0.0 represent 
the same thing because they count up the number of distinct values.

Peter F. Patel-Schneider

---

From: John Cowan <[email-masked]>
Date: Tue, 9 Jul 2013 22:38:53 -0400

Peter F. Patel-Schneider scripsit:

> JSON numbers are unlike the numeric type system of any programming
> language that I had been familiar with, even LISP.  

Basic and Perl are both flonum-only, at least unless you set appropriate
options.

-- 
Is a chair finely made tragic or comic? Is the          John Cowan
portrait of Mona Lisa good if I desire to see           [email-masked]
it? Is the bust of Sir Philip Crampton lyrical,         http://ccil.org/~cowan
epical or dramatic?  If a man hacking in fury
at a block of wood make there an image of a cow,
is that image a work of art? If not, why not?               --Stephen Dedalus

---

From: John Cowan <[email-masked]>
Date: Tue, 9 Jul 2013 22:42:15 -0400

> would be counter-productive and to me a non-starter.

Apparently, there are already practical problems with interchanging
64-bit longs: google for [json large numbers bugs] for lots of reports.
While there are apparently int32-only systems out there, it's clearly
understood that they are unusually restricted: such is not the case
for JavaScript-hosted implementations.

-- 
By Elbereth and Luthien the Fair, you shall     [email-masked]
have neither the Ring nor me!  --Frodo          http://www.ccil.org/~cowan

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 10 Jul 2013 05:16:15 +0200

On Jul 10, 2013, at 04:42, John Cowan <[email-masked]> wrote:

> google for [json large numbers bugs]

Next up: google for »number overflow bugs«.

Somehow, it seems that handling large numbers is not entirely a problem specific to JSON.

Any interchange format that exchanges numbers between ecosystems with different number formats will run into this.  That is not a valid argument for saying JSON should be a single-ecosystem format.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jul 2013 23:13:23 -0500

> for JavaScript-hosted implementations.

So let's survey implementations.

ECMAScript, Perl, Python, libjv, Jansson, json-c -- all have doubles.

Heimdal's not-really-intended-for-external-use-(not-yet) JSON
implementation only does 32-bit integers.  This one doesn't count, and
it'd be easy to add double support to it if it mattered.  Are there
any C JSON libraries that do NaN coding?

Those are the ones I know, use, and/or contribute to.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Wed, 10 Jul 2013 00:41:39 -0400

> ECMAScript, Perl, Python, libjv, Jansson, json-c -- all have doubles.

My own implementation in Scheme will interpret a number with a decimal
point or an exponent as an IEEE double, except on the few Schemes
that don't support inexact numbers.  An integer will be interpreted
as an exact integer up to the system limit, which varies from 24-bit
to limited only by memory size.

-- 
Being understandable rather than obscurantist poses certain
risks, in that one's opinions are clear and therefore     | John Cowan
falsifiable in the light of new data, but it has the      | [email-masked]
advantage of encouraging feedback from others.  --James A. Matisoff

---

From: Stephan Beal <[email-masked]>
Date: Wed, 10 Jul 2013 11:47:59 +0200

On Wed, Jul 10, 2013 at 1:44 AM, Peter F. Patel-Schneider <
[email-masked]> wrote:

> That's a very unhappy situation.   My interest in JSON is to consume data
> in JSON documents (mostly to use as input into representation systems that
> also use the W3C semantic web languages RDF and OWL). If JSON is ambiguous
> (e.g., as to whether 0.0 and 0 encode/stand for/represent the same thing)
> then JSON isn't very suitable for transmitting data, at least for me.
>

While i think we will all agree that, at a technically pedantic level,
you're absolutely right, JSON has been in heavy use for about 10(?) years
now with _relatively_ few instances of this causing a problem. Implementors
tend to use whatever default limits the platform provides (e.g. 32-bit on
32-bit platforms and 64 on 64-bit, and 6-digit precision in doubles seems
to be conventional in C libraries). People using high-precision/very
large/very small numbers are certainly aware of the limitations/portability
problems, and will (possibly after falling on their face with JSON) pick a
different format. That's all fine and good - i haven't seen anyone here
argue that JSON needs to be _the_ data format. It needs to be a _useful_
format for a wide range of applications, and it is that even if it's
hard-coded to be limited to 31-bit integer ranges. In my implementations i
have had to be very aware of system-level precision limits, but i simply
document them, add build options to use, e.g. 64-bit integers if available,
and leave it at that. Those details fall comfortably into the normal range
of "implementation defined" details, IMO, and do _not_ (IMO) fall into
JSON's realm of authority (JSON just needs to tell me the BNF for reading a
number, though one could argue that the BNF should/does also imply certain
limits). It would be impossible to enforce that arbitrary implementations
must support arbitrarily long numbers, just as it would be silly to
arbitrarily limit JSON to, say, 20-bit precision.

Using your case of 0 vs 0.0. The vast, vast majority of JSON consumers are
JavaScript, and JS doesn't differentiate between doubles and integers, so 0
is, in effect equivalent to 0.0. In fact, there are few real-world
applications using JSON where the two are _not_ equivalent (barring
scientific, high-precision, math-centric apps, of course, and those should
probably be looking for a different format which guarantees them their
desired ranges/limits).

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 10 Jul 2013 03:07:04 -0700

On 07/10/2013 02:47 AM, Stephan Beal wrote:
> On Wed, Jul 10, 2013 at 1:44 AM, Peter F. Patel-Schneider 
>     That's a very unhappy situation.   My interest in JSON is to consume
>     data in JSON documents (mostly to use as input into representation
>     systems that also use the W3C semantic web languages RDF and OWL). If
>     JSON is ambiguous (e.g., as to whether 0.0 and 0 encode/stand
>     for/represent the same thing) then JSON isn't very suitable for
>     transmitting data, at least for me.
>
>
> now with _relatively_ few instances of this causing a problem.

Relatively is one of these weasel words that we all use.  I certainly agree 
that JSON is useful for transmitting certain kinds of data.

However, the working group mailing archives contain evidence that there are 
indeed significant problems when using JSON to portably interchange data, 
particularly  binary data.

> Implementors tend to use whatever default limits the platform provides (e.g. 
> 32-bit on 32-bit platforms and 64 on 64-bit, and 6-digit precision in 
> doubles seems to be conventional in C libraries).
> People using high-precision/very large/very small numbers are certainly 
> aware of the limitations/portability problems, and will (possibly after 
> falling on their face with JSON) pick a different format.

Are they really aware of all the potential problems?   And just what counts as 
high-precision/very large/very small?  Does 0. belong to any of these categories?

> That's all fine and good - i haven't seen anyone here argue that JSON needs 
> to be _the_ data format. It needs to be a _useful_ format for a wide range 
> of applications, and it is that even if it's hard-coded to be limited to 
> 31-bit integer ranges. In my implementations i have had to be very aware of 
> system-level precision limits, but i simply document them, add build options 
> to use, e.g. 64-bit integers if available, and leave it at that. Those 
> details fall comfortably into the normal range of "implementation defined" 
> details, IMO, and do _not_ (IMO) fall into JSON's realm of authority (JSON 
> just needs to tell me the BNF for reading a number, though one could argue 
> that the BNF should/does also imply certain limits). It would be impossible 
> to enforce that arbitrary implementations must support arbitrarily long 
> numbers, just as it would be silly to arbitrarily limit JSON to, say, 20-bit 
> precision.
>

I would appreciate some evidence to back up the claim that the vast, vast 
majority of JSON is handled in an environment where the JSON numbers 0 and 0.0 
do indeed represent the same thing. The RDF W3C workiing group is in the last 
stages of putting its stamp of approval on JSON-LD, which presents the JSON 
numbers 0 and 0.0 to RDF as being different.

> ___________________
Peter F. Patel-Schneider

---

From: Stephan Beal <[email-masked]>
Date: Wed, 10 Jul 2013 12:15:22 +0200

On Wed, Jul 10, 2013 at 12:07 PM, Peter F. Patel-Schneider <
[email-masked]> wrote:

> Relatively is one of these weasel words that we all use.

Agreed.

>
> However, the working group mailing archives contain evidence that there
> are indeed significant problems when using JSON to portably interchange
> data, particularly  binary data.

Such cases are their own fault: JSON never, every claimed to be able to
support binary data. Wrong tool for the job, period.

> Implementors tend to use whatever default limits the platform provides
> (e.g. 32-bit on 32-bit platforms and 64 on 64-bit, and 6-digit precision in
> Are they really aware of all the potential problems?   And just what
> counts as high-precision/very large/very small?  Does 0. belong to any of
> these categories?

Excellent point - no, they are most certainly not aware of all potential
problems, and that's why they rely on the system-level defaults - because
they know that very smart people have spent time implementing them and
making sure they are reasonable for a wide variety of use cases. Least
common denominator. It would be impossible to expect every JSON parser
implementor to understand the intricacies of binary encodings of floating
point values. (i've implement 3 or 4 JSON parsers in C and C++ and have
never read any of the relevant IEEE docs - i rely on the system to define
my legal ranges).

>> Using your case of 0 vs 0.0. The vast, vast majority of JSON consumers
>> are JavaScript, and JS doesn't differentiate between doubles and integers,
>> so 0 is, in effect equivalent to 0.0. In fact, there are few real-world
> majority of JSON is handled in an environment where the JSON numbers 0 and
> 0.0 do indeed represent the same thing.

You're right, and i'll see if i can find some properly collected
statistics. My "evidence" is purely anecdotal - 90+% of the applications i
work with which use JSON are www-based or www-bound.

The RDF W3C workiing group is in the last stages of putting its stamp of
> approval on JSON-LD, which presents the JSON numbers 0 and 0.0 to RDF as
> being different.
>

Different standard - doesn't interest me ;). i don't have a single
application where the difference between 0 and 0.0 is relevant to the
outcome of a calculation (with the minor exception of maybe output
formatting).

---

From: Stephan Beal <[email-masked]>
Date: Wed, 10 Jul 2013 12:24:31 +0200

> work with which use JSON are www-based or www-bound.

While this doesn't count as evidence, it might help support the anecdote:

http://en.wikipedia.org/wiki/JSON

"It [JSON] is used primarily to transmit data between a server and web
application, serving as an alternative to XML."

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 10 Jul 2013 04:26:20 -0700

On 07/10/2013 03:24 AM, Stephan Beal wrote:
> On Wed, Jul 10, 2013 at 12:15 PM, Stephan Beal <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     statistics. My "evidence" is purely anecdotal - 90+% of the applications
>     i work with which use JSON are www-based or www-bound.
>
>

If it is in Wikipedia, it must be true.  :-)

So that means that I can also count on the infobox information that JSON is 
"Extended from JavaScript", right?

peter

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 10 Jul 2013 14:18:37 +0200

> majority of JSON is handled in an environment where the JSON numbers 0 and
0.0
> numbers 0 and 0.0 to RDF as being different.

That's just wrong. When converting JSON-LD to RDF both 0 and 0.0 will result
in a "0"^^xsd:integer.

--
Markus Lanthaler
@markuslanthaler

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 10 Jul 2013 05:29:40 -0700

On 07/10/2013 03:15 AM, Stephan Beal wrote:
>
> On Wed, Jul 10, 2013 at 12:07 PM, Peter F. Patel-Schneider 
[...]
> support binary data. Wrong tool for the job, period.

That's one opinion (which I agree with, by the way), but there appears to be 
significant support for a different opinion.

[...]
> Excellent point - no, they [users] are most certainly not aware of all 
> potential problems, and that's why they rely on the system-level defaults - 
> because they know that very smart people [the IETF JSON working group?] have 
> spent time implementing them and making sure they are reasonable for a wide 
> variety of use cases. Least common denominator.

But what is this least common denominator?  There was a (probably not serious) 
post that suggested that 5-bit JSON numbers would be viable.  There were 
serious posts suggesting that one should not count on anything more than 
32-bit (signed) integers, i.e., that 0.1 or even 0.0 might be rejected by some 
JSON implementations.

[...]
> outcome of a calculation (with the minor exception of maybe output formatting).

I find this particular attitude very troubling, even spoken in jest.  You may 
not care whether the JSON numbers 0 and 0.0 represent different things, but 
others do, and are about to push an answer into a W3C recommendation.   In my 
opinion, it should be the goal of any standard (or quasi-standard) setting 
body to try to cover all the reasonable cases (without, of course, getting 
bogged down on things like how many Unicode surrogate characters can dance on 
the head of a JSON string).
Peter F. Patel-Schneider

---

From: Stephan Beal <[email-masked]>
Date: Wed, 10 Jul 2013 14:49:25 +0200

On Wed, Jul 10, 2013 at 2:29 PM, Peter F. Patel-Schneider <
[email-masked]> wrote:

> That's one opinion (which I agree with, by the way), but there appears to
> be significant support for a different opinion.
>

That's the beauty of it - in the context of the JSON RFC it's not an
opinion - it's a technical fact. Unless the WG wants to extend JSON in
incompatible ways (which they are prohibited from doing, from what i
understand) there is zero chance of this revision of it supporting JSON.

>> because they know that very smart people [the IETF JSON working group?]
>> have spent time implementing them and making sure they are reasonable for a
>> wide variety of use cases. Least common denominator.
>>
>
> But what is this least common denominator?  There was a (probably not
> serious) post that suggested that 5-bit JSON numbers would be viable.
>  There were serious posts suggesting that one should not count on anything
> more than 32-bit (signed) integers, i.e., that 0.1 or even 0.0 might be
> rejected by some JSON implementations.
>

As long as JSON does not specify a numeric precision (and IMO it "MUST NOT"
;) then any precision which can be represented per the grammar is legal.
i.e. any precision for which we can represent digits 0 to 9 (which rules
out the 0-bit precision someone suggested ;).

> I find this particular attitude very troubling, even spoken in jest.

It was not intended as a jest. The ONLY JSON documents which interest me
are the original RFC and the one being drafted now. i have ZERO
applications for which there is a semantic difference between 0 and 0.0,
-0, +0, and all the other zeroes. They are the proverbial "noisy minority,"
for which much of the fuss has been raised regarding numbers, but which
have zero impact on "the majority" of applications (==none i've ever worked
on in nearly 30 years of programming).

>  You may not care whether the JSON numbers 0 and 0.0 represent different
> things, but others do, and are about to push an answer into a W3C
> recommendation.

And the current RFC does not distinguish, so  what's the problem if it
continues not to? JSON, as it is now, is NOT a format for high-precision
numerics because the RFC _clearly_ and _unambiguously_ does _not_ cover the
details important to such cases (e.g. 0!==0.0 and a myriad of others which
have been mentioned in recent weeks). If JSON had been designed for that
then i'm quite certain that Doug would have covered those topics.

In my opinion, it should be the goal of any standard (or quasi-standard)
> setting body to try to cover all the reasonable cases (without, of course,
> getting bogged down on things like how many Unicode surrogate characters
> can dance on the head of a JSON string)

One of the charters of the WG is to NOT break JSON compatibility, so their
hands are tied in this regard. They only way to introduce that level of
detail is if one "breaks" JSON vis-a-vis many of the existing
implementations, or otherwise breaks them in the sense of, "they worked
yesterday but are no longer compliant under the new definition" (kinda like
what happened to poor old Pluto - it's big enough to have more moons than
Earth but is no longer considered a planet).

---

From: "Peter F. Patel-Schneider" <[email-masked]>
Date: Wed, 10 Jul 2013 05:59:51 -0700

Oops, right you are, the spec says that non-zero fractional parts are float, 
and 0.0 appears to have a zero fractional part.  My fault for not reading 
enough, particularly now that the situation is spelled out in a part of the 
document that I have been complaining about.

So now we have things the other way around - 0 and 0.0 represent the same 
thing namely the integer 0 in JSON-LD.   This is certainly a decision, and a 
decision, I think, somewhat different from the one taken by ECMAScript-based 
implementations of JSON (although it does share with them the situation that 0 
and 0.0 represent the same thing.

My apologies for misstating the situation,

peter

---

From: John Cowan <[email-masked]>
Date: Wed, 10 Jul 2013 10:03:14 -0400

> understand) there is zero chance of this revision of it supporting JSON.

JSON cannot support binary data *as a distinct type*.  It can of course
support binary data in a variety of other ways by convention:  as an
array of 0s and 1s or non-negative numbers < 256 or < 65536, or as an
array containing a string whose 16-bit whatever-they-ares represent the
binary object 16 bits at a time.

> out the 0-bit precision someone suggested ;).

It doesn't, actually.  A JSON implementation can set any numerical limits
it likes, from 2^53 to 2^-53, or from 0 to 0 (exclusive).

-- 
My confusion is rapidly waxing          John Cowan
For XML Schema's too taxing:            [email-masked]
    I'd use DTDs                        http://www.ccil.org/~cowan
    If they had local trees --
I think I best switch to RELAX NG.

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 10 Jul 2013 11:47:50 -0700

> > to practical problems like preventing use of 64-bit longs for timestamps

I am not saying there can't be problems there (and I have gotten feature
requests for optional processing limitations to conform to Javascript
representation); but rather that these are also successfully and widely
used. So it is fine to define or recommend safe(r) profiles that take
commonly encountered limitations into account.
But just to take care not to consider this strict limitation on all usage.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 10 Jul 2013 11:55:41 -0700

On Wed, Jul 10, 2013 at 3:07 AM, Peter F. Patel-Schneider <
[email-masked]> wrote:

>> [email-masked] <mailto:[email-masked]>> wrote:
>>
> Relatively is one of these weasel words that we all use.  I certainly
> agree that JSON is useful for transmitting certain kinds of data.
>

Same can be said about any data formats in wide usage.

This whole discussion could go on about XML as well: XML does not define
any numeric types at format level. Additional specifications (schema, specs
that require one of schema languages) have been defined to handle mapping
of potentially unbound numbers. When you think of JSON as counter-part to
XML specification, you may be less surprised to see physical limitations --
those may well belong to another layer. XML Schema has wide variety of
numeric types as well, including unlimited precision ones.

> data, particularly  binary data.

Why so? Base64 encoding is the de-facto standard for this, and very widely
used.
Of all things discussed, this seems among lesser problems.

And yes, even for transferring large amounts of binary data it is possible
to add incremental (.... streaming) processing. I use this for
synchronizing gigabyte sized blocks on a distributed storage system
(although usually using binary encoding of JSON called Smile: but
functionally JSON also works and is used for system tests).

>  That's all fine and good - i haven't seen anyone here argue that JSON
>> needs to be _the_ data format. It needs to be a _useful_ format for a wide
>> range of applications, and it is that even if it's hard-coded to be limited
>> to 31-bit integer ranges. In my implementations i have had to be very aware
>> of system-level precision limits, but i simply document them, add build
>> options to use, e.g. 64-bit integers if available, and leave it at that.
>> Those details fall comfortably into the normal range of "implementation
>> defined" details, IMO, and do _not_ (IMO) fall into JSON's realm of
>> authority (JSON just needs to tell me the BNF for reading a number, though
>> one could argue that the BNF should/does also imply certain limits). It
>> would be impossible to enforce that arbitrary implementations must support
>> arbitrarily long numbers, just as it would be silly to arbitrarily limit
>> JSON to, say, 20-bit precision.
>>
> 0.0 do indeed represent the same thing. The RDF W3C workiing group is in
> the last stages of putting its stamp of approval on JSON-LD, which presents
> the JSON numbers 0 and 0.0 to RDF as being
>

I would rather argue that cases where equality matters are minority use
cases.
So whether they are equal at format level is completely irrelevant.

You seem to be implying that this is a common concern -- I do not buy that
claim.
It does exist, but it is much more favored by theoretical thinkers than
actual system implementors.

-+ Tatu +-

---

From: Jacob Davies <[email-masked]>
Date: Wed, 10 Jul 2013 14:32:42 -0700

> 64-bit longs: google for [json large numbers bugs] for lots of reports.

The canonical one was Twitter IDs, where they had to introduce a
(string) field in their JSON APIs called "id_str" which is not
guaranteed to be equal to the obsolete-but-included (number) field
"id".

> for JavaScript-hosted implementations.

Yes, this is why I think the specification itself should note that for
interoperability with Javascript, the Javascript size limitations must
be respected, even if the specification does not require that in
itself. The number of JSON-to-Javascript implementations or uses must
surely outnumber all other uses of JSON by several orders of magnitude
so I'd argue that as far as "running code" goes the limitation on
practical, interoperable number size is quite real and surprising to
people trying to send data to Javascript from non-JS environments
where larger numbers are available. (I went so far in the Java
libraries I wrote as to encode/decode longs & Big* as strings.)

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 10 Jul 2013 21:10:07 -0700

> so I'd argue that as far as "running code" goes the limitation on

Any data to back that up? I would not assume this -- JSON is being used a
lot for service-to-service integration, as well as by native mobile clients.
Javascript as client is certainly important, but I would not assume any
particular majority (even simple one), and certainly not an order of
magnitude.

My experience wrt problem reports differs from that of John's, in that
while feature requests have been filed to allow limitation or truncation,
these have not been highly voted or actively followed. User community also
has not brought this up as a significant issue either.
I suspect this is because it is relatively easy to work around the problem
if and when it occurs. Developers are surprisingly resourceful in solving
problems.

> libraries I wrote as to encode/decode longs & Big* as strings.)

Mentioning limitations is reasonable, and Javascript can be used as a
common example. There is nothing wrong in outlining challenges, best
practices. But I don't like fearmongering.

Javascript's limited support for numbers is not a JSON-specific problem --
same goes for all data formats as well; for example, XML. This is why some
concerns wrt JSON seem overblown to me.

-+ Tatu +-

---

From: R S <[email-masked]>
Date: Wed, 10 Jul 2013 22:42:42 -0700

Is this thread making any progress over the minimal edit I proposed? I
think not, fwiw.

- Rob

---

From: Francis Galiegue <[email-masked]>
Date: Thu, 11 Jul 2013 22:47:26 +0200

On Wed, Jul 10, 2013 at 12:46 AM, Peter F. Patel-Schneider
<[email-masked]> wrote:
[...]
> grammar then what's the point of JSON?

Some JSON parsers support {de,}serializing numbers of an arbitrary
precision and/or scale. Castrating these languages just so that they
limit their possible numeric representations is castrating the value
of JSON itself.

The number production grammar is perfect as it is, I see no reason to change it.

--
Francis Galiegue, [email-masked]
JSON Schema in Java: http://json-schema-validator.herokuapp.com

---

From: Nico Williams <[email-masked]>
Date: Thu, 11 Jul 2013 16:03:52 -0500

> limit their possible numeric representations is castrating the value

That's not responsive to Peter's comment.  Just as we shouldn't leave
out bignums, we cannot require bignum support either, and people who
create open APIs with the intention to interop will have to face this
(and related) problems.

> The number production grammar is perfect as it is, I see no reason to change it.

The *grammar* is, the semantics are underspecified.

This, I think, is more fodder for: publish an update noting the
divergent interpretations, *perhaps*, in the same update, also
identify a most-likely-to-interop subset of JSON, and finally and
separately publish Internet JSON -- a subset of JSON with much less
room for interpretation and therefore much higher likelihood of
interoperability.

---

From: Jorge Chamorro <[email-masked]>
Date: Thu, 11 Jul 2013 23:09:06 +0200

> The number production grammar is perfect as it is, I see no reason to change it.

+1

-- 
( Jorge )();

---

From: Francis Galiegue <[email-masked]>
Date: Thu, 11 Jul 2013 23:28:16 +0200

On Thu, Jul 11, 2013 at 11:03 PM, Nico Williams <[email-masked]> wrote:
[...]

> The *grammar* is, the semantics are underspecified.

I don't believe the JSON RFC should _require_ any semantics. Advice,
yes, why not. But API programmers are well aware that they have to
deal with their own language's/other people's languages limitations.
So, even then it does not look necessary at all.

---

From: Nico Williams <[email-masked]>
Date: Thu, 11 Jul 2013 16:57:07 -0500

> I do not deny this. However:

Great!  :)

> So, even then it does not look necessary at all.

Well, OK, but whether we're dealing with APIs or protocols (or plain
document formats), we need to concern ourselves with compatibility
(APIs) and interoperability (protocols).  This requires more than
grammar.  If we specify no semantics then we're stuck with either the
maximal interpretation of the grammar (binary strings! bignums! allow
dup names! [since no specific handling of dup names can be considered
a maximal interpretation...]) or an agreed subset, de facto or de
jure.

Given that we're here, that we have formed a WG tasked with looking
into this, I think it's proper to look at the question of semantics.
You don't want to -- fair and noted, but we have a charter that had
consensus, and we may have consensus to actually do something in this
space, with or without you.  At some point the chairs will have to
take action and make some consensus calls (I know, they have, but
their earlier consensus calls were not, i think, phrased usefully _in
retrospect_; they may yet take another swing and succeed).

My proposed consensus calls:

1a) Minimal edit noting divergent interpretations -- yes or no?

1b) If not, should we conclude without further ado, or consider (3)
below?  -- conclude or work on I-JSON?

2) If yes to (1a), note a likely-to-interop subset of JSON in the same
update -- yes or no?

3) Regardless of the answers to (1) and (2), should we re-charter to
include publishing of an Internet JSON as a work item -- yes or no?

That should suffice for now.  If we get consensus for (1a), (2),
and/or (3), we'll surely need more consensus calls later, but I think
the content of a minimal update and a likely-to-interop subset of JSON
are both possible to produce "mechanically", so to speak, leaving only
minor controversies plus any controversies in producing I-JSON.

Nico
--

---

From: Francis Galiegue <[email-masked]>
Date: Fri, 12 Jul 2013 00:41:36 +0200

On Thu, Jul 11, 2013 at 11:57 PM, Nico Williams <[email-masked]> wrote:
[...]
> (APIs) and interoperability (protocols).

I agree, but then it is up to the protocols to define that. Say you
want to exchange prices over JSON, then it makes sense to require that
numbers be limited to two digits after the decimal point and, why not,
require that scientific notation not be used. (and 0.01 cannot be
represented reliably with a 64bit IEEE 754 floating point number, too)

But I believe this is not the role of JSON proper to define that.

> [...] If we specify no semantics then we're stuck with either the

I don't believe duplicate object names are quite the same level. This
was rather an oversight of RFC 4627 imho.

---

From: Tim Bray <[email-masked]>
Date: Thu, 11 Jul 2013 15:56:30 -0700

> represented reliably with a 64bit IEEE 754 floating point number, too)

Well yeah, but most programmers want to use generic non-app-sensitive
libraries to deserialize JSON into structs or objects or database records
or whatever, and if you send a 78-digit number that will usually break.  So
as long as you have pre-agreement and a flexible enough library, you’re
probably good.  This is the main reason I thought I-JSON was worth
defining, to maximize the chance that you can use a nice simple library at
both ends and have the lowest possible chance of breakage. -T

---

From: Nico Williams <[email-masked]>
Date: Thu, 11 Jul 2013 18:20:47 -0500

>> represented reliably with a 64bit IEEE 754 floating point number, too)

Right, we could do that.  We thought we would do that for all JSON
applications, and it turns out we can't quite do it.  It'd be nice,
then, to have a reference (or three) subset(s) of JSON, to facilitate
interoperable application protocol specifications that want to use
JSON.

> libraries to deserialize JSON into structs or objects or database records or
> whatever, and if you send a 78-digit number that will usually break.  So as
> long as you have pre-agreement and a flexible enough library, you’re
> probably good.  This is the main reason I thought I-JSON was worth defining,
> to maximize the chance that you can use a nice simple library at both ends
> and have the lowest possible chance of breakage. -T

+1.

I think we've reached a conclusion.  It's time to confirm it.  I
believe that conclusion is: RFC4627 is what it is, we can't agree
exactly on what that is, and while we can explain some/many of the
ways that it's been interpreted, we really need a
very-likely-to-interop subset (even or more than one such subsets) of
the JSON defined in that RFC.

Nico
--