Subject: [Json] All those number ideas

From: Tim Bray <[email-masked]>
Date: Mon, 23 Sep 2013 18:57:34 -0700

How about we enhance the text following the ABNF in the “Numbers” section
to say:

This specification allows implementations to set limits on the range of
numbers accepted. Since software which correctly implements IEEE 754-2008
[IEEE754] is generally available and widely used, good interoperability can
be achieved by limiting numbers in JSON texts to those within the precision
and magnitude expressible in an IEEE 754 binary64 (double precision)
number. Note that when such software is used, numbers which are integers,
in the range [(-2**53)+1, (2**53)-1], and are represented without "frac"
parts, for example as 3 not 3.0, are interoperable in the sense that
implementations will agree exactly on the numeric values.

Attempting to represent numbers that cannot be exactly encoded as an IEEE
754:2008 binary64 number, such as 1E400, 9007199254740993, or
3.141592653589793238462643383279, may cause interoperability problems.

Numbers which represent zero without a sign, for example as 0 or 0.0 not -0
or +0.0, are interoperable in the sense that software implementations will
agree on the zero value. Signed zeros are significant in some
numerically-intensive applications, but implementations which read JSON
texts cannot be relied upon to preserve that distinction.

---

From: R S <[email-masked]>
Date: Mon, 23 Sep 2013 19:11:13 -0700

On Mon, Sep 23, 2013 at 6:57 PM, Tim Bray <[email-masked]> wrote:

>
>  Signed zeros are significant in some numerically-intensive applications,
> but implementations which read JSON texts cannot be relied upon to preserve
> that distinction.
>

Aren't these implementations just buggy?

This works in the JavaScript implementations I checked:

> var input = '{"pos":0, "neg":-0}'
undefined
> var obj = JSON.parse(input)
undefined
> 1/obj.pos
Infinity
> 1/obj.neg
-Infinity

- Rob

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 23 Sep 2013 19:37:08 -0700

<no hat>

On Sep 23, 2013, at 7:11 PM, R S <[email-masked]> wrote:

>  Signed zeros are significant in some numerically-intensive applications, but implementations which read JSON texts cannot be relied upon to preserve that distinction.
> 

Not that I can see from RFC 4627. It is silent on the meaning of numbers.

> > var obj = JSON.parse(input)

JSON != JavaScript, of course.

--Paul Hoffman

---

From: R S <[email-masked]>
Date: Mon, 23 Sep 2013 19:43:22 -0700

> JSON != JavaScript, of course.

I think the issue has more to do with IEEE754, but I am not a floating
point lawyer.

n.b. that the sign of zero may be disregarded by a language's equality
operator (it is in JavaScript)

$ ruby -v
ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]

$ irb
2.0.0p247 :001 > require 'yajl'
 => true

2.0.0p247 :002 > parser = Yajl::Parser.new
 => #<Yajl::Parser:0x00000000c0f6c0>

2.0.0p247 :003 > a = parser.parse('{"a":0.0, "b":-0.0}')
 => {"a"=>0.0, "b"=>-0.0}

2.0.0p247 :004 > a['a'] == a['b']
 => true

2.0.0p247 :005 > 1/a['a']
 => Infinity

2.0.0p247 :006 > 1/a['b']
 => -Infinity

- Rob

---

From: "John Levine" <[email-masked]>
Date: 24 Sep 2013 02:50:04 -0000

>Aren't these implementations just buggy?

No.  It's perfectly legitimate to have an implementation that only
handles integers, if you know that's all your application will need.
They don't do signed zeros.

There are also still some non-IEEE floating point formats in use that
don't reliably preserve the sign of a zero.

R's,
John

---

From: Tim Bray <[email-masked]>
Date: Mon, 23 Sep 2013 20:29:20 -0700

So, we observe that two different json libraries in one language have
incompatible behavior. I think this is evidence that the proposed language
is useful.  -T

On Mon, Sep 23, 2013 at 7:43 PM, R S <[email-masked]> wrote:

> On Mon, Sep 23, 2013 at 7:37 PM, Paul Hoffman <[email-masked]>wrote:
>
>>
>>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

---

From: R S <[email-masked]>
Date: Mon, 23 Sep 2013 20:46:02 -0700

Fully disagree with this rationale and the proposed text. The rationale is
wrong because it can be used to allow an exception for any buggy
implementation, and the text is wrong because "implementations which read
JSON texts" can be relied upon to preserve negative zeros.

- Rob

---

From: John Cowan <[email-masked]>
Date: Tue, 24 Sep 2013 00:25:01 -0400

Tim Bray scripsit:

> in the range [(-2**53)+1, (2**53)-1], 

Since [] means something different in JSON than it does here, I suggest
"in the range (-2**53)+1 to (2&&53)-1 inclusive" instead.

Otherwise +1.

-- 
John Cowan   http://ccil.org/~cowan    [email-masked]
We want more school houses and less jails; more books and less arsenals;
more learning and less vice; more constant work and less crime; more
leisure and less greed; more justice and less revenge; in fact, more of
the opportunities to cultivate our better natures.  --Samuel Gompers

---

From: John Cowan <[email-masked]>
Date: Tue, 24 Sep 2013 00:31:38 -0400

> operator (it is in JavaScript)

Just so.  Indeed, a language which conforms either absolutely or
conditionally to IEEE 754 MUST treat 0.0 and -0.0 as equal, even though
they don't produce equal results when certain functions are applied
to them.

> incompatible behavior. I think this is evidence that the proposed language

Sorry, which two libraries?

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
Adam [...] did not want the apple for the apple's sake, he wanted it only
because it was forbidden. The mistake was not forbidding the serpent;
then he would have eaten the serpent. --Mark Twain

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 Sep 2013 06:49:03 +0200

On Sep 24, 2013, at 05:46, R S <[email-masked]> wrote:

> Fully disagree with this rationale and the proposed text. The rationale is wrong because it can be used to allow an exception for any buggy implementation, and the text is wrong because "implementations which read JSON texts" can be relied upon to preserve negative zeros.

One's complement is a bit out of fashion, no?

As Paul said: JSON is silent on the specific number representation to be used by a receiving implementation.  There is a general expectation that conversion (commonly from JSON's decimal form to *some* binary form) is happening on decoding, and the specifics of that conversion, including the set of representable numbers and how the set of numbers representable in JSON is mapped to that set, is implementation specific.  (RFC 4627 just fails to say that these may differ in precision, not only range; this is an erratum, not a feature.)

Integer-only implementations of JSON are acceptable.
Integers outside of the range between -(2**53)+1 and +(2**53)-1 are endangered in practice, as floating-point-only implementations with that precision abound.
An implementation that cannot distinguish 2**53 from 2**53+1 is exactly as buggy as one that cannot distinguish +0.0 from -0.0: not buggy at all.

The much more interesting point was that there are implementations that can distinguish +0.0 from -0.0, but don't distinguish +0 from -0. These implementations then go on to differ in whether they distinguish +0e0 from -0e0!
All bets are off here.  

Many JSON implementations distinguish 0 from 0.0.  I'm still waiting for a report from the first implementation that distinguishes 0.0 from 0.00; there would be nothing wrong with that.

Grüße, Carsten

PS.: Plus signs (and other arithmetic) in the above for exposition only; they are not allowed in JSON.

---

From: R S <[email-masked]>
Date: Mon, 23 Sep 2013 21:56:01 -0700

> Integer-only implementations of JSON are acceptable.

Tim's proposal reads "Since software which correctly implements IEEE
754-2008 [IEEE754] is generally available and widely used".

Should it instead read "Since software which [in]correctly implements IEEE
754-2008 [IEEE754] is generally available and widely used" ?

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 Sep 2013 07:27:07 +0200

My summary from the discussion so far:

On Sep 24, 2013, at 03:57, Tim Bray <[email-masked]> wrote:

> How about we enhance the text following the ABNF in the “Numbers” section to say:
> 
> This specification allows implementations to set limits on the range of numbers accepted. Since software which correctly

Drop the "correctly" (doesn't add anything).

> implements IEEE 754-2008 [IEEE754] is generally available and widely used, good interoperability can be achieved by limiting numbers in JSON texts to those within the precision and magnitude expressible in an IEEE 754 binary64 (double precision) number. Note that when such software is used, numbers which are integers, in the range [(-2**53)+1, (2**53)-1], and are represented without "frac" parts, for example as 3 not 3.0,

or exp parts...
Maybe say "use only the int part and optionally the minus".

> are interoperable in the sense that implementations will agree exactly on the numeric values.

> 
> Attempting to represent numbers that cannot be exactly encoded as an IEEE 754:2008 binary64 number, such as 1E400, 9007199254740993, or 3.141592653589793238462643383279, may cause interoperability problems.

As I mentioned before, 1.1 "cannot be exactly encoded as an IEEE 754:2008 binary64 number", and there are fewer interop problems with that than with the first two examples you give here.
Also, it is not the attempt to represent 9007199254740993, but the expectation that an implementation will be able to distinguish 9007199254740993 from 9007199254740992, that causes interop problems (same with 3.141592653589793238462643383279).

> Numbers which represent zero without a sign, for example as 0 or 0.0 not -0 or +0.0,

+0.0 isn't JSON.

> are interoperable in the sense that software implementations will agree on the zero value. Signed zeros are significant in some numerically-intensive applications, but implementations which read JSON texts cannot be relied upon to preserve that distinction.

(Maybe be more specific that this is about the distinction between 0.0 and -0.0, not about the presence or absence of a sign, even though that is actually the same in JSON.)

Grüße, Carsten

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 24 Sep 2013 14:05:41 +0200

>> var obj = JSON.parse(input)

However, ecmascript's JSON.stringify does not preserve the sign, and so

  1 / JSON.parse(JSON.stringify(JSON.parse('-0')))

is positive infinity. Since `(-0).toString()` and `String(-0)` are `0`
that should not be too surprising, but such anomalies attract bugs, and
sure enough, Opera 12.x's JSON.parse does not preserve the zero sign.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Jorge Chamorro <[email-masked]>
Date: Tue, 24 Sep 2013 14:15:05 +0200

On 24/09/2013, at 03:57, Tim Bray wrote:

> numbers which are integers, in the range [(-2**53)+1, (2**53)-1], and are represented without "frac" parts

I'd move that minus out of the parenthesis: [-(2**53)+1, (2**53)-1]

-- 
( Jorge )();

---

From: Tim Bray <[email-masked]>
Date: Tue, 24 Sep 2013 13:37:44 -0700

Carsten, I like your edits, but:

On Mon, Sep 23, 2013 at 10:27 PM, Carsten Bormann <[email-masked]> wrote:

> > Attempting to represent numbers that cannot be exactly encoded as an
> IEEE 754:2008 binary64 number, such as 1E400, 9007199254740993, or
> As I mentioned before, 1.1 "cannot be exactly encoded as an IEEE 754:2008
> binary64 number", and there are fewer interop problems with that than with
> the first two examples you give here.
> Also, it is not the attempt to represent 9007199254740993, but the
> expectation that an implementation will be able to distinguish
> 9007199254740993 from 9007199254740992, that causes interop problems (same
> with 3.141592653589793238462643383279).
>

I totally don’t get what you’re trying to say about 1.1.  It seems to me
that despite the fact that lots of decimal fractions can’t be represented
in binary FP, conforming to 754 gives, as we say “good interoperability”,
in the sense that people who worry about arithmetic mean.  And that
round-tripping 1.1 through JSON is generally unlikely to break your
program. Maybe you’re arguing for more exactness, something like
“9007199254740993 (2**53+1) and 3.141592653589793238462643383279 express
more precision than is available in an IEEE754 binary64, and IE400 exceeds
the greatest possible magnitude expressible in such a number.”

If I’m missing your point (quite likely), please provide alternative
language that  would give better guidance about the kinds of numeric values
to avoid, and why.

 -T

> > are interoperable in the sense that software implementations will agree
> on the zero value. Signed zeros are significant in some
> (Maybe be more specific that this is about the distinction between 0.0 and
> -0.0, not about the presence or absence of a sign, even though that is
> actually the same in JSON.)
>
> Grüße, Carsten
>
>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 24 Sep 2013 23:43:13 +0200

> I totally don’t get what you’re trying to say about 1.1.  

I'm trying to say there is no difference between 1.1 and the other numbers with respect to the question:  "can it be exactly encoded as an IEEE 754:2008 binary64 number"?

The difference that you are most likely after is that most implementations will convert this number back to decimal 1.1, even though it is internally represented as  1.100000000000000088817841970012523233890533447265625.

But we shouldn't try to single out the numbers that "cannot be exactly encoded as an IEEE 754:2008 binary64 number", because there are lot of those, and not all of their uses cause interop problems.

> It seems to me that despite the fact that lots of decimal fractions can’t be represented in binary FP, conforming to 754 gives, as we say “good interoperability”, in the sense that people who worry about arithmetic mean.  And that round-tripping 1.1 through JSON is generally unlikely to break your program.

That depends on whether the 1.1 was meant to be precise or not.  You cannot say by looking at the number.

> Maybe you’re arguing for more exactness, something like “9007199254740993 (2**53+1) and 3.141592653589793238462643383279

But again, you cannot say by looking at the JSON whether the given precision actually matters or not.  That somebody generated 3.141592653589793238462643383279 with all those digits may be absolutely crucial for the application or the 3.141592653589793115997963468544185161590576171875 you actually get from IEEE 754 may well be good enough.  Again, you cannot tell by looking at the JSON.

> express more precision

1.1 expresses a *lot* of precision.  It is precisely 10 % more than 1.0...
There is no semantic difference in JSON between
1.1
and
1.100000000000000000000000000000000000000000000000000000

> than is available in an IEEE754 binary64, and IE400 exceeds the greatest possible magnitude expressible in such a number.”  

Converting 1e400 into binary64 leads to a more catastrophic loss of precision, indeed.
(But if the application is trying to express an upper limit on an electron count, there still is no practical difference between 1e400 and Infinity.)

> If I’m missing your point (quite likely), please provide alternative language that  would give better guidance about the kinds of numeric values to avoid, and why.

How about:

Representing numbers that cannot be exactly encoded as an IEEE 754:2008 binary64 number, such as 1E400, 9007199254740993, 3.141592653589793238462643383279, or even 1.1, may cause interoperability problems if the application relies on the decimally encoded JSON number being exactly preserved in the decimal-to-binary conversion during the decoding process.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Tue, 24 Sep 2013 15:07:30 -0700

On Tue, Sep 24, 2013 at 2:43 PM, Carsten Bormann <[email-masked]> wrote:

> How about:
>
> Representing numbers that cannot be exactly encoded as an IEEE 754:2008
> binary64 number, such as 1E400, 9007199254740993,
> 3.141592653589793238462643383279, or even 1.1, may cause interoperability
> problems if the application relies on the decimally encoded JSON number
> being exactly preserved in the decimal-to-binary conversion during the
> decoding process.
>

OK, but it feels to me that there’s an important difference between the
normal binary/decimal fraction you get with 1.1, and the
guaranteed-breakage you’re getting with the others.

---

From: Tim Bray <[email-masked]>
Date: Tue, 24 Sep 2013 15:20:54 -0700

I meant to say: OK, but it feels to me that there’s an important difference
between the normal binary/decimal friction you get with 1.1, and the
guaranteed-breakage you’re getting with the others.  Friction not fraction.

---

From: "John Levine" <[email-masked]>
Date: 25 Sep 2013 00:39:02 -0000

>The difference that you are most likely after is that most implementations will convert this number back to decimal 1.1, even though
>it is internally represented as  1.100000000000000088817841970012523233890533447265625.

My zSeries mainframe represents 1.1 as 1.100000 and will generally
print that value as 1.1 or maybe 1.10 depending on the picture.
That's because it has decimal floating point.  (See the z/Architecture
Principles of Operation, chapter 20.)

Really, anyone who thinks that JSON is tied to IEEE or any other
particular internal numeric representation is just mistaken, and
the sooner we get past that misconception, the better.

R's,
John

---

From: John Cowan <[email-masked]>
Date: Tue, 24 Sep 2013 21:55:09 -0400

> Principles of Operation, chapter 20.)

Those decimal floats are in fact IEEE 754:2008 compliant.

I note that if you compile a HLL runtime with gcc
-funsafe-math-optimizations, it no longer distinguishes 0.0 and -0.0.

> the sooner we get past that misconception, the better.

+1

-- 
John Cowan      [email-masked]
        "Not to know The Smiths is not to know K.X.U."  --K.X.U.

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 25 Sep 2013 14:19:17 +1000

My guess is that neither 3.141592653589793238462643383279 nor 1.1 cause interop problems (in general). I assume (hope) basically all JSON parsers will parse both, even if they approximate the number in their internal binary representation. I doubt that rounding causes many interop problems as rounding comes with the territory when dealing with floats. For instance, equality testing is almost always dodgy with floats (regardless of whether JSON is involved), and < and/or > should be used instead.

Perhaps say:
  If you exchange numbers expecting no more precision or range than
  available with an IEEE 64-bit double (eg 52 bits of precision,
  which is about 16 decimal digits), then a JSON number will be
 interoperable in the sense that implementations will
 approximate the number within the expected precision. If you expect
 a greater precision or range then problems of rounding can happen.

For integers, on the other hand, we need a different warning. Perhaps:

  If you exchange an integer x, where -2**53 < x < 2**53, in JSON
 without a fraction or exponent part, the number will be interoperable
  in the sense that most implementations will agree on the value precisely.
  Integers outside this range, or expressed with fraction or exponent parts,
  can face problems of being approximated, or not treated as integers
  (eg returned by an API only as a double).

  Note: 2**53 = 9007199254740992

--
James Manger

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tim Bray
Sent: Wednesday, 25 September 2013 8:21 AM
To: Carsten Bormann
Cc: [email-masked]
Subject: Re: [Json] All those number ideas

I meant to say: OK, but it feels to me that there’s an important difference between the normal binary/decimal friction you get with 1.1, and the guaranteed-breakage you’re getting with the others.  Friction not fraction.

On Tue, Sep 24, 2013 at 3:07 PM, Tim Bray <[email-masked]<mailto:[email-masked]>> wrote:
On Tue, Sep 24, 2013 at 2:43 PM, Carsten Bormann <[email-masked]<mailto:[email-masked]>> wrote:
How about:

OK, but it feels to me that there’s an important difference between the normal binary/decimal fraction you get with 1.1, and the guaranteed-breakage you’re getting with the others.

Grüße, Carsten

---

From: Jorge Chamorro <[email-masked]>
Date: Wed, 25 Sep 2013 11:04:18 +0200

>   Note: 2**53 = 9007199254740992

Even though that's ok, I'd prefer to see the +1 and the -1 after the -/+ 2**53s.

If not, people tend to think that 2**53 is ok too.

If they see the +/-1 they'll remember it, if instead it's got to be 'seen' by the less-than-s, it may easily pass unnoticed.

-- 
( Jorge )();

---

From: Tim Bray <[email-masked]>
Date: Wed, 25 Sep 2013 17:01:49 -0700

I’m inclined to adopt James’ language here, edited to flow more like the
other cautions, but the “expecting no more precision or range than IEEE”
seems like about the right warning to give.

On Tue, Sep 24, 2013 at 9:19 PM, Manger, James H <
[email-masked]> wrote:

> My guess is that neither 3.141592653589793238462643383279 nor 1.1 cause
> interop problems (in general). I assume (hope) basically all JSON parsers
> will parse both, even if they approximate the number in their internal
> binary representation. I doubt that rounding causes many interop problems
> as rounding comes with the territory when dealing with floats. For
> instance, equality testing is almost always dodgy with floats (regardless
> of whether JSON is involved), and < and/or > should be used instead.****
>
> ** **
>
> Perhaps say:****
>
>   If you exchange numbers expecting no more precision or range than****
>
>   available with an IEEE 64-bit double (eg 52 bits of precision,****
>
>   which is about 16 decimal digits), then a JSON number will be****
>
>  interoperable in the sense that implementations will****
>
>  approximate the number within the expected precision. If you expect****
>
>  a greater precision or range then problems of rounding can happen.****
>
> ** **
>
> ** **
>
> For integers, on the other hand, we need a different warning. Perhaps:****
>
> ** **
>
>   If you exchange an integer x, where -2**53 < x < 2**53, in JSON****
>
>  without a fraction or exponent part, the number will be interoperable****
>
> ****
>   Integers outside this range, or expressed with fraction or exponent
> parts,****
>
>   can face problems of being approximated, or not treated as integers****
>
>   (eg returned by an API only as a double).****
>
> ** **
>
>   Note: 2**53 = 9007199254740992****
>
>   ****
>
> ** **
>
> --****
>
> James Manger****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Tim Bray
> *Sent:* Wednesday, 25 September 2013 8:21 AM
> *To:* Carsten Bormann
> *Cc:* [email-masked]
> *Subject:* Re: [Json] All those number ideas****
>
> ** **
>
> I meant to say: OK, but it feels to me that there’s an important
> difference between the normal binary/decimal friction you get with 1.1, and
> the guaranteed-breakage you’re getting with the others.  Friction not
> fraction.****
>
> ** **
>
> On Tue, Sep 24, 2013 at 3:07 PM, Tim Bray <[email-masked]> wrote:***
> *
>
> On Tue, Sep 24, 2013 at 2:43 PM, Carsten Bormann <[email-masked]> wrote:****
>
> How about:
>
> decoding process.****
> ** **
> guaranteed-breakage you’re getting with the others.****
>
>  ****
>
>
> Grüße, Carsten****
>
> ** **
>
> ** **
>

---

From: Tony Hansen <[email-masked]>
Date: Wed, 25 Sep 2013 23:01:40 -0400

On 9/25/2013 8:01 PM, Tim Bray wrote:
> I’m inclined to adopt James’ language here, edited to flow more like
> the other cautions, but the “expecting no more precision or range than
> IEEE” seems like about the right warning to give.

I can live with James' language.

Tony Hansen