Subject: [Json] Interop problem report with number nasties

From: Tim Bray <[email-masked]>
Date: Sat, 21 Sep 2013 16:09:18 -0700

Deron Meranda commented on my blog at
https://www.tbray.org/ongoing/When/201x/2013/09/18/Editing-JSON#c1379796840.898519

None of these have bit me, but maybe they’re worth worrying about. Deron’s
comment:
======================================================
I know from having been involved with several JSON implementations in
different languages, that there are still some real-world interoperability
concerns that have not yet been addressed. These include:

* Some implementations can not handle strings with the character U+0000,
however encoded or escaped. This most often occurs when trying to embed raw
binary data into a JSON value, and usually affects C language based
implementations.

* Negative zeros, whether -0 or -0.0 are distinct from 0 and 0.0. The IEEE
floating point standard, as well as JavaScript, require the negative sign
for floating-point zeros to be preserved. Other implementations may not. So
encoding a -0.0 in JSON may lead to interoperability problems. I've not
seen any implementations that preserves negatives for integer zeros.

* Integer vs. floating point. The JSON spec makes no distinction between
integer or floating point numbers. However most, but not all, JSON
implementations do, so again this can lead to interoperability problems.
Some implementations may treat 1.0, 1E3, 1000E-2, etc. as integers; as
mathematically they are. Adding some language about the "interpretation" of
numbers in JSON may be welcome.

---

From: John Cowan <[email-masked]>
Date: Sat, 21 Sep 2013 21:47:57 -0400

> binary data into a JSON value, and usually affects C language based

I think this should be included.

> seen any implementations that preserves negatives for integer zeros.

This too.

> numbers in JSON may be welcome.

I can't see this as making any difference: it is purely a difference
in the internal model, which is no concern of this RFC.

-- 
Real FORTRAN programmers can program FORTRAN    John Cowan
in any language.  --Ed Post                     [email-masked]

---

From: Tim Bray <[email-masked]>
Date: Sat, 21 Sep 2013 22:54:35 -0700

Replying to myself. The point about embedded zeros seems worth a one-line
warning; as an old C-lang hacker, I can certainly believe in the
possibility of breakage.

Can someone provide an example of a situation where +/-0 would cause an
actual practical problem? I can’t.

As for integer-or-not, pfui, that’s up to implementers to not be stupid
with.  -T

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 22 Sep 2013 09:30:57 +0200

On Sep 22, 2013, at 01:09, Tim Bray <[email-masked]> wrote:

> So encoding a -0.0 in JSON may lead to interoperability problems. 

As with the other number issues, it is not the encoding of a specific value by a sender that leads to an interoperability problem, but the sender relying on a specific interpretation of that value by the recipient.
Here, an interoperability problem would be caused if the sender relies on the recipient preserving the negativeness of the number, or on it differentiating it from a positive zero.

Say, if the required application semantics is that of a lookup table (alist), and a sender sends;

[[-2, "joe"],
 [-1, "jill"],
 [-0, "bob"],
 [0, "jack"],
 [1, "alice"],
 [2, "john"]]

Similar, a sender might rely on the recipient to preserve differences in encoding the same number, such as 0 vs. 0.0 vs. 0e0.  Example:

[[0, "joe"],
 [0.0, "jill"],
 [0.00, "bob"],
 [0e0, "jack"],
 [0e00, "alice"]]

Many programming languages make a strong distinction between NR1 (1) on one hand and NR2 (1.0) and NR3 (1e0) on the other hand.  Since the JSON spec says numbers look like they do in programming languages, inferring that there is a difference between them may be an excusable misinterpretation.  (I don't think anyone would justify relying on distinguishing 0e0 and 0e00, and only very few people would claim that 0.0 clearly differs from 0.00.)

I don't want to beat dead horses, but the interoperability issues come from us not defining the data model, but relying on implementers to pick it up from their background knowledge -- that background knowledge does differ in such details.

Example 2: 1 vs. 1.0.  https://jira.talendforge.org/browse/TDI-26517
is a random example of a bug report highlighting a data model problem.
The bug reporter expected the JSON library to make available to the
application a number as an Integer type independent of whether it was
notated sign-digits only (ISO 6093 "NR1", "integer") or has a decimal
point and/or an exponent (ISO 6093 "NR2/NR3", "real").  They were
rebuffed by the developer stating that JSON numbers notated as "real"s
can't be integers in the decoder's data model.  (Historically, most
programming languages have made such a distinction in number literals.
While RFC 4627 says, "The representation of numbers is similar to that
used in most programming languages.", this is expressly about the
representation, not the data model.)

Grüße, Carsten

PS.: Fun with just two implementations:

>> require 'json'
=> true
>> JSON.load('[-0]')
=> [0]
>> JSON.load('[-0.0]')
=> [-0.0]
>> require 'oj'
=> true
>> Oj.load('[-0]')
=> [0]
>> Oj.load('[-0.0]')
=> [-0.0]

But:

>> Oj.load('[-0e0]')
=> [0]
>> JSON.load('[-0e0]')
=> [-0.0]

So this is an example where
1) the apparent floatingpointness of the number influences the preservation of the negativeness
2) the interpretation of the floatingpointness differs between implementations.

---

From: John Cowan <[email-masked]>
Date: Sun, 22 Sep 2013 11:23:57 -0400

> actual practical problem? I can’t.

tl;dr answer: Yes, but in those circumstances other problems with JSON
and JSON implementations probably swamp it.

The introduction of -0.0 into IEEE 754 primarily serves the needs of
scientific computing, where it may be important to correctly preserve the
sign of a result which is larger than the largest representable negative
float, namely -2^-149.  Negative zero represents an unspecified number x
such that -2^-149 < x <= 0.  By the same token, ordinary 0.0 represents
an unspecified number y such that 0 <= y < 2^-149.

Although 0.0 and -0.0 compare equal, applying certain functions to them
produce results that are unequal.  (Just another reason why floating-point
numbers aren't really numbers.)  In particular, dividing by 0.0 produces
positive infinity, but dividing by -0.0 produces negative infinity,
which are as different as they can be.  Similarly, atan2(0.0, -0.0) is
an approximation to \pi, whereas atan2(-0.0, -0.0) is an approximation
to -\pi.  (The function atan2(y, x) computes the principal arc tangent
of the complex number x + yi; see <https://en.wikipedia.org/wiki/Atan2>
for motivation and explanations.)

The applicability to JSON is, of course, that if scientific data is being
passed via JSON, it should reach the other end accurately and as-is.
Against this is the fact that there is no JSON representation for
positive infinity, negative infinity, or NaN, so they would have to
be sent as strings, and sending -0.0 as a string wouldn't be that much
of an additional burden.  What is more, most languages don't guarantee
either accurate printing or accurate reading of floating-point numbers.

A safe approach, then, would be to send floats as quoted strings of
hex digits.  In practice, people probably won't bother and will feed
arbitrary floats to JSON generators, which will print out whatever
the implementation of the floating-point printing routine (often
ultimately sprintf()) they are using prints out, most likely without
bothering to check for the four special cases.  This is what happens
in Scheme, which *does* require accuracy in floating-point reading and
printing, and (in later versions of the standard) prescribes syntax
for non-finite numbers, but a good many implementations get it wrong:
see <http://trac.sacrideo.us/wg/wiki/NonFiniteNumbers> for details.

-- 
If you have ever wondered if you are in hell,       John Cowan
it has been said, then you are on a well-traveled   http://www.ccil.org/~cowan
road of spiritual inquiry.  If you are absolutely   [email-masked]
sure you are in hell, however, then you must be
on the Cross Bronx Expressway.          --Alan Feuer, NYTimes, 2002-09-20

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 23 Sep 2013 08:57:58 -0700

<no hat>

On Sep 21, 2013, at 4:09 PM, Tim Bray <[email-masked]> wrote:

> * Some implementations can not handle strings with the character U+0000, however encoded or escaped. This most often occurs when trying to embed raw binary data into a JSON value, and usually affects C language based implementations.

Those are broken implementations, not non-interoperable implementations. There is no sense in this WG trying to list the valid Unicode characters that some broken implementations have problems with.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 23 Sep 2013 09:00:31 -0700

<no hat>

> Here, an interoperability problem would be caused if the sender relies on the recipient preserving the negativeness of the number, or on it differentiating it from a positive zero.

+1. There are all sorts of ways that JSON parsers can interpret numbers that will surprise other implementations but make perfect sense on the platform in question. I don't see any value to trying to list all of these.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 23 Sep 2013 09:05:00 -0700

<no hat>

On Sep 21, 2013, at 4:09 PM, Tim Bray <[email-masked]> wrote:

> * Integer vs. floating point. The JSON spec makes no distinction between integer or floating point numbers. However most, but not all, JSON implementations do,

Fully agree, but...

> so again this can lead to interoperability problems.

No, it leads to an interpretation problem.

> Some implementations may treat 1.0, 1E3, 1000E-2, etc. as integers; as mathematically they are. Adding some language about the "interpretation" of numbers in JSON may be welcome.

That sounds useful. Proposed text:

JSON numbers can be interpreted in many ways. For example, the number 1 and the number 1.0 might be considered the same by some interpreters but different in others; similarly, 0, +0, and -0 might be considered to be the same or different. JSON encoders should not make any assumptions how JSON decoders will handle any particular type of number.

--Paul Hoffman

---

From: "John Levine" <[email-masked]>
Date: 23 Sep 2013 16:28:34 -0000

>+1. There are all sorts of ways that JSON parsers can interpret numbers that will surprise other implementations but make perfect
>sense on the platform in question. I don't see any value to trying to list all of these.

+0.5.  I worry that a list of surprises will be misinterpreted as
definitive, particularly if someone comes up with some corner case
later that we didn't think of.

I do think it's reasonable give examples, something along the lines of

JSON parsers may interpret strings that represent numbers that are
mathematically equal as different values.  Examples might be, 0, +0
and -0, or 1, 1.0, and 0.1e1.

R's,
John

---

From: John Cowan <[email-masked]>
Date: Mon, 23 Sep 2013 12:44:39 -0400

Paul Hoffman scripsit:

> JSON numbers can be interpreted in many ways. For example, the number
> 1 and the number 1.0 might be considered the same by some interpreters
> but different in others; similarly, 0, +0, and -0 might be considered to
> be the same or different. JSON encoders should not make any assumptions
> how JSON decoders will handle any particular type of number.

+1, but needs reformulation into interoperability language, something like this:

   JSON numbers can be interpreted in many ways. For example, the number 1
   and the number 1.0 might be considered the same by some processes but
   different by others; similarly, 0, +0, and -0 might be considered to be
   the same or different. Therefore, it is not interoperable for a process
   that generates JSON numbers to make assumptions about how processes that
   consume them will handle the numbers.

-- 
John Cowan    http://ccil.org/~cowan  [email-masked]
Arise, you prisoners of Windows / Arise, you slaves of Redmond, Wash,
The day and hour soon are coming / When all the IT folks say "Gosh!"
It isn't from a clever lawsuit / That Windowsland will finally fall,
But thousands writing open source code / Like mice who nibble through a wall.
        --The Linux-nationale by Greg Baker

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 23 Sep 2013 09:51:38 -0700

>   consume them will handle the numbers.

+1 to John's wording over mine.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Mon, 23 Sep 2013 13:12:31 -0400

Paul Hoffman scripsit:

> Those are broken implementations, not non-interoperable
> implementations. There is no sense in this WG trying to list the valid
> Unicode characters that some broken implementations have problems with.

At least 9 out of the 16 JSON implementations written in C and listed
at json.org have some form of the U+0000 bug.  I think that makes it well
worth warning about.  The hall of shame: LibU, json-c, json-parser,
WJElement, M's JSON parser aka mjson, cJSON, Jansson, parson, nxjson.
I determined this by code inspection rather than testing, so there may
be others.  Some exhibit the bug only in the form of a call to parse
JSON from a C string.

-- 
"But I am the real Strider, fortunately,"       John Cowan
he said, looking down at them with his face     [email-masked]
softened by a sudden smile.  "I am Aragorn son  http://www.ccil.org/~cowan
of Arathorn, and if by life or death I can
save you, I will."  --LotR Book I Chapter 10

---

From: Tim Bray <[email-masked]>
Date: Mon, 23 Sep 2013 11:18:11 -0700

<editorial-hat>
So far, all the changes we’ve made are of the form “If you do $X,
$JSON_construct_Y will be interoperable in the sense that implementations
will $Z. If you don’t problems of the form $W can happen.”

I think it would be useful if we could stay with that rhythm, as anything
that can be phrased that way is pretty clearly within our charter, and
furthermore has the advantage of being actionable.

So I think it’d be good if people who are proposing changes (including me)
could try to package their thoughts up that way.
</editorial-hat>

On Mon, Sep 23, 2013 at 9:28 AM, John Levine <[email-masked]> wrote:

> >+1. There are all sorts of ways that JSON parsers can interpret numbers
> that will surprise other implementations but make perfect
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 23 Sep 2013 11:27:56 -0700

<no hat>

On Sep 23, 2013, at 11:18 AM, Tim Bray <[email-masked]> wrote:

> <editorial-hat>
> So far, all the changes we’ve made are of the form “If you do $X, $JSON_construct_Y will be interoperable in the sense that implementations will $Z. If you don’t problems of the form $W can happen.”
> 
> I think it would be useful if we could stay with that rhythm, as anything that can be phrased that way is pretty clearly within our charter, and furthermore has the advantage of being actionable.
> 
> So I think it’d be good if people who are proposing changes (including me) could try to package their thoughts up that way.  
> </editorial-hat>

For numbers like -0.0, do we have any clue at all about what ($X $Z) is? I propose we don't.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Mon, 23 Sep 2013 11:36:51 -0700

> For numbers like -0.0, do we have any clue at all about what ($X $Z) is? I

Well, that could be evidence that this isn’t worth putting in, which I’m OK
with because I don't think it’s really actionable.  But let’s see.

“JSON numbers which use 0 or 0.0 for zero, as opposed to -0 or -0.0, will
be interoperable in that software implementations will agree on the zero
value. The difference between 0 and -0 is significant in some
numerically-intensive applications, but implementations which read JSON
texts cannot be relied upon to preserve that distinction.”

The equivalent text for x.0 vs x is left as an exercise for someone who
thinks it’s important.

---

From: John Cowan <[email-masked]>
Date: Mon, 23 Sep 2013 17:21:28 -0400

> texts cannot be relied upon to preserve that distinction.”

+1

> The equivalent text for x.0 vs x is left as an exercise for someone who

I certainly don't.  From the Scheme viewpoint, x.0 is an inexact integer,
whereas x is an exact integer, but both are considered integers.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
You cannot enter here.  Go back to the abyss prepared for you!  Go back!
Fall into the nothingness that awaits you and your Master.  Go! --Gandalf