Subject: [Json] AD Evaluation of draft-ietf-json-i-json-03

From: Pete Resnick <[email-masked]>
Date: Thu, 16 Oct 2014 16:55:09 -0500

It's been a lousy month. I'm finally getting to my AD Evaluations for 
this group. Here's the i-json eval; you should see text-sequences in a 
few hours.

Generally this document is looking good, but some things need tightening 
up or clarification. I think only my comment about section 3 is likely 
to cause any consternation. By section number:

2.2 - The "MUST NOT assume" and "MUST NOT expect" language is too mushy. 
You should put the requirement on the generators clearly:

OLD
    Software which implements IEEE 754-2008 binary64 (double precision)
    numbers [IEEE754] is generally available and widely used.
    Implementations which generate I-JSON messages MUST NOT assume that
    receiving implementations can process numeric values with greater
    magnitude or precision than provided by those numbers.
NEW
    Software that implements IEEE 754-2008 binary64 (double precision)
    numbers [IEEE754] is generally available and widely used, and
    receiving implementations might not be able to process numeric values
    that can't be represented as such. Therefore, implementations which
    generate I-JSON messages MUST NOT generate numeric values with
    magnitude or precision greater than provided by those numbers.

On the second bit, I think switching to a SHOULD NOT actually makes it 
stronger:

OLD
    In particular, an I-JSON sender MUST NOT expect a receiver to treat
    an integer whose absolute value is greater than 9007199254740991
    (i.e., that is outside the range [-(2**53)+1, (2**53)-1]) as an exact
    value.
NEW
    In particular, an I-JSON sender SHOULD NOT generate a numeric integer
    value greater than 9007199254740991 (i.e., that is outside the range
    [-(2**53)+1, (2**53)-1]), since a receiving implementation is likely
    not to treat it as an exact value.

That way you're using a proper 2119 SHOULD NOT:

    there may exist valid reasons in particular circumstances when the
    particular behavior is acceptable or even useful, but the full
    implications should be understood and the case carefully weighed
    before implementing...

You could make it a simple "MUST NOT generate", but the way it was 
written didn't seem like the WG was on board with that. I'm OK with it 
either way.

2.3 - Again, the "MUST NOT assume" makes this mushy:

OLD
    Implementations which generate I-JSON messages MUST NOT assume that
    the order of object members in those messages is available to
    software which receives them.

Here, I think using a MAY for the *receiving* side actually makes it 
stronger:

    The order of object members in an I-JSON message does not change the
    meaning of an I-JSON message. A receiving implementation MAY treat
    two I-JSON messages as equivalent if they differ only in the order of
    the object members.

3 - In the rest of this document, we're putting tight constraint on the 
generators of I-JSON messages. That's exactly what this document ought 
to do. But the first paragraph of this section is putting constraints on 
all interpreters. This seems bogus to me. If I write a client that 
receives I-JSON, I've got my choice of what to do with broken I-JSON. I 
can generate an error; I can use it but warn the user; I can rewrite it 
to make it proper I-JSON; I can post copies of it (and the name of the 
generator software) on my website called, "Idiots who can't implement 
I-JSON". Saying that there are circumstances (which depend on the 
protocol in question) where you MUST discard the message (say, using it 
in a security circumstance) is fine. What I object to is this protocol 
document saying that I MUST NOT act on it in any way. Let me suggest a 
replacement:

    A major advantage of using I-JSON is that receivers can avoid
    ambiguous semantics in the JSON messages it receives. This allows
    receivers to reject or otherwise disregard messages which do not
    conform to the requirements in this document for I-JSON messages.
    Protocols that use I-JSON message can be written so that receiving
    implementation are required to reject (or, as in the case of security
    protocols, not trust) messages that do not satisfy the constraints of
    I-JSON.

The second paragraph I'm OK with.

4.1 - s/avoid the use of/do not use (or you can reverse the order of the 
sentence and say "SHOULD NOT use")

4.2 -

OLD
    Such a policy is often referred to as "Must-Ignore" and is expressed
    with language such as "When receiving software encounters a protocol
    element which it does not recognize, it MUST NOT change its behavior
    as a consequence, and in particular must not fail."  The converse
    policy, often referred to as "Must-Understand", does not tolerate the
    introduction of new protocol elements, and while this has proven
    necessary in certain protocol designs, in general it has been found
    to be overly restrictive and brittle.

First of all, this is the first time I can remember seeing the terms 
"Must-Ignore" and "Must-Understand" as some sort of catch phrases, so I 
think "often referred to" is probably overstated. I also find the use of 
the terms "software" and "behavior" here kind of icky. Let me suggest an 
alternative:

NEW
    This can be referred to as a "Must-Ignore" policy, meaning that when
    an implementation encounters a protocol element which it does not
    recognize, it should treat the rest of the protocol transaction as if
    the new element simply did not appear, and in particular MUST NOT
    treat this as an error condition. The converse "Must-Understand"
    policy does not tolerate the introduction of new protocol elements,
    and while this has proven necessary in certain protocol designs, in
    general it has been found to be overly restrictive and brittle.

And then a small editorial change to the last paragraph: s/MUST NOT 
produce behavior changes/MUST be ignored.

Everything else seems good to go.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Tim Bray <[email-masked]>
Date: Thu, 16 Oct 2014 15:51:57 -0700

These almost all look like improvements to me and, absent screaming,
I'll fold them in. Except for:

>    magnitude or precision greater than provided by those numbers.

I suggest rolling in the improvement with the sentence that followed
the OLD, to read

Software which implements IEEE 754-2008 binary64 (double
precision) numbers <xref target="IEEE754"/> is generally available and
widely used, and receiving implementations might not be able to
process numeric values that can’t be represented as such.
Therefore, implementations which generate I-JSON messages MUST NOT
generate numeric values with greater magnitude or precision
than an IEEE 754 double precision number provides, for
example 1E400 or 3.141592653589793238462643383279.

>    not to treat it as an exact value.

I assume your omission of “absolute” is an oversight not an
intentional change?  Thus:

In particular, an I-JSON sender SHOULD NOT generate a numeric
integer whose absolute value is greater than 9007199254740991 (i.e.,
that is outside the range [-(2**53)+1, (2**53)-1]),
since a receiving implementation is unlikely to treat it
as an exact value.

> 3 - What
> I object to is this protocol document saying that I MUST NOT act on it in
> any way. Let me suggest a replacement:
>

This feels like a substantial change from where the WG had consensus.
I could live with it.  Others?

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Oct 2014 01:29:54 +0200

On 17 Oct 2014, at 00:51, Tim Bray <[email-masked]> wrote:

> greater magnitude or precision
> than an IEEE 754 double precision number provides

— is 1e-400 in or out?
— this appears to need a definition that allows me to derive the exact set of JSON numbers (which are decimal) that do not provide more precision than a binary64 provides (it took about 25 years to get libraries that get the binary-to-decimal conversion for this mostly right, CVE-2010-4645 and CVE-2010-4476 being good cases in point; I hope we are not actually requiring this of every I-JSON transmitter, or, worse, requiring the detection of superfluous precision by every I-JSON receiver).

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Oct 2014 01:38:16 +0200

Oh, and I missed this one:

> since a receiving implementation is unlikely to treat it

Is 9007199254740992 a “numeric integer”?
Is 900719925474099.2e1?
Do I have to make sure all my values that are larger than 2**53 have fractional components?
(That conflicts with the precision thing above.)

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Thu, 16 Oct 2014 17:04:24 -0700

Carsten, I don’t think it’s that bad. For floating-point quantities,
if inside your program they’re actually 754-compliant you can just use
%g or %ld or whatever on them and you should be safe.

If you’re dealing with bignums inside your program then yeah, your
JSON generator is going to have to have a normalize_to_ieee function
and that’s a good thing, because otherwise you could accidentally end
up sending monster values that would actually break real-world
receivers.  What are some scenarios where this would be hard to comply
with?

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: "Manger, James" <[email-masked]>
Date: Fri, 17 Oct 2014 13:13:49 +1100

+1 to Carsten's comment.
Pete Resnick's suggestion is not an improvement.

ECMAScript 5.1 §9.8.1 "ToString Applied to the Number Type" specifies that integers with up to 20 digits are written as plain integers (no exponent), eg 9555444333222111000.
I-JSON better not make this normal output from JSON.stringify() invalid.

The I-JSON rules on numbers are aimed primarily at people *designing* JSON types. If you have an integer field (eg a serial number) that will always be < 2^53 then a JSON number is a good choice. If you have a 64-bit integer field then a JSON number is a bad choice -- your spec for this type cannot call itself I-JSON.

"MUST NOT expect" and "MUST NOT assume" apply to *designers*.

--
James Manger

-----Original Message-----
From: json [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Friday, 17 October 2014 10:30 AM
To: Tim Bray
Cc: Pete Resnick; [email-masked]
Subject: Re: [Json] AD Evaluation of draft-ietf-json-i-json-03

> than an IEEE 754 double precision number provides

- is 1e-400 in or out?
- this appears to need a definition that allows me to derive the exact set of JSON numbers (which are decimal) that do not provide more precision than a binary64 provides (it took about 25 years to get libraries that get the binary-to-decimal conversion for this mostly right, CVE-2010-4645 and CVE-2010-4476 being good cases in point; I hope we are not actually requiring this of every I-JSON transmitter, or, worse, requiring the detection of superfluous precision by every I-JSON receiver).

Grüße, Carsten

_______________________________________________
json mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/json

---

From: Pete Resnick <[email-masked]>
Date: Fri, 17 Oct 2014 13:19:39 -0500

> I-JSON better not make this normal output from JSON.stringify() invalid.

Why not? I thought this was *exactly* what the purpose of I-JSON: To 
limit otherwise valid JSON productions within some particular 
constraints. If your implementation passes numbers to JSON.stringify() 
that are too large to be interpreted by an I-JSON interpreter, that 
output from JSON.stringify is *invalid* I-JSON and you've implemented 
the I-JSON spec incorrectly.

> The I-JSON rules on numbers are aimed primarily at people*designing*  JSON types. If you have an integer field (eg a serial number) that will always be<  253  then a JSON number is a good choice. If you have a 64-bit integer field then a JSON number is a bad choice -- your spec for this type cannot call itself I-JSON.
>    

MUST and SHOULD and the like to describe issues of interoperability and 
potential harm. If I am handed a block of text that purports to be 
I-JSON, whatever produced it best have followed I-JSON requirements in 
order to interoperate with me. That means that if the implementation is 
going to use a generic JSON.stringify() function, it's going to need to 
(i.e., MUST) do a range check beforehand. This is about implementation, 
not design.

> "MUST NOT expect" and "MUST NOT assume" apply to*designers*.
>    

Except that they are misuses of the term "MUST NOT". I don't know how to 
implement a "MUST NOT expect" or "MUST NOT assume".

pr

---

From: Tim Bray <[email-masked]>
Date: Fri, 17 Oct 2014 11:26:53 -0700

Yeah, what Pete said. If your ECMAScript program wants to interoperate
with the rest of the Internet, it really needs to be a little bit
careful about the size of numbers it emits.

On Fri, Oct 17, 2014 at 11:19 AM, Pete Resnick
<[email-masked]> wrote:
> ECMAScript 5.1 §9.8.1 "ToString Applied to the Number Type" specifies that
> integers with up to 20 digits are written as plain integers (no exponent),
> eg 9555444333222111000.
> Why not? I thought this was *exactly* what the purpose of I-JSON: To limit
> otherwise valid JSON productions within some particular constraints. If your
> implementation passes numbers to JSON.stringify() that are too large to be
> interpreted by an I-JSON interpreter, that output from JSON.stringify is
> *invalid* I-JSON and you've implemented the I-JSON spec incorrectly.
>
> The I-JSON rules on numbers are aimed primarily at people *designing* JSON
> types. If you have an integer field (eg a serial number) that will always be
> < 253 then a JSON number is a good choice. If you have a 64-bit integer
> field then a JSON number is a bad choice -- your spec for this type cannot
> call itself I-JSON.
>
>
>
> potential harm. If I am handed a block of text that purports to be I-JSON,
> whatever produced it best have followed I-JSON requirements in order to
> interoperate with me. That means that if the implementation is going to use
> a generic JSON.stringify() function, it's going to need to (i.e., MUST) do a
> range check beforehand. This is about implementation, not design.
>
> Pete Resnick <http://www.qualcomm.com/~presnick/>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Oct 2014 20:28:53 +0200

On 17 Oct 2014, at 20:19, Pete Resnick <[email-masked]> wrote:
> Why not? I thought this was *exactly* what the purpose of I-JSON: To limit otherwise valid JSON productions within some particular constraints. If your implementation passes numbers to JSON.stringify() that are too large to be interpreted by an I-JSON interpreter, that output from JSON.stringify is *invalid* I-JSON and you've implemented the I-JSON spec incorrectly.

I think James’ point was that there is no known interoperability problem with exchanging, say, 2**55.  So the text, as changed, rules out interchange that is unproblematic.

Grüße, Carsten

> JSON.stringify(Math.pow(2, 55))
'36028797018963970'
> JSON.stringify(Math.pow(2, 55)+10)
'36028797018963976'
> JSON.stringify(Math.pow(2, 55)-10)
'36028797018963960'
> JSON.stringify(Math.pow(2, 55)+20)
'36028797018963980'
> JSON.stringify(Math.pow(2, 55)-20)
‘36028797018963948'

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 17 Oct 2014 14:35:33 -0400

> careful about the size of numbers it emits.

I would take a totally different tack and eliminate floats entirely from I-JSON.

I have never seen a protocol use of a float as opposed to a decimal
fraction. Lattitude and Longitude are decimal fractions, not floats.

If we are doing floats then either we don't care about exact precision
or we should declare an enhanced version of the spec that can encode
floats as hex fractions rather than decimal.

Yes, I know you don't want to extend JSON but the inescapable
consequence of deciding not to modify your base technology is that you
are not going to have a result that is fit for every purpose. If
people want to encode science data in a format that can round trip
then JSON-I is not going to be that format. Regardless of what
curlicues or constraints you attempt there is no possibility that
implementations will be sufficiently robust to rely on.

---

From: Pete Resnick <[email-masked]>
Date: Fri, 17 Oct 2014 13:46:03 -0500

> I think James’ point was that there is no known interoperability problem with exchanging, say, 2**55.  So the text, as changed, rules out interchange that is unproblematic.

Ah, that's a different kettle of fish, but makes the original text no 
less problematic. If there's an interoperability problem, use a MUST NOT 
and tell me what that interoperability problem is. If there isn't an 
interoperability problem, don't use MUST NOT. I'm not committed to my 
text if I've gotten the requirement wrong; the text just needs to state 
a proper requirement (if there is one).

pr

---

From: John Cowan <[email-masked]>
Date: Fri, 17 Oct 2014 20:57:39 -0400

Pete Resnick scripsit:

> constraints. If your implementation passes numbers to
> JSON.stringify() that are too large to be interpreted by an I-JSON
> interpreter, that output from JSON.stringify is *invalid* I-JSON and
> you've implemented the I-JSON spec incorrectly.

Doubtless: if you don't comply, you are non-compliant.  But "is" does not
imply "ought".  Whatever rules I-JSON imposes ought not to disallow the
behavior of ECMAScript JSON encoders and decoders.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
A: "Spiro conjectures Ex-Lax."
Q: "What does Pat Nixon frost her cakes with?"
  --"Jeopardy" for generative semanticists

---

From: Pete Resnick <[email-masked]>
Date: Sat, 18 Oct 2014 16:13:58 -0500

> behavior of ECMAScript JSON encoders and decoders.

"Is" implies "ought" *in the context of implementing the standard*. As a 
general rule, we don't write compliance standards in the IETF; we write 
interoperability standards. "MUST" means "do it this way or you're not 
going to interoperate with someone who is also implementing this 
standard". If you are implementing an I-JSON encoder, you *ought* not 
blindly pass things to an ECMAScript JSON encoder, because an I-JSON 
decoder might fail to interpret what you send correctly. In the context 
of I-JSON, the behavior of an ECMAScript encoder *is* disallowed by the 
standard, in the sense that if you use one, you're going to produce some 
JSON which does not interoperate with other I-JSON implementations. (In 
James terminology, the normal output from JSON.stringify() *can* be 
invalid I-JSON.) If you want to use your ECMAScript JSON encoder to 
implement regular old JSON, have at it; that's not disallowed. Heck, if 
you want to use it to produce scrambled eggs, more power to you if you 
can pull it off. But it's not appropriate (by itself, without adding on 
extra code or external modules) to use it to produce I-JSON.

I hope we're saying the same thing in two different ways, just looking 
at opposite ends of the elephant. Otherwise, I'm pretty mystified.

pr

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 19 Oct 2014 00:54:25 +0200

On 18 Oct 2014, at 23:13, Pete Resnick <[email-masked]> wrote:
> 
> If you are implementing an I-JSON encoder, you *ought* not blindly pass things to an ECMAScript JSON encoder, because an I-JSON decoder might fail to interpret what you send correctly.

I don’t think this is what the WG intended to do.

Since the ECMAscript JSON encoder is the 800 pound gorilla here (and everybody else will make sure they can read at least that), and the ECMAscript number system is the most restricted one that we want to cater for, there is no point disallowing anything that comes up naturally out of the ECNAscript JSON encoder.

In other words: If we define I-JSON in such a way that it requires extra work from a JavaScript programmer, it will not see much use.  All the extra work goes to everyone else.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Sat, 18 Oct 2014 16:11:57 -0700

Right, but I think we’re actually in harmony.  ECMAScript specifies
that numbers are IEEE 754 doubles, so there should be no impedance
mismatch with I-JSON.

What our draft said was “don’t count on any more precision/range than
754 doubles, and in particular don’t try to interchange integers with
more than53 bits” and I think it did an OK job of saying that.  Pete
points out accurately that it would be more within the spirit of 2119
language to say “I-JSON producers MUST NOT produce anything that falls
outside those bounds” and what I think we’re learning is that
specifying the forbidden behavior turns out to be hard.

Note that languages which have built-in bignums that silently Do The
Right Thing, e.g. Ruby and most Lisps, are more apt to produce busted
I-JSON than any modern ECMAScript implementation.

I have no objection to what Pete’s trying to accomplish here but I
have to confess that I so far haven’t been able to think of good
language for specifying it.  Anyone have  suggestion?

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 19 Oct 2014 01:32:00 +0200

> I-JSON than any modern ECMAScript implementation.

(Or any language that has 64-bit numbers, which is pretty much “any modern language but JavaScript” at this point.)

> language for specifying it.  Anyone have  suggestion?

We spent quite some time in the WG coming up with text that is capturing the results of the discussion.

Unfortunately, there is no way to look at a JSON file and say whether the conversion from the source’s internal number representation to the JSON file represents the intent of the programmer, under the restrictions of I-JSON.  If you see 36028797018963970, there is no way to find out if the programmer intended that to be a different number from 36028797018963968 or not (it isn’t in I-JSON).

So trying to phrase this in MUST/MUST NOT predicates on the JSON file will not work.

Grüße, Carsten

PS.:
36028797018963968 is the exact decimal representation of the number 2**55 (which ECMAscript can represent exactly), but:
> JSON.stringify(36028797018963968)
'36028797018963970’
There are actually good reasons for this behavior, but there is little point in turning I-JSON into a dissertation about binary to decimal conversion.
There is also absolutely no point in ruling out 36028797018963968.

---

From: John Cowan <[email-masked]>
Date: Sat, 18 Oct 2014 21:42:25 -0400

Carsten Bormann scripsit:

> Since the ECMAscript JSON encoder is the 800 pound gorilla here
> (and everybody else will make sure they can read at least that),
> and the ECMAscript number system is the most restricted one that we
> want to cater for, there is no point disallowing anything that comes
> up naturally out of the ECNAscript JSON encoder.

That is exactly what I meant.  I-JSON *could* set any limitations it wants
to (and then I-JSON users would have to conform).  But we don't want to set
limitations that disallow what ECMAscript allows.

> In other words: If we define I-JSON in such a way that it requires
> extra work from a JavaScript programmer, it will not see much use.
> All the extra work goes to everyone else.

+1

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
I could dance with you till the cows come home.  On second thought,
I'd rather dance with the cows when you come home.
        --Rufus T. Firefly

---

From: Pete Resnick <[email-masked]>
Date: Sun, 19 Oct 2014 12:29:30 -0500

> So trying to phrase this in MUST/MUST NOT predicates on the JSON file will not work.

I have read the above a dozen times, and I'm still a bit baffled.

If I look at some data, and I can't tell what the intent of the 
generator of that data was regarding the semantics of the data, that is 
by definition an interoperability problem. We agree that if someone 
sends me 36028797018963970 as a numeric value in a JSON text (leave 
aside I-JSON), I can't tell whether they meant it to be identical to 
36028797018963968 or not. I hope we also agree that this means if I act 
on such a value, I might end up doing the wrong thing. Both sides should 
be told, "Using such a value is non-interoperable; you'd need out of 
band information to know what to do with it." That's all that "MUST NOT" 
means in the context of IETF documents.

Are we simply disagreeing about the *efficacy* of saying "MUST NOT"? 
That even if we make it a requirement of the spec, it will be ignored 
because people won't actually implement the range check? IETF specs are 
voluntary standards; they're not laws. I thought the whole point of 
I-JSON was, "If you generate this, the other side will always interpret 
it correctly." If the spec doesn't say, "In I-JSON, don't use >53 bit 
integers", what's the point of saying *anything* about number ranges?

I am mystified.

pr

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 19 Oct 2014 22:19:54 +0200

On 19 Oct 2014, at 19:29, Pete Resnick <[email-masked]> wrote:
> If I look at some data, and I can't tell what the intent of the generator of that data was regarding the semantics of the data, that is by definition an interoperability problem. We agree that if someone sends me 36028797018963970 as a numeric value in a JSON text (leave aside I-JSON), I can't tell whether they meant it to be identical to 36028797018963968 or not.

The benefit of agreeing on something like I-JSON is to take out that ambiguity.
With JSON, I really don’t know, because clearly these are different numbers, and JSON is not bound to JavaScript and the limitations of JavaScript’s number space in any way.
With I-JSON, I know that the intention of sending 36028797018963970 cannot have been different from the intention of sending 36028797018963968.  (In all likelihood, this is a floating point value, measuring and computing which has had a larger internal uncertainty than that difference.)

> I hope we also agree that this means if I act on such a value, I might end up doing the wrong thing. Both sides should be told, "Using such a value is non-interoperable; you'd need out of band information to know what to do with it." That's all that "MUST NOT" means in the context of IETF documents.
> 
> Are we simply disagreeing about the *efficacy* of saying "MUST NOT"? That even if we make it a requirement of the spec, it will be ignored because people won't actually implement the range check?

If we agreed on outlawing 36028797018963970 (because it is not exact), or 36028797018963968 (because it is not the minimum-number-of-required-digits representation that a modern floating-point binary to decimal conversion routing will generate), we would indeed lose users: Doing such a check is not inexpensive, and libraries are not prepared today to do this check.  Worse, a constrained implementation may not want to include expensive minimum-number-of-required-digits floating point code (we are talking about more code here than one would need for decent crypto).
So, independent of whether users of such a specification want to conform, it seems unwise to impose the cost on them.

> IETF specs are voluntary standards; they're not laws. I thought the whole point of I-JSON was, "If you generate this, the other side will always interpret it correctly." If the spec doesn't say, "In I-JSON, don't use >53 bit integers", what's the point of saying *anything* about number ranges?

JSON doesn’t have integers or floating point values, it only has numbers.
So the spec says:  If the objective is exactness, use an integer number within [-(2**53)+1, (2**53)-1], because for most implementations exactness is cheap to check in this range.  If an inexact number is fine, do not expect more resolution than binary64 provides, because we don’t expect receivers to have more than that.  Unfortunately, JSON does not distinguish exact and inexact numbers*), so there is no way to check for compatibility with the objective.  Retroactively classifying the floating point representations into exact and inexact ones (e.g., NR1 is always exact, NR2/NR3 are always inexact) ten years after JSON took hold doesn’t make sense.

                              oOo

(Unless specifically asked, I am going to shut up on this issue now, as the utility of I-JSON is palpable but limited, and it is already having a detrimental effect on specifications such as YANG-JSON that I didn’t see coming, but also don’t know how to curb.)

Grüße, Carsten

*) JSON was extracted from a programming language, and very few programming languages have number systems that are sophisticated enough to make that distinction.

---

From: Pete Resnick <[email-masked]>
Date: Sun, 19 Oct 2014 15:43:09 -0500

Aha. I think this is the bit that made it clear:

> So the spec says:  If the objective is exactness, use an integer number within [-(2**53)+1, (2**53)-1], because for most implementations exactness is cheap to check in this range.  If an inexact nu…

So basically what we're saying is, "Any numeric value that can be 
exactly represented by IEEE 754-2008 binary64 (double precision) numbers 
(e.g., any integer in the range [-(2**53)+1, (2**53)-1]), is perfectly 
safe to use in I-JSON. The semantics of any I-JSON numeric value that 
cannot be exactly represented by an IEEE 754-2008 binary64 (double 
precision) number are undefined, SHOULD NOT be used, and if you need 
numbers outside of that range with exact values, use a string instead." 
I'm perfectly cool with that. It's all of the "MUST NOT assume/expect" 
stuff that bugged me, but saying that the semantics are undefined seems 
just fine.

So, here's a replacement for the first two paragraphs of the section, 
rewritten that way. Does this work for folks?

Software which implements IEEE 754-2008 binary64 (double precision)
numbers [IEEE754] is generally available and widely used. In an
I-JSON implementation, numeric values that can be exactly represented
using the magnitude or precision of such numbers (e.g., any integer
in the range [-(2**53)+1, (2**53)-1]) are perfectly safe to use. The
semantics of any numeric value that cannot be exactly represented by
an IEEE 754-2008 binary64 (double precision) number (e.g., 1E400 or
3.141592653589793238462643383279) are undefined and I-JSON messages
SHOULD NOT include such numbers, since a receiver may be unable to
treat such numbers as an exact value.

pr

---

From: Tim Bray <[email-masked]>
Date: Sun, 19 Oct 2014 14:03:52 -0700

We’re on the right trail, but I don’t think “the semantics are
undefined” is the right wording. The semantics are clear, it’s just
that interchange may fail to produce the expected results if those
results rely on numeric accuracy better than IEEE754 can provide.
Also the “perfectly safe” language feels a little chatty to me.
Slight re-spin:

Software which implements IEEE 754-2008 binary64 (double precision)
numbers [IEEE754] is generally available and widely used. In an
I-JSON implementation, numeric values that can be exactly represented
using the magnitude and precision of such numbers (e.g., any integer
in the range [-(2**53)+1, (2**53)-1]) are fully interoperable.
Numeric values that cannot be exactly represented by such numbers
(e.g., 1E400 or 3.141592653589793238462643383279)  can lead to
interoperability problems and I-JSON messages
SHOULD NOT include such numbers.

On Sun, Oct 19, 2014 at 1:43 PM, Pete Resnick <[email-masked]> wrote:
>> So the spec says:  If the objective is exactness, use an integer number
>> within [-(2**53)+1, (2**53)-1], because for most implementations exactness
>> is cheap to check in this range.  If an inexact number is fine, do not
>> expect more resolution than binary64 provides, because we don’t expect
>> receivers to have more than that.  Unfortunately, JSON does not distinguish
>> exact and inexact numbers*), so there is no way to check for compatibility
>> with the objective.  Retroactively classifying the floating point
>> representations into exact and inexact ones (e.g., NR1 is always exact,
>> NR2/NR3 are always inexact) ten years after JSON took hold doesn’t make
>> sense.
>>
>
>
> So basically what we're saying is, "Any numeric value that can be exactly
> represented by IEEE 754-2008 binary64 (double precision) numbers (e.g., any
> integer in the range [-(2**53)+1, (2**53)-1]), is perfectly safe to use in
> I-JSON. The semantics of any I-JSON numeric value that cannot be exactly
> represented by an IEEE 754-2008 binary64 (double precision) number are
> undefined, SHOULD NOT be used, and if you need numbers outside of that range
> with exact values, use a string instead." I'm perfectly cool with that. It's
> all of the "MUST NOT assume/expect" stuff that bugged me, but saying that
> the semantics are undefined seems just fine.
>

---

From: "Manger, James" <[email-masked]>
Date: Mon, 20 Oct 2014 10:46:51 +1100

We can't say "numeric values that cannot be exactly represented [by a binary64 number] SHOULD NOT be included". If you are handling a floating point quantity you generally don't expected to be exact. Programs have generally already made an approximation to put the quantity in a 64-bit float, and generally need to make another approximation to represented the quantity with decimal digits.

I believe the closest binary64 value to PI is:
   0x1.921fb54442d18p1 (java notation)
This 64-bit number is *exactly* equal to the following 49-digit number:
   3.141592653589793115997963468544185161590576171875  
But PI to 49-digits is (diff underlined):
   3.141592653589793238462643383279502884197169399375
                    ---------------------------------

So what should be sent in an I-JSON message?
It depends.
If you don't need much precision you could send:
   3.14159

If you want 64-bit float precision you ought to send:
   3.141592653589793
or 3.1415926535897931
or 3.1415926535897932
or 0.3141592653589793e1
or something similar with, say, 15-17 significant digits.
You should NOT send the exact 49-digit decimal version of a binary64 value.

If you want more precision you ought to pick another syntax (eg a string)
   "3.141592653589793238462643383279502884197169399375"

The text in draft-ietf-json-i-json-03 captures this better than the alternatives suggested in this thread to date.

--
James Manger

-----Original Message-----
From: Tim Bray [mailto:[email-masked]] 
Sent: Monday, 20 October 2014 8:04 AM
To: Pete Resnick
Cc: Carsten Bormann; John Cowan; Manger, James; [email-masked]
Subject: Re: [Json] AD Evaluation of draft-ietf-json-i-json-03

We’re on the right trail, but I don’t think “the semantics are undefined” is the right wording. The semantics are clear, it’s just that interchange may fail to produce the expected results if those results rely on numeric accuracy better than IEEE754 can provide.
Also the “perfectly safe” language feels a little chatty to me.
Slight re-spin:

Software which implements IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is generally available and widely used. In an I-JSON implementation, numeric values that can be exactly represented using the magnitude and precision of such numbers (e.g., any integer in the range [-(2**53)+1, (2**53)-1]) are fully interoperable.
Numeric values that cannot be exactly represented by such numbers (e.g., 1E400 or 3.141592653589793238462643383279)  can lead to interoperability problems and I-JSON messages SHOULD NOT include such numbers.

On Sun, Oct 19, 2014 at 1:43 PM, Pete Resnick <[email-masked]> wrote:
>> So the spec says:  If the objective is exactness, use an integer 
>> number within [-(2**53)+1, (2**53)-1], because for most 
>> implementations exactness is cheap to check in this range.  If an 
>> inexact number is fine, do not expect more resolution than binary64 
>> provides, because we don’t expect receivers to have more than that.  
>> Unfortunately, JSON does not distinguish exact and inexact numbers*), 
>> so there is no way to check for compatibility with the objective.  
>> Retroactively classifying the floating point representations into 
>> exact and inexact ones (e.g., NR1 is always exact,
>> NR2/NR3 are always inexact) ten years after JSON took hold doesn’t 
>> make sense.
>>
>
>
> exactly represented by IEEE 754-2008 binary64 (double precision) 
> numbers (e.g., any integer in the range [-(2**53)+1, (2**53)-1]), is 
> perfectly safe to use in I-JSON. The semantics of any I-JSON numeric 
> value that cannot be exactly represented by an IEEE 754-2008 binary64 
> (double precision) number are undefined, SHOULD NOT be used, and if 
> you need numbers outside of that range with exact values, use a string 
> instead." I'm perfectly cool with that. It's all of the "MUST NOT 
> assume/expect" stuff that bugged me, but saying that the semantics are undefined seems just fine.
>
> numbers [IEEE754] is generally available and widely used. In an I-JSON 
> implementation, numeric values that can be exactly represented using 
> the magnitude or precision of such numbers (e.g., any integer in the 
> range [-(2**53)+1, (2**53)-1]) are perfectly safe to use. The

---

From: Pete Resnick <[email-masked]>
Date: Sun, 19 Oct 2014 21:30:53 -0500

On 10/19/14 6:46 PM, Manger, James wrote:
> On 10/19/14 4:03 PM, Tim Bray wrote:
>    

FWIW, I am fine with the above; I'm happy to leave the precise language 
to you.

> We can't say "numeric values that cannot be exactly represented [by a binary64 number] SHOULD NOT be included". If you are handling a floating point quantity you generally don't expected to be exac…

I see what you mean. Just for information: The language currently in the 
document is:

I-JSON
messages SHOULD NOT include numbers which express greater magnitude
or precision than an IEEE 754 double precision number provides, for
example 1E400 or 3.141592653589793238462643383279.

Maybe going along those same lines (talking in terms of SHOULD NOT 
include numeric values that *express* greater magnitude or precision 
than binary64) is what needs to be said.

> The text in draft-ietf-json-i-json-03 captures this better than the alternatives suggested in this thread to date.

Again, the rest of the text I'm happy to see. It's only the "MUST NOT 
assume/expect" language that I am objecting to. My other suggestions 
were just by way of trying to get at what Carsten was pointing out. If 
you can clarify the rest, I'm sure that will work for me.

pr

---

From: John Cowan <[email-masked]>
Date: Mon, 20 Oct 2014 10:57:57 -0400

Pete Resnick scripsit:

> perfectly safe to use in I-JSON.

+1

> The semantics of any I-JSON numeric value that cannot be exactly
> undefined, SHOULD NOT be used,

No, that is much too strong.  36028797018963968 and 36028797018963970
are both well-defined (by the 800-lb gorilla and other systems) as the
binary64 number whose canonical representation is 3.6028797018964e+16.

> and if you need numbers outside of that range with exact values, use a
> string instead."

+1, the key being "with exact values".

> It's all of the "MUST NOT assume/expect" stuff that bugged me,

I suggested that language, I believe (others may have done so too)
based on how the Unicode Standard talks about composed vs. decomposed
equivalents, like A-with-acute (composed) and A followed by combining
acute (decomposed).  These are equivalent, but NOT in the sense that no
process is permitted to distinguish between them.  Trivially, a process
that decomposes or composes running text distinguishes, in the sense
that one representation is left alone and the other is changed.

Instead, what Unicode requires is that no process assume that
any other process makes a distinction between equivalent forms.
In other words, the distinction can't conformantly be used to encode
out-of-band information.  Similarly, I think the distinction between
36028797018963968, 36028797018963970, and 3.6028797018964e+16 can't be
used in I-JSON to encode a meaningful distinction.  That is not the same
as banning two of them or saying their semantics is undefined.

If you say this is not an appropriate use of MUST NOT, then I'd say
change it to "cannot assume" and move on.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
In politics, obedience and support are the same thing.  --Hannah Arendt

---

From: John Cowan <[email-masked]>
Date: Mon, 20 Oct 2014 11:00:55 -0400

> SHOULD NOT include such numbers.

This is all right as far as it goes, but I don't think it goes far enough.
See my immediately previous post.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
I don't know half of you half as well as I should like, and I like less
than half of you half as well as you deserve.  --Bilbo