Subject: Re: [Json] JSON Schema (Was: JSON Concluded? Well, maybe not)

From: Pete Cordell <[email-masked]>
Date: Mon, 1 Jan 2018 14:30:27 +0000

On the JSON schema topic, Andy Newton and I, with the help of a number 
of people active in the IETF, have been working on something called 
"JSON Content Rules".  The current I-D is at:

   https://www.ietf.org/id/draft-newton-json-content-rules-09.txt

The aim is for it to be easy to write, and easy to understand.  It's a 
"JSON-like" syntax, as it was felt restricting it to pure JSON made the 
syntax harder to write and read.

There's a few more details at the following address, including details 
of implementations so far:

   http://json-content-rules.org

We're happy to discuss it with anyone, either here, using my send 
address, or via [email-masked] .

Happy New Year!

Pete.

On 31/12/2017 18:50, Tim Bray wrote:
> If I were going to make an argument for keeping the WG alive, it would 
> require someone to get energized and submit an I-D for a plausible JSON 
> schema system, or maybe even a much-needed formalization of JSONPath. We 
> use JSONPath a lot at AWS.
> 
> On Sun, Dec 31, 2017 at 10:43 AM, Richard Gibson 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     On the topic of JSON normalization, I believe that
>     canonicaljson-spec
>     <http://gibson042.github.io/canonicaljson-spec/> covers all cases
>     while respecting prior art like
>     https://tools.ietf.org/html/draft-staykov-hu-json-canonical-form-00
>     <https://tools.ietf.org/html/draft-staykov-hu-json-canonical-form-00> and
>     RFC 7638. I'd like to get it published as an RFC to handle scenarios
>     like those mentioned by Anders Rundgren, but am not sure how to go
>     about doing so. Is this a good place to ask for assistance?
> 
>     On Sun, Dec 31, 2017 at 12:24 AM, Anders Rundgren
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         Congratulations everybody to the revised JSON RFC!
> 
>         Does this mean that JSON is "done" for good?
> 
>         Probably not because the concept I have mentioned from time to
>         time, the ability adding a digital signature to a JSON object
>         (in contrast to signing arbitrary Base64Url-encoded data), is
>         still very much alive.  In fact, there is an I-D in preparation
>         aiming at reducing the current proliferation of "DIY-standards"
>         for dealing with this highly requested feature.  The only real
>         challenge is agreeing on a suitable way "normalizing" JSON data
>         during parsing and serialization.  Such a scheme will be like an
>         extended version of I-JSON (RFC7493), potentially having an
>         impact on "ordinary" uses of JSON as well.
> 
>         Happy New [and optionally signed] JSON Year
>         Anders Rundgren
> 
>         _______________________________________________
>         json mailing list
>         [email-masked] <mailto:[email-masked]>
>         https://www.ietf.org/mailman/listinfo/json
>         <https://www.ietf.org/mailman/listinfo/json>
> 
> 
> 
>     _______________________________________________
>     json mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/json
>     <https://www.ietf.org/mailman/listinfo/json>
> 
> 
> 
> 
> -- 
> - Tim Bray (If you’d like to send me a private message, see 
> https://keybase.io/timbray)
> 
> 
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 1 Jan 2018 12:17:17 -0500

​I can't see the need for stringify there. By which I mean, my code runs
and I do not use javascript or stringify anywhere.

​Where I think folk tend to go really wrong is doing the stuff people
decided they wanted with XML Signature which was to have the following:

Unsigned object:

<object>
   <blahdyblah/>
</object>

Signed object:

<object>
   <blahdyblah/>
   <signature>
      {path transform}
      {etc}
   </signature>
</object>

And it worked exactly as well as could be expected.

On Mon, Jan 1, 2018 at 1:43 AM, Anders Rundgren <
[email-masked]> wrote:

> On 2018-01-01 07:27, Phillip Hallam-Baker wrote:
>
>> The use case that I believe gives rise to the belief canonicalization is
>> necessary is the need to sign a data object in the same JSON document.
>>
>
> Yes, but this is indeed only a belief.  The following in-object signature
> would validate in any JSON system compliant with ECMA:
>
> {
>   "now": "2017-04-16T11:23:06Z",
>   "escapeMe": "\u20ac$\u000F\u000aA'\u0042\u0022\u005c\\\"\/",
>   "numbers": [1e+30,4.5,6],
>   "signature": {
>     "alg": "ES256",
>     "jwk": {
>       "kty": "EC",
>       "crv": "P-256",
>       "x": "_gow8fcS3Dx9z6j57U5q8tunnRBdrgLU9A7CZTYCnqU",
>       "y": "bdfJGraBVL5aPj38TG4tHwxpU2VKwG1XBp0wQfCLOFQ"
>     },
>     "val": "OfTUed3gOO58ZPcP5ZXpL6OA9iyHskfo0zKI60H6XriGNNNb5sugGz80nqf
> yJ25jXPRYGAWcjH-1KGy5_eQuJQ"
>   }
> }
>
>
>   // Perform normalization
>   var clone = Object.assign({}, signedObject.signature);     // Clone
> "signature" child object
>   var signature = decodeBase64URL(clone.val);                // Get
> signature value
>   delete signedObject.signature.val;                         // Remove
> signature value property from signed object
>   var data = convertToUTF8(JSON.stringify(signedObject));    // Get
> normalized JSON string (signed data)
>   signedObject.signature = clone;                            // Restore
> signed object
>   // Do the crypto now
>
> That's clearly not rocket science.
>
> Anders
>
>
> I have this need frequently:
>>
>> { {A = [... JSON Data...]} { Signature (A) } }
>>
>> The way I solve this using JSON is to BASE-64 encode:
>>
>> { { A = base64(... JSON Data...)} { Signature (A) } }
>>
>> Which is brutal but entirely effective and only expands the data by 33%
>>
>> For an efficient encoding, I use JSON-B which is a strict superset of
>> JSON with additional code points that allow binary data to be spliced in.
>>
>> JSON only uses bytes in the range 1-127 for tagging. Bytes in the range
>> 128-255 can only occur inside strings. Which means all the other values can
>> be repurposed
>>
>> In JSON-B, the binary sequence 01 02 03 can be encoded as either "AQID" or
>>
>> 88 03 01 02 03
>>
>> The first byte says that this is a binary value with an 8 bit length and
>> the final block of the sequence (i.e. there is only one block in this
>> sequence).
>>
>> The second byte is the length.
>>
>> The rest is the data.
>>
>>
>> I would much rather extend JSON to support efficient encoding of any
>> binary value than develop a canonicalization hack that could only ever work
>> for JSON.
>>
>> http://www.prismproof.org/Documents/draft-hallambaker-jsonbcd.html
>>
>>
>>
>>
>

---

From: Anders Rundgren <[email-masked]>
Date: Mon, 1 Jan 2018 18:55:01 +0100

On 2018-01-01 18:17, Phillip Hallam-Baker wrote:
> ​I can't see the need for stringify there. By which I mean, my code runs and 
> I do not use javascript or stringify anywhere.

The only "hard" requirement is that the JSON tool adheres to ECMA's ES6+ JSON processing rules.  With the exception of "Number" normalization [1] this is a trivial matter [2].

By putting your JSON data in Base64 or using special JSON dialects you can of course do things differently.

> ​Where I think folk tend to go really wrong is doing the stuff people > decided they wanted with XML Signature which was to have the following:
> 
> Unsigned object:
> 
> <object>
>     <blahdyblah/>
> </object>
> 
> Signed object:
> 
> <object>
>     <blahdyblah/>
>     <signature>
>        {path transform}
>        {etc}
>     </signature>
> </object>
> 

The sample and its associated scheme does (by design) NOT perform any sophisticated textual processing. Maybe it only covers 80% of the "market".  Is that a showstopper?

Anders

1] https://www.ecma-international.org/ecma-262/8.0/index.html#sec-tostring-applied-to-the-number-type
2] https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashMap.html

> On Mon, Jan 1, 2018 at 1:43 AM, Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Yes, but this is indeed only a belief.  The following in-object signature would validate in any JSON system compliant with ECMA:
> 
>     {
>          "val": "OfTUed3gOO58ZPcP5ZXpL6OA9iyHskfo0zKI60H6XriGNNNb5sugGz80nqfyJ25jXPRYGAWcjH-1KGy5_eQuJQ"
>        }
>     }
> 
> 
>        // Perform normalization
>        var clone = Object.assign({}, signedObject.signature);     // Clone "signature" child object
>        var signature = decodeBase64URL(clone.val);                // Get signature value
>        delete signedObject.signature.val;                         // Remove signature value property from signed object
>        var data = convertToUTF8(JSON.stringify(signedObject));    // Get normalized JSON string (signed data)
>        signedObject.signature = clone;                            // Restore signed object
>        // Do the crypto now
> 
>         For an efficient encoding, I use JSON-B which is a strict superset of JSON with additional code points that allow binary data to be spliced in.
> 
>         JSON only uses bytes in the range 1-127 for tagging. Bytes in the range 128-255 can only occur inside strings. Which means all the other values can be repurposed
> 
>         The first byte says that this is a binary value with an 8 bit length and the final block of the sequence (i.e. there is only one block in this sequence).
> 
>         I would much rather extend JSON to support efficient encoding of any binary value than develop a canonicalization hack that could only ever work for JSON.
> 
>         http://www.prismproof.org/Documents/draft-hallambaker-jsonbcd.html <http://www.prismproof.org/Documents/draft-hallambaker-jsonbcd.html>
> 
> 
> 
> 
>

---

From: Richard Gibson <[email-masked]>
Date: Tue, 2 Jan 2018 11:33:54 -0500

On Mon, Jan 1, 2018 at 1:10 AM, Anders Rundgren <
[email-masked]> wrote:

> On 2018-01-01 00:39, Richard Gibson wrote:
>
>> On Sun, Dec 31, 2017 at 4:34 PM, Anders Rundgren <
>> [email-masked] <mailto:[email-masked]>>
>> wrote:
>>
>>     For normalization it turns out that JSON.parse() and JSON.stringify()
>> as specified by ECMA and already supported by the most widely available
>> JSON tools is all you need!
>>
>>
>> That is definitely not true. A trivial counterexample:
>> https://jsbin.com/luyoveyuqu/edit?js,console,output
>>
>
> Right, JSON.stringify() doesn't not canonicalize but preserve property
> order [1] + normalize data which for "crypto-safe" JSON applications is
> entirely sufficient.
>

JSON.stringify assumes information that is explicitly not conveyed by JSON
(in which an object is "an unordered collection of zero or more name/value
pairs") [1], and both its number serialization [2] and string serialization
[3] specify aspects that harm compatibility (the former having arbitrary
value-dependent branches, the latter being capable of producing invalid
UTF-8 octet sequences that represent unpaired surrogate code
points—unacceptable for exchange outside of a closed ecosystem [4]). JSON
is a general language-agnostic interchange format, and ECMAScript
JSON.stringify is *not* a JSON canonicalization solution.

[1]: https://tools.ietf.org/html/rfc8259#section-1
[2]:
http://ecma-international.org/ecma-262/7.0/#sec-tostring-applied-to-the-number-type
[3]: http://ecma-international.org/ecma-262/7.0/#sec-quotejsonstring
[4]: https://tools.ietf.org/html/rfc8259#section-8.1

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 3 Jan 2018 08:06:36 +0100

On 2018-01-02 17:33, Richard Gibson wrote:

<snip>

> JSON.stringify assumes information that is explicitly not conveyed 
> by JSON (in which an object is "an unordered collection of zero or 
> more name/value pairs") [1], and both its number serialization [2]
> and string serialization [3] specify aspects that harm compatibility
> (the former having arbitrary value-dependent branches, the latter being
> capable of producing invalid UTF-8 octet sequences that represent unpaired
> surrogate code points—unacceptable for exchange outside of a closed
> ecosystem [4]). JSON is a general language-agnostic interchange format,
> and ECMAScript JSON.stringify is *not* a JSON canonicalization solution.
> 
> [2]: http://ecma-international.org/ecma-262/7.0/#sec-tostring-applied-to-the-number-type

JSON is a format defined the IETF and ECMA in some kind of cooperation.

ECMA's ECMAScript defines a set of ("fairly" easy to implement) JSON processing rules which are fully compatible [1] with the JSON format.

The idea using ES6+ number normalization was originally proposed by an esteemed member of the now concluded JOSE WG:
https://www.ietf.org/mail-archive/web/jose/current/msg05323.html

Following that path, I (accidentally) discovered the "missing link", ES6+'s preservation [2] of the original property ordering which in my mind is quite logical [3] for "ordinary" JSON usage as well.

A Java based reference implementation has been successfully tested against Chrome, Node.js, Firefox and Safari.

Feel free taking a test ride on: https://mobilepki.org/jose-jcs/home
Note that it outlaws signing a JSON "Number" expressed as "1.0" since the correct (according to ES6+) representation is "1".

Anders

1] possibly with the exception of "unpaired surrogate code points" which I admittedly know zilch about
2] with properties named as integers as a for practical purposes insignificant exception
3] Making data come out in the order it was specified addresses human aspects like debugging and documentation of JSON data

---

From: Richard Gibson <[email-masked]>
Date: Wed, 3 Jan 2018 12:25:02 -0500

I'm really not sure where the disconnect between us is coming from... JSON
specifies unordered keys for objects, arbitrary-precision numbers, and
strings that can contain code points excluded from UTF-8. As such,
ECMAScript JSON.stringify is *not* suitable for JSON canonicalization,
though it may suffice for certain Javascript-centric or otherwise
restricted subsets thereof as addressed by JOSE. I believe that a general
solution like http://gibson042.github.io/canonicaljson-spec/ is needed to
avoid this seemingly endless series of special cases [1] [2] [3] [etc]
(including those that have nothing to do with cryptographic signatures or
encryption), and therefore wish to work towards an RFC describing it.
There's no problem if you're uninterested, or even if you believe that a
general solution is unwarranted, but there _is_ a problem if you believe
that JSON.stringify already provides one.

[1]: https://tools.ietf.org/html/draft-staykov-hu-json-canonical-form-00
[2]: https://tools.ietf.org/html/rfc7638#section-3.3
[3]: https://keybase.io/docs/api/1.0/canonical_packings#json
[etc]: http://wiki.laptop.org/go/Canonical_JSON

On Wed, Jan 3, 2018 at 2:06 AM, Anders Rundgren <
[email-masked]> wrote:

>> pairs") [1], and both its number serialization [2]
> ECMA's ECMAScript defines a set of ("fairly" easy to implement) JSON
> processing rules which are fully compatible [1] with the JSON format.
>
> The idea using ES6+ number normalization was originally proposed by an
> esteemed member of the now concluded JOSE WG:
> Following that path, I (accidentally) discovered the "missing link",
> ES6+'s preservation [2] of the original property ordering which in my mind
> is quite logical [3] for "ordinary" JSON usage as well.
>
> Note that it outlaws signing a JSON "Number" expressed as "1.0" since the
> correct (according to ES6+) representation is "1".
>
> Anders
>
> 3] Making data come out in the order it was specified addresses human
> aspects like debugging and documentation of JSON data
>

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 4 Jan 2018 06:35:37 +0100

Dear Richard,

I'm in no way against an RFC for canonical JSON. On the contrary, I whish you luck with your JSON canonicalization endeavors!

There is no real disconnect, there is a difference in opinion of what is [most] important.

In my opinion a cross-platform system based on JSON messaging MUST consider the limitations in both ECMAScript JSON processing and in JSON itself.  The latter are quite severe and every now and then cause people starting new projects to "fix JSON" like: https://www.tjson.org/

Creating a signature scheme relying on ECMAScript JSON processing rules may indeed be regarded as a short-cut, but availability typically trumps perfection.

Anders

On 2018-01-03 18:25, Richard Gibson wrote:
> I'm really not sure where the disconnect between us is coming from... JSON specifies unordered keys for objects, arbitrary-precision numbers, and strings that can contain code points excluded from UTF-8. As such, ECMAScript JSON.stringify is *not* suitable for JSON canonicalization, though it may suffice for certain Javascript-centric or otherwise restricted subsets thereof as addressed by JOSE. I believe that a general solution like http://gibson042.github.io/canonicaljson-spec/ is needed to avoid this seemingly endless series of special cases [1] [2] [3] [etc] (including those that have nothing to do with cryptographic signatures or encryption), and therefore wish to work towards an RFC describing it. There's no problem if you're uninterested, or even if you believe that a general solution is unwarranted, but there _is_ a problem if you believe that JSON.stringify already provides one.
> 
> On Wed, Jan 3, 2018 at 2:06 AM, Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
> 
>         JSON.stringify assumes information that is explicitly not conveyed by JSON (in which an object is "an unordered collection of zero or more name/value pairs") [1], and both its number serialization [2]
>         [1]: https://tools.ietf.org/html/rfc8259#section-1 <https://tools.ietf.org/html/rfc8259#section-1>
>         [2]: http://ecma-international.org/ecma-262/7.0/#sec-tostring-applied-to-the-number-type <http://ecma-international.org/ecma-262/7.0/#sec-tostring-applied-to-the-number-type>
>         [3]: http://ecma-international.org/ecma-262/7.0/#sec-quotejsonstring <http://ecma-international.org/ecma-262/7.0/#sec-quotejsonstring>
>         [4]: https://tools.ietf.org/html/rfc8259#section-8.1 <https://tools.ietf.org/html/rfc8259#section-8.1>
> 
> 
>     https://www.ietf.org/mail-archive/web/jose/current/msg05323.html <https://www.ietf.org/mail-archive/web/jose/current/msg05323.html>
> 
>     Feel free taking a test ride on: https://mobilepki.org/jose-jcs/home <https://mobilepki.org/jose-jcs/home>