Subject: [Json] I-D: draft-rundgren-json-canonicalization-scheme-02

From: Anders Rundgren <[email-masked]>
Date: Thu, 20 Dec 2018 04:12:28 +0100

https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-02

Abstract:
    Cryptographic operations like hashing and signing depend on that the
    target data does not change during serialization, transport, or
    parsing.  By applying the rules defined by JCS (JSON Canonicalization
    Scheme), data provided in the JSON [RFC8259] format can be exchanged
    "as is", while still being subject to secure cryptographic
    operations.  JCS achieves this by building on the serialization
    formats for JSON primitives as defined by ECMAScript [ES6],
    constraining JSON data to the I-JSON [RFC7493] subset, and through a
    platform independent property sorting scheme.

    The intended audiences of this document are JSON tool vendors, as
    well as designers of JSON based cryptographic solutions.

////
In addition to some wordsmithing, this revision is supposed to make it
clearer that JCS is not a traditional canonicalization scheme working
on the text level, but a serialization scheme providing a canonical
form of JSON data conforming to a strict interpretation of I-JSON.

The serialization-only scheme makes implementation straightforward
and improves performance.

A list of Open Source implementations is also included in this revision.

Enjoy!

Anders

---

From: Bret Jordan <[email-masked]>
Date: Fri, 21 Dec 2018 14:04:36 -0700

Anders,

Would you be willing to put this in a Google Doc and share it out so that I can more easily comments and provide suggestions?  

Thanks,
Bret
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

> On Dec 19, 2018, at 8:12 PM, Anders Rundgren <[email-masked]> wrote:
> 
> _______________________________________________
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 22 Dec 2018 05:46:32 +0100

> Would you be willing to put this in a Google Doc and share it out so that I can more easily comments and provide suggestions?

Hi Bret,

The specification is already supported by an "issue" system:
https://github.com/cyberphone/json-canonicalization/issues
It is quite simple to use.  Questions may also be put there.

Anyway, here is a content-wise identical specification but in a more reader-friendly format
https://cyberphone.github.io/doc/security/draft-rundgren-json-canonicalization-scheme.html
which I guess could be put in Google Docs.  Feel free doing that, the text is not copyrighted in any way.

The I-D source language is XML and pretty unreadable :-|

Happy Christmas all you JSON aficionados out there!

Thanks,
Anders
>> On Dec 19, 2018, at 8:12 PM, Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 22 Dec 2018 16:21:56 -0500

An obvious wart here is that keys are sorted as UTF-16 and output as UTF-8 because of the internal representation in current software.

This strikes me as a bad long-term approach - the sort order should be identical to the output representation - aka UTF-8. This would mean that implementations which currently use UTF-16 internally would need to convert the text before sorting, but lead to a better long-term standard as software moves away from the "worst of all worlds" UTF-16:

https://utf8everywhere.org/

(in particular since this draft does not specify an endianness for the utf-16 code units, it could easily lead to bugs amongst software only tested on ASCII input which sorts the same way regardless of the endianness of the codepoints)

Regards,

Bron.

On Thu, Dec 20, 2018, at 14:13, Anders Rundgren wrote:
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
> 

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 [email-masked]

---

From: Anders Rundgren <[email-masked]>
Date: Sun, 23 Dec 2018 11:02:44 +0100

On 2018-12-22 22:21, Bron Gondwana wrote:
> An obvious wart here is that keys are sorted as UTF-16 and 
> output as UTF-8 because of the internal representation in current software.

Hi Bron,

Since JCS is in a fairly advanced state including some pilot usage, I created an issue for this:
https://github.com/cyberphone/json-canonicalization/issues/6

That UTF-8 is the only reasonable external representation of text is clear.

That the JCS specification (at the time of writing) sorts properties based on big endian UTF-16 code units is indeed for maintaining optimal performance on legacy software platforms, but also due to JSON itself which specifies Unicode string escapes as UTF-16 constants. If there actually is a problem using UTF-16 for sorting, wouldn’t this affect JSON as well?

Just to make things a bit more complicated, other JSON canonicalization schemes rather prescribe using Unicode code points:
https://tools.ietf.org/html/rfc7638
https://gibson042.github.io/canonicaljson-spec/

I am not in any way married to using UTF-16 as the foundation for sorting but since sorting is an internal operation, I do not see that as a necessity for interoperability.

https://unicode.org/faq/utf_bom.html states the following regarding the different UTF variants:
"The conversions between all of them are algorithmically based, fast and lossless"

> https://utf8everywhere.org/

The lack of UTF-8 string escapes remains an issue in JSON.  I don't expect this will change.

> (in particular since this draft does not specify an endianness for the utf-16 code units, it could easily lead to bugs amongst software only tested on ASCII input which sorts the same way regardles…

I believe UTF-16 has big endian as default but you are right, it could be explicit as well.  Fixed in next version.  Thanx!

There is quite a lot of test data for JCS which deals with this as well as other issues:
https://github.com/cyberphone/json-canonicalization/tree/master/testdata

Regards,
Anders

---

From: John Cowan <[email-masked]>
Date: Sun, 23 Dec 2018 15:39:06 -0500

On Sun, Dec 23, 2018 at 5:02 AM Anders Rundgren <
[email-masked]> wrote:

That the JCS specification (at the time of writing) sorts properties based
> on big endian UTF-16 code units is indeed for maintaining optimal
> performance on legacy software platforms, but also due to JSON itself which
> specifies Unicode string escapes as UTF-16 constants. If there actually is
> a problem using UTF-16 for sorting, wouldn’t this affect JSON as well?
>

It is indeed annoying that none of Java, C#, or JavaScript have standard
methods for comparing strings in Unicode codepoint order.   At least in
Java and C#, however, such routines are easily written and not inherently
less efficient (in a big-O sense) than native routines.  The standard
technique is to change code units in the range D800-DFFF to F800-FFFF and
vice versa (an O(n) operation) and then run a native comparison (also
O(n)).  In Java this behavior can be provided as a class implementing the
standard interface Comparator.

The fact that the external representation of Astral Plane characters uses
two consecutive \unnnn sequences is irrelevant.  JSON parsers and writers
that use UTF-8 or UTF-32 as their native representations are perfectly able
to deal with this convention, just as XML parsers and writers in
Java/C#/JavaScript are able to convert between the external &#nnnnnn;
representation and  two consecutive UTF-16 code units.

-- 
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
So they play that [tune] on their fascist banjos, eh?
        --Great-Souled Sam

---

From: "Bron Gondwana" <[email-masked]>
Date: Sun, 23 Dec 2018 21:02:57 -0500

> I am not in any way married to using UTF-16 as the foundation for sorting but since sorting is an internal operation, I do not see that as a necessity for interoperability.

I'm not sure what you mean by this - since you can't be canonical without sorting, everyone has to sort the same way.

> "The conversions between all of them are algorithmically based, fast and lossless"

That's true - I have no overall objection to whatever works, I'm just sad that UTF-16 is continuing to get baked into things.

> I believe UTF-16 has big endian as default but you are right, it could be explicit as well. Fixed in next version. Thanx!

No worries. That's definitely a good thing to specify.

Cheers,

Bron.

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 24 Dec 2018 05:00:29 +0000

Hello John, others,

On 2018/12/24 05:39, John Cowan wrote:
> vice versa

Please be careful. There are codepoints in the range E000-F7FF, too. 
These are in the Private Use Area (PUA), so the chances that there's 
some data there is not very high, but still, it's better to do this by
moving D800-DFFF to F800-FFFF and shifting down all of E000-FFFF to 
D800-F7FF. Otherwise, just one spurious codepoint in the PUA can ruin 
sort order interoperability between UTF-8 and UTF-16.

Regards,   Martin.

> (an O(n) operation) and then run a native comparison (also

---

From: John Cowan <[email-masked]>
Date: Mon, 24 Dec 2018 00:49:36 -0500

You're right, of course.