Subject: Re: [Json] In "praise" of UTF-16

From: Anders Rundgren <[email-masked]>
Date: Mon, 2 Sep 2019 21:48:19 +0200

Thanx John and Tim, your input was much appreciated!

Anders

On 2019-08-31 20:21, Tim Bray wrote:
> It's incorrect to say that nobody complains about UTF-16, I do so all the time. But it doesn’t do any good.
> 
> I don't see any problem with using UTF-16 this way.
> 
> On Sat, Aug 31, 2019 at 8:39 AM Anders Rundgren <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Hi JSON experts,
>     Pardon the subject line, I'm by no means an UTF-16 aficionado.
> 
>     That UTF-16 has been deprecated by the industry at large for EXTERNAL representation of textual data is completely understandable.
> 
>     However, an I-D dealing with canonical JSON serialization is currently in the IETF ISE queue got criticism for using UTF-16 encoding INTERNALLY for sorting properties/keys.
>     I don't see why since the only purpose of the sorting is creating a defined order.   That sorting on UTF-8 or UTF-32 would give another result is true but for the stated purpose that is of no importance.
> 
>     In addition, JSON itself also depends on UTF-16 encoding for \uhhhh constants and AFAIK nobody have complained about that.
>     Example: A smiley Emoji has the Unicode value U+1F600 but would in a JSON escape sequence be represented as \ud83d\ude00
> 
>     The reason for preferring UTF-16 in this particular case is simply because JavaScript, Windows and Java use UTF-16 as internal representation.  That's obviously a slight platform bias but the my Go and Python implementations show that the UTF-16 requirement in practice is a no-issue.
> 
>     According to the Unicode standard UTF-16 belongs to the set of supported fully interchangeable encodings.
> 
>     WDYT?
> 
>     thanx,
>     Anders
>     https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-06
> 
>     _______________________________________________
>     json mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/json
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 2 Sep 2019 22:59:59 +0200

Hi Anders,

> On Aug 31, 2019, at 17:39, Anders Rundgren <[email-masked]> wrote:
> 
> I-D dealing with canonical JSON serialization is currently in the IETF ISE queue

I don’t really care much about this draft, as I have pointed out.
(Mainly not because deterministic encoding is wrong, but because the reasons you want to define a “standard” for it are wrong.  But that has been said already in different messages.)

So please do go ahead and taint your scheme further with UTF-16 legacy stuff.

For those people who missed the early history of ISO 10646 (aka Unicode):
Unicode was supposed to be the 16-bit character set to end all 8-bit character sets.
We were all supposed to move to UCS-2 (a 2-byte, 16-bit representation where the standards bodies couldn’t even decide the endianness, so we actually have two of those).

One important language designed in the early-to-mid-1990s got stuck with this: Java.  As did Windows.
Pioneers get shot full of arrows.  
The .NET system designed for Windows as a competitor to the JVM (Java Virtual Machine), as well as JavaScript (which is not Java but usurped the name), also inherited 16-bit characters.

Then it dawned people that 16 bits were not enough, and after a short excursion to a 31-bit system, UTF-16 was invented as a hack to accept more than 63488 [sic] codepoints, with the built-in problem that the “astral” codepoints (beyond 2**16) were infrequent enough that people built endless amounts of bugs into software that used it.  

The 16-bit legacy embedded into Java and its contemporaries could not be stamped out since.  The main reason is that while UTF-16 is a hack that hurts, it does not hurt enough to sustain deployment of a fix (remember that even Python is still struggling this very year with getting its Unicode fixes deployed).

JSON of course inherited the 16-bit legacy from JavaScript in its string escape scheme, but otherwise did come out pretty much unharmed and managed to focus on UTF-8, like everything else relevant to interchange.

Defining a deterministic encoding scheme (“canonicalization”) for JSON in 2019 that needs a detour through UTF-16-land looks like a cruel joke.  If you only ever care about Java and its contemporaries, it may actually seem practical to you.  Given that most map keys will be ASCII anyway, people will certainly find shortcuts (also known as sleeping interoperability problems) around the issue.  A protocol designer would not touch this with a 16-foot pole.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 2 Sep 2019 16:17:45 -0500

On Mon, Sep 02, 2019 at 10:59:59PM +0200, Carsten Bormann wrote:
> Defining a deterministic encoding scheme (“canonicalization”) for JSON
> in 2019 that needs a detour through UTF-16-land looks like a cruel
> joke.  [...]

Is that what this is about?!

Oh no.

Every JSON implementor needs to deal with UTF-8, but not every JSON
implementor needs to deal with UTF-16.

Speaking as a maintainer and developer of one popular JSON
implementation that only supports UTF-8, I'm loather to have to add
support for transliterating to UTF-16 code for canonicalization
purposes.

I understand that for many implementors the use of UTF-16 for this would
be far, _far_ more convenient than UTF-8, and that were I one of them,
I'd probably be on the other side.  For this reason I think it would be
fair for either camp to end up on the rough side of consensus, and if it
be my side so be it.

But first we should find what the consensus is.  Has there been a
consensus call on this?

Whatever we do here will make some unhappy.  Like endianness, we can
have some notion of purity (network byte order) and yet often prefer
that which is most widely implemented in hardware (little endian byte
order).  In Internet standards we _tend_ to prefer purity.

> [...].  If you only ever care about Java and its contemporaries, it may
> actually seem practical to you.  Given that most map keys will be
> ASCII anyway, people will certainly find shortcuts (also known as
> sleeping interoperability problems) around the issue.  A protocol
> designer would not touch this with a 16-foot pole.

+1

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 2 Sep 2019 23:20:10 +0200

> But first we should find what the consensus is.  Has there been a

This is not a consensus document.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 2 Sep 2019 23:34:52 +0200

On Sep 2, 2019, at 23:17, Nico Williams <[email-masked]> wrote:
> 
> I'm loather to have to add

It’s not that bad.

If you have the code point cp, you can translate this to a comparison value by doing

(cp < 0x10000 ? cp << 16 : (cp << 10) + 0xD4000000)

(Untested code alert.  
This does not give you valid UTF-16, but something that will compare the same. 
I think.)

A similar transform can be applied to UTF-8 with a little state machine, but I’m too tired to code this up right now.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 2 Sep 2019 16:40:48 -0500

> If you have the code point cp, you can translate this to a comparison value by doing

Yes, I'm aware.  It's not that bad.  It's not that bad for the other
camp either (since they must already have UTF-{8, 16} transliteration.

Neither choice here is the end of the world for anyone.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 2 Sep 2019 23:48:54 +0200

> camp either (since they must already have UTF-{8, 16} transliteration.

It’s trivial for the UTF-16 side because JSON needs conversion to UTF-8 already.
Only if you then want to carry around the JSON as UTF-16 inside your program (which appears to be something that some Java people like) the whole thing becomes ugly.
 
> Neither choice here is the end of the world for anyone.

No, because we don’t care :-)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 2 Sep 2019 18:09:38 -0500

On Mon, Sep 02, 2019 at 03:56:09PM -0600, Joe Hildebrand wrote:
> I agree with Carsten. The use of utf-16 here is just one more reason to
> think this is a bad idea.

Well, yes, there's that too.

We all know that canonicalization should just not be necessary.  In
practice, to not need canonicalization requires additional functionality
somewhere.

For example, the Heimdal ASN.1 compiler will (for types where it's
requested) include a _save field in the emitted structures that has a
copy of the sub-string of the octet string being decoded that
corresponds to that structure.  E.g., you can request that the
TBSCertificate structure include a _save field whose value will be the
original encoding of a tbsCertificate field of a Certificate.  This
makes it trivial to validate the signature in a Certificate.

I'm starting to think that JSON parsers are going to need a similar
feature...

Nico
--

---

From: Rob Sayre <[email-masked]>
Date: Mon, 2 Sep 2019 16:30:10 -0700

On Mon, Sep 2, 2019 at 2:49 PM Carsten Bormann <[email-masked]> wrote:

> Only if you then want to carry around the JSON as UTF-16 inside your
> program (which appears to be something that some Java people like) the
> whole thing becomes ugly.
>

Doesn't this argument miss the escaping syntax that JSON requires?

'To escape an extended character that is not in the Basic Multilingual
Plane, the character is represented as a 12-character sequence, encoding
the UTF-16 surrogate pair.  So, for example, a string containing only the G
clef character (U+1D11E) may be represented as "\uD834\uDD1E".'

<https://www.rfc-editor.org/rfc/rfc8259.txt>, section 7.

thanks,
Rob

---

From: Nico Williams <[email-masked]>
Date: Mon, 2 Sep 2019 23:08:56 -0500

> clef character (U+1D11E) may be represented as "\uD834\uDD1E".'

Ah, good point.  Objection withdrawn.

---

From: Anders Rundgren <[email-masked]>
Date: Tue, 3 Sep 2019 07:17:25 +0200

{
   "mildIrony": "Thanx Nico and Carsten for your \"encouraging\" input",
   "applicableEmojis": "\ud83d\udca9\ud83d\ude01"
}

More seriously, you are wasting your precious cycles on already solved and quite simple problems:
https://chromium.googlesource.com/android_tools/+/78ccfd5f7e4880597fd90c61453a3be0e7aee5f0/sdk/sources/android-21/java/lang/Character.java#2058
The slight platform "bias" doesn't actually matter, it is the result that counts.

Personally, I found it more worthwhile looking into the [previously] unsolved problem: canonical serialization of 64-bit doubles.
This was clearly out of my league but I got some excellent help on the way.  The JCS number test suite passes without a glitch for Java, JavaScript, C#, Python and Go:
https://github.com/cyberphone/json-canonicalization/tree/master/testdata#es6-numbers
This scheme also have a clear platform bias.  In fact, the entire JCS specification (modulo sorting) is 100% compatible with ES6's JSON.stringify().

Anyway, if you think that JCS is controversial, that's nothing compared to what it is targeted for:
https://lists.w3.org/Archives/Public/public-payments-wg/2019Sep/0000.html
The 5-level signature depth required made the [previously] mandatory B64U wrapping schemes look really stupid.

thanx,
Anders
https://mobilepki.org/jws-jcs/home

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 Sep 2019 07:32:28 +0200

On Sep 3, 2019, at 01:30, Rob Sayre <[email-masked]> wrote:
> Doesn’t this argument miss the escaping syntax that JSON requires?

A (sanely designed) canonical encoding won’t ever use that, so you don’t have to implement it even at the decoder.

> 'To escape an extended character that is not in the Basic Multilingual Plane, the character is represented as a 12-character sequence, encoding the UTF-16 surrogate pair.  So, for example, a string containing only the G clef character (U+1D11E) may be represented as “\uD834\uDD1E".'

Yes, but you never escape these.  The only ones that you do escape are the one you MUST:

   … the characters that MUST be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F).  [p. 8]

So, even if your string source is non-UTF-8, all escaping can be done *after* UTF-8 conversion, looking at the ASCII characters (high bit of the byte unset) only.

(For those listening in and not understanding what the disagreement could possibly be here:
It is a common misunderstanding of the cited paragraph at the start of page 9 of RFC 8259 that this mandates escaping astral code points.  No, it just says how you do it if you want to.  But you don’t want to.)

Grüße, Carsten

---

From: Rob Sayre <[email-masked]>
Date: Mon, 2 Sep 2019 22:42:06 -0700

On Mon, Sep 2, 2019 at 10:32 PM Carsten Bormann <[email-masked]> wrote:

> It is a common misunderstanding of the cited paragraph at the start of
> page 9 of RFC 8259 that this mandates escaping astral code points.  No, it
> just says how you do it if you want to.  But you don’t want to.)
>

I don't think it mandates escaping those astral code points. It says "may".
That means encoders and decoders must deal with that UTF-16 representation
in order to interoperate. I agree that it's not necessary in an end-to-end
UTF-8 pipeline.

thanks,
Rob

---

From: Anders Rundgren <[email-masked]>
Date: Tue, 3 Sep 2019 07:46:01 +0200

The (IMO) only real problem with JCS is in Appendix E:
https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-06#appendix-E

Why is that?  Because you typically expect tools and libraries to do all the gory stuff.

In this particular case a part of the canonicalization process may spill over to the application.
There are easy workarounds but workarounds is always a source of problems.

IF JCS gets firm adoption this problem could be more or less nullified but that's another story.

thanx,
Anders

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 3 Sep 2019 08:00:39 +0200

Any deterministic encoding scheme always has an application layer canonicalization issue.
It’s just slightly worse for (I-)JSON because its generic data model has so few common data types, so more work is at the application layer, mapping to those data types.

Grüße, Carsten

> On Sep 3, 2019, at 07:46, Anders Rundgren <[email-masked]> wrote:
> 
> [email-masked]

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 4 Sep 2019 07:18:39 +0200

> It’s just slightly worse for (I-)JSON because its generic data model has so few common data types, so more work is at the application layer, mapping to those data types.

A schema based system could manage this in a for the application developer transparent way at the expense of an additional backing string variable for each emulated data type.

Anders