Subject: [Json] In "praise" of UTF-16

From: Anders Rundgren <[email-masked]>
Date: Sat, 31 Aug 2019 17:39:21 +0200

Hi JSON experts,
Pardon the subject line, I'm by no means an UTF-16 aficionado.

That UTF-16 has been deprecated by the industry at large for EXTERNAL representation of textual data is completely understandable.

However, an I-D dealing with canonical JSON serialization is currently in the IETF ISE queue got criticism for using UTF-16 encoding INTERNALLY for sorting properties/keys.
I don't see why since the only purpose of the sorting is creating a defined order.   That sorting on UTF-8 or UTF-32 would give another result is true but for the stated purpose that is of no importance.

In addition, JSON itself also depends on UTF-16 encoding for \uhhhh constants and AFAIK nobody have complained about that.
Example: A smiley Emoji has the Unicode value U+1F600 but would in a JSON escape sequence be represented as \ud83d\ude00

The reason for preferring UTF-16 in this particular case is simply because JavaScript, Windows and Java use UTF-16 as internal representation.  That's obviously a slight platform bias but the my Go and Python implementations show that the UTF-16 requirement in practice is a no-issue.

According to the Unicode standard UTF-16 belongs to the set of supported fully interchangeable encodings.

WDYT?

thanx,
Anders
https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-06

---

From: "John Levine" <[email-masked]>
Date: 31 Aug 2019 13:58:28 -0400

In article <[email-masked]> you write:
>The reason for preferring UTF-16 in this particular case is simply because JavaScript, Windows and Java use UTF-16 as
>internal representation.  That's obviously a slight platform bias but the my Go and Python implementations show that the
>UTF-16 requirement in practice is a no-issue.

I wish UTF-16 had never been invented, but since it was, and
Javascript and JSON have a lot of historical connection to UTF-16,
using it as the canonical sort order is OK.

---

From: Tim Bray <[email-masked]>
Date: Sat, 31 Aug 2019 11:21:09 -0700

It's incorrect to say that nobody complains about UTF-16, I do so all the
time. But it doesn’t do any good.

I don't see any problem with using UTF-16 this way.

On Sat, Aug 31, 2019 at 8:39 AM Anders Rundgren <
[email-masked]> wrote:

> That UTF-16 has been deprecated by the industry at large for EXTERNAL
> representation of textual data is completely understandable.
>
> However, an I-D dealing with canonical JSON serialization is currently in
> the IETF ISE queue got criticism for using UTF-16 encoding INTERNALLY for
> sorting properties/keys.
> I don't see why since the only purpose of the sorting is creating a
> defined order.   That sorting on UTF-8 or UTF-32 would give another result
> is true but for the stated purpose that is of no importance.
>
> In addition, JSON itself also depends on UTF-16 encoding for \uhhhh
> constants and AFAIK nobody have complained about that.
> Example: A smiley Emoji has the Unicode value U+1F600 but would in a JSON
> escape sequence be represented as \ud83d\ude00
>
> The reason for preferring UTF-16 in this particular case is simply because
> JavaScript, Windows and Java use UTF-16 as internal representation.  That's
> obviously a slight platform bias but the my Go and Python implementations
> show that the UTF-16 requirement in practice is a no-issue.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>