Subject: [Json] Json and U+08A1 and related cases (was: Re: Barry Leiba's Discuss on draft-ietf-json-i-json-05: (with DISCUSS and COMMENT))

From: John C Klensin <[email-masked]>
Date: Wed, 21 Jan 2015 14:29:22 -0500

(Resending from my IETF address, which did not appear in the
first version I received but that doesn't work on some of the
multiple mailing lists copied)
(Changing subject line per Barry's request and addressing that
one issue only.)

--On Wednesday, January 21, 2015 13:04 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

>...
>> outside the context of a locale.
> 
> There's no guarantee that all text in a Fula locale (e.g. on a
> computer with a Fula OS, if there ever is such a thing)

Note that isn't "a Fula locale", but "a Fula locale that uses
the Arabic script to write the language".  There are no known
such locales of even country-scope; Fula is normally written in
Latin characters and has been for at least a few centuries.
There are lots of problems with the analogy,  but thinking about
Fula-in-Arabic in ordinary locale terms is a little bit like
thinking about Japanese-exclusively-in-Romanji (and Romanji with
a few characters that don't appear anywhere else in Latin
script) as a locale.  Probably possible to make it work, but not
the way most of us usually think about things.

> will
> have Behs with Hamzas above represented with U+08A1 (composed)
> rather than decomposed.
> بٔ U+0628 U+0654 (decomposed) may get in there from Arabic
> text easily.

I presume that is one of the reasons TUS 7.0 says
"preferentially" should use the precombined (composed) form
rather than the decomposed one rather than something we would
recognize as "MUST".

>> John Klensin
>> has explained the problem well in Section 2 of
>> draft-klensin-idna-5892upd-unicode70-03.txt.  You may wish to
>> review that work, and ask i8n program for a copy of the draft
>> statement, because it may have security implications on your
>> work, in as much as i-json is used to pass identifiers.
> 
> Given that JSON implementations, in contrast to IDNA, compare
> object member names codepoint-by-codepoint, the much more
> obvious addition to the security section would be to point out
> the potential consequences for precomposed/decomposed
> confusions in general. The chance that this becomes an actual
> issue is much much higher (although still rather low) in e.g.
> French or German than in Fula, where Arabic isn't even the
> main script used for writing the language.

Agreed.  And, fwiw, that would be my preferred solution.   I
think Fula and U+08A1 are important only as illustrative
examples of what appears to me to be a very nasty situation when
the nature of the identifier(s) or their uses does not provide a
reliable language context.

> P.S.: Please note that the comments above don't mean that I'm
> happy with the inclusion of U+08A1 in Unicode 7.0.0, and that
> I sincerely hope the Unicode Consortium will weight the
> problems of identifier confusability higher in their future
> decisions.

Agreed, although the way in which the U+08A1 decision has been
defended, and the antecedents for it, do not predict to that
result.  See below.

--On Wednesday, January 21, 2015 10:22 -0600 Nico Williams
<[email-masked]> wrote:

> I thought that NFC was closed to new precompositions though new
> precompositions might be added to Unicode.  That is, the NFC
> form of U+08A1 must be the same as the NFD form of U+08A1,
> which is to say: U+0628 U+0654.
> 
> Is my memory wrong about that?

That is the understanding that several -- I dare to say most or
all of  the IDNAbis WG participants -- of us had.  What has
actually occurred either violates that assumption or introduces
an extra case, depending on how one looks at the problem.   We
understood that, if precompositions (your term) are added, they
would have decompositions and NFC of the character would yield
the decomposed form.  That understanding derived both from what
the WG was told and from text in the Unicode Standard and UAX
#15 (see draft-klensin-idna-5892upd-unicode70-02, especially
Section 2.1, for details and references).  

But, while U+08A1 is abstract-character-identical and even
plausible-name-identical to U+0628 U+0654, it does _not_
decompose into the latter.  Instead, NFD(U+08A1) = NFC(U+08A1) =
U+08A1.  NFC (U+0628 U+0654) is U+0628 U+0654 as one would
expect from the stability rules; from that perspective, it is
the failure of U+08A1 to have a (non-identity) decomposition
that is the issue.

     john

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 21 Jan 2015 11:35:30 -0800

I see nothing here about JSON. As pointed out, JSON implementations compare codepoint by codepoint. Please clarify, or please let's drop this from the JSON WG list.

--Paul Hoffman

> On Jan 21, 2015, at 11:29 AM, John C Klensin <[email-masked]> wrote:
> 
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 Jan 2015 13:42:54 -0600

On Wed, Jan 21, 2015 at 02:29:22PM -0500, John C Klensin wrote:
> an extra case, depending on how one looks at the problem.   [...]

Because...

> decompose into the latter.  Instead, NFD(U+08A1) = NFC(U+08A1) =

...this is a desirable property of that particular character, or because 
the UC screwed up?  See below.

> the failure of U+08A1 to have a (non-identity) decomposition

Is it identical, as rendered as well as semantically, to U+0628 U+0654?

If U+08A1 identical to U+0628 U+0654 in every way then I think the UC
erred.  If it is not, then U+08A1 strikes me as a new case that IDNA
should treat as though NFC(U+08A1) == U+0628 U+0654 (because what else
could IDNA reasonably do??).  In what ways is U+08A1 not identical to
U+0628 U+0654? (besides, of course, being a different codepoint
sequence)

Nico
--

---

From: Shawn Steele <[email-masked]>
Date: Wed, 21 Jan 2015 20:07:25 +0000

>> precompositions might be added to Unicode.  That is, the NFC form of 
>> U+08A1 must be the same as the NFD form of U+08A1, which is to say: 
>> U+0628 U+0654.
> 

> That is the understanding that several -- I dare to say most or all of  the IDNAbis WG participants -- of us had.  

My understanding of the rules is that no new characters could be added that would cause normalization of existing codepoints to change.  (They could be added with both precomposed and decomposed forms, but only if all of the codepoints were new, which seems unlikely as new script additions seem to not like to have both forms).

Anyway, so the behavior would be that any character that is added that happens to look like a different character sequence is therefore not the same as the other character.  In other words, by definition, U+08A1 is not the same thing as U+0628 U+0654, whether they look alike or not.

This is similar to spelling Hawai'I with or without an ʻokina (Hawaiʻi).  A human thinks they look the same, but they aren't the same.  

I don't know the history of why this code point was added, and cannot argue about whether or not it "should" have just used the existing code points, however I'm happy that smart people that know a lot more about scripts than I do felt that this was the right way to handle this character.  And, even if they got it "wrong" (for whatever value of "wrong"), really there are a lot of quirks about Unicode that are far more interesting than this character, so I'm not going to second guess the UTC.

The Unicode process is not the same as the IETF process, however it is possible to participate.  If people really feel this strongly about how Unicode encodes characters, then they should participate in the process.  It's not helpful to second guess what they did.  It's even less helpful for IDN to try to attempt to work around a problem by using codepoints in a manner that doesn't conform to The Unicode Standard.  (See Zawgyi for an extreme example).  

The UTC/Unicode/UTR15 decided that U+08A1 doesn't decompose to U+0628 U+0654.  IDN shouldn't try to change that behavior.  At this point if folks really feel strongly that Unicode did the wrong thing, then they should work with the UTC to try to convince them (realizing that at this point it would need an amazing case to either break the stability rules, deprecate the new code point, &/or publish some sort of addendum/errata).

-Shawn

---

From: [email-masked]
Date: Wed, 21 Jan 2015 15:33:12 -0500

> the failure of U+08A1 to have a (non-identity) decomposition

If U+08A1 had such a decomposition, it would violate Unicode's
no-new-NFC rule.  What it violates is the (false) assumption that
base1 + combining is never confusable with a canonically
non-equivalent base2.  Even outside Arabic there are already
such cases:

U+1D92 LATIN SMALL LETTER E WITH RETROFLEX HOOK is not canonically
equivalent to U+0065 LATIN SMALL LETTER E plus U+0322 COMBINING
RETROFLEX HOOK BELOW (and ditto for some other Latin letters).

U+047D CYRILLIC SMALL LETTER OMEGA WITH TITLO is not canonically
equivalent to U+0461 CYRILLIC SMALL LETTER OMEGA plus U+0483 COMBINING
CYRILLIC TITLO (and ditto for capital omega)

U+2A25 PLUS SIGN WITH DOT BELOW is not canonically
equivalent to U+002B PLUS SIGN plus U+0323 COMBINING DOT BELOW
(and ditto for many other math symbols).

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
What is the sound of Perl?  Is it not the sound of a [Ww]all that people
have stopped banging their head against?  --Larry

---

From: Shawn Steele <[email-masked]>
Date: Thu, 22 Jan 2015 08:31:10 +0000

> I do not think it is similar to that case.  as I understand it, there is no _in principle_ way one could distinguish these character/character sequences in practice, even with attention to the appearance of characters.

I'm confused one is used for one thing and the other is used for another thing.  So they're distinguishable.  And you can tell the difference by context.  They may look the same, but there are numerous other homographs.

I'd also argue that eszett and ss is an even "worse" case (from my understanding).  At least everyone knows that though some words have preferred forms, the ss is often alternate spelling for eszett.  They're the same script, and either could appear in a word.  At least in the U+08A1 case my understanding is that it should not appear interchangeably with the homograph, so only malicious users would use the wrong one in the wrong place, whereas in German there are legitimate reasons for confusing the two forms.  If we can solve German (bundle or block), then we can solve this case.

> We're just trying to understand.  

It seems to go beyond understanding.  Oversimplifying, but Unicode folks have said "they're different, they're used in different contexts.  Yes they look the same, but they're different and it's not appropriate to mix them."  That seems fairly easy to understand, however there is still disagreement about whether or not they should be "the same". 

> The reports keep getting worse, however.

Then participate :)  There are very smart people in the UTC.  There have been some unfortunate things in hindsight, however Unicode learns and does better.  I'm reasonably confident that they're doing the best possible job of accommodating the requests and requirements with the appropriate amount of consideration.  Sometimes I have different opinions, but this isn't an easy space, and the more regular UTC participants know way more than I about how scripts work.

One of the other comments started to move beyond the "it shouldn't be this way, why is it this way?" and started asking how to handle it as it is obviously a homograph and that could be confusing to users.  I think that this should be handled as any other homograph, which seems to lead to prohibit (inappropriate in this case because it would prevent words from being spelled as designed), bundle, or block one if the other is registered first.  

-Shawn