Subject: [Json] Barry Leiba's Discuss on draft-ietf-json-i-json-05: (with DISCUSS and COMMENT)

From: "Barry Leiba" <[email-masked]>
Date: Mon, 19 Jan 2015 14:43:57 -0800

Barry Leiba has entered the following ballot position for
draft-ietf-json-i-json-05: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to http://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
http://datatracker.ietf.org/doc/draft-ietf-json-i-json/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

This should be quite simple to sort out:

-- Section 2.1 --

   Object member names, and string values in arrays and object members,
   MUST NOT include code points which identify Surrogates or
   Noncharacters.

Where are the definitions of "Surrogates" and "Noncharacters"?  Because
you say they MUST NOT be included, I think they need to be defined in
normative reference(s) and cited here (they're not defined in 3629, nor
does 3620 cite a definition).

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

The RFC Editor will make a lot of "which" -> "that" changes.  Just be
aware.

-- Sections 3 and 4 --
Going in the other direction from Jürgen's comments, I think these
sections are unremarkable, and would just suggest changing the 2119
language to English words instead.  This is all giving sage advice, not
defining protocol.

-- Section 5 --
I don't normally comment on "Acknowledgments" sections, and please take
this as you will and do as you think best.  You mention the contributions
of Douglas Crockford, and I wonder whether you might also mention the
contributions of Ecma International TC39.

---

From: Pete Resnick <[email-masked]>
Date: Mon, 19 Jan 2015 21:22:11 -0600

The codepoints used for UTF-16 surrogate pairs (U+D800 -> U+DFFF) are 
mentioned in 3629 in the first paragraph at the top of page 5 
<https://tools.ietf.org/html/rfc3629#page-5>, though I'm surprised not 
to see a reference to 2781, which also talks about surrogates. There is 
discussion of non-characters (as well as surrogates) in RFC 3454 (which 
is referenced by 6885).

None of those are great citations. We could simply cite Unicode 
<http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf>.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Tim Bray <[email-masked]>
Date: Mon, 19 Jan 2015 19:25:24 -0800

Yup, those are Unicode notions, Unicode is the right reference for them.
On Jan 19, 2015 7:22 PM, "Pete Resnick" <[email-masked]> wrote:

> mentioned in 3629 in the first paragraph at the top of page 5 <
> https://tools.ietf.org/html/rfc3629#page-5>, though I'm surprised not to
> see a reference to 2781, which also talks about surrogates. There is
> discussion of non-characters (as well as surrogates) in RFC 3454 (which is
> referenced by 6885).
>
> None of those are great citations. We could simply cite Unicode <
> http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf>.
>
> pr
>
> --
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Eliot Lear <[email-masked]>
Date: Tue, 20 Jan 2015 08:24:15 +0100

Hi,

Sorry for the late comment, but...

The IAB is preparing a cautionary statement regarding Unicode 7.0.0 and
specifically the introduction of pre-composed characters involving HAMZA
where it is impossible to typographically distinguish the precomposed
and decomposed characters outside the context of a locale.  John Klensin
has explained the problem well in Section 2 of
draft-klensin-idna-5892upd-unicode70-03.txt.  You may wish to review
that work, and ask i8n program for a copy of the draft statement,
because it may have security implications on your work, in as much as
i-json is used to pass identifiers.

Eliot

On 1/20/15 4:25 AM, Tim Bray wrote:
> On Jan 19, 2015 7:22 PM, "Pete Resnick" <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     The codepoints used for UTF-16 surrogate pairs (U+D800 -> U+DFFF)
>     are mentioned in 3629 in the first paragraph at the top of page 5
>     <https://tools.ietf.org/html/rfc3629#page-5>, though I'm surprised
>     not to see a reference to 2781, which also talks about surrogates.
>     There is discussion of non-characters (as well as surrogates) in
>     RFC 3454 (which is referenced by 6885).
>
>     Pete Resnick<http://www.qualcomm.com/~presnick/
>     <http://www.qualcomm.com/%7Epresnick/>>
>     <tel:%2B1%20%28858%29651-4478>
>
>     [email-masked] <mailto:[email-masked]>

---

From: Eliot Lear <[email-masked]>
Date: Tue, 20 Jan 2015 16:06:02 +0100

Just one further comment: I would not anticipate ANY change to the
specification over this, but you may think it worth mentioning as an
upper layer matter.  Or not.

Eliot

On 1/20/15 8:24 AM, Eliot Lear wrote:
> The IAB is preparing a cautionary statement regarding Unicode 7.0.0
> and specifically the introduction of pre-composed characters involving
> HAMZA where it is impossible to typographically distinguish the
> precomposed and decomposed characters outside the context of a
> locale.  John Klensin has explained the problem well in Section 2 of
>>     <https://tools.ietf.org/html/rfc3629#page-5>, though I'm
>>     surprised not to see a reference to 2781, which also talks about
>>     surrogates. There is discussion of non-characters (as well as
>>     surrogates) in RFC 3454 (which is referenced by 6885).
>>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 21 Jan 2015 13:04:22 +0900

Hello Eliot,

Many thanks for the information. Some comments below. Note that I'm not 
assuming that the IAB statement will contain as much 'near misses' as 
your explanation below, but I think it's better to address potential 
errors before it's too late. And yes, in this sense it would be good to 
get a draft of that statement.

I have also taken the liberty of adding the IDNA mailing list, because 
most of the Unicode side experts are on that list but not on the json list.

> The IAB is preparing a cautionary statement regarding Unicode 7.0.0 and

I very much hope that the statement will be specific to the actual 
concerns, and not, as this sentence seems to suggest, about Unicode 
7.0.0 in general. The later would be throwing out the baby with the 
bathwater.

> and decomposed characters

Please note that first, there are no cases where it's *impossible* to 
typographically distinguish precomposed and decomposed characters, or 
some other usually not distinguished pairs (see below for examples). 
There's always the choice of choosing (for a font designer) of choosing 
somewhat different glyphs, or for a rendering engine to include some 
distinctions (such as showing the precomposed variant with a separated 
combining character).

Second, there are other cases where in general, no typographic 
distinction is used. க and ௧ (Tamil 'ka' and '1') would be a good example.

> outside the context of a locale.

There's no guarantee that all text in a Fula locale (e.g. on a computer 
with a Fula OS, if there ever is such a thing) will have Behs with 
Hamzas above represented with U+08A1 (composed) rather than decomposed.
بٔ U+0628 U+0654 (decomposed) may get in there from Arabic text easily.

> John Klensin

Given that JSON implementations, in contrast to IDNA, compare object 
member names codepoint-by-codepoint, the much more obvious addition to 
the security section would be to point out the potential consequences 
for precomposed/decomposed confusions in general. The chance that this 
becomes an actual issue is much much higher (although still rather low) 
in e.g. French or German than in Fula, where Arabic isn't even the main 
script used for writing the language.

Regards,   Martin.

P.S.: Please note that the comments above don't mean that I'm happy with 
the inclusion of U+08A1 in Unicode 7.0.0, and that I sincerely hope the 
Unicode Consortium will weight the problems of identifier confusability 
higher in their future decisions.

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 Jan 2015 10:22:10 -0600

On Wed, Jan 21, 2015 at 01:04:22PM +0900, "Martin J. Dürst" wrote:
> member names codepoint-by-codepoint, the much more obvious addition
> to the security section would be to point out the potential
> consequences for precomposed/decomposed confusions in general. The
> chance that this becomes an actual issue is much much higher
> (although still rather low) in e.g. French or German than in Fula,
> where Arabic isn't even the main script used for writing the
> language.

A JSON implementation could normalize as it hashes (if it hashes) the
keys, though, of course, RFC7159 says nothing about this.

Implementations of form-insensitive Unicode character string comparison
and hashing functions do exist that basically normalize one _character_
(not codepoint) at a time.  The reason for doing it this way is
performance: there's no need to allocate memory dynamically for this,
and in many cases most characters in the input string(s) have just one
form or are already in canonical form.

> P.S.: Please note that the comments above don't mean that I'm happy
> with the inclusion of U+08A1 in Unicode 7.0.0, and that I sincerely
> hope the Unicode Consortium will weight the problems of identifier
> confusability higher in their future decisions.

I thought that NFC was closed to new precompositions though new
precompositions might be added to Unicode.  That is, the NFC form of
U+08A1 must be the same as the NFD form of U+08A1, which is to say:
U+0628 U+0654.

Is my memory wrong about that?

Nico
--

---

From: Barry Leiba <[email-masked]>
Date: Wed, 21 Jan 2015 11:34:47 -0500

All, can we please take this up in a separate message thread?  It's a
useful discussion to have, but it has nothing to do with my DISCUSS
and comments on the document.

Thanks,
Barry

---

From: John C Klensin <[email-masked]>
Date: Wed, 21 Jan 2015 14:26:32 -0500

(Changing subject line per Barry's request and addressing that
one issue only.)

--On Wednesday, January 21, 2015 13:04 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

>...
> There's no guarantee that all text in a Fula locale (e.g. on a
> computer with a Fula OS, if there ever is such a thing)

Note that isn't "a Fula locale", but "a Fula locale that uses
the Arabic script to write the language".  There are no known
such locales of even country-scope; Fula is normally written in
Latin characters and has been for at least a few centuries.
There are lots of problems with the analogy,  but thinking about
Fula-in-Arabic in ordinary locale terms is a little bit like
thinking about Japanese-exclusively-in-Romanji (and Romanji with
a few characters that don't appear anywhere else in Latin
script) as a locale.  Probably possible to make it work, but not
the way most of us usually think about things.

> will
> have Behs with Hamzas above represented with U+08A1 (composed)
> rather than decomposed.

I presume that is one of the reasons TUS 7.0 says
"preferentially" should use the precombined (composed) form
rather than the decomposed one rather than something we would
recognize as "MUST".

>> draft-klensin-idna-5892upd-unicode70-03.txt.  You may wish to
>> review that work, and ask i8n program for a copy of the draft
>> statement, because it may have security implications on your
>> work, in as much as i-json is used to pass identifiers.
> 
> Given that JSON implementations, in contrast to IDNA, compare
> object member names codepoint-by-codepoint, the much more
> obvious addition to the security section would be to point out
> the potential consequences for precomposed/decomposed
> confusions in general. The chance that this becomes an actual
> issue is much much higher (although still rather low) in e.g.
> French or German than in Fula, where Arabic isn't even the
> main script used for writing the language.

Agreed.  And, fwiw, that would be my preferred solution.   I
think Fula and U+08A1 are important only as illustrative
examples of what appears to me to be a very nasty situation when
the nature of the identifier(s) or their uses does not provide a
reliable language context.

> P.S.: Please note that the comments above don't mean that I'm
> happy with the inclusion of U+08A1 in Unicode 7.0.0, and that
> I sincerely hope the Unicode Consortium will weight the
> problems of identifier confusability higher in their future
> decisions.

Agreed, although the way in which the U+08A1 decision has been
defended, and the antecedents for it, do not predict to that
result.  See below.

--On Wednesday, January 21, 2015 10:22 -0600 Nico Williams
<[email-masked]> wrote:

> precompositions might be added to Unicode.  That is, the NFC
> form of U+08A1 must be the same as the NFD form of U+08A1,
> which is to say: U+0628 U+0654.
> 

That is the understanding that several -- I dare to say most or
all of  the IDNAbis WG participants -- of us had.  What has
actually occurred either violates that assumption or introduces
an extra case, depending on how one looks at the problem.   We
understood that, if precompositions (your term) are added, they
would have decompositions and NFC of the character would yield
the decomposed form.  That understanding derived both from what
the WG was told and from text in the Unicode Standard and UAX
#15 (see draft-klensin-idna-5892upd-unicode70-02, especially
Section 2.1, for details and references).  

But, while U+08A1 is abstract-character-identical and even
plausible-name-identical to U+0628 U+0654, it does _not_
decompose into the latter.  Instead, NFD(U+08A1) = NFC(U+08A1) =
U+08A1.  NFC (U+0628 U+0654) is U+0628 U+0654 as one would
expect from the stability rules; from that perspective, it is
the failure of U+08A1 to have a (non-identity) decomposition
that is the issue.

     john

---

From: Asmus Freytag <[email-masked]>
Date: Wed, 21 Jan 2015 12:36:04 -0800

> confusability higher in their future decisions. 

Given that an almost exactly parallel issue existed for a code point 
used in several Western European languages from Unicode 1.0, I think it 
would be wrong for Unicode to suddenly change the way non-Western 
scripts are encoded. Treating this particular case in isolation obscures 
the real issue and will possibly prevent or delay a proper design of 
identifier repertoires.

To put the issue in general terms, it concerns the fact that there are 
homographs in Unicode. These are two distinct code point sequences with 
normally identical appearance. In particular, the concern is with 
homographs that are present after normalization.

I'm very much on board with highlighting that issue, which is 
that*applying **NFC does not eliminate homographs*. And, having just 
finished the exercise of reviewing the full repertoire of modern scripts 
for suitability for the DNS root zone, I can attest that there are more 
homographs than people might initially suspect, and quite a few of them 
in Latin.

In most, but not all cases, one of the two is a code point or code point 
sequence that exists for a very specialized purpose. Often only one of 
the forms is actually used in general orthographies and only one of the 
forms would therefore be expected to occur in a set of mnemonic identifiers.

Unfortunately, it is not always the composite one that should be 
supported. For example, Unicode has several non-normalizable Latin 
digraphs that are encoded for special usage scenarios; in these cases 
the individual code points must be supported.  In some cases, a letter 
and digit may be homographs (the example of க and ௧ (Tamil 'ka' and '1') 
as mentioned in the preceding post). Both would be supported for 
different purposes. Finally, in some cases, there's a combining mark 
that (given its name and general appearance) might be expected to yield 
the same appearance when applied to some base letters, as certain 
precomposed forms. In Latin, this applies to combining overlays, 
because, on principle, the Unicode standard does not decompose 
orthographic characters for which the shape is derived by striking 
through part or all of the letter form.

Like the case of the Arabic script, any such characters needed for an as 
yet unencoded Latin orthography, would be encoded with a composite glyph 
shape, but without decomposition.

*The proper response for IDNA2008* would be to inventorize these cases 
and *strongly warn* that they not be incorporated unexamined into 
general repertoires; or, if they have to be supported, that Label 
Generation Rulesets (aka IDN tables) support context or variant rules 
that prevent these from co-occurring in any minimal pair of labels.

For language-specific IDN tables, it's often possible to eliminate one 
or the other alternative.

For example, a Danish IDN table would rule out 0338 (combining slash), 
so that <o, 0338> cannot exist alongside o-slash. For a Fula-specific 
IDN table, one would rule out the combining Hamza - it has not place in 
that orthography.

Eliminating any particular homographs on an ad-hoc basis in IDN2008 by 
making one of the code points INVALID does not solve the general 
problem, but unnecessarily prevents language-specific solutions in a way 
that is at best inconsistent and at worst discriminatory.

The Fula character is a good example of a pseudo-decomposable character 
that is needed for consistent encoding of a hitherto not fully supported 
orthography, while the code point sequence serves a specialized purpose 
elsewhere.

It is very important, that whatever the solution is decided on for 
IDNA2008, that IETF not haphazardly single out a particular instance of 
a general pattern.

A./

---

From: "Matthew A. Miller" <[email-masked]>
Date: Wed, 21 Jan 2015 13:43:19 -0700

Replying in the (hopefully not vain) attempt to strip the recipient list
to something our poor system can handle.

Continue the conversation on this thread.

Thanks,

- -- your JSON WG co-chair