From nobody Thu Oct 1 06:50:52 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C7FF71A6F11 for ; Thu, 1 Oct 2015 06:50:50 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.91 X-Spam-Level: X-Spam-Status: No, score=-3.91 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, GB_I_LETTER=-2, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J3Nj0RljoTW0 for ; Thu, 1 Oct 2015 06:50:47 -0700 (PDT) Received: from bsa2.jck.com (ns.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 00A9F1A6EE7 for ; Thu, 1 Oct 2015 06:50:46 -0700 (PDT) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1ZheFu-000L0F-Ru; Thu, 01 Oct 2015 09:50:42 -0400 Date: Thu, 01 Oct 2015 09:50:37 -0400 From: John C Klensin To: Peter Saint-Andre - &yet , Tom Worster , Alexey Melnikov Message-ID: <588752141F4228C805E674FC@JcK-HP8200.jck.com> In-Reply-To: <560C5149.5090607@andyet.net> References: <560C5149.5090607@andyet.net> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 13:50:51 -0000 --On Wednesday, September 30, 2015 15:16 -0600 Peter Saint-Andre - &yet wrote: > Hi Tom, thanks for the note. >=20 > My feeling is that we phrased things in a slightly wrong way, > because we assumed that case-mapping applies primarily or only > to uppercase and titlecase characters. I think this was more a > matter of communication (because people think of case mapping > as something needed only with respect to uppercase > characters), whereas it obviously applies more generally > (i.e., applying Unicode Default Case Folding will result in > mapping of the code points you mention here). >=20 > We could do something like this in the nickname spec... >... =20 > NEW >=20 > 3. Case Mapping Rule: Unicode Default Case Folding MUST > be applied, > as defined in the Unicode Standard [Unicode] (at the > time > of this writing, the algorithm is specified in Chapter > 3 of > [Unicode7.0]). The primary result of doing so is that > uppercase > characters are mapped to lowercase characters. In > applications > that prohibit conflicting nicknames, this rule helps > to reduce > the possibility of confusion by ensuring that = nicknames > differing only by case (e.g., "stpeter" vs. "StPeter") > would not > be presented to a human user at the same time. >... Peter, While your proposed text is an improvement, the desire of many people for a magic "just tell me what to do" formula, one that lets them avoid understanding the issues, may call for a little more: (1) First, toCaseFold is _not_ toLowerCase. Saying "The primary result of doing so is that uppercase characters are mapped to lowercase characters" is true for toCaseFold, but it has other effects that are information-losing and may be counterintuitive in some locales and situations. =20 (2) Second, probably as a result of having IDNA in the lead, we've gotten sloppy about language and operations and should probably start untangling that before it gets people in trouble. The Unicode Standard, at least as I understand it, is fairly clear that the most important (and really only safe) use of toCaseFold is as part of a comparison operation. Using your example it is entirely reasonable to treat, "stpeter" and "StPeter" as equivalent in a comparison operation, but accepting one string and changing it to the other for display may not be a really good idea. While that transformation may be acceptable (although I would be surprised if there were no people who share your surname who could consider "stpeter" or "Stpeter" unacceptable and might even believe that "StPeter" is an unacceptable substitute for "St. Peter"), it also points out the dangers of using Basic Latin script examples to illustrate situations in which even more extended Latin script, much less other scripts, may raise more complex issues. Because IDNA is essentially a workaround because changing the DNS comparison rules was impractical for several reasons, we ended up using toCaweFold to map characters and strings into others in IDNA2003 but PRECIS implementations that do not have the same constraints would, in general, be better off confining the use of toCaseFold, or even toLowerCase, to comparison operations. =20 (3) Because toCaweFold loses information when used for more than comparison (for comparison, it merely contributes to what some people would consider false positives for matching) involves some controversial decisions and, because of stability requirements, cannot be changed even if the controversies are resolved in other ways, we end up with, e.g., toCaseFold ("Nu=C3=9F") -> "nuss" which is considered an acceptable transformation in some places that identify themselves as speaking/using German and two different unacceptable errors in others. Again, this will almost always be much more serious if the transformation is used to map and replace strings than if it is used to compare (fwiw, that particular example is part of a continuing disagreement between IDNA2008 and, among others, German domain registry authorities on one side and UTC and UTR 46 on the other). (4) If the motivation is really to avoid confusion, the correct confusion-blocking rule for Latin script (but not others) and many languages that use it (but certainly not all) involves moving beyond toCaseFold and treating all "decorated" characters (characters normally represented by glyphs consisting of a Basic Latin character and one or more diacritical or equivalent markings) compare equal to their base characters, e.g., "=C3=A1" = not only matches "=C3=81" but also "a" and "A" and, as an = unfortunate side-effect, maybe "=C3=80" and "=C3=A0" as well. This is bad = news for languages in which decorated Latin characters are used to represent phonetically and conceptually different characters, not just pronunciation variations. I am not qualified to evaluate "how bad". In addition, extrapolations from this principle about Latin script to unrelated scripts will almost certainly lead to serious errors and/or additional confusion. More on this and Tom's question below... =20 > On 9/29/15 3:28 PM, Tom Worster wrote: >> Peter, Alexey, >>=20 >> I think there is an ambiguity in the specification of case >> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >... >> But there are 55 code points in Unicode 7.0.0 that change >> under default case folding that are neither uppercase nor >> titlecase characters, 12 of which are Lowercase_Letter. I >> suspect this stems from a confusion between Unicode case >> mapping and case folding.=20 Yes, I think so. See above, but, if I were making the rules, I would say "never use toCaseFold where case mapping is intended and, in particular, where one wants to substitute one string for another rather than checking a pair of strings for equivalence or perhaps telling users what would be considered equivalent". That interpretation is, I believe, consistent with most of the Unicode FAQ text you have quoted and other Unicode statements. However I have lost that argument before and hope, given decisions that have been made and deployed, that I was wrong. But there is another issue... >... >> The nickname profile can be corrected or the algorithm >> clarified. I'm not sure what to do with a Proposed Standard >> RFC. Errata? Can the case mapping rule be changed in IANA? >> https://www.iana.org/assignments/precis-parameters/profiles/U >> sernameCaseMap ped.txt >> e.g. to "Apply Unicode default case folding" Almost certainly not... an "update" revision of the spec would be needed. At least a few of the characters you questioned raise another issue: >... >> Ll; 03D0; C; 03B2; # GREEK BETA SYMBOL >> Ll; 03D1; C; 03B8; # GREEK THETA SYMBOL >> Ll; 03D5; C; 03C6; # GREEK PHI SYMBOL >> Ll; 03D6; C; 03C0; # GREEK PI SYMBOL >> Ll; 03F0; C; 03BA; # GREEK KAPPA SYMBOL >> Ll; 03F1; C; 03C1; # GREEK RHO SYMBOL >> Ll; 03F5; C; 03B5; # GREEK LUNATE EPSILON SYMBOL >... >> Ll; 1FBE; C; 03B9; # GREEK PROSGEGRAMMENI >> Nl; 2160; C; 2170; # ROMAN NUMERAL ONE > (etc) >> So; 24B6; C; 24D0; # CIRCLED LATIN CAPITAL LETTER A >> So; 24B7; C; 24D1; # CIRCLED LATIN CAPITAL LETTER B >... Those examples, and others, are independent of their Unicode categories, not characters used in writing "words" of normal languages. Most of them are inherently confusable with the similar-looking letters, e.g., U+2160 and U+2170 with upper and lower-case "I" (and ("i") respectively or U+03D0 and its relationship to "=CE=B2". The latter also raises the now-purely-academic question of whether a "variant letterform", such as U+03D0, violates the Unicode principle that different code points are not assigned to different glyph forms of the same latter, but those kinds of questions are another thing that makes these discussions difficult, especially for those who don't want to get involved with even script-specific or locale-specific details. To the extent possible, we dealt with such characters in IDNA2008 by identifying them as DISALLOWED, but PRECIS permits enough additional flexibility to, as you have noticed, allow people who don't understand what they are doing (or who are trying to avoid that necessity) to get themselves and their users into a lot of trouble. Fewer easy answers here than one would like and would expect in some alternate and easier reality. best, john From nobody Thu Oct 1 09:15:19 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 456501B2DC5 for ; Thu, 1 Oct 2015 09:15:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.901 X-Spam-Level: X-Spam-Status: No, score=-3.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, GB_I_LETTER=-2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oZHIewbe-yjO for ; Thu, 1 Oct 2015 09:15:11 -0700 (PDT) Received: from smtp98.iad3a.emailsrvr.com (smtp98.iad3a.emailsrvr.com [173.203.187.98]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1839E1B2DBF for ; Thu, 1 Oct 2015 09:15:10 -0700 (PDT) Received: from smtp13.relay.iad3a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp13.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 341931002CA; Thu, 1 Oct 2015 12:15:09 -0400 (EDT) Received: by smtp13.relay.iad3a.emailsrvr.com (Authenticated sender: fsb-AT-thefsb.org) with ESMTPSA id C94C81002CC; Thu, 1 Oct 2015 12:15:06 -0400 (EDT) X-Sender-Id: fsb@thefsb.org Received: from [10.0.1.2] (c-73-4-147-142.hsd1.ma.comcast.net [73.4.147.142]) (using TLSv1 with cipher DES-CBC3-SHA) by 0.0.0.0:465 (trex/5.4.2); Thu, 01 Oct 2015 16:15:09 GMT User-Agent: Microsoft-MacOutlook/14.5.5.150821 Date: Thu, 01 Oct 2015 12:15:03 -0400 From: Tom Worster To: , John C Klensin , Peter Saint-Andre - &yet , Alexey Melnikov Message-ID: Thread-Topic: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname References: <560C5149.5090607@andyet.net> <588752141F4228C805E674FC@JcK-HP8200.jck.com> In-Reply-To: <588752141F4228C805E674FC@JcK-HP8200.jck.com> Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable Archived-At: Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 16:15:16 -0000 Thanks, John, for the most interesting email I got this week:) It raises a question I would like to ask not only to you. As an implementer working through the specs, I got the clear impression that, while the PRECIS framework can potentially be useful for various things, the four profiles are explicitly specified exclusively for the purpose of string comparison operations on usernames, passwords and nicknames. If someone uses them for some other purpose and thereby causes problems, it's not RFC 7613's fault. The question is, did I get the wrong impression? If not then the discussion of how the profiles should specify case map/folding, which is specified only in the profiles, is simpler. The points to resolve being, as John put it, the "false positives". Fwiw, the (immature and inadequately tested) implementation is open at https://github.com/tom--/precis Tom On 10/1/15, 9:50 AM, "John C Klensin" wrote: >--On Wednesday, September 30, 2015 15:16 -0600 Peter Saint-Andre >- &yet wrote: > >> Hi Tom, thanks for the note. >>=20 >> My feeling is that we phrased things in a slightly wrong way, >> because we assumed that case-mapping applies primarily or only >> to uppercase and titlecase characters. I think this was more a >> matter of communication (because people think of case mapping >> as something needed only with respect to uppercase >> characters), whereas it obviously applies more generally >> (i.e., applying Unicode Default Case Folding will result in >> mapping of the code points you mention here). >>=20 >> We could do something like this in the nickname spec... >>... > =20 >> NEW >>=20 >> 3. Case Mapping Rule: Unicode Default Case Folding MUST >> be applied, >> as defined in the Unicode Standard [Unicode] (at the >> time >> of this writing, the algorithm is specified in Chapter >> 3 of >> [Unicode7.0]). The primary result of doing so is that >> uppercase >> characters are mapped to lowercase characters. In >> applications >> that prohibit conflicting nicknames, this rule helps >> to reduce >> the possibility of confusion by ensuring that nicknames >> differing only by case (e.g., "stpeter" vs. "StPeter") >> would not >> be presented to a human user at the same time. >>... > >Peter, > >While your proposed text is an improvement, the desire of many >people for a magic "just tell me what to do" formula, one that >lets them avoid understanding the issues, may call for a little >more: > >(1) First, toCaseFold is _not_ toLowerCase. Saying "The primary >result of doing so is that uppercase characters are mapped to >lowercase characters" is true for toCaseFold, but it has other >effects that are information-losing and may be counterintuitive >in some locales and situations. > >(2) Second, probably as a result of having IDNA in the lead, >we've gotten sloppy about language and operations and should >probably start untangling that before it gets people in trouble. >The Unicode Standard, at least as I understand it, is fairly >clear that the most important (and really only safe) use of >toCaseFold is as part of a comparison operation. Using your >example it is entirely reasonable to treat, "stpeter" and >"StPeter" as equivalent in a comparison operation, but accepting >one string and changing it to the other for display may not be a >really good idea. While that transformation may be acceptable >(although I would be surprised if there were no people who share >your surname who could consider "stpeter" or "Stpeter" >unacceptable and might even believe that "StPeter" is an >unacceptable substitute for "St. Peter"), it also points out the >dangers of using Basic Latin script examples to illustrate >situations in which even more extended Latin script, much less >other scripts, may raise more complex issues. Because IDNA is >essentially a workaround because changing the DNS comparison >rules was impractical for several reasons, we ended up using >toCaweFold to map characters and strings into others in IDNA2003 >but PRECIS implementations that do not have the same constraints >would, in general, be better off confining the use of >toCaseFold, or even toLowerCase, to comparison operations. > >(3) Because toCaweFold loses information when used for more than >comparison (for comparison, it merely contributes to what some >people would consider false positives for matching) involves >some controversial decisions and, because of stability >requirements, cannot be changed even if the controversies are >resolved in other ways, we end up with, e.g., > toCaseFold ("Nu=C3=9F") -> "nuss" >which is considered an acceptable transformation in some places >that identify themselves as speaking/using German and two >different unacceptable errors in others. Again, this will >almost always be much more serious if the transformation is used >to map and replace strings than if it is used to compare (fwiw, >that particular example is part of a continuing disagreement >between IDNA2008 and, among others, German domain registry >authorities on one side and UTC and UTR 46 on the other). > >(4) If the motivation is really to avoid confusion, the correct >confusion-blocking rule for Latin script (but not others) and >many languages that use it (but certainly not all) involves >moving beyond toCaseFold and treating all "decorated" characters >(characters normally represented by glyphs consisting of a Basic >Latin character and one or more diacritical or equivalent >markings) compare equal to their base characters, e.g., "=C3=A1" not >only matches "=C3=81" but also "a" and "A" and, as an unfortunate >side-effect, maybe "=C3=80" and "=C3=A0" as well. This is bad news for >languages in which decorated Latin characters are used to >represent phonetically and conceptually different characters, >not just pronunciation variations. I am not qualified to >evaluate "how bad". In addition, extrapolations from this >principle about Latin script to unrelated scripts will almost >certainly lead to serious errors and/or additional confusion. > >More on this and Tom's question below... >=20 >> On 9/29/15 3:28 PM, Tom Worster wrote: >>> Peter, Alexey, >>>=20 >>> I think there is an ambiguity in the specification of case >>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>... >>> But there are 55 code points in Unicode 7.0.0 that change >>> under default case folding that are neither uppercase nor >>> titlecase characters, 12 of which are Lowercase_Letter. I >>> suspect this stems from a confusion between Unicode case >>> mapping and case folding. > >Yes, I think so. See above, but, if I were making the rules, I >would say "never use toCaseFold where case mapping is intended >and, in particular, where one wants to substitute one string for >another rather than checking a pair of strings for equivalence >or perhaps telling users what would be considered equivalent". >That interpretation is, I believe, consistent with most of the >Unicode FAQ text you have quoted and other Unicode statements. >However I have lost that argument before and hope, given >decisions that have been made and deployed, that I was wrong. >But there is another issue... > >>... >>> The nickname profile can be corrected or the algorithm >>> clarified. I'm not sure what to do with a Proposed Standard >>> RFC. Errata? Can the case mapping rule be changed in IANA? >>> https://www.iana.org/assignments/precis-parameters/profiles/U >>> sernameCaseMap ped.txt >>> e.g. to "Apply Unicode default case folding" > >Almost certainly not... an "update" revision of the spec would >be needed. > >At least a few of the characters you questioned raise another >issue: > >>... >>> Ll; 03D0; C; 03B2; # GREEK BETA SYMBOL >>> Ll; 03D1; C; 03B8; # GREEK THETA SYMBOL >>> Ll; 03D5; C; 03C6; # GREEK PHI SYMBOL >>> Ll; 03D6; C; 03C0; # GREEK PI SYMBOL >>> Ll; 03F0; C; 03BA; # GREEK KAPPA SYMBOL >>> Ll; 03F1; C; 03C1; # GREEK RHO SYMBOL >>> Ll; 03F5; C; 03B5; # GREEK LUNATE EPSILON SYMBOL >>... >>> Ll; 1FBE; C; 03B9; # GREEK PROSGEGRAMMENI >>> Nl; 2160; C; 2170; # ROMAN NUMERAL ONE >> (etc) >>> So; 24B6; C; 24D0; # CIRCLED LATIN CAPITAL LETTER A >>> So; 24B7; C; 24D1; # CIRCLED LATIN CAPITAL LETTER B >>... > >Those examples, and others, are independent of their Unicode >categories, not characters used in writing "words" of normal >languages. Most of them are inherently confusable with the >similar-looking letters, e.g., U+2160 and U+2170 with upper and >lower-case "I" (and ("i") respectively or U+03D0 and its >relationship to "=CE=B2". The latter also raises the >now-purely-academic question of whether a "variant letterform", >such as U+03D0, violates the Unicode principle that different >code points are not assigned to different glyph forms of the >same latter, but those kinds of questions are another thing that >makes these discussions difficult, especially for those who >don't want to get involved with even script-specific or >locale-specific details. To the extent possible, we dealt >with such characters in IDNA2008 by identifying them as >DISALLOWED, but PRECIS permits enough additional flexibility to, >as you have noticed, allow people who don't understand what they >are doing (or who are trying to avoid that necessity) to get >themselves and their users into a lot of trouble. > >Fewer easy answers here than one would like and would expect in >some alternate and easier reality. > >best, > john > From nobody Thu Oct 1 10:27:44 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3671F1B2DFC for ; Thu, 1 Oct 2015 10:27:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.91 X-Spam-Level: X-Spam-Status: No, score=-1.91 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9rBY1ne5YmQg for ; Thu, 1 Oct 2015 10:27:31 -0700 (PDT) Received: from bsa2.jck.com (ns.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2E1081B2E2C for ; Thu, 1 Oct 2015 10:27:31 -0700 (PDT) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Zhhdf-000LWC-6n; Thu, 01 Oct 2015 13:27:27 -0400 Date: Thu, 01 Oct 2015 13:27:22 -0400 From: John C Klensin To: Tom Worster , precis@ietf.org, Peter Saint-Andre - &yet , Alexey Melnikov Message-ID: In-Reply-To: References: <560C5149.5090607@andyet.net> <588752141F4228C805E674FC@JcK-HP8200.jck.com> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2015 17:27:36 -0000 --On Thursday, October 01, 2015 12:15 -0400 Tom Worster wrote: > Thanks, John, for the most interesting email I got this week:) :-) I wish that this work were a lot less interesting, but so it goes. > It raises a question I would like to ask not only to you. As an > implementer working through the specs, I got the clear > impression that, while the PRECIS framework can potentially be > useful for various things, the four profiles are explicitly > specified exclusively for the purpose of string comparison > operations on usernames, passwords and nicknames. If someone > uses them for some other purpose and thereby causes problems, > it's not RFC 7613's fault. I'm in a bit of a difficult position on this because I was never a big PRECIS fan. It is probably best for you to assume that the reasons are very technical (most are) but the main reason why I didn't spot the paragraph you found and provide that explanation before or during IETF Last Call is that I simply gave up (after getting the WG to agree to resolve some problems I considered much more significant). So, if you have questions about intent, Peter, ALexey, or others may be better qualified to answer them. > The question is, did I get the wrong impression? > > If not then the discussion of how the profiles should specify > case map/folding, which is specified only in the profiles, is > simpler. The points to resolve being, as John put it, the > "false positives". Yes and no. Maybe it is useful to think about it this way. If I'm a user, I want the systems and interfaces to be as intuitive as possible relative to whatever I already know or think I do. If I'm looking at text written in a particular language, I expect the identifiers embedded in that text to obey that language's rules (or what I think they are). That argues for a highly localized environment in which, e.g., French-as-used-in-France and France-as-used-in-Quebec have different case matching rules and Arabic-language use of Arabic script is a lot different from Persian-language use of Arabic script even before one gets to "two sets of digits" or an interesting discussion known to some specialists as "a digit is a digit". On the other hand, if I'm a system or library designer who really wants to develop a single library, based around a single profile and/or easily-switched tables, I really want a single worldwide approach because trying to work things out separately for every language, script, and location is almost impossible to think about (and trying probably leads to madness) Each of those groups is justified in believing that the other group is living in an alternate, and profoundly unrealistic, reality. For some purposes, we can have a common code base and allow considerable user choice, but if you and I could make different choices about whether two valid strings match as identifiers, we (and everyone else) are in big trouble. In IDNA2008, we tried (and failed, btw) to simply disallow all of the characters that led to difficult edge cases. PRECIS made a different, profile-based, set of decisions that, on the one hand, allow doing things that may be more intuitive for users and that, on the other, may lead to surprises or contradictions. For basic Latin script, the case-matching problem was solved with Multics circa 40 years ago, albeit brutally (a solution carried into Unix): if something doesn't match at the bitstring level, it doesn't match and people just need to get used to it. That causes obvious problems with expectations and assumptions, but is at least 100% ambiguous. As soon as one starts marching down the path of case-independence for non-ASCII and then non-Latin characters, it gets hard to explain to users -- users whom we've managed to convince that computers are smart and adaptable -- why "Color" and "color" should match but "colour" (or even "couleur" or "Farbe") don't match and are different. > Fwiw, the (immature and inadequately tested) implementation is > open at https://github.com/tom--/precis Good luck. john From nobody Sun Oct 11 08:28:32 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3CE831A8F45; Sun, 11 Oct 2015 08:28:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.261 X-Spam-Level: X-Spam-Status: No, score=-1.261 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_SE=0.35, MIME_8BIT_HEADER=0.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id engRcj1UI8Sa; Sun, 11 Oct 2015 08:28:26 -0700 (PDT) Received: from mail.frobbit.se (mail.frobbit.se [IPv6:2a02:80:3ffe::176]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 139121A8F42; Sun, 11 Oct 2015 08:28:17 -0700 (PDT) Received: from [192.168.1.199] (frobbit.cust.teleservice.net [85.30.128.225]) by mail.frobbit.se (Postfix) with ESMTPSA id 68ACD2148C; Sun, 11 Oct 2015 17:28:15 +0200 (CEST) From: "Patrik =?utf-8?b?RsOkbHRzdHLDtm0=?=" To: "Peter Saint-Andre - &yet" Date: Sun, 11 Oct 2015 17:28:14 +0200 Message-ID: <3A77F7E3-CA6C-438A-B9C8-08A2D3F36249@frobbit.se> In-Reply-To: <55E7898F.4070405@andyet.net> References: <20150902220443.12714.44152.idtracker@ietfa.amsl.com> <55E77FEE.8050209@andyet.net> <55E780DE.6060402@cs.tcd.ie> <55E7898F.4070405@andyet.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_2DAB0D9B-9A27-4040-960E-B37192728DB3_="; micalg=pgp-sha1; protocol="application/pgp-signature" X-Mailer: MailMate (1.9.2r5141) Archived-At: Cc: draft-ietf-precis-mappings.ad@ietf.org, precis@ietf.org, draft-ietf-precis-mappings.shepherd@ietf.org, stpeter@jabber.org, The IESG , draft-ietf-precis-mappings@ietf.org, precis-chairs@ietf.org, Stephen Farrell Subject: Re: [precis] Stephen Farrell's No Objection on draft-ietf-precis-mappings-11: (with COMMENT) X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 15:28:27 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_2DAB0D9B-9A27-4040-960E-B37192728DB3_= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On 3 Sep 2015, at 1:43, Peter Saint-Andre - &yet wrote: > On 9/2/15 5:06 PM, Stephen Farrell wrote: > >> >> Hiya, >> >> On 03/09/15 00:02, Peter Saint-Andre - &yet wrote: > > >>>> >>>> - The write-up didn't tell me if this is likely to be >>>> considered as neutral by the Unicode folks, or as >>>> something non-neutral. I'm curious about that, but not >>>> concerned much. I do think the IESG should be aware >>>> though if this is something that might cause e.g. >>>> liaison fun for the IAB later on. (I'm not saying I >>>> think it does/would btw, just that I'd like to know if >>>> it might.) > >>> >>> What do you mean by "neutral"? > >> >> So I was being too oblique eh:-) Not something I'm good at >> I guess. >> >> I'm wondering if this contains anything about which they'd >> go ballistic. > > > I'm just a lowly shepherd, but I don't think so... I do not think so either, given it tries to be sort of neutral regarding = some mappings. See for example how it discusses the "sharp s" issue, whic= h in reality is an issue with versioning in Unicode. paf --=_MailMate_2DAB0D9B-9A27-4040-960E-B37192728DB3_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlYagA4ACgkQrMabGguI182u2QCcDNMgjgV4+zYUZZAeTdZy+xV3 I2QAn25doGrOBNTQ9gpINg/Z8tWydPME =1Buc -----END PGP SIGNATURE----- --=_MailMate_2DAB0D9B-9A27-4040-960E-B37192728DB3_=-- From nobody Mon Oct 26 14:51:26 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1459A1A1BC8 for ; Mon, 26 Oct 2015 14:51:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 1.099 X-Spam-Level: * X-Spam-Status: No, score=1.099 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, GB_I_LETTER=-2, MANGLED_LIST=2.3, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aIvoBvgyyevh for ; Mon, 26 Oct 2015 14:51:22 -0700 (PDT) Received: from mail-oi0-x232.google.com (mail-oi0-x232.google.com [IPv6:2607:f8b0:4003:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 790AB1A1BBE for ; Mon, 26 Oct 2015 14:51:22 -0700 (PDT) Received: by oifu63 with SMTP id u63so66552691oif.2 for ; Mon, 26 Oct 2015 14:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=from:subject:to:references:cc:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=9+eX1/bJgJm+VlALlrf8zc9PeKBYFYxds7YJPFGCeIA=; b=v7JuatuTs7JFWNXFt1U7f/czWJe8NWmn3B+NeWUWwy/6VDbE4jyqjYyUwNFs2yOY7Q UZZa6DqTzgxOkAbm0dxRYvG02TbG5eCkr+wTloprtDRz5TnDieXrA9aJ1EaSbk3nt02F k2t6t4iuhCwqtjbOgURNM8/YD7ol5f8ubYDdJDTLo4OSTzMhWunH4zMkygpWa3Tbfizy 0DWqxcuUNu/mzKsiZ5OuD8x12mqoEmcy9FxN0d0C6dzPjeL2/afRzQiBu2qko4YXwps9 hL+8ZfZmPnIwmsVRz2el3tGsA99tIKNGn114MhVATMS8umgH4F3SmLbdmxm2uiZuA+Cz aTIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:to:references:cc:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=9+eX1/bJgJm+VlALlrf8zc9PeKBYFYxds7YJPFGCeIA=; b=fLAQP2oef6/ifqULI0+wG2MYui2NHYNOPKoXHio+/tTrD2AscPAYF3m/nOlEJDkJhq /Iep5PHhb1Kl5hK7hC9TZHZIwEQWetRDXaD7IeB3iqa47LmzmT0wY4GkEU/tmnF430pB jL0Jjh6sOOuMVf1GadET9Qs9W7P59JNdxVlz7cDe6XT5YkRHNtPNvJ2o6T2oHk/C6EAA 5bP2QQBauOMnErUUyq4Bb8vfBcRNJQidCiJsV7PS1UemDcquZdYtpP4clYOikDitzvtw 9RMdpeWxnc29PY8fUFka0zoAXQfEqz5L0A6Sj65QgLcKRVPOIxb9q4MyFIQQbAQWwLnh nKNg== X-Gm-Message-State: ALoCoQnffQgG67H5aP8kctTIBGtESjNfEitWHE49UAyI3Bv3iA5sobFKRmsOATkAw4Bf/LYOVp02 X-Received: by 10.202.200.140 with SMTP id y134mr24775666oif.5.1445896281724; Mon, 26 Oct 2015 14:51:21 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id q2sm15846000obf.22.2015.10.26.14.51.19 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Oct 2015 14:51:20 -0700 (PDT) From: Peter Saint-Andre - &yet To: John C Klensin , Tom Worster , Alexey Melnikov References: <560C5149.5090607@andyet.net> <588752141F4228C805E674FC@JcK-HP8200.jck.com> Message-ID: <562EA055.3030404@andyet.net> Date: Mon, 26 Oct 2015 15:51:17 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <588752141F4228C805E674FC@JcK-HP8200.jck.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2015 21:51:25 -0000 My apologies for the delayed reply. Comments inline. On 10/1/15 7:50 AM, John C Klensin wrote: > --On Wednesday, September 30, 2015 15:16 -0600 Peter Saint-Andre > - &yet wrote: > >> Hi Tom, thanks for the note. >> >> My feeling is that we phrased things in a slightly wrong way, >> because we assumed that case-mapping applies primarily or only >> to uppercase and titlecase characters. I think this was more a >> matter of communication (because people think of case mapping >> as something needed only with respect to uppercase >> characters), whereas it obviously applies more generally >> (i.e., applying Unicode Default Case Folding will result in >> mapping of the code points you mention here). >> >> We could do something like this in the nickname spec... >> ... > >> NEW >> >> 3. Case Mapping Rule: Unicode Default Case Folding MUST >> be applied, >> as defined in the Unicode Standard [Unicode] (at the >> time >> of this writing, the algorithm is specified in Chapter >> 3 of >> [Unicode7.0]). The primary result of doing so is that >> uppercase >> characters are mapped to lowercase characters. In >> applications >> that prohibit conflicting nicknames, this rule helps >> to reduce >> the possibility of confusion by ensuring that nicknames >> differing only by case (e.g., "stpeter" vs. "StPeter") >> would not >> be presented to a human user at the same time. >> ... > > Peter, > > While your proposed text is an improvement, Happy to hear it. All I intended was a slight clarification. > the desire of many > people for a magic "just tell me what to do" formula, one that > lets them avoid understanding the issues, may call for a little > more: There is always a need for more when it comes to i18n. > (1) First, toCaseFold is _not_ toLowerCase. Saying "The primary > result of doing so is that uppercase characters are mapped to > lowercase characters" is true for toCaseFold, By "primary" I meant two things: (1) lowercasing is what happens to the preponderance of code points and (2) this is the result that most people care about. > but it has other > effects that are information-losing and may be counterintuitive > in some locales and situations. Indeed. > (2) Second, probably as a result of having IDNA in the lead, > we've gotten sloppy about language and operations and should > probably start untangling that before it gets people in trouble. Where is the right place to do that untangling? (I doubt that it is the precis-nickname document.) > The Unicode Standard, at least as I understand it, is fairly > clear that the most important (and really only safe) use of > toCaseFold is as part of a comparison operation. Thanks for noting that. For example, Section 5.18 of Unicode 8.0.0 says: Caseless matching is implemented using case folding, which is the process of mapping characters of different case to a single form, so that case differences in strings are erased. Case folding allows for fast caseless matches in lookups because only binary comparison is required. It is more than just conversion to lowercase. > Using your > example it is entirely reasonable to treat, "stpeter" and > "StPeter" as equivalent in a comparison operation, but accepting > one string and changing it to the other for display may not be a > really good idea. While that transformation may be acceptable > (although I would be surprised if there were no people who share > your surname who could consider "stpeter" or "Stpeter" > unacceptable and might even believe that "StPeter" is an > unacceptable substitute for "St. Peter"), I do receive email at stpeter@gmail.com intended for st.peter@gmail.com but that's a separate topic... > it also points out the > dangers of using Basic Latin script examples to illustrate > situations in which even more extended Latin script, much less > other scripts, may raise more complex issues. Because IDNA is > essentially a workaround because changing the DNS comparison > rules was impractical for several reasons, we ended up using > toCaweFold to map characters and strings into others in IDNA2003 > but PRECIS implementations that do not have the same constraints > would, in general, be better off confining the use of > toCaseFold, or even toLowerCase, to comparison operations. Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does it make sense for this nickname specification to differ in this respect from the published RFCs? Shall we file errata against those documents? (This might apply only to RFC 7613, which says to apply case folding as part of the enforcement process - when exactly to apply case folding is not stipulated by RFC 7564.) > (3) Because toCaweFold loses information when used for more than > comparison (for comparison, it merely contributes to what some > people would consider false positives for matching) involves > some controversial decisions and, because of stability > requirements, cannot be changed even if the controversies are > resolved in other ways, we end up with, e.g., > toCaseFold ("Nuß") -> "nuss" > which is considered an acceptable transformation in some places > that identify themselves as speaking/using German and two > different unacceptable errors in others. Again, this will > almost always be much more serious if the transformation is used > to map and replace strings than if it is used to compare (fwiw, > that particular example is part of a continuing disagreement > between IDNA2008 and, among others, German domain registry > authorities on one side and UTC and UTR 46 on the other). Agreed. > (4) If the motivation is really to avoid confusion, the correct > confusion-blocking rule for Latin script (but not others) and > many languages that use it (but certainly not all) involves > moving beyond toCaseFold and treating all "decorated" characters > (characters normally represented by glyphs consisting of a Basic > Latin character and one or more diacritical or equivalent > markings) compare equal to their base characters, e.g., "á" not > only matches "Á" but also "a" and "A" and, as an unfortunate > side-effect, maybe "À" and "à" as well. This is bad news for > languages in which decorated Latin characters are used to > represent phonetically and conceptually different characters, > not just pronunciation variations. I am not qualified to > evaluate "how bad". In addition, extrapolations from this > principle about Latin script to unrelated scripts will almost > certainly lead to serious errors and/or additional confusion. I would not be comfortable going that far... > More on this and Tom's question below... > >> On 9/29/15 3:28 PM, Tom Worster wrote: >>> Peter, Alexey, >>> >>> I think there is an ambiguity in the specification of case >>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >> ... >>> But there are 55 code points in Unicode 7.0.0 that change >>> under default case folding that are neither uppercase nor >>> titlecase characters, 12 of which are Lowercase_Letter. I >>> suspect this stems from a confusion between Unicode case >>> mapping and case folding. > > Yes, I think so. See above, but, if I were making the rules, I > would say "never use toCaseFold where case mapping is intended > and, in particular, where one wants to substitute one string for > another rather than checking a pair of strings for equivalence > or perhaps telling users what would be considered equivalent". > That interpretation is, I believe, consistent with most of the > Unicode FAQ text you have quoted and other Unicode statements. > However I have lost that argument before and hope, given > decisions that have been made and deployed, that I was wrong. > But there is another issue... > >> ... >>> The nickname profile can be corrected or the algorithm >>> clarified. I'm not sure what to do with a Proposed Standard >>> RFC. Errata? Can the case mapping rule be changed in IANA? >>> https://www.iana.org/assignments/precis-parameters/profiles/U >>> sernameCaseMap ped.txt >>> e.g. to "Apply Unicode default case folding" > > Almost certainly not... an "update" revision of the spec would > be needed. Yes, RFC 7613 would need to be updated via a separate spec or 7613bis. > At least a few of the characters you questioned raise another > issue: > >> ... >>> Ll; 03D0; C; 03B2; # GREEK BETA SYMBOL >>> Ll; 03D1; C; 03B8; # GREEK THETA SYMBOL >>> Ll; 03D5; C; 03C6; # GREEK PHI SYMBOL >>> Ll; 03D6; C; 03C0; # GREEK PI SYMBOL >>> Ll; 03F0; C; 03BA; # GREEK KAPPA SYMBOL >>> Ll; 03F1; C; 03C1; # GREEK RHO SYMBOL >>> Ll; 03F5; C; 03B5; # GREEK LUNATE EPSILON SYMBOL >> ... >>> Ll; 1FBE; C; 03B9; # GREEK PROSGEGRAMMENI >>> Nl; 2160; C; 2170; # ROMAN NUMERAL ONE >> (etc) >>> So; 24B6; C; 24D0; # CIRCLED LATIN CAPITAL LETTER A >>> So; 24B7; C; 24D1; # CIRCLED LATIN CAPITAL LETTER B >> ... > > Those examples, and others, are independent of their Unicode > categories, not characters used in writing "words" of normal > languages. Most of them are inherently confusable with the > similar-looking letters, e.g., U+2160 and U+2170 with upper and > lower-case "I" (and ("i") respectively or U+03D0 and its > relationship to "β". The latter also raises the > now-purely-academic question of whether a "variant letterform", > such as U+03D0, violates the Unicode principle that different > code points are not assigned to different glyph forms of the > same latter, but those kinds of questions are another thing that > makes these discussions difficult, especially for those who > don't want to get involved with even script-specific or > locale-specific details. To the extent possible, we dealt > with such characters in IDNA2008 by identifying them as > DISALLOWED, but PRECIS permits enough additional flexibility to, > as you have noticed, allow people who don't understand what they > are doing (or who are trying to avoid that necessity) to get > themselves and their users into a lot of trouble. This is certainly the case for FreeformClass in PRECIS. I would hope that we took a safer path with the IdentifierClass (e.g., U+03D0 is disallowed there). > Fewer easy answers here than one would like and would expect in > some alternate and easier reality. Always. :( After all that, I have 3 questions: (1) Is my proposed text enough of a clarification that we should make that change before the nickname I-D is published as an RFC? (2) Should we modify draft-ietf-precis-nickname so that case folding is applied only as part of comparison and not as part of enforcement? If so, should we make that change before this document is published as an RFC? (3) Should we update RFC 7613 so that case folding is applied only as part of comparison and not as part of enforcement? Peter From nobody Tue Oct 27 06:52:40 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B126A1A89A7 for ; Tue, 27 Oct 2015 06:52:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EFU2wXWT8N1P for ; Tue, 27 Oct 2015 06:52:38 -0700 (PDT) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1BAC91A89A8 for ; Tue, 27 Oct 2015 06:52:25 -0700 (PDT) Received: by oifu63 with SMTP id u63so77680570oif.2 for ; Tue, 27 Oct 2015 06:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=ikreFNwCqN8BBI6fyNXjtodTVHWldRXsdudZd7sUlpY=; b=DT3tkoSc/5nT7R2GzY3tqEUf5/Fuud3ieBvDDu8oSsoGCJVx2R8cHPlGwddtS2Jp7n J/djLmHw54Q/tn9mdlGx8T0ymQ6EkQe/7svRvxK36emI7Pyjp+UZXbmtHbAUMJb4PtB8 aP241O8M4tFMyeZiKg+cUyAJfuPNuwuNRlQuhuXFOPVOOTs2N6bk5M0Bjcp2TOuDVcGr Lv50MCL9LGqVSaJ30rTqMSTTPxViTIbrmqfHpDQXpokSB1pBRCnzUuK+g5qdUWUvTjEU cdDOoJzidG6CP2Soif8ehL7bp97QijGrd4eTUuAsS/bnBaEjXTb+AJozAAUHYhogkSo9 8U2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=ikreFNwCqN8BBI6fyNXjtodTVHWldRXsdudZd7sUlpY=; b=IwAbGxd75mU40YubPwOXpinw2FOPj7McjrErXoPn6x0wIR3mIENUSvlD8U21pPSG3M K2CCmFTVYgdyml6htqqJUzQAY/OoLON1hHxRGQYj9JK2S1p4ft1DrOFVxCHM00pW6yyS jSicw4tDpA10dfyaOpP5I+FKLTFYvq9ftGsEdPn9ZQf+Ty5MGmYLEqbSpV86KR8ra3a5 +p6NIYYUuakZb9nNwLJfOQltqpyJ2xQvR0+/9h4BEcEUUDokJeittnWPlLc6qvsUUjTP 3cw58S68P06Ft8YxcnF+0ZedPz/BL669WqnBP9vhwVFzpqFT7Z080bI4VTCYOmckSUll XB8w== X-Gm-Message-State: ALoCoQm0bVm5PPcSrbXgeFyCdhugNdbX9ZFs+J2lWgTQJYErY2ebTP7TXGoTns6Y9TYQ4FL1oI0b X-Received: by 10.202.80.67 with SMTP id e64mr27274678oib.125.1445953944183; Tue, 27 Oct 2015 06:52:24 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id tb8sm17438398obc.4.2015.10.27.06.52.22 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 27 Oct 2015 06:52:22 -0700 (PDT) To: Peter Saint-Andre - &yet , John C Klensin , Tom Worster , Alexey Melnikov References: <560C5149.5090607@andyet.net> <588752141F4228C805E674FC@JcK-HP8200.jck.com> <562EA055.3030404@andyet.net> From: Peter Saint-Andre Message-ID: <562F8194.2020108@andyet.net> Date: Tue, 27 Oct 2015 07:52:20 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <562EA055.3030404@andyet.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 13:52:39 -0000 This issue has greater urgency now because draft-ietf-precis-nickname is now in AUTH48... On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: > After all that, I have 3 questions: > > (1) Is my proposed text enough of a clarification that we should make > that change before the nickname I-D is published as an RFC? I think so. > (2) Should we modify draft-ietf-precis-nickname so that case folding is > applied only as part of comparison and not as part of enforcement? If > so, should we make that change before this document is published as an RFC? Although it seems to be the case that Unicode case folding is primarily designed for the purpose of matching (i.e., comparison), I have a concern that applying the PRECIS case mapping rule after applying the normalization and directionality rules might have unintended consequences that we haven't had a chance to consider yet. The PRECIS framework expresses a preference (actually a hard requirement) for applying the rules in a particular order. We made a late change to the username profiles (RFC 7613), such that width mapping is applied first (in order to accommodate fullwidth and halfwidth characters in certain East Asian scripts). Making a late change to the nickname profile also concerns me, even though both of these late changes seem reasonable on the face of it. I will try to find time to think about this further in the next 24 hours. > (3) Should we update RFC 7613 so that case folding is applied only as > part of comparison and not as part of enforcement? That is less urgent so I suggest that we address the nickname spec first. Peter From nobody Tue Oct 27 07:12:30 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EFE891A8A23 for ; Tue, 27 Oct 2015 07:12:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.61 X-Spam-Level: X-Spam-Status: No, score=-2.61 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jt_BbFj-5UKJ for ; Tue, 27 Oct 2015 07:12:27 -0700 (PDT) Received: from bsa2.jck.com (bsa2.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6B1231A8A1D for ; Tue, 27 Oct 2015 07:12:27 -0700 (PDT) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Zr4z6-000Ops-Ob; Tue, 27 Oct 2015 10:12:20 -0400 Date: Tue, 27 Oct 2015 10:12:15 -0400 From: John C Klensin To: Peter Saint-Andre , Tom Worster , Alexey Melnikov Message-ID: <6EB77470DA21E4ED85940C28@JcK-HP8200.jck.com> In-Reply-To: <562F8194.2020108@andyet.net> References: <560C5149.5090607@andyet.net> <588752141F4228C805E674FC@JcK-HP8200.jck.com> <562EA055.3030404@andyet.net> <562F8194.2020108@andyet.net> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 14:12:29 -0000 I am most of the way through a response now... should be able to mail it within an hour or so if I'm not interrupted again; later today in any event. john --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre wrote: > This issue has greater urgency now because > draft-ietf-precis-nickname is now in AUTH48... > > On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: > >> After all that, I have 3 questions: >> >> (1) Is my proposed text enough of a clarification that we >> should make that change before the nickname I-D is published >> as an RFC? > > I think so. > >> (2) Should we modify draft-ietf-precis-nickname so that case >> folding is applied only as part of comparison and not as part >> of enforcement? If so, should we make that change before this >> document is published as an RFC? > > Although it seems to be the case that Unicode case folding is > primarily designed for the purpose of matching (i.e., > comparison), I have a concern that applying the PRECIS case > mapping rule after applying the normalization and > directionality rules might have unintended consequences that > we haven't had a chance to consider yet. The PRECIS framework > expresses a preference (actually a hard requirement) for > applying the rules in a particular order. We made a late > change to the username profiles (RFC 7613), such that width > mapping is applied first (in order to accommodate fullwidth > and halfwidth characters in certain East Asian scripts). > Making a late change to the nickname profile also concerns me, > even though both of these late changes seem reasonable on the > face of it. I will try to find time to think about this > further in the next 24 hours. > >> (3) Should we update RFC 7613 so that case folding is applied >> only as part of comparison and not as part of enforcement? > > That is less urgent so I suggest that we address the nickname > spec first. > > Peter > From nobody Tue Oct 27 07:23:22 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE5891A8A55 for ; Tue, 27 Oct 2015 07:23:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uLoje8h1umPM for ; Tue, 27 Oct 2015 07:23:20 -0700 (PDT) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6BF071A8A3B for ; Tue, 27 Oct 2015 07:23:20 -0700 (PDT) Received: by oies66 with SMTP id s66so120586265oie.1 for ; Tue, 27 Oct 2015 07:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=QaFRaDJGSDkXFdfGKjEWLz+WNOg6IFYEYzSKGzWfNic=; b=np5tyh+Wxu4cR/0QgToSBzwQOWFuK5TQ6tRq9s2/nCRfWKwNvzCPdnF+6UkWHQAiSZ cEzDxSspj7hAix8UWvTwgfkRTxzXFTRItvCN637M6fljve5iRXBa5aPDv2R4IRE0qI9p zy+G+oYUZdqQWcFi8ENdIJO3X6Pj9eeHbXNTblPv/+DziPb5dv2VEpourWhAadp4a2it HYKNNqPEJTF/87X69U5BYMYnFyxIY8GtDRFMT1BOCI4QjaaI0QUdasiCk8/1VjKQc7tG 15L9c6KAR548LSgA7Dsb1Piab7j8HrTVRgleAA2UH9cF/U2QjaG3c46fQ03ZwyKKpCB1 9rzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=QaFRaDJGSDkXFdfGKjEWLz+WNOg6IFYEYzSKGzWfNic=; b=fpM5e8xwdmx9l8wqC6CcKU0JdX7xxoR4ThsIvHUcDj56EFf+Ki85VrGdfjRLjj9rZx wSNqwtAWKxiZIGdgwe1dGt8TaAPPl4rlWTTUfpRmSM3O3Biz119GfCROtv3y/ThD+RAr E7jpuiGZ03DUHOtXSGSqSQIvH2M4a17TqIk0ctQzVB1dgEfEEUJP/0Vsdt8fVaIgjQV9 J5nsdCSmM8mzFfWbxRhqRx8Q78RpuIHW1cj8t/Xn7A4VKHL4jNp9+Enx751iN5T2BBv3 kl4dhQ/sHbOvzEMraD7VSQm0h662++hR6TaetFnBqYa5Yjk8YENtSGEuqK5N2QdHKdcZ Kpwg== X-Gm-Message-State: ALoCoQlTTKSYtN3gb4go5TUTfHoYyjC78ZUx5DBp1CIz1nz2FCKiHnDBqCgdB2/hptOapbLaNWhc X-Received: by 10.202.77.142 with SMTP id a136mr27659770oib.123.1445955799682; Tue, 27 Oct 2015 07:23:19 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id s19sm17269594oet.7.2015.10.27.07.23.17 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Oct 2015 07:23:18 -0700 (PDT) To: John C Klensin , Tom Worster , Alexey Melnikov References: <560C5149.5090607@andyet.net> <588752141F4228C805E674FC@JcK-HP8200.jck.com> <562EA055.3030404@andyet.net> <562F8194.2020108@andyet.net> <6EB77470DA21E4ED85940C28@JcK-HP8200.jck.com> From: Peter Saint-Andre Message-ID: <562F88D4.6070609@andyet.net> Date: Tue, 27 Oct 2015 08:23:16 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <6EB77470DA21E4ED85940C28@JcK-HP8200.jck.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 14:23:22 -0000 Thanks, John! On 10/27/15 8:12 AM, John C Klensin wrote: > I am most of the way through a response now... should be able to > mail it within an hour or so if I'm not interrupted again; later > today in any event. > > john > > > --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre > wrote: > >> This issue has greater urgency now because >> draft-ietf-precis-nickname is now in AUTH48... >> >> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >> >>> After all that, I have 3 questions: >>> >>> (1) Is my proposed text enough of a clarification that we >>> should make that change before the nickname I-D is published >>> as an RFC? >> >> I think so. >> >>> (2) Should we modify draft-ietf-precis-nickname so that case >>> folding is applied only as part of comparison and not as part >>> of enforcement? If so, should we make that change before this >>> document is published as an RFC? >> >> Although it seems to be the case that Unicode case folding is >> primarily designed for the purpose of matching (i.e., >> comparison), I have a concern that applying the PRECIS case >> mapping rule after applying the normalization and >> directionality rules might have unintended consequences that >> we haven't had a chance to consider yet. The PRECIS framework >> expresses a preference (actually a hard requirement) for >> applying the rules in a particular order. We made a late >> change to the username profiles (RFC 7613), such that width >> mapping is applied first (in order to accommodate fullwidth >> and halfwidth characters in certain East Asian scripts). >> Making a late change to the nickname profile also concerns me, >> even though both of these late changes seem reasonable on the >> face of it. I will try to find time to think about this >> further in the next 24 hours. >> >>> (3) Should we update RFC 7613 so that case folding is applied >>> only as part of comparison and not as part of enforcement? >> >> That is less urgent so I suggest that we address the nickname >> spec first. >> >> Peter >> From nobody Tue Oct 27 10:32:29 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2C3D71ACD2B for ; Tue, 27 Oct 2015 10:32:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.39 X-Spam-Level: X-Spam-Status: No, score=0.39 tagged_above=-999 required=5 tests=[BAYES_50=0.8, GB_I_LETTER=-2, MANGLED_LIST=2.3, RCVD_IN_DNSWL_LOW=-0.7, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4fOd9Ag5GAvh for ; Tue, 27 Oct 2015 10:32:16 -0700 (PDT) Received: from bsa2.jck.com (ns.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EB7B41ACD1D for ; Tue, 27 Oct 2015 10:32:15 -0700 (PDT) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Zr86T-000Prn-L1; Tue, 27 Oct 2015 13:32:09 -0400 Date: Tue, 27 Oct 2015 13:32:04 -0400 From: John C Klensin To: Peter Saint-Andre - &yet , Tom Worster , Alexey Melnikov Message-ID: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 17:32:22 -0000 Response to Monday's note immediately below; response to today's follows it. My apologies, but it is probably important to read both. My further apologies for the length of this note, but I think we are in deep trouble here, trouble that is aggravated by precis-mappings and precis-nickname both being post-approval and that, as far as I know, there are no future plans for PRECIS work (having precis-nickname in AUTH48 just emphasizes that -- see comment at end).=20 --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - &yet wrote: > My apologies for the delayed reply. Comments inline. A few remarks below... I can't tell whether we disagree or whether at least one of us, probably me, are not being adequately clear. (Material on which we fairly clearly agree elided.) > On 10/1/15 7:50 AM, John C Klensin wrote: >> --On Wednesday, September 30, 2015 15:16 -0600 Peter >> Saint-Andre - &yet wrote: >... >> Peter, >>=20 >> While your proposed text is an improvement, >=20 > Happy to hear it. All I intended was a slight clarification. But I'm not certain we are there yet... >> the desire of many >> people for a magic "just tell me what to do" formula, one = that >> lets them avoid understanding the issues, may call for a >> little more: >=20 > There is always a need for more when it comes to i18n. But I think it is a little more that that. I've heard several times, including in PRECIS meetings, requests for "just tell me what to do and make sure it isn't complicated" (or "I don't want to have to think about, much less understand, the issues"). We can debate whether giving in to those requests in the I18n case is wise. I think it leads directly to conclusions equivalent to "I understand my own script and writing system (or think I do) and therefore, since all writing systems must be pretty much the same, I understand all of the core issues in terms of my script and understanding". That, in turn, leads directly to the "how do you spell 'Z=C3=BCrich'?" and "all spellings of 'Zuerich' = should be treated as equivalent" discussion that sounded like they dominated a BOF at IETF 93. Now I actually think it is reasonable for someone to ask for a library that will do the job most of the time and that will almost never cause their users or customers to get angry at them. But, if we are going to call what we do "standards", they should contain sufficient information that would-be library authors can know what to do ... or understand that they are in over their heads. And, for these particular cases, we may need to explain, or help the library authors explain, why some cases will fail and, indeed, get users mad at vendors. =20 >> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >> primary result of doing so is that uppercase characters are >> mapped to lowercase characters" is true for toCaseFold, >=20 > By "primary" I meant two things: (1) lowercasing is what > happens to the preponderance of code points and (2) this is > the result that most people care about. If I parse the above correctly, I think you are wrong. I think what most people want, care about, and think they are getting, is lower case conversion, i.e., an operation that preserves lower case characters and converts upper case characters to the equivalent lower case. toCaseFold isn't that operation. It is a much more complex and subtle operation that, as well as converting upper case characters to lower case, sometimes converts lower case characters to different lower case characters (or strings of them). It also requires a fairly good understanding of Unicode (not just a relevant script) and historical Unicode decisions to predict its behavior and to have any hope of explaining that behavior to users. If one is trying to compare (as distinct from converting), then toCaseFold may be exactly what it wanted. but it is really hard to explain or justify that in terms of "nicknames" or "aliases", which are about conversion. And, if one hopes to explain what is going on to users in terms of "lower casing", then toCaseFold is just the wrong operation. That is what toLowerCase is for and the two operations are just not equivalent. FWIW and purely by coincidence wrt PRECIS and this document, I had a conversation a few days ago with an expert on Arabic (and Persian) calligraphy and writing systems (and good general knowledge of writing systems) who is quite insistent that any procedure we use for case-insensitive matching (e.g., case folding) is discriminatory, inconsistent, and just badly-thought-out if that same procedure doesn't treat isolated, initial, and medial forms of the same character as equivalent. He further strengthens his case (sic) by noting that Unicode case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, unambiguously a lower-case character) to U+03C3 (GREEK SMALL LETTER SIGMA), a relationship that depends entirely on positional use and not case. He also believes the same relationships should apply to all other scripts that make form distinctions for some characters based on positions in a string and for which Unicode has chosen to assign different code points. Even if there were wide acceptance of his view, Unicode stability principles would prevent changing toCaseFold (or CaseFolding.txt), but this is more evidence that what toCaseFold does and does not do is going to be hard to explain to either casual users or to writing system experts whose primary experience is not with the Greek-Latin-Cyrillic group. =20 I don't think we want to say "these matching rules are somewhat arbitrary and irrational, but, if you don't like it, blame Unicode and not us", if only because it is our choice to use those matching rules. More below. >... >> (2) Second, probably as a result of having IDNA in the lead, >> we've gotten sloppy about language and operations and should >> probably start untangling that before it gets people in >> trouble. >=20 > Where is the right place to do that untangling? (I doubt that > it is the precis-nickname document.) I agree that precis-nickname isn't the ideal place. I also believe that you and it are the innocent victims of the situation. At the same time, I don't believe IETF should be producing incomplete, ambiguous, erroneous, or misleading standards because no one could get around to doing the right foundational work. =20 >> The Unicode Standard, at least as I understand it, is fairly >> clear that the most important (and really only safe) use of >> toCaseFold is as part of a comparison operation. >=20 > Thanks for noting that. For example, Section 5.18 of Unicode > 8.0.0 says: >=20 > Caseless matching is implemented using case folding, which > is the > process of mapping characters of different case to a > single form, so > that case differences in strings are erased. Case folding > allows for > fast caseless matches in lookups because only binary > comparison is > required. It is more than just conversion to lowercase. Right. But, again, when its use is appropriate (a very controversial topic in itself with our painful IDNA history with Final Sigma, Eszett and the case-independent versus position-independent controversy called out above as examples) that is "matches in lookups" (what I've described elsewhere as "comparison only"). Not creating or defining nicknames or aliases. And that _is_ a problem for this document. >> Using your >> example it is entirely reasonable to treat, "stpeter" and >> "StPeter" as equivalent in a comparison operation, but >> accepting one string and changing it to the other for display >> may not be a really good idea. While that transformation may >> be acceptable (although I would be surprised if there were no >> people who share your surname who could consider "stpeter" or >> "Stpeter" unacceptable and might even believe that "StPeter" >> is an unacceptable substitute for "St. Peter"), >=20 > I do receive email at stpeter@gmail.com intended for > st.peter@gmail.com but that's a separate topic... One that is relevant because it "works" as a side-effect of a decision Google has made about mailbox name equivalence, a decision that, IMO, will sooner or later get someone into a lot of trouble and, more important, a decision and matching rule that PRECIS, AFAICT, does not allow and that IDNA unambigiously forbids. >> it also points out the >> dangers of using Basic Latin script examples to illustrate >> situations in which even more extended Latin script, much = less >> other scripts, may raise more complex issues. Because IDNA >> is essentially a workaround because changing the DNS >> comparison rules was impractical for several reasons, we >> ended up using toCaweFold to map characters and strings into >> others in IDNA2003 but PRECIS implementations that do not >> have the same constraints would, in general, be better off >> confining the use of toCaseFold, or even toLowerCase, to >> comparison operations. >=20 > Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does > it make sense for this nickname specification to differ in > this respect from the published RFCs? Shall we file errata > against those documents? (This might apply only to RFC 7613, > which says to apply case folding as part of the enforcement > process - when exactly to apply case folding is not stipulated > by RFC 7564.) To the extent to which this is a "botched that because the WG didn't understand the issues well enough" conclusion, it would be entirely reasonable to generate an updating RFC that repairs 7613 and/or 7564, even doing so in an addendum to precis-nickname if that is the only way to do that expeditiously. Per the above, we really don't want to give library routine writers bad instructions. As I understand it, the current position of the RFC Editor and IESG is that technical specification errors discovered in retrospect or after people start using a spec are not appropriate topics for errata. If the WG is not willing to do any of those things, then I suggest that precis-nickname at least needs to contain a very clear warning notice about this situation (see my response to your question 1 below). >> (3) Because toCaweFold loses information when used for more >> than comparison (for comparison, it merely contributes to >> what some people would consider false positives for matching) >> involves some controversial decisions and, because of >> stability requirements, cannot be changed even if the >> controversies are resolved in other ways, we end up with, >> e.g., >> toCaseFold ("Nu=C3=9F") -> "nuss" >> which is considered an acceptable transformation in some >> places that identify themselves as speaking/using German and >> two different unacceptable errors in others. Again, this = will >> almost always be much more serious if the transformation is >> used to map and replace strings than if it is used to compare >> (fwiw, that particular example is part of a continuing >> disagreement between IDNA2008 and, among others, German >> domain registry authorities on one side and UTC and UTR 46 on >> the other). >=20 > Agreed. See "warning notice" comment above and question 1 response = below. > (4) If the motivation is really to avoid confusion, the >> correct confusion-blocking rule for Latin script (but not >> others) and many languages that use it (but certainly not >> all) involves moving beyond toCaseFold and treating all >> "decorated" characters (characters normally represented by >> glyphs consisting of a Basic Latin character and one or more >> diacritical or equivalent markings) compare equal to their >> base characters, e.g., "=C3=A1" not only matches "=C3=81" but = also >> "a" and "A" and, as an unfortunate side-effect, maybe = "=C3=80" >> and "=C3=A0" as well. This is bad news for languages in = which >> decorated Latin characters are used to represent phonetically >> and conceptually different characters, not just pronunciation >> variations. I am not qualified to evaluate "how bad". In >> addition, extrapolations from this principle about Latin >> script to unrelated scripts will almost certainly lead to >> serious errors and/or additional confusion. >=20 > I would not be comfortable going that far... In case it isn't clear, I would not be either. But it is where getting sloppy about this stuff could easily take us. It is worth noting that it also identifies one of the difficulties with doing a global system to be applied to many types of applications (like the PRECIS work) and then applying it in user interface software that end users will expect to be localized to their assumptions because it has been mapped or translated into their language (if one normally speaks Upper Slobbovian but has some familiarity with English, an application interface in English will probably be expected to be "foreign", odd, and maybe even inconsistent with whatever expectations exist. But, if the interface is in Upper Slobbovian, the natural and reasonable assumption will be the matching should conform to normal Upper Slobbovian conventions. FWIW, a matching rule that says: (i) Two instances of a base character with the same diacritical mark(s) match. (ii) Two instances of a base character with different diacritical mark(s) do not match. (iii) Two instances of a base character, one with diacritical mark(s) and the other without any decoration match. Is precisely correct and normal behavior for at least one language that uses Latin script. It is also the normal practice for at least one Latin script transcription system that is used by a large fraction of a billion people (maybe more). >> More on this and Tom's question below... >>=20 >>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>> Peter, Alexey, >>>>=20 >>>> I think there is an ambiguity in the specification of case >>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>> ... >>>> But there are 55 code points in Unicode 7.0.0 that change >>>> under default case folding that are neither uppercase nor >>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>> suspect this stems from a confusion between Unicode case >>>> mapping and case folding. In the context of the above, a different way to say the same thing is that people are looking at toCaseFold and assuming (and explaining things in terms of) toLowerCase. toCaseFold works the way it is expected to and those 55 code points are, more or less, collateral damage to get to a matching algorithm that favors false positives over false negatives and various edge cases (including in "edge cases" languages spoken by, and script variations used by, millions of people). >... > After all that, I have 3 questions: Personal opinions about answers... > (1) Is my proposed text enough of a clarification that we > should make that change before the nickname I-D is published > as an RFC? I think the clarification is an improvement and is important enough to incorporate (I know that is the answer to a slightly different question). However, I think it is inadequate without a serious warning about the situation. That warning could appear in either this document or RFC 7613 (or 7613bis) with a pointer from the other, but, unless you want to revise 7613 now, this one is handy. Comment about possible text below. > (2) Should we modify draft-ietf-precis-nickname so that case > folding is applied only as part of comparison and not as part > of enforcement? If so, should we make that change before this > document is published as an RFC? Yes. If something is used for "enforcement", it should be lower casing or something else that can be explained to people who are ordinarily familiar with one or more of the scripts that make case distinctions. However, viewed in the light of this discussion, the whole "enforcement" concept becomes a little dicey, especially if, as I believe but don't have time to verify, the transformations performed by toLowerCase are not a proper subset of those performed by toCaseFold. > (3) Should we update RFC 7613 so that case folding is applied > only as part of comparison and not as part of enforcement? I think that is necessary. Following up on the comment above, I would prefer that the current Section 3.2.2 (3) of RFC 7613 either point to Unicode Lower Casing or contain a warning along the lines of that below. ---------- --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre wrote: > This issue has greater urgency now because > draft-ietf-precis-nickname is now in AUTH48... >=20 > On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >=20 >> After all that, I have 3 questions: >>=20 >> (1) Is my proposed text enough of a clarification that we >> should make that change before the nickname I-D is published >> as an RFC? >=20 > I think so. See above. >> (2) Should we modify draft-ietf-precis-nickname so that case >> folding is applied only as part of comparison and not as part >> of enforcement? If so, should we make that change before this >> document is published as an RFC? >=20 > Although it seems to be the case that Unicode case folding is > primarily designed for the purpose of matching (i.e., > comparison), "Seems" is a little weak. The Unicode Standard is really quite specific about that. > I have a concern that applying the PRECIS case > mapping rule after applying the normalization and > directionality rules might have unintended consequences that > we haven't had a chance to consider yet. The PRECIS framework > expresses a preference (actually a hard requirement) for > applying the rules in a particular order. We made a late > change to the username profiles (RFC 7613), such that width > mapping is applied first (in order to accommodate fullwidth > and halfwidth characters in certain East Asian scripts). > Making a late change to the nickname profile also concerns me, > even though both of these late changes seem reasonable on the > face of it. I will try to find time to think about this > further in the next 24 hours. First, a hint for the consideration process: there is a reason why Unicode now supports a unified case folding and normalization operation. My recollection is that it is not only more efficient to perform both operations at once (rather than looking in one table and then the other), but that there are some order-dependent or priority-dependent cases. The very fact that this issue exists (and is coming up again) this late in the process (7613 published in August, WG winding down and not, e.g., meeting next week) calls at least the PRECIS quality of review and some fairly fundamental model issues into question. I first raised that issue a rather long time ago but have continued to hope that we have an approximation to "good enough" without going back and rethinking everything. =20 The right solution, IMO, is that, if RFC 7613 is to rationalize or explain the operation in terms of converting upper case characters to lower case, then it should be using toLowerCase because that is what the operation does. After a quick look at 7613, amending/updating it to simply convert to lower case would be straightforward (and would not raise the ordering issue called out above). It would presumably require another IETF Last Call, however and I'd hope we would see some serious discussion within the WG (and with UTC) before making the change and about how it is explained. If we are not willing to make a change that significant and/or if we conclude that the WG (and perhaps the IETF) have completely run out of energy for dealing with i18n issues [1], then I suggest that we introduce some additional text. I've just spent a half-hour trying to find the AUTH48 copy of precis-nickname (aka RFC-to-be-7700), but the RFC Editor has apparently changed naming conventions and the various queue entry pages all point to the -19 I-D and not the current working copy so I can't try to match text and insertion point to what is there already. The suggestion is a patch (and a hack), not a good fix but something like it is probably the least drastic measure that would yield something that doesn't contain unexplained known defects. Rough version of suggested text (possibly to go after your revised paragraph and following up my comments in my 1 October note). Some of the terminology needs checking which I can do if you want to go this route: 'Users of this specification should note that the concept of "lower case conversion" is somewhat elusive and more dependent on the conventions of different languages and notation systems that use the same script than may appear obvious at first glance, especially if that glance is at Basic Latin characters (i.e., the ASCII letter repertoire). Unicode provides two different mapping procedures that produce lower-case characters, but they have different effects and results for many characters. The more conservative one, typically appropriately applicable when lower case forms are needed, is actual lower-casing (embodied in the Unicode operation toLowerCase). A more radical operation, normally suitable only for string matching in situations in which it is better to consider uncertain cases as matching than to treat them as distinct, is called "Case Folding" (Unicode operation toCaseFold). While the two operations will often produce the same results, Case Folding maps some lower case characters into others and performs other transformations that may be intuitively reasonable and expected for some users and quite astonishing (or just wrong) to others. There may be no practical alternative, especially if the operations are to be used for mapping or enforcement, to developers of PRECIS-dependent understanding that the cases in which the two yield different results require careful understanding of the relevant user base and its needs [2].' >> (3) Should we update RFC 7613 so that case folding is applied >> only as part of comparison and not as part of enforcement? >=20 > That is less urgent so I suggest that we address the nickname > spec first. Unless you (or someone else here) have a plausible plan to continue and revitalize the WG and assign it that revision work (and bring everyone actively participating up to the level needed to easily understand this discussion thread and feel embarrassed for not spotting the problems), I think we need to assume that this is our last shot. Absent an active and committed WG, "do this first" could easily be equivalent to "don't get around to the other, ever". I think that the particular set of issues that started this thread as a known defect in the PRECIS specs, both nickname and 7613 and that we are obligated to either fix the problems or at least explain them. The above warning text is an attempt to explain and identify the problems even if it does not actually provide a solution. If it were published as part of precis-nickname, it could include a statement to the effect that it should also be treated as an update to 7613 or, if the IESG and RFC Editor would agree in advance to accept, rather than bury, the thing, I suppose we could publish it in precis-nickname and create an erratum to 7613 indicating that it should have included some form of that statement. Neither option implies a huge amount of work to update 7613. But I think that making the changes of (2) without doing anything about (3) makes the two documents inconsistent with each other and that would be an additional known defect. Procedural question: given that precis-nickname is in AUTH48 as of yesterday and I don't see anything blocking publication next week if you and Barry sign off on the revised text that the WG hasn't seen, does someone need to file a pro forma objection/ appeal to block that until this is sorted out and the WG has a chance to review proposed publication text? best, john [1] I believe our collective inability to deal with the within-script character forms that do not normalize to each other because of language-dependent or other usage factors can be taken as evidence of having run out of energy, but it is probably in the interest of finishing the PRECIS work to try to treat that as a separate issue. [2] Not unlike the reason to differentiate between NFC and NFKC and understand the effects of each. From nobody Tue Oct 27 19:10:01 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6E02C1B32F6 for ; Tue, 27 Oct 2015 19:09:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.601 X-Spam-Level: X-Spam-Status: No, score=-1.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, GB_I_LETTER=-2, MANGLED_LIST=2.3, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FIAV0hYQtD4v for ; Tue, 27 Oct 2015 19:09:54 -0700 (PDT) Received: from mail-ob0-x22b.google.com (mail-ob0-x22b.google.com [IPv6:2607:f8b0:4003:c01::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 77E291B32F3 for ; Tue, 27 Oct 2015 19:09:53 -0700 (PDT) Received: by obcqt19 with SMTP id qt19so186603221obc.3 for ; Tue, 27 Oct 2015 19:09:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=mAO/CQMq6aR0df0UClkBIytL2PcHKEmKXyNVAg9xdMM=; b=a9RWY/zWs88w32idqaRbIPHJCTA18X4g1g3jJx9phBkFMC7oGulA4waqjUtRggypPI 02oKwONAQqWT6nuzehntC4sZC4Bh21mXZisvAtkbmP98YqBS9/A8fI9S85lP24XMMhUr gRpv2vXyskzHhELjVdC+f6uGcZENNbMuv76JAsP5r2XT6b7PaBcoefT2otNdUIEIs4Fw wJg+gJhoK+8ZeQQu0vuGj1Gwx0/NaiwQ2km0/gUfN9FpoLX4I9ekFoL0RwuWrwDvE+Xw E53+UvbnaktSIZPpgW1XODfgFhuWPh1B8pffEsHfW1MH6jhQwz/bifdXXVyVKazl8JFL LGQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=mAO/CQMq6aR0df0UClkBIytL2PcHKEmKXyNVAg9xdMM=; b=GXmzOG5gWn73L3up3MjRn0kRRTh4ALtEGCvsdRwlO3yfwa8nlMNzNv5Cj1Q57F3z/a gIFW2nidkDcIYzFMiQf0PwFMIbovFbRzQAQDWriuIxfLHi3KO1Hizw2Hq+vwC71+vNyc lOyQLfncuOJEE3KtcMhj4xhR8dRPqWTGjHGNydx+nbG0GNhGBrw2LgqQ79q2rrR2DoHM hX7vHbN9qrDbYI6gxP692r25E1M3a9tXUYZxlUPdj5Iw/200xTdniODrsvhvZrt6uI0c 4KR9s3bkafHYvTJOJc/HHZ0GswxjqaTudJ0VSnkjC3rCUvxmMuz1VWxt56el54WerjDv nh8g== X-Gm-Message-State: ALoCoQnuTZ5OZk1RAmdrgjeEwtOt7AZiLR0XgphuRPSePwv6OpN/yIZ/L+8OkmXNCCbIanOGfFRM X-Received: by 10.182.153.101 with SMTP id vf5mr5167552obb.81.1445998193065; Tue, 27 Oct 2015 19:09:53 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id o8sm18864883obi.6.2015.10.27.19.09.51 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Oct 2015 19:09:51 -0700 (PDT) To: John C Klensin , Tom Worster , Alexey Melnikov References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> From: Peter Saint-Andre Message-ID: <56302E6D.5030901@andyet.net> Date: Tue, 27 Oct 2015 20:09:49 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 02:09:59 -0000 On 10/27/15 11:32 AM, John C Klensin wrote: > Response to Monday's note immediately below; response to today's > follows it. My apologies, but it is probably important to read > both. My further apologies for the length of this note, but I > think we are in deep trouble here, Internationalization always seems to be a matter of how deep the trouble is... > trouble that is aggravated by > precis-mappings and precis-nickname both being post-approval and > that, as far as I know, there are no future plans for PRECIS > work (having precis-nickname in AUTH48 just emphasizes that -- > see comment at end). We had not planned to work on PRECIS because we thought we were done for awhile. If that's not the case and we need to fix things, then so be it. Whether there is sufficient and continued energy for such work is another question. Personally I don't want us to have broken RFCs out there. > --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - > &yet wrote: > >> My apologies for the delayed reply. Comments inline. > > A few remarks below... I can't tell whether we disagree or > whether at least one of us, probably me, are not being > adequately clear. (Material on which we fairly clearly agree > elided.) > > >> On 10/1/15 7:50 AM, John C Klensin wrote: >>> --On Wednesday, September 30, 2015 15:16 -0600 Peter >>> Saint-Andre - &yet wrote: >> ... >>> Peter, >>> >>> While your proposed text is an improvement, >> >> Happy to hear it. All I intended was a slight clarification. > > But I'm not certain we are there yet... Agreed. The text I proposed addressed only a very small part of the problem. >>> the desire of many >>> people for a magic "just tell me what to do" formula, one that >>> lets them avoid understanding the issues, may call for a >>> little more: >> >> There is always a need for more when it comes to i18n. > > But I think it is a little more that that. I've heard several > times, including in PRECIS meetings, requests for "just tell me > what to do and make sure it isn't complicated" (or "I don't want > to have to think about, much less understand, the issues"). We > can debate whether giving in to those requests in the I18n case > is wise. I think it leads directly to conclusions equivalent to > "I understand my own script and writing system (or think I do) > and therefore, since all writing systems must be pretty much the > same, I understand all of the core issues in terms of my script > and understanding". That, in turn, leads directly to the "how > do you spell 'Zürich'?" and "all spellings of 'Zuerich' should > be treated as equivalent" discussion that sounded like they > dominated a BOF at IETF 93. > > Now I actually think it is reasonable for someone to ask for a > library that will do the job most of the time and that will > almost never cause their users or customers to get angry at > them. But, if we are going to call what we do "standards", they > should contain sufficient information that would-be library > authors can know what to do ... or understand that they are in > over their heads. And, for these particular cases, we may need > to explain, or help the library authors explain, why some cases > will fail and, indeed, get users mad at vendors. > > >>> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >>> primary result of doing so is that uppercase characters are >>> mapped to lowercase characters" is true for toCaseFold, >> >> By "primary" I meant two things: (1) lowercasing is what >> happens to the preponderance of code points and (2) this is >> the result that most people care about. > > If I parse the above correctly, I think you are wrong. I think > what most people want, care about, and think they are getting, > is lower case conversion, i.e., an operation that preserves > lower case characters and converts upper case characters to the > equivalent lower case. toCaseFold isn't that operation. It is > a much more complex and subtle operation that, as well as > converting upper case characters to lower case, sometimes > converts lower case characters to different lower case > characters (or strings of them). It also requires a fairly good > understanding of Unicode (not just a relevant script) and > historical Unicode decisions to predict its behavior and to have > any hope of explaining that behavior to users. If one is > trying to compare (as distinct from converting), then toCaseFold > may be exactly what it wanted. but it is really hard to explain > or justify that in terms of "nicknames" or "aliases", which are > about conversion. And, if one hopes to explain what is going > on to users in terms of "lower casing", then toCaseFold is just > the wrong operation. That is what toLowerCase is for and the > two operations are just not equivalent. My recollection, quite possibly inaccurate or incomplete, from at least one and I think several in-person meetings of the PRECIS WG was: just use Unicode Default Case Folding because if you use anything else or try to roll your own you will be fubar forever. I do not recall any discussion of the issues you have raised in this thread (e.g., about the inadvisability of using case folding for anything but comparison operations) until the last few weeks. However, I freely admit that's probably because, through my own faults and ignorance, I didn't understand what you were saying. > FWIW and purely by coincidence wrt PRECIS and this document, I > had a conversation a few days ago with an expert on Arabic (and > Persian) calligraphy and writing systems (and good general > knowledge of writing systems) who is quite insistent that any > procedure we use for case-insensitive matching (e.g., case > folding) is discriminatory, inconsistent, and just > badly-thought-out if that same procedure doesn't treat isolated, > initial, and medial forms of the same character as equivalent. > He further strengthens his case (sic) by noting that Unicode > case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, > unambiguously a lower-case character) to U+03C3 (GREEK SMALL > LETTER SIGMA), a relationship that depends entirely on > positional use and not case. He also believes the same > relationships should apply to all other scripts that make form > distinctions for some characters based on positions in a string > and for which Unicode has chosen to assign different code > points. Even if there were wide acceptance of his view, Unicode > stability principles would prevent changing toCaseFold (or > CaseFolding.txt), but this is more evidence that what toCaseFold > does and does not do is going to be hard to explain to either > casual users or to writing system experts whose primary > experience is not with the Greek-Latin-Cyrillic group. > > I don't think we want to say "these matching rules are somewhat > arbitrary and irrational, but, if you don't like it, blame > Unicode and not us", if only because it is our choice to use > those matching rules. More below. > > >> ... >>> (2) Second, probably as a result of having IDNA in the lead, >>> we've gotten sloppy about language and operations and should >>> probably start untangling that before it gets people in >>> trouble. >> >> Where is the right place to do that untangling? (I doubt that >> it is the precis-nickname document.) > > I agree that precis-nickname isn't the ideal place. I also > believe that you and it are the innocent victims of the > situation. At the same time, I don't believe IETF should be > producing incomplete, ambiguous, erroneous, or misleading > standards because no one could get around to doing the right > foundational work. Agreed. I too want to get this right, even though it's not a lot of fun and it's certainly more work than I thought I was signing up for at the NEWPREP BoF years ago. >>> The Unicode Standard, at least as I understand it, is fairly >>> clear that the most important (and really only safe) use of >>> toCaseFold is as part of a comparison operation. >> >> Thanks for noting that. For example, Section 5.18 of Unicode >> 8.0.0 says: >> >> Caseless matching is implemented using case folding, which >> is the >> process of mapping characters of different case to a >> single form, so >> that case differences in strings are erased. Case folding >> allows for >> fast caseless matches in lookups because only binary >> comparison is >> required. It is more than just conversion to lowercase. > > Right. But, again, when its use is appropriate (a very > controversial topic in itself with our painful IDNA history with > Final Sigma, Eszett and the case-independent versus > position-independent controversy called out above as examples) > that is "matches in lookups" (what I've described elsewhere as > "comparison only"). Not creating or defining nicknames or > aliases. And that _is_ a problem for this document. I'm not convinced that things are as bad as you think. If we say in draft-ietf-precis-nickname that the case mapping rule is to be applied only as part of comparison and not as part of enforcement - which I think is really what we care about (e.g., to prevent spoofing of users in chat rooms) - then I think we might be most of the way there. >>> Using your >>> example it is entirely reasonable to treat, "stpeter" and >>> "StPeter" as equivalent in a comparison operation, but >>> accepting one string and changing it to the other for display >>> may not be a really good idea. While that transformation may >>> be acceptable (although I would be surprised if there were no >>> people who share your surname who could consider "stpeter" or >>> "Stpeter" unacceptable and might even believe that "StPeter" >>> is an unacceptable substitute for "St. Peter"), >> >> I do receive email at stpeter@gmail.com intended for >> st.peter@gmail.com but that's a separate topic... > > One that is relevant because it "works" as a side-effect of a > decision Google has made about mailbox name equivalence, a > decision that, IMO, will sooner or later get someone into a lot > of trouble and, more important, a decision and matching rule > that PRECIS, AFAICT, does not allow and that IDNA unambigiously > forbids. > >>> it also points out the >>> dangers of using Basic Latin script examples to illustrate >>> situations in which even more extended Latin script, much less >>> other scripts, may raise more complex issues. Because IDNA >>> is essentially a workaround because changing the DNS >>> comparison rules was impractical for several reasons, we >>> ended up using toCaweFold to map characters and strings into >>> others in IDNA2003 but PRECIS implementations that do not >>> have the same constraints would, in general, be better off >>> confining the use of toCaseFold, or even toLowerCase, to >>> comparison operations. >> >> Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does >> it make sense for this nickname specification to differ in >> this respect from the published RFCs? Shall we file errata >> against those documents? (This might apply only to RFC 7613, >> which says to apply case folding as part of the enforcement >> process - when exactly to apply case folding is not stipulated >> by RFC 7564.) > > To the extent to which this is a "botched that because the WG > didn't understand the issues well enough" conclusion, it would > be entirely reasonable to generate an updating RFC that repairs > 7613 and/or 7564, even doing so in an addendum to > precis-nickname if that is the only way to do that > expeditiously. Per the above, we really don't want to give > library routine writers bad instructions. As I understand it, > the current position of the RFC Editor and IESG is that > technical specification errors discovered in retrospect or after > people start using a spec are not appropriate topics for errata. > If the WG is not willing to do any of those things, then I > suggest that precis-nickname at least needs to contain a very > clear warning notice about this situation (see my response to > your question 1 below). I think we'll probably need to fix 7613 and 7564. I am hoping we can fix nickname now so that it is less incorrect than the other two. That doesn't necessarily mean we won't need to also further fix nickname later on. Granted, we were supposed to avoid this problem by working on all of the PRECIS specs simultaneously. Clearly we have not avoided the problem, so we need to solve it one way or another. If that means bis for them all, we need to deal with it. >>> (3) Because toCaweFold loses information when used for more >>> than comparison (for comparison, it merely contributes to >>> what some people would consider false positives for matching) >>> involves some controversial decisions and, because of >>> stability requirements, cannot be changed even if the >>> controversies are resolved in other ways, we end up with, >>> e.g., >>> toCaseFold ("Nuß") -> "nuss" >>> which is considered an acceptable transformation in some >>> places that identify themselves as speaking/using German and >>> two different unacceptable errors in others. Again, this will >>> almost always be much more serious if the transformation is >>> used to map and replace strings than if it is used to compare >>> (fwiw, that particular example is part of a continuing >>> disagreement between IDNA2008 and, among others, German >>> domain registry authorities on one side and UTC and UTR 46 on >>> the other). >> >> Agreed. > > See "warning notice" comment above and question 1 response below. > >> (4) If the motivation is really to avoid confusion, the >>> correct confusion-blocking rule for Latin script (but not >>> others) and many languages that use it (but certainly not >>> all) involves moving beyond toCaseFold and treating all >>> "decorated" characters (characters normally represented by >>> glyphs consisting of a Basic Latin character and one or more >>> diacritical or equivalent markings) compare equal to their >>> base characters, e.g., "á" not only matches "Á" but also >>> "a" and "A" and, as an unfortunate side-effect, maybe "À" >>> and "à" as well. This is bad news for languages in which >>> decorated Latin characters are used to represent phonetically >>> and conceptually different characters, not just pronunciation >>> variations. I am not qualified to evaluate "how bad". In >>> addition, extrapolations from this principle about Latin >>> script to unrelated scripts will almost certainly lead to >>> serious errors and/or additional confusion. >> >> I would not be comfortable going that far... > > In case it isn't clear, I would not be either. But it is where > getting sloppy about this stuff could easily take us. It is > worth noting that it also identifies one of the difficulties > with doing a global system to be applied to many types of > applications (like the PRECIS work) and then applying it in user > interface software that end users will expect to be localized to > their assumptions because it has been mapped or translated into > their language (if one normally speaks Upper Slobbovian but has > some familiarity with English, an application interface in > English will probably be expected to be "foreign", odd, and > maybe even inconsistent with whatever expectations exist. But, > if the interface is in Upper Slobbovian, the natural and > reasonable assumption will be the matching should conform to > normal Upper Slobbovian conventions. FWIW, a matching rule > that says: > > (i) Two instances of a base character with the same > diacritical mark(s) match. > (ii) Two instances of a base character with different > diacritical mark(s) do not match. > (iii) Two instances of a base character, one with > diacritical mark(s) and the other without any decoration > match. > > Is precisely correct and normal behavior for at least one > language that uses Latin script. It is also the normal practice > for at least one Latin script transcription system that is used > by a large fraction of a billion people (maybe more). That is indeed sobering. >>> More on this and Tom's question below... >>> >>>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>>> Peter, Alexey, >>>>> >>>>> I think there is an ambiguity in the specification of case >>>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>>> ... >>>>> But there are 55 code points in Unicode 7.0.0 that change >>>>> under default case folding that are neither uppercase nor >>>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>>> suspect this stems from a confusion between Unicode case >>>>> mapping and case folding. > > In the context of the above, a different way to say the same > thing is that people are looking at toCaseFold and assuming (and > explaining things in terms of) toLowerCase. toCaseFold works > the way it is expected to and those 55 code points are, more or > less, collateral damage to get to a matching algorithm that > favors false positives over false negatives and various edge > cases (including in "edge cases" languages spoken by, and script > variations used by, millions of people). Sadly I suspect that is an accurate description of the current state of affairs (modulo my comment above about PRECIS WG discussions at one or more IETF meetings). >> ... >> After all that, I have 3 questions: > > Personal opinions about answers... > >> (1) Is my proposed text enough of a clarification that we >> should make that change before the nickname I-D is published >> as an RFC? > > I think the clarification is an improvement and is important > enough to incorporate (I know that is the answer to a slightly > different question). > > However, I think it is inadequate without a serious warning > about the situation. Yes. > That warning could appear in either this > document or RFC 7613 (or 7613bis) with a pointer from the other, > but, unless you want to revise 7613 now, this one is handy. I suspect that we need to revise 7613. I suspect that we might also need to revise 7564 (at least with respect to the order in which operations are applied, since there has been some confusion among implementers). Well, we always knew that we would need to revise them. Just not so soon. > Comment about possible text below. > >> (2) Should we modify draft-ietf-precis-nickname so that case >> folding is applied only as part of comparison and not as part >> of enforcement? If so, should we make that change before this >> document is published as an RFC? > > Yes. If something is used for "enforcement", it should be lower > casing or something else that can be explained to people who are > ordinarily familiar with one or more of the scripts that make > case distinctions. > > However, viewed in the light of this discussion, the whole > "enforcement" concept becomes a little dicey, especially if, as > I believe but don't have time to verify, the transformations > performed by toLowerCase are not a proper subset of those > performed by toCaseFold. My initial thought is that case mapping doesn't belong in the nickname enforcement operation at all - only in the comparison operation. >> (3) Should we update RFC 7613 so that case folding is applied >> only as part of comparison and not as part of enforcement? > > I think that is necessary. Following up on the comment above, I > would prefer that the current Section 3.2.2 (3) of RFC 7613 > either point to Unicode Lower Casing or contain a warning along > the lines of that below. Unlike the nickname profile (which I think can be cleaned up by moving the case mapping rule to the comparison operation and continuing to use Unicode Default Case Folding), I think you are right that for the UsernameCaseMapped profile we probably want Unicode Lower Casing. Thus the likely need, sooner rather than later, for 7613bis. > > ---------- > > --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre > wrote: > >> This issue has greater urgency now because >> draft-ietf-precis-nickname is now in AUTH48... >> >> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >> >>> After all that, I have 3 questions: >>> >>> (1) Is my proposed text enough of a clarification that we >>> should make that change before the nickname I-D is published >>> as an RFC? >> >> I think so. > > See above. > >>> (2) Should we modify draft-ietf-precis-nickname so that case >>> folding is applied only as part of comparison and not as part >>> of enforcement? If so, should we make that change before this >>> document is published as an RFC? >> >> Although it seems to be the case that Unicode case folding is >> primarily designed for the purpose of matching (i.e., >> comparison), > > "Seems" is a little weak. The Unicode Standard is really quite > specific about that. > >> I have a concern that applying the PRECIS case >> mapping rule after applying the normalization and >> directionality rules might have unintended consequences that >> we haven't had a chance to consider yet. The PRECIS framework >> expresses a preference (actually a hard requirement) for >> applying the rules in a particular order. We made a late >> change to the username profiles (RFC 7613), such that width >> mapping is applied first (in order to accommodate fullwidth >> and halfwidth characters in certain East Asian scripts). >> Making a late change to the nickname profile also concerns me, >> even though both of these late changes seem reasonable on the >> face of it. I will try to find time to think about this >> further in the next 24 hours. > > First, a hint for the consideration process: there is a reason > why Unicode now supports a unified case folding and > normalization operation. My recollection is that it is not only > more efficient to perform both operations at once (rather than > looking in one table and then the other), but that there are > some order-dependent or priority-dependent cases. > > The very fact that this issue exists (and is coming up again) > this late in the process (7613 published in August, WG winding > down and not, e.g., meeting next week) calls at least the PRECIS > quality of review and some fairly fundamental model issues into > question. I first raised that issue a rather long time ago but > have continued to hope that we have an approximation to "good > enough" without going back and rethinking everything. > > The right solution, IMO, is that, if RFC 7613 is to rationalize > or explain the operation in terms of converting upper case > characters to lower case, then it should be using toLowerCase > because that is what the operation does. After a quick look at > 7613, amending/updating it to simply convert to lower case would > be straightforward (and would not raise the ordering issue > called out above). It would presumably require another IETF > Last Call, however and I'd hope we would see some serious > discussion within the WG (and with UTC) before making the change > and about how it is explained. > > If we are not willing to make a change I'm willing. It would, as you note, require some careful thinking and review to make sure that we got it (more) right this time. > that significant and/or > if we conclude that the WG (and perhaps the IETF) have > completely run out of energy for dealing with i18n issues [1], > then I suggest that we introduce some additional text. I've > just spent a half-hour trying to find the AUTH48 copy of > precis-nickname (aka RFC-to-be-7700), but the RFC Editor has > apparently changed naming conventions and the various queue > entry pages all point to the -19 I-D and not the current working > copy so I can't try to match text and insertion point to what is > there already. http://www.rfc-editor.org/authors/rfc7700.txt > The suggestion is a patch (and a hack), not a > good fix but something like it is probably the least drastic > measure that would yield something that doesn't contain > unexplained known defects. > > Rough version of suggested text (possibly to go after your > revised paragraph and following up my comments in my 1 October > note). Some of the terminology needs checking which I can do if > you want to go this route: > > 'Users of this specification should note that the > concept of "lower case conversion" is somewhat elusive > and more dependent on the conventions of different > languages and notation systems that use the same script > than may appear obvious at first glance, especially if > that glance is at Basic Latin characters (i.e., the > ASCII letter repertoire). Unicode provides two > different mapping procedures that produce lower-case > characters, but they have different effects and results > for many characters. The more conservative one, > typically appropriately applicable when lower case forms > are needed, is actual lower-casing (embodied in the > Unicode operation toLowerCase). A more radical > operation, normally suitable only for string matching in > situations in which it is better to consider uncertain > cases as matching than to treat them as distinct, is > called "Case Folding" (Unicode operation toCaseFold). > While the two operations will often produce the same > results, Case Folding maps some lower case characters > into others and performs other transformations that may > be intuitively reasonable and expected for some users > and quite astonishing (or just wrong) to others. There > may be no practical alternative, especially if the > operations are to be used for mapping or enforcement, to > developers of PRECIS-dependent understanding that the > cases in which the two yield different results require > careful understanding of the relevant user base and its > needs [2].' Thanks. I am not sure if we need something like that if we move case mapping (here, case folding) to the comparison operation only - but something like that might still be appropriate. >>> (3) Should we update RFC 7613 so that case folding is applied >>> only as part of comparison and not as part of enforcement? >> >> That is less urgent so I suggest that we address the nickname >> spec first. > > Unless you (or someone else here) have a plausible plan to > continue and revitalize the WG and assign it that revision work > (and bring everyone actively participating up to the level > needed to easily understand this discussion thread and feel > embarrassed for not spotting the problems), I think we need to > assume that this is our last shot. Absent an active and > committed WG, "do this first" could easily be equivalent to > "don't get around to the other, ever". As mentioned, I don't want to have broken RFCs out there. > I think that the particular set of issues that started this > thread as a known defect in the PRECIS specs, both nickname and > 7613 and that we are obligated to either fix the problems or at > least explain them. The above warning text is an attempt to > explain and identify the problems even if it does not actually > provide a solution. If it were published as part of > precis-nickname, it could include a statement to the effect that > it should also be treated as an update to 7613 or, if the IESG > and RFC Editor would agree in advance to accept, rather than > bury, the thing, I suppose we could publish it in > precis-nickname and create an erratum to 7613 indicating that it > should have included some form of that statement. Neither > option implies a huge amount of work to update 7613. But I > think that making the changes of (2) without doing anything > about (3) makes the two documents inconsistent with each other > and that would be an additional known defect. > > Procedural question: given that precis-nickname is in AUTH48 as > of yesterday and I don't see anything blocking publication next > week if you and Barry sign off on the revised text that the WG > hasn't seen, There is no revised text yet. That's why we're having this discussion. > does someone need to file a pro forma objection/ > appeal to block that until this is sorted out and the WG has a > chance to review proposed publication text? I see no reason to invoke the specter of appeals quite yet. Everyone is working in good faith to do the right thing and get this mess cleaned up. > [1] I believe our collective inability to deal with the > within-script character forms that do not normalize to each > other because of language-dependent or other usage factors can > be taken as evidence of having run out of energy, Or in my case simple ignorance of some of the relevant issues and examples. It's not easy to know about all of this. > but it is > probably in the interest of finishing the PRECIS work to try to > treat that as a separate issue. Probably. > [2] Not unlike the reason to differentiate between NFC and NFKC > and understand the effects of each. Another thing that's not easy to grok in fulness. Peter From nobody Wed Oct 28 13:06:14 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BBD531ACE55 for ; Wed, 28 Oct 2015 13:06:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.601 X-Spam-Level: X-Spam-Status: No, score=-1.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, GB_I_LETTER=-2, MANGLED_LIST=2.3, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pUmh_0KGkDvx for ; Wed, 28 Oct 2015 13:06:08 -0700 (PDT) Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5DA871ACE17 for ; Wed, 28 Oct 2015 13:06:08 -0700 (PDT) Received: by obbza9 with SMTP id za9so16197421obb.1 for ; Wed, 28 Oct 2015 13:06:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=PeyWlkSyebBrtdIuW6hvHGvJqlCzEERN1eUu5IqRCa8=; b=ZEV/zuNfrfecjpmoUtTSF8/tcoa/Oya8MgdfpjqDTppiN/n9euKR9m8rYmC81sixzK ShWi+WUOPeDhf9WglKDQERD9PknRij+hlB0zZAO8PTQ97CbiNob4L5TFUyCy1YD9EG9I HrWBnX1kfbNMkoPN0lAok0pqLfNmj05y7cqkrSDe13/GkK61q8B9w4WWH4fvNuvCaf0r 3vTuaZddr5en3NKk2BmjUNXXIq5fYbj0ditjMD1HzVpk730sWEGC+kX1//gVRxNd1R5+ Sp2du69vQOSm+6xeQtsauROdB55mslg2MhDfXDgHcg+vom6JsIXmimWCqeVfXTfiP4RH ciEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=PeyWlkSyebBrtdIuW6hvHGvJqlCzEERN1eUu5IqRCa8=; b=CgK+Gr/bkiBBUyQ3TnZsWiWh+HLoyFvvniHOnYX+9ZXySBmvJF0YrjDIP2/IpQdUwv OdGD0iAERW4/1yK1tJ0OQsVTbaWA5IQeXSf2QBMqFfRRfbkhTLcaB2NFRag70lutqTJM RPxs24JIMa+OxTKAgK73wcf/dxBhScAAcp1qb2hR3Xknsl1iedBz+CUUdXN+bHzst6VB 9DCbZwuDh88/EtOzKzUjq7f7KJKPuc20B6pz3IuyUYWgvanle3bkNIGpr1ni+9WruhTV gPx2+2FbdEywXI5uMJ+H72O33caJJHeK+XEcP7b5DDn2M953sUVQZBtzqQJ8fakjnte2 UebA== X-Gm-Message-State: ALoCoQkPDVvYee5+ykuVs8JDreaukjPU4ErA4QL/xRAKiN+1iXxXg8ppHF8TLobml6v1ATHJ0eaK X-Received: by 10.60.173.42 with SMTP id bh10mr29749836oec.58.1446062767517; Wed, 28 Oct 2015 13:06:07 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id r124sm3718311oia.24.2015.10.28.13.06.05 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Oct 2015 13:06:06 -0700 (PDT) To: John C Klensin , Tom Worster , Alexey Melnikov References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> From: Peter Saint-Andre Message-ID: <56312AAC.1000300@andyet.net> Date: Wed, 28 Oct 2015 14:06:04 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <56302E6D.5030901@andyet.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 20:06:13 -0000 I propose the following text changes: ### OLD 3. Case Mapping Rule: Uppercase and titlecase characters MUST be mapped to their lowercase equivalents using Unicode Default Case Folding as defined in the Unicode Standard [Unicode] (at the time of this writing, the algorithm is specified in Chapter 3 of [Unicode7.0]). In applications that prohibit conflicting nicknames, this rule helps to reduce the possibility of confusion by ensuring that nicknames differing only by case (e.g., "stpeter" vs. "StPeter") would not be presented to a human user at the same time. NEW 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, as defined in the Unicode Standard [Unicode] (at the time of this writing, the algorithm is specified in Chapter 3 of [Unicode7.0]). The primary result of doing so is that uppercase characters are mapped to lowercase characters. In applications that prohibit conflicting nicknames, this rule helps to reduce the possibility of confusion by ensuring that nicknames differing only by case (e.g., "stpeter" vs. "StPeter") would not be presented to a human user at the same time. ### (The foregoing was previously sent to the list.) ### OLD 2.3. Enforcement An entity that performs enforcement according to this profile MUST prepare a string as described in Section 2.2 and MUST also apply the rules specified in Section 2.1. The rules MUST be applied in the order shown. After all of the foregoing rules have been enforced, the entity MUST ensure that the nickname is not zero bytes in length (this is done after enforcing the rules to prevent applications from mistakenly omitting a nickname entirely, because when internationalized characters are accepted, a non-empty sequence of characters can result in a zero-length nickname after canonicalization). 2.4. Comparison An entity that performs comparison of two strings according to this profile MUST prepare each string and enforce the rules as specified in Sections 2.2 and 2.3. The two strings are to be considered equivalent if they are an exact octet-for-octet match (sometimes called "bit-string identity"). NEW 2.3. Enforcement An entity that performs enforcement according to this profile MUST prepare a string as described in Section 2.2 and MUST also apply the following rules specified in Section 2.1 in the order shown: 1. Additional Mapping Rule 2. Normalization Rule 3. Directionality Rule After all of the foregoing rules have been enforced, the entity MUST ensure that the nickname is not zero bytes in length (this is done after enforcing the rules to prevent applications from mistakenly omitting a nickname entirely, because when internationalized characters are accepted, a non-empty sequence of characters can result in a zero-length nickname after canonicalization). 2.4. Comparison An entity that performs comparison of two strings according to this profile MUST prepare each string as specified in Section 2.2 and MUST apply the following rules specified in Section 2.1 in the order shown: 1. Additional Mapping Rule 2. Case Mapping Rule 3. Normalization Rule 4. Directionality Rule The two strings are to be considered equivalent if they are an exact octet-for-octet match (sometimes called "bit-string identity"). ### In addition, some variation on John's proposed text about toLowerCase vs. toCaseFold might be appropriate at the end of Section 4; however, I'm still not sure that is necessary if we move the case mapping rule to the comparison operation. Peter On 10/27/15 8:09 PM, Peter Saint-Andre wrote: > On 10/27/15 11:32 AM, John C Klensin wrote: >> Response to Monday's note immediately below; response to today's >> follows it. My apologies, but it is probably important to read >> both. My further apologies for the length of this note, but I >> think we are in deep trouble here, > > Internationalization always seems to be a matter of how deep the trouble > is... > >> trouble that is aggravated by >> precis-mappings and precis-nickname both being post-approval and >> that, as far as I know, there are no future plans for PRECIS >> work (having precis-nickname in AUTH48 just emphasizes that -- >> see comment at end). > > We had not planned to work on PRECIS because we thought we were done for > awhile. If that's not the case and we need to fix things, then so be it. > Whether there is sufficient and continued energy for such work is > another question. Personally I don't want us to have broken RFCs out there. > >> --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - >> &yet wrote: >> >>> My apologies for the delayed reply. Comments inline. >> >> A few remarks below... I can't tell whether we disagree or >> whether at least one of us, probably me, are not being >> adequately clear. (Material on which we fairly clearly agree >> elided.) >> >> >>> On 10/1/15 7:50 AM, John C Klensin wrote: >>>> --On Wednesday, September 30, 2015 15:16 -0600 Peter >>>> Saint-Andre - &yet wrote: >>> ... >>>> Peter, >>>> >>>> While your proposed text is an improvement, >>> >>> Happy to hear it. All I intended was a slight clarification. >> >> But I'm not certain we are there yet... > > Agreed. The text I proposed addressed only a very small part of the > problem. > >>>> the desire of many >>>> people for a magic "just tell me what to do" formula, one that >>>> lets them avoid understanding the issues, may call for a >>>> little more: >>> >>> There is always a need for more when it comes to i18n. >> >> But I think it is a little more that that. I've heard several >> times, including in PRECIS meetings, requests for "just tell me >> what to do and make sure it isn't complicated" (or "I don't want >> to have to think about, much less understand, the issues"). We >> can debate whether giving in to those requests in the I18n case >> is wise. I think it leads directly to conclusions equivalent to >> "I understand my own script and writing system (or think I do) >> and therefore, since all writing systems must be pretty much the >> same, I understand all of the core issues in terms of my script >> and understanding". That, in turn, leads directly to the "how >> do you spell 'Zürich'?" and "all spellings of 'Zuerich' should >> be treated as equivalent" discussion that sounded like they >> dominated a BOF at IETF 93. >> >> Now I actually think it is reasonable for someone to ask for a >> library that will do the job most of the time and that will >> almost never cause their users or customers to get angry at >> them. But, if we are going to call what we do "standards", they >> should contain sufficient information that would-be library >> authors can know what to do ... or understand that they are in >> over their heads. And, for these particular cases, we may need >> to explain, or help the library authors explain, why some cases >> will fail and, indeed, get users mad at vendors. >> >> >>>> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >>>> primary result of doing so is that uppercase characters are >>>> mapped to lowercase characters" is true for toCaseFold, >>> >>> By "primary" I meant two things: (1) lowercasing is what >>> happens to the preponderance of code points and (2) this is >>> the result that most people care about. >> >> If I parse the above correctly, I think you are wrong. I think >> what most people want, care about, and think they are getting, >> is lower case conversion, i.e., an operation that preserves >> lower case characters and converts upper case characters to the >> equivalent lower case. toCaseFold isn't that operation. It is >> a much more complex and subtle operation that, as well as >> converting upper case characters to lower case, sometimes >> converts lower case characters to different lower case >> characters (or strings of them). It also requires a fairly good >> understanding of Unicode (not just a relevant script) and >> historical Unicode decisions to predict its behavior and to have >> any hope of explaining that behavior to users. If one is >> trying to compare (as distinct from converting), then toCaseFold >> may be exactly what it wanted. but it is really hard to explain >> or justify that in terms of "nicknames" or "aliases", which are >> about conversion. And, if one hopes to explain what is going >> on to users in terms of "lower casing", then toCaseFold is just >> the wrong operation. That is what toLowerCase is for and the >> two operations are just not equivalent. > > My recollection, quite possibly inaccurate or incomplete, from at least > one and I think several in-person meetings of the PRECIS WG was: just > use Unicode Default Case Folding because if you use anything else or try > to roll your own you will be fubar forever. I do not recall any > discussion of the issues you have raised in this thread (e.g., about the > inadvisability of using case folding for anything but comparison > operations) until the last few weeks. However, I freely admit that's > probably because, through my own faults and ignorance, I didn't > understand what you were saying. > >> FWIW and purely by coincidence wrt PRECIS and this document, I >> had a conversation a few days ago with an expert on Arabic (and >> Persian) calligraphy and writing systems (and good general >> knowledge of writing systems) who is quite insistent that any >> procedure we use for case-insensitive matching (e.g., case >> folding) is discriminatory, inconsistent, and just >> badly-thought-out if that same procedure doesn't treat isolated, >> initial, and medial forms of the same character as equivalent. >> He further strengthens his case (sic) by noting that Unicode >> case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, >> unambiguously a lower-case character) to U+03C3 (GREEK SMALL >> LETTER SIGMA), a relationship that depends entirely on >> positional use and not case. He also believes the same >> relationships should apply to all other scripts that make form >> distinctions for some characters based on positions in a string >> and for which Unicode has chosen to assign different code >> points. Even if there were wide acceptance of his view, Unicode >> stability principles would prevent changing toCaseFold (or >> CaseFolding.txt), but this is more evidence that what toCaseFold >> does and does not do is going to be hard to explain to either >> casual users or to writing system experts whose primary >> experience is not with the Greek-Latin-Cyrillic group. >> >> I don't think we want to say "these matching rules are somewhat >> arbitrary and irrational, but, if you don't like it, blame >> Unicode and not us", if only because it is our choice to use >> those matching rules. More below. >> >> >>> ... >>>> (2) Second, probably as a result of having IDNA in the lead, >>>> we've gotten sloppy about language and operations and should >>>> probably start untangling that before it gets people in >>>> trouble. >>> >>> Where is the right place to do that untangling? (I doubt that >>> it is the precis-nickname document.) >> >> I agree that precis-nickname isn't the ideal place. I also >> believe that you and it are the innocent victims of the >> situation. At the same time, I don't believe IETF should be >> producing incomplete, ambiguous, erroneous, or misleading >> standards because no one could get around to doing the right >> foundational work. > > Agreed. I too want to get this right, even though it's not a lot of fun > and it's certainly more work than I thought I was signing up for at the > NEWPREP BoF years ago. > >>>> The Unicode Standard, at least as I understand it, is fairly >>>> clear that the most important (and really only safe) use of >>>> toCaseFold is as part of a comparison operation. >>> >>> Thanks for noting that. For example, Section 5.18 of Unicode >>> 8.0.0 says: >>> >>> Caseless matching is implemented using case folding, which >>> is the >>> process of mapping characters of different case to a >>> single form, so >>> that case differences in strings are erased. Case folding >>> allows for >>> fast caseless matches in lookups because only binary >>> comparison is >>> required. It is more than just conversion to lowercase. >> >> Right. But, again, when its use is appropriate (a very >> controversial topic in itself with our painful IDNA history with >> Final Sigma, Eszett and the case-independent versus >> position-independent controversy called out above as examples) >> that is "matches in lookups" (what I've described elsewhere as >> "comparison only"). Not creating or defining nicknames or >> aliases. And that _is_ a problem for this document. > > I'm not convinced that things are as bad as you think. If we say in > draft-ietf-precis-nickname that the case mapping rule is to be applied > only as part of comparison and not as part of enforcement - which I > think is really what we care about (e.g., to prevent spoofing of users > in chat rooms) - then I think we might be most of the way there. > >>>> Using your >>>> example it is entirely reasonable to treat, "stpeter" and >>>> "StPeter" as equivalent in a comparison operation, but >>>> accepting one string and changing it to the other for display >>>> may not be a really good idea. While that transformation may >>>> be acceptable (although I would be surprised if there were no >>>> people who share your surname who could consider "stpeter" or >>>> "Stpeter" unacceptable and might even believe that "StPeter" >>>> is an unacceptable substitute for "St. Peter"), >>> >>> I do receive email at stpeter@gmail.com intended for >>> st.peter@gmail.com but that's a separate topic... >> >> One that is relevant because it "works" as a side-effect of a >> decision Google has made about mailbox name equivalence, a >> decision that, IMO, will sooner or later get someone into a lot >> of trouble and, more important, a decision and matching rule >> that PRECIS, AFAICT, does not allow and that IDNA unambigiously >> forbids. >> >>>> it also points out the >>>> dangers of using Basic Latin script examples to illustrate >>>> situations in which even more extended Latin script, much less >>>> other scripts, may raise more complex issues. Because IDNA >>>> is essentially a workaround because changing the DNS >>>> comparison rules was impractical for several reasons, we >>>> ended up using toCaweFold to map characters and strings into >>>> others in IDNA2003 but PRECIS implementations that do not >>>> have the same constraints would, in general, be better off >>>> confining the use of toCaseFold, or even toLowerCase, to >>>> comparison operations. >>> >>> Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does >>> it make sense for this nickname specification to differ in >>> this respect from the published RFCs? Shall we file errata >>> against those documents? (This might apply only to RFC 7613, >>> which says to apply case folding as part of the enforcement >>> process - when exactly to apply case folding is not stipulated >>> by RFC 7564.) >> >> To the extent to which this is a "botched that because the WG >> didn't understand the issues well enough" conclusion, it would >> be entirely reasonable to generate an updating RFC that repairs >> 7613 and/or 7564, even doing so in an addendum to >> precis-nickname if that is the only way to do that >> expeditiously. Per the above, we really don't want to give >> library routine writers bad instructions. As I understand it, >> the current position of the RFC Editor and IESG is that >> technical specification errors discovered in retrospect or after >> people start using a spec are not appropriate topics for errata. >> If the WG is not willing to do any of those things, then I >> suggest that precis-nickname at least needs to contain a very >> clear warning notice about this situation (see my response to >> your question 1 below). > > I think we'll probably need to fix 7613 and 7564. I am hoping we can fix > nickname now so that it is less incorrect than the other two. That > doesn't necessarily mean we won't need to also further fix nickname > later on. > > Granted, we were supposed to avoid this problem by working on all of the > PRECIS specs simultaneously. Clearly we have not avoided the problem, so > we need to solve it one way or another. If that means bis for them all, > we need to deal with it. > >>>> (3) Because toCaweFold loses information when used for more >>>> than comparison (for comparison, it merely contributes to >>>> what some people would consider false positives for matching) >>>> involves some controversial decisions and, because of >>>> stability requirements, cannot be changed even if the >>>> controversies are resolved in other ways, we end up with, >>>> e.g., >>>> toCaseFold ("Nuß") -> "nuss" >>>> which is considered an acceptable transformation in some >>>> places that identify themselves as speaking/using German and >>>> two different unacceptable errors in others. Again, this will >>>> almost always be much more serious if the transformation is >>>> used to map and replace strings than if it is used to compare >>>> (fwiw, that particular example is part of a continuing >>>> disagreement between IDNA2008 and, among others, German >>>> domain registry authorities on one side and UTC and UTR 46 on >>>> the other). >>> >>> Agreed. >> >> See "warning notice" comment above and question 1 response below. >> >>> (4) If the motivation is really to avoid confusion, the >>>> correct confusion-blocking rule for Latin script (but not >>>> others) and many languages that use it (but certainly not >>>> all) involves moving beyond toCaseFold and treating all >>>> "decorated" characters (characters normally represented by >>>> glyphs consisting of a Basic Latin character and one or more >>>> diacritical or equivalent markings) compare equal to their >>>> base characters, e.g., "á" not only matches "Á" but also >>>> "a" and "A" and, as an unfortunate side-effect, maybe "À" >>>> and "à" as well. This is bad news for languages in which >>>> decorated Latin characters are used to represent phonetically >>>> and conceptually different characters, not just pronunciation >>>> variations. I am not qualified to evaluate "how bad". In >>>> addition, extrapolations from this principle about Latin >>>> script to unrelated scripts will almost certainly lead to >>>> serious errors and/or additional confusion. >>> >>> I would not be comfortable going that far... >> >> In case it isn't clear, I would not be either. But it is where >> getting sloppy about this stuff could easily take us. It is >> worth noting that it also identifies one of the difficulties >> with doing a global system to be applied to many types of >> applications (like the PRECIS work) and then applying it in user >> interface software that end users will expect to be localized to >> their assumptions because it has been mapped or translated into >> their language (if one normally speaks Upper Slobbovian but has >> some familiarity with English, an application interface in >> English will probably be expected to be "foreign", odd, and >> maybe even inconsistent with whatever expectations exist. But, >> if the interface is in Upper Slobbovian, the natural and >> reasonable assumption will be the matching should conform to >> normal Upper Slobbovian conventions. FWIW, a matching rule >> that says: >> >> (i) Two instances of a base character with the same >> diacritical mark(s) match. >> (ii) Two instances of a base character with different >> diacritical mark(s) do not match. >> (iii) Two instances of a base character, one with >> diacritical mark(s) and the other without any decoration >> match. >> >> Is precisely correct and normal behavior for at least one >> language that uses Latin script. It is also the normal practice >> for at least one Latin script transcription system that is used >> by a large fraction of a billion people (maybe more). > > That is indeed sobering. > >>>> More on this and Tom's question below... >>>> >>>>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>>>> Peter, Alexey, >>>>>> >>>>>> I think there is an ambiguity in the specification of case >>>>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>>>> ... >>>>>> But there are 55 code points in Unicode 7.0.0 that change >>>>>> under default case folding that are neither uppercase nor >>>>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>>>> suspect this stems from a confusion between Unicode case >>>>>> mapping and case folding. >> >> In the context of the above, a different way to say the same >> thing is that people are looking at toCaseFold and assuming (and >> explaining things in terms of) toLowerCase. toCaseFold works >> the way it is expected to and those 55 code points are, more or >> less, collateral damage to get to a matching algorithm that >> favors false positives over false negatives and various edge >> cases (including in "edge cases" languages spoken by, and script >> variations used by, millions of people). > > Sadly I suspect that is an accurate description of the current state of > affairs (modulo my comment above about PRECIS WG discussions at one or > more IETF meetings). > >>> ... >>> After all that, I have 3 questions: >> >> Personal opinions about answers... >> >>> (1) Is my proposed text enough of a clarification that we >>> should make that change before the nickname I-D is published >>> as an RFC? >> >> I think the clarification is an improvement and is important >> enough to incorporate (I know that is the answer to a slightly >> different question). >> >> However, I think it is inadequate without a serious warning >> about the situation. > > Yes. > >> That warning could appear in either this >> document or RFC 7613 (or 7613bis) with a pointer from the other, >> but, unless you want to revise 7613 now, this one is handy. > > I suspect that we need to revise 7613. I suspect that we might also need > to revise 7564 (at least with respect to the order in which operations > are applied, since there has been some confusion among implementers). > > Well, we always knew that we would need to revise them. Just not so soon. > >> Comment about possible text below. >> >>> (2) Should we modify draft-ietf-precis-nickname so that case >>> folding is applied only as part of comparison and not as part >>> of enforcement? If so, should we make that change before this >>> document is published as an RFC? >> >> Yes. If something is used for "enforcement", it should be lower >> casing or something else that can be explained to people who are >> ordinarily familiar with one or more of the scripts that make >> case distinctions. >> >> However, viewed in the light of this discussion, the whole >> "enforcement" concept becomes a little dicey, especially if, as >> I believe but don't have time to verify, the transformations >> performed by toLowerCase are not a proper subset of those >> performed by toCaseFold. > > My initial thought is that case mapping doesn't belong in the nickname > enforcement operation at all - only in the comparison operation. > >>> (3) Should we update RFC 7613 so that case folding is applied >>> only as part of comparison and not as part of enforcement? >> >> I think that is necessary. Following up on the comment above, I >> would prefer that the current Section 3.2.2 (3) of RFC 7613 >> either point to Unicode Lower Casing or contain a warning along >> the lines of that below. > > Unlike the nickname profile (which I think can be cleaned up by moving > the case mapping rule to the comparison operation and continuing to use > Unicode Default Case Folding), I think you are right that for the > UsernameCaseMapped profile we probably want Unicode Lower Casing. Thus > the likely need, sooner rather than later, for 7613bis. > >> >> ---------- >> >> --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre >> wrote: >> >>> This issue has greater urgency now because >>> draft-ietf-precis-nickname is now in AUTH48... >>> >>> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >>> >>>> After all that, I have 3 questions: >>>> >>>> (1) Is my proposed text enough of a clarification that we >>>> should make that change before the nickname I-D is published >>>> as an RFC? >>> >>> I think so. >> >> See above. >> >>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>> folding is applied only as part of comparison and not as part >>>> of enforcement? If so, should we make that change before this >>>> document is published as an RFC? >>> >>> Although it seems to be the case that Unicode case folding is >>> primarily designed for the purpose of matching (i.e., >>> comparison), >> >> "Seems" is a little weak. The Unicode Standard is really quite >> specific about that. >> >>> I have a concern that applying the PRECIS case >>> mapping rule after applying the normalization and >>> directionality rules might have unintended consequences that >>> we haven't had a chance to consider yet. The PRECIS framework >>> expresses a preference (actually a hard requirement) for >>> applying the rules in a particular order. We made a late >>> change to the username profiles (RFC 7613), such that width >>> mapping is applied first (in order to accommodate fullwidth >>> and halfwidth characters in certain East Asian scripts). >>> Making a late change to the nickname profile also concerns me, >>> even though both of these late changes seem reasonable on the >>> face of it. I will try to find time to think about this >>> further in the next 24 hours. >> >> First, a hint for the consideration process: there is a reason >> why Unicode now supports a unified case folding and >> normalization operation. My recollection is that it is not only >> more efficient to perform both operations at once (rather than >> looking in one table and then the other), but that there are >> some order-dependent or priority-dependent cases. >> >> The very fact that this issue exists (and is coming up again) >> this late in the process (7613 published in August, WG winding >> down and not, e.g., meeting next week) calls at least the PRECIS >> quality of review and some fairly fundamental model issues into >> question. I first raised that issue a rather long time ago but >> have continued to hope that we have an approximation to "good >> enough" without going back and rethinking everything. >> >> The right solution, IMO, is that, if RFC 7613 is to rationalize >> or explain the operation in terms of converting upper case >> characters to lower case, then it should be using toLowerCase >> because that is what the operation does. After a quick look at >> 7613, amending/updating it to simply convert to lower case would >> be straightforward (and would not raise the ordering issue >> called out above). It would presumably require another IETF >> Last Call, however and I'd hope we would see some serious >> discussion within the WG (and with UTC) before making the change >> and about how it is explained. >> >> If we are not willing to make a change > > I'm willing. It would, as you note, require some careful thinking and > review to make sure that we got it (more) right this time. > >> that significant and/or >> if we conclude that the WG (and perhaps the IETF) have >> completely run out of energy for dealing with i18n issues [1], >> then I suggest that we introduce some additional text. I've >> just spent a half-hour trying to find the AUTH48 copy of >> precis-nickname (aka RFC-to-be-7700), but the RFC Editor has >> apparently changed naming conventions and the various queue >> entry pages all point to the -19 I-D and not the current working >> copy so I can't try to match text and insertion point to what is >> there already. > > http://www.rfc-editor.org/authors/rfc7700.txt > >> The suggestion is a patch (and a hack), not a >> good fix but something like it is probably the least drastic >> measure that would yield something that doesn't contain >> unexplained known defects. >> >> Rough version of suggested text (possibly to go after your >> revised paragraph and following up my comments in my 1 October >> note). Some of the terminology needs checking which I can do if >> you want to go this route: >> >> 'Users of this specification should note that the >> concept of "lower case conversion" is somewhat elusive >> and more dependent on the conventions of different >> languages and notation systems that use the same script >> than may appear obvious at first glance, especially if >> that glance is at Basic Latin characters (i.e., the >> ASCII letter repertoire). Unicode provides two >> different mapping procedures that produce lower-case >> characters, but they have different effects and results >> for many characters. The more conservative one, >> typically appropriately applicable when lower case forms >> are needed, is actual lower-casing (embodied in the >> Unicode operation toLowerCase). A more radical >> operation, normally suitable only for string matching in >> situations in which it is better to consider uncertain >> cases as matching than to treat them as distinct, is >> called "Case Folding" (Unicode operation toCaseFold). >> While the two operations will often produce the same >> results, Case Folding maps some lower case characters >> into others and performs other transformations that may >> be intuitively reasonable and expected for some users >> and quite astonishing (or just wrong) to others. There >> may be no practical alternative, especially if the >> operations are to be used for mapping or enforcement, to >> developers of PRECIS-dependent understanding that the >> cases in which the two yield different results require >> careful understanding of the relevant user base and its >> needs [2].' > > Thanks. > > I am not sure if we need something like that if we move case mapping > (here, case folding) to the comparison operation only - but something > like that might still be appropriate. > >>>> (3) Should we update RFC 7613 so that case folding is applied >>>> only as part of comparison and not as part of enforcement? >>> >>> That is less urgent so I suggest that we address the nickname >>> spec first. >> >> Unless you (or someone else here) have a plausible plan to >> continue and revitalize the WG and assign it that revision work >> (and bring everyone actively participating up to the level >> needed to easily understand this discussion thread and feel >> embarrassed for not spotting the problems), I think we need to >> assume that this is our last shot. Absent an active and >> committed WG, "do this first" could easily be equivalent to >> "don't get around to the other, ever". > > As mentioned, I don't want to have broken RFCs out there. > >> I think that the particular set of issues that started this >> thread as a known defect in the PRECIS specs, both nickname and >> 7613 and that we are obligated to either fix the problems or at >> least explain them. The above warning text is an attempt to >> explain and identify the problems even if it does not actually >> provide a solution. If it were published as part of >> precis-nickname, it could include a statement to the effect that >> it should also be treated as an update to 7613 or, if the IESG >> and RFC Editor would agree in advance to accept, rather than >> bury, the thing, I suppose we could publish it in >> precis-nickname and create an erratum to 7613 indicating that it >> should have included some form of that statement. Neither >> option implies a huge amount of work to update 7613. But I >> think that making the changes of (2) without doing anything >> about (3) makes the two documents inconsistent with each other >> and that would be an additional known defect. >> >> Procedural question: given that precis-nickname is in AUTH48 as >> of yesterday and I don't see anything blocking publication next >> week if you and Barry sign off on the revised text that the WG >> hasn't seen, > > There is no revised text yet. That's why we're having this discussion. > >> does someone need to file a pro forma objection/ >> appeal to block that until this is sorted out and the WG has a >> chance to review proposed publication text? > > I see no reason to invoke the specter of appeals quite yet. Everyone is > working in good faith to do the right thing and get this mess cleaned up. > >> [1] I believe our collective inability to deal with the >> within-script character forms that do not normalize to each >> other because of language-dependent or other usage factors can >> be taken as evidence of having run out of energy, > > Or in my case simple ignorance of some of the relevant issues and > examples. It's not easy to know about all of this. > >> but it is >> probably in the interest of finishing the PRECIS work to try to >> treat that as a separate issue. > > Probably. > >> [2] Not unlike the reason to differentiate between NFC and NFKC >> and understand the effects of each. > > Another thing that's not easy to grok in fulness. > > Peter > > From nobody Wed Oct 28 13:54:56 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0E66D1B5DEA for ; Wed, 28 Oct 2015 13:54:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.601 X-Spam-Level: X-Spam-Status: No, score=-1.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, GB_I_LETTER=-2, MANGLED_LIST=2.3, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 08nZWDCc5Eik for ; Wed, 28 Oct 2015 13:54:49 -0700 (PDT) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8263E1B5DE2 for ; Wed, 28 Oct 2015 13:54:49 -0700 (PDT) Received: by oies66 with SMTP id s66so12386565oie.1 for ; Wed, 28 Oct 2015 13:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=MehwFRTfb6njP34dppIDIlHcRculEGLtKZ2fUL+49SU=; b=QrJjyhIAD3KvVLt87sqPhrhQwSToObttn5eWUQ9JnH+VDWWXwfyv8mEezikMq9SACw 7o4HX944apufQ14deyL9YC472anQdz54UtrObOoz/3AlCr2e6u++bDliV9Aglh0coPut qM1cdqyVGFPOxW4mXA2qbWd+Ox6eIiVUhs06/zPI5lKgpz4wtXSeTMnHZo4muaUhzSE0 T1rHTH9usMmCAmmkBNQdsJvAlxVZz4MbygDm5IJGPKkWFBUv+oO+qOizpyZrBvyBZJbR d0ivCX1myEfNYcZbLN9aCCu3l4f0S6pQAli8ss+aNsGVqmJ2Q693ILMw6kah0Euwol9R fhoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=MehwFRTfb6njP34dppIDIlHcRculEGLtKZ2fUL+49SU=; b=dRN8+CxgE1uPXL/pk9eglyvWJi4w/G8BWobTSK3Up1vjYoJCxxSG93/ewTercWWnHi uS73n2y0xqwbx5Lt1mcQ/iOdoZRk7wV+1f/+M2XjYn1vlKVJP4zpf/vRU6d8DxEq64x1 ScganRhsr8qZ9pNE2+XWaqEpekqUDPrIDoyPL3/07dWN1jddSbGEi2WfnfyW99Bf2nSh R1Y+GQlZtq3J+BX4wjR2umXK6VQjsO/gOYrj8C/LkMK1nsGj+9OBBOjk8Y/3rPEsA+K7 0zZiG4hFaR9L77oI9pv/3edjt2KgA9xOBvlbL1CmxhYFIynEyXSiPUwVrBKedaPPM9qE DkdA== X-Gm-Message-State: ALoCoQk4dcMnd1jYNb5odOIJzozMDGfbhplMD9C09+lEy4awxLhGIETSAWKaZsV8abhgQqMtJU9h X-Received: by 10.202.183.137 with SMTP id h131mr6363260oif.58.1446065688765; Wed, 28 Oct 2015 13:54:48 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id s127sm20726444oia.21.2015.10.28.13.54.46 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Oct 2015 13:54:47 -0700 (PDT) To: John C Klensin , Tom Worster , Alexey Melnikov References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> From: Peter Saint-Andre Message-ID: <56313616.8000801@andyet.net> Date: Wed, 28 Oct 2015 14:54:46 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <56312AAC.1000300@andyet.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 20:54:55 -0000 And here is another correction in Section 3... OLD Regarding examples 5, 6, and 7: applying Unicode Default Case Folding to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL LETTER SIGMA (U+03C3), and doing so during comparison would result in matching the nicknames in examples 5 and 6; however, because the PRECIS mapping rules do not account for the special status of GREEK SMALL LETTER FINAL SIGMA (U+03C2), the nicknames in examples 5 and 7 or examples 6 and 7 would not be matched. NEW Regarding examples 5, 6, and 7: applying Unicode Default Case Folding to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL LETTER SIGMA (U+03C3), and the same is true of GREEK SMALL LETTER FINAL SIGMA (U+03C2); therefore, the comparison operation defined in Section 2.4 would result in matching of the nicknames in examples 5, 6, and 7. On 10/28/15 2:06 PM, Peter Saint-Andre wrote: > I propose the following text changes: > > ### > > OLD > > 3. Case Mapping Rule: Uppercase and titlecase characters MUST be > mapped to their lowercase equivalents using Unicode Default Case > Folding as defined in the Unicode Standard [Unicode] (at the time > of this writing, the algorithm is specified in Chapter 3 of > [Unicode7.0]). In applications that prohibit conflicting > nicknames, this rule helps to reduce the possibility of confusion > by ensuring that nicknames differing only by case (e.g., > "stpeter" vs. "StPeter") would not be presented to a human user > at the same time. > > NEW > > 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, > as defined in the Unicode Standard [Unicode] (at the time > of this writing, the algorithm is specified in Chapter 3 of > [Unicode7.0]). The primary result of doing so is that uppercase > characters are mapped to lowercase characters. In applications > that prohibit conflicting nicknames, this rule helps to reduce > the possibility of confusion by ensuring that nicknames > differing only by case (e.g., "stpeter" vs. "StPeter") would not > be presented to a human user at the same time. > > ### > > (The foregoing was previously sent to the list.) > > ### > > OLD > > 2.3. Enforcement > > An entity that performs enforcement according to this profile MUST > prepare a string as described in Section 2.2 and MUST also apply the > rules specified in Section 2.1. The rules MUST be applied in the > order shown. > > After all of the foregoing rules have been enforced, the entity MUST > ensure that the nickname is not zero bytes in length (this is done > after enforcing the rules to prevent applications from mistakenly > omitting a nickname entirely, because when internationalized > characters are accepted, a non-empty sequence of characters can > result in a zero-length nickname after canonicalization). > > 2.4. Comparison > > An entity that performs comparison of two strings according to this > profile MUST prepare each string and enforce the rules as specified > in Sections 2.2 and 2.3. The two strings are to be considered > equivalent if they are an exact octet-for-octet match (sometimes > called "bit-string identity"). > > NEW > > 2.3. Enforcement > > An entity that performs enforcement according to this profile MUST > prepare a string as described in Section 2.2 and MUST also apply the > following rules specified in Section 2.1 in the order shown: > > 1. Additional Mapping Rule > 2. Normalization Rule > 3. Directionality Rule > > After all of the foregoing rules have been enforced, the entity MUST > ensure that the nickname is not zero bytes in length (this is done > after enforcing the rules to prevent applications from mistakenly > omitting a nickname entirely, because when internationalized > characters are accepted, a non-empty sequence of characters can > result in a zero-length nickname after canonicalization). > > 2.4. Comparison > > An entity that performs comparison of two strings according to this > profile MUST prepare each string as specified in Section 2.2 and > MUST apply the following rules specified in Section 2.1 in the order > shown: > > 1. Additional Mapping Rule > 2. Case Mapping Rule > 3. Normalization Rule > 4. Directionality Rule > > The two strings are to be considered equivalent if they are an exact > octet-for-octet match (sometimes called "bit-string identity"). > > ### > > In addition, some variation on John's proposed text about toLowerCase > vs. toCaseFold might be appropriate at the end of Section 4; however, > I'm still not sure that is necessary if we move the case mapping rule to > the comparison operation. > > Peter > > On 10/27/15 8:09 PM, Peter Saint-Andre wrote: >> On 10/27/15 11:32 AM, John C Klensin wrote: >>> Response to Monday's note immediately below; response to today's >>> follows it. My apologies, but it is probably important to read >>> both. My further apologies for the length of this note, but I >>> think we are in deep trouble here, >> >> Internationalization always seems to be a matter of how deep the trouble >> is... >> >>> trouble that is aggravated by >>> precis-mappings and precis-nickname both being post-approval and >>> that, as far as I know, there are no future plans for PRECIS >>> work (having precis-nickname in AUTH48 just emphasizes that -- >>> see comment at end). >> >> We had not planned to work on PRECIS because we thought we were done for >> awhile. If that's not the case and we need to fix things, then so be it. >> Whether there is sufficient and continued energy for such work is >> another question. Personally I don't want us to have broken RFCs out >> there. >> >>> --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - >>> &yet wrote: >>> >>>> My apologies for the delayed reply. Comments inline. >>> >>> A few remarks below... I can't tell whether we disagree or >>> whether at least one of us, probably me, are not being >>> adequately clear. (Material on which we fairly clearly agree >>> elided.) >>> >>> >>>> On 10/1/15 7:50 AM, John C Klensin wrote: >>>>> --On Wednesday, September 30, 2015 15:16 -0600 Peter >>>>> Saint-Andre - &yet wrote: >>>> ... >>>>> Peter, >>>>> >>>>> While your proposed text is an improvement, >>>> >>>> Happy to hear it. All I intended was a slight clarification. >>> >>> But I'm not certain we are there yet... >> >> Agreed. The text I proposed addressed only a very small part of the >> problem. >> >>>>> the desire of many >>>>> people for a magic "just tell me what to do" formula, one that >>>>> lets them avoid understanding the issues, may call for a >>>>> little more: >>>> >>>> There is always a need for more when it comes to i18n. >>> >>> But I think it is a little more that that. I've heard several >>> times, including in PRECIS meetings, requests for "just tell me >>> what to do and make sure it isn't complicated" (or "I don't want >>> to have to think about, much less understand, the issues"). We >>> can debate whether giving in to those requests in the I18n case >>> is wise. I think it leads directly to conclusions equivalent to >>> "I understand my own script and writing system (or think I do) >>> and therefore, since all writing systems must be pretty much the >>> same, I understand all of the core issues in terms of my script >>> and understanding". That, in turn, leads directly to the "how >>> do you spell 'Zürich'?" and "all spellings of 'Zuerich' should >>> be treated as equivalent" discussion that sounded like they >>> dominated a BOF at IETF 93. >>> >>> Now I actually think it is reasonable for someone to ask for a >>> library that will do the job most of the time and that will >>> almost never cause their users or customers to get angry at >>> them. But, if we are going to call what we do "standards", they >>> should contain sufficient information that would-be library >>> authors can know what to do ... or understand that they are in >>> over their heads. And, for these particular cases, we may need >>> to explain, or help the library authors explain, why some cases >>> will fail and, indeed, get users mad at vendors. >>> >>> >>>>> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >>>>> primary result of doing so is that uppercase characters are >>>>> mapped to lowercase characters" is true for toCaseFold, >>>> >>>> By "primary" I meant two things: (1) lowercasing is what >>>> happens to the preponderance of code points and (2) this is >>>> the result that most people care about. >>> >>> If I parse the above correctly, I think you are wrong. I think >>> what most people want, care about, and think they are getting, >>> is lower case conversion, i.e., an operation that preserves >>> lower case characters and converts upper case characters to the >>> equivalent lower case. toCaseFold isn't that operation. It is >>> a much more complex and subtle operation that, as well as >>> converting upper case characters to lower case, sometimes >>> converts lower case characters to different lower case >>> characters (or strings of them). It also requires a fairly good >>> understanding of Unicode (not just a relevant script) and >>> historical Unicode decisions to predict its behavior and to have >>> any hope of explaining that behavior to users. If one is >>> trying to compare (as distinct from converting), then toCaseFold >>> may be exactly what it wanted. but it is really hard to explain >>> or justify that in terms of "nicknames" or "aliases", which are >>> about conversion. And, if one hopes to explain what is going >>> on to users in terms of "lower casing", then toCaseFold is just >>> the wrong operation. That is what toLowerCase is for and the >>> two operations are just not equivalent. >> >> My recollection, quite possibly inaccurate or incomplete, from at least >> one and I think several in-person meetings of the PRECIS WG was: just >> use Unicode Default Case Folding because if you use anything else or try >> to roll your own you will be fubar forever. I do not recall any >> discussion of the issues you have raised in this thread (e.g., about the >> inadvisability of using case folding for anything but comparison >> operations) until the last few weeks. However, I freely admit that's >> probably because, through my own faults and ignorance, I didn't >> understand what you were saying. >> >>> FWIW and purely by coincidence wrt PRECIS and this document, I >>> had a conversation a few days ago with an expert on Arabic (and >>> Persian) calligraphy and writing systems (and good general >>> knowledge of writing systems) who is quite insistent that any >>> procedure we use for case-insensitive matching (e.g., case >>> folding) is discriminatory, inconsistent, and just >>> badly-thought-out if that same procedure doesn't treat isolated, >>> initial, and medial forms of the same character as equivalent. >>> He further strengthens his case (sic) by noting that Unicode >>> case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, >>> unambiguously a lower-case character) to U+03C3 (GREEK SMALL >>> LETTER SIGMA), a relationship that depends entirely on >>> positional use and not case. He also believes the same >>> relationships should apply to all other scripts that make form >>> distinctions for some characters based on positions in a string >>> and for which Unicode has chosen to assign different code >>> points. Even if there were wide acceptance of his view, Unicode >>> stability principles would prevent changing toCaseFold (or >>> CaseFolding.txt), but this is more evidence that what toCaseFold >>> does and does not do is going to be hard to explain to either >>> casual users or to writing system experts whose primary >>> experience is not with the Greek-Latin-Cyrillic group. >>> >>> I don't think we want to say "these matching rules are somewhat >>> arbitrary and irrational, but, if you don't like it, blame >>> Unicode and not us", if only because it is our choice to use >>> those matching rules. More below. >>> >>> >>>> ... >>>>> (2) Second, probably as a result of having IDNA in the lead, >>>>> we've gotten sloppy about language and operations and should >>>>> probably start untangling that before it gets people in >>>>> trouble. >>>> >>>> Where is the right place to do that untangling? (I doubt that >>>> it is the precis-nickname document.) >>> >>> I agree that precis-nickname isn't the ideal place. I also >>> believe that you and it are the innocent victims of the >>> situation. At the same time, I don't believe IETF should be >>> producing incomplete, ambiguous, erroneous, or misleading >>> standards because no one could get around to doing the right >>> foundational work. >> >> Agreed. I too want to get this right, even though it's not a lot of fun >> and it's certainly more work than I thought I was signing up for at the >> NEWPREP BoF years ago. >> >>>>> The Unicode Standard, at least as I understand it, is fairly >>>>> clear that the most important (and really only safe) use of >>>>> toCaseFold is as part of a comparison operation. >>>> >>>> Thanks for noting that. For example, Section 5.18 of Unicode >>>> 8.0.0 says: >>>> >>>> Caseless matching is implemented using case folding, which >>>> is the >>>> process of mapping characters of different case to a >>>> single form, so >>>> that case differences in strings are erased. Case folding >>>> allows for >>>> fast caseless matches in lookups because only binary >>>> comparison is >>>> required. It is more than just conversion to lowercase. >>> >>> Right. But, again, when its use is appropriate (a very >>> controversial topic in itself with our painful IDNA history with >>> Final Sigma, Eszett and the case-independent versus >>> position-independent controversy called out above as examples) >>> that is "matches in lookups" (what I've described elsewhere as >>> "comparison only"). Not creating or defining nicknames or >>> aliases. And that _is_ a problem for this document. >> >> I'm not convinced that things are as bad as you think. If we say in >> draft-ietf-precis-nickname that the case mapping rule is to be applied >> only as part of comparison and not as part of enforcement - which I >> think is really what we care about (e.g., to prevent spoofing of users >> in chat rooms) - then I think we might be most of the way there. >> >>>>> Using your >>>>> example it is entirely reasonable to treat, "stpeter" and >>>>> "StPeter" as equivalent in a comparison operation, but >>>>> accepting one string and changing it to the other for display >>>>> may not be a really good idea. While that transformation may >>>>> be acceptable (although I would be surprised if there were no >>>>> people who share your surname who could consider "stpeter" or >>>>> "Stpeter" unacceptable and might even believe that "StPeter" >>>>> is an unacceptable substitute for "St. Peter"), >>>> >>>> I do receive email at stpeter@gmail.com intended for >>>> st.peter@gmail.com but that's a separate topic... >>> >>> One that is relevant because it "works" as a side-effect of a >>> decision Google has made about mailbox name equivalence, a >>> decision that, IMO, will sooner or later get someone into a lot >>> of trouble and, more important, a decision and matching rule >>> that PRECIS, AFAICT, does not allow and that IDNA unambigiously >>> forbids. >>> >>>>> it also points out the >>>>> dangers of using Basic Latin script examples to illustrate >>>>> situations in which even more extended Latin script, much less >>>>> other scripts, may raise more complex issues. Because IDNA >>>>> is essentially a workaround because changing the DNS >>>>> comparison rules was impractical for several reasons, we >>>>> ended up using toCaweFold to map characters and strings into >>>>> others in IDNA2003 but PRECIS implementations that do not >>>>> have the same constraints would, in general, be better off >>>>> confining the use of toCaseFold, or even toLowerCase, to >>>>> comparison operations. >>>> >>>> Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does >>>> it make sense for this nickname specification to differ in >>>> this respect from the published RFCs? Shall we file errata >>>> against those documents? (This might apply only to RFC 7613, >>>> which says to apply case folding as part of the enforcement >>>> process - when exactly to apply case folding is not stipulated >>>> by RFC 7564.) >>> >>> To the extent to which this is a "botched that because the WG >>> didn't understand the issues well enough" conclusion, it would >>> be entirely reasonable to generate an updating RFC that repairs >>> 7613 and/or 7564, even doing so in an addendum to >>> precis-nickname if that is the only way to do that >>> expeditiously. Per the above, we really don't want to give >>> library routine writers bad instructions. As I understand it, >>> the current position of the RFC Editor and IESG is that >>> technical specification errors discovered in retrospect or after >>> people start using a spec are not appropriate topics for errata. >>> If the WG is not willing to do any of those things, then I >>> suggest that precis-nickname at least needs to contain a very >>> clear warning notice about this situation (see my response to >>> your question 1 below). >> >> I think we'll probably need to fix 7613 and 7564. I am hoping we can fix >> nickname now so that it is less incorrect than the other two. That >> doesn't necessarily mean we won't need to also further fix nickname >> later on. >> >> Granted, we were supposed to avoid this problem by working on all of the >> PRECIS specs simultaneously. Clearly we have not avoided the problem, so >> we need to solve it one way or another. If that means bis for them all, >> we need to deal with it. >> >>>>> (3) Because toCaweFold loses information when used for more >>>>> than comparison (for comparison, it merely contributes to >>>>> what some people would consider false positives for matching) >>>>> involves some controversial decisions and, because of >>>>> stability requirements, cannot be changed even if the >>>>> controversies are resolved in other ways, we end up with, >>>>> e.g., >>>>> toCaseFold ("Nuß") -> "nuss" >>>>> which is considered an acceptable transformation in some >>>>> places that identify themselves as speaking/using German and >>>>> two different unacceptable errors in others. Again, this will >>>>> almost always be much more serious if the transformation is >>>>> used to map and replace strings than if it is used to compare >>>>> (fwiw, that particular example is part of a continuing >>>>> disagreement between IDNA2008 and, among others, German >>>>> domain registry authorities on one side and UTC and UTR 46 on >>>>> the other). >>>> >>>> Agreed. >>> >>> See "warning notice" comment above and question 1 response below. >>> >>>> (4) If the motivation is really to avoid confusion, the >>>>> correct confusion-blocking rule for Latin script (but not >>>>> others) and many languages that use it (but certainly not >>>>> all) involves moving beyond toCaseFold and treating all >>>>> "decorated" characters (characters normally represented by >>>>> glyphs consisting of a Basic Latin character and one or more >>>>> diacritical or equivalent markings) compare equal to their >>>>> base characters, e.g., "á" not only matches "Á" but also >>>>> "a" and "A" and, as an unfortunate side-effect, maybe "À" >>>>> and "à" as well. This is bad news for languages in which >>>>> decorated Latin characters are used to represent phonetically >>>>> and conceptually different characters, not just pronunciation >>>>> variations. I am not qualified to evaluate "how bad". In >>>>> addition, extrapolations from this principle about Latin >>>>> script to unrelated scripts will almost certainly lead to >>>>> serious errors and/or additional confusion. >>>> >>>> I would not be comfortable going that far... >>> >>> In case it isn't clear, I would not be either. But it is where >>> getting sloppy about this stuff could easily take us. It is >>> worth noting that it also identifies one of the difficulties >>> with doing a global system to be applied to many types of >>> applications (like the PRECIS work) and then applying it in user >>> interface software that end users will expect to be localized to >>> their assumptions because it has been mapped or translated into >>> their language (if one normally speaks Upper Slobbovian but has >>> some familiarity with English, an application interface in >>> English will probably be expected to be "foreign", odd, and >>> maybe even inconsistent with whatever expectations exist. But, >>> if the interface is in Upper Slobbovian, the natural and >>> reasonable assumption will be the matching should conform to >>> normal Upper Slobbovian conventions. FWIW, a matching rule >>> that says: >>> >>> (i) Two instances of a base character with the same >>> diacritical mark(s) match. >>> (ii) Two instances of a base character with different >>> diacritical mark(s) do not match. >>> (iii) Two instances of a base character, one with >>> diacritical mark(s) and the other without any decoration >>> match. >>> >>> Is precisely correct and normal behavior for at least one >>> language that uses Latin script. It is also the normal practice >>> for at least one Latin script transcription system that is used >>> by a large fraction of a billion people (maybe more). >> >> That is indeed sobering. >> >>>>> More on this and Tom's question below... >>>>> >>>>>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>>>>> Peter, Alexey, >>>>>>> >>>>>>> I think there is an ambiguity in the specification of case >>>>>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>>>>> ... >>>>>>> But there are 55 code points in Unicode 7.0.0 that change >>>>>>> under default case folding that are neither uppercase nor >>>>>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>>>>> suspect this stems from a confusion between Unicode case >>>>>>> mapping and case folding. >>> >>> In the context of the above, a different way to say the same >>> thing is that people are looking at toCaseFold and assuming (and >>> explaining things in terms of) toLowerCase. toCaseFold works >>> the way it is expected to and those 55 code points are, more or >>> less, collateral damage to get to a matching algorithm that >>> favors false positives over false negatives and various edge >>> cases (including in "edge cases" languages spoken by, and script >>> variations used by, millions of people). >> >> Sadly I suspect that is an accurate description of the current state of >> affairs (modulo my comment above about PRECIS WG discussions at one or >> more IETF meetings). >> >>>> ... >>>> After all that, I have 3 questions: >>> >>> Personal opinions about answers... >>> >>>> (1) Is my proposed text enough of a clarification that we >>>> should make that change before the nickname I-D is published >>>> as an RFC? >>> >>> I think the clarification is an improvement and is important >>> enough to incorporate (I know that is the answer to a slightly >>> different question). >>> >>> However, I think it is inadequate without a serious warning >>> about the situation. >> >> Yes. >> >>> That warning could appear in either this >>> document or RFC 7613 (or 7613bis) with a pointer from the other, >>> but, unless you want to revise 7613 now, this one is handy. >> >> I suspect that we need to revise 7613. I suspect that we might also need >> to revise 7564 (at least with respect to the order in which operations >> are applied, since there has been some confusion among implementers). >> >> Well, we always knew that we would need to revise them. Just not so soon. >> >>> Comment about possible text below. >>> >>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>> folding is applied only as part of comparison and not as part >>>> of enforcement? If so, should we make that change before this >>>> document is published as an RFC? >>> >>> Yes. If something is used for "enforcement", it should be lower >>> casing or something else that can be explained to people who are >>> ordinarily familiar with one or more of the scripts that make >>> case distinctions. >>> >>> However, viewed in the light of this discussion, the whole >>> "enforcement" concept becomes a little dicey, especially if, as >>> I believe but don't have time to verify, the transformations >>> performed by toLowerCase are not a proper subset of those >>> performed by toCaseFold. >> >> My initial thought is that case mapping doesn't belong in the nickname >> enforcement operation at all - only in the comparison operation. >> >>>> (3) Should we update RFC 7613 so that case folding is applied >>>> only as part of comparison and not as part of enforcement? >>> >>> I think that is necessary. Following up on the comment above, I >>> would prefer that the current Section 3.2.2 (3) of RFC 7613 >>> either point to Unicode Lower Casing or contain a warning along >>> the lines of that below. >> >> Unlike the nickname profile (which I think can be cleaned up by moving >> the case mapping rule to the comparison operation and continuing to use >> Unicode Default Case Folding), I think you are right that for the >> UsernameCaseMapped profile we probably want Unicode Lower Casing. Thus >> the likely need, sooner rather than later, for 7613bis. >> >>> >>> ---------- >>> >>> --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre >>> wrote: >>> >>>> This issue has greater urgency now because >>>> draft-ietf-precis-nickname is now in AUTH48... >>>> >>>> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >>>> >>>>> After all that, I have 3 questions: >>>>> >>>>> (1) Is my proposed text enough of a clarification that we >>>>> should make that change before the nickname I-D is published >>>>> as an RFC? >>>> >>>> I think so. >>> >>> See above. >>> >>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>> folding is applied only as part of comparison and not as part >>>>> of enforcement? If so, should we make that change before this >>>>> document is published as an RFC? >>>> >>>> Although it seems to be the case that Unicode case folding is >>>> primarily designed for the purpose of matching (i.e., >>>> comparison), >>> >>> "Seems" is a little weak. The Unicode Standard is really quite >>> specific about that. >>> >>>> I have a concern that applying the PRECIS case >>>> mapping rule after applying the normalization and >>>> directionality rules might have unintended consequences that >>>> we haven't had a chance to consider yet. The PRECIS framework >>>> expresses a preference (actually a hard requirement) for >>>> applying the rules in a particular order. We made a late >>>> change to the username profiles (RFC 7613), such that width >>>> mapping is applied first (in order to accommodate fullwidth >>>> and halfwidth characters in certain East Asian scripts). >>>> Making a late change to the nickname profile also concerns me, >>>> even though both of these late changes seem reasonable on the >>>> face of it. I will try to find time to think about this >>>> further in the next 24 hours. >>> >>> First, a hint for the consideration process: there is a reason >>> why Unicode now supports a unified case folding and >>> normalization operation. My recollection is that it is not only >>> more efficient to perform both operations at once (rather than >>> looking in one table and then the other), but that there are >>> some order-dependent or priority-dependent cases. >>> >>> The very fact that this issue exists (and is coming up again) >>> this late in the process (7613 published in August, WG winding >>> down and not, e.g., meeting next week) calls at least the PRECIS >>> quality of review and some fairly fundamental model issues into >>> question. I first raised that issue a rather long time ago but >>> have continued to hope that we have an approximation to "good >>> enough" without going back and rethinking everything. >>> >>> The right solution, IMO, is that, if RFC 7613 is to rationalize >>> or explain the operation in terms of converting upper case >>> characters to lower case, then it should be using toLowerCase >>> because that is what the operation does. After a quick look at >>> 7613, amending/updating it to simply convert to lower case would >>> be straightforward (and would not raise the ordering issue >>> called out above). It would presumably require another IETF >>> Last Call, however and I'd hope we would see some serious >>> discussion within the WG (and with UTC) before making the change >>> and about how it is explained. >>> >>> If we are not willing to make a change >> >> I'm willing. It would, as you note, require some careful thinking and >> review to make sure that we got it (more) right this time. >> >>> that significant and/or >>> if we conclude that the WG (and perhaps the IETF) have >>> completely run out of energy for dealing with i18n issues [1], >>> then I suggest that we introduce some additional text. I've >>> just spent a half-hour trying to find the AUTH48 copy of >>> precis-nickname (aka RFC-to-be-7700), but the RFC Editor has >>> apparently changed naming conventions and the various queue >>> entry pages all point to the -19 I-D and not the current working >>> copy so I can't try to match text and insertion point to what is >>> there already. >> >> http://www.rfc-editor.org/authors/rfc7700.txt >> >>> The suggestion is a patch (and a hack), not a >>> good fix but something like it is probably the least drastic >>> measure that would yield something that doesn't contain >>> unexplained known defects. >>> >>> Rough version of suggested text (possibly to go after your >>> revised paragraph and following up my comments in my 1 October >>> note). Some of the terminology needs checking which I can do if >>> you want to go this route: >>> >>> 'Users of this specification should note that the >>> concept of "lower case conversion" is somewhat elusive >>> and more dependent on the conventions of different >>> languages and notation systems that use the same script >>> than may appear obvious at first glance, especially if >>> that glance is at Basic Latin characters (i.e., the >>> ASCII letter repertoire). Unicode provides two >>> different mapping procedures that produce lower-case >>> characters, but they have different effects and results >>> for many characters. The more conservative one, >>> typically appropriately applicable when lower case forms >>> are needed, is actual lower-casing (embodied in the >>> Unicode operation toLowerCase). A more radical >>> operation, normally suitable only for string matching in >>> situations in which it is better to consider uncertain >>> cases as matching than to treat them as distinct, is >>> called "Case Folding" (Unicode operation toCaseFold). >>> While the two operations will often produce the same >>> results, Case Folding maps some lower case characters >>> into others and performs other transformations that may >>> be intuitively reasonable and expected for some users >>> and quite astonishing (or just wrong) to others. There >>> may be no practical alternative, especially if the >>> operations are to be used for mapping or enforcement, to >>> developers of PRECIS-dependent understanding that the >>> cases in which the two yield different results require >>> careful understanding of the relevant user base and its >>> needs [2].' >> >> Thanks. >> >> I am not sure if we need something like that if we move case mapping >> (here, case folding) to the comparison operation only - but something >> like that might still be appropriate. >> >>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>> only as part of comparison and not as part of enforcement? >>>> >>>> That is less urgent so I suggest that we address the nickname >>>> spec first. >>> >>> Unless you (or someone else here) have a plausible plan to >>> continue and revitalize the WG and assign it that revision work >>> (and bring everyone actively participating up to the level >>> needed to easily understand this discussion thread and feel >>> embarrassed for not spotting the problems), I think we need to >>> assume that this is our last shot. Absent an active and >>> committed WG, "do this first" could easily be equivalent to >>> "don't get around to the other, ever". >> >> As mentioned, I don't want to have broken RFCs out there. >> >>> I think that the particular set of issues that started this >>> thread as a known defect in the PRECIS specs, both nickname and >>> 7613 and that we are obligated to either fix the problems or at >>> least explain them. The above warning text is an attempt to >>> explain and identify the problems even if it does not actually >>> provide a solution. If it were published as part of >>> precis-nickname, it could include a statement to the effect that >>> it should also be treated as an update to 7613 or, if the IESG >>> and RFC Editor would agree in advance to accept, rather than >>> bury, the thing, I suppose we could publish it in >>> precis-nickname and create an erratum to 7613 indicating that it >>> should have included some form of that statement. Neither >>> option implies a huge amount of work to update 7613. But I >>> think that making the changes of (2) without doing anything >>> about (3) makes the two documents inconsistent with each other >>> and that would be an additional known defect. >>> >>> Procedural question: given that precis-nickname is in AUTH48 as >>> of yesterday and I don't see anything blocking publication next >>> week if you and Barry sign off on the revised text that the WG >>> hasn't seen, >> >> There is no revised text yet. That's why we're having this discussion. >> >>> does someone need to file a pro forma objection/ >>> appeal to block that until this is sorted out and the WG has a >>> chance to review proposed publication text? >> >> I see no reason to invoke the specter of appeals quite yet. Everyone is >> working in good faith to do the right thing and get this mess cleaned up. >> >>> [1] I believe our collective inability to deal with the >>> within-script character forms that do not normalize to each >>> other because of language-dependent or other usage factors can >>> be taken as evidence of having run out of energy, >> >> Or in my case simple ignorance of some of the relevant issues and >> examples. It's not easy to know about all of this. >> >>> but it is >>> probably in the interest of finishing the PRECIS work to try to >>> treat that as a separate issue. >> >> Probably. >> >>> [2] Not unlike the reason to differentiate between NFC and NFKC >>> and understand the effects of each. >> >> Another thing that's not easy to grok in fulness. >> >> Peter >> From nobody Wed Oct 28 14:53:08 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 452901A0121 for ; Wed, 28 Oct 2015 14:53:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.601 X-Spam-Level: X-Spam-Status: No, score=-1.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, GB_I_LETTER=-2, MANGLED_LIST=2.3, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s-RDbN19rW1h for ; Wed, 28 Oct 2015 14:53:00 -0700 (PDT) Received: from mail-ob0-x232.google.com (mail-ob0-x232.google.com [IPv6:2607:f8b0:4003:c01::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8165D1A011B for ; Wed, 28 Oct 2015 14:53:00 -0700 (PDT) Received: by obbwb3 with SMTP id wb3so18670429obb.0 for ; Wed, 28 Oct 2015 14:53:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=Z8EuDYSJx4ujZi7wl6BJ8CP4X+QxPGBiUC0uNXXLb7M=; b=dVsyCwmRnhI18CdfcUC+dmYqbnXfMpDZbdUuewx+WtbHcFOOwaQ9A6Whcw1IzoutXK eU6J3lp6hZKopHEDfFvTxrGHQgi5BHCt4zBbjAEzjHuN+HTR1anqZVYyGlJtnOXCVAfr oH9hjC6MwoZ8r3585L/m0yrmuQOmpHtVP7U1ZMCxtdepXnCscGTzxv7vojY6jK8fikWr Fu+JFF00cVmHOrbxSQw7g+ng5bljoOcqBglGHBVrCdO7/dCcMYiWH56Yc30+oyXy1NKk 7kP2k7wM4dKQGcVVn1MN4zjzhkdD25OaIQ7fqoKARG6c5HN0e58kYcCxZ6eAbKW6a2YP pUOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=Z8EuDYSJx4ujZi7wl6BJ8CP4X+QxPGBiUC0uNXXLb7M=; b=JBCTStsqIEzukBM3/q8UcFStjFjt4Sd1X8iTfxzfDdAdl8e7ir6Chdvhl0LVZRKuud M1GFhCjA10AYEO8vonRUY1nF2tXkZYVsrNyEyV6blwJizlDjg+LPQiPVpxAQPgRnWFXW EY4UDUgEIe9rBVSyxXtaWvz0DH/Ko0YUMKtt732qnJEfUcew3s9mf8BDHnJRaJ4fTAET +k9sG8swn7c0gvMtA35RBN4+PiqfT+Y7kysufxok0BJZoMCb8eUKt7KKS2rrsQTtr4cT imoIvbqJ51QFQjj8Y33MgFVbJIAhSwYIQbqiiJypcY8aWbnJt/wZlf16bok2vmhbvm6B ROJA== X-Gm-Message-State: ALoCoQkkwn2oeWUpz82Fc+pTK8VUsSNlvsYqsYgFTRm+2NFzIa2khzzmgTUFfP51O9jw7uPQCIDM X-Received: by 10.60.173.42 with SMTP id bh10mr30045673oec.58.1446069179795; Wed, 28 Oct 2015 14:52:59 -0700 (PDT) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id t192sm20844854oie.29.2015.10.28.14.52.57 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Oct 2015 14:52:58 -0700 (PDT) To: John C Klensin , Tom Worster , Alexey Melnikov References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> From: Peter Saint-Andre Message-ID: <563143B9.7020707@andyet.net> Date: Wed, 28 Oct 2015 15:52:57 -0600 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <56313616.8000801@andyet.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2015 21:53:06 -0000 Example 7 needs to be corrected, too, in accordance with CaseFolding.txt. On 10/28/15 2:54 PM, Peter Saint-Andre wrote: > And here is another correction in Section 3... > > OLD > > Regarding examples 5, 6, and 7: applying Unicode Default Case Folding > to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL LETTER > SIGMA (U+03C3), and doing so during comparison would result in > matching the nicknames in examples 5 and 6; however, because the > PRECIS mapping rules do not account for the special status of GREEK > SMALL LETTER FINAL SIGMA (U+03C2), the nicknames in examples 5 and 7 > or examples 6 and 7 would not be matched. > > NEW > > Regarding examples 5, 6, and 7: applying Unicode Default Case Folding > to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL LETTER > SIGMA (U+03C3), and the same is true of GREEK SMALL LETTER FINAL > SIGMA (U+03C2); therefore, the comparison operation defined in > Section 2.4 would result in matching of the nicknames in examples 5, > 6, and 7. > > On 10/28/15 2:06 PM, Peter Saint-Andre wrote: >> I propose the following text changes: >> >> ### >> >> OLD >> >> 3. Case Mapping Rule: Uppercase and titlecase characters MUST be >> mapped to their lowercase equivalents using Unicode Default Case >> Folding as defined in the Unicode Standard [Unicode] (at the time >> of this writing, the algorithm is specified in Chapter 3 of >> [Unicode7.0]). In applications that prohibit conflicting >> nicknames, this rule helps to reduce the possibility of confusion >> by ensuring that nicknames differing only by case (e.g., >> "stpeter" vs. "StPeter") would not be presented to a human user >> at the same time. >> >> NEW >> >> 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, >> as defined in the Unicode Standard [Unicode] (at the time >> of this writing, the algorithm is specified in Chapter 3 of >> [Unicode7.0]). The primary result of doing so is that uppercase >> characters are mapped to lowercase characters. In applications >> that prohibit conflicting nicknames, this rule helps to reduce >> the possibility of confusion by ensuring that nicknames >> differing only by case (e.g., "stpeter" vs. "StPeter") would not >> be presented to a human user at the same time. >> >> ### >> >> (The foregoing was previously sent to the list.) >> >> ### >> >> OLD >> >> 2.3. Enforcement >> >> An entity that performs enforcement according to this profile MUST >> prepare a string as described in Section 2.2 and MUST also apply the >> rules specified in Section 2.1. The rules MUST be applied in the >> order shown. >> >> After all of the foregoing rules have been enforced, the entity MUST >> ensure that the nickname is not zero bytes in length (this is done >> after enforcing the rules to prevent applications from mistakenly >> omitting a nickname entirely, because when internationalized >> characters are accepted, a non-empty sequence of characters can >> result in a zero-length nickname after canonicalization). >> >> 2.4. Comparison >> >> An entity that performs comparison of two strings according to this >> profile MUST prepare each string and enforce the rules as specified >> in Sections 2.2 and 2.3. The two strings are to be considered >> equivalent if they are an exact octet-for-octet match (sometimes >> called "bit-string identity"). >> >> NEW >> >> 2.3. Enforcement >> >> An entity that performs enforcement according to this profile MUST >> prepare a string as described in Section 2.2 and MUST also apply the >> following rules specified in Section 2.1 in the order shown: >> >> 1. Additional Mapping Rule >> 2. Normalization Rule >> 3. Directionality Rule >> >> After all of the foregoing rules have been enforced, the entity MUST >> ensure that the nickname is not zero bytes in length (this is done >> after enforcing the rules to prevent applications from mistakenly >> omitting a nickname entirely, because when internationalized >> characters are accepted, a non-empty sequence of characters can >> result in a zero-length nickname after canonicalization). >> >> 2.4. Comparison >> >> An entity that performs comparison of two strings according to this >> profile MUST prepare each string as specified in Section 2.2 and >> MUST apply the following rules specified in Section 2.1 in the order >> shown: >> >> 1. Additional Mapping Rule >> 2. Case Mapping Rule >> 3. Normalization Rule >> 4. Directionality Rule >> >> The two strings are to be considered equivalent if they are an exact >> octet-for-octet match (sometimes called "bit-string identity"). >> >> ### >> >> In addition, some variation on John's proposed text about toLowerCase >> vs. toCaseFold might be appropriate at the end of Section 4; however, >> I'm still not sure that is necessary if we move the case mapping rule to >> the comparison operation. >> >> Peter >> >> On 10/27/15 8:09 PM, Peter Saint-Andre wrote: >>> On 10/27/15 11:32 AM, John C Klensin wrote: >>>> Response to Monday's note immediately below; response to today's >>>> follows it. My apologies, but it is probably important to read >>>> both. My further apologies for the length of this note, but I >>>> think we are in deep trouble here, >>> >>> Internationalization always seems to be a matter of how deep the trouble >>> is... >>> >>>> trouble that is aggravated by >>>> precis-mappings and precis-nickname both being post-approval and >>>> that, as far as I know, there are no future plans for PRECIS >>>> work (having precis-nickname in AUTH48 just emphasizes that -- >>>> see comment at end). >>> >>> We had not planned to work on PRECIS because we thought we were done for >>> awhile. If that's not the case and we need to fix things, then so be it. >>> Whether there is sufficient and continued energy for such work is >>> another question. Personally I don't want us to have broken RFCs out >>> there. >>> >>>> --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - >>>> &yet wrote: >>>> >>>>> My apologies for the delayed reply. Comments inline. >>>> >>>> A few remarks below... I can't tell whether we disagree or >>>> whether at least one of us, probably me, are not being >>>> adequately clear. (Material on which we fairly clearly agree >>>> elided.) >>>> >>>> >>>>> On 10/1/15 7:50 AM, John C Klensin wrote: >>>>>> --On Wednesday, September 30, 2015 15:16 -0600 Peter >>>>>> Saint-Andre - &yet wrote: >>>>> ... >>>>>> Peter, >>>>>> >>>>>> While your proposed text is an improvement, >>>>> >>>>> Happy to hear it. All I intended was a slight clarification. >>>> >>>> But I'm not certain we are there yet... >>> >>> Agreed. The text I proposed addressed only a very small part of the >>> problem. >>> >>>>>> the desire of many >>>>>> people for a magic "just tell me what to do" formula, one that >>>>>> lets them avoid understanding the issues, may call for a >>>>>> little more: >>>>> >>>>> There is always a need for more when it comes to i18n. >>>> >>>> But I think it is a little more that that. I've heard several >>>> times, including in PRECIS meetings, requests for "just tell me >>>> what to do and make sure it isn't complicated" (or "I don't want >>>> to have to think about, much less understand, the issues"). We >>>> can debate whether giving in to those requests in the I18n case >>>> is wise. I think it leads directly to conclusions equivalent to >>>> "I understand my own script and writing system (or think I do) >>>> and therefore, since all writing systems must be pretty much the >>>> same, I understand all of the core issues in terms of my script >>>> and understanding". That, in turn, leads directly to the "how >>>> do you spell 'Zürich'?" and "all spellings of 'Zuerich' should >>>> be treated as equivalent" discussion that sounded like they >>>> dominated a BOF at IETF 93. >>>> >>>> Now I actually think it is reasonable for someone to ask for a >>>> library that will do the job most of the time and that will >>>> almost never cause their users or customers to get angry at >>>> them. But, if we are going to call what we do "standards", they >>>> should contain sufficient information that would-be library >>>> authors can know what to do ... or understand that they are in >>>> over their heads. And, for these particular cases, we may need >>>> to explain, or help the library authors explain, why some cases >>>> will fail and, indeed, get users mad at vendors. >>>> >>>> >>>>>> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >>>>>> primary result of doing so is that uppercase characters are >>>>>> mapped to lowercase characters" is true for toCaseFold, >>>>> >>>>> By "primary" I meant two things: (1) lowercasing is what >>>>> happens to the preponderance of code points and (2) this is >>>>> the result that most people care about. >>>> >>>> If I parse the above correctly, I think you are wrong. I think >>>> what most people want, care about, and think they are getting, >>>> is lower case conversion, i.e., an operation that preserves >>>> lower case characters and converts upper case characters to the >>>> equivalent lower case. toCaseFold isn't that operation. It is >>>> a much more complex and subtle operation that, as well as >>>> converting upper case characters to lower case, sometimes >>>> converts lower case characters to different lower case >>>> characters (or strings of them). It also requires a fairly good >>>> understanding of Unicode (not just a relevant script) and >>>> historical Unicode decisions to predict its behavior and to have >>>> any hope of explaining that behavior to users. If one is >>>> trying to compare (as distinct from converting), then toCaseFold >>>> may be exactly what it wanted. but it is really hard to explain >>>> or justify that in terms of "nicknames" or "aliases", which are >>>> about conversion. And, if one hopes to explain what is going >>>> on to users in terms of "lower casing", then toCaseFold is just >>>> the wrong operation. That is what toLowerCase is for and the >>>> two operations are just not equivalent. >>> >>> My recollection, quite possibly inaccurate or incomplete, from at least >>> one and I think several in-person meetings of the PRECIS WG was: just >>> use Unicode Default Case Folding because if you use anything else or try >>> to roll your own you will be fubar forever. I do not recall any >>> discussion of the issues you have raised in this thread (e.g., about the >>> inadvisability of using case folding for anything but comparison >>> operations) until the last few weeks. However, I freely admit that's >>> probably because, through my own faults and ignorance, I didn't >>> understand what you were saying. >>> >>>> FWIW and purely by coincidence wrt PRECIS and this document, I >>>> had a conversation a few days ago with an expert on Arabic (and >>>> Persian) calligraphy and writing systems (and good general >>>> knowledge of writing systems) who is quite insistent that any >>>> procedure we use for case-insensitive matching (e.g., case >>>> folding) is discriminatory, inconsistent, and just >>>> badly-thought-out if that same procedure doesn't treat isolated, >>>> initial, and medial forms of the same character as equivalent. >>>> He further strengthens his case (sic) by noting that Unicode >>>> case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, >>>> unambiguously a lower-case character) to U+03C3 (GREEK SMALL >>>> LETTER SIGMA), a relationship that depends entirely on >>>> positional use and not case. He also believes the same >>>> relationships should apply to all other scripts that make form >>>> distinctions for some characters based on positions in a string >>>> and for which Unicode has chosen to assign different code >>>> points. Even if there were wide acceptance of his view, Unicode >>>> stability principles would prevent changing toCaseFold (or >>>> CaseFolding.txt), but this is more evidence that what toCaseFold >>>> does and does not do is going to be hard to explain to either >>>> casual users or to writing system experts whose primary >>>> experience is not with the Greek-Latin-Cyrillic group. >>>> >>>> I don't think we want to say "these matching rules are somewhat >>>> arbitrary and irrational, but, if you don't like it, blame >>>> Unicode and not us", if only because it is our choice to use >>>> those matching rules. More below. >>>> >>>> >>>>> ... >>>>>> (2) Second, probably as a result of having IDNA in the lead, >>>>>> we've gotten sloppy about language and operations and should >>>>>> probably start untangling that before it gets people in >>>>>> trouble. >>>>> >>>>> Where is the right place to do that untangling? (I doubt that >>>>> it is the precis-nickname document.) >>>> >>>> I agree that precis-nickname isn't the ideal place. I also >>>> believe that you and it are the innocent victims of the >>>> situation. At the same time, I don't believe IETF should be >>>> producing incomplete, ambiguous, erroneous, or misleading >>>> standards because no one could get around to doing the right >>>> foundational work. >>> >>> Agreed. I too want to get this right, even though it's not a lot of fun >>> and it's certainly more work than I thought I was signing up for at the >>> NEWPREP BoF years ago. >>> >>>>>> The Unicode Standard, at least as I understand it, is fairly >>>>>> clear that the most important (and really only safe) use of >>>>>> toCaseFold is as part of a comparison operation. >>>>> >>>>> Thanks for noting that. For example, Section 5.18 of Unicode >>>>> 8.0.0 says: >>>>> >>>>> Caseless matching is implemented using case folding, which >>>>> is the >>>>> process of mapping characters of different case to a >>>>> single form, so >>>>> that case differences in strings are erased. Case folding >>>>> allows for >>>>> fast caseless matches in lookups because only binary >>>>> comparison is >>>>> required. It is more than just conversion to lowercase. >>>> >>>> Right. But, again, when its use is appropriate (a very >>>> controversial topic in itself with our painful IDNA history with >>>> Final Sigma, Eszett and the case-independent versus >>>> position-independent controversy called out above as examples) >>>> that is "matches in lookups" (what I've described elsewhere as >>>> "comparison only"). Not creating or defining nicknames or >>>> aliases. And that _is_ a problem for this document. >>> >>> I'm not convinced that things are as bad as you think. If we say in >>> draft-ietf-precis-nickname that the case mapping rule is to be applied >>> only as part of comparison and not as part of enforcement - which I >>> think is really what we care about (e.g., to prevent spoofing of users >>> in chat rooms) - then I think we might be most of the way there. >>> >>>>>> Using your >>>>>> example it is entirely reasonable to treat, "stpeter" and >>>>>> "StPeter" as equivalent in a comparison operation, but >>>>>> accepting one string and changing it to the other for display >>>>>> may not be a really good idea. While that transformation may >>>>>> be acceptable (although I would be surprised if there were no >>>>>> people who share your surname who could consider "stpeter" or >>>>>> "Stpeter" unacceptable and might even believe that "StPeter" >>>>>> is an unacceptable substitute for "St. Peter"), >>>>> >>>>> I do receive email at stpeter@gmail.com intended for >>>>> st.peter@gmail.com but that's a separate topic... >>>> >>>> One that is relevant because it "works" as a side-effect of a >>>> decision Google has made about mailbox name equivalence, a >>>> decision that, IMO, will sooner or later get someone into a lot >>>> of trouble and, more important, a decision and matching rule >>>> that PRECIS, AFAICT, does not allow and that IDNA unambigiously >>>> forbids. >>>> >>>>>> it also points out the >>>>>> dangers of using Basic Latin script examples to illustrate >>>>>> situations in which even more extended Latin script, much less >>>>>> other scripts, may raise more complex issues. Because IDNA >>>>>> is essentially a workaround because changing the DNS >>>>>> comparison rules was impractical for several reasons, we >>>>>> ended up using toCaweFold to map characters and strings into >>>>>> others in IDNA2003 but PRECIS implementations that do not >>>>>> have the same constraints would, in general, be better off >>>>>> confining the use of toCaseFold, or even toLowerCase, to >>>>>> comparison operations. >>>>> >>>>> Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does >>>>> it make sense for this nickname specification to differ in >>>>> this respect from the published RFCs? Shall we file errata >>>>> against those documents? (This might apply only to RFC 7613, >>>>> which says to apply case folding as part of the enforcement >>>>> process - when exactly to apply case folding is not stipulated >>>>> by RFC 7564.) >>>> >>>> To the extent to which this is a "botched that because the WG >>>> didn't understand the issues well enough" conclusion, it would >>>> be entirely reasonable to generate an updating RFC that repairs >>>> 7613 and/or 7564, even doing so in an addendum to >>>> precis-nickname if that is the only way to do that >>>> expeditiously. Per the above, we really don't want to give >>>> library routine writers bad instructions. As I understand it, >>>> the current position of the RFC Editor and IESG is that >>>> technical specification errors discovered in retrospect or after >>>> people start using a spec are not appropriate topics for errata. >>>> If the WG is not willing to do any of those things, then I >>>> suggest that precis-nickname at least needs to contain a very >>>> clear warning notice about this situation (see my response to >>>> your question 1 below). >>> >>> I think we'll probably need to fix 7613 and 7564. I am hoping we can fix >>> nickname now so that it is less incorrect than the other two. That >>> doesn't necessarily mean we won't need to also further fix nickname >>> later on. >>> >>> Granted, we were supposed to avoid this problem by working on all of the >>> PRECIS specs simultaneously. Clearly we have not avoided the problem, so >>> we need to solve it one way or another. If that means bis for them all, >>> we need to deal with it. >>> >>>>>> (3) Because toCaweFold loses information when used for more >>>>>> than comparison (for comparison, it merely contributes to >>>>>> what some people would consider false positives for matching) >>>>>> involves some controversial decisions and, because of >>>>>> stability requirements, cannot be changed even if the >>>>>> controversies are resolved in other ways, we end up with, >>>>>> e.g., >>>>>> toCaseFold ("Nuß") -> "nuss" >>>>>> which is considered an acceptable transformation in some >>>>>> places that identify themselves as speaking/using German and >>>>>> two different unacceptable errors in others. Again, this will >>>>>> almost always be much more serious if the transformation is >>>>>> used to map and replace strings than if it is used to compare >>>>>> (fwiw, that particular example is part of a continuing >>>>>> disagreement between IDNA2008 and, among others, German >>>>>> domain registry authorities on one side and UTC and UTR 46 on >>>>>> the other). >>>>> >>>>> Agreed. >>>> >>>> See "warning notice" comment above and question 1 response below. >>>> >>>>> (4) If the motivation is really to avoid confusion, the >>>>>> correct confusion-blocking rule for Latin script (but not >>>>>> others) and many languages that use it (but certainly not >>>>>> all) involves moving beyond toCaseFold and treating all >>>>>> "decorated" characters (characters normally represented by >>>>>> glyphs consisting of a Basic Latin character and one or more >>>>>> diacritical or equivalent markings) compare equal to their >>>>>> base characters, e.g., "á" not only matches "Á" but also >>>>>> "a" and "A" and, as an unfortunate side-effect, maybe "À" >>>>>> and "à" as well. This is bad news for languages in which >>>>>> decorated Latin characters are used to represent phonetically >>>>>> and conceptually different characters, not just pronunciation >>>>>> variations. I am not qualified to evaluate "how bad". In >>>>>> addition, extrapolations from this principle about Latin >>>>>> script to unrelated scripts will almost certainly lead to >>>>>> serious errors and/or additional confusion. >>>>> >>>>> I would not be comfortable going that far... >>>> >>>> In case it isn't clear, I would not be either. But it is where >>>> getting sloppy about this stuff could easily take us. It is >>>> worth noting that it also identifies one of the difficulties >>>> with doing a global system to be applied to many types of >>>> applications (like the PRECIS work) and then applying it in user >>>> interface software that end users will expect to be localized to >>>> their assumptions because it has been mapped or translated into >>>> their language (if one normally speaks Upper Slobbovian but has >>>> some familiarity with English, an application interface in >>>> English will probably be expected to be "foreign", odd, and >>>> maybe even inconsistent with whatever expectations exist. But, >>>> if the interface is in Upper Slobbovian, the natural and >>>> reasonable assumption will be the matching should conform to >>>> normal Upper Slobbovian conventions. FWIW, a matching rule >>>> that says: >>>> >>>> (i) Two instances of a base character with the same >>>> diacritical mark(s) match. >>>> (ii) Two instances of a base character with different >>>> diacritical mark(s) do not match. >>>> (iii) Two instances of a base character, one with >>>> diacritical mark(s) and the other without any decoration >>>> match. >>>> >>>> Is precisely correct and normal behavior for at least one >>>> language that uses Latin script. It is also the normal practice >>>> for at least one Latin script transcription system that is used >>>> by a large fraction of a billion people (maybe more). >>> >>> That is indeed sobering. >>> >>>>>> More on this and Tom's question below... >>>>>> >>>>>>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>>>>>> Peter, Alexey, >>>>>>>> >>>>>>>> I think there is an ambiguity in the specification of case >>>>>>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>>>>>> ... >>>>>>>> But there are 55 code points in Unicode 7.0.0 that change >>>>>>>> under default case folding that are neither uppercase nor >>>>>>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>>>>>> suspect this stems from a confusion between Unicode case >>>>>>>> mapping and case folding. >>>> >>>> In the context of the above, a different way to say the same >>>> thing is that people are looking at toCaseFold and assuming (and >>>> explaining things in terms of) toLowerCase. toCaseFold works >>>> the way it is expected to and those 55 code points are, more or >>>> less, collateral damage to get to a matching algorithm that >>>> favors false positives over false negatives and various edge >>>> cases (including in "edge cases" languages spoken by, and script >>>> variations used by, millions of people). >>> >>> Sadly I suspect that is an accurate description of the current state of >>> affairs (modulo my comment above about PRECIS WG discussions at one or >>> more IETF meetings). >>> >>>>> ... >>>>> After all that, I have 3 questions: >>>> >>>> Personal opinions about answers... >>>> >>>>> (1) Is my proposed text enough of a clarification that we >>>>> should make that change before the nickname I-D is published >>>>> as an RFC? >>>> >>>> I think the clarification is an improvement and is important >>>> enough to incorporate (I know that is the answer to a slightly >>>> different question). >>>> >>>> However, I think it is inadequate without a serious warning >>>> about the situation. >>> >>> Yes. >>> >>>> That warning could appear in either this >>>> document or RFC 7613 (or 7613bis) with a pointer from the other, >>>> but, unless you want to revise 7613 now, this one is handy. >>> >>> I suspect that we need to revise 7613. I suspect that we might also need >>> to revise 7564 (at least with respect to the order in which operations >>> are applied, since there has been some confusion among implementers). >>> >>> Well, we always knew that we would need to revise them. Just not so >>> soon. >>> >>>> Comment about possible text below. >>>> >>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>> folding is applied only as part of comparison and not as part >>>>> of enforcement? If so, should we make that change before this >>>>> document is published as an RFC? >>>> >>>> Yes. If something is used for "enforcement", it should be lower >>>> casing or something else that can be explained to people who are >>>> ordinarily familiar with one or more of the scripts that make >>>> case distinctions. >>>> >>>> However, viewed in the light of this discussion, the whole >>>> "enforcement" concept becomes a little dicey, especially if, as >>>> I believe but don't have time to verify, the transformations >>>> performed by toLowerCase are not a proper subset of those >>>> performed by toCaseFold. >>> >>> My initial thought is that case mapping doesn't belong in the nickname >>> enforcement operation at all - only in the comparison operation. >>> >>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>> only as part of comparison and not as part of enforcement? >>>> >>>> I think that is necessary. Following up on the comment above, I >>>> would prefer that the current Section 3.2.2 (3) of RFC 7613 >>>> either point to Unicode Lower Casing or contain a warning along >>>> the lines of that below. >>> >>> Unlike the nickname profile (which I think can be cleaned up by moving >>> the case mapping rule to the comparison operation and continuing to use >>> Unicode Default Case Folding), I think you are right that for the >>> UsernameCaseMapped profile we probably want Unicode Lower Casing. Thus >>> the likely need, sooner rather than later, for 7613bis. >>> >>>> >>>> ---------- >>>> >>>> --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre >>>> wrote: >>>> >>>>> This issue has greater urgency now because >>>>> draft-ietf-precis-nickname is now in AUTH48... >>>>> >>>>> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >>>>> >>>>>> After all that, I have 3 questions: >>>>>> >>>>>> (1) Is my proposed text enough of a clarification that we >>>>>> should make that change before the nickname I-D is published >>>>>> as an RFC? >>>>> >>>>> I think so. >>>> >>>> See above. >>>> >>>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>>> folding is applied only as part of comparison and not as part >>>>>> of enforcement? If so, should we make that change before this >>>>>> document is published as an RFC? >>>>> >>>>> Although it seems to be the case that Unicode case folding is >>>>> primarily designed for the purpose of matching (i.e., >>>>> comparison), >>>> >>>> "Seems" is a little weak. The Unicode Standard is really quite >>>> specific about that. >>>> >>>>> I have a concern that applying the PRECIS case >>>>> mapping rule after applying the normalization and >>>>> directionality rules might have unintended consequences that >>>>> we haven't had a chance to consider yet. The PRECIS framework >>>>> expresses a preference (actually a hard requirement) for >>>>> applying the rules in a particular order. We made a late >>>>> change to the username profiles (RFC 7613), such that width >>>>> mapping is applied first (in order to accommodate fullwidth >>>>> and halfwidth characters in certain East Asian scripts). >>>>> Making a late change to the nickname profile also concerns me, >>>>> even though both of these late changes seem reasonable on the >>>>> face of it. I will try to find time to think about this >>>>> further in the next 24 hours. >>>> >>>> First, a hint for the consideration process: there is a reason >>>> why Unicode now supports a unified case folding and >>>> normalization operation. My recollection is that it is not only >>>> more efficient to perform both operations at once (rather than >>>> looking in one table and then the other), but that there are >>>> some order-dependent or priority-dependent cases. >>>> >>>> The very fact that this issue exists (and is coming up again) >>>> this late in the process (7613 published in August, WG winding >>>> down and not, e.g., meeting next week) calls at least the PRECIS >>>> quality of review and some fairly fundamental model issues into >>>> question. I first raised that issue a rather long time ago but >>>> have continued to hope that we have an approximation to "good >>>> enough" without going back and rethinking everything. >>>> >>>> The right solution, IMO, is that, if RFC 7613 is to rationalize >>>> or explain the operation in terms of converting upper case >>>> characters to lower case, then it should be using toLowerCase >>>> because that is what the operation does. After a quick look at >>>> 7613, amending/updating it to simply convert to lower case would >>>> be straightforward (and would not raise the ordering issue >>>> called out above). It would presumably require another IETF >>>> Last Call, however and I'd hope we would see some serious >>>> discussion within the WG (and with UTC) before making the change >>>> and about how it is explained. >>>> >>>> If we are not willing to make a change >>> >>> I'm willing. It would, as you note, require some careful thinking and >>> review to make sure that we got it (more) right this time. >>> >>>> that significant and/or >>>> if we conclude that the WG (and perhaps the IETF) have >>>> completely run out of energy for dealing with i18n issues [1], >>>> then I suggest that we introduce some additional text. I've >>>> just spent a half-hour trying to find the AUTH48 copy of >>>> precis-nickname (aka RFC-to-be-7700), but the RFC Editor has >>>> apparently changed naming conventions and the various queue >>>> entry pages all point to the -19 I-D and not the current working >>>> copy so I can't try to match text and insertion point to what is >>>> there already. >>> >>> http://www.rfc-editor.org/authors/rfc7700.txt >>> >>>> The suggestion is a patch (and a hack), not a >>>> good fix but something like it is probably the least drastic >>>> measure that would yield something that doesn't contain >>>> unexplained known defects. >>>> >>>> Rough version of suggested text (possibly to go after your >>>> revised paragraph and following up my comments in my 1 October >>>> note). Some of the terminology needs checking which I can do if >>>> you want to go this route: >>>> >>>> 'Users of this specification should note that the >>>> concept of "lower case conversion" is somewhat elusive >>>> and more dependent on the conventions of different >>>> languages and notation systems that use the same script >>>> than may appear obvious at first glance, especially if >>>> that glance is at Basic Latin characters (i.e., the >>>> ASCII letter repertoire). Unicode provides two >>>> different mapping procedures that produce lower-case >>>> characters, but they have different effects and results >>>> for many characters. The more conservative one, >>>> typically appropriately applicable when lower case forms >>>> are needed, is actual lower-casing (embodied in the >>>> Unicode operation toLowerCase). A more radical >>>> operation, normally suitable only for string matching in >>>> situations in which it is better to consider uncertain >>>> cases as matching than to treat them as distinct, is >>>> called "Case Folding" (Unicode operation toCaseFold). >>>> While the two operations will often produce the same >>>> results, Case Folding maps some lower case characters >>>> into others and performs other transformations that may >>>> be intuitively reasonable and expected for some users >>>> and quite astonishing (or just wrong) to others. There >>>> may be no practical alternative, especially if the >>>> operations are to be used for mapping or enforcement, to >>>> developers of PRECIS-dependent understanding that the >>>> cases in which the two yield different results require >>>> careful understanding of the relevant user base and its >>>> needs [2].' >>> >>> Thanks. >>> >>> I am not sure if we need something like that if we move case mapping >>> (here, case folding) to the comparison operation only - but something >>> like that might still be appropriate. >>> >>>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>>> only as part of comparison and not as part of enforcement? >>>>> >>>>> That is less urgent so I suggest that we address the nickname >>>>> spec first. >>>> >>>> Unless you (or someone else here) have a plausible plan to >>>> continue and revitalize the WG and assign it that revision work >>>> (and bring everyone actively participating up to the level >>>> needed to easily understand this discussion thread and feel >>>> embarrassed for not spotting the problems), I think we need to >>>> assume that this is our last shot. Absent an active and >>>> committed WG, "do this first" could easily be equivalent to >>>> "don't get around to the other, ever". >>> >>> As mentioned, I don't want to have broken RFCs out there. >>> >>>> I think that the particular set of issues that started this >>>> thread as a known defect in the PRECIS specs, both nickname and >>>> 7613 and that we are obligated to either fix the problems or at >>>> least explain them. The above warning text is an attempt to >>>> explain and identify the problems even if it does not actually >>>> provide a solution. If it were published as part of >>>> precis-nickname, it could include a statement to the effect that >>>> it should also be treated as an update to 7613 or, if the IESG >>>> and RFC Editor would agree in advance to accept, rather than >>>> bury, the thing, I suppose we could publish it in >>>> precis-nickname and create an erratum to 7613 indicating that it >>>> should have included some form of that statement. Neither >>>> option implies a huge amount of work to update 7613. But I >>>> think that making the changes of (2) without doing anything >>>> about (3) makes the two documents inconsistent with each other >>>> and that would be an additional known defect. >>>> >>>> Procedural question: given that precis-nickname is in AUTH48 as >>>> of yesterday and I don't see anything blocking publication next >>>> week if you and Barry sign off on the revised text that the WG >>>> hasn't seen, >>> >>> There is no revised text yet. That's why we're having this discussion. >>> >>>> does someone need to file a pro forma objection/ >>>> appeal to block that until this is sorted out and the WG has a >>>> chance to review proposed publication text? >>> >>> I see no reason to invoke the specter of appeals quite yet. Everyone is >>> working in good faith to do the right thing and get this mess cleaned >>> up. >>> >>>> [1] I believe our collective inability to deal with the >>>> within-script character forms that do not normalize to each >>>> other because of language-dependent or other usage factors can >>>> be taken as evidence of having run out of energy, >>> >>> Or in my case simple ignorance of some of the relevant issues and >>> examples. It's not easy to know about all of this. >>> >>>> but it is >>>> probably in the interest of finishing the PRECIS work to try to >>>> treat that as a separate issue. >>> >>> Probably. >>> >>>> [2] Not unlike the reason to differentiate between NFC and NFKC >>>> and understand the effects of each. >>> >>> Another thing that's not easy to grok in fulness. >>> >>> Peter >>>