From nobody Mon Nov 2 00:12:14 2015 Return-Path: X-Original-To: precis@ietf.org Delivered-To: precis@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id EE2831A8908; Mon, 2 Nov 2015 00:12:10 -0800 (PST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: X-Test-IDTracker: no X-IETF-IDTracker: 6.7.2 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <20151102081210.32066.68736.idtracker@ietfa.amsl.com> Date: Mon, 02 Nov 2015 00:12:10 -0800 Archived-At: Cc: precis@ietf.org Subject: [precis] I-D Action: draft-ietf-precis-mappings-12.txt X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 08:12:11 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Preparation and Comparison of Internationalized Strings Working Group of the IETF. Title : Mapping characters for PRECIS classes Authors : Yoshiro YONEYA Takahiro Nemoto Filename : draft-ietf-precis-mappings-12.txt Pages : 13 Date : 2015-11-01 Abstract: The framework for preparation, enforcement, and comparison of internationalized strings ("PRECIS") defines several classes of strings for use in application protocols. Because many protocols perform case-sensitive or case-insensitive string comparison, it necessary to define methods for case mapping. In addition, both the Internationalized Domain Names in Applications (IDNA) and the PRECIS problem statement describe mappings for internationalized strings that are not limited to case, but include width mapping and mapping of delimiters and other special characters that can be taken into consideration. This document provides guidelines for designers of PRECIS profiles and describes several mappings that can be applied between receiving user input and passing permitted code points to internationalized protocols. In particular, this document describes both locale-dependent and context-depending case mappings as well as additional mappings for delimiters and special characters. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-precis-mappings/ There's also a htmlized version available at: https://tools.ietf.org/html/draft-ietf-precis-mappings-12 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-precis-mappings-12 Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org. Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Nov 2 01:33:22 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DBBD51B35B9 for ; Mon, 2 Nov 2015 01:33:20 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.598 X-Spam-Level: X-Spam-Status: No, score=0.598 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_EQ_JP=1.244, HOST_EQ_JP=1.265, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GNQMVZHUBk6G for ; Mon, 2 Nov 2015 01:33:19 -0800 (PST) Received: from mail.kmd.keio.ac.jp (mail.kmd.keio.ac.jp [IPv6:2001:200:167:2e90::164]) by ietfa.amsl.com (Postfix) with ESMTP id B42A11B35B7 for ; Mon, 2 Nov 2015 01:33:18 -0800 (PST) Received: from t20010c4000003024982351dc3e90d577.v6.meeting.ietf94.jp (t20010c4000003024982351dc3e90d577.v6.meeting.ietf94.jp [IPv6:2001:c40:0:3024:9823:51dc:3e90:d577]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kmd.keio.ac.jp (Postfix) with ESMTPSA id 846117FCC4 for ; Mon, 2 Nov 2015 18:33:17 +0900 (JST) (envelope-from t.nemo10@kmd.keio.ac.jp) From: Takahiro Nemoto Content-Type: multipart/signed; boundary="Apple-Mail=_6B2E1638-212F-475A-8EFF-C63A6BB8B870"; protocol="application/pkcs7-signature"; micalg=sha1 Date: Mon, 2 Nov 2015 18:33:11 +0900 References: <20151102081210.32066.68736.idtracker@ietfa.amsl.com> To: precis@ietf.org Message-Id: <3FAB3F2D-E822-4442-8E33-A8A904D9CB04@kmd.keio.ac.jp> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) X-Mailer: Apple Mail (2.2104) Archived-At: Subject: [precis] Fwd: I-D Action: draft-ietf-precis-mappings-12.txt X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 09:33:21 -0000 --Apple-Mail=_6B2E1638-212F-475A-8EFF-C63A6BB8B870 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Dear all, Yoneya-san and I submitted the latest version of precis mappings = document. We revised the document to address comments from IETF LC and by the IESG = review. Please let me know if you have any questions and comments. Your feedbacks will be greatly appreciated. Regards, Nemo > =E8=BB=A2=E9=80=81=E3=81=95=E3=82=8C=E3=81=9F=E3=83=A1=E3=83=83=E3=82=BB= =E3=83=BC=E3=82=B8=EF=BC=9A >=20 > =E5=B7=AE=E5=87=BA=E4=BA=BA: internet-drafts@ietf.org > =E4=BB=B6=E5=90=8D: [precis] I-D Action: = draft-ietf-precis-mappings-12.txt > =E6=97=A5=E4=BB=98: 2015=E5=B9=B411=E6=9C=882=E6=97=A5 17:12:10 JST > =E5=AE=9B=E5=85=88: > CC: precis@ietf.org >=20 >=20 > A New Internet-Draft is available from the on-line Internet-Drafts = directories. > This draft is a work item of the Preparation and Comparison of = Internationalized Strings Working Group of the IETF. >=20 > Title : Mapping characters for PRECIS classes > Authors : Yoshiro YONEYA > Takahiro Nemoto > Filename : draft-ietf-precis-mappings-12.txt > Pages : 13 > Date : 2015-11-01 >=20 > Abstract: > The framework for preparation, enforcement, and comparison of > internationalized strings ("PRECIS") defines several classes of > strings for use in application protocols. Because many protocols > perform case-sensitive or case-insensitive string comparison, it > necessary to define methods for case mapping. In addition, both the > Internationalized Domain Names in Applications (IDNA) and the PRECIS > problem statement describe mappings for internationalized strings > that are not limited to case, but include width mapping and mapping > of delimiters and other special characters that can be taken into > consideration. This document provides guidelines for designers of > PRECIS profiles and describes several mappings that can be applied > between receiving user input and passing permitted code points to > internationalized protocols. In particular, this document describes > both locale-dependent and context-depending case mappings as well as > additional mappings for delimiters and special characters. >=20 >=20 > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-ietf-precis-mappings/ >=20 > There's also a htmlized version available at: > https://tools.ietf.org/html/draft-ietf-precis-mappings-12 >=20 > A diff from the previous version is available at: > https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-precis-mappings-12 >=20 >=20 > Please note that it may take a couple of minutes from the time of = submission > until the htmlized version and diff are available at tools.ietf.org. >=20 > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ >=20 > _______________________________________________ > precis mailing list > precis@ietf.org > https://www.ietf.org/mailman/listinfo/precis --Apple-Mail=_6B2E1638-212F-475A-8EFF-C63A6BB8B870 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKUzCCBOYw ggLOoAMCAQICCl0jroAl+7c0vpAwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMCSlAxFTATBgNV BAoMDFdJREUgUHJvamVjdDEfMB0GA1UEAwwWbWVtYmVycyBvcmllbnRlZCBDQSAwMzAeFw0xNTA2 MjQwNDE4NDhaFw0xNzA2MzAyMzU5NTlaMG0xCzAJBgNVBAYTAkpQMRUwEwYDVQQKDAxXSURFIFBy b2plY3QxHzAdBgNVBAMMFjE1MzcgVGFrYWhpcm8gTmVtb3RvIDQxJjAkBgkqhkiG9w0BCQEWF3Qu bmVtbzEwQGttZC5rZWlvLmFjLmpwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtV4V 6xVvC26RiAerOT4jzv3akEADrmJs+PCBwj+/LI78liZMg/CMXs3Fpri/D5h7b/9VzZSgGcNpux2r OSMWiT0zMnstuefSIHvp0oEF5f9hQDZYGlbojG11A/wqewC8d0Y/9Fflm2AylgLee0tUpnap0E+o ZFz46wKxmC1cl74UqVsfi8IQgS+5wbTygT5fcVGeQpzevFMdf3Bd4ZuNNf1FLbAVqGo9P7g5jhCv NMWFDNhL8odmcfeVhX1JOVYe16fsWePZki1YeENZX9IQOMmHbF3qg5yPLUr76JiW3GhaP1p9649r RtLFGVypfKZ3r4fUxb3OMsdMeil1dtao/QIDAQABo4GvMIGsMAsGA1UdDwQEAwIEsDAiBgNVHREE GzAZgRd0Lm5lbW8xMEBrbWQua2Vpby5hYy5qcDAvBgNVHR8EKDAmMCSgIqAghh5odHRwOi8vbW9j YS53aWRlLmFkLmpwL2NybC5jcmwwHwYDVR0jBBgwFoAUlpT82BlfTL9FMdyssrhbVQdMiw8wJwYD VR0lBCAwHgYIKwYBBQUHAwQGCCsGAQUFBwMDBggrBgEFBQcDAjANBgkqhkiG9w0BAQsFAAOCAgEA Mq8UBxidKprmf9IPx7GLt645beGZgRGbD74VegiFgFPum+Z6ksYKzxHB4zjsGqHzZ3VPzE5WCWjn 8iPEEcc2CSeg75/sZ2HSuNZUMKFVl20hQ+9SHw+in1iM729tnTfvQe0Btby5JMNjCUYSb9mVhpcH 0h3Ii0Xh0ZQ9gdBDG0vsWvhcJuZ0hZ/LvW7v7R+FvWEXJKSgTjsZEjwso7lMYpT769tjNtuzXFSe rsb1K6iPsws10YGhV/Kz+mO2S7rpR035tv7fEaXAMiJz6sxalejIROutC8kinwe1PASn1qs4P2Vi H7BM2GnxNlc+r0SlxT/ibhKTxsi8WKggKQscEs9FgJv3c+F0S1XIqMiKXgjGJMdoZv4cIJ71h5NH pM6RSRvO7W8oqwc+EaynN0XdTCpunmI+z+AJvKGpr+lVdVNKqNPsUTb9aCf9wJGTSlwv+OhdD4g0 NEvorTiZ+DbS3/EbFXIwDgj8v8SdkWLKBshTALXG2wgm7ewi4h3bxQbngToDPd4iunu6iQpmeXd0 FPChwKTXtx7VVCZSHWB08U/PTvasmyPx4V6AZtXWbYz/BlZX+Cq9Y7MECOziluC0OSM/v0I1xDo+ AoTHy080xi1VrVTBoi68rxgxu4FIjKlGfg9sQgylRLHv6j/YNn3rGCD0RAXtT/5BnrhW8Xb6Zfgw ggVlMIIDTaADAgECAgsA3pfLDt1eq5DA7TANBgkqhkiG9w0BAQsFADA+MQswCQYDVQQGEwJKUDEV MBMGA1UECgwMV0lERSBQcm9qZWN0MRgwFgYDVQQDDA9XSURFIFJPT1QgQ0EgMDMwHhcNMTMwNjAz MTIwMDI0WhcNMjMwNjMwMTQwNTEyWjBFMQswCQYDVQQGEwJKUDEVMBMGA1UECgwMV0lERSBQcm9q ZWN0MR8wHQYDVQQDDBZtZW1iZXJzIG9yaWVudGVkIENBIDAzMIICIjANBgkqhkiG9w0BAQEFAAOC Ag8AMIICCgKCAgEAnX+tN/1KWgtexAc658h4NY5IvJbShg3gIpoITVP8IAFhcFSed6t7RQaKomEF P0NZxL4+rntl6F80MB/2freB2pGkCxA81tC1gRw8ZkgWMS9+qPZYOAW5hBOMjrykrSI7+NyGno7b L2Sso/b+FRmHASqWJ05LqOs64EX3g40sNjFNx2AHQXoas1HdWNvj8QpZipCPv1ZFdvUlu9eZ2CQj 7rd9bvUBGrdCuV2rBb1vn5LbtjHgYsw/Ic5f8PLlJ0BOPdYYRjtgHW1UNzERxGnambcH+dRE4L0+ ThCPNQiJu7nxBvkMG9t5K+1b/rF8BuvYJrM3BpYtc50y3YmAFTrqlkE85iPsGe1BP8/s6EphkSBe KBq1aYSds4rhiM+tr5dSCKoowiNR41OWC7kEbew5bmiM0MDqqhJx4QKs0LsgngjwdwR1gyCWrR4D waQn13H+fbEj9C4g3jy5wIS6wI/pgjfBtYZNlR6LbRwqmq98e5TLmbeooDcTpBQVlfozDVGib0u2 BT3toc1GZeXO1aL41xyeCnMaKdlxWLcymp1Pue7FJEQwW5J5vKkD08m5DE0IX0i9R2aHgm5tN4AJ BWSo2dc9w6HhD/PjugaMBhTIDdgkxibAjVxI8grPwDLdLWDwwhl7+ZPRTQ17Qe+nqjF/w3kOhiWY AKBTBMGcvt/ZpYECAwEAAaNdMFswHQYDVR0OBBYEFJaU/NgZX0y/RTHcrLK4W1UHTIsPMB8GA1Ud IwQYMBaAFDhrU7/rSxuLAPgA6CJlDQCm847qMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0G CSqGSIb3DQEBCwUAA4ICAQCDysFnIkZoio7HmwBZ0MaIv6REQtCweXy4P0LVka1HwqoqepCgOBZJ Y2MMXE6ZUxbuT+HEsskkiMjloh4xCRyL9ds3zZ6gMhDvx3pEgGhe7iusJI/bfCe10ZIGTcrWaZwo hN9ubbHXyC47QqDTh6OjEMosEjnWTUzm6H9NsoKQNYKQCCpP4bNJO1urUueHR74FuH5i6taI/RAm 1qdoRoeH7qwL2Lvg/kPLyJE3hUiNpw3VQSC29Tx9KUgvltodiwX6tu5oroM+MgywMWwgNlmKEYmq a9DWeXIBzA47m3l9BruoqXXdsUTzzF7K/bkflJWNIhCzaJfkFnJqGacuJiBFnjeqy0aOZ5gWvG3r BpJgGro8Pdgz/9NHTXe49zhFmR1eeDDNNmWL2U060NjwxCBZKe1UQs+7qiCKsybHwv7e43UB2LZo dD356WLtAb3jwslVHah2ZGzX2LihURz4s7UO8XiqFO0GAtcFbTj0if30ex7EHVe7Y+xg/MCmu2Dy 1P071fNrkZhEaZcrU1PVubfZQJG5mAEW3K4HDMwZl7SH624K/s8QpWZgBoz1FYvh3W0dj1y/i+dA i3hmwRlCx07+jUA4FhWHCI/8KxZLg/z4jj5F8BVI7FLOr1rgFOPdU2JjPDIaKIj0Ggrun7KtNRyE ect37auh6rCZTfL3SAwvUDGCAqUwggKhAgEBMFMwRTELMAkGA1UEBhMCSlAxFTATBgNVBAoMDFdJ REUgUHJvamVjdDEfMB0GA1UEAwwWbWVtYmVycyBvcmllbnRlZCBDQSAwMwIKXSOugCX7tzS+kDAJ BgUrDgMCGgUAoIIBJzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0x NTExMDIwOTMzMTJaMCMGCSqGSIb3DQEJBDEWBBRVjalQ6V6eARZ/sOm+ELmaXDTelzBiBgkrBgEE AYI3EAQxVTBTMEUxCzAJBgNVBAYTAkpQMRUwEwYDVQQKDAxXSURFIFByb2plY3QxHzAdBgNVBAMM Fm1lbWJlcnMgb3JpZW50ZWQgQ0EgMDMCCl0jroAl+7c0vpAwZAYLKoZIhvcNAQkQAgsxVaBTMEUx CzAJBgNVBAYTAkpQMRUwEwYDVQQKDAxXSURFIFByb2plY3QxHzAdBgNVBAMMFm1lbWJlcnMgb3Jp ZW50ZWQgQ0EgMDMCCl0jroAl+7c0vpAwDQYJKoZIhvcNAQEBBQAEggEASpwqSQcBhzFgFNdlAHms xuD/7jVf0mhWPDaUrFoHgPVsBuDE62pEI+mnBSs+3BVSJPza/qmLJQhIKBCqDt28yD8iyAwbK0+V J9WgYBuYL9j4jPi71MDPscj7TvUaMybu5OpQGuD0gB7mQgTrxJrFfQwuD1m2f1RMVOMbUw4WMAFa E61N57ydOfOzEv0UuHYRFzRpRR0ij50eP1lOdkmtz08H5Iec8OwYp9Y5A6PddwgAqgY31ymjY+pw n8aprLL6HGZ6Vcjsp//iqdC7Bq8lljdAZQQxU6upmI8gSAE5ZE1HIhv1jGo1jHBsiri14ei4zUi4 nU33rm9nLOvEmjo2EAAAAAAAAA== --Apple-Mail=_6B2E1638-212F-475A-8EFF-C63A6BB8B870-- From nobody Mon Nov 2 20:42:39 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D23CE1B2DAA for ; Mon, 2 Nov 2015 20:42:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.601 X-Spam-Level: X-Spam-Status: No, score=-1.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, GB_I_LETTER=-2, MANGLED_LIST=2.3, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FWYXuZ6LeWrI for ; Mon, 2 Nov 2015 20:42:32 -0800 (PST) Received: from mail-ob0-x231.google.com (mail-ob0-x231.google.com [IPv6:2607:f8b0:4003:c01::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5E18C1B2DB2 for ; Mon, 2 Nov 2015 20:42:27 -0800 (PST) Received: by obdgf3 with SMTP id gf3so4023833obd.3 for ; Mon, 02 Nov 2015 20:42:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=bsNcwZ7JIeUyVZz993LA6Mf+6Rm3KlXpGD6yTVEDxEE=; b=Y78JRIm9jVAT5GMezmGGezDrNWBLKz+4bvnyPu3XJ8AgU9bp3YIPmMdkwvK2u5XK00 9DPLZYgTzNrIKsPehqIT8HVNbrjd6KVUtexc0rJp7rsCviB0p6Y/yH7vbKzhDIIteAny shzs8pu0UCR0tL/vycqWrRLIWlRXs/kfRTaHg5MJvL+pRAbYE5LrDoElPVYy3ExMxFgJ 8GYuu85cBQGiYOoqz13wrRSz5g0nMczlk+nqTTHGWTb0JgLJpnH/72bYu/fpXHk6lrfk 65Q4Bq/w8AkZSITeijam4hBi9/WRU9bjDh+rqOWVsXt9w21/yh/G8CzzT8j6fJva8uFJ /9Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=bsNcwZ7JIeUyVZz993LA6Mf+6Rm3KlXpGD6yTVEDxEE=; b=gWUwlJNwSbxzYWx8XIcNkKvfZHqAXu82Qfoiu2SrsW87tKosFMi0BRgon9apITCEqm zg9LShbk4AONEl1HqM/UAXlpMKnA515F7teUP3VDG2uf6G0Ueyk7Y3EYYERBxgVV1VQy WZHjboIh/hC9ZSe5BFsyt7oMCPBTyIrwqdEx3BNKERCmpx/MUCIGDldQMv8s1+M1oGbP zAlujJF9sTPI0V1rG2RrSP+I6dHsjTIXZpiGVvt4/FZ/B2kyK1EUz6Hcs7iOs04fkfax EzKBuWtX8SSF9cpAtiZ1B1iUllVy25pVt6oT3TWdDyczsUnzGmZIq07HVZ4U0cw2L6ao 2RKg== X-Gm-Message-State: ALoCoQmYDmzLi2P0JqwHoXU9LAE0fPvusKsQA7XD9QZCT+Egx/f7V0nU0y2WEc7ATEdzv1o7XYXC X-Received: by 10.182.129.138 with SMTP id nw10mr10225787obb.24.1446525746670; Mon, 02 Nov 2015 20:42:26 -0800 (PST) Received: from aither.local (c-73-34-202-214.hsd1.co.comcast.net. [73.34.202.214]) by smtp.googlemail.com with ESMTPSA id v76sm10419815oie.18.2015.11.02.20.42.24 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Nov 2015 20:42:25 -0800 (PST) To: John C Klensin , Tom Worster , Alexey Melnikov References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> From: Peter Saint-Andre Message-ID: <56383B2F.6080505@andyet.net> Date: Mon, 2 Nov 2015 21:42:23 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <563143B9.7020707@andyet.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 04:42:38 -0000 For ease of reviewing only, and with no presumption that these proposed changes have been accepted by anyone, I have asked the RFC Editor to provisionally update the document in AUTH48 as outlined in the messages I have sent to the list. The resulting file is here: http://www.rfc-editor.org/authors/rfc7700.txt Despite those caveats, if at all possible I would prefer to find an acceptable solution for publishing this RFC now without undue further delays (in part because draft-ietf-simple-chat has been held on this document for almost 3 years!). That does not mean, as I said earlier in this thread, that I want to have broken RFCs out there, but I think we can fix this one acceptably now and then update it again in strict coherence with updates to RFC 7564 and RFC 7613. I am committed to getting things right, but I am also committed to not holding up other people's work for years and years on end. Peter On 10/28/15 3:52 PM, Peter Saint-Andre wrote: > Example 7 needs to be corrected, too, in accordance with CaseFolding.txt. > > On 10/28/15 2:54 PM, Peter Saint-Andre wrote: >> And here is another correction in Section 3... >> >> OLD >> >> Regarding examples 5, 6, and 7: applying Unicode Default Case Folding >> to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL LETTER >> SIGMA (U+03C3), and doing so during comparison would result in >> matching the nicknames in examples 5 and 6; however, because the >> PRECIS mapping rules do not account for the special status of GREEK >> SMALL LETTER FINAL SIGMA (U+03C2), the nicknames in examples 5 and 7 >> or examples 6 and 7 would not be matched. >> >> NEW >> >> Regarding examples 5, 6, and 7: applying Unicode Default Case Folding >> to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL LETTER >> SIGMA (U+03C3), and the same is true of GREEK SMALL LETTER FINAL >> SIGMA (U+03C2); therefore, the comparison operation defined in >> Section 2.4 would result in matching of the nicknames in examples 5, >> 6, and 7. >> >> On 10/28/15 2:06 PM, Peter Saint-Andre wrote: >>> I propose the following text changes: >>> >>> ### >>> >>> OLD >>> >>> 3. Case Mapping Rule: Uppercase and titlecase characters MUST be >>> mapped to their lowercase equivalents using Unicode Default Case >>> Folding as defined in the Unicode Standard [Unicode] (at the >>> time >>> of this writing, the algorithm is specified in Chapter 3 of >>> [Unicode7.0]). In applications that prohibit conflicting >>> nicknames, this rule helps to reduce the possibility of >>> confusion >>> by ensuring that nicknames differing only by case (e.g., >>> "stpeter" vs. "StPeter") would not be presented to a human user >>> at the same time. >>> >>> NEW >>> >>> 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, >>> as defined in the Unicode Standard [Unicode] (at the time >>> of this writing, the algorithm is specified in Chapter 3 of >>> [Unicode7.0]). The primary result of doing so is that uppercase >>> characters are mapped to lowercase characters. In applications >>> that prohibit conflicting nicknames, this rule helps to reduce >>> the possibility of confusion by ensuring that nicknames >>> differing only by case (e.g., "stpeter" vs. "StPeter") would not >>> be presented to a human user at the same time. >>> >>> ### >>> >>> (The foregoing was previously sent to the list.) >>> >>> ### >>> >>> OLD >>> >>> 2.3. Enforcement >>> >>> An entity that performs enforcement according to this profile MUST >>> prepare a string as described in Section 2.2 and MUST also apply the >>> rules specified in Section 2.1. The rules MUST be applied in the >>> order shown. >>> >>> After all of the foregoing rules have been enforced, the entity MUST >>> ensure that the nickname is not zero bytes in length (this is done >>> after enforcing the rules to prevent applications from mistakenly >>> omitting a nickname entirely, because when internationalized >>> characters are accepted, a non-empty sequence of characters can >>> result in a zero-length nickname after canonicalization). >>> >>> 2.4. Comparison >>> >>> An entity that performs comparison of two strings according to this >>> profile MUST prepare each string and enforce the rules as specified >>> in Sections 2.2 and 2.3. The two strings are to be considered >>> equivalent if they are an exact octet-for-octet match (sometimes >>> called "bit-string identity"). >>> >>> NEW >>> >>> 2.3. Enforcement >>> >>> An entity that performs enforcement according to this profile MUST >>> prepare a string as described in Section 2.2 and MUST also apply the >>> following rules specified in Section 2.1 in the order shown: >>> >>> 1. Additional Mapping Rule >>> 2. Normalization Rule >>> 3. Directionality Rule >>> >>> After all of the foregoing rules have been enforced, the entity MUST >>> ensure that the nickname is not zero bytes in length (this is done >>> after enforcing the rules to prevent applications from mistakenly >>> omitting a nickname entirely, because when internationalized >>> characters are accepted, a non-empty sequence of characters can >>> result in a zero-length nickname after canonicalization). >>> >>> 2.4. Comparison >>> >>> An entity that performs comparison of two strings according to this >>> profile MUST prepare each string as specified in Section 2.2 and >>> MUST apply the following rules specified in Section 2.1 in the order >>> shown: >>> >>> 1. Additional Mapping Rule >>> 2. Case Mapping Rule >>> 3. Normalization Rule >>> 4. Directionality Rule >>> >>> The two strings are to be considered equivalent if they are an exact >>> octet-for-octet match (sometimes called "bit-string identity"). >>> >>> ### >>> >>> In addition, some variation on John's proposed text about toLowerCase >>> vs. toCaseFold might be appropriate at the end of Section 4; however, >>> I'm still not sure that is necessary if we move the case mapping rule to >>> the comparison operation. >>> >>> Peter >>> >>> On 10/27/15 8:09 PM, Peter Saint-Andre wrote: >>>> On 10/27/15 11:32 AM, John C Klensin wrote: >>>>> Response to Monday's note immediately below; response to today's >>>>> follows it. My apologies, but it is probably important to read >>>>> both. My further apologies for the length of this note, but I >>>>> think we are in deep trouble here, >>>> >>>> Internationalization always seems to be a matter of how deep the >>>> trouble >>>> is... >>>> >>>>> trouble that is aggravated by >>>>> precis-mappings and precis-nickname both being post-approval and >>>>> that, as far as I know, there are no future plans for PRECIS >>>>> work (having precis-nickname in AUTH48 just emphasizes that -- >>>>> see comment at end). >>>> >>>> We had not planned to work on PRECIS because we thought we were done >>>> for >>>> awhile. If that's not the case and we need to fix things, then so be >>>> it. >>>> Whether there is sufficient and continued energy for such work is >>>> another question. Personally I don't want us to have broken RFCs out >>>> there. >>>> >>>>> --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - >>>>> &yet wrote: >>>>> >>>>>> My apologies for the delayed reply. Comments inline. >>>>> >>>>> A few remarks below... I can't tell whether we disagree or >>>>> whether at least one of us, probably me, are not being >>>>> adequately clear. (Material on which we fairly clearly agree >>>>> elided.) >>>>> >>>>> >>>>>> On 10/1/15 7:50 AM, John C Klensin wrote: >>>>>>> --On Wednesday, September 30, 2015 15:16 -0600 Peter >>>>>>> Saint-Andre - &yet wrote: >>>>>> ... >>>>>>> Peter, >>>>>>> >>>>>>> While your proposed text is an improvement, >>>>>> >>>>>> Happy to hear it. All I intended was a slight clarification. >>>>> >>>>> But I'm not certain we are there yet... >>>> >>>> Agreed. The text I proposed addressed only a very small part of the >>>> problem. >>>> >>>>>>> the desire of many >>>>>>> people for a magic "just tell me what to do" formula, one that >>>>>>> lets them avoid understanding the issues, may call for a >>>>>>> little more: >>>>>> >>>>>> There is always a need for more when it comes to i18n. >>>>> >>>>> But I think it is a little more that that. I've heard several >>>>> times, including in PRECIS meetings, requests for "just tell me >>>>> what to do and make sure it isn't complicated" (or "I don't want >>>>> to have to think about, much less understand, the issues"). We >>>>> can debate whether giving in to those requests in the I18n case >>>>> is wise. I think it leads directly to conclusions equivalent to >>>>> "I understand my own script and writing system (or think I do) >>>>> and therefore, since all writing systems must be pretty much the >>>>> same, I understand all of the core issues in terms of my script >>>>> and understanding". That, in turn, leads directly to the "how >>>>> do you spell 'Zürich'?" and "all spellings of 'Zuerich' should >>>>> be treated as equivalent" discussion that sounded like they >>>>> dominated a BOF at IETF 93. >>>>> >>>>> Now I actually think it is reasonable for someone to ask for a >>>>> library that will do the job most of the time and that will >>>>> almost never cause their users or customers to get angry at >>>>> them. But, if we are going to call what we do "standards", they >>>>> should contain sufficient information that would-be library >>>>> authors can know what to do ... or understand that they are in >>>>> over their heads. And, for these particular cases, we may need >>>>> to explain, or help the library authors explain, why some cases >>>>> will fail and, indeed, get users mad at vendors. >>>>> >>>>> >>>>>>> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >>>>>>> primary result of doing so is that uppercase characters are >>>>>>> mapped to lowercase characters" is true for toCaseFold, >>>>>> >>>>>> By "primary" I meant two things: (1) lowercasing is what >>>>>> happens to the preponderance of code points and (2) this is >>>>>> the result that most people care about. >>>>> >>>>> If I parse the above correctly, I think you are wrong. I think >>>>> what most people want, care about, and think they are getting, >>>>> is lower case conversion, i.e., an operation that preserves >>>>> lower case characters and converts upper case characters to the >>>>> equivalent lower case. toCaseFold isn't that operation. It is >>>>> a much more complex and subtle operation that, as well as >>>>> converting upper case characters to lower case, sometimes >>>>> converts lower case characters to different lower case >>>>> characters (or strings of them). It also requires a fairly good >>>>> understanding of Unicode (not just a relevant script) and >>>>> historical Unicode decisions to predict its behavior and to have >>>>> any hope of explaining that behavior to users. If one is >>>>> trying to compare (as distinct from converting), then toCaseFold >>>>> may be exactly what it wanted. but it is really hard to explain >>>>> or justify that in terms of "nicknames" or "aliases", which are >>>>> about conversion. And, if one hopes to explain what is going >>>>> on to users in terms of "lower casing", then toCaseFold is just >>>>> the wrong operation. That is what toLowerCase is for and the >>>>> two operations are just not equivalent. >>>> >>>> My recollection, quite possibly inaccurate or incomplete, from at least >>>> one and I think several in-person meetings of the PRECIS WG was: just >>>> use Unicode Default Case Folding because if you use anything else or >>>> try >>>> to roll your own you will be fubar forever. I do not recall any >>>> discussion of the issues you have raised in this thread (e.g., about >>>> the >>>> inadvisability of using case folding for anything but comparison >>>> operations) until the last few weeks. However, I freely admit that's >>>> probably because, through my own faults and ignorance, I didn't >>>> understand what you were saying. >>>> >>>>> FWIW and purely by coincidence wrt PRECIS and this document, I >>>>> had a conversation a few days ago with an expert on Arabic (and >>>>> Persian) calligraphy and writing systems (and good general >>>>> knowledge of writing systems) who is quite insistent that any >>>>> procedure we use for case-insensitive matching (e.g., case >>>>> folding) is discriminatory, inconsistent, and just >>>>> badly-thought-out if that same procedure doesn't treat isolated, >>>>> initial, and medial forms of the same character as equivalent. >>>>> He further strengthens his case (sic) by noting that Unicode >>>>> case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, >>>>> unambiguously a lower-case character) to U+03C3 (GREEK SMALL >>>>> LETTER SIGMA), a relationship that depends entirely on >>>>> positional use and not case. He also believes the same >>>>> relationships should apply to all other scripts that make form >>>>> distinctions for some characters based on positions in a string >>>>> and for which Unicode has chosen to assign different code >>>>> points. Even if there were wide acceptance of his view, Unicode >>>>> stability principles would prevent changing toCaseFold (or >>>>> CaseFolding.txt), but this is more evidence that what toCaseFold >>>>> does and does not do is going to be hard to explain to either >>>>> casual users or to writing system experts whose primary >>>>> experience is not with the Greek-Latin-Cyrillic group. >>>>> >>>>> I don't think we want to say "these matching rules are somewhat >>>>> arbitrary and irrational, but, if you don't like it, blame >>>>> Unicode and not us", if only because it is our choice to use >>>>> those matching rules. More below. >>>>> >>>>> >>>>>> ... >>>>>>> (2) Second, probably as a result of having IDNA in the lead, >>>>>>> we've gotten sloppy about language and operations and should >>>>>>> probably start untangling that before it gets people in >>>>>>> trouble. >>>>>> >>>>>> Where is the right place to do that untangling? (I doubt that >>>>>> it is the precis-nickname document.) >>>>> >>>>> I agree that precis-nickname isn't the ideal place. I also >>>>> believe that you and it are the innocent victims of the >>>>> situation. At the same time, I don't believe IETF should be >>>>> producing incomplete, ambiguous, erroneous, or misleading >>>>> standards because no one could get around to doing the right >>>>> foundational work. >>>> >>>> Agreed. I too want to get this right, even though it's not a lot of fun >>>> and it's certainly more work than I thought I was signing up for at the >>>> NEWPREP BoF years ago. >>>> >>>>>>> The Unicode Standard, at least as I understand it, is fairly >>>>>>> clear that the most important (and really only safe) use of >>>>>>> toCaseFold is as part of a comparison operation. >>>>>> >>>>>> Thanks for noting that. For example, Section 5.18 of Unicode >>>>>> 8.0.0 says: >>>>>> >>>>>> Caseless matching is implemented using case folding, which >>>>>> is the >>>>>> process of mapping characters of different case to a >>>>>> single form, so >>>>>> that case differences in strings are erased. Case folding >>>>>> allows for >>>>>> fast caseless matches in lookups because only binary >>>>>> comparison is >>>>>> required. It is more than just conversion to lowercase. >>>>> >>>>> Right. But, again, when its use is appropriate (a very >>>>> controversial topic in itself with our painful IDNA history with >>>>> Final Sigma, Eszett and the case-independent versus >>>>> position-independent controversy called out above as examples) >>>>> that is "matches in lookups" (what I've described elsewhere as >>>>> "comparison only"). Not creating or defining nicknames or >>>>> aliases. And that _is_ a problem for this document. >>>> >>>> I'm not convinced that things are as bad as you think. If we say in >>>> draft-ietf-precis-nickname that the case mapping rule is to be applied >>>> only as part of comparison and not as part of enforcement - which I >>>> think is really what we care about (e.g., to prevent spoofing of users >>>> in chat rooms) - then I think we might be most of the way there. >>>> >>>>>>> Using your >>>>>>> example it is entirely reasonable to treat, "stpeter" and >>>>>>> "StPeter" as equivalent in a comparison operation, but >>>>>>> accepting one string and changing it to the other for display >>>>>>> may not be a really good idea. While that transformation may >>>>>>> be acceptable (although I would be surprised if there were no >>>>>>> people who share your surname who could consider "stpeter" or >>>>>>> "Stpeter" unacceptable and might even believe that "StPeter" >>>>>>> is an unacceptable substitute for "St. Peter"), >>>>>> >>>>>> I do receive email at stpeter@gmail.com intended for >>>>>> st.peter@gmail.com but that's a separate topic... >>>>> >>>>> One that is relevant because it "works" as a side-effect of a >>>>> decision Google has made about mailbox name equivalence, a >>>>> decision that, IMO, will sooner or later get someone into a lot >>>>> of trouble and, more important, a decision and matching rule >>>>> that PRECIS, AFAICT, does not allow and that IDNA unambigiously >>>>> forbids. >>>>> >>>>>>> it also points out the >>>>>>> dangers of using Basic Latin script examples to illustrate >>>>>>> situations in which even more extended Latin script, much less >>>>>>> other scripts, may raise more complex issues. Because IDNA >>>>>>> is essentially a workaround because changing the DNS >>>>>>> comparison rules was impractical for several reasons, we >>>>>>> ended up using toCaweFold to map characters and strings into >>>>>>> others in IDNA2003 but PRECIS implementations that do not >>>>>>> have the same constraints would, in general, be better off >>>>>>> confining the use of toCaseFold, or even toLowerCase, to >>>>>>> comparison operations. >>>>>> >>>>>> Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does >>>>>> it make sense for this nickname specification to differ in >>>>>> this respect from the published RFCs? Shall we file errata >>>>>> against those documents? (This might apply only to RFC 7613, >>>>>> which says to apply case folding as part of the enforcement >>>>>> process - when exactly to apply case folding is not stipulated >>>>>> by RFC 7564.) >>>>> >>>>> To the extent to which this is a "botched that because the WG >>>>> didn't understand the issues well enough" conclusion, it would >>>>> be entirely reasonable to generate an updating RFC that repairs >>>>> 7613 and/or 7564, even doing so in an addendum to >>>>> precis-nickname if that is the only way to do that >>>>> expeditiously. Per the above, we really don't want to give >>>>> library routine writers bad instructions. As I understand it, >>>>> the current position of the RFC Editor and IESG is that >>>>> technical specification errors discovered in retrospect or after >>>>> people start using a spec are not appropriate topics for errata. >>>>> If the WG is not willing to do any of those things, then I >>>>> suggest that precis-nickname at least needs to contain a very >>>>> clear warning notice about this situation (see my response to >>>>> your question 1 below). >>>> >>>> I think we'll probably need to fix 7613 and 7564. I am hoping we can >>>> fix >>>> nickname now so that it is less incorrect than the other two. That >>>> doesn't necessarily mean we won't need to also further fix nickname >>>> later on. >>>> >>>> Granted, we were supposed to avoid this problem by working on all of >>>> the >>>> PRECIS specs simultaneously. Clearly we have not avoided the >>>> problem, so >>>> we need to solve it one way or another. If that means bis for them all, >>>> we need to deal with it. >>>> >>>>>>> (3) Because toCaweFold loses information when used for more >>>>>>> than comparison (for comparison, it merely contributes to >>>>>>> what some people would consider false positives for matching) >>>>>>> involves some controversial decisions and, because of >>>>>>> stability requirements, cannot be changed even if the >>>>>>> controversies are resolved in other ways, we end up with, >>>>>>> e.g., >>>>>>> toCaseFold ("Nuß") -> "nuss" >>>>>>> which is considered an acceptable transformation in some >>>>>>> places that identify themselves as speaking/using German and >>>>>>> two different unacceptable errors in others. Again, this will >>>>>>> almost always be much more serious if the transformation is >>>>>>> used to map and replace strings than if it is used to compare >>>>>>> (fwiw, that particular example is part of a continuing >>>>>>> disagreement between IDNA2008 and, among others, German >>>>>>> domain registry authorities on one side and UTC and UTR 46 on >>>>>>> the other). >>>>>> >>>>>> Agreed. >>>>> >>>>> See "warning notice" comment above and question 1 response below. >>>>> >>>>>> (4) If the motivation is really to avoid confusion, the >>>>>>> correct confusion-blocking rule for Latin script (but not >>>>>>> others) and many languages that use it (but certainly not >>>>>>> all) involves moving beyond toCaseFold and treating all >>>>>>> "decorated" characters (characters normally represented by >>>>>>> glyphs consisting of a Basic Latin character and one or more >>>>>>> diacritical or equivalent markings) compare equal to their >>>>>>> base characters, e.g., "á" not only matches "Á" but also >>>>>>> "a" and "A" and, as an unfortunate side-effect, maybe "À" >>>>>>> and "à" as well. This is bad news for languages in which >>>>>>> decorated Latin characters are used to represent phonetically >>>>>>> and conceptually different characters, not just pronunciation >>>>>>> variations. I am not qualified to evaluate "how bad". In >>>>>>> addition, extrapolations from this principle about Latin >>>>>>> script to unrelated scripts will almost certainly lead to >>>>>>> serious errors and/or additional confusion. >>>>>> >>>>>> I would not be comfortable going that far... >>>>> >>>>> In case it isn't clear, I would not be either. But it is where >>>>> getting sloppy about this stuff could easily take us. It is >>>>> worth noting that it also identifies one of the difficulties >>>>> with doing a global system to be applied to many types of >>>>> applications (like the PRECIS work) and then applying it in user >>>>> interface software that end users will expect to be localized to >>>>> their assumptions because it has been mapped or translated into >>>>> their language (if one normally speaks Upper Slobbovian but has >>>>> some familiarity with English, an application interface in >>>>> English will probably be expected to be "foreign", odd, and >>>>> maybe even inconsistent with whatever expectations exist. But, >>>>> if the interface is in Upper Slobbovian, the natural and >>>>> reasonable assumption will be the matching should conform to >>>>> normal Upper Slobbovian conventions. FWIW, a matching rule >>>>> that says: >>>>> >>>>> (i) Two instances of a base character with the same >>>>> diacritical mark(s) match. >>>>> (ii) Two instances of a base character with different >>>>> diacritical mark(s) do not match. >>>>> (iii) Two instances of a base character, one with >>>>> diacritical mark(s) and the other without any decoration >>>>> match. >>>>> >>>>> Is precisely correct and normal behavior for at least one >>>>> language that uses Latin script. It is also the normal practice >>>>> for at least one Latin script transcription system that is used >>>>> by a large fraction of a billion people (maybe more). >>>> >>>> That is indeed sobering. >>>> >>>>>>> More on this and Tom's question below... >>>>>>> >>>>>>>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>>>>>>> Peter, Alexey, >>>>>>>>> >>>>>>>>> I think there is an ambiguity in the specification of case >>>>>>>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>>>>>>> ... >>>>>>>>> But there are 55 code points in Unicode 7.0.0 that change >>>>>>>>> under default case folding that are neither uppercase nor >>>>>>>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>>>>>>> suspect this stems from a confusion between Unicode case >>>>>>>>> mapping and case folding. >>>>> >>>>> In the context of the above, a different way to say the same >>>>> thing is that people are looking at toCaseFold and assuming (and >>>>> explaining things in terms of) toLowerCase. toCaseFold works >>>>> the way it is expected to and those 55 code points are, more or >>>>> less, collateral damage to get to a matching algorithm that >>>>> favors false positives over false negatives and various edge >>>>> cases (including in "edge cases" languages spoken by, and script >>>>> variations used by, millions of people). >>>> >>>> Sadly I suspect that is an accurate description of the current state of >>>> affairs (modulo my comment above about PRECIS WG discussions at one or >>>> more IETF meetings). >>>> >>>>>> ... >>>>>> After all that, I have 3 questions: >>>>> >>>>> Personal opinions about answers... >>>>> >>>>>> (1) Is my proposed text enough of a clarification that we >>>>>> should make that change before the nickname I-D is published >>>>>> as an RFC? >>>>> >>>>> I think the clarification is an improvement and is important >>>>> enough to incorporate (I know that is the answer to a slightly >>>>> different question). >>>>> >>>>> However, I think it is inadequate without a serious warning >>>>> about the situation. >>>> >>>> Yes. >>>> >>>>> That warning could appear in either this >>>>> document or RFC 7613 (or 7613bis) with a pointer from the other, >>>>> but, unless you want to revise 7613 now, this one is handy. >>>> >>>> I suspect that we need to revise 7613. I suspect that we might also >>>> need >>>> to revise 7564 (at least with respect to the order in which operations >>>> are applied, since there has been some confusion among implementers). >>>> >>>> Well, we always knew that we would need to revise them. Just not so >>>> soon. >>>> >>>>> Comment about possible text below. >>>>> >>>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>>> folding is applied only as part of comparison and not as part >>>>>> of enforcement? If so, should we make that change before this >>>>>> document is published as an RFC? >>>>> >>>>> Yes. If something is used for "enforcement", it should be lower >>>>> casing or something else that can be explained to people who are >>>>> ordinarily familiar with one or more of the scripts that make >>>>> case distinctions. >>>>> >>>>> However, viewed in the light of this discussion, the whole >>>>> "enforcement" concept becomes a little dicey, especially if, as >>>>> I believe but don't have time to verify, the transformations >>>>> performed by toLowerCase are not a proper subset of those >>>>> performed by toCaseFold. >>>> >>>> My initial thought is that case mapping doesn't belong in the nickname >>>> enforcement operation at all - only in the comparison operation. >>>> >>>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>>> only as part of comparison and not as part of enforcement? >>>>> >>>>> I think that is necessary. Following up on the comment above, I >>>>> would prefer that the current Section 3.2.2 (3) of RFC 7613 >>>>> either point to Unicode Lower Casing or contain a warning along >>>>> the lines of that below. >>>> >>>> Unlike the nickname profile (which I think can be cleaned up by moving >>>> the case mapping rule to the comparison operation and continuing to use >>>> Unicode Default Case Folding), I think you are right that for the >>>> UsernameCaseMapped profile we probably want Unicode Lower Casing. Thus >>>> the likely need, sooner rather than later, for 7613bis. >>>> >>>>> >>>>> ---------- >>>>> >>>>> --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre >>>>> wrote: >>>>> >>>>>> This issue has greater urgency now because >>>>>> draft-ietf-precis-nickname is now in AUTH48... >>>>>> >>>>>> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >>>>>> >>>>>>> After all that, I have 3 questions: >>>>>>> >>>>>>> (1) Is my proposed text enough of a clarification that we >>>>>>> should make that change before the nickname I-D is published >>>>>>> as an RFC? >>>>>> >>>>>> I think so. >>>>> >>>>> See above. >>>>> >>>>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>>>> folding is applied only as part of comparison and not as part >>>>>>> of enforcement? If so, should we make that change before this >>>>>>> document is published as an RFC? >>>>>> >>>>>> Although it seems to be the case that Unicode case folding is >>>>>> primarily designed for the purpose of matching (i.e., >>>>>> comparison), >>>>> >>>>> "Seems" is a little weak. The Unicode Standard is really quite >>>>> specific about that. >>>>> >>>>>> I have a concern that applying the PRECIS case >>>>>> mapping rule after applying the normalization and >>>>>> directionality rules might have unintended consequences that >>>>>> we haven't had a chance to consider yet. The PRECIS framework >>>>>> expresses a preference (actually a hard requirement) for >>>>>> applying the rules in a particular order. We made a late >>>>>> change to the username profiles (RFC 7613), such that width >>>>>> mapping is applied first (in order to accommodate fullwidth >>>>>> and halfwidth characters in certain East Asian scripts). >>>>>> Making a late change to the nickname profile also concerns me, >>>>>> even though both of these late changes seem reasonable on the >>>>>> face of it. I will try to find time to think about this >>>>>> further in the next 24 hours. >>>>> >>>>> First, a hint for the consideration process: there is a reason >>>>> why Unicode now supports a unified case folding and >>>>> normalization operation. My recollection is that it is not only >>>>> more efficient to perform both operations at once (rather than >>>>> looking in one table and then the other), but that there are >>>>> some order-dependent or priority-dependent cases. >>>>> >>>>> The very fact that this issue exists (and is coming up again) >>>>> this late in the process (7613 published in August, WG winding >>>>> down and not, e.g., meeting next week) calls at least the PRECIS >>>>> quality of review and some fairly fundamental model issues into >>>>> question. I first raised that issue a rather long time ago but >>>>> have continued to hope that we have an approximation to "good >>>>> enough" without going back and rethinking everything. >>>>> >>>>> The right solution, IMO, is that, if RFC 7613 is to rationalize >>>>> or explain the operation in terms of converting upper case >>>>> characters to lower case, then it should be using toLowerCase >>>>> because that is what the operation does. After a quick look at >>>>> 7613, amending/updating it to simply convert to lower case would >>>>> be straightforward (and would not raise the ordering issue >>>>> called out above). It would presumably require another IETF >>>>> Last Call, however and I'd hope we would see some serious >>>>> discussion within the WG (and with UTC) before making the change >>>>> and about how it is explained. >>>>> >>>>> If we are not willing to make a change >>>> >>>> I'm willing. It would, as you note, require some careful thinking and >>>> review to make sure that we got it (more) right this time. >>>> >>>>> that significant and/or >>>>> if we conclude that the WG (and perhaps the IETF) have >>>>> completely run out of energy for dealing with i18n issues [1], >>>>> then I suggest that we introduce some additional text. I've >>>>> just spent a half-hour trying to find the AUTH48 copy of >>>>> precis-nickname (aka RFC-to-be-7700), but the RFC Editor has >>>>> apparently changed naming conventions and the various queue >>>>> entry pages all point to the -19 I-D and not the current working >>>>> copy so I can't try to match text and insertion point to what is >>>>> there already. >>>> >>>> http://www.rfc-editor.org/authors/rfc7700.txt >>>> >>>>> The suggestion is a patch (and a hack), not a >>>>> good fix but something like it is probably the least drastic >>>>> measure that would yield something that doesn't contain >>>>> unexplained known defects. >>>>> >>>>> Rough version of suggested text (possibly to go after your >>>>> revised paragraph and following up my comments in my 1 October >>>>> note). Some of the terminology needs checking which I can do if >>>>> you want to go this route: >>>>> >>>>> 'Users of this specification should note that the >>>>> concept of "lower case conversion" is somewhat elusive >>>>> and more dependent on the conventions of different >>>>> languages and notation systems that use the same script >>>>> than may appear obvious at first glance, especially if >>>>> that glance is at Basic Latin characters (i.e., the >>>>> ASCII letter repertoire). Unicode provides two >>>>> different mapping procedures that produce lower-case >>>>> characters, but they have different effects and results >>>>> for many characters. The more conservative one, >>>>> typically appropriately applicable when lower case forms >>>>> are needed, is actual lower-casing (embodied in the >>>>> Unicode operation toLowerCase). A more radical >>>>> operation, normally suitable only for string matching in >>>>> situations in which it is better to consider uncertain >>>>> cases as matching than to treat them as distinct, is >>>>> called "Case Folding" (Unicode operation toCaseFold). >>>>> While the two operations will often produce the same >>>>> results, Case Folding maps some lower case characters >>>>> into others and performs other transformations that may >>>>> be intuitively reasonable and expected for some users >>>>> and quite astonishing (or just wrong) to others. There >>>>> may be no practical alternative, especially if the >>>>> operations are to be used for mapping or enforcement, to >>>>> developers of PRECIS-dependent understanding that the >>>>> cases in which the two yield different results require >>>>> careful understanding of the relevant user base and its >>>>> needs [2].' >>>> >>>> Thanks. >>>> >>>> I am not sure if we need something like that if we move case mapping >>>> (here, case folding) to the comparison operation only - but something >>>> like that might still be appropriate. >>>> >>>>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>>>> only as part of comparison and not as part of enforcement? >>>>>> >>>>>> That is less urgent so I suggest that we address the nickname >>>>>> spec first. >>>>> >>>>> Unless you (or someone else here) have a plausible plan to >>>>> continue and revitalize the WG and assign it that revision work >>>>> (and bring everyone actively participating up to the level >>>>> needed to easily understand this discussion thread and feel >>>>> embarrassed for not spotting the problems), I think we need to >>>>> assume that this is our last shot. Absent an active and >>>>> committed WG, "do this first" could easily be equivalent to >>>>> "don't get around to the other, ever". >>>> >>>> As mentioned, I don't want to have broken RFCs out there. >>>> >>>>> I think that the particular set of issues that started this >>>>> thread as a known defect in the PRECIS specs, both nickname and >>>>> 7613 and that we are obligated to either fix the problems or at >>>>> least explain them. The above warning text is an attempt to >>>>> explain and identify the problems even if it does not actually >>>>> provide a solution. If it were published as part of >>>>> precis-nickname, it could include a statement to the effect that >>>>> it should also be treated as an update to 7613 or, if the IESG >>>>> and RFC Editor would agree in advance to accept, rather than >>>>> bury, the thing, I suppose we could publish it in >>>>> precis-nickname and create an erratum to 7613 indicating that it >>>>> should have included some form of that statement. Neither >>>>> option implies a huge amount of work to update 7613. But I >>>>> think that making the changes of (2) without doing anything >>>>> about (3) makes the two documents inconsistent with each other >>>>> and that would be an additional known defect. >>>>> >>>>> Procedural question: given that precis-nickname is in AUTH48 as >>>>> of yesterday and I don't see anything blocking publication next >>>>> week if you and Barry sign off on the revised text that the WG >>>>> hasn't seen, >>>> >>>> There is no revised text yet. That's why we're having this discussion. >>>> >>>>> does someone need to file a pro forma objection/ >>>>> appeal to block that until this is sorted out and the WG has a >>>>> chance to review proposed publication text? >>>> >>>> I see no reason to invoke the specter of appeals quite yet. Everyone is >>>> working in good faith to do the right thing and get this mess cleaned >>>> up. >>>> >>>>> [1] I believe our collective inability to deal with the >>>>> within-script character forms that do not normalize to each >>>>> other because of language-dependent or other usage factors can >>>>> be taken as evidence of having run out of energy, >>>> >>>> Or in my case simple ignorance of some of the relevant issues and >>>> examples. It's not easy to know about all of this. >>>> >>>>> but it is >>>>> probably in the interest of finishing the PRECIS work to try to >>>>> treat that as a separate issue. >>>> >>>> Probably. >>>> >>>>> [2] Not unlike the reason to differentiate between NFC and NFKC >>>>> and understand the effects of each. >>>> >>>> Another thing that's not easy to grok in fulness. >>>> >>>> Peter >>>> From nobody Tue Nov 3 05:16:29 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 61FC81B3358 for ; Tue, 3 Nov 2015 05:16:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.601 X-Spam-Level: X-Spam-Status: No, score=-1.601 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, GB_I_LETTER=-2, MANGLED_LIST=2.3, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xv76hMbV7AaH for ; Tue, 3 Nov 2015 05:16:21 -0800 (PST) Received: from smtp66.iad3a.emailsrvr.com (smtp66.iad3a.emailsrvr.com [173.203.187.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7E70B1B3352 for ; Tue, 3 Nov 2015 05:16:20 -0800 (PST) Received: from smtp9.relay.iad3a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp9.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 7653D380470; Tue, 3 Nov 2015 08:16:19 -0500 (EST) Received: by smtp9.relay.iad3a.emailsrvr.com (Authenticated sender: fsb-AT-thefsb.org) with ESMTPSA id DB376380401; Tue, 3 Nov 2015 08:16:16 -0500 (EST) X-Sender-Id: fsb@thefsb.org Received: from [10.0.1.2] (c-73-4-147-142.hsd1.ma.comcast.net [73.4.147.142]) (using TLSv1 with cipher DES-CBC3-SHA) by 0.0.0.0:465 (trex/5.5.4); Tue, 03 Nov 2015 08:16:19 -0500 User-Agent: Microsoft-MacOutlook/14.5.7.151005 Date: Tue, 03 Nov 2015 08:16:13 -0500 From: Tom Worster To: Peter Saint-Andre , John C Klensin , Alexey Melnikov Message-ID: Thread-Topic: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> <56383B2F.6080505@andyet.net> In-Reply-To: <56383B2F.6080505@andyet.net> Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 13:16:27 -0000 Peter, This is much better and, as far as the normative language goes, seems correct and unambiguous. The informative sentence in 2.1 Rule 3: "The primary result of doing so is that uppercase characters are mapped to lowercase characters." is good but I think it's worth spending a few more words to spell out that "primary" implies exceptions. "While the primary result is that uppercase characters are mapped to lowercase characters, there are exceptions." It might nudge a few fore implementers to understand that toLowerCase() isn't the right thing. Tom On 11/2/15, 11:42 PM, "Peter Saint-Andre" wrote: >For ease of reviewing only, and with no presumption that these proposed >changes have been accepted by anyone, I have asked the RFC Editor to >provisionally update the document in AUTH48 as outlined in the messages >I have sent to the list. The resulting file is here: > >http://www.rfc-editor.org/authors/rfc7700.txt > >Despite those caveats, if at all possible I would prefer to find an >acceptable solution for publishing this RFC now without undue further >delays (in part because draft-ietf-simple-chat has been held on this >document for almost 3 years!). That does not mean, as I said earlier in >this thread, that I want to have broken RFCs out there, but I think we >can fix this one acceptably now and then update it again in strict >coherence with updates to RFC 7564 and RFC 7613. I am committed to >getting things right, but I am also committed to not holding up other >people's work for years and years on end. > >Peter > >On 10/28/15 3:52 PM, Peter Saint-Andre wrote: >> Example 7 needs to be corrected, too, in accordance with >>CaseFolding.txt. >> >> On 10/28/15 2:54 PM, Peter Saint-Andre wrote: >>> And here is another correction in Section 3... >>> >>> OLD >>> >>> Regarding examples 5, 6, and 7: applying Unicode Default Case >>>Folding >>> to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL >>>LETTER >>> SIGMA (U+03C3), and doing so during comparison would result in >>> matching the nicknames in examples 5 and 6; however, because the >>> PRECIS mapping rules do not account for the special status of GREEK >>> SMALL LETTER FINAL SIGMA (U+03C2), the nicknames in examples 5 and >>>7 >>> or examples 6 and 7 would not be matched. >>> >>> NEW >>> >>> Regarding examples 5, 6, and 7: applying Unicode Default Case >>>Folding >>> to GREEK CAPITAL LETTER SIGMA (U+03A3) results in GREEK SMALL >>>LETTER >>> SIGMA (U+03C3), and the same is true of GREEK SMALL LETTER FINAL >>> SIGMA (U+03C2); therefore, the comparison operation defined in >>> Section 2.4 would result in matching of the nicknames in examples >>>5, >>> 6, and 7. >>> >>> On 10/28/15 2:06 PM, Peter Saint-Andre wrote: >>>> I propose the following text changes: >>>> >>>> ### >>>> >>>> OLD >>>> >>>> 3. Case Mapping Rule: Uppercase and titlecase characters MUST be >>>> mapped to their lowercase equivalents using Unicode Default >>>>Case >>>> Folding as defined in the Unicode Standard [Unicode] (at the >>>> time >>>> of this writing, the algorithm is specified in Chapter 3 of >>>> [Unicode7.0]). In applications that prohibit conflicting >>>> nicknames, this rule helps to reduce the possibility of >>>> confusion >>>> by ensuring that nicknames differing only by case (e.g., >>>> "stpeter" vs. "StPeter") would not be presented to a human >>>>user >>>> at the same time. >>>> >>>> NEW >>>> >>>> 3. Case Mapping Rule: Unicode Default Case Folding MUST be >>>>applied, >>>> as defined in the Unicode Standard [Unicode] (at the time >>>> of this writing, the algorithm is specified in Chapter 3 of >>>> [Unicode7.0]). The primary result of doing so is that >>>>uppercase >>>> characters are mapped to lowercase characters. In applications >>>> that prohibit conflicting nicknames, this rule helps to reduce >>>> the possibility of confusion by ensuring that nicknames >>>> differing only by case (e.g., "stpeter" vs. "StPeter") would >>>>not >>>> be presented to a human user at the same time. >>>> >>>> ### >>>> >>>> (The foregoing was previously sent to the list.) >>>> >>>> ### >>>> >>>> OLD >>>> >>>> 2.3. Enforcement >>>> >>>> An entity that performs enforcement according to this profile MUST >>>> prepare a string as described in Section 2.2 and MUST also apply >>>>the >>>> rules specified in Section 2.1. The rules MUST be applied in the >>>> order shown. >>>> >>>> After all of the foregoing rules have been enforced, the entity >>>>MUST >>>> ensure that the nickname is not zero bytes in length (this is done >>>> after enforcing the rules to prevent applications from mistakenly >>>> omitting a nickname entirely, because when internationalized >>>> characters are accepted, a non-empty sequence of characters can >>>> result in a zero-length nickname after canonicalization). >>>> >>>> 2.4. Comparison >>>> >>>> An entity that performs comparison of two strings according to >>>>this >>>> profile MUST prepare each string and enforce the rules as >>>>specified >>>> in Sections 2.2 and 2.3. The two strings are to be considered >>>> equivalent if they are an exact octet-for-octet match (sometimes >>>> called "bit-string identity"). >>>> >>>> NEW >>>> >>>> 2.3. Enforcement >>>> >>>> An entity that performs enforcement according to this profile MUST >>>> prepare a string as described in Section 2.2 and MUST also apply >>>>the >>>> following rules specified in Section 2.1 in the order shown: >>>> >>>> 1. Additional Mapping Rule >>>> 2. Normalization Rule >>>> 3. Directionality Rule >>>> >>>> After all of the foregoing rules have been enforced, the entity >>>>MUST >>>> ensure that the nickname is not zero bytes in length (this is done >>>> after enforcing the rules to prevent applications from mistakenly >>>> omitting a nickname entirely, because when internationalized >>>> characters are accepted, a non-empty sequence of characters can >>>> result in a zero-length nickname after canonicalization). >>>> >>>> 2.4. Comparison >>>> >>>> An entity that performs comparison of two strings according to >>>>this >>>> profile MUST prepare each string as specified in Section 2.2 and >>>> MUST apply the following rules specified in Section 2.1 in the >>>>order >>>> shown: >>>> >>>> 1. Additional Mapping Rule >>>> 2. Case Mapping Rule >>>> 3. Normalization Rule >>>> 4. Directionality Rule >>>> >>>> The two strings are to be considered equivalent if they are an >>>>exact >>>> octet-for-octet match (sometimes called "bit-string identity"). >>>> >>>> ### >>>> >>>> In addition, some variation on John's proposed text about toLowerCase >>>> vs. toCaseFold might be appropriate at the end of Section 4; however, >>>> I'm still not sure that is necessary if we move the case mapping rule >>>>to >>>> the comparison operation. >>>> >>>> Peter >>>> >>>> On 10/27/15 8:09 PM, Peter Saint-Andre wrote: >>>>> On 10/27/15 11:32 AM, John C Klensin wrote: >>>>>> Response to Monday's note immediately below; response to today's >>>>>> follows it. My apologies, but it is probably important to read >>>>>> both. My further apologies for the length of this note, but I >>>>>> think we are in deep trouble here, >>>>> >>>>> Internationalization always seems to be a matter of how deep the >>>>> trouble >>>>> is... >>>>> >>>>>> trouble that is aggravated by >>>>>> precis-mappings and precis-nickname both being post-approval and >>>>>> that, as far as I know, there are no future plans for PRECIS >>>>>> work (having precis-nickname in AUTH48 just emphasizes that -- >>>>>> see comment at end). >>>>> >>>>> We had not planned to work on PRECIS because we thought we were done >>>>> for >>>>> awhile. If that's not the case and we need to fix things, then so be >>>>> it. >>>>> Whether there is sufficient and continued energy for such work is >>>>> another question. Personally I don't want us to have broken RFCs out >>>>> there. >>>>> >>>>>> --On Monday, October 26, 2015 15:51 -0600 Peter Saint-Andre - >>>>>> &yet wrote: >>>>>> >>>>>>> My apologies for the delayed reply. Comments inline. >>>>>> >>>>>> A few remarks below... I can't tell whether we disagree or >>>>>> whether at least one of us, probably me, are not being >>>>>> adequately clear. (Material on which we fairly clearly agree >>>>>> elided.) >>>>>> >>>>>> >>>>>>> On 10/1/15 7:50 AM, John C Klensin wrote: >>>>>>>> --On Wednesday, September 30, 2015 15:16 -0600 Peter >>>>>>>> Saint-Andre - &yet wrote: >>>>>>> ... >>>>>>>> Peter, >>>>>>>> >>>>>>>> While your proposed text is an improvement, >>>>>>> >>>>>>> Happy to hear it. All I intended was a slight clarification. >>>>>> >>>>>> But I'm not certain we are there yet... >>>>> >>>>> Agreed. The text I proposed addressed only a very small part of the >>>>> problem. >>>>> >>>>>>>> the desire of many >>>>>>>> people for a magic "just tell me what to do" formula, one that >>>>>>>> lets them avoid understanding the issues, may call for a >>>>>>>> little more: >>>>>>> >>>>>>> There is always a need for more when it comes to i18n. >>>>>> >>>>>> But I think it is a little more that that. I've heard several >>>>>> times, including in PRECIS meetings, requests for "just tell me >>>>>> what to do and make sure it isn't complicated" (or "I don't want >>>>>> to have to think about, much less understand, the issues"). We >>>>>> can debate whether giving in to those requests in the I18n case >>>>>> is wise. I think it leads directly to conclusions equivalent to >>>>>> "I understand my own script and writing system (or think I do) >>>>>> and therefore, since all writing systems must be pretty much the >>>>>> same, I understand all of the core issues in terms of my script >>>>>> and understanding". That, in turn, leads directly to the "how >>>>>> do you spell 'Z=C3=BCrich'?" and "all spellings of 'Zuerich' should >>>>>> be treated as equivalent" discussion that sounded like they >>>>>> dominated a BOF at IETF 93. >>>>>> >>>>>> Now I actually think it is reasonable for someone to ask for a >>>>>> library that will do the job most of the time and that will >>>>>> almost never cause their users or customers to get angry at >>>>>> them. But, if we are going to call what we do "standards", they >>>>>> should contain sufficient information that would-be library >>>>>> authors can know what to do ... or understand that they are in >>>>>> over their heads. And, for these particular cases, we may need >>>>>> to explain, or help the library authors explain, why some cases >>>>>> will fail and, indeed, get users mad at vendors. >>>>>> >>>>>> >>>>>>>> (1) First, toCaseFold is _not_ toLowerCase. Saying "The >>>>>>>> primary result of doing so is that uppercase characters are >>>>>>>> mapped to lowercase characters" is true for toCaseFold, >>>>>>> >>>>>>> By "primary" I meant two things: (1) lowercasing is what >>>>>>> happens to the preponderance of code points and (2) this is >>>>>>> the result that most people care about. >>>>>> >>>>>> If I parse the above correctly, I think you are wrong. I think >>>>>> what most people want, care about, and think they are getting, >>>>>> is lower case conversion, i.e., an operation that preserves >>>>>> lower case characters and converts upper case characters to the >>>>>> equivalent lower case. toCaseFold isn't that operation. It is >>>>>> a much more complex and subtle operation that, as well as >>>>>> converting upper case characters to lower case, sometimes >>>>>> converts lower case characters to different lower case >>>>>> characters (or strings of them). It also requires a fairly good >>>>>> understanding of Unicode (not just a relevant script) and >>>>>> historical Unicode decisions to predict its behavior and to have >>>>>> any hope of explaining that behavior to users. If one is >>>>>> trying to compare (as distinct from converting), then toCaseFold >>>>>> may be exactly what it wanted. but it is really hard to explain >>>>>> or justify that in terms of "nicknames" or "aliases", which are >>>>>> about conversion. And, if one hopes to explain what is going >>>>>> on to users in terms of "lower casing", then toCaseFold is just >>>>>> the wrong operation. That is what toLowerCase is for and the >>>>>> two operations are just not equivalent. >>>>> >>>>> My recollection, quite possibly inaccurate or incomplete, from at >>>>>least >>>>> one and I think several in-person meetings of the PRECIS WG was: just >>>>> use Unicode Default Case Folding because if you use anything else or >>>>> try >>>>> to roll your own you will be fubar forever. I do not recall any >>>>> discussion of the issues you have raised in this thread (e.g., about >>>>> the >>>>> inadvisability of using case folding for anything but comparison >>>>> operations) until the last few weeks. However, I freely admit that's >>>>> probably because, through my own faults and ignorance, I didn't >>>>> understand what you were saying. >>>>> >>>>>> FWIW and purely by coincidence wrt PRECIS and this document, I >>>>>> had a conversation a few days ago with an expert on Arabic (and >>>>>> Persian) calligraphy and writing systems (and good general >>>>>> knowledge of writing systems) who is quite insistent that any >>>>>> procedure we use for case-insensitive matching (e.g., case >>>>>> folding) is discriminatory, inconsistent, and just >>>>>> badly-thought-out if that same procedure doesn't treat isolated, >>>>>> initial, and medial forms of the same character as equivalent. >>>>>> He further strengthens his case (sic) by noting that Unicode >>>>>> case folding maps U+03C2 (GREEK SMALL LETTER FINAL SIGMA, >>>>>> unambiguously a lower-case character) to U+03C3 (GREEK SMALL >>>>>> LETTER SIGMA), a relationship that depends entirely on >>>>>> positional use and not case. He also believes the same >>>>>> relationships should apply to all other scripts that make form >>>>>> distinctions for some characters based on positions in a string >>>>>> and for which Unicode has chosen to assign different code >>>>>> points. Even if there were wide acceptance of his view, Unicode >>>>>> stability principles would prevent changing toCaseFold (or >>>>>> CaseFolding.txt), but this is more evidence that what toCaseFold >>>>>> does and does not do is going to be hard to explain to either >>>>>> casual users or to writing system experts whose primary >>>>>> experience is not with the Greek-Latin-Cyrillic group. >>>>>> >>>>>> I don't think we want to say "these matching rules are somewhat >>>>>> arbitrary and irrational, but, if you don't like it, blame >>>>>> Unicode and not us", if only because it is our choice to use >>>>>> those matching rules. More below. >>>>>> >>>>>> >>>>>>> ... >>>>>>>> (2) Second, probably as a result of having IDNA in the lead, >>>>>>>> we've gotten sloppy about language and operations and should >>>>>>>> probably start untangling that before it gets people in >>>>>>>> trouble. >>>>>>> >>>>>>> Where is the right place to do that untangling? (I doubt that >>>>>>> it is the precis-nickname document.) >>>>>> >>>>>> I agree that precis-nickname isn't the ideal place. I also >>>>>> believe that you and it are the innocent victims of the >>>>>> situation. At the same time, I don't believe IETF should be >>>>>> producing incomplete, ambiguous, erroneous, or misleading >>>>>> standards because no one could get around to doing the right >>>>>> foundational work. >>>>> >>>>> Agreed. I too want to get this right, even though it's not a lot of >>>>>fun >>>>> and it's certainly more work than I thought I was signing up for at >>>>>the >>>>> NEWPREP BoF years ago. >>>>> >>>>>>>> The Unicode Standard, at least as I understand it, is fairly >>>>>>>> clear that the most important (and really only safe) use of >>>>>>>> toCaseFold is as part of a comparison operation. >>>>>>> >>>>>>> Thanks for noting that. For example, Section 5.18 of Unicode >>>>>>> 8.0.0 says: >>>>>>> >>>>>>> Caseless matching is implemented using case folding, which >>>>>>> is the >>>>>>> process of mapping characters of different case to a >>>>>>> single form, so >>>>>>> that case differences in strings are erased. Case folding >>>>>>> allows for >>>>>>> fast caseless matches in lookups because only binary >>>>>>> comparison is >>>>>>> required. It is more than just conversion to lowercase. >>>>>> >>>>>> Right. But, again, when its use is appropriate (a very >>>>>> controversial topic in itself with our painful IDNA history with >>>>>> Final Sigma, Eszett and the case-independent versus >>>>>> position-independent controversy called out above as examples) >>>>>> that is "matches in lookups" (what I've described elsewhere as >>>>>> "comparison only"). Not creating or defining nicknames or >>>>>> aliases. And that _is_ a problem for this document. >>>>> >>>>> I'm not convinced that things are as bad as you think. If we say in >>>>> draft-ietf-precis-nickname that the case mapping rule is to be >>>>>applied >>>>> only as part of comparison and not as part of enforcement - which I >>>>> think is really what we care about (e.g., to prevent spoofing of >>>>>users >>>>> in chat rooms) - then I think we might be most of the way there. >>>>> >>>>>>>> Using your >>>>>>>> example it is entirely reasonable to treat, "stpeter" and >>>>>>>> "StPeter" as equivalent in a comparison operation, but >>>>>>>> accepting one string and changing it to the other for display >>>>>>>> may not be a really good idea. While that transformation may >>>>>>>> be acceptable (although I would be surprised if there were no >>>>>>>> people who share your surname who could consider "stpeter" or >>>>>>>> "Stpeter" unacceptable and might even believe that "StPeter" >>>>>>>> is an unacceptable substitute for "St. Peter"), >>>>>>> >>>>>>> I do receive email at stpeter@gmail.com intended for >>>>>>> st.peter@gmail.com but that's a separate topic... >>>>>> >>>>>> One that is relevant because it "works" as a side-effect of a >>>>>> decision Google has made about mailbox name equivalence, a >>>>>> decision that, IMO, will sooner or later get someone into a lot >>>>>> of trouble and, more important, a decision and matching rule >>>>>> that PRECIS, AFAICT, does not allow and that IDNA unambigiously >>>>>> forbids. >>>>>> >>>>>>>> it also points out the >>>>>>>> dangers of using Basic Latin script examples to illustrate >>>>>>>> situations in which even more extended Latin script, much less >>>>>>>> other scripts, may raise more complex issues. Because IDNA >>>>>>>> is essentially a workaround because changing the DNS >>>>>>>> comparison rules was impractical for several reasons, we >>>>>>>> ended up using toCaweFold to map characters and strings into >>>>>>>> others in IDNA2003 but PRECIS implementations that do not >>>>>>>> have the same constraints would, in general, be better off >>>>>>>> confining the use of toCaseFold, or even toLowerCase, to >>>>>>>> comparison operations. >>>>>>> >>>>>>> Unfortunately we didn't do that in RFC 7564 or RFC 7613. Does >>>>>>> it make sense for this nickname specification to differ in >>>>>>> this respect from the published RFCs? Shall we file errata >>>>>>> against those documents? (This might apply only to RFC 7613, >>>>>>> which says to apply case folding as part of the enforcement >>>>>>> process - when exactly to apply case folding is not stipulated >>>>>>> by RFC 7564.) >>>>>> >>>>>> To the extent to which this is a "botched that because the WG >>>>>> didn't understand the issues well enough" conclusion, it would >>>>>> be entirely reasonable to generate an updating RFC that repairs >>>>>> 7613 and/or 7564, even doing so in an addendum to >>>>>> precis-nickname if that is the only way to do that >>>>>> expeditiously. Per the above, we really don't want to give >>>>>> library routine writers bad instructions. As I understand it, >>>>>> the current position of the RFC Editor and IESG is that >>>>>> technical specification errors discovered in retrospect or after >>>>>> people start using a spec are not appropriate topics for errata. >>>>>> If the WG is not willing to do any of those things, then I >>>>>> suggest that precis-nickname at least needs to contain a very >>>>>> clear warning notice about this situation (see my response to >>>>>> your question 1 below). >>>>> >>>>> I think we'll probably need to fix 7613 and 7564. I am hoping we can >>>>> fix >>>>> nickname now so that it is less incorrect than the other two. That >>>>> doesn't necessarily mean we won't need to also further fix nickname >>>>> later on. >>>>> >>>>> Granted, we were supposed to avoid this problem by working on all of >>>>> the >>>>> PRECIS specs simultaneously. Clearly we have not avoided the >>>>> problem, so >>>>> we need to solve it one way or another. If that means bis for them >>>>>all, >>>>> we need to deal with it. >>>>> >>>>>>>> (3) Because toCaweFold loses information when used for more >>>>>>>> than comparison (for comparison, it merely contributes to >>>>>>>> what some people would consider false positives for matching) >>>>>>>> involves some controversial decisions and, because of >>>>>>>> stability requirements, cannot be changed even if the >>>>>>>> controversies are resolved in other ways, we end up with, >>>>>>>> e.g., >>>>>>>> toCaseFold ("Nu=C3=9F") -> "nuss" >>>>>>>> which is considered an acceptable transformation in some >>>>>>>> places that identify themselves as speaking/using German and >>>>>>>> two different unacceptable errors in others. Again, this will >>>>>>>> almost always be much more serious if the transformation is >>>>>>>> used to map and replace strings than if it is used to compare >>>>>>>> (fwiw, that particular example is part of a continuing >>>>>>>> disagreement between IDNA2008 and, among others, German >>>>>>>> domain registry authorities on one side and UTC and UTR 46 on >>>>>>>> the other). >>>>>>> >>>>>>> Agreed. >>>>>> >>>>>> See "warning notice" comment above and question 1 response below. >>>>>> >>>>>>> (4) If the motivation is really to avoid confusion, the >>>>>>>> correct confusion-blocking rule for Latin script (but not >>>>>>>> others) and many languages that use it (but certainly not >>>>>>>> all) involves moving beyond toCaseFold and treating all >>>>>>>> "decorated" characters (characters normally represented by >>>>>>>> glyphs consisting of a Basic Latin character and one or more >>>>>>>> diacritical or equivalent markings) compare equal to their >>>>>>>> base characters, e.g., "=C3=A1" not only matches "=C3=81" but also >>>>>>>> "a" and "A" and, as an unfortunate side-effect, maybe "=C3=80" >>>>>>>> and "=C3=A0" as well. This is bad news for languages in which >>>>>>>> decorated Latin characters are used to represent phonetically >>>>>>>> and conceptually different characters, not just pronunciation >>>>>>>> variations. I am not qualified to evaluate "how bad". In >>>>>>>> addition, extrapolations from this principle about Latin >>>>>>>> script to unrelated scripts will almost certainly lead to >>>>>>>> serious errors and/or additional confusion. >>>>>>> >>>>>>> I would not be comfortable going that far... >>>>>> >>>>>> In case it isn't clear, I would not be either. But it is where >>>>>> getting sloppy about this stuff could easily take us. It is >>>>>> worth noting that it also identifies one of the difficulties >>>>>> with doing a global system to be applied to many types of >>>>>> applications (like the PRECIS work) and then applying it in user >>>>>> interface software that end users will expect to be localized to >>>>>> their assumptions because it has been mapped or translated into >>>>>> their language (if one normally speaks Upper Slobbovian but has >>>>>> some familiarity with English, an application interface in >>>>>> English will probably be expected to be "foreign", odd, and >>>>>> maybe even inconsistent with whatever expectations exist. But, >>>>>> if the interface is in Upper Slobbovian, the natural and >>>>>> reasonable assumption will be the matching should conform to >>>>>> normal Upper Slobbovian conventions. FWIW, a matching rule >>>>>> that says: >>>>>> >>>>>> (i) Two instances of a base character with the same >>>>>> diacritical mark(s) match. >>>>>> (ii) Two instances of a base character with different >>>>>> diacritical mark(s) do not match. >>>>>> (iii) Two instances of a base character, one with >>>>>> diacritical mark(s) and the other without any decoration >>>>>> match. >>>>>> >>>>>> Is precisely correct and normal behavior for at least one >>>>>> language that uses Latin script. It is also the normal practice >>>>>> for at least one Latin script transcription system that is used >>>>>> by a large fraction of a billion people (maybe more). >>>>> >>>>> That is indeed sobering. >>>>> >>>>>>>> More on this and Tom's question below... >>>>>>>> >>>>>>>>> On 9/29/15 3:28 PM, Tom Worster wrote: >>>>>>>>>> Peter, Alexey, >>>>>>>>>> >>>>>>>>>> I think there is an ambiguity in the specification of case >>>>>>>>>> mapping in RFC 7613 and draft-ietf-precis-nickname-19. >>>>>>>>> ... >>>>>>>>>> But there are 55 code points in Unicode 7.0.0 that change >>>>>>>>>> under default case folding that are neither uppercase nor >>>>>>>>>> titlecase characters, 12 of which are Lowercase_Letter. I >>>>>>>>>> suspect this stems from a confusion between Unicode case >>>>>>>>>> mapping and case folding. >>>>>> >>>>>> In the context of the above, a different way to say the same >>>>>> thing is that people are looking at toCaseFold and assuming (and >>>>>> explaining things in terms of) toLowerCase. toCaseFold works >>>>>> the way it is expected to and those 55 code points are, more or >>>>>> less, collateral damage to get to a matching algorithm that >>>>>> favors false positives over false negatives and various edge >>>>>> cases (including in "edge cases" languages spoken by, and script >>>>>> variations used by, millions of people). >>>>> >>>>> Sadly I suspect that is an accurate description of the current state >>>>>of >>>>> affairs (modulo my comment above about PRECIS WG discussions at one >>>>>or >>>>> more IETF meetings). >>>>> >>>>>>> ... >>>>>>> After all that, I have 3 questions: >>>>>> >>>>>> Personal opinions about answers... >>>>>> >>>>>>> (1) Is my proposed text enough of a clarification that we >>>>>>> should make that change before the nickname I-D is published >>>>>>> as an RFC? >>>>>> >>>>>> I think the clarification is an improvement and is important >>>>>> enough to incorporate (I know that is the answer to a slightly >>>>>> different question). >>>>>> >>>>>> However, I think it is inadequate without a serious warning >>>>>> about the situation. >>>>> >>>>> Yes. >>>>> >>>>>> That warning could appear in either this >>>>>> document or RFC 7613 (or 7613bis) with a pointer from the other, >>>>>> but, unless you want to revise 7613 now, this one is handy. >>>>> >>>>> I suspect that we need to revise 7613. I suspect that we might also >>>>> need >>>>> to revise 7564 (at least with respect to the order in which >>>>>operations >>>>> are applied, since there has been some confusion among implementers). >>>>> >>>>> Well, we always knew that we would need to revise them. Just not so >>>>> soon. >>>>> >>>>>> Comment about possible text below. >>>>>> >>>>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>>>> folding is applied only as part of comparison and not as part >>>>>>> of enforcement? If so, should we make that change before this >>>>>>> document is published as an RFC? >>>>>> >>>>>> Yes. If something is used for "enforcement", it should be lower >>>>>> casing or something else that can be explained to people who are >>>>>> ordinarily familiar with one or more of the scripts that make >>>>>> case distinctions. >>>>>> >>>>>> However, viewed in the light of this discussion, the whole >>>>>> "enforcement" concept becomes a little dicey, especially if, as >>>>>> I believe but don't have time to verify, the transformations >>>>>> performed by toLowerCase are not a proper subset of those >>>>>> performed by toCaseFold. >>>>> >>>>> My initial thought is that case mapping doesn't belong in the >>>>>nickname >>>>> enforcement operation at all - only in the comparison operation. >>>>> >>>>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>>>> only as part of comparison and not as part of enforcement? >>>>>> >>>>>> I think that is necessary. Following up on the comment above, I >>>>>> would prefer that the current Section 3.2.2 (3) of RFC 7613 >>>>>> either point to Unicode Lower Casing or contain a warning along >>>>>> the lines of that below. >>>>> >>>>> Unlike the nickname profile (which I think can be cleaned up by >>>>>moving >>>>> the case mapping rule to the comparison operation and continuing to >>>>>use >>>>> Unicode Default Case Folding), I think you are right that for the >>>>> UsernameCaseMapped profile we probably want Unicode Lower Casing. >>>>>Thus >>>>> the likely need, sooner rather than later, for 7613bis. >>>>> >>>>>> >>>>>> ---------- >>>>>> >>>>>> --On Tuesday, October 27, 2015 07:52 -0600 Peter Saint-Andre >>>>>> wrote: >>>>>> >>>>>>> This issue has greater urgency now because >>>>>>> draft-ietf-precis-nickname is now in AUTH48... >>>>>>> >>>>>>> On 10/26/15 3:51 PM, Peter Saint-Andre - &yet wrote: >>>>>>> >>>>>>>> After all that, I have 3 questions: >>>>>>>> >>>>>>>> (1) Is my proposed text enough of a clarification that we >>>>>>>> should make that change before the nickname I-D is published >>>>>>>> as an RFC? >>>>>>> >>>>>>> I think so. >>>>>> >>>>>> See above. >>>>>> >>>>>>>> (2) Should we modify draft-ietf-precis-nickname so that case >>>>>>>> folding is applied only as part of comparison and not as part >>>>>>>> of enforcement? If so, should we make that change before this >>>>>>>> document is published as an RFC? >>>>>>> >>>>>>> Although it seems to be the case that Unicode case folding is >>>>>>> primarily designed for the purpose of matching (i.e., >>>>>>> comparison), >>>>>> >>>>>> "Seems" is a little weak. The Unicode Standard is really quite >>>>>> specific about that. >>>>>> >>>>>>> I have a concern that applying the PRECIS case >>>>>>> mapping rule after applying the normalization and >>>>>>> directionality rules might have unintended consequences that >>>>>>> we haven't had a chance to consider yet. The PRECIS framework >>>>>>> expresses a preference (actually a hard requirement) for >>>>>>> applying the rules in a particular order. We made a late >>>>>>> change to the username profiles (RFC 7613), such that width >>>>>>> mapping is applied first (in order to accommodate fullwidth >>>>>>> and halfwidth characters in certain East Asian scripts). >>>>>>> Making a late change to the nickname profile also concerns me, >>>>>>> even though both of these late changes seem reasonable on the >>>>>>> face of it. I will try to find time to think about this >>>>>>> further in the next 24 hours. >>>>>> >>>>>> First, a hint for the consideration process: there is a reason >>>>>> why Unicode now supports a unified case folding and >>>>>> normalization operation. My recollection is that it is not only >>>>>> more efficient to perform both operations at once (rather than >>>>>> looking in one table and then the other), but that there are >>>>>> some order-dependent or priority-dependent cases. >>>>>> >>>>>> The very fact that this issue exists (and is coming up again) >>>>>> this late in the process (7613 published in August, WG winding >>>>>> down and not, e.g., meeting next week) calls at least the PRECIS >>>>>> quality of review and some fairly fundamental model issues into >>>>>> question. I first raised that issue a rather long time ago but >>>>>> have continued to hope that we have an approximation to "good >>>>>> enough" without going back and rethinking everything. >>>>>> >>>>>> The right solution, IMO, is that, if RFC 7613 is to rationalize >>>>>> or explain the operation in terms of converting upper case >>>>>> characters to lower case, then it should be using toLowerCase >>>>>> because that is what the operation does. After a quick look at >>>>>> 7613, amending/updating it to simply convert to lower case would >>>>>> be straightforward (and would not raise the ordering issue >>>>>> called out above). It would presumably require another IETF >>>>>> Last Call, however and I'd hope we would see some serious >>>>>> discussion within the WG (and with UTC) before making the change >>>>>> and about how it is explained. >>>>>> >>>>>> If we are not willing to make a change >>>>> >>>>> I'm willing. It would, as you note, require some careful thinking and >>>>> review to make sure that we got it (more) right this time. >>>>> >>>>>> that significant and/or >>>>>> if we conclude that the WG (and perhaps the IETF) have >>>>>> completely run out of energy for dealing with i18n issues [1], >>>>>> then I suggest that we introduce some additional text. I've >>>>>> just spent a half-hour trying to find the AUTH48 copy of >>>>>> precis-nickname (aka RFC-to-be-7700), but the RFC Editor has >>>>>> apparently changed naming conventions and the various queue >>>>>> entry pages all point to the -19 I-D and not the current working >>>>>> copy so I can't try to match text and insertion point to what is >>>>>> there already. >>>>> >>>>> http://www.rfc-editor.org/authors/rfc7700.txt >>>>> >>>>>> The suggestion is a patch (and a hack), not a >>>>>> good fix but something like it is probably the least drastic >>>>>> measure that would yield something that doesn't contain >>>>>> unexplained known defects. >>>>>> >>>>>> Rough version of suggested text (possibly to go after your >>>>>> revised paragraph and following up my comments in my 1 October >>>>>> note). Some of the terminology needs checking which I can do if >>>>>> you want to go this route: >>>>>> >>>>>> 'Users of this specification should note that the >>>>>> concept of "lower case conversion" is somewhat elusive >>>>>> and more dependent on the conventions of different >>>>>> languages and notation systems that use the same script >>>>>> than may appear obvious at first glance, especially if >>>>>> that glance is at Basic Latin characters (i.e., the >>>>>> ASCII letter repertoire). Unicode provides two >>>>>> different mapping procedures that produce lower-case >>>>>> characters, but they have different effects and results >>>>>> for many characters. The more conservative one, >>>>>> typically appropriately applicable when lower case forms >>>>>> are needed, is actual lower-casing (embodied in the >>>>>> Unicode operation toLowerCase). A more radical >>>>>> operation, normally suitable only for string matching in >>>>>> situations in which it is better to consider uncertain >>>>>> cases as matching than to treat them as distinct, is >>>>>> called "Case Folding" (Unicode operation toCaseFold). >>>>>> While the two operations will often produce the same >>>>>> results, Case Folding maps some lower case characters >>>>>> into others and performs other transformations that may >>>>>> be intuitively reasonable and expected for some users >>>>>> and quite astonishing (or just wrong) to others. There >>>>>> may be no practical alternative, especially if the >>>>>> operations are to be used for mapping or enforcement, to >>>>>> developers of PRECIS-dependent understanding that the >>>>>> cases in which the two yield different results require >>>>>> careful understanding of the relevant user base and its >>>>>> needs [2].' >>>>> >>>>> Thanks. >>>>> >>>>> I am not sure if we need something like that if we move case mapping >>>>> (here, case folding) to the comparison operation only - but something >>>>> like that might still be appropriate. >>>>> >>>>>>>> (3) Should we update RFC 7613 so that case folding is applied >>>>>>>> only as part of comparison and not as part of enforcement? >>>>>>> >>>>>>> That is less urgent so I suggest that we address the nickname >>>>>>> spec first. >>>>>> >>>>>> Unless you (or someone else here) have a plausible plan to >>>>>> continue and revitalize the WG and assign it that revision work >>>>>> (and bring everyone actively participating up to the level >>>>>> needed to easily understand this discussion thread and feel >>>>>> embarrassed for not spotting the problems), I think we need to >>>>>> assume that this is our last shot. Absent an active and >>>>>> committed WG, "do this first" could easily be equivalent to >>>>>> "don't get around to the other, ever". >>>>> >>>>> As mentioned, I don't want to have broken RFCs out there. >>>>> >>>>>> I think that the particular set of issues that started this >>>>>> thread as a known defect in the PRECIS specs, both nickname and >>>>>> 7613 and that we are obligated to either fix the problems or at >>>>>> least explain them. The above warning text is an attempt to >>>>>> explain and identify the problems even if it does not actually >>>>>> provide a solution. If it were published as part of >>>>>> precis-nickname, it could include a statement to the effect that >>>>>> it should also be treated as an update to 7613 or, if the IESG >>>>>> and RFC Editor would agree in advance to accept, rather than >>>>>> bury, the thing, I suppose we could publish it in >>>>>> precis-nickname and create an erratum to 7613 indicating that it >>>>>> should have included some form of that statement. Neither >>>>>> option implies a huge amount of work to update 7613. But I >>>>>> think that making the changes of (2) without doing anything >>>>>> about (3) makes the two documents inconsistent with each other >>>>>> and that would be an additional known defect. >>>>>> >>>>>> Procedural question: given that precis-nickname is in AUTH48 as >>>>>> of yesterday and I don't see anything blocking publication next >>>>>> week if you and Barry sign off on the revised text that the WG >>>>>> hasn't seen, >>>>> >>>>> There is no revised text yet. That's why we're having this=20 >>>>>discussion. >>>>> >>>>>> does someone need to file a pro forma objection/ >>>>>> appeal to block that until this is sorted out and the WG has a >>>>>> chance to review proposed publication text? >>>>> >>>>> I see no reason to invoke the specter of appeals quite yet. Everyone=20 >>>>>is >>>>> working in good faith to do the right thing and get this mess cleaned >>>>> up. >>>>> >>>>>> [1] I believe our collective inability to deal with the >>>>>> within-script character forms that do not normalize to each >>>>>> other because of language-dependent or other usage factors can >>>>>> be taken as evidence of having run out of energy, >>>>> >>>>> Or in my case simple ignorance of some of the relevant issues and >>>>> examples. It's not easy to know about all of this. >>>>> >>>>>> but it is >>>>>> probably in the interest of finishing the PRECIS work to try to >>>>>> treat that as a separate issue. >>>>> >>>>> Probably. >>>>> >>>>>> [2] Not unlike the reason to differentiate between NFC and NFKC >>>>>> and understand the effects of each. >>>>> >>>>> Another thing that's not easy to grok in fulness. >>>>> >>>>> Peter From nobody Tue Nov 3 05:42:25 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 90A871B33B6 for ; Tue, 3 Nov 2015 05:42:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.61 X-Spam-Level: X-Spam-Status: No, score=-2.61 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gUQYHNwVY--A for ; Tue, 3 Nov 2015 05:42:23 -0800 (PST) Received: from bsa2.jck.com (ns.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 299291B33B7 for ; Tue, 3 Nov 2015 05:42:23 -0800 (PST) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Ztbqo-000HWu-CA; Tue, 03 Nov 2015 08:42:14 -0500 Date: Tue, 03 Nov 2015 08:42:09 -0500 From: John C Klensin To: Tom Worster , Peter Saint-Andre , Alexey Melnikov Message-ID: In-Reply-To: References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> <56383B2F.6080505@andyet.net> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 13:42:24 -0000 --On Tuesday, November 03, 2015 08:16 -0500 Tom Worster wrote: > The informative sentence in 2.1 Rule 3: > > "The primary result of doing so is that uppercase > characters are mapped to lowercase characters." > > is good but I think it's worth spending a few more words to > spell out that "primary" implies exceptions. > > "While the primary result is that uppercase characters are > mapped to lowercase characters, there are exceptions." > > It might nudge a few fore implementers to understand that > toLowerCase() isn't the right thing. But Tom, for many purposes, forms, and language-script combinations, toLowerCase() is _exactly_ the right thing. For others, if one adopts the principle of doing no harm, whether toLowerCase() or toCaseFold() are likely to do less harm depends on some rather subtle issues including the perspective from which "harm" is viewed. I think one could more accurately restate your comment above as "... to understand that toLowerCase() is sometimes not the right thing and that sometimes toCaseFold() is sometimes not the right thing either. I could take a shot at a paragraph explaining that if it is what people want. Otherwise, I'd be very careful about getting further into that space than the present text goes. Personally, I think the text is still dancing around the issue too much, rather than addressing it, but I may be in the rough. john From nobody Tue Nov 3 06:19:40 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 159BA1A0423 for ; Tue, 3 Nov 2015 06:19:39 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fqw4Gkbe5HxZ for ; Tue, 3 Nov 2015 06:19:37 -0800 (PST) Received: from smtp90.iad3a.emailsrvr.com (smtp90.iad3a.emailsrvr.com [173.203.187.90]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 34F431A19EC for ; Tue, 3 Nov 2015 06:19:14 -0800 (PST) Received: from smtp4.relay.iad3a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp4.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 0391128069A; Tue, 3 Nov 2015 09:19:14 -0500 (EST) Received: by smtp4.relay.iad3a.emailsrvr.com (Authenticated sender: fsb-AT-thefsb.org) with ESMTPSA id EDC01280688; Tue, 3 Nov 2015 09:19:12 -0500 (EST) X-Sender-Id: fsb@thefsb.org Received: from [10.0.1.2] (c-73-4-147-142.hsd1.ma.comcast.net [73.4.147.142]) (using TLSv1 with cipher DES-CBC3-SHA) by 0.0.0.0:465 (trex/5.5.4); Tue, 03 Nov 2015 09:19:14 -0500 User-Agent: Microsoft-MacOutlook/14.5.7.151005 Date: Tue, 03 Nov 2015 09:19:08 -0500 From: Tom Worster To: John C Klensin , Peter Saint-Andre , Alexey Melnikov Message-ID: Thread-Topic: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> <56383B2F.6080505@andyet.net> In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 14:19:39 -0000 On 11/3/15, 8:42 AM, "John C Klensin" wrote: --On Tuesday, November 03, 2015 08:16 -0500 Tom Worster > wrote: > >> The informative sentence in 2.1 Rule 3: >> >> "The primary result of doing so is that uppercase >> characters are mapped to lowercase characters." >> >> is good but I think it's worth spending a few more words to >> spell out that "primary" implies exceptions. >> >> "While the primary result is that uppercase characters are >> mapped to lowercase characters, there are exceptions." >> >> It might nudge a few fore implementers to understand that >> toLowerCase() isn't the right thing. > >But Tom, for many purposes, forms, and language-script >combinations, toLowerCase() is _exactly_ the right thing. For >others, if one adopts the principle of doing no harm, whether >toLowerCase() or toCaseFold() are likely to do less harm depends >on some rather subtle issues including the perspective from >which "harm" is viewed. > >I think one could more accurately restate your comment above as >"... to understand that toLowerCase() is sometimes not the right >thing and that sometimes toCaseFold() is sometimes not the right >thing either. You're quite right while my last sentence was not. Please humor me and pretend I hadn't written it in that email. Tom From nobody Tue Nov 3 06:52:22 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 34F851A1B12 for ; Tue, 3 Nov 2015 06:52:21 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.901 X-Spam-Level: X-Spam-Status: No, score=-3.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, GB_I_LETTER=-2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pmiygWtgmQKm for ; Tue, 3 Nov 2015 06:52:19 -0800 (PST) Received: from smtp66.iad3a.emailsrvr.com (smtp66.iad3a.emailsrvr.com [173.203.187.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D4C5C1A1B0E for ; Tue, 3 Nov 2015 06:52:19 -0800 (PST) Received: from smtp25.relay.iad3a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp25.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id A01D7180654 for ; Tue, 3 Nov 2015 09:52:18 -0500 (EST) Received: by smtp25.relay.iad3a.emailsrvr.com (Authenticated sender: fsb-AT-thefsb.org) with ESMTPSA id 3AC61180630 for ; Tue, 3 Nov 2015 09:52:18 -0500 (EST) X-Sender-Id: fsb@thefsb.org Received: from [10.0.1.2] (c-73-4-147-142.hsd1.ma.comcast.net [73.4.147.142]) (using TLSv1 with cipher DES-CBC3-SHA) by 0.0.0.0:465 (trex/5.5.4); Tue, 03 Nov 2015 09:52:18 -0500 User-Agent: Microsoft-MacOutlook/14.5.7.151005 Date: Tue, 03 Nov 2015 09:52:14 -0500 From: Tom Worster To: Message-ID: Thread-Topic: 2 questions from an app developer References: In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable Archived-At: Subject: Re: [precis] 2 questions from an app developer X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 14:52:21 -0000 Now there's some traffic in the PRECIS list, I'd like to ask this question again, phrased differently. Afaict, the etiology of this implementer's non-minimal astonishment is: o Password is based on OpaqueString Profile o OpaqueString Profile is based on FreeformClass o FreeformClass uses Exceptions (F) from RFC 5892 sec. 2.6 o Exceptions (F) disallows MIDDLE DOT except under CONTEXTO o CONTEXTO rule MIDDLE DOT in RFC 5892 A.3 says "Between 'l' (U+006C) characters only, used to permit the Catalan character ela geminada to be expressed." o Therefore, for example ih=CE=B1tePa=C2=A7s=CF=89rdrul=C2=B7lz is valid ih=CE=B1tePa=C2=A7s=CF=89rdrul=C2=B7ze is invalid Authoring a validation error message that helps the user understand and fix it was a challenge. "Password may not contain the =C2=B7 character except as part of a Catalan character ela geminada," is a cute easter egg[1] but not much use. I imagine IDNA would not want MIDDLE DOTs in domain names and some identifiers because of spoofing but that concern is specific to that domain and surely not to passwords. I don't know Catalan but I use MIDDLE DOTs for a variety of purposes, not quite daily but often enough to know it's been OPT-SHIFT-9 since very early MacOS. It's a useful character so I suspect people will encounter this rule. RFCs 7564 and 7613 are done and dusted so my question is: did I decode the specs correctly? Tom [1] Reminds me of PHP's infamous "Parse error: syntax error, unexpected T_PAAMAYIM_NEKUDOTAYIM" On 9/14/15, 9:23 AM, "Tom Worster" wrote: >Hi, > >Do I understand right that an RFC 7613 password must not contain a MIDDLE >DOT (U+00B7) unless both the previous and next characters are LATIN SMALL >LETTER L (U+006C)? > >Are test vectors available for either of the RFC 7564 string classes or >of the RFC 7613 ID and password profiles? > >If this is not the right place for these questions, please steer me in >the right direction. > >Thanks for your consideration. > >Tom Worster From nobody Tue Nov 3 09:38:27 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 660511A1A7E for ; Tue, 3 Nov 2015 09:38:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.61 X-Spam-Level: X-Spam-Status: No, score=-4.61 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, GB_I_LETTER=-2, RCVD_IN_DNSWL_LOW=-0.7, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JtVUC1XpYQcR for ; Tue, 3 Nov 2015 09:38:22 -0800 (PST) Received: from bsa2.jck.com (ns.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 577431A1A92 for ; Tue, 3 Nov 2015 09:38:22 -0800 (PST) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1ZtfXF-000Hyj-IV; Tue, 03 Nov 2015 12:38:17 -0500 Date: Tue, 03 Nov 2015 12:38:12 -0500 From: John C Klensin To: Tom Worster , precis@ietf.org Message-ID: In-Reply-To: References: X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Subject: Re: [precis] 2 questions from an app developer X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 17:38:25 -0000 --On Tuesday, November 03, 2015 09:52 -0500 Tom Worster wrote: > Now there's some traffic in the PRECIS list, I'd like to ask > this question again, phrased differently. >=20 > Afaict, the etiology of this implementer's non-minimal > astonishment is: >=20 > o Password is based on OpaqueString Profile >=20 > o OpaqueString Profile is based on FreeformClass >=20 > o FreeformClass uses Exceptions (F) from RFC 5892 sec. 2.6 >=20 > o Exceptions (F) disallows MIDDLE DOT except under CONTEXTO >=20 > o CONTEXTO rule MIDDLE DOT in RFC 5892 A.3 says "Between 'l' > (U+006C) characters only, used to permit the Catalan character > ela geminada to be expressed." >=20 > o Therefore, for example >=20 > ih=CE=B1tePa=C2=A7s=CF=89rdrul=C2=B7lz is valid >=20 > ih=CE=B1tePa=C2=A7s=CF=89rdrul=C2=B7ze is invalid >=20 > Authoring a validation error message that helps the user > understand and fix it was a challenge. "Password may not > contain the =C2=B7 character except as part of a Catalan = character > ela geminada," is a cute easter egg[1] but not much use. Of course, the nice, simple, message is "prohibited everywhere". See below. > I imagine IDNA would not want MIDDLE DOTs in domain names and > some identifiers because of spoofing Not just spoofing. The rule that you cite serves to allow MIDDLE DOT (U+00B7), which would normally be DISALLOWED as Punctuation, to be used in that one case. It is not, as you seem to have inferred, a rule that restricts the use of a character that would otherwise be allowed (PVALID). > but that concern is > specific to that domain and surely not to passwords.=20 Well, it depends. Punctuation characters were DISALLOWED for multiple reasons, including the observation that many parsers consider many (but certainly not all) of them to be string terminators or introducers of one flavor or another and at least some rendering arrangements, including those who use substitute graphics when a preferred glyph for a code point is not available, are fairly liberal about swapping them around. Perhaps because it is so easily written, there are also a lot of dot and dot-like characters and even a lot of middle-type ones, some of which are punctuation or other symbols and some of which are letters or numbers. I suppose that translates to "spoofing problem", but spoofing was not the main IDNA motivation. Whether, in this day and age, the advantages of a password that can't be typed without special knowledge outweigh the disadvantages of a password that may be hard to use consistently in contexts that require parsing or rendering, is a judgment call. Given that, maybe "not to passwords" is true, but "surely" is not so clear. > I don't > know Catalan but I use MIDDLE DOTs for a variety of purposes, > not quite daily but often enough to know it's been OPT-SHIFT-9 > since very early MacOS. It's a useful character so I suspect > people will encounter this rule. What I'm about to say is based on trying to adopt a very broad i18n perspective. I'm not questioning or trying to denounce your choices, which I assume are reasonable. Almost everyone has a candidate or two for "favorite character outside the Basic Latin letter range" or, more generally, "favorite character outside the normal letter range of my most common use of my favorite script". Such characters are useful in their own right, for special emphasis, to add security by obscurity in various places where can can be reasonably assured that they won't be prominent in, e.g., password guessing files, and so on. There was a mini-stink when we started transitioning to IDNA2008 about assorted currency symbols which people wanted to use, not just in currency contexts. =20 If the goal is global interoperability, then using them (or counting on having them available) is a bad idea. That applies most strongly to identifier-type things, but even applies to things like passwords: if you ever find yourself traveling to a place where you need to use a keyboard or IME that is optimized for a completely different script of set of assumptions, you could find that you have "secured" yourself out of all ability to access the relevant information. That might have advantages too, but you --and any users you are advising through error messages or the like-- should probably understand the tradeoffs. > RFCs 7564 and 7613 are done and dusted so my question is: did > I decode the specs correctly? I'll let someone else comment on that. Based on comments at several IETF meetings (more clear there than on-list), several people and organizations had the main (or only) goal / success criterion for PRECIS that they would end up having something that could be applied by both implementers and users, and used globally, without any deep understanding, evaluation of circumstances, or specific localizations... ideally with decisions compatible with what they have already been doing. To the extent to which those are the criteria, every case you and a few others have come up with -- sensitivity to the difference between toLowerCase and toCaseFold, individual characters that behave or are defined in a way that seems astonishing, the nasty problems with Turkic languages written in Latin script, asserted ambiguities in the specs that really should be interpreted based on local context-- demonstrate that this is harder and requires far more understanding than people would like, i.e., that, by those criteria, PRECIS has simply failed and we need to get used to, and document and explain, a more complex world. Some of the problem is a disconnect between IETF priorities and Unicode ones, with the latter emphasizing large blocks of running text in known languages rather than strings that may not be words or language-associated at all (most of our identifiers and most good passwords fall into that category). The rest is more fundamental: several people have observed that the creation stories of a number of religions explain the diversity and incompatibilities among human languages (and, by extension, writing systems) in terms of deliberate, divine, intervention to make things difficult. Whether one accepts those explanations or not, it is clear that many of these problems did not originate with the Internet, Unicode, or PRECIS. :-( We do what we can and I have no delusion about achieving perfection. I just wish we could abandon the goal of simplifying it to the point of application that is globally predictable by people who are not willing to understand the issues and, instead, allow for localization where needed and do a better job of explaining the issues and tradeoffs. john From nobody Wed Nov 4 07:37:09 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E2C4C1B31DD for ; Wed, 4 Nov 2015 07:37:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BHau8zhcdwcu for ; Wed, 4 Nov 2015 07:37:07 -0800 (PST) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4E8801B31DC for ; Wed, 4 Nov 2015 07:37:07 -0800 (PST) Received: by iofz202 with SMTP id z202so57322621iof.2 for ; Wed, 04 Nov 2015 07:37:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=andyet_net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=k5fj2eFyiqM2jlU2ba4/pqWwiNuWOIISGeWFN9ynyMU=; b=Jp6n0ZMW7rZUSSK42WhH62xdWPU63JueYRj/PJKPZ7hmZhSx5augV9Y9LWU2PReBr6 Li3exRv/tL8CJRlXpkyAIsj6fR2lcvV+/5zhKP8ksKe4W47hB60kxZKPZD7p2faMlidz oWPJoPGWpJc3ni9HBjW6xPQW/+rbVVctbmwZeYXCiMVuX6lQ1JmJqK+w71ojYzk34MDc dcw7WpIXmYiQsX8MA7c+xRDAWZVYpkSQ0JsLoW5StvsO2K1T/GhjmRLIIkRoeYMG60Zc x2Fqnz8lxoTus6Y1oy3QyS87szOhp3akP4L2ftd8ewdyn0tVIVmT3Cu8sDaDqhKMt7aP rDYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=k5fj2eFyiqM2jlU2ba4/pqWwiNuWOIISGeWFN9ynyMU=; b=BYmB1/WP01dW62UgiaH+RnoLhAEjX9130E37MgCc9yfiPivH8vwHClI/yWN9WZbGEj 9TF/FLStBjDF/pLR32bEGMs0SA/rpeOa3CVg3HBjHHYx+GXc8hxAgNdFhy5hqY5aHJop 7n2wq7efHVaG/WLEJ8TMKFBRdd5wrzNnY1k6OslJQZbarY2Bu7JQanTsHd/jf1wTsANV aOW0ttmA5MLP4/D5KWIzaiTOLQo3RIDe1/NA2CO1mPldPfHgH2zFvPoVan/vLf+mjv/o GR3u6IkeFQRHhWm3fKCZYPKxS34OcO3fAVCVok0ew3QrwQtPbCI8kfV4JaOtlXvZyGJV fYzw== X-Gm-Message-State: ALoCoQlXEjOIxLLhwvOpUw3v0PJZfLZQyypuLdHHgK/l7emjqoVvDscziIW9/+2R0c79J8rT9pga X-Received: by 10.107.136.216 with SMTP id s85mr3574381ioi.142.1446651426466; Wed, 04 Nov 2015 07:37:06 -0800 (PST) Received: from aither.local ([2601:282:4201:ef5b:65c1:aacb:2a70:f39d]) by smtp.googlemail.com with ESMTPSA id rt5sm1128936igb.20.2015.11.04.07.37.04 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Nov 2015 07:37:05 -0800 (PST) To: John C Klensin , Tom Worster , Alexey Melnikov References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> <56383B2F.6080505@andyet.net> From: Peter Saint-Andre Message-ID: <563A261F.2040904@andyet.net> Date: Wed, 4 Nov 2015 08:37:03 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2015 15:37:09 -0000 On 11/3/15 6:42 AM, John C Klensin wrote: > > > --On Tuesday, November 03, 2015 08:16 -0500 Tom Worster > wrote: > >> The informative sentence in 2.1 Rule 3: >> >> "The primary result of doing so is that uppercase >> characters are mapped to lowercase characters." >> >> is good but I think it's worth spending a few more words to >> spell out that "primary" implies exceptions. >> >> "While the primary result is that uppercase characters are >> mapped to lowercase characters, there are exceptions." >> >> It might nudge a few fore implementers to understand that >> toLowerCase() isn't the right thing. > > But Tom, for many purposes, forms, and language-script > combinations, toLowerCase() is _exactly_ the right thing. For > others, if one adopts the principle of doing no harm, whether > toLowerCase() or toCaseFold() are likely to do less harm depends > on some rather subtle issues including the perspective from > which "harm" is viewed. > > I think one could more accurately restate your comment above as > "... to understand that toLowerCase() is sometimes not the right > thing and that sometimes toCaseFold() is sometimes not the right > thing either. > > I could take a shot at a paragraph explaining that if it is what > people want. Otherwise, I'd be very careful about getting > further into that space than the present text goes. Personally, > I think the text is still dancing around the issue too much, > rather than addressing it, but I may be in the rough. That may be. We could simply remove the offending sentence: OLD 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, as defined in the Unicode Standard [Unicode] (at the time of this writing, the algorithm is specified in Chapter 3 of [Unicode7.0]). The primary result of doing so is that uppercase characters are mapped to lowercase characters. In applications that prohibit conflicting nicknames, this rule helps to reduce the possibility of confusion by ensuring that nicknames differing only by case (e.g., "stpeter" vs. "StPeter") would not be presented to a human user at the same time. NEW 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, as defined in the Unicode Standard [Unicode] (at the time of this writing, the algorithm is specified in Chapter 3 of [Unicode7.0]). In applications that prohibit conflicting nicknames, this rule helps to reduce the possibility of confusion by ensuring that nicknames differing only by case (e.g., "stpeter" vs. "StPeter") would not be presented to a human user at the same time. If we do that, we're no longer dancing around issues. Peter From nobody Wed Nov 4 08:51:45 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C43211B33C3 for ; Wed, 4 Nov 2015 08:51:36 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e5N-2kPAq957 for ; Wed, 4 Nov 2015 08:51:35 -0800 (PST) Received: from smtp74.iad3a.emailsrvr.com (smtp74.iad3a.emailsrvr.com [173.203.187.74]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 292EE1B3381 for ; Wed, 4 Nov 2015 08:51:35 -0800 (PST) Received: from smtp26.relay.iad3a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp26.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 0ECB480435; Wed, 4 Nov 2015 11:51:34 -0500 (EST) Received: by smtp26.relay.iad3a.emailsrvr.com (Authenticated sender: fsb-AT-thefsb.org) with ESMTPSA id A7C9C80312; Wed, 4 Nov 2015 11:51:30 -0500 (EST) X-Sender-Id: fsb@thefsb.org Received: from [10.0.1.2] (c-73-4-147-142.hsd1.ma.comcast.net [73.4.147.142]) (using TLSv1 with cipher DES-CBC3-SHA) by 0.0.0.0:465 (trex/5.5.4); Wed, 04 Nov 2015 11:51:34 -0500 User-Agent: Microsoft-MacOutlook/14.5.7.151005 Date: Wed, 04 Nov 2015 11:51:27 -0500 From: Tom Worster To: Peter Saint-Andre , John C Klensin , Alexey Melnikov Message-ID: Thread-Topic: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> <56383B2F.6080505@andyet.net> <563A261F.2040904@andyet.net> In-Reply-To: <563A261F.2040904@andyet.net> Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 7bit Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2015 16:51:37 -0000 On 11/4/15, 10:37 AM, "Peter Saint-Andre" wrote: > >That may be. We could simply remove the offending sentence: > >OLD > > 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, > as defined in the Unicode Standard [Unicode] (at the time of this > writing, the algorithm is specified in Chapter 3 of > [Unicode7.0]). The primary result of doing so is that uppercase > characters are mapped to lowercase characters. In applications > that prohibit conflicting nicknames, this rule helps to reduce > the possibility of confusion by ensuring that nicknames differing > only by case (e.g., "stpeter" vs. "StPeter") would not be > presented to a human user at the same time. > >NEW > > > 3. Case Mapping Rule: Unicode Default Case Folding MUST be applied, > as defined in the Unicode Standard [Unicode] (at the time of this > writing, the algorithm is specified in Chapter 3 of > [Unicode7.0]). In applications that prohibit conflicting > nicknames, this rule helps to reduce the possibility of > confusion by ensuring that nicknames differing only by case > (e.g., "stpeter" vs. "StPeter") would not be presented to a > human user at the same time. > >If we do that, we're no longer dancing around issues. Splendid idea! Tom From nobody Wed Nov 4 12:05:06 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5B2731B33BC for ; Wed, 4 Nov 2015 12:05:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.61 X-Spam-Level: X-Spam-Status: No, score=-2.61 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hmnv1qiSzBHp for ; Wed, 4 Nov 2015 12:05:03 -0800 (PST) Received: from bsa2.jck.com (bsa2.jck.com [70.88.254.51]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E38B51B33BB for ; Wed, 4 Nov 2015 12:04:59 -0800 (PST) Received: from [198.252.137.10] (helo=JcK-HP8200.jck.com) by bsa2.jck.com with esmtp (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Zu4Id-000LFO-ES; Wed, 04 Nov 2015 15:04:51 -0500 Date: Wed, 04 Nov 2015 15:04:46 -0500 From: John C Klensin To: Peter Saint-Andre , Tom Worster , Alexey Melnikov Message-ID: <7407A8C7DB30384DF67C0EA5@JcK-HP8200.jck.com> In-Reply-To: <563A261F.2040904@andyet.net> References: <0347834EBDC481BD99BDBE67@JcK-HP8200.jck.com> <56302E6D.5030901@andyet.net> <56312AAC.1000300@andyet.net> <56313616.8000801@andyet.net> <563143B9.7020707@andyet.net> <56383B2F.6080505@andyet.net> <563A261F.2040904@andyet.net> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline X-SA-Exim-Connect-IP: 198.252.137.10 X-SA-Exim-Mail-From: john-ietf@jck.com X-SA-Exim-Scanned: No (on bsa2.jck.com); SAEximRunCond expanded to false Archived-At: Cc: precis@ietf.org Subject: Re: [precis] Ambiguity in specification of case mapping in RFC 7613 and draft-ietf-precis-nickname X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2015 20:05:04 -0000 --On Wednesday, November 04, 2015 08:37 -0700 Peter Saint-Andre wrote: >> I could take a shot at a paragraph explaining that if it is >> what people want. Otherwise, I'd be very careful about >> getting further into that space than the present text goes. >> Personally, I think the text is still dancing around the >> issue too much, rather than addressing it, but I may be in >> the rough. > > That may be. We could simply remove the offending sentence: >... > NEW > > > 3. Case Mapping Rule: Unicode Default Case Folding MUST > be applied, > as defined in the Unicode Standard [Unicode] (at the > time of this > writing, the algorithm is specified in Chapter 3 of > [Unicode7.0]). In applications that prohibit > conflicting > nicknames, this rule helps to reduce the possibility of > confusion by ensuring that nicknames differing only by > case > (e.g., "stpeter" vs. "StPeter") would not be presented > to a > human user at the same time. > > If we do that, we're no longer dancing around issues. Different dance, I think, but we may have different objectives. If we keep the explanatory text, it should be accurate. The sentence you removed, by itself, is misleading, so, in that sense, getting rid of it improves accuracy and is an improvement. On the other hand, if we remove the explanation, we encourage those who build libraries to simply apply toCaseFold and believe they don't need to think further about the situation and those who build user-facing systems to call those libraries and do the same thing, secure in the belief that the IETF told them things would be ok. The problem remains that, while toCaseFold works well and as predicted in most locales, in "minority" locales and script and language contexts, it produces results that are surprising and that may be bizarre. It doesn't even catch/match all of the cases one would prefer as well as matching some things that shouldn't be (probably the more harmless of the two types of errors). let me say this strongly: unless the IETF is willing to take the position that we are interested in and supporting an Internet only for "majority" languages and writing systems ("majority" defined by "what works well with Unicode generally and toCaseFold in particular), then a document like this almost certainly needed a paragraph warning about local issues and sensitivities. I can imagine several ways to do that and have suggested outlines for some of them. But just dropping text to avoid saying anything wrong except by implication doesn't do much for me, largely because I believe that an IETF that takes such a position would have outlived its usefulness to the global Internet community. john From nobody Sat Nov 21 05:12:44 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A041D1A9031 for ; Sat, 21 Nov 2015 05:12:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_50=0.8, FREEMAIL_FROM=0.001, GB_I_LETTER=-2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kPAukRguRxdv for ; Sat, 21 Nov 2015 05:12:42 -0800 (PST) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C5C2A1A9035 for ; Sat, 21 Nov 2015 05:12:41 -0800 (PST) Received: from christihudtsmbp.fritz.box ([77.0.27.29]) by mail.gmx.com (mrgmx001) with ESMTPSA (Nemesis) id 0LpKKr-1aWRus3Psr-00fB2t for ; Sat, 21 Nov 2015 14:12:39 +0100 From: Christian Schudt Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-Id: <3A5E5283-7BCC-41DA-A183-6872695D91DC@gmx.de> Date: Sat, 21 Nov 2015 14:12:38 +0100 To: precis@ietf.org Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) X-Mailer: Apple Mail (2.3096.5) X-Provags-ID: V03:K0:YZND4FwTElHtwZy1Fc9irTwDT8t2xUdy0zOXzLtpIOJF4qc8ftK n5QS3uP2o7JMuhxpMSqTyw4KkRt4+7MMd/k9z1hsCi+t/Y+tLwS2C0WfiXJ9bmtnkfF7DK9 jF0kGHpZRmq6BKaw9MZwzDAPAOOQDezShuIX5UwRjeW77jB3b9E+3dirT13/yMHeSU7s5Ca vCHDYcim6SiTHXBWC4OAA== X-UI-Out-Filterresults: notjunk:1;V01:K0:koD5G+YTbI4=:HXsiZKWVJIodK8Q4E9CkBA o0FEJB9+/lrRdILEgiqNh7T1lWxom3qmceAOWa8e3jfzSAGIDO127jQ9yv4Vf68lHrY9aGJ0a SAFCCrn9NdFNAPtxyq9FX9/3+azCZ2pnkBQ2nu3t7AGcAeEnFGRYHz5Nf4B/AE8LyCLQtesc4 KOXJqrYm+eEtMofcGEO0T2whRhP0BM+cIYVBmZRpXogQFNw0N12aFsg71RTWTkEOvl/SGA2OJ Xz/LWZUqHuB9MxR81rYvKyoDCQoriNyWI1Y7T4q256LnHLWsob8pZO7SH9bzRwOfFcMENas+/ kBSUUfJs04qbiQJN2SQFsutW8VZaaSyO7ddICWcp90fEAMqQb+T0Rm/v8w/Zjgsdhuk5TwWFU HZgmgpc3oNZx+Xd97Ou672D5nXx0mAsiXcABVEpwWyGtIGI925xJ+omUo2/niCGXvtLePONRC Ci/rr6SNf/L3UIocE7naKoq9y8EW6RtCoaUeiFRZUY0fuQOUMqWlpXWso6I8S3+bLZsrUoaUF FCG30GsUTvOSvSf0kHKXrOlgQ45Y7y1x13Ksdj/s2ZU9WelpnbO3U9qvvqjSVE3+OLQOp4xm7 bnXYl5qofu17w/PQvjaIDVVhKolKCm177a2a0tAHiuW+Bdtg5T5xv5Db//rQrgQR02CZ4vBWj bj7AZA1r72fru/SK2vpgel1+6kKIa7Qw3v4IFGsE2+nzuqn5yyU4bnP6kL15qqrZN4Qq5Yt15 BfSDrNqf8LemqGqzQy3yZbUG7/By2jg+XthXzC05E6ciKDN+h1jnXEt/tgsXGa0A7TXX/TcHN ROiILpg Archived-At: Subject: [precis] U+212B (ANGSTROM SIGN) in Usernames X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 13:12:43 -0000 Hi, can you please help answering the following question: When doing enforcement of a string of the UsernameCaseMappedProfile as = per RFC 7613 =C2=A7 3.2.2 and the string contains U+212B (ANGSTROM = SIGN), I would first do the preparation and it would be disallowed by = the IdentifierClass because it has a compatibility equivalent (which is = U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE). However, if I stick to the order rules specified in the Precis Core = Framework (RFC 7564 =C2=A7 7), the string would first be normalized with = NFC (RFC 7613 =C2=A7 4.2.2.4), becoming U+00C5, and then later would = pass the IdentifierClass check. If I stick to the rules in RFC 7613 =C2=A7 3.2.2, such a string would be = disallowed. If I stick to the rules in RFC 7564 =C2=A7 7, it would be allowed. What=E2=80=99s correct behavior? Preparation (RFC 7613 =C2=A7 3.2.1) has introduced a workaround for the = HasCompat issue, but only for full- and halfwidth characters (which = U+212B is not). Generally I don=E2=80=99t understand why RFC 7613 violates the order of = rules (compared to RFC 7564 =C2=A7 7): Doing the preparation (checking = the IdentifierClass) first as opposed to do it after the normalization. Thanks for your help and comments, =E2=80=94 Christian= From nobody Sat Nov 21 08:25:28 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B4D7A1ACE42 for ; Sat, 21 Nov 2015 08:25:26 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.702 X-Spam-Level: X-Spam-Status: No, score=-0.702 tagged_above=-999 required=5 tests=[BAYES_20=-0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wKJjn_v68hOP for ; Sat, 21 Nov 2015 08:25:24 -0800 (PST) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A12091ACE41 for ; Sat, 21 Nov 2015 08:25:24 -0800 (PST) Received: from [192.168.123.105] (unknown [75.83.2.34]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 58D8A509BD for ; Sat, 21 Nov 2015 11:25:22 -0500 (EST) From: Sean Leonard Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Message-Id: <38EC381E-C707-480E-A203-EEA403764B8A@seantek.com> Date: Sat, 21 Nov 2015 08:25:21 -0800 To: precis@ietf.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) Archived-At: Subject: [precis] Mapping Cryptographic Passwords in PKCS #8 X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 16:25:26 -0000 Hello PRECIS: Recently I proposed the application/pkcs8-encrypted media type = registration. I thought I would ping this list (now) as well as the = Unicode mailing list (eventually) to weigh in on some parameters that = could use PRECIS. What are your thoughts? In = , application/pkcs8-encrypted is proposed for the PKCS #8 = EncryptedPrivateKeyInfo type. For purposes of our discussion, the format = takes as input an opaque octet string (any octet in the range 00h-FFh, = of any length), and executes various specified algorithms; the result is = a decrypted private key. The most common algorithm is PBKDF2, but any = algorithm can be used (including, for example, a raw symmetric = encryption algorithm such as AES-256). PKCS #8 uses PKCS #5. PKCS #8 and PKCS #5 are embedded in a variety of = other protocols and formats, including PKCS #12. PKCS #8 punts on the issue of character encoding. It says that ASCII or = UTF-8 could be used, but doesn=92t enforce anything in particular. PKCS = #12 specifies UTF-16LE with a terminating NULL character (00h 00h). In the application/pkcs8-encrypted registration, I thought it might be = wise to allow senders and receivers to specify how input (whether user = input or otherwise) gets mapped to the octet string, since it=92s not = part of the format. However my concern at that time was to reflect IANA = character sets, rather than profiles of Unicode. Currently, I am thinking that the parameter =93charset=94 should be = renamed to =93mapping=94 (or =93pw-mapping=94), and =93mapping=94 (or = =93pw-mapping=94) can include the following special values: *pkcs12 for PKCS #12 *precis for RFC 7613 Section 4 (Passwords) Thank you, Sean The relevant part of the template (proposed 2015-11-04) is: Optional parameters: charset: When the private key encryption algorithm incorporates a = =93password" that is an octet string, a mapping between user input and = the octet string is desirable. PKCS #5 [RFC2898] Section 3 recommends = "that applications follow some common text encoding rules"; it then = suggests, but does not recommend, ASCII and UTF-8. This parameter = specifies the charset that a recipient SHOULD attempt first when mapping = user input to the octet string. It has the same semantics as the charset = parameter from text/plain, except that it only applies to the user=92s = input of the password. There is no default value. ualg: When the charset is a Unicode-based encoding, this parameter is a = space-delimited list of Unicode algorithms that a recipient SHOULD first = attempt to apply to the Unicode user input in succession, in order to = derive the octet string. The list of algorithm keywords is defined by = [UNICODE]. =93Tailored operations=94 are operations that are sensitive = to language, which must be provided as an input parameter. If a tailored = operation is called for, the exclamation mark followed by the [BCP47] = language tag specifies the language. For example, "toNFD = toNFKC_Casefold!tr" first applies Normalization Form D, followed by = Normalization Form KC with Case Folding in the Turkish language, = according to [UNICODE] and [UAX31]. The default value of this parameter = is empty, and leaves the matter of whether to normalize, case fold, or = apply other transformations unspecified. From nobody Mon Nov 23 17:31:24 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EB9C01B2B53 for ; Mon, 23 Nov 2015 17:31:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 1.098 X-Spam-Level: * X-Spam-Status: No, score=1.098 tagged_above=-999 required=5 tests=[BAYES_50=0.8, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5O_Pu-UAgy87 for ; Mon, 23 Nov 2015 17:31:19 -0800 (PST) Received: from APC01-PU1-obe.outbound.protection.outlook.com (mail-pu1apc01on0111.outbound.protection.outlook.com [104.47.126.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7781D1B2B51 for ; Mon, 23 Nov 2015 17:31:18 -0800 (PST) Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=duerst@it.aoyama.ac.jp; Received: from [133.2.210.64] (133.2.210.64) by OS1PR01MB0134.jpnprd01.prod.outlook.com (10.161.228.15) with Microsoft SMTP Server (TLS) id 15.1.331.20; Tue, 24 Nov 2015 01:31:13 +0000 To: Sean Leonard References: <5641BCB5.4060409@it.aoyama.ac.jp> From: =?UTF-8?Q?Martin_J._D=c3=bcrst?= Organization: Aoyama Gakuin University Message-ID: <5653BDDB.9020203@it.aoyama.ac.jp> Date: Tue, 24 Nov 2015 10:31:07 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [133.2.210.64] X-ClientProxiedBy: TY1PR0201CA0005.apcprd02.prod.outlook.com (25.164.90.143) To OS1PR01MB0134.jpnprd01.prod.outlook.com (25.161.228.15) X-Microsoft-Exchange-Diagnostics: 1; OS1PR01MB0134; 2:W2NvGTHtSc+yEXu7AKKwZcYaKX+IOPDe8x/4NmsC0B31wF988MUOHoIAxw3KG1Q+IIG/PUsxvilc6LdpsS/WuBOuyrB/mDDkvZaZqGHe0FRMI1w95Kyeh6NSaa4uClC1quTE9VLfGMbguvm6y9ceKg==; 3:7rLu3gLygXN6iF6V5mfqRJbi/w76tn3YykdkTyRhEqE90sSkowjUn1wm6Xs3ePFK6q24BDb0F9sYUu99F4/pW1BFLlBJ0+gS/y1My0rJwQsVTb1Js9Vxtwoja8zJgZjY; 25:QIWO7K8/6t7nYQf9hvZHxwOw6cHYegHbweqd61e15ZDkAPaUGASP+vUW8coU4C91x+cOxC/ZBOO4sQ4tGqj3XheRbUFcdMJWQ+TIIDv2Xo/japNgt5IQKvft2KoWXIR0QfZ/6+dHS/NzoiXqAkB9ZjaSDdzuWE/KR3z4nuz/9L10lXDYSezvO3U2Ox55uJfdTqiyBVi7UXv/aHOgNKTvVgmwmAfYlitgMwiZwkvhg8Y943ruP1kfz90uMQydHiUVWJ+/Qche67n8GoMbGECYSg==; 4:9dQzNy5gaSsIXTNCy60ApZHekKg/3fBOWCgN2BvHuJBJE2IrisAPGl0yU7B+jmW36vMmi0QWTgcU65uFAaMvXfAeyA7w+Nm3vsnU3UBKHePv7Ks/+pBLrEBBcJILf4pQ/5o9uZqX/O6ZHDaBPu/rc1uWGz8yT0KokmNetbTIc3ailLUJG1uAi8ad+ZBj5QY82SXIWVIUUnYZf6411q4IxPxxS81+Q5oUpHdt/EF6Y4o24E8YaEswSEbaf4sHfgXO5T2lYQg0Jg2+EAm1JrWCVxIVHmFH8vB81RYAlq1AVLEg4lUQagKWjZFF7wzSrqYPSitv8gj2P9KvmWPtkTlKBEwsgZA8ckfP7wk+Q1aw6Civprb1ylUFOMDgqh9Oeh8B X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:OS1PR01MB0134; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(2401047)(8121501046)(520078)(5005006)(3002001)(10201501046); SRVR:OS1PR01MB0134; BCL:0; PCL:0; RULEID:; SRVR:OS1PR01MB0134; X-Forefront-PRVS: 0770F75EA9 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(6049001)(6009001)(24454002)(189002)(52034003)(377454003)(479174004)(15594002)(199003)(51444003)(87976001)(87266999)(40100003)(59896002)(15975445007)(5007970100001)(5004730100002)(50466002)(106356001)(74482002)(561944003)(54356999)(77096005)(189998001)(83506001)(110136002)(2950100001)(5001960100002)(122386002)(4001350100001)(86362001)(23676002)(92566002)(19580395003)(19580405001)(33656002)(81156007)(80316001)(101416001)(65816999)(3846002)(105586002)(551544002)(50986999)(5008740100001)(66066001)(65956001)(97736004)(76176999)(42186005)(64126003)(586003)(6116002)(65806001)(47776003)(3940600001); DIR:OUT; SFP:1102; SCL:1; SRVR:OS1PR01MB0134; H:[133.2.210.64]; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:0; LANG:en; Received-SPF: None (protection.outlook.com: it.aoyama.ac.jp does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtPUzFQUjAxTUIwMTM0OzIzOmhOanNoVHpHZUgwZTdvaDlscEE2WDB0ZVNP?= =?utf-8?B?ZXhMSXdaVnBEaGxTQlJEN3dXRy9RTzdyNk1lcm12eGVaMWRhL2drVFoyZktD?= =?utf-8?B?dXNQRjdUNlJSMjNURC9TUVlYaEc0a29uaGxPVTFkREgwMHVXUHNaMXFWVzh4?= =?utf-8?B?bU40eVBXSm5vRVVJeWRRd3ZGblJLRDFlbzFLcGhuQk1SV2IveXB4TWNvWGlr?= =?utf-8?B?bkluNXNxcmZESHZMcUI4bU1IUTBpSklWOXlGNmVXMVcyRzhxRFEvdHV6VXRZ?= =?utf-8?B?eTRtcVJFUlB3dE9ucjFUQWhHSEg2OEdrV0FnRWZ5SXJXenEyZ1k3NjlTY01y?= =?utf-8?B?K2NJdHA3WXd4cVhpUjg5a1k3OGI4a1ZuQUtjb21lS3ZoTXlkSklZbGpMdWJi?= =?utf-8?B?bitiR0g5bW1SaGdhSHpSVW9FUUFERWhUV3dKZDVDRG9IK0h6UU5VL1ZBL3ZY?= =?utf-8?B?UmRTQTM0cm1WMWNWdGhnb2NXLzJCbFVnczk3WlJPb1lucXFqbUZpb1VLZGVR?= =?utf-8?B?YUJxeUZ2M0M3VHJPZ21TMXVjTHFxYXBSYlBpQjZyZzhyMjBMa3pXM1oyUytW?= =?utf-8?B?MmpXYW5LbVhoUVIrRTRDQzVTWTdjejZISVU0Zk4wR0J2ei90bXg5dHg1Tkt6?= =?utf-8?B?dTV2K2luK0gzYnIyRmNXYmszd3pEdkpKMzFzQ3VhdDMyM3p4MG1IVTZCYnJE?= =?utf-8?B?VHJScDZEcmhJNVZISzdqQVRYTmlkV3hHc2FRaEFRNmtYTTRvNDRwYk9WMUIy?= =?utf-8?B?MEtzVll5ayszVmw5R3hzbUI0YUt3eG41SmdqOEt3dmVrWTBReTh0QW5lUEQ1?= =?utf-8?B?a0Nzek4zV25SRE81blJvMnRtczVzcEcyWG9Oc2RLV3VGZ1NkaUVVRnN4REdP?= =?utf-8?B?eFB4c1N4K2ljTG9IT1l2WUY0Z2F4VEVQWGdKaWV0WmFkRk1SZkU2K0MrSEFC?= =?utf-8?B?c3dvTVpNc2lQNHY0LzlUUys3SnpoVndhRFl2V3VtM0s2aUpLdHBDZjFUV3ZU?= =?utf-8?B?SDBHOGo1bGpVb2tDVnVOUGFzVTdCd2FQemp4cDVtVVpjbVQ3STc5SGtCREp5?= =?utf-8?B?Y0tTVmM1SGJSeXZmMEFkVlFWb0NXNXh1Zk1TRUtHQkdpakhyaVZ2WUk3emJ0?= =?utf-8?B?TnpNMEFOT1l4aWNOVmhtM2w1bHJjdko5TDJBSTk3ajF3dlp6L2I3NWdjY3I4?= =?utf-8?B?TkNqZUZyRHFGM3A3Yi9wbEVZejYyUzlyVE5IelkzV0FQYjVBOGlvUGVOV25m?= =?utf-8?B?c0JUc2VLZ3NZdnVHNjM2WlRzdHM0aDZTU0doMHNRQ0lwR3VSb0ZQWStWREJV?= =?utf-8?B?NGpsM3JIaXlPMlYzYkpKbDJ3bXVkczlGajVqaFVXWXFFRHNLT1N6Z0R3aGdy?= =?utf-8?B?VEt0blNhR1UzNnphWm9KYTBLbENMV0kxMTNwcXdoZFZHNitYWi9neFdzdGUx?= =?utf-8?B?ZmNWQkpNcDd5LzhBMjZxOHlCMGdubkVSemJqRUI1Z2taekNsaEtnc2g4WE5z?= =?utf-8?B?UDVDU1VBZE8vbDVhSlVrU25nVFFMeG5rMm5xQUhsNkNCN0hxOHJDUWR5a3Fs?= =?utf-8?B?dmREOTc1MksrVzJsTXBUd2g1anpFZTdyV1c0YzQxb3FlcmFsR0VWbldQM1lW?= =?utf-8?B?VzBrNVVjbDBSRWdhNEZ5TlhoTGMvWkdSVU5MMk5EeWtRYUhTSE85ejNZQzJx?= =?utf-8?B?WTFxeVEzSGZwUFgxNG9FVTlGZnhkMHFKS0ZMbVIyU0FRSTdZYVAvY2FyMGk3?= =?utf-8?B?NnFiTXlWR2xFanE4dURrVjZMQ0xIUndMcTk3anZ5V3NsSDVobDlHTEdRVUxx?= =?utf-8?B?NXYwaTRTcytadWZrdExyQUZTQkdldkpMVkJneFBGVTdBVFNEa2Q4V3ZOT3Bw?= =?utf-8?B?SklCQjlnenFuZWJYSzBxRWsrYnpNZ0xuSVFjUlpPOXZBblAxMnJ2SWh5cGtB?= =?utf-8?Q?gizTzW/QJIEhQcqgujnuWUsO7apRYI=3D?= X-Microsoft-Exchange-Diagnostics: 1; OS1PR01MB0134; 5:e4hc9g+H/SDK3OMD8LnYArH8GeUqHGKt7gl0clQGv0Iu7WgsRvqNN7ec2dtP2rPGC3zRnnZBBoxKyOpVnnMpR2eP6fE7CNDhWPmKftk8frZLVXlFfTtCUaPaNfJmzicXENEExF5SeGm4G0JUi7eHGg==; 24:rRXzIDi3uW4mA4SxnQxXmKq6fAXwFQxMjVagvfDCNa4FjcXJIN5hmwioWGWU1TLgipolp0Tysrb4OMMYG8XLbvJKGK4VjWuTEJV6kUs4AuQ= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: it.aoyama.ac.jp X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Nov 2015 01:31:13.1686 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: OS1PR01MB0134 Archived-At: Cc: "media-types@iana.org" , "precis@ietf.org" Subject: Re: [precis] [media-types] Internet media type application/pkcs8-encrypted rev 2 X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2015 01:31:23 -0000 Hello Sean, I have cc'ed the precis mailing list because some of what I'll write below is relevant for the discussion you have started there. This is also the reason why I'm keeping most the previous context. On 2015/11/11 00:25, Sean Leonard wrote: > Hello Martin, > > On Nov 10, 2015, at 1:45 AM, Martin J. Dürst wrote: > >> Hello Sean, >> >> I have a few questions re. your registration below. >> >> On 2015/11/05 14:57, Sean Leonard wrote: >>> Hello: >>> >>> To keep this moving, trying a different thing. Please review. >>> >>> Sean >>> >>> ***** >>> >>> Type name: application >>> >>> Subtype name: pkcs8-encrypted >>> >>> Required parameters: N/A >>> >>> Optional parameters: >>> charset: When the private key encryption algorithm incorporates a “password" that is an octet string, a mapping between user input and the octet string is desirable. PKCS #5 [RFC2898] Section 3 recommends "that applications follow some common text encoding rules"; it then suggests, but does not recommend, ASCII and UTF-8. This parameter specifies the charset that a recipient SHOULD attempt first when mapping user input to the octet string. It has the same semantics as the charset parameter from text/plain, except that it only applies to the user’s input of the password. There is no default value. >> >> Why does it say "This parameter specifies the charset that a recipient SHOULD attempt *first*" here? Can't that encoding just be specified as such? >> >> At least for future, similar efforts, it would be extremely desirable to not leave character encoding open like this, but just to nail it down to UTF-8. > > There seems to be something of a “cultural disconnect” between the security people and the I18N/UI/UX people. > > The I18N/UI/UX people want well-defined interfaces that work with users “in their own language”, whether that language is visual, aural, tactile, symbolic, pictorial, etc. Invariably this involves Unicode and a large character repertoire such as 💩 and 大便所. > > In contrast, the security people find open-ended things like Unicode to be anathema and would much rather restrict the range of inputs to a small and preferably uniformly distributed set of values. And there are good reasons for that, because when you introduce bias into cryptographic protocols, it turns out that it is a lot easier to cryptanalyze the results. > > The common security protocols that I have seen that take passwords, hand-wave about character sets and encodings and define the password to be an octet string. This is great for universality but bad for human input. PBKDF2 (PKCS #5, on which this PKCS #8 EncryptedPrivateKeyInfo registration is based) is a leading example of the “octet string” approach. Ultimately, the algorithms don’t care what encoding it’s in, as long as they get a blob of bits (octets). > > My knowledge of implementations of PKCS #5/#8/#12 suggests that there are many applications out there that give zero thought to the encoding issue, which means that they will take user input “As-Is”, i.e., in the current code page. > > Note that PKCS #12 defines the input to this structure as a UTF-16LE encoded character string, *with* a terminating U+0000 NULL character (i.e., the octets 00 00). This is really “weird” except of course for the fact that Microsoft invented it and then shipped it without too much thought, in which case, all weirdness can be explained. > > It is a design criteria that if you extract such an EncryptedPrivateKeyInfo blob from a PKCS #12 file, that you should be able to process it. If you specify UTF-8 as the one, single, true encoding of the password for application/pkcs8-encrypted, that can’t happen. That's just fine, in this specific case. I have explicitly prefaced my remark above with "At least in the future". But if we know that the password is encoded in UTF-16LE, then why doesn't your registration just say "This parameter specifies the charset" rather than the handwavy "This parameter specifies the charset that a recipient SHOULD attempt *first*". > Furthermore, UTF-8 is not uniformly distributed across the octet range. If your users are in US-English they are highly likely to have octets in 20-7E. Octets in 00-1F will be pretty rare. And if you choose scalar values randomly in Unicode (regardless of assignment), you will see a *lot* of F0-F4 but virtually none in 00-7F. And in spite of all this, octets F5-FF will *never* appear in UTF-8. > > It turns out that we have a pretty good source of uniformity and universality: characters in the US-ASCII range 20-7E. Many password input boxes will only accept US-ASCII and so user’s non-US-English keyboards will switch to US-ASCII mode for the purpose of providing input to such boxes. What matters is not so much the specific characters, so much as a reasonable selection of arbitrary buttons that a user can push *across a wide range of devices*. This ends up giving you 5-6 bits of entropy per user input. So the need for UTF-8 or any particular encoding is actually not as great as some people perceive. My comment was specifically trying to say: If you use something more than US-ASCII, make it UTF-8. I think that's also the general policy of the IETF. As for entropy, the entropy needs to be measured over the whole string. It's clear that in UTF-8 bytes, a password in the ASCII range is shorter than a similar-length (in terms of charaters) password in a non-Latin script. The entropy of each byte will be lower, but the entropy of the overall password should be about the same. Something that's very important for passwords is how easy they are to remember for actual people. It should be obvious that it's easier for somebody to remember a password in the language/script they use every day than in some foreign gibberish. > Overall I think that a standard such as IEEE 802.11 strikes a reasonable balance. (See 802.11-2012 Annex M.4, which is informative, but is pretty much the worldwide de-facto standard practice.) In 802.11, the input to PBKDF2 is between 8-63 ASCII-encoded characters in the range 20-7E, or 64 hexadecimal characters that convert directly to 32 octets. So it's up to 63 ASCII characters but only up to 32 octets that may e.g. be used for UTF-8? That doesn't strike me as a reasonable balance; it puts a much stronger length limitation on some scripts outside ASCII. > *** > To answer your questions directly: >> Why does it say "This parameter specifies the charset that a recipient SHOULD attempt *first*" here? >> Can't that encoding just be specified as such? > > > The parameter is not cryptographically protected so it is subject to tampering or substitution. Furthermore, a good-faith but naïve sender may put some encoding (e.g., UTF-8) but not have the means to verify that the encoding actually works, because the user did not supply the password. Basically it’s a good-faith first effort, but this parameter can’t meaningfully restrict what the sender or receiver attempt to do. That essentially applies to any single parameter in any single media type registration, and in much more of what the IETF does. Yet this is virtually never called out, because otherwise, IETF documents would be full of such stuff and very hard to read. > Also, I am not sure how to specify the NULL suffix in the PKCS #12-extracted case. That may suggest that you are going down the wrong path here. > I suppose it could just be “+0” or something. > >> >>> ualg: When the charset is a Unicode-based encoding, this parameter is a space-delimited list of Unicode algorithms that a recipient SHOULD first attempt to apply to the Unicode user input in succession, in order to derive the octet string. The list of algorithm keywords is defined by [UNICODE]. “Tailored operations” are operations that are sensitive to language, which must be provided as an input parameter. If a tailored operation is called for, the exclamation mark followed by the [BCP47] language tag specifies the language. For example, "toNFD toNFKC_Casefold!tr" first applies Normalization Form D, followed by Normalization Form KC with Case Folding in the Turkish language, according to [UNICODE] and [UAX31]. The default value of this parameter is empty, and leaves the matter of whether to normalize, case fold, or apply other transformations unspecified. >> >> "When the charset is": Is this the charset parameter, or the actual encoding of the password? > > Admittedly this was vague. First draft. I am not sure what it should be. Per PKCS #5, the "Actual Encoding" is just an octet string of arbitrary length. > > I would limit this to cases when the charset parameter is present and defined. Makes it easier. > >> >> What is a "Unicode algorithm”? > > Conformance Clause D17. Well, this, via the term "Named Unicode Algorithm" points to table 3.1 (page 93 in Unicode V 8.0). >> Reading on and looking at the examples, the intent becomes clearer, at least to somebody who has seen things such toNFD and toNFKC and Casefold, but I hope we can avoid "specification by example" here. > > In fairness, “toNFD” and “toNFKC” are not defined terms. However, NFD (D118) and NFKC (D121) are. Yes, but not as (Named) Unicode Algorithms. > I would rather not create Yet Another Registry of things. I'd agree in principle. > The terms are in fact defined in [UNICODE] in the conformance clauses. Yes, but there are many other things defined there, too. > My usability perception is that if people really want to use Unicode in their passwords, canonicalization is a very useful property to preserve. Case folding/case mapping are not so useful, as most systems like to have case-sensitive passwords for greater entropy, but “most systems” is not “all systems” so we shouldn’t preclude the use of case algorithms. As for other algorithms such as line breaking, character segmentation, Hangul syllable name generation, etc., the short answer is “I don’t know”. (These are all reasons why people stick with ASCII passwords, by the way.) Line breaking, character segmentation, Hangul syllable name generation,... are completely irrelevant for passwords and passphrases. Also, many algorithms come with options or parameters. >> Also, if there is indeed a list of algorithm identifiers in [UNICODE], then it would be good to give a Section number. Is the intent that each and every algorithm named somewhere in [UNICODE] is implemented? My rough guess would be that the average password input implementation implements only the identity transform. [I would of course be positively surprised if I were wrong.] > > See above; main thing that worries me is Normalization Forms. > >> >> Also, references for [UNICODE], [BCP47], and [UAX31] should be give so that this registration is self-containing. > > Ok. > > Another possibility is that this registration goes back to “rev 1”, i.e., no optional parameters about the character encoding at all. I think that is perfectly defensible. But it is not particularly i18n-friendly. I'm not sufficiently familiar with the format and the actual use cases, but my suggestion would be to check what's actually out there in the field (such as the Microsoft UTF-16LE including final NULL), and select or create a list of parameters/algorithms (with a registry if it turns out to be needed). To that, add a way to reference PRECIS, even if it's not currently used, because that includes the expertise/recommendations of experts. The current proposal just essentially saying: Unicode may define some of the pieces you may want to use here, and may have labels for them, so just give it a try. I'm not at all sure this will help interoperability, except by similar accidents like the Microsoft one that you described above. Regards, Martin. > Regards, > > Sean > >> >> Regards, Martin. >> >>> Encoding considerations: binary >>> >>> Security considerations: >>> Carries a cryptographic private key. See Section 6 of RFC 5958. >>> EncryptedPrivateKeyInfo PKCS #8 data contains exactly one private key. Poor password choices, weak algorithms, or improper parameter selections (e.g., insufficient salting rounds) will make the confidential payloads much easier to compromise. >>> >>> Interoperability considerations: >>> PKCS #8 is a widely recognized format for private key information on all modern cryptographic stacks. The encrypted variation in this registration, EncryptedPrivateKeyInfo (Section 3, Encrypted Private Key Info, of RFC 5958, and Section 6 of PKCS #8), is less widely used for exchange than PKCS #12, but it is much simpler to implement. The contents are exactly one private key (with optional attributes), so the possibility for hidden "easter eggs" in the payload such as unexpected certificates or miscellaneous secrets is drastically reduced. >>> >>> Published specification: >>> PKCS #8 v1.2, November 1993 (republished as RFC 5208, May 2008); RFC 5958, August 2010 >>> >>> Applications that use this media type: >>> Machines, applications, browsers, Internet kiosks, and so on, that support this standard allow a user to import, export, and exercise a single private key. >>> >>> Fragment identifier considerations: N/A >>> >>> Additional information: >>> >>> Deprecated alias names for this type: N/A >>> Magic number(s): None. >>> File extension(s): .p8e >>> Macintosh file type code(s): N/A >>> >>> Person & email address to contact for further information: >>> Sean Leonard >>> >>> Intended usage: COMMON >>> >>> Restrictions on usage: None. >>> >>> Author: >>> RSA, EMC, IETF >>> >>> Change controller: The IETF >>> >>> Provisional registration? (standards tree only): No >>> >>> >>> >>> _______________________________________________ >>> media-types mailing list >>> media-types@ietf.org >>> https://www.ietf.org/mailman/listinfo/media-types >>> > From nobody Tue Nov 24 01:56:52 2015 Return-Path: X-Original-To: precis@ietfa.amsl.com Delivered-To: precis@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 70FEB1B2ED5 for ; Tue, 24 Nov 2015 01:56:51 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.399 X-Spam-Level: X-Spam-Status: No, score=0.399 tagged_above=-999 required=5 tests=[BAYES_50=0.8, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pV57a2EoI29k for ; Tue, 24 Nov 2015 01:56:45 -0800 (PST) Received: from mxout-08.mxes.net (mxout-08.mxes.net [216.86.168.183]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 948911B2ED1 for ; Tue, 24 Nov 2015 01:56:45 -0800 (PST) Received: from [192.168.123.7] (unknown [75.83.2.34]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 600FA509BB; Tue, 24 Nov 2015 04:56:43 -0500 (EST) To: =?UTF-8?Q?Martin_J._D=c3=bcrst?= References: <5641BCB5.4060409@it.aoyama.ac.jp> <5653BDDB.9020203@it.aoyama.ac.jp> From: Sean Leonard Message-ID: <565433DF.50304@seantek.com> Date: Tue, 24 Nov 2015 01:54:39 -0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <5653BDDB.9020203@it.aoyama.ac.jp> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms030004020303030104030100" Archived-At: Cc: "media-types@iana.org" , "precis@ietf.org" Subject: Re: [precis] [media-types] Internet media type application/pkcs8-encrypted rev 2 X-BeenThere: precis@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Preparation and Comparison of Internationalized Strings List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2015 09:56:51 -0000 This is a cryptographically signed message in MIME format. --------------ms030004020303030104030100 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable On 11/23/2015 5:31 PM, Martin J. D=C3=BCrst wrote: > Hello Sean, > > I have cc'ed the precis mailing list because some of what I'll write=20 > below is relevant for the discussion you have started there. This is=20 > also the reason why I'm keeping most the previous context. > > On 2015/11/11 00:25, Sean Leonard wrote: >> Hello Martin, >> >> On Nov 10, 2015, at 1:45 AM, Martin J. D=C3=BCrst =20 >> wrote: >> >>> Hello Sean, >>> >>> I have a few questions re. your registration below. >>> >>> On 2015/11/05 14:57, Sean Leonard wrote: >>>> Hello: >>>> >>>> To keep this moving, trying a different thing. Please review. >>>> >>>> Sean >>>> >>>> ***** >>>> >>>> Type name: application >>>> >>>> Subtype name: pkcs8-encrypted >>>> >>>> Required parameters: N/A >>>> >>>> Optional parameters: >>>> charset: When the private key encryption algorithm incorporates a=20 >>>> =E2=80=9Cpassword" that is an octet string, a mapping between user i= nput=20 >>>> and the octet string is desirable. PKCS #5 [RFC2898] Section 3=20 >>>> recommends "that applications follow some common text encoding=20 >>>> rules"; it then suggests, but does not recommend, ASCII and UTF-8.=20 >>>> This parameter specifies the charset that a recipient SHOULD=20 >>>> attempt first when mapping user input to the octet string. It has=20 >>>> the same semantics as the charset parameter from text/plain, except = >>>> that it only applies to the user=E2=80=99s input of the password. Th= ere is=20 >>>> no default value. >>> >>> Why does it say "This parameter specifies the charset that a=20 >>> recipient SHOULD attempt *first*" here? Can't that encoding just be=20 >>> specified as such? >>> >>> At least for future, similar efforts, it would be extremely=20 >>> desirable to not leave character encoding open like this, but just=20 >>> to nail it down to UTF-8. >> >> There seems to be something of a =E2=80=9Ccultural disconnect=E2=80=9D= between the=20 >> security people and the I18N/UI/UX people. >> >> The I18N/UI/UX people want well-defined interfaces that work with=20 >> users =E2=80=9Cin their own language=E2=80=9D, whether that language i= s visual,=20 >> aural, tactile, symbolic, pictorial, etc. Invariably this involves=20 >> Unicode and a large character repertoire such as =F0=9F=92=A9 and =E5=A4= =A7=E4=BE=BF=E6=89=80. >> >> In contrast, the security people find open-ended things like Unicode=20 >> to be anathema and would much rather restrict the range of inputs to=20 >> a small and preferably uniformly distributed set of values. And there = >> are good reasons for that, because when you introduce bias into=20 >> cryptographic protocols, it turns out that it is a lot easier to=20 >> cryptanalyze the results. >> >> The common security protocols that I have seen that take passwords,=20 >> hand-wave about character sets and encodings and define the password=20 >> to be an octet string. This is great for universality but bad for=20 >> human input. PBKDF2 (PKCS #5, on which this PKCS #8=20 >> EncryptedPrivateKeyInfo registration is based) is a leading example=20 >> of the =E2=80=9Coctet string=E2=80=9D approach. Ultimately, the algori= thms don=E2=80=99t care=20 >> what encoding it=E2=80=99s in, as long as they get a blob of bits (oct= ets). >> >> My knowledge of implementations of PKCS #5/#8/#12 suggests that there = >> are many applications out there that give zero thought to the=20 >> encoding issue, which means that they will take user input =E2=80=9CAs= -Is=E2=80=9D,=20 >> i.e., in the current code page. >> >> Note that PKCS #12 defines the input to this structure as a UTF-16LE=20 >> encoded character string, *with* a terminating U+0000 NULL character=20 >> (i.e., the octets 00 00). This is really =E2=80=9Cweird=E2=80=9D excep= t of course for=20 >> the fact that Microsoft invented it and then shipped it without too=20 >> much thought, in which case, all weirdness can be explained. >> >> It is a design criteria that if you extract such an=20 >> EncryptedPrivateKeyInfo blob from a PKCS #12 file, that you should be = >> able to process it. If you specify UTF-8 as the one, single, true=20 >> encoding of the password for application/pkcs8-encrypted, that can=E2=80= =99t=20 >> happen. > > That's just fine, in this specific case. I have explicitly prefaced my = > remark above with "At least in the future". > > But if we know that the password is encoded in UTF-16LE, then why=20 > doesn't your registration just say "This parameter specifies the=20 > charset" rather than the handwavy "This parameter specifies the=20 > charset that a recipient SHOULD attempt *first*". See below... > > >> Furthermore, UTF-8 is not uniformly distributed across the octet=20 >> range. If your users are in US-English they are highly likely to have = >> octets in 20-7E. Octets in 00-1F will be pretty rare. And if you=20 >> choose scalar values randomly in Unicode (regardless of assignment),=20 >> you will see a *lot* of F0-F4 but virtually none in 00-7F. And in=20 >> spite of all this, octets F5-FF will *never* appear in UTF-8. >> >> It turns out that we have a pretty good source of uniformity and=20 >> universality: characters in the US-ASCII range 20-7E. Many password=20 >> input boxes will only accept US-ASCII and so user=E2=80=99s non-US-Eng= lish=20 >> keyboards will switch to US-ASCII mode for the purpose of providing=20 >> input to such boxes. What matters is not so much the specific=20 >> characters, so much as a reasonable selection of arbitrary buttons=20 >> that a user can push *across a wide range of devices*. This ends up=20 >> giving you 5-6 bits of entropy per user input. So the need for UTF-8=20 >> or any particular encoding is actually not as great as some people=20 >> perceive. > > My comment was specifically trying to say: If you use something more=20 > than US-ASCII, make it UTF-8. I think that's also the general policy=20 > of the IETF. As for entropy, the entropy needs to be measured over the = > whole string. It's clear that in UTF-8 bytes, a password in the ASCII=20 > range is shorter than a similar-length (in terms of charaters)=20 > password in a non-Latin script. The entropy of each byte will be=20 > lower, but the entropy of the overall password should be about the same= =2E > > Something that's very important for passwords is how easy they are to=20 > remember for actual people. It should be obvious that it's easier for=20 > somebody to remember a password in the language/script they use every=20 > day than in some foreign gibberish. Actually, the most important criteria for passwords is that they are=20 inputtable: capable of being input into a computing system. The=20 passphrase "=C3=A9clairs sont si d=C3=A9licieuX" is nice and memorable, b= ut if you=20 cannot put in "=C3=A9" (or "X") in the computer systems where you're goin= g to=20 use the thing, you're screwed. Viewed through this lens, many cryptographic systems are so-defined to=20 accept secret input as octet strings, where neither the mathematical=20 algorithms nor the protocols inherently restrict the range of the=20 octets. Whether "=C3=A9" maps to E9, C3 A9, E9 00, or 82 [code page 437] = is=20 relevant because if the user tries to input =C3=A9 but the math part does= n't=20 get the original octets on the input, it won't work. Scenarios where PKCS #8 blobs might be accessed at device boot-time are=20 possible. In those scenarios, the character set/encoding of the system=20 may well be something other than UTF-8. Regardless of the encoding, the=20 boot-time input methods may be severely restricted. (There is a recent=20 thread on the Unicode mailing list, related to passwords, that raises=20 this issue.) PKCS #8/PKCS #5 mention ASCII and UTF-8 as complementary=20 possibilities--this registration does not attempt to disturb that. The handwavy text, "This parameter specifies the charset that a=20 recipient SHOULD attempt *first*", recognizes the mathematical reality.=20 Suppose an Evildoer (pick your preferred one) encodes all of its private = keys with the parameter charset=3Dus-ascii. But the actual input has some= =20 character =C3=B1 that is clearly outside the range. When the Good Guys=E2= =84=A2 try=20 to break the encryption, if they only attempt characters in us-ascii=20 range, they are never going to decrypt the thing. Ditto for UTF-8=20 encoding if the correct input (meaning the input that actually results=20 successfully decrypting the private key payload) contains FFh somehow,=20 or some overlong encoding. If this parameter were baked into the PKCS #8 format, perhaps a stronger = claim could be made in the specification text. (Compare with PKCS #12.)=20 But as it stands, the parameter is just a "hint"...and for a long time,=20 implementations will probably not bother to put this parameter in. > >> Overall I think that a standard such as IEEE 802.11 strikes a=20 >> reasonable balance. (See 802.11-2012 Annex M.4, which is informative, = >> but is pretty much the worldwide de-facto standard practice.) In=20 >> 802.11, the input to PBKDF2 is between 8-63 ASCII-encoded characters=20 >> in the range 20-7E, or 64 hexadecimal characters that convert=20 >> directly to 32 octets. > > So it's up to 63 ASCII characters but only up to 32 octets that may=20 > e.g. be used for UTF-8? That doesn't strike me as a reasonable=20 > balance; it puts a much stronger length limitation on some scripts=20 > outside ASCII. Take a look at 802.11-2012 Annex M.4. Here is a direct link:=20 . When the input is 64 hexadecimal characters, PBKDF2 is bypassed: the=20 resulting 32 octets are used directly as the RSNA pre-shared key (PSK).=20 The advantage of defining the protocol that way is that a user interface = (aka, password prompt) for Wi-Fi devices only needs to have one text box = field: the implementation then auto-detects which path to use based on=20 the length. Otherwise, all Wi-Fi devices on the planet would have to=20 have user interfaces designed with additional dropdowns, checkboxes,=20 radio buttons, etc. to pick between the (ASCII) password and direct=20 input of the PSK. (It stands to reason that every password input for=20 every SSID has a corresponding valid 64-hexacdecimal-character PSK.) 802.11-2012 does not define an algorithm for cases when the passphrase=20 has characters that encode outside of the range of 32 to 126. I do not=20 know what other implementations do worldwide if non-ASCII characters are = entered. As some have observed (on the Unicode mailing list), some=20 operating systems, e.g., iOS, always enforce Latin script for password=20 box entry. > >> *** >> To answer your questions directly: >>> Why does it say "This parameter specifies the charset that a=20 >>> recipient SHOULD attempt *first*" here? >>> Can't that encoding just be specified as such? >> >> >> The parameter is not cryptographically protected so it is subject to=20 >> tampering or substitution. Furthermore, a good-faith but na=C3=AFve se= nder=20 >> may put some encoding (e.g., UTF-8) but not have the means to verify=20 >> that the encoding actually works, because the user did not supply the = >> password. Basically it=E2=80=99s a good-faith first effort, but this=20 >> parameter can=E2=80=99t meaningfully restrict what the sender or recei= ver=20 >> attempt to do. > > That essentially applies to any single parameter in any single media=20 > type registration, and in much more of what the IETF does. Yet this is = > virtually never called out, because otherwise, IETF documents would be = > full of such stuff and very hard to read. Perhaps. First of all such warnings about "not being cryptographically=20 protected" tend to show up in many Security Considerations sections, of=20 which I have seen many in the IETF. Second of all, the content inside=20 PKCS #8 EncryptedPrivateKeyInfo *is* cryptographically protected (to a=20 certain extent, depends on the algorithms being used), but an=20 implementation really needs to be vigilant not to treat the stuff=20 outside, i.e., the media type parameters, as the same as on the inside. > >> Also, I am not sure how to specify the NULL suffix in the PKCS=20 >> #12-extracted case. > > That may suggest that you are going down the wrong path here. > >> I suppose it could just be =E2=80=9C+0=E2=80=9D or something. >> >>> >>>> ualg: When the charset is a Unicode-based encoding, this parameter=20 >>>> is a space-delimited list of Unicode algorithms that a recipient=20 >>>> SHOULD first attempt to apply to the Unicode user input in=20 >>>> succession, in order to derive the octet string. The list of=20 >>>> algorithm keywords is defined by [UNICODE]. =E2=80=9CTailored operat= ions=E2=80=9D=20 >>>> are operations that are sensitive to language, which must be=20 >>>> provided as an input parameter. If a tailored operation is called=20 >>>> for, the exclamation mark followed by the [BCP47] language tag=20 >>>> specifies the language. For example, "toNFD toNFKC_Casefold!tr"=20 >>>> first applies Normalization Form D, followed by Normalization Form=20 >>>> KC with Case Folding in the Turkish language, according to=20 >>>> [UNICODE] and [UAX31]. The default value of this parameter is=20 >>>> empty, and leaves the matter of whether to normalize, case fold, or = >>>> apply other transformations unspecified. >>> >>> "When the charset is": Is this the charset parameter, or the actual=20 >>> encoding of the password? >> >> Admittedly this was vague. First draft. I am not sure what it should=20 >> be. Per PKCS #5, the "Actual Encoding" is just an octet string of=20 >> arbitrary length. >> >> I would limit this to cases when the charset parameter is present and = >> defined. Makes it easier. >> >>> >>> What is a "Unicode algorithm=E2=80=9D? >> >> Conformance Clause D17. > > Well, this, via the term "Named Unicode Algorithm" points to table 3.1 = > (page 93 in Unicode V 8.0). > > >>> Reading on and looking at the examples, the intent becomes clearer,=20 >>> at least to somebody who has seen things such toNFD and toNFKC and=20 >>> Casefold, but I hope we can avoid "specification by example" here. >> >> In fairness, =E2=80=9CtoNFD=E2=80=9D and =E2=80=9CtoNFKC=E2=80=9D are = not defined terms. However, NFD=20 >> (D118) and NFKC (D121) are. > > Yes, but not as (Named) Unicode Algorithms. > >> I would rather not create Yet Another Registry of things. > > I'd agree in principle. > >> The terms are in fact defined in [UNICODE] in the conformance clauses.= > > Yes, but there are many other things defined there, too. Yes... If you meant to refer to [UNICODE] Table 3.1 (page 93, Unicode V8.0) as=20 a canonical list of names...I do not find that satisfactory because the=20 only entries for normalization are "Normalization" and "Identifier=20 Normalization", but the most important kind of transformation is to pick = which Normalization Form you want: NFC, NFD, NFKC, or NFKD. (I am under=20 the impression that NFKC is the "Best" for passwords, but the PRECIS=20 profile for passwords uses NFC. Not sure exactly why but I will leave=20 that one alone.) Really, algorithms to do character transformations, prohibitions,=20 substitutions, etc. in the Internet context, would seem to fall under=20 PRECIS. Therefore maybe the proper pre-existing "registry" of things is=20 just PRECIS profiles. Developing from my last PRECIS e-mail on the topic, I am leaning towards = a single parameter: pw-mapping with special values: *pkcs12 =3D UTF-16LE with U+0000 NULL terminator *precis =3D PRECIS password profile, i.e., OpaqueString from Section 4= =20 for RFC 7613 (always UTF-8) *precis-XXX =3D PRECIS profile named by XXX *hex =3D hexadecimal input: as with 802.11, the input is mapped to= =20 0-9, A-F, and then converted directly to octets. If there are an odd=20 number of hex digits, the final digit 0 is appended, or an error=20 condition may be raised. *dtmf =3D The characters "0"-"9", "A"-"D", "*", and "#", which map t= o=20 their corresponding ASCII codes. (This is to support restricted-input=20 devices, i.e., telephones and telephone-like equipment.) Otherwise, pw-mapping is a charset. The parameter previously called "ualg" is interesting but maybe it's a=20 bit over-specified. If *precis-XXX can do the job, I am fine with=20 specifying "pw-mapping" only, and removing "ualg". Personally I don't see why PRECIS prohibits control characters like HT=20 in passwords. It seems to me that those sorts of characters are legit=20 for password purposes. But that is just my personal view and I don't=20 care enough about HT, BEL, ENQ, or similar in this context to fight=20 about it. :) > >> My usability perception is that if people really want to use Unicode=20 >> in their passwords, canonicalization is a very useful property to=20 >> preserve. Case folding/case mapping are not so useful, as most=20 >> systems like to have case-sensitive passwords for greater entropy,=20 >> but =E2=80=9Cmost systems=E2=80=9D is not =E2=80=9Call systems=E2=80=9D= so we shouldn=E2=80=99t preclude the=20 >> use of case algorithms. As for other algorithms such as line=20 >> breaking, character segmentation, Hangul syllable name generation,=20 >> etc., the short answer is =E2=80=9CI don=E2=80=99t know=E2=80=9D. (The= se are all reasons why=20 >> people stick with ASCII passwords, by the way.) > > Line breaking, character segmentation, Hangul syllable name=20 > generation,... are completely irrelevant for passwords and passphrases.= > > Also, many algorithms come with options or parameters. > > >>> Also, if there is indeed a list of algorithm identifiers in=20 >>> [UNICODE], then it would be good to give a Section number. Is the=20 >>> intent that each and every algorithm named somewhere in [UNICODE] is = >>> implemented? My rough guess would be that the average password input = >>> implementation implements only the identity transform. [I would of=20 >>> course be positively surprised if I were wrong.] >> >> See above; main thing that worries me is Normalization Forms. >> >>> >>> Also, references for [UNICODE], [BCP47], and [UAX31] should be give=20 >>> so that this registration is self-containing. >> >> Ok. >> >> Another possibility is that this registration goes back to =E2=80=9Cre= v 1=E2=80=9D,=20 >> i.e., no optional parameters about the character encoding at all. I=20 >> think that is perfectly defensible. But it is not particularly=20 >> i18n-friendly. > > I'm not sufficiently familiar with the format and the actual use=20 > cases, but my suggestion would be to check what's actually out there=20 > in the field (such as the Microsoft UTF-16LE including final NULL),=20 > and select or create a list of parameters/algorithms (with a registry=20 > if it turns out to be needed). To that, add a way to reference PRECIS, = > even if it's not currently used, because that includes the=20 > expertise/recommendations of experts. > > The current proposal just essentially saying: Unicode may define some=20 > of the pieces you may want to use here, and may have labels for them,=20 > so just give it a try. I'm not at all sure this will help=20 > interoperability, except by similar accidents like the Microsoft one=20 > that you described above. Right. Well, maybe PRECIS to the rescue? David Wheeler's aphorism is=20 that all problems in computer science can be solved by another level of=20 indirection. If PRECIS can handle this stuff, that's good enough for me. Sean > > Regards, Martin. > >> Regards, >> >> Sean >> >>> >>> Regards, Martin. >>> >>>> Encoding considerations: binary >>>> >>>> Security considerations: >>>> Carries a cryptographic private key. See Section 6 of RFC 5958. >>>> EncryptedPrivateKeyInfo PKCS #8 data contains exactly one private=20 >>>> key. Poor password choices, weak algorithms, or improper parameter=20 >>>> selections (e.g., insufficient salting rounds) will make the=20 >>>> confidential payloads much easier to compromise. >>>> >>>> Interoperability considerations: >>>> PKCS #8 is a widely recognized format for private key information=20 >>>> on all modern cryptographic stacks. The encrypted variation in this = >>>> registration, EncryptedPrivateKeyInfo (Section 3, Encrypted Private = >>>> Key Info, of RFC 5958, and Section 6 of PKCS #8), is less widely=20 >>>> used for exchange than PKCS #12, but it is much simpler to=20 >>>> implement. The contents are exactly one private key (with optional=20 >>>> attributes), so the possibility for hidden "easter eggs" in the=20 >>>> payload such as unexpected certificates or miscellaneous secrets is = >>>> drastically reduced. >>>> >>>> Published specification: >>>> PKCS #8 v1.2, November 1993 (republished as RFC 5208, May 2008);=20 >>>> RFC 5958, August 2010 >>>> >>>> Applications that use this media type: >>>> Machines, applications, browsers, Internet kiosks, and so on, that=20 >>>> support this standard allow a user to import, export, and exercise=20 >>>> a single private key. >>>> >>>> Fragment identifier considerations: N/A >>>> >>>> Additional information: >>>> >>>> Deprecated alias names for this type: N/A >>>> Magic number(s): None. >>>> File extension(s): .p8e >>>> Macintosh file type code(s): N/A >>>> >>>> Person & email address to contact for further information: >>>> Sean Leonard >>>> >>>> Intended usage: COMMON >>>> >>>> Restrictions on usage: None. >>>> >>>> Author: >>>> RSA, EMC, IETF >>>> >>>> Change controller: The IETF >>>> >>>> Provisional registration? (standards tree only): No >>>> >>>> >>>> >>>> _______________________________________________ >>>> media-types mailing list >>>> media-types@ietf.org >>>> https://www.ietf.org/mailman/listinfo/media-types >>>> >> --------------ms030004020303030104030100 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC CfYwggSvMIIDl6ADAgECAhEA4CPLFRKDU4mtYW56VGdrITANBgkqhkiG9w0BAQsFADBvMQsw CQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4 dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290 MB4XDTE0MTIyMjAwMDAwMFoXDTIwMDUzMDEwNDgzOFowgZsxCzAJBgNVBAYTAkdCMRswGQYD VQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNP TU9ETyBDQSBMaW1pdGVkMUEwPwYDVQQDEzhDT01PRE8gU0hBLTI1NiBDbGllbnQgQXV0aGVu dGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAImxDdp6UxlOcFIdvFamBia3uEngludRq/HwWhNJFaO0jBtgvHpRQqd5jKQi3xdh TpHVdiMKFNNKAn+2HQmAbqUEPdm6uxb+oYepLkNSQxZ8rzJQyKZPWukI2M+TJZx7iOgwZOak +FaA/SokFDMXmaxE5WmLo0YGS8Iz1OlAnwawsayTQLm1CJM6nCpToxDbPSBhPFUDjtlOdiUC ISn6o3xxdk/u4V+B6ftUgNvDezVSt4TeIj0sMC0xf1m9UjewM2ktQ+v61qXxl3dnUYzZ7ifr vKUHOHaMpKk4/9+M9QOsSb7K93OZOg8yq5yVOhM9DkY6V3RhUL7GQD/L5OKfoiECAwEAAaOC ARcwggETMB8GA1UdIwQYMBaAFK29mHo0tCb3+sQmVO8DveAky1QaMB0GA1UdDgQWBBSSYWuC 4aKgqk/sZ/HCo/e0gADB7DAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAd BgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwEQYDVR0gBAowCDAGBgRVHSAAMEQGA1Ud HwQ9MDswOaA3oDWGM2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9BZGRUcnVzdEV4dGVybmFs Q0FSb290LmNybDA1BggrBgEFBQcBAQQpMCcwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVz ZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQELBQADggEBABsqbqxVwTqriMXY7c1V86prYSvACRAj mQ/FZmpvsfW0tXdeDwJhAN99Bf4Ss6SAgAD8+x1banICCkG8BbrBWNUmwurVTYT7/oKYz1gb 4yJjnFL4uwU2q31Ypd6rO2Pl2tVz7+zg+3vio//wQiOcyraNTT7kSxgDsqgt1Ni7QkuQaYUQ 26Y3NOh74AEQpZzKOsefT4g0bopl0BqKu6ncyso20fT8wmQpNa/WsadxEdIDQ7GPPprsnjJT 9HaSyoY0B7ksyuYcStiZDcGG4pCS+1pCaiMhEOllx/XVu37qjIUgAmLq0ToHLFnFmTPyOInl tukWeh95FPZKEBom+nyK+5swggU/MIIEJ6ADAgECAhAaQkrPJ/nEG3M8lirbnsnnMA0GCSqG SIb3DQEBCwUAMIGbMQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVy MRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDFBMD8GA1UE AxM4Q09NT0RPIFNIQS0yNTYgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1h aWwgQ0EwHhcNMTUwMjAyMDAwMDAwWhcNMTYwMjAyMjM1OTU5WjAlMSMwIQYJKoZIhvcNAQkB FhRkZXYraWV0ZkBzZWFudGVrLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AM+J9tKgDs1LQtaDc+c4E58tCUQRNZiWbM1drOhLq2oSj75LuxPrrZy4XGjWoQFq9qGYHgiv boRKoo2guKi2R5xrf/pZ27ELY5jCa3BQs4Q2YwziXWM5rktnFL2amO2MMwhuo0n7OIMDYftv TEun2mOrnJ3G53zvawQdMbuRiHSNwc7DzWJwAjZQWFO8OF+BgNPzMDQGmzYQ4MVNk8NX5ecN VbfusUU1wZOlfdy8lWTfcASDqa9jLoaAiSx2ay4q7/5m0BOWqOKZCS0f7MmIUtSqODEoiZkd 7oCj5MRKOg2ZLia33JN+oS5rTKN8rs4u4ZF5g8zypHS5TY/l3pNRlrcCAwEAAaOCAfIwggHu MB8GA1UdIwQYMBaAFJJha4LhoqCqT+xn8cKj97SAAMHsMB0GA1UdDgQWBBQaZuXe8vDwU+ja pyFW3zSvIW6TxjAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAgBgNVHSUEGTAXBggr BgEFBQcDBAYLKwYBBAGyMQEDBQIwEQYJYIZIAYb4QgEBBAQDAgUgMEYGA1UdIAQ/MD0wOwYM KwYBBAGyMQECAQEBMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMF0GA1UdHwRWMFQwUqBQoE6GTGh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1NI QTI1NkNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgZAGCCsGAQUF BwEBBIGDMIGAMFgGCCsGAQUFBzAChkxodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9DT01PRE9T SEEyNTZDbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3J0MCQGCCsGAQUF BzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wHwYDVR0RBBgwFoEUZGV2K2lldGZAc2Vh bnRlay5jb20wDQYJKoZIhvcNAQELBQADggEBAHiH/zXr779dLLJNLp6yW/RQvJSYuNbEqauQ BWLZgvCF8FJjcS+y+bhN7RpTSVaEymbxnpJWFg0Qs6us6rXtttU6MF2JhTuxtu6yWW0EYy80 KGOt6zJRTxa46kAhAOqw4KPQ5RTvf/NT+2Qu8b2edlh+3t6f8iTQOAL/BNHnmr4QvpfnMLa8 NMAqd4FLOcSU9JKBlb5YtsEdfM3nBt4m95pfhpww79gB+zoXMoVaI+lfQHhkGth6mlaTkFUV DT2pqq98XgESpObFEngb8ZRftBVoZfgJ4ajXcZnhKaqYEr23EtY207KgKb4LAXA84wtPcFfJ rQaDArapR7xd2GNs+eYxggRBMIIEPQIBATCBsDCBmzELMAkGA1UEBhMCR0IxGzAZBgNVBAgT EkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RP IENBIExpbWl0ZWQxQTA/BgNVBAMTOENPTU9ETyBTSEEtMjU2IENsaWVudCBBdXRoZW50aWNh dGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhAaQkrPJ/nEG3M8lirbnsnnMA0GCWCGSAFlAwQC AQUAoIICYTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEx MjQwOTU0MzlaMC8GCSqGSIb3DQEJBDEiBCDg/OSWtiZwTMa+a3T+hu0dlmq9MJ1TZhVOjxF1 Q8DONzBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjALBglghkgBZQMEAQIwCgYIKoZI hvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3 DQMCAgEoMIHBBgkrBgEEAYI3EAQxgbMwgbAwgZsxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJH cmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBD QSBMaW1pdGVkMUEwPwYDVQQDEzhDT01PRE8gU0hBLTI1NiBDbGllbnQgQXV0aGVudGljYXRp b24gYW5kIFNlY3VyZSBFbWFpbCBDQQIQGkJKzyf5xBtzPJYq257J5zCBwwYLKoZIhvcNAQkQ AgsxgbOggbAwgZsxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIx EDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMUEwPwYDVQQD EzhDT01PRE8gU0hBLTI1NiBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFp bCBDQQIQGkJKzyf5xBtzPJYq257J5zANBgkqhkiG9w0BAQEFAASCAQCiq17o1yxyIrVaZ1ix KMBjFilLvHi4q6ORHL7pf0/z2LyPLKhcLa2gTEfC5dSSLC71o0qodzjqfku1lCM7Z0w+K+5V NMGqQnqeC8gmHJBMDIk9XEecSs/PduenpGi7gZXJdB+mywvoOpz/8q1OoNillZ66gFgbGFDV BF2Yrv3lCLbgz2AS9LAxdOL3btKz/yi15GKM6xlGNc9+lB9DJ+rIq4rQQw/sNR7Pgc5GF5Zs Is0Fogh0xu9ArHTaSl7Fr1VnfQODyTA7l+SOBLoCqi0muXrV0nXeu6ZHnQazkh0Czt6bKNcV RfLcXDq/4+3h1eZXvPoDtnLQUprEHFmLjug6AAAAAAAA --------------ms030004020303030104030100--