Subject: [openpgp] User ID conventions (it's not really a RFC2822 name-addr)

From: Daniel Kahn Gillmor <[email-masked]>
Date: Mon, 16 Sep 2019 18:35:07 -0400

Hey OpenPGP folks--

I'd like to have a clearer undersatnding about the actual conventions
for OpenPGP User IDs in the context of e-mail.  The standards currently
say that the convention is an RFC2822 "name-addr", but (as detailed
below), that does not appear to be the actual convention in practice.

While we're updating RFC 4880, we should fix the standards to reflect
reality.  There are two proposals at the end that i'd love feedback on.
I prefer proposal 2.

Claims about name-addr
----------------------

RFC 4880 says the following:

    5.11.  User ID Packet (Tag 13)

    A User ID packet consists of UTF-8 text that is intended to represent
    the name and email address of the key holder.  By convention, it
    includes an RFC 2822 [RFC2822] mail name-addr, but there are no
    restrictions on its content.  The packet length in the header
    specifies the length of the User ID.

RFC4880bis repeats the above, and adds:

    5.13.2.  User ID Attribute Subpacket

    […]

    A User ID Attribute subpacket, just like a User ID packet, consists
    of UTF-8 text that is intended to represent the name and email
    address of the key holder.  By convention, it includes an RFC 2822
    [RFC2822] mail name-addr, but there are no restrictions on its
    content.  For devices using OpenPGP for device certificates, it may
    just be the device identifier.  The packet length in the header
    specifies the length of the User ID.

Both of these references to rfc 2822 are problematic.  Real user IDs
don't look like this, and other implementations won't parse things this
way either, so the implementers might be led astray by this
documentation.

User ID convention is not a name-addr
-------------------------------------

Here are a few concrete reasons why the convention is not actually an
RFC 2822 name-addr:

 a) name-addr in RFC 2822 is defined to be a US-ASCII field, potentially
    charset-switched with RFC 2047 extensions in at least the
    display-name part.  But User IDs are native UTF-8.  For example,
    compare the following strings:

     1) Björn Björnson <[email-masked]>
     2) Bj=?utf-8?q?=C3=B6?=rn Bj=?utf-8?q?=C3=B6?=rnson <[email-masked]>

    We expect User IDs to look like (1), even though (2) is technically
    an RFC 2822 mail-addr.  We don't want people to generate user IDs
    like (2), and we don't want implementations to try to apply RFC 2047
    decoding to the contents of a user ID packet to be able to display
    it.

 b) name-addr doesn't allow non-quoted internal commas or apostrophes,
    so the following common User ID patterns are not technically
    name-addrs either, though implementations generate them, and people
    use them just fine in the real world:

     3) Acme Industries, Inc. <[email-masked]>
     4) Michael O'Brian <[email-masked]>
     5) Smith, John <[email-masked]>

 c) in RFC 2822, a <name-addr> is not the same as a "mailbox" -- a
    "mailbox" is either a "name-addr" (which contains an "addr-spec") or
    an "addr-spec" on its own.  But we have many examples in flight
    today of user IDs that are just a raw "addr-spec" (without
    angle-brackets), and those tend to be accepted by many OpenPGP
    implementations:

     6) [email-masked]

 d) the "display-name" part of an RFC 2822 "name-addr" is a "phrase" (a
    series of "word"s, which are either "atom"s or "quoted-string"s).
    An "atom" cannot contain the "@" symbol, so the display-name cannot
    contain an unquoted @.  However, due to infelicities in common
    interfaces, we also see a large number of user IDs that simply
    replicate the addr-spec as though it were the domain name.  This is
    not a valid name-addr, but it is accepted by most OpenPGP
    implementations.

    For example:

     7) [email-masked] <[email-masked]>

These differences between RFC 2822's name-addr and the actual user IDs
in use today suggest that the guidance that they are "by convention" a
name-addr is a mistake, and a potentially damaging one at that.  It's
likely to cause implementers to do expensive implementations of the
complex name-addr syntax, which they then have to make exceptions for
when they encounter all the real-world counterexamples.

At the same time, we don't want implementers to each have their own
arbitrary deviations from the convention -- the more uniform we can make
the convention, the more likely we'll be to have interoperability.

Goals
-----

AFAICT, there is one main, uncontroversial technical goal for an
e-mail-focused OpenPGP implementation when dealing with user IDs:

 A) extract the addr-spec

    If the implementation can't figure out the addr-spec, they can't use
    the certificate to learn how to contact.  and if the implementation
    can't index internally by addr-spec, then they can't find the
    appropriate certificate to use when trying to contact a given e-mail
    address.

What we really want is for every implementation to do this in a robust
and predictable way, including for all of the common non-mail-addr forms
described above.

Are there any other goals that people think this convention should
cover?

Some (possibly-contentious) additional goals:

 B) accepting UTF-8 addr-specs

    recent RFCs about internationalization accept non-ASCII characters
    in domain names and local-parts of the addr-spec:

    https://tools.ietf.org/html/rfc6530#section-10.1
    https://tools.ietf.org/html/rfc6532#section-3.2

    do we expect user agents to be able to extract addr-specs that look
    like:

        [email-masked]
        [email-masked]

    (These examples are from
    https://en.m.wikipedia.org/wiki/International_email)

 C) accepting really unusual addr-specs:

    the addr-spec definition formally includes some really bizarre
    structures that (while probably in use on some legacy systems) are a
    really bad idea.  For example, localparts that are wrapped in
    double-quotes but otherwise contain forbidden characters can be
    problematic:

       "Abc@def"@example.com
       "Fred Bloggs"@example.com

    It looks to me like RFC 5322 even allows CFWS in the local-part,
    ugh.  Do we expect user agents to do anything sensible with these
    addresses?

A non-goal (does anyone want this?):

 D) be able to distinguish the "comment" from the "name" in display-name:

    Despite several implementations appearing to distinguish "Comment"
    from "Name" in the display-part, it's not clear that anyone *does*
    anything with that information, so it's mainly clutter and
    confusion.  On top of that, there are probably more useless comments
    than useful ones, so i'd be happy to let this misfeature die out.

Proposal 1: unicode maybe-wrapped addr-spec
-------------------------------------------

We can address goals A, B, and C with some sort of language that
acknowledges reality if we accept the following:

 * addr-spec from RFC 5322 is augmented by the definitions
   in RFC 6532 section 3

 * there is no structure that we care about in what we would have called
   the "display-name" part of the supposed name-addr.

Then the user ID convention becomes (again, assuming atext as augmented
by 6532 §3):

    pgp-uid-prefix-char    = atext / specials

    pgp-uid-convention     = addr-spec /
                             *pgp-uid-prefix-char "<" addr-spec ">"

Proposal 2: simplify, simplify
------------------------------

Proposal 1 is still pretty ugly due to the inherent complexities of
addr-spec itself.

We can simplify the formal addr-spec greatly if:

 - we don't allow CFWS or quoted-string in the local-part, and
 - we don't allow CFWS or domain-literal addresses in the domain, and
 - we drop all the obsolete variants ("obs-*" labels in RFC 5322 ABNF)

CFWS is "comments and folding whitespace".  Dropping comments is
justified by the argument that comments can go elsewhere in the user ID.
Folding-whitespace isn't necessary due to the structure of the user
ID itself -- we're not in an e-mail message header.  Dropping obsolete
parts is justified because they're obsolete.  Dropping quoted-string is
justified because it's rarely used, and likely to break in reality.  And
dropping domain-literal parts is justified because no one delivers
e-mail to raw IP addresses anyway.

Note that yes, this will discard some legitimate (if odd) addresses
(e.g. ones with CFWS or quoted-string), and it may fail to recognize
some legacy (odd) user IDs (obs-* or domain-literal).  But we're
describing a convention here, not making a normative statement, and we
can do much better than the convention we were describing earlier but
pretty much every implementation fails to follow.

Using the definitions in RFC 5322 and RFC 5234, as augmented by RFC 6532
section 3, we can implement this simplification like so:

    pgp-addr-spec          = dot-atom-text "@" dot-atom-text
    
    pgp-uid-prefix-char    = atext / specials
    
    pgp-uid-convention     = pgp-addr-spec /
                             *pgp-uid-prefix-char "<" pgp-addr-spec ">"

Note that every pgp-addr-spec is by definition an addr-spec (though not
all addr-specs are a pgp-addr-spec).

I believe that proposal 2 is closer to what most implementations do
today, and it handles goals A and B.  I don't mind it failing at goal
C because of how much simpler the matching rule is.

Conclusion
----------

My preference is to replace the text about User ID conventions in RFC
4880bis with proposal 2, but i'd be open to hearing other suggestions if
anyone has them.

        --dkg

PS in researching other ways to solve this problem, i came up with an
   approach that relies on Unicode character properties, in particular
   Grapheme_Base and Grapheme_Extend as a way to exclude control chars
   and other non-printables.  This is a more sophisticated/nuanced
   approach than the RFC 6532 ABNF extensions to atext.  But specifying
   it requires a character class set subtraction operation (you want to
   subtract "<" and ">" and "@" and " " from the Grapheme_* classes),
   which isn't listed in IETF's ABNF definition in RFC 5324.  And
   implementing it requires a toolkit capable of discerning and acting
   on Unicode properties (e.g. the python regex module from PyPi, but
   not the re module from python's stdlib).  That's too bad, because
   6532 §3 effectively makes things like U+200B ZERO WIDTH JOINER
   allowable within dot-atom-text, which is uncomfortable and weird.
   But other implementers reliant on 6532 might accept such a localpart
   anyway. These costs don't appear to be worth the minor gain compared
   to proposal 2, so i've stopped attempting to document that approach.
   If anyone wants to take a crack at it though, i'm happy to share my
   notes.

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Mon, 16 Sep 2019 21:43:54 -0400

As usual, when you try to implement something, you find the missing
pieces. :P

>     pgp-uid-prefix-char    = atext / specials

The above line appears in both proposals, but it contains a mistake.  It
should read instead:

     pgp-uid-prefix-char    = atext / specials / SPACE

The following python3 code implements proposal 2 using python's built-in
re module (i have not tested it with python2, given python2's clunky
unicode support).

    import re

    specials = r'[()<>\[\]:;@\\,."]'
    atext = "[-A-Za-z0-9!#$%&'*+/=?^_`{|}~\x80-\U0010ffff]"
    dot_atom_text = atext + r"+(?:\." + atext + "+)*"
    pgp_addr_spec = dot_atom_text + "@" + dot_atom_text
    pgp_uid_prefix_char = "(?:" + atext + "|" + specials + "| )"
    addr_spec_raw = "(?P<addr_spec_raw>" + pgp_addr_spec + ")"
    addr_spec_wrapped = pgp_uid_prefix_char + "*<(?P<addr_spec_wrapped>" + pgp_addr_spec + ")>"
    pgp_uid_convention = "^(?:" + addr_spec_raw + "|" + addr_spec_wrapped + ")$"

    pgp_uid_convention_re = re.compile(pgp_uid_convention, re.UNICODE)

    m = pgp_uid_convention_re.search(uid)

If there's a resultant match object, then the pgp-addr-spec can be found
in either m["addr_spec_raw"] or m["addr_spec_wrapped"], depending on
whether there are angle-brackets involved or not.

Note the atext definition is the extended form of atext.

anyway, i hope this clarification is useful.

     --dkg

PS if the above is useful, anyone should feel free to reuse any of the
   above code in any context under any license.  If you do so, and you
   want to provide an attribution so that complaints can be directed
   properly, that's fine, but i have no need for credit if you don't
   want to bother.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 16 Sep 2019 22:25:00 -0400

Thanks for all of this.

    > Proposal 2: simplify, simplify

I like it too.

    > C because of how much simpler the matching rule is.

Agreed.

    > If anyone wants to take a crack at it though, i'm happy to share my

Wow :-)

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Jon Callas <[email-masked]>
Date: Thu, 19 Sep 2019 11:09:58 -0700

> below), that does not appear to be the actual convention in practice.

A simple place to start is that RFC 6531 seems to be the present upgrade/replacement for RFC2822. There have been interim ones and I didn't spend a lot of time sorting out details. Whatever the present successor is, let's just call it R.

> reality.  There are two proposals at the end that i'd love feedback on.

I think I do, too, with your addenda. 

Rewinding to history and second if not first principles, the User ID field is ideally free-form text. The most common free-form text is an email address, and that's described by R, and I think your proposal 2.

However, there are a lot of places where people have put human-readable text in (e.g. "XYZ Group Release Signing Key") that doesn't have a machine readable context at all. There have also been X.500 Distinguished Names, Common Names, and so on put there. There have been other outré uses as well. As long as we don't end up with it being *precisely* an email address, meaning that we outlaw just plan text, I think it's fine.

At one time, I really wanted User Attribute Packets (created for having images as IDs), to have other types of structured data including email addresses. The reasons for not doing it then (it's a pain, and who's going to support it) apply today. It's still a good idea, though.

If whatever you come up with precludes text with no machine meaning, I think you hit the flip side of having a defined type of ID: there's lots of things that already don't meet it, and people will just ignore what's there because history.

Operationally, I think that supporting Just Text merely means that if the field fails parsing rules (like Proposal 2) then it's Just Text and we're moving on.

	Jon

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Fri, 20 Sep 2019 14:20:48 -0400

On Thu 2019-09-19 11:09:58 -0700, Jon Callas wrote:
> Operationally, I think that supporting Just Text merely means that if
> the field fails parsing rules (like Proposal 2) then it's Just Text
> and we're moving on.

Yep, completely agreed.  Nothing in my commentary in this thread is
meant to change the fundamental nature of the User ID as a UTF-8 string
at its core.

The goal is simply to more realistically state what the common
convention is for User IDs that are likely to be used in the e-mail
context.

        --dkg

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Sun, 29 Sep 2019 13:03:25 +0200

On Fri 2019-09-20 14:20:48 -0400, Daniel Kahn Gillmor wrote:
> Nothing in my commentary in this thread is meant to change the
> fundamental nature of the User ID as a UTF-8 string at its core.
>

I've opened this as a merge request on the draft:

https://gitlab.com/openpgp-wg/rfc4880bis/merge_requests/23

        --dkg