Subject: [EAI] Status summary on SimpleDowngrade (was: Re: AppsDir Review of draft-ietf-eai-simpledowngrade-03)

From: John C Klensin <[email-masked]>
Date: Sat, 05 May 2012 11:04:10 -0400

Hi.

I've been trying to watch the discussion, rather than
participating in it, in the hope that I could instead summarize
conclusions and get us moving forward.   Note that the comments
and conclusions below are my summaries and extrapolations from
what has been said on the list and are tentative: if anyone
disagrees in any substantive way, speak up.  Circa 50 messages
in the last couple of weeks later...

(1) I infer that we have general agreement on the basic EAI IMAP
and POP specs.   If we do not, people had better speak up
quickly.  We will review those documents quickly on the interim
meeting Jabber chat, but, in the absence of significant issues
then, I assume that the WG thinks the drafts are ready to go to
the IESG and will start doing my final WG Chair
(pre-Shepherd-writeup) check.

(2) A separate note will follow about the other downgrade spec
(draft-ietf-eai-popimap-downgrade).

(3) Re the Simple Downgrade draft, I believe that, with the
possible exception of one or two people or issues in the rough,
the WG has concluded:

(3.1) The Security Considerations should mention issues that are
specific to receiver-end downgrading and to this spec.  John
Levine's request to include a DKIM pointer along with "signed
body parts" falls into that category, but note that there is
already text suggesting that the implementer better be really
careful when altering messages that are signed in any way in the
Framework spec (RFC 6530).  A pointer to the Security
Considerations section of the basic IMAP spec would be in order
if the author likes (i.e., I consider the question of whether
the pointers that effectively incorporate the IMAP spec by
reference make such an additional reference/comment unnecessary
to be a matter of author discretion unless the WG feels _very_
strongly to the contrary).

Like Claudio, I generally tend to prefer to err on the side of
too much explanation and detail rather than too little.
However, no matter what we do in this spec, anyone who tries to
implement it without a good understanding of the POP and/or IMAP
specs and the documents that define the EAI changes to POP and
IMAP is going to be in really serious trouble.  I think we need
to keep in mind that giving people enough detail that they might
believe they don't need to understand the specs that this one
supplements does no one any favors.  And I think that tips the
balance significantly toward not repeating (even by summary and
reference) a lot of material here that appears elsewhere.

I think it is also worth noting in this discussion that the
split-UA model that underlies both IMAP and POP implies a
certain intimacy between client and server that does not exist
with, e.g., normal SMTP transit relationships.  Yes, we have
protocols that permit different clients to interwork with the
same servers and vice versa, but a great many things depend on
the assumption that the IMAP (or POP) client-server relationship
involves a reasonably high level of trust in each other and in
the channel -- the relationship is nowhere near as easy to
compromise as, e.g., the one between a pair of SMTP relays or,
historically, the DNS.

(3.2) As a partial followup to the above, I think we have all
known since the EAI effort started that this work would
inevitably require updates, or at least careful review, to any
security mechanism that involves integrity checks that are
sensitive to the media type of the body parts, that involve
signatures over either external or internal headers, or that use
email addresses as identifiers.  I've just reviewed the charter
and I can find absolutely no support for the idea that it is our
problem.  The same comment applies to issues of UI presentation
of messages involving valid or invalid signatures even though I
think this work emphasizes how little deep thought the IETF has
given to that problem.  So, for those who want to pursue those
reviews and upgrades, my very best wishes, but take it somewhere
else.

(3.3) There are a number of ways in which the popimap-downgrade
draft, by virtue of taking on a harder task and possibly because
it has had more time to mature in the WG, is simply more
comprehensive than this one.  That is ok because it is really
another symptom of the tradeoffs between a comprehensive, but
complex, approach and one that is "good enough" for some set of
cases but much simpler.  But, picking up a comment of Claudio's
from several days ago, the WG may want to consider whether this
document should reference that one for additional discussion of
some of the risks and issues, especially those that appear in
cases that are less easily understood.

(3.4) Environments in which complete EAI collections are rolled
out at once and made mandatory or in which upgraded POP and IMAP
clients and servers are completely deployed before UTF8SMTP is
enabled in the delivery MTA are simply irrelevant to these two
downgrade specs.  If it not clear in the POP and IMAP docs that
downgrading is never going to be invoked in those cases, then,
despite (1) above, they need to be fixed.  Whether they are
fixed or not, the two downgrade specs are still unaffected.
These documents exist only for the real-world situations in
which those who control the servers cannot completely control
the end-user's choice of clients _and_ who have chosen to not
respond to the combination of a legacy client and a message that
requires UTF8SMTP features with an error condition equivalent to
"message inaccessible to you until you upgrade (or use
webmail)".  

I don't think anyone has seriously suggested "pretend the
message isn't there".  Regardless of what the specs say, every
competent POP and (especially) IMAP server implementation in the
world --except those that are designed to sit behind delivery
MTAs or mail stores that will reject or drop problem messages--
has mechanisms for dealing with messages that are sufficiently
broken that they can't be delivered across the interface.  I've
seen POP3 implementations that put message numbers of 0 or
snarky comments in the scan listing response to LIST and respond
to RETR with "-ERR junk message".  Similarly, an IMAP server
might respond to SEARCH ALL by including the message numbers of
such messages, interpret the option of leaving out messages with
non-text and non-Message body parts from more selective SEARCH
command responses, and then respond to FETCH NN .. with
something like "* NN FETCH (BOGUS)" or "* FETCH NN (BOGUS (22)
invalid message format)".  If either causes the client to get
annoyed, that is more or less the intended behavior.  Most of us
have also seen clients and occasionally servers self-destruct
under equivalent circumstances, but that is, sadly, how it goes.
Perhaps IMAP and/or POP should be updated to explicitly allow
per-message "junk; cannot convert and download" responses but,
again, not this WG's problem.

Going back to the discussion that led to the simpledowngrade
spec, we should not lose track of the fact that a "you need to
upgrade" response will be the right strategy for at least two
kinds of environments, the enterprise one in which the intended
message will be "we told you to upgrade, you lose until you do"
and the large, don't-really-care, provider in which the intended
message will be "if you don't want to keep your client up to
date, too bad for you".  That is ultimately the argument in
favor of popimap-downgrade over simpledowngrade: if it is worth
going to the trouble of downgrading at all, one might as well
exert the extra effort to do it comprehensively.  Fortunately,
the WG doesn't need to resolve that argument and, if we need a
better description of it, this document is probably not the
right place.

   john

---

From: "John Levine" <[email-masked]>
Date: 5 May 2012 19:43:15 -0000

> That is ultimately the argument in favor of popimap-downgrade over
> simpledowngrade: if it is worth going to the trouble of downgrading
> at all, one might as well exert the extra effort to do it
> comprehensively.

That's the argument, but it's one that I don't find at all persuasive.

If a user has a non-EAI MUA, the most a downgrade can realistically do
to create something the MUA can display is what simpledowngrade
describes.  Anything beyond that is a quality of implementation issue
that has nothing to do with interoperability.  As soon as you rewrite
anything, signatures break, heuristics that look for patterns in the
header lines don't work any more, and it's just a losing battle to try
to make it look like a real non-EAI message.

If a message's headers contain EAI addresses, there's no no way a
non-EAI MUA can respond to them directly, so there's nothing to
standardize with addresses beyond ensuring that one can't reply to
them.  I'm not aware of any MUA that would do anything useful with
Downgraded-Something headers, and I believe I am far from alone in
thinking that any work that might be invested in making a MUA use them
would be far better spent making the MUA handle native EAI.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sat, 05 May 2012 17:05:54 -0400

--On Saturday, May 05, 2012 19:43 +0000 John Levine
<[email-masked]> wrote:

>> That is ultimately the argument in favor of popimap-downgrade
>> over simpledowngrade: if it is worth going to the trouble of
>> downgrading at all, one might as well exert the extra effort
>> to do it comprehensively.
> 
> If a user has a non-EAI MUA, the most a downgrade can
> realistically do to create something the MUA can display is
> what simpledowngrade describes.  Anything beyond that is a
> quality of implementation issue that has nothing to do with
> interoperability.  As soon as you rewrite anything, signatures
> break, heuristics that look for patterns in the header lines
> don't work any more, and it's just a losing battle to try to
> make it look like a real non-EAI message.
>...

John (and others),

(Co-Chair hat pulled firmly over ears)

Let me be blunt, with the understanding that I agreed to take on
this WG because there seemed to be general agreement that one of
our major shared goals was to move this work forward
efficiently, much more efficiently than we have been doing
lately. 

Unless someone is going to argue strongly that the more complex
popimap-downgrade spec should simply be dropped, which scenarios
you (or others) find persuasive is simply not interesting.  You
predict that one scenario will dominate and that certain things
are important and therefore find some approaches unpersuasive.
Others make different predictions and/or assessments of
importance.  Starting from a variation on some of Ned's
comments, others of us may have yet a third prediction.

No matter how persuasive each of us may find our individual
predictions, there are two important realities.  First, we are
all making those predictions without specific experience or data
about this particular type of situation.  Second, people who
implement and deploy these things are thoroughly likely to do
what they think is right, regardless of what the IETF predicts
or thinks is good for them.

Another scenario we haven't discussed at all is among the likely
ones in some situations.  That is that the POP or IMAP server
generates an alert or special log entry to what is often called
a "Help Desk" with the user's identity and the fact that a
message that requires EAI capability has landed in the mailbox
so that the Help Desk personnel can do something proactive and
entirely out of band.  Really low implementation cost and, for
those environments in which the "Help Desk" can force upgrades,
a very efficient and thoroughly likely one.

Similarly, none of our beliefs about how time is best spent are
likely to amount to anything.  To generalize from what Ned has
pointed out several times, implementers have audiences
(sometimes called "customers") and are likely to end up doing
what those audiences want, whether we think it is efficient or
not.  

So I continue to suggest that we just do as competent a job as
we can of documenting the protocol alternatives that subsets of
us think are reasonable and then move on, independent of who
those who favor (or predict) one scenario feel about the others.
The alternative, it seems to me, is to try to reach agreement
about those scenarios.  All I can say about trying to reach
agreement on one scenario is that it is certain to cost us a lot
of time and that, based on the discussions of the last several
weeks, the odds that many people will be persuaded to change
their positions are really low.  

Recommendation: Those of us who care about (note that I didn't
say "like") each of the scenarios try to make sure that the
protocols and actions that result are well-thought-out,
well-documented, and, ideally, clear about the scenarios for
which they are optimized.  Let's be sure that the base EAI POP
and IMAP documents provide the right references and linkages to
whatever downgrade specs are out there and that they are
otherwise agnostics.  And let's save arguments about
persuasiveness or plausibility for some time post-deployment
when we've actually accumulated some serious experience.

best,
   john

p.s. Nothing personal: you were merely the first person who
stuck his head up and made a "presuasiveness" point today.

---

From: "John R Levine" <[email-masked]>
Date: 5 May 2012 17:42:42 -0400

> Second, people who implement and deploy these things are thoroughly 
> likely to do what they think is right, regardless of what the IETF 
> predicts or thinks is good for them.

Of course.  To restate the obvious, standards tell how to write 
independent subsystems so they interoperate with each other.  Your alarm 
scenario is entirely within one system, so it presents nothing to 
standardize.

If people concretely expect to write MUAs that do something useful with 
the Downgraded-foo headers, then we might as well publish a document that 
says what those headers should contain.  But it's my impression that the 
plan is to add them just in case anyone is interested, which strikes me as 
mildly counterproductive.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sat, 05 May 2012 19:36:26 -0400

--On Saturday, May 05, 2012 17:42 -0400 John R Levine
<[email-masked]> wrote:

>> Second, people who implement and deploy these things are
>> thoroughly  likely to do what they think is right, regardless
>> of what the IETF  predicts or thinks is good for them.
> 
> Of course.  To restate the obvious, standards tell how to
> write independent subsystems so they interoperate with each
> other.  Your alarm scenario is entirely within one system, so
> it presents nothing to standardize.

To restate the equally obvious, the thing that is important
about that scenario is that it represents a case in which no
downgrading at all is needed or likely to be useful, making both
of our downgrading specs irrelevant.
 
> If people concretely expect to write MUAs that do something
> useful with the Downgraded-foo headers, then we might as well
> publish a document that says what those headers should
> contain.  But it's my impression that the plan is to add them
> just in case anyone is interested, which strikes me as mildly
> counterproductive.

Kazunori Fujiwara and his colleagues can (and should) speak for
themselves, but I suspect that they already have such MUAs
implemented.  Certainly they had those MUAs when the
experimental specs specified very similar downgrading in
transit.  It would be quite easy to modify an MUA, and to move
the downgrading code from an MTA to a POP/IMAP server, if one
had that code already.  If I'm correct, there is both running
code for the popimap-downgrade case and the Downgraded-foo
headers and a community with a long history of routinely dealing
with non-ASCII characters who think that approach is desirable.
The combination is a lot more than "in case anyone is
interested".  One can speculate on how many others are likely to
support it, but that is more speculation.

Independent of that, the main argument for Downgraded-foo, as
I've understood it, is not that MUAs will be able to do anything
useful with it but that displaying those headers will permit an
intelligent and determined user to figure out a lot more about
what was going on with the original message than they could do
with a more simplistic downgrading model.  With most MUAs, such
headers would appear only with a "full" header display and I
think most normal users almost never look at full header
displays (even if they know how to activate them).  But one
could say that few users ever look at, or know how to access and
interpret, trace fields, yet we spend a lot of energy trying to
get them right.  

>From my own experience, I'd expect a typical user to freak out
about a downgraded (and un-reply-able) EAI-requiring message
appearing in a traditional MUA, regardless of the downgrade
model that was used.  But, if our Japanese colleagues tell us
that either their MUAs or their users --who have many years of
experience working in environments use multiple non-European
scripts-- are different from what I think of as a typical user,
I am quite sure I don't have enough knowledge or experience to
tell them that they are wrong.

    john

---

From: [email-masked]
Date: Sat, 05 May 2012 21:36:51 -0700 (PDT)

> That's the argument, but it's one that I don't find at all persuasive.

I don't really find it persuasive either, but not for some the reasons you
give.

> to make it look like a real non-EAI message.

Unfortunately, one of your examples is demonstrably off target while the other
actually favors popimap-downgrade.

More specifically, as I pointed out in a previous message, AFAICT this supposed
"signature breaking" is nothing but a will o' the wisp. Yes, it will break DKIM
and DomainKeys, but those aren't intended to be end-to-end mechanisms. (And if
they used that way, they encounter a plethora of much more serious problems.)
The actual end-to-end signature mechanisms in use are things like PGP or
S/MIME. which operate either by using multipart/signed or through embeddings in
text parts. Since multipart/signed requires 7bit encoding of content, the issue
it has is with EAI proper, not with downgrading. And I fail to see an issue
specific to downgrading with text part embedddings either.

There may also be issues with EAI when signature schemes use addresses as
identifiers in certificates, or when header lines are copied into secure
enclosures, but these again have everything to do with EAI and nothing to do
with downgrading per se.

Tests on headers are another matter, however. This is Sieve territory. Sieve
specifies that encoded words in headers should be decoded and converted to
utf-8 before tests are performed. There can be issues with this when the
charset is unknown but EAI downgrading always produces utf-8 encoded words,
which of course decode as an identity transformatin.

The net result is Sieve scripts running on a non-EAI client will actually work
better on messages transformed by popimap-downgrade and not simpledowngrade,
because the former preserves more header fields.

And yes, there are any number of nonstandard facilities out there for this sort
of thing, e.g., procmail, but as I've said previously, I don't see any
obligation to spend time worrying about them.

But all this is really beside the point, because the real issue isn't
rewriting, it's the likely loss of critical address information. For better or
worse, there's no useful way to present an EAI address in a non-EAI client. And
that's why I don't think there's a persuasive case to try and support "better"
downgrading.

> them.

Exactly, although the issues actually transcend replies. While I completely
disagree with Ted's conclusions about the loss of validation anchors, he is
absolutely correct in stating that their loss is an unavoidable consequence of
downgrading.

> I'm not aware of any MUA that would do anything useful with

Couldn't agree more. There is simply no point in spending a bunch of time
implementing complex downgrading schemes and then implementing even more
complex upgrading schemes in clients when it's actually easier to
simply support EAI in the client.

				Ned

---

From: [email-masked]
Date: Sat, 05 May 2012 22:13:36 -0700 (PDT)

> comments, others of us may have yet a third prediction.

Well, as I've stated before, I actually don't find it especially persuasive
that either of these documents will see much use in their intended roles. That
said, I've never said there is no utility here, and to be honest I'm really not
sure in the "whole mailbox permanent downgrade" which approach makes more
sense. 

> or thinks is good for them.

Sure, none of us have any actual deployment experience with EAI, which is part
of why I haven't argued in favor of focusing on the whole mailbox downgrade
scenario I view as most likely.

> a very efficient and thoroughly likely one.

I'm dubious as to the extent that proactive approaches like this will be used.
But there are plenty of environments out there where the choice of client is
not under the user's control. Those environments will simply upgrade to an
EAI-capable client when the need arises and the users will have no say in the
matter.

> what those audiences want, whether we think it is efficient or

Well, yes and no. I have no problem believing that user demand for EAI will
drive EAI deployment in some, and perhaps many, cases. But in my experience
*how* that happens is not going to be under the user's control.

> their positions are really low.

Fair point.

> when we've actually accumulated some serious experience.

Seems like a reasonable approach to me, especially since I happen  to think the
likely deployment scenario may be rather different from what we're envisioning.

				Ned

---

From: Claudio Allocchio <[email-masked]>
Date: Sun, 6 May 2012 10:07:04 +0200 (CEST)

> Seems like a reasonable approach to me, especially since I happen to 
> think the likely deployment scenario may be rather different from what 
> we're envisioning.

with just my WG member hat on (and the ancient one of implmemeter and 
provider of gateway services - a very similar scenario to downgrading) I 
can only FULLY support this reccomendation and approach!

Very strange deployment we cannot even think of in our way to see the 
world can happen, <joke> also driven by users/groups which believes that 
the correct way to send a personal message to somebody is to post it on a 
social network messages board </joke>

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 6 May 2012 10:10:21 +0200

On 05/06/2012 01:36 AM, John C Klensin wrote:
> transit.

This makes sense. A great deal of sense.

> It would be quite easy to modify an MUA, and to move
> had that code already. 

The conditions and O() expectations differ. In one case, the code is
expected to store and forward a single message. In the other, it's
expected to open up to around 10-20k messages, sometimes with read-only
access.

Creating Downgraded-Cc is easy if you store and forward. Less so if you
have read-only write access to the mailbox and the client expects you to
do it for 10k messages, fast. UID FETCH 1:* (RFC822.SIZE FLAGS) for
instance. Servers commonly have to do that for datasets which won't fit
in available RAM.

BTW, I don't think popimap-downgrade discusses what happens if a
mischievous sender already adds downgraded-* fields?

Arnt

---

From: "John Levine" <[email-masked]>
Date: 6 May 2012 14:42:55 -0000

> predicts or thinks is good for them.

You're doubtless correct, but now I'm totally confused.  I think I'm
seeing the suggestion that we provide a range of downgrade options,
none of which can interoperate, and implementers will implement one of
them or maybe do something totally different.  What's the benefit of
putting anything in a standard beyond the advice that a downgrade has
to get rid of non-ASCII text and not introduce addresses that could be
replied to by mistake?

As a straw man, here's another unimplemented downgrade strategy.

We all seem to agree that it would be a good thing for users to see
the addresses of their correspondents.  Let's say the message has
these headers, with the capitalized strings standing for UTF-8.

From: <ABDUL@AA> acomment
To: <BIBI@BB> bcomment
Cc: <CHEN@CC> ccomment

The downgraded message headers are rewritten to these:

From: <[email-masked]> ABDUL@AA acomment
To: <[email-masked]> BIBI@BB bcomment
Cc: <[email-masked]> CHEN@CC ccomment

Since comment text can be MIME encoded, the user can now see the
original addresses.  The p1, p2, and p3 addresses are indexes into a
database shared between the MDA and MSA; the addresses in the comment
fields are just comments.  When the user responds to this message, the
MSA recognizes the proxy.local addresses, looks them up in the
database, and replaces them with the EAI addresses.  It might tidy up
the comments to remove extra copies of the EAI addresses, but that's a
quality of implementation issue.  For the rest of the headers and MIME
part headers, you do something, whatever one of the existing proposals
does.

There are all sorts of ways this could fail, e.g., a user sends mail
through another MSA, but I will wave my hands and say that my users
don't do that, or if they do, they get what they deserve.

If we're going to list possible downgrade strategies, why stop at the
two in the existing I-D's?  Should we publish some like this?  Why or
why not? 

R's,
John

---

From: [email-masked]
Date: Sun, 06 May 2012 07:50:28 -0700 (PDT)

> to get rid of non-ASCII text and not introduce addresses that could be

There are some who are arguing that all of this needs to be marked as
informational, not standards-track. One of the possible, and even likely,
outcomes of that is that implementors will feel *no* need to implement things
in a consistent fashion. So not being on the standards track could reduce
the availability of validation anchors even more than it needs to be.

Regardless, I see your point that documenting multiple approaches has
its own issues, almost independnent of what those approaches are.

> part headers, you do something, whatever one of the existing proposals

Very clever. I note that some MUAs *only* show the associated phrase text, with
the result that there are already cases where addresses are duplicated into the
phrase text so that users can see the actual address. This could be seen
as expanding on that.

> don't do that, or if they do, they get what they deserve.

Yes, least astonishment principle violations seem pretty likely in such a
setup, especially given how users take their laptops on th road...

				Ned

---

From: John C Klensin <[email-masked]>
Date: Sun, 06 May 2012 14:09:08 -0400

--On Sunday, May 06, 2012 14:42 +0000 John Levine
<[email-masked]> wrote:

> You're doubtless correct, but now I'm totally confused.  I
> think I'm seeing the suggestion that we provide a range of
> downgrade options, none of which can interoperate, and
> implementers will implement one of them or maybe do something
> totally different.  What's the benefit of putting anything in
> a standard beyond the advice that a downgrade has to get rid
> of non-ASCII text and not introduce addresses that could be
> replied to by mistake?

See Ned's note.

> We all seem to agree that it would be a good thing for users
> to see the addresses of their correspondents.  Let's say the
> message has these headers, with the capitalized strings
> standing for UTF-8.
> 
> Since comment text can be MIME encoded, the user can now see
> the original addresses.  The p1, p2, and p3 addresses are
> indexes into a database shared between the MDA and MSA; the
> addresses in the comment fields are just comments.  When the
> user responds to this message, the MSA recognizes the
> proxy.local addresses, looks them up in the database, and
> replaces them with the EAI addresses.  It might tidy up the
> comments to remove extra copies of the EAI addresses, but
> that's a quality of implementation issue.  For the rest of the
> headers and MIME part headers, you do something, whatever one
> of the existing proposals does.
> 
> There are all sorts of ways this could fail, e.g., a user
> sends mail through another MSA, but I will wave my hands and
> say that my users don't do that, or if they do, they get what
> they deserve.

Variations on this theme in which elements of the delivery
system tried to supply the MSA with specific information was
considered very early in the WG's life and again when we first
started discussing Submission server downgrading.  It was
rejected then, but nothing prevents you from bringing it up
again.

> If we're going to list possible downgrade strategies, why stop
> at the two in the existing I-D's?  Should we publish some like
> this?  Why or why not? 

I think it is clear that we could generate a number of different
proposals if we were so inclined, each with different
assumptions about implementation strategies, deployment
strategies, interactions among client systems and POP/IMAP
servers, delivery MTAs, storage and submission servers, and/or
user behavior.

Speaking as a co-chair with a perceived mandate to get this work
wrapped up, pushing for documenting more (standards track or
not) different approaches as a prerequisite for completing the
existing four documents would be an excellent strategy for very
long delays.  I'm sure you don't intend that, but unless the WG
has consensus that such delays are appropriate, I feel a need to
push back against any more proposals.  An alternative would be
to put text into the the POP and IMAP specs indicating that
downgrading is an issue and that there are several proposals
around which might be published in the future.  We could then
publish those two documents, shut down the WG, and leave
everything else to the tender mercies of individual submissions.
My guess is that approach would not be popular with those who
think the current two proposals are fairly well developed and
likely to get traction in the community, but I'm happy to do
whatever the WG likes as long as it doesn't involve another year
or more of delays (unless the WG clearly agrees to that).

I suggest we put the question of whether we are in a hurry or
not on the agenda of the Jabber chat and then issue a consensus
call.  If you are serious about this proposal, I suggest you
generate an I-D so that we can put the question of whether the
WG wants to consider and cope with it on the Jabber chat agenda
as well.

    john

---

From: Claudio Allocchio <[email-masked]>
Date: Sun, 6 May 2012 22:56:05 +0200 (CEST)

> say that my users don't do that, or if they do, they get what

There are good reasons to be very careful in promoting some field to a 
different scope in order to supply missing or useful information to act on 
inside a message header or evelope if the used filed is unstructured, and 
vaguely  specified in scope.

A true story to explain it:

in the '70s, the IBM MVS "SMTPNOTE" mail system (a quite primitive batch 
mail system) looked like

  //SENDNOTE EXEC PGM=IEBGENER
  //SYSIN DD DUMMY
  //SYSPRINT DD SYSOUT=*
  //SYSUT2 DD SYSOUT=(B,SMTP)
  //SYSUT1 DD *
  HELO MVSHOST
  MAIL FROM:<[email-masked]>
  RCPT TO:<[email-masked]>
  DATA
  From: "some comment here" <[email-masked]>
  To: "some comment here" <[email-masked]>
  Subject: Test message from MVS using SMTP

  This is a line in the body of the note.
  .
  QUIT

The first attempt to build a gateway between early RFC821/822 and MVS used 
the "some comment here" filed to convey informaton to be used by the 
gateways... a pity that the chosen fields (something like 
<*informatiohere>) was interpreted by MVS JCL as a set of undocumented but 
existing instructions, and the SMTPNOTE job had to run in EXEC mode (e.g. 
superuser). The System/390 mainframe of Geneva CERN IT division crashed 
and all jobs and data in the batch queues were lost. 5 minutes later I had 
on the phone a quite angry IT division director (and later IETF chair), 
Brian Carpenter.

> or more of delays (unless the WG clearly agrees to that).

Now that hopefully I do not crash any more mainframes, I'm not in favour 
of leaving open to any (even good) alternate ideas the already slippery 
path of "downgrading". It's just MHO, of course... my cents, but let's 
stay with the current 2 proposals, and see what will be implemented out 
there.

---

From: "John R Levine" <[email-masked]>
Date: 6 May 2012 17:30:48 -0400

> vaguely  specified in scope.

I agree.  I did say I didn't think we should publish my hack.  But it 
seems to me that anything beyond simpledowngrade runs a similar risk.

> Now that hopefully I do not crash any more mainframes, I'm not in favour of 
> leaving open to any (even good) alternate ideas the already slippery path of 
> "downgrading". It's just MHO, of course... my cents, but let's stay with the 
> current 2 proposals, and see what will be implemented out there.

I'm not thrilled about them, but I suppose that if we publish them as 
informational they'd be mostly harmless.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sun, 06 May 2012 19:22:17 -0400

--On Sunday, May 06, 2012 22:56 +0200 Claudio Allocchio
<[email-masked]> wrote:

> There are good reasons to be very careful in promoting some
> field to a different scope in order to supply missing or
> useful information to act on inside a message header or
> evelope if the used filed is unstructured, and vaguely
> specified in scope.

Yes.  And that is one of the advantages of the "separate header
fields" architecture of popimap-downgrade in comparison to trick
encoding and embedding hacks of various sorts.  Of course there
are disadvantages as well but, since we've been telling mail
implementers for many years to ignore (or purge) header fields
they don't recognize, the odds of unintended consequences are
vastly reduced.

> in the '70s, the IBM MVS "SMTPNOTE" mail system (a quite
> primitive batch mail system) looked like
>...

Yeah.  We did a bit better with VM/CMS version of that problem,
largely because we didn't try to integrate code going to or from
the Internet gateway with the (almost non-existent) operating
system mail facility.  I think that was rather more good fortune
than any particular insight on our part, but the issues with
trying to encode information in places that other software might
misunderstand were pretty clear.
 
>...
> Now that hopefully I do not crash any more mainframes, I'm not
> in favour of leaving open to any (even good) alternate ideas
> the already slippery path of "downgrading". It's just MHO, of
> course... my cents, but let's stay with the current 2
> proposals, and see what will be implemented out there.

Speaking personally rather than as co-chair, I agree.

   john

---

From: [email-masked]
Date: Mon, 07 May 2012 14:17:53 +0900 (JST)

> From: John C Klensin <[email-masked]>

I read the message now because of Japanesde holiday week.

In-transit-downgrading worked at the experimental spec and
implementing new MUA that understands Downgraded-* header fields may
be easy.

But, In my opinion, it is better to implement and deploy EAI capable
MUAs rather than to implement MUAs which understand popimap downgraded
messages.

The popimap downgrade should be used to inform that the receiver
received EAI messages and should update his/her MUA.

--
Kazunori Fujiwara, JPRS <[email-masked]>

---

From: [email-masked]
Date: Mon, 07 May 2012 14:50:00 +0900 (JST)

> From: Arnt Gulbrandsen <[email-masked]>

"message/global" messages should not contain "Downgraded-*" header
fields.

If an message contains both non-ASCII header value and Downgraded-*
header fields, the message is broken.

A senario which show the broken case:
  - a user uses fetchmail and procmail to retreive messages from pop servers
  - and forward them to non-ASCII e-mail address.
  - where fetchmail does not know UTF8 capability

 ->  The forwarded message contains both
     - Downgraded header fields
     - non-ASCII header fields (Received, Return-Path, Delivered-To, ...)
       which came from non-ASCII RCPT TO in SMTP.

Otherwise, the broken case came from malicious entities.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 7 May 2012 08:45:33 +0200

> header fields, the message is broken.

Yes. But what should happen?

Should the message be stripped of its wrongness at delivery time? In
that case popimap-downgrade cannot apply to mail stored before it was
deployed, or an time-consuming migration is required to deploy it.

Should the message be edited on disk when popimap-downgrade sees it? In
that case popimap-downgrade can only be deployed for mailboxes to which
the server has write access. Further, if that's done for IMAP, the
document needs to specify that the server issue a new UIDVALIDITY.

The third alternative is to keep th message in RAM and discard it at the
end of the mailbox session. But in that case even frequent and until now
cheap operations like UID FETCH 1:* RFC822.SIZE become expensive.

Arnt

---

From: [email-masked]
Date: Mon, 07 May 2012 20:17:52 +0900 (JST)

> Should the message be stripped of its wrongness at delivery time?

It is out of scope of pop/imap.
Bogus messages are delivered to pop/imap mailboxes as is.

> In

POP server or IMAP server shuold not change messages when a client
support UTF8 capability. It is out of scope of pop/imap downgrading.

> Should the message be edited on disk when popimap-downgrade sees it?

A user may use multiple POP/IMAP clients. Some clients may support
UTF8 capability, and others may not support UTF8 capability.

Then, the POP/IMAP server must keep the messages as is.
And the server may have downgraded version additonally (as a cache).

> In

The UIDVALIDITY value should be calculated by the original message.

Is the UIDVALIDITY for the original message same as the UIDVALIDITY
for the downgraded message ?

> cheap operations like UID FETCH 1:* RFC822.SIZE become expensive.

You can keep the message in the disk cache.
The downgrading occurs when the first lookup from non-UTF8 clients.

Is it OK?

---

From: "John Levine" <[email-masked]>
Date: 7 May 2012 12:35:41 -0000

>cheap operations like UID FETCH 1:* RFC822.SIZE become expensive.

I would think that if your downgrade were something like simpledowngrade
you could downgrade on the fly, as the message passes from the message store
through the POP or IMAP server and out to the client.  As far as I can tell
it doesn't depend on any interactions between headers, so you can do it one
line or header at a time.

R's,
John

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 14 May 2012 15:04:07 +0100

On 07/05/2012 12:17, [email-masked] wrote:
>> From: Arnt Gulbrandsen<[email-masked]>
Just to clarify one thing:

A message in IMAP is immutable (i.e. it can't change its basic 
properties, other than IMAP flags/annotations). The message is uniquely 
identified by UIDVALIDITY (per mailbox value) + its UID.
So if the message changes in the IMAP mailstore, it has to get a new 
UID, or the UIDVALIDITY for the mailbox has to change. The latter 
signals clients that all of its cached messages are no longer available 
and have to be purged from the cache.

---

From: John C Klensin <[email-masked]>
Date: Mon, 14 May 2012 11:02:21 -0400

--On Monday, May 14, 2012 15:04 +0100 Alexey Melnikov
<[email-masked]> wrote:

> properties, other than IMAP flags/annotations). The message is
> uniquely identified by UIDVALIDITY (per mailbox value) + its
> UID.
> So if the message changes in the IMAP mailstore, it has to get
> a new UID, or the UIDVALIDITY for the mailbox has to change.
> The latter signals clients that all of its cached messages are
> no longer available and have to be purged from the cache.

Right.  I think that, among other things that means that, if I'm
a server and I try the strategy of converting/downgrading the
message when it enters the mail store and then keeping both
versions and deciding which one to deliver based on what the
client asks for, I'd better be _really_ alert and careful about
the above.  I'm not even certain it is possible to get right, at
least while preserving UIDVALIDITY at all.

I started thinking about a slightly different, but related,
problem over the weekend having to do with what happens when a
user accesses a mail store with an EAI-compliant IMAP client, an
EAI non-compliant IMAP client, a webmail client (probably
compliant), and compliant and non-complaint POP clients that use
the notorious "leave mail on server" option and that do so in
different orders.  One can get a bad headache rather easily.  My
use of the term "variant of message" rather than "downgraded
message" in my "critical path issues" note earlier today was a
consequence of that because, in many respects, a "downgraded
message" (simple or otherwise) isn't really a transformed copy
of the same message -- it is a new message with a familial or
derivative relationship to the original one that the server
chooses to deliver instead.

best,
   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 May 2012 17:10:16 +0200

On 05/14/2012 05:02 PM, John C Klensin wrote:
> I'm not even certain it is possible to get right, at

I think simpledowngrade gets it right, since it doesn't touch any
non-EAI message at all. popimap-downgrade has a bit of a problem, since
it depends on downgraded-* not being there for non-EAI input messages.

I think that perhaps the IMAP EAI document should say something like:

 A client which is updated to add UTF8=* should be aware that if it
 has cached any part of an internationalized message before
 it was extended with UTF8=* support, then that part may be
 some sort of compatibility rendering by the server. Clearing the
 cache when the upgrade is deployed may be advisable.

Arnt

---

From: [email-masked]
Date: Mon, 14 May 2012 08:29:00 -0700 (PDT)

> it depends on downgraded-* not being there for non-EAI input messages.

It's more than a problem, it's a disaster in the making.

The problem is that you cannot make *any* assumptions about the presence or
absence of downgraded-* fields. This is because they're certain to leak - 
consider the case of a non-EAI client that resends a message (an operation
that's likely to retain these headers) to an EAI client that resends the
message again, this time adding resent-* fields that contain EAI addresses.
Now you have a perfectly legitimate EAI message that contains downgraded-*
fields.

And simpledowngrade doesn't really do much better, because you now end up
with invalid addresses in the regular from and to fields.

This is why I continue to believe that all of this stuff is not only
going to generate support calls, they're going to be the sort of calls where
figuring out what happened requires real expertise.

>  cache when the upgrade is deployed may be advisable.

And what about download-and-delete clients (which exist for both POP and IMAP)?
Yes, such clients won't suffer from UIDVALIDITY issues, but the messages are
essentially irreversibly corrupt at this point. 

				Ned

---

From: [email-masked]
Date: Mon, 14 May 2012 09:06:28 -0700 (PDT)

> chooses to deliver instead.

If you want to turn that headache into a migraine, consider what happens when a
single EAI client alternately accesses a mailbox on the campany LAN, which
allows direct access, and through a capability-filtering corporate firewall
that removes the EAI capability from the server announcement.

And yes, such firewalls do exist for POP, IMAP, and SMTP. And you can't assume
they filter all the protocols in a consistent way either - does that upgrade
this to a cluster headache?

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 14 May 2012 12:37:31 -0400

--On Monday, May 14, 2012 09:06 -0700 Ned Freed
<[email-masked]> wrote:

>> user accesses a mail store with an EAI-compliant IMAP client,
>> an EAI non-compliant IMAP client, a webmail client (probably
>> compliant), and compliant and non-complaint POP clients that
>> use the notorious "leave mail on server" option and that do
>> so in different orders.  One can get a bad headache rather
>> easily.  My use of the term "variant of message" rather than
>> "downgraded message" in my "critical path issues" note
>> earlier today was a consequence of that because, in many
>> respects, a "downgraded message" (simple or otherwise) isn't
>> really a transformed copy of the same message -- it is a new
>> message with a familial or derivative relationship to the
>> original one that the server chooses to deliver instead.
> 
> If you want to turn that headache into a migraine, consider
> what happens when a single EAI client alternately accesses a
> mailbox on the campany LAN, which allows direct access, and
> through a capability-filtering corporate firewall that removes
> the EAI capability from the server announcement.
> 
> And yes, such firewalls do exist for POP, IMAP, and SMTP. And
> you can't assume they filter all the protocols in a consistent
> way either - does that upgrade this to a cluster headache?

Yep.   At some level, this is all part of "all of the
alternatives to just get everything upgraded are pretty horrible
and no one is hugely less horrible than the others", but it is
yet another situation about which the IMAP spec should probably
call out (and hence yet another reason for putting the text in
the IMAP spec rather than the POP one).  As promised during the
Jabber chat meeting, I'll try to get text to the list as soon as
I finish with a couple of other things (include the draft
minutes), but please help me and the WG get the tone of that
text right.

   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 May 2012 18:51:17 +0200

> It's more than a problem, it's a disaster in the making.

Perhaps.

> absence of downgraded-* fields. [...]

OK for me — I don't.

> with invalid addresses in the regular from and to fields.

IMO, that's the irreducible minimum problem. It's a nasty one, no
question, but if we're going to have EAI, we have to pay that price.

> figuring out what happened requires real expertise.

Yes.

> essentially irreversibly corrupt at this point. 

Yes. A serious problem.

But I think it's not relevant to that document, since there's nothing to
do about it. Fetchmail, for instance, can add EAI support, but there's
nothing it can do about the mail it has deleted, so why bother to talk
about that?

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 May 2012 19:00:29 +0200

> essentially irreversibly corrupt at this point. 

I have a feeling that during this transition, the late implementers are
going to feel more pain than the early ones.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Mon, 14 May 2012 13:32:01 -0400

--On Monday, May 14, 2012 19:00 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

>> And what about download-and-delete clients (which exist for
>> both POP and IMAP)? Yes, such clients won't suffer from
>> UIDVALIDITY issues, but the messages are essentially
>> irreversibly corrupt at this point. 
> 
> I have a feeling that during this transition, the late
> implementers are going to feel more pain than the early ones.

That is both poetic justice (for a change) and implies
accelerating pressure to upgrade (probably A Good Thing).

   john

---

From: [email-masked]
Date: Mon, 14 May 2012 11:01:23 -0700 (PDT)

> question, but if we're going to have EAI, we have to pay that price.

Not if you eliminate as many downgrade scenarios as possible.

> Yes.

Any time a support call requires real expertise you're past front line and
even second line suport, and at that point costs can easily go up by a factor
of 10 or more.

> nothing it can do about the mail it has deleted, so why bother to talk

I'm sorry, but I strongly disagree. It's absolutely vital that these
specifications list all the known consequences of their use, *especially*
those consequences that cannot be mitigated, so that people deploying
this stuff can make informed choices about how to set things up, and indeed,
whether or not to deploy it at all.

I can almost accept the argument that these things can be covered in some other
document, but the problem with that is that both the consequences and the
mitigation methods are inextricably linked to what downgrade approach is used.
We see any example right here: Downgrade-* headers have different issues than
insertion of invalid addresses.

				Ned

---

From: [email-masked]
Date: Mon, 14 May 2012 11:18:19 -0700 (PDT)

> accelerating pressure to upgrade (probably A Good Thing).

I'm not sure this is really relevant, but FWIW...

I don't see how this is really any different from the deployment of MIME,
or ESMTP, or many other things.

It's been my experience that the main factor isn't when, but who. If you're an
open source developer, for example, you're not under direct financial
pressure... And you don't have to care about customer support if you don't want
to. So you have a lot of control over your pain level, and most (wisely) choose
to keep that level low.

The same goes for the commercial entities whose software runs on vast numbers
of deployed systems. You're in the envious position of having your stuff be
treated as the defacto standard more often than not. You can release
egregiously broken crap but as long as it self-interoperates it becomes
everyone else's problem to deal with.

The implementors in the middle are the ones who get screwed. They're under
direct financial pressure to deliver, they have to deal with "but it works fine
when foo talks to itself, that means you must be the one who is broken", and so
on.

As for what the middle group sees, the distribution of pain tends to be
tricameral over time. The first hump is the early implementor hump, where the
first set of interoperability issues are shaken out. The second is when the big
players get on board, and invariably screw something up that you have no choice
but to deal with. (See above.) And the final hump, which is smaller but with a
long tail, is dealing with the crap that cannot or will not be upgraded.

Of course there's an added hit, which is that whenever you enter you have your
own bugs and deployment issues to sort out. And in the case of EAI those are
going to be nonnegligable. The best way to handle this is to try and get
between the humps. If you wait too long you'll overlap that long tail - which I
guess is the case you're thinking of - and that's not going to be pleasant. But
given that most implementors of this sort don't have full control over their
release schedule, getting the timing right can be very tricky. You're probably
best off implementing as soon as possible, because that gives you a shot at
being between the first two humps.

So I guess this is a longwinded way of saying I come to more or less the same
conclusion but for different reasons.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 14 May 2012 15:08:05 -0400

--On Monday, May 14, 2012 11:18 -0700 Ned Freed
<[email-masked]> wrote:

> I don't see how this is really any different from the
> deployment of MIME, or ESMTP, or many other things.
>...
 
> So I guess this is a longwinded way of saying I come to more
> or less the same conclusion but for different reasons.

Ned, I think "yes".

I think there is one small difference between EAI --perhaps not
generally, but at the IMAP/POP end of things.  I have no idea
how much of a difference it makes.  A system operator,
especially one at the "we are giving things away for free that
are worth what you pay for them and are big enough to not care"
end of the spectrum can avoid all issues with client interfaces
to the mail store by simply refusing to advertise EAI capability
at the delivery server.  Nothing requiring EAI capabilities ever
makes it into the mail store, so it makes no difference whether
POP or IMAP clients (or webmail interfaces) are upgraded or not.
The MIME design avoided that particular control point and, with
the possible exception of using SIZE to negotiate really big
message transfers, the early/core SMTP extensions were about
making things work more efficiently or smoothly, not about
whether or not the messages went through.  

I think that difference is going to affect our deployment
dynamics, including where at least some of the pain appears and
who it affects.  But I can't even guess at scenarios beyond that
other than the one we've periodically predicted for the basic
SMTP and Message Format extensions, i.e., that deployment and
use is likely to spread outward from particularly-interested
language / script communities.

But I don't see the above --whether correct or not-- affecting
our documents in any way.

best,
   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 14 May 2012 21:40:30 +0200

> whether or not to deploy it at all.

All of the documents from this WG are going to be read mostly by people
who are implementing some EAI thing or another, or are already
implementing. You're talking about people who might/will deploy, which
is a different audience.

I'm not at all opposed to having a document about deployment issues, I
think it would be a fine thing, but I'm definitely not volunteering to
write that.

Arnt

---

From: [email-masked]
Date: Mon, 14 May 2012 13:56:20 -0700 (PDT)

> implementing. You're talking about people who might/will deploy, which

I'm doing nothing of the sort. There are *implementation* choices to be
made here, including but not limited to which downgrade approach to use.

This was actually the case even with one downgrade mechanism: Implementations
choices don't stop at simply writing a module that performs a downgrade
operation. There are a myriad of choices and tradeoffs as to how this interacts
with the server, and the consequences of downgrade, including the
non-mitigatable ones, are absolutely a part of that.

But even that weren't the case, the minute you proposed an alternative
downgrade mechanism, you bought in to all of this stuff. And with all due
respect, I suggest you start dealing with that reality.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 15 May 2012 09:14:18 +0200

> made here, including but not limited to which downgrade approach to use.

Sorry, I guess I misunderstood.

> respect, I suggest you start dealing with that reality.

Do you have more concrete suggestions? Because I've thought about
implementation discussion and advice and in the case of simpledowngrade
I've even tried to write some, and failed. My foresight/insight is not
good enough to offer anything useful.

Arnt

---

From: "John Levine" <[email-masked]>
Date: 15 May 2012 14:26:21 -0000

>> respect, I suggest you start dealing with that reality.

I have to agree with Arnt, all approaches to dealing with EAI mail to
non-EAI recipients are horrible.  If you do something different the
horror changes but doesn't go away.

If you reject EAI messages to non-EAI recipients, people will
complain.  If you deliver downgraded messages, they're not repliable
and people will complain.  If you hide them or deliver placeholders
saying to upgrade your MUA, people will complain.  If your MDAs notice
EAI mail to non-EAI users and your concierge service goes out to
upgrade the user's mail software, they'll complain that it's not a
convenient time, my mail works fine, what are you doing, go away.

I suppose some webmail systems will be able to upgrade their entire
infrastructure, and only then globally set the EAI flag on all their
incoming MTAs, but short of that, what do you suggest?

R's,
John

---

From: Douglas Otis <[email-masked]>
Date: Tue, 15 May 2012 11:12:55 -0700

On 5/14/12 11:01 AM, [email-masked] wrote:
> >>>> I'm not even certain it is possible to get right, at least
> >>>> while preserving UIDVALIDITY at all.
> >>>>
> >>> I think simpledowngrade gets it right, since it doesn't touch
> >>> any non-EAI message at all. popimap-downgrade has a bit of a
> >>> problem, since it depends on downgraded-* not being there for
> >>> non-EAI input messages.
> >>
Dear Ned,

One of the problems with EAI is many expect to leverage 
From-header-field domains to reference "authorization" records.  IMHO, 
that approach has serious problems.   Sources can still be authenticated 
via HELO/EHLO or StartTLS as a means to exclude phishing sources that 
otherwise place this type of email at extreme risk.  The HELO/EHLO must 
be encoded using A-Labels so in that sense it is immune to change.  As 
insurance against spoofing, conversion symmetry should be checked.

> >> The problem is that you cannot make *any* assumptions about the
> >> presence or absence of downgraded-* fields. [...]
>
> > OK for me — I don't.
>
> >> And simpledowngrade doesn't really do much better, because you
> >> now end up with invalid addresses in the regular from and to
> >> fields.

Perhaps multiple From header identities might become useful.  One with 
escaped local-part and A-label domain with the other as UTF-8 encoded 
that might be downgraded could represent a reasonable interim solution 
that simple downgrade still permits.

>  Not if you eliminate as many downgrade scenarios as possible.

Malefactors are very good at looking like one of these cases.

> >> This is why I continue to believe that all of this stuff is not
> >> only going to generate support calls, they're going to be the
> >> sort of calls where figuring out what happened requires real
> >> expertise.
>
> > Yes.
>
>  Any time a support call requires real expertise you're past front
>  line and even second line suport, and at that point costs can easily
>  go up by a factor of 10 or more.

This is where use of EHLO and optional dual From header field encoding 
might result in fewer support issues.  Those that occur could be 
explained in a FAQ in conjunction with a web based conversion tool.

> >>> A client which is updated to add UTF8=* should be aware that if
> >>> it has cached any part of an internationalized message before
> >>> it was extended with UTF8=* support, then that part may be some
> >>> sort of compatibility rendering by the server. Clearing the
> >> And what about download-and-delete clients (which exist for both
> >> POP and IMAP)? Yes, such clients won't suffer from UIDVALIDITY
> >> issues, but the messages are essentially irreversibly corrupt at
> >> this point.
>
> > Yes. A serious problem.
>
> > But I think it's not relevant to that document, since there's
> > nothing to do about it. Fetchmail, for instance, can add EAI
> > support, but there's nothing it can do about the mail it has
> > deleted, so why bother to talk about that?
>
>  specifications list all the known consequences of their use,
>  *especially* those consequences that cannot be mitigated, so that
>  people deploying this stuff can make informed choices about how to
>  set things up, and indeed, whether or not to deploy it at all.
>
>  I can almost accept the argument that these things can be covered in
>  some other document, but the problem with that is that both the
>  consequences and the mitigation methods are inextricably linked to
>  what downgrade approach is used. We see any example right here:
>  Downgrade-* headers have different issues than insertion of invalid
>  addresses.

Ensure possible solutions are not prevented.  Reluctance to implement 
dual identities within the From header field is understandable.  It 
breaks things like DKIM.  IMHO, EHLO in conjunction with ATPS type of 
authorization scheme provides a strategy that can adapt to the current 
email infrastructure as a reasonable means to mitigate risk.  It seems 
most ESPs hope to avoid reliance on EHLOs which would then require a 
scheme like ATPS.  ATPS can easily scale to cover any number of ESP 
services being used. This scheme can also supplant a need to preserve 
header fields as a means to mitigate risk.

Regards,
Douglas Otis

---

From: SM <[email-masked]>
Date: Tue, 15 May 2012 11:39:13 -0700

Hi Doug,
At 11:12 15-05-2012, Douglas Otis wrote:
>From-header-field domains to reference "authorization" 
>records.  IMHO, that approach has serious problems.   Sources can 
>still be authenticated via HELO/EHLO or StartTLS as a means to 
>exclude phishing sources that otherwise place this type of email at 
>extreme risk.  The HELO/EHLO must be encoded using A-Labels so in 
>that sense it is immune to change.  As insurance against spoofing, 
>conversion symmetry should be checked.

One of the problems is that there are problems outside the realm of 
EAI which are taken to EAI to resolve an issue outside EAI.

>Ensure possible solutions are not prevented.  Reluctance to 
>implement dual identities within the From header field is 
>understandable.  It breaks things like DKIM.  IMHO, EHLO in 
>conjunction with ATPS type of authorization scheme provides a 
>strategy that can adapt to the current email infrastructure as a 
>reasonable means to mitigate risk.  It seems

I am aware that someone else was, in my opinion, selectively quoting 
RFC 5321. :-)

Regards,
-sm

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 May 2012 17:21:11 -0400

--On Tuesday, May 15, 2012 14:26 +0000 John Levine
<[email-masked]> wrote:

>>> But even that weren't the case, the minute you proposed an
>>> alternative downgrade mechanism, you bought in to all of
>>> this stuff. And with all due respect, I suggest you start
>>> dealing with that reality.
> 
> I have to agree with Arnt, all approaches to dealing with EAI
> mail to non-EAI recipients are horrible.  If you do something
> different the horror changes but doesn't go away.

I think we have consensus on that.  Does anyone significantly
disagree?  If not, can we stop repeating it?

> complain.  If you deliver downgraded messages, they're not
> repliable and people will complain.  If you hide them or
> deliver placeholders saying to upgrade your MUA, people will
> complain.  If your MDAs notice EAI mail to non-EAI users and
> your concierge service goes out to upgrade the user's mail
> software, they'll complain that it's not a convenient time, my
> mail works fine, what are you doing, go away.

Sure.  Such is life.  And I haven't seen any recent disagreement
about this either.

> I suppose some webmail systems will be able to upgrade their
> entire infrastructure, and only then globally set the EAI flag
> on all their incoming MTAs, but short of that, what do you
> suggest?

The bottom line is that we have two choices:

	(1) Deploy EAI, expect a certain amount of pain as
	people do (or don't), and live with it.
	
	(2) Decide that fully internationalized email isn't
	worth the trouble after all.

We decided against (2) a rather long time ago.

At the risk of sounding optimistic. I do believe that, in
practice, this isn't going to turn out as badly as several of us
have been predicting.  A lot of the reason goes back, again, to
discussions we had when the WG first started up (and earlier).
If our hypotheses from that time are correct, it will basically
not be feasible for an email provider in areas where
internationalized addresses are important to avoid supporting
them (MTA, IMAP/POP server, IMAP/POP client/ MUA/ Webmail, etc.
In other areas, things will go more slowly, but the downgrade
problem won't be much of an issue -- it will take uses with
international correspondence very little time to figure out
that, if getting messages through is important to them, they
need to maintain ASCII return addresses and avoid pushing things
with EAI features.

That doesn't make the issues with, e.g., IMAP/POP downgrading
any less real. But we probably aren't headed into the sort of
global email interoperability disaster that the recent thread(s)
could be interpreted to imply.

   john

---

From: Claudio Allocchio <[email-masked]>
Date: Tue, 15 May 2012 23:39:33 +0200 (CEST)

> disagree?  If not, can we stop repeating it?

correct, we all agree.

> Sure.  Such is life.  And I haven't seen any recent disagreement

another +1

> We decided against (2) a rather long time ago.

otherwise we have just being playing games all the time, which is not 
true.

> need to maintain ASCII return addresses and avoid pushing things

I can add that, being a sort of a dinosaur myself, I've personally being 
running an ASCII only MUA for a long time, even while I was receiving 
already lots of UTF8 content messages, and just eventually decided to 
upgrade my MUA to an ITF8 compliant one not because I was fed up if the 
strange things happening while opening UTF8 contente bodyparts, but 
because I wanted other features in the MUA which had nothing to do with 
UTF8 support, and the UTF8 support just came is as an accident. I guess a 
very similar scenario can happen with EAI downgrading, and that's not so 
dramatic.

> could be interpreted to imply.

Indeed I'm not worried... and if this helps people in getting better MUAs 
installed, I'll be very happy.

all the best!

---

From: Douglas Otis <[email-masked]>
Date: Tue, 15 May 2012 15:09:27 -0700

On 5/15/12 11:39 AM, SM wrote:
>> From-header-field domains to reference "authorization" records.  
>> IMHO, that approach has serious problems.   Sources can still be 
>> authenticated via HELO/EHLO or StartTLS as a means to exclude 
>> phishing sources that otherwise place this type of email at extreme 
>> risk.  The HELO/EHLO must be encoded using A-Labels so in that sense 
>> it is immune to change.  As insurance against spoofing, conversion 
>> symmetry should be checked.
Dear SM,

Agreed.  Which is why leaving options open as simpledowngrade has done 
is likely the best approach.
>> authorization scheme provides a strategy that can adapt to the 
>> current email infrastructure as a reasonable means to mitigate risk.  
>> It seems
I suspect those concerned about security need to carefully consider how 
EHLO/HELO SMTP parameters/StartTLS/etc might be used to secure email.  
While ESPs may not relish this challenge, it can be made to work in a 
fashion similar to that of early UUCP, which should not prove disruptive.

I'll admit to making inferences from RFC5321 LDH DNS label restrictions 
and those in RFC6531.  Whether or not RFC6531 had been written, UTF-8 
email addresses have been in common use in Asia for several years, well 
ahead of written standards.

It seems a shame the IETF did not suffer the transition pain years ago 
and use UTF-8 in DNS, now used in LLMNR and mDNS.  The compression 
afforded makes visibility a PITA.  Problems related to UTF-8 would have 
been resolved long ago within far less complex environments.  In Asia it 
is fairly common to see UTF-8 headers and UTF-8 encoded sub-domains 
resolving hostnames.  This is likely done to assist customers over the 
A-Label speed bump who are using poorly supported clients.  In other 
words, don't assume that only valid A-labels resolve resource records.

Regards,
Douglas Otis

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2012 21:59:24 -0400

Doug and SM,

This note is written as co-chair in the hope of focusing the
discussion...

--On Tuesday, May 15, 2012 15:09 -0700 Douglas Otis
<[email-masked]> wrote:

>...
>> One of the problems is that there are problems outside the
>> realm of  EAI which are taken to EAI to resolve an issue
>> outside EAI.
 
> Agreed.  Which is why leaving options open as simpledowngrade
> has done is likely the best approach.

I don't know what SM's intent was, but mine is to say "not our
problem, please take it elsewhere" every time an issue comes up
that exists independent of the EAI work.

I'm sorry you weren't able to make the interim meeting/ call on
Monday, but I think there is reasonable agreement within the WG
that the only way were are going to make progress toward getting
things wrapped up and to the IETF is to avoid debates about
"best approaches" where legacy POP or IMAP clients are accessing
upgraded servers and trying to access SMTPUTF8 messages.  We
seem to be in agreement that all solutions to that problem other
than upgrading the clients are bad enough that "best" is not
meaningful.  Please see and, if appropriate, comment on, the
proposed new text I posted a short time ago.

>>> conjunction with ATPS type of  authorization scheme provides
>>> a strategy that can adapt to the  current email
>>> infrastructure as a reasonable means to mitigate risk.   It
>>> seems
>> I am aware that someone else was, in my opinion, selectively
>> quoting  RFC 5321. :-)
> I suspect those concerned about security need to carefully
> consider how EHLO/HELO SMTP parameters/StartTLS/etc might be
> used to secure email.  While ESPs may not relish this
> challenge, it can be made to work in a fashion similar to that
> of early UUCP, which should not prove disruptive.

Take it somewhere else.

> I'll admit to making inferences from RFC5321 LDH DNS label
> restrictions and those in RFC6531.  Whether or not RFC6531 had
> been written, UTF-8 email addresses have been in common use in
> Asia for several years, well ahead of written standards.

"In common use" is inconsistent with what the WG has been told
by those of its active participants, several of whom who come
from Asia and are familiar with (or operating) major email
systems there.  It is also inconsistent with recent inquiries
I've received about EAI status from governments, ISP/ESPs, and
others in South and Southeast Asia.  

But suppose you are correct?  What would you have the WG do
differently?  Or at all, other than getting on with its work and
being as little distracted by irrelevancies as possible.

> It seems a shame the IETF did not suffer the transition pain
> years ago and use UTF-8 in DNS, now used in LLMNR and mDNS.
> The compression afforded makes visibility a PITA.  Problems
> related to UTF-8 would have been resolved long ago within far
> less complex environments.  In Asia it is fairly common to see
> UTF-8 headers and UTF-8 encoded sub-domains resolving
> hostnames.  This is likely done to assist customers over the
> A-Label speed bump who are using poorly supported clients.  In
> other words, don't assume that only valid A-labels resolve
> resource records.

The issue of non-IDNA encodings for domain names was dealt with
at length in RFC 6055 and is not an EAI problem.  Please take it
elsewhere.

Please consider this note a warning that the WG Co-chairs
believe that the WG has consensus that it is important to move
things along and at least rough consensus that delays and
diversions into other topics are a threat to EAI-conforming
deployment and encouragement for non-interoperable local
solutions.  As a result, my (and probably our) patience for
off-topic postings -- including postings relevant to email or
i18n issues but not specific to EAI -- is extremely limited.
Please stop it.

   regards,
    john

---

From: Douglas Otis <[email-masked]>
Date: Thu, 17 May 2012 15:17:14 -0700

> proposed new text I posted a short time ago.
Dear John,

I'll attempt to be more clear.  The drafts look good and on track.  Good 
job.  An approach offering a visible and functional incentive to upgrade 
is an excellent choice.  It seems other related areas need to consider 
the impact UTF-8 or lack of support will have on security is not be 
mitigated without changes made elsewhere.  To be clear, this was 
responding somewhat holistically to criticisms of the approach, and that 
there is no ideal solution.  In other words, I too think this has found 
general agreement.
> being as little distracted by irrelevancies as possible.
The term "common" suggested occurrences too frequent to justify 
filtering, and not to imply a predominate number of UTF-8 local-parts et 
al, sans negotiation and not necessarily pertaining to ISP/ESP or 
government agencies, but regionally isolated to specific interests.  
This was to support rapidly moving this effort forward as there is a 
clear desire to use real names when sussing out the message author and 
where dependence on display names is not safe.
> Please consider this note a warning that the WG Co-chairs believe that 
> the WG has consensus that it is important to move things along and at 
> least rough consensus that delays and diversions into other topics are 
> a threat to EAI-conforming deployment and encouragement for 
> non-interoperable local solutions. As a result, my (and probably our) 
> patience for off-topic postings -- including postings relevant to 
> email or i18n issues but not specific to EAI -- is extremely limited.
Most concerns have been addressed in the specifications and represent 
excellent work.

Although already discussed in the draft, some have suggested UTF-8 
addresses only require simple A-Label conversions.  From a security 
stand point, this conversion must also ensure invalid code points are 
omitted.  Prior conversion routines may not offer this assurance.  It 
seems the draft might recommend something to help resolve this concern.  
Perhaps symmetry checks?

References to rfc5890 by follow-on documents is lacking.  While it might 
seem redundant, email can be stored for prolonged periods.  For example, 
there have been changes to UTF-8 domain encoding.  It seems such changes 
may impact UTF-8 domain handling.   As such, this deserves a reference 
to better highlight this aspect of interchange compatibility requirements.

Regards,
Douglas Otis