Subject: [EAI] Unhappy with the complexity of pop-imap-downgrade

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 30 Jan 2012 13:17:04 +0100

TL;DR: Change downgrading to support what's most important to the end 
user, and drop the rest. Lessen the amount of code provided to implement 
downgrade to the minimum possible.

Rationale: While I find the downgrading document is well written, the 
downgrade itself is IMO much too complex to be worthwhile. There's too 
much to implement, too many cases to test, etc.

It defines new header fields that are intended to be parsable by new 
code and even encourages parsing them. That's not good. We want people 
to spend their time and energy implementing EAI natively, not writing 
clientside parsers for Downgraded-Header-Field-X.

Details: Below I give section numbers and suggested action.

Section 4 (about new parsable header fields, Downgraded-This-That):

New text: "Server MAY generate Downgraded- fields. Clients MUST NOT rely 
on such fields or try to parse such fields."

Section 5 (about downgrading various syntax elements):

5.1.1: If any Received field cannot be sent to non-EAI POP or IMAP 
clients, then all Received fields are silently suppressed.

5.1.4: Any comments that require EAI are silently excised when the 
header field is sent to an non-EAI client.

As an aside, I really, really hate this:
   Date: 14 aug 2011 12:00:00 +0200 (Mitteleuropäische Zeit)

5.1.5: Any MIME values that require EAI MAY be silently excised, or MAY 
be converted to use RFC2231 syntax.

5.1.8: I like the algorithm, but I would prefer that the exact address 
be explicitly undefined, so noone is tempted to detect EAI by looking at 
a magic group name.

Btw. For reasons I cannot at the moment remember, I generate

    "[email-masked]" <[email-masked]>

instead of an empty group. I'm sure I had a reason, though.

5.1.9: Seems to overlap with 4. I think 5.1.9 is a better location than 
5 for this.

5.2.x: I would change most of these to be examples showing the effect of 
the relevant 5.1.x rule.

6 (about MIME bodypart fields): Replace with "If any MIME values (e.g. a 
file name in content-disposition) require EAI, these values are silently 
excised before showing the header field to the client. If any other part 
of a bodypart fields requires EAI, the entire field is silently excised."

7. Strike about half, particularly the bit about parsing downgrade-*. 
Add a sentence such as "Sending EAI to non-EAI aware recipients offers 
new opportunities for misunderstandings, spoofing and deception, if the 
recipient is unable to read or understand a part of the message's meaning".

I apologise for suggesting such large changes so late.

Arnt

---

From: "John Levine" <[email-masked]>
Date: 2 Feb 2012 15:59:07 -0000

>downgrade to the minimum possible.

I have to agree.  The current document is basically a detailed design
to tunnel EAI messages through hostile MUAs that is so complex that it
seems unlikely to me that two implementations would ever match in
practice.  If that's what you want to do, you can always wrap the
messages as message/global attachments.

Arnt's suggested changes seem right, chop it down to the minimum
needed to produce a version of a message that won't break MUAs.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Thu, 02 Feb 2012 11:19:31 -0500

--On Thursday, February 02, 2012 15:59 +0000 John Levine
<[email-masked]> wrote:

>> TL;DR: Change downgrading to support what's most important to
>> the end  user, and drop the rest. Lessen the amount of code
>> provided to implement  downgrade to the minimum possible.
> 
> I have to agree.  The current document is basically a detailed
> design to tunnel EAI messages through hostile MUAs that is so
> complex that it seems unlikely to me that two implementations
> would ever match in practice.  If that's what you want to do,
> you can always wrap the messages as message/global attachments.
> 
> Arnt's suggested changes seem right, chop it down to the
> minimum needed to produce a version of a message that won't
> break MUAs.

<role cochair>

For the moment, let me note that there are other options and
that circumstances may differ in different environments.  Just
to help with the discussion (this isn't a proposal yet, much
less one that I'm ready to advocate), note that this entire
downgrading model is something that happens between the delivery
MTA, the mail store, and the MUA.   We make strong assumptions
here and elsewhere about that being a relatively tight,
mutual-consent, relationship.   That is not traditionally IETF
territory.  More important, the nature of interoperability
issues are quite different from having multiple, independent,
systems at different edges of the Internet.

Noting that I will continue to leave consensus decisions up to
Joseph, I also note that two voices (or even a few more) don't
change the presumed WG consensus to move forward with the
current spec as is.  The fact that we've gone a few years
without any discussion along this line certainly entitles the
existing document to the presumption that it is the way the WG
wants to go.

I wonder if it would be possible to define a family of
self-consistent downgrade options, such that we could preserve
the current model for those environments that needed it and were
willing to go to the trouble of implementing (or finding
implementations) and deploying it, a second option along the
lines the Arnt and John L are outlining, and possibly a third or
fourth option.  

For example, it occurs to me that, for some environments, the
right response to a retrieval request from a legacy IMAP or POP
client for a message that requires SMTPUTF8 capability might be
to leave the message in the mail store and deliver a synthesized
cousin of an DSN to the client that identifies the message and
suggests finding an upgraded client.

Going that route would take some work, but perhaps not much more
work than constructing the stripped-down version of Downgrade.
Possibly most of the hard parts are done already.

Is that worth thinking about as a group?

   john

</role>

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 2 Feb 2012 17:36:45 +0100

To be honest I think it's not.

This isn't the common path. Sending mail to people who cannot really read it is an exceptional activity, not the kind of thing that warrants very much server code or several different protocol variants.

Arnt

---

From: "John R Levine" <[email-masked]>
Date: 5 Feb 2012 20:42:32 -0500

> The fact that we've gone a few years without any discussion along this 
> line certainly entitles the existing document to the presumption that it 
> is the way the WG wants to go.

You're right, although it might also mean that nobody's been paying 
attention recently.  It feels to me like a vestige of the downgrade on 
demand model that EAI rejected for mail transport.

> the current model for those environments that needed it

Possibly.  In order of distance from the current text, I'd suggest:

1.  Since downgrades aren't reversible, there isn't really an interop 
reason to specify the details.  Say that for headers that contain UTF8 
text, do some combination of MIME encoding where possible (most headers), 
redacting parts of them (address headers), and throwing them away (trace 
headers).

2.  To deliver an EAI message to a non-EAI MUA, wrap it as a 
message/global, copy whatever headers you can into the wrapper message, 
and it's the MUA's problem.

3.  John's suggestion, synhesize a DSN that says you need a better MUA and 
deliver that to the user.

4.  For the rest of EAI, we made the decision that the EAI and legacy mail 
streams are parallel but separate, if you want to send an EAI message and 
there isn't an EAI path to the recipient, you lose.  Why should this be 
any different?  If the user doesn't have an EAI MUA, he can't pick up mail 
from an EAI POP or IMAP server.

My inclination leans toward #2, since MUAs that don't support EAI can 
often nonetheless display UTF-8 text.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 6 Feb 2012 12:17:45 +0100

Fwiw, the reason for me to implement a kind of downgrade is that EAI mail may appear in folders whet you don't expect it, such as the one in which you store this mailing list.

It would be rather irritating if, say, your EAI-ignorant smartphone could not read any mail in that folder merely because someone posted a message to the list using EAI. Downgrade needs to be good enough to let you read the rest of your mail and have some idea of what's in the mail you cannot properly read. Locking your phone out of the entire folder would not be good.

Arnt

---

From: [email-masked]
Date: Mon, 06 Feb 2012 20:19:20 +0900 (JST)

The purpose of post-delivery message downgrading is
that POP/IMAP servers can send/remove internalionalized messages
to traditional POP/IMAP clients
and receivers can know they received internationalized messages.

No popimap-downgrading, POP/IMAP server could not send EAI messages to
traditional clients and POP server cound not remove the messages.

> From: "John R Levine" <[email-masked]>
> text, do some combination of MIME encoding where possible (most
> headers), redacting parts of them (address headers), and throwing them
> away (trace headers).

Current draft.
Tried not to break current MUAs. (Except allowing empty from:).

> message/global, copy whatever headers you can into the wrapper
> message, and it's the MUA's problem.

Do you think that end user understands what happend?

> 3.  John's suggestion, synhesize a DSN that says you need a better MUA
> and deliver that to the user.
> 
> 4.  For the rest of EAI, we made the decision that the EAI and legacy
> mail streams are parallel but separate, if you want to send an EAI
> message and there isn't an EAI path to the recipient, you lose.  Why
> should this be any different?  If the user doesn't have an EAI MUA, he
> can't pick up mail from an EAI POP or IMAP server.

Any MUA can access EAI POP servers which offer EAI messages for the MUA.

Port number and protocols are the same.

--
Kazunori Fujiwara, JPRS <[email-masked]>

---

From: "John R Levine" <[email-masked]>
Date: 6 Feb 2012 10:15:26 -0500

> Fwiw, the reason for me to implement a kind of downgrade is that EAI mail may appear in folders whet you don't expect it, such as the one in which you store this mailing list.

Understood, but there's simpler ways to do that than to specify a detailed 
non-reversible downgrade scheme.

Regards,

---

From: "John R Levine" <[email-masked]>
Date: 6 Feb 2012 10:37:32 -0500

> Tried not to break current MUAs. (Except allowing empty from:).

Agreed, but we could strip out a great deal of the detail and just say 
here are the three techniques you can use to fix headers, use whichever 
ones you want.  Since the transformations aren't reversible and the user 
can't reply to a message with an EAI return address, the details don't 
really matter.

> Do you think that end user understands what happend?

I don't know, but I don't see any reason to think that it's more or less 
baffling than a message full of Downgraded-this and redacted that headers. 
The IETF has never been very good at user interface design.

> Port number and protocols are the same.

We could simplify section 3.1 of RFC5721bis and section 4 or RFC5738bis, 
so that if the client hasn't said UTF8, requests to retrieve EAI messages 
fail.  They currently offer a choice between fail and downgrade, just take 
out the downgrade choice.

Regards,

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 7 Feb 2012 11:01:52 +0100

On 02/06/2012 04:15 PM, John R Levine wrote:
>> Fwiw, the reason for me to implement a kind of downgrade is that EAI
>> mail may appear in folders whet you don't expect it, such as the one
>> in which you store this mailing list.
> 
> Understood, but there's simpler ways to do that than to specify a
> detailed non-reversible downgrade scheme.

Exactly. _A_ downgrade is justified, not a high-fidelity one.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Tue, 07 Feb 2012 11:55:39 -0500

<hat=co-chair>

Folks,

I'd like to suggest that people dial down the rhetoric and try
extra-hard to listen to and understand each other because there
may be a lot of different, valid, perspectives on the subject.

A few of us have pointed out that the IETF doesn't do well with
UI designs, but, while there may be some assumptions, UI designs
are not really the issue here.  What is an issue is the
relationship among the delivery MTA, the message store, the IMAP
or POP server, the user and her MUA (and IMAP or POP client),
and maybe some other pieces.  Operational deployment policies
may be important too: if I'm an operator or IT department in an
organization that is basically ASCII-oriented and I'm not going
to advertise SMTPUTF8 capability from the delivery MTA until I
have EAI-competent IMAP and POP clients available and have told
people to convert to them, my attitude toward downgrades may be
very different from, e.g., being in a mostly non-ASCII
environment in which people are very anxious to enable SMTPUTF8
and non-ASCII addresses on the delivery servers but where I
expect the deployment rate for upgraded clients to have a very
long tail.  

For obvious reasons, the IETF has traditionally stayed away from
trying to put requirements on those operational relationships
among consenting (and usually connected) parties.    None of us
really know each other's operational environments or
assumptions; we know even less about environments and
assumptions that are not represented in the WG.

So, while I think it is useful for us to explore and, if
appropriate, document, a range of options for different
scenarios, I think we should skip trying to tell others what
they do or do not need.  

I noted that, for one scenario, I'd like to explore mechanisms
for giving the IMAP or POP user a clear "there are messages
waiting for you that you aren't going to see until you switch
clients or find a different access method" message.  I don't
think that is the only answer, or even a "best" answer, just
that it matches at least one plausible scenario.

So, while I'm open to other ideas, I'd like to see those who
think that scenarios other than the one anticipated by the
current pop-imap-downgrade are important to prepare I-Ds
describing the scenarios they anticipate and what the
corresponding protocol would look like, both in enough detail
that the WG can look at the drafts and figure out whether there
is consensus on publication.  As part of this, I think we should
try to modify the current pop-imap-downgrade draft to include a
description of the scenario(s) that would justify its "preserve
all possible information even if it cannot be used in automated
replies" approach.

Please remember that our target is still to have the set of
POP-IMAP documents in the hands of the IESG well before the
Paris IETF so, if you are going to write, write quickly.  That
said, I don't know that there would be a big problem handling
additional scenarios/protocol specifications as individual
submissions if that were necessary.

It would be nice to end up with no more than three
scenario-solution combinations: no downgrade (refuse to
deliver), full information preservation, and somewhere in
between.  But I see no need to require that at this point, nor
any possibility of doing so.  Let's see some drafts.

If anyone has better ideas that would help us move forward
efficiently, please say so.

    john

---

From: "John R Levine" <[email-masked]>
Date: 7 Feb 2012 23:37:36 -0500

> may be a lot of different, valid, perspectives on the subject.

You're right, it really depends on your model of how users will make the 
transition from legacy to EAI mail.  Here's some possibilities I can think 
of when an MTA starts handling EAI mail:

1.  Users don't have to upgrade, but get EAI mail anyway: the server 
creates some version of EAI messages for legacy clients, downgraded or 
wrapped.  If they're serious about supporting legacy clients, they'd need 
some way for legacy users to respond to EAI mail, e.g., the server creates 
local forwarding addresses for EAI addresses in downgraded or wrapped 
messages, and substitutes them into the headers.  (I am most definitely 
not suggesting we spec out the way that would work.)

2. Users are expected to upgrade: server can synthesize DSNs that say 
"upgrade your MUA like we told you to if you want to see your mail."

3. Users who want EAI mail upgrade, users who don't care, don't: the 
server remembers which users are EAI enabled, based on whether they've 
sent EAI mail or set the UTF8 flag in a POP or IMAP session.  Server 
rejects incoming EAI mail to non-EAI users.  For legacy POP and IMAP 
sessions from EAI users, it synthesizes DSNs that say "use your EAI mail 
program."

I can see any of these being reasonable, depending on the user community 
and its relationship to the mail service.  But I have to say, the current 
downgrade where you can mostly see messages but not respond to them seems 
cruel.  If you want to provide backward compatibility for your legacy 
users, it should let them use mail like they're accustomed to using it.

R's,
John

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 08 Feb 2012 19:54:40 +0900

> submissions if that were necessary.

Quick question: Is downgrade an integral part of the POP-IMAP documents 
set, or could it be treated as a separate thing, so that the main POP 
and IMAP documents could move ahead even if we have to take some more 
time with downgrade?

Regards,   Martin.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 8 Feb 2012 12:13:49 +0100

> and IMAP documents could move ahead even if we have to take some more

I'm not sure.

As far as I could tell, none of the IMAP, POP, SMTP or Submission
extensions involve a promise on part of the server to support IMAP/POP
downgrading. But I find that a little surprising, so maybe there is a
promise I've overlooked?

Arnt

---

From: John C Klensin <[email-masked]>
Date: Wed, 08 Feb 2012 07:06:13 -0500

--On Tuesday, February 07, 2012 23:37 -0500 John R Levine
<[email-masked]> wrote:

>...
> I can see any of these being reasonable, depending on the user
> community and its relationship to the mail service.  But I
> have to say, the current downgrade where you can mostly see
> messages but not respond to them seems cruel.  If you want to
> provide backward compatibility for your legacy users, it
> should let them use mail like they're accustomed to using it.

It is maybe worth noting that one of the old jokes about X.400
was the the only way to reply to most X.400 messages involved
picking up the phone (either to respond directly or to find a
route).  We know how much X.400 traffic there is today, probably
in part as a consequence.  On the other hand, people can
eliminate those problems by upgrading (either an MUA or a
provider) which, for X.400 users who didn't like the "reply by
phone" option, there was mostly the possibility of "upgrading"
to Internet mail.  Many did.

So I would suggest that cruelty lies somewhat in the eye of the
beholder.  And, while circumstances differ, if it encourages
upgrading, maybe one should be in favor of a little cruelty :-(

    john

---

From: John C Klensin <[email-masked]>
Date: Wed, 08 Feb 2012 07:43:38 -0500

--On Wednesday, February 08, 2012 19:54 +0900 "\"Martin J.
Dürst\"" <[email-masked]> wrote:

> Quick question: Is downgrade an integral part of the POP-IMAP
> documents set, or could it be treated as a separate thing, so
> that the main POP and IMAP documents could move ahead even if
> we have to take some more time with downgrade?

Your quick question doesn't have a quick answer.  It would
almost certainly be possible to restructure the POP and IMAP
documents to make a downgrade model (very broadly construed and
probably including the explicit "there is a message here that
you can't read" possibility that John Levine and I have been
poking at) a future task.  Whether the IETF Last Call and the
IESG would let us get away with that is a different question --
the IETF has often, but not always, been uncomfortable with
situations that require transitions but where the documents come
with no plan or discussion of a strategy.

In addition and even more important, the WG has so far assumed
that some sort of plan is needed to deal with the case of a
delivery MTA that accepts SMTPUTF8 messages but that something
between there and the user can't handle them.   I think it might
be hard to get consensus in the WG to walk away from that
position although, if there were enough interest, we could ask
the question (while remembering that the IETF community might
push back and at least require a thorough analysis if not a
protocol).

Speaking personally, I'd also hope the WG would ask the question
as to whether "POP and IMAP now; downgrade after we take some
more time" would actually buy anything.  Remembering that our
target isn't to get documents published but to get systems and
capabilities deployed, unless there was evidence that there were
important actors who would deploy without these
post-delivery-MTA issues being resolved by the WG, splitting
downgrade handling into a separate, later, task might actually
result in a longer time before deployment occurred (because we'd
have to spend time revising and reviewing the POP and IMAP
documents to deal with the change rather than focusing that
energy on getting the downgrading model resolved.

    john

---

From: John C Klensin <[email-masked]>
Date: Wed, 08 Feb 2012 08:03:00 -0500

--On Wednesday, February 08, 2012 12:13 +0100 Arnt Gulbrandsen
<[email-masked]> wrote:

>...
> As far as I could tell, none of the IMAP, POP, SMTP or
> Submission extensions involve a promise on part of the server
> to support IMAP/POP downgrading. But I find that a little
> surprising, so maybe there is a promise I've overlooked?

The problem is that the server can't know and the promise should
not be necessary in the long term (everything past the delivery
server EAI-capable).  Trying to draw a picture that covers all
of the cases, including closely-connected message stores and
ones to whom messages are passed (via LMTP, shared NFS-like
storage, or otherwise), POP/IMAP clients with varying
capabilities, and so on. is really illuminating about how many
cases there are, what information has to be available in what
places, etc.  A delivery server _could_ promise to always
downgrade a message before depositing it in or transferring it
to the the mail store but that is exactly the behavior we want
to discourage in order to favor EAI-capable mail stores.

    john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 8 Feb 2012 14:38:18 +0100

That answer makes perfect sense to me. A fine, coherent design.

Arnt