Subject: [EAI] FWD: eai - Requested session has been scheduled for IETF 85

From: John C Klensin <[email-masked]>
Date: Mon, 22 Oct 2012 13:10:41 -0400

Hi.

We are still going around in circles with the IMAP document
(5738bis) relative to:

(1) Exactly what a server announcement of UTF8=ONLY means and
whether it adds sufficient value to justify the distinction
between it and UTF8=ACCEPT.

(2) What the distinction is between client-side "ENABLE
UTF8=ACCEPT" and "ENABLE UTF8=ONLY" and whether they are
actually different in practice.  If they are not, would we like
to eliminate them in favor of a single "ENABLE UTF8" (even if
the server announcements remain separate)?   If we retain both,
we need to provide a clear explanation as to when each should be
used (e.g., whether they can be used only in conjunction with
server capability announcements of "...ACCEPT" or "...ONLY"
respectively?

Because these are significant issues that require WG consensus,
not just matters of editorial tuning, and it is now clear that
we are not going to be finished in under two weeks (as I/we had
hoped), we have scheduled an IETF 85 meeting slot at 9AM on
Friday 9 November.  The slot is two hours long, but I don't
expect/ intend that we will meet for more than one.

The people who have been most engaged in the discussion so far
will be meeting earlier in the week and will have a clear
summary and, we hope, a proposal ready for Friday morning and
circulated to this mailing list earlier.

Sorry about this although it may be worth noting that more
checking and discussion of the document in the WG might have
prevented the problem.

best,
   john

---------- Forwarded Message ----------
Date: Monday, October 22, 2012 09:31 -0700
From: "\"IETF Secretariat\"" <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked],
[email-masked], [email-masked]
Subject: eai - Requested session has been scheduled for IETF 85

Dear Wanda Lo,

The session(s) that you have requested have been scheduled.
Below is the scheduled session information followed by
the original request. 

eai Session 1 (2:00:00)
    Friday, Morning Session I 0900-1100
    Room Name: Room 209
    ---------------------------------------------

Request Information:

---------------------------------------------------------
Working Group Name: 
Area Name: 
Session Requester: 

Number of Sessions: 1
Length of Session(s):  2 Hours
Number of Attendees: 50

[...]

---------- End Forwarded Message ----------

---

From: "Jiankang YAO" <[email-masked]>
Date: Tue, 23 Oct 2012 10:41:36 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, October 23, 2012 1:10 AM
Subject: [EAI] FWD: eai - Requested session has been scheduled for IETF 85

> server capability announcements of "...ACCEPT" or "...ONLY"

UTF8=ACCEPT: accepts UTF-8 and other encodings; It means that the server accept UTF-8, but not refuses other encoding.
UTF8=ONLY: only accepts UTF-8. It means that the server only accepts UTF-8, and refuses other encoding.

The explaination above is reasonable?

If yes, I think that the meaning has been expressed in the current draft.

Jiankang Yao

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 23 Oct 2012 18:03:10 +0300

> between it and UTF8=ACCEPT.

I'm hoping to avoid implementing message downgrading on the IMAP server side. So I'm thinking about allowing users to permanently switch to UTF8=ONLY mode (I guess there should also be some way to switch back). So I think it's a good idea to tell clients explicitly that there are UTF8 messages and no downgrading is possible. To me it seems that UTF8=ACCEPT allows at least potentially downgrading while UTF8=ONLY doesn't. So keeping them distinct would be nice.

This also reminds me that I think it would be a good idea to mention explicitly in the RFC that the UTF8=* capabilities (like all other capabilities) may be different for different users:

* OK [CAPABILITY IMAP4rev1]
a LOGIN user pass
a OK [CAPABILITY IMAP4rev1 UTF8=ONLY] Logged in.

> actually different in practice.  

I don't think there is any difference.

> If they are not, would we like

I think ENABLE UTF8 would be clearer, but I don't really care much.

> expect/ intend that we will meet for more than one.

I'll try to be in Jabber.

---

From: "Jiankang YAO" <[email-masked]>
Date: Wed, 24 Oct 2012 09:40:48 +0800

----- Original Message ----- 
From: "Timo Sirainen" <[email-masked]>
To: "IETF EAI" <[email-masked]>
Sent: Tuesday, October 23, 2012 11:03 PM
Subject: Re: [EAI] FWD: eai - Requested session has been scheduled for IETF85

> I'm hoping to avoid implementing message downgrading on the IMAP server side.
>

There had a jabber meeting in May. The consensus or summary for this meeting is here http://www.ietf.org/mail-archive/web/ima/current/msg04820.html . It decided that downgrading is stll kept in the draft after revising.

If downgrading is still there, UTF-8=ONLY makes transition smooth as it is said in the draft's appendix "The "UTF8=ONLY" mechanism simplifies diagnosis of interoperability  problems when legacy support goes away. "

Jiankang Yao

>
> So I'm thinking about allowing users to permanently switch to UTF8=ONLY mode (I guess there should also be some way to >switch back). So I think it's a good idea to tell clients explicitly that there are UTF8 messages and no downgrading is possible. >To me it seems that UTF8=ACCEPT allows at least potentially downgrading while UTF8=ONLY doesn't. So keeping them >distinct would be nice.
>

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 30 Oct 2012 13:14:45 +0000

> I'm hoping to avoid implementing message downgrading on the IMAP server side. So I'm thinking about allowing users to permanently switch to UTF8=ONLY mode (I guess there should also be some way to…
I tend to agree.
> a OK [CAPABILITY IMAP4rev1 UTF8=ONLY] Logged in.
Might be a good idea to add a clarifying example. On a somewhat related 
note, it might be worth actually documenting (in a separate draft) that 
CAPABILITY answer can change after TLS/authentication. Implementors 
"know", but as far as I remember RFC 3501 doesn't say that.
> I don't think there is any difference.
+1.
> I think ENABLE UTF8 would be clearer, but I don't really care much.

I don't care much either, but "UTF8" is not a registered IMAP capability and ENABLE accept a list of capabilities and not arbitrary tokens.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 30 Oct 2012 13:47:14 -0400

>> I'm hoping to avoid implementing message downgrading on the IMAP server
>> side. So I'm thinking about allowing users to permanently switch to
>> UTF8=ONLY mode (I guess there should also be some way to switch back). So I
>> think it's a good idea to tell clients explicitly that there are UTF8
>> messages and no downgrading is possible. To me it seems that UTF8=ACCEPT
>> allows at least potentially downgrading while UTF8=ONLY doesn't. So keeping
>> them distinct would be nice.
>
> I tend to agree.

And here's exactly the sort of "UTF8=ONLY is underspecified" thing
that I mean: I don't see where the current spec says that UTF8=ONLY
doesn't allow downgrading.  I don't see where the current spec says
anywhere near enough with respect to what UTF8=ONLY changes about how
the server works.

So, Timo (and Alexey), what makes you think that a server that
advertises UTF8=ONLY will never do downgrading?  And what makes you
think that knowing that would help any clients, given that downgrading
is meant for clients that don't understand any of this in the first
place?

Barry

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 30 Oct 2012 19:57:13 +0200

On 30.10.2012, at 19.47, Barry Leiba wrote:

> advertises UTF8=ONLY will never do downgrading?  

I think that's the only reasonable conclusion based on what its intention was. Although the current draft still explicitly only talks about the mailbox names being in UTF8. If that truly is the only reason for the capability, I don't see any point in having it, since the extra effort to implement mUTF7 is almost nothing. The only complexity I see is related to downgrading messages.

> And what makes you

I'm actually interested in knowing what the current most widely used clients would do if they saw UTF8 mailbox names and UTF8 headers. I would actually guess that they would handle them pretty well (except maybe the UTF8 email addresses). If they did, I think it would be okay for UTF8=ONLY server to simply assume that the client is UTF8-capable and just send it UTF8 data instead of simply giving errors for anything it tries to do.

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 30 Oct 2012 20:16:44 +0200

> I'm actually interested in knowing what the current most widely used clients would do if they saw UTF8 mailbox names and UTF8 headers. I would actually guess that they would handle them pretty well…

A quick test shows that Thunderbird and Apple Mail can handle UTF8 subject/from headers without any problems. Thunderbird was able to list and subscribe to UTF8 mailbox name correctly. When showing the mailbox in the actual list, both of them showed the UTF8 characters wrong and neither could select the mailbox (or in any case showed the mailbox as empty).

If anyone wants to try, I can send IP+user+password for the test server privately.

---

From: John C Klensin <[email-masked]>
Date: Tue, 30 Oct 2012 15:39:15 -0400

--On Tuesday, October 30, 2012 19:57 +0200 Timo Sirainen
<[email-masked]> wrote:

>> And what makes you
>> think that knowing that would help any clients, given that
>> downgrading is meant for clients that don't understand any of
>> this in the first place?
> 
> I'm actually interested in knowing what the current most
> widely used clients would do if they saw UTF8 mailbox names
> and UTF8 headers. I would actually guess that they would
> handle them pretty well (except maybe the UTF8 email
> addresses). If they did, I think it would be okay for
> UTF8=ONLY server to simply assume that the client is
> UTF8-capable and just send it UTF8 data instead of simply
> giving errors for anything it tries to do.

(personal opinion - chair hat off)

So you think that it is ok for a server to violate a protocol if
only if it announces it first -- in an announcement an
un-upgrade client will essentially not see?    If so, I want to
propose an SMTP extension that is announced in the EHLO response:
   Extension name: HEADERS-IN-INNER-SLOBBOVIAN-ENGLISH
   Client command or parameter response: none, client just has
to accept it.

Inner-Slobbovian-English is just like normal English, except
that a few words have their definitions transposed.  For
example, "From" and "Reply-To" in Inner-Slobbovian-English have
the same meanings as "Reply-to" and "From", respectively, in
English.  As with your tests of just sending UTF-8 header fields
to selected clients, this would work most of the time.  It would
cause havoc that the user would have a lot of trouble diagnosing
the rest of the time.

Really, I don't want to go there.  And I would hope that the
IETF community generally, and the IESG in particular, would not
approve an extension to a standard whose purpose is to allow the
server to unilaterally decide to violate the base protocol.

   sorry,
     john

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 30 Oct 2012 23:20:40 +0200

On 30.10.2012, at 21.39, John C Klensin wrote:

> only if it announces it first

Not exactly, not by default. I'm interested in having an option on the IMAP/POP3 server where the user can decide if he wants to use the "old email" or the "new UTF8 email". The default would be the "old email". An UTF8 capable client could enable the "new UTF8 email" for the session. But also the user (or admin..) could force enabling the "new UTF8 email" even if the client itself didn't explicitly support it.

Pretty much all IMAP clients can already deal with 8bit email headers in one way or another, because 8bit headers have already existed for a long time now (hotmail did it for a long time, spam mails still contain it, and most IMAP servers simply send them to clients). So assuming a user normally uses UTF8 capable IMAP client, but sometimes (e.g. on a phone) uses non-UTF8 capable client where the choices are to either a) not be able to read emails at all or b) violate some RFCs but still read something that is more or less readable on that client, I think b) isn't such a bad option. (This is assuming there would be no c) downgrading option.)

So those a) and b) options are the only possibilities of what I think UTF8=ONLY could mean. I don't know if it's necessarily a good idea for it to mean b), but then again it's not very useful for it to mean a) either. At least with b) the clients can possibly still access some/all emails, while with a) the user is only getting one error message after another, maybe never seeing any emails at all.

BTW. I heard of test results also for Outlook 2007: Doesn't show UTF8 headers properly, probably assumes they're in the system's default charset (CP-1252 in this case). Still the email headers were understandable enough. Of course if the subject was in some heavily non-ASCII language like Chinese it probably wouldn't be readable. But the email body would still be readable even if the headers weren't.

So I'm finding it difficult to think of a reason why I would spend a lot of time implementing the downgrading functionality when it's anyway going to be unnecessary some day in future, and even today it wouldn't really be necessary for most users since they could read the emails anyway (although maybe less perfectly than with the downgrading code). And note that I'm only talking about IMAP/POP3 side, on SMTP side I can see how downgrading is much more useful, but luckily I'm not implementing an SMTP server. :)

---

From: Timo Sirainen <[email-masked]>
Date: Tue, 30 Oct 2012 23:40:27 +0200

On 30.10.2012, at 23.20, Timo Sirainen wrote:

> And note that I'm only talking about IMAP/POP3 side, on SMTP side I can see how downgrading is much more useful, but luckily I'm not implementing an SMTP server. :)

On a related note, there are also some new potentially fun combinations for email clients, like:

 * IMAP server advertises URLAUTH and UTF8=ACCEPT
 * SMTP server advertises BURL but not SMTPUTF8

Few clients support URLAUTH/BURL currently, but if they did .. and if UTF8 support was added to the client, at least the current behavior of "first save the email to IMAP server's Sent mailbox and then send it via SMTP" would suddenly fail in the middle if it was first saved to IMAP server using UTF8 headers and then SMTP layer noticed it wouldn't work because SMTPUTF8 wasn't advertised.

---

From: John C Klensin <[email-masked]>
Date: Tue, 30 Oct 2012 18:11:00 -0400

--On Tuesday, October 30, 2012 23:20 +0200 Timo Sirainen
<[email-masked]> wrote:

>> So you think that it is ok for a server to violate a protocol
>> if only if it announces it first
> 
> Not exactly, not by default. I'm interested in having an
> option on the IMAP/POP3 server where the user can decide if he
> wants to use the "old email" or the "new UTF8 email". The
> default would be the "old email". An UTF8 capable client could
> enable the "new UTF8 email" for the session. But also the user
> (or admin..) could force enabling the "new UTF8 email" even if
> the client itself didn't explicitly support it.

This is just out of band negotiation of a violation (or, if you
prefer, non-standard extension) to the protocol.  Given the
assumed close relationship between IMAP servers and clients (we
don't use the term "split-UA" for nothing), I don't see anything
preventing you from doing this.   There might be a philosophical
discussion as to whether the result should be called "IMAP" or
"TimoIMAP", but who cares.  

On other hand, servers who don't know what clients are being
used with them or clients that, in practice and notwithstanding
some of the "split-UA" assumptions, have no control over what
server is being used, need an in-band mechanism.  EAI's choices
were to use capability announcements or to invent a new
mechanism.   And the WG selected the former, probably for good
reason.

In particular, if the server is set to assume and send "new UTF8
email" no matter what, and the user comes along with a legacy
client that does Bad Things when sent UTF-8, I wouldn't
recommend being the person who gets to debug it or tell the user
why that option was arranged for his convenience... unless I was
in an enterprise situation in which I got to say "you ran a
non-(corporate-)standard client and I'm not interested in your
problems".

All of this might be different if having the server advertise
the capability or having the client enable it was really a big
deal.  But, as others have pointed out, it really isn't.  The
only advantage your strategy would gain is a short window in
which some legacy clients would be able to accept these messages
without being upgraded.  That seems to me to be a rotten
tradeoff against throwing something at the user that might cause
bad behavior with other clients, but YMMD.

And, again, it is _really_ late in the process to be having this
discussion.

best,
   john

---

From: "John Levine" <[email-masked]>
Date: 30 Oct 2012 22:47:30 -0000

>Pretty much all IMAP clients can already deal with 8bit email headers in one way or another,
>because 8bit headers have already existed for a long time now ...

Back before my time, the EAI group went through extreme backflips to
try and provide a downgrade path so that EAI mail could more or less
work with non-EAI mail systems.  It became clear that the pain from
places where the sort-of-downgrade didn't work far outweighed the
benefits.

If your MUA is 95% of the way to handling EAI, that's great. It should
be easy to persuade its authors to make the tiny tweaks needed to do
the other 5% and tell the IMAP server that it does UTF8.

As far as I am concerned, I am only interested in two scenarios:

A) fully EAI aware software, does all of the EAI stuff

B) full downgrade, IMAP server continues to work with all existing
clients, yea back unto Outlook 1995 or whatever.

It is specifically not a goal to add kludges intended to allow people
to use non-EAI software to do mostly EAI stuff.  Please don't ask us
to go there.  Use that effort to upgrade your MUA instead.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 31 Oct 2012 00:59:51 +0200

> "TimoIMAP", but who cares.  

Yes, agreed.

> mechanism.   And the WG selected the former, probably for good

The discussion here I think is mainly about the choice of:

a) Don't have UTF8=ONLY capability at all

b) Have UTF8=ONLY capability only mean that mUTF-7 is not supported in mailbox list names.

c) Same as b) + also not support any kind of message downgrading. Some way of server not allowing access to UTF8 emails, most likely by not allowing client to select any mailboxes at all.

d) My thought of maybe it meaning that same as assuming client had issued ENABLE UTF8=ACCEPT.

The current draft seems to point towards b), but I don't think that makes sense at all (mUTF-7 code is very simple to implement, especially compared to downgrading related code). Technically I think UTF8=ONLY should mean c), but from user's point of view that is just about as useless as replacing IMAP4rev1 capability with IMAP5, meaning that the client couldn't access the mails at all anyway then. Of course, that might eventually be a good thing that users don't attempt to use old clients against new servers. I don't really know.

> non-(corporate-)standard client and I'm not interested in your

I wouldn't expect servers to be set for the forced-UTF8 mode outside small/corporate setups, at least for many more years. Also I don't think there are any clients that do Very Bad Things when they see UTF8, since spam has already forced them to deal with it for 10+ years.

> bad behavior with other clients, but YMMD.

Depends on how short or long the window is going to be. In the mean time I think the servers are going to need the force-feature in any case. But I guess eventually that feature could be disabled and non-downgrade-capable servers could simply not allow non-UTF8 capable clients to do anything. Of course, a long time later after "all" clients support it the servers might be too lazy to actually bother enforcing that.. So I think all of the solutions are all in all about temporary (dis)advantages that finally become non-issues.

> And, again, it is _really_ late in the process to be having this

I'm trying to keep this mainly about "What does UTF8=ONLY mean?"

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 31 Oct 2012 01:21:42 +0200

> clients, yea back unto Outlook 1995 or whatever.

Yes, I think those two are the important cases. And A) is (hopefully) the only one relevant in the far future.

> to go there.  Use that effort to upgrade your MUA instead.

I don't think it's about kludges. It's mainly a question of which one is better:

a) IMAP client shows all emails in some way. Yes, violating some RFCs, just like may spams already do, so maybe they won't show up correctly, but pretty much always they do show up in some way. Theoretically there are some issues with this that make it worse than not showing the emails at all, but overall I'd think they are almost nonexistent (or, guesstimating affecting <0,01% of users). In the far future this won't be an issue at all. Temporarily this may help some users to use non-UTF8 clients that otherwise would be completely unusable.

b) IMAP client doesn't show any emails in any way. The server simply says that you're using too old software and you need to upgrade. A clean solution from the protocol point of view, but not so awesome during the temporary transition. In the far future this won't be an issue at all either. For next few years I'm planning on making this an option on the server side, allowing it to be enabled per-user and/or globally.

Also does the RFC really even need to explicitly specify either behavior? It could simply be a hint to the client/user/debugger that "you really need to support this UTF8 stuff" and the server could decide whether to implement a) or b) behavior. Couldn't it be even recommended that "for the first few years it's a good idea to implement b) by default, but eventually a) is fine too"? I'm thinking that this is going to be the reality whether the RFC says it or not.

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 31 Oct 2012 01:35:58 +0200

> to go there.  Use that effort to upgrade your MUA instead.

Also another thought from practicality point of view:

1. Admin upgrades SMTP server to be UTF8 capable.
2. New emails arrive with UTF8 headers and get stored using regular mbox/maildir format.
3. Admin still uses an old generic non-UTF8 capable IMAP/POP3 server.

Regardless of what EAI RFCs say, what is going to happen is that the old non-UTF8 IMAP/POP3 server is going to send the mails to IMAP/POP3 clients using UTF8 headers. Of course this is mainly the admin's fault of not using compatible software, but I don't think the above is a very far fetched scenario.

---

From: Timo Sirainen <[email-masked]>
Date: Wed, 31 Oct 2012 02:04:19 +0200

I know I'm sending too many mails now about this already, but I think after writing the previous ones I've finally formed an opinion about the UTF8=ACCEPT vs. UTF8=ONLY issue:

No IMAP client is ever going to implement UTF8=ONLY in a different way than UTF8=ACCEPT. If the server supports only UTF8=ONLY, the clients' possibilities are then to:

a) Not support UTF8=*: The server will then fail the commands with "Sorry dude, you need to upgrade your IMAP client to use this awesome new server."

b) Recognize UTF8=ONLY but not actually support it: The client will fail the login with "Sorry dude, but I was too lazy to implement proper support for this server, but not too lazy to add this very helpful error message saying you can't use this server."

The end result is the same. The client can't use the server. In the far future neither of these things happen. In the mean time I seriously doubt any client developer will add anything like b), even with a different error message.

So my opinion: Merge UTF8=ACCEPT and UTF8=ONLY into a single UTF8 capability. Leave it to the server implementation to figure out what to do if the client doesn't issue ENABLE UTF8. It's fine to reject such clients or to downgrade (more, less or none at all) such clients.

---

From: "John R Levine" <[email-masked]>
Date: 30 Oct 2012 20:33:08 -0400

These are all questions that we addressed months ago.  It's much, much too 
late to reopen them now.

> I don't think it's about kludges. ...
>
> a) IMAP client shows all emails in some way. Yes, violating some RFCs,

That's a kludge, by definition.  Don't do that.

The whole point of the downgrade docs (have you read them?) is to describe 
ways to show EAI messages to to non-EAI clients.

> b) IMAP client doesn't show any emails in any way. The server simply 
> says that you're using too old software and you need to upgrade.

That's an option, but not a great one since the server will fill up with 
mail that the user can neither see nor delete.  We also discussed this 
months ago.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 31 Oct 2012 11:36:55 -0400

Timo says...
> So my opinion: Merge UTF8=ACCEPT and UTF8=ONLY into a single UTF8
> capability. Leave it to the server implementation to figure out what to do if the
> client doesn't issue ENABLE UTF8. It's fine to reject such clients or to
> downgrade (more, less or none at all) such clients.

That's my thought as well.  As I've said, I think ONLY was useful when
we had a larger set of options, and its existence now is a vestige
from that time.  But when we decided to drop the other options (ALL,
APPEND, and USER), we *did* decide specifically to keep ONLY, which is
why we need to have the discussion now about why we decided that at
the time.  It may well be that I've just forgotten the details, and
the distinction is still important.

b