Subject: [EAI] Proposed revised text for IMAP (5738bis) and related documents

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2012 19:52:51 -0400

Hi.

I propose the following changes to the IMAP
[draft-ietf-eai-5738bis] and POP [draft-ietf-eai-5721bis] specs
to reflect the discussion on Monday's Jabber-chat/ virtual
meeting.  These are subject to editorial discretion by the
authors of that document, i.e., I'm trying to get the ideas
outlined in my Monday note down but am not particularly
attached to the suggested text.  In particular, I've tried to
reflect both my Monday note, the Jabber conversation, and a few
other notes to the list within the last week.  That has
resulted in somewhat more words than even I would like. I've
put them in on the assumption that it is easier to cut things
that are unnecessary out than to add text; the next round of
decisions are up to the WG and the authors/editors.

An important point in the text below may not be obvious, but I
think we should move away from talking about "downgraded" or
"downconverted" messages.  In the context of other mail
protocols, both imply conversions, such as those of
content-transfer-encodings, that preserve all information.
I've used "variant message" and its synomyms/ variants(sic)
below. I note that simpledowngrade uses "synthetic message",
which would be equally satisfactory.   Anyone who has a strong
preference should speak up; I do hope we can settle on one or
the other of the "variant" or "synthetic" alternatives rather
than having to explain to the community and IESG why we are
using both terms.

I am prepared, given the text below, to explain why we don't
get rid of the term "downgrade" entirely, but, if others
disagree, please speak up (especially if you are willing to do
the work).

Specific changes:

(IMAP.1) Introduction, second paragraph.  Replace the paragraph
by:

	Most of this specification assumes that the IMAP server
	will be operating in a fully internationalized environment,
	i.e., one in which all clients accessing the server will be
	able to accept non-ASCII message header fields and other
	information as specified in Section 3.  At least during a
	transition period, that assumption will not be realistic
	for many environments; the issues involved are discussed in
	Section 7 below.

(IMAP.2) New Section 7 (subsequent sections to be renumbered):

	7. Dealing With Legacy Clients

	In most situations, it will be difficult or impossible for
	the implementer or operator of an IMAP (or POP) server to
	know whether all of the clients that might access it, or
	the associated mail store more generally, will be able to
	support the facilities defined in this document.  In almost
	all cases, servers who conform to this specification will
	have to be prepared to deal with clients that do not enable
	the relevant capabilities.  Unfortunately, there is no
	completely satisfactory way to do so other than for systems
	that wish to receive email that requires SMTPUTF8
	capabilities to be sure that all components of those
	systems -- including IMAP and other clients selected by
	users -- are upgraded appropriately.

	Choices available to the server when a message that
	requires SMTPUTF8 is encountered and the client doesn't
	enable UTF-8 capability include hiding the problematic
	message(s) as outlined elsewhere in this specification,
	creating in band or out of band notifications or error
	messages, or somehow trying to create a variation on the
	message with the intention of providing useful information
	to that client about what has occurred.  Such variant
	messages cannot be actual substitutes for the original
	message: it will rarely be possible to reply to (either at
	all or without loss of information), new header fields or
	specialized constructs  for server-client communication
	may go beyond the requirements of, e.g., RFC 5322 and may
	consequently confuse some legacy mail user agents
	(including IMAP clients) or otherwise not provide the
	expected information to users.  There are also tradeoffs in
	constructing variants of the original message between
	accepting complexity and additional computation costs in
	order to try to preserve as much information as possible
	(for example, in [draft-ietf-eai-popimap-downgrade]) and
	trying to minimize those costs while still providing useful
	information (for example, in
	[draft-ietf-eai-simpledowngrade]). 

	Because such messages are really variations on the original
	ones, not really "downgraded" (ones although that
	terminology is often used for convenience), they inevitably
	have relationships to the original ones that the IMAP
	specification [RFC3501] did not anticipate.  In particular,
	digital signatures computed over the original message will
	often not be applicable to the variant version and servers
	that may be accessed by the same user with different clients
	or methods (e.g., POP or webmail systems in addition to
	IMAP or IMAP clients with different capabilities) will need
	to exert extreme care to be sure that UIDVALIDITY behaves
	as the user would expect.  Those issues may be especially
	sensitive if the server caches the variant message or
	computes and stores it when the message arrives with the
	intent of making either form available depending on client
	capabilities. 

	The best (or "least bad") approach for any given
	environment will depend on local conditions, local
	assumptions about user behavior, the degree of control the
	server operator has over client usage and upgrading, the
	options that are actually available, and so on.  It is
	impossible, at least at the time, to give good advice that
	will apply to all situations, or even particular profiles
	of situations, other than "upgrade legacy clients as soon
	as possible".

(IMAP.3) Change the second paragraph of "Security
Considerations" to read;

	Special considerations, some of them with security
	implications, occur if a server that conforms to this
	specification is accessed by a client that does not and in
	some more complex situations in which a given message is
	accessed by multiple clients that might use different
	protocols and/or support different capabilities.  Those
	issues are discussed in Section 7 above.

(POP.1) Most of the fourth paragraph of Section 3.1 ("Mail
stores are either ASCII or native UTF-8, and clients either
issue...") needs to disappear.   Considering a separate note
that I just sent (Subject: POP3 Spec and mail drop/ Mail store
organization") and the instructions from the Jabber chat to the
POP3 editors to get rid of excess "UTF8" parameters, this
suggestion may need considerable rewriting, but, following the
theme above, I suggest something like:

	Normal operation for UTF-8 maildrops will be for both
	servers and clients to support the extension discussed in
	this specification.  Upgrading of both clients and servers
	is the only fully satisfactory way to support the
	capabilities offered by the "UTF8" extension and SMTPUTF8
	mail more generally.  Servers must, however, anticipate the
	possibility of a client attempting to access a message that
	requires this extension without having issued the "UTF8"
	command.  There are no completely satisfactory responses
	for that case other than upgrading the client to support
	this specification.  One solution, unsatisfactory because
	the user may be confused by being able to access the
	message through some means and not others, is that a server
	MAY choose to reject the command to retrieve the message as
	discussed in Section 5.  Other alternatives, including the
	possibility of creating and delivering variant form of the
	message, are discussed in Section 7 of the IMAP UTF-8
	specification [draft-ietf-eai-5738bis].

IMO, we can safely just lose the extended discussion of sizes,
etc., in the existing paragraph.  Part of the reason for
shifting the terminology from "downgraded message" to "variant
form of the message" is that no one sane would expect sizes,
etc. to remain the same.  If others think it is needed, I
suggest adding as much of it as possible to the IMAP text
(IMAP.2 above) and then moving the rest to the Security
Considerations section of the POP3 doc (where there is now a
back-reference pointing to 3.1, that reference would need to be
corrected in any event).

(POP.2) In Section 4, drop from "Mechanisms for 7-bit..."
through the end of the paragraph and replace by:

	When the server is not in UTF-8 mode and the message
	requires that mode, requests to download the message MAY
	be rejected (as specified in the next section) or the
	various other alternatives outlined in Section 3.1 above
	and in Section 7 of the IMAP UTF-8 specification
	[draft-ietf-eai-5738bis], including creation and delivery
	of variations on the original message, MAY be considered.

(POP.3) Section 5, last sentence.  I recommend just dropping
the parenthetical note, but the authors are welcome to propose
another way to dance around the issue.  Note that one of the
goals of this set of suggestions is to eliminate all reference
to the "downgrade" specs from the POP3 UTF-8 document and,
ideally, get "downgrade" (and, worse, "downconvert") out of the
POP3 UTF-8 document's vocabulary.

(POPIMAP-Downgrade.1) "Security Considerations": Consider
whether any of this material should be moved to either the new
Section 7 or the Security Considerations section of the IMAP
spec.  I believe that any material that would apply to any
"downgrading" transformation should be moved to the IMAP spec
(the paragraph starting "Rewriting" is an obvious candidate)
and referenced from here (and SimpleDowngrade) but, in the
interest of getting these notes out without more delay, I
haven't tried to do the work of identifying what, exactly
should be moved.

(SimpleDowngrade.1) Arnt, please review Section 1 Paragraph 4.
"High-fidelity" might easily be read to imply "no information
loss".  My instinct (personal opinion) is that you just drop
everything from ", since implementing" to the end of the
paragraph, but other suggestions that support the "only
horrible alternatives" and then talk about tradeoffs would be
welcome.  In particular (personal opinion again), if one
considers providing sufficient explanations to enable customer
support as part of the package, I think it is sensible to argue
that your downgrading method is more trouble than implementing
5721bis and 5738bis (Note that your references should point to
the new docs, not the earlier ones -- your note under "Open
Issues is superfluous because the chance of this document being
approved except in a package with 5721bis and 5738bis is
exactly zero -- please let's not make it hard for the RFC
Editor... or easy for them to make mistakes).

(SimpleDowngrade.2) Note the interim meeting consensus that
"invalid addresses" are to be avoided.  I haven't noticed
objections to that decision on the mailing list, so, unless I
missed something important, I presume it will stand.

(SimpleDowngrade.3) See the observations about message sizes in
POP.1 above, think about whether this discussion should be
abstracted somewhat and moved to the new Section 7 (or Security
Considerations) of the IMAP spec, and dropped from here.  Note
that any variant/synthesized message is going to have this
problem.

(SimpleDowngrade.4) Given the on-list discussion about UIDs and
UIDVALIDITY, do you think a Downgraded/synthesized message has
the same UID value as the non-ASCII original or a different
one?  I think the text should be clear -- and then either moved
to the IMAP spec or checked to be sure that popimap-downgrade
is consistent.

best regards,
   john

---

From: Barry Leiba <[email-masked]>
Date: Wed, 16 May 2012 22:42:05 -0400

I like the proposal in general, and don't have major comments about
it.  Lots of text tweaking and error corrections, of course, but that
all needs to wait until text is folded into the actual documents, and
the document editors have had their ways with it.

> to the IMAP spec or checked to be sure that popimap-downgrade

This is, of course, a very tricky point, but I'll note one thing:
It's generally assumed that all clients see the same UIDVALIDITY for a
mailbox, and the same UIDs for each message... but I don't see
anything that necessitates that, since every client deals with UIDs
independently of all others.  I suggest, then, that we don't have to
worry about UIDVALIDITY, and that servers that do this message
substitution simply present the same UID for a given message,
regardless of whether that message is sent to the client in UTF-8 or
not.  My (compliant) client will see a different message with that UID
than your non-compliant one will, but they will conceptually refer to
the same message, and will, therefore, have substantially the same
content.

The failure mode of this approach is that when the user upgrades your
client to the new, newly compliant version, and that version uses the
same local message cache as the earlier version, it retains the
"synthetic" message, and doesn't know that it needs to refresh its
cache.  A clean way for an updated client to know that a cached
message has been synthesized would be a good way out of this, so maybe
we should consider that (an IMAP flag?).

Barry

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2012 23:25:13 -0400

--On Wednesday, May 16, 2012 22:42 -0400 Barry Leiba
<[email-masked]> wrote:

> I like the proposal in general, and don't have major comments
> about it.  Lots of text tweaking and error corrections, of
> course, but that all needs to wait until text is folded into
> the actual documents, and the document editors have had their
> ways with it.
> 
>> (SimpleDowngrade.4) Given the on-list discussion about UIDs
>> and UIDVALIDITY, do you think a Downgraded/synthesized
>> message has the same UID value as the non-ASCII original or a
>> different one?  I think the text should be clear -- and then
>> either moved to the IMAP spec or checked to be sure that
>> popimap-downgrade is consistent.
> 
> This is, of course, a very tricky point, but I'll note one
> thing: It's generally assumed that all clients see the same
> UIDVALIDITY for a mailbox, and the same UIDs for each
> message... but I don't see anything that necessitates that,
> since every client deals with UIDs independently of all
> others.  I suggest, then, that we don't have to worry about
> UIDVALIDITY, and that servers that do this message
> regardless of whether that message is sent to the client in
> UTF-8 or not.  My (compliant) client will see a different
> message with that UID than your non-compliant one will, but
> they will conceptually refer to the same message, and will,
> therefore, have substantially the same content.

This works for me, but my primary concern is that we not leave
the issue up in the air.  We should, IMO, pick something and be
consistent about it or state clearly that we have no intention
of doing so.

> The failure mode of this approach is that when the user
> upgrades your client to the new, newly compliant version, and
> that version uses the same local message cache as the earlier
> version, it retains the "synthetic" message, and doesn't know
> that it needs to refresh its cache.  A clean way for an
> updated client to know that a cached message has been
> synthesized would be a good way out of this, so maybe we
> should consider that (an IMAP flag?).

As usual, I'm reluctant to do anything in EAI that has
non-specific implications.  An IMAP flag
("synthesized_message"?) would certainly do the job, but so
would something as simple as either recommending that client
upgrades treat UIDVALIDITY as false and rebuild the cache on the
assumption that _some_ message-or-flag-affecting capability,
storage format, etc. might have changed.  As a trivial example
that is not EAI-specific (but part of which came up an earlier
rant), suppose that non-EAI client version 1.98 tries to store
text/plain information whenever possible in ISO-2022-JP (or
8859-1) even if that means verifying that there are no
signatures present and converting from whatever form arrives
even if it UTF-8.  I can think of several practical reasons why
one might want to do that.  Now suppose that client version 2.0,
in recognition of other changes in the world and preparatory for
version 2.1 which has full EAI support, decides to bite the
bullet and store everything locally in Unicode form (UTF-16BE if
that is the local OS preference) and maybe to do a little less
converting.  I think it would want to resynchronize and
reevaluate the cache, not try to convert old messages to
reconstruct the guessed-at original form.  Right?

But I'd defer to the IMAP experts on the list.

    john

---

From: "Jiankang YAO" <[email-masked]>
Date: Fri, 18 May 2012 16:59:11 +0800

thanks to John's hard work and kind proposed changes.

If the following comments or issuses are addressed in the next version of 4 documents, is there any other issuse which will 
not be addressed?

If there is none, I think that it will be a big step forward when 4 new version of documens appear.

Jiankang Yao

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, May 17, 2012 7:52 AM
Subject: [EAI] Proposed revised text for IMAP (5738bis) and related documents

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 22 May 2012 23:25:25 -0400

> thanks to John's hard work and kind proposed changes.

Just trying to move things along.

> If the following comments or issuses are addressed in the next
> version of 4 documents, is there any other issuse which will 
> not be addressed?

Certainly the "get rid of extra 'UTF-8' parameters in favor of
using only ENABLE (or equivalent) changes that we discussed on
the call need to be made.  I'm not aware of anything else of
significance.

> when 4 new version of documens appear.

It will be a big step when we get new drafts that reflect these
changes either way.  And I anxiously await new versions.

   johm

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 25 May 2012 15:31:17 +0100

>> therefore, have substantially the same content.
Right. There is only an issue if the same clients sees different 
versions of the same message with the same UIDVALIDITY+UID.
Possible cases for this are:
1). An EAI-capable client was talking to a non-EAI capable server. The 
server was upgraded to support EAI.
2). An non EAI-capable client was talking to an EAI capable server. The 
client was upgraded to support EAI.
3). An EAI capable client accesses EAI capable mailstore sometimes 
directly and sometimes through IMAP proxies that remove the EAI capability.

I suppose we could warn about cases 1 and 2 (In the 1st case the server 
should change UIDVALIDITY. In the 2nd case the client should blow away 
its cache, even if the UIDVALIDITY hasn't changed.)
> storage format, etc. might have changed.
Exactly.
> As a trivial example
I think some text about upgrading clients and servers would be helpful.

---

From: "Jiankang YAO" <[email-masked]>
Date: Thu, 31 May 2012 11:37:25 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, May 17, 2012 7:52 AM
Subject: [EAI] Proposed revised text for IMAP (5738bis) and related documents

> decisions are up to the WG and the authors/editors.
...
> POP3 editors to get rid of excess "UTF8" parameters, 
>

Does it mean to  get rid of excess "UTF8" parameter "argument", and remove the 3.2 section
"USER Argument to UTF8 Capability".

From Joseph's Jabber summary http://www.ietf.org/mail-archive/web/ima/current/msg04803.html, 

it seems that there is no summary related to POP.

Jiankang Yao

>this