Subject: [EAI] POP3 Spec and mail drop/ Mail store organization

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2012 18:11:41 -0400

These are more or less nits but some of them are a fairly large
ones (and the IMAP spec should be checked to be sure it doesn't
suffer from any of the same problems).

The second paragraph of Section 3.1 starts "Maildrops can
natively store UTF-8 or be limited to ASCII.".  Essentially the
same statement appears at the beginning of paragraph 4 only it
says "mail store" rather than "maildrop".

Issues:

(1) I'm not quite sure what the difference is between a
"maildrop" and a "mail store".  If there isn't one, the
terminology should be harmonized.  FWIW, RFC 1939 mentions
"maildrop" in several places but never mentions a "mail store"
(or "mailstore").  If there is a difference, the document should
probably go to a little more effort to identify it.  

(2) The second paragraph of Section 3.1 starts "Maildrops can
natively store UTF-8 or be limited to ASCII."  As the inclusive
statement that appears to be, it is flat wrong.  First, if the
maildrop uses Unicode, we don't care what is stored "natively",
with various flavors of UTF-16 or UTF-32 being likely (and
common).  We don't even care if messages are stored in whatever
Charset arrives over the wire (and, indeed, storage of messages
in the form in which they were delivered may be common).  We
require that headers be either ASCII or UTF-8 on the wire, but
that says almost nothing about either maildrop storage form or
message content.  What we care about is that, if the messages
are delivered across a POP3 interfaces with UTF-8 enabled, the
header material is in UTF-8 on the wire and on delivery to the
POP client.

(3) The language about UTF-8 maildrops and messages can easily
be read to imply that a message that arrives with

  From: [email-masked]
  To:   [email-masked]
  [...]
  Content-type: text/plain; charset="GB2312"
  Content-transfer-encoding: base64
  [...]

Is required to be decoded, converted to 'charset="UTF-8"', and
then reencoded to base64.   I don't think that is our intention:
the conversion is burdensome and POP servers may not even have
the capability for all possible Charsets.  But, whether it is or
is not, the spec should be absolutely clear about what our
requirement actually is.

   john

---

From: Barry Leiba <[email-masked]>
Date: Wed, 16 May 2012 22:03:48 -0400

> probably go to a little more effort to identify it.

I think there is no difference.  I think 1939 used "maildrop" because
the model was the real post-office model: mail gets dropped there, and
the POP client picks it up (and it no longer resides at the drop, but
is now at the client).  With UIDL and the proliferation of "leave on
server" features, the maildrop has in many use cases given way to the
mail store model, where the server provides long-term storage for the
messages, even after they're picked up by (copied to, in this model)
the client.

I think we should stick with the terminology used in 1939.[1]

Barry

[1] Said with full realization that the extent to which that looks
like a year might be telling us that we're trying to live in the past.

---

From: "Jiankang YAO" <[email-masked]>
Date: Fri, 25 May 2012 17:32:00 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, May 17, 2012 6:11 AM
Subject: [EAI] POP3 Spec and mail drop/ Mail store organization

> probably go to a little more effort to identify it.  

yes, it should stick to maildrop.

> header material is in UTF-8 on the wire and on delivery to the

From my point of view, I see no problem of specifying "Maildrops can
 natively store UTF-8 or be limited to ASCII."

UTF-8 stored "natively", I think, is stored in UTF-8, not UTF-16 or UTF-32  or 
some Content-transfer-encoding: base64.

Section 3.1 said " Maildrops can natively store UTF-8 or be limited to ASCII.  UTF-8
   mode has no effect on messages in an ASCII-only maildrop. 
 Messages
   in native UTF-8 maildrops can be ASCII or UTF-8 using
   internationalized headers [RFC6532] and/or 8bit content-transfer-
   encoding, as defined in MIME Section 2.8 [RFC2045]."

Messages with some Content-transfer-encoding: base64, I think, is a kind of "limited to ASCII" based on the text above.

> then reencoded to base64.   I don't think that is our intention:

From the current words in section 3.1, I have seen no such meaning too.

> is not, the spec should be absolutely clear about what our

will try to refine it.

Jiankang Yao
>   john
> 
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Sun, 27 May 2012 10:13:29 -0400

--On Friday, May 25, 2012 17:32 +0800 Jiankang YAO
<[email-masked]> wrote:

>...
 
>> natively store UTF-8 or be limited to ASCII."  As the
>> inclusive statement that appears to be, it is flat wrong.
>> First, if the maildrop uses Unicode, we don't care what is
>> stored "natively", with various flavors of UTF-16 or UTF-32
>> being likely (and common).  We don't even care if messages
>> are stored in whatever Charset arrives over the wire (and,
>> indeed, storage of messages in the form in which they were
>> delivered may be common).  We require that headers be either
>> ASCII or UTF-8 on the wire, but that says almost nothing
>> about either maildrop storage form or message content.  What
>> we care about is that, if the messages are delivered across a
>> POP3 interfaces with UTF-8 enabled, the header material is in
>> UTF-8 on the wire and on delivery to the POP client.
>> 
> 
> From my point of view, I see no problem of specifying
> "Maildrops can  natively store UTF-8 or be limited to ASCII."

Either I didn't explain the problem very well or we are caught
on a problem with English, or both.   If you say, especially in
IETF-speak, "Maildrops can natively store UTF-8 or be limited to
ASCII", there is an implication of "and there are no other
choices".  It is possible to read it as implying "or they can
store it in any of several other ways", but that would be
uncommon.  At best, the statement is ambiguous.

As far as limiting the maildrop to UTF-8 or its ASCII subset
(and nothing else), I'd expect many systems to store data in
whatever form the operating system/file system uses, converting
to and from UTF-8 only to handle data on the wire.  So some
systems might use one of the UTF-16 or UTF-32 variations
internally, reflecting whoever they handled text.  They might
even use GB2312 or what the IETF calls ISO-2022-JP.    While
some of us might think that maintaining UTF-8 in the maildrop
would be a good idea, there are good reasons why it might not
be.  Far more important, to the best of my knowledge we have
never told an implementer how data must be represented while it
is stored on their systems.  There is also no justification for
our doing do except, maybe, for shared file system environments
(far outside EAI's scope): only data that appear on the wire can
affect interoperability.

> UTF-8 stored "natively", I think, is stored in UTF-8, not
> UTF-16 or UTF-32  or  some Content-transfer-encoding: base64.

Yes, that is how I read your comment.  I just think that it is
necessary to allow systems to _store_ data in whatever native
form works for them, not to try to somehow force them into UTF-8.

> Section 3.1 said " Maildrops can natively store UTF-8 or be
> limited to ASCII.  UTF-8    mode has no effect on messages in
> an ASCII-only maildrop.   Messages
>    internationalized headers [RFC6532] and/or 8bit
> content-transfer-    encoding, as defined in MIME Section 2.8
> [RFC2045]."

Let me say this a different way: I don't think we have any
business saying anything at all about "UTF-8 maildrops" except
as a short form for "maildrops that are capable of receiving and
sending information as Unicode codes encoded in UTF-8",    I
think the above hopelessly confuses "UTF-8 mode" (an EAI
characteristic that describes certain capabilities, commands,
and functionality), a UTF-8-capable maildrop (one that can
receive information in UTF-8 form, store it, and deliver/return
it in UTF-8 form without loss of information), and and the
actual internal ("native") storage form of a maildrop.

> Messages with some Content-transfer-encoding: base64, I think,
> is a kind of "limited to ASCII" based on the text above.

Sure it is.  But whether the maildrop should store content in
base64 or store it in some decoded form is an implementation
decision as long as (i) there is no information loss, and (ii)
for IMAP, the server pays careful attention to UIDVALIDITY as
discussed earlier on the list.

> From the current words in section 3.1, I have seen no such

Then why do you consider C-T-E;base64 to be relevant above.
Again, I'm not at all sure we have any substantive disagreement.
But it is _very_ easy to misunderstand the current text.

>> the capability for all possible Charsets.  But, whether it is
>> or is not, the spec should be absolutely clear about what our
>> requirement actually is.
>> 
> 
> will try to refine it.

thanks.
   john