Subject: [EAI] LANG, UTF-8, and POP3

From: John C Klensin <[email-masked]>
Date: Wed, 16 May 2012 17:52:24 -0400

Questions for the working group:

(1) In reading the POP3 document, I notice that it is possible
for a client to request and a server to process LANG without
supporting UTF8.  In the interest of simplification of
variations and with the knowledge that very few languages can be
completely supported by ASCII (and English is _not_ one of
them), I suggest we consider requiring UTF8 capability if LANG
is supported/requested.  

The logical alternative is for a server to have to support both
UTF-8 and, e.g., transliterated / ASCII-mapped message forms for
the various languages and, at least IMO, we really don't want to
go there.

UTF8 without LANG makes sense (elementary EAI/ SMTPUTF8 support
for addresses and headers) but I suggest that LANG without UTF8
does not

Note that adopting this change requires no change to the
document other than a well-placed MUST and maybe a one sentence
explanation.

    john

---

From: Barry Leiba <[email-masked]>
Date: Wed, 16 May 2012 21:56:55 -0400

> for addresses and headers) but I suggest that LANG without UTF8

I don't see how they're related.  The server's being able to put
non-English on protocol messages, and having its messages translated
into 17 languages, is entirely unrelated to its ability to handle
UTF-8 in email address fields.

On the other hand, I can't see anyone looking at this spec and saying,
"Well, I can't/won't do the rest of it, but the LANG command looks
interesting.  I think I'll just implement that."

Barry

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 22 May 2012 13:06:37 +0900

> UTF-8 in email address fields.

Well, except that most languages, to be written decently, need non-ASCII 
characters.

> interesting.  I think I'll just implement that."

Agreed. Also, the other way round, I can't see anyone looking at the 
spec and saying: Implementing LANG is way too complex, I'll never get 
that done. A trivial implementation just means that the following two 
things have to work:

       C: LANG Foo
       S: -ERR invalid language Foo

       C: LANG
       S: -ERR Server is unable to list languages

I'm not very familiar with POP3, but maybe this even could be shortened 
to a single uniform reply:

       C: LANG [Foo]
       S: -ERR Server only has a dummy implementation of the LANG command

Implementer's mileage may vary, but I have serious difficulties 
imagining that this would take much work to get implemented.

So given the above, I'd go a step further than John and fold the two 
capabilities together completely.

Regards,   Martin.

---

From: Barry Leiba <[email-masked]>
Date: Tue, 22 May 2012 08:58:57 -0400

> Well, except that most languages, to be written decently, need non-ASCII

So?  What does being able to put non-ASCII characters in the
human-readable messages have to do with being able to handle UTF-8 in
email address fields?

b

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 22 May 2012 15:11:02 +0200

We have two options:

1. The server is permitted to issue "unknown command" on LANG if it does
UTF8 but not LANG.
2. The server is permitted to issue "unimplemented command" on LANG if
it does UTF8 but not LANG.

Pardon my yawn.

Arnt

---

From: John C Klensin <[email-masked]>
Date: Tue, 22 May 2012 12:21:40 -0400

--On Tuesday, May 22, 2012 08:58 -0400 Barry Leiba
<[email-masked]> wrote:

>>>> UTF8 without LANG makes sense (elementary EAI/ SMTPUTF8
>>>> support for addresses and headers) but I suggest that LANG
>>>> without UTF8 does not
>>> 
>>> I don't see how they're related.  The server's being able
>>> to put non-English on protocol messages, and having its
>>> messages translated into 17 languages, is entirely unrelated
>>> to its ability to handle UTF-8 in email address fields.
>> 
>> Well, except that most languages, to be written decently,
>> need non-ASCII characters.

"Most" is a bit of an understatement.  Not even English
qualifies (although many texts in English do).

> human-readable messages have to do with being able to handle
> UTF-8 in email address fields?

Barry, please reread what I wrote:  IMO, UTF-8 without LANG
makes perfectly good sense and email address fields and similar
information is certainly part of the reason.  I was just
objecting to LANG without UTF-8, which I think is a setup for
the use of *and battles over) arbitrary character codings.

I do not believe this is a big enough deal to justify holding up
the document, but the "LANG without..." case strikes me as
unsanitary and in conflict what a lot of what other IETF groups
have been saying about languages and characters sets.  Most of
those discussions would actually argue for "LANG implies UTF-8
with no mechanism for specifying anything else" but, for these
specs, my personal design sense says that "if LANG is specified,
UTF8 MUST be too" fits the model better.  YMMD.

   john

best,
  john

---

From: John C Klensin <[email-masked]>
Date: Tue, 22 May 2012 16:01:27 -0400

--On Tuesday, May 22, 2012 15:11 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> We have two options:
> 
> 1. The server is permitted to issue "unknown command" on LANG
> if it does UTF8 but not LANG.
> 2. The server is permitted to issue "unimplemented command" on
> LANG if it does UTF8 but not LANG.

Because I really hate the "try to guess what character code the
string might be in" case, I pretty much agree.  There is a third
alternative,

	3. LANG implies UTF-8 for anything that might be
	affected by LANG but not otherwise.

But that takes us right back to "UTF-8 for some things but not
others" issue that we rejected on the Jabber chat.

Wrt what message the server should issue if the client issues
LANG but not UTF8, I share your yawn but would prefer that
something be picked.

   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 22 May 2012 22:11:13 +0200

> others" issue that we rejected on the Jabber chat.

We have that anyway. Consider a server which speaks LANG and a client 
which groks EAI. That pair may still deal with a good old MIME message 
containing

   content-type: text/plain; charset=iso-646-10

If you feel strongly about this, I'll go along on the strength of your 
intuition. A sense of design is very often prescient, even if it nothing 
concrete is known to support it at first.

Arnt

PS: I've seen ISO 646 in real mail this millenium.

---

From: "Jiankang YAO" <[email-masked]>
Date: Fri, 25 May 2012 15:47:36 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, May 17, 2012 5:52 AM
Subject: [EAI] LANG, UTF-8, and POP3

> document other than a well-placed MUST and maybe a one sentence

LANG without supporting UTF8 is not good. LANG must have UTF8.
I will try to clarify some words in the new version.

Jiankang Yao

>    john
> 
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Fri, 25 May 2012 04:07:12 -0400

--On Friday, May 25, 2012 15:47 +0800 Jiankang YAO
<[email-masked]> wrote:

>...

Excellent, thanks.   The clarification could be as simple as a
sentence in the LANG description that says "Because of the need
to use non-ASCII characters to represent almost all languages in
a comprehensive way, the LANG option MUST NOT be specified
unless UTF8 is also specified".  But I think that is a matter
for editorial discretion.

Anyone who objects to such a change should speak up, and
explain, immediately.

   john

---

From: [email-masked]
Date: Fri, 25 May 2012 12:06:49 -0700 (PDT)

I have to say I don't really care for this approach, because it effectively
requires a server to implement UTF8 (difficult) in order to implement LANG
(easy).

I'd rather say that the client's use of LANG implicitly authorizes the server
to send replies in UTF-8, no matter what language ends up getting selected (or
not selected).

But if people really prefer the dependency, I can live with it.

				Ned

P.S. Looking at the specification, another question: Why was the approach of
enabling the use of UTF-8 in USER and PASS taken rather than simply requiring
AUTH support in order to suport UTF8? This is a case where coupling makes
considerably more sense.

---

From: John C Klensin <[email-masked]>
Date: Fri, 25 May 2012 15:59:01 -0400

Ned,

Just trying to make sure we are seeing the issues the same way,
even if we still reach different conclusions.

--On Friday, May 25, 2012 12:06 -0700 Ned Freed
<[email-masked]> wrote:

> I have to say I don't really care for this approach, because
> it effectively requires a server to implement UTF8 (difficult)
> in order to implement LANG (easy).

First, ancient history now, but I'd contend that standalone LANG
would/should have been out of scope for EAI.  One can argue that
it has utility with traditional ASCII addresses and headers, but
not a lot... and it should have been a separate proposal, not
handled through EAI.  RFC 5721's title reinforces that view: it
is called "POP3 Support for UTF-8", not, e.g., "Assorted i18n
extensions for POP3".  Given that LANG was included in 5721, I
hope that we can just skip dealing with that issue, but...

Second, while there are several languages for which, with care,
one can construct reply messages in ASCII only (e.g., English;
German if the "ue", "oe", etc., conventions are acceptable; I
suppose Chinese if one were willing to write in Pinyin without
tone marks) the vast majority, both Latin-script-based and
otherwise, require non-ASCII characters and that, these days,
effectively means UTF-8.    One could avoid that problem and
make LANG independent of UTF-8 by specifying a mechanism for
encoded words or some other escape convention in the relevant
strings, but the current spec doesn't discuss that.   In
addition, creating a new case in which escapes or encoded words
are needed sort of violates the EAI goal of creating a migration
path toward a clean, UTF-8-based, target rather than
instantiating more kludges forever.

So I think that clean LANG support pretty much requires handling
of replies in UTF-8.  The current draft doesn't say that, so we
need to do _something_.

> I'd rather say that the client's use of LANG implicitly
> authorizes the server to send replies in UTF-8, no matter what
> language ends up getting selected (or not selected).

That would work too.  It seems to me that the argument for
requiring UTF-8 option support instead is the same as the one
that caused us to take the parameters out of IMAP -- we don't
want to encourage partway implementations and the marginal
utility of LANG without full EAI support is probably small.  If
one disagrees with either of those reasons, then, yes, a LANG
that intrinsically enables UTF-8 replies without having any
effect or whether i18n headers are permitted or not would be
more sensible.

Is that consistent with your analysis?

> P.S. Looking at the specification, another question: Why was
> the approach of enabling the use of UTF-8 in USER and PASS
> taken rather than simply requiring AUTH support in order to
> suport UTF8? This is a case where coupling makes considerably
> more sense.

Randy?  Chris?

    john

---

From: [email-masked]
Date: Fri, 25 May 2012 13:19:00 -0700 (PDT)

> hope that we can just skip dealing with that issue, but...

It's the same sort of argument that was used, back in the day, against MIME:
The WG was chartered to internationalize email content, so what's all this
multimedia crap doing in there?

As you might guess, that past experience doesn't give me a lot of sympathy for
this argument.

> of replies in UTF-8.  The current draft doesn't say that, so we

Absolutely, which is why I made the proposal below.

> Is that consistent with your analysis?

No, not really. It would be one thing if EAI was taking POP3 from
no international capabilities to the full gamut. In that case, or
if there was something equivalent to enhanced status codes in POP3,
the incremental step LANG provides would be small enough not to bother.

But it isn't and there aren't. MIME and SASL already gave POP3
internationalized bodies, header text, usernames, and passwords. So there are a
lot of people out there using POP3 who potentially have a problem dealing with
POP3 errors as the stand, especially given the crappy error code situation. 

It also would be different if UTF8 was a snap to implement. But that's not the
case either. LANG is *far* simpler than implementing UTF8, mostly because of
downgrading.

But again, if people prefer the dependency, I can live with it.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Fri, 25 May 2012 17:28:51 -0400

Ok.  IMO, Ned's position is entirely reasonable.  I don't think
I agree, but the difference is mostly a matter of taste.  Up to
the WG.  Those who have opinions, speak up.  If no one other
than Ned and myself care, I'll either defer to Ned's
implementation experience or ask Joseph to flip a coin.

   john

--On Friday, May 25, 2012 13:19 -0700 Ned Freed
<[email-masked]> wrote:

>> Ned,
> 
>> Just trying to make sure we are seeing the issues the same
>> way, even if we still reach different conclusions.
> 
>> > it effectively requires a server to implement UTF8
>> > (difficult) in order to implement LANG (easy).
> 
>> First, ancient history now, but I'd contend that standalone
>> LANG would/should have been out of scope for EAI.  One can
>> argue that it has utility with traditional ASCII addresses
>> and headers, but not a lot... and it should have been a
>> separate proposal, not handled through EAI.  RFC 5721's title
>> reinforces that view: it is called "POP3 Support for UTF-8",
>> not, e.g., "Assorted i18n extensions for POP3".  Given that
>> LANG was included in 5721, I hope that we can just skip
>> dealing with that issue, but...
> 
> It's the same sort of argument that was used, back in the day,
> against MIME: The WG was chartered to internationalize email
> content, so what's all this multimedia crap doing in there?
> 
> As you might guess, that past experience doesn't give me a lot
> of sympathy for this argument.
> 
>> Second, while there are several languages for which, with
>> care, one can construct reply messages in ASCII only (e.g.,
>> English; German if the "ue", "oe", etc., conventions are
>> acceptable; I suppose Chinese if one were willing to write in
>> Pinyin without tone marks) the vast majority, both
>> Latin-script-based and otherwise, require non-ASCII
>> characters and that, these days, effectively means UTF-8.
>> One could avoid that problem and make LANG independent of
>> UTF-8 by specifying a mechanism for encoded words or some
>> other escape convention in the relevant strings, but the
>> current spec doesn't discuss that.   In addition, creating a
>> new case in which escapes or encoded words are needed sort of
>> violates the EAI goal of creating a migration path toward a
>> clean, UTF-8-based, target rather than instantiating more
>> kludges forever.
> 
>> So I think that clean LANG support pretty much requires
>> handling of replies in UTF-8.  The current draft doesn't say
>> that, so we need to do _something_.
> 
>> > authorizes the server to send replies in UTF-8, no matter
>> > what language ends up getting selected (or not selected).
> 
>> utility of LANG without full EAI support is probably small.
>> If one disagrees with either of those reasons, then, yes, a
>> LANG that intrinsically enables UTF-8 replies without having
>> any effect or whether i18n headers are permitted or not would
>> be more sensible.
> 
> No, not really. It would be one thing if EAI was taking POP3
> from no international capabilities to the full gamut. In that
> case, or if there was something equivalent to enhanced status
> codes in POP3, the incremental step LANG provides would be
> small enough not to bother.
> 
> internationalized bodies, header text, usernames, and
> passwords. So there are a lot of people out there using POP3
> who potentially have a problem dealing with POP3 errors as the
> stand, especially given the crappy error code situation. 
> 
> It also would be different if UTF8 was a snap to implement.
> But that's not the case either. LANG is *far* simpler than
> implementing UTF8, mostly because of downgrading.
>

---

From: Barry Leiba <[email-masked]>
Date: Fri, 25 May 2012 19:30:54 -0400

> implementation experience or ask Joseph to flip a coin.

Well, I've already said that I don't see the reason to tie them
together, but was willing to agree with your (John's) proposal.  Ned
appears to be in the same boat that I am, so we now have two who think
that UTF-8 support in error messages (LANG) does not need to be tied
to UTF-8 support for email addresses.

Not a strong opinion, and I'm happy to accept it either way, but my
preference would be to keep them separate.

Barry

Barry

---

From: John C Klensin <[email-masked]>
Date: Sat, 26 May 2012 02:26:02 -0400

--On Friday, May 25, 2012 19:30 -0400 Barry Leiba
<[email-masked]> wrote:

>> Ok.  IMO, Ned's position is entirely reasonable.  I don't
>> think I agree, but the difference is mostly a matter of
>> taste.  Up to the WG.  Those who have opinions, speak up.
>>  If no one other than Ned and myself care, I'll either defer
>> to Ned's implementation experience or ask Joseph to flip a
>> coin.
> 
> together, but was willing to agree with your (John's)
> proposal.  Ned appears to be in the same boat that I am, so we
> now have two who think that UTF-8 support in error messages
> (LANG) does not need to be tied to UTF-8 support for email
> addresses.
> 
> Not a strong opinion, and I'm happy to accept it either way,
> but my preference would be to keep them separate.

Note that there are two separate issues here:

(1) Is support for UTF-8 (the charset) required if LANG is used,
i.e., if LANG is offered, does the server have the right to
transmit responses in UTF-8 (the charset)?  The current version
of the spec actually does say "yes".  IMO, it does so in a way
that is a bit too hard to find -- my initial quick reading
didn't find it, which is what started this discussion thread and
proposal.  Worse, what it does say is "This and subsequent
protocol-level human-readable text is encoded in the UTF-8
charset" which could be a little over-broad because _all_ POP3
exchanges are "human-readable text", thereby possibly making the
LANG capability a superset of the UTF8 one.  I don't think
anyone intends, but that is what a plausible reading of the text
says.

(2) Given a "yes, if LANG is used the server must be able to
send +OK and -ERR replies un UTF-8 (the charset)" answer, is the
best way to enable that:

(2.1) Make that a property of LANG, such that using the option
authorizes the server to send non-ASCII responses (in UTF-8, the
charset)?   (This is Ned's proposal and probably what the
existing text was intended to say.)

(2.2) Require support for the whole EAI package in order for
LANG to be used, i.e., the LANG capability requires the UTF8
capability.  (This is my original proposal, consistent with what
I thought was the "don't allow implementing pieces and subsets"
conclusion about IMAP parameters on the call.)

If the preference is 2.1, then does anyone want to review the
IMAP parameter question?

For the record, I actually don't have a strong preference
between 2.1 and 2.2.   I just want to be sure that we make a
decision and that whatever decision we make is completely clear
and obvious in the final text.

   john

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 26 May 2012 09:34:23 +0200

Maybe I'm woozy from too much travel now, but I can't tell which question the imap parameter one refers to.

Arnt

---

From: Barry Leiba <[email-masked]>
Date: Sat, 26 May 2012 09:02:25 -0400

Yes, and I prefer 2.1 but would accept 2.2.

b

On Saturday, May 26, 2012, John C Klensin wrote:

> <[email-masked] <javascript:;>> wrote:
>
> >...
> [email-masked] <javascript:;>

---

From: John C Klensin <[email-masked]>
Date: Sat, 26 May 2012 09:26:14 -0400

After thinking about this for a few more hours, I've changed my
mind.  See below.

--On Saturday, May 26, 2012 02:26 -0400 John C Klensin
<[email-masked]> wrote:

> send +OK and -ERR replies un UTF-8 (the charset)" answer, is
> the best way to enable that:
> 
> authorizes the server to send non-ASCII responses (in UTF-8,
> the charset)?   (This is Ned's proposal and probably what the
> capability.  (This is my original proposal, consistent with
> what I thought was the "don't allow implementing pieces and
> subsets" conclusion about IMAP parameters on the call.)
> 

Writing as an individual participant only...

Mixing the principle of making minimal changes at this late date
with Ned's reasoning, I've changed my mind about preferred
outcome.  I propose that we go with 2.1 and that we change the
second sentence of the Section 2 discussion of LANG in 5721bis
to read something like:

	"The LANG capability and command permit a POP3 client to
	negotiate which language the server uses when sending
	human-readable text in replies and changes the character
	set used in those replies from ASCII to Unicode encoded
	in UTF-8."

That correctly explains what the LANG capability and command
actually do up front, rather than burying the UTF-8 part in what
appears to be almost an offhand comment.

Others may still favor 2.2; if so, they should speak up.

best,
   john

---

From: [email-masked]
Date: Sat, 26 May 2012 06:59:08 -0700 (PDT)

> 	set used in those replies from ASCII to Unicode encoded

This wording seems fine to me - it makes it clear where UTF-8 can
now be used.

> appears to be almost an offhand comment.

It was sufficiently offhand that I missed it completely.

				Ned

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 29 May 2012 18:58:58 +0900

I'm fine with 2.1. In case we went for 2.2, I'd again want to propose 
that we'd actually not have a separate LANG capability, because 
implementing the LANG command on the server is, as Ned has said, fairly 
trivial, and even more so if it's only a trivial version (i.e. there's 
only a single language to 'choose' for, which is perfectly okay 
according to how I understand the spec).

Regards,   Martin.

---

From: Tony Hansen <[email-masked]>
Date: Tue, 29 May 2012 08:59:29 -0400

> appears to be almost an offhand comment.

I'm happy with this text.

     Tony

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 30 May 2012 12:32:08 +0100

> I'm happy with this text.

Works for me as well.