Subject: Re: [EAI] LANG, UTF-8, and POP3

From: Shawn Steele <[email-masked]>
Date: Tue, 29 May 2012 23:07:14 +0000

While, technically I agree that LANG shouldn't depend on UTF8, and I think it's not really needed, I'm happy with keeping it.  I'm also happy with requiring UTF8 because, while LANG might be easier, certainly UTF8 is much more important to the entire mail ecosystem and I'd much prefer developers spend time getting UTF8 right.  If that makes one extra app have UTF8 because they wanted LANG, I can live with that :)

So I prefer the 2.2:  Use of LANG requires UTF8 support, because that pushes the message that UTF8 support is important.

-Shawn

---

From: Barry Leiba <[email-masked]>
Date: Tue, 29 May 2012 19:49:39 -0400

> So I prefer the 2.2:  Use of LANG requires UTF8 support, because that pushes
> the message that UTF8 support is important.

Sorry: I have to push back on this rather strongly.
We should not be making protocol decisions because we want to push any
messages about importance.  We need to make protocol decisions based
on what's right for the protocol.

Barry

---

From: [email-masked]
Date: Tue, 29 May 2012 16:52:12 -0700 (PDT)

> on what's right for the protocol.

Although I disagree with Shawn, I wasn't going to comment on it until I
saw this.

Barry, you are of course correct. Pushing decisions about what's important
through protocol decisions is a bad thing, and especially when it results
in a coupling that doesn't need to be there.

				Ned

---

From: Shawn Steele <[email-masked]>
Date: Wed, 30 May 2012 00:58:24 +0000

> through protocol decisions is a bad thing

In that case LANG shouldn't be there.  It's doing the same thing in reverse.  Either LANG's important for UTF-8 mail, or it's not.  But it was argued that at this point it should be left in basically for processes, which seems way worse than for "what's important".

And, although I buy the higher level philosophical point, as devil's advocate isn't every protocol decision a decision about what's important?  Eg: cool but unimportant things are often just left out.  Other things get a MUST because they're the right thing to do, not necessarily because there are technical limitations that force their presence.  For example, 3492 is very permissive, allowing Punycode most anywhere.  However 6530 discourages that practice in EAI.

If LANG is permitted without UTF-8 mail, then there are a bunch of questions about code pages, encodings, etc.  Technically they could probably be solved, or ignored, but it works way better with UTF-8 mail.  If a solution is solved (or ignored) that allows LANG not requiring UTF-8, then I MUST test LANG and no UTF-8 for interoperability, even if I have no intention in supporting the LANG w/o UTF8 case in my client or server.  That protocol decision adds overhead and complexity for every application developer for a scenario that we've all pretty much agreed is a not a desirable case.  That means less time fixing real bugs or adding meaningful extensions to the standard.  So I think it's fair to ask for a baseline of support for new features, in this case supporting UTF8 when supporting LANG.

Making protocol decisions without considering the total cost of those decisions is also a bad thing.

That said, I may have misused the word "important".  LANG opens up the problems of other scripts and stuff, which leads to the 3 options mentioned before:  Require UTF8, restrict it to ASCII, or figure out how to declare/encode stuff in other encodings.  Since our entire set of RFCs is trying to advocate UTF8 over the current mixed up mail encoding story, the 3rd option seems a non-starter.  So requiring UTF8 for LANG seems reasonable as the simplest way to get reliable LANG support.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 May 2012 00:20:00 -0400

Speaking personally only.  I think we have a combination of a
misunderstanding or two and a legitimate philosophical
disagreement here.  Mainly...

--On Wednesday, May 30, 2012 00:58 +0000 Shawn Steele
<[email-masked]> wrote:

>> > We should not be making protocol decisions because we want
>> > to push any  messages about importance.  We need to make
>> > protocol decisions based  on what's right for the protocol.
> 
>> Barry, you are of course correct. Pushing decisions about
>> what's important through protocol decisions is a bad thing
> 
> In that case LANG shouldn't be there.  It's doing the same
> thing in reverse.  Either LANG's important for UTF-8 mail, or
> it's not.  But it was argued that at this point it should be
> left in basically for processes, which seems way worse than
> for "what's important".

I don't think anyone has argued "for processes".  Some,
including myself, have argued that the time to get rid of (or
not have) LANG was long ago and that, if we are going to make
progress, we should not reopen old issues without compelling
need.  That really isn't a process issue but a somewhat
different management one.  That said, there is a strong case for
removing LANG entirely that has nothing to do with importance.
While "ask the server to do what you want" is always a tempting
design issue, good protocol designs are almost always minimal,
with as few extra options or features as possible.  It is also
almost always better to use a canonical form on the wire and let
the systems at each end convert that to local form as necessary
than to convert an "each client needs to convert from the
canonical form to what it needs" problem into one in which each
server has to support all possible forms to make clients happy.

The case is not that straightforward for POP3 and LANG for two
reasons.  One is that POP3's reply structure is not particularly
friendly to a canonical form approach.  It differs in that
respect from systems that return canonical codes that contain
most of the needed information as part of the effort response.
The other is that the list of preferred languages approach
permits any given server to make its own decisions about what
languages to support with, at least in principle, no pressure to
support everything.  Whether those two characteristics add
enough value to overcome what I believe should be a bias against
complexity and _any_ extra commands and capabilities unless they
are clearly necessary, is a question on which the WG needs to
decide (and, I suggest, decided many months ago).

> And, although I buy the higher level philosophical point, as
> devil's advocate isn't every protocol decision a decision
> about what's important?  Eg: cool but unimportant things are
> often just left out.  Other things get a MUST because they're
> the right thing to do, not necessarily because there are
> technical limitations that force their presence.  For example,
> 3492 is very permissive, allowing Punycode most anywhere.
> However 6530 discourages that practice in EAI.

Right.  However...

> If LANG is permitted without UTF-8 mail, then there are a
> bunch of questions about code pages, encodings, etc.
> Technically they could probably be solved, or ignored, but it
> works way better with UTF-8 mail.  If a solution is solved (or
> ignored) that allows LANG not requiring UTF-8, then I MUST
> test LANG and no UTF-8 for interoperability, even if I have no
> intention in supporting the LANG w/o UTF8 case in my client or
> server.  That protocol decision adds overhead and complexity
> for every application developer for a scenario that we've all
> pretty much agreed is a not a desirable case.  That means less
> time fixing real bugs or adding meaningful extensions to the
> standard.  So I think it's fair to ask for a baseline of
> support for new features, in this case supporting UTF8 when
> supporting LANG.

I don't see this problem.  You are making the same assumption
about what LANG specifies that Ned and I did after a few
readings, i.e., that LANG didn't imply/require character set
support at all.  What LANG actually says is that, if it is
enabled, the server gets to send messages associated with the
negotiated language in UTF-8 coding.  So no code page or
encoding issues, just UTF-8.  And your interoperability testing
issue is limited to being sure that a client that enables LANG
can handle receiving UTF-8 responses (presumably not
burdensome).  The server burden is one of keeping different
response tables for each language you choose to support... and
you need not support any at all, or even have such tables, to
conform to the requirements of the spec.

The fact that wasn't crystal-clear is exactly why I proposed new
text and placement to clarify the relationship.

> of those decisions is also a bad thing.

Agreed.

> That said, I may have misused the word "important".  LANG
> opens up the problems of other scripts and stuff, which leads
> to the 3 options mentioned before:  Require UTF8, restrict it
> to ASCII, or figure out how to declare/encode stuff in other
> encodings.  Since our entire set of RFCs is trying to advocate
> UTF8 over the current mixed up mail encoding story, the 3rd
> option seems a non-starter.  So requiring UTF8 for LANG seems
> reasonable as the simplest way to get reliable LANG support.

And that is what the current text says and the clarification
reinforces.

best,
    john

---

From: Shawn Steele <[email-masked]>
Date: Wed, 30 May 2012 05:05:46 +0000

Well, regardless of the methodology, it sounds like we're in violent agreement with the result :)

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Tuesday, May 29, 2012 9:20 PM
To: Shawn Steele; Ned Freed; Barry Leiba
Cc: [email-masked]
Subject: Re: [EAI] LANG, UTF-8, and POP3

Speaking personally only.  I think we have a combination of a misunderstanding or two and a legitimate philosophical disagreement here.  Mainly...

--On Wednesday, May 30, 2012 00:58 +0000 Shawn Steele <[email-masked]> wrote:

>> > We should not be making protocol decisions because we want to push 
>> > any  messages about importance.  We need to make protocol decisions 
>> > based  on what's right for the protocol.
> 
>> Barry, you are of course correct. Pushing decisions about what's 
>> important through protocol decisions is a bad thing
> 
> In that case LANG shouldn't be there.  It's doing the same thing in 
> reverse.  Either LANG's important for UTF-8 mail, or it's not.  But it 
> was argued that at this point it should be left in basically for 
> processes, which seems way worse than for "what's important".

I don't think anyone has argued "for processes".  Some, including myself, have argued that the time to get rid of (or not have) LANG was long ago and that, if we are going to make progress, we should not reopen old issues without compelling need.  That really isn't a process issue but a somewhat different management one.  That said, there is a strong case for removing LANG entirely that has nothing to do with importance.
While "ask the server to do what you want" is always a tempting design issue, good protocol designs are almost always minimal, with as few extra options or features as possible.  It is also almost always better to use a canonical form on the wire and let the systems at each end convert that to local form as necessary than to convert an "each client needs to convert from the canonical form to what it needs" problem into one in which each server has to support all possible forms to make clients happy.

The case is not that straightforward for POP3 and LANG for two reasons.  One is that POP3's reply structure is not particularly friendly to a canonical form approach.  It differs in that respect from systems that return canonical codes that contain most of the needed information as part of the effort response.
The other is that the list of preferred languages approach permits any given server to make its own decisions about what languages to support with, at least in principle, no pressure to support everything.  Whether those two characteristics add enough value to overcome what I believe should be a bias against complexity and _any_ extra commands and capabilities unless they are clearly necessary, is a question on which the WG needs to decide (and, I suggest, decided many months ago).

> And, although I buy the higher level philosophical point, as devil's 
> advocate isn't every protocol decision a decision about what's 
> important?  Eg: cool but unimportant things are often just left out.  
> Other things get a MUST because they're the right thing to do, not 
> necessarily because there are technical limitations that force their 
> presence.  For example,

Right.  However...

> If LANG is permitted without UTF-8 mail, then there are a bunch of 
> questions about code pages, encodings, etc.
> Technically they could probably be solved, or ignored, but it works 
> way better with UTF-8 mail.  If a solution is solved (or
> ignored) that allows LANG not requiring UTF-8, then I MUST test LANG 
> and no UTF-8 for interoperability, even if I have no intention in 
> supporting the LANG w/o UTF8 case in my client or server.  That 
> protocol decision adds overhead and complexity for every application 
> developer for a scenario that we've all pretty much agreed is a not a 
> desirable case.  That means less time fixing real bugs or adding 
> meaningful extensions to the standard.  So I think it's fair to ask 
> for a baseline of support for new features, in this case supporting 
> UTF8 when supporting LANG.

I don't see this problem.  You are making the same assumption about what LANG specifies that Ned and I did after a few readings, i.e., that LANG didn't imply/require character set support at all.  What LANG actually says is that, if it is enabled, the server gets to send messages associated with the negotiated language in UTF-8 coding.  So no code page or encoding issues, just UTF-8.  And your interoperability testing issue is limited to being sure that a client that enables LANG can handle receiving UTF-8 responses (presumably not burdensome).  The server burden is one of keeping different response tables for each language you choose to support... and you need not support any at all, or even have such tables, to conform to the requirements of the spec.

The fact that wasn't crystal-clear is exactly why I proposed new text and placement to clarify the relationship.

Agreed.

> That said, I may have misused the word "important".  LANG opens up the 
> problems of other scripts and stuff, which leads to the 3 options 
> mentioned before:  Require UTF8, restrict it to ASCII, or figure out 
> how to declare/encode stuff in other encodings.  Since our entire set 
> of RFCs is trying to advocate
> UTF8 over the current mixed up mail encoding story, the 3rd option 
> seems a non-starter.  So requiring UTF8 for LANG seems reasonable as 
> the simplest way to get reliable LANG support.

And that is what the current text says and the clarification reinforces.

best,
    john

---

From: [email-masked]
Date: Tue, 29 May 2012 21:50:05 -0700 (PDT)

> In that case LANG shouldn't be there.  It's doing the same thing in reverse. 

It does nothing of the sort.

> Either LANG's important for UTF-8 mail, or it's not.

You're once again falling into the trap Barry warned about. This is about
creating standards with the greatest possible utility, not about trying to
assess importance, or worse, trying to use standards language to force a notion
of what we happen to think is important down people's throats.

Given the limitations of POP error codes, by providing the ability to select a
language and return responses in UTF-8, LANG provides a better client
internationalization experience. This improvement is actually largely
independent of whether the UTF8 capability is used to provide support for
downloading EAI messages, especially since email already has significant
internationalization facilities.

This argues strongly for it to be possible to implement the LANG capability
without also requiring the ability to use the UTF8 capability.

> But it was argued that at this point it should be left in basically for

That certainly isn't the point I've been making. This has essentially nothing
to do with process and everything to do with the utility of the outcome.

> And, although I buy the higher level philosophical point, as devil's advocate
> isn't every protocol decision a decision about what's important?

Nope. It's about utility. Very different.

> Eg: cool but unimportant things are often just left out.  Other things get a
> MUST because they're the right thing to do, not necessarily because there are
> technical limitations that force their presence.  For example, 3492 is very
> permissive, allowing Punycode most anywhere.  However 6530 discourages that
> practice in EAI.

That's the result of a technical assessment. (It happens to be one I actually
disagree with, but I accept that it's the consensus of the group.)

> If LANG is permitted without UTF-8 mail, then there are a bunch of questions
> about code pages, encodings, etc.

Nobody is saying that LANG should provide the ability to produce responses in
any charset other than UTF-8. There's only one code page and one encoding on
the table.

You appear to be confusing the UTF8 *capability*, which provides the
ability to download EAI messages in POP, with the UTF-8 *charset*. When
people talk about LANG requiring UTF8, they are talking about
coupling the capabilities with each other. They are most definitely not
talking about allowing LANG to return responses in charsets other than UTF-8.
That's completely off in left field.

> Technically they could probably be solved, or ignored, but it works way better
> with UTF-8 mail.  If a solution is solved (or ignored) that allows LANG not
> requiring UTF-8,

See above. This is effectively a strawman. All we are saying is that servers
should be allowed to support the LANG *capability* without also requiring
support for the UTF8 *capability*.

> then I MUST test LANG and no UTF-8 for interoperability, even if I have no
> intention in supporting the LANG w/o UTF8 case in my client or server.

Again, this is not about being able to use other charsets with LANG. As for the
UTF8 capability, whether or not we require servers to support it if they
support LANG has no impact on your test cases, for several reasons:

(1) Nothing about server-side requirements has any impact on what clients may 
    do - a client may choose to activate neither capability, LANG by itself,
    UTF8 by itself, or both. So if you implement both, your test cases have
    to cover all of the combinations in order to account for all possible
    client behavior.

(2) If as you say, your client is going to require both capabilities, then the
    only server it is going to work with is one that offers both extensions,
    and that's the only case you need to test. This means you may not work
    with some servers, but that's your choice as an implementor to make.

(3) The only place these capabilities actually overlap would be if there are
    internationalized error responses specific to UTF8 capability. That's
    not much overlap, so even if you decide to support one without the other
    in your client, it's not a significant test burden since the tests are
    mostly indepedent of each other.

> That protocol decision adds overhead and complexity for every application
> developer for a scenario that we've all pretty much agreed is a not a desirable case. 

It does no such thing. See above.

> That means less time fixing real bugs or adding meaningful extensions to the
> standard.  So I think it's fair to ask for a baseline of support for new
> features, in this case supporting UTF8 when supporting LANG.

I'm afraid as a result of an apparent misunderstanding of the discussion, your
analysis is flawed and therefore your conclusions are erroneous.

> problems of other scripts and stuff, which leads to the 3 options mentioned
> before:  Require UTF8, restrict it to ASCII, or figure out how to
> declare/encode stuff in other encodings.

> Since our entire set of RFCs is trying to advocate UTF8 over the current mixed
> up mail encoding story, the 3rd option seems a non-starter.  So requiring UTF8
> for LANG seems reasonable as the simplest way to get reliable LANG support.

Again, since nobody is proposing anything like that, this amounts to nothing
but a strawman.

				Ned

---

From: Shawn Steele <[email-masked]>
Date: Wed, 30 May 2012 07:52:01 +0000

> This is about creating standards with the greatest possible utility, not about trying to
> assess importance, or worse, trying to use standards language to force a notion of 
> what we happen to think is important down people's throats.

Features is not necessarily == utility.  Standards, by necessity, "force what the authors happen to think is important down people's throats."  No matter how excellent the standard, people can be found that disagree with that standard, either in large parts or small parts.

Standards are limited to what the authors think is important.  (If they didn't think it was important, they wouldn't waste time including it in the standard).  Hopefully there is consensus and a great process driving those decisions.  I think we've done a decent job.

For example, if we were striving for "greatest possible utility", I'd suggest that a more flexible approach would have been to not create "SMTPUTF8", but rather "SMTPCP", with a mechanism for selecting from an unlimited set of code pages.  We all decided that UTF8 and consistency was more important for EAI than allowing people to pick their favorite code page.  We sacrificed potential utility for the uniformity we perceived was more important.  Someone stuck maintaining a system with some awkward code page may well have preferred if we'd come to a different conclusion.

Standards, IMO, are around to create interoperable systems.  Many permitted variations make the systems more complex and hinder the interoperability.  So, IMO, it's completely fair to require UTF8 for LANG in order to reduce the complexity of the system.  I do not think that allowing additional code pages, particularly since it would introduce ambiguity or some complex declaration system, would provide more "utility".  In fact I think it'd be less useful than a LANG that was unambiguous in its encoding.

But I'd like to point out that it shouldn't be "forcing what the authors think is important down people's throats".  It should be what the WG has decided is realistic and reasonable for the entire ecosystem and the long term goals of the standard and related standards.  IMO EAI cannot succeed without widespread UTF8 adoption, so adding things to an EAI standard that encourage deviating from UTF8 could only delay EAI adoption, conflicting with the core goal of EAI adoption that this WG is trying to achieve.

Anyway, this is a philosophical discussion is sort of orthogonal to what we're doing here.  I'm fine with what John's been proposing, though there seems to be some disagreement since John just said "That is what the current text says" and Ned said "nobody is proposing anything like that" :)

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 May 2012 06:50:29 -0400

> violent agreement with the result :)

At this stage, I think that is what is important.

   john

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 May 2012 07:51:07 -0400

Shawn,

Two added comments, just because the design philosophy
discussion is interesting (at least to me).  Again, speaking
personally only.

--On Wednesday, May 30, 2012 07:52 +0000 Shawn Steele
<[email-masked]> wrote:

>...
> For example, if we were striving for "greatest possible
> utility", I'd suggest that a more flexible approach would have
> been to not create "SMTPUTF8", but rather "SMTPCP", with a
> mechanism for selecting from an unlimited set of code pages.
> We all decided that UTF8 and consistency was more important
> for EAI than allowing people to pick their favorite code page.
> We sacrificed potential utility for the uniformity we
> perceived was more important.

While you may see that in terms of importance, I see it as
maximizing interoperability by permitting only one form on the
wire.  One would see greater utility only if either all servers
or all clients supported all possible code pages, which is a
near-impossibility.   If not, an unlimited number of code pages
approach would require every client to support both its
preferred code page(s) and one or more that it was pretty sure
everyone would support so it could convert.  (As
standards-writers, we could have made that choice easier by
making something mandatory to implement; that something would
probably be UTF-8).  The result would be that a client
implementer would need to support its preferred code page(s),
UTF-8, and conversion between UTF-8 and one or more of those
code pages.  That is not either improved functionality nor
reduced implementation complexity.  It enables a performance
optimization for clients that run on operating systems that
prefer non-Unicode internally, but, given that these reply
strings tend to be very short, that optimization is trivial.  Of
course, if there is no mandatory to implement code page (other
than ASCII) on the server, and both server and client optimize
flexibility by supporting only their own code pages,
"flexibility" would translate into an interoperability horror.

For context, I believe we would not have adopted the multiple
code page/encoding "charset" approach with text/plain had it
been clear that Unicode and UTF-8 would be available and
successful.  That was not at all obvious in 1992.  Even if it
had been, UTF-8 support wasn't even near widely deployed and
supported enough to make trying to force its use (even on the
wire) plausible.  (I note that this wasn't long after the
emerging HTML community effectively decided that, if they used
8859-1, they would be finished with i18n problems -- we at least
knew that was a non-starter.)

> Someone stuck maintaining a
> system with some awkward code page may well have preferred if
> we'd come to a different conclusion.

Maybe.  But, if they wanted to interoperate with others, I don't
know how strong the preference would have been given that they
would have had to support mapping to and from UTF-8 for that
code page anyway.

> Standards, IMO, are around to create interoperable systems.
> Many permitted variations make the systems more complex and
> hinder the interoperability.  So, IMO, it's completely fair to
> require UTF8 for LANG in order to reduce the complexity of the
> system.  I do not think that allowing additional code pages,
> particularly since it would introduce ambiguity or some
> complex declaration system, would provide more "utility".  In
> fact I think it'd be less useful than a LANG that was
> unambiguous in its encoding.

Exactly.

> But I'd like to point out that it shouldn't be "forcing what
> the authors think is important down people's throats".  It
> should be what the WG has decided is realistic and reasonable
> for the entire ecosystem and the long term goals of the
> standard and related standards.  IMO EAI cannot succeed
> without widespread UTF8 adoption, so adding things to an EAI
> standard that encourage deviating from UTF8 could only delay
> EAI adoption, conflicting with the core goal of EAI adoption
> that this WG is trying to achieve.

Again, no one opposed LANG with arbitrary code pages.  My
initial readings of the spec was that it didn't say anything at
all on the subject, which meant that we either needed to specify
that it used UTF-8 coding on replies or to descend into
"ambiguity or some complex declaration system".  This discussion
has been entirely about exactly how the "if you are going to
enable LANG, replies MUST be in UTF-8" condition was going to be
specified.
 
> Anyway, this is a philosophical discussion is sort of
> orthogonal to what we're doing here.  I'm fine with what
> John's been proposing, though there seems to be some
> disagreement since John just said "That is what the current
> text says" and Ned said "nobody is proposing anything like
> that" :)

I actually don't think so.  A really careful reading of the text
actually turns up the requirement for replies in UTF-8 so my
suggested modified text clarifies the situation but doesn't
actually change anything.  After another look at I-D this
morning, I believe that we should also change the abstract and
possibly even the title, to clarify that two separate and
orthogonal capabilities are being specified.  But even if the WG
agreed to those changes, they would be clarifications to text to
reduce confusion and increase the odds of correct
implementations, not alterations to the spec that would affect
any correct implementation.

   john

---

From: [email-masked]
Date: Wed, 30 May 2012 07:17:03 -0700 (PDT)

> Features is not necessarily == utility.  Standards, by necessity, "force what
> the authors happen to think is important down people's throats."  No matter how
> excellent the standard, people can be found that disagree with that standard,
> either in large parts or small parts.

Of course features are not equivalent to utility. That's exactly the point of
this discussion. You're arguing for what amounts to a feature - coupling of
LANG to UTF8. I'm arguing that the feature has considerable negative utility,
not having it has negligable cost, and therefore it has no business being
there.

> Standards are limited to what the authors think is important.  (If they
> didn't think it was important, they wouldn't waste time including it in the
> standard).  Hopefully there is consensus and a great process driving those
> decisions.  I think we've done a decent job.

I actually disagee, but that's irrelevant at this point.

> For example, if we were striving for "greatest possible utility", I'd suggest
> that a more flexible approach would have been to not create "SMTPUTF8", but
> rather "SMTPCP", with a mechanism for selecting from an unlimited set of code
> pages.

Sorry, that's just flat out incorrect. There may have been a time when being
able to use multiple charsets resulted in an overall increase in utility
because of lack of widespread support for UTF-8, but that time is long since
past. It's especially true given the total hashup that's been made of various
CJK charsets.

And there's no way you can demand that servers be able to supply all possible
code pages. If you made that a requirement, it would either be ignored or
nobody would implement the extension. That's just reality.

> We all decided that UTF8 and consistency was more important for EAI than
> allowing people to pick their favorite code page.

No, we decided that having something that was implementable was more
important than having something that was not.

> We sacrificed potential utility for the uniformity we perceived was more
> important.  Someone stuck maintaining a system with some awkward code page may
> well have preferred if we'd come to a different conclusion.

Nope. You're comparing something that's implementable with something that is
not. There is no utility in something that will never see the light of day.

> Standards, IMO, are around to create interoperable systems.  Many permitted
> variations make the systems more complex and hinder the interoperability.  So,
> IMO, it's completely fair to require UTF8 for LANG in order to reduce the
> complexity of the system. 

You have not provided the slightest scintilla of evidence that separating the
two actually reduces significant added implementation complexity. I, OTOH, have
analyzed the situation fairly carefully and shown that the added complexity is
somewhere between nonexistent to negligable.

> I do not think that allowing additional code pages, particularly since it
> would introduce ambiguity or some complex declaration system, would provide
> more "utility".

And once again you're arguing a strawman. For the last time, nobody is arguing
for requiring support of additional code pages. That's not part of any
proposal anyone has made. Read the proposed text if you don't believe me.

> In fact I think it'd be less useful than a LANG that was unambiguous in its
> encoding.

Please explain why there's anything ambiguous about requiring the use of UTF-8.

> But I'd like to point out that it shouldn't be "forcing what the authors
> think is important down people's throats".  It should be what the WG has
> decided is realistic and reasonable for the entire ecosystem and the long term
> goals of the standard and related standards.  IMO EAI cannot succeed without
> widespread UTF8 adoption, so adding things to an EAI standard that encourage
> deviating from UTF8 could only delay EAI adoption, conflicting with the core
> goal of EAI adoption that this WG is trying to achieve.

You need to stop piling up the strawmen. There is nothing about allowing
LANG to be used separately from UTF8 that will discourage use of the UTF8
capability.

> Anyway, this is a philosophical discussion is sort of orthogonal to what
> we're doing here.  I'm fine with what John's been proposing, though there seems
> to be some disagreement since John just said "That is what the current text
> says" and Ned said "nobody is proposing anything like that" :)

John has apparently missed all your claims about how the revised LANG proposal
introduces the need to support additional code pages.

				Ned

---

From: Shawn Steele <[email-masked]>
Date: Wed, 30 May 2012 16:40:06 +0000

> Sorry, that's just flat out incorrect. There may have been a time when 
> being able to use multiple charsets resulted in an overall increase in utility
> because of lack of widespread support for UTF-8, but that time is long 
> since past. It's especially true given the total hashup that's been made of
> various CJK charsets.

I was being facetious.   John said it wasn't "importance" but "maximizing interoperability" that led to this decision.  I think that it is important that interoperability be maximized, so that seems like picking nits to me.

And now I'm really confused, if code pages are so evil, why not require LANG to respond in UTF8?  (Which is what I understand John is trying to agree on language for.)  The WG seems agreed that there's about 1 language that ASCII is sufficient for, so it seems like having localized responses would immediately require a non-ASCII response.

IMO if LANG replies MUST be in UTF-8, then you're a large part of the way towards supporting UTF8 already?  Like, how can you reply in UTF8 if you haven't negotiated a UTF8 connection?  If I use LANG w/o using UTF8, then how does that work?  That's where I'm a bit confused.

-Shawn

---

From: [email-masked]
Date: Wed, 30 May 2012 09:52:40 -0700 (PDT)

> And now I'm really confused, if code pages are so evil, why not require LANG
> to respond in UTF8?

I dislike shouting, but it seems to be in order here:

THAT'S EXACTLY WHAT IS BEING PROPOSED. ABSOLUTELY NOBODY IS PROPOSING ALLOWING
ANYTHING OTHER THAN UTF-8 WHEN LANG IS USED. YOU ARE MAKING UP ISSUES WHERE
NONE EXIST.

*Please* read the text. I defy you to find even the slightest mention of
allowing any other charset, any other code page, any other encoding.

LANG does two things:

(1) It allows the server to respond in utf-8 instead being restricted to ASCII.
(2) It allows the client to request responses in different language, on the
    condition that the server is prepared to respond in that language.

That's it. There is nothing else there. When we talk about the coupling of LANG
with UTF8 (note the spelling - that's *not* a charset), we're talking about
there being a connection between suporting LANG and supporting the UTF8
capability, which enables client access to EAI messages without downgrading.

> (Which is what I understand John is trying to agree on language for.)

And your understanding is simply incorrect.

> The WG seems agreed that there's about 1 language that ASCII is sufficient
> for, so it seems like having localized responses would immediately
> require a non-ASCII response.

Which is why LANG is useful.

> IMO if LANG replies MUST be in UTF-8, then you're a large part of the way
>  towards supporting UTF8 already?

No you are not. If you believe that, you need to look at the various
downgrade specifications a *lot* more carefully.

> Like, how can you reply in UTF8 if you haven't negotiated a UTF8 connection? 
> If I use LANG w/o using UTF8, then how does that work?  That's where I'm a bit
> confused.

See above. These are separate things.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 30 May 2012 14:11:11 -0400

--On Wednesday, May 30, 2012 16:40 +0000 Shawn Steele
<[email-masked]> wrote:

> And now I'm really confused, if code pages are so evil, why
> not require LANG to respond in UTF8?  (Which is what I
> understand John is trying to agree on language for.)  The WG
> seems agreed that there's about 1 language that ASCII is
> sufficient for, so it seems like having localized responses
> would immediately require a non-ASCII response.

Please read the spec.  In particular, the introduction to
draft-ietf-eai-rfc5721bis-04 says (second paragraph of Section
1): 

	"a mechanism to support UTF-8 characters in protocol
	level response strings as well as the ability to
	negotiate a language for such response strings."

and, in the third paragraph of the "Discussion" in Section 2:

	"This and subsequent protocol-level human-readable text
	is encoded in the UTF-8 charset."

That is really fairly clear and certainly doesn't suggest
returning anything but UTF-8 strings.  It has, IMO, only one
problem, which is that the two pieces of text are a little bit
buried and sound narrative rather than normative.   My proposed
fixes -- the modification to the actual description of LANG and
the suggestion that the WG consider changes to the title and/or
abstract -- simply clarify and highlight those statements; they
do not change the protocol spec in any way.
 
> IMO if LANG replies MUST be in UTF-8, then you're a large part
> of the way towards supporting UTF8 already?  Like, how can you
> reply in UTF8 if you haven't negotiated a UTF8 connection?  If
> I use LANG w/o using UTF8, then how does that work?  That's
> where I'm a bit confused.

Sending the LANG command and having it accepted constitutes
negotiating a a connection over which UTF-8 (the charset, not
the other Capability in this spec) will be sent.  See above.
Given that, I'm not sure what you are talking about, so one of
us is probably _very_ confused.  I don't think it is me (or Ned
or Barry).

best,
    john

---

From: Shawn Steele <[email-masked]>
Date: Wed, 30 May 2012 18:59:45 +0000

I'm happy with it, thanks :)

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Wednesday, May 30, 2012 11:11 AM
To: Shawn Steele; Ned Freed
Cc: Barry Leiba; [email-masked]
Subject: Re: [EAI] LANG, UTF-8, and POP3

--On Wednesday, May 30, 2012 16:40 +0000 Shawn Steele <[email-masked]> wrote:

> And now I'm really confused, if code pages are so evil, why not 
> require LANG to respond in UTF8?  (Which is what I understand John is 
> trying to agree on language for.)  The WG seems agreed that there's 
> about 1 language that ASCII is sufficient for, so it seems like having 
> localized responses would immediately require a non-ASCII response.

That is really fairly clear and certainly doesn't suggest returning anything but UTF-8 strings.  It has, IMO, only one problem, which is that the two pieces of text are a little bit
buried and sound narrative rather than normative.   My proposed
fixes -- the modification to the actual description of LANG and the suggestion that the WG consider changes to the title and/or abstract -- simply clarify and highlight those statements; they do not change the protocol spec in any way.
 
> IMO if LANG replies MUST be in UTF-8, then you're a large part of the 
> way towards supporting UTF8 already?  Like, how can you reply in UTF8 
> if you haven't negotiated a UTF8 connection?  If I use LANG w/o using 
> UTF8, then how does that work?  That's where I'm a bit confused.

Sending the LANG command and having it accepted constitutes negotiating a a connection over which UTF-8 (the charset, not the other Capability in this spec) will be sent.  See above.
Given that, I'm not sure what you are talking about, so one of us is probably _very_ confused.  I don't think it is me (or Ned or Barry).

best,
    john