Subject: [EAI] [Technical Errata Reported] RFC6531 (4996)

From: RFC Errata System <[email-masked]>
Date: Sun, 16 Apr 2017 13:30:54 -0700 (PDT)

The following errata report has been submitted for RFC6531,
"SMTP Extension for Internationalized Email".

--------------------------------------
You may review the report below and at:
http://www.rfc-editor.org/errata_search.php?rfc=6531&eid=4996

--------------------------------------
Type: Technical
Reported by: Vitaliy V. Tokarev <[email-masked]>

Section: 3.3

Original Text
-------------
The definition of <atext> is extended to permit both the RFC 5321
definition and a UTF-8 string. That string MUST NOT contain any
of the ASCII graphics or control characters.

Corrected Text
--------------
The definition of <atext> is extended to permit both the RFC 5321
definition and a UTF-8 string. That string MUST NOT contain any
of the Extended ASCII graphics (%d128-255) or control characters.

Notes
-----
The question is: what is "ASCII graphics characters"? Either meant that is possible to transmit 8bit characters, but they should not be in range %d128-255. Or something else. A better clarification is appreciated.

Instructions:
-------------
This erratum is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party  
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC6531 (draft-ietf-eai-rfc5336bis-16)
--------------------------------------
Title               : SMTP Extension for Internationalized Email
Publication Date    : February 2012
Author(s)           : J. Yao, W. Mao
Category            : PROPOSED STANDARD
Source              : Email Address Internationalization
Area                : Applications
Stream              : IETF
Verifying Party     : IESG

---

From: [email-masked]
Date: Sun, 16 Apr 2017 17:07:45 -0700 (PDT)

First and foremost, the suggested change in this errata isn't quite right. As
the errata itself notes, ASCII by definition specifies characters in the range
%d0-%d127, and does not include characters in the range %d128-255, so the
suggested text doesn't work.

Second, I note that when you want to know how something is defined, the place
to look is the formal definition. And the good news is the formal definition of
the modified atext in RFC 6531 is clear and correct:

  atext   =/  UTF8-non-ascii
    ; extend the implicit definition of atext in
    ; RFC 5321, Section 4.1.2, which ultimately points to
    ; the actual definition in RFC 5322, Section 3.2.3

That said, there are multiple legitimate issue with the bullet point in
question.

The first sentence of the bullet point does not match the formal specification.
In particular, it fails to say that it's the non-ASCII subset of UTF-8 that's
being added to atext. So the current text needs to be changed from:

      The definition of <atext> is extended to permit both the RFC 5321
      definition and a UTF-8 string. 

to something like:

      The definition of <atext> is extended to permit both the RFC 5321
      character repetiore and the non-ASCII subset of UTF-8.

I think this qualifies as a legitimate technical correction.

The second sentence also has issues:

      That string MUST NOT contain any of the ASCII graphics or control
      characters.

It's not clear to me why repeating the prohibition on the parts of ASCII not
allowed in atext is necessary or even desireable, especially if the first
sentence is clarified.

But even if you accept that the prohibition is useful, there are problems.

The term "ASCII control character" is well defined as meaning the range
%d0-%d31, and that range was and is forbidden for use in atext.

The term "ASCII graphic character", OTOH, is not well defined. It's sometimes
used as a synonym for "ASCII printable character", in other cases it refers
to characters in an "extendened ASCII" that fall in the range  %d127-%d255.

So this basically a mess. My suggestion would be to change it to read
something like:

     This does not add any additional ASCII characters to the atext repetoire;
     The string MUST NOT contain ASCII control characters or RFC 5321 specials.

I believe this also qualifies as legitimate technical correction.

				Ned

> The question is: what is "ASCII graphics characters"? Either meant that is
> possible to transmit 8bit characters, but they should not be in range
> %d128-255. Or something else. A better clarification is appreciated.

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: "Jiankang Yao" <[email-masked]>
Date: Tue, 18 Apr 2017 14:27:42 +0800

Hello all,

   Thanks a lot for Vitaliy V. Tokarev's suggested possible Errata based on the sentence below
"The definition of <atext> is extended to permit both the RFC 5321
definition and a UTF-8 string. That string MUST NOT contain any
of the ASCII graphics or control characters."

My personal feeling is that it might not be an Errata.

The term "UTF-8 string " is used everywhere in EAI RFCs, for examples

1)In section 7.1 RFC6530, which said   " Permits the use of UTF-8 strings in email addresses, both local
      parts and domain names."
2)In section 3.2  RFC6531, which said  "   An SMTP server that announces the SMTPUTF8 extension MUST be prepared
   to accept a UTF-8 string [RFC3629] in any position in which RFC 5321
   specifies that a <mailbox> can appear."

So we have this sentence "The definition of <atext> is extended to permit both the RFC 5321
definition and a UTF-8 string. "
 It means that <atext> allows both RFC 5321 definition and a UTF-8 string.
Since UTF-8 string includes both non-ASCII UTF-8 and ASCII UTF-8, some readers may mis-understand that <atext> can allow some  ASCII graphics or control characters. 
In order to emphasize that  <atext> does not allow any
of the ASCII graphics or control characters, we add this sentence "That string MUST NOT contain any
      of the ASCII graphics or control characters."

Because  "UTF-8 string " is used everywhere in EAI RFCs, in order to avoid the possible misunderstanding,  I remembered that the WG  suggested to add this sentence " That string MUST NOT contain any
of the ASCII graphics or control characters." when discussing this issue.

Since the formal <atext> syntax is clearly defined, I think that this sentence
"The definition of <atext> is extended to permit both the RFC 5321
definition and a UTF-8 string. That string MUST NOT contain any
of the ASCII graphics or control characters." will not cause any problems, and it will help the readers to understand the formal <atext> syntax and avoid the possible misunderstanding.

I hope that this clarification is clear.

Thanks a lot.

Jiankang Yao

From: RFC Errata System
Date: 2017-04-17 04:30
To: yaojk; maowei_ietf; ben; alissa; aamelnikov; john-ietf; jyee
CC: vitaliy.tokarev; ima; rfc-editor
Subject: [Technical Errata Reported] RFC6531 (4996)
The following errata report has been submitted for RFC6531,
"SMTP Extension for Internationalized Email".

Section: 3.3

Original Text
-------------

Corrected Text
--------------

Notes
-----

Instructions:
-------------

---

From: Vitaliy T <[email-masked]>
Date: Tue, 18 Apr 2017 18:01:06 +0300

Hello,

> My personal feeling is that it might not be an Errata.

That's OK. I did not intended to be correct. Basically this was just a question.

> Since UTF-8 string includes both non-ASCII UTF-8 and ASCII UTF-8, some
> readers may mis-understand that <atext> can allow some  ASCII graphics or
> control characters.
> Because  "UTF-8 string " is used everywhere in EAI RFCs, in order to avoid
> the possible misunderstanding,  I remembered that the WG  suggested to add
> this sentence " That string MUST NOT contain any

Sorry, but there is no clear definition what the "ASCII graphics
[characters]" is.

> of the ASCII graphics or control characters." will not cause any problems,
> and it will help the readers to understand the formal <atext> syntax and
> avoid the possible misunderstanding.

So, that means there is enough the current ABNF definition:

  atext =/ UTF8-non-ascii

I suggest either to clarify what ASCII graphics characters are or just
remove that sentence. Without such a clarification the readers must
hope that they understood you right and do the next things:

1) accepts only valid UTF-8 strings (there is no a single octet
character in range %d80-FF, anything that starts from %d80 is a
multi-octet character);
2) accepts printable characters within <atext> definition as described
in [RFC5321] and [RFC5322]. E.g. accepts ASCII characters "!", "#",
"^", "{", "}", etc.;
3) do NOT add new or own rules to existing ones.

As Ned Freed said the definition of ASCII graphics is not well
defined. For instance, someone may look at

  pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html

and will see that graphic characters are crossing with the definition
of <atext> [RFC5322]. Someone else may think about Extended ASCII
graphic characters (still they are not permitted in UTF-8). And others
will ignore that sentence, because [RFC5322] clearly states what ASCII
characters are permitted within <atext>.

In the end, thank you very much for your responses. In my case, I
simply ignored that sentence and used only the new ABNF rules (still
keeping hope that I do it right). I think it is possible to keep the
current text of [RFC6531] as is, but public this Errata for those who
will be confused.

-- 
With Best Regards,
Vitaliy V. Tokarev

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Apr 2017 12:57:43 -0400

> That's OK. I did not intended to be correct. Basically this

If it is a question, then use of the errata system to report it
is entirely inappropriate.  RFC Editor and IESG, please mark the
proposed erratum as "rejected" for this reason and the more
substantive ones given on this list.

Vitaliy, 

I need to apologize for sounding annoyed, but the errata process
is very heavy-duty and costs a lot of people (including me in my
former WG co-chair capacity) a lot of time, time that is largely
wasted in this case.  If you have a question, please raise it on
the mailing list, even if that question is "is this a mistake in
the spec or do I just not understand the terminology or what is
going on".   If the answer is that it is an error, the errata
process works much better because you will have already
established some signs of consensus on the issues.

That said, see below.

>>  It means that <atext> allows both RFC 5321 definition and a
>>  UTF-8 string. Since UTF-8 string includes both non-ASCII
>> UTF-8 and ASCII UTF-8, some readers may mis-understand that
>> <atext> can allow some  ASCII graphics or control characters.
>> of the ASCII graphics or control characters, we add this
>> sentence "That string MUST NOT contain any
>> Because  "UTF-8 string " is used everywhere in EAI RFCs, in
>> order to avoid the possible misunderstanding,  I remembered
>> that the WG  suggested to add this sentence " That string
>> MUST NOT contain any

> Sorry, but there is no clear definition what the "ASCII
> graphics [characters]" is.

Sure there is.   It is in the ASCII Standard, cited as [ASCII]
in the document.  If that is inconvenient to find, see SeIf
nothing else, see Section 4.2 of RFC 20.

>...
> I suggest either to clarify what ASCII graphics characters are
> or just remove that sentence. Without such a clarification the
> readers must hope that they understood you right and do the
> next things:

While I'm not a huge fan of the way 6531 was written, it was
good enough, and clear enough, to achieve WG and IETF consensus.
Let me suggest that you are having a slightly different problem
from the one you describe.  6531 (and the other SMTPUTF8 core
documents) make a number of assumptions.  They all depend on
6530 for definitions and context and are quite clear about that.
In addition, they are add-ons to other specs for basic email
specs and, to some degree at least, for basic understanding of
internationalization and character coding issues.  In
retrospect, 6530 should probably have been a bit more explicit
about the latter, but it seemed obvious.   If one reads the
SMTPUTF8 documents in isolation, without that context and
knowledge, they probably say a lot of confusing or incomplete
things.  

That might not have been the right set of decisions as to where
these documents stopped (although I think it probably was), but
they were definitely consensus decisions, make after a good deal
of discussion.

> defined. For instance, someone may look at

Ned and I disagree.   See above.   Probably 6531 (and 6530)
should have referenced RFC 20 in addition to the base ASCII
spec, but there was a lot of controversy in the community about
that at the time.

> and will see that graphic characters are crossing with the
> definition of <atext> [RFC5322]. Someone else may think about
> Extended ASCII graphic characters (still they are not
> permitted in UTF-8).

I don't understand that statement.  UTF-8 is simply a coding of
Unicode that includes all valid Unicode code points.  See RFC
3629 or the Unicode Standard.

>   And others will ignore that sentence,
> because [RFC5322] clearly states what ASCII characters are
> permitted within <atext>.
> 
> In the end, thank you very much for your responses. In my
> case, I simply ignored that sentence and used only the new
> ABNF rules (still keeping hope that I do it right). I think it
> is possible to keep the current text of [RFC6531] as is, but
> public this Errata for those who will be confused.

See above.  I think the authors (again, both of 6531 and
6530)should be happy to accept an erratum that asks for
additional clarification or references, but I'm unconvinced that
you have found an error, or even a major omission, in the text.

And, my disturbance at the use of the errata mechanism
notwithstanding, thanks for bringing this up -- if there are
parts of the specs that aren't clear to a reasonable reader, we
need to know about it.

best,
    john

---

From: Vitaliy T <[email-masked]>
Date: Tue, 18 Apr 2017 20:47:43 +0300

> substantive ones given on this list.

It is a question which should be identified as Errata. IMHO.

> established some signs of consensus on the issues.

Yes, I understand. Your time is NOT wasted, be sure. You helped me and
probably you may help others (like me).

> nothing else, see Section 4.2 of RFC 20.

You have rise another issue on the topic. Neither [RFC6530] or
[RFC6531] have a reference to [RFC20]. And a reference to it leads to
the question in this errata.

> spec, but there was a lot of controversy in the community about

If [RFC6531] will have a reference to [RFC20] then the sentence from
[RFC6531] even more confusing. Let's see what we have.

1. As [RFC20] states in section "4.2", for instance, the character "+"
is considered as a graphic character.

2. In [RFC6531] has been said in section "3.3":

3. In [RFC5322] has been said in section "3.2.3"

  atext = ALPHA / DIGIT / ; Printable US-ASCII
      "!" / "#" / ; characters not including
       "$" / "%" / ; specials. Used for atoms.
       "&" / "’" /
       "*" / "+" /
       "-" / "/" /
       "=" / "?" /
       "^" / "_" /
       "‘" / "{" /
       "|" / "}" /
       "~"

4. And later in [RFC6531] has been said:

   atext =/ UTF8-non-ascii

What statement is truth? May be a correction should look like this:

  atext = ALPHA / DIGIT / UTF8-non-ascii

> you have found an error, or even a major omission, in the text.

If you see that there is no mistake, then we can close the question.

Sorry, no offends, perhaps, I am wrong. Thanks for your time.

-- 
With Best Regards,
Vitaliy V. Tokarev

---

From: [email-masked]
Date: Tue, 18 Apr 2017 10:22:30 -0700 (PDT)

> nothing else, see Section 4.2 of RFC 20.

John, the problem is that by that definition, the sentence shifts from being
confusing to flat-out incorrect. The definition of atext is RFC 5322 is:

                       "&" / "'" /
                       "*" / "+" /
                       "-" / "/" /
                       "=" / "?" /
                       "^" / "_" /
                       "`" / "{" /
                       "|" / "}" /
                       "~"

But according to section 4.2 of RFC 20, everything in atext other than ALPHA
and DIGIT qualifies as a "graphics character". So the assertion that when atext
is composed of a "UTF8 string" it MUST NOT contain "graphics characters"
directly contradicts the ABNF definition of atext. The fact is, only
the specials subset of graphics characters are excluded.

And FWIW, I also stand by my assertion that the term "ASCII graphics
characters" is not well defined - although perhaps "no longer well defined"
would be a better way to put it. There is nothing in the terminology section
that points a reader to the definition in ASCII or RFC 20. And based on what I
see in other documents, the common use definition has shifted away from what's
in the ASCII specification to something which has more utility in the
modern world.

In any case, IMO there are multiple technical problems with the bullet
point in question - WG consensus on the text notwithstanding.

				Ned