Subject: [apps-discuss] I-D Action: draft-ietf-appsawg-malformed-mail-02.txt

From: [email-masked]
Date: Sat, 19 May 2012 01:44:43 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Applications Area Working Group Working Group of the IETF.

	Title           : Advice for Safe Handling of Malformed Messages
	Author(s)       : Murray S. Kucherawy
	Filename        : draft-ietf-appsawg-malformed-mail-02.txt
	Pages           : 16
	Date            : 2012-05-19

   The email ecosystem has long had a very permissive set of common
   processing rules in place, despite increasingly rigid standards
   governing its components, ostensibly to improve the user experience.
   The handling of these come at some cost, and various components are
   faced with decisions about whether or not to permit non-conforming
   messages to continue toward their destinations unaltered, adjust them
   to conform (possibly at the cost of losing some of the original
   message), or outright rejecting them.

   This document includes a collection of the best advice available
   regarding a variety of common malformed mail situations, to be used
   as implementation guidance.  It must be emphasized, however, that the
   intent of this document is not to standardize malformations or
   otherwise encourage their proliferation.  The messages are manifestly
   malformed, and the code and culture that generates them needs to be
   fixed.  Nevertheless, many malformed messages from otherwise
   legitimate senders are in circulation and will be for some time, and,
   unfortunately, commercial reality shows that we cannot simply reject
   or discard them.  Accordingly, this document presents recommendations
   for dealing with them in ways that seem to do the least additional
   harm until the infrastructure is tightened up to match the standards.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-appsawg-malformed-mail-02.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-appsawg-malformed-mail-02.txt

The IETF datatracker page for this Internet-Draft is:
https://datatracker.ietf.org/doc/draft-ietf-appsawg-malformed-mail/

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sat, 19 May 2012 08:48:52 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: Saturday, May 19, 2012 1:45 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: I-D Action: draft-ietf-appsawg-malformed-mail-02.txt
> 
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories. This draft is a work item of the Applications Area Working
> Group Working Group of the IETF.
> 
> [...]

Trying to breathe life back into this work...

This version incorporates a lot of feedback I received off-list.  Contributors acknowledged accordingly!

I've also changed the document's state to Informational from BCP, now that I understand how we use the latter term here.  However, it uses normative language to provide its advice, so I'm wondering if this would qualify as an applicability statement regarding email, meaning it should aim for Standards Track, or if instead I should avoid use of RFC2119 language and make leave it Informational.

Comments welcome.

-MSK

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 19 May 2012 11:55:39 +0200

On Sat 19/May/2012 11:28:41 +0200 Murray S. Kucherawy wrote:
> 
> I've also changed the document's state to Informational from BCP,
> now that I understand how we use the latter term here.

Just to make sure, I grasp that "here" BCP refers to issues internal
to the standardization process, as RFC 2026 auto-referentially states.
 Its Section 5, however, seems to mention that as the third of three
possibilities:

   A BCP document [...] is a vehicle by which the IETF community can
   define and ratify the community's best current thinking on a
   statement of principle or on what is believed to be the best way
   to perform some operations or IETF process function.

> Comments welcome.

One surprising statement is
				
   The following use of unbalanced quotation marks:

      To: "Joe <[email-masked]>

   should be interpreted as:

      To: "Joe <[email-masked]>"@example.net
				
   where "example.net" is the domain name or host name of the
   handling agent making the interpretation.

If that were a "From:" field, wouldn't it represent an attack path
against example.com's ADSP settings?  IMHO, a DKIM verifier should
make an exception to that rule, and tag the message as discardable, if
that's the case.

---

From: Barry Leiba <[email-masked]>
Date: Sat, 19 May 2012 09:52:37 -0400

> now that I understand how we use the latter term here.  However, it
> uses normative language to provide its advice, so I'm wondering if
> this would qualify as an applicability statement regarding email,
> meaning it should aim for Standards Track, or if instead I should
> avoid use of RFC2119 language and make leave it Informational.

I would be very uncomfortable making it Standards Track, because of
the many comments against "standardizing" the handling of
malformations -- making it look too much like declaring the
malformations to be acceptable in the standard.  I actually do think
that Applicability Statement is the right classification for this
document, but hesitate to use it because of that concern.

If consensus gets behind making it an Applicability Statement at
Proposed Standard, with all the appropriate text stressing that this
is standardizing mechanisms for coping with malformations, and that
the malformations themselves remain in violation of all specified
standards, then I'm happy to go there.  But I'd really need to see
strong consensus for that, rather than anything particularly rough.

Otherwise, yes, Informational looks good.

So let's make that an explicit call for comments: Is there a way we
can put this forth in the document so that we can get the consensus I
describe above?

Barry, responsible AD

---

From: "John Levine" <[email-masked]>
Date: 19 May 2012 15:56:30 -0000

> aim for Standards Track, or if instead I should avoid use of RFC2119
> language and make leave it Informational.

I'd prefer that it stay informational.  It seems to me that standards
should tell people how best to interoperate, and in this case the best
thing to do is to read the fripping specs and use the correct syntax.
This draft is offering advice how to try to minimize the damage when
attempting to recover from mistakes, which is something else.

I'd also suggest that the draft emphasize that many kinds of errors
are strong indicators that a message is spam or contains malware, so
the best recovery may well be to reject it or throw it away.
Conversely, if senders want their mail to be delivered, one of the
easiest ways to make it not look like spam is to make it syntactically
correct.

R's,
John

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 20 May 2012 04:01:06 +0000

> From: John Levine [mailto:[email-masked]]
> Sent: Saturday, May 19, 2012 8:56 AM
> To: [email-masked]
> Cc: Murray S. Kucherawy
> Subject: Re: [apps-discuss] I-D Action: draft-ietf-appsawg-malformed-mail-02.txt
> 

Based on the few comments on this topic so far, I'd be just fine leaving it as Informational.

> I'd also suggest that the draft emphasize that many kinds of errors are
> strong indicators that a message is spam or contains malware, so the
> best recovery may well be to reject it or throw it away.

I certainly agree with the latter.  I think the former is a little more dangerous, because there are lots of errors that are innocent or ignorant rather than attempts to deceive.  That doesn't make them right, but it does mean drastic measures can have unwanted side effects, just like assuming a failed SPF check or DKIM signature validation is automatically a sign of foul play.

Let's see, how about this as a new Section 1.3, "General Considerations" or something:

Many deviations from what [MAIL] specifies are considered by some receivers to be strong indications that the message is undesirable, i.e., is spam or contains malware.  Such receivers quickly decide that the best handling choice is simply to reject or discard the message.  This means malformations caused by innocent misunderstandings or ignorance of proper syntax can cause messages with no ill intent also to fail to be delivered.

Senders that want to ensure message delivery are best advised to adhere strictly to [MAIL], as well as observe other industry best practices such as may be published either by the IETF or independently from time to time.

?

-MSK

---

From: "John Levine" <[email-masked]>
Date: 20 May 2012 07:35:41 -0000

>> strong indicators that a message is spam or contains malware, ...

>I certainly agree with the latter.  I think the former is a little more
>dangerous, because there are lots of errors that are innocent or
>ignorant rather than attempts to deceive.  That doesn't make them right,
>but it does mean drastic measures can have unwanted side effects, just
>like assuming a failed SPF check or DKIM signature validation is
>automatically a sign of foul play.

Depends on the error.  There are highly accurate blacklists that work
by looking for technical errors in spam that are characteristic of
botware.  I certainly am not proposing that you say what the bot
signatures are, just that certain kinds of errors strongly suggest
that it's a message best rejected or discarded.

>Many deviations from what [MAIL] specifies are considered by some
>receivers to be strong indications that the message is undesirable,
>i.e., is spam or contains malware.  Such receivers quickly decide that
>the best handling choice is simply to reject or discard the message. 

Yes.

>This means malformations caused by innocent misunderstandings or
>ignorance of proper syntax can cause messages with no ill intent also to
>fail to be delivered.

Maybe, depends on the filter.  See above.

>Senders that want to ensure message delivery are best advised to adhere
>strictly to [MAIL], as well as observe other industry best practices
>such as may be published either by the IETF or independently from time
>to time.

Yes

R's,
John

---

From: Barry Leiba <[email-masked]>
Date: Sun, 20 May 2012 11:28:21 -0400

> Many deviations from what [MAIL] specifies are considered by some receivers to be
> strong indications that the message is undesirable, i.e., is spam or contains malware.
> Such receivers quickly decide that the best handling choice is simply to reject or discard
> the message.  This means malformations caused by innocent misunderstandings or
> ignorance of proper syntax can cause messages with no ill intent also to fail to be delivered.

This text looks good to me.
John says, "Maybe, depends on the filter," for the last sentence.
Well, sure, but that's why it says "can cause", and not "will cause"

> Senders that want to ensure message delivery are best advised to adhere strictly to
> [MAIL], as well as observe other industry best practices such as may be published either
> by the IETF or independently from time to time.

I think this is really a case where that terse method of reference
doesn't work well.  For one thing, it's not just RFC 5322, but also
RFC 2045.  And maybe RFC 6376 (by the way, please update the DKIM
reference to 6376 from 4871) and RFC 4408(bis).

So perhaps, "adhere strictly to the relevant standards (including, but
not limited to, [MAIL],
[MIME], and [DKIM]), as well as"....

Matching that, I might also go back and change the first sentence at
the top to this:
"Many deviations from the standards specifications are considered by
some receivers"....

Barry

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 21 May 2012 17:38:38 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
> Sent: Sunday, May 20, 2012 8:28 AM
> To: Murray S. Kucherawy
> doesn't work well.  For one thing, it's not just RFC 5322, but also RFC
> 2045.  And maybe RFC 6376 (by the way, please update the DKIM reference
> to 6376 from 4871) and RFC 4408(bis).
> 
> not limited to, [MAIL], [MIME], and [DKIM]), as well as"....
> 

I've been looking for an acceptable way to say "all email standards".  Recently the question was asked how one would refer to the DNS and all of its related standards in general, and "STD13" was the answer given.  If you look at STD13 (RFC1034), it's updated by a dozen or more other things, including DNSSEC and such.  That makes it a good reference for DNS work, because you get pointers to all this other important stuff.

We don't quite have that in email, because STD11 is RFC822 which has been twice replaced.  One would have to chase the chain to RFC2822 (which is not a standard) and then to RFC5322, and in any case the pointers off to stuff like MIME and DKIM don't exist because they didn't update those documents.  In the DKIM case, they merely updated a registry.

But all that said, your suggestion is better than what I have, so I'll hack it in.  Thanks.

-MSK

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 21 May 2012 20:00:09 +0200

On Mon 21/May/2012 19:38:38 +0200 Murray S. Kucherawy wrote:

> In the DKIM case, they merely updated a registry.

For SPF, it was said that updating SMTP is out of the question, both
procedurally and because it's a bad idea...