Subject: Re: [apps-discuss] Comments on Malformed Message BCP draft

From: Alessandro Vesely <[email-masked]>
Date: Fri, 15 Apr 2011 16:20:31 +0200

On 15/Apr/11 14:50, Keith Moore wrote:
> I'm strongly opposed to MTAs "fixing" malformed messages (other than
> submission servers fixing a small number of known problems caused by
> broken mail clients).

+1, standardizing fixes implies a standard status for malformed messages.

> If an MTA does anything at all when it thinks that a message is
> malformed, it should be to bounce it _exactly as it received it
> originally_.

Bouncing is not quite practical.  Rejecting is more viable, but it has
to be coordinated with any backup MX.

> MTAs trying to fix malformed messages, at best, mask problems further
> upstream that should be fixed.   At worst, they exacerbate existing
> problems and make such problems harder to diagnose.
> 
> Keith

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 15 Apr 2011 09:44:16 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Alessandro Vesely
> Sent: Friday, April 15, 2011 7:21 AM
> To: ietf-822
> Cc: apps-discuss
> Subject: Re: [apps-discuss] Comments on Malformed Message BCP draft
> 

The intended status of this document doesn't give any kind of standard status to malformations.  It recommends the safest way of handling them if you're in an environment where you have to do so rather than simply rejecting them.  And the reality is that we (the industry) usually have to do that, so it seems like a good idea to share the collected wisdom about the best/safest way to do so.

The fact that MTAs will do one thing with, for example, malformed MIME and browsers/MUAs do something else is a real problem.

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 15 Apr 2011 09:47:50 -0700

> The intended status of this document doesn't give any kind of standard status to malformations.  It recommends the safest way of handling them if you're in an environment where you have to do so ra…

Perhaps we need to add a new document classification, given this long-term need 
to acknowledge and deal with real-world operational pragmatics and the benefit 
of regularizing them:  Best Current Bad Practices

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Dave Cridland <[email-masked]>
Date: Fri, 15 Apr 2011 17:50:22 +0100

On Fri Apr 15 17:44:16 2011, Murray S. Kucherawy wrote:
> The fact that MTAs will do one thing with, for example, malformed  
> MIME and browsers/MUAs do something else is a real problem.

I'd be willing to bet that such differences represent a vector for  
(for example) mail-based trojans.

I agree that in the event that MIME is incorrect, there is strong  
pressure to do *something* with it, and we should try to ensure that  
that something is aligned.

It may be that the discussion suggests rejecting, in which case I  
suggest the document should clearly explain why, and what the  
implications of not doing so are, beyond "it makes some problems  
harder to diagnose".

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Dave Cridland <[email-masked]>
Date: Fri, 15 Apr 2011 17:54:30 +0100

On Fri Apr 15 17:47:50 2011, Dave CROCKER wrote:
>> The intended status of this document doesn't give any kind of  
>> standard status to malformations.  It recommends the safest way of  
>> handling them if you're in an environment where you have to do so  
>> rather than simply rejecting them.  And the reality is that we  
>> (the industry) usually have to do that, so it seems like a good  
>> idea to share the collected wisdom about the best/safest way to do  
>> so.
> 
> 
> Perhaps we need to add a new document classification, given this  
> long-term need to acknowledge and deal with real-world operational  
> pragmatics and the benefit of regularizing them:  Best Current Bad  
> Practices

I've heard the suggestion of LWP - Least Worst Practise - before.

Dave.

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 15 Apr 2011 09:56:20 -0700

> I've heard the suggestion of LWP - Least Worst Practise - before.

I like that /much/ better than what I suggested.  Concise, accurate and 
appropriately apologetic.

d/

-- 

   Dave Crocker

---

From: Simon Tyler <[email-masked]>
Date: Fri, 15 Apr 2011 17:26:08 +0000

> appropriately apologetic.

I like this to. It sums it all perfectly - there is no right solution to
most of these problems. A best practise implies goodness. A least worst
practise implies the minimum badness which is eactly the problem.

Simon

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Apr 2011 13:38:50 -0400

> to be coordinated with any backup MX.

Bouncing is absolutely what should happen if the message is merely malformed.  Otherwise, the sender has no idea that his message didn't arrive (or why), and nothing will ever be done to fix the problem.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Apr 2011 13:48:35 -0400

I do think there's room for some sort of "if you must do this bad thing, please do it this way" document series from IETF.

However, I don't think that would be applicable to having third party mail relays repair malformed messages.  I think that's no longer "best way to do something that has the potential to do harm" and closer to "how to make a bad situation worse".  

Of course, the devil is in the details.

Keith

On Apr 15, 2011, at 12:56 PM, Dave CROCKER wrote:

> I like that /much/ better than what I suggested.  Concise, accurate and appropriately apologetic.
> 
> d/
> 
> -- 
> 
>  Dave Crocker

---

From: Dave Cridland <[email-masked]>
Date: Fri, 15 Apr 2011 18:50:36 +0100

On Fri Apr 15 18:38:50 2011, Keith Moore wrote:
> Bouncing is absolutely what should happen if the message is merely  
> malformed.  Otherwise, the sender has no idea that his message  
> didn't arrive (or why), and nothing will ever be done to fix the  
> problem.

But the problem is that the message didn't arrive. The reason is that  
it's malformed, but that's not the problem that people care most  
about. Now, *we* may care, but that's a wholly different thing, and  
largely irrelevant to the average user.

Bouncing has problems too - it's trivial to use such a server to  
bounce malformed MIME back to some other address which then processes  
the MIME and allows some malware through.

As I said before, differences in error handling behaviour may result  
in malware vectors being available. If you standardize the error  
handling (to whatever you like - pass through, bounce, or reject)  
then the net result is that exploits of this form cannot happen.

Dave.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 15 Apr 2011 10:58:28 -0700

> From: Keith Moore [mailto:[email-masked]]
> Sent: Friday, April 15, 2011 10:49 AM
> To: [email-masked]
> Cc: Dave Cridland; ietf-822; General discussion of application-layer protocols; Murray S. Kucherawy
> I do think there's room for some sort of "if you must do this bad
> thing, please do it this way" document series from IETF.
> 
> However, I don't think that would be applicable to having third party
> mail relays repair malformed messages.  I think that's no longer "best
> way to do something that has the potential to do harm" and closer to
> "how to make a bad situation worse".
> 

I don't think this work is targeted at intermediaries.  In fact, I'd be completely fine with expressly saying it's meant to address processing at ingress MTAs only.

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Apr 2011 14:00:22 -0400

By default, MTAs should not care about the contents of a message.  To do so is a layering violation.   The message should be opaque and the MTA should look only at the envelope.   The only "standard" exception is 8bit to 7bit conversion, which hopefully is rarely an issue these days.  

So the fact that a message is "malformed" should not prevent delivery of that message.

But if MTAs are going to care about contents of messages (say, for the purpose of filtering spam or viruses, which pretty much everyone will acknowledge as necessary measures against evil these days), then there will inevitably be cases where the MTAs are unable to parse those contents, presumably because the messages are malformed.  If the spam or virus filter can reliably tell that the message is spam or a virus, dropping it is the proper thing to do.  But if the filter cannot tell, or cannot parse the message because it is malformed, bouncing it is probably better.

The problem of using MIME to send malware exists whether or not the message is bounced.  Bouncing messages, if properly done, does not amplify the problem, because there should never be more than one bounce address.  Either the message containing malware is delivered (in which case the recipient UA deals with the threat) or it is bounced (in which case the "return-path" UA deals with the threat).   And again, if properly done, bouncing should not "mask" the source of the malware, because the Received headers should be present in the returned content in any case.  

Keith

On Apr 15, 2011, at 1:50 PM, Dave Cridland wrote:

> But the problem is that the message didn't arrive. The reason is that it's malformed, but that's not the problem that people care most about. Now, *we* may care, but that's a wholly different thing, and largely irrelevant to the average user.
> 
> Bouncing has problems too - it's trivial to use such a server to bounce malformed MIME back to some other address which then processes the MIME and allows some malware through.
> 
> As I said before, differences in error handling behaviour may result in malware vectors being available. If you standardize the error handling (to whatever you like - pass through, bounce, or reject) then the net result is that exploits of this form cannot happen.
> 
> Dave.
> --

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Apr 2011 14:03:42 -0400

Well, I'm all for having submission servers (listening to port 587, requiring authentication, etc.) check for *822/MIME syntax errors, and correct obvious flaws in the message content.  Though I still think that sometimes it's better if they bounce those messages, so that users can know that their MUAs are broken.  Of course there will be pressure on ISPs to support messages generated by broken MUAs.

Accepting mail submissions on port 25 is something that should have been phased out 10 years ago.

Keith

---

From: "J.D. Falk" <[email-masked]>
Date: Fri, 15 Apr 2011 11:26:03 -0700

> practise implies the minimum badness which is eactly the problem.

+1

But in the meantime, I think the BCP stream is the right place for this document.

--
J.D. Falk
the leading purveyor of industry counter-rhetoric solutions

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 15 Apr 2011 23:25:49 +0200

I found the advice rather naïve. Not bad, but also not very 
sophisticated, and I fear that extending it to be comprehensive will not 
produce a readable document.

Still, I can see value in a document whose abstract says "here are some 
of the bad things you must expect to receive, and not just from 
spammers". Is that the goal of your draft?

Arnt

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 15 Apr 2011 15:20:50 -0700

> From: Arnt Gulbrandsen [mailto:[email-masked]]
> Sent: Friday, April 15, 2011 2:26 PM
> To: Murray S. Kucherawy
> Cc: [email-masked]; [email-masked]

It's meant only as a starting point.  And I think extending it, if done carefully, will produce something readable and valuable.

> spammers". Is that the goal of your draft?

"...and what the consensus view is about what to do with them," precisely.

---

From: Larry Masinter <[email-masked]>
Date: Fri, 15 Apr 2011 18:45:57 -0700

One might note that the same kind of difficult debate over 'error handling' has been at the root of much controversy in HTML land.

Don't many MTAs actually implement the same "sniffing" for content-type and charset as is being proposed in

http://tools.ietf.org/html/draft-ietf-websec-mime-sniff-02

It was my impression they did.

Is "wrong content-type" a "malformed message"?

Larry

--

http://larry.masinter.net

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
Sent: Friday, April 15, 2011 3:21 PM
To: Arnt Gulbrandsen
Cc: [email-masked]; [email-masked]
Subject: Re: [apps-discuss] Comments on Malformed Message BCP draft

_______________________________________________

apps-discuss mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Nathaniel Borenstein <[email-masked]>
Date: Sat, 16 Apr 2011 09:27:35 +0100

Although all my basic instincts are in agreement with Keith's position -- the only way you will ever persuade message senders to comply more fully with the standards is to reject their non-compliant messages -- my recent experience has turned me 180 degrees and convinced me that this position is hopelessly idealistic.  

As some of you know, I have only recently returned to work at a company (Mimecast) whose primary business includes the maintenance and operation of a large-scale MTA.  The situation has changed a lot in the last ten (not to say twenty) years, and in many ways not for the better.  Today's commercial reality is a race to the bottom that more or less guarantees that MTAs will do precisely what Keith says they should not do, which means, to my mind, that it behooves us to document our collective wisdom on the least harmful and most consistent way to do it.

A single example should make this clear:  Nearly every business now receives its mail via some third party email security mechanism -- perhaps an appliance, perhaps locally installed code, and increasingly a cloud-based service.  Competition among these third parties is fierce, and inevitably standards compliance is far less of a driver of business decisions than customer satisfaction.  Imagine, then, that company X, which enforces the standards very strictly, as Keith advocates, manages to "steal" a customer from company Y, which does not.  All of a sudden -- and I assure you this really happens -- a host of slightly-misformatted but customer-desired messages which were delivered by company Y will start getting blocked by company X.  The customer, inevitably, will scream bloody murder.  They won't care that company X is doing a better job of enforcing the standards, they will only care that company X is -- wrongly, in their eyes -- blocking desirable streams of messages that the previous vendor delivered with no visible problem.  The generalization is clear:  the financial and business incentive is to deliver every message that you can possibly figure out how to deliver, so that your service doesn't appear "inferior" to customers who don't give a rat's rump about the details of standards compliance.

I'm not sure why this came as such a surprise to me, as it is actually just another instance of Postel's Law.  Being liberal in what we accept means, in this case, accepting and delivering any message where we believe, with a high level of confidence, that the sender's intentions are clear despite its standards violations.

I can tell you that nothing the IETF says is likely to have any effect on whether or not Mimecast tries to "fix" and deliver such messages -- we do and we will, as do nearly all of our competitors, because although we care what the IETF says, we have no choice but to care what our paying customers say even more.  However, if the IETF offered guidance on the least harmful way to do this, the odds are good that we would follow it.  And I think it would be better if vendors who felt the need to "fix" messages would at least be mutually consistent in how they fix them.

One (probably controversial) idea that might reduce our collective angst would be to specify that, when an MTA fixes such a message, it also generates an explanatory/warning message back to the sender or the sender's postmaster (perhaps limited to 1 message per day per malformation type).  That way we might not be exacerbating existing problems quite as much as Keith and other (myself included) fear.  And the senders of misformatted messages might eventually fix their code, if only to shut up the annoying warning messages.  -- Nathaniel

On Apr 15, 2011, at 1:50 PM, Keith Moore wrote:

> I'm strongly opposed to MTAs "fixing" malformed messages (other than submission servers fixing a small number of known problems caused by broken mail clients).
> If an MTA does anything at all when it thinks that a message is malformed, it should be to bounce it _exactly as it received it originally_.
> 
> MTAs trying to fix malformed messages, at best, mask problems further upstream that should be fixed.   At worst, they exacerbate existing problems and make such problems harder to diagnose.
> 
> Keith
> 
> On Apr 14, 2011, at 3:07 PM, Murray S. Kucherawy wrote:
> 
>> This is some work starting up in the APPS area.  Please comment on the apps-discuss list if you’re interested in participating.
>>  
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Simon Tyler
>> Sent: Thursday, April 14, 2011 2:59 AM
>> To: [email-masked]
>> Subject: [apps-discuss] Comments on Malformed Message BCP draft
>>  
>> Hi,
>>  
>> Having read the Malformed Message BCP draft I am interested in getting some discussion going on its content and to find the best way forward.
>>  
>> For those who missed it, the draft is at:
>> 
>> https://www1.tools.ietf.org/html/draft-kucherawy-mta-malformed-00
>> 
>> I have a few comments on it.
>> 
>> One thing that concerns me is the proposal that processing should stop when certain malformations are identified.
>> 
>> For example it is proposed that should a badly wrapped header field be found (quite how we define this is left open, I presume a line that does not start with a valid header field name followed by a colon) then the processing agent should treat this as the end of the header. My feeling is that this opens up a greater potential hole than the one closed and that can be exploited.
>> 
>> An example of the type of issue this could is cause is that should such a malformation occur before the MIME header fields in the header then this would cause the rest of the header and the message body to be treated as plain text. This could cause content analysis system to fail as they would not interpret the MIME content in the way that was presumably intended.
>> 
>> Given that these recommendations are unlikely to be followed by all clients and servers, I feel that this suggestion will allow content through an agent without suitable processing. My preference on this particular malformation would be to continue processing the header fields, this is based on the assumption that what follows the malformed header field is more likely to be further header fields and not body content. What we do with the malformed header field I am less certain about. We could just ignore it or we could treat it as part of the previous header field - both will be right as often as they wrong. I would welcome some additional thoughts on this.
>> 
>> I have similar feelings about some of the other suggestions including the lack of a MIME-Version header. We cannot ignore intended meaning just because a non-compliant application made a small error in the header fields. Everyone will be best served if we subject such messages to more analysis, not less.
>> 
>> On the whole I think a set of guidelines in this area is good but it will be hard to reach consensus without agreement on some basic underlying principles.  I would suggest that one such principle is to try to do what the intended recipient would most likely prefer, which is generally to fix and deliver non-spam messages.
>> 
>> I would also propose some additions to the draft. At Mimecast we see a lot of messages generated by all sorts of systems and amongst these we see a lot of different kinds of message malformations.
>> 
>> I'll suggest more as I think of them but for starters here are a few:
>> 
>> 1. Excessively long lines in both headers and body. I commonly see lines that are several hundred Kbs in length. These are often valid messages in the sense that the content is desired by the receiver and in all respects other than line length are well formed. Obviously a limit has to be enforced and I would like to find a consensus on what sort of limit is reasonable. Initially I felt 8K was a good limit - it is after all 8 times the limit in RFC 5321. But it appears that this is too small a limit in real situations. When the limit is exceeded, what is the best option – a rejection or  forced line wrap. I am open to both. 
>> 
>> 2. Invalid characters in headers. I often see headers with un-encoded 8bit characters. These are often displayed correctly to the recipient where the client happens upon the correct character set by virtue of chance.
>>  
>> 3. 8bit characters in MIME sections with a content-transfer-encoding of 7bit.
>> 
>> If you have read this far then I think you will agree with me that Murray has made a good start on a much needed set of guidelines. Now let's see if we can support him to expand on the work he has done and reach a consensus on the best approaches.
>> 
>> Simon
>> <ATT00001..txt>
>

---

From: Jaap Akkerhuis <[email-masked]>
Date: Sat, 16 Apr 2011 10:39:13 +0200

And the senders of misformatted messages might eventually fix
    their code, if only to shut up the annoying warning messages.

And the easiest way to do that is declare these as spam, so nothing will
be fixed in the end.

	jaap

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Apr 2011 10:45:37 -0400

> A single example should make this clear:  Nearly every business now receives its mail via some third party email security mechanism -- perhaps an appliance, perhaps locally installed code, and incr…

This is why I think the checking for syntax and either correct or rejection of malformed messages needs to be done on the sender's side, by the MSA.

But there's also the opposite argument - say that handling of incoming messages moves from company X to company Y, and malformed messages that were blocked by company X are now passed by company Y.  As a result, the client company receiving such messages, now finds itself subject to drastically increased attacks that use malformed messages (and UA susceptibility to them) as a vector.   Now the client will scream bloody murder because Y did not block such messages.

Both scenarios argue for more uniform handling of messages, and/or more strict adherence to *822/MIME syntax. so there will be less variation between how different MTAs/providers handle mail.     (Though the mail screen/delivery providers might want to distinguish themselves from one another by how strict/tolerant/clever they are, thus increasing the variation between providers.)

But yes, the desire for uniform handling might argue for defining a standard syntax verifier (with some recommendations about correction, where appropriate) and encouraging MSAs to implement it.   The desire for more uniform handling of messages might also argue for trying to move such verification/correction to MSAs and having downstream MTAs treat such messages as opaque, so that there will be fewer opportunities for messages to be subjected to variable amounts of verification/correction.  

(Honestly, I have long thought that traditional SMTP store-and-forward these days is more trouble than it's worth, and what we need is to reduce the number of hops that a message endures as it goes from sender to recipient.  To this end, I'd support some sort of pass-through extension to SMTP to allow SMTP to continue to be used as a means to let messages get through firewalls, but without actually doing store-and-forward when not necessary.)

> I'm not sure why this came as such a surprise to me, as it is actually just another instance of Postel's Law.  Being liberal in what we accept means, in this case, accepting and delivering any mess…

Postel's Law was great in ARPAnet days, but I have long wondered how well it scales to a network with billions of users and thousands or tens of thousands of implementations.

But I don't really have any problem in principle with correcting messages where "the sender's intentions are clear".  In practice, I wonder how many cases exist for which code can reliably detect that "the sender's intentions are clear".   For example, a missing Date header field - sure the MSA or downstream MTA can supply one, but it doesn't really know the date/time at which the sender caused the message to be sent.

> One (probably controversial) idea that might reduce our collective angst would be to specify that, when an MTA fixes such a message, it also generates an explanatory/warning message back to the sen…

Well, if there were a standard set of tests to apply to outgoing mail, and these tests were implemented by MSAs, those MSAs could presumably make such data available to sending UAs and/or their users.    And yes, hopefully MUA vendors would make sure that messages generated by their MUAs passed such tests.  That would be, IMO, the best of all possible outcomes.

Keith

---

From: Peter Koch <[email-masked]>
Date: Sat, 16 Apr 2011 19:54:49 +0200

On Sat, Apr 16, 2011 at 09:27:35AM +0100, Nathaniel Borenstein wrote:

> The generalization is clear:  the financial and business incentive is to deliver every message that you can possibly figure out how to deliver, so that your service doesn't appear "inferior" to customers who don't give a rat's rump about the details of standards compliance.

Sure, the same way taxis will speed to outperform their competitors striving
to improve their own customers' experience.  Something is wrong in this picture.
Standards are only partly about that customer experience, they are about
interoperability, security, scalability, and, at least back in the day, overall
architecture.  This theme fits painfully nicely into that IETF80 technical
plenary about the post standardization world and dominance of applications.

> I'm not sure why this came as such a surprise to me, as it is actually just another instance of Postel's Law.  Being liberal in what we accept means, in this case, accepting and delivering any mess…

I believe the Robustness Principle has been abused and perverted for too
long now.  You can only be so liberal that you are still able to be conservative
on the emitting side. The problems in Murray's draft are with those cases where
the intent was not clear or not clear enough to allow or support that balance.

> However, if the IETF offered guidance on the least harmful way to do this, the odds are good that we would follow it.  And I think it would be better if vendors who felt the need to "fix" messages would at least be mutually consistent in how they fix them.

First, it is about interoperability issues, not an abstract "compliance", i.e.,
we're already in "following the rules by spirit rather than letter" space. 
Who can I as an operator file a bug report with when there is a standard that
says "A" rather than "B" and a BCP that says "well, you should really treat B,
which should not have happened, this way"?  If there's an ambiguity in the standard
that harms interoperability and/or poses security risks, shouldn't it be corrected
in the standard for that to be self consistent?

> That way we might not be exacerbating existing problems quite as much as Keith and other (myself included) fear.  And the senders of misformatted messages might eventually fix their code, if only to shut up the annoying warning messages.  -- Nathaniel

Or they'd add additional code to suppress these warnings and market that
as a feature?

-Peter

---

From: Tony Finch <[email-masked]>
Date: Mon, 18 Apr 2011 13:08:13 +0100

Dave Cridland <[email-masked]> wrote:
>
> It may be that the discussion suggests rejecting, in which case I suggest the
> document should clearly explain why, and what the implications of not doing so
> are, beyond "it makes some problems harder to diagnose".

It does not make sense to have a uniform policy for dealing with corrupt
messages. Some kinds of corruption are caused by common legitimate
software, in which case you will want to treat it leniently; others are
caused by malware or rare kinds of incompetence, in which case it makes
more sense to reject. You can only determine which is which based on
operational experience, and the least-worst response changes from time to
time.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rockall, Malin, Hebrides: South 5 to 7, occasionally gale 8 at first in
Rockall and Malin, veering west or northwest 4 or 5, then backing southwest 5
or 6 later. Rough or very rough. Occasional rain. Moderate or good,
occasionally poor.

---

From: Eric Burger <[email-masked]>
Date: Mon, 18 Apr 2011 08:10:55 -0400

This whole discussion makes me think what we are really doing is changing 2822 to make these messages "well formed."

Is that the intent?

On Apr 18, 2011, at 8:08 AM, Tony Finch wrote:

> [email-masked]

---

From: Tony Finch <[email-masked]>
Date: Mon, 18 Apr 2011 13:19:49 +0100

Murray S. Kucherawy <[email-masked]> wrote:
>
> I don't think this work is targeted at intermediaries.  In fact, I'd be
> completely fine with expressly saying it's meant to address processing
> at ingress MTAs only.

If you are going to make that kind of restriction, it should happen at
submission servers only.

There is a history of MX servers making "helpful" fix-ups to messgaes
(e.g. inserting missing message-id or from headers) before handing them
over to anti-spam software, which can make spam/legit checks invalid in
both the positive and negative senses. So I think MXs should be as
transparent as possible so that downstream security software is less
likely to have interop problems. Intermediate relays should also be as
transparent as possible.

Tony.

---

From: Tony Finch <[email-masked]>
Date: Mon, 18 Apr 2011 13:47:08 +0100

Nathaniel Borenstein <[email-masked]> wrote:
>
> One (probably controversial) idea that might reduce our collective angst
> would be to specify that, when an MTA fixes such a message, it also
> generates an explanatory/warning message back to the sender or the
> sender's postmaster (perhaps limited to 1 message per day per
> malformation type).

That's likely to cause too much spam backscatter.

Tony.

---

From: John C Klensin <[email-masked]>
Date: Mon, 18 Apr 2011 08:48:58 -0400

--On Monday, April 18, 2011 13:19 +0100 Tony Finch
<[email-masked]> wrote:

>> I don't think this work is targeted at intermediaries.  In
>> fact, I'd be completely fine with expressly saying it's meant
>> to address processing at ingress MTAs only.
> 
> If you are going to make that kind of restriction, it should
> happen at submission servers only.
> 
> There is a history of MX servers making "helpful" fix-ups to
> messgaes (e.g. inserting missing message-id or from headers)
> before handing them over to anti-spam software, which can make
> spam/legit checks invalid in both the positive and negative
> senses. So I think MXs should be as transparent as possible so
> that downstream security software is less likely to have
> interop problems. Intermediate relays should also be as
> transparent as possible.

Of course, this is exactly what RFCs 4409, 5068, and 5321 say:
considerable flexibility for making fixes for the submission
server, which is presumably under the control of (or at least
responsible to) the sender; relays don't mess with message
contents until they get to the delivery server.  Various
operational considerations have modified that somewhat but, if
this specification doesn't apply to intermediaries, it changes
nothing except, as Eric Burger points out, the practical
definition of well-formed messages.

IMO, if we want to do that (and I'd personally not favor it), we
should do so in some 5322bis, not try to sneak the changes in
via a BCP that is inconsistent with 5322.

    john

---

From: Ned Freed <[email-masked]>
Date: Mon, 18 Apr 2011 09:45:34 -0700 (PDT)

> operational experience, and the least-worst response changes from time to

Bingo. This is precisely the concern I have with this effort - the IETF
consensus process operates in a time scale that will make it very difficult to
capture a sensible set of recommendations in a specification. And even if we do
manage to capture something useful, it will need to be updated regularly to
remain useful. Are we prepared to do that?

And it isn't a binary choice between rejection and fixup either. What sort of
fixup makes sense can change over time.

Consider the invalid header line case. There was a period a few years ago
when it seemed like half the email agents out there couldn't consistently
fold header lines. As a result the strategy of accepting such messages
and not terminating the header seemed to work best.

But as this issue was starting to disappear, a new one, where some group of
agents decided that the blank line between the header and body was  superfluous
and could be omitted, started to show up. A strategy of treating an invalid
header line as a stopping point works best in this case. But who's to say
what's going to happen next month?

This sort of behavior shifting over time is especially prevalent in certain
problem areas like CJK charset usage and labeling.

Another aspect to this that hasn't been discussed yet is where some
agent can only tolerate a subset of what the standards allow. Is it OK
to reformat messages to avoid using legitimate but neverthless problematic
features?

My personal favorite of these was the one where, several years back, a very
popular product refused to accept MIME messages where the 
content-tranfer-encoding field appeared prior to the content-type in the
message header.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Mon, 18 Apr 2011 20:13:32 +0100

> fixup makes sense can change over time.

The draft suggests an intermediate option, which is to process the message
using a grammar with more lenient handling of error cases but to pass on
the message unchanged (if it gets passed on). A lot of the risk comes
from doing this (or rather, from doing this inconsistently). I think I'd
like to make a distinction between a relay (which is transparent) and a
security gateway (which does fixups). They should have consistent
behaviour, by which I mean that if a transparent relay is presented with a
corrupt message, it should treat it in the same way as a standard parser
would treat the same message after it has been fixed up by a security
gateway.

Whether a message gets rejected or not is a somewhat different matter. I
think submission servers can and should be a lot stricter than an MX can
be. Either way, every system needs to parse borderline cases more
consistently.

Tony.

---

From: Douglas Otis <[email-masked]>
Date: Mon, 18 Apr 2011 15:56:08 -0700

On 4/18/11 5:48 AM, John C Klensin wrote:
>> Murray S. Kucherawy<[email-masked]>  wrote:
Fully Agree.

-Doug

---

From: Barry Leiba <[email-masked]>
Date: Tue, 19 Apr 2011 14:40:20 -0400

On reading all the comments about this, and thinking about it myself,
I'm of a very mixed mind.

First: I have no sympathy for the comments that we should fix this
stuff in 5322, and not in some "add-on".  This is *not* "fixing"
anything.  This is *not* saying that any of the "malformed" messages
are now valid.  This is not changing anything at all in 5322.  What
this is doing is acknowledging that senders often violate 5322, and
that those violations are *wrong*.  What it adds is that it also
acknowledges the reality that, as Nathaniel and others have said, we
can't just throw those wrong messages away, and there's some value in
agreeing how to handle them.  This document -- or its final version --
is an attempt to document that agreement.

Agents along the way -- MSAs, MTAs, MDAs, and MUAs -- will make their
guesses and fix-ups, and I do think it's in the best interest of
everyone for us to document less-harmful avenues to take, as well as
roads to hell.  So I support this document for that reason.

On the other hand, Ned's right that the "best" (or least bad, or
whatever) way to handle each situation *is* likely to change over
time, so locking the advice into a BCP might not work well.  It's also
clear that some malformations will do best with complicated heuristics
beyond what'll be recommended here.  The appearance of a non-header
line in a message header is a perfect example of that.  Consider these
two fragments:

1:
   Date: xxx
   Subject: this is a badly
   continued header line
   MIME-Version: 1.0

   This is the body of the message.

2:
   Date: xxx
   MIME-Version: 1.0
   Subject: this is the subject
   I've improperly terminated the header here.
   This is the rest of the body.

The right answer for the two is different.  In (1), we don't want to
assume the "continued header line" is the beginning of the body, and
in (2) we don't want to try to treat the "I've improperly" line as a
continuation of the subject.  A better answer will be to look ahead a
bit to try to re-establish context and make a better guess than can be
done simply by applying a rule.

And yet number three here will break that too:

3:
   Date: xxx
   MIME-Version: 1.0
   Subject: this is the subject
   I've improperly terminated the header here.
   Look: You know it's complicated.

   This is rest of the body.

On the other hand, we know that some of these issues are
straightforward.  Why make everyone figure it all out from scratch?

In the end, the best we can do is to make recommendations to try to
get some consistency.  I think it's worth doing a document like this,
but it's not at all straightforward, and we'll have to be very careful
at every step to make a few things clear:

1. The appearance of these broken messages is BAD, and the BEST thing
is to fix the software that's generating them.

2. Sometimes, it really IS the right thing to
reject/refuse/bounce/whatever-you-want-to-call-it the message.

3. We do or don't have a sense of a reasonably safe guess to make for
this particular malformation.

4. When we do have a reasonably safe guess, here's what it is.

Barry