Subject: Re: [apps-discuss] Malformed mail guidance document (draft-ietf-appsawg-malformed-mail)

From: Alessandro Vesely <[email-masked]>
Date: Mon, 05 Mar 2012 11:44:36 +0100

On 01/Mar/12 05:51, Scott Kitterman wrote:
> On Wednesday, February 29, 2012 07:56:41 PM Murray S. Kucherawy wrote:
>> Hi all,
> ...
>> When it reappears, I'd love to see some additional review comments,
>> especially the submission of new cases that should be included in the first
>> version.  I imagine there are many.
> 
> One thought that immediately came to mind is that it might be useful to 
> mention case changes in header fields in paragraph 7.  "Helpful" MTAs doing 
> this have been responsible for more than a few hard to troubleshoot DKIM 
> failures.

More non-malformation cases occur on MIME rewriting.  It may be
helpful to encourage tool developers to stick to the original choice
of token / quoted-string in MIME parameters (format="flowed" vs.
format=flowed see http://tools.ietf.org/html/rfc2045#section-5.1 )
as well as keeping the original boundaries, if at all possible.

---

From: Ned Freed <[email-masked]>
Date: Mon, 05 Mar 2012 07:25:31 -0800 (PST)

> More non-malformation cases occur on MIME rewriting.

It's more like malformations in the original message get exposed if it's
rewritten for whatever reason. You can sometimes avoid or at least defer this
by avoiding unnecessary rewriting; unfortunatley people always seem to come up
with more reasons for doing it.

> It may be

IMO both of these are really bad suggestions. The first assumes that all agents
tolerate superfluous syntax and some actually require it; my experience has
been exactly the reverse.

The second effectively either mandates forced encoding to base64 of all parts
or multi-pass processing to implement. The former often causes more problems
than it solves, the latter is a both a PITA to implement as well as being more
error-prone.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 05 Mar 2012 19:23:32 +0100

On 05/Mar/12 16:25, Ned Freed wrote:
> rewritten for whatever reason.

Hm... MIME doesn't seem to say that either is better than the other.
Based on what you write below, I'd derive that token is better than
quoted-string, provided that quotes are actually superfluous.  Hence,
the I-D could state that that's the canonical syntax, implying that
DKIM signers (and verifiers) may get smoother if they convert that way
before playing their magic.

I recall unsuccessfully looking for such a statement.  Scripts may opt
for always putting quotes in order to avoid to find out whether they
are actually needed.  Would an informational I-D have the authority to
tell them that they shouldn't do so?  As a side note, quotes around
HTML attributes tend to become always mandatory.

> You can sometimes avoid or at least defer this by avoiding
> unnecessary rewriting; unfortunately people always seem to come up 

Yes, when they came out they looked as smart enhancements :-/

---

From: Ned Freed <[email-masked]>
Date: Tue, 06 Mar 2012 00:04:14 -0800 (PST)

> before playing their magic.

I'm unclear as to the relevance. If you're transforming MIME content in some
way - and if you're not why are you bothering to mess around with the headers -
the content will be altered and any DKIM signature is going to be invalidated
no matter what you do to the quotes.

> I recall unsuccessfully looking for such a statement.

There's isn't one. MIME doesn't discuss canonical forms or what representations
are "better". At the time we were much more concerned with getting a
specification out the door than we were on spending what would have certainly
been lots of time arguing about optimal representations. Nor, frankly, did we
have operational experience on which to base such choices. It took 10 years at
least to accumulate that, by which time it was far too late to even consider
adding that to MIME.

Remember that this was 20 years ago. PEM, the first signature scheme
for email, was still under development.

> Scripts may opt

I don't see a problem with saying that, but a document that says that and that
alone doesn't seem worth it to me. I'm also dubious that anything would change
significantly because of making such a recommendation.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 06 Mar 2012 16:12:35 +0100

On 06/Mar/12 09:04, Ned Freed wrote:
> 
>> Hm... MIME doesn't seem to say that either is better than the
>> other. Based on what you write below, I'd derive that token is
>> better than quoted-string, provided that quotes are actually
>> superfluous.  Hence, the I-D could state that that's the
>> canonical syntax, implying that DKIM signers (and verifiers) may
>> get smoother if they convert that way before playing their
>> magic.
> 
> I'm unclear as to the relevance. If you're transforming MIME
> content in some way - and if you're not why are you bothering to
> mess around with the headers - the content will be altered and any
> DKIM signature is going to be invalidated no matter what you do to
> the quotes.

Not necessarily.  The message you replied to was sent to apps-discuss
with Content-Type: text/plain; charset=us-ascii, but I received it
from mail.ietf.org with Content-Type: text/plain; charset="us-ascii".
Thus, the message was rewritten, and its signature would have been
broken if the Content-Type field had been signed (which should be the
behavior of choice, according to RFCs 6376-6377.)

As for the relevance, I admit it's a statistically irrelevant corner
case, which looks more like a DKIM shortcoming than a MIME fault.

> I don't see a problem with saying that, but a document that says
> that and that alone doesn't seem worth it to me. I'm also dubious
> that anything would change significantly because of making such a
> recommendation.

Yet, the less MIME rewriting, the more DKIM deployment is possible.

---

From: Ned Freed <[email-masked]>
Date: Wed, 07 Mar 2012 18:35:52 -0800 (PST)

> behavior of choice, according to RFCs 6376-6377.)

But that only supports my argument - the list also adds a footer to the
message, invalidating any signature on the message body. Who cares about the
quotes on a header field when the body was altered?

> case, which looks more like a DKIM shortcoming than a MIME fault.

Well, I offered to work on the MIME canonicalization problem for DKIM, which
happens to be a problem I've given a lot of thought to. Nobody showed even the
slightest interest, so...

				Ned

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 14 Mar 2012 19:48:58 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Scott Kitterman
> Sent: Wednesday, February 29, 2012 8:52 PM
> To: [email-masked]
> Subject: Re: [apps-discuss] Malformed mail guidance document (draft-ietf-appsawg-malformed-mail)
> 
> > especially the submission of new cases that should be included in the
> > first version.  I imagine there are many.
> 
> mention case changes in header fields in paragraph 7.  "Helpful" MTAs
> doing this have been responsible for more than a few hard to
> troubleshoot DKIM failures.

That's certainly a concern but I'm not sure it's on topic for this draft.  Altering header field name cases is not strictly a malformation, it's a questionable handling choice that doesn't really alter the semantics or interpretation of the message in a meaningful way.  And DKIM does have a way (relaxed header canonicalization mode) to deal with it.

A separate document that lists annoying MTA habits which don't actually result in non-compliant messages would quite possibly be a lot larger than this one.  :-)

-MSK

---

From: Ned Freed <[email-masked]>
Date: Wed, 14 Mar 2012 15:36:06 -0700 (PDT)

> A separate document that lists annoying MTA habits which don't actually result in non-compliant messages would quite possibly be a lot larger than this one.  :-)

Completely agree. This needs to focus on malformations that actually change
RFC5322/MIME message semantics and which cause problems for end users. There's
plenty of graund to cover here.

If you want a header rewriting one that *does* affect end users, the one that
immediately comes to mind isn't, or rather shouldn't, be a malformation per se,
but some clients treat it as such: Some clients get all pissy unless headers
appear in a certain order. I don't think mentioning specific clients is any
help here (especially since it would involve some industry heavyweights who
really should have known better), but a specific example of headers that have
had ordering dependencies would be Content-Type: and Content-Transfer-Encoding.
Most clients put CT before CTE, but not all MIME downgraders do, resulting in
... interesting behavior.

Now, you can argue that this behavior malforms the MIME headers because MIME
says header ordering must be preserved. OTOH, it doesn't really talk about
the downgrading case, so it isn't clear this applies when downgrading.

Another interesting example of header ordering issues shows up when Received:
fields are stripped from messages. (This is unfortunately pretty common to have
happen, especially at final delivery.) The obvious problem with this is good
luck on tracing the message or figuring out where it was delayed, but a less
obvious one shows up when some clients attempt to process Resent-* fields. It
seems some of them depend on there being Received: fields in between the
sets of Resent-* fields. Blow away the Received: fields and ... oops.

				Ned