Subject: [marf] Some thoughts on draft-ietf-marf-as-02

From: Steve Atkins <[email-masked]>
Date: Wed, 11 Jan 2012 19:28:00 -0800

Some comments on https://datatracker.ietf.org/doc/draft-ietf-marf-as/?include_text=1

Most of the document looks fine, but I have some concerns about the sections that deal with unsolicited reports.

Section 5. Solicited and Unsolicited Reports

"Scored high by spam filters" really isn't a good reason to send an abuse report. Sending mail that "looks like" spam according to a mechanical content filter isn't an AUP violation, and reports about it aren't going to be actionable as there's nothing wrong with sending mail with a high spamassassin score. While, arguably, reports of mail that "looks like" spam might be a basis for investigation by an abuse desk as to whether a customer is doing something wrong it's not a very useful basis - if there is a source of mail on your network that's found offensive by recipients, you'll find out fairly quickly by receiving informed abuse reports, manually or from this-is-spam-button driven feedback loops, so reports of mail that "looks like" spam aren't going to provide much data that the abuse desk won't already have via other channels. 

Non-actionable reports - whether they be false positives or not - reduce the efficiency of an abuse desk. Let's just not mention reports of mail scored high by spam filters, and quietly leave it as part of "anything else the sender considers".

(Automated reports of spam-like content may well be useful as part of an "avoid false positives" sort of feedback loop, but not as an abuse report. And automated reports of mail that has virus or malware content might be appropriate, even though spam-like content isn't.)

The DKIM signer of a message is definitely a good target for solicited reports as part of a feedback loop. For unsolicited reports it's much less clear that that's the case. There are some mechanical reasons - the d= value is an identifier for reputation, not a domain intended to receive email, so we don't want to imply that [email-masked] is automatically a good place to report mail that's signed with d=transactional.blighty.com. In a broader sense, though, it's likely that the DKIM signing domain is the author of the mail - and if the mail is objectionable, they're often the least useful place to notify. In the case of mail sent through an ESP, the DKIM d= domain is likely to end up at the customer, rather than at the ESP where it's more likely to be handled correctly.

That's one instance of a more general issue of sending automated, unsolicited reports. Identifying an appropriate recipient for a report is *hard* to do automatically, while "shotgunning" reports to multiple email addresses because one of them might be appropriate is unhelpful. Sending them in a format that's intended primarily to be machine-readable and handled automatically by the entity receiving the reports leads to the problem of knowing what email address at what entity is intended to handle machine-readable reports. There's no standard for it, and there's a risk that those machine-readable reports will be discarded or ignored if they're sent to role addresses chosen at random. 

Section 8 discusses this issue again, and somewhat better. Would it make sense to move the mention of DKIM to section 8, where it would be in a better context?

Section 7. Receiving and Processing Complaint-Based Solicited Reports

"Point 3 - Mail operators SHOULD utilize an automated system to receive and process these reports". 

That's really an engineering and operational decision. If I only receive two or three ARF reports a day, and my MUA or ticketing system can display them correctly, then handling them as part of my usual customer support / abuse desk workflow is likely to be more reliable (and much less expensive to implement) than an automated system. An ARF based FBL is still of value to me, but automating it would be a poor decision. RFC 6449 Section 4.4 says pretty much that. I think we'd be better just pointing people at that section of RFC 6449, rather than saying that automation is a SHOULD.

Section 8. Generating and Handling Unsolicited Reports

Is authentication failure abusive? If it is, we should say why (and in what context). If not, it shouldn't be in this document.

I'm not sure it's true that "even recipient systems that haven't asked for ARF format reports handle them at least as well as any other format such as plain text, with or without a copy of the message attached."

I know for sure that there are ticketing systems in use that don't handle anything other than plain text particularly well. They will not handle ARF reports as well as they would handle plain text. 

Unexpected multipart/* formats (such as the multipart/report type used by ARF) aren't necessarily going to be handled well even by MIME aware MUAs. Similarly, message/feedback-report isn't necessarily going to be handled well by MIME aware MUAs.

As an example, here's a snapshot of an ARF report displayed by Mail.app: https://skitch.com/steveatkins/g2bkt/inbox-27642-messages-33-unread

And here's one displayed by gmail: https://skitch.com/steveatkins/g2bck/inbox-27642-messages-33-unread

And Outlook Express: https://skitch.com/steveatkins/g2bpj/192.168.80.89

All three of those are more sophisticated than the web-based ticketing systems commonly used at smaller ISPs.

(Alpine, OTOH, displayed it just fine - https://skitch.com/steveatkins/g2bq8/steve-infrastructure-mail )

Ticketing systems are one of the last bits of the email system that don't handle MIME particularly robustly, if at all, and abuse desks are (quite reasonably) wary of clicking on attachments - all attachments, including attached message/rfc822. Quite a few ISPs (including national US ISPs) ask reporters explicitly to not forward email as an attachment, rather to paste the headers and body of the mail being reported into a new message, due to either tool limitations or a policy not to open attachments for security reasons. 

Everybody can handle the lowest common denominator of text/plain, but that's about it.

So… I don't think it's valid to say that recipient systems that are not advertised to accept ARF will render ARF format messages in a way that will lead to them being handled. At the very least, anyone sending unsolicited messages in ARF format must assume that recipients will not be able to see the ARF metadata, and should include all information needed in human readable format in the first, text/plain section of the report. Also, they should ensure that the report is readable when viewed as plain text, to give low end ticketing systems as much help as possible - that might includes advice on appropriate MIME encoding and choice of character sets and so on. And they should be aware that the report may be discarded or ignored due to it's format.

I don't think that experience purely of sending unsolicited ARF reports can really lead to the claim that all recipients handle them well - as the recipients that *don't* handle them well are simply going to discard or ignore the report, rather than trying to work out why it was sent. Abuse desks generally have more than enough actionable reports in their queue to waste time trying to decipher ones that they can't read.

IIRC, John includes a link to a web-based rendering of each report in the plain text part of the unsolicited reports he sends out (not a bad idea, if you have the infrastructure for it), so they may be more widely readable by abuse desks than an ARF format report would be.

Cheers,
  Steve

---

From: "John Levine" <[email-masked]>
Date: 12 Jan 2012 07:51:44 -0000

>I'm not sure it's true that "even recipient systems that haven't asked
>for ARF format reports handle them at least as well as any other format
>such as plain text, with or without a copy of the message attached."
>
>I know for sure that there are ticketing systems in use that don't
>handle anything other than plain text particularly well. They will not
>handle ARF reports as well as they would handle plain text. 

Oh, sure, but I also used to get a certain number of "please send this
as an attachement."  I didn't say it works flawlessly, just that in
my experience, at least, it's no worse than anything else. I'm also
getting a trickle of responses saying "ARF? Great, we handle that automatically."

> Quite a few ISPs (including national
>US ISPs) ask reporters explicitly to not forward email as an attachment,

Still?  I can't rememnber the last time I got a request to resend with
the message pasted in from an ISP I'd ever heard of.

R's,
John

---

From: Shmuel (Seymour J.) Metz <[email-masked]>
Date: Thu, 12 Jan 2012 09:54:40 -0500

In <[email-masked]>, on
01/11/2012
   at 07:28 PM, Steve Atkins <[email-masked]> said:

>Quite a few ISPs (including national US ISPs) ask reporters
>explicitly to not forward email as an attachment, rather to paste the
>headers and body of the mail being reported into a new message, 

While others reject it as spam unless you put it in an attachment :-(

>Everybody can handle the lowest common denominator of text/plain,

Would that it were true.

-- 
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2        <http://patriot.net/~shmuel>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 12 Jan 2012 17:32:34 +0100

> Some comments on https://datatracker.ietf.org/doc/draft-ietf-marf-as/?include_text=1

Thank you, Steve.  Glad to read from you again on this list.

> "Scored high by spam filters" really isn't a good reason to send an
> abuse report.

Yet, for good messages, that use case of ARF would very closely
resemble the intended use of its ancestor NDN.

> Sending mail that "looks like" spam according to a mechanical
> content filter isn't an AUP violation, and reports about it aren't
> going to be actionable as there's nothing wrong with sending mail
> with a high spamassassin score. [...]
> 
> Non-actionable reports - whether they be false positives or not -
> reduce the efficiency of an abuse desk. Let's just not mention
> reports of mail scored high by spam filters, and quietly leave it
> as part of "anything else the sender considers".

So this is a case where it is useful to know whether the
report-trigger was auto or manual.  Previous mumbling is here:
http://www.ietf.org/mail-archive/web/marf/current/msg01233.html

> (Automated reports of spam-like content may well be useful as part
> of an "avoid false positives" sort of feedback loop, but not as an
> abuse report.

This reasoning seems to imply it is better to add a new Feedback-Type
rather than a new field with a human/auto indication.  (Assuming we
won't then need, say, a manually-discovered-virus type.)

> The DKIM signer of a message is definitely a good target for
> solicited reports as part of a feedback loop. For unsolicited
> reports it's much less clear that that's the case. There are some
> mechanical reasons - the d= value is an identifier for reputation,
> not a domain intended to receive email, so we don't want to imply
> that [email-masked] is automatically a good place
> to report mail that's signed with d=transactional.blighty.com.

That can be covered by reporting-discovery.

> In a broader sense, though, it's likely that the DKIM signing
> domain is the author of the mail - and if the mail is
> objectionable, they're often the least useful place to notify. In
> the case of mail sent through an ESP, the DKIM d= domain is likely
> to end up at the customer, rather than at the ESP where it's more
> likely to be handled correctly.

IMHO, ESPs should keep current and apply BCP 167.

> That's one instance of a more general issue of sending automated,
> unsolicited reports. Identifying an appropriate recipient for a
> report is *hard* to do automatically, while "shotgunning" reports
> to multiple email addresses because one of them might be
> appropriate is unhelpful. Sending them in a format that's intended
> primarily to be machine-readable and handled automatically by the
> entity receiving the reports leads to the problem of knowing what
> email address at what entity is intended to handle machine-readable
> reports. There's no standard for it, and there's a risk that those
> machine-readable reports will be discarded or ignored if they're
> sent to role addresses chosen at random.

Reporting-discovery can only cover authenticated mail (DKIM and/or
SPF).  For the rest, only the relay's IP address can be a meaningful
automated feedback target, AFAICS.  But then we'd also need to tell
network providers what to do when they receive ARF messages at their
abuse-mailboxes, since they are neither authors nor senders.

> That's really an engineering and operational decision. If I only
> receive two or three ARF reports a day, and my MUA or ticketing
> system can display them correctly, then handling them as part of my
> usual customer support / abuse desk workflow is likely to be more
> reliable (and much less expensive to implement) than an automated
> system. An ARF based FBL is still of value to me, but automating it
> would be a poor decision.

How about

  3.  Mail operators are urged to utilize a system that understands
      ARF messages and displays and processes them adequately, along
      the guidelines of Section 4 of [RFC6449].  Automated systems
      SHOULD receive and process these reports as discussed in
      Section 4.4 of [RFC6449].
?

> I don't think that experience purely of sending unsolicited ARF
> reports can really lead to the claim that all recipients handle
> them well - as the recipients that *don't* handle them well are
> simply going to discard or ignore the report, rather than trying to
> work out why it was sent.

Senders of unsolicited reports need some kind of acknowledgment,
sometimes, especially from new consumers that are not yet well
characterized.  Not boilerplate, but a few crisply handwritten lines
that help to decide whether to continue sending, or stop for the time
being.  Would such actions then turn the FBL into a "solicited" one?

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Jan 2012 10:22:29 -0800

As participant:

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Atkins
> Sent: Wednesday, January 11, 2012 7:28 PM
> To: Message Abuse Report Format working group
> Subject: [marf] Some thoughts on draft-ietf-marf-as-02
> 
> [...]
> reduce the efficiency of an abuse desk. Let's just not mention reports
> of mail scored high by spam filters, and quietly leave it as part of
> "anything else the sender considers".

I agree, and actually I agree enough that I would advocate for addition of a SHOULD NOT.  The question is one of wording.  Perhaps:

A report generator SHOULD NOT generate automated reports based on inline content analysis tools that apply subjective rules.  This can cause reports that, because of their subjective nature, are not actionable by report receivers, which wastes valuable operator time in processing them.

> The DKIM signer of a message is definitely a good target for solicited
> reports as part of a feedback loop. For unsolicited reports it's much
> less clear that that's the case. There are some mechanical reasons -
> the d= value is an identifier for reputation, not a domain intended to
> receive email, so we don't want to imply that
> [email-masked] is automatically a good place to report
> mail that's signed with d=transactional.blighty.com. In a broader
> sense, though, it's likely that the DKIM signing domain is the author
> of the mail - and if the mail is objectionable, they're often the least
> useful place to notify. In the case of mail sent through an ESP, the
> DKIM d= domain is likely to end up at the customer, rather than at the
> ESP where it's more likely to be handled correctly.

This is true, but I don't think that means the advice at the bottom of Section 5 is wrong, since it only suggests the DKIM-verified domain is a "reasonable candidate".  Perhaps the right thing to do is describe cases where that'll be a good guess, but a wrong one.

> unsolicited reports. Identifying an appropriate recipient for a report
> is *hard* to do automatically, while "shotgunning" reports to multiple
> email addresses because one of them might be appropriate is unhelpful.
> Sending them in a format that's intended primarily to be machine-
> readable and handled automatically by the entity receiving the reports
> leads to the problem of knowing what email address at what entity is
> intended to handle machine-readable reports. There's no standard for
> it, and there's a risk that those machine-readable reports will be
> discarded or ignored if they're sent to role addresses chosen at
> random.
> 
> Section 8 discusses this issue again, and somewhat better. Would it
> make sense to move the mention of DKIM to section 8, where it would be
> in a better context?

Yes, that seems reasonable.

I also think Section 8 should start at "abuse@" (referenced in Section 7) as being the only reasonable default.

> receive two or three ARF reports a day, and my MUA or ticketing system
> can display them correctly, then handling them as part of my usual
> customer support / abuse desk workflow is likely to be more reliable
> (and much less expensive to implement) than an automated system. An ARF
> based FBL is still of value to me, but automating it would be a poor
> decision. RFC 6449 Section 4.4 says pretty much that. I think we'd be
> better just pointing people at that section of RFC 6449, rather than
> saying that automation is a SHOULD.

Works for me.

> Is authentication failure abusive? If it is, we should say why (and in
> what context). If not, it shouldn't be in this document.

There are cases where one might think it is, such as ADSP violations for sites that put a lot of faith in it, but it's not universal.  Additional detail might be helpful there.

> handle ARF reports as well as they would handle plain text.

I imagine the text there envisions a system that handles in an automated fashion anything that it understands, and leaves the rest to operators to figure out.

> Unexpected multipart/* formats (such as the multipart/report type used
> by ARF) aren't necessarily going to be handled well even by MIME aware
> MUAs. Similarly, message/feedback-report isn't necessarily going to be
> handled well by MIME aware MUAs.

I don't think that paragraph means to include MUAs in the agents it's discussing, but rather is referring to automated systems.  Perhaps that should be made clearer here.

> So... I don't think it's valid to say that recipient systems that are not
> advertised to accept ARF will render ARF format messages in a way that
> will lead to them being handled. At the very least, anyone sending
> unsolicited messages in ARF format must assume that recipients will not
> be able to see the ARF metadata, and should include all information
> needed in human readable format in the first, text/plain section of the
> report. Also, they should ensure that the report is readable when
> viewed as plain text, to give low end ticketing systems as much help as
> possible - that might includes advice on appropriate MIME encoding and
> choice of character sets and so on. And they should be aware that the
> report may be discarded or ignored due to it's format.

I wouldn't object to including this advice while avoiding use of normative language.  That is, I wouldn't want to say "SHOULD put all the useful data in the text/plain part of an ARF report", because then why bother at all?

-MSK

---

From: Steve Atkins <[email-masked]>
Date: Mon, 23 Jan 2012 11:09:52 -0800

> A report generator SHOULD NOT generate automated reports based on inline content analysis tools that apply subjective rules.  This can cause reports that, because of their subjective nature, are no…

This does overlap with malware detection, though. Malware detection is based on subjective heuristics, and does have occasional false positives, but I think a report of an IP address sending malware would be considered actionable by an ISP abuse desk where a report of it sending "spam" based on subjective heuristics wouldn't.

I'm not sure what the distinction is other than "likely to be actionable".

> This is true, but I don't think that means the advice at the bottom of Section 5 is wrong, since it only suggests the DKIM-verified domain is a "reasonable candidate".  

It's not wrong, but it's a bit misleading. Calling out a DKIM-verified domain in particular as a good candidate suggests it's one that should be used, even though it's often going to be a bad candidate and the heuristics needed to work out whether it's appropriate or not aren't trivial.

> Perhaps the right thing to do is describe cases where that'll be a good guess, but a wrong one.

Perhaps. I think moving it to section 8 helps resolve the problem, as the context is there.

> I also think Section 8 should start at "abuse@" (referenced in Section 7) as being the only reasonable default.

Probably, yes. That leaves open the question of "which domain", though. The existing wording is sensible, but very vague on implementation details (sensibly - it's something tricky to do well, and non-trivial to even do non-abusively). The more we try and get into details there, the more we're going to hit problems.

> There are cases where one might think it is, such as ADSP violations for sites that put a lot of faith in it, but it's not universal.  Additional detail might be helpful there.

Yup. It's not obvious which cases this means, so we probably need more detail if we're going to mention it.

(Does this overlap with draft-ietf-marf-dkim-reporting and draft-ietf-marf-spf-reporting?)

> I imagine the text there envisions a system that handles in an automated fashion anything that it understands, and leaves the rest to operators to figure out.

Operators who are using typical MUAs or low-end CRM systems will have difficulty handling ARF reports at all, and may only have convenient access to the text/plain part of the message. If the intent is that the report is acted on, it has to be able to be read and acted on by those operators as well as by automation.

> I don't think that paragraph means to include MUAs in the agents it's discussing, but rather is referring to automated systems.  Perhaps that should be made clearer here.

If you're sending solicited reports that's a sensible distinction. I'm not sure it is in this context.

> I wouldn't object to including this advice while avoiding use of normative language.  That is, I wouldn't want to say "SHOULD put all the useful data in the text/plain part of an ARF report", becau…

You're sending unsolicited reports to people whose capabilities you have fairly limited information about. You would like them to be able to process that report using whatever system you happen to end up being delivered to. That includes ARF-aware full automation in some cases, and it includes very basic web-based MUAs (ticketing systems) in other cases.

If you want the report to be easily processed by the ARF-aware automation then all the information needs to be available as ARF metadata. If you want the report to be easily processed by an MUA then it needs to be easily readable using a text/plain MUA.

If you want it to be easily processed by both types of recipient, and you don't know which you're sending it to, then it needs to both have ARF metadata and be easily readable using a text/plain MUA.

Anyone sending ARF-only reports with a non-actionable plain text part today is making the same sort of decision as someone who sent text/html-only email a decade ago. It's great for recipients who can read it, but many can't. That doesn't mean that we need normative language, but making it clear that if you're sending unsolicited reports that are not readable by a text/plain level MUA you may well be wasting your time and, more importantly, the time of the recipient of your report is something we should probably mention.

"Senders SHOULD NOT assume that the recipient of an unsolicited report is able to make full use of an ARF format report. Unless the sender has reason to believe that all recipients of an unsolicited report can conveniently handle ARF format then they SHOULD ensure that the report is clear and contains enough information to be handled appropriately when viewed in a typical MUA or ticketing system."

There's another can of worms - if we're talking about sending unsolicited email to addresses guessed at via heuristics, do we also want to talk about "MUST provide an easy way to opt-out of all future reports"?

Cheers,
  Steve

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Jan 2012 11:24:12 -0800

> Sent: Monday, January 23, 2012 11:10 AM
> Subject: Re: [marf] Some thoughts on draft-ietf-marf-as-02
> 
> > A report generator SHOULD NOT generate automated reports based on
> > inline content analysis tools that apply subjective rules.  This can
> > cause reports that, because of their subjective nature, are not
> > actionable by report receivers, which wastes valuable operator time in
> > processing them.
> 
> This does overlap with malware detection, though. Malware detection is
> based on subjective heuristics, and does have occasional false
> positives, but I think a report of an IP address sending malware would
> be considered actionable by an ISP abuse desk where a report of it
> sending "spam" based on subjective heuristics wouldn't.
> 

Ah, true.  But reducing this to just talk about "likely to be actionable" fails to point out the very distinction you've made, which I think is a good one.

How about:

OLD

   However, it is inadvisable to generate automated reports based on
   inline content analysis tools that apply subjective evaluation rules.
   This can cause reports that, because of their subjective nature, are
   not actionable by report receivers, which wastes valuable operator
   time in processing them.

NEW

   However, it is inadvisable to generate automated reports based on
   inline content analysis tools that apply subjective evaluation rules,
   such as filters that attempt to classify spam or phishing.
   This can cause reports that, because of their subjective nature, are
   not actionable by report receivers, which wastes valuable operator
   time in processing them.  By contrast, virus detection reports are
   far more likely to be actionable.

> > This is true, but I don't think that means the advice at the bottom
> > of Section 5 is wrong, since it only suggests the DKIM-verified domain
> > is a "reasonable candidate".
> 
> It's not wrong, but it's a bit misleading. Calling out a DKIM-verified
> domain in particular as a good candidate suggests it's one that should
> be used, even though it's often going to be a bad candidate and the
> heuristics needed to work out whether it's appropriate or not aren't
> trivial.
> 
> > I also think Section 8 should start at "abuse@" (referenced in
> > Section 7) as being the only reasonable default.
> 
> Probably, yes. That leaves open the question of "which domain", though.
> The existing wording is sensible, but very vague on implementation
> details (sensibly - it's something tricky to do well, and non-trivial
> to even do non-abusively). The more we try and get into details there,
> the more we're going to hit problems.

Have a look at -03 (now posted) and see what you think.

> >> Is authentication failure abusive? If it is, we should say why (and
> >> in what context). If not, it shouldn't be in this document.
> >
> > There are cases where one might think it is, such as ADSP violations
> > for sites that put a lot of faith in it, but it's not universal.
> > Additional detail might be helpful there.
> 

No, I don't think so.  (And I think we should avoid doing so.)

> >> handle anything other than plain text particularly well. They will
> >> not handle ARF reports as well as they would handle plain text.
> >
> > I imagine the text there envisions a system that handles in an
> > automated fashion anything that it understands, and leaves the rest to
> > operators to figure out.
> 
> Operators who are using typical MUAs or low-end CRM systems will have
> difficulty handling ARF reports at all, and may only have convenient
> access to the text/plain part of the message. If the intent is that the
> report is acted on, it has to be able to be read and acted on by those
> operators as well as by automation.

Have a look at -03 and see what you think.

> There's another can of worms - if we're talking about sending
> unsolicited email to addresses guessed at via heuristics, do we also
> want to talk about "MUST provide an easy way to opt-out of all future
> reports"?

Yes, that's probably good to include as well.  Absent objection, I'll add that in -04.

-MSK