Subject: [spfbis] Review of draft-ietf-spfbis-experiment-05

From: Barry Leiba <[email-masked]>
Date: Sun, 22 Apr 2012 16:58:23 -0400

Here's a review of experiment-05.  Some of these comments are purely
editorial, but some address the tone of the document, the tendency to
deprecate Sender ID, and general issues of pronouncements and opinions.
 Please seriously consider addressing those: I don't think such
pronouncements, opinions, and bias are in place in this document (see my
first comment below).

-- Abstract --

OLD
   After six years, sufficient experience and evidence have been
   collected that the experiment thus created can be considered
   concluded, and a single protocol can be advanced.  This document
   presents those findings.

I would prefer that this document avoid making pronouncements and stating
these kinds of opinions ("a single protocol can be advanced").  The point
of this draft is to document results.  The actual SPFbis doc will do the
advancement of SPF.  It would be better if statements about the fate of
Sender ID be left for other granfalloons.  I suggest ending the sentence
with "concluded".

Note that RFC 4406 uses "Sender ID", with a space and no dash (except in
the IESG Note).  Please use that orthography consistently here.

-- Introduction --

OLD
   The two protocols made use of this policy
   statement and some specific (but different) logic

NEW (clarity, even though "same" is in the next paragraph)
   The two protocols made use of this same policy
   statement and some specific (but different) logic

OLD
   Due to the absence of consensus behind one or the other, and because
   Sender-ID supported use of the same policy statement defined by SPF,
   the IESG at the time was concerned that an implementation of
   Sender-ID might erroneously apply that statement to a message and,
   depending on selected recipient actions, could improperly interfere
   with message delivery.

This seems to have a significant SPF bias.  May I suggest this?:

NEW
   Due to the absence of consensus behind one or the other and because
   SPF and Sender-ID supported use of the same policy statement with
   different semantics, the IESG at the time was concerned that
   implementations of SPF or Sender-ID might erroneously apply a
   statement that had been published with the semantics of the other,
   and, depending on selected recipient actions, could improperly
   interfere with message delivery.

OLD
   It is important to note that this document makes no direct technical
   comparison of the two protocols in terms of correctness, weaknesses,
   or use case coverage.  The email community at large has already done
   that.

I suggest "has already done that through its deployment choices."
 Otherwise, one might wonder where the comparison that's been done is
documented.

-- Analysis --

OLD
   2.  Of the records retrieved, fewer than 3% requested processing of
       messages using the PRA algorithm, which was an essential part of
       Sender-ID.

I suggest "which is an essential part of Sender ID."  There seems no reason
for past tense.

OLD
   3.  Although the two mechanisms often used different email addresses
       as the subject being evaluated, no data collected showed any
       substantial operational benefit (e.g., cheaper processing,
       improved accuracy) to using Sender-ID over SPF.

I suggest "to using either mechanism over the other."

-- Conclusions --

OLD
   It is standard procedure within the IETF to document as standard
   those protocols and practices that have come into sufficient common
   use as to become part of the basic infrastructure.

This seems unnecessary and out of place.  I suggest removing it.

OLD
   In light of the this and the analysis in the previous section, the
   following conclusions are supported:
NEW
   In light of the analysis in the previous section, the
   following conclusions are supported:

OLD
   1.  The experiment comprising the series of RFCs defining the
       SUBMITTER SMTP extension, the Sender-ID mechanism, the Purported
       Responsible address algorithm, and SPF, should be considered
       concluded.

Citations for all have appeared earlier, but I think it would be helpful to
put RFC citations for them here... consider it an excutive summary.  This
is one of the problems with the choice of non-RFC-based citation tags.  I
suggest this, which uses parentheses instead of citations:

NEW
   1.  The experiment comprising the series of RFCs defining the
       SUBMITTER SMTP extension (RFC 4405), the Sender-ID mechanism
      (RFC 4406), the Purported Responsible address algorithm (RFC 4407),
       and SPF (RFC 4408), should be considered concluded.

OLD
   3.  The absence of significant adoption of the [SUBMITTER] extension,
       [SENDER-ID], and [PRA], indicates that there is not a strong
       community prepared to develop those mechanisms beyond
       experimental status.

I would MUCH rather see this reported factually, without further opinion.
 Something like this:

NEW
   3.  The absence of significant adoption of the [SUBMITTER] extension,
       [SENDER-ID], and [PRA], indicates that there is not a strong
       community deploying and using these protocols.

OLD
   4.  Continued widespread use of [SPF] indicates it is worthy of
       consideration for the Standards Track.

I suggest this:
NEW
   4.  [SPF] has widespread implementation and deployment, comparable
       to that of many Standards Track protocols.

-- Security Considerations --

OLD
   This document contains information for the community, akin to an
   implementation report, and does not introduce any new security
   concerns.  Its implications could, in fact, resolve some.

That last sentence seems odd.  Unless you want to be specific about it, I
suggest removing it.

-- Informative References --

We can always argue whether an Informational document, by its nature, can
have "normative" references.  Still, in the sense that we use it here,
references that are central to the understanding of the document at hand
are usually considered to be normative.  DNS is arguable, but I'd say no.
 I think the references to PRA, SENDER-ID, SPF, and SUBMITTER are normative.

-- Appendix A.  Experiences Developing SPF --

This is really about the RR type issue, not about SPF in general.  I
suggest saying that in the section title.  Maybe, "Background on the RR
Type Issue", or some such.

--
Barry

---

From: Scott Kitterman <[email-masked]>
Date: Sun, 22 Apr 2012 17:12:41 -0400

>    interfere with message delivery.

The current text is correct.  The issue was Sender ID reusing SPF policy 
statements (of which there were a large number published before this change 
was introduced into Sender ID) not a generic "they both use the same data".

I think the other changes you proposed are all good and make the document 
better for its intended purpose.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 22 Apr 2012 21:27:59 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Scott Kitterman
> Sent: Sunday, April 22, 2012 2:13 PM
> To: [email-masked]
> Subject: Re: [spfbis] Review of draft-ietf-spfbis-experiment-05
> 
> The current text is correct.  The issue was Sender ID reusing SPF
> policy statements (of which there were a large number published before
> this change was introduced into Sender ID) not a generic "they both use
> the same data".

Is it necessary to be that precise in this document's context?

> I think the other changes you proposed are all good and make the
> document better for its intended purpose.

+1.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Sun, 22 Apr 2012 17:38:14 -0400

On Sunday, April 22, 2012 09:27:59 PM Murray S. Kucherawy wrote:
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Scott Kitterman Sent: Sunday, April 22, 2012 2:13 PM
> > To: [email-masked]

It was an important enough issue at the time that IESG [1] and IAB [2] appeals 
on the practice were filed.  I think pretending that this was an even handed 
data reuse issue is flat out wrong.  In short, on this issue, yes.

Scott K

[1] http://www.ietf.org/iesg/appeal/mehnle-2005-08-25.txt
[2] http://www.iab.org/appeals/2006-2/appeal-against-iesg-decision-by-julian-
mehnle-8-february-2006/

P.S. This is unrelated to the question of hard feelings about MARID, which we 
are supposed to avoid.  This is related to post-MARID design decisions made by 
Microsoft.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Apr 2012 03:03:10 +0000

> Sent: Sunday, April 22, 2012 2:38 PM
> To: [email-masked]
> It was an important enough issue at the time that IESG [1] and IAB [2]
> appeals on the practice were filed.  I think pretending that this was
> an even handed data reuse issue is flat out wrong.  In short, on this
> issue, yes.

How does it further frame or resolve the investigation into which one has enjoyed more uptake than the other?  At this point, that's the filter it appears we should be using.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Sun, 22 Apr 2012 23:28:41 -0400

On Monday, April 23, 2012 03:03:10 AM Murray S. Kucherawy wrote:
> > Of Scott Kitterman Sent: Sunday, April 22, 2012 2:38 PM
> > To: [email-masked]
> How does it further frame or resolve the investigation into which one has
> enjoyed more uptake than the other?  At this point, that's the filter it
> appears we should be using.

Since Sender ID makes use of SPF records, it is possible to make the claim 
that lack of publishing Sender ID specific records is not a sign of disinterest 
in Sender ID, but in fact a sign that few senders find problems with publishing 
an SPF records for both.

Since a large number of domains (I don't have a number I can support, but it's 
clear that SPF was widely adopted before the Sender ID RFCs were published) 
published records before they were used for both protocols, I think it's an 
important point that all those domains were opt-ed in to the Sender ID 
experiment involuntarily.  

There are any number of ways we could re-write history to make things more 
palatable, but I think it's important that what we say be correct.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Apr 2012 03:40:21 +0000

Comments inline (alas):

From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
Sent: Sunday, April 22, 2012 1:58 PM
To: [email-masked]
Subject: [spfbis] Review of draft-ietf-spfbis-experiment-05

OLD

[MSK: I don't think that's correct.  Sender ID has a substantially higher processing cost given that it requires accepting the DATA part of the message and has an obviously higher cost to extract the various identifiers the PRA algorithm considers.  SPF, purely compute-wise, is cheaper.  However, their accuracy is comparable.  If we want to be clear, we can say their accuracies are about the same, but SPF is operationally cheaper.]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Apr 2012 03:51:45 +0000

> Sent: Sunday, April 22, 2012 8:29 PM
> To: [email-masked]
> Since Sender ID makes use of SPF records, it is possible to make the
> claim that lack of publishing Sender ID specific records is not a sign
> of disinterest in Sender ID, but in fact a sign that few senders find
> problems with publishing an SPF records for both.
> 
> Since a large number of domains (I don't have a number I can support,
> but it's clear that SPF was widely adopted before the Sender ID RFCs
> were published) published records before they were used for both
> protocols, I think it's an important point that all those domains were
> opt-ed in to the Sender ID experiment involuntarily.

Taken together, this says early SPF adopters became unwitting participants in the Sender ID experiment, but it didn't make a difference in the end because it didn't cause any problems.  If that's the case, then why is highlighting who was "here" first (in terms of the DNS mechanism) an important thing?

> There are any number of ways we could re-write history to make things
> more palatable, but I think it's important that what we say be correct.

I think there's a difference between deliberate historical revisionism and an effort to avoid details that don't appear to be relevant to the matter at hand.  The actual history is well-documented if the curious want to go unearth it.  We aren't trying to hide anything; we're just trying to (and need to) avoid showing bias.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 00:02:55 -0400

On Monday, April 23, 2012 03:51:45 AM Murray S. Kucherawy wrote:
> > Of Scott Kitterman Sent: Sunday, April 22, 2012 8:29 PM
> > To: [email-masked]
> Taken together, this says early SPF adopters became unwitting participants
> in the Sender ID experiment, but it didn't make a difference in the end
> because it didn't cause any problems.  If that's the case, then why is
> highlighting who was "here" first (in terms of the DNS mechanism) an
> important thing?
> I think there's a difference between deliberate historical revisionism and
> an effort to avoid details that don't appear to be relevant to the matter
> at hand.  The actual history is well-documented if the curious want to go
> unearth it.  We aren't trying to hide anything; we're just trying to (and
> need to) avoid showing bias.

Facts are not bias.  Wanting to sweep them under the rug is bias.  What I am 
asking for is the exact opposite of bias.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 00:07:09 -0400

On Monday, April 23, 2012 03:40:21 AM Murray S. Kucherawy wrote:
> Comments inline (alas):
> 
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Barry Leiba Sent: Sunday, April 22, 2012 1:58 PM
> To: [email-masked]
> [MSK: I don't think that's correct.  Sender ID has a substantially higher
> processing cost given that it requires accepting the DATA part of the
> message and has an obviously higher cost to extract the various identifiers
> the PRA algorithm considers.  SPF, purely compute-wise, is cheaper. 
> However, their accuracy is comparable.  If we want to be clear, we can say
> their accuracies are about the same, but SPF is operationally cheaper.]

SPF is also not fundamentally incompatible with existing internet standards in 
a way that prevents it from being advanced, documented in a different appeal 
[1].  One very good answer to "Why advance SPF and not Sender ID?" is that 
there are standards issues with Sender ID that don't exist with SPF.  Even if 
they had equivalent operational complexity/cost, that would, IMO, be enough to 
be determinant.

Scott K

[1] http://www.ietf.org/iesg/appeal/leibzon-2005-08-29.txt

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Apr 2012 04:08:33 +0000

> Sent: Sunday, April 22, 2012 9:03 PM
> To: [email-masked]
> Facts are not bias.  Wanting to sweep them under the rug is bias.  What
> I am asking for is the exact opposite of bias.

At this point I think I'll leave it to working group consensus.  I don't have any desire, nor I believe does Barry, to obscure facts that are relevant to the question.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 00:25:58 -0400

On Monday, April 23, 2012 04:08:33 AM Murray S. Kucherawy wrote:
> I don't have any desire, nor I believe does Barry, to obscure facts that are
> relevant to the question.

I don't believe you or Barry does either.

Scott K

---

From: Barry Leiba <[email-masked]>
Date: Mon, 23 Apr 2012 00:52:29 -0400

Ah... Then the whole item 3 needs to be reworded to say what is actually
the case with respect to processing cost and accuracy.  If there actually
is a benefit to SPF here, say so.

Barry

On Sunday, April 22, 2012, Murray S. Kucherawy wrote:

>  Comments inline (alas):****
>
> ** **
>
> *From:* [email-masked] <javascript:_e({}, 'cvml',
> [email-masked]');> [mailto:[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>]
> *On Behalf Of *Barry Leiba
> *Sent:* Sunday, April 22, 2012 1:58 PM
> *To:* [email-masked] <javascript:_e({}, 'cvml', [email-masked]');>
> *Subject:* [spfbis] Review of draft-ietf-spfbis-experiment-05****
>
> ** **
>
> OLD****
>
>    3.  Although the two mechanisms often used different email addresses***
> *
>
>        as the subject being evaluated, no data collected showed any****
>
>        substantial operational benefit (e.g., cheaper processing,****
>
>        improved accuracy) to using Sender-ID over SPF.****
>
> ** **
>
> I suggest "to using either mechanism over the other."****
>
>
> [MSK: I don’t think that’s correct.  Sender ID has a substantially higher
> their accuracies are about the same, but SPF is operationally cheaper.]***
> *
>
> ** **
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Apr 2012 05:02:51 +0000

Actually, I don't think we need to get into the benefits of one over the other.  We're merely measuring and reporting on adoption since publication, not the "why" part.  That's already covered in the "people voted with their feet" bit.

So the net change, I think, is to remove the mention of cheaper processing, and just say the accuracy of the two is comparable.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
Sent: Sunday, April 22, 2012 9:52 PM
To: Murray S. Kucherawy
Cc: [email-masked]
Subject: Re: [spfbis] Review of draft-ietf-spfbis-experiment-05

Ah... Then the whole item 3 needs to be reworded to say what is actually the case with respect to processing cost and accuracy.  If there actually is a benefit to SPF here, say so.

---

From: John Leslie <[email-masked]>
Date: Mon, 23 Apr 2012 06:07:52 -0400

Barry Leiba <[email-masked]> wrote:
>...

   Both of these try to put words in the mouth of an IESG of many years
ago (before either Barry or I was scribing). I advise against doing so.
Please, consider the actual IESG statement:

--------------------------------cut here--------------------------------
Date: Wed, 22 Sep 2004 15:52:55 -0400
From: The IESG <[email-masked]>
Subject: WG Action: Conclusion of MTA Authorization Records in DNS (marid)
To: [email-masked]
Cc: [email-masked],
        "Marshall T. Rose" <[email-masked]>,
        Andrew Newton <[email-masked]>

The MTA Authorization Records in DNS (MARID) working group in the
Applications Area has concluded.
   
The IESG contact persons are Ted Hardie and Scott Hollenbeck.
   
The mailing list will remain active.
   
After an assessment of the current state of the MARID working group,
its charter, and its milestones, the working group chairs and Area
Advisor concluded that the MARID working group should be terminated.
   
The group was originally chartered with a very tight time frame, with
the expectation that a focused group of engineers would be able to
produce in relatively short order a standard in the area of
DNS-stored policies related to and accessible by MTAs. The group has
had no lack of energy. From the outset, however, the working group
participants have had fundamental disagreements on the nature of the
record to be provided and the mechanism by which it would be checked.
Technical discussion of the merits of these mechanisms has not swayed
their proponents, and what data is available on existing deployments
has not made one choice obviously superior. Each represents
trade-offs, and the working group has not succeeded in establishing
which trade-offs are the most appropriate for this purpose. These
assessments have been difficult in part because they have been moved
out of the realm of pure engineering by the need to evaluate IPR and
licensing related to at least one proposal in the light of a variety
of licenses associated with the deployed base of MTAs.
   
Efforts to reach consensus by compromise and by inclusion have been
attempted on multiple occasions. Despite early hopes of success
after each such attempt, post-facto recycling of technical issues
which these efforts should have closed has shown that the group
remains divided on very basic issues. The working group chairs and
Area Advisor are agreed that the working group has no immediate
prospect of achieving its primary milestone:

Aug 04 Submit working group document on MTA Authorization Record in DNS to PS
--------------------------------cut here--------------------------------

   Fundamentally, IMHO, the IESG of 2004 considered MARID a failed WG.

--------------------------------cut here--------------------------------
Rather than spin in place, the working group chairs and Area Advisor believe
that the best way forward is experimentation with multiple proposals
and a subsequent review of deployment experience. The working group
chairs and Area Advisor intend to ask that the editors of existing
working group drafts put forward their documents as non-working group
submissions for Experimental RFC status. Given the importance of the
world-wide email and DNS systems, it is critical that IETF-sponsored
experimental proposals likely to see broad deployment contain no
mechanisms that would have deleterious effects on the overall system.
The Area Directors intend, therefore, to request that the
experimental proposals be reviewed by a focused technology
directorate. This review group has not yet been formed but, as with
all directorates, its membership will be publicly listed at
http://www.ietf.org/u/ietfchair/directorates.html once it has been
constituted.

Concluding a group without it having achieved its goals is never a
pleasant prospect, and it is always tempting to believe that just a
small amount of additional time and energy will cause consensus to
emerge. After careful consideration, however, the working group
chairs and area advisor have concluded that such energy would be
better spent on gathering deployment experience.
--------------------------------cut here--------------------------------

   The WG list remained quite active for a while, among other things
speaking unkindly of the "directorate" mentioned above. In due course
Individual I-Ds were submitted which became RFCs 4405-4408.

   We should be careful in saying anything in spfbis-experiment which
puts words beyond the above into the mouth of the 2004 IESG. I believe
at the time the above message was sent, the IESG believed that a limited
group of reviewers (the "directorate") would be able to enforce sanity
on a discussion which seemed to the 2004 IESG to be religious posturing.
(That did not happen, of course.)

   But I don't believe my interpretation belongs in spfbis-experiment,
either. I don't believe either "OLD" or "NEW" from Barry's email is
appropriate. I could suggest:
" 
" Since the MARID WG was closed without reaching consensus on either,
" the 2004 IESG left it to each sub-group to describe the usage they
" proposed in Experimental RFCs, with evaluation to be done by some
" other body after a period of experimentation.
"
" This document will discuss the effects of applying different semantics
" to the same DNS record but makes no claim to tell the whole story.

   I'm not at all attached to that particular wording: I just believe
that we need to be careful in attributing intent to the 2004 IESG.
If, OTOH, you get Ted Hardie's approval to either of those statments,
it'd be OK with me.

--
John Leslie <[email-masked]>

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 06:50:06 -0400

Snip.

That's all true, but not, I think relevant to the question of record reuse. There was consensus in MARID that reuse of SPF records for Sender ID PRA assessments was technically inappropriate.

I don't think it's correct to apply words about the MARID shutdown to changes that were made later.

Scott K

---

From: John Leslie <[email-masked]>
Date: Mon, 23 Apr 2012 07:19:30 -0400

(I see by the threading that Scott replied to my email...)

Scott Kitterman <[email-masked]> wrote:
> 
> Snip.
> 
> That's all true, but not, I think relevant to the question of record
> reuse.

   Perhaps Scott might say _what_ he thinks not relevant. Mostly, I was
quoting the IESG post closing the MARID WG.

> There was consensus in MARID that reuse of SPF records for Sender ID
> PRA assessments was technically inappropriate.

   Um... No.

   There was a widespread belief among MARID participants that Sender ID
use of already published SPF records was inappropriate, but it was never
a WG consensus.

> changes that were made later.

   Scott loses me here. I agree that the Sender ID RFCs were not exactly
what was discussed before MARID closed -- neither was the SPF RFC...
But I was only criticizing wording which purported to state an IESG
_intent_.

   The IESG posting I quoted (in its entirety) is by definition the best
source for IESG intent.

   (Barry and I know firsthand how IESG statements like this are
generated.)

   We also have, of course, the IESG statement in the RFCs in question,
which also represents an IESG consensus (though IMHO probably less work
went into it than into closing a very-active WG):

--------------------------------cut here--------------------------------
RFC 4405          SMTP Responsible Submitter Extension        April 2006
...
   The following documents  (RFC 4405, RFC 4406, RFC 4407, and RFC 4408)
   are published simultaneously as Experimental RFCs, although there is
   no general technical consensus and efforts to reconcile the two
   approaches have failed.  As such, these documents have not received
   full IETF review and are published "AS-IS" to document the different
   approaches as they were considered in the MARID working group.

   The IESG takes no position about which approach is to be preferred
   and cautions the reader that there are serious open issues for each
   approach and concerns about using them in tandem.  The IESG believes
   that documenting the different approaches does less harm than not
   documenting them.

   Note that the Sender ID experiment may use DNS records that may have
   been created for the current SPF experiment or earlier versions in
   this set of experiments.  Depending on the content of the record,
   this may mean that Sender-ID heuristics would be applied incorrectly
   to a message.  Depending on the actions associated by the recipient
   with those heuristics, the message may not be delivered or may be
   discarded on receipt.

   Participants relying on Sender ID experiment DNS records are warned
   that they may lose valid messages in this set of circumstances.
   Participants publishing SPF experiment DNS records should consider
   the advice given in section 3.4 of RFC 4406 and may wish to publish
   both v=spf1 and spf2.0 records to avoid the conflict.

   Participants in the Sender-ID experiment need to be aware that the
   way Resent-* header fields are used will result in failure to receive
   legitimate email when interacting with standards-compliant systems
   (specifically automatic forwarders which comply with the standards by
   not adding Resent-* headers, and systems which comply with RFC 822
   but have not yet implemented RFC 2822 Resent-* semantics).  It would
   be inappropriate to advance Sender-ID on the standards track without
   resolving this interoperability problem.

   The community is invited to observe the success or failure of the two
   approaches during the two years following publication, in order that
   a community consensus can be reached in the future.
--------------------------------cut here--------------------------------

   (Perhaps Scott sees some IESG "changes" in this; but I don't...)

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 07:49:52 -0400

>    (I see by the threading that Scott replied to my email...)

Sorry about that.

>    (Perhaps Scott sees some IESG "changes" in this; but I don't...)

It is likely that I am not sufficiently separating myself from the issue since I 
do have strong feelings on the issues of both Microsoft's and the IESG's roles 
in all this, so after this I'll drop it.  I will, however, make one more 
attempt.

Here is a quote from the IESG statement:

>    this set of experiments.  

The IESG did not say something like "SPF and Sender-ID supported use of the 
same policy statement with different semantics".  That sounds much closer to 
"Sender-ID supported use of the same policy statement defined by SPF" to me.

If the group wants to stick closely to what the IESG at the time said, then, 

>    interfere with message delivery.

Murray's text (labeled "OLD" here) very closely reflects what the IESG said and 
not any kind of bias, but whatever the group wants is fine by me.

Scott K

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 Apr 2012 07:52:55 -0400

Scott Kitterman wrote:
> Snip.
> 
> That's all true, but not, I think relevant to the question of record reuse. There was consensus 
> in MARID that reuse of SPF records for Sender ID PRA assessments was technically inappropriate.

Inappropriate in what way was not always agreed.

> I don't think it's correct to apply words about the MARID shutdown to changes that were made later.

As an active participant before marid, during marid and after marid, 
and maybe because my perspective of how all these ideas can work 
together, my assessment was much different. I didn't see precisely 
what the SPF v1.0 group were concern about, but then again at the 
time, I still had a hard time seeing how the PRA would address the #1 
key concern where SPF broke down - path independent.  After all, that 
is what gave birth to the other proposals and it was the MARKETING 
reason presented in the news rags.

Putting aside all subjective politics, the technical problems when CEP 
was presented with its PRA concept:

   - XML format for DNS records,

   - PAYLOAD PRA defeated the high focus with SMTP IP and Envelope 
methods
     to address the accept bounce issue,

The PAYLOAD/PRA concerned was resolved with the SUBMITTER idea.  And I 
believe it was pretty clear we didn't want to add more DNS overhead 
with XML records and processing, it didn't seem this will help make it 
a standard and there was also high interest to consider the new thing 
- a new RR type. Considering, over 80% of the use cases did not 
require the PRA, IMO, it was technically appropriate to consider 
getting rid of the XML format and used a single source language to 
covered both.

The claim that there was potential for errors seems to be based on the 
idea that implementators and publishers would do it wrong - not that 
the protocols were incorrect because they were both technically 
distinct with a tag.

I didn't get that part of it, and FWIW, I felt that it could be 
integrated smoothly without problems.  Waste? Perhaps. Lower use 
cases? Perhaps.   But to me, it was more did PRA help (not cure) with 
the SPFv1.0 path problem that was the #1 criticism it had.

I just feel if the SPFBIS focus was on the integrated experiment 
question which would include resolving the DNS record conflict claims, 
then there would be a different set of interest.

-- 
Hector Santos, CTO
http://www.santronics.com
http://hector.wildcatblog.com

---

From: Dotzero <[email-masked]>
Date: Mon, 23 Apr 2012 08:57:59 -0400

> their accuracies are about the same, but SPF is operationally cheaper.]

Murray,

It is absolutely incorrect to say that their accuracies are about the
same. I can consistently game PRA to get a neutral outcome regardless
of what the originating domain publishes in it's record. If there is a
Sender field then that is the PRA per the spec. I cannot do the the
same for SPF.

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 23 Apr 2012 10:26:54 -0400

No hat.

> asking for is the exact opposite of bias.

This doesn't respond to Murray's question of relevance.

It might be that there were a large number of people unwittingly
pulled into an experiment when the publication decisions were made.
That does not help us in any way to determine the relative uptake of
the different technologies.

The document is about the results of the experiment, to the extent
there was one, and not about the conditions of the experiment as such.
Quite frankly, if we had to do an evaluation of this as an experiment,
we would have to criticise the experimental design in many ways, and
the accidental pollution of the sample base is the least of the
problems.  But that's not what we're trying to do.

So, without discussing the factuality of the claims, why are they
relevant?

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 23 Apr 2012 10:30:42 -0400

Moderator note.

On Mon, Apr 23, 2012 at 06:50:06AM -0400, Scott Kitterman wrote:

> reuse. There was consensus in MARID that reuse of SPF records for

We are not going to debate on this list whether there was or was not
consensus in MARID on anything.

Thank you.

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 23 Apr 2012 14:34:03 +0000

> From: Dotzero [mailto:[email-masked]]
> Sent: Monday, April 23, 2012 5:58 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]

Based on the data collected about actual live mail streams, SPF and Sender ID reach the same pass/fail conclusion about messages at least 80%, and as much as 95%, of the time.  Do you have different data?

We're specifically not doing a comparison of the weaknesses of the two.  We're only citing empirical data.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 10:51:49 -0400

On Monday, April 23, 2012 10:26:54 AM Andrew Sullivan wrote:
> > Facts are not bias.  Wanting to sweep them under the rug is bias.  What I
> > am asking for is the exact opposite of bias.
> 

I think it's highly relevant to how the data on usage is interpreted.

Scott K

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 23 Apr 2012 10:59:51 -0400

No hat.

> I think it's highly relevant to how the data on usage is interpreted.

How?  I don't see it.  If you're arguing for inclusion, the burden of
proof is on you.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 11:00:23 -0400

> consensus in MARID on anything.

As long as we equally don't debate what was in the IESG/directorate's minds 
when they shut down MARID, I think that's wonderful.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 11:00:56 -0400

> How?  I don't see it.  If you're arguing for inclusion, the burden of

No.  I'm arguing that the current text remain.

Scott K

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 23 Apr 2012 11:08:05 -0400

> when they shut down MARID, I think that's wonderful.

I fully agree, although we do have statements from the IESG.
Particularly, we have the IESG notes on the published documents, which
stated that there was no technical consensus.  Since those are notes
from the IESG, we can correctly read it as making a claim about the
judgement of the IESG at the time of publication.  But going any
further is a bad idea, I very strongly agree.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Dotzero <[email-masked]>
Date: Mon, 23 Apr 2012 11:13:35 -0400

> We're specifically not doing a comparison of the weaknesses of the two.  We're only citing empirical data.

You stated that their accuracies are comparable. Given the known
weakness of PRA (based on emperical data/testing), that is an
incorrect statement. The whole point of SPF (and presumably SIDF) is
to mitigate abuse. You have not provided complete details about the
dataset you are referring to so it is hard to draw detailed conlusions
regarding key points:

1) What percentage of the dataset does not have a Sender field?
2) For data points where there is a Sender field, what percentage of
those data points have a Sender field that is aligned with the Mail
>From (and conceivably From)?
3) What percentage of the messages were "abusive"? Would the mail
stream selected be a likely target for the particular type of abuse I
am pointing out? If not, why would you expect to see this particular
type of abuse?
4) To what extent is the dataset representative of the mail streams
that various types and sizes of receivers might receive? That is, is
the dataset truly representative or are there potential issues with
self selection, etc?

---

From: Barry Leiba <[email-masked]>
Date: Mon, 23 Apr 2012 12:13:56 -0400

> I'm arguing that the current text remain.

Scott is right that because what's there has rough consensus, he does not
need to establish consensus to leave it there.

That said, I think the current text ill-serves the document, so let me try
again.  How about this, which I think does not attempt to attribute
anything but the Experimental requirement to the old IESG:

OLD
   Due to the absence of consensus behind one or the other, and because
   Sender-ID supported use of the same policy statement defined by SPF,
   the IESG at the time was concerned that an implementation of
   Sender-ID might erroneously apply that statement to a message and,
   depending on selected recipient actions, could improperly interfere
   with message delivery.  As a result, the IESG required the
   publication of all of these documents as Experimental, and requested
   that the community observe deployment and operation of the protocols
   over a period of two years from the date of publication in order to
   determine a reasonable path forward.

NEW
   Consensus did not clearly support one protocol over the other, and
   there was significant concern that the two would conflict in some
   significant operational situations, interfering with message delivery.
   The IESG required the publication of all of these documents as
   Experimental, and requested that the community observe deployment
   and operation of the protocols over a period of two years from the date
   of publication in order to determine a reasonable path forward.

Does that work?  I think it says what it needs to say without trying to
either blame anyone or get into historical details that are not necessary
in this document.

Barry

---

From: Barry Leiba <[email-masked]>
Date: Mon, 23 Apr 2012 12:19:32 -0400

> Still, in the sense that we use it
> here, references that are central to the understanding of the document
> at hand are usually considered to be normative.  DNS is arguable, but
> I'd say no.  I think the references to PRA, SENDER-ID, SPF, and
> SUBMITTER are normative.

You think that SPF or Sender ID could function without the DNS
specification???  That is, what is it you think might be arguable about
classing it as a normative reference?

[Barry]  No.  I'm saying that you need to understand SPF, etc, to
understand this doc, but you probably don't need to understand DNS to
understand this doc.  Normativity isn't transitive.

That said, because there is talk in here about the RR types, I'm happy if
we also say that DNS is a normative reference.  I'm not happy to have the
RFC 440x docs NOT be normative.

Barry

---

From: Commerco WebMaster <[email-masked]>
Date: Mon, 23 Apr 2012 12:10:39 -0600

On 4/23/2012 10:13 AM, Barry Leiba wrote:
> Scott is right that because what's there has rough consensus, he does
> not need to establish consensus to leave it there.
>
> That said, I think the current text ill-serves the document, so let me
> try again.  How about this, which I think does not attempt to attribute
> either blame anyone or get into historical details that are not
> necessary in this document.
>
> Barry
>

New looks cleaner.  Would it be possible (and does it make sense) to add 
something like -

"Measurement data gathered for both the Sender-ID and SPF experiments 
indicate SPF adoption levels in the community as the preferred protocol, 
thus offering the reasonable path forward."

- to the end of the paragraph?

Alan M.

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 15:08:09 -0400

> either blame anyone or get into historical details that are not necessary

I think it is biased towards Sender ID by pretending an equivalence that is 
incorrect, but it's not worth arguing over.  Go ahead with it as far as I'm 
concerned.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 23 Apr 2012 15:45:58 -0400

On Monday, April 23, 2012 11:13:35 AM Dotzero wrote:
> > Based on the data collected about actual live mail streams, SPF and Sender
> > ID reach the same pass/fail conclusion about messages at least 80%, and
> > as much as 95%, of the time.  Do you have different data?
> > 

Perhaps it would be enough to say that there isn't any evidence of accuracy 
improvement associated with the additional complexity of Sender ID?

Scott K

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 Apr 2012 16:39:07 -0400

> either blame anyone or get into historical details that are not necessary

I believe another approach for the report is to restructure the 
introduction outlining:

   1) Restating the industry problem the protocols attempted address,
   2) Summarize their goals and approaches, and
   3) Summarize the concerns and conflicts.

and allow the collection 6+ years of deployment experiences show if 
the original problem(s) was addressed with the goals and approaches 
and what were the concerns and conflicts. I also would include insight 
if any new industry solutions altered the goals and approaches and 
perhaps help resolve some of the conflicts and concerns.  The latter 
point is not a focus of replacing, but how the deployment decisions 
was altered that may help explain the 6+ years deployment data.

If there was a "bias" for lack of a better term, it was the advocates 
for a backend only solution versus a more complete and integrated 
total solution with all mail system components involved.  Despite the 
extra complexity, this is what SenderID brought to the table as 
described in its intro:

    This document describes a mechanism such that receiving Mail Transfer
    Agents (MTAs), Mail Delivery Agents (MDAs), and/or Mail User Agents
    (MUAs) can recognize mail in the above category and take appropriate
    action.  For example, an MTA might refuse to accept a message, an MDA
    might discard a message rather than placing it into a mailbox, and an
    MUA might render that message in some distinctive fashion.

The conflict was not just cited DNS records between SPFv1 and SPFv2, 
but other considerations as well:

    - DNS record conflicts (SPFv1 vs SPFv2)
    - DNS optimization (considering new RR type)
    - PRA Payload concern (use of SUBMITTER)
    - Marking vs Rejection - both must have the same result/outcome 
for security reasons,
      - 4405 is more RFC2119 forceful with REJECT only than 4406, 4407 
and 4408.
    - Helping users where the backend was yet compliant (or just 
marking it)

---

From: Douglas Otis <[email-masked]>
Date: Mon, 23 Apr 2012 14:01:01 -0700

On 4/23/12 8:13 AM, Dotzero wrote:
> On Mon, Apr 23, 2012 at 10:34 AM, Murray S. Kucherawy<[email-masked]>  wrote:
Dear Mike,

Agreed.  Serious security concerns remain.  Publishing domain alignments 
of various message elements might better illustrate these concerns.

Both PRA and Mail parameters suffer similar weaknesses as neither are 
assured to be visible.  Unlike the PRA, the Mail parameter establishes 
the return-path, however SPF records lack any assurance which identity 
is intended to correspond with outbound server addresses.  Instances 
where PRA accommodates normal SMTP function (without misuse of Resent-* 
header fields) allows this scheme to be more compliant.  This compliance 
carries added risk as it requires application of similar out-of-band 
information necessary for making exceptions as those with the Mail 
parameters, although the Mail parameter offers less selection 
uncertainty and much less associated overhead.

The SPF HELO/EHLO option does not require use of out-of-band 
information, does not require exceptions, nor is there more than one 
possible identity within a message exchange.

While HELO/EHLO identity is recognized by SPF, SPF lacks a scope list 
making it clear whether use was intended exclusively for the HELO/EHLO 
to authenticate outbound servers, or to authorize third-party servers 
that might carry a domain's messages.   Source authentication using 
HELO/EHLO offers content assurances lacking with either the PRA or Mail 
parameters.  Additionally, only the HELO/EHLO identity is able to ensure 
messages not directly originating from their servers can be safely 
rejected, which to prevent spoofing,  requires a separately specified 
 From header field alignment requirement.  IMHO, the situation faced 
today would have been better resolved by agreeing to a scope-id list 
that included MFROM, PRA, and HELO.

The PRA and Mail parameter identities are used to guess whether a 
particular server may have been authorized to issue the message or 
whether the server is normally not compatible with either scheme.  
Clearly, this lacks a solid basis for ensuring message interchange.  By 
not holding the outbound server accountable for issuing spoofed 
messages, users that have had their accounts compromised are less likely 
to be made aware of this dangerous situation and there is less incentive 
for ensuring account security.

Regards,
Douglas Otis

---

From: Hector Santos <[email-masked]>
Date: Mon, 23 Apr 2012 17:17:38 -0400

> that included MFROM, PRA, and HELO.

I think this works only for protected a GOOD DOMAIN playing the rules. 
But I don't see it address the legacy abuse problem where by session 
only contains:

     EHLO      invalid-input
     MAIL FROM valid-input-but-spoofed-domain

How do you address this?  To me, thats the problem.  The problem is 
the the good guy playing by the rules, its the bad guy that is not 
behaving as we him wish to to.

---

From: S Moonesamy <[email-masked]>
Date: Mon, 23 Apr 2012 14:50:10 -0700

Hello,

Barry Leiba, as an individual, suggested text at 
http://www.ietf.org/mail-archive/web/spfbis/current/msg01397.html 
There is also a discussion about normative references at 
http://www.ietf.org/mail-archive/web/spfbis/current/msg01398.html

It's difficult for me to identify the other issues in 
draft-ietf-spfbis-experiment-05 based on the messages posted to this 
thread.  I'll ask the document editor to post a new draft based on 
the latest round of discussions.  Although it might not reflect the 
outcome of the those discussions, it might provide some clarity.  The 
working group can then suggest text for proposed changes.

I'll remind the working group of the moderator note posted by Andrew 
Sullivan at http://www.ietf.org/mail-archive/web/spfbis/current/msg01388.html

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: Douglas Otis <[email-masked]>
Date: Mon, 23 Apr 2012 15:31:36 -0700

On 4/23/12 2:17 PM, Hector Santos wrote:
>  Douglas Otis wrote:
> >
> > While HELO/EHLO identity is recognized by SPF, SPF lacks a scope
> > list making it clear whether use was intended exclusively for the
> > HELO/EHLO to authenticate outbound servers, or to authorize
> > third-party servers that might carry a domain's messages. Source
> > authentication using HELO/EHLO offers content assurances lacking
> > with either the PRA or Mail parameters. Additionally, only the
> > HELO/EHLO identity is able to ensure messages not directly
> > originating from their servers can be safely rejected, which to
> > prevent spoofing, requires a separately specified From header
> > field alignment requirement. IMHO, the situation faced today would
> > have been better resolved by agreeing to a scope-id list that
> > included MFROM, PRA, and HELO.
>
>  I think this works only for protected a GOOD DOMAIN playing the
>  rules. But I don't see it address the legacy abuse problem where by
>  session only contains:
>
>  EHLO invalid-input MAIL FROM valid-input-but-spoofed-domain

Dear Hector,

Not sure what was meant by EHLO invalid-input.

Spoofing EHLO domains should represent adequate and safe grounds for 
rejection.  Adding a From header field domain alignment requirement 
against HELO/EHLO domains would extend HELO/EHLO rejections to include 
this field as well.  No receiver should have a problem rejecting 
messages attempting to spoof HELO/EHLO domains, nor should there be any 
reason to make alignment exceptions based upon HELO/EHLO domains (this 
might be amended to include ATPS).

>  behaving as we him wish to to.

With IPv6, you need to assume there is essentially an endless supply of 
IP address prefixes, where the same has always been true for domain 
names.  Restrictions on use of IPv4 addresses has been largely 
responsible for excluding most email abuse.  That is why it is vitally 
important to track BOTH the domain and source IP address.  This tracking 
permits locating compromised systems and determining which reputation 
service had a problem with the message.  Unfortunately, many SPF records 
include %i and %s macros which thwart mitigation efforts and their 
usefulness for identifying their outbound address space.   By capturing 
the IP address in results (which is missing in Authentication-Results 
headers), critical tracking ability is retained.

There soon will be a sea change in how abuse in general can be handled.  
Any unknown entity must be rate-limited until behaviors can be 
determined.  Responding to abuse is often after-the-fact where 
malefactors will have already moved on to exploiting different resources.

Regards,
Douglas Otis

---

From: Dave Crocker <[email-masked]>
Date: Mon, 23 Apr 2012 11:58:01 -0700

wfm.

d/

-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 24 Apr 2012 17:46:37 +0200

> problems.  But that's not what we're trying to do.

Discussing the experimental setup is not our main task.  However, I
think that if we have precise critics to make we should go ahead,
especially if our experience could be useful for other experiments.
For example, we could answer why it took so many more years than
anticipated in order to reach a community consensus.

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 24 Apr 2012 12:06:26 -0400

> anticipated in order to reach a community consensus.

I really don't think there is value in going down this road.  Personally, I 
don't think technical considerations were a significant factor in what was 
decided in 2006, so any attempt to understand the technical aspects of the 
decision in retrospect are doomed to failure, confusion, and doubt.

Scott K

---

From: Dave Crocker <[email-masked]>
Date: Sun, 22 Apr 2012 14:43:41 -0700

On 4/22/2012 1:58 PM, Barry Leiba wrote:
> I would prefer that this document avoid making pronouncements and
> stating these kinds of opinions ("a single protocol can be advanced").
>   The point of this draft is to document results.

+1

> This seems to have a significant SPF bias.  May I suggest this?:

SPF defined the record.  It was there first.  Sender ID came afterwards, 
creating the overload.  A 'bias' in this case has rather solid 
historical precedent.

>     interfere with message delivery.

My above comment notwithstanding, this wording looks fine to me, and in 
general I concur with this sort of neutral language.

>   Otherwise, one might wonder where the comparison that's been done is

ok.

> I suggest "to using either mechanism over the other."

Hmmm.  Different issue:  Often the email addresses are the same.  It's 
the /fields/ that are /always/ different.  I suggest changing the 
wording to say "email address fields".

Tidbits of other cleanup:

      3.  Although the two mechanisms often use different email address
          fields for evaluation, no data collected showed any
          substantial operational benefit (e.g., cheaper processing,
          improved accuracy) to using one mechanism over the other..

> I would MUCH rather see this reported factually, without further
> opinion.  Something like this:
>
> NEW

Here, I think I disagree.  Part of the analysis can reasonably note that 
the analysis period has gone a long time and that a failure to archive 
much traction after this long is not likely to produce better traction 
in the future.

> That last sentence seems odd.  Unless you want to be specific about it,
> I suggest removing it.

ok.

> We can always argue whether an Informational document, by its nature,
> can have "normative" references.

There's nothing to argue about.  Many Informational documents are 
specifications.  Specifications dictate behavior (within the context of 
the specifications.) Some of the behaviors that are dictated 
fundamentally rely on external specifications that are cited. Language 
that dictates behavior is called 'normative'.  This includes such 
essential citations.

I think this is a reasonable summary of the issue:

http://stackoverflow.com/questions/6420522/what-does-normative-and-non-normative-mean-in-reference-to-xml

I suspect the envisioned debate is caused by confusing normative-ness 
with standards status...

d/
-- 
  Dave Crocker
  bbiw.net

---

From: Dave Crocker <[email-masked]>
Date: Mon, 23 Apr 2012 10:09:35 -0700

On 4/23/2012 9:19 AM, Barry Leiba wrote:
> understand this doc.

Of course you do.

naming hierarchy.  wildcards.  RRs (TXT).

All of those are core DNS constructs and they need to be understood for 
the SPF spec, at the risk of otherwise misconfiguring or misapplying SPF.

>  Normativity isn't transitive.

Hold on.  Just a sec.  OK.  My head is now back together.  The 
profundity of the deep philosophical point took me by surprise.

And yet I think I agree with you, if the point is that any citations 
that are normative need to be cited as normative on a case-by-case 
basis, as appropriate to the document.

That said, one would hope that there are some fairly concrete guidelines 
for what qualifies. Otherwise it's all ad hoc-ery.

d/

-- 
  Dave Crocker
  bbiw.net

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 16:56:32 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of John Leslie
> Sent: Monday, April 23, 2012 3:08 AM
> To: Barry Leiba
> Cc: [email-masked]

I asked, and he's not interested in getting involved due to workloads, and probably some PTSD.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 18:33:37 +0000

I'm also having sending problems this morning, so apologies for the apparent delay and the burst that will occur when the backlog clears.  And these tweaks didn't make it into -06 since they were just released from moderation.

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Crocker
> Sent: Sunday, April 22, 2012 2:44 PM
> To: Barry Leiba
> Cc: [email-masked]

Done.

> much traction after this long is not likely to produce better traction

I think those are practically synonymous.

> classing it as a normative reference?

OK, [DNS] is now Normative.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 19:16:11 +0000

[resending; original appears to have gone missing]

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Dotzero
> Sent: Monday, April 23, 2012 8:14 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> incorrect statement.

Suppose PRA was a random number generator rather than a heuristic.  If the data show that SPF and Sender ID thus concur 95% of the time, then I'd say SPF is as accurate as a random number generator.

The result is not an analysis of the compared mechanisms, and we specifically say that already.  It only observes that they concur at a specific rate.  And given that, it doesn't make sense to claim one is substantially more accurate than the other, because it plainly isn't.

> The whole point of SPF (and presumably SIDF) is to mitigate abuse. You 
> have not provided complete details about the dataset you are referring 
> to so it is hard to draw detailed conlusions regarding key
> points:
> 

I don't agree that any of these are material to answering the question.

Suppose you're comparing two commercial spam filters.  You don't get to know the guts of them because that's proprietary, but you can observe the black box output, which is to say they each tag x% of mail as spam or not spam.  Now suppose that after your evaluations, the report says both of them tag messages the same way 95% of the time.  I claim that, processing speeds not withstanding, they're basically the same in terms of accuracy.  I'm not saying they are 95% accurate, but I am saying that 95% of the time their accuracy (whatever it is) is the same.  Now, assuming you don't actually care about the difference between 95% and 95.1%, are you going to claim this is an invalid test, merely because you don't know the internal mechanics of the two filters?

> the dataset truly representative or are there potential issues with 

If self-selection of the data renders a data set invalid, then the entire document is invalid.  The data we have come from a handful volunteers who did the work to construct and execute the surveys and report findings.  We have the fortune that at least two of them are sizable participants (Hotmail and Cisco).

If you would like to run such a survey yourself and ensure your view of the world is also represented, or procure such surveys from whatever set of sources you think would make the results more even, then by all means please do so.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 16:46:33 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dotzero
> weakness of PRA (based on emperical data/testing), that is an incorrect
> statement.

> The whole point of SPF (and presumably SIDF) is to mitigate
> abuse. You have not provided complete details about the dataset you are
> referring to so it is hard to draw detailed conlusions regarding key
> points:
> 

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 16:52:03 +0000

> From: Murray S. Kucherawy
> Sent: Tuesday, April 24, 2012 9:47 AM
> To: [email-masked]
> Subject: RE: [spfbis] Review of draft-ietf-spfbis-experiment-05
> 
> If self-selection of the data renders a data set invalid, then the
> entire document is invalid.  The data we have come from a handful
> volunteers who did the work to construct and execute the surveys and
> report findings.  We have the fortune that at least two of them are
> sizable participants (Hotmail and Cisco).
> 
> If you would like to run such a survey yourself and ensure your view of
> the world is also represented, or procure such surveys from whatever
> set of sources you think would make the results more even, then by all
> means please do so.

Moreover, one will always be able to argue that some corner of the email ecosystem was excluded from our report unless we get universal participation.  That means this task is impossible.  I'd like to publish this before I expire.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 16:53:15 +0000

> Behalf Of Commerco WebMaster
> Sent: Monday, April 23, 2012 11:11 AM
> To: [email-masked]
> New looks cleaner.  Would it be possible (and does it make sense) to
> add something like -
> 
> indicate SPF adoption levels in the community as the preferred
> protocol, thus offering the reasonable path forward."

Works for me.  Others?

-MSK