Subject: [spfbis] Request for experimental evidence

From: Andrew Sullivan <[email-masked]>
Date: Fri, 3 Feb 2012 16:51:59 -0500

Dear colleagues,

One of our tasks as a WG is to bring to an end the experiment started
by RFC 4408 and other documents published at the same time.

To that end, your chairs would like evidence about the results of the
experiment.  

Note that we are asking specifically for evidence; evidence of
effectiveness and, particularly, evidence in respect of relative
effectiveness between SPF and Sender-ID would be extremely helpful.
This would all be evidence gleaned from operational experience with
one or more of the published RFCs.

This is _not_ a request for an analysis of that evidence, or any sort
of meta-commentary on it.  It is especially not a request for any
analysis of the differences nor any architecural commentary on how one
or the other would work under this or that condition.  It's just
straight actual from-the-field evidence: this is an empirical
experiment, and therefore we want empirical data.

Please post anything you have to the list.  We ask that participants
not discuss the different observations that are posted; if you have
different evidence to offer, please just post it without regard to
others' postings.  Thank you.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 03 Feb 2012 17:27:19 -0500

> by RFC 4408 and other documents published at the same time.

It was never clear to me what the experiment was supposed to be, so I've no 
idea what evidence would be appropriate.  Ultimately, I think Sender ID has 
been replaced by DKIM and that SPF and DKIM are complementary technolgies, so 
I'm not sure how to reply.

Can we get away with:

The SPF/Sender ID experiment was overtaken by the development of DK/DKIM and 
so Sender ID is obsoleted by those other technologies?

Scott K

---

From: Commerco WebMaster <[email-masked]>
Date: Fri, 03 Feb 2012 15:54:36 -0700

On 2/3/2012 3:27 PM, Scott Kitterman wrote:
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
>

+1

I would also add that I see the overall experiment process as being 
defined by its purpose, its ability to meet its purpose and finally its 
success in adoption by the Internet community.

In the case of purpose, perhaps simply domain name reputation protection 
through a process which defines simply how domain name holders can 
explicitly declare from what MTAs that email represented as being sent 
from their domain may be sent.

In the case of meeting its purpose, does SPF adequately address its 
purpose?  In most cases, yes.  In our own experience, absolutely yes.

Finally, adoption by the community.  Arguably, SPF is one of the more 
widely adopted facilities for achieving its stated purpose.

Arguably, the SPF experiment has gone well for its implementers.

I also agree with Scott's assertion that SPF and DKIM are complementary 
technologies.

Alan M

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 03 Feb 2012 14:58:12 -0800

> so Sender ID is obsoleted by those other technologies?

The simplest form of experimental question could be:

1. Has the world essentially dropped one of these (or both).

Clearly spf is still heavily used, so this translates into whether there remains 
significant support for Sender-ID.  I believe the answer is no, but I don't any 
empirical basis for this.

A more elaborate question could be:

2. Are there features from one that gained support and would be worth merging 
into the other?

Given my own answer to #1, above, that translates into "Are there parts of 
Sender-ID worth salvaging?"

Given the narrow scope, it is likely that any items deems reasonable to merge 
into SPF will have to wait for a later SPFbisbis effort, but it could be worth 
documenting.

Of general interest would be:

3. Document the deployment experiences of using SPF and/or Sender-ID.

This would be a plusses and minuses history.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 3 Feb 2012 18:56:18 -0500

Dear colleagues,

On Fri, Feb 03, 2012 at 05:27:19PM -0500, Scott Kitterman wrote:

> It was never clear to me what the experiment was supposed to be, so
> I've no idea what evidence would be appropriate.

While I explicitly asked that the experiment evidence not include
meta-analysis, commentary, and so on, and that it only have empirical
evidence of whatever kind, I think Scott Kitterman's point is a
reasonable one: "The question is wrong."

Therefore, this subthread is for "The question is wrong" responses.
If you just want to say "+1" that's ok with me.  If you want to
outline the ways in which you think the question is wrong or the
experiment was flawed, that's ok with me too.  Please do not add
commentary about others' remarks, however, just as I originally
asked.  We're _collecting_ right now, not analyzing.

Thanks,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 3 Feb 2012 18:57:02 -0500

Dear colleagues,

For the purposes of this collection exercise, any of the questions
Dave outlines below, or indeed any other question you think has been
answered by the experiment, is a fine one.  The constraint is that you
have to have empirical evidence of some kind.  Note the distinction
Dave usefully makes in his (1): "I believe the answer is no, but I
don't any empirical basis for this."  (Thank you for making the
distinction, Dave.)  We will have opportunities later to discuss
beliefs, conclusions, and so on.  But right now, it's important that
we collect whatever empirical evidence we have, however sketchy, and
do it with as little bias as we can.

I am sometimes willing to discuss experimental metholology to the
point that your ears bleed.  It was once one of my areas of
specialization.  But for right now, we need to focus on such empirical
evidence as we can scare up.

Thanks,

A

On Fri, Feb 03, 2012 at 02:58:12PM -0800, Dave CROCKER wrote:
> Clearly spf is still heavily used, so this translates into whether
> there remains significant support for Sender-ID.  I believe the
> answer is no, but I don't any empirical basis for this.
> 
> 
> 2. Are there features from one that gained support and would be
> worth merging into the other?
> 
> Given my own answer to #1, above, that translates into "Are there
> parts of Sender-ID worth salvaging?"
> 
> Given the narrow scope, it is likely that any items deems reasonable
> to merge into SPF will have to wait for a later SPFbisbis effort,
> but it could be worth documenting.
> 
> 

-- 
Andrew Sullivan
[email-masked]

---

From: Pete Resnick <[email-masked]>
Date: Fri, 3 Feb 2012 21:36:22 -0600

On 2/3/12 4:58 PM, Dave CROCKER wrote:
>> I've no
> there remains significant support for Sender-ID.  I believe the answer 
> is no, but I don't any empirical basis for this.

[Speaking without hats]

Point of clarification: In addition to wanting some "empirical basis" 
for whether there remains significant support for Sender-ID, I think we 
also want some specifics (i.e., empirical basis) for how "clearly spf is 
still heavily used". That is, I think we want the 
result-of-the-experiment document to actually say how much deployment 
there is of each technology, and what kind of deployments. It doesn't 
have to be extensive, but I think we do want the document to have some 
numbers.

> 2. Are there features from one that gained support and would be worth 
> merging into the other?
>
> to merge into SPF will have to wait for a later SPFbisbis effort, but 
> it could be worth documenting.
>
>

Agreed on these.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 3 Feb 2012 19:51:28 -0800

I imagine to the IESG at the time, the "experiment" was the simultaneous deployment of the two technologies (or perhaps more accurately, their simultaneous release into the wild as experimental RFCs) and an observation of the impacts of doing so.  The issue they had, as I recall, was that the two technologies acted based on the same signal from the purported sender (the SPF record), but could (often, in theory) reach different conclusions.  It was this collision that caused them not to release either as Proposed Standard, and add the IESG Note to all four documents.

Thus to my mind, resolving the "experiment" itself involves questions such as these:

1) Was this simultaneous deployment harmful, as the IESG feared?

2) Do SPF and Sender-ID actually come to similar or equal conclusions most of the time?  Or perhaps more generally, how often do they disagree?

...and then one of:

3a) If they are essentially or even approximately the same, which one is simpler and thus merits advancement?
	
3b) If they are essentially or even approximately the same, which one is more widely deployed and thus worth standardization?

I have opinions about all of these except (2), which requires a lot of collected data to answer.  I'll know sometime this weekend if my own data collected about DKIM happens to include SPF and Sender-ID data so that I could offer an answer (hopefully one of several such data sets).

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 3 Feb 2012 20:10:37 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
> Sent: Friday, February 03, 2012 7:51 PM
> To: [email-masked]
> Subject: Re: [spfbis] "The question is wrong" thread (was: Request for experimental evidence)
> 
> I have opinions about all of these except (2), which requires a lot of
> collected data to answer.  I'll know sometime this weekend if my own
> data collected about DKIM happens to include SPF and Sender-ID data so
> that I could offer an answer (hopefully one of several such data sets).

The report came back earlier than I expected.

In 95.51% of the cases for which I have both SPF and Sender-ID data (only about 125k messages), the two agreed (i.e., both passed or neither passed).

I'm quite sure there are far larger data sets than the one I have, but I hope that gets us started.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 3 Feb 2012 20:14:29 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Pete Resnick
> Sent: Friday, February 03, 2012 7:36 PM
> To: [email-masked]
> Cc: [email-masked]; Dave CROCKER; Scott Kitterman
> Subject: Re: [spfbis] Request for experimental evidence
> 
> also want some specifics (i.e., empirical basis) for how "clearly spf
> is still heavily used". That is, I think we want the result-of-the-
> experiment document to actually say how much deployment there is of
> each technology, and what kind of deployments. It doesn't have to be
> extensive, but I think we do want the document to have some numbers.

I think this is a difficult task to complete given the fact that it's impossible to tell from random polling whether a given site, sending or receiving, has deployed either technology.  We'd have to rely on a survey of some kind, which necessarily makes some guesses about the masses that don't submit answers.

-MSK

---

From: Paul Midgen <[email-masked]>
Date: Sat, 4 Feb 2012 11:57:09 +0000

I have a data collection effort underway at Hotmail that should wrap up around late March and will share the data with this list.

I agree with Murray that deployment will be hard to gauge, but it will be easy to compare the results of both methods against the same message, for a huge volume of mail with a very large number of unique sending domains over a reasonably long period of time.

In my opinion the true purpose of the experiment was to determine which method produced a measurable advantage in robust determination of path authorization. When it is shown that both ended up doing pretty much the same thing, there should be little reasonable resistance to taking the simplest most expedient path forward.

Hopefully others will share data as well.

-p

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
Sent: Friday, February 03, 2012 8:14 PM
To: [email-masked]
Subject: Re: [spfbis] Request for experimental evidence

> for whether there remains significant support for Sender-ID, I think 
> we also want some specifics (i.e., empirical basis) for how "clearly 
> spf is still heavily used". That is, I think we want the 
> result-of-the- experiment document to actually say how much deployment 
> have to be extensive, but I think we do want the document to have some numbers.

-MSK

---

From: Paul Midgen <[email-masked]>
Date: Sat, 4 Feb 2012 12:03:13 +0000

I mention a rather large set of data being accumulated in another message to the list.

Analysis of smaller data sets supports Murray's ~95% value. I've just taken to saying >95% of the time SPF and SenderID do the same thing. I'll add a few other bits of information to the analysis but that'll just be additional color (e.g. PRA breakdowns, comparison to MailFrom, etc.)

-p

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
Sent: Friday, February 03, 2012 8:11 PM
To: [email-masked]
Subject: Re: [spfbis] "The question is wrong" thread (was: Request for experimental evidence)

-MSK

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 04 Feb 2012 13:14:59 +0100

On 04/Feb/12 05:14, Murray S. Kucherawy wrote:
>> From: ietf.org On Behalf Of Pete Resnick
>
>> I think we want the result-of-the-experiment document to actually
>> say how much deployment there is of each technology, and what
>> kind of deployments. It doesn't have to be extensive, but I think
>> we do want the document to have some numbers.
> 
> I think this is a difficult task to complete given the fact that
> it's impossible to tell from random polling whether a given site,
> sending or receiving, has deployed either technology.  We'd have to
> rely on a survey of some kind, which necessarily makes some guesses
> about the masses that don't submit answers.

Yes, it is much easier to sample the published policies base.
http://eggert.org/meter/spf.html and http://spf-all.com/ come to mind.

For checking, besides Murray's collected data for DKIM, there is
Google's DMARC reporting.  It allows, among other things, to count
failing spf checks for messages naively forwarded to gmail accounts,
e.g., presumably for messages I sent to this or similar list, I got:

      <spf>
        <domain>ietf.org</domain>
        <result>hardfail</result>
      </spf>

Another way to get numbers on checking results is to extract them from
our own log files, where we know how we check SPF on incoming
messages.  That is obviously skewed, but can provide a trend.  Worth?

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 04 Feb 2012 19:02:49 -0800

>> 1. Has the world essentially dropped one of these (or both).
...
> Point of clarification: In addition to wanting some "empirical basis" for
> whether there remains significant support for Sender-ID, I think we also want
> some specifics (i.e., empirical basis) for how "clearly spf is still heavily
> used". That is, I think we want the result-of-the-experiment document to

<http://eggert.org/meter/spf.html>

I believe Lars had nothing to do with SPF development or any of the related 
technologies. That is, I believe he qualifies as an independent source.

The URL has more detail, but he cites 65% global deployment.

d/
-- 

   Dave Crocker

---

From: Philip Gladstone <[email-masked]>
Date: Mon, 06 Feb 2012 08:46:12 -0500

> To that end, your chairs would like evidence about the results of the
The problem that was widespread at the time of the introduction of SPF 
-- the joe-job has ceased to be an issue (as far as I am concerned). I 
feel that SPF has contributed to the ending of that particular scourge.

However, the scarcity of deployed SRS (Sender Rewriting Scheme) 
implementations remains a barrier today to more widespread use of 
hardfail in SPF. This also extends to some (social) websites which send 
email to your 'friends' and forge it to come from you.

Philip

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 6 Feb 2012 10:29:30 -0500

Dear colleagues,

On Mon, Feb 06, 2012 at 08:46:12AM -0500, Philip Gladstone wrote:

> The problem that was widespread at the time of the introduction of
> SPF -- the joe-job has ceased to be an issue (as far as I am
> concerned). I feel that SPF has contributed to the ending of that
> particular scourge.
> 
> hardfail in SPF. This also extends to some (social) websites which
> send email to your 'friends' and forge it to come from you.

This is not to pick on Philip Gladstone, but to remind everyone that
what we're looking for is empirical evidence.  What we need are actual
observations of SPF deployment.  The first paragraph above gives an
evaluation of what SPF did, and the second paragraph has a claim about
an SPF dependency.  How unused is hardfail in SPF?  (Whether that is
because of the reason offered or another one is a separate question,
and not a matter of evidence.)

Please restrict your postings in this thread to empirical evidence.

Thanks,

A

-- 
Andrew Sullivan
[email-masked]

---

From: S Moonesamy <[email-masked]>
Date: Mon, 06 Feb 2012 08:17:21 -0800

>concerned). I feel that SPF has contributed to the ending of that 

Paul Midgen mentioned that he can share some data with the working 
group ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg00271.html 
).  Such information is useful for the work on the document which 
concludes the Sender-ID and SPF experiments.  I'll quote part of 
Andrew's message ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg00255.html ):

   "This is _not_ a request for an analysis of that evidence, or any sort
    of meta-commentary on it.  It is especially not a request for any
    analysis of the differences nor any architecural commentary on how one
    or the other would work under this or that condition.  It's just
    straight actual from-the-field evidence: this is an empirical
    experiment, and therefore we want empirical data."

Saying that SPF or Sender-ID has contributed to solve X is not empirical data.

Scott Kitterman posted the following comment ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg00258.html ):

   "It was never clear to me what the experiment was supposed to be,
    so I've no idea what evidence would be appropriate."

There are different ways to assess the experiments.  The working 
group can consider the number of implementations, it can consider 
deployment, e.g. DNS records for Sender-ID and SPF, or the problems 
the experiments sought to address.  There will be ample opportunity 
to discuss about which metric to use.  For now, if you think that you 
have data which is relevant, post it to the list.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: Philip Gladstone <[email-masked]>
Date: Mon, 06 Feb 2012 11:44:00 -0500

On 2/6/2012 10:29 AM, Andrew Sullivan wrote:
> Thanks, A 
OK!

I have an archive of ~280,000 SPF records from the Alexa top 1M web 
hosts. Not exactly the list of people that send email, but representative.

Qualifiers used:

           +a 223030
         +all   2117
       +aspmx     25
      +exists   1415
          +ip    507
         +ip4 331927
         +ip6    833
        +ipv4    640
        +mail     44
          +mx 215065
          +p4     27
         +ptr  36507
         +spf     43
           -a     33
         -all  52810
         -ip4     43
         ?all  53887
      ?exists     66
           ~a     31
          ~al     29
         ~all 164812

Note that the vast majority of '+' is implicit. Also, I have eliminated 
the infrequent checks -- there is a fair amount of garbage in these 
records (and maybe that is indicative of a problem). There is some 
garbage in the table above.

Philip

-- 
Philip Gladstone
Distinguished Engineer
Product Development
[email-masked]
Phone: +1 978-ZEN-TOAD (+1 978 936 8623)
Google: +1 978 800 1010
Ham radio: N1DQ

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 06 Feb 2012 12:04:59 -0500

> Saying that SPF or Sender-ID has contributed to solve X is not empirical

If I put it in terms of:

Before I deployed an SPF record ending in -all (fail), on my personal domains 
I used to get dozens to hundreds of bounce back emails from joe job attempts 
on and almost daily bases.

Within a few months of deploying this record the level of traffic dropped to 
essentially nothing and it's been years since I recall getting a single such 
message.

Is that empirical data (that was my experience as a small domain owner and I 
suspect similar to Phil's)?

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 06 Feb 2012 12:23:28 -0500

On Monday, February 06, 2012 10:29:30 AM Andrew Sullivan wrote:
> How unused is hardfail in SPF? 

It's not used at all, since there's no such thing as hardfail in SPF.  I think 
being careful and precise about language in this discussion is important.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 09:37:32 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Scott Kitterman
> Sent: Monday, February 06, 2012 9:05 AM
> To: [email-masked]
> Before I deployed an SPF record ending in -all (fail), on my personal
> domains I used to get dozens to hundreds of bounce back emails from joe
> job attempts on and almost daily bases.
> 
> Within a few months of deploying this record the level of traffic
> dropped to essentially nothing and it's been years since I recall
> getting a single such message.
> 
> Is that empirical data (that was my experience as a small domain owner
> and I suspect similar to Phil's)?

I think such anecdotal evidence answers a question as to SPF's efficacy, but not the question of resolution of the experiment declared by the IESG (or, at least, not directly).  In particular, it makes no statement about whether or not Sender-ID had or would have had the same or similar impact.

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 06 Feb 2012 12:42:19 -0500

On Monday, February 06, 2012 09:37:32 AM Murray S. Kucherawy wrote:
> > From: [email-masked] [mailto:[email-masked]] On Behalf
> > Of Scott Kitterman Sent: Monday, February 06, 2012 9:05 AM
> > To: [email-masked]
> I think such anecdotal evidence answers a question as to SPF's efficacy, but
> not the question of resolution of the experiment declared by the IESG (or,
> at least, not directly).  In particular, it makes no statement about
> whether or not Sender-ID had or would have had the same or similar impact.

That's true, but there's no way to go back and tell.  I think it is useful on 
the basis of the idea that got suggested here somewhere that if they both 
work, we should just use the simpler one (SPF).  That thought can be extended 
to, if SPF works, we should just use it.  Unless there's a showing of superior 
efficacy of some kind, there's no reason to take on the inherent technical and 
IPR complexities of Sender-ID.

Scott K

---

From: Philip Gladstone <[email-masked]>
Date: Mon, 06 Feb 2012 12:46:48 -0500

> suspect similar to Phil's)?
Over the last 4 days, I see 57 IP addresses forwarding emails that were 
originated from my domain. I *think* that all of these are actually 
valid forwards (but not doing the appropriate SRS rewriting), though I 
suspect that there are implementation problems in certain SPF 
implementations.

For example,

98.136.219.60: ostmaster
98.138.215.201: ostmaster
98.138.91.117: postmaster
98.138.91.60: postmaster
98.139.165.31: ostmaster
98.139.165.55: ostmaster

This says that the addresses forwarded mail from postmaster@mydomain and 
ostmaster@mydomain. I suspect that the 'ostmaster' is a bug in the 
evaluation of the %{l1r-} macro.

I also see

209.68.2.48: bounces.wxqc

This indicates that (probably) wxqc-bounces was the originator and the 
%{l1r-} did not get evaluated correctly (two components were returned)

128.186.98.3: ouncces.wxqc

This is weirdest -- it looks like another bug. (I see this particular 
error from two other IPs as well.

Philip

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 06 Feb 2012 10:28:50 -0800

> I have an archive of ~280,000 SPF records

This prompts a question worth considering:

      Do we care about adoption via publication of SPF records, versus adoption 
via evaluation of SPF records?

It's trivial to publish a record, but it does still take the decision and some 
effort.  So knowledge of publication percentages is helpful.

In contrast, it is rather expensive to start paying attention to SPF records on 
the receive side, since it requires extra software AND requires additional 
policy decisions.   That is, how should the additional information be /used/?

Adoption by publication is important, of course, but it only represents an 
inexpensive hope of utility.

Adoption by evaluation is, IMO, the critical information.

d/
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 06 Feb 2012 10:31:39 -0800

On 2/3/2012 7:51 PM, Murray S. Kucherawy wrote:
> I imagine to the IESG at the time, the "experiment" was the simultaneous deployment of the two technologies (or perhaps more accurately, their simultaneous release into the wild as experimental RFCs) and an observation of the impacts of doing so.

This highlights a very basic and very serious problem we are faced with: 
guessing what the IESG wanted.

Absent a clear statement, how can we know we are satisfying the requirement to 
comment on the 'experiment'?  We do not even know what it was or was supposed to be.

d/
-- 

   Dave Crocker

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 6 Feb 2012 13:39:24 -0500

Dear colleagues,

On Mon, Feb 06, 2012 at 10:31:39AM -0800, Dave CROCKER wrote:

> This highlights a very basic and very serious problem we are faced
> with: guessing what the IESG wanted.
> 
> Absent a clear statement, how can we know we are satisfying the
> requirement to comment on the 'experiment'?  We do not even know
> what it was or was supposed to be.

All of that is true, but it is also futile: "the IESG" is not the same
body it was then, and anyway asking people about what their intentions
were in the past is not likley to yield a proposition amenable to
verification.  

Speaking only for myself for a moment, this is actually why I think we
want anything that resembles empirical evidence about any of this
experiment.  If we are to go to the rest of the IETF and say, "We
believe this experiment is over," people will be right to ask what
basis we have for that assertion.  We don't know what the experimental
terms are, but we can point to such evidence as we have and say that
it seems to demonstrate [insert our conclusions here]. 

Speaking as chair again, I strongly discourage us to attempt to work
out what the experimental purpose was.  It's too bad nobody wrote it
down at the time.  But they didn't, and we have to live with that fact
of the world.  The lack of experimental goals, however, is indeed why
we have a thread for complaints that we cannot tell what would qualify
as evidence about the experiment.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Dotzero <[email-masked]>
Date: Mon, 6 Feb 2012 14:03:13 -0500

> as evidence about the experiment.

In light of Andrews comments, I would recommend that we not spend any
time considering SenderID. Based on the results we cans imply say that
it failed to gain traction in the wild (particularly from a receiver
evaluation side) and leave it at that. The working group charter is
focused on SPF and NOT SenderID.

Mike

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 11:09:14 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dotzero
> Sent: Monday, February 06, 2012 11:03 AM
> To: Andrew Sullivan
> Cc: [email-masked]
> Subject: Re: [spfbis] "The question is wrong" thread
> 

What evidence would you present to support that claim?

We need to do better than anecdotal, I believe.

---

From: Dotzero <[email-masked]>
Date: Mon, 6 Feb 2012 14:17:05 -0500

> What evidence would you present to support that claim?

The only significant receiver that I am aware of that validates
SenderID is Microsoft and they are supportive of SPF as part of DMARC
looking forward. I'll flip it around. if someone or someones want to
come forward and argue that SenderID has a thriving  implementer base
then I'd love to see the list.

> We need to do better than anecdotal, I believe.

You missed the (implied) point in my first statement. SenderID is out
of scope for the charter of the SPFbis working group. If others wish
to then they can seek to form a SenderIDbis working group. I am not
recommending this, just pointing out that this group does not have an
obligation to speak to that experiement. My comment about SenderID
adoption should really be considered as an aside.

Mike

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 11:23:06 -0800

> From: Dotzero [mailto:[email-masked]]
> Sent: Monday, February 06, 2012 11:17 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]

I don't think it's reasonable to ask anyone to prove a negative.

> of scope for the charter of the SPFbis working group. If others wish to
> then they can seek to form a SenderIDbis working group. I am not

I disagree.  The IESG Note attached to both Sender ID and SPF indicates that the two are in direct conflict, which is why both were relegated to Experimental status.  If I were on the IESG that will receive this working group's documents and that issue had been completely ignored or only marginally addressed, I would object to the documents' advancement.

I don't think this is a way out.  We need to answer the question one way or another.  It's in our charter.

-MSK

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 6 Feb 2012 14:52:48 -0500

> time considering SenderID. Based on the results we cans imply say that
                             ^^^^^^^^^^^^^^^^^^^^

Let me try again.  We are attempting to determine, in the present
stage, _what those results are_ and not what they mean.  We need to
collect the data before we can analyze it.  We cannot throw away one
pile of evidence on the basis that we already know what the answer is.

> focused on SPF and NOT SenderID.

This is certainly true, but given that they were both released at the
same time using the same facilities on the Internet, it is going to be
hard to move ahead without making a determination about the way these
things interacted with one another.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 06 Feb 2012 11:58:11 -0800

> were in the past is not likley to yield a proposition amenable to

I concur, and in fact was mostly interested in highlighting that casting any of 
this in terms of what "the IESG" wants or wanted will not be productive. However 
some of the discussion has attempted to presume what was being requested by the 
IESG.

The history of this activity is an 'experiment' only in the sense of 
test-marketing.  Seeing whether something will be a success in the market.  Of 
course, there can be many interesting questions about what it means to be a 
success, including comparative worth between competing solutions.

So, really, my core point is that we need to agree on what questions we need to 
answer.  Any empirical data could be helpful, but without specific questions 
that we agree need to be answered the degree of utility will be accidental.

I originally posted some suggested questions for us to answer.  You and others 
have posted additional questions.  All look interesting, but we probably should 
prune the set to a few basic issues that could help with strategic choices.

d/
-- 

   Dave Crocker

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 6 Feb 2012 15:22:45 -0500

On Mon, Feb 06, 2012 at 11:58:11AM -0800, Dave CROCKER wrote:
> 
> I originally posted some suggested questions for us to answer.  You
> and others have posted additional questions.  All look interesting,
> but we probably should prune the set to a few basic issues that
> could help with strategic choices.

I'm clearly having a bad chair day, because I'm having a devil of a
time getting this point across.  While I agree that it would be
considerably easier to prune the questions we think we're trying to
answer, I believe that we might face hard questions later if we
exclude some classes of evidence from consideration or if we decide
too early to rule some question out of consideration.  That is why
(despite it going against everything in my personal make-up) I have
been resisting trying to shape the question too much in this initial
round.  

I know that there are many people who may find this approach
frustrating or annoying, or who might think that this is simply a
waste of time.  It seems to me, however, that we are less likely to
overlook something during our evaluation if we have collected as much
as possible from the beginning.

Had the "experiment" actually contained any sort of statement about
what propositions were under test, I would likely urge a different
evaluation methodology.  But it didn't, so I didn't.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Dotzero <[email-masked]>
Date: Mon, 6 Feb 2012 15:29:31 -0500

>> time considering SenderID. Based on the results we cans imply say that

I leave it to others then to provide data on SenderID.

I'm not saying throw away a pile of evidence on the basis that we
already know what the answer is. I'm saying that data regarding
SenderID and the success or failure of THAT experiment is generally
orthogonal to the SPF experiment and it's success or failure in the
wild. SenderID data should only be of interest regarding the SPF
experiment to the extent that such data directly impacts discussion of
the success or failure (empirically) of the SPF experiment. At the
close of the MARID working group there were basically two competing
camps - each supporting one of the specs. I was lukewarm towards
SenderID but found it unacceptable because a flaw in the PRA algorithm
(related to use of the Sender field) allowed malicious emails to
consistently get a neutral even for domains publishing a -all.

Mike

---

From: S Moonesamy <[email-masked]>
Date: Mon, 06 Feb 2012 13:52:12 -0800

Dear SPFBIS Working Group,

If WG participants believe that there are or there will be two 
competing camps here, I suggest that they give the matter some more 
thought.  We have a charter which spells out the working group 
deliverables.  My WG co-chair has already explained that we are asking for:

  (i)  Reviews of RFC 4408

  (ii) Experimental evidence

I would like to see some focus here instead of discussions about 
issues, how to conclude the experiments or what the IESG wants.  Some 
of you may view the questions as fuzzy.  I would like to be able to say:

   The Working Group has carefully considered this matter and took the
   following decision.

It will take us longer to get to that point if we start with 
open-ended discussions.  I understand that some WG participants might 
not be familiar with the IETF process.  We are also here to explain 
the non-technical details.  Please post your questions to this 
mailing list or to either of the WG co-chairs.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 14:07:14 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Andrew Sullivan
> Sent: Monday, February 06, 2012 12:23 PM
> To: [email-masked]

As has been pointed out, the IESG that added the Note Well to those other documents isn't the IESG we have today.  But really, all that matters is the IESG we have today (or, at least, the one that will review the output of this working group).

Thus, would it be beneficial to ask the current IESG what it thinks the experiment is or was, or what they'd like to see from us in terms of a resolution?  Since they're the ones that get to decide, maybe they're the ones we should ask.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 14:10:38 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of S Moonesamy
> Sent: Monday, February 06, 2012 1:52 PM
> To: [email-masked]
> Cc: Andrew Sullivan

I think we are going in that direction, though it's very early in that process.  The request for RFC4408 reviews is quite clear, but the request for evidence of the results of the experiment isn't answerable without understanding exactly what we think the experiment was, and that currently seems to be an open-ended question.  Otherwise, I think we need some pretty specific guidance about what data you're after.

-MSK

---

From: Dotzero <[email-masked]>
Date: Mon, 6 Feb 2012 17:32:16 -0500

> I think we are going in that direction, though it's very early in that process.  The request for RFC4408 reviews is quite clear, but the request for evidence of the results of the experiment isn't…

This is why I am asking for a consensus or a call by the chairs that
the item (ii) Experimental evidence presented should be constrained by
item  (i)  Reviews of RFC 4408 and the fact that RFC 4408 is the focus
of this working group. There were in fact two concurrent experiments.
The charter of the group does not include RFC 4406 or RFC 4407.

Mike

---

From: SM <[email-masked]>
Date: Mon, 06 Feb 2012 14:20:16 -0800

Hi Murray,
At 02:07 PM 2/6/2012, Murray S. Kucherawy wrote:
>As has been pointed out, the IESG that added the Note Well to those 
>other documents isn't the IESG we have today.  But really, all that 
>matters is the IESG we have today (or, at least, the one that will 
>review the output of this working group).

It's an IESG Note.  The Note Well is about IETF Contributions.

>Thus, would it be beneficial to ask the current IESG what it thinks 
>the experiment is or was, or what they'd like to see from us in 
>terms of a resolution?  Since they're the ones that get to decide, 
>maybe they're the ones we should ask.

I don't think you'll get the answer you are waiting for. :-)

Regards,
-sm

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 14:35:12 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of SM
> Sent: Monday, February 06, 2012 2:20 PM
> To: [email-masked]

Sorry, yes, the IESG Note.

> >Thus, would it be beneficial to ask the current IESG what it thinks the
> >experiment is or was, or what they'd like to see from us in terms of a
> >resolution?  Since they're the ones that get to decide, maybe they're
> >the ones we should ask.
> 

What answer do you think I'm waiting for?

It's clear that any answer is better than guessing, and that my guess is different than those of others.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 14:37:13 -0800

> Sent: Monday, February 06, 2012 2:32 PM
> To: Murray S. Kucherawy
> Cc: [email-masked]

On the contrary, the charter lists as our first milestone:

     Aug 2012:    A document describing the SPF/Sender-ID experiment
             and its conclusions to the IESG for publication.

That clearly involves RFCs 4406 and 4407, because they defined Sender-ID, and all three of them included the IESG Note, which created the experiment (and without clearly defining it, as Andrew pointed out).

-MSK

---

From: Commerco WebMaster <[email-masked]>
Date: Mon, 06 Feb 2012 15:39:40 -0700

> evaluation methodology.  But it didn't, so I didn't.

How about this - can we look at the "experiment" from an MTA adoption 
standpoint.

There are a number of well known SMTP/POP server applications that exist 
on various OS platforms out there.

Can we easily determine how many of these (both Open Source and 
Commercially paid for products) actually support SPF vs SenderID as a 
measure of success of one vs the other?

It would at least offer an answer as to the success of the experiment 
from an MTA implementer standpoint in software which seemingly most 
benefits from SPF or SenderID.

 From an adopter point of view, I could offer yet another anecdotal 
story regarding SPF, but I'm guessing our Chair might be unhappy, so for 
these, probably the only thing we can do is reach out to those who have 
done some independent research from DNS servers as to the adoption of 
SPF RR and TXT RRs which contain the SPF v1 marker.

In a strange sense, I suppose the support for and implementation of the 
SPF RR in DNS itself might be considered something of a win for the SPF 
experiment.

Best,

Alan M

---

From: S Moonesamy <[email-masked]>
Date: Mon, 06 Feb 2012 14:48:49 -0800

>The charter of the group does not include RFC 4406 or RFC 4407.

I'll discuss the matter with my WG co-chair.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: John Leslie <[email-masked]>
Date: Mon, 6 Feb 2012 18:11:12 -0500

> I don't think you'll get the answer you are waiting for. :-)

   More exactly, it's doubtful he'd get _any_ timely answer.

   Any AD can add a Management Item for such a request. Depending on
how full a particular IESG agenda is, it may or may not get discussion.
Either way, they look for one AD to take an action item to draft a
reply. In this case, that pretty much means Pete Resnick or Barry
Leiba. I don't think either of them would volunteer.

   If some AD does volunteer, it's a seemingly random variable how
long it will take. Sometimes it's only two weeks; two months is
closer to the norm; both Barry and I have seen these items drag on
longer. Once a reply _is_ drafted, there's _always_ some AD that
thinks it needs changes.

   Lather, rinse, repeat.

   I suggest a different paradigm: assume the IESG would simply like
to see the Sender-ID experiment wrapped up, and they don't really
care all that much whether Sender-ID ever gets advanced to Proposed
Standard.

   Thus, they're likely to be happy with any justification we give
for declaring the Sender-ID experiment to be complete. Our problem
lies elsewhere: if we annoy the pro-Sender-ID folks too much, they'll
raise issues during LastCall of our document declaring Sender-ID to
have finished experimental stage. This could slow things considerably.

   All we really need out of this is to declare the Sender-ID use
of v=spf obsolete. To justify that, we need only prove that SPF use
of v=spf greatly exceeds Sender-ID use of it.

   I suggest we concentrate on gathering such evidence.

--
John Leslie <[email-masked]>

---

From: SM <[email-masked]>
Date: Mon, 06 Feb 2012 15:27:17 -0800

Hi John,
>    More exactly, it's doubtful he'd get _any_ timely answer.

I hope that your answer answers Murray's question.

As for the rest of your message, insert smiley. :-)

Regards,
-sm

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 Feb 2012 15:31:05 -0800

> Sent: Monday, February 06, 2012 3:27 PM
> To: John Leslie
> Cc: [email-masked]

It tells me that may not exactly be a timely path to get some direction about what will satisfy our deliverable, but apart from that, I still feel like we're taking at best an educated guess about the question we're supposed to answer.

-MSK

---

From: SM <[email-masked]>
Date: Mon, 06 Feb 2012 16:31:56 -0800

Hi Murray,
At 03:31 PM 2/6/2012, Murray S. Kucherawy wrote:
>satisfy our deliverable, but apart from that, I still feel like 
>we're taking at best an educated guess about the question we're 
>supposed to answer.

Yes.

BTW, I understand your point.  Give me some time to think about it.

Regards,
-sm

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 06 Feb 2012 19:22:46 -0800

On 2/6/2012 12:22 PM, Andrew Sullivan wrote:
> time getting this point across.

There was a TV station in Chicago, many years ago, that would post a sign on the 
screen when it was having technical difficulties.  It showed a carton figure of 
a guy on a balcony, looking and pointing "away".  The caption read "It's not 
you.  It's not us.  It's THEM!".  Which is to say that, I think you're doing 
fine.  You've pressed for resolution of an important point.  It just happens to 
be one that is difficult, IMO.

I think the underlying problem here is our having to answer questions without 
our being told what they are.  Hence, we need to find out

 >  While I agree that it would be
> too early to rule some question out of consideration.

And indeed, that ought to inform our pruning process.  But I suspect we can't do 
that without getting some interim review from the IESG.

My suggestion is that we try to formulate the questions that we think make sense 
to answer and then cycle them to the IESG for comment, before actually 
formulating answers.

s/

-- 

   Dave Crocker

---

From: Pete Resnick <[email-masked]>
Date: Tue, 7 Feb 2012 01:00:22 -0600

With AD hat on.

Let me put a slightly happier spin on John's message, which I think is 
essentially correct.

On 2/6/12 5:11 PM, John Leslie wrote:
> SM<[email-masked]>  wrote:
>    

So, let me say up front that I would immediately volunteer for this if 
asked. However, I don't think it's necessary.

> longer. Once a reply _is_ drafted, there's _always_ some AD that

I think if you ask for a *formal* answer, something like the above is 
likely to happen. Or worse, they'll say, "Pete, you obviously didn't 
scope the charter well enough. Go back to the group and recharter that 
section." If simply asked the question without pomp and circumstance, I 
think it slightly more likely that the IESG will come to me (probably 
not Barry; we spare the new guys crap until they ask for it ;-) ) and 
say, "We said what we wanted in the charter. It's your WG to manage; go 
give them an answer."

But really, the fact of the matter is that if you all come up with an 
interpretation of this sentence:

     The working group will produce a document describing the course of
     the SPF/Sender-ID experiment, thus bringing that experiment to a
     formal conclusion.

and that interpretation sounds reasonable to me (which we can determine 
now), when it comes time for Last Call/IESG Evaluation, I will make it 
quite clear to the IESG that the WG has fulfilled it's charter 
obligation. Yes, we can wring our hands about some AD coming out of the 
woodwork and disagreeing, but (a) I think the likelihood of that is 
exceedingly low, and (b) if we're going to worry about that, I can 
provide a much longer list to worry about too.

The fact is that the IESG thought the above sentence was sufficient to 
explain the deliverable. If we come up with a reasonable interpretation 
of that deliverable, we'll be fine and I will defend it to the hilt.

> care all that much whether Sender-ID ever gets advanced to Proposed

I think that's pretty much true, adding only, "the IESG would simply 
like to see the SPF *and* Sender-ID experiments wrapped up, and presumes 
that the charter is correct and the outcome of the experiment leads one 
to conclude that SPF should get advanced to Proposed Standard and 
Sender-ID should not".

> for declaring the Sender-ID experiment to be complete.

Sure, with the caveat that the shepherd and/or I will be doing a 
document writeup, and (like the rest of the WG and the chairs, I hope) 
we will want to be able to document that the WG did its due diligence to 
come up with a justification for the conclusion of the experiment.

> Our problem

Similarly, due diligence should address the above as well.

>     I suggest we concentrate on gathering such evidence.

This is why I like Andrew and SM's approach to this question: The WG was 
chartered on the assumption (see 1, 2, and 3 in the charter) that there 
exists community consensus that SPF has been more successfully deployed 
than Sender-ID and that therefore SPF should move forward and that no 
further work need be done on Sender-ID. The first order of business is 
to document that SPF *was* more successfully deployed than Sender-ID (by 
enumerating the amount of deployment of each and the "relative 
effectiveness" of each), and that's what I take Andrew to mean by 
providing "experimental evidence".

It would be exceedingly amusing were we to discover that there is *more* 
deployment of Sender-ID or that it has *more* relative effectiveness. We 
might decide to re-charter in that horrifyingly unlikely event. :-)

Assuming none of these very low likelihood events occurs, gathering data 
on how much and how effectively (relative to each other) SPF and 
Sender-ID got deployed seems like the right thing to be doing.

pr

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 07 Feb 2012 08:41:07 -0800

On 2/6/2012 11:00 PM, Pete Resnick wrote:
> The fact is that the IESG thought the above sentence was sufficient to explain
> the deliverable. If we come up with a reasonable interpretation of that
> deliverable, we'll be fine and I will defend it to the hilt.

Pete,

You could well be right, of course. However there's enough history with 
late-stage surprises with the IESG to make it worth considering a relatively 
modest exercise.

It will help /us/ to settle on the list of basic questions we decide to answer, 
in satisfaction of this charter requirement.  It will likely help the process to 
run that presumably short list past the IESG for comment.

d/
-- 

   Dave Crocker

---

From: John Leslie <[email-masked]>
Date: Tue, 7 Feb 2012 12:05:50 -0500

>> of that deliverable, we'll be fine and I will defend it to the hilt.

   (I don't understand how such a nice guy as Pete survives in the arena
of being an AD.)

   I thoroghly agree with Pete here.

> late-stage surprises with the IESG to make it worth considering a 
> relatively modest exercise.

   I disagree with Dave, at least partly wearing my NarrativeScribe hat.

   Such exercises are _not_ modest to the IESG.

> It will help /us/ to settle on the list of basic questions we decide to 
> answer, in satisfaction of this charter requirement.  It will likely help 
> the process to run that presumably short list past the IESG for comment.

   Pete has made us an offer we _really_should_not_ refuse. He has
offered to initiate this discussion wearing his ResponsibleAD hat. It
would be an abuse of process to escalate that to the full IESG unless
and until we have a problem with our ResponsibleAD's interpretation of
the charter item (which Pete has already said he'll mostly defer to us
to write).

   No, Dave, it would not help to escalate this to the IESG now.

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 07 Feb 2012 18:14:19 +0100

>> of v=spf greatly exceeds Sender-ID use of it.

To clarify this point, let me quote the results of Microsoft's wizard
http://www.anti-spamtools.org/ varying the "Scope" radio button:

The Purported Responsible Address (PRA) only
spf2.0/pra [mechanisms omitted] -all

The MAIL FROM (or reverse-path) address only (2 records)
v=spf1 [mechanisms omitted] -all
spf2.0/pra ?all

Both <-- this was the wizard's initial setting
v=spf1 [mechanisms omitted] -all

It doesn't seem to make sense to analyze the published records.  Thus
we can only analyze what's deployed on the checking side.  However, we
cannot assume that mail domains choose their MTA software based on
Sender-ID/SPF support only.  Moreover, some MTA software licenses are
not compatible with OSP or RANDZ.  Therefore, counting existing
software developments, or deployments thereof, will not result in a
true representation of the corresponding effectiveness.

> Assuming [no] low likelihood events occur, gathering data on how
> much and how effectively (relative to each other) SPF and Sender-ID
> got deployed seems like the right thing to be doing.

I'd randomly select some Sender-ID-compliant MTAs that are able to log
"mfrom" and "pra" separately.  We may consider SPF equivalent to
Sender-ID with just "mfrom".  That is to say, let's evaluate how
relevant is the contribution of "pra" relative to "pra" + "mfrom", for
such MTAs.  In effect, it's that "pra" part that we are going to tear
down.  It is reasonable to evaluate it as we go.

The problem is, the last time I checked hotmail, I obtained an ehlo
response with no 250-SUBMITTER line.  Are there any fully compliant
Sender-ID deployments out there?

---

From: S Moonesamy <[email-masked]>
Date: Tue, 07 Feb 2012 10:07:04 -0800

>To clarify this point, let me quote the results of Microsoft's wizard

I could quote what the WG Chairs have been saying but it might be 
futile as it seems that nobody is listening.  If WG participants 
ignore the requests from the WG Chairs, I take it that it is would 
not be unfair for the WG Chairs not to listen to what the WG 
participants say on this thread.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 7 Feb 2012 15:38:48 -0500

Dear colleagues,

On Tue, Feb 07, 2012 at 01:00:22AM -0600, Pete Resnick wrote:

> The WG was chartered on the assumption (see 1, 2, and 3 in the
> charter) that there exists community consensus that SPF has been
> more successfully deployed than Sender-ID and that therefore SPF
> should move forward and that no further work need be done on
> Sender-ID. The first order of business is to document that SPF *was*
> more successfully deployed than Sender-ID (by enumerating the amount
> of deployment of each and the "relative effectiveness" of each), and
> that's what I take Andrew to mean by providing "experimental
> evidence".

That's right, it was, but SM and I can say something more formally:

We have been avoiding trying to interpret what would qualify as
empirical evidence of the end of the experiment, partly in an attempt
to draw out participants' views about the experiment, and partly
because the experimental design did not include a clear statement of
the proposition being tested.

Nevertheless, participants want a clearer statement of what evidence
is desired.  Our AD has offered to take seriously the WG's
good-faith efforts to understand the experiment, and therefore that's
what we will do.

We understand the experiment to have consisted of the simultaneous
deployment of two protocols using the same data from the DNS.  We
interpret the experiment to have been an effort to see which protocol
would be more widely deployed, or more effective, or both.  Therefore,
any evidence about relative deployment, or relative efficacy, would be
welcome.  Evidence of migration from one technology to another would
also be important evidence (really, evidence about relative
deployment).  

Note that this is still _not_ a request for evaluations of the
trade-off.  For instance, if we discovered that the two technologies
were approximately equal in efficacy, then one might make an argument
that one or the other ought to be preferred because it is simpler and
just as effective.  That, however, is a _conclusion_, and not evidence
as such.  Let's please hold off on the conclusions until we have
gathered such evidence as we have.

Best regards,

Andrew and SM

-- 
Andrew Sullivan
[email-masked]

---

From: John Leslie <[email-masked]>
Date: Tue, 7 Feb 2012 16:12:00 -0500

NB: although our WGCs are asking us not to discuss/analyze data, I think
this one thread deserves just a bit more discussion. (But I will not
respond to Dave much longer.)

Dave CROCKER <[email-masked]> wrote:
> 
> 
> Just to clarify, having Pete be the one to bring the list of questions
> to the IESG and -- to make the distinction that Pete raised in his note
> -- to cast it as an informal query and response, are fine.

   Should Pete want to do that, I wouldn't object (but I might send him
private email about what he's getting into).

> My concern is with waiting until the end of our work to get this issue
> reviewed at all by the IESG.

   Reporting on Sender-ID is hardly "the end of our work". Check the
charter milestones...

> For anyone who misunderstand the basis of the concern I am raising,
> or who thinks that an individual AD can make assurances about the
> IESG, I encourage them to review the history of the YAM working group.

   I think Dave and I will have to agree to disagree here. I was scribing
those IESG telechats; and I think my Narrative Minutes support my belief
the IESG of that time was determined to not repeat a YAM-style charter.

   Regardless, today's IESG is different; and I agree with Pete's belief
that his strong support of our "document decribing the SPF/Sender-ID
experiment" will be sufficient to convince other ADs.

   (Where we may get "late surprises" is at IETF-wide LastCall; and
no prior IESG statement can stop those.)

---

From: Philip Gladstone <[email-masked]>
Date: Thu, 09 Feb 2012 09:57:16 -0500

On 2/6/2012 12:46 PM, Philip Gladstone wrote:
>>
> Over the last 4 days, I see 57 IP addresses forwarding emails that 
> were originated from my domain. I *think* that all of these are 
> actually valid forwards (but not doing the appropriate SRS rewriting), 
> though I suspect that there are implementation problems in certain SPF 
> implementations.
>
> For example,
>
> 98.136.219.60: ostmaster
> This says that the addresses forwarded mail from postmaster@mydomain 
> and ostmaster@mydomain. I suspect that the 'ostmaster' is a bug in the 

I think that this points out a potential issue:

* It is not easy/generally possible to find where a broken 
implementation of SPF is. While I can see where the bad DNS request 
comes from, this doesn't help as the address belongs to an DNS recursive 
name server. I don't know how to solve it, but it is an operational issue.

Philip