Subject: Re: [spfbis] WGLC for draft-ietf-spfbis-experiment

From: "John R Levine" <[email-masked]>
Date: 25 Apr 2012 23:03:20 -0400

> I believe this doc is meant to present actual results observed in 
> operation. Are there results that show that mail has been rejected under 
> the conditions you describe?

Rejected?  Almost nobody rejects on SPF or S-ID failure, so that seems 
relatively unlikely.  But has mail been misfiled due to scoring systems 
that use S-ID?  Who knows?

The IESG insisted in putting a big skull and crossbones on each of 4405 through 
4408 about how bad things might happen due to the DNS reuse, so it seems a bit 
odd to say nothing at all about it.

Our numbers say that somewhere between 5% and 20% of mail gets different 
results with S-ID and SPF checks.  Since we saw almost no S-ID records 
published, the reasonable assumption is that most of those different answers 
are due to S-ID reusing SPF records.  I don't know how many of those different 
results led to mishandling the mail, but it seems rash to assume that none of 
them did, and it does pretty clearly indicate that in practice, S-ID will get 
the wrong answer (in the sense of not what the domain's owner intended) a 
significant fraction of the time.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 Apr 2012 00:05:16 -0400

>> operation. Are there results that show that mail has been rejected 
>> under the conditions you describe?
> 
> relatively unlikely.  

-1.  Our operator's local site policy default setup is to 
REJECT-ON-FAIL. Almost nobody also means somebody so it is relatively 
likely to happen.

-- 
Hector Santos, CTO
http://www.santronics.com
http://hector.wildcatblog.com

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 26 Apr 2012 05:02:29 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John R Levine
> Sent: Wednesday, April 25, 2012 8:03 PM
> To: [email-masked]
> Subject: Re: [spfbis] WGLC for draft-ietf-spfbis-experiment
> 
> Our numbers say that somewhere between 5% and 20% of mail gets
> different results with S-ID and SPF checks.  Since we saw almost no S-
> ID records published, the reasonable assumption is that most of those
> different answers are due to S-ID reusing SPF records.

I don't agree that that's the only reasonable assumption.  It seems more likely to me that different answers might also come from the fact that the PRA produced a different domain than the MAIL FROM domain in those cases, resulting in a different policy being evaluated altogether, independent of the DNS reuse question.

I don't think the SIDF re-use of SPF records is an important thing to call out in the experiment document because its contribution to the stats is hypothetical.  On the other hand, this could very well be something to mention in Security Considerations of RFC4408bis itself.

-MSK

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 Apr 2012 01:40:44 -0400

+1

---

From: "John Levine" <[email-masked]>
Date: 26 Apr 2012 12:19:57 -0000

>I don't agree that that's the only reasonable assumption.  It seems more likely to me
>that different answers might also come from the fact that the PRA produced a
>different domain than the MAIL FROM domain in those cases, resulting in a different
>policy being evaluated altogether, independent of the DNS reuse question.

Except that there isn't anything like 5% of the senders publishing
S-ID records.  Those checks probably did use a different domain, but
if they checked it against an SPF record intended for envelope
domains, that's a bug.

Imagine a hypothetical survey, in which we ask mail system managers
who publish SPF records whether they also intend those records to be
used for Sender-ID.  I would expect the overwhelming response to be
"What's Sender-ID?"

>I don't think the SIDF re-use of SPF records is an important thing to call out in the
>experiment document because its contribution to the stats is hypothetical.

I suppose if we're all OK with the implicit messge that the skull and
crossbones was silly, we could drop it.  But I'm kind of surprised
nobody on the IESG has mentioned it.  Short memories, perhaps.

R's,
John

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 26 Apr 2012 13:57:14 +0000

> From: John Levine [mailto:[email-masked]]
> Sent: Thursday, April 26, 2012 5:20 AM
> To: [email-masked]
> Cc: Murray S. Kucherawy
> >I don't agree that that's the only reasonable assumption.  It seems
> >more likely to me that different answers might also come from the fact
> >that the PRA produced a different domain than the MAIL FROM domain in
> >those cases, resulting in a different policy being evaluated
> altogether, independent of the DNS reuse question.
> 
> Except that there isn't anything like 5% of the senders publishing S-ID
> records.  Those checks probably did use a different domain, but if they
> checked it against an SPF record intended for envelope domains, that's
> a bug.

I don't think this is the right document for that kind of analysis.  In my opinion, we should keep it focused on the question of what got adopted, as that's enough to answer the "consensus" question from the IESG note and advance SPF to the Standards Track.

And we've seen how even collecting and analyzing data about what's in use, separate from efficacy, can rathole on a bunch of hypothetical and orthogonal questions even though that analysis isn't subjective.  It's scary to think about how long an in-depth analysis about those other questions would take to reach consensus.

-MSK

---

From: "John Levine" <[email-masked]>
Date: 26 Apr 2012 15:40:29 -0000

>adopted, as that's enough to answer the "consensus" question from the
>IESG note and advance SPF to the Standards Track.

OK.  I take it that the consensus of the group is that the warning
about DNS record reuse was too or trivial to be worth addressing.

We don't have an enormous amount of data, but we do have some.

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Thu, 26 Apr 2012 08:57:09 -0700

On 4/26/2012 6:57 AM, Murray S. Kucherawy wrote:
> I don't think this is the right document for that kind of analysis.
> In my opinion, we should keep it focused on the question of what got

+1

This document does not need to venture into conjecture.

Such analysis might be fun and even productive, but it's not needed for 
this document and it's clear this document should do only the minimum it 
needs to, to satisfy its reason for being written.

d/
-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 26 Apr 2012 16:53:09 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of John Levine
> Sent: Thursday, April 26, 2012 8:40 AM
> To: [email-masked]
> Cc: Murray S. Kucherawy

...in this document.  (To be clear about my position, at any rate.)

---

From: Douglas Otis <[email-masked]>
Date: Thu, 26 Apr 2012 10:48:41 -0700

> I don't think the SIDF re-use of SPF records is an important thing to call out in the experiment document because its contribution to the stats is hypothetical.  On the other hand, this could very…
Dear Murray,

Agreed.  The RFC4408bis document is a good location for this type of 
warning.

As an aside, the DMARC group said large organizations have been 
implementing their combined SPF+DKIM strategy since 2007.  Would they be 
willing to make last minute contributions?  This seems appropriate since 
this appears where SPF is headed based upon the level of corporate buy in.

BTW, prior statements had been based on information presented by Sam 
Masiello IIRC.  I had not yet discovered the latest document.  Many of 
the links were stale.   Not that this parallel effort seems anything 
like shades of the past, but is there a reason why your DMARC draft is 
not published as an IETF document?

Regards,
Douglas Otis

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 26 Apr 2012 17:56:23 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Douglas Otis
> Sent: Thursday, April 26, 2012 10:49 AM
> To: [email-masked]
> implementing their combined SPF+DKIM strategy since 2007.  Would they
> be willing to make last minute contributions?  This seems appropriate
> since this appears where SPF is headed based upon the level of
> corporate buy in.

What sort of contributions?

> not published as an IETF document?

Mostly change control reasons, in the same way that the ESTG kept control of DKIM until they were prepared to hand it off to the IETF.  But IETF processing of DMARC is indeed the end goal.

But again, I think this is off-topic for spfbis.

-MSK, juggling hats quite a bit here

---

From: Douglas Otis <[email-masked]>
Date: Thu, 26 Apr 2012 12:11:28 -0700

Dear Murray,

In essence, points made by Hotmail et al  was in regard to quality of 
results compared against that of Sender-ID.  It sounded like these 
results were seen as sufficient even for those who were stalwart 
Sender-ID advocates.  The highlights of this information suggested that 
it strongly supported claims made by John Levine, for example.  While 
this might seem like a reach, perhaps it would also help satisfy those 
less familiar with SPF+DKIM in conjunction with "Organizational Domain" 
policy assertions.  It seems "Organizational Domain" is new to the 
IETF.  But since this is where traction is growing in establishing 
greater levels of policy enforcement, it seems this information could 
offer rather convincing support of the notion for dropping Sender-ID 
when the policy function is better handled by SPF in conjunction with 
DKIM and Organizational Domain policies.

Regards,
Douglas Otis

---

From: Paul Midgen <[email-masked]>
Date: Thu, 26 Apr 2012 22:43:23 -0700

Hotmail hat on.

Teeing up Hotmail and/or Microsoft as SenderID advocates == not the 
droids we're looking for.

The reality is that the folks for whom this was a political (or other) 
crusade are long gone, and it's just a business (for the enterprise 
groups like Exchange) or technical (for Hotmail) issue to be evaluated 
on the merits of the benefit it provides. For Hotmail it was simple: the 
end result in terms of user experience are pretty much the same either 
way, most folks use SPF, SPFbis is now an IETF working group, ergo we 
will get behind SPF because speeding it along to Standards Track means 
other interesting work can happen.

So I don't see the point in trying to record some kind of historical 
note about the intentions of folks publishing SPF1 records with respect 
to how SenderID verifiers would use those records. Outside of 
hypothetical discussions, it has proven immaterial in practice.

It is safe to say that in terms of received mail volume and source 
domain diversity, Hotmail was the largest SenderID validator who used 
authentication results in making delivery decisions. We never ran into 
an issue where SenderID caused a problem that SPF wouldn't have. We 
never experienced any kind of security issue (e.g. elevation of 
privilege, policy avoidance) due to use of SenderID with SPF1 records 
that was of sufficient magnitude to be reported by a sender or 
recipient, or be noticed by our own security teams.

I am NOT saying such things aren't possible, others have proven they 
are. Just that we have no proof a real live bad guy exploited that 
potential at Hotmail in its history using SenderID, and since Hotmail is 
moving to SPF a future example is unlikely to obtain. The move away from 
SenderID actually had some drawbacks requiring workarounds, but these 
things weren't deterrent enough to stop the move to SPF.

So, the purpose of the "experiment conclusion" document is to be a 
historical record of the SPF/SenderID experiment and identify SPF(bis) 
as the way forward, and I believe the current draft does that well enough.

No offense intended to anyone on this list, but operational consensus 
was unambiguously clear before SPFbis was formed. All subsequent 
debate/discussion hasn't really changed that fact, or will ultimately 
alter the outcome. We're here to do the work of documenting history and 
reductionary refinement of SPF.

On 26-Apr/12 12:11 PM, Douglas Otis wrote:
> Sender-ID advocates.  The highlights of this information suggested 
> that it strongly supported claims made by John Levine, for example.  
> While this might seem like a reach, perhaps it would also help satisfy 
> those less familiar with SPF+DKIM in conjunction with "Organizational 
> Domain" policy assertions.  It seems "Organizational Domain" is new to 
> the IETF.  But since this is where traction is growing in establishing 
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis

---

From: Paul Midgen <[email-masked]>
Date: Sat, 28 Apr 2012 12:08:07 -0700

WG chairs: having read/commented/suggested changes to several revisions 
of draft-ietf-spfbis-experiment please consider this my official +1 in 
favor of approving the WG last-call text.