Subject: [spfbis] How about using SurveyMonkey to ask mail admins how they use SPF?

From: Alessandro Vesely <[email-masked]>
Date: Tue, 24 Apr 2012 18:00:16 +0200

Murray amd all,

we often concluded that it's not feasible to complement the experiment
I-D with data about SPF checks deployment.  Perhaps, we can just ask
it, no?

I attach proposed questions.  Opinions?

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 24 Apr 2012 12:49:01 -0400

No hat.

> I-D with data about SPF checks deployment.  Perhaps, we can just ask

Whom do we ask?  How do we select that sample? 

Moreover, how do we make sure that the sample isn't biased?  What
you're asking for is a self-selected sample.  They're notoriously
tricky to use.  (I don't know very much, but I know really quite a lot
about survey design, and I urge all of you not to give me the
opportunity to bore you.)

A

-- 
Andrew Sullivan
[email-masked]

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 24 Apr 2012 19:17:10 +0200

> Whom do we ask?  How do we select that sample? 

Posting to the mailing lists we know of.

> tricky to use.

Well, I think we'll be able to see if answers make some sense.  In
addition, it may allow us and the readers of the experiment to take
the pulse of the current interest in SPF.

> (I don't know very much, but I know really quite a lot about survey
> design, and I urge all of you not to give me the opportunity to
> bore you.)

If you have some practical advice, I think it wouldn't be boring.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 24 Apr 2012 14:11:04 -0400

No hat.

> Posting to the mailing lists we know of.

This is approximately the worst kind of sample there is, I'm afraid.
It's actually no better than those "surveys" you see on the sidebar of
webpages.  The number that comes out of it is a number, all right, but
it doesn't measure anything.

> Well, I think we'll be able to see if answers make some sense. 

How?  What that really says is, "I already know the answer; I just
want to be able to point at something to back it up."  I actually
think such surveys have negative value, because they tend to be
designed to give the presupposed outcome.

Surveys are excellent ways to find out what people believe about
themselves.  They're a lousy way to find out whether those beliefs are
true.

The evidence we have does not ask anyone to provide their opinion
about what they think they do.  Instead, it takes actual evidence from
deployed systems.  I grant you that the cross-section of deployed
systems is narrow.  It would be considerably better if we had a way of
soliciting logs from a wider array of mail providers, or if we had a
tool that people could run across their mail logs to answer certain
questions.  In the absence of someone stepping up to do all that work
(and to convince people to share their logs with us), I don't think a
survey of admins is the right answer.  I'd prefer to stick with
narrowly empirical data and draw conclusions from that, with the
caveat that we are assuming these mail systems are representative of
all mail systems.  

 In
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis

-- 
Andrew Sullivan
[email-masked]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 24 Apr 2012 18:27:10 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Andrew Sullivan
> Sent: Tuesday, April 24, 2012 11:11 AM
> To: [email-masked]
> Subject: Re: [spfbis] How about using SurveyMonkey to ask mail admins how they use SPF?
> 
> The evidence we have does not ask anyone to provide their opinion about
> what they think they do.  Instead, it takes actual evidence from
> narrowly empirical data and draw conclusions from that, with the caveat
> that we are assuming these mail systems are representative of all mail
> systems.

A note to the effect of your last sentence seems appropriate to add to the document.

-MSK

---

From: "John Levine" <[email-masked]>
Date: 24 Apr 2012 19:35:02 -0000

>questions.

I suppose, but the reality of the mail ecosystem is that a small group
of providers handles a large fraction of the mail, with an extremely
long tail of medium and small systems that handle ever smaller mail
streams.

We happen to have data from a couple of the largest systems, and we
have empirical evidence that gives us a pretty good idea of what other
large mail sysetms do with SPF.  So it may be narrow, but it's not
small.  I find it hard to imagine that we would find anything
interestingly different if we could survey 10,000 other mail systems.

Does anyone really believe that there is significant use of type 99
records anywhere?  Does anyone but Ale think that there are
significant numbers of mail recipients doing reject on SPF fail?  (Or
even if there were, that there is any possibility of amending RFC
5321?)  There is nothing we could plausibly learn from a survey that
would change what we're going to do in the next few months: we'll
clean up the nits in 4408, deprecate type 99 records, perhaps figure
out some politically acceptable way to say that nobody uses Sender-ID,
and that's it.

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 25 Apr 2012 12:17:55 +0200

> long tail of medium and small systems that handle ever smaller mail

Given that reality, I think it's not fair nor desirable to treat a few
large providers as representative of many small ones.

> large mail systems do with SPF.  So it may be narrow, but it's not

In theory, 10,000 providers with 10,000 messages per day move as much
traffic as a provider with a hundred million messages.  That's not
unworthy, but difficult to reach.

> Does anyone but Ale think that there are significant numbers of
> mail recipients doing reject on SPF fail?

I don't think that: I just don't know it.  That's why I'm thinking to
take a survey.  The other possibility is to run the test I mentioned
in http://www.ietf.org/mail-archive/web/spfbis/current/msg01224.html

I have to hurry up...

> There is nothing we could plausibly learn from a survey that would
> change what we're going to do in the next few months: we'll clean
> up the nits in 4408, deprecate type 99 records, perhaps figure out
> some politically acceptable way to say that nobody uses Sender-ID, 
> and that's it.

I agree on the schedule (albeit with a mental reservation).  However,
some of our readers may be developers of SPF tools, and I think they
would be interested to know how many people will take the burden to
answer a survey on SPF.

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 25 Apr 2012 13:09:47 +0200

> it doesn't measure anything.

An other way to find mail admins would be to write to the postmaster@
address of a number of domains, which is obviously not practicable.
Given the way that domain names, network connections, and much MTA
software are distributed, I see no other way to address the category.

> want to be able to point at something to back it up."

That's only true for the DNS question.

> I actually think such surveys have negative value, because they
> tend to be designed to give the presupposed outcome.

I'll try and avoid that tendency the best I can.

> themselves.  They're a lousy way to find out whether those beliefs are

Right.  So you'd say the survey should focus on evaluation and
expectations about SPF in general?  Doesn't a value such as "I believe
our domain deploys some form of SRS rewriting" measure some
predisposition toward adapting to SPF requirements?

> about what they think they do.  Instead, it takes actual evidence
> from deployed systems.

Yes, and that is probably ways more objective.  I'm not proposing to
/replace/ any data we already have, but to try and corroborate and
augment it.

> I grant you that the cross-section of deployed systems is narrow.
> It would be considerably better if we had a way of soliciting logs
> from a wider array of mail providers, or if we had a tool that
> people could run across their mail logs to answer certain 
> questions.

I'd agree, if it were feasible in a month.

> In the absence of someone stepping up to do all that work (and to
> convince people to share their logs with us), I don't think a 
> caveat that we are assuming these mail systems are representative
> of all mail systems.

OTOH, those of us (possibly an extended "us") who'd step up to do that
work would have to build a mail-admins channel first.  Wouldn't it be
neat to take a survey before investing in such an effort?

I can well understand that the WG considers the data collected thus
far sufficient for our needs.  What I fail to grasp is why one would
refuse to consider additional data.  However skewed, data cannot have
a negative value.  Negative information --or entropy-- is when the
data is forgotten or otherwise hidden.  Do we think a survey could
sort out such an effect??

---

From: Hector Santos <[email-masked]>
Date: Wed, 25 Apr 2012 09:50:17 -0400

> sort out such an effect??

I don't. Never mind that you won't get the sampling needed, surveys 
can be skewed in any direction you want. Plus, without a doubt, based 
on my engineering training and nearly 25+ years of real mail system 
design of proprietary and open standards, I would have a completely 
different set of R&D questions.  I don't need to concur with Mr. 
Sullivan, but I doubt we would even be able to get agreement on what 
should be the survey questions.

To me, this is not rocket science.  Just considering one 
FEATURE-BY-DESIGN part with SPF - POLICIES.  In principle, what 
policies are affordable depends on type of service, scale and mode of 
operation. For example, who can afford to use a strong policy like -ALL?

   - LARGE PUBLIC SITES SITES, ESP, ISP, HOSTING sites
        could LESS afford to offer/use -ALL

   - SMALL TO MID PRIVATE/PUBLIC SITES
        could begin to look at -ALL

   - HOBBY, MOM&POP TO SMALL PRIVATE/PUBLIC SITES,
        could afford more to use -ALL

and that also relates to the affordability of local deployment options 
for Marking vs Rejection.   Thats not a hard core rule, its generally 
considered that the smaller you are the more nimble you are with 
advanced technology.  Many larger companies simply do not have the 
luxury to "change" as fast as one wants, just drop this or that and 
switch to something else.  Sure, they can talk about, go to meetings, 
trade shows, hear presentations and even present them, etc. That 
doesn't always mean they actually can afford the change.  Big ISP can 
announce they will begin doing this or that, perhaps the Quality 
Circle meetings had help desk people say they can same money, and they 
do it, but find out quickly it wasn't actually cost effective with 
other things come out of the word works. BIG ESP can announce they 
will enforce XYZ in 2005, only to see in 2012, they don't.  Perhaps 
they tried and saw it wasn't working well for them.

I will suggest there are two central difference with many in the mail 
market:

   - Those who prefer to push delivery decisions to users vs those who 
also have
     system level backend focuses.  The latter has been where the big 
changes are
     because traditionally it was really never the transport system 
business to
     get involved in these decisions, at least not automatically 
without making it
     an local site option.

   - Those who prefer to focus on GOOD MAIL filtering vs BAD MAIL 
filtering.

Almost everything we do and basically "fight" about and also vendors 
"compete" when it comes to mail, has its genesis with either one or 
both.  What they offer, how they do it, what works or doesn't work for 
one, is not always the same across the board, etc.

There is no right or wrong, but unfortunately, it breaks down when 
compromise is lost in seeking the commonality - the protocol mechanics 
or basically "physics" of the mail system we all work with.

-- 
Hector Santos, CTO
http://www.santronics.com
http://hector.wildcatblog.com

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 27 Apr 2012 08:00:04 +0200

On Fri 27/Apr/2012 07:53:49 +0200 Hector Santos wrote:
> Alessandro Vesely wrote:
> 
>> Do we think a survey could sort out such an effect??
> 

Here's what I've been able to come up with:

  https://www.surveymonkey.com/s/SPF-deployment-survey

Please complete it.  It will only take three minutes (ten questions on
a single page).

Please help reaching all mail admins by posting that link to
specialized mailing lists and mail admins you are in touch with,
worldwide.  Thank you in advance!

---

From: Hector Santos <[email-masked]>
Date: Fri, 27 Apr 2012 08:37:17 -0400

> Please complete it.  It will only take three minutes (ten questions on

Out of curiosity, I completely the survey.

> worldwide.  Thank you in advance!

I would like to support your effort, but I don't believe this survey 
caters to all market base profiles, for example, as a vendor, I had to 
check all the options for all the questions since all of them applied. 
  Overall, IMO this survey should not have any bearing on 4408BIS 
decision making.  The functional (What we want) and technical (how it 
is done) specification should be agnostic to the level of scale and 
usage of any one or more particular protocol feature.

There of many examples to show but just using the question with the 
option:

   [_] An SPF result of FAIL causes the message to be rejected immediately

is moot because it is already part of the RFC4405-4408 functional and 
technical specification that is part of the protocol as an deployment 
option.  The skewed result showing 33% is meaningless unless the 
surveys intent is to change the specification to eliminate this 
existing RFC4408 deployment option for 4408BIS.

What would had of been interesting is to extract the type of operation 
the survey user has for a concentrated motivation in his specific mode 
of SPF deployment.  For example,

    Please describe your role as a SPF implementator or deployment:

    [_] Vendor (for resell, others to use)
    [_] Operators of vendor software (commercial, free, open source, etc)
    [_] ISV or 3rd party developers (write plug-ins for vendor software)
    [_] In-house, custom, provisional developers (not for others to use)
    [_] Other role

This allows you to see the level of protocol support, what features 
are used, what can they do or not do, etc. For example, a vendor 
generally needs to make available all options and protocol features 
available for operators to choose, where other types will just be 
privy to a need specific world of smaller needs and what features are 
turned on or off, etc.

Very different motivations, and IMV a robust protocol is designed for 
all roles despite what one group, dominant or otherwise, may suggest 
is better for all.

IMV, its interesting, but this survey can not change the functional 
and technical specification of 4408BIS.  A proper survey should 
attempt to see where there are protocol conflicts and use this to make 
the corrections, i.e. like the above question related to 
REJECT-ON-FAIL currently allowed by the RFC4408 specification. The 
suggestion and implication of a "Almost Nobody" does this is moot 
unless again the intent is to propose the removal of rejection 
semantics from the 4408 specification

---

From: Hector Santos <[email-masked]>
Date: Fri, 27 Apr 2012 10:41:32 -0400

A follow up note:

Hector Santos wrote:
> 
> There of many examples to show but just using the question with the option:
> 
> option.  The skewed result showing 33% is meaningless unless the surveys 
> intent is to change the specification to eliminate this existing RFC4408 
> deployment option for 4408BIS.

To show there is no bias, lets assume the percentage here showed 
something that was 80% of more for immediate rejection deployment 
majority.

Does that have any significant meaning the current MARK-ON-FAIL 
receiver option is no longer significant due to a low method usage and 
therefore should be removed from the specification so that 
REJECT-ON-FAIL is the only consistent deployment expectation?

No, I submit it does not because its a valid protocol option, 
dangerous in my personal view, but nonetheless valid as part of the 
current functional specs already established in 4408.

But IMV, the survey might include questions related to the 4408 lack 
of a technical specification for MARK-ON-FAIL option left undefined or 
left open-ended by design:

    Please describe how MARK-ON-FAIL is implemented in your SPF server:

    [_] Received-SPF: fail is recorded
    [_] Authentication-Result: spf=fail is recorded
    [_] SPF fail mail is quarantined into a "junk email" or similar 
user folder.
    [_] SPF fail mail is bundled with user's MUA POP3 protocol mail 
pickup.

and related:

    What kind of MUA portal access if allowed on your server?

    [_] Online MUA, i.e. Web Mail, Mobile device, i.e THIN DEVICE
    [_] Offline MUA with POP3 access,
    [_] Mixed Online/Offline MUA with IMAP access
    [_] Other

For systems that offer a full range of multi-device mail access 
portals as illustrated in the following diagram, a common protocol 
that works with all methods at the backend and does not allow for 
security loopholes is critical important.

    http://beta.winserver.com/public/wcrpc/wcrpcnet7.png

Assuming an IETF protocol-based offline MUA as the primary method and 
the only method tp consider to access mail is IMV a critical fault for 
a protocol such as SPF.

While the survey does not ask these protocol deployment questions, I 
am hoping the WG will recognize the consideration come 4408 updating 
with 4408BIS.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 27 Apr 2012 08:29:18 -0700

> opportunity to bore you.)

Just to add another common negative into the mix:

Surveys are typically good for recording personal preferences and 
attitudes and strikingly bad at recording 'behaviors', especially 
estimates of performance numbers.

Hence, a survey like this is probably going to measure respondent's 
beliefs and attitudes, rather than producing hard numbers.

As such, I can't imagine any benefit in the survey.  We don't need the 
information it will produce.

d/

-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 27 Apr 2012 16:35:53 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Crocker
> Sent: Friday, April 27, 2012 8:29 AM
> To: Andrew Sullivan
> Cc: [email-masked]

I'm also concerned with anonymity and data pollution.  If you were to ask for answers about the DNS and MTA surveys, I can tell you which domain said what when and how fast.  If I were to look at one response from a survey of people, there's subjectivity in the answer.  And each DNS server or MTA polled gets exactly one Boolean answer, but there's no guarantee of that in a human survey.

Even with the mush around the record re-use question, I think conclusions based on the DNS and MTA surveys will be substantially more defensible.

-MSK

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 27 Apr 2012 20:07:49 +0200

> estimates of performance numbers.

Yup.  I don't get why this is deemed negative.

> beliefs and attitudes, rather than producing hard numbers.

We already have some hard numbers.

> information it will produce.

However we'll craft the SPF spec, that is going to redound upon mail
admins' beliefs and attitudes.  If we are better informed, we can
possibly be more effective.