Subject: Re: [spfbis] Suggestion for an additional deliverable

From: Douglas Otis <[email-masked]>
Date: Tue, 03 Jan 2012 11:36:44 -0800

On 12/30/11 6:11 PM, Hector Santos wrote:
> Douglas Otis wrote:
>> Is there any interest in assuring use of mx mechanism will work 
>> properly with domains publishing more than 5 MX records? 
> Can you illustrate the MX concern and whether there is an SPF problem 
> here?
Hector,

Most often this occurred as result of "default" SPF records applied 
where none were found.  A common practice.  A reduction in limits of 
subsequent hosts being resolved in many scripts is of course welcome.  
IIRC, t-online.de had more than 5 MX records, which now has 5 and still 
no SPF record.  This problem was caused for senders not attempting to 
use SPF.

There remains a significant discrepancy between implementation and 
specification.  Many implementations do not accommodate common MX record 
use.  The recommended subsequent recursion limit has not been followed 
for good reason.  A specification that offers no warning or recommended 
minimum has not helped to ensure interchange.
>> Many current SPF libraries process a maximum of 5 rather than the 
>> specified 10 to circumvent a possible 100+ DNS transactional 
>> sequence.  The impact is this may cause intermittent failures 
>> difficult to troubleshoot.  What makes this contentious is possible 
>> macro expansion of local-parts against cached SPF records.
>
> Doug, its been what?  nearly 9, 10 years? I have yet to see a 
> situation that was not managed with the SPF implementation, especially 
> with recursive parts where limits are placed.
We have a large complaint center.:^)   At a recent MAAWG meeting,  I led 
an ISP specific discussion regarding SPF use.  There was consensus that 
rejecting failing SPF records ending with "-all" cause a large number of 
complaints.  SPF registration is poorly understood and often misapplied 
even now.  SPF records offering a pass do not necessarily confirm the 
domain originating the message either.

In general, efforts are aimed at improving message acceptance rates.  
This has led some phishing sensitive senders to enter into private 
arrangements to ensure invalid DKIM messages are rejected.  Even DKIM's 
low failure rate still has some wanting a fallback alternative.  IMHO, 
address registration will remain and will become increasingly problematic.

Few are willing to process million per second log entries.  The SPF 
macro flexibility offered in terms of requesting ANY RR by the SPF 
"exists" mechanism, made against hosts derived from components of 
local-part addresses and script is not something easily ascertained.  
Email handles requests primarily originating directly or indirectly from 
compromised systems.  Email's lack of source authentication has resulted 
in several orders of magnitude greater criminal use when compared to 
HTTP related sources.  A poorly considered comparison when used to 
dismiss concerns.
> I have yet to see or hear of any exploit that circumvents SPF 
> implementations.  Its all theoretical with the presumption the 
> implementation is not aware of aggressive abuse.
Few recipients expending resources processing SPF macros are likely to 
be the target of any attack and few retain a log of this activity.  The 
targeted victim or the domain used to stage the attack need not appear 
anywhere within attacking messages.  If SPF required specific resources 
be used either by type or by label, there might be a way to ascertain 
SPF's involvement.  SPF does neither.  The flexibility offered by SPF 
macros also means use of domain reputation will prove ineffective at 
mitigating this threat as well.

-Doug

---

From: Hector Santos <[email-masked]>
Date: Tue, 03 Jan 2012 16:47:07 -0500

Douglas Otis wrote:
> where none were found.  A common practice. 

Agreed, and probably the initial WIZARDS had a lot to do with that 
because initially when there was skepticism, the default was a neutral 
policy - a very wasteful policy especially when nothing else is 
available to give it any weight - negatively or positively.  SPF-BIS 
should make a note that any attempt to explicitly declare a SPF policy 
SHOULD NOT be a wasteful policy for SPF receiver to process.

> A reduction in limits of 

Not sure of the issue here since there is no SPF record for t-online.de.

But overall, any DNS lookup for records will have limits placed. 
Including for MX and expansions.  Robust DNS Clients are expected to 
be part of the SPF tool chest.

> for good reason.  

Where is the proof that it hasn't been followed?   It would not be 
compliant if it did not control recursion.

> A specification that offers no warning or recommended 

Well, maybe there is some truth to this "theoretically" but I have not 
seen any evidence of it.  Limits are prudent SPF design otherwise you 
will get bitten.

> domain originating the message either.

I totally disagree with that consensus purely on the basis of its 
audience.  Its like a gathering of all Republicans all agreeing that 
Obama is a BAD person.  No truth to it.  If I was in the Reputation 
Business, you can bet your boots I would have a strategic concern 
about how SPF does not HELP the business.

I look at the technical merits.  A Hard Policy is the most powerful, 
the most deterministic, the most beneficial and lowest overhead you 
can get.  Of course, if you don't honor it, then you are undermining 
the Domain wishes, defeating its purpose and you will get a lot of 
overhead.

Thats the problem I have with this GROUP - a majority of people who 
simply NEVER gave SPF any light in day - EVER. I mean it. So their 
blurts of failure is burned in and there was absolutely NEVER any 
evidence of that. That is why it had prosper despite all these 
erroneous negative statements about it.  It is can stated in trade 
groups, doesn't make it true whatsoever.

Now, if you said weaker indeterminate wasteful policies like SOFT or 
NEUTRAL, I agreed that that is a serious problem.

But not the hard policy.

> address registration will remain and will become increasingly problematic.

DKIM has failed because it destroyed its HARD POLICY framework, making 
it weak and indeterminate, wasteful and so relaxed that no one 
required to following any kind of strong assertion about its presence 
in email.

> HTTP related sources.  A poorly considered comparison when used to 

Email's lack of ability to offer strong HARD policies is whats keeping 
this all open-ended and wasteful.  SPF offers strong deterministic 
policies. So does DKIM with ADSP and its extensions with ACL and ATPS. 
  If you wish to ignore these strong domain policies, then I don't how 
anyone can make a claim that there is a solution to any of these work, 
at any level.  It will simply never WORK without some level of strong 
DOMAIN policies that receivers follow.

It doesn't matter if its TRUST related - if we have support for TRUST 
working today, the same concept applies - you TRUST the domain or not 
and that information has to be passed to receivers one way or another.

No matter what it is - hard core TRUE/FALSE policies are required 
about failure and the expectations by originating owners, domains or 
just plain people  making a declaration about the way they do things.

> mitigating this threat as well.

The only issue with SPF macros is the recursion potential and Doug, 
that is already 100% dealt with.  Old news.

The bottom line, there is no systemic problem with SPF and all the 
blurted catastrophic concerns about SPF is just that - erroneous fear 
mongering to replace it with other things and none of it was ever true 
nor materialized.

It can be stated with any technology or protocol, including SMTP, FTP, 
NNTP, HTTP, etc, there is always some level of it being employed 
wrong.  That absolutely does not suggest its universally true or the 
protocol is inherently problematic.

Finally, I know each site has its own statistics, but looking at our 
3-4 days for the DNS cache of SPF records, 2 of 613 had macros rules 
and they were attempts to record usage. And among those 2, one of 
those had a failed syntax.  Go figure.

Here was the break down:

- Total               :   613
- Total Bad           :    15
- Total SPF1          :   583
- Total SPF2          :    15  (SENDER-ID)

# of directives

- Total MX            :   435
- Total PTR           :   325
- Total IP4           :   900
- Total IP6           :     3
- Total A             :   437
- Total INCLUDE       :   145
- Total REDIRECT      :     5
- Total EXISTS        :     2

Policy Types:

- Total PASS          :     6
- Total FAIL          :   110
- Total SOFTFAIL      :   411
- Total NEUTRAL       :    86 (17 implicit)

Directives with Policy Prefixes:

- Total TAG PASS      :     0
- Total TAG FAIL      :     1
- Total TAG SOFTFAIL  :     0
- Total TAG NEUTRAL   :     2

These numbers mean number, but only to show the high amount of 
HARD+SOFTFAIL.

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

---

From: Douglas Otis <[email-masked]>
Date: Wed, 04 Jan 2012 15:36:50 -0800

On 1/3/12 1:47 PM, Hector Santos wrote:
> >> Can you illustrate the MX concern and whether there is an SPF
> >> problem here?
> > Hector,
> >
>  because initially when there was skepticism, the default was a
>  neutral policy - a very wasteful policy especially when nothing else
>  is available to give it any weight - negatively or positively.
>  SPF-BIS should make a note that any attempt to explicitly declare a
>  SPF policy SHOULD NOT be a wasteful policy for SPF receiver to
>  process.
Hector,

Not exactly.  Some will substitute "No SPF Record" with a synthetic 
"v=spf1 a mx/24 ?all".  This could produce a difficult to diagnose loss 
of DSNs or sporadic junk folder message placements.

> > A reduction in limits of subsequent hosts being resolved in many
> > scripts is of course welcome. IIRC, t-online.de had more than 5 MX
> > records, which now has 5 and still no SPF record. This problem was
> > caused for senders not attempting to use SPF.

By not having an SPF record, some receivers employ "synthetic" records 
instead.  The mx mechanism is flaky when generally applied largely 
because Wayne Schlitt changed the number of mx records his SPF library 
processed from 10 to 5.  While a commendable change, it led to 
interchange issues which demand no more than 5 MX records be published, 
especially for domains making use of mx mechanisms in their records!  
The publisher does not control processing limits employed by recipients 
and the specification offers no minimum a requirement.

Some have also suggested those affected by a long sequence of bogus host 
name requests result from improper DNS configuration.  Per RFC2308, the 
SOA minimum field has had two previous meanings but now provides the 
requested duration of negative responses.  Unfortunately, due to 
problems of blocked access leading to support calls, often recipients 
will override requested negative caching durations.

Windows 2000 supports negative caching as specified in RFC2308, but with 
modifications of low maximum limits. Negative responses are held for the 
number of seconds specified in the NegativeCacheTime value with a 
default limit of 300 seconds.  The value of NegativeCacheTime can be set 
to 0.  However, it can not be greater than 15 minutes.  For Windows 
2008, the dnscmd supports /maxnegativecachettl with a default of 10 
minutes.  Bind 8 and 9 also provide global max-ncache-ttl to limit 
negative caching.  Cisco offers similar methods to limit negative 
caching using the command dns set max-negcache-ttl that defaults to 1 hour.

It would cost malefactors little of their resources to publish an SPF 
record:
"v=spf1 exists:%{l}0.victim-domain exists:%{l}1.victim 
exists:%{l}2.victim-domain exists:%{l}3.victim-domain 
exists:%{l}4.victim-domain exists:%{l}5.victim-domain 
exists:%{l}6.victim-domain exists:%{l}7.victim-domain 
exists:%{l}8.victim-domain exists:%{l}9.victim-domain +all"

This record will generate no traffic from the attacker's DNS after the 
first message, but may generate an endless series of 10 A record 
transactions/message  against a victim domain when either the local-part 
in their return-path in their spam campaign does not repeat, or repeats 
after a sequence that is beyond negative cache limits commonly 
employed.  Mx mechanisms can extend this sequence to 100/message.  Our 
logs show malefactors have no problem not repeating.  The source of 
these messages is typically a script running on a compromised system 
able to generate random local-part values.  Most email is emitted from 
compromised systems where neither SPF nor Authentication-Results header 
fields identify or confirm the transmitter.  The authentication-Results 
header field even fails to capture the source IP address!

>  be part of the SPF tool chest.

DNS clients and script processing libraries are not controlled by SPF 
record publishers.  A client being robust does not help targeted victims 
either.  Spammers use a technique some describe as a snowshoe method.  
Levels low enough to hide within the noise.  SPF macros allow an attack 
to be broadly distributed at levels not noticed by any recipient, but 
still focused against a single victim.  The victim that can not be 
deduced by message content, where victims are impacted to the point of 
having their link saturated with UDP traffic from an unlimited number of 
sources that can't be blocked or mitigated.

When dealing with security, once demonstrated, seldom does one ask 
whether it is currently being exploited before looking for solutions.  
With SPF, the macro scripts are able to exploit far too much of the 
recipient's resources.   Even a 1 to 1 reflected network amplification, 
rather than 150 to 1, or 50 to 1 represents a serious concern.  
Especially when a DDoS can be implemented concurrently with spamming.   
Often DDoS attacks are used to create a diversion or to slow 
distribution of intercept patterns.

>  compliant if it did not control recursion.

Check a selection of SPF libraries and test asserted limits.  There is 
no required minimum.  There is a recommended maximum where some 
libraries assert no limit or use the previous 20 SPF record recursive 
depth as a limit.  Since record publishers are not processing the 
record, correct results depend upon undefined minimum processing 
limits.  In the case of those developed by Wayne, this might be 5, and 
not 10 subsequent record  resolutions.

> > A specification that offers no warning or recommended minimum has
> > not helped to ensure interchange.
>  Well, maybe there is some truth to this "theoretically" but I have
>  not seen any evidence of it. Limits are prudent SPF design otherwise
>  you will get bitten.

Those _not_ using SPF are and have been affected!

> > We have a large complaint center.:^) At a recent MAAWG meeting,
> > I led an ISP specific discussion regarding SPF use. There was
> > consensus that rejecting failing SPF records ending with "-all"
> > cause a large number of complaints. SPF registration is poorly
> > understood and often misapplied even now. SPF records offering a
> > pass do not necessarily confirm the domain originating the message
> > either.
>  audience.

This was not my opinion.   A redeeming feature of SPF was using scripts 
to determine which RBL currently lists the domain.

>  the Domain wishes, defeating its purpose and you will get a lot of

A large provider is wise not to reject on the basis of SPF failure when 
this generates expensive support calls.

>  groups, doesn't make it true whatsoever.

Initially, many supported Sender-ID.  Redmond offered $50K apiece to 
have it implemented.  After some time, it became apparent Sender-ID 
would not resolve the phishing problem and it made path registration 
dependent upon email modification.  Few now are willing to support 
Sender-ID.  It also seems many view SPF as little more than a tool that 
can be used to investigate delivery issues, or to setup whitelists.  Use 
of local-part macros or the exists mechanism would degrade or destroy 
even this limited utility.  IMHO, the exists mechanism and local-part 
macros should be deprecated, with more stringent error limits placed on 
SPF processing libraries.

>  NEUTRAL, I agreed that that is a serious problem.

This is expressing a desire to assert control over outbound services.  
IMHO, outbound domains should stand on their own merit.  Use of "-all" 
causes additional grief for little benefit.  Domains prone to being 
phished should use DKIM, and hope recipient's MUAs check for invalid 
pre-pended header fields. :^)

>  DKIM has failed because it destroyed its HARD POLICY framework,
>  making it weak and indeterminate, wasteful and so relaxed that no one

In the Repute WG some expressed a desire for SPF or DKIM results to 
accrue against associated domains.  DKIM never confirms who sent the 
message, nor intended recipients.  Requiring SPF domains to pass in 
conjunction with their DKIM signatures would make DKIM far less robust, 
where some even desire a fall-back for failed DKIM signatures.  Anyone 
blocking a domain on the basis of DKIM or SPF results is at risk, since 
this evidence never authenticates the domain actually responsible for 
transmitting the message.  It is not wise to block the wrong domain.

>  Email's lack of ability to offer strong HARD policies is whats
>  keeping this all open-ended and wasteful. SPF offers strong
>  deterministic policies. So does DKIM with ADSP and its extensions
>  with ACL and ATPS. If you wish to ignore these strong domain
>  policies, then I don't how anyone can make a claim that there is a
>  solution to any of these work, at any level. It will simply never
>  WORK without some level of strong DOMAIN policies that receivers
>  follow.

I agree with a need to curtail unauthenticated transmission and receipt 
of email.  ATSP can only function properly with the authentication of 
outbound servers.  While HELO/SPF could fulfill that role, the Internet 
environment now demands cryptographic methods.

>  that is already 100% dealt with. Old news.

It has not been handled, only presumed handled.  Email/DNS traffic is 
not analogous with other Internet traffic.  Web services streamline 
transmitter authentication.  Email avoids transmitter authentication and 
expects recipients to grant unknown entities script based access to 
their DNS resources. :^O

>  mongering to replace it with other things and none of it was ever
>  true nor materialized.

An inability to discover evidence of a threat does not represent 
evidence of no threat.  There are plenty of active threats not detected 
until someone looks.  Carberp was a good example that affected several 
US government agencies and financial institutions by never escalating 
permissions.  While you might not regularly see a DDoS attack, not 
everyone is that fortunate.  SPF automatically grants access to DNS  
with SPF scripts running on behalf of any malefactor.

http://blog.trendmicro.com/?p=34170

I suspect that in your SPF sampling you failed to accurately define 
which elements represent a macro.  IMHO, % script expansion and various 
canned "mechanisms" should be considered active (recipient resource 
dependent) script components that increase the number of unique DNS 
transactions.

-Doug

---

From: Hector Santos <[email-masked]>
Date: Wed, 04 Jan 2012 20:21:32 -0500

Doug,

If all this is about the need to have learned limits written in 
SPF-BIF. I agree.  My point was that that it was learned long ago and 
the issues, i.e. required attention to recursion, buffer overflow 
and/or stack related issues, were addressed, trapped and mitigated. 
DNS caching is very important for today's modern SMTP system that is 
employing additional DNS related technologies.

Sure, all these points are possible SPFBIF material that needs to be 
convey.

> dependent) script components that increase the number of unique DNS 

No Doug. For the sample, it was two of 600+ SPF records with macros 
and both used it with EXISTS. One of those had a bad syntax for 
v=spf1, so the record was bad.

Let me see the current accumulated cache. In this collection, over 
1000+ SPF records.  Only 2 with SPF macros.   The same two as a matter 
of fact with the same one record with the bad syntax.

Based on this, Macros are rarely used and I'm not surprised because 
historically only those trying to "track" usage were interested in them.

---

From: Philip Gladstone <[email-masked]>
Date: Thu, 05 Jan 2012 15:56:20 -0500

On 1/4/2012 8:21 PM, Hector Santos wrote:
>> which elements represent a macro.  IMHO, % script expansion and 
>> various canned "mechanisms" should be considered active (recipient 
>> resource dependent) script components that increase the number of 
>> unique DNS transactions.
>

In my list of SPF records, I have over 80 distinct records (after 
eliminating the ip addresses, domain names etc) which use macros. A few 
of these records are invalid. I don't know what volume of traffic is 
sent with SPF records that use macros....

The macros are used in exists:, exp=, include:, a:, mx:, redirect=

The actual macros were: %{d} %{h} %{ir} %{i} %{l1r+} %{l1r-} %{l} %{o} 
%{r} %{s}  -- i.e. nobody is using v, p, c, or t.

The actual clauses were (for some value in <string>)

a:%{ir}.<string>
a:<string>.%{d}
a:<string>.%{o}
exists:%{ir}.<string>
exists:%{ir}._ip.%{l}.<string>
exists:%{ir}.mrbl.%{d}
exists:%{i}.%{l1r-}.user.%{d}
exists:%{i}.%{l}.<string>
exists:%{i}.3/86400.rate.%{d}
exists:%{i}.<string>
exists:%{i}._.%{h}._.%{s}.<string>
exists:%{i}._.%{h}._.%{s}._.%{r}.<string>
exists:%{l1r+}.<string>
exists:%{l}.%{ir}.<string>
exists:%{l}.%{i}.<string>
exists:%{l}.%{o}.%{i}.%{h}.<string>
exists:%{l}.<string>
exists:%{l}.x.%{ir}.<string>
exists:%{s}.%{i}.<string>
exists:%{s}.<string>
exists:%{s}.EmailAddr.%{i}.<string>
exists:%{s}.S.%{i}.<string>
exists:%{s}.sending-to.%{r}.from-ip-address-%{i}.<string>
exists:%{s}_%{i}_<string>
exists:CL.%{i}.FR.%{s}.<string>
exists:CL.%{i}.FR.%{s}.HE.%{h}.<string>
exists:I.%{i}.F.%{l}.%{o}.H.%{h}.<string>
exists:IP%{i}.H%{h}.D%{d}.<string>
exists:_h.%{h}._l.%{l}._o.%{o}._i.%{i}._s.%{s}._r.%{r}.<string>
exists:_h.%{h}._l.%{l}._o.%{o}._i.%{i}._spf.%{d}
exists:h_%{h}-i_%{i}-%{s}.<string>
exists:i%{i}.f%{s}.<string>
exp=<string>.%{d}
include:%{l}.%{o}.<string>
include:<string>.%{d}
mx:<string>.%{d}
redirect=%{l}.<string>
redirect=%{o}.<string>
redirect=<string>.%{d}

Philip

-- 
Philip Gladstone
Distinguished Engineer
Product Development
[email-masked]
Phone: +1 978-ZEN-TOAD (+1 978 936 8623)
Google: +1 978 800 1010
Ham radio: N1DQ

---

From: Stuart D Gathman <[email-masked]>
Date: Thu, 05 Jan 2012 16:57:10 -0500

Long ago, Nostradamus foresaw that on 01/04/2012 06:36 PM, Douglas Otis 
would write:
> "v=spf1 a mx/24 ?all".  This could produce a difficult to diagnose 
> loss of DSNs or sporadic junk folder message placements.
Whatever problem you are worried about here, it is not an SPF problem.  
The spec is already clear that any result from such practices (and I 
myself am such a practitioner) is *not* an SPF result and MUST NOT be 
reported as such.

In the discussion of the Authentication-Results header field, we did 
talk about needing a way to report such heuristic results (but not as 
SPF results).

---

From: Hector Santos <[email-masked]>
Date: Thu, 05 Jan 2012 19:23:37 -0500

> include:%{l}.%{o}.<string>

Of course, every site business/social personal network community will 
have different stats, so I don't to imply one case is ever the norm. 
As a former ESP site (i.e. free email accounts, etc) during the 90s 
(turned off in 98/99), we still get a very high hit rate of anonymous 
abuse, including all the blasting of CIS account hits and attempts to 
target our user base.  This represented an immediate and huge DNS 
overhead when SPF was implemented and executed at the MAIL FROM.  So 
it was pretty obvious we needed to delay it until the RCPT TO was 
validated.  This alone provided a 60% DNS overhead savings.  In 
addition, SPF results are cached and hard failures are remembered and 
not repeated (DNS lookup) again until a certain time.

The point is for our product implementation (not just our own support 
site usage) used by customers, is by the time a SPF query is done, a 
large percentage of would be "abusers" would be already filtered and 
what remains has a lesser abusive quality of SPF records.  This was a 
primary focus because we didn't want DNS overhead.

My SPF experience and view is macros are highly isolated to those that 
have an interest in tracking and feedback which I recall was the 
primary focus and interest.

I agree that macro limits should be conveyed and written into SPF-BIF. 
  I just don't think it was a major issue as its seems to be implied 
here and certainly not something which warrants any major flash as a 
security concern.  If it was, I would be the first to stop using SPF. 
  But was never the case and I think veteran SPF implementations have 
learned that long ago.

The only concern I always had was the wasteful policies that 
contributed to DNS query waste that only resulted in indeterminate (No 
Decision) results, like NEUTRAL and also SOFTFAIL unless it was 
coupled with secondary criteria to make a decision for which AFAIK, 
there is no standard SOFTFAIL+OTHER heuristic used.

Of the 80 you indicated in your sample, what percentage of those were 
abusive?  More importantly, which ones are exerting a high processing 
overhead because certainly not all possible macro expansions are 
abusive and the waste case is a recursive overhead.  That is the key 
limit (recursion) all SPF implementation MUST have and if SPF/SPF-BIS 
does not convey this, then is something that needs to be done. 
Without a doubt.

My view is a legitimate domain with a large email network of providers 
that requires a complex set of macros probably needs a thorough review 
if its network and that includes a suggestion that perhaps SPF is 
probably not for them, especially if the SPF result is always an 
wasteful indeterminate result when none of its macro rules are 
satisfy.  Its a major burden on SPF receivers to process SPF rules 
that end up with no results.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 05 Jan 2012 21:36:13 -0800

On 1/5/12 4:23 PM, Hector Santos wrote:
> >> Based on this, Macros are rarely used and I'm not surprised
> >> because historically only those trying to "track" usage were
> >> interested in them.
> >
> > eliminating the ip addresses, domain names etc) which use macros. A
> > few of these records are invalid. I don't know what volume of
> > traffic is sent with SPF records that use macros....
> >
> > The actual macros were: %{d} %{h} %{ir} %{i} %{l1r+} %{l1r-} %{l}
> > %{o} %{r} %{s} -- i.e. nobody is using v, p, c, or t.
>
Hector,

Excluding SPF processing for messages with invalid recipients offers 
spammers immediate feedback for which messages contain invalid 
recipients.  The macro aspect of SPF is highly problematic.  A spammer 
can encode the recipient into the return-path local-part along with an 
SPF record containing "exists:%{l}.<string>.%{o}" where their DNS logs 
will then indicate which recipients were accepted and processed. :^(

>  My SPF experience and view is macros are highly isolated to those
>  that have an interest in tracking and feedback which I recall was the
>  I agree that macro limits should be conveyed and written into
>  SPF-BIF. I just don't think it was a major issue as its seems to be
>  implied here and certainly not something which warrants any major
>  flash as a security concern. If it was, I would be the first to stop
>  using SPF. But was never the case and I think veteran SPF
>  implementations have learned that long ago.

Unfortunately, this misses the DDoS concern entirely!  It is NOT in the 
malefactor's interest to attack recipients whose resources are exploited 
by their accepted messages.   Thousands of other equally unwitting 
accomplices can saturate links to any victim's domain with DNS UDP 
traffic.  The local-part macro might represent a random value that will 
never be cached for use by subsequent messages.  This feature represents 
a real security concern since there is no affordable mitigation strategy 
to deal with this, where victims will have few clues of the attack source.

>  contributed to DNS query waste that only resulted in indeterminate
>  (No Decision) results, like NEUTRAL and also SOFTFAIL unless it was

When there is a HARDFAIL most ESPs know better than to reject the 
message.  The majority of email traffic is being emitted by compromised 
systems.  Why would anyone offer this type of criminal activity FREE 
access to SPF macro script processing able hit any third-party DNS 
server with significantly increased gain.   Other server to server 
services prevent abuse from reaching this level by directly 
authenticating the source in the most efficient method possible.  
Accepting a message, then processing dozens or hundreds of potentially 
non-cachable DNS transactions just to select which messages to place 
into a junk folder is irresponsible.

While much of the residential address space has been mapped out and 
thereby excluded, IPv6 is about to open vast unmapped regions too large 
to even make prefix queries against.  Security will soon depend upon 
everyone recognizing they are all part of the Internet ecosystem and 
that they must not ignore their own pollution, even when it does not 
appear to be affecting them directly.

-Doug

---

From: Hector Santos <[email-masked]>
Date: Fri, 06 Jan 2012 03:24:51 -0500

Douglas Otis wrote:

>> This represented an immediate and huge DNS
> recipients.  

Its a red-herring to suggest they will actually learn to STOP trying 
and thats the fallacy in your consideration the negative feedback 
means anything to them.

Dynamic RCPT TO validation and returning a 550 is going to happen 
regardless of SPF or anything.  The goal is to give immediate feedback 
and if this going to happen 50-60% of the time pretty persistently, 
every day, there will be a tremendous amount of overhead in payload 
downloads and SPF or any other DNS lookup technologies.  Coupled when 
a large percentage of the domains are bogus or NXDOMAIN, there is a 
serious amount of DNS related SMTP session slow downs. This is not 
conjecture. I have 9 years of statistic showing all this.

     http://www.winserver.com/public/spamstats.wct

The history shows the overhead before and after December 2003 when the 
delayed validation was implemented with immediate DNS overhead 
reduction and SMTP time improvement.

> The macro aspect of SPF is highly problematic.  

You have no real proof of this Doug and if there is, you need to be 
very specific on what kind of macros are used and how it is 
problematic. I already stated SPF system SHOULD have recursive and DNS 
lookup limits for SPF.   Even for MX lookups for SMTP clients, there 
are limits, so its not just SPF related.

> A spammer 

Even if this is a systemic record used by spammers, which is not by 
far, if they can't get pass RCPT TO,  there is no SPF query.

> to deal with this, where victims will have few clues of the attack source.

I don't see and have never heard of any problem when there are LIMITS 
Doug.

> message.  

Then they are going against the SPF specification.

> The majority of email traffic is being emitted by compromised 
> server with significantly increased gain.   

You are making it sound like this is wide spread. It is not and any 
DoS potential is already managed. I fail to see where you are going 
with all this.  Are you trying to abolish SPF like you always tried 
with all your predictions which simply never materialized?

> appear to be affecting them directly.

And your point regarding SPF and how this applies to SPF-BIF?

---

From: Douglas Otis <[email-masked]>
Date: Fri, 06 Jan 2012 19:24:40 -0800

On 1/6/12 12:24 AM, Hector Santos wrote:
>  Douglas Otis wrote:
> > Hector,
> >
> > Excluding SPF processing for messages with invalid recipients
> > offers spammers immediate feedback for which messages contain
> > invalid recipients.
>  regardless of SPF or anything. The goal is to give immediate
>  feedback and if this going to happen 50-60% of the time pretty
>  persistently, every day, there will be a tremendous amount of
>  overhead in payload downloads and SPF or any other DNS lookup
>  technologies. Coupled when a large percentage of the domains are
>  bogus or NXDOMAIN, there is a serious amount of DNS related SMTP
>  session slow downs. This is not conjecture. I have 9 years of
>  statistic showing all this.
>
>  The history shows the overhead before and after December 2003 when
>  the delayed validation was implemented with immediate DNS overhead

Dear Hector,

You are asking the right questions.  Many domains delay refusal until 
after the data phase to obscure their reason and endure the increased 
SMTP latency.  Offering immediate valid recipient feedback means 
recipient checks will be less effective at restricting malefactor access 
to user inboxes, to SPF processing engines, or to Call-Back checks.  
Initiating Call-Back checks to test the viability of the return-path 
increases SMTP latency at the expense of consuming another domain's 
resources.  IMHO, lack of a practical outbound MTA authentication 
methodology has led to what might be considered inconsiderate behaviors.

>  problematic. I already stated SPF system SHOULD have recursive and
>  DNS lookup limits for SPF. Even for MX lookups for SMTP clients,
>  there are limits, so its not just SPF related.

Indeed most SPF libraries impose limits on DNS transactions permitted 
per message.  The current specification permits up to 100 randomized 
transactions targeting any unrelated victim not evident within the 
message and still provide a PASS result.  With today's level of abuse 
exceeding 90%, there is a real need to vet message sources.  
Unfortunately, SPF expects recipients to determine whether a return-path 
domain has authorized the last MTA of an indeterminate sequence of 
MTAs.  Such a process always produces indeterminate results, as is now 
the case.

That is not to say SPF provides no value.  SPF in its simplest form 
offers a method to determine which reputation service lists the 
authorized outbound MTAs or to place them into a white-list.  This 
function works best for SPF records lacking seldom used macro 
expansions, the exists mechanism, or modify default host domains.  Any 
of which is seldom done.

> > A spammer can encode the recipient into the return-path local-part
> > along with an SPF record containing "exists:%{l}.<string>.%{o}"
> > where their DNS logs will then indicate which recipients were
> > accepted and processed. :^(
>

Giving malefactors RCPT TO validity feedback permits quick evasion of 
this check.

> > Unfortunately, this misses the DDoS concern entirely! It is NOT in
> > the malefactor's interest to attack recipients whose resources are
> > exploited by their accepted messages. Thousands of other equally
> > unwitting accomplices can saturate links to any victim's domain
> > with DNS UDP traffic. The local-part macro might represent a
> > random value that will never be cached for use by subsequent
> > messages. This feature represents a real security concern since
> > there is no affordable mitigation strategy to deal with this, where
> > victims will have few clues of the attack source.
>

It would be unreasonable to expect ESPs with their large bandwidths to 
notice anything.  What other server authentication processes grant 
unknown malefactors similar levels of access and control over these 
powerful recipient resources?   Lengthy SPF based transactions are 
carried out by any number of recipient domains happily unaware whether 
records referenced by local-parts of return-paths emit transactions 
targeting innocent uninvolved services that nevertheless may be vital 
and have nothing to do with email.  What would you expect victims to 
see?  What will victims do after their links are saturated with UDP 
traffic from thousands of sources.  Have everyone cease use of email?  
Have everyone cease use of DNS?

>  Then they are going against the SPF specification.

The specification says software can mark or reject a message.  There are 
no requirements to reject.  The strongest language is a should consider 
rejecting e-mail from invalid domains.

The specification also makes a remarkable and inaccurate statement in 
the introduction:

"An additional benefit to mail receivers is that after the use of an 
identity is verified, local policy decisions about the mail can be made 
based on the sender's domain, rather than the host's IP address. This is 
advantageous because reputation of domain names is likely to be more 
accurate than reputation of host IP addresses."

With today's level of abuse, most now outsource email services.  As 
such, outbound MTAs are commonly shared with thousands of domains.  
While a domain may authorize outbound servers, it would be unfounded 
conjecture whether the authorizing domain intended any specific message 
be sent to any specific recipient.  However, the IP addresses of 
outbound MTAs are weakly authenticated by the TCP transport.  Unless the 
EHLO has been validated against the domain of the return-path, mere 
authorization is never stronger than IP address authentication.  The 
outbound MTA is obligated to exclude abusive sources since recipients 
can only guess who was the responsible domain.

> > systems. Why would anyone offer this type of criminal activity
> > FREE access to SPF macro script processing able hit any third-party
> > DNS server with significantly increased gain.
>

Are you saying DDoS has been solved or that your are certain these 
attacks are not related to SPF?  Realistic consideration of SPF benefits 
should significantly limit its automatic use.  Mail-From authorization 
is unsuitable for abating spam since it offers no incontrovertible 
evidence of abuse.  It is also unsuitable for abating spoofing, since it 
is normally not seen.  SPF is also unable to deal with IPv6 related 
changes as source addresses become even more indeterminate.

> > thereby excluded, IPv6 is about to open vast unmapped regions too
> > large to even make prefix queries against. Security will soon
> > depend upon everyone recognizing they are all part of the Internet
> > ecosystem and that they must not ignore their own pollution, even
> > when it does not appear to be affecting them directly.
>

Looking at your web status page, it is clear much of your vetting 
process depends upon using the resources of others.  Others that just 
might be DDoS targets as well.  SPF would benefit from the 
simplification of the SPF definition by removing much of its seldom used 
flexibility.

The recommendation was to deprecate the exists and local-part macros and 
to remove terminal strings from any mechanism.  The few domains making 
use of these features will be prompted by minor problems that they need 
to simplify their records.  The list of legitimate domains sending email 
responsibly is comparatively small.  Why not immediately accept messages 
from those that have authenticated the HELO domain of the transmitting 
server?  That could be done within one DNS transaction.  After which, 
reputation can be safely and effectively applied against authenticated 
domains that are in full control of the assessed transactions.  ATSP can 
then prevent unauthorized transmissions using an additional DNS transaction.

These changes should ensure responsible and responsive ESPs will have a 
prosperous future and better mitigate the current levels of abuse.  All 
of this keeps you in the driver's seat.  We are on the same side after all.

Regards,
Doug Otis

---

From: Hector Santos <[email-masked]>
Date: Sat, 07 Jan 2012 01:27:05 -0500

> rejecting e-mail from invalid domains.

Perhaps, the term MARK needs to be defined in SPF-BIF.  What is MARKING?

My opinion on the topic:

A HARDFAIL MARK still represents a condition of FAILURE.

In other words,  HARDFAIL MARK should still separate a transaction 
from the rest as a very suspicious transaction with a very high 
confidence and zero FALSE positive state.  If a SPF receiver is not 
going to honor the outright SPF DOMAIN HARDFAIL policy result is a 
100% unacceptable bad transaction with 100% confidence, then all 
liabilities will be transferred to the RECEIVER who is now 
intentionally ignorant of a 100% unacceptable condition.  There is no 
claim here of plausible denial for a SPF compliant receiver.  All 
liabilities passes to it when it ignores a Domain HardFail policy result.

The ultimate outcome SHOULD be the same for either.  SPF-BIF should 
make that very clear. When it comes to a HardFail result, a decision 
to mark and accept and still pass on the transaction to recipient(s), 
is highly dangerous and goes against the main purpose of what SPF is 
trying to achieve, especially with HardFail declarations.

While moving it to a spam box is acceptable idea, but it must be 
understood a HARDFAIL MARK outcome is to match the a HARDFAIL REJECT 
outcome where the recipients are not subject to harm, the bottom line.

To me, MARKING is great for the indeterminate results simply because 
no decision can be made and IMV the receiver SHOULD not be rejecting 
purely on these indeterminate results.

Again, a HARDFAIL is very different in all this. Its Black and White, 
there is no grey here.

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 07 Jan 2012 11:50:54 +0100

On 05/Jan/12 22:57, Stuart D Gathman wrote:
> Long ago, Nostradamus foresaw that on 01/04/2012 06:36 PM, Douglas
> Otis would write:
>>

I don't agree on this.  If there were problems evaluating bestguess,
then there would be problems in case that domain actually published
such policy.

> The spec is already clear that any result from such practices (and
> I myself am such a practitioner) is *not* an SPF result and MUST
> NOT be reported as such.

Correct.

> talk about needing a way to report such heuristic results (but not as

An additional method is needed for that.  We might speculate whether
that would be a case of "addition of any enhancements that have
already gained widespread support" or if rechartering is needed.  The
consensus seems to be not to schedule a deliverable for bestgess at
this stage.

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 07 Jan 2012 12:11:29 +0100

On 06/Jan/12 06:36, Douglas Otis wrote:
>>  [I]t was pretty obvious we needed to delay it until the RCPT TO was
>>  validated. This alone provided a 60% DNS overhead savings.
> 

I agree this is a weakness of SPF.  The current charter does not allow
us to ban local macros, but that could be done for v=spf3.  If there
is agreement on that, spfbis could include some text more or less like so:

   Note: reporting mechanisms are being improved, and it is suggested
         that domain owners use those instead of DNS-tracking.  Future
         versions of SPF may deprecate "exists" mechanisms that use
         local macros.

jm2c

---

From: Hector Santos <[email-masked]>
Date: Sun, 08 Jan 2012 00:16:12 -0500

>          versions of SPF may deprecate "exists" mechanisms that use

The problem with this SPF macros discussion is its being generalized 
as bad as a whole.  That is far from any truth.

So if there is a problem with SPF macro expansions, there needs to be 
specific examples of how and where this problem begins and what it causes.

Is the "exists" directive the only problem with?  If so, why?

How about the "include" and "redirect" directives?

In my experience of the issues that had came up very early on, what 
the lack of limits.  It was immediately apparent that SPF 
implementations needs to pay attention to recursive limits.

For SPF-BIF, in the name of not having an revision change the core 
specification, I believe, the security topics need to expanded and 
begin to have semantics and guidelines that imply something like:

      SPF implementations MAY impose strict limits and policies SHOULD be
      very attentive to reduce complexity, directives, mechanisms and 
macros
      that will cause extensive DNS operations. SPF receivers MAY impose
      limits on the amount of recursive include, redirects or the number
      of exists directives.

For our package,we added the limits and we never seen issues again.  I 
thought other packages also implemented limits, but according to Doug, 
some SPF libraries do not.

That may because its old, not supported, whatever. But for the 
supported packages, we learned this long ago and limits need to be 
stated in SPF-BIF.

---

From: Philip Gladstone <[email-masked]>
Date: Mon, 09 Jan 2012 09:56:53 -0500

> specific examples of how and where this problem begins and what it 
Exactly. A powerful use of the local part macros is to be able to cause 
email that is sent (by any MTA) from an invalid user to be rejected.

There are still lots of forwarding services that don't do SRS or similar 
and thus cause problems with SPF. I wanted to be able to permit these 
uses, but only when the email is sent from known good usernames.

In general, making a system more flexible is a good thing as it allows 
people to use it in ways that the designers did not consider. I would 
firmly resist any deprecation of macros *unless* equivalent 
functionality was provided in another way.

Philip

---

From: Douglas Otis <[email-masked]>
Date: Tue, 10 Jan 2012 11:54:31 -0800

On 1/9/12 6:56 AM, Philip Gladstone wrote:
> Exactly. A powerful use of the local part macros is to be able to 
> cause email that is sent (by any MTA) from an invalid user to be 
> rejected.
> There are still lots of forwarding services that don't do SRS or 
> similar and thus cause problems with SPF. I wanted to be able to 
> permit these uses, but only when the email is sent from known good 
> usernames.
Dear Philip,

Use of local-part macros to vet valid email-addresses allows malefactors 
unlimited access to a mechanism that validates local-part guesses.  In 
other words, spammers are being welcomed.  Once any valid local-part has 
been determined and is intended to be used from any MTA, this will 
produce a PASS based simply on the presents of guessed local-parts.  A 
poorly considered method for preventing abuse that makes PASS results 
completely meaningless.
> functionality was provided in another way. 
Use of SPF macros in unintended ways creates real risk. Authenticating 
outbound MTAs offers a scalable solution for generally determining 
whether a message might be trusted.   S/MIME or OpenPGP offers a viable 
alternative for ensuring who originated a message.  Use of SPF 
local-part macros is never safe for any purpose.

Regards,
Doug Otis

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 10 Jan 2012 15:01:45 -0500

On Tuesday, January 10, 2012 11:54:31 AM Douglas Otis wrote:
> >>> is agreement on that, spfbis could include some text more or less
> >>> 
> >>> like so:

This is all a bunch of hand waving.  Do you have any evidence for real world 
occurences of these alleged problems?

Scott K

---

From: Philip Gladstone <[email-masked]>
Date: Tue, 10 Jan 2012 15:24:01 -0500

On 1/10/2012 2:54 PM, Douglas Otis wrote:
> Dear Philip,
>
> Use of local-part macros to vet valid email-addresses allows 
> malefactors unlimited access to a mechanism that validates local-part 
> guesses.  In other words, spammers are being welcomed.  Once any valid 
> local-part has been determined and is intended to be used from any 
> MTA, this will produce a PASS based simply on the presents of guessed 
> local-parts.  A poorly considered method for preventing abuse that 
> makes PASS results completely meaningless.
>> In general, making a system more flexible is a good thing as it 
>> allows people to use it in ways that the designers did not consider. 
>> I would firmly resist any deprecation of macros *unless* equivalent 
> whether a message might be trusted.   S/MIME or OpenPGP offers a 
> viable alternative for ensuring who originated a message.  Use of SPF 

Doug

I understand that you wouldn't make use of this feature.   However I am 
free to design my SPF record in any way that I like that meets my needs. 
By your reasoning, '+all' would not be allowed either as it makes SPF 
meaningless as well.

In my sample of SPF records, '-all' beats '~all' beats '+all' beats 
'all'  (in terms of counts) -- with +all being about 5% of -all  (sample 
not weighted in any particular way).

If you have some concrete examples of the risk that the use of the SPF 
local-part macro poses, then lets discuss them.

Philip

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 11 Jan 2012 19:09:39 +0100

> local-part macro poses, then lets discuss them.

Please, let's not.  Such kinds of discussion won't bring us much.

It is a fact that there is people who consider SPF more complicate
than it needs to be for mail sites that use MSAs correctly.  We may
want to estimate whether the user base --both publishers and
checkers-- would increase sensibly if we offered a simplified version.

We don't need to enumerate the possible risks once more.

jm2c

---

From: Hector Santos <[email-masked]>
Date: Wed, 11 Jan 2012 17:17:34 -0500

> We don't need to enumerate the possible risks once more.

For our package, an MSA is an authorized, authenticated session and 
thus SPF and other stuff are skipped.  It is for unknown transactions 
like done at a MDA where it is operationally not required to do an 
authorization to receive a local recipient message where SPF by 
supportive receivers apply it.  Not saying the MSA can't apply SPF, 
but I don't see where it can help against the compromised MSA user 
client because the user is not one to be using an SPF record.

I believe if there is a specific, known, loophole that is not 
currently covered by SPF implementations in the field, nor specified, 
mentioned, written in RFC4408, then it should be discussed and covered 
by SPF-BIF.

Section 10.1 Processing Limits, covers the security idea of limits. 
But it only mentions one hard limit of 10 for MX and PTR. It 
recommends a unspecified minimum for others like INCLUDE which is more 
of a concern because of its recursive nature.   We have a limit of 20 
here. Probably more than enough to allow a domain to general an 
include recursion level of 20! Each of these directive whether with 
macros or not, needs to have limits.

So in my view, it will probably be a good idea to avoid the "guessing 
game" and begin to add semantics about specific recommended limits.

Doug is suggesting of changing the 10 for MX/PTR to maybe 5.  Where 
did 10 come from to begin with?  I presume is what was some SMTP 
client were already doing when sending out mail - they have a MX 
expansion limit of 10. Maybe the 10 come from the author's experience 
with this.

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 11 Jan 2012 17:34:29 -0500

> macros or not, needs to have limits.

This is not correct.  What is says is:

   SPF implementations MUST limit the number of mechanisms and modifiers
   that do DNS lookups to at most 10 per SPF check, including any
   lookups caused by the use of the "include" mechanism or the
   "redirect" modifier.  

The alleged hole does not exist.

Scott K

---

From: Derek Diget <[email-masked]>
Date: Wed, 11 Jan 2012 17:39:02 -0500 (EST)

On Jan 11, 2012 at 17:17 -0500, Hector Santos wrote:
=>Section 10.1 Processing Limits, covers the security idea of limits. But it
=>only mentions one hard limit of 10 for MX and PTR. It recommends a unspecified
=>minimum for others like INCLUDE which is more of a concern because of its
=>recursive nature.   We have a limit of 20 here. Probably more than enough to
=>allow a domain to general an include recursion level of 20! Each of these
=>directive whether with macros or not, needs to have limits.
=>
=>So in my view, it will probably be a good idea to avoid the "guessing game"
=>and begin to add semantics about specific recommended limits.
=>
=>Doug is suggesting of changing the 10 for MX/PTR to maybe 5.  Where did 10
=>come from to begin with?  I presume is what was some SMTP client were already
=>doing when sending out mail - they have a MX expansion limit of 10. Maybe the
=>10 come from the author's experience with this.

I have been reading this thread and wanting to chime in...

I have always read the first sentence of the 3rd paragraph of RFC 4408 
section 10.1 (starts with "SPF implementations MUST limit") to mean that 
an SPF record that requires more that 10 DNS lookups to be a PermError 
and the SPF library should not do a 11th lookup.  Period/end-stop.

The MX limit that is mentioned in later paragraphs is still constrained 
by the first sentence's (10 lookup) limit.  I would almost say that the 
4th paragraph ('When evaluating the "mx"') is redundant to the 3rd 
paragraph except for the mention of the %{p} macro.

(I can't speak for code writers of the MTA we use, but I think that is 
the behavior I have seen.  11 DNS lookup causes a PermError to be 
returned by check_host() ).

-- 
***********************************************************************
Derek Diget                            Office of Information Technology
Western Michigan University - Kalamazoo  Michigan  USA - www.wmich.edu/
***********************************************************************

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 11 Jan 2012 14:41:02 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Derek Diget
> Sent: Wednesday, January 11, 2012 2:39 PM
> To: [email-masked]
> Subject: Re: [spfbis] Local macros strike again, was Suggestion...
> 
> section 10.1 (starts with "SPF implementations MUST limit") to mean
> that an SPF record that requires more that 10 DNS lookups to be a
> PermError and the SPF library should not do a 11th lookup.  Period/end-
> stop.
> 

If people feel this is not clear enough in 4408, the bis effort would certainly be in a position to clarify.

---

From: Hector Santos <[email-masked]>
Date: Wed, 11 Jan 2012 17:47:19 -0500

> The alleged hole does not exist.

You're right, missed that statement.

---

From: Hector Santos <[email-masked]>
Date: Wed, 11 Jan 2012 22:02:21 -0500

> returned by check_host() ).

I think its a matter of what DNS client/API the SPF software used.

As I always understood what were the practical issues, it was the two 
limits, one based for on recursion (i.e. Stack) limit and one limit 
for the expansion of the MX host record.

Recursion is largely dependend on the API on how it repeats a function 
recursively, i.e first record has a INCLUDE, which has a INCLUDE, 
which has as INCLUDE, and so on. A well coded, elegant recursive 
function, OOPS or otherwise, will do this recursively and there could 
be stack limits.

Viruses are out there that specifically look for these "overflow" 
design errors because depending on the exception trap restart logic, 
if any, the virus payload is placed (byte wise) at the function Epilog 
location where the OS will return from the function.  That is your 
classic buffer over/under flow virus that started when the eEye 
security firm (startup at the time) exposed *blackmailed Microsoft" by 
showing how easy it can be done back in 1999/2000.  All future buffer 
overflow viruses were based on the eEye exposure.

The MX expansion length support is well known for SMTP clients when 
doing a MX lookup.  Some larger domains, ESP, like YAHOO.COM, AOL.COM 
has very large MX expansions compared to most systems.   With Yahoo, 
when fully expanded, you get 13 IP addresses.

V:\wc5beta>testwcdns /q=mx /nd- /nc yahoo.com
TESTWCDNS v2.7 (c) copyright 2004-2012 Santronics Software, Inc
- DNS Cache Enabled   : No
- DNS Servers         : 208.247.131.210, 192.168.1.254
- DNS Resource Type   : MX
- DNS Lookup Timeout  : 10 secs
- DNS Max Records     : 64
- MX Expand Multi-Home: Yes
- MX Delay Lookup     : No

* Host: [yahoo.com]
count : 13
===========================================================================
RR    D#  |    TTL | PRE | HOST                       | DATA
===========================================================================
MX     0  |    590 |   1 | mta7.am0.yahoodns.net      | 209.191.88.254
MX     1  |    590 |   1 | mta7.am0.yahoodns.net      | 66.94.237.64
MX     2  |    590 |   1 | mta7.am0.yahoodns.net      | 66.94.237.139
MX     3  |    590 |   1 | mta7.am0.yahoodns.net      | 66.94.238.147
MX     4  |    590 |   1 | mta7.am0.yahoodns.net      | 67.195.103.233
MX     5  |    590 |   1 | mta7.am0.yahoodns.net      | 74.6.136.65
MX     6  |    590 |   1 | mta7.am0.yahoodns.net      | 74.6.140.64
MX     7  |    590 |   1 | mta7.am0.yahoodns.net      | 98.139.175.225
MX     8  |    590 |   1 | mta5.am0.yahoodns.net      | 67.195.168.230
MX     9  |    590 |   1 | mta5.am0.yahoodns.net      | 66.94.236.34
MX     10 |    590 |   1 | mta6.am0.yahoodns.net      | 98.137.54.238
MX     11 |    590 |   1 | mta6.am0.yahoodns.net      | 98.139.54.60
MX     12 |    590 |   1 | mta6.am0.yahoodns.net      | 67.195.103.232
===========================================================================

but with AOL.COM, you get 19 IP addresses in the MX expansion:

V:\wc5beta>testwcdns /q=mx /nd- /nc aol.com
TESTWCDNS v2.7 (c) copyright 2004-2012 Santronics Software, Inc
- DNS Cache Enabled   : No
- DNS Servers         : 208.247.131.210, 192.168.1.254
- DNS Resource Type   : MX
- DNS Lookup Timeout  : 10 secs
- DNS Max Records     : 64
- MX Expand Multi-Home: Yes
- MX Delay Lookup     : No

* Host: [aol.com]
count : 19
============================================================================
RR    D#  |    TTL | PRE | HOST                       | DATA
============================================================================
MX     0  |    256 |  15 | mailin-04.mx.aol.com       | 205.188.103.2
MX     1  |    256 |  15 | mailin-04.mx.aol.com       | 205.188.146.194
MX     2  |    256 |  15 | mailin-04.mx.aol.com       | 64.12.90.34
MX     3  |    256 |  15 | mailin-04.mx.aol.com       | 64.12.90.66
MX     4  |    256 |  15 | mailin-04.mx.aol.com       | 64.12.138.161
MX     5  |    256 |  15 | mailin-01.mx.aol.com       | 205.188.159.42
MX     6  |    256 |  15 | mailin-01.mx.aol.com       | 64.12.90.1
MX     7  |    256 |  15 | mailin-01.mx.aol.com       | 64.12.90.98
MX     8  |    256 |  15 | mailin-01.mx.aol.com       | 205.188.59.194
MX     9  |    256 |  15 | mailin-01.mx.aol.com       | 205.188.146.193
MX     10 |    256 |  15 | mailin-02.mx.aol.com       | 205.188.190.1
MX     11 |    256 |  15 | mailin-02.mx.aol.com       | 64.12.90.65
MX     12 |    256 |  15 | mailin-02.mx.aol.com       | 64.12.139.193
MX     13 |    256 |  15 | mailin-02.mx.aol.com       | 205.188.103.1
MX     14 |    256 |  15 | mailin-03.mx.aol.com       | 64.12.90.33
MX     15 |    256 |  15 | mailin-03.mx.aol.com       | 64.12.90.97
MX     16 |    256 |  15 | mailin-03.mx.aol.com       | 205.188.59.193
MX     17 |    256 |  15 | mailin-03.mx.aol.com       | 205.188.156.193
MX     18 |    256 |  15 | mailin-03.mx.aol.com       | 205.188.190.2
============================================================================

So what exactly is the LIMIT with the MX in SPF?  For an SMTP client, 
it is not operationally required to try all 19 IP address for each 
Outbound Queued attempt. It probably needs to remember what it last 
tried, but to there is no requirement that a SMTP client is 
technically required to try all 19.

So if the DNS client API the SPF software is using some limit on how 
many MX expanded IPs list are returned in order to compare with the 
connecting IP address, it is conceivable that the IP is not within the 
returned list and a failure can occur.

So the LIMIT on the MX should be discussed on what precises it means 
and if limits are placed, the DOMAIN record owner needs to be aware 
that this can happen to them.

Fortunately, I believe, the issue is isolated to the very few domains 
that do have such high MX expanded results, like YAHOO.COM, and 
AOL.COM.  There are not many other domains with that large MX farm of 
machines.

But its is a design consideration and in our last work with our DNS 
Client software, I believe the internal limit was set at 30, 
sufficient to cover yahoo.com, aol.com and probably everyone else.

But if lets say a 10 limit means MX expanded limited, then like this 
testwcdns shows when I add the /c=10 count limit:

V:\wc5beta>testwcdns /q=mx /nd- /nc /c=10 aol.com
TESTWCDNS v2.7 (10/05/2011 DEBUG C++ 6.0) (c) copyright 2004-2012 
Santronics Software, Inc
- DNS Cache Enabled   : No
- DNS Servers         : 208.247.131.210, 192.168.1.254
- DNS Resource Type   : MX
- DNS Lookup Timeout  : 10 secs
- DNS Max Records     : 10
- MX Expand Multi-Home: Yes
- MX Delay Lookup     : No

* Host: [aol.com]
count : 10
=============================================================================================
RR    D#  |    TTL | PRE | HOST                                | DATA
=============================================================================================
MX     0  |   2798 |  15 | mailin-02.mx.aol.com                | 
64.12.139.193
MX     1  |   2798 |  15 | mailin-02.mx.aol.com                | 
205.188.103.1
MX     2  |   2798 |  15 | mailin-02.mx.aol.com                | 
205.188.190.1
MX     3  |   2798 |  15 | mailin-02.mx.aol.com                | 
64.12.90.65
MX     4  |   2798 |  15 | mailin-03.mx.aol.com                | 
205.188.59.193
MX     5  |   2798 |  15 | mailin-03.mx.aol.com                | 
205.188.156.193
MX     6  |   2798 |  15 | mailin-03.mx.aol.com                | 
205.188.190.2
MX     7  |   2798 |  15 | mailin-03.mx.aol.com                | 
64.12.90.33
MX     8  |   2798 |  15 | mailin-03.mx.aol.com                | 
64.12.90.97
MX     9  |   2798 |  15 | mailin-04.mx.aol.com                | 
64.12.90.34
=============================================================================================

then as I said, the SPF receiver and the DOMAIN owner needs to be 
aware that an SPF receiver MX lookup can return a IP list that is not 
covering the connecting IP address.

---

From: Hector Santos <[email-masked]>
Date: Thu, 12 Jan 2012 00:36:20 -0500

Murray S. Kucherawy responded to Derek:

>> Derek Diget stated:

+1

IMV, Section 10.1 is clear in covering the main real practical issues, 
specifically the two limits dealing with:

    - Recursion limits with directives such as include, redirect, and
    - Array sizes/length of DNS RR records returned.

However, I think these two paragraphs can be further clarified:

    When evaluating the "mx" and "ptr" mechanisms, or the %{p} macro,
    there MUST be a limit of no more than 10 MX or PTR RRs looked up and
    checked.

    SPF implementations SHOULD limit the total amount of data obtained
    from the DNS queries.  For example, when DNS over TCP or EDNS0 are
    available, there may need to be an explicit limit to how much data
    will be accepted to prevent excessive bandwidth usage or memory usage
    and DoS attacks.

Is the 10 limit for MX/PTR lookups a limit for the number of records 
returned or it is the limit of MX/RR directives that are declared in 
SPF statements?

It appears to me that first paragraph addresses the latter - a limit 
of 10 for 10 different MX/PTR directives in SPF statements, and the 
2nd paragraph addresses an unspecified recommendation of limits for 
the number of records returned in a MX or PTR lookup, especially as a 
result of a initial truncated query and switch from UDP to TCP.

In anything should be considered, possibly a statement can can added 
to the 2nd paragraph or separate paragraph:

    ..... However, keep in mind, the limit 10 for MX/PTR is not 
related to
    limiting the MX expansion of IP addresses to 10 records only or 10 
PTR
    host records returned to 10 only.  Any limit for DATA records MUST
    consider the possibility larger email domains may exceed the SPF
    implementation limit in MX expanded records or PTR records returned.

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 11 Jan 2012 23:47:23 -0600

Hector Santos <[email-masked]> wrote:

>> If people feel this is not clear enough in 4408, the bis effort would
>certainly be 
>> in a position to clarify.
>
>+1
>

Hector,

I would encourage you to review the entire section again. The concept of recursion limits does not appear in RFC 4408. This was used for quite awhile in pre-4408 drafts of the spec while 4408 focuses more directly on lookup limits.

I suspect you may be inadvertently conflating internal implementation design decisions in your code base that were made before 4408 was published with what's in the RFC.

Scott K

---

From: Derek Diget <[email-masked]>
Date: Thu, 12 Jan 2012 14:22:06 -0500 (EST)

On Jan 12, 2012 at 00:36 -0500, Hector Santos wrote:
=>IMV, Section 10.1 is clear in covering the main real practical issues,
=>specifically the two limits dealing with:
=>
=>   - Recursion limits with directives such as include, redirect, and
=>   - Array sizes/length of DNS RR records returned.

Disagree.  I don't see where RFC 4408 deals with the size of the DNS RR 
returned for a query.

=>However, I think these two paragraphs can be further clarified:
=>
=>   When evaluating the "mx" and "ptr" mechanisms, or the %{p} macro,
=>   there MUST be a limit of no more than 10 MX or PTR RRs looked up and
=>   checked.

Note the "RRs looked up" phrase.  I read it to mean that check_host() is 
limited to looking up no more that 10 DNS Resource Records.  It does not 
say anything about the size (number) of the returned data set.

=>   SPF implementations SHOULD limit the total amount of data obtained
=>   from the DNS queries.  For example, when DNS over TCP or EDNS0 are
=>   available, there may need to be an explicit limit to how much data
=>   will be accepted to prevent excessive bandwidth usage or memory usage
=>   and DoS attacks.
=>
=>Is the 10 limit for MX/PTR lookups a limit for the number of records returned
=>or it is the limit of MX/RR directives that are declared in SPF statements?

I read it to be the total number of DNS _queries_ a SPF record can cause 
to be looked up.

As an example, the SPF library I am using fails with a PermError on 
latimes.com since their includes: never stop.  It returns PermError 
after the ninth lookup for cs1.com's SPF record.  The 10th (really 
first) lookup was the initial query for latimes.com's SPF record.  (The 
samples of delivery attempts I have are all from what appear to be 
infected machines and thus I don't see any legitimate mail being 
rejected.)

---

From: Hector Santos <[email-masked]>
Date: Thu, 12 Jan 2012 14:52:12 -0500

> I suspect you may be inadvertently conflating internal implementation design decisions in your code base that were made before 4408 was published with what's in the RFC.

There is some truth to this. For some things, I didn't immediate 
update the code when the RFC was finally published.

But for SPF-BIF, I still think the developer implementation point 
needs to be made, because if it wasn't the first, it was the among the 
first bug reports we got related to certain abusive records with deep 
includes.  This is very serious because most developers or programmers 
will naturally use recursive software designs where the #1 issue will 
be a stack limitation and that is the cat's meow for the Classic 
Buffer Overrun viruses.  That is what they look for. Its a security 
concern that implementations SHOULD|MUST be aware of.

I would also suggest that these two paragraphs:

    MTAs or other processors MAY also impose a limit on the maximum
    amount of elapsed time to evaluate check_host().  Such a limit SHOULD
    allow at least 20 seconds.  If such a limit is exceeded, the result
    of authorization SHOULD be "TempError".

    Domains publishing records SHOULD try to keep the number of "include"
    mechanisms and chained "redirect" modifiers to a minimum.

indirectly cover the idea of redundant, recursive calls. In one form, 
system performance degrades and the total time elapsed can reach a 
high value. So the suggestion of having a global timer correlates to 
such issues, and for the 2nd paragraph, term "chained" can be viewed 
as form of "recursion."

So while it may not specific state the term, in my engineering 
opinion, attention and guideline to recursive related possible 
overhead SHOULD|MUST be provided in SPF-BIF.

For SPF, when a SPF-Proc() is written starting with the initial 
SPF-PROC() where its parsing reaches a INCLUDE and REDIRECT, its a 
natural recursive call to SPF-PROC() again.  To avoid this stack 
limits, you need to either have a recursive limit or programmatically 
redesign the code to be non-recursive.

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 12 Jan 2012 15:57:25 -0500

On Thursday, January 12, 2012 02:52:12 PM Hector Santos wrote:
> > I would encourage you to review the entire section again. The concept of
> > recursion limits does not appear in RFC 4408. This was used for quite
> > awhile in pre-4408 drafts of the spec while 4408 focuses more directly
> > on lookup limits.
> > 
> > I suspect you may be inadvertently conflating internal implementation
> > design decisions in your code base that were made before 4408 was
> > published with what's in the RFC.

When I updated pyspf fo the RFC 4408 processing limits I left the recursion 
depth code in place.  The RFC 4408 lookup based limits are tight enough that 
it was unlikely the recursion limit would ever be triggered, so I can 
understand how that works.

> concern that implementations SHOULD|MUST be aware of.

The DNS lookup limits as currently specified address this concern as well.  
There isn't a need for specifying special recursion depth limits for 
interoperability purposes.  The 10 DNS lookup limit for mechanisms/modifiers 
solves the problem as long as an implementation can handle a recursion depth 
of 10 (which is needed in any case).  

This doesn't preclude implementers from being smart about how the approach it.  
In the case of (once again) pyspf, we have a check for the trivial recursion 
loop of a domain including itself and immediately raise an error (the first 
include in the latimes.com record mentioned elsewhere in the thread has this 
kind of error).

> such issues, and for the 2nd paragraph, term "chained" can be viewed

They do, but so does:

   SPF implementations MUST limit the number of mechanisms and modifiers
   that do DNS lookups to at most 10 per SPF check, ...

Recursion is just a special case of too many lookups.  I don't think it needs 
special treatment to support interoperability or security.  Also the first 
paragraph you mention there is referring to such a global timer, so I think 
it's fine as is.

> overhead SHOULD|MUST be provided in SPF-BIF.

I don't understand how recursion can be a problem for an implementation that 
fully implements the RFC 4408 processing limits.  This is the relevant text 
from the last pre-MARID draft:

   SPF clients must be prepared to handle records that are set up
   incorrectly or maliciously.  SPF clients MUST perform loop detection,
   limit SPF recursion, or both.  If an SPF client chooses to limit
   recursion depth, then at least a total of 20 redirects and includes
   SHOULD be supported.  (This number should be enough for even the most
   complicated configurations.)
   
   If a loop is detected, or if more than 20 subqueries are triggered,
   an SPF client MAY abort the lookup and return the result "unknown".

   Regular non-recursive lookups due to mechanisms like "a" and "mx" or
   due to modifiers like "exp" do not count toward this total.

Since each include/redirect counts as a mechanism/modifier that needs a DNS 
lookup it's not possible to get to more than half this recursion limit if the 
RFC 4408 limits are used.

> redesign the code to be non-recursive.

Yes, but that's an implementation detail and not an interoperabliity 
requirement.

Scott K

---

From: Hector Santos <[email-masked]>
Date: Thu, 12 Jan 2012 16:22:44 -0500

>    complicated configurations.)

Ahhhh, That's where we got our 20 limit from! :)

> RFC 4408 limits are used.

Maybe it is the current TEXT went from a draft Clear and Definitive 
developer understanding to a more vague description.   The issue is 
recursion as that is how most designs will do it, so I am scratching 
my head as to why the precise technical term and problem description 
was altered/removed. Sure, you can get around it with a more complex 
non-recursive design, hence, technically there is no stack related 
limit, but now a worst case of requiring a global timer.  Question, 
does pyspf implement a global timer?

> Yes, but that's an implementation detail and not an interoperabliity 

The entire section is about Implementation requirements.  It is an 
interoperability problem if 10 is used and a DOMAIN just happens to 
requires 11.  So on the flip side, the section is also an 
implementation guideline for domain administrators to limit their 
records to 10 deep recursion levels.

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 12 Jan 2012 17:00:16 -0500

On Thursday, January 12, 2012 04:22:44 PM Hector Santos wrote:
> > I don't understand how recursion can be a problem for an implementation
> > that fully implements the RFC 4408 processing limits.  This is the
> > relevant text> 
> > Since each include/redirect counts as a mechanism/modifier that needs a
> > DNS lookup it's not possible to get to more than half this recursion
> > limit if the RFC 4408 limits are used.
> 

The limits defined in RFC 4408 are not just about recursion.  They are about 
limiting DNS lookups.  They limit recursion even more than the old limits did, 
but they do other things too.

The limits are difficult to understand because they are complex, but I think 
they are reasonable and there's no evidence they need to be different than they 
are.  Once we get into the work of the group, I think it would be worthwhile 
to see if there is a clearer way to explain them.

No, pyspf doesn't implement a global timer, it's per lookup.  It probably 
should (it's on my TODO list).

> records to 10 deep recursion levels.

They are implementation requirements, but specifics of recursive versus non-
recursive design and how to process records that are within the processing 
limits are implementation design and not requirements.  I don't think we need 
to specify the means and methods of meeting those requirements.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 12 Jan 2012 14:07:15 -0800

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Scott Kitterman
> Sent: Thursday, January 12, 2012 2:00 PM
> To: [email-masked]
> Subject: Re: [spfbis] Section 10.1 Processing Limits
> 
> They are implementation requirements, but specifics of recursive versus
> non- recursive design and how to process records that are within the
> processing limits are implementation design and not requirements.  I
> don't think we need to specify the means and methods of meeting those
> requirements.

In fact, we SHOULD NOT.  :-)

-MSK

---

From: Hector Santos <[email-masked]>
Date: Thu, 12 Jan 2012 22:04:09 -0500

Scott Kitterman wrote:

>> HLS:

Scott, I think we are being a little anal on the idea that I was
advocating a specific coding method over another.  Far from it.  I do
see your point of what you are saying; that the MUST 10 limit will
cover all coding concerns and "SHOULD" be enough to avoid Stack
overflows for recursive implementations.  What I am stating is that
the INCLUDE/REDIRECT is not only a DNS lookup, but also a high, if not
a higher performance penalty than the DNS lookup, by the shear nature
it may be done recursively.

Going over section 10.1, here is what I suggest.

In paragraph 6, 1st sentence, add the word combined

    SPF implementations MUST limit the _combined_ number of mechanisms
    and modifiers that do DNS lookups to at most 10 per SPF check,
    including any lookups caused by the use of the "include" mechanism
    or the "redirect" modifier.

Based on that statement, we can possibly add a new sentence:

    This hard limitation implies SPF domain administrators are
    limited to SPF policy statements containing at most 10
    mechanisms that perform DNS lookups, specifically; A, PTR and MX.

I personally believe that a limit related to recursion different than
the 10 combined limit is warranted.  Many domains just start with a
SPF with a bunch of includes, for example:

Initial Lookup domain1:

    v=spf1  include:domain2 include:domain3 include:domain4 -all

Lookup #1 domain2:

    v=spf1  include:domain5 include:domain6 include:domain7 -all

Lookup #2 domain5:

    v=spf1  include:domain8 include:domain9 include:domain10 -all

Lookup #3 domain8:

    v=spf1  include:domain11 include:domain12 include:domain13 -all

Lookup #4 domain11:

    v=spf1  include:domain14 include:domain15 include:domain16 -all

Lookup #5 domain14:

    v=spf1  include:domain17 include:domain18 include:domain19 -all

Lookup #6 domain17:

    v=spf1  include:domain20 include:domain21 include:domain22 -all

Lookup #7 domain20:

    v=spf1  include:domain23 include:domain24 include:domain25 -all

Lookup #8  domain23:

    v=spf1  include:domain26 include:domain27 include:domain28 -all

Lookup #9 domain26:

    v=spf1  include:domain29 include:domain30 include:domain31 -all

Lookup #10 domain29:

    v=spf1  include:domain32 include:domain33 include:domain34 -all

The above would the recursive or "chained" order of the include
lookups. Sure, ridiculous, but we came across such abuse, rare,
trapped, so you don't about it, but it's there.

Perhaps, we should consider a lower limit for recursive
INCLUDE/REDIRECT mechanisms.

Finally, for MX and PTR records, we need to possibly state some
practical limits and/or point of that limits in RR records return can
fail a comparison under some scenarios.

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jan 2012 00:18:15 -0500

> it may be done recursively.

I disagree.  To recurse or not to recurse is a choice for the implementer.  It 
doesn't affect interoperability.

>     mechanisms that perform DNS lookups, specifically; A, PTR and MX.

redirect and include would be on this list as they inevitably cause DNS 
lookups.

> trapped, so you don't about it, but it's there.

I've seen records structured like this.  As you've described it here it could 
easily exceed the 10 DNS lookup limit, so I continue to disagree that from an 
interoperabliity perspective anything about recursion is different than DNS 
lookups and needs any different limits.

I think I've demonstrated the existing limits are stricter than what you were 
advocating for.  Absent some actual operational issues the technical limits 
should stay as they are, but I do think they could be explained better.

> fail a comparison under some scenarios.

MX and PTR already have separate limits (10 each).

Scott K

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 13 Jan 2012 12:34:46 +0100

> should (it's on my TODO list).

SPF doesn't seem to provide a computationally complete language,
however complex check_host() may be.  Thus, it should be feasible to
evaluate an SPF record statically, up to DNS hiccups, and measure its
prominent characteristics.  For example, given a domain name, one
could try and compute the following:

* number of terms appearing, including recursion,
* range of results that can be actually obtained,
* default result,
* number and type of IPs that can possibly result in a "pass",
* number of terms actually evaluated (average),
* number of necessary lookups (average),
* number of necessary lookups that are cacheable (average),
* TTL of cacheable data (average),
* number of necessary lookups requiring a TCP session (average),
* response time taken for the lookups (average),
* number of unreachable terms,
* number of reachable unknown modifiers,
* number of reachable unknown mechanisms,
* other errors?  What else?

Local macros can hinder such computation, but it is still possible to
try "postmaster" or similar heuristics.  Ditto for {%ir}.

Such a dictionary of metrics may lead to a better understanding of the
spec.  It will teach how to publish better SPF records.  That
specification can be a (reusable) part of the SPF experiment
description mentioned as the second deliverable.

I'd assign top score to a domain that tells "pass" for a tiny number
of IPs only, within UDP size, having one or more days of cache, and
without any further lookup.  Bottom score for domains that always
return PermError, requiring 10 or more uncacheable TCP lookups for
doing so.

Beyond the simple extreme cases, it is difficult to provide weights
that allow to end up with a single number --the SPF score-- for a
given domain.  I don't think we should delve into specifying precise
weights, but some other independent service provider might want to do
so, and it may result in a Good Thing (TM).

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jan 2012 12:15:15 -0500

> doesn't affect interoperability.

I guess that is why disable DNS Recursion is an option and it can have 
cause interoperability issues in variant results.

> redirect and include would be on this list as they inevitably cause DNS 

+1.

     This hard limitation implies SPF domain administrators are
     limited to SPF policy statements containing at most 10
     mechanisms that perform DNS lookups, specifically; A, PTR, MX,
     INCLUDE and REDIRECT.

> should stay as they are, but I do think they could be explained better.

If systems are going to limit by the # of directives, then its 
reasonable to look at a different lookup order such as resolving parts 
or the entire record first, before recursive calls are made, e.g.;

      v=spf1  include:domain2 ip4:1.2.3.4 ip4:5.6.7.8 
ptr:mx.domain2.com -all

I may want to do the no overhead rules, i.e. IP4, first to avoid short 
changing a valid result before the 10 limit is reached.

> MX and PTR already have separate limits (10 each).

IMO, 10 is unreasonable low to address the larger domains that have MX 
expanded results over 10.  No way I will limit this to 10 and not 
expect to see false positives/negatives.

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jan 2012 13:05:31 -0500

Hector Santos wrote:
> expanded results over 10.  No way I will limit this to 10 and not expect 
> to see false positives/negatives.

A following to reflect what I stated offlist to another participant 
discussing the limit.

It should be keep in mind there is a different between a SMTP client 
doing a normal MX lookup versus what SPF requires for an MX lookup.

For SMTP client, it is not required to expand the entire set of MX 
host.  This may be a factor in an SMTP client queuing algorithms, but 
overall, when doing an MX lookup, you MAY or MAY NOT get the IP 
records already expanded for you.  That depends on your DNS server, it 
may be a caching or its uplink is a DNS cached server.   The SMTP may 
get first example, 2-3 MX Host records, and only expand the first one 
(another DNS lookup) to begin the outbound mail session and the odds 
are high it can make the trip in this set of IPs without the 
additional complete DNS lookups to fully expand the MX records.

But for SPF, you have to choice but to do non-delay, full MX 
expansion, otherwise the potential is high to not be able to get the 
IP necessary to do the comparison with the client's connecting IP address.

Limiting this to 10 resultant records is, in my engineering opinion, 
extremely too low with a higher chance of causing SPF failures. Of 
course, only the larger domains will be affected.

AOL.COM would be one with a full expanded set of 19 IP addresses, but 
fortunately, in this case, it doesn't apply because their SPF is:

        v=spf1 ptr:mx.aol.com ?all

and for their valid IP clients, it returns only 1 host record.

But for a domain server with virtual domain hosting, returning more 
than 10 should not be unexpected.

    nslookup -query=ptr 208.247.131.9

9.131.247.208.in-addr.arpa      name = groups.winserver.com
9.131.247.208.in-addr.arpa      name = catinthebox.net
9.131.247.208.in-addr.arpa      name = pop3.winserver.com
9.131.247.208.in-addr.arpa      name = mail.catinthebox.net
9.131.247.208.in-addr.arpa      name = ftp.catinthebox.net
9.131.247.208.in-addr.arpa      name = secure.winserver.com
9.131.247.208.in-addr.arpa      name = ntbbs.winserver.com
9.131.247.208.in-addr.arpa      name = news.winserver.com
9.131.247.208.in-addr.arpa      name = ntbbs.santronics.com
9.131.247.208.in-addr.arpa      name = mail.winserver.com
9.131.247.208.in-addr.arpa      name = listserv.winserver.com
9.131.247.208.in-addr.arpa      name = winserver.com
9.131.247.208.in-addr.arpa      name = mail.santronics.com
9.131.247.208.in-addr.arpa      name = dkim.winserver.com

On a related note, does the ABNF, allow for multiple domains for a 
mechanism?

For example, if I wanted a SPF record for the above, could I use 
something like this?

    v=spf1 ptr:winserver.com,catinthebox.com,santronics.com -all

or would need to be done in three calls?

    v=spf1 ptr:winserver.com ptr:catinthebox.com ptr:santronics.com -all

Anyway, I believe 10 RR records is to short, especially for SPF MX 
lookups and I would to suggest that we either expand the SPF-BIF 
discussion about this and/or make new recommendations.  Something like 
perhaps:

    Limit of 32 for MX expanded IP records returns (DNS 4 bytes records)
    Limit of 16 for PTR host domain record (DNS string records)

The 16 is a SWAG for PTR and have no issue with some other number or 
keeping it at 10 which assumes a statistical average 51.2 byte size of 
domain strings. But for MX, 10 is way too short.

---

From: Douglas Otis <[email-masked]>
Date: Fri, 13 Jan 2012 15:31:57 -0800

On 1/11/12 2:39 PM, Derek Diget wrote:
>
>  On Jan 11, 2012 at 17:17 -0500, Hector Santos wrote: =>Section 10.1
>  Processing Limits, covers the security idea of limits. But it =>only
>  mentions one hard limit of 10 for MX and PTR. It recommends a
>  unspecified =>minimum for others like INCLUDE which is more of a
>  concern because of its =>recursive nature. We have a limit of 20
>  here. Probably more than enough to =>allow a domain to general an
>  include recursion level of 20! Each of these =>directive whether with
>  macros or not, needs to have limits. => =>So in my view, it will
>  probably be a good idea to avoid the "guessing game" =>and begin to
>  add semantics about specific recommended limits. => =>Doug is
>  suggesting of changing the 10 for MX/PTR to maybe 5. Where did 10
>  =>come from to begin with? I presume is what was some SMTP client
>  were already =>doing when sending out mail - they have a MX expansion
>  limit of 10. Maybe the =>10 come from the author's experience with
>  this.
Dear Hector,

This was simply explaining the impact caused by a change Wayne Schlitt 
made to host name limits in his SPF library.  He restricted the number 
of host names resolved to 5.  This caused a problem for t-online.de who 
never published SPF records but were subjected to "default" SPF records 
such as "a mx/24" and that had 8 hosts in their MX RRset, for example. :^(

The original concept was to permit any amount that could fit within any 
number of SPF records where only recursion was limited to 20.  Several 
expressed concern about the amount of traffic a recursion approach might 
cause.  Testing back in 2006 came across several libraries still using 
recursion as a limit, and some limiting hosts to the number returned 
within a single DNS transaction for a requested RRset, and the 5 host 
name limit used by Wayne instead of the 10 indicated in the draft.

>  I have always read the first sentence of the 3rd paragraph of RFC
>  4408 section 10.1 (starts with "SPF implementations MUST limit") to
>  mean that an SPF record that requires more that 10 DNS lookups to be
>  a PermError and the SPF library should not do a 11th lookup.
>  Period/end-stop.

Dear Derek,

The mechanism and modification limit is expressed in section 10.1
,--
SPF implementations MUST limit the number of _mechanisms_ and 
_modifiers_ that do DNS lookups to at most 10 per SPF check, including 
any lookups caused by the use of the "include" mechanism or the 
"redirect" modifier.
'-
Please note the term "mechanism" and not DNS transaction or imprecise 
lookup that might imply the existence of an intervening agent.  
"include" is a mechanism and "redirect" is a modifier that is counted as 
a mechanism.

The lookup limit expressed in section 5.4 "mx"
,--
This mechanism matches if <ip> is one of the MX hosts for a domain 
name.  MX = "mx" [ ":" domain-spec ] [ dual-cidr-length ] check_host() 
first performs an MX lookup on the <target-name>. Then it performs an 
address lookup on _each_ MX name returned. The <ip> is compared to each 
returned IP address. To prevent Denial of Service (DoS) attacks, more 
than 10 MX names MUST NOT be looked up during the evaluation of an "mx" 
mechanism (see Section 10 
<http://tools.ietf.org/html/rfc4408#section-10>). If any address 
matches, the mechanism matches.
'--

Each "mx" mechanisms can separately invoke address resolution 
transactions over a maximum of 10 host names.  As the dos draft 
demonstrated, a compliant library processed 10 mechanisms  where each 
mechanism in turn induced 10 subsequent DNS transactions to resolve each 
host's IP address set.  Tallying these DNS transactions is 10 times ( 1 
mx rrset + 10 host/a) or 10 times 11 for 110.  Of course, one must 
include the first transaction for the initial SPF record.   This bring 
the total DNS transactions used to resolve a single email address to 
111.  This is no where near 10 many keep suggesting.

>  The MX limit that is mentioned in later paragraphs is still
>  constrained by the first sentence's (10 lookup) limit. I would
>  almost say that the 4th paragraph ('When evaluating the "mx"') is
>  redundant to the 3rd paragraph except for the mention of the %{p}
>  macro.
>
>
>  (I can't speak for code writers of the MTA we use, but I think that
>  is the behavior I have seen. 11 DNS lookup causes a PermError to be

Perhaps you were confused by the term mechanism.  A mechanism such as 
"mx" or "ptr" performs DNS transactions to obtain RRsets and then up to 
10 subsequent transactions for each element of the RRset.  In other 
words, 110 DNS transactions.  This is made more dangerous when 
transactions are based upon hosts defined by email-address local-parts.  
As such, it would be wrong to suggest SPF results are for the entire 
domain or that they can be cached.  The local-part concept runs afoul of 
Repute definitions or efforts to combine the different domain results.

Regards,
Doug Otis

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jan 2012 20:42:44 -0500

Doug,

Since I'm not familiar with the idea of a "default" SPF record, I 
don't quite understand why a receiver would applied a "SPF default" to 
a domain that did make any SPF explicit declaration. Why would you not 
expect to see problems with this?  If this part of the specification?

That said, what we have (in our package) is a macro-based filtering 
language that includes a macro called DIP for DOMAIN::IP association. 
Its a lightweight LMAP rule that sysops can add to their filters.   We 
recommend to add them for KNOWN transactions that do not have SPF 
records and only do so if its clear to them that a DIP would 
accurately apply.  These are the DIP rules I have added over the years 
for our SMTP server:

;------------------------------------------------------------
Reason DIP whitelisted
;------------------------------------------------------------

accept if %DIP% = above.proper.com[208.184.76.39]
accept if %DIP% = bbsnets.com[66.8.157.90]
accept if %DIP% = dlois.com[216.220.225.104]
accept if %DIP% = fbg.collection.com[71.40.6.42]
accept if %DIP% = fbg.collection.com[71.40.6.43]
accept if %DIP% = hello.sd.dreamhost.com[66.33.201.150]
accept if %DIP% = ietf.org[132.151.6.22]
accept if %DIP% = imc.org[208.184.76.39]
accept if %DIP% = isdg.net[208.247.131.9]
accept if %DIP% = jck.com[209.187.148.211]
accept if %DIP% = lists.dsafe.org[66.33.206.23]
accept if %DIP% = lists.php.net[216.92.131.4]
accept if %DIP% = lists.xml.org[209.202.168.102]
accept if %DIP% = lists.xml.org[209.202.168.106]
accept if %DIP% = mail.imc.org[207.182.41.81]
accept if %DIP% = mail.imc.org[208.184.76.39]
accept if %DIP% = mail.songbird.com[72.52.113.17]
accept if %DIP% = maximus.com[12.150.181.2]
accept if %DIP% = mmx.engelschall.com[195.27.130.252]
accept if %DIP% = nic.fr[192.134.7.248]
accept if %DIP% = orgill.com[204.118.96.135]
accept if %DIP% = republico.estv.ipv.pt[193.137.7.30]
accept if %DIP% = tbsp.com[173.196.180.194]
accept if %DIP% = tclbbs.com[68.252.52.41]
accept if %DIP% = tclbbs.com[68.252.52.42]
accept if %DIP% = tclbbs.com[68.252.52.43]
accept if %DIP% = tclbbs.com[68.252.52.44]
accept if %DIP% = tclbbs.com[68.252.52.45]
accept if %DIP% = tclbbs.com[68.252.52.46]
accept if %DIP% = v2.listbox.com[207.8.214.5]
accept if %DIP% = v2.listbox.com[207.8.214.6]
accept if %DIP% = v2.listbox.com[208.58.1.195]
accept if %DIP% = wingate.com[202.180.113.230]
accept if %DIP% = ww-bbs.com[63.168.37.2]
accept if %DIP% = www1.ietf.org[132.151.6.22]

Note, these are ACCEPTS, so there is no harm per se compared to DIP 
Reject rules.

Similarly, for EHLO/HELP LMAP checks, we provide active examples in 
the default filter script file for rules that apply to us:

;------------------------------------------------------------
Reason HELO/EHLO mismatch
;------------------------------------------------------------

; The following rules basically says that only the machine
; at 208.247.131.* are allowed to use the domains listed
; as part of the HELO/EHLO command.   These are specifically
; for SSI. You should add your own domains with the IP address
; of where WCSMTP is running using similar rejection rules.

Reject if .santronics.com  in .%CDN% and %CIP% !in 208.247.131.*
Reject if .winserver.com   in .%CDN% and %CIP% !in 208.247.131.*
Reject if .isdg.net        in .%CDN% and %CIP% !in 208.247.131.*
Reject if .catinthebox.net in .%CDN% and %CIP% !in 208.247.131.*

Again these "default", SPF-like rules SHOULD only applied where there 
is precise knowledge.  I can't envision a general "SPF Default" rule 
applicable to all non-SPF domains.  Is there such a thing?

---

From: Douglas Otis <[email-masked]>
Date: Sat, 14 Jan 2012 06:27:07 -0800

On 1/13/12 5:42 PM, Hector Santos wrote:
>  Douglas Otis wrote:
> > Dear Hector,
> >
> > This was simply explaining the impact caused by a change Wayne
> > Schlitt made to host name limits in his SPF library. He restricted
> > the number of host names resolved to 5. This caused a problem for
> > t-online.de who never published SPF records but were subjected to
> > "default" SPF records such as "a mx/24" and that had 8 hosts in
> > their MX RRset, for example. :^(
>
>  Doug,
>
>  don't quite understand why a receiver would applied a "SPF default"
>  to a domain that did make any SPF explicit declaration. Why would you
>  not expect to see problems with this? If this part of the
>  specification?

Dear Hector,

AFAIK, t-online.de never published SPF records.  Some implementers 
leveraged SPF built-in functions to make "best guesses" about domains 
lacking SPF records.  This tactic is found in many SPF implementations 
as it will be for years.  When a guess by thousands of recipient 
libraries fail to provide consistent results, as it was for t-online.de, 
users believed the fault was with t-online.de, not realizing the actions 
of SPF foo.  As a result, this created a "defacto" rule for those not 
publishing SPF records, "Do not publish more than 5 hosts in your MX RRset."

>  Its a lightweight LMAP rule that sysops can add to their filters.
>  We recommend to add them for KNOWN transactions that do not have SPF
>  accurately apply. These are the DIP rules I have added over the
>  years for our SMTP server:

The concern is not with sysops using macros, although authenticated 
sources will provide better consistency and security.

The concern is that SPF allows _any_ domain to run SPF macros leveraging 
the powerful resources of recipient's email service providers.  Use of 
local-part macros means SPF results are NOT cache-able and always 
require repeated macro execution.  This is true even for cached SPF 
records!  Whether it is 10 or 100 transactions per message that might be 
generated, these transactions represent a FREE attack for any malefactor.

These additional transactions are unlikely noticed or logged by email 
service providers.  Their trigger is any email message that forwarded 
the malefactor's Mail From parameter as required by SMTP.  When a 
malefactor's SPF record targets some victim domain, messages sent to 
thousands of different domains will trigger their Distributed Denial of 
Service attack with a high level of amplification.  An attack where 
victims never see the messages, and where those reflecting the attack 
will likely remain unaware of their role.  None of the messages or SPF 
records need to contain the victim's domain since even this can be 
synthesized by SPF macros.

Virtually all other services authenticate with a brief exchange 
cryptographically verified within a few transactions that:
  1) factors are cache-able  (spf + local-part macros NOT cache-able)
  2) results are cache-able  (spf + local-part macros NOT cache-able)
  3) minimal demand on recipients (NOT cache-able and 100+ transactions/msg)

After the high resource expenditure, the email provider reflecting their 
resources remains an enigma, and only their IP address have been 
authorized.  The domain involved could be one of the 100K new seen daily.

>  applicable to all non-SPF domains. Is there such a thing?

Unfortunately, many consider making a guess is okay.  All of this risk, 
guesswork, and wasteful use of resources disappears with the use of 
normal server authentication.  Email service providers need to 
AUTHENTICATE to protect users, their reputations, and to function 
properly with IPv6.

Regards,
Doug Otis

---

From: Hector Santos <[email-masked]>
Date: Wed, 18 Jan 2012 05:46:19 -0500

Douglas Otis wrote:
> as it will be for years.  

Who?

This must be completed isolated.

I have a hard time understanding why would any system screw around 
with a guessing game  of "SPF default record" with completely not 
possibility for actual accuracy and yet, expect there would not be 
problem or rather begin to use this as some basis that something is 
broken about SPF!  I don't get it Doug. This would is already random. 
I see no way to presume there is a default SPF record without falling 
into trouble.

Now, if it was fundamentally the case that Senders *always* used an 
machine among the Receiveer network of MX host IP addresses, then I 
can see some default rules.  Sure.  But that is not reality. There is 
no requirement that a Sender machine must be part of any RECEIVER (MX) 
network.  This applies closer to smaller systems but not with larger 
network systems and not even with smaller systems.

I just don't see why you think this has any value.

---

From: Philip Gladstone <[email-masked]>
Date: Wed, 18 Jan 2012 09:40:27 -0500

> I just don't see why you think this has any value.
I think that the question could be rephrased as:

For a particular receiving MTA, for all mail senders who do not publish 
an SPF record, does the addition of "a mx/24 ~all" improve spam 
detection accuracy or not?

It is clear that using this record will cause all spam to be marked as 
~. Depending on your spam detection algorithm, this may be the level 
that you would select if there was no SPF processing. If this is the 
case, then the "a mx/24" record will only improve the situation.

Philip

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 18 Jan 2012 10:05:05 -0500

> I just don't see why you think this has any value.

The SPF "Best Guess" approach was supported early in SPF development 
(2003/2004) as a method to help bootstrap the protocol.  At the time, it was 
clearly described as something that was only even potentially useful for 
making positive assertions.  If someone based SPF rejections on a Best Guess 
record they were very mistaken.  I don't doubt this happened, but I don't 
think it's fundamentally an issue with the SPF protocol.  

The most that 4408bis might want to deal with the practice is to explicitly 
tell people not to make up SPF records for senders.

This, BTW, has nothing to do with local macros.

Also, a bit further up in the mail, the information about Wayne Schlitt's 
libspf2 is a bit misleading.  The limit of 5 pre-dated RFC 4408.  The current 
version of libspf2 uses the RFC 4408 limit of 10, so this notion that one of 
the editors of RFC 4408 had to use different limits in their implementation is 
not correct.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 18 Jan 2012 10:21:08 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Hector Santos
> Sent: Wednesday, January 18, 2012 2:46 AM
> To: [email-masked]

It's not.  A Google search for "best guess SPF" comes up with numerous records, and I've seen it referenced in white papers.  And I know for a fact that Gmail uses it.

The best definition I've seen is here: http://www.openspf.net/FAQ/Best_guess_record

I think part of the -bis effort should include an appendix that talks about this, albeit informatively, just so that a real consensus definition exists someplace.

-MSK

---

From: Hector Santos <[email-masked]>
Date: Fri, 20 Jan 2012 03:40:09 -0500

> think it's fundamentally an issue with the SPF protocol.  

Well, I recall something about, but since it only made sense to me 
from a LOCAL POLICY and accumulated KNOWN of domain information, not 
applied to anonymous system, we must of been on different wave lengths.

There is no logic when its applied to anonymous senders and the only 
conceivable rule is one based on an integrated multi-behavior MTA 
sender/receiver "All-In-One" machine/network. In other words, 
something like our own system that is a single MTA with multiple 
rules, receiver/router/sender.   Its a MSA when the user authenticates 
and mail is received, its a MDA when the user DOES NOT authenticate 
for local mail only reception and a thread does the routing for remote 
mail feeding it to the sender thread(s).   So such a rule would apply 
to us, but I assume it to be the case for other systems.

I agree, if such a MX/24 best guess rule is applied, it can only be 
used, at best, for passing. But if one tries to work a fail or soft 
results, IMO, that is completely wrong to do since there is simply no 
requirement for the sender/receiver to be associated via MX.  MX is 
for receiving only. Its not about sending. I recall the IETF-SMTP 
crowd a while back making a strong point of this to me when I was 
winging an ideal of using MX names formatted with sender IP 
address/mask information - a way to get information about the senders 
using an existing set of information most likely to exist for correct 
SMTP setups. I only thought about it because our own system is 
all-in-one multi-threaded receiver/sender/router.  So for 99% of our 
SMTP customers, the sender IP is the going to be among the MX IPs they 
have setup.

> tell people not to make up SPF records for senders.

+1, good idea because it doesn't make sense for anonymous senders. 
However, I do see value when you known the sender but they don't have 
an SPF record and you know for sure they have a fixed DOMAIN::IP 
association as I pointed out to Doug with DIP rules.  Just a method to 
white list known senders with known domain/ip associations.  I don't 
know if that should mentioned or not. But its not SPF, IMO.

---

From: Hector Santos <[email-masked]>
Date: Fri, 20 Jan 2012 04:06:10 -0500

Hector Santos wrote:

> sender/receiver "All-In-One" machine/network. In other words, something 
> like our own system that is a single MTA with multiple rules, 
> receiver/router/sender.   Its a MSA when the user authenticates and mail 
> is received, its a MDA when the user DOES NOT authenticate for local 
> mail only reception and a thread does the routing for remote mail 
> feeding it to the sender thread(s).   So such a rule would apply to us, 
> but I assume it to be the case for other systems.

Correction: "but I can't assume ....."

---

From: Hector Santos <[email-masked]>
Date: Fri, 20 Jan 2012 05:02:13 -0500

Philip Gladstone wrote:

>> Hector Santos:
> ~. 

My Opinion.

I would think an ? (Neutral) would more closer follow technical 
expectations.  A Softfail (~) presumes a specific operational SMTP 
setup for the sender's receiver network being integrated in some 
fashion and since this is not a requirement, presuming a "level" of 
failure would be, IMO, erroneous if its to be reported as a SoftFail 
and there was never an official domain declaration, nor expectation, 
for this practice - a form of misrepsentation and tort.

> Depending on your spam detection algorithm, this may be the level

IMO, only to get a PASS, at best, which can only yield a data point 
indicating the sender/receiver network is the same or related via the 
"a mx/24" rule. What value does this type of sender/receiver 
operational knowledge offer?

IMO, anything less than a PASS and more than unknown/neutral, i.e, 
soft fail, would have to be determined using some non-SPF logic. 
Perhaps an MX IP "RBL" check or perhaps an CBV on the return path 
which MUST NOT be invalid per 5321.

So it seems to me, this is more about "leveraging" SPF reporting for 
some form of "Failure" checking entirely unrelated to SPF.

Maybe the question is how much does a "A MX/24" rule apply today as 
compared to yester-years?  One can suggest that with multi-core and 
faster OS/hardware, tons of virtual memory, the TCO can be lowered by 
scaling up (more CPU) and reducing scaling out (less machines).  In 
this vain, I can see this rule being increasingly more applicable or 
imaginable to be true using a Pareto 80% of the time, but its still 
not a requirement for systems to be setup like this.

---

From: Hector Santos <[email-masked]>
Date: Sun, 22 Jan 2012 00:55:14 -0500

Hi Murray,

I guess it all depends on how its written, but my only concern is that 
no one should get bent out of shape about trying to implement a 
"default rule" that would apply to all. I say that from the point of 
view that REJECTION is a strong part of SPF and this best guess idea 
can only serve a purpose for "reporting," at best, for one data point:

     The sender and receiver are part of the same network.

If this was a fundamental concept for all, i.e. part of the spec, then 
I can see the default "rule" applying for filters and IMO, we really 
never would need SPF.

But that is not an SMTP setup operational requirement, so I find no 
value in this default rule other than getting some "interesting data 
point" about the sender network setup.  The Sender/Receiver is part of 
the same network. So what?  It means nothing, at least I don't see it 
unless a NON-SPF logic is done to see if the MX is black listed or 
doesn't exist.

Per SMTP, an MX is not a requirement and we use a fall back to the A 
record.  If that fails, depending on the implementation it could be 
filtered. i.e. many systems use a "NO MX/TRIED A record Once" concept 
when sending mail.

--

Murray S. Kucherawy wrote:
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
> 
>

---

From: Philip Gladstone <[email-masked]>
Date: Mon, 23 Jan 2012 11:18:16 -0500

There is significant value in distinguishing neutral from positive. This 
is when this signal is an input into an anti-spam system. A positive SPF 
eval will (typically) allow slightly spammier looking emails through, 
whereas a neutral would be slightly stricter.

Philip

On 1/22/2012 12:55 AM, Hector Santos wrote:
>>> Behalf Of Hector Santos
>> It's not.  A Google search for "best guess SPF" comes up with
>> numerous records, and I've seen it referenced in white papers.  And I
>> know for a fact that Gmail uses it.
>>
>> The best definition I've seen is here:
>> http://www.openspf.net/FAQ/Best_guess_record
>>
>> I think part of the -bis effort should include an appendix that talks
>> about this, albeit informatively, just so that a real consensus
>> definition exists someplace.
>>
>> -MSK

---

From: Commerco WebMaster <[email-masked]>
Date: Mon, 23 Jan 2012 11:03:56 -0700

Perhaps it is also important to remember that SPF is not really an anti 
spam system per se, rather it is a standard which primarily confirms 
that a domain name has indeed been permitted by its holders to send 
messages from a defined set of MTAs.  Obviously, SPF does have some uses 
as regards reducing spam in situations where a bad guy may be hijacking 
a domain name for their spam and thus besmirching the reputation of the 
domain name in the eyes of the receivers of said spam.

So, if a spammer sends a message via a domain which they actually do 
hold (and thus can control DNS and consequentially create appropriate 
SPF or TXT records), their mail would certainly be legitimate from an 
SPF standpoint.  The value of SPF in this case is that the spammer 
cannot then go on to claim that they were not the originators of the 
mail, or that somehow their mail servers were hijacked by some unknown 
bad guy.

A positive SPF eval should always confirm that the sending domain name 
was permitted to send, by DNS configuration, a message via the MTA from 
which the message came.  Thus, all the intended valid email for a domain 
name should make it through to the receiving MTA, while additional MTA 
processing would be needed to address the spammier looking email which 
made it through, as it was before SPF ever existed.

Best,

Alan M.