Subject: [spfbis] SPF and EAI

From: "John Levine" <[email-masked]>
Date: 12 Jul 2012 00:45:27 -0000

Since Doug's brought up EAI, here's a few comments from someone who's
been active in the EAI WG.

Executive summary: we have a problem.

An EAI (now officially called SMTPUTF8) address has a UTF-8 localpart
and a UTF-8 domain.  The domain has to a U-label, that is, limited to
characters that can be translated to an A-label (xn--whatever) and
looked up in the DNS.  There is no such restriction on the localpart.
It can be any UTF-8 that fits in 64 octets.  Technically, the
definition of A-labels changed between 2003 and 2008, in practice the
difference only affects two obscure characters that nobody has seen in
the wild, so we can ignore it.

It is reasonable to say that the macros that expand to domain names, o
d p h r and their various transformations are turned into A-labels,
since as I understand it, the main reason to use macros is to concoct
a name that is to be looked up in the DNS.

For macros that expand to local parts, s l and their variants, you
can't do that.  The local part does not have to be a U-label, so in
general, you can't turn it into a domain name.  This isn't entirely a
new problem; ASCII localparts can contain arbitrary characters that no
sane person would put into a domain name, e.g. A#$!%")z, and
localparts are in principle case sensitive and domain names aren't, so
localpart macros have always been ambiguous.

My suggestion would be to say that the domain names are turned into
A-labels, but s and l macros are deprecated, since there's no way to
turn them into domain names short of some crock like base64 encoding
them which would not be backward compatible.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 11 Jul 2012 21:29:27 -0400

> them which would not be backward compatible.

Localpart macros that can't be represented as A labels are pointless, ASCII or 
UTF-8.  Rather than deprecate them why not just limit them to things that can 
be represented that way?  Yes, it imposes some limitations on theoretical 
future use, but I think it would be backward compatible and support about as 
much of SMTPUTF8 as we can without having to recharter.

Scott K

---

From: Dave Crocker <[email-masked]>
Date: Wed, 11 Jul 2012 18:58:01 -0700

> Executive summary: we have a problem.

Eventually, perhaps.

On the average, it is not wise to make any significant adjustments for 
new standards that have yet to achieve widespread deployment.

EAI has a very, very rocky history.  Perhaps this latest round of effort 
will indeed gain traction.  In spite of the obvious and major need for 
something like EAI, we are nonetheless left with a poor history -- both 
before and during the current effort -- and with specifications that 
remain a bit troubling.

My advice is that the current SPFbis effort should continue focus on 
actual SPF experience.  If there are small changes that can aid in EAI 
use, then it's probably safe to add them.  I doubt there are any however.

Save the larger changes for when EAI has established its operational 
success.

d/
-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: "John Levine" <[email-masked]>
Date: 12 Jul 2012 02:20:51 -0000

>be represented that way?

Well, OK.  If an address shows up with a local part that's not a U-label, and
the SPF record contains %l, what happens?

For that matter, if the local part is @*#&$)(! ";  what happens now?

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 11 Jul 2012 22:59:41 -0400

On Thursday, July 12, 2012 02:20:51 AM John Levine wrote:
> >Localpart macros that can't be represented as A labels are pointless, ASCII
> >or UTF-8.  Rather than deprecate them why not just limit them to things
> >that can be represented that way?
> 
> Well, OK.  If an address shows up with a local part that's not a U-label,
> and the SPF record contains %l, what happens?
> 

It won't match, so it's not a pass/neutral/etc.  I think it's the same for 
this.

At least this is my interpretation.  IIRC it doesn't explicitly say that, but 
given the way it all works, I don't see how it could be anything else.  

I think localpart macros are rare enough that no one need any sleep over them 
not supporting the full range of UTF-8, but they are in use so it's out of 
scope to ditch them (I believe we resolved that already).

Clearly we need some clarification on exactly how to go about this, but I don't 
see a major issue.  Since a localpart that can't be represented as an A label 
can't work as a localpart macro, the worst is there are some localparts that 
you can't use.  There is no case where the limitation causes things to fail 
open.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 11 Jul 2012 23:47:46 -0700

> scope to ditch them (I believe we resolved that already).

My data from the experiment document show 220 records using macros of any
kind, out of over 150,000 responding.  Only 18 of those contain local part
macros.  I haven't checked Philip's data but I expect it to be similar.

I would support deprecating them if this is a real problem.

I'm also fine with Scott's proposed solution, if we can agree on it.

-MSK

---

From: Dave Crocker <[email-masked]>
Date: Thu, 12 Jul 2012 07:41:20 -0700

> I would support deprecating them if this is a real problem.

given how little they are used and the inherent complexity added by 
macros, wouldn't it make sense to deprecate them anyhow?

d/

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 12 Jul 2012 10:43:16 -0400

> macros, wouldn't it make sense to deprecate them anyhow?

We already went through this once and decided not.  I don't think EAI changes 
that.

Scott K

---

From: Dave Crocker <[email-masked]>
Date: Thu, 12 Jul 2012 07:59:24 -0700

> We already went through this once and decided not.  I don't think EAI changes

I am reacting to the statistic that Murray provided, not to EAI issues.

A feature that adds significantly complexity in design, development and 
operation (debugging) but has gained so little use is a feature worth 
dropping.

d/

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 12 Jul 2012 11:12:14 -0400

On Thursday, July 12, 2012 07:59:24 AM Dave Crocker wrote:
> > We already went through this once and decided not.  I don't think EAI
> > changes that.
> 

We've already been through this once.  It's issue 
http://trac.tools.ietf.org/wg/spfbis/trac/ticket/14 and the chairs marked it 
closed/wontfix.  We have enough to do just doing everything once, I'd ask we 
try and avoid repeats.

Scott K

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 12 Jul 2012 18:06:54 +0200

> It won't match, so it's not a pass/neutral/etc.  I think it's the same for 

No, wait.  That makes sense for constructs like (Section 9.1.2)

   v=spf1 exists:_h.%{h}._l.%{l}._o.%{o}._i.%{i}._spf.%{d} ?all

However, a local part may also appear in an explanation (Section 6.2)

   "See http://%{d}/why.html?s=%{S}&i=%{I}"

Since SPF is not giving a possibility to URL-encode @*#&$)(! "; it is
not going to add it now for UTF-8.  However, what if an SPF
implementation needs to return an UTF-8 reply from a server which does
not implement SMTPUTF8?

> you can't use.  There is no case where the limitation causes things to fail 

Agreed.  How about s/US-ASCII/UTF-8/?

It makes sense to try and convert to an A-label whatever string is not
US-ASCII /just before/ doing a query, as a general rule.  Failed
conversions result in permerror.  With such kind of rule, SPF can
accept either encoding in input, both for parameters and SPF records.
 Mail admins who are so savvy as to limit local parts to U-labels can
expect the above "exists" to work untouched.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 12 Jul 2012 20:49:18 -0700

> closed/wontfix.  We have enough to do just doing everything once, I'd ask
I believe that ticket was closed based on the argument that originally
opened it, namely that local part macros are a DNS load or security
consideration.  I agree that this argument failed to gain traction, and we
closed it on that basis.

What I'm talking about (and Dave is agreeing to) is that macros in general,
and local-part macros in particular, see even less use than Sender ID, so
it's within charter to remove them in RFC4408bis on that basis alone.  It
has nothing to do with the original argument in tracker item 14.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 00:10:50 -0400

> has nothing to do with the original argument in tracker item 14.

The charter says "removal of unused features".  This is different than type SPF 
because type SPF was redundant, this is not.

I don't think it's in our charter to break existing, published SPF record 
without a compelling case.  I don't think rarely used is at all compelling.  I 
don't think simplification is at all compelling either.  

Yes, writing an SPF library would be simpler without macros, but I don't think 
there's a lot of new library writing going on today.  It was done 5 - 7 years 
ago, so whatever theoretical benefit simplification would bring, in practice 
there won't be any.

I checked and in pyspf there are, by my count (and including blank lines, 
comments, and tests), 169 lines out of 1,986 related in any way to macros.  
Removing macros would make things slightly simpler for whatever future library 
writers there may be, but it removes a tool from the box that some people find 
useful.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 12 Jul 2012 21:58:33 -0700

> because type SPF was redundant, this is not.

I think the issue of redundancy is inapposite.  It's an unused feature,
pure and simple.  The statistics show far less less than 1% use.

Why do we need to keep something as complicated as macro expansion when
almost nobody is using it?

Perhaps more importantly: Why was it okay for us to decide Sender ID is de
facto obsolete, while even stronger evidence (by orders of magnitude)
against macros must be disregarded?  If it would help, I could add up the
number of lines in sid-milter that added Sender ID support for comparison.
It, too, was relatively small.

> don't think simplification is at all compelling either.

Purely from a procedural standpoint, I disagree.  SPF is currently
Experimental, and moving to Proposed Standard is an ideal time to chop out
things that fewer than two tenths of one percent of the deployed base is
using.  That comprises part of what we learned from the experiment.  I
don't agree that we are mandated to cater to such a tiny number of sites.

The remaining question is whether the working group believes use by 200 out
of 150,000 warrants keeping the feature.  To use your own argument, if in
5-7 years we've seen only this much use of SPF macros, I'm doubtful it's
going to increase.

But, to be fair, we don't have a hard-and-fast definition of "unused" when
going through this exercise, so it's left to consensus. Others will have to
weigh in.

We do have precedent, however, as a guide.  For example, when DKIM went to
Draft Standard, we chopped a few things (most notably "g=") because they
had seen almost zero adoption.  According to the RFC4871 interoperability
report, after 8.1M signed messages were processed, only 434 of keys were
found that had "g=" tags in them.  So the use wasn't zero, and someone
apparently found them potentially useful, but the decision was made that
there was not enough adoption to warrant keeping the complexity in the
specification.

However, we didn't remove "l=", much as we wanted to, because its use was
around 3.7%.  Same with "x=" (3.4%) and "z=" (5.9%).  Use of SPF macros
falls somewhere in between those and "g=".

There are other examples of this tactic going back as far as RFC821 (SAML
and SOML).

You argued that the complexity isn't that big based on lines of code.  It
wasn't for "g=" either.  I removed 162 lines of code that time from
OpenDKIM, and we removed a small section of the RFC.

As is often quoted around these parts: "“Perfection is achieved not when
there is nothing left to add, but when there is nothing left to take away."

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 01:45:56 -0400

> almost nobody is using it?

We gain nothing from dropping it and we break a small number of existing 
records.

> It, too, was relatively small.

We didn't decide that.  It says right in the charter that Sender ID has been 
judged not to merit further development.  If some group has the energy to go 
work on SenderID and can get a working group chartered, more power to them.  
They'll first have to get some standards incompatibilities out of the way (and 
I don't think they can - this is discussed in the various appeals associated 
with it when it was published).  In any case, should Sender ID live or die 
isn't this working group's call and by whatever mechanism that was decided, 
it's irrelevant to the work of this group (the charter even says it had 
enjoyed wide scale deployment - deployment wasn't the issue).

> don't agree that we are mandated to cater to such a tiny number of sites.

It seems a bit odd that if we learned that as a lesson of the experiment, we 
didn't think important enough to mention in draft-ietf-spfbis-experiment.

>From a purely procedural standpoint you are right.  For largely political 
reasons it was put in Experimental.  The reason the charter is written as 
tightly as it is though is to keep people from using that excuse to break the 
protocol.  It's going from experimental to standards track, but it's not a 
normal case.

> 5-7 years we've seen only this much use of SPF macros, I'm doubtful it's

Because there's no compelling reason to remove it.  It's not unused and for 
whatever corner cases it is used for, I believe it's important (it's not easy 
to use well, so anyone who's using it does, I think need it).  While catering 
to corner cases isn't something we should focus on, I see no need to go out of 
our way to break them either.

> There are other examples of this tactic going back as far as RFC821 (SAML

Did any of these precedents involve working groups where the charter said only 
unused features could be removed? If not, I don't think they are relevant.

> there is nothing left to add, but when there is nothing left to take away."

I'm well aware of that, but nothing left to take away can be defined in 
different ways.  IMO, if (absent some really compelling evidence of which I've 
seen no sign) anyone has to go republish SPF records after we're finished we 
haven't lived up to the spirit of the charter no matter how finely people want 
to parse the words to claim we have.

   Changes to the SPF specification will be limited to the correction
   of errors, removal of unused features, addition of any enhancements
   that have already gained widespread support, and addition of
  clarifying language. 

I don't think any of those apply here.

I think the benefits of removal are almost entirely theoretical and the benefits 
of keeping macros in general and "l" specifically are very practical.  

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 12 Jul 2012 23:32:48 -0700

> We gain nothing from dropping it and we break a small number of existing

I don't agree that we gain nothing from dropping it.  We gain simplicity.
Perhaps the value of simplicity in a Proposed Standard is being
underestimated.

> didn't think important enough to mention in draft-ietf-spfbis-experiment.

Why would use of SPF macros be germane to a document whose function is to
compare adoption rates of SPF versus Sender ID, and not their specific
components?

As it happens, the data collection we did for that effort includes
something that's interesting to this one.  I don't think that's odd at all.

Because there's no compelling reason to remove it.

But there is.

> unused features could be removed? If not, I don't think they are relevant.

Yes, all of them, since it's normal IETF procedure when upgrading the
status of something on the standards track.  You might want to review
RFC6410 Section 2.2, which covers advancement from Proposed Standard to
Internet Standard (Draft Standard has been eliminated since DKIM was
updated), as it includes as one of the criteria:

   (3) There are no unused features in the specification that greatly
       increase implementation complexity.

Macros are both relatively unused and increase complexity.  I'm trudging
through Section 8 in my review now.

The only "outs" here to me are the facts that (a) we're moving from
Experimental to Proposed Standard, which RFC6410 doesn't cover (although I
believe it's still useful to apply these criteria as good practice), and
(b) the definition of "unused" in this context is fuzzy.

I believe we have a stronger Proposed Standard by removing this or anything
else that yields no operational or community benefit.  Section 8 is not a
trivial piece of work for an implementer to follow and get right.  No, it's
not impossible, but I don't agree that it's at all straightforward either.

> I don't think any of those apply here.
I don't agree.  The statistics speak clearly to me that macros are an
unused feature, and "removal of unused features" appears in the very
charter text you cited.

I've said my piece here.  Others will have to weigh in.  However, I ask
that the co-chairs open this as a new tracker item as a separate issue from
#14.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 02:53:06 -0400

>        increase implementation complexity.

Requiring unused features to be removed is not at all the same thing as 
requiring used features be kept (which is what the SPFbis charter says).  
Additionally, while I agree that macros increase complexity, I wouldn't agree 
with the idea that they greatly increase complexity.

(and with that, I'll leave it be too)

Scott K

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jul 2012 06:59:45 -0400

The premise that an documented feature in a specification is unused is 
now a basis for removal,  opens the highly subjective door to 
popularity contests and "Who's Who Using What" reasonings.  Case in 
point, the statement "its an unused feature, pure and simple" is not a 
fact.   If this premise is used across the board, I sincerely doubt we 
would have extremely wide support for (or it would be very hard to 
support) protocols when a "less popular" feature is always open for be 
subjectively removed and pulled from under your feet.  Fortunately, it 
hasn't and doesn't work that way.

I would prefer to discuss the proposal of removing a (SPF) feature on 
the basis there is a technical and/or security problem.  Odds are 
good, it is probably inherently less used anyway on that basis and 
IMO, worthy of discussion.

IMO, the issues with EAI appear to be complex, therefore unsettled and 
a moving target.

-- 
HLS

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 13 Jul 2012 10:17:19 -0400

Chair hat on.

On Thu, Jul 12, 2012 at 07:59:24AM -0700, Dave Crocker wrote:
> A feature that adds significantly complexity in design, development
> and operation (debugging) but has gained so little use is a feature
> worth dropping.

As you know, I was quite uncomfortable with the decision about the SPF
type not because I think TYPE 99 is real useful, but because our
charter says that we are to drop _unused_ features.  Dave, at the time
you argued for an interpretation of "unused" that was really "not used
very much", where the value of "very much" had to be evaluated by the
WG.  We avoided having to make a decision about that, however, because
we concluded that there was a significant ambiguity in the published
RFC and that therefore we would be required to make a change.

This case is not like that: there's no requirement to use these macros
at all in order to get the protocol working, and even though the
protocol is complicated by them we have positive evidence that some
people are using them, which presumably means that they can't achieve
their desired goal through some other (equivalent) functionality
already available in RFC 4408.  Therefore, I cannot see how we are in
any way chartered to ditch this feature, unless we opt for Dave's
somewhat strained interpretation of "unused".  

Now, with my hat off, why can't the document simply note that there
are significant issues entailed by local part macros, that they won't
work at all with internationalized addresses, and leave it at that for
now?

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 10:30:31 -0400

> work at all with internationalized addresses, and leave it at that for

It can and should do exactly that.

Scott K

---

From: Commerco WebMaster <[email-masked]>
Date: Fri, 13 Jul 2012 08:56:20 -0600

> work at all with internationalized addresses, and leave it at that for

+1.  Although I do appreciate the benefits Murray points out in reducing 
complexity of a specification, there is certainly no requirement to 
implement the marcros in SPF.  A note mentioning the downsides of doing 
so with this optional facility within the SPF spec should suffice and 
seems a good compromise.

As to this issue vs the DNS SPF type 99 issue, while I also shared the 
concern in dropping the use of SPF type 99 in the current spec, it was 
done so because of its potential for "breaking things".  Stretching the 
charter in that case to ensure a better working specification was almost 
certainly worth it.

Best,

Alan M.

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 11:14:17 -0400

On Friday, July 13, 2012 08:56:20 AM Commerco WebMaster wrote:
...
> so with this optional facility within the SPF spec should suffice and 
...

Just to make sure we're all on the same page about this ...

By, "no requirement to implement the macros in SPF", I gather you mean "no 
requirement to use macros in an SPF record".  Support for macros as part of 
receiver processing of SPF is not optional.

Scott K

---

From: Dave Crocker <[email-masked]>
Date: Fri, 13 Jul 2012 08:18:26 -0700

> with the idea that they greatly increase complexity.

      1. "Unused" is a statistical label.  It does not mean that there 
are literally no implementations or deployments.  It means that it has 
not gained serious use.  Anything that's been available this long and 
has reached such a tiny percentage of deployment is unused.  (Murray's 
comparison to Sender-ID statistics brings this home nicely, IMO.)

      2. A recursion construct is inherently complex.  It invites both 
software and configuration bugs and they are naturally more difficult to 
debug, especially across the Internet with different administrations and 
where the parties are not actively debugging at the same time.

The more simple a specification, the easier it is for a wider range of 
readers to understand it and the easier it is for a wider range of 
implementers and operators to follow it correctly.

The cost of any feature is exponentially higher than we tend to think, 
because it affects so many implementations and installations.  On 
Internet scale, that's a very, very large number.

d/

---

From: Commerco WebMaster <[email-masked]>
Date: Fri, 13 Jul 2012 09:36:11 -0600

> receiver processing of SPF is not optional.

Sorry about that.  Just me being publisher centric.  Yes, you 
interpreted my intent correctly.

Alan M.

---

From: "John Levine" <[email-masked]>
Date: 13 Jul 2012 15:51:38 -0000

>> work at all with internationalized addresses, and leave it at that for

I'm more worried about the broken by design aspect of macros,
particularly localpart macros, since they let bad guys inject
arbitrary text strings into the expanded text.  In the example of
generating an explanatory URL in an exp: record, a suitably forged
bounce address can inject ? & and other interesting characters into
the URL.  Or it can inject arbitrary invalid characters into domain
names to be looked up, which DNS client libraries may defend against
and may not.  (How many people have checked to see what happens if you
feed ^D or ^Z or characters with the high bit set to your local
version of gethostbyname?)

I realize there are upper case versions that do percent expansion, but
if we leave macros in, we can't force people to use them.  Given how
few people have found them useful, I'd rather just deprecate them.

R's,
John

PS: In case it's not obvious, this is not the same issue as Doug's
alleged DNS storms.

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 13 Jul 2012 11:53:16 -0400

Hat on.

On Fri, Jul 13, 2012 at 08:18:26AM -0700, Dave Crocker wrote:
> 
>      1. "Unused" is a statistical label.  It does not mean that
> there are literally no implementations or deployments.  It means
> that it has not gained serious use.  Anything that's been available
> this long and has reached such a tiny percentage of deployment is
> unused.

Just to be clear: you mean that as a proposed interpretation, not a
statement of fact, right?  

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 13 Jul 2012 18:18:09 +0200

On Fri 13/Jul/2012 16:17:19 +0200 Andrew Sullivan wrote:
> already available in RFC 4408.

+1, type 99 has an obvious and painless replacement, while replacing
an already deployed authorization scheme that uses local-parts can be
plainly unfeasible.

jm2c

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jul 2012 12:42:15 -0400

Dave Crocker wrote:
> not gained serious use.  

So now we need to define "serious."

Rhetorically, is it related to vendor size, influence, IETF or trade 
group meetings, attendance, not just # of vendors but which ones, 
industry size, etc?  I am sure we get that point. I hope.

I don't think SPF macro language falls in that category of 
considerations. Personally see it is rarely used (according to a quick 
dns cache read), it is nonetheless in play. I also have not seen DNS 
Storms in the 9+ years for product deployment among thousands of 
operators. I believe the limits may be part of the security blanket. 
(I thought this issue was dropped because in fact, we do have limits?)

Finally, isn't this two different issues? macro expansion versus EAI 
text conversion methods?  I presume one is about overhead and the 
other complexity/correctiveness/capability.

-- 
HLS

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jul 2012 13:25:51 -0400

> It can and should do exactly that.

IMV, since this continues to be a moving target with associations 
across many applications levels, there is an increase need to 
understand this in general.  So I think it is better to described it 
as best as we know it now. Personally, one that often comes into logic 
play is when conversion is expected or passthru. i.e. should the 
conversion be dynamic or once when the input stream is received?

I do think that no matter what, even when the receiver does all it 
suppose to do correctly, if the result is a calculation failure then 
we should consider what the SPF result should be.

    NONE?
    SOFTFAIL?

To me, NONE is ok, but too much or on-going promotion to NONE which 
there is intent to use SPF does give receivers the natural server 
abuse protection right to take action against this unnecessary overhead.

The same issue applies to the TEMPFAIL/SERVFAIL issue and what should 
be returned.  Receivers have an inherent right to control abuse and 
redundancy (never ending) errors, therefore a temporary error "SHOULD" 
be allowed to be promoted to an permanent error.

-- 
HLS

---

From: Dave Crocker <[email-masked]>
Date: Fri, 13 Jul 2012 11:53:38 -0700

On 7/13/2012 7:17 AM, Andrew Sullivan wrote:
>   Dave, at the time

Good to know that I am sometimes consistent, since I am still putting 
forward that view...

> somewhat strained interpretation of "unused".

Besides re-raising the question of whether 'unused' requires the 
mathematical perfection on 'none' versus 'extremely few', you have 
introduced another aspect of 'used' that is often debated.

That is, the distinction between "in the code" vs. "over the wire". They 
are very different kinds of usage, of course.

My view is that putting something in the specification essentially 
requires implementation in code and that that's expensive.  Yes, many 
scenarios of non-use operationally can formally justify not implementing 
in code, but I believe it is more typical to have it in the code but be 
unexercised.

Stated differently:  If it's in the spec, there will be many 
implementations.  That's expensive.  If we have solid evidence that it 
will be (essentially) unused operationally, we have caused the industry 
to waste significant software development effort AND deployed software 
that is not well-tested by the crucible of actual use.

> work at all with internationalized addresses, and leave it at that for

cost and bugs.

d/

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Jul 2012 16:03:17 -0400

> in code, but I believe it is more typical to have it in the code but be 

   May I remind everyone that "coded but not tested" is a sure-fire
recipe for disaster?

> that is not well-tested by the crucible of actual use.

   I understand that folks who have already implemented SPF look at
every "change" as yet another opportunity for disaster; but there _will_
be others implementing from scratch...

> Andrew Sullivan <[email-masked]> wrote:
> 

   I don't think "simply note" will cover it.

   We have to deal with changes in the environment since RFC4408:
specifically
- IPv6, and
- internationalized email addresses.

   It is not instantly obvious how to deal with these in SPF macro
expansion. We _could_ work it out -- or we could deprecate them.

   "Deprecate" feels right to me. Please understand, "deprecate" means
" 
" senders SHOULD NOT publish SPF records that make use of the feature
" in question. Its use is NOT RECOMMENDED and it might be removed entirely
" in future updates to the protocol. Such features do, however, remain
" part of the SPF protocol and receiving systems MUST support them unless
" this specification says otherwise.

   So, anything we mark "deprecated" we don't have to handle well: we
can say essentially "Use at your own risk."

   I think I agree with Dave here. (But no doubt Dave will find a way
to prove me wrong. ;^)

--
John Leslie <[email-masked]>

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Jul 2012 16:45:50 -0400

The difference is that original functional macro support was already 
part of the development expense long ago and it is also a requirement 
on the receiver side.  So with a proposed removal, we are now talking 
of additional expenses:

  - Programmers to remove it,
  - Testers to continue to see if it (removal) didn't break anything else,
  - Documentators to explain or add/change the operators and 
references guides,
  - Teaching support people that new complaints may be reported when 
the removal
    fails to support any published advanced SPF records.

That all new expenses "serious" implementators need to consider when 
"change (removal)" is proposed for an original protocol feature.

-- 
HLS

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 17:10:31 -0400

>    May I remind everyone that "coded but not tested" is a sure-fire

In 2005, before there was an SPF test suite, I was grateful that Philip 
Gladstone had published SPF records with macros.   It was an invaluable source 
of test cases that helped me fix a bunch of bugs in the pyspf macro processing 
code, so I know exactly what you mean.

Fortunately, the situation is different now.  You can go to 
http://www.openspf.org/Test_Suite and get a copy of a comprehensive test suite 
that is believed to cover all the RFC 4408 requirements.  It includes section 
number citations of what requirements are being tested and where the SPF 
community (after a lot of discussion) felt RFC 4408 had some ambiguity about 
what the correct result was.  

It's been stable for almost three years.  It has been used both to validate 
correctness in existing implementations and by developers working from scratch 
to assist them through the process of development.

I don't anticipate significant changes based on what's been done so far.  The 
only things that need to be done is to set the ambiguous desired results to 
the 4408bis correct result (I believe that the current draft resolves all of 
them).

So yes, untested code is bad, but if you aren't doing comprehensive testing, 
you've got no one to blame but yourself.  The community has already done the 
hard part of figuring out the exact test cases you need to run. 

> be others implementing from scratch...

All of the "new" implementations I'm aware of have been done via integration 
of existing open source libraries that provide the core functionality of the 
protocol.  There are C, Java, Python, Perl, and .Net libraries available that 
I know of.  See http://www.openspf.org/Implementations .

Once again, anyone who ignores the existing wealth of code out there to reuse 
has already taken a gun and aimed it squarely at their feet.  People who want 
to do it the hard way are going to get about the result you would expect no 
matter what we put in 4408bis.

> - internationalized email addresses.

Other than a few places in the spec where we say A instead of A and AAAA, IPv6 
is already fully addressed (and I'm unaware of any implementations that didn't 
assume that anyway).  I don't think the coming of IPv6 presents any new 
issues.

As we've discussed, I think the internationalized email addresses are almost 
as easy. You can only use things in SPF records (including those with macro 
parts) that can be represented as an A label.  Once again, no impact to the 
protocol.  The only effect is a limitation on what internationalized localparts 
can be represented by the "l" macro.  So something you couldn't do before, you 
still can't do.

> can say essentially "Use at your own risk."

I think it's perfectly obvious how to deal with this issues and other than a 
few cases of "you can't do that" I don't think there's any problem at all.

The potentially significant operational issue is that some records which use 
macros can't be cached.  RFC 4408 (and our draft) already address this:

Note: Domains should avoid using the "s", "l", "o", or "h" macros in
   conjunction with any mechanism directive.  Although these macros are
   powerful and allow per-user records to be published, they severely
   limit the ability of implementations to cache results of check_host()
   and they reduce the effectiveness of DNS caches.

I have no objection to beefing that up, but I'm not aware of anyone have actual 
operational issues as a result of this (unlike PTR, which we've deprecated - I 
know of cases where those cause real problems).

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 13 Jul 2012 14:12:57 -0700

> be others implementing from scratch...
This is largely my concern.  It's virtually guaranteed* that there will be
some industry participant that can't use any of the existing open source or
even commercial implementations for whatever local infrastructure reason
and has to implement from scratch.  We should make their jobs as easy as
possible, lest interoperability or even security issues be introduced.

-MSK

* Please don't ask me to give a precise quantitative definition for this.

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 17:20:32 -0400

> * Please don't ask me to give a precise quantitative definition for this.

For this corner case, I think the detailed test suite we have already proven 
out mitigates the risk significantly.  If there are test cases that are 
missing, I'll be glad to make sure they are documented and added (spf-discuss 
is the best place for that discussion I think).  We've been through an entire 
from scratch development cycle with http://james.apache.org/jspf/ - you'll see 
they even reference the test suite in their release announcements.

Unless their local reasons include idiocies like not using tests in 
development, I don't think there is much risk here nor does removing macros 
significantly simplify the work of implementing SPF from scratch.

Scott K

---

From: "John Levine" <[email-masked]>
Date: 13 Jul 2012 21:27:18 -0000

>that is believed to cover all the RFC 4408 requirements.

I took a look.  With respect to macros, it has one test called
dorky-sentinel with a space in a local part, although it's not obvious
to me what the correct answer is supposed to be, one called
upper-macro that tests that upper cased macros do percent encoding,
and invalid-domain-long-via-macro which checks for a macro expansion
that creates an excessively long domain name.  There's also a bunch of
other tests that check that macros do what they're supposed to with
normal input.

Other than the one about the space, I don't see any tests for what
happens if the text being expanded contains unexpected hostile
characters.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 17:36:28 -0400

On Friday, July 13, 2012 09:27:18 PM John Levine wrote:
> >http://www.openspf.org/Test_Suite and get a copy of a comprehensive test
> >suite that is believed to cover all the RFC 4408 requirements.
> 

Should be easy enough to add.  What would you consider hostile?

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 13 Jul 2012 17:43:58 -0400

> Should be easy enough to add.  What would you consider hostile?

I ask, not just because of the test suite, macro question, but if there are 
potential characters that should be prohibited from SPF records generally due 
to security considerations, I think we should address it in the draft.

Scott K

---

From: "John Levine" <[email-masked]>
Date: 14 Jul 2012 01:02:59 -0000

>to security considerations, I think we should address it in the draft.

It's not characters in SPF records, it's characters in SMTP bounce addresses.
Try some records with %s and %l, and bounce addresses that contain 0x04, 0x1a,
0xa0 and other characters most DNS software doesn't expect.

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 14 Jul 2012 14:59:41 +0200

> 0xa0 and other characters most DNS software doesn't expect.

Many multibyte characters vulnerabilities were found and fixed during
the last decade, so I believe IDN software is robust and reliable
enough nowadays.  AFAICS, the one thing we have to take care for is
servers/implementations that cannot avail themselves of up-to-date
libraries.  Possibly, they should raise permerror as soon as they see
a signed byte, and mention they're not SMTPUTF8 in any reply or mark.

IMHO, using UTF-8 for both records and parameters makes for smoother
specs, implementations, and deployment.  That does not mean U-labels
are mandatory.  Until the number of non EAI-compliant servers becomes
irrelevant, publishers will find it convenient to convert domain names
to A-labels.  An annoyance that will be forgotten afterwards.

PS:  Sorry for having overlooked uppercase macros in a previous
message of mines.  Anyway, malicious exp's are a known issue.

---

From: Dave Crocker <[email-masked]>
Date: Sat, 14 Jul 2012 08:33:09 -0700

On 7/13/2012 2:27 PM, John Levine wrote:
> dorky-sentinel with a space in a local part,

Bringing this around to the other active thread...

Along with various contributions that innovated core technology -- eg, 
ip and tcp -- and styles of collaboration -- open collaboration -- the 
Internet's history demonstrated the essential benefit of 
interoperability testings, specifically in contrast with bench testing 
against suites.

It's not that test suites are useless -- they can be extremely efficient 
at finding common problems.  However, they are insufficient for 
producing working product.  What produces working products is direct 
interaction with other implementations.

When code is deployed that is not used, it does not receive this kind of 
testing.  Ever.  The extremely low level of deployment macros have 
received after this long make clear that such code is unlikely to get 
that testing.

This makes the code a kind of time-bomb, waiting  possibly for years, 
for some unusual situation which /does/ invoke it and demonstrate that 
it does not work properly.

and returning to the Subject thread:  this is also why it is advisable 
to delay adopting brand new component standards into other standards, 
until there is more field experience.  That field experience provides 
the interoperability testing and development of operational knowledge 
about the new component.

d/

---

From: Dave Crocker <[email-masked]>
Date: Sat, 14 Jul 2012 08:34:41 -0700

> statement of fact, right?

It's the perspective/definition I've been operating with for a couple of 
decades, at least.  Possible 3 or 4.  And I didn't create it.

d/

---

From: "John Levine" <[email-masked]>
Date: 14 Jul 2012 17:51:55 -0000

>Many multibyte characters vulnerabilities ...

This issue has nothing whatsoever to do with multibyte characters.

R's,
John

---

From: S Moonesamy <[email-masked]>
Date: Sat, 14 Jul 2012 11:29:49 -0700

Hi Dave,
At 08:33 14-07-2012, Dave Crocker wrote:
>It's not that test suites are useless -- they can be extremely 
>efficient at finding common problems.  However, they are 
>insufficient for producing working product.  What produces working 
>products is direct interaction with other implementations.

Yes.

>This makes the code a kind of time-bomb, waiting  possibly for 
>years, for some unusual situation which /does/ invoke it and 
>demonstrate that it does not work properly.

That was one of the arguments which was taken into consideration to 
encourage discussion about some of the issues.  Even if all the code 
paths are exercised that unusual situation can still happen.  It 
helps in the decision-making if it can be demonstrated that X does 
not work properly.  Having input from several WG participants is also 
helpful.  That's why WG participants are asked to weigh in on an issue.

Regards,
S. Moonesamy

---

From: Douglas Otis <[email-masked]>
Date: Sat, 14 Jul 2012 21:56:37 -0700

On 7/13/12 2:36 PM, Scott Kitterman wrote:
>>> http://www.openspf.org/Test_Suite and get a copy of a
>>> comprehensive test suite that is believed to cover all the RFC
>>> 4408 requirements.
>> 
>> dorky-sentinel with a space in a local part, although it's not
>> obvious to me what the correct answer is supposed to be, one
>> called upper-macro that tests that upper cased macros do percent
>> encoding, and invalid-domain-long-via-macro which checks for a
>> macro expansion that creates an excessively long domain name.
>> There's also a bunch of other tests that check that macros do what
>> they're supposed to with normal input.
>> 

Dear Scott,

Since SMTPUTF8 permits UTF-8 mail-parameters that may affect either the
local-part or the domain of email-addresses, it seems unlikely adequate
test coverage will be found based on older specifications that expected
mail parameters were limited to US-ASCII.  This is no longer an SMTP
constraint based upon the standards track RFC6530 extending both RFC5321
and RFC5322.

Use of SPF records to filter local-part values independent of the domain
is an expensive operation for receivers that may negatively impact their
DNS cache or other third-parties whenever a component of the local-part
constructs domains that may not appear in the email-address or anywhere
in the email.  A static local-part minimizes the impact of macro
expansions, especially since no limit is imposed on the number of
no-domain errors that might be generated.

A simple macro expansion of a local-part label can direct an unlimited
number of DNS queries against any third-party domain derived from a
single cached SPF record.  Simple deprecation of the local-part macro
expansion guards against a potential form of reflected and amplified
abuse where a pass result is possible subsequent to as many as 100
random sub-domain queries against any uninvolved third-party that may
not be evident anywhere in the email.

Page 9:

Was:

   Implementations must take care to correctly extract the <domain> from
   the data given with the SMTP MAIL FROM command as many MTAs will
   still accept such things as source routes (see [RFC5321], Appendix
   C), the %-hack (see [RFC1123]), and bang paths (see [RFC1983]).
   These archaic features have been maliciously used to bypass security
   systems.

Change to:

The EHLO/HELO parameter must use A-Label encoding.  However that may
not be true for the mail parameter.  Implementations must take care to
correctly extract the <domain> from the data given with the SMTP MAIL
command as many MTAs will still accept such things as source routes (see
[RFC5321], Appendix C), the %-hack (see [RFC1123]), and bang paths (see
[RFC1983]). These archaic features have been maliciously used to bypass
security systems.

There should be some combination of routines available to ensure UTF-8
parameters do not use illegal code-points.  One method might use
UTF-8(a) -> A-Label(a) conversion strategies that may not include
assurances against illegal code-points.  Confirmation that A-Label(a) ->
UTF-8(b) provides the same string as UTF-8(a) should offer assurances
against illegal code points without any additional routines.
Elimination of illegal code-points is required to guard against visually
deceptive strings.

The domain portion of a UTF-8 encoded email address should be converted
to an A-Label form using a method that ensures against illegal
code-points, only the A-Label version of the domain should be passed to
the SPF library.  Otherwise, results will be undefined.

Dealing with the local-part of a UTF-8 encoded email-address is
problematic.  One solution could replace the local-part with the value
"postmaster" defined in Section 2.2 and 4.3.  If there is a library in
use that has not deprecated the operation of the '%l' macro, a simple
string replacement still offers predictable results.  Ideally, any
record that includes '%l' macro should return the result "none".

Page 10:

Was:
   A result of "none" means that no records were published by the domain
   or that no checkable sender domain could be determined from the given
   identity.  The checking software cannot ascertain whether or not the
   client host is authorized.

Change to:
   A result of "none" means that no records were published by the domain
   or that no checkable sender domain could be determined from the given
   identity.  The checking software cannot ascertain whether or not the
   client host is authorized.  Use of a deprecated local-part macro
   should return the result of "none" or alternatively checked
   against the identity where local-part is replaced by the string
   "postmaster".

Page 27:

Was:
   Note: In general, the domain "A" cannot reliably use a redirect to
   another domain "B" not under the same administrative control.  Since
   the <sender> stays the same, there is no guarantee that the record at
   domain "B" will correctly work for mailboxes in domain "A",
   especially if domain "B" uses mechanisms involving localparts.  An
   "include" directive is generally be more appropriate.

Change to:
   Note: In general, the domain "A" cannot reliably use a redirect to
   another domain "B" not under the same administrative control.  Since
   the <sender> stays the same, there is no guarantee that the record at
   domain "B" will correctly work for mailboxes in domain "A".  An
   "include" directive is generally more appropriate.

Page 31:

Was:
l = local-part of <sender>

Change to:
Deprecated l = local-part of <sender>

Page 32:

Was:
The "l" macro expands to just the localpart.

Change to:
The "l" macro is deprecated and may not expand to components of the
local-part.

Page 37:

Was:
       2.  The "MAIL FROM" identity could have additional information in
           the localpart that cryptographically identifies the mail as
           coming from an authorized source.  In this case, such an SPF
           record could be used:

              "v=spf1 mx exists:%{l}._spf_verify.%{d} -all"

           Then, a specialized DNS server can be set up to serve the
           _spf_verify subdomain that validates the localpart.  Although
           this requires an extra DNS lookup, this happens only when the
           email would otherwise be rejected as not coming from a known
           good source.
           Note that due to the 63-character limit for domain labels,
           this approach only works reliably if the localpart signature
           scheme is guaranteed either to only produce localparts with a
           maximum of 63 characters or to gracefully handle truncated
           localparts.

Change to:

	Use of DNS resource records to individually validate an entire
population of a domain's users is detrimental to the operation of DNS,
and inappropriately burdens receiving domains.  Domain-wide
verifications should be used instead, such as DKIM for example.

Regards,
Douglas Otis

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 16 Jul 2012 06:58:06 -0400

Hat on.

> " this specification says otherwise.

We have positive evidence, in the form of published records with
local-part macros in them, that people are using this feature or at
least think they are.

Given that is the case, it seems to me that we have a process issue
here.  I'd like those who support deprecation or removal of the
feature to explain how their proposed action is consistent with the
charter we have.  Dave Crocker has already offered his view, which is
(to oversimplify) that "unused" has a meaning slightly different from
its plain English meaning, and that local-part macros can therefore be
removed.  What about the case for deprecation?

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 09:34:22 -0400

> removed.  What about the case for deprecation?

I think I started the idea of deprecation with PTR, so I'll explain my 
thinking and how I think it's consistent with the charter.

PTR is unnecessary.  If we'd known better in 2005, I think we'd have dropped 
it.  There are, as far as I'm aware, always better ways to construct an SPF 
record that don't use PTR.  It's slow and unreliable for a variety of reasons 
I won't repeat here.

Consistent with the charter, we cannot, however much I would like to, remove 
it now.  It is in broad use and no matter who's definition of in use one 
selects, it's in use.

To me 'deprecated' means that for now it's still an official part of the 
protocol, but you should migrate away from it because it might be removed in 
the future.  It's a way to stay compatible with existing records and the 
charter, but provide a path to eventual removal (no idea if that would ever 
actually happen without bumping the SPF version number and no need to decide 
now).

Given all that, I think making something deprecated is consistent with the 
charter.  

So far, in my opinion, we've identified some slight theoretical disadvantages 
for macros in general, slightly more for the ones that can't be cached, and 
yet again slightly more for the local-part macro.  I don't see a good case for 
deprecating these, but I think it's within the working group's purview to make 
that decision.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 09:42:23 -0400

> verifications should be used instead, such as DKIM for example.

You are using deprecated in a way different than the one I've defined in the 
current draft.  My use of deprecated is "works as before, but stop using, it 
may go away in the future".  Yours is "ignore it, but don't raise an error."

The former is backwards compatible and consistent with the charter.  The 
latter is not on either point.  

Last I checked, the chairs had ruled removing macros out of scope, so I'm 
going to hold your suggestions for text until that discussion plays out.

Scott K

---

From: John Leslie <[email-masked]>
Date: Mon, 16 Jul 2012 09:56:07 -0400

> removed.  What about the case for deprecation?

   Simply, "deprecate" is not "remove". From the charter:
" 
" Specifically out-of-scope for this working group:
" 
" * Revisiting past technical arguments where consensus was reached in
"   the MARID working group, except where review is reasonably
"   warranted based on operational experience.
" 
" * Discussion of the merits of SPF.
" 
" * Discussion of the merits of Sender-ID in preference to SPF.
" 
" * Extensions to the SPF protocols.
" 
" * Removal of existing features that are in current use.
" 
" Discussion of extensions to the SPF protocols or removal of
" existing features shall only be discussed after completion of
" current charter items in anticipation of rechartering the working
" group.

   The definition of "deprecate" quite clearly states that the feature
is not being removed (even if we threaten to do so in the future).
We only say "SHOULD not publish," understanding that there may be
valid reasons to publish and still claim compliance with this Proposed
Standard.

   Furthermore, in defense of Dave (who certainly doesn't _need_ me
defending him!):

- "in current use" is not an exact antonym of "unused"; something may
  be a non-current use and not run afoul of this charter language;

- we can never prove the absence of any use whatsoever, so there must
  be some judgment involved: judging the existence of "current use"
  (at the time the charter was written) requires more effort than we
  should reasonably expend.

   Besides, "deprecate" is not "remove"...

> Best,
> 
> A
> 
> -- 
> Andrew Sullivan
> [email-masked]
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis

---

From: Dave Crocker <[email-masked]>
Date: Mon, 16 Jul 2012 06:59:47 -0700

On 7/16/2012 6:42 AM, Scott Kitterman wrote:
>   My use of deprecated is "works as before, but stop using, it

I believe your intent with 'deprecated' is roughly the same as what was 
done for RFC 2822 (now RFC 5322) which used the term "obsolete".  So 
it's worth looking at the way it dealt with the matter.  (The difference 
in label isn't interesting to me.  Looking at the meaning is.)

The text about this in RFC 5322 is:

> 3.1. Introduction
...
>    In some of the definitions, there will be non-terminals whose names
>    start with "obs-".  These "obs-" elements refer to tokens defined in
>    the obsolete syntax in section 4.  In all cases, these productions
>    are to be ignored for the purposes of generating legal Internet
>    messages and MUST NOT be used as part of such a message.  However,
>    when interpreting messages, these tokens MUST be honored as part of
>    the legal syntax.  In this sense, section 3 defines a grammar for the
>    generation of messages, with "obs-" elements that are to be ignored,
>    while section 4 adds grammar for the interpretation of messages.

I read this as meaning:  If you are conforming to the latest 
specification, you must not /generate/ data using these constructs. 
However you will still receive and correctly process data that was 
generated using these constructs.

I believe the language in RFC 5322 is compatible with the interpretation 
that Scott has for 'deprecated'.

d/

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 10:50:48 -0400

> that Scott has for 'deprecated'.

It's very close.  I think the difference is that my definition is SHOULD NOT and 
that one is MUST NOT.  So far the only thing that's marked 'deprecated' in -03 
is PTR and the related macro.  I think SHOULD NOT is the appropriate thing to 
say.  I think MUST NOT is too strong.  

Like you, I'm not very interested in the label and don't mind changing it if 
something else is better.  I originally picked deprecated because it seemed to 
me to be a widely used term that programmers would generally understand.  It's 
perhaps more important that DNS administrators understand it, so I'm open to 
suggestions.

Scott K

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 16 Jul 2012 17:22:03 +0200

On Mon 16/Jul/2012 15:56:07 +0200 John Leslie wrote:
>> I'd like those who support deprecation or removal of the feature
>> to explain how their proposed action is consistent with the 
>> charter we have.
> 
> " [...]

I happened to write both in favor and against removal of local macros:
http://www.ietf.org/mail-archive/web/spfbis/current/msg00184.html
http://www.ietf.org/mail-archive/web/spfbis/current/msg01821.html

Let me please try and clarify my thought.  We already have the
following text in the draft:

  Note: Domains should avoid using the "s", "l", "o", or "h" macros in
  conjunction with any mechanism directive.

I'd be fine with s/should/SHOULD/.

IMHO, we won't get more than that by adding deprecation.  But we might
get less:  By deprecating all those macros rather than just %{p}, we
might dilute deprecation itself, and risk that the next WG will just
confirm such status without actually removing the unwanted features.

jm2c

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Jul 2012 13:02:05 -0400

> say.  I think MUST NOT is too strong.  

PTR is deprecated?

Sorry for not being able to provide the time to focus of the WG 
details of whats been done here, but this command is highly used and 
it is very useful.

It serves a very useful purpose for IPs that have multiple A records 
and SMTP software that use the first round robin A record for a HELO. 
  There are plenty of SMTP software that do this.  I submit this is 
the case with the much higher volume of smaller sites, businesses and 
operations that have limited IPs but plenty of subdomains for IPs.

-1 to DEPRECATING this feature.

Can you remind us (or just me) of the reasoning behind this move?  I'm 
certainly not to deprecate PTR in our implementation. In a quick scan 
of DNS cache, I see at least 20-25% of the records using PTR macros.

If deprecating is the consensus, no doubt, SHOULD NOT is the ONLY 
option.  I prefer that would be more of guideline than any sort of 
strong 2119 encouragement where it may make a replacement SPF 
statement more complex, longer and possibly impracticable to be 
dynamic (a new SPF statement update is needed for every new subdomain 
added to the single IP).  PTR allows the SPF statement to be generic 
for systems that have multiple A records and the SMTP software is 
randomly and/or in round robin fashion using the first host address it 
gets.

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 13:09:06 -0400

On Monday, July 16, 2012 01:02:05 PM Hector Santos wrote:
> Scott Kitterman wrote:
> > It's very close.  I think the difference is that my definition is SHOULD
> > NOT and that one is MUST NOT.  So far the only thing that's marked
> > 'deprecated' in -03 is PTR and the related macro.  I think SHOULD NOT is
> > the appropriate thing to say.  I think MUST NOT is too strong.
> 
> PTR is deprecated?
> 

I have yet to run into a situation that can't be better expressed in other 
ways.  I think both the list archives and the draft are pretty clear about why 
we did it.  

Scott K

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Jul 2012 13:27:52 -0400

> Can you remind us (or just me) of the reasoning behind this move (PTR Deprecation)?  
> I'm  certainly not to deprecate PTR in our implementation. In a quick scan of 
> DNS cache, I see at least 20-25% of the records using PTR macros.

16.5% in this quick cache to be more snapshot precise are using PTR in 
their SPF statement.

I suggest we have a higher occurrence of a zero PTR overhead for SPF 
specifically when increasingly at the SMTP receiver level more and 
more systems are performing PTR lookups as a SMTP policy.

So any reasoning that a PTR lookup generally a higher overhead DNS 
expense lookup, is a problem for SPF is a weaker reason today by the 
time the SPF query is made. Assuming the SMTP and SPF software are 
using the same DNS query/cache, any SPF PTR lookup would be in the 
cache .  We had a major update two years ago to make sure all DNS API 
related components in our total Internet Hosting framework 
(telnet/rlogin, nntp, ftp, web, smtp, pop, etc) used a common DNS 
API/Cache to reduce the redundancy and overhead with the new 
associated DNS security related protocols.

-- 
HLS

Hector Santos wrote:
> Scott Kitterman wrote:
>>
>> It's very close.  I think the difference is that my definition is 
>> SHOULD NOT and that one is MUST NOT.  So far the only thing that's 
>> marked 'deprecated' in -03 is PTR and the related macro.  I think 
>> SHOULD NOT is the appropriate thing to say.  I think MUST NOT is too 
>> strong. 
> 
> PTR is deprecated?
> 
> Sorry for not being able to provide the time to focus of the WG details 
> of whats been done here, but this command is highly used and it is very 
> useful.
> 
> It serves a very useful purpose for IPs that have multiple A records and 
> SMTP software that use the first round robin A record for a HELO.  There 
> are plenty of SMTP software that do this.  I submit this is the case 
> with the much higher volume of smaller sites, businesses and operations 
> that have limited IPs but plenty of subdomains for IPs.
> 
> certainly not to deprecate PTR in our implementation. In a quick scan of 
> DNS cache, I see at least 20-25% of the records using PTR macros.
> 
> strong 2119 encouragement where it may make a replacement SPF statement 
> more complex, longer and possibly impracticable to be dynamic (a new SPF 
> statement update is needed for every new subdomain added to the single 
> IP).  PTR allows the SPF statement to be generic for systems that have 
> multiple A records and the SMTP software is randomly and/or in round 
> robin fashion using the first host address it gets.
> 

-- 
HLS

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Jul 2012 13:38:39 -0400

> ways.  I think both the list archives and the draft are pretty clear about why 

Really?

    Note: This mechanism has been deprecated because it is slow, it is
    not as reliable as other mechanisms in cases of DNS errors, and it
    places a large burden on the arpa name servers.  If used, proper PTR
    records must be in place for the domain's hosts and the "ptr"
    mechanism should be one of the last mechanisms checked.  After many
    yaers of SPF deployment experience it has been concluded it is
    unnecessary and more reliable alternatives used instead.

This hurt smaller, small business and operations systems who are 
operating with limited IPs and and relying on virtual domains hosting.

In addition, the overhead (and "slowness") is already performed at the 
SMTP level, and I submit increasingly so, especially with the larger 
ESP, such as AOL and others that now require a PTR record to be 
available at the connection level.  In other words, by the time any 
in-sessionS SMTP sub-process does any SPF lookup, the PTR records will 
already be in the DNS cache - zero overhead.

This is not a good reason to deprecate it just like it never was a 
good reason to get rid of it (PTR in general) long ago in the first 
place, especially in the past years where queries did have a higher 
network impact with lower bandwidths.

There is more harm done with this change.

Besides, I don't recall any voting done for this.

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 13:40:16 -0400

On Monday, July 16, 2012 01:38:39 PM Hector Santos wrote:
...
...

Probably because the IETF doesn't vote on things.  ;-)

Scott K

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Jul 2012 13:57:55 -0400

> Probably because the IETF doesn't vote on things.  ;-)

So does this means the change was made with no or limited input and 
practical concerns of others?

This PTR deprecation will made the statements more complex, longer to 
add every possible A/PTR matching sequence - very impractical.  The 
fact there are already a significant amount of usage will force people 
into a change mode required to avoid increasing errors where it does 
not exist today for them.

I think what was lost this decision is that many SMTP software use the 
first host domain found for a HELO/EHLO which is one of the reasons 
performing HELO validations in general is not reliable - this 
configuration history has been the case for a long time.

But if a SPF PTR policy is made that says the IP much match one of the 
A records, this a strong policy and very easy, simpler SPF policy to 
have for many sites that have long been in the less than ideal 
configuration history mode.

Now this change is telling people, they must fix their entire SMTP 
setups (which shouldn't be expected to happen then and now) and now 
they are being forced to increase the sizes of their SPF records with 
all the A sub/domain names, and when any new A record is added to the 
IP, they must change their SPF records again.

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 14:21:44 -0400

> practical concerns of others?

No.  It means the IETF runs on rough consensus which is not getting 51% of the 
people in the room to agree with you.  It's been discussed and as far as I'm 
recall, you're the only person to speak against the change.

> IP, they must change their SPF records again.

Within the last month, I've helped people fix things that turned out to be 
nothing to do with SPF, but bad rDNS setups.  It's an unreliable feature and 
not necessary.  Note that deprecated doesn't mean MUST NOT use.  It means 
SHOULD NOT.  If there is such a thing as a setup that would be hard to 
transition, then there will be lots of time to do so.  PTR as a mechanism is 
not going anywhere in the short term.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 16 Jul 2012 13:01:05 -0700

> removed.  What about the case for deprecation?

I concur with Dave, and the reasoning behind his position.  Given the data
available, both PTR and macros introduce complexity but, in the six years
since SPF was first published as an Experimental RFC, have enjoyed
approximately zero adoption (but not zero).  They don't add enough utility
to justify their added complexity.

It's already been decided that PTR is to be deprecated, and our definition
of "deprecated" here seems to be approximately what we want.  Based on
statistics gathered during our previous work, PTR and macros are in the
same ballpark in terms of adoption.  I believe we are being inconsistent if
we don't apply the same logic to both of them.

Since we are going out with the first version of SPF on the standards
track, I would prefer not to include large chunks of text that have to do
with things people aren't really using or things we don't want them to use
anymore.  This thing should be as lean as possible.

Other working groups have gone through this process and used "unused" in a
very-low-but-not-zero way.  I cited DKIM as an example.  I realize we're
not bound by the process defined in RFC6410 because SPF is advancing to,
but not along, the Standards Track.  Rather, we're bound by our charter.  I
can say that when I crafted our charter, I had in mind the use of "unused"
that I got from the previous work, i.e., almost-but-not-zero qualifies as
"unused".  I suppose it's a matter of conversation as to whether the IESG
had the same assumption when they accepted it, since the charter is
basically our operating agreement with them.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 16 Jul 2012 16:10:40 -0400

>basically our operating agreement with them.

Except being unused was not part of the rationale for deprecating PTR. The rationale was that it didn't work well, was unreliable, and was unnecessary.  Unused didn't figure into it at all.  There's no inconsistency here.

I think it's within the charter to deprecate (but not remove) anything we conclude is technically a bad idea, used or not.

Scott K

---

From: Dave Crocker <[email-masked]>
Date: Mon, 16 Jul 2012 13:29:10 -0700

> I think it's within the charter to deprecate (but not remove) anything we conclude is technically a bad idea, used or not.

As I recall, we also saw very poor statistics of use for PTR and that 
that /did/ factor into the group's decision process.

In any event, the charter authorizes removal of 'unused' feature, module 
agreeing on the meaning of unused...

The distinction between deprecating versus removing can be pursued 
separately, IMO.  It's important but independent.

d/

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 16 Jul 2012 13:31:10 -0700

On Mon, Jul 16, 2012 at 1:10 PM, Scott Kitterman <[email-masked]> wrote:

>
> Except being unused was not part of the rationale for deprecating PTR. The
> rationale was that it didn't work well, was unreliable, and was
> unnecessary.  Unused didn't figure into it at all.  There's no
> inconsistency here.
>

I think something that sees use by less than 0.2% of the user base
qualifies as "unnecessary" as well.  I don't have any data about the
reliability question.

-MSK

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Jul 2012 16:31:11 -0400

> to justify their added complexity.

Either we are mixing apples and oranges at this point, does my 16% 
snapshot usage of incoming SPF statements with PTR command still 
doesn't mean anything?

> of "deprecated" here seems to be approximately what we want.  

I would like to see when when exactly there was a clear group rough 
consensus, time frame and/or with a link decision to deprecate the 
PTR.  It must of been some off-list discussion because I don't recall 
such a public decision.

What I do recall was a wide agreement that there shouldn't be a 
security problem because of the limits already written and used in 
practice thus why there was never a problem before.

> anymore.  This thing should be as lean as possible.

But PTR is being used and with proper usages, understanding, 
controlled and and no harm. Where in your deployment information does 
it show otherwise?

> basically our operating agreement with them.

Well, things are know to fall thru the crack and this appears to be 
one of them. You can't lump such generality like this to pull the rug 
from deployment feet.  This cause time and money and when they don't 
see a good reason to listen, they won't.  Do you honestly believe SPF 
receivers are going to stop supporting features that COULD OCCUR even 
thing in your mind believe it is "unused?"   Even if you believe its 
1% usage, today there is 0% ERROR because of it.  Deprecation means at 
least 1% ERROR rate and a presumption everyone will follow the NEW SPF 
and thus go down to zero.  How long do you expect that to happen?

No matter what, the so called "complexity" still need to be coded in 
order to support any presumed legacy of PTR usage.

-- 
HLS

---

From: S Moonesamy <[email-masked]>
Date: Mon, 16 Jul 2012 17:41:08 -0700

At 13:31 16-07-2012, Hector Santos wrote:
>PTR.  It must of been some off-list discussion because I don't 
>recall such a public decision.

On March 7, Arthur Thisell raised an issue about deprecating the 
"ptr: mechanism and the %{p} macro" (Issue #27).

The following is from the minutes for the SPFBIS session at IETF83:

    Issue #27 is about the PTR mechanism and %{p} macro. There was support on
    the mailing list for adding "SHOULD NOT" text about this. There was support
    for "SHOULD NOT" from the Jabber room. The issue was closed pending the new
    document.

I posted a message to the WG mailing list about the minutes ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg01482.html ).

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: "John Levine" <[email-masked]>
Date: 17 Jul 2012 05:09:10 -0000

>feature to explain how their proposed action is consistent with the

As far as I can tell, localpart macros are no more popular than type
99 records.  In both cases, we've found some people publishing them,
but it's down in the noise.

Also, in both cases, the current spec is broken.  For localpart
macros, the problem is that localparts can contain any character from
%d32 (space) through %d126 (tilde), which includes a lot of characters
that aren't valid in host names, don't normally occur in DNS names
(particularly given the provisioning problems in DNS names that
contain characters like spaces and backslashes), and in the exp:
context, can be syntactically significant in URLs such as / & ? and %.
While there's an option to percent-encode localparts, it's just an
option.  EAI will make this much worse, but it's broken now without
EAI.

While this is fixable, given how little used it is, it makes more
sense to deprecate it than to waste time trying to harden it.  That's
particularly true because hardening would probably require treating
syntactically interesting characters differently, and that opens a
whole can of worms of backward compatibility with largely non-existent
current practice.

I realize that nobody's reported problems with hostile characters
blowing up localpart macros, but I think it's likely that's because so
few people use them that the code has never been stressed, not because
it's robust.  I suppose I can concoct some SPF records with localpart
expansions, start firing mail with funky localparts at Scott and other
people who use SPF, and see what happens.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 17 Jul 2012 11:36:11 -0400

> people who use SPF, and see what happens.

I don't have a strong objection to deprecation.  I do object to removal.

One factor that is different than PTR and Type 99 though is that we (or at 
least I) concluded that neither of those features were needed to support any 
unique functional requirements.  You could remove/deprecate them and not affect 
the ability of ADMDs to publish SPF records that accomplished everything they 
did with records based on RFC 4408.

Removal/deprecation of macros does impact at least one use case that's difficult 
to replicate and there might be others.  I don't have personal knowledge of 
how or why publishers actually use macros in the wild except for the logging 
record suggested in RFC 4408 section 9.1:

Trying to figure out SPF deployments is hard.  The above use of macros is one 
of the few reliable ways to get feedback on where receivers think your mail is 
coming from (yes, we have RFC 6652 and perhaps DMARC at some point in the 
future, but neither of those mechanisms have the deployment to support being a 
full replacement for this use of macros).

Deprecation does not immediately harm ADMDs making use of macros and, if it 
turns out there's a need in the future, could conceivably be reversed.  Before 
going beyond that point (i.e. removal) we need to understand what unique use 
cases the small fraction of ADMDs that use macros are using them for to see if 
we are dropping functionality that leaves valid (if narrow) use cases 
unaddressed.

Scott K

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 17 Jul 2012 12:30:25 -0400

Still hatted.  Maybe it's cooking my brain, but . . .

On Tue, Jul 17, 2012 at 11:36:11AM -0400, Scott Kitterman wrote:
> to replicate and there might be others.

the above the "process issue" I was referring to.  The argument is
that we can remove local part macros because we have scanty evidence
of their use; but as Scott is pointing out, they provide a piece of
functionality otherwise unavailable, and we have detected their use
(at least, their publication).

Moreover, _all_ our evidence about SPF is scanty: we didn't do the
kind of widespread investigation that would allow us to make highly
defensible strong statements about SPF.  That seemed ok to me as the
basis for experiment conclusion, given the way our charter constrained
us.  If we're now going to use that evidence as a reason to remove a
capability (one that I personally dislike), then I worry we are
exceeding our permitted activities.  If people are really going to
push this, then I want to have a discussion with my co-chair and our
AD about the matter.

One might well argue that we _should_ be free to remove this feature,
both because it is little used and because it is dangerous.  (On the
latter, I still am mulling over John Levine's argument.)  That isn't
the same thing as saying that our charter permits us that freedom.  So
far, I haven't convinced myself.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Dave Crocker <[email-masked]>
Date: Tue, 17 Jul 2012 09:43:19 -0700

> I don't have a strong objection to deprecation.  I do object to removal.

fwiw, I'm fine with deprecating rather than removal.  I'd prefer 
removal, but would consider it progress to mark the feature deprecated.

if/when the group does choose deprecated, I think we'll need a separate 
thread to get consensus on its precise meaning.  (I would prefer that we 
/not/ debate the issue right now.)

d/

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 17 Jul 2012 12:58:15 -0400

> /not/ debate the issue right now.)

Unless we know what deprecated means, we can hardly make a decision to do it.  
I think the definition needs to come first.  The current -03 draft (I have a 
comment from msk to clean this up a bit, but it doesn't affect the meaning) is:

1.3.4.  Deprecated

   There are several [RFC4408] features that are marked "deprecated".
   In the context of this document, deprecated means that senders SHOULD
   NOT publish SPF records that make use of the feature in question.
   Its use is NOT RECOMMENDED and it might be removed entirely in future
   updates to the protocol.  Such features do, however, remain part of
   the SPF protocol and receiving systems MUST support them unless this
   specification says otherwise.

As I mentioned earlier, this is different than the 5321 definition of obsolete 
primarily by virtue of being a SHOULD NOT vice a MUST NOT.

Since PTR and the "p" macro are already marked deprecated based on previous WG 
discussion, if this isn't the definition we want, I think now is the time to 
find out.

Scott K

---

From: Dotzero <[email-masked]>
Date: Tue, 17 Jul 2012 13:42:20 -0400

> far, I haven't convinced myself.

I lean towards deprecation. I would have leaned that way even before
John Levines argument but that just strengthens my position.

Even if some small protion of the implementation universe feels there
is value, I do believe that as we move from Experimental to Standards
Track it is important for us to consider how local part macros fir in
with the overall goal and whether we expect additional adoption beyond
the small core of advocates, many of whom have been SPF activists
since day 1 (MARID). I don't take this position lightly and in fact,
going into the SPFbis process I was willing to leave well enough alone
simply to see things move forward. I'm now convinced that it is
appropriate to consider the future of local part macros and whether to
deprecate or even remove them from the standard.

Mike

---

From: S Moonesamy <[email-masked]>
Date: Tue, 17 Jul 2012 10:59:39 -0700

>but it's down in the noise.

I'll pick the above as a starting point.  The WG Chair asked the 
following question [1]:

    "What about the case for deprecation?"

In my view the issue of SPF RR Type and TXT RR was about choosing one 
of the two RRs.  The WG Chair previously mentioned that:

    "The were people on the mailing list in favor of using the TXT RR
     appeared to be a modest majority and there were people who argued
     for SPF RR Type on the grounds of DNS hygiene. The Chair explained
     that it appears as an empirical matter, overwhelmingly, the TXT RR
     is used but RRTYPE 99 does not qualify under the unused provision
     of the SPFBIS charter."

There was also Issue #14 and the following was mentioned:

    'Nobody argued that the working group must deprecate local-part macros.
     The Chair closed the issue with "won't fix". The Chair will reconsider
     the issue if there is text to go in the Security considerations section
     of the document. John Levine pointed out that as there is only one person
     in favor of the argument, it should be closed.'

    'Pete Resnick, as Area Director, explained what he understood from the
     discussion of Issue #14: the working group sufficiently considered the
     issue and agreed not to deprecate the feature; it can add clarification
     in the Security Considerations section.'

I'll quote John Leslie [2] out of context:

   'Besides, "deprecate" is not "remove"...'

I would not use that in an explanation if the working group is asked 
about the conclusion of the "local-part" issue.  Scott Kitterman 
mentioned that he does not have a strong objection to deprecation 
[3].  John Levine mentioned that "the current spec is broken" 
[4].  That argument probably came up in the arguments about Issue 
#14.  There was agreement previously not to deprecate the 
feature.  There is new information available.

Even if there isn't any objection against deprecation I would still 
like to see comments from WG participants about the case for 
deprecation.  If you have not commented yet please do.

Regards,
S. Moonesamy

1. http://www.ietf.org/mail-archive/web/spfbis/current/msg01845.html
2. http://www.ietf.org/mail-archive/web/spfbis/current/msg01848.html
3. http://www.ietf.org/mail-archive/web/spfbis/current/msg01871.html
4. http://www.ietf.org/mail-archive/web/spfbis/current/msg01870.html

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Jul 2012 14:42:47 -0400

SM, thanks for your follow up:

IMV, there are apples and oranges being mixed up in this SPF WG 
thread.  I am wondering if I should bother to bring that up but I 
don't think it is very clear what is exactly wanted and seems to be 
going going over board by lumping it all together - PTR directive and 
%p macro.

My recollection in the IETF83 Jabber session was that it wasn't cut 
and dry and Andrew wanted to get list input to make the case 
for/against.  I don't recall seeing many inputs here, if only Doug 
Otis I seem to recall.

My position was always that SPF receivers can not practically remove 
support (for backward compatibility reasons) and specifically for PTR. 
Macros such as %p I already felt was more unnecessary, has less 
utility (IMV) and sure, it is hardly used.

But PTR itself is very popular and perhaps I was too naive to believe 
anyone would take it serious to remove a high utility item instantly 
creating errors for SPF sessions.  No matter what, SPF receivers MUST 
support it.  Recommending Publishers not use it is another thing, so 
perhaps this is more about a BCP.

PTR appears to me to be among the highest use directive among the 
cache I see and we should not be surprise because the WIZARDS out 
there makes it part of the easy setup defaults.   By far, the #1 SPF 
statement (among my cache) is:

     v=spf1 a mx ptr ~all

Now, it seems we are mixing apples and oranges here with inconsistent 
reasons for deprecation either PTR or %p or both:

   - Scott has a Slowness reason written in 03
   - I believe Doug cited security reasons, and now
   - Several has indicated "unused" reasons, and
   - Some indicated a complexity reason for new developers.

So I think there are different angles to how the features is viewed.

As I recall, the main reason against deprecation was that it was under 
control with limits as there was no security issue.

Nonetheless, with PTR itself, we will be losing a high utility in 
functionality and what it offers.  Using the PTR directive makes it 
dynamic for 1 to MANY IP::A records sites to use a simple SPF policy 
command like above created by a number of wizards.

-- 
HLS

S Moonesamy wrote:
> On March 7, Arthur Thisell raised an issue about deprecating the "ptr: 
> mechanism and the %{p} macro" (Issue #27).
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Jul 2012 11:48:35 -0700

On Tue, Jul 17, 2012 at 9:43 AM, Dave Crocker <[email-masked]> wrote:

>
> fwiw, I'm fine with deprecating rather than removal.  I'd prefer removal,
> but would consider it progress to mark the feature deprecated.
>

I'd accept this compromise.  +1.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Jul 2012 12:11:26 -0700

On Tue, Jul 17, 2012 at 9:30 AM, Andrew Sullivan <[email-masked]>wrote:

> defensible strong statements about SPF.

I'm surprised by this characterization.  I think the data we collected
represents a reasonable cross-section of the deployment, is quite
comprehensive, and is fairly strong considering multiple independent
surveys were done.

Charter questions aside, let me ask this: What data would you consider
acceptable to justify deprecation or removal of a feature?  What's the bar
here?

It may be that we did collect the data, but simply didn't need to use it
for the questions asked to resolve the experiment issue.

> far, I haven't convinced myself.
One of the questions I'm asking myself here is: Should we go to PS with a
feature that is almost unused and is dangerous, even though it was there
before and a tiny number of people appear to be using it?  If I were
sitting on SECDIR, I'd make some noise about this during Last Call, for
example.  I'd prefer to go out without such concerns.  The counter-argument
I've heard is "It's only PS; when going to IS, the bar would be much
higher."  I'd rather not wait that long if it can be avoided.

If we can solve that problem under the "unused" clause in the charter (and
I'm of the opinion that it does fit), then I think that's a win.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 17 Jul 2012 15:21:45 -0400

> statement (among my cache) is:

Which is a brain dead recommendation.  It's not random that openspf.org no 
longer provides a wizard.  The old one produced awful garbage like this and no 
one wrote a new one yet.  The one thing that all SPF wizards (AFAIK) have in 
common is they aren't much good.

> command like above created by a number of wizards.

I have yet to run across a domain that couldn't write a correct/complete SPF 
record without PTR.  It's more than just slowness.  The PTR name of the IP 
address is very often not under control of the ADMD.  Errors are common.  I 
recently saw a case where the PTR lookups were getting SERVFAIL (the 
persistent kind) results and interfering with mail delivery.  It's slow, 
unreliable, and unnecessary.  Additionally, to the extent it is useful it's 
only useful for up to ten PTR records.  Then you're either stuck with random 
answers (match/not match based on if the relevant PTR name is in the first 10) 
or some other method.  That some other method works just fine if the number is 
one through nine as well.

That said, it's marked deprecated, not removed, so existing records won't get 
broken.  You're correct that it appears in too many records to simply remove 
it.

Scott K

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Jul 2012 16:02:32 -0400

Hi Scott,

I think the way it is written PTR is being deprecated which to some 
will mean it does not need to be coded and/or provides the excuse why 
it wasn't in the advent of an error dispute - finger pointing begins. 
  PTR as written in 03, it is parenthetically written as deprecated. 
That can provide the erroneous sense it is no longer necessary.

Just like you represented the OpenSPF.org wizard no longer provides 
the "brain dead" expert system answer to operator input, this is more 
of a publishing recommendation and thus the BIS should be written in 
that manner.

IOW, this is more about a BCP to not publish SPF policies with any 
higher overhead DNS query requirement, not just PTR.  It should be 
noted that increasingly more systems have SMTP frontend logic before 
SPF takes place which do place an PTR requirement on the client. This 
removes the "slowness" concern written in 03 in regards to SPF.

Overall, for this version of SPF, PTR is not deprecated in my view. It 
is just recommended not to be used (published) for possible DNS 
performance reasons.  Are there other higher overhead directives? If 
so, then if SLOWNESS is a concern for PTR, should they be lumped 
together in a SPF DNS Performance section?

Maybe complete removal can be reserved for SPF v3.0, but for SPF v2.0 
it needs to remain and the way its written now possibly helping the 
perception it is "unused" it can be removed via deprecation.

Thanks

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 17 Jul 2012 16:07:11 -0400

On Tuesday, July 17, 2012 04:02:32 PM Hector Santos wrote:
> > longer provides a wizard.  The old one produced awful garbage like this
> > and no one wrote a new one yet.  The one thing that all SPF wizards
> > (AFAIK) have in common is they aren't much good.
> > 
> > I have yet to run across a domain that couldn't write a correct/complete
> > SPF record without PTR.  It's more than just slowness.  The PTR name of
> > the IP address is very often not under control of the ADMD.  Errors are
> > common.  I recently saw a case where the PTR lookups were getting
> > SERVFAIL (the persistent kind) results and interfering with mail
> > delivery.  It's slow, unreliable, and unnecessary.  Additionally, to the
> > extent it is useful it's only useful for up to ten PTR records.  Then
> > you're either stuck with random answers (match/not match based on if the
> > relevant PTR name is in the first 10) or some other method.  That some
> > other method works just fine if the number is one through nine as well.
> > 
> > That said, it's marked deprecated, not removed, so existing records won't
> > get broken.  You're correct that it appears in too many records to simply
> > remove it.

Please read the definition of deprecated in -03 (that I mailed to the list 
earlier today).  I hope that will resolve your concerns.  Additionally, I 
emailed a diff from Alessandro yesterday that I plan for -04 to add additional 
discussion in Section 9 about the importance of minimizing the DNS cost of 
published records.

I do think that your concerns are either resolved with the -03 definition of 
deprecated or on their way to being resolved with what's planned for -04.

Scott K

---

From: S Moonesamy <[email-masked]>
Date: Tue, 17 Jul 2012 13:22:21 -0700

Hi Hector,
>for/against.  I don't recall seeing many inputs here, if only Doug 

I'll explain my view.  The minutes of the SPFBIS session at IETF83 is 
quite extensive.    I use the information in there to understand what 
was discussed and what conclusions were reached.  Andrew can ask for 
list input to make the case for or against.  If WG participants do 
not make the case for or against there is much the WG Chairs can 
do.  There are times when there is significant input and times when 
there are a few individuals discussing about an issue.  I prefer to 
see more input on the issues.

>support (for backward compatibility reasons) and specifically for 
>PTR. Macros such as %p I already felt was more unnecessary, has less 

I am not looking at that part of the specification as I write this 
message.  My reading of the SPFBIS Charter is that it restricts 
changes so that the working group does not end up writing a 
specification which is not compatible with the existing RFC.

>But PTR itself is very popular and perhaps I was too naive to 
>believe anyone would take it serious to remove a high utility item 
>instantly creating errors for SPF sessions.  No matter what, SPF 
>receivers MUST support it.  Recommending Publishers not use it is 
>another thing, so perhaps this is more about a BCP.

One of the individual comments I made about a different issue was 
that "deprecation can be done by clients not sending it but receivers 
supporting it, meaning that things will continue parsing the header 
for some time".  For what it is worth I was commenting about how that 
can be done, and not arguing for or against deprecation.   That gets 
you the "MUST support".

If you ask me whether it should be a BCP my answer is that I don't 
know as I have not thought about it yet.  If you ask me to think 
about it  I'll need information which has not been considered 
previously (see 
http://www.ietf.org/mail-archive/web/spfbis/current/msg01869.html as 
an example).

>Now, it seems we are mixing apples and oranges here with 
>inconsistent reasons for deprecation either PTR or %p or both:
>

Yes.  I'll point out that it is up to WG participants to explain the 
angles so that the working group ends up with a better view of the problem.

>As I recall, the main reason against deprecation was that it was 
>under control with limits as there was no security issue.

Some of the arguments are in the message at 
http://www.ietf.org/mail-archive/web/spfbis/current/msg00808.html

>command like above created by a number of wizards.

I suggest reviewing the text at 
http://tools.ietf.org/html/draft-ietf-spfbis-4408bis-03#section-5.5 
If you or anyone else has an objection to Section 5.5 I'll need 
information to determine why the conclusion is incorrect.

Regards,
S. Moonesamy

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 17 Jul 2012 16:27:38 -0400

> comprehensive, and is fairly strong considering multiple independent

The problem is that we started from domains that we knew or somehow
had contact with, and not a cross-section of "the Internet".  I don't
even know how you'd do a really defensible sample for this sort of
thing.   

We had enough anyway, IMO, to justify concluding the experiment and
following through on our charter.  I don't think we had enough to get
creative with our interpretation of "used" so that we can call things
unused when we have clear published examples.

> acceptable to justify deprecation or removal of a feature?  What's the bar

There is, as SM has already pointed out, a difference between
"deprecation" as we've styled it, and removal.  And given that I am
engaged here as co-chair, I don't feel comfortable asking the question
without considering the charter.  But people seem to be converging
around "deprecation", with some reluctance, and I'm certainly
comfortable with that approach.

> feature that is almost unused and is dangerous, even though it was there

The "dangerous" issue is one that I'm still thinking about.  Some of
what John Levine says upthread concerns me because I suspect it
applies to a number of other things.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 17 Jul 2012 16:35:36 -0400

> applies to a number of other things.

I think dangerous puts it backwards.  Some have stated a concern about safety 
due to a number of reasons such as potential lack of testing of macro related 
code.  Saying it may not be safe because of X, Y, and Z (some of which I think 
is a reasonable concern) is not the same as it's dangerous because of X, Y, 
and Z.  I think that given the charter's bias towards compatibility with RFC 
4408, there is a burden on those wanting to break backwards compatibility that 
goes well beyond "We think that in theory ....".

That said, deprecation doesn't break backward compatibility, so if that's what 
we converge on, it's a non-issue.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Jul 2012 14:30:01 -0700

On Tue, Jul 17, 2012 at 1:22 PM, S Moonesamy <[email-masked]> wrote:

> One of the individual comments I made about a different issue was that
> "deprecation can be done by clients not sending it but receivers supporting
> it, meaning that things will continue parsing the header for some time".
>  For what it is worth I was commenting about how that can be done, and not
> arguing for or against deprecation.   That gets you the "MUST support".
>

I'm a little confused by this, now that you mention it.  If the force of
this is that verifiers have to continue to support PTR and macros and
whatever else we formally decide to deprecate, I wonder what we've gained.
The verifier is where all of the real work is done and where the risk is
arguably highest.

The sender is advised to stop using them, but there's really no impact if
they don't.

What's the point?

Wouldn't it be better to say that verifiers SHOULD ignore deprecated
features, but if they don't, they should simply be ignored (rather than
returning "permerror")?

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Jul 2012 14:37:27 -0700

> even know how you'd do a really defensible sample for this sort of
That's true for the TDP survey (which queried domains with which any of its
reporting sites had contact).  The Cisco survey polled the top million
domains based on Alexa rankings, not domains with which it had contact.
Still not a true broad Internet survey given the number of domains that do
exist, but as you say it's not clear how to construct a 100% defensible
survey.

I wonder if people would be satisfied with a survey that can be shown to be
statistically significant yet not biased in any "bad" way.  The
absoluteness of some of the replies here have me doubting it, but
statistics is a well-established discipline.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 17 Jul 2012 19:17:19 -0400

"Murray S. Kucherawy" <[email-masked]> wrote:

>that

That would also be substantially backward incompatible.  As an example, it would make:

v=spf1 ptr -all

and

v=spf1 -all

into equivalent records. I don't think that's what we'd want.  FWIW, that first record isn't a theoretical one. That was the cisco.com SPF record for several years.

What deprecation achieves is a path towards removal without pulling the rug out from under people. I'd expect in a couple of years or so we'll be back to advance along the standards track. At that time, I'd expect the charter to cover removal of features that are either unused or were previously marked deprecated.

Scott K

---

From: Kurt Andersen <[email-masked]>
Date: Tue, 17 Jul 2012 16:25:06 -0700

Apologies if this distributes through the list twice - I've been
having fun getting my mail client to do the right thing...

--Kurt

On Tue, Jul 17, 2012 at 8:36 AM, Scott Kitterman <[email-masked]> wrote:
> coming from. . .

Speaking personally, this usage is exactly what the postmaster team at
a previous employer had in mind when we implemented SPF. We actually
used the %{l} macro to blacklist commonly spoofed localparts with a
-exists construct and then followed that with a "survey" entry that
would tell us who was spoofing our domain (by IP and localpart). With
over 300 domains under management, macro expansion was critical to
doing this in a manageable fashion. We received daily query reports
from our domain's authoritative name servers. Only "bad" mail hit
those "expensive" lookups, not the majority good mail that was
accepted with the initial ip4 entries.

Unfortunately, we ran into a problem much like the hypothetical cases
that have been in discussion here. A widely deployed perimeter email
appliance in China had a broken SPF implementation which choked on the
macro expansion of the SPF records and, since we could not get any
traction with the vendor, management elected to discontinue the use of
all SPF records rather than dealing with the frequent delivery
failures to a growing market segment.

Regretfully, I think that we should deprecate the %{l} macro, but
Murray's point that deprecation does not solve the problem of broken
(or flawed) implementations on checkers is a valid point. Deprecation
paves the way to eliminate it - but not for a few years or the next
"version" of the spec. In the meantime, everyone who implements SPF
checkers still bears the burden of doing it right - or we have to
define an alternate result for non-implemented quasi-optional elements
(different than PERMERROR).

Cheers,
  Kurt Andersen

---

From: S Moonesamy <[email-masked]>
Date: Tue, 17 Jul 2012 16:55:44 -0700

Hi Murray,
At 14:30 17-07-2012, Murray S. Kucherawy wrote:
>I'm a little confused by this, now that you mention it.  If the 
>force of this is that verifiers have to continue to support PTR and 
>macros and whatever else we formally decide to deprecate, I wonder 
>what we've gained.  The verifier is where all of the real work is 
>done and where the risk is arguably highest.

I was trying to explain "MUST support" in terms of a previous discussion.

>The sender is advised to stop using them, but there's really no 
>impact if they don't.
>
>What's the point?
>
>features, but if they don't, they should simply be ignored (rather 
>than returning "permerror")?

In terms of a protocol we can remove a feature or we can deprecate 
it.  Let's say that we decide to make "ptr" obsolete.  Some parsers 
might break, some parsers might ignore that.  If we want a transition 
we can deprecate a feature.  We can look at this in terms of senders 
do not generate and receivers will process.  We could also have 
chosen the "please ignore" approach where receivers will not 
process.  The "verifiers SHOULD ignore deprecated features" is a 
generic recommendation.   If we want to see an immediate impact we 
are forgoing gradual transition.

Regards,
S. Moonesamy

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 17 Jul 2012 21:54:07 -0400

No hat.

> exist, but as you say it's not clear how to construct a 100% defensible

Right.  If I wanted to attack the samples, I'd say that they were either
biased in favour of domains "we know" or else in favour of large sites
that have different use patterns than small sites.  Or something like
that.  The point is that these samples are not a random sample of "the
Internet", which seems like a pretty tall order.

Aside: I've often thought (especially lately) that the problem we have
is that we're trying to measure things in much the way that the
physical and sometimes social sciences do, and what we probably want
to do is something more like economic modelling.  The problem, of
course, is that economic modelling is well known to produce almost
useless results most of the time :)

> statistics is a well-established discipline.

Our problem in this case is not a statistics problem, but a sampling
one.  What we need is a representative sample of the population.  If
Dave thinks that we've been arguing about semantic problems by
discussing what "unused" means, I can't wait until we discuss what is
"representative".  It still seems to me to be much easier here to
deprecate but not remove the feature, but I still haven't thought very
hard about the "dangerous" claim.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: S Moonesamy <[email-masked]>
Date: Tue, 17 Jul 2012 20:35:07 -0700

At 16:17 17-07-2012, Scott Kitterman wrote:
>into equivalent records. I don't think that's what we'd want.  FWIW, 
>that first record isn't a theoretical one. That was the cisco.com 
>SPF record for several years.

Here's the text in Section 5.5 of draft-ietf-spfbis-4408bis-03:

   "This mechanism tests whether the DNS reverse-mapping for <ip> exists
    and correctly points to a domain name within a particular domain.
    This mechanism is deprecated and SHOULD NOT be used."

   'Note: This mechanism has been deprecated because it is slow, it is
    not as reliable as other mechanisms in cases of DNS errors, and it
    places a large burden on the arpa name servers.  If used, proper PTR
    records must be in place for the domain's hosts and the "ptr"
    mechanism should be one of the last mechanisms checked.  After many
    yaers of SPF deployment experience it has been concluded it is
    unnecessary and more reliable alternatives used instead.'

 From Section 1.3.4:

   'There are several [RFC4408] features that are marked "deprecated".
    In the context of this document, deprecated means that senders SHOULD
    NOT publish SPF records that make use of the feature in question.
    Its use is NOT RECOMMENDED and it might be removed entirely in future
    updates to the protocol.  Such features do, however, remain part of
    the SPF protocol and receiving systems MUST support them unless this
    specification says otherwise.'

After reading the above I end up with diverse interpretations of what 
to do.  I suggest forgetting about Section 1.3.4 for now.  Please 
review Section 5.5 to determine whether there is any 
ambiguity.  Please consider whether any proposed change has any 
adverse effect on other parts of the draft.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 18 Jul 2012 11:10:47 +0200

On Tue 17/Jul/2012 21:21:45 +0200 Scott Kitterman wrote:
>> Recommending Publishers not use it is another thing, so perhaps
>> this is more about a BCP.
>> 
> Which is a brain dead recommendation.  It's not random that
> openspf.org no longer provides a wizard.  The old one produced
> awful garbage like this and no one wrote a new one yet.  The one
> thing that all SPF wizards (AFAIK) have in common is they aren't
> much good.

Good point!  Should the definition of "Deprecated" address wizards?
For example (after incorporating Murray's change):

 There are several [RFC4408] features that are marked "deprecated".
 In the context of this document, deprecated means that senders SHOULD
 NOT publish SPF records that make use of the features in question,
 and tools that generate generic SPF records based on users' input
 MUST NOT propose them.  Use of these features is not advised, as
 they might be removed entirely in future updates to the protocol.
 Such features do, however, remain part of the SPF protocol and
 receiving systems MUST support them unless this specification says
 otherwise.

That way, people would be able to tell which wizards are not compliant
with 4408bis.

---

From: Derek Diget <[email-masked]>
Date: Wed, 18 Jul 2012 13:05:26 -0400 (EDT)

On Jul 17, 2012 at 16:55 -0700, S Moonesamy wrote:
=>At 14:30 17-07-2012, Murray S. Kucherawy wrote:
=>> I'm a little confused by this, now that you mention it.  If the force of
=>> this is that verifiers have to continue to support PTR and macros and
=>> whatever else we formally decide to deprecate, I wonder what we've gained.
=>> The verifier is where all of the real work is done and where the risk is
=>> arguably highest.
=>
=>I was trying to explain "MUST support" in terms of a previous discussion.
=>
=>> The sender is advised to stop using them, but there's really no impact if
=>> they don't.
=>> 
=>> What's the point?
=>> 
=>> Wouldn't it be better to say that verifiers SHOULD ignore deprecated
=>> features, but if they don't, they should simply be ignored (rather than
=>> returning "permerror")?
=>
=>In terms of a protocol we can remove a feature or we can deprecate it.  Let's
=>say that we decide to make "ptr" obsolete.  Some parsers might break, some
=>parsers might ignore that.  If we want a transition we can deprecate a
=>feature.  We can look at this in terms of senders do not generate and
=>receivers will process.  We could also have chosen the "please ignore"
=>approach where receivers will not process.  The "verifiers SHOULD ignore
=>deprecated features" is a generic recommendation.   If we want to see an
=>immediate impact we are forgoing gradual transition.

At what point do we have to say that the changes proposed for 4408bis 
will force the SPF record to jump to v=spf3 because the protocol has 
changed enough that interoperability issues (deprecated/removed/?added? 
items) are introduced?

I know (or think) this line of thought is probably outside the group's 
charter, but as a email admin, I am starting to feel that the more that 
is deprecated or removed, the more that I will run into issues with a 
sender saying one thing (either a "legacy" 4408 SPF record or an updated 
4408bis SPF record) and my (receiver) software evaluating it differently 
than the publisher wanted.

<off-topic rant>
Heck, I have had an open ticket with @kohls.com for almost a weak about 
their broken (dual TXT) SPF record.  Don't get me started on how I had 
to tell one of the DNS admins how to use dig to look it up. :(
</off-topic rant>

-- 
***********************************************************************
Derek Diget                            Office of Information Technology
Western Michigan University - Kalamazoo  Michigan  USA - www.wmich.edu/
***********************************************************************

---

From: Hector Santos <[email-masked]>
Date: Wed, 18 Jul 2012 14:44:10 -0400

I don't think its a good idea to blame wizards, especially branding 
any specific ones in a specification.  IMO, they were not the problem 
and I am not convince there was ever a problem requiring to deprecate 
specifically PTR. Its going to cause more harm, than good to promote 
the deprecation. I am not looking forward to changing all our SPF 
policies for our domains using PTR.

--
HLS

---

From: Stuart Gathman <[email-masked]>
Date: Thu, 19 Jul 2012 00:49:48 -0400

PTR not needed:

One use case of PTR is when a home user is forced to send mail through 
his ISP, and the ISP provides no SPF record of its own, but the smtp 
servers all end in "isp.com".  A typical SPF record for this situation 
is "v=spf1 ?ptr:isp.com -all" - which documents plausible origins for 
the home users personal email domain.  How is this accomplished without ptr?

PTR is slow and loads arpa servers:

MTAs already fetch the PTR records.  While you could argue that they 
shouldn't, as long as they do, PTR is the fastest DNS mechanism in SPF 
since it is already cached.

Wierd chars on %p substitution:

Since only a *validated* ptr is substituted, you are guaranteed that the 
substitution came from a DNS A or AAAA record, and that the receivers 
DNS machinery was able to look it up.  If you are concerned about A/AAAA 
RRs with strange chars, the standard could require receivers to ignore A 
and AAAA RRs  with strange chars when validating PTR.

%p ambiguity:

The definition says "validated domain name of <ip>" - but there can be 
more than 1 (up to 10 according to the processing limits).  If the 
choice is arbitrary, this may result in varying SPF result depending on 
which name is chosen.  In practice, though, %p would be used with exists 
for logging - but still, it is an ambiguity.
The definition of %p says "validated name of <ip>" - but there can be

---

From: "John Levine" <[email-masked]>
Date: 19 Jul 2012 16:12:26 -0000

>the home users personal email domain.  How is this accomplished without ptr?

Everyone I know does it by including the ISP's SPF record.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 19 Jul 2012 12:28:09 -0400

> Everyone I know does it by including the ISP's SPF record.

Yes, but he specified the case where the ISP doesn't publish one, so that's not 
much of a solution.  

I'm not much of a fan of his solution either.  

Back in ancient history, when I was stuck in this kind of a situation I sent 
myself a bunch of mail to an external mailbox, figured out the MTA hostname 
naming pattern, made use of dig to figure out which names in that pattern 
existed, and constructed a list of ip4: mechanisms to add to my SPF record.  
That's not much fun to do, but ultimately produces a much better solution than 
ptr.

Scott K

---

From: "John Levine" <[email-masked]>
Date: 19 Jul 2012 20:30:03 -0000

>That's not much fun to do, but ultimately produces a much better solution than 

Makes sense.  Beyond some point, it's not your correspondents' job to
help work around your lousy configuration.

R's,
John