Subject: Re: [spfbis] 4408bis update

From: Arthur Thisell <[email-masked]>
Date: Fri, 17 Aug 2012 06:58:13 -0700 (PDT)

>   The macro ABNF does not in itself limit macro expansion so that it
>   results in a valid domain-spec.  Nonsense such as this is permitted:
>
>   %{s.-+,/_=.+,..+.}
>
>   Just because the macro ABNF allows it, does not make it valid.  The
>   result of macro expansion MUST still be a valid domain-spec.

I don't understand this section.   The macro %{s.-+,/_=.+,..+.} is perfectly valid, although having multiple identical split delimiters is redundant.  It says to take the <sender>, split it into multiple parts based on the characters ".-+_=", and then glue them back together with dots.

As far as being a valid domain-spec, that is bogus.  A domain-spec, as defined in the I-D can have macros that are substituted, but SPF doesn't do recursive macro expansion.

As far as whether the result is a valid domain name, that is spelled out a few paragraphs above where it says "When the result of macro expansion is used in a domain name query, ...".  It doesn't matter if the result isn't a valid host name, the DNS allows for 8-bit data in labels, except \0.  Some of the DNS RFCs even give examples of such, and you can enter these labels in bind-format zone files.  It may well be that a particular site has valid local parts that can't easily be put into the DNS or queried or due to the "case insensitive" nature of the DNS, multiple EAI local-parts map into a single DNS label.  If such is the case, those sites should not use macros that use the local-part.

I understand the problem with SPF allowing people to shoot themselves in the foot.   I understand that marcos are, in general, not used much, and bugs in the SPF library, the OS (gethostbyame), etc. can easily lurk in little used areas.  I understand that things evolve and things that could be done in the past can no longer be used most of the time.

If we were designing a new protocol, or a new version of SPF, I would be quite open to the idea of getting rid of or restricting macros, but we aren't.   There are some things, such as exist:${ir}._spf.example.com that have to be used if the number of ip addresses that a ADMD uses exceeds all other encoding methods.  Arguing that macros can be gotten rid of because there aren't many of them would be like arguing that since there are only 13 root name server IP addresses, we can get rid of root name servers.

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 17 Aug 2012 10:23:19 -0400

On Friday, August 17, 2012 06:58:13 AM Arthur Thisell wrote:
> I don't understand this section.   The macro %{s.-+,/_=.+,..+.} is perfectly
> valid, although having multiple identical split delimiters is
> redundant.  It says to take the <sender>, split it into multiple parts
> based on the characters ".-+_=", and then glue them back together with
> dots.
> 
> As far as being a valid domain-spec, that is bogus.  A domain-spec, as
> defined in the I-D can have macros that are substituted, but SPF doesn't do
> recursive macro expansion.
> 
> As far as whether the result is a valid domain name, that is spelled out a
> few paragraphs above where it says "When the result of macro expansion is
> used in a domain name query, ...".  It doesn't matter if the result isn't a
> valid host name, the DNS allows for 8-bit data in labels, except \0.  Some
> of the DNS RFCs even give examples of such, and you can enter these labels
> in bind-format zone files.  It may well be that a particular site has valid
> local parts that can't easily be put into the DNS or queried or due to the
> "case insensitive" nature of the DNS, multiple EAI local-parts map into a
> single DNS label.  If such is the case, those sites should not use macros
> that use the local-part.

OK, so I got that bit a bit wrong (as well as the localpart/EAI text).  
Thanks. I think I can safely delete this addition.  Please suggest appropriate 
text for the last sentence of the paragraph in section 8.1 that discusses the 
"s", "l", and "o" macros.  I'm not sure how to put it.

Scott K

---

From: "John Levine" <[email-masked]>
Date: 17 Aug 2012 17:11:09 -0000

>"s", "l", and "o" macros.  I'm not sure how to put it.

The point is that those macros can allow hostile senders to inject
arbitrary character strings into the names in DNS queries.  So you
better be sure that the libraries that handle the queries don't assume
that the names are well formed, or consist only of printing ASCII
characters.

If you want to mention EAI, note that an EAI address consists of an
arbitrary UTF-8 mailbox and a domain which may be a U-label (UTF-8) or
A-label (punycoded.)  For %o it would probably be a good idea to
convert U- to A-labels before lookup.  But other than that, addresses
can and probably will contain characters that are difficult or
impossible to provision in DNS software, so don't use %s or %l in
expansions that are intended to produce names to be looked up.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 17 Aug 2012 23:09:14 -0400

On Friday, August 17, 2012 05:11:09 PM John Levine wrote:
> >Thanks. I think I can safely delete this addition.  Please suggest
> >appropriate text for the last sentence of the paragraph in section 8.1
> >that discusses the "s", "l", and "o" macros.  I'm not sure how to put it.
> 

I think "there may be bugs in software" is not a working group issue.  If 
people have DNS servers answering arbitrary queries from the outside world, 
all these things can already be queried for.  Fundamentally any naive 
assumptions DNS software developers make are not our problem.

If we have some real experience with real problems (I've certainly seen both 
with use of PTR and relying on DNS over TCP) then we should consider if we are 
proposing something unreliable.  I don't think we ought to do too much hand 
wringing over uncertain issues that may or may not exist.

> expansions that are intended to produce names to be looked up.

Why not?  If I can generate a query for it, but it can't be provisioned, all 
that can happen is it doesn't match.  Under RFC 4408, that's what would happen 
and so I think we just need to say that if the domain has a local-parts that 
can't be looked up, then don't use %s or %l in your record.  I have a poor try 
at this in -04, but it needs redoing.

Scott K

---

From: "John Levine" <[email-masked]>
Date: 18 Aug 2012 19:33:16 -0000

>I think "there may be bugs in software" is not a working group issue.

It's not just bugs, although I think it would polite to point out that
it's likely to send stuff to your resolver library that the library is
not prepared to handle.  (I do not share Arthur's confidence that
resolver libraries have all been hardened against obscure inputs.)

The worse problem is that the interface to resolver libraries is
surprisingly ill specified once you get very far away from host names.
Most painfully, dot and null (zero) are valid characters in a component
of a domain name, but most resolver libraries don't deal with that.

So let's say your SPF record has

   exists:%{l}.users.example.com

The bounce address is [email-masked], so it looks up
john.doe.users.example.com.  Is that first dot a delimiter between two
components or part of the first component?  What if the address is
[email-masked]?  Does that create a valid or an invalid domain
name?

Before you say this is totally off the wall, look at the RNAME field
of the SOA RR, which has the same problem, the first component of the
name is a mailbox.  I have seen people stick a dot in there using \.
which works in master files, but as far as I know not in resolver
libraries.

Since approximately nobody has noticed this before, it just confirms
that approximately nobody uses macros.  That's why I don't suggest
trying to nail down the right thing to do, just note that macros allow
hostile senders to inject arbitrary text into your DNS queries, so you
should defend against that.  EAI makes it somewhat more likely that
names will include hard to handle text, but the problem's been there
all along.

R's,
John

---

From: Hector Santos <[email-masked]>
Date: Sat, 18 Aug 2012 21:12:12 -0400

> which works in master files, but as far as I know not in resolver

One poor resolver should not suggest its applies to all.

> that approximately nobody uses macros.

Incredible. Darn if spf.messagelabs.com (just one of many) doesn't use 
the exist macros for its customers SPF related services.  Our 
bnymellon.com customer has this hardfail -ALL policy:

        v=spf1 include:spf.messagelabs.com
               include:pacificlife.com
               ip4:198.181.8.23
               ip4:198.181.8.24
               ip4:207.218.135.22
               ip4:207.218.135.23
               ip4:147.249.37.42
               mx
               -all

and the first include yields:

        v=spf1 exists:%{ir}.nets.messagelabs.com

What it seems to suggest is maybe people only used certain macros, 
those that are not going to cause issues or questions like the one you 
suggest.

> That's why I don't suggest trying to nail down the right thing to do, 
> just note that macros allow hostile senders to inject arbitrary text 
> into your DNS queries, so you should defend against that. 

IMO, SPF macros is not the key issue here. But I do agree, we should 
point out the particular complexities where there might be some 
conflicts and developers (also publishers) need to be aware of.

Per your example %{l}, using local parts has always been problematic 
for protocols, including SMTP and especially within DNS, i.e. user 
level granularity concepts.

> EAI makes it 
> somewhat more likely that names will include hard to handle text, but 
> the problem's been there all along.

Right, SPF lives in the same screwed up world of EMAIL.

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Sat, 18 Aug 2012 22:17:24 -0400

> names will include hard to handle text, but the problem's been there

The way I have addressed this is to remove the attempt at an EAI discussion in 
the local-part macro, which I think is a can of worms that we don't need to 
open and to rework the untrusted data security consideration into three sub-
sections, one if which is about macros:

          <xref target="macros"/> allow senders to inject arbitrary text (any
          non-null <xref target="US-ASCII"/> character) into your DNS queries.
          It is necesary to be prepared for hostile or unexpected content.

That's in for -05 and we can tweak it from there.

Scott K

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 20 Aug 2012 14:06:38 -0400

No hat.

> which works in master files, but as far as I know not in resolver

Apparently the resolver libraries you used didn't read RFC 1035 when
implementing (this is not surprising; a large number of them seem not
to have done).  See, for example, this text from 1035 section 5.1:

\X              where X is any character other than a digit (0-9), is
                used to quote that character so that its special meaning
                does not apply.  For example, "\." can be used to place
                a dot character in a label.

This is about master files, which are "text files that contain RRs in
text form."

In the wire format, of course, the separator dot doesn't appear, so we
don't have a problem distinguising "." on the wire from the others.

> that approximately nobody uses macros.

Either that, or people's resolver libararies are somewhat better than
you seem to think in coping with features of the DNS that have been
there since day 1 (approximately the same text as I quote above is in
RFC 883).  

> should defend against that.

Note that I have no objection to text along those lines.  I'd prefer
"be prepared to deal with" instead of "defend against".  I don't think
that to be a substantive change, since it was something that was
always possible.  

Best,

A

-- 
Andrew Sullivan
[email-masked]