Subject: [spfbis] New issue? What identity is being reported/checked?

From: Alessandro Vesely <[email-masked]>
Date: Fri, 09 Nov 2012 17:29:21 -0500

Section 7, *Recording The Result* talks about results in general.

The following sentence in 7.1, *The Received-SPF Header Field* might
generate confusion, as it seems to imply that the "helo" identity is
mandatory while "MAIL FROM" IS OPTIONAL:

   That is, at least "client-ip", "helo", and, if the
   "MAIL FROM" identity was checked, "envelope-from".

The recommendation to use one field for each identity that was checked
has disappeared.  (Why?)  It can only be inferred from the examples.

The second paragraph of Section 2.2, *The MAIL FROM Identity* has a
couple of issues.  One is that it says:

                In this case, there is no explicit sender mailbox, and
   such a message can be assumed to be a notification message from the
   mail system itself.

The cited section of RFC 5321 says exactly the opposite.  That is, if
the message is a compliant notification, you can assume that the
envelope sender is empty, not the other way around.

Anyway,it is the next sentence that might cause some real states of
confusion about identities:

                        When the reverse-path is null, this document
   defines the "MAIL FROM" identity to be the mailbox composed of the
   local-part "postmaster" and the "HELO" identity (which might or might
   not have been checked separately before).

If that were a definition, it should have been given in Section 1.3.3,
*Mail From Definition*.  Isn't that cumbersome?  Does it make any real
protocol difference if we simply say that when the "MAIL FROM" identity
is empty, verifying helo raises from SHOULD to MUST?

Consider:

   Received-SPF: fail (mybox.example.org: domain of
                     [email-masked] does not designate
                     192.0.2.1 as permitted sender)
                     identity=mailfrom; client-ip=192.0.2.1;
                     envelope-from="[email-masked]";

Does that apply the definition correctly?

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 09 Nov 2012 19:24:40 -0500

> Does that apply the definition correctly?

Where the current text is unclear, please provide a recommended alternative.

Changing from "Use helo to construct an artificial mail from in this manner" to 
"If mail from is empty helo checking is mandatory" is a protocol change.  It 
may usually (and perhaps always) achieve the same result, but I think it's 
precisely the kind of change we need to be careful about making due to risks 
of inadvertent consequences.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sat, 10 Nov 2012 08:11:43 -0800

I agree that the first "if" referring to MAIL FROM is a little odd, at
least without the surrounding context.  But I also agree with Scott: Please
post some specific suggested changes, like:

OLD:
blah blah

NEW:
blah2 blah2

It's much easier to see what you have in mind then.

-MSK

On Fri, Nov 9, 2012 at 4:24 PM, Scott Kitterman <[email-masked]> wrote:

> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
>

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 11 Nov 2012 21:14:59 +0100

> Where the current text is unclear, please provide a recommended alternative.

Changes are as follows (see also xml patch attached).  I also reword
the previous protocol change (exemption when useless) as with no
commas the "by applying" might be associated to the helo check inside
the "if".

ORIGINAL (4408)
 [RFC2821] allows the reverse-path to be null (see Section 4.5.5 in
 RFC 2821).  In this case, there is no explicit sender mailbox, and
 such a message can be assumed to be a notification message from the
 mail system itself.  When the reverse-path is null, this document
 defines the "MAIL FROM" identity to be the mailbox composed of the
 localpart "postmaster" and the "HELO" identity (which may or may not
 have been checked separately before).

 SPF clients MUST check the "MAIL FROM" identity.  SPF clients check
 the "MAIL FROM" identity by applying the check_host() function to the
 "MAIL FROM" identity as the <sender>.

OLD (bis-08):
 SPF verifiers MUST check the ""MAIL FROM" identity if a completed
 "HELO" check has not reached a definitive policy result by applying
 the check_host() function to the "MAIL FROM" identity as the
 <sender>.

 [RFC5321] allows the reverse-path to be null (see Section 4.5.5 in
 [RFC5321]).  In this case, there is no explicit sender mailbox, and
 such a message can be assumed to be a notification message from the
 mail system itself.  When the reverse-path is null, this document
 defines the "MAIL FROM" identity to be the mailbox composed of the
 local-part "postmaster" and the "HELO" identity (which might or might
 not have been checked separately before).

NEW:
 SPF verifiers MUST check the "MAIL FROM" identity, by applying the
 check_host() function to the "MAIL FROM" identity as the <sender>,
 except for the two cases following:

 o  A final policy decision has already been reached by completing a
    check on the "HELO" identity.  In this case, unless its result
    will be recorded (Section 7), it is useless to check the "MAIL
    FROM" identity.

 o  SMTP allows the "MAIL FROM" identity to be null (Section 4.5.5 in
    [RFC5321]).  In this case, SPF verifiers MUST check the "HELO"
    identity instead.  According to Section 4.3, that implies
    synthesizing a mailbox composed of the local-part "postmaster" and
    the "HELO" identity.  Note that such check might have been carried
    out separately before.

> Changing from "Use helo to construct an artificial mail from in
> this manner" to "If mail from is empty helo checking is mandatory"
> is a protocol change.  It may usually (and perhaps always) achieve
> the same result,

An example where the achieved results differ is given above.  The new
wording doesn't allow unintended interpretations, hopefully:  That's
what I'm asking the list to verify.

> but I think it's precisely the kind of change we need to be careful
> about making due to risks of inadvertent consequences.

Agreed.  On the other hand, ill-defined rules bear the risk of latent
misinterpretations, whose consequences may show up anytime.

BTW, the "exemption when useless" isn't in RFC 4408 either.
--- draft-ietf-spfbis-4408bis-08.xml	2012-11-11 17:49:31.000000000 +0100
+++ draft-ietf-spfbis-4408bis-08+.xml	2012-11-11 20:33:10.000000000 +0100
@@ -233,20 +233,25 @@
       </section>
       <section title="The &quot;MAIL FROM&quot; Identity" anchor="mfrom-ident">
         <t>
-          SPF verifiers MUST check the ""MAIL FROM" identity if a completed
-          "HELO" check has not reached a definitive policy result by
-          applying the check_host() function to the "MAIL FROM" identity as
-          the &lt;sender&gt;.
-        </t>
-        <t>
-          <xref target="RFC5321"/> allows the reverse-path to be null (see
-          Section 4.5.5 in <xref target="RFC5321"/>).  In this case, there is no
-          explicit sender mailbox, and such a message can be assumed to be a
-          notification message from the mail system itself.  When the
-          reverse-path is null, this document defines the "MAIL FROM"
-          identity to be the mailbox composed of the local-part "postmaster"
-          and the "HELO" identity (which might or might not have been
-          checked separately before).
+          SPF verifiers MUST check the "MAIL FROM" identity, by applying the
+          check_host() function to the "MAIL FROM" identity as the
+          &lt;sender&gt;, except for the two cases following:
+        <list style="symbols">
+          <t>
+            A final policy decision has already been reached by completing a
+            check on the "HELO" identity.  In this case, unless its result
+            will be recorded (<xref target="results-headers"/>), it is useless
+            to check the "MAIL FROM" identity.
+          </t>
+          <t>
+            SMTP allows the "MAIL FROM" identity to be null (Section 4.5.5 in
+            <xref target="RFC5321"/>).  In this case, SPF verifiers MUST check
+            the "HELO" identity instead.  According to <xref target="initial"/>,
+            that implies synthesizing a mailbox composed of the local-part
+            "postmaster" and the "HELO" identity.  Note that such check might
+            have been carried out separately before.
+          </t>
+        </list>
         </t>
       </section>
       <section title="Publishing Authorization">

---

From: Hector Santos <[email-masked]>
Date: Sun, 11 Nov 2012 16:21:43 -0500

>     the "HELO" identity.  Note that such check might have been carried

I would add (with a text change above to indicate three cases):

    o  Some other authentication/authorization check is made that does not
       require SPF to be applied any longer.  A good example is checking
       the RCPT TO address for local user existence prior to performing
       SPF checks (or any other DNS related protocol check). In 
practice, this
       consideration can reduce DNS/SPF overhead by the same 
percentage as
       the RCPT TO rejections detected, in some field observations as 
high
       as 60% reduction in SPF necessity for the current session.

-- 
HLS

---

From: Hector Santos <[email-masked]>
Date: Sun, 11 Nov 2012 16:34:42 -0500

Hector Santos wrote:
>       SPF checks (or any other DNS related protocol check). In practice, 
>       this consideration can reduce DNS/SPF overhead by the same percentage
>       as the RCPT TO rejections detected, in some field observations as 
>       high as 60% reduction in SPF necessity for the current session.

Keep in mind this optimization suggestion is on par with RFC 2821/5321 
section 3.3 regarding checking the acceptability of the reverse-path 
(5321.MAIL FROM).

    3.3 Mail Transactions

    .....

    Despite the apparent scope of this requirement, there are
    circumstances in which the acceptability of the reverse-path may
    not be determined until one or more forward-paths (in RCPT
    commands) can be examined.  In those cases, the server MAY
    reasonably accept the reverse-path (with a 250 reply) and then
    report problems after the forward-paths are received and
    examined.  Normally, failures produce 550 or 553 replies.

For example, if your SMTP site on average rejects RCPT TO 60% of the 
time, then delaying the SPF processing on MAIL FROM (and/or HELO) 
until RCPT TO is determined will reduce the SPF, thus the DNS overhead 
by 60% as well.

These are the sort of insights accumulated from the 9+ years of 
experimenting that will benefit the BIS work item.

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Sun, 11 Nov 2012 20:17:46 -0500

>     the "HELO" identity.  Note that such check might have been carried

Your proposal does, in fact, change the protocol, if only slightly.  A 
constructed "MAIL FROM" check (which is what's in RFC 4408 and the current 
4408bis draft) is not, in my opinion, the same as a HELO check.  How about 
this instead:

    If the <sender> is null (see Section 4.5.5 of [RFC5321]),  such a message
    can be assumed to be a notification message from the mail system itself.
    The "MAIL FROM" in such cases is constructed using the local-part
    "postmaster" and the "HELO" name for the domain part of the "MAIL FROM".

> BTW, the "exemption when useless" isn't in RFC 4408 either.

That's correct.  This was discussed on the list before the change was made.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 11 Nov 2012 20:19:48 -0800

The second sentence sounds like it could be a description of something
being done by the agent sending the message, rather than the module doing
SPF evaluation.

I suggest:

If the <sender> is null (see Section 4.5.5 of [RFC5321]), such a message
can be assumed to be an notification message from the mail system itself.
The module doing SPF evaluation should instead act as if the "MAIL FROM"
address consisted of the local-part "postmaster" and domain part set to the
parameter that was passed with the "HELO" or "EHLO" command.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 11 Nov 2012 20:25:45 -0800

>  the check_host() function to the "MAIL FROM" identity as the

This could probably be fixed just by sticking commas after "identity" and
"result".

>     will be recorded (Section 7), it is useless to check the "MAIL

I would strike the last sentence:

"A final policy decision has already been reached by completing a check on
the "HELO" identity, unless the result of this check is to be recorded (see
Section 7)."

It's not necessary to make the "useless" point.

>     the "HELO" identity.  Note that such check might have been carried

Can you give an example of a possible interpretation of the OLD text that
is not possible with the NEW text?  I'm still having trouble seeing the
problem we're trying to fix here.

In particular, this bullet point does not describe a case in which the MAIL
FROM check would be skipped; rather it describes how to go about doing it
for the "null" case, which doesn't flow from the proposed earlier paragraph.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 11 Nov 2012 21:57:46 -0800

This suggestion made me go back and do some reading.  Others should pipe up
if I'm wrong, but it seems to me that although RFC4408 and the current bis
document probably imply that rejection is meant to be done inline with the
HELO or MAIL FROM command, it doesn't actually say that anywhere.  For all
we know, the proposed rejection replies come in response to any of HELO,
MAIL FROM, one or more of the RCPT TOs, or even DATA, and still meets the
spec(s).

Should we say explicitly that although inline rejection is common, the
rejection can be done anywhere in the sequence based on local policy
requirements (i.e., other checks)?  Or is being ambiguous actually
appropriate here?

I'm not certain this is the right place to put this paragraph or one like
it, but it does seem to be a reasonable suggestion, especially if other
operators on this list have similar practices and data to share.

-MSK

On Sun, Nov 11, 2012 at 1:21 PM, Hector Santos <[email-masked]> wrote:

> this
>       consideration can reduce DNS/SPF overhead by the same percentage as
>       the RCPT TO rejections detected, in some field observations as high
> ______________________________**_________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/spfbis<https://www.ietf.org/mailman/listinfo/spfbis>
>

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 12 Nov 2012 11:12:15 +0100

>     "postmaster" and the "HELO" name for the domain part of the "MAIL FROM".

MAIL FROM and <sender> should be swapped, in that text.  <sender> is
the argument of check_host(), and can never be null.  MAIL FROM can be
null, or can be constructed otherwise by the SMTP client.

BTW, rather than repeating the following for each identity, we should
state once, in Section 2.0, that:

   Checking the "X" identity means applying the check_host() function
   to the "X" identity as the <sender>, for "X" = "MAIL FROM", "HELO".

Now for that spurious definition.  Since we apply a function, its
parameter is a dummy variable.  Thus, to say

   define MAIL FROM := HELO;
   MF-result = check_host(MAIL FROM);

 is the same as saying

   MF-result = check_host(HELO);

 except for the silent dropping of the spurious definition before
reporting the SPF result --which is what I'm trying to correct.

Thus, what remains is to say that that result is what should be used
to make policy decisions.  We /have/ to be ambiguous here, because we
don't specify that part.  In no case we could prevent a receiver from
behaving specially when MAIL FROM is null.  So what we really would
have said is that for simplistic policies that just consider a single
SPF result, such result, which is normally check_host(MAIL FROM), is
check_host(HELO) when MAIL FROM is null.  Correct?

How about:

  If the "MAIL FROM" identity is null (see Section 4.5.5 of
  [RFC5321]) the principal SPF result is obtained by checking the
  "HELO" identity instead.

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 12 Nov 2012 11:58:43 +0100

On Mon 12/Nov/2012 05:25:45 +0100 Murray S. Kucherawy wrote:
> On Sun, Nov 11, 2012 at 12:14 PM, Alessandro Vesely <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> "A final policy decision has already been reached by completing a
> check on the "HELO" identity, unless the result of this check is to
> be recorded (see Section 7)."
>

Agreed, we could put such kind of explanations to the /changes/
appendix, whenever we write it.

> Can you give an example of a possible interpretation of the OLD text
> that is not possible with the NEW text?  I'm still having trouble
> seeing the problem we're trying to fix here.

On reporting the result, according to the existing definition, the
verifier shall tag it as the result of checking the "MAIL FROM" identity.

> In particular, this bullet point does not describe a case in which
> the MAIL FROM check would be skipped; rather it describes how to go
> about doing it for the "null" case, which doesn't flow from the
> proposed earlier paragraph.

Correct.

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 12 Nov 2012 12:24:46 +0100

On Sun 11/Nov/2012 22:21:43 +0100 Hector Santos wrote:
>>  [...]
>     SPF checks (or any other DNS related protocol check). In practice, this

Albeit correct, that is outside SPF.  We could as well say that SPF
can be checked after DNSBLs.  It sounds like uselessly complicating
the subject, since we don't actually specify receiver policy.  Recall
http://www.ietf.org/mail-archive/web/spfbis/current/msg00494.html
(part of the "exemption when useless" discussion).

---

From: Hector Santos <[email-masked]>
Date: Mon, 12 Nov 2012 10:56:46 -0500

I never did agree or buy into the lack of a receiver policy argument. 
  This is a protocol.  You can do a State Machine flow chart!  There 
are boundary conditions and limits and very little options at each 
state that one can do. You can literally write down a SPF process 
model with its input arguments and expected outputs - plug and play 
black box function generator!

Reminder, SPF is a significant DNS overhead protocol with continued 
unknown query waste of domains.  If SPF implementations can reduce the 
need to query DNS by 60% (each site will vary), then it would be 
prudent to leverage this potential reduction in overhead and SPF 
waste.  This benefits the SPF network as a whole.

--
HLS

-- 
HLS

---

From: Franck Martin <[email-masked]>
Date: Mon, 12 Nov 2012 18:40:03 +0000

The document would gain clarity if the MAIL FROM was called something else. It entertains confusion with the mail from aka RFC5321from.

Can't we call it something like the spf identity and defines what it is: rfc5321from if present otherwise postmaster@<helo string> ?

From: "Murray S. Kucherawy" <[email-masked]<mailto:[email-masked]>>
Date: Sunday, November 11, 2012 9:57 PM
To: Hector Santos <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Alessandro Vesely <[email-masked]<mailto:[email-masked]>>
Subject: Re: [spfbis] New issue? What identity is being reported/checked?

This suggestion made me go back and do some reading.  Others should pipe up if I'm wrong, but it seems to me that although RFC4408 and the current bis document probably imply that rejection is meant to be done inline with the HELO or MAIL FROM command, it doesn't actually say that anywhere.  For all we know, the proposed rejection replies come in response to any of HELO, MAIL FROM, one or more of the RCPT TOs, or even DATA, and still meets the spec(s).

Should we say explicitly that although inline rejection is common, the rejection can be done anywhere in the sequence based on local policy requirements (i.e., other checks)?  Or is being ambiguous actually appropriate here?

I'm not certain this is the right place to put this paragraph or one like it, but it does seem to be a reasonable suggestion, especially if other operators on this list have similar practices and data to share.

-MSK

On Sun, Nov 11, 2012 at 1:21 PM, Hector Santos <[email-masked]<mailto:[email-masked]>> wrote:
Alessandro Vesely wrote:
NEW:
 SPF verifiers MUST check the "MAIL FROM" identity, by applying the
 check_host() function to the "MAIL FROM" identity as the <sender>,
 except for the two cases following:

--
HLS

_______________________________________________
spfbis mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/spfbis

---

From: Hector Santos <[email-masked]>
Date: Mon, 12 Nov 2012 14:05:54 -0500

reverse-path/address, return-path/address are well understood SMTP terms.

Franck Martin wrote:
> ------------------------------------------------------------------------
> 

-- 
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 12 Nov 2012 15:34:18 -0500

I've always assumed it could be done at any time and I think when you do it is 
really a local configuration matter and not something we need to explicitly 
discuss.  Postfix, as an example, will, by default, not reject until after Mail 
>From even if it's based on data from connect or HELO as a work around for 
interoperability issues seen with other MTAs (I don't know what issues or what 
MTAs).

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 12 Nov 2012 15:39:28 -0500

>   [RFC5321]) the principal SPF result is obtained by checking the

No, because I don't think "Check HELO" is the same thing as "Check Mail From 
constructed from postmaster and HELO"

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 12 Nov 2012 15:41:32 -0500

On Monday, November 12, 2012 10:56:46 AM Hector Santos wrote:
...
> I never did agree or buy into the lack of a receiver policy argument. 
...
It doesn't matter.  With some VERY limited exceptions that's what's documented 
in RFC 4408 and what we are chartered to update.  Let's hold discussions about 
expanded scope until we have 4408bis done.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 12 Nov 2012 15:43:12 -0500

On Monday, November 12, 2012 06:40:03 PM Franck Martin wrote:
> The document would gain clarity if the MAIL FROM was called something else.
> It entertains confusion with the mail from aka RFC5321from.
> 
> Can't we call it something like the spf identity and defines what it is:
> rfc5321from if present otherwise postmaster@<helo string> ?

We did take a stab at doing something like this in one draft and it turned out 
to be more confusing.  Mail From is reasonably well understood and we have a 
definition in the document.  I think that is sufficient.

Scott K

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 12 Nov 2012 16:14:23 -0500

With my chair hat on,

> expanded scope until we have 4408bis done.

I couldn't have said this better myself.  What Scott said, please.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Hector Santos <[email-masked]>
Date: Mon, 12 Nov 2012 18:17:33 -0500

Which is what? I'm lost again in what the WGCs are looking for.

-- 
HLS

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 12 Nov 2012 18:29:03 -0500

> Which is what? I'm lost again in what the WGCs are looking for.

No expanded scope, please.  That means that talking about things
outside what's in 4408 is not part of our charter.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 13 Nov 2012 09:49:33 +0100

> constructed from postmaster and HELO"

And the difference between them is?

---

From: Hector Santos <[email-masked]>
Date: Tue, 13 Nov 2012 05:54:28 -0500

> outside what's in 4408 is not part of our charter.

Does that include AUTH-RES?  DKIM?  DMARC? and anything related to 
reporting and reputation, like SPAMHAUS or gearing up for it? IOW, the 
Reorganization?

I believe I have been advocating nothing but the pure codification of 
this SPF protocol and implementation insights based on the 9+ years of 
field operations for the BIS which I believe is on par for standard 
(or typical) BIS work.

-- 
HLS

---

From: S Moonesamy <[email-masked]>
Date: Tue, 13 Nov 2012 12:26:57 -0800

Hi Hector,
At 02:54 13-11-2012, Hector Santos wrote:
>reporting and reputation, like SPAMHAUS or gearing up for it? IOW, 
>the Reorganization?

DKIM is not part of the 4408bis work.  DMARC is not part of the 
4408bis work.  I haven't seen any messages on the SPFBIS mailing list 
about doing reputation work.

Regards,
S. Moonesamy

---

From: Hector Santos <[email-masked]>
Date: Tue, 13 Nov 2012 16:13:32 -0500

S Moonesamy wrote:
> DKIM is not part of the 4408bis work.  DMARC is not part of
> the 4408bis work.  I haven't seen any messages on the 
> SPFBIS mailing list about doing reputation work.

Nevertheless, they are there.  Its good to know we are getting focused 
on BIS.

Thanks

-- 
Sincerely

Hector Santos
http://www.santronics.com