Subject: [apps-discuss] Adoption of draft-kucherawy-received-state?

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 11 Jan 2012 13:06:40 -0800

Can I get a show of support (or objection) to processing this through APPSAWG?

Mykyta already gave a +1 to this before it was asked, but others have not said so explicitly.

I'm game, of course.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 12 Jan 2012 12:46:09 -0800

For evidence that there's interest in this work, or at least a lot of input to it, visit http://www.imc.org/ietf-smtp/mail-archive/.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Murray S. Kucherawy
Sent: Wednesday, January 11, 2012 1:07 PM
To: [email-masked]
Subject: [apps-discuss] Adoption of draft-kucherawy-received-state?

I'm game, of course.

-MSK

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 13 Jan 2012 08:23:27 -0800

> Can I get a show of support (or objection) to processing this through APPSAWG?

+1

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: SM <[email-masked]>
Date: Fri, 13 Jan 2012 09:01:46 -0800

>Can I get a show of support (or objection) to processing this through APPSAWG?

This draft registers a new optional clause that can be used in trace 
fields.  I support the processing.

BTW, the normative reference to RFC 5322 ([MAIL] reference in ABNF) is missing.

Regards,
-sm

---

From: John C Klensin <[email-masked]>
Date: Fri, 13 Jan 2012 13:53:39 -0500

Murray and others,

You can consider this an objection to AppsAWG handling the
document in this form or one of the first comments in the
dicussion of it.

SMTP rather carefully defines "trace information" or "trace
headers" and then limits that category to two types of
information, the Received header fields and the Return-path one.
I think I know why Jon Postel did it that way, but it is pure
reconstruction: if I actually knew at the time, I've long since
forgotten.

In any event, we have discovered repeatedly (and sometimes
painfully) that the the Received header field is not well
designed for extensibility.   It depends on an ordered,
context-dependent, reserved keyword model (something I don't
think anyone would do today) and then switches to an Atom-String
pair model for additional clauses.  

While I have no problem at all with the type of facility the
draft proposes, I think that, if we are going to entertain this
sort of thing, we should have a serious discussion about how far
we want to go in extending "Received:" and when it is time to
add one or more additional Trace header fields, possibly with a
typology of functions that each serves (beyond "inserted by a
transit MTA or something else") and with a syntax that is
clearly name-value based, rather than even partially dependent
on parser recognition of specific keywords.

Let's not turn "Received:" further into a kludge just because it
is there.

    john

--On Thursday, January 12, 2012 12:46 -0800 "Murray S.
Kucherawy" <[email-masked]> wrote:

> For evidence that there's interest in this work, or at least a
> lot of input to it, visit
> http://www.imc.org/ietf-smtp/mail-archive/.
> 
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Murray S.
> Kucherawy Sent: Wednesday, January 11, 2012 1:07 PM

---

From: "John Levine" <[email-masked]>
Date: 14 Jan 2012 23:52:07 -0000

>Let's not turn "Received:" further into a kludge just because it

I see your point, but I'm trying to weigh the relative uglitude of
adding more clauses in Received: lines against adding yet more trace
headers.  IANA already has a registry for
Additional-registered-clauses at
http://www.iana.org/assignments/mail-parameters, but as far as I can
tell, there's no registry of trace headers.

On the third hand, there probably should be a registry of trace headers,
since there's nothing I can find now beyond the extremely incomplete list
in section 3.6.7 of RFC 5322.

R's,
John

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 14 Jan 2012 16:48:23 -0800

> headers" and then limits that category to two types of

What is the import of this observation, relative to the current proposal?

You appear to be intending it as a constraint, but I don't understand how, 
specifically.

> designed for extensibility.

I don't recall seeing these claims before or documentation that they are valid. 
  Can you provide some citations?

Note that the definition of the Received: header field provides a registry for 
addition of new clauses, so you appear to be saying that extensibility is not 
viable.

>        It depends on an ordered,

If you are saying that the specific placement of the clause is essential, then 
please document this requirement -- beyond what is in the RFC5321 spec and the 
proposed clause spec -- both in terms of what you believe is acceptable and 
different from the current specification and also the basis for this.

>       we should have a serious discussion about how far
> transit MTA or something else") and

Please review the latest version, specifically the 'Discussion' section, which 
provides exactly the consideration you are calling for.

 >      with a syntax that is

I don't understand the requirement you are proposing, since the "name" of a 
name/value pair is a specific keyword.  Hence, your saying "rather than" appears 
to be contradictory.

You seem to be meaning something else, but I can't guess what.

> Let's not turn "Received:" further into a kludge just because it

How is it already a kludge?

d/

-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Sat, 14 Jan 2012 20:57:52 -0500

--On Saturday, January 14, 2012 23:52 +0000 John Levine
<[email-masked]> wrote:

>> Let's not turn "Received:" further into a kludge just because
>> it is there.
> 
> I see your point, but I'm trying to weigh the relative
> uglitude of adding more clauses in Received: lines against
> adding yet more trace headers.  IANA already has a registry for
> http://www.iana.org/assignments/mail-parameters, but as far as
> I can tell, there's no registry of trace headers.

There is no registry of trace headers because there are, and
have been since RFC 821, only two of them: "Received:" and
"Return-path:".   I'm suggesting that the time may have come to
add one or more rather than continuing to overload "Received:".
If we were to start adding additional ones, a registry would
certainly be in order.  Unless we do, a registry would just be
work for its own sake, at least IMO.

> On the third hand, there probably should be a registry of
> trace headers, since there's nothing I can find now beyond the
> extremely incomplete list in section 3.6.7 of RFC 5322.

See above.

And here we go down that rathole:  I hope that 5321 and 5322 are
consistent, but, if there were any respect in which they were
not, we get back to the old question of which one is
authoritative: 5321 because creating the things and putting them
in is the responsibility of the MTA or 5322 because they are,
after all, header field names.

    john

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 14 Jan 2012 19:03:17 -0800

On 1/14/2012 5:57 PM, John C Klensin wrote:
>  I'm suggesting that the time may have come to

"continuing to"?  please explain the history of overloading that has taken 
place; I'm not aware of it.

Please then explain how the current proposal exceeds a reasonable enhancement of 
the Received field.  "received" refers to transitions.  The enhancement merely 
documents more transitions.

Please also note that creating a separate kind of header field, such as for 
transitions /within/ a host, will then require correlating the /within/ fields 
and the /between/ (Received:) fields.  If you think there is anything about the 
current situation that is a kluge, you ain't seen nothing' yet.

> after all, header field names.

What is it that will cause us to go down that rathole?  There's nothing in the 
current proposal that creates the potential for loss of synchrony between 5321 
and 5322.  So how is your concern relevant here?

d/

-- 

   Dave Crocker

---

From: "John R. Levine" <[email-masked]>
Date: 14 Jan 2012 22:55:01 -0500

> "Return-path:".

If you believe what the RFCs say, there are many more trace headers than 
those two.  RFC 5322 makes a special case of Resent-foo: fields, but they 
act an awful lot like trace headers.  RFC 5451 says that 
Authentication-Results "should be treated as a Trace Field."  RFC 5436 
says "The 'Auto-Submitted' header field is considered a 'trace field'". 
RFC 4408 says that "The Received-SPF header field is a trace field."  RFC 
5518 says "VBR-Info is a 'trace header field'". That horse left the barn a 
long time ago. RFC 5537 also lists a bunch of trace fields for usenet, 
which isn't mail, but it might be a good idea to reserve them to avoid 
collisions.

> add one or more rather than continuing to overload "Received:".

It's certainly time for a trace field registry.  And I suppose that if we
have one, adding new trace fields isn't that big a deal.

R's,
John

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sat, 14 Jan 2012 20:30:43 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John Levine
> Sent: Saturday, January 14, 2012 3:52 PM
> Subject: Re: [apps-discuss] Adoption of draft-kucherawy-received-state?
> 
> headers.  IANA already has a registry for Additional-registered-clauses
> at http://www.iana.org/assignments/mail-parameters, but as far as I can

In fact, this was added by RFC5321, and it is this mechanism that this draft is attempting to employ.  So if augmenting Received is a bad idea, I'm left wondering why RFC5321 deliberately enabled it.

-MSK

---

From: SM <[email-masked]>
Date: Sun, 15 Jan 2012 00:54:07 -0800

Hi John,
At 19:55 14-01-2012, John R. Levine wrote:
>If you believe what the RFCs say, there are many more trace headers 
>than those two.  RFC
>  5322 makes a special case of Resent-foo: fields, but they act an 
> awful lot like trace headers.  RFC 5451 says that 
> Authentication-Results "should be treated as a Trace Field."  RFC 
> 5436 says "The 'Auto-Submitted' header field is considered a 'trace 
> field'". RFC 4408 says that "The Received-SPF header field is a 
> trace field."  RFC 5518 says "VBR-Info is a 'trace header field'". 
> That horse left the barn a long time ago. RFC 5537 also lists a 
> bunch of trace fields for usenet, which isn't mail, but it might be 
> a good idea to reserve them to avoid collisions.

The Auto-Submitted header field is defined in RFC 3834.  It 
references RFC 2822 normatively.  RFC 5451 defines 
Authentications-Results and references RFC 5322 normatively.  RFC 
5518 (VBR-Info) references RFC 5322 normatively.  RFC 4408 references 
RFC 2822 normatively.  RFC 5518 likely used the same path as RFC 
5451.  RFC 3834 does not mention Trace Field but RFC 5436 does.

I would say that some of the header fields were defined as "trace 
fields" to restrict the ordering of existing "trace fields header 
lines".  RFC 5322 contains a requirement not to reorder "trace header 
fields".  Most of the header fields can be traced backed to RFC 5322 
as that document defines a syntax for text messages.  Beliefs in RFCs 
could be traced back to the feature sought.

Regards,
-sm

---

From: SM <[email-masked]>
Date: Sun, 15 Jan 2012 02:53:20 -0800

Hi Murray,
At 20:30 14-01-2012, Murray S. Kucherawy wrote:
>In fact, this was added by RFC5321, and it is this mechanism that 
>this draft is attempting to employ.  So if augmenting Received is a 
>bad idea, I'm left wondering why RFC5321 deliberately enabled it.

I'll comment on the Additional-registered-clauses part only.  The 
Additional-registered-clauses registry was added, in my opinion, to 
keep matters simple.  What made it into RFC 5321 IANA Considerations 
Section is:

   "The registry will contain clause names, a description, a summary of
    the syntax of the associated String, and a reference.  As new clauses
    are defined, they may, in principle, specify creation of their own
    registries if the Strings consist of reserved terms or keywords rather
    than less restricted strings.  As with link and protocol identifiers,
    additional clauses may be registered only by standardization or by way
    of an RFC-documented, IESG-approved, Experimental protocol extension.
    The additional clause name space is for identification and is not
    limited in size: the IESG is encouraged to approve on the basis of clear
    documentation, actual use or strong signs that the clause will be
    used, and a distinct requirement rather than preferences about the
    properties of the clause itself."

The Additional-registered-clauses sub-registry is currently empty as 
the clause is not used in any RFC. I don't have an opinion at the 
moment about what should or should not go into the registry.

Regards,
-sm

---

From: John C Klensin <[email-masked]>
Date: Sun, 15 Jan 2012 10:51:43 -0500

--On Saturday, January 14, 2012 22:55 -0500 "John R. Levine"
<[email-masked]> wrote:

>>> http://www.iana.org/assignments/mail-parameters, but as far
>>> as I can tell, there's no registry of trace headers.
>> 
> If you believe what the RFCs say, there are many more trace
> headers than those two.  RFC 5322 makes a special case of
> Resent-foo: fields, but they act an awful lot like trace
> headers.  RFC 5451 says that Authentication-Results "should be
> treated as a Trace Field."  RFC 5436 says "The
> 'Auto-Submitted' header field is considered a 'trace field'".
> RFC 4408 says that "The Received-SPF header field is a trace
> field."  RFC 5518 says "VBR-Info is a 'trace header field'".
> That horse left the barn a long time ago. RFC 5537 also lists
> a bunch of trace fields for usenet, which isn't mail, but it
> might be a good idea to reserve them to avoid collisions.

This is the sort of disconnect between the specifications about
SMTP MTAs and the specifications of header fields and their
names that I mentioned in an earlier note and to which Dave
Crocker took significant exception.  While I don't want to go on
a quest for specific language, I believe that, from the MTA
point of view, part of the definition of "trace field" is that
they are inserted by MTAs, in transit.  From that point of view,
while fields like the Resent-* set are used to trace the
handling of a message (but so are Sender and maybe Message-ID),
they are MUA functions and hence not trace fields in the MTA
sense.

Some of the things you list above are probably consistent with
the MTA definition; others are probably not.

I think that distinction is worth preserving and, despite some
excessively casual language, don't think we have breached it
yet.  Then again, I think the multi-level distinction between
envelope and content information in X.400 is one of the few
important desirable distinctions between their model and ours
(in which transport tracing information is put into message
headers because we don't have a better place).  YMMD, of course.

> have one, adding new trace fields isn't that big a deal.

I don't have any problem with such a registry.  It seems to me
that you've done most of the work of identifying what should be
in it.  Personally, I'd rather see either separate subregistries
for MTA and MUA trace info or a field that distinguishes the
former from the latter.  Do you want to generate the I-D?  I'm
happy to review and/or help as needed.

best,
   john

---

From: Ned Freed <[email-masked]>
Date: Sun, 15 Jan 2012 07:53:41 -0800 (PST)

>   Can you provide some citations?

While I am unaware of any standards usage of this particular extensibility
feature, ever since it was defined there has been some ad-hoc use of it. I have
not observed any issues with this usage. To the extent there have been
problems, it's been with obsolete forms that have never worked well and jsut a
general failure to implement basic stuff like date-time parsing propoerly.

> addition of new clauses, so you appear to be saying that extensibility is not

If that's indeed the case, then this is a serious problem that trascends the
present discussion. Consider: The claim is that a fairly important feature of a
particalur standard cannot be used because if it is sure to cause
interoperability problems. If so, then this pretty clearly fails to meet the
criteria for standards advancment: Stuff like this is supposed to be removed,
redone, or at least severerly restricted before advancing past proposed. At an
absolute minimum the registry would need to be closed to new entries.

+1

> > transit MTA or something else") and

I have found additional header fields to be be more problematic than extending
Received:. For one thing, agents tend to assume that Received: is the only
field deserving such treatement and many do not even have a list of additional
fields to copy when moving trace information around. (And this operation does
happen in various cases.) And even if such a list is maintained, there's always
the update problem when a new trace field is defined.

For another, while it's fairly safe to rely on headers of the same type
staying in the same order, the same cannot always be said for fields of
different types.

> How is it already a kludge?

Whether or not it is a kludge is of precisely zero interest to me. It's the
mechanim we have. We have a problem to solve here, and unfortunately all of the
possible solutions have issues. In my judgement out of all the available
options the best is to extend Received:, messy syntax and all. All of the other
options, including but not limited to additional trace fields, replacing
Received: with something else, replacing Received; while retaining generation
of Received; for backwards compatibility, are even worse.

And the worst option is effectively saying that addttional trace information
can never be added again. That would really suck.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 15 Jan 2012 19:58:13 +0100

On 15/Jan/12 16:53, Ned Freed wrote:
> 
> We have a problem to solve here, and unfortunately all of the 
> possible solutions have issues. In my judgement out of all the
> available options the best is to extend Received:, messy syntax and
> all. All of the other options, including but not limited to
> additional trace fields, replacing Received: with something else,
> replacing Received; while retaining generation of Received; for
> backwards compatibility, are even worse.

This judgment is difficult to understand, for me at least, without
considering an actual alternative.  For an example trace field to be
paired with "Received:", I'd pick "Sent:".  Consider:

  Received: from mauve.mrochek.com (mauve.mrochek.com [66.59.230.40])
    by ietfa.amsl.com (Postfix) with ESMTP id 489FA21F845E;
    Sun, 15 Jan 2012 08:15:57 -0800 (PST)
  Sent: to ietfa.amsl.com to-addr 12.22.58.30 to-port 25
    by mauve.mrochek.com by-addr 66.59.230.40 by-port 1438
    date "Sun, 15 Jan 2012 08:15:57 -0800 (PST)"
  Received: from dkim-sign.mauve.mrochek.com by mauve.mrochek.com
    (PMDF V6.1-1 #35243) id <[email-masked]> for
    [email-masked]; Sun, 15 Jan 2012 08:15:54 -0800 (PST)
  ...

That "Sent:" field is to be written on the fly, and if the message is
rejected (5xx) or delayed (4xx), then the field gets discarded.

There is a lot of redundant data.  It wastes bandwidth, but can be
used to check proper ordering, synchronization, and consistency.  The
format can be similar and new at the same time.  Both fields can be
extended, but no coordination between their formats is to be mandated.

An advantage of the new field, for the problem at hand, is that it
allows to make a final statement rather than forecasting a state.  It
would be possible to tell this is the 4th attempt in 15 minutes, as
after greylisting.

> And the worst option is effectively saying that additional trace
> information can never be added again. That would really suck.

+1.

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 15 Jan 2012 11:46:12 -0800

Folks,

> have one, adding new trace fields isn't that big a deal.

It would help for folks to explain what the specific need for a "trace fields" 
registry is.  I'm not seeing it.

A registry is for the purpose of coordination.  It allows participants to know 
what exists, either for:

      a)  discovering to use the information, or

      b)  discovery to avoid conflicts in assigning new values

Registries incur a cost, so the associated benefit needs to be quite clear.

      1.  What coordination purpose will be served by the new registry?

We already have a registry to fields[1], so the 'trace' registry would be 
redundant with a subset of the that existing registry.  Redundancy is usually bad.

      2.  Why is redundancy acceptable, here?

d/

[1]  http://tools.ietf.org/html/rfc3864
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 15 Jan 2012 11:54:17 -0800

> a quest for specific language, I believe that, from the MTA

Since I'm cited by name, I'll clarify that what I took exception to were 
assertions of fact that I believe to be wrong and that I asked to see documented.

Unfortunately, anyone can make any claim they feel like making.  Consequently, 
responsible technical discussion means that the making of a claim does indeed 
require someone's doing the work of providing specifics.

Typically, the person making an affirmative assertion of fact carries the 
responsibility for substantiating it.

d/
-- 

   Dave Crocker

---

From: Ned Freed <[email-masked]>
Date: Sun, 15 Jan 2012 11:51:09 -0800 (PST)

> extended, but no coordination between their formats is to be mandated.

We went over all of these tradeoffs in tedious detail back in the MIME work.
The original proposal for MIME headers (written by Bob Smart and myself) was 
to add additional paired fields as you are doing here. Once you work through
all the details, all sorts of problems emerge - it's just not possible to keep
the fields ordered due to vagarities of agent processing. This then led to
proposals for linkage ids, but the complexity is such that implementations are
pretty much guaranteed to botch it. (And now you're back to either extending
Received: or abusing the semantics of some existing clause - exactly what the
new field is trying to avoid.)

Encoded words were then proposed by Keith Moore, and rather obviously had
better operational characteristics. Sure, there were bobbles with
implementations that didn't really support proper syntax processing, but in
hindsight it was clearly the right choice.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 15 Jan 2012 20:08:33 -0000

>general failure to implement basic stuff like date-time parsing propoerly.

I've written my share of received header parsers (it's unavoidable if
you want to send a non-trivial number of abuse reports) and I will
agree that extra clauses are not an issue.  The problem, as usual, is
people who don't read the spec at all before they start coding, and
there's not much we can do about that.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 15 Jan 2012 20:16:13 -0000

>> It's certainly time for a trace field registry.  And I suppose
>> that if we have one, adding new trace fields isn't that big a deal.
>

Sure.  My list above was generated using my favorite semantic analysis
tool, grep, on my directory full of RFCs.

I sort of see your distinction between MTA and MUA headers, although I
don't see anything in the list other than Resent-* that strikes me as
something an MUA would do.  The closest is Auto-Submitted: for SIEVE
mailto: notifications, something that's in the MDA or maybe MSA.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 15 Jan 2012 20:18:17 -0000

>      1.  What coordination purpose will be served by the new registry?

Unlike other headers, trace headers can occur multiple times in a
message, and their order means something.  This should be of use to
code that parses headers.

>      2.  Why is redundancy acceptable, here?

I suppose that adding a trace flag to the existing registry would do the trick.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Mon, 16 Jan 2012 07:57:40 -0800 (PST)

> there's not much we can do about that.

Exactly. And given eixsting use of nonstandard clauses any parser that fails to
handle them is not materially different from one that fails to handle other
syntax details.

I completely fail to see why we should be catering to this level of brokennness
at this late date.

					Ned

---

From: Ned Freed <[email-masked]>
Date: Mon, 16 Jan 2012 08:09:26 -0800 (PST)

>       2.  Why is redundancy acceptable, here?

I have to agree with John Levine here - why can't this just be an added flag in
the existiing header registry. The cost of that should be *far* lower than a
new registry. In fact after experience with having additional purpose-specific
media type registries (something we should never have allowed), I am *strongly* 
opposed to overlapping registries of any sort. (Right now I owe IANA a response
about how in the blazes to address the current multiple registries for media
types.)

				Ned

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 16 Jan 2012 09:24:14 -0800

> about how in the blazes to address the current multiple registries for media

I am still left with the basic concern, independent of whether this is a field 
in an existing registry or is a new registry:

      What is the utility of marking some fields as 'trace' fields?

      What coordination does it facilitate?  I really do not understand the point.

In the spec, it's useful to have the label 'trace' for education, to 
conceptually aggregate some fields.  But what is the /functional/ benefit of the 
label?

d/
-- 

   Dave Crocker

---

From: Tony Finch <[email-masked]>
Date: Mon, 16 Jan 2012 23:31:54 +0000

John C Klensin <[email-masked]> wrote:
> we want to go in extending "Received:"

Wouldn't this just be re-hashing a discussion that we already had quite
recently? Murray's draft uses an extensibility hook (the
Additional-Registered-Clauses ABNF production) that was added in RFC 5321.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Viking, North Utsire: Southerly 4 or 5. Slight or moderate. Showers then fair.
Good.

---

From: "John Levine" <[email-masked]>
Date: 17 Jan 2012 05:19:18 -0000

>      What coordination does it facilitate?  I really do not understand the point.

Please see the message I sent yesterday, which answers those questions.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Tue, 17 Jan 2012 07:37:03 -0800 (PST)

>       What is the utility of marking some fields as 'trace' fields?

An obvious one is certain types of forwarding and digesting, where you
want to strip trace information. Logging of trace also comes to mind. Our
software has several lists of fields in it that it uses for such purposes; it
would be helpful not to have to look through dozens of RFCs to figure out
the content of such lists.

>       What coordination does it facilitate?  I really do not understand the

Not sure what you mean by "coordination".

See above.

					Ned

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 17 Jan 2012 08:48:39 -0800

> code that parses headers.

Summary:

      Your note and Ned's suggest to me a reasonable basis for adding a column 
to the existing message header fields registry.

      (Picky detail:  Return-path isn't supposed to appear more than one.  So 
the "multiple times" is currently limited to Received:.

Also...

This has been an interesting exercise. I had not previously noticed that RFC5321 
and RFC5322 have conflicting definitions of 'trace'.  RFC5321 equates it only to 
Received.  RFC5322 uses it to describe a class of fields.

If we are going to start doing more formal things about "trace" stuff, we are 
going to need to resolve this disparity.  In addition, I suspect some folk are 
looking to increase the number of fields counted as <trace> and we're going to 
need to consider each field carefully, since the class gets special treatment.

RFC5322:

    3.6:

    More importantly, the trace header fields and resent
    header fields MUST NOT be reordered, and SHOULD be kept in blocks
    prepended to the message.  See sections 3.6.6 and 3.6.7 for more
    information.
    ...
    | Field          | Min    | Max number | Notes                      |
    +----------------+--------+------------+----------------------------+
    | trace          | 0      | unlimited  | Block prepended - see      |
    |                |        |            | 3.6.7                      |

    3.6.7:

    trace           =   [return]
                        1*received

    return          =   "Return-Path:" path CRLF

    received        =   "Received:" *received-token ";" date-time CRLF

RFC5321:

    4.1.1.4:

    When the SMTP server accepts a message either for relaying or for
    final delivery, it inserts a trace record (also referred to
    interchangeably as a "time stamp line" or "Received" line) at the top
    of the mail data.

    4.4.  Trace Information

    When an SMTP server receives a message for delivery or further
    processing, it MUST insert trace ("time stamp" or "Received")
    information at the beginning of the message content, as discussed in
    Section 4.1.1.4.

-- 

   Dave Crocker

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Jan 2012 09:33:44 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave CROCKER
> Sent: Tuesday, January 17, 2012 8:49 AM
> Subject: Re: [apps-discuss] possibleTrace fields registry
> 
> This has been an interesting exercise. I had not previously noticed
> that RFC5321 and RFC5322 have conflicting definitions of 'trace'.
> RFC5321 equates it only to Received.  RFC5322 uses it to describe a
> class of fields.

I don't see this as being a conflict.  RFC5322 defines a trace field in general terms, and RFC5321 discusses one specific instance of a trace field, namely Received, as being the only one SMTP itself cares about.  It doesn't mean there aren't others, but an MTA doesn't do anything special with them (i.e., doesn't add any new ones).

-MSK

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 17 Jan 2012 10:06:31 -0800

> I don't see this as being a conflict.  RFC5322 defines a trace field in general terms, and RFC5321 discusses one specific instance of a trace field, namely Received, as being the only one SMTP itse…

Except that that's not what RFC5321 says.  Your interpretation actually requires 
some linguistic creativity.  RFC5321 does not note a larger concept, with 
Received being an instance.  It simply uses the one term to refer to the one 
header field.  (IMO, RFCs 821 and 822 created the problem of redundant and 
sometimes divergent definitions, though I hadn't noticed this one before.)

So I think that what you've really done is suggest a simple and reasonable fix, 
requiring only a small amount of wording change to RFC 5321.

When <trace> was a loosely-used term, the disparity didn't matter much.  But 
since there is talk of formalizing the construct further, we should be careful 
about the details and not relay on assumptions about interpretation of 
specification text.  Note, for example, the view that /any/ trace field can be 
repeated...

d/

-- 

   Dave Crocker

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 17 Jan 2012 19:36:24 +0100

> RFC5321 equates it only to Received.  RFC5322 uses it to describe a

RFC 5321 also defines Return-Path, of course.  As both definitions are
given in Section 4.4, one could derive that they are both "Trace
Information".

---

From: Nico Williams <[email-masked]>
Date: Tue, 17 Jan 2012 14:44:49 -0600

>   information at the beginning of the message content, as discussed in

I read 4.4 as implying that "Received" (a.k.a. "time stamp") is an
example of known trace headers.

I read 4.1.1.4 as implying that "Received" is the only then-known trace header.

I don't read either as excluding other trace headers -- it'd be quite
a stretch to do so.

At most we have an erratum against RFC5321, but a need to update it?
just for this?  IMO: no need to update RFC5321.

Nico
--

---

From: "John Levine" <[email-masked]>
Date: 18 Jan 2012 02:10:50 -0000

I wrote a draft to add a trace column to the existing header field
registry.  This is mostly about the registry; SM's draft (which I
didn't know he was writing) is more about what trace fields are.

R's,
John

------------------- snippo -----------------
A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : A Registry for Mail Trace Fields
	Author(s)       : Taughannock Networks
	Filename        : draft-levine-trace-header-registry-00.txt
	Pages           : 4
	Date            : 2012-01-17

   SMTP mail software adds trace header fields to messages as they pass
   through the mail system.  This document updates the definition of
   trace header fields, and adds trace field information to the
   registries for Permanent Message Header Fields and Provisional
   Message Header Fields.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-levine-trace-header-registry-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-levine-trace-header-registry-00.txt

---

From: Ned Freed <[email-masked]>
Date: Tue, 17 Jan 2012 18:20:43 -0800 (PST)

> didn't know he was writing) is more about what trace fields are.

Would it make sense to you and SM to combine them?

				Ned

---

From: S Moonesamy <[email-masked]>
Date: Tue, 17 Jan 2012 21:09:41 -0800

Hi John,
>didn't know he was writing) is more about what trace fields are.

Yes.  I wrote about Trace fields in response to the latest 
discussions on this list.  draft-moonesamy-mail-trace-fields-00 
provides some background information about Trace fields, hence the 
reference to RFC 4871 (obsolete).   BTW, I already know about the 
invalid RFC822 reference.  The draft does not mention the registry 
part as you mentioned that you might work on that.

I only covered mail-related specifications mentioning Trace fields up 
to "VBR-Info" as they are related to the DKIM work.  RFC 5598 is not 
mentioned as it uses RFC 2505 (antispam) to reference "trace information".

I don't feel strongly about the question of Trace fields.  I'll 
comment on draft-levine-trace-header-registry-00 as we looked at the 
topic from different viewpoints.  Section 3.6 of RFC 5322 discusses 
about "trace header field" and has a requirement for not reordering 
them and a recommendation for keeping them in blocks (see issues in 
Section 6 of draft-moonesamy-mail-trace-fields-00).  Section 3.1 of 
draft-levine-trace-header-registry-00 keeps the requirement and drops 
the recommendation.  Such a change would generate some on-list discussion.

My preference is to use "trace information" to discuss about 
documenting the progress of a message through the e-mail system 
(second paragraph of Section 3.1 of your draft).  Section 3.6 of RFC 
5322 mentions that:

   "the trace header fields and resent header fields MUST NOT be reordered"

I'll defer to the editor of RFC 5322 as to what exactly is a Trace 
field as it is not clear given the text quoted above and what's in 
Section 3.6.7.

As a reply to Ned's question, I am ok with combining the two drafts 
if John thinks that the material is appropriate for the topic he is tackling.

Regards,
S. Moonesamy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Jan 2012 22:23:30 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ned Freed
> Sent: Tuesday, January 17, 2012 6:21 PM
> To: John Levine
> Cc: [email-masked]
> Subject: Re: [apps-discuss] Draft for trace fields registry
> 

I like the idea of doing all of the "trace header field" cleanup work we're talking about in a single document.

However, all of this seems orthogonal to the processing of the "received-state" draft, because I still think tagging Received fields in that way is the right thing to do.

---

From: John C Klensin <[email-masked]>
Date: Wed, 18 Jan 2012 17:35:17 -0500

Hi.

I had a major system failure on Sunday, may have lost some hours
of email, and, while I got an alternate server up early Monday,
I have been only only slowly digging out.  The discussion seems
to have progressed significantly after I dropped out, including
a couple of new I-Ds.  While I hope those could be consolidated
to the degree possible, I think the idea of getting these things
registered is just great.

It may still be worth clarifying the point/suggestion I was
trying to make.

First, my concern was that loading too much disparate
information into "Received:", and especially intermixing
post-delivery MUA information with in transit MTA information
might not be a sufficiently good idea that we should do it
automatically and without thinking.   I didn't say "don't do
it", especially wrt this specific proposal.  I did suggest that
it was worth examining the question.

Second, please remember that, while RFC 5321 requires RFC
5322-conforming date-time fields, the <daytime> production of
RFC 821 is a little different.   The 821 syntax is not quite a
proper subset of the 5321/5322 syntax (e.g., seconds are
required in 821 and not in 5321/5322, 821 permitted two-digit
years, etc.).  Both 821 and 5321 require that both FROM and BY
appear, but I've seen that rule broken innumerable times.  5321
permits "TCP-info" to be supplied as a comment to BY (in
<Extended-domain>); 821 does not.  I've also seen liberties
taken with the specific format of the <date-time> (or
<datetime>) fields, especially by systems on the other side of
gateways that, e.g., use ISO dates (both 821 and 5321 warn
against (5321 says "MUST NOT") later systems messing with those
fields to try to "correct" them).  The multiple-path FOR form is
still seen in the wild, despite the security issues and language
in 5321 deprecating it. If multiple paths or mailboxes are used
with FOR, the beginning of the [Additional-Registered-Clauses]
becomes ambiguous, at least in theory.

I have no doubt that Ned, or anyone else even half as competent,
can sort this out and has implementations that do so.  I believe
that any MUA that is really as robust as 5322 MUAs are supposed
to be (indeed required to be to deal with other common erroneous
practices) can get it right.  But we see intermediate MTAs and
post-delivery filters trying to check these fields for bogus
information as part of spam assessments, and some of those are
not too bright.  

So what I was suggesting, and continue to suggest, is that it
may be time to think about whether it is better to keep adding
fields to "Received:" or whether one or more additional header
fields are in order.  Clearly the disadvantage of additional
ones is that something that treats anything appearing between
the first and last "Received:" header fields other than more of
them as bogus would get into trouble.  The advantage is that
something that wants to find one of these things for processing
purposes would know exactly where to look, rather than having to
parse "Received:" fields, looking for the relevant information,
and trusting that someone hasn't used the same
<Additional-Registered-Clauses> for something else (despite the
SHOULD NOT) in 5321 Section 4.4).

If we concluded that additional header fields, with different
names, were appropriate, it might still be the case that the
current "received-state" proposal would belong within
"Received:".  I'm just suggesting that we ask the question.

     john

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 18 Jan 2012 15:20:45 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of John C Klensin
> Sent: Wednesday, January 18, 2012 2:35 PM
> First, my concern was that loading too much disparate information into
> "Received:", and especially intermixing post-delivery MUA information
> with in transit MTA information might not be a sufficiently good idea that we should do it
> it", especially wrt this specific proposal.  I did suggest that it was
> worth examining the question.

I had thought there was objection to doing the received-state work by using Additional-Registered-Clauses, when my own MTA implementation experience suggests to me that it's actually ideal and that a new header field wouldn't be right for this.  Rather, what I'm seeing now is just an observation that the whole "trace field" thing is a little under-defined and we need to fix that.  I can buy that if we agree the two topics are in fact separate.

And an interesting observation: When I wrote RFC5451, my goal was to call Authentication-Results a trace field specifically because preserving the order of addition of Received fields with this mixed in is an important hint to figuring out where, and perhaps why, message authentication work ran into problems.  So I only wanted to piggyback on the "don't reorder or change this" rule that had already been laid out rather than re-stating it.  However, it also has an end-user purpose, which you correctly called post-delivery MUA information.  That part isn't exactly trace information per se, or maybe it is but it's actually MUA-actionable.  So that header field is actually a mix of both types of header field, I suppose: Agents should all follow trace header field handling rules, but it actually contains actionable stuff.  I'm hoping it doesn't thus deserve its own hybrid category...

One reason I prefer a Received extension for the received state work is because moving it into its own header field requires repetition of a bunch of information (hostname and date, for starters) that's already present in a corresponding Received field.  Thus, it seems it would add more clutter than it's worth to go that route.

> So what I was suggesting, and continue to suggest, is that it may be
> time to think about whether it is better to keep adding fields to
> "Received:" or whether one or more additional header fields are in
> order.

I think perhaps the amalgamated draft we seem to see forming between John's and SM's proposals might do well to talk about situations where each approach (Additional-Registered-Clause vs. new header field) is more appropriate.

> The advantage is that something that wants to find one of these things
> for processing purposes would know exactly where to look, rather than
> having to parse "Received:" fields, looking for the relevant
> information, and trusting that someone hasn't used the same
> <Additional-Registered-Clauses> for something else (despite the SHOULD
> NOT) in 5321 Section 4.4).

So far since the parsing of Received is mostly 1*(key value) up until the date part, parsing hasn't been a problem for systems I've worked on.  Most systems appropriately ignore things they don't understand.  However, naturally, I don't claim to speak for anything more than a fraction of the Received parsers out there.

-MSK

---

From: John C Klensin <[email-masked]>
Date: Wed, 18 Jan 2012 19:16:02 -0500

--On Wednesday, January 18, 2012 15:20 -0800 "Murray S.
Kucherawy" <[email-masked]> wrote:

>> it", especially wrt this specific proposal.  I did suggest
>> that it was worth examining the question.

> I had thought there was objection to doing the received-state
> work by using Additional-Registered-Clauses, when my own MTA
> implementation experience suggests to me that it's actually
> ideal and that a new header field wouldn't be right for this.
> Rather, what I'm seeing now is just an observation that the
> whole "trace field" thing is a little under-defined and we
> need to fix that.  I can buy that if we agree the two topics
> are in fact separate.

First, this is exactly the discussion I wanted to encourage.
There is a huge difference, IMO, between "MTA implementation
experience suggests to me that it's actually ideal and that a
new header field wouldn't be right for this" and "let's push
this into "Received:" because "Received:" is there.  

And, yes, the topics are mostly -- but not entirely-- separate.

> And an interesting observation: When I wrote RFC5451, my goal
> was to call Authentication-Results a trace field specifically
> because preserving the order of addition of Received fields
> with this mixed in is an important hint to figuring out where,
> and perhaps why, message authentication work ran into
> problems.  So I only wanted to piggyback on the "don't reorder
> or change this" rule that had already been laid out rather
> than re-stating it.  However, it also has an end-user purpose,
> which you correctly called post-delivery MUA information.
> That part isn't exactly trace information per se, or maybe it
> is but it's actually MUA-actionable.  So that header field is
> actually a mix of both types of header field, I suppose:
> Agents should all follow trace header field handling rules,
> but it actually contains actionable stuff.  I'm hoping it
> doesn't thus deserve its own hybrid category...

While I share your hope that it does not, the other line that is
being crossed here is between "used primarily for debugging" (a
phrase that, if I recall, appears both in 5321 and 821) in
contexts that imply "mostly  parsed and interpreted by humans"
and "actionable by MUAs" (which implies a lot of parsing and
interpretation by machines).  The tradeoffs may still favor
putting everything in "Received:" header fields, or my favor
splitting it up in some way.  I actually don't have a position
on the subject; I am only asking that we discuss it rather than
dropping into an accidental default.

> One reason I prefer a Received extension for the received
> state work is because moving it into its own header field
> requires repetition of a bunch of information (hostname and
> date, for starters) that's already present in a corresponding
> Received field.  Thus, it seems it would add more clutter than
> it's worth to go that route.

Makes sense.

>> may be time to think about whether it is better to keep
>> adding fields to "Received:" or whether one or more
>> additional header fields are in order.
> 
> I think perhaps the amalgamated draft we seem to see forming
> between John's and SM's proposals might do well to talk about
> situations where each approach (Additional-Registered-Clause
> vs. new header field) is more appropriate.

If that happened, my concern and, for this issue, interest would
shift entirely to the discussion of that draft and away from
your specific draft.
 
>> The advantage is that something that wants to find one of
>> these things for processing purposes would know exactly where
>> to look, rather than having to parse "Received:" fields,
>> looking for the relevant information, and trusting that
>> someone hasn't used the same <Additional-Registered-Clauses>
>> for something else (despite the SHOULD NOT) in 5321 Section
>> 4.4).
> 
> So far since the parsing of Received is mostly 1*(key value)
> up until the date part, parsing hasn't been a problem for
> systems I've worked on.  Most systems appropriately ignore
> things they don't understand.  However, naturally, I don't
> claim to speak for anything more than a fraction of the
> Received parsers out there.

Ack.

    john

---

From: SM <[email-masked]>
Date: Wed, 18 Jan 2012 16:19:58 -0800

Hi Murray,
At 15:20 18-01-2012, Murray S. Kucherawy wrote:
>And an interesting observation: When I wrote RFC5451, my goal was to 
>call Authentication-Results a trace field specifically because 
>preserving the order of addition of Received fields with this mixed 
>in is an important hint to figuring out where, and perhaps why, 
>message authentication work ran into problems.  So I only wanted to 
>piggyback on the "don't reorder or change this" rule that had 
>already been laid out rather than re-stating

Several other specifications have followed the piggyback path.  In my 
opinion, the question is worth discussing as it is not clear whether 
it is an acceptable practice.

>  it.  However, it also has an end-user purpose, which you correctly 
> called post-delivery MUA information.  That part isn't exactly 
> trace information per se, or maybe it is but it's actually 
> MUA-actionable.  So that header field is actually a mix of both 
> types of header field, I suppose: Agents should all follow trace 
> header field handling rules, but it actually contains actionable 
> stuff.  I'm hoping it doesn't thus deserve its own hybrid category...

It's easier to pretend not to see this part instead of trying to come 
up with a hybrid category. :-)

>I think perhaps the amalgamated draft we seem to see forming between 
>John's and SM's proposals might do well to talk about situations 
>where each approach (Additional-Registered-Clause vs. new header 
>field) is more appropriate.

I haven't discussed the approach to take with John Levine.  I don't 
have a preference at the moment.

Regards,
-sm

---

From: SM <[email-masked]>
Date: Wed, 18 Jan 2012 15:58:46 -0800

Hi John,
>of email, and, while I got an alternate server up early Monday,

I hope that everything is fine now.

>registered is just great.

I sent John Levine the XML.  There may be a few controversies down 
the road as the topic touches both RFC 5321 and 5322.

>becomes ambiguous, at least in theory.

That was issue #37.

>information as part of spam assessments, and some of those are

Yes.

I tested the example in draft-kucherawy-received-state with some CPAN 
code and it did not cause any problems.  I don't know the effects on 
legacy code.  I vaguely recall coming across a parser bug a few years 
back when parsing a "Received:" header field inserted by a MTA implementation.

Regards,
-sm

---

From: Ned Freed <[email-masked]>
Date: Thu, 19 Jan 2012 03:44:01 -0800 (PST)

> it was worth examining the question.

John, while I agree that adding post-delivery information to Received:
fields would be bad, I'm completely failing to see how the current proposal does
anything of the sort. It's entirely about what sort of queue the 
message ends up in and all of the cases clearly correspond to states
prior to final delivery.

If you think a rule about not adding MUA stuff to Received: fields needs
to be stated explicity, the place for that is either in RFC 5321bis or in
some sort of clarifying document. It makes no sense for it to be part of
the present proposal.

> becomes ambiguous, at least in theory.

And once again I'm failing to see any relevance. Yes, parsing Recieved: fields
in the wild has some complications - in fact one of the complictions is the use
of nonstandard clauses, which like it or not are present in the wild.  There's
nothing we can do at this point to change any of this. But the present proposal
is to add a clause with a single keyword arugment. I fail to see a parsing
problem in that.

> information as part of spam assessments, and some of those are

And those implementations are already failing on real world messages. I'm
not seeing how this is going to have significant added impact.

> SHOULD NOT) in 5321 Section 4.4).

Again, the problems with Received: field parsing pale in comparison with the
problems associated with having multiple order-dependent interrelated trace
fields. Agents that reorder multiple occurances of the same field are rare,
agents that lose or are incapable of "seeing" the relative order of multiple
fields, not so much.

Sieve is actually an example of this. Sieve can perform tests on all occurances
of a given field or it can be restricted to look at the first, second, Nth,
N-1th, etc. occurance. So as long as you have some idea of how many Received:
fields are added prior to delivery in a given environment (and you usually do),
you can write tests that check the correct field or fields and don't get
confused by stuff provided by stuff outside the administrative domain (which is
inherently unreliable).

But there's no way to test something like "the Foo: field that appears after
the Nth Bar: field". And an extension that does that is going to be very, very
ugly, no matter how you slice it. And that ugliness is going to make using such
an extension quite problematic, even for fairly experienced script
constructors.

Again, we went through all of this stuff in tedious detail back when MIME
header internationalization was being standardized. The encoded-word propsoal
certainly has had issues (mostly poor quality implementation - for some reason
people just don't seem to get that certain activities have to be performed in a
specific order), but it was small beer compared to the problems we would have
had with multiple interconnected fields.

> "Received:".  I'm just suggesting that we ask the question.

And my answer, I'm afraid, is still a very emphatic NO.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 19 Jan 2012 11:51:01 -0500

--On Thursday, January 19, 2012 03:44 -0800 Ned Freed
<[email-masked]> wrote:

> John, while I agree that adding post-delivery information to
> Received: fields would be bad, I'm completely failing to see
> how the current proposal does anything of the sort. It's
> entirely about what sort of queue the  message ends up in and
> all of the cases clearly correspond to states prior to final
> delivery.

I thought I was clear in my initial note.  If I wasn't, I
apologize.  Let me try to say it differently:

It is reasonable to hypothesize that there is some information
that people might want to capture between when the message
leaves the Submission server and when it arrives on the user's
screen-equivalent that should not be loaded onto Received
fields.  One can make the argument that some things should be in
different header fields from "Received:" on the basis of parsing
idiosyncrasies associated with the changing definition of
"Received:" over the years (which I overstressed), concerns
about arbitrarily-long header fields (as with parsing, more a
concern for lower-quality implementations than high-quality
ones), precedent, or aesthetics, but the hypothesis remains.
My guess is that at least part of the dividing line will work
out to be associated with states before or after final delivery,
but that was an example, not a position.  Perhaps, for example,
some authentication information should be separated out as well.
Perhaps not.

I believe that we should take that issue up and start developing
a theory and guidelines before adding more stuff to "Received:".
I believe that any WG that is going to take up specifications
that propose adding "Received:" clauses should be obligated to
address this more philosophical boundary question about what
belongs there and, as has been pointed out, the issue of an
appropriate registry for such clauses.

I've expressed absolutely no opinion about whether the data
elements proposed in this particular specification belong in
"Received:" or not.  My guess is that they probably do.   If
that opinion is generally shared, I don't see any problem moving
ahead with this specification _as long as a serious effort is
underway in parallel to address the broader issue.

> If you think a rule about not adding MUA stuff to Received:
> fields needs to be stated explicity, the place for that is
> either in RFC 5321bis or in some sort of clarifying document.
> It makes no sense for it to be part of the present proposal.

Never suggested it should be.  All I've suggested is that, if
AppsAWG is going to take up this sort of proposal, it should be
prepared  to take on that classification question --more broadly
stated than "MUA stuff in 'Received:'" as well.   I haven't even
said "solve the general problem before addressing the specific
proposal".

>...
> And once again I'm failing to see any relevance. Yes, parsing
> Received: fields in the wild has some complications - in fact
> one of the complictions is the use of nonstandard clauses,
> which like it or not are present in the wild.  There's nothing
> we can do at this point to change any of this. But the present
> proposal is to add a clause with a single keyword arugment. I
> fail to see a parsing problem in that.

See above.

>> I have no doubt that Ned, or anyone else even half as
>> competent, can sort this out and has implementations that do
>> so.  I believe that any MUA that is really as robust as 5322
>> MUAs are supposed to be (indeed required to be to deal with
>> other common erroneous practices) can get it right.  But we
>> see intermediate MTAs and post-delivery filters trying to
>> check these fields for bogus information as part of spam
>> assessments, and some of those are not too bright.
> 
> And those implementations are already failing on real world
> messages. I'm not seeing how this is going to have significant
> added impact.

Again, with the understanding that what I'm saying is "please
can we examine the tradeoffs" and not "don't do this", let me
suggest where significant added impact might occur.  Suppose I
have a moderately fragile real world implementation that is not
now failing because it was written on the assumption that
"Received:" fields are used for debugging information and it
doesn't actually have to parse them at all.  If it does
minimally parse them -- e.g., for loop detection purposes (which
mostly requires only that it recognize "Received:" header fields
that it might have added -- its belief in the "debugging" model
is such that it can and does simply ignore any "Received:"
header field that confuses it.   Now we come along and put
information in the "Received:" field to which it actually needs
to pay attention.   Its sloppy parsing and willingness to
discard confusing ones keeps it from getting to that information.

That actually does have significant added impact relative to
having the information in a separate header field that can be
analyzed without having to first parse "Received:" into its
components.  Of course, whether that "significant added impact"
is worse (because the weak implementation can't find the
information) or actually an improvement (because that
implementation might be encouraged to clean up its act) is one
of those tradeoffs I've been talking about.  As you know from
experience, I'm inclined to favor the "get them to clean up
their act" plan for such situations.  But I also recognize that
it sometimes works poorly enough to believe there is a tradeoff
that ought to be considered.

But, again, "considered" and "tradeoffs" arsn't "I think there
is anything wrong with this proposal".  I just want to see us
ask the question in a serious way.

> Again, the problems with Received: field parsing pale in
> comparison with the problems associated with having multiple
> order-dependent interrelated trace fields. Agents that reorder
> multiple occurances of the same field are rare, agents that
> lose or are incapable of "seeing" the relative order of
> multiple fields, not so much.
>...

Again, this (including the Sieve issue) is exactly the
discussion of tradeoffs I was hoping to encourage.  I'm not as
convinced as you are that the issues were settled with MIME i18n
and encoded words, but that is another matter.  And it seems to
me that the Sieve example argues against even using different
header fields/names for post-final-delivery information, at
least without a degree of hair-splitting about the definition of
final delivery  that we have so far managed to avoid.  If the
answer turns out to be "No, let's keep putting everything into
'Received:'", I have no problem with that.

    john

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 20 Jan 2012 18:39:12 +0100

(Subject changed according to topics separation)

On 19/Jan/12 12:44, Ned Freed wrote:
> Sieve is actually an example of this. Sieve can perform tests on
> all occurances of a given field or it can be restricted to look at
> the first, second, Nth, N-1th, etc. occurance. So as long as you
> have some idea of how many Received: fields are added prior to
> delivery in a given environment (and you usually do), you can write
> tests that check the correct field or fields and don't get confused
> by stuff provided by stuff outside the administrative domain (which
> is inherently unreliable).
> 
> But there's no way to test something like "the Foo: field that
> appears after the Nth Bar: field". And an extension that does that
> is going to be very, very ugly, no matter how you slice it. And
> that ugliness is going to make using such an extension quite
> problematic, even for fairly experienced script constructors.

Good point.  However, we already have that situation: see e.g.
http://www.rfc-editor.org/errata_search.php?rfc=5451&eid=2818
(Hey, I did get Murray's approval before posting that :-)

Perhaps, "the Foo: field of the Nth trace stanza" is easier.  We
should just convene that "Received:" fields delimit trace stanzas.

Of course, all the RFCs that mention "trace header fields" say that
these go at the top of the header.  Some of them explicitly mention
"Received:" (quotes below), but also RFC 5322 and RFC 5518 obviously
imply such positioning if we assume that "Received:" is always
present.  Thus, it seems feasible to introduce this concept, in one of
the trace- I-Ds, perhaps?

Quotes (emphasis added):
RFC 4408:
 The Received-SPF header field is a trace field (see [RFC2822] Section
 3.6.7) and SHOULD be prepended to the existing header, above the
 *Received*: field that is generated by the SMTP receiver.

RFC 5436:
                                 If the implementation retains the
 "Received" fields from the triggering message (see above), the "Auto-
 Submitted" field MUST be placed above those "*Received*" fields,
 serving as a boundary between the ones from the triggering message
 and those that will be part of the notification message.

RFC 5451:
     On the presumption that internal MTAs are fully compliant with
     Section 3.6 of [MAIL], and the compliant internal MTAs are using
     their own host names or the ADMD's DNS domain name as the
     "authserv-id" token, the header field proposed here should always
     appear above a *Received*: header added by a trusted MTA.

RFC 6376:
  INFORMATIVE IMPLEMENTATION NOTE: The easiest way to achieve this
  is to insert the DKIM-Signature header field at the beginning of
  the header block.  In particular, it may be placed before any
  existing *Received* header fields.  This is consistent with treating
  DKIM-Signature as a trace header field.