Subject: Re: [apps-discuss] Review of draft-melnikov-smtp-priority-14

From: Alexey Melnikov <[email-masked]>
Date: Fri, 01 Jun 2012 11:28:01 +0100

On 31/05/2012 08:49, Dave Crocker wrote:
> This review is at my own initiative.

Dave,
Thank you for the review.

I've applied most of your editorial changes (and spelling fixes) as is. 
I've deleted them from my reply.
I will reply to different parts of your review in separate messages.

>> 2.  Conventions Used in This Document
>>
>>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>>    document are to be interpreted as described in [RFC2119] when they
>>    appear in ALL CAPS.  These words also appear in this document in
>>    lower case as plain English words, absent their normative meanings.
>
> "when they appear in ALL CAPS".  sigh. In other words, this document 
> is modifying RFC2119.
>
> Does anyone seriously expect this new nuance of usage to be read and 
> remembered by average readers of this document?

> FWIW, it took me about 3 minutes to make the relevant changes, below. 
> Case-sensitivity in semantics invites comprehension problems here. 
> Worse, there is absolutely no need or benefit in creating the problem 
> for this document.  At a minimum, please do not try to modify IETF 
> document writing policy on the fly.

I will let IESG weight in on this. I've implemented what I was told 
earlier. I don't have a strong opinion on this.

>> 4.1.  Handling of the MT-PRIORITY parameter by the receiving SMTP server
>>
>>    The following rules apply to SMTP transactions in a server that
>>    supports the MT-PRIORITY parameter:
>>
>>    1.  A conforming SMTP server MUST NOT refuse a MAIL FROM command
>>        based on the absence of the MT-PRIORITY parameter.
>
>
> hmmm.   For some operational environments, it will be essential to 
> have /all/ handling conform to the priority policy model in force for 
> the environment.  In those cases, the absence of the parameter would 
> be a violation of the spec.

Good point. Deleted.

>>
>>    5.  If no priority has been determined by the above, the server may
>>        use its normal policies to set the message's priority.  By
>>        default, each message has priority 0.
>
> ouch.  This chooses a particular model for meaning of the numbers, and 
> without having defined the model beforehand.
>
> That is, I think the problem with this default is that it really is 
> assuming a particular policy for meaning of the values, namely that 
> average mail is mid-priority, assuming the numbers have linear semantics.

Why is this a problem (commenting on the last sentence quoted above)?

>>    however, allow an untrusted source to lower its own message's
>>    priorities -- consider, for example, an email marketer that
>>    voluntarily sends its marketing messages at low priority.
>
> To beat the point a bit deader:  this assumes a particular policy for 
> the meaning of the priority values.  Worse, it also appears to 
> contradict the earlier default of 0, but perhaps I've misunderstood that.

Yes, you misunderstood. The example talking about marketing messages is 
an example of a sender that explicitly requests priority below 0.

>>    450 or 550 reply code.  The corresponding enhanced status code MUST
>>    be X.7.TBD1 [RFC2034] if the determined priority level is below the
>>    lowest priority currently acceptable for the receiving SMTP server.
>>    Note that this condition might be temporary, in cases where the
>>    server is temporarily operating in a mode where only high priority
>>    messages are accepted for transfer and delivery.
>
> Given a range of 0-9, what qualifies as high priority and what 
> qualifies as low?
>
> Perhaps:
>
>    Note that this condition might be temporary.  In some environments, 
> operational policies might permit periods of operation that relay only 
> higher priority messages and defer lower priority ones.  Such handling 
> choices need to be specified for that operational environment..

Ok. I used a slightly modified version of your suggested text.

>> 4.4.  Mailing lists and Aliases
>>
>>    Several options exist to translate the address of an email recipient
>
> "options"?  Perhaps you mean mechanisms or services?

Yes.

> And they really aren't translating an address but are doing a form of 
> message transmission (relaying, or the like).

Suggested replacement?

>>    into one or more other addresses.  Examples for this are aliases and
>>    mailing lists [RFC5321].
>>
>>    If a recipient address is to be translated and/or expanded when
>>    delivered to an alias or a mailing list, the translating or expanding
>>    entity (MTA, etc.)  SHOULD retain the MT-PRIORITY parameter value for
>>    all expanded and/or translated addresses.
>
> This appears to expect the extension's semantics to survive delivery 
> and re-posting via a mailing list.  Remember that a mailing list posts 
> a /new/ message.  Offhand, this requirement seems to go considerably 
> beyond normal SMTP options and beyond most Internet Mail standards 
> services...

This is the same as for the DSN SMTP extension. So no, this is not the 
first time.
>
> At the least, this needs considerably more extensive and careful 
> documentation that it is given here.
>
>
>> 4.5.  Gatewaying a message into a foreign environment
>>
>>    The following rules govern the behavior of a conforming MTA, when
>>    gatewaying a message that was received via the SMTP protocol, into a
>>    foreign (non-SMTP) environment:
>>
>>    1.  If the destination environment is unable to provide an equivalent
>>        of the MT-PRIORITY parameter, the conforming MTA SHOULD behave as
>>        if it is relaying to a non-conformant SMTP server (Section 4.3).
>
> Given the variable semantics that apply here -- per the different 
> policy possibilities -- what does it mean to be an equivalent of the 
> MT-PRIORITY parameter, in specific technical terms?

The point is that if I am trying to gateway to the Royal Pigeon Mail 
Service and it doesn't support anything similar to priorities at the 
protocol level, then this is treated as relaying to a non-conformant 
SMTP server.

>>    2.  If the destination environment is capable of providing an
>>        equivalent of the MT-PRIORITY parameter, the conforming MTA
>>        SHOULD behave as if it is relaying to a conformant SMTP server
>>        (Section 4.2), converting the MT-PRIORITY value to the equivalent
>>        in the destination environment.
>
> Is this really enough technical and operational detail to cover 
> gatewaying?  Perhaps it is, but it feels to me as if it isn't.  Then 
> again, I don't know what more/else to suggest.

I don't know either :-).

>> 4.6.  Interaction with DSN SMTP Extension
>>
>>    An MTA which also conforms to [RFC3461] SHOULD apply the same
>>    priority value to delivery reports (whether for successful delivery
>>    or failed delivery) it generates for a message.
>
> In many operational environments, control messages get higher priority 
> (or lower priority) than payload messages.

What is a control message?

Any suggested text?

>>    Note that rejections based on priority (see Section 4.1) or priority+
>>    message size combination SHOULD only be done by an MSA (i.e. they
>>    SHOULD NOT be done by MTAs/MDAs), because the MSA has a link to the
>
> This presumes that an MSA knows the priority-related policies of 
> downstream MTAs and MDAs.

I don't think so. Why do you say that?

>>    Mail User Agents (MUAs) which generated the message and is in a
>>    position to perform a corrective action, such as resubmission of the
>
> resubmission with lower priority is a big deal but seems to get a 
> rather casual reference here.
>
>
>>    message with lower priority, converting or truncating the message to
>>    make it smaller, etc.  Such actions usually require user interaction.
>>    For this reason it is also important for MUAs to support enhanced
>>
>>
>
>>    with a lower one.  Additionally, the retry interval and/or default
>>    timeout before non-delivery report is generated can be lower (more
>>    aggressive) for messages of higher priority.
>
> oh?  "can be"?

Should have been "MAY be".

> This sort of language looks like it is specifying something but it 
> isn't.  And the reference to what "can be" carries no cautions or 
> directions meaningful to implementers.
>
>
>>    Note that as this SMTP extension requires some sort of trust
>>    relationship between a sender and a receiver and thus some for of
>
>    for -> form  (?)

Yes.

> This is treated as a side comment (by introducing it with "Note") but 
> I believe it is in fact a core predicate for the extension.
>
>
>>    authentication (whether using SMTP AUTH, TLS, IP address whitelist,
>>    etc.), so senders using this SMTP extension will not be subject to
>>    greylisting [GREYLISTING], unless they are unauthorized to use this
>>    SMTP extension, due to an explicit policy decision or a
>>    misconfiguration error.
>>
>>    In order to make implementations of this extension easier, this SMTP
>>    extension only allows a single priority for all recipients of the
>>    same message.
>
> This is quite a good and clear simplification point.  However note 
> that a model which permits selective support of priority values winds 
> up going counter to that goal of simplification.

Well, simplification doesn't apply to everything in this extension. Some 
tradeoffs were made....

>>
>> 5.2.  Timely Delivery
>>
>>    An important constraint (usually associated with higher priority
>>    levels) is that messages with high priority have some delivery time
>>    constraints.  In some cases, higher priorities mean a shorter maximum
>>    time allowed for delivery.
>>
>>    Unextended SMTP does not offer a service for timely delivery.  The
>>    "Deliver By SMTP Service Extension" (DELIVERBY Extension) defined in
>>    [RFC2852] is an example of an SMTP extension providing a service that
>>    can be used to implement such constraints.  But note that use of the
>>    DELIVERBY extension alone does not guarantee any priority processing.
>
> It seems as if this section introduces an issue but does not actually 
> deal with it.  Perhaps there should be some discussion of the possible 
> (or required?) interaction between the two extensions it discusses?

There is no issue and no real interaction in this specific case. A 
client that wants to use both against a server that supports both should 
consider this issue.

>> 8.  Deployment Considerations
>>
>>    If multiple DNS MX records are used to specify multiple servers for a
>>    domain in section 5 of [RFC5321], it is strongly advised that all of
>
> "strongly advised"?
>
> This seems the sort of thing that needs normative language yet it is 
> carefully avoided.  That is, by saying 'strongly' the issue is moved 
> towards the asking why it is not stated normatively.  I'd guess this 
> needs a SHOULD, possibly with discussion of permissible exceptions 
> (such as the open Internet...?)

This section applies to system administrators. It is quite different 
from the rest of the document which contains mostly protocol level 
requirements. So yes, I avoided usage of RFC 2119 keywords here on purpose.
Suggestions for alternatives are welcomed.

> However note that the 'strongly advised' presumes instantaneous 
> implementation on all hosts within the trust environment.
> Since that isn't possible, it's not clear how the advice of this 
> section is practical.

Nothing is instantaneous, so I am not very concerned ;-).

Once some servers start supporting this extension a sysadmin can remove 
MXes for servers which don't support it. Upgrading all servers nearly 
simultaneously is another option.

>> 10.  Security Considerations
>>
>>    Message Submission Agents MUST implement a policy that only allows
>>    authenticated users (or only certain groups of authenticated users)
>>    to specify message transfer priorities, and MAY restrict maximum
>>    priority values different groups of users can request, or MAY
>>    override the priority values specified by MUAs.
>
> Presumably the normative MSA language is meant "for those MSAs 
> supporting this extension"?

Of course. I don't think this needs saying.

---

From: Ned Freed <[email-masked]>
Date: Fri, 01 Jun 2012 09:18:43 -0700 (PDT)

A few comments are in order here.

> >    particularly important during emergencies for first responders and
> >    for environments such as military messaging, where messages have high
> >    operational significance, and the consequences of extraneous delay
> >    can be significant.
> >
> >    In order for an SMTP receiver to be able to send higher priority

>     send -> relay

Given the imprecise nature of this document overall, it's a stretch to believe
that being precise here is really necessary, but if it is, this fixes only part
of the problem. The message may have been received via SMTP, SUBMIT, or
something else. And the higher *or* *lower* priority may apply to whatever the
next step is: SMTP, LMTP, or something else.

If you want to be precise here, it needs to say something like:

      In order for an MSA, MTA, or MDA to be able to relay, deliver, or
      otherwise process higher priority messages first, ...

> >    messages first, there needs to be a mechanism to communicate (during
> >    both Message Submission [RFC6409] and Message Transfer [RFC5321]) the
> >    priority of each message.  This specification defines this mechanism
> >    by specification of an SMTP [RFC5321] extension.
> >
> >    Implementors of this document might also consider implementing
> >    [PRIORITY-TUNNELING] which talks about tunneling of Message Transfer
> >    Priority information through Message Transfer Agents (MTAs) that do
> >    not support this extension, using a new message header field
> >    [RFC5322].

> Consider replacing above paragraph with:

>     In order to permit end-to-end use of this extension across email
> infrastructure that does not support it, a companion tunneling mechanism
> is defined in [PRIORITY-TUNNELING] through use of a new message header
> field [RFC5322].

This is better. These specifications are not just for implementors.

> "when they appear in ALL CAPS".  sigh. In other words, this document is
> modifying RFC2119.

On the contrary, all this is doing is being specific about the way RFC 2119 has
actually been used ever since it was created. I can easily produce a long list
of RFCs containing instances of these words in lower case where, even though it
wasn't stated, the intent *clearly* was for lower case versions of these words
to have their regular meanings.

And even if I were to accept the notion that this document specifies a novel
derivation of RFC 2119 (the notion that it *modifies* it being patently
absurd), that ship has also sailed. I can also easily point to RFCs that do
things like specify ALLS CAPS versions of these words have one meaning, Mixed
Case another, and lower case yet another.

> remembered by average readers of this document?

The better question is does anyone seriously expect anyone to assume, given
abundant evidence to the contrary, that the lower case versions of these words
will be taken to have RFC 2119 meaning?

> document writing policy on the fly.

You're the one who is arguing for a modification to the *actual* IETF document
writing policy here.

> >    The formal syntax use the Augmented Backus-Naur Form (ABNF) [RFC5234]
> >    notation including the core rules defined in Appendix B of RFC 5234
> >    [RFC5234].
> >
> >    In examples, "C:" and "S:" indicate lines sent by the client and
> >    server respectively.  Line breaks that do not start with a new "C:"
> >    or "S:" exist for editorial reasons and are not a part of the
> >    protocol.
> >
> >    This document uses the term "priority" specifically in relation to
> >    the internal treatment of a message by the server: messages with
> >    higher priorities may be given expedited handling, and those with

>     may -> can

> >    lower priorities may be handled only as resources become available.

>     may -> can

Again, given the loosey-goosey nature of this document overall, I don't think
precision is really required, but if it is, both of these changes are somewhat
objectionable.

Like it or not, "may" and "can" are *not* synonyms. "Can" is more assertive,
and carries with it a flavor that's closer to "should", whereas "may" is
expresses a possibilit with no preference.

We're talking about prioritization methodology here, specifically expedited
handling and deferred processing. There can be overhead associated with both of
these, and we should not be encourage the use of these techniques when it isn't
absolutely necessary. 

And yes, I'm being very pedantic here. That's what you're going to get when
you elect to go down this path.

> >
> > 3.  Definition of the Priority SMTP Extension
> >
> >    The Priority SMTP service extension is defined as follows:
> >
> >
> >
> >
> > Melnikov & Carlberg     Expires November 25, 2012               [Page 3]
> > 
> > Internet-Draft  Message Transfer Priority SMTP Extension        May 2012
> >
> >
> >    1.  The textual name of this extension is "Priority Message
> >        Handling".
> >
> >    2.  The EHLO keyword value associated with this extension is "MT-
> >        PRIORITY".
> >
> >    3.  The EHLO keyword has no parameters.  Parameters are reserved for
> >        possible future extensions and MUST be ignored by clients that
> >        don't understand them.
> >
> >    4.  No additional SMTP verbs are defined by this extension.
> >
> >    5.  One optional parameter ("MT-PRIORITY") is added to the MAIL FROM
> >        command.  The value associated with this parameter is a decimal
> >        integer number from -9 to 9 (inclusive) indicating the priority
> >        of the email message.  The syntax of the MT-PRIORITY parameter is
> >        described by the <priority-value> ABNF non-terminal defined in
> >        Section 6.  Higher numbers mean higher priority.

> As a minor point, the form of -9 to 9 seems more complicated than
> useful.  Is there a need for 19 values?  I'd think that 0 to 9 would be
> sufficient.

Well, to be fair, the specification only requires support for six distinct
values:

   A message priority is a decimal integer in the range from -9 to 9
   (inclusive).  SMTP servers compliant with this specification are not
   required to support all 19 distinct priority levels (i.e. to treat
   each priority value as a separate priority), but they MUST implement
   at least the following 6 distinct priority levels: -4, -2, 0, 2, 4,
   9.  I.e. an implementation that only supports the 6 priority levels
   will internally round up a syntactically valid level that isn't
   supported to the next higher supported number.  For example, such an
   implementation will treat priority values below and including -4 as
   priority -4, priority -3 as priority -2, and all priorities starting
   from 5 are treated as priority 9.  An SMTP server MAY support more
   than 6 different priority levels.  Decision about which levels to
   support in addition to the 6 mentioned above is a local matter.

Now, I have no idea what it actually means to "support" six values, but I
assume it means that if you only make a processing distinction between five or
fewer values total, if you support only one "lower than normal" priority, or
you support fewer than three "higher than normal" priorities, you cannot
implement this specification. (Note I'm assuming 0 is "normal" here - see
below.)

Of course there's a trivial way around this: You simply state that, as a matter
of implementation policy, you only support X number and here's the mapping.
This would allow, say, a system that only does the old X.400 non-urgent,
normal, urgent thing to claim support.

> For that matter, why are many values needed?  What is the basis for
> choosing to have a range of more than a few values?

IMO this is one of the rare cases where X.400 was spot-on correct.

> This appears to be the only place the provides semantics to the priority
> number.  As such, it should be more thorough.  The critical issue in
> assigning the number, I think, is trying to get independent originators
> to assign numbers according roughly the same criteria.  For example, if
> I label "average" messages I send as as 0 and you label them as 1, then
> your messages get preferential treatment over mine, although that was
> not your intent.  (Assuming you're not trying to game the service...)

> This issue goes to the core of the usage model for the feature:  It
> really needs an environment tailored to its use, with all participants
> not only within the same trust system, but sharing the same priority
> assignment model/policy.  It's probably good for this document to permit
> a range of assignment models, but should note that an operational
> requirement for use of the extension is that an assignment policy be
> defined for all participants.

> This document might, then, offer a candidate model, to be use by default
> and absent one specific to an environment.

Well, this goes to the problem I have with the entire document - I have no real
idea what the policy associated with these priorities is supposed to be. We
have a prioritization capability in our product and I can tell you what it
does, but s to whether or not it satisfies the intended target of this
specification, I realy couldn't say.

More generally, in a modern, high performance MTA that's capable of processing
large numbers of messages simultaneously, there's a significant problem
actually implementing mechanisms that are guaranteed to preferentially process
certain messages.

For example, suppose I have several dozen threads/processes/whatever connected
up to a given system, all busy transferring large messages. A higher priority
message comes in. If I shove this message on the thread that is the closest to
being done, there may still be a significant delay. OTOH, if I start a new
connection to handle it, what if I hit the limit on the number of connections
the remote MTA will accept from me?

Put another way, if you assume a fairly strict policy is needed, your point
about this only being applicable in an environment that's specifically tailored
for it is, if anything, an massive understatement. For this to provide a real
priority boost with high reliability it's actually necessary to closely
coordinate operational policies across all involved ADMDs. And not to put too
fine a point on it, the level of technical acumen needed to actually do this
sort of thing is something I've observed to be the exception and not the rule.

All that said, I think this actually argues for putting in some statements
as to what this document does *not* do. It calls for best effort to 
provide preferential processing, nothing more. It does *not* provide
any sort of guarantees. 

And if that's not the case, then this needs to be experimental.

> >    6.  The maximum length of a MAIL FROM command line is increased by 15
> >        octets by the possible addition of a space, the MT-PRIORITY
> >        keyword and value.
> >
> >    7.  The MT-PRIORITY extension is valid for the submission service
> >        [RFC6409] and LMTP [RFC2033].
> >
> > 4.  Handling of messages received via SMTP
> >
> >    This section describes how a conforming SMTP server should handle any
> >    messages received via SMTP.
> >

> hmmm.   For some operational environments, it will be essential to have
> /all/ handling conform to the priority policy model in force for the
> environment.  In those cases, the absence of the parameter would be a
> violation of the spec.

Interesting point. I'm not sure what, if anything, to do about it.

> >    2.  If any of the associated <esmtp-value>s (as defined in Section
> >        4.1.2 of [RFC5321]) are not syntactically valid, or if there is
> >        more than one MT-PRIORITY parameter in a particular MAIL FROM
> >        command, the server MUST return an error, for example "501 syntax
> >        error in parameter" (with 5.5.2 Enhanced Status Code [RFC2034]).
> >
> >    3.  When inserting a Received header field as specified in Section
> >        4.4 of [RFC5321], the compliant MTA/MSA SHOULD include the
> >        "PRIORITY" clause whose syntax is specified in Section 6.
> >
> >
> >
> > Melnikov & Carlberg     Expires November 25, 2012               [Page 4]
> > 
> > Internet-Draft  Message Transfer Priority SMTP Extension        May 2012
> >
> >
> >    4.  If the sending SMTP client specified the MT-PRIORITY parameter to
> >        the MAIL FROM command, then the value of this parameter is the
> >        message priority.
> >

>     may -> can

> without having defined the model beforehand.

I for one was assuming 0 was the normal, default priority. But now that you
mention it, at least this needs to be made explicit.

> average mail is mid-priority, assuming the numbers have linear semantics.

That sure was what I was assuming.

> >    The SMTP server MUST NOT allow upgraded priorities from untrusted
> >    (e.g. unauthenticated) or insufficiently trusted sources.  (One
> >    example of an "insufficiently trusted source" might be an SMTP sender
> >    which authenticated using SMTP AUTH, but which is not explicitly
> >    whitelisted to use the SMTP MT-PRIORITY service.)  The server MAY,

> It is good that this issue is raised, but it really requires an earlier,
> separate and careful discussion of the need for this extension to be
> used within an administered trust environment.  Given such an
> introductory section, the above text would be sufficient.

Agreed.

> In its absence, the reference to a whitelist is based on the very large
> assumption that participants in the extension are whitelisted.  This is
> not document elsewhere.

True.

> contradict the earlier default of 0, but perhaps I've misunderstood that.

I think that's a misunderstanding.

I generally agree with the remainder of your comments, although I think some
of them become less important as long as it's clear this doesn't provide
guaranteees, but they still should be addressed.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Fri, 01 Jun 2012 21:15:39 +0200

> earlier. I don't have a strong opinion on this.

So we don't follow IETF consensus documents anymore for process and 
documentation conventions?  The IESG establishes those details at its whim?

> Why is this a problem (commenting on the last sentence quoted above)?

It's quite basic.

The document specifies a standard but does not specify it completely. 
First, it periodically relies on undocumented policy assumptions. 
Second, policies will vary from one environment to the next.

As I note in the review, proper operation of a QOS mechanism requires 
all participants to operate according to the same policies.  I gave the 
example of two posters meaning 'average priority' but happening to 
choose different values for that semantic.  To get proper system 
operation, you and I and the other guy all need to choose the same value 
for the same semantic.  That requires a definition of the semantic; in 
this case that means rules for when to assign specific values.

In effect, the current document is a syntax mechanism for expressing 
priority, but it doesn't really specify actual priority semantics for 
making assignments, although it does hint at a particular set of 
semantics/policies.

So the document is both incomplete and overly rigid, IMO.  That's why I 
suggested providing an explicit and complete policy section as a 
default, but designed to allow alternative policy sections for other 
environments.

> an example of a sender that explicitly requests priority below 0.

But there is no inherent meaning to "low priority", absent a policy 
statement that defines the meaning of values.  Low is relative to other 
values, but which ones?  In some environments -5 is low and in others 0 
is low and I'll bet some others will treat 5 as low.

>> message transmission (relaying, or the like).

      Several types of mechanisms exist to redirect or forward messages 
to alternative or multiple addresses.[RFC5598]  Examples for this are 
aliases and mailing lists [RFC5321].

      If a message is subject to such processing, the Mediator node 
[rfc5598, Sec 2.1] SHOULD retain the MT-PRIORITY parameter value for all 
expanded and/or translated addresses.

> What is a control message?

Chatter about handling activity is distinct from traffic comprising user 
payload (messages) even if a user is the recipient.  The chatter 
consists of control messages.  A delivery report is a control message.

> Any suggested text?

    Messages concerning email handling, such as delivery reports, 
constitute control information rather than direct message payload.  In 
some environments, control traffic is subject to higher priority 
handling and in others it receives the same priority as the message that 
caused the report to be generated.  Determining the priority to assign 
is a matter for the policy statements applying to the environment in 
which [RFC3461] is supported.

> I don't think so. Why do you say that?

A rejection means that the message is outside an acceptable range. 
That's a policy for the environment and it does not make sense for an 
MSA to have one set of policies and MTAs to have another.

> client that wants to use both against a server that supports both should

Specifications that say an implementer should consider something but 
which gives no guidance about the consideration aren't doing much 
useful, in my view.

As this set of text was proceeding, I thought it was going to provide 
some useful guidance about possible uses of the combined options, since 
it seemed like the combination could be, well, useful.

> requirements. So yes, I avoided usage of RFC 2119 keywords here on purpose.

I think that is merely confusing.  Normative direction to operators is 
an entirely reasonable part of a specification.  Since that's really 
what you are doing, go ahead and use officially normative language.

> Suggestions for alternatives are welcomed.

Just convert to the normative terms that are appropriate.

> Nothing is instantaneous, so I am not very concerned ;-).

So the document is strongly advising something that is impossible to 
achieve, during a transition period.  It's worth considering how to 
handle that.

> simultaneously is another option.

Is it practical?

> Of course. I don't think this needs saying.

And yet the document says exactly that sort of qualifier earlier: "An 
MTA which also conforms to [RFC3461]..."

d/

-- 
  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 01 Jun 2012 20:59:05 +0100

Hi Ned,

>      otherwise process higher priority messages first, ...
Yeah, I think this is less readable than what I have. I am sure readers 
understand the meaning of what is being said.
> This is better. These specifications are not just for implementors.
Yes, already used in -15.
> You're the one who is arguing for a modification to the *actual* IETF 

Agreeing with that.

> you elect to go down this path.
:-)
> implement this specification. (Note I'm assuming 0 is "normal" here - see
More or less, yes.
> normal, urgent thing to claim support.

Right. And there is already such a mapping in Appendix B.

> specification, I realy couldn't say.

I hope I clarified this in -16. Also look at Appendix A and Appendix B.

> provide preferential processing, nothing more. It does *not* provide

Fair enough, I will add some text about that. Best effort was certainly 
the intent.

> mention it, at least this needs to be made explicit.

I believe the document already says that, but I will double check.

Yes.

>> introductory section, the above text would be sufficient.

Ok.

>> assumption that participants in the extension are whitelisted.  This is

Hmm, Ok.

> I think that's a misunderstanding.

I already replied in another message that it is.

> guaranteees, but they still should be addressed.

I believe I addressed most of them in -15 and -16. I might need to add a 
couple of clarifying sentences as per above.

---

From: Ned Freed <[email-masked]>
Date: Fri, 01 Jun 2012 13:36:45 -0700 (PDT)

OK, let me take a look at -16 and I'll see what else, if anything, needs to be
said. (I was looking at -15.)

				Ned

---

From: Ned Freed <[email-masked]>
Date: Fri, 01 Jun 2012 16:56:51 -0700 (PDT)

> said. (I was looking at -15.)

Well, I looked, and now I'm even more confused. The statements about
implementations having to support at least six levels (with no indication
of what support means) are still there, but so are the Appendices about
mappings with far fewer levels.

So let me put this in a form of a question. Suppose I have an implementation
that is capable of storing and transferring all of the MT-PRIORITY levels,
but internally is only capable of doing the three X.400 levels. Would such an
implementation be able to offer this extension?

				Ned

P.S. The text about defaults and what is the normal level is now fine.

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 02 Jun 2012 20:39:43 +0100

Hi Dave,

>> On 31/05/2012 08:49, Dave Crocker wrote:
  [...]
> statement that defines the meaning of values.  Low is relative to 
> other values, but which ones?  In some environments -5 is low and in 
> others 0 is low and I'll bet some others will treat 5 as low.

To clarify I've changed that to "a negative priority". This is good 
enough for this example.

> [rfc5598, Sec 2.1] SHOULD retain the MT-PRIORITY parameter value for 

Ok, I've used your text.

  [...]
> some useful guidance about possible uses of the combined options, 
> since it seemed like the combination could be, well, useful.

The client can decide to use smaller values. In this case it is up to 
the client to pick them, this is not controlled by the site policy.

  [...]
> MTA which also conforms to [RFC3461]..."

RFC 3461 is a separate specification, so mentioning it explicitly is, 
IMHO, quite appropriate.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 04 Jun 2012 22:15:25 +0200

Alexey,

> the client to pick them, this is not controlled by the site policy.

Once again:  it is not reasonable to have different clients apply 
different semantics (policies) for choosing values.  The values need to 
have consistent meaning across the entire the trust environment that is 
supporting this mechanism.

Otherwise, it won't work properly.

The environment will be left with individual clients taking more than 
their fair share. Or trying to.

Absent very specific rules to be applied consistently across the trust 
environment, what is most likely is that every client will always claim 
top priority and no one will actually get it.  (This is a well-known 
phenomenon for this sort of game-theoretic condition.)

d/

---

From: Ned Freed <[email-masked]>
Date: Mon, 04 Jun 2012 14:47:39 -0700 (PDT)

> Otherwise, it won't work properly.

More precisely, it may "work" in some sense, but it may not deliver the results
you expect, which given what I understand the intended uses to be may be
problematic.

OTOH, it may actually work very well, if for no other reason than most modern
mail systems are able to deliver messages in a matter of seconds most of the
time, which will make it difficult for a human user to observe any tangible
difference for different priorities.

And in practice since prioritization is very complex - to the point where, as
I've pointed out previously, the most obvious strategies may actually result in
increased latency, especially under high load conditions - so aligning behavior
across multiple implementations may prove to be a practical impossibility.

All that said, I am completly at a loss as to what, if anything, to do about
all of this. To nail down what prioritization means in an operational sense
requires a far more detailed model of how MSA/MTA/MDAs work than we currently
have. And I given what I know about the internals of various MTAs, I despair
of finding any sort of model that is simultaneously sufficiently general
and sufficiently accurate that we could even talk about this stuff sensibly.

There's a reason why every specification I've seen that mentions email
prioritization, going back as far as FIPS PUB 98 (RFC 841) and including X.400,
GOSIP, various LAN email systems, either omits entirely any description of what
priorization actually means or contains nothing but a bunch of handwaving.

> phenomenon for this sort of game-theoretic condition.)

I have no good explanation for it, but the evidence I've seen says otherwise:
Quite a few existing systems support message prioritization, but I've yet to
encounter a case where everybody claims the top priority. (Users mostly seem to
ignore it, even when the controls for setting it are in plain view.) I rather
suspect it's a combination of factors: (1) As noted above, it has no observable
effect a lot of the time, (2) Interfaces rarely let average users bump the
default priority, meaning you have to do it every time, and (3) It can have
unwanted side effects, e.g., shortening timeouts, using up quotas, or enabling
generation of delivery receipts.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Mon, 04 Jun 2012 19:14:34 -0400

--On Monday, June 04, 2012 14:47 -0700 Ned Freed
<[email-masked]> wrote:

>...
> you expect, which given what I understand the intended uses to
> be may be problematic.

I mostly agree, but see below.

> across multiple implementations may prove to be a practical

Yes in principle.  In practice, I've seen one set of communities
that have repeatedly expressed high levels of demand for
prioritization.  Those communities are military or ones with
aspirations to behave like military ones.  For those
communities, "messages from the General get more delivery
priority than those from the Lieutenant" have obvious meaning
and obvious intent.  Moreover, the obvious response, especially
if someone is a communications officer in that chain of command,
is to salute and say, "yessir, we will transmit the General's
message with instructions to process it at higher priority".
If what happens after that is that this information is passed
along but no MTA does a thing about it, your observation about a
second or two takes over, there is a report back to the General
of complete conformance to his wishes and recognition of his
exalted status, and everyone is happy.   And, if that trick
doesn't work, it is always possible to artificially delay  the
Lieutenant's mail for a while, even if the General has no mail
in the queue, just to help prove that the General is important.
Simply dumping the Lieutenant's mail into the queue when it
arrives while trying to immediately process the General's mail
for delivery or forwarding would probably be a sufficient
implementation.   I don't have enough data to claim that
scenario occurs many times around the world every day, but it
has got to be close.

That view leads to an unattractive variation on Ned's question...

>...

I agree about the model issues (and have said so).  But suppose
the above scenario, for which the model is good enough within a
chain of command and irrelevant outside it, _is_ the use case.
The question is then whether it is appropriate for the IETF to
support this type of essentially "do little but get folks to
feel good" option.  I'd normally say "no".  But, if the reality
is that people will demand mail prioritization --I suggest that
they will as long as there are Generals and they outrank
Lieutenants and maybe as long as there are mail service
providers who can figure out how to charge one class of
customers more for exactly the same service because that groups
is willing to pay to be important-- and, that by standardizing
something we can at least do a security analysis and contain
interoperability issues, then maybe we should just hold our
noses (or Alexey's) and do it.

   john

---

From: Pete Resnick <[email-masked]>
Date: Mon, 4 Jun 2012 18:35:47 -0500

On 6/4/12 6:14 PM, John C Klensin wrote:

> ...if the reality

Speaking as the sponsoring AD, this is where I ended up. I find much of 
this exercise silly; I find more than 5 "priorities" complete overkill, 
and I think the likelihood that in a modern SMTP system any of these 
priorities will cause a significant change in delivery time (or order, 
for that matter) to be exceedingly low. That said, there is a community 
that insists on attempting this, and it is not a completely insular 
community; they will insist on implementations not of their own making 
to do "the right thing" about this. Given that, I'd prefer we document 
it and see if it gets deployment in any kind of interoperable way. If it 
doesn't, we move it to Historic and move on with our lives.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Pete Resnick <[email-masked]>
Date: Mon, 4 Jun 2012 18:57:22 -0500

On 6/4/12 6:35 PM, Pete Resnick wrote:
> Speaking as the sponsoring AD, this is where I ended up. I find much 
> of this exercise silly; I find more than 5 "priorities" complete 
> overkill, and I think the likelihood that in a modern SMTP system any 
> of these priorities will cause a significant change in delivery time 
> (or order, for that matter) to be exceedingly low. That said, there is 
> a community that insists on attempting this, and it is not a 
> completely insular community; they will insist on implementations not 
> of their own making to do "the right thing" about this. Given that, 
> I'd prefer we document it and see if it gets deployment in any kind of 
> interoperable way. If it doesn't, we move it to Historic and move on 
> with our lives.

Sorry, should have ended with the appropriate question: Does anybody 
feel that we should *not* be moving forward and documenting this?

pr

---

From: Claudio Allocchio <[email-masked]>
Date: Tue, 5 Jun 2012 08:35:09 +0200 (CEST)

On Mon, 4 Jun 2012, Pete Resnick wrote:

> Speaking as the sponsoring AD, this is where I ended up. I find much of this 
> exercise silly; I find more than 5 "priorities" complete overkill, and I 
> think the likelihood that in a modern SMTP system any of these priorities 
> will cause a significant change in delivery time (or order, for that matter) 
> to be exceedingly low. That said, there is a community that insists on 
> attempting this, and it is not a completely insular community; they will 
> insist on implementations not of their own making to do "the right thing" 
> about this. Given that, I'd prefer we document it and see if it gets 
> deployment in any kind of interoperable way. If it doesn't, we move it to 
> Historic and move on with our lives.

+1 +1 +1

Also given the combination of other "actions" which happens to an e-mail 
message when traversing current MTAs and MUAs: Highest Priority which 
meets a graylisting in the middle... need other examples?

Let's see... and we check later on.

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: John C Klensin <[email-masked]>
Date: Tue, 05 Jun 2012 06:07:15 -0400

--On Monday, June 04, 2012 20:18 -0400 ken carlberg
<[email-masked]> wrote:

> 
> On Jun 4, 2012, at 7:14 PM, John C Klensin wrote:
> 
>> Yes in principle.  In practice, I've seen one set of
>> communities that have repeatedly expressed high levels of
>> demand for prioritization.  Those communities are military or
>> ones with aspirations to behave like military ones.  For those
> so what you describe above is what I've called role-based
> prioritization.  its the easiest form of prioritization to
> implement and deploy because its typically associated with a
> role (sargent, general, etc.) and its generally stagnant and
> stays associated with a user or device.  several systems for
> both voice and messaging have been built over the past 50
> years or so using this form of prioritization.

Yes, but it is a little more than that.  Referring back to a
recent conversation about authentication, I note that most
military organizations have a very negative view about corporals
impersonating generals... and the means to deliver unambiguous
negative reinforcement to support that view.  Impersonating your
boss in a civilian company might get you fired, but would be
unlikely to get you shot (I can think of an exception or two,
but let's not go there).   While that isn't authentication in
the sense that we normally use the term, it can be a reasonable
surrogate for many purposes.  If this were just "role-based",
one would need to reopen the authentication and authorization
issues and to do so for inter-organizational mail.   I won't say
"impossible", but one could spend a rather long time exploring
that family of rat holes.
 
> another type of system (that I worked on about 25 years ago)
> is what I refer to as content-based prioritization, where the
> prioritization is attributed to the content of the message.
> So in this case, one could have a case where the user (or a
> proxy) decides and associates a priority based on the
> information in the message to be sent.  So in this case, you
> could have a case where the lowly Sargent trumps the General
> (kind a cool :-).  And by far, this is the hardest system to
> build/deploy because one needs to avoid the trap that Dave
> brought up earlier in that given the freedom, everyone wants
> to have the highest priority.

Sorry, the Sargent never trumps the General.  He may be sending
a message on behalf of a body that can, with that body's
authorization, or may, more generally, have authorizations the
General does not, but...

See Ned's comment, with which my experience agrees.  Also think
about whether you can make an anecdotal inference about message
and topic priority from this thread of messages.  It shows that
neither of us want to, and are assigning, "highest priority" to
handling this thread, sitting anxiously in front of our screens
waiting for the next message to arrive so it can be answered
instantly.  Now that it prioritization in the human part of the
process --message reading priority, message answering priority,
and message dispatching priority-- not in the transport part.
But the recipient generally doesn't care about the
human-versus-transport distinction.  The distinction clearly
makes no difference to the total or delivery times of a
communications round trip, etc.  I suggest that is a practical
counterexample to assertions that everyone always wants highest
priority.

Indeed, while (as far as I know) it has never been proposed as
an Internet mail extension, partially because it is best
implemented in the sending MUA or between it and the sending
Submission Server, experiments run many years ago made a
convincing case for a "sit on this message for an hour or two
before sending it out just in case I change my mind" service.
That, again, would be just the opposite from "I want highest
priority and for my message to be delivered immediately or
sooner".

> In the past few years, some communities have had to rely on
> the prioritization made available in X.400.  However, these
> and other communities wish to migrate to SMTP, hence the
> interest in produce the draft-melnikov-smtp-priority draft.
> So what i wanted to point out is that people have indeed
> worked on these systems and gained experience in the subject
> area, and we'd like to migrate this to SMTP, as opposed to
> just relying on proprietary hacks.

I'm tempted to say that, if you want the X.400 service model,
you should be looking to improve on MIXER, not SMTP.  I might
also note that that the round trip (message out, reply, message
back) time of many X.400 transactions in progress was such that
prioritization.  In practice, X.400 also needed it -- what was
it that Einar Steffrud used to say about X.400 delivery in a
millisecond world?  Days?  But, more important, there is an
authentication and authorization infrastructure built into the
X.400 model with bilateral trust agreements between management
(and hence transmission-authorizing) entities and each such
entity required to take responsibility for authentication and
authorization of anyone initiating mail.  Whether one trusts the
results of that model or not is another issue but the structure
is there.  Attempts to retrofit that structure onto Internet
mail without extensive hand waving have been rather
unsuccessful.  And that is ultimately one of the key problems
with trying to use this particular proposal between unrelated
administrative domains.

   john

---

From: Dave Crocker <[email-masked]>
Date: Tue, 05 Jun 2012 13:45:03 +0200

My hope is that I can respond to Ned without contradicting him.  I'm
prefacing with that statement because this is proving a challenging
topic having -- as he notes -- some unexpected characteristics...

On 6/4/2012 11:47 PM, Ned Freed wrote:
> OTOH, it may actually work very well, if for no other reason than
> most modern mail systems are able to deliver messages in a matter of
> seconds most of the time, which will make it difficult for a human
> user to observe any tangible difference for different priorities.

Priority is a mechanism intended to deal with situations of limited
resources.  In other guises, that's an aspect of queuing.  An
observation made by Kleinrock and others about such queuing is that it
only works well in periods of transient congestion.  In protracted
periods the only thing that suffices is more capacity and in periods of
no queuing, there's no need for the mechanism.

Perceiving no difference equates to "we don't need the mechanism".

The modification for some prioritization -- the role-based usage that's
been cited -- is protracted declining of service to lower rank
originators. That's no longer 'queuing' in an operational sense, it's a
termination of service for some.  Besides the military example, that's
what happens for telephone emergency mode, as I understand it.

So as soon as we say that user's perceive no difference, we are saying
that this mechanism isn't needed.

Therefore we need to define use cases that /do/ need to show a
difference.  I believe the military case and the emergency service case
have plenty of experience in that regard, establishing the value of
traffic segmentation.

> All that said, I am completly at a loss as to what, if anything, to
> do about all of this. To nail down what prioritization means in an
> operational sense requires a far more detailed model of how
> MSA/MTA/MDAs work than we currently have.

I think the basic syntax of prioritization is simple:  differential
labeling of the object with a rank-ordering value.  Handling of labeled
objects will, at a minimum, simply mean taking higher-labeled stuff
before lower-labeled stuff.

In more sophisticated queue-management situations, handling could be
more complex.  (I've just heard of recent work by van Jacobsen, for IP
routers, that provides an example, but the details don't matter here.)

> There's a reason why every specification I've seen that mentions
> email prioritization, going back as far as FIPS PUB 98 (RFC 841) and
> including X.400, GOSIP, various LAN email systems, either omits
> entirely any description of what priorization actually means or
> contains nothing but a bunch of handwaving.

That's why I'm suggesting partitioning mechanism syntax from assignment
semantics.  Environments that insist on using the mechanism have the
obligation to deal with this hard stuff.   But we don't have to, both
because we can't do it now and because it needs to be different for
different environments.

However, as usual, this sort of partitioning only works when one or two
real-world cases are applied.  IMO, this means that this spec needs to
be accompanied by two specifications for the policy-side that provides
details for using the mechanism in different environments.  Hmmm.
Military.  Emergency Services.  That's two.

>> The environment will be left with individual clients taking more
>> than their fair share. Or trying to.
>
>> Absent very specific rules to be applied consistently across the
>> trust environment, what is most likely is that every client will
>> always claim top priority and no one will actually get it. (This
>> is a well-known phenomenon for this sort of game-theoretic
>> condition.)
>
> I have no good explanation for it, but the evidence I've seen says
> otherwise: Quite a few existing systems support message
> prioritization, but I've yet to encounter a case where everybody
> claims the top priority.

But what you've described are situations where prioritization didn't
matter.  So user's had no incentive to use it.

Although I can't cite papers, I recall hearing of extensive research
(and experience in non-Internet contexts) that demonstrate the "everyone
sets to max" phenomenon.

On 6/5/2012 1:35 AM, Pete Resnick wrote:
>  I find more than 5 "priorities" complete
> overkill,

That's my assumption, to.  One can easily argue for 3.

In any event, for more than 5, I suggest that there needs to be explicit 
justification that is based on documented real-world models.  (I don't 
care whether they are Internet-based.  If there are useful models 
elsewhere, we can assume they might show up on the Internet.)

>    and I think the likelihood that in a modern SMTP system any
> (or order, for that matter) to be exceedingly low.

1. Most systems experience little-to-no queuing.

2. Until an environment has a meaningful pattern of queuing, no this 
mechanism isn't need.

3. Some environments are known to experience queuing.  That's why the 
combination of military and emergency should suffice as justification, 
IMO.  However they don't seem to be getting referenced to provide their 
substance.

d/

---

From: ken carlberg <[email-masked]>
Date: Mon, 4 Jun 2012 20:18:10 -0400

> scenario occurs many times around the world every day, but it

so what you describe above is what I've called role-based prioritization.  its the easiest form of prioritization to implement and deploy because its typically associated with a role (sargent, general, etc.) and its generally stagnant and stays associated with a user or device.  several systems for both voice and messaging have been built over the past 50 years or so using this form of prioritization.

another type of system (that I worked on about 25 years ago) is what I refer to as content-based prioritization, where the prioritization is attributed to the content of the message.  So in this case, one could have a case where the user (or a proxy) decides and associates a priority based on the information in the message to be sent.  So in this case, you could have a case where the lowly Sargent trumps the General (kind a cool :-).  And by far, this is the hardest system to build/deploy because one needs to avoid the trap that Dave brought up earlier in that given the freedom, everyone wants to have the highest priority.

In the past few years, some communities have had to rely on the prioritization made available in X.400.  However, these and other communities wish to migrate to SMTP, hence the interest in produce the draft-melnikov-smtp-priority draft.  So what i wanted to point out is that people have indeed worked on these systems and gained experience in the subject area, and we'd like to migrate this to SMTP, as opposed to just relying on proprietary hacks.

cheers,

-ken

---

From: ken carlberg <[email-masked]>
Date: Tue, 5 Jun 2012 07:23:15 -0400

> authorization, or may, more generally, have authorizations the

yes, the Sargent can trump the General.  The simple example is a General is at his headquarters and sends a message describing new operational responsibilities for his staff -- mundane stuff.  The Sargent is at a remote Atoll monitoring live Satcom images and sees that a nuclear bomb has just exploded and sends a message to that effect.  In cases where there is contention of resources between message switching/forwarding centers/servers, the Sargent's message trumps the General's.  I was part of the team that wrote the code to do this. (and sorry, I can't go into any indepth descriptions of the systems, operating environment, etc.  I just wanted to point out that this form of content-prioritization has been done in the past).

> you should be looking to improve on MIXER, not SMTP.  I might

actually, my understanding is that there is no interest in the X.400 service model.  There is an interest, though, in the specification of a prioritization capability in SMTP, which has been accomplished in X.400.  A slight, but subtle difference.

cheers,

-ken

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 06 Jun 2012 09:27:52 +0900

> what happens for telephone emergency mode, as I understand it.

That's definitely what happened in Japan after the 2011-03-11 
earthquake, and to a lesser extent in other situations.

Regards,   Martin.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 7 Jun 2012 13:42:54 +0100

Hi Dave,

> have plenty of experience in that regard, establishing the value of

I believe both of these are mentioned in the 1st paragraph of the Introduction.
If you think something is missing, please suggest some text.

> Military.  Emergency Services.  That's two.

I am working to address this (-16 doesn't quite do that).

> In any event, for more than 5, I suggest that there needs to be explicit justification that is based on documented real-world models.  (I don't care whether they are Internet-based.  If there are useful models elsewhere, we can assume they might show up on the Internet.)

Hmm, Appendix A?

Also see the new Appendix E in -16.

> 2. Until an environment has a meaningful pattern of queuing, no this mechanism isn't need.
> 
> 3. Some environments are known to experience queuing.  That's why the combination of military and emergency should suffice as justification, IMO.  However they don't seem to be getting referenced to provide their substance.

Well, there are Appendices A, B and C. But as I said above, I will work on more text.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 07 Jun 2012 15:47:23 +0200

> If you think something is missing, please suggest some text.

"Mentioned" is an accurate choice of vocabulary.

By calling for use cases, I mean sufficient description to make the 
nature, needs and dynamics sufficiently clear that they motivate design 
choices.  This is in contrast with having the document assume that the 
reader knows what the use requires.

>> In any event, for more than 5, I suggest that there needs to be explicit justification that is based on documented real-world models.  (I don't care whether they are Internet-based.  If there are…

Probably.

So... For more than 6, I suggest that...

> Also see the new Appendix E in -16.

well that argues from some additional room, but provides no rationale 
for having a space more than 3 times larger.

Since the values are transmitted as text, no that you don't really need 
to specify specific numbers, other than to say it is a decimal number. 
That is, the specific range is a function of the specific profile that 
defines the semantics of the numbers.

> Well, there are Appendices A, B and C. But as I said above, I will work on more text.

ack

d/

---

From: Ned Freed <[email-masked]>
Date: Mon, 11 Jun 2012 07:29:50 -0700 (PDT)

This entire discussion has been growing increasingly abstract, and that's
rarely a good thing in standards work.

Rather than continue in that direction, I decided to do something I should
have done a lot sooner: I implemented the draft.

<disclaimer>

The fact that I have implemented this draft in Oracle Messaging Server does not
imply any commitment on the part of Oracle to actually support this extension.
And should Oracle decide to support it, this doesn't mean it will be done
in any particular release or in any particular time frame.

</disclaimer>

(Sorry about having to include that, but rules are rules.)

Implementation work is usually very instructive, and this time was no
exception. I'll list the simple things I learned first:

(1) The draft says that the extension is valid for LMTP, but doesn't give any
    guidance as to what that means. This could be interpreted as saying that
    an implementation that uses LMTP has to support the MT-PRIORITY extension 
    there to be compliant. That would in turn mean that prioritization has
    to be done on the LMTP server side, which may be a bit difficult as there's
    no queue there. Our implementation, and I suspect many others, handles
    prioritization on the LMTP client side.

    That said, it's entirely possible for the MT-PRIORITY to affect the LMTP
    server by changing processing or I/O handling in some way, or affect
    subsequent store access behavior, or may be needed just so it can be
    logged. So the extension should be allowed over LMTP, but some clarifying
    text is needed to say an implementation MAY choose to handle
    prioritization on the client side of the LMTP connection only.

(2) The draft also allows MT-PRIORITY on submission, which of course makes
    complete sense. However, given the overall state of MUAs in the world
    today,  it is next to certain that clients are going to be used that don't
    support it. (And please don't try the line that this can always be dealt
    with by requiring the use of certain clients. I'm talking about the real
    world, not fantasyland.) As such, there are going to be cases where the
    MSA needs to attach an MT-PRIORITY to messages that don't have one. It
    would be good to mention this, but even if it's not discussed there needs
    to be an enhanced status code defined to indicate when it has been done,
    and more generally to indicate when the MT-PRIORITY has been upgraded.

(3) Implementations that support Sieve need to provide a way for Sieves to
    test the current MT-PRIORITY value. I implemented this as an environment
    item because that's a place that allows for implementation-specific values.
    The alternative would be to do it as a full-blown Sieve extension and add
    it to the envelope test. Given that MT-PRIORITY is an SMTP extension the
    envelope test is the natural place to do it, but a problem with that may
    be it restricts the test to implementations and situations where the
    envelope test itself is supported. This is not a problem for our
    implementation, but it may be a problem for others.

    Another issue is the lack of a defined comparator that will work with
    MT-PRIORITY values. i;ascii-numeric doesn't handle signed comparisons.
    An i;ascii-integer comparator is needed, so I added one and I think I'll
    go ahead and add the definition to the comparator registry.

    It may also be a bad idea to add all this in the current draft, especially
    given where this draft is in the process. But it needs to be done somewhere.

(4) The draft doesn't say anything about logging. The current MT-PRIORITY
    value does appear in Received: fields, but that's not the same thing.
    Logging of how MT-PRIORITY is used is going to be a requirement in some
    environments, so a general suggestion along the lines of "MT-PRIORITY
    values SHOULD be logged as part of any logging of message transactions" is
    in order.

(5) The suggested text for the X.3.TBD3 error code doesn't conform to the
    requirement that the text begin with the new priority value. (Note that
    the new error code suggested under (2) should have the same requirement.)

(6) The draft talks about greylisting, but fails to mention that
    connection-level and SMTP HELO/HELO greylisting options exist, and when
    these options are used there's an issue if the trust relationship is
    established through the use of SASL. This needs to be pointed out.

I also spotted a minor typo in the Introduction: sattelite -> satellite.

The draft also spells out numbers in some places and writes them as
actual numbers in others, making it hard to search for things. I don't care
which approach is used as long as it is consistent.

Now on to the more difficult stuff.

I first need to say a few words about how our product is implemented. We have
separate entities we call "channels" that messages can be routed to based on
various criteria. Each channel has it's own queue for messages, and can be
separately configured as to how many delivery threads to use, how frequently to
retry, what IP address and port to use, and so on and so forth. Additionally,
each queue supports three priorities, basically meaning higher priority
messages get tended to first and will be retried more frequently.

I realized right away that not only does a single channel approach not suffice
because a single channel doesn't have enough available priorities (the draft
requires six; see section 5), but also because even if we were to add
additional queue priorities, it's unlikely that that would actually create a
useful distinction between all the different priorities. For a useful
distinction to exist, especially across so many different levels, you need
access to things like different connections that shape traffic differently, and
in our world that means different channels.

So what I did was provide various hooks so that routing can be adjusted based
on priority. The obvious way to configure things would be to have two channels
and route three MT-PRIORITY levels to each one, but of course many other
arrangements are possible, up to and including ones that fullly support 19
distinct levels.

But there's no way I can guarantee that things will be configured this way. I
can't even test for it, since I have no way of testing external conditions. And
this is true no matter what implementation strategy I use, because any
sufficiently flexible configuration mechanism can be used to defeat priority
handling just as easily as it can be used to enforce it.

What this means is that the six level business is really an operational
requirement not an implementation requirement. What the implementation needs to
do is provide the *means* to support at least six priority levels. That's all
that can be reasonably expected, and the draft really needs to make that clear.

This then raises the question of whether or not we should make use of this
extension contingent on operational support of six levels. I think the answer
to that is a rather emphatic "no" - it should be left to the service profile to
decide how many levels they actually need. And that also needs to be made very
clear - in particular, statements as to supporting six levels being a
requirement, which sure sound like an operational requirement to me, need to
either be qualified or removed.

But this in turn raises another question: Should there be some indication of
what service profile is being provided? Proposals have been made previously to
do things like include the available priorities in the EHLO response. There are
two problems with that: (1) It doesn't actually identify the profile, and (2)
There's nothing a client can usefully do with the information.

A better alternative would be to include some sort of profile name in the EHLO
response. This wouldn't be for operational use, but rather for auditing
purposes. Of course an implementation like ours, and I suspect most others,
would not be done in a profile-specific way, so this would have to be a
configuration option. And I'm not sufficiently fond of the idea to say it
should be done. I'm only offering it as a suggestion.

Notifications next. The draft says that notifications SHOULD be processed at
the same priority as the message they are responding to. Having now implemented
this, I'm somewhat tempted to make that a MUST, because if you try and do it
any other way it's like opening a pandora's box of failure cases. Perhaps a
warning to the effect that, "Regardless of the approach chosen, schemes which
cause notifications to be completely lost MUST NOT be used."

But the draft then goes on to say that:

   For delivery reports received by an MTA, processing rules specified
   in Section 4.1 apply.  But note that while it might be tempting to
   handle all delivery reports (a.k.a.  DSNs) at their stated priority,
   under the assumption that failure notices need to get through quickly
   in some situations, but such a policy creates an exposure to fake-DSN
   attacks if sources of DSNs can't be reliably established.

This seemed to make sense until I tried to implement it, at which point I
realized it makes no sense at all. The problem is fundamental: There is in
general no way to tell whether or not you're dealing with a DSN. Yes, we
have defined format for DSNs, but there are still plenty of things out there
that generate DSNs in a different format. 

Even if we were to require support for NOTARY (and thus for our DSN format), we
need to be able to make priority decisions during envelope processing, and the
only indication there is the empty MAIL FROM. But while DSNs are required to
use an empty MAIL FROM, the converse isn't true: Plenty of things other than
DSNs use empty MAIL FROM.

And if that wasn't enough, the implication here seems to be that DSNs are
intrinsicly harder or more difficult to authenticate. Huh? That would seem to
imply that DSNs arrive from some sort of alternate source, or we're supposed to
base authentication decisions on the MAIL FROM. I don't know of any of the
former and the latter is a *really* bad idea. DSNs are just messages that
arrive from some source, and that source is either appropriately authenticated
or it isn't. Yes, you can make make additional choices once you've received the
message based on the  content being a DSN, but that's no different than any
other sort of content-based decisions. This isn't X.400, where DSNs aren't
actually messages.

I therefore strongly recommend dropping the entire paragraph except for the
first sentence.

And that's about it. I will say that having implementing the draft in what
I think is is a useful way, I'm a lot more confident that this extension
makes sense.

I'll try to follow up on some of the other traffic on this topic in a bit.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 21 Jun 2012 14:47:30 +0100

Hi Ned,
Thank you for very constructive comments.

>      prioritization on the LMTP client side.
A very good point. Yes.
>      prioritization on the client side of the LMTP connection only.
I've added a new section on this in -19.
>      MSA needs to attach an MT-PRIORITY to messages that don't have one.

This was always allowed. It is implied from the procedure of calculating the
priority as specified by the sender (and using 0 if not specified), then
changing it due to the local policy.

Actually, I thought the current text was quite clear:

<t>The SMTP server MAY also alter the message priority (to lower or
     to raise it) in order to enforce some other site policy.
     For example an MSA might have a mapping table that assigns priorities
     to messages based on authentication credentials.
</t>

I added "(Note that this also includes the case when the priority is not 
explicitly specified.)" after the first sentence quoted above.

I will also added an extra example that demonstrates that.

>      It
I changed the existing Enhanced Status Code for "the priority was 
lowered" to "the priority was changed". Pete asked for it earlier 
anyway, but I just didn't see much use case, so didn't do that. I don't 
think there is a need in  2 Enhanced Status Codes, because the selected 
priority is communicated back in the response anyway.

>      it to the envelope test.

Personally I have no preferences and I think either will work. 
Considering that you implemented the extended environment test, we might 
as well use that.

> Given that MT-PRIORITY is an SMTP extension the

I think doing this in a separate document would be best at this stage in 
the process.

>      values SHOULD be logged as part of any logging of message transactions" is

Agreed. Added in -19.

>      the new error code suggested under (2) should have the same requirement.)

You mean where it is first defined? I copied the text from the IANA 
registration section to make this clear.

>      established through the use of SASL. This needs to be pointed out.

Added.

> I also spotted a minor typo in the Introduction: sattelite ->  satellite.

Fixed.

> which approach is used as long as it is consistent.

I think I fixed these.

> Now on to the more difficult stuff.

I mostly agree with these comments. They will be addressed in -20.