Subject: [Jmap] MessageSubmission last call

From: Neil Jenkins <[email-masked]>
Date: Mon, 28 Aug 2017 11:44:09 +1000

Hello all,

I'm getting back to work on the JMAP spec stuff, and first order of
business is getting the MessageSubmission change merged. There were a
few minor issues people had raised on the pull request, all of which
I've now resolved. I think this is just about ready to merge, so I'm
putting out a call for any last comments, clarifications or objections
then I'll merge this next week: https://github.com/jmapio/jmap/pull/99
Cheers,
Neil.

---

From: Stan Kalisch <[email-masked]>
Date: Mon, 28 Aug 2017 13:55:07 -0400

Should this part of the spec speak to what JMAP will/will not do if a "Return-Path:" header is submitted as part of a message?

Stan

> On Aug 27, 2017, at 9:44 PM, Neil Jenkins <[email-masked]> wrote:
> 
> Hello all,
> 
> I'm getting back to work on the JMAP spec stuff, and first order of business is getting the MessageSubmission change merged. There were a few minor issues people had raised on the pull request, all of which I've now resolved. I think this is just about ready to merge, so I'm putting out a call for any last comments, clarifications or objections then I'll merge this next week: https://github.com/jmapio/jmap/pull/99
> 
> Cheers,
> Neil.
> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 28 Aug 2017 19:11:12 -0700

Overall, I like this approach and support this change. Some substantive 
issues:

On the DeliveryStatus, I don't like having "isQueued" and "delivered" as 
separate properties because it allows a silly state (isQueued = true, 
delivered = yes). I think it'd be better to have one string field that 
doesn't allow silly states. Perhaps: status = {queued, failed, 
delivered, unknown} where:
   queued: in a local mail queue and status will change once it exits 
the local mail queues. This means the smtpReply, if present, may change.
   failed: permanent delivery failure for recipient. The smtpReply is 
final.
   delivered: successfully delivered to recipient. The smtpReply, if 
present, is final.
   unknown: unknown or other status; additional information may be 
present in DSN message for recipient. The smtpReply may or may not 
change for this status.
I'm neutral on whether the other three DSN action values are included 
{delayed, expanded, relayed}; they seem harmless as a client can just 
treat them as synonyms for 'unknown' and helpful as clients that cared 
wouldn't have to fetch & parse the DSN. But I understand there's a 
desire to keep things simpler so I'm fine either way.

The "SHOULD NOT expose ... MDN messages" is fine. The "SHOULD NOT expose 
DSN" is a problem for failed DSNs because that means any client that 
fails to build a UI for per-recipient DeliveryStatus / failed will 
result in silently lost mail. I think hiding of failure DSNs needs to be 
a client-requested opt-in feature and clients MUST provide users with a 
mechanism to indicate delivery failure if the client requests that 
feature. I'd be interested in other opinions on this point. I'm not a 
fan of client opt-in features, but I don't see a better alternative in 
this case.

For DeliveryStatus, I think we need some discussion about how status for 
addresses that don't exactly match one of the rcptTo addresses are 
reported. There are three cases: 1. there's a single DSN that has an 
original recipient matching a rcptTo address (but for a different actual 
recipient). 2. there are multiple DSNs with different addresses that 
match the same original recipient. 3. there are DSNs that don't match 
any original rcptTo. I'd suggest servers may correlate for case 1, but I 
think case 2 & 3 should be punted (the server doesn't correlate DSN; it 
just leaves it in the INBOX). Punting on case 1 is also ok. But the spec 
needs to advise on the correlation model to get consistent behavior.

For smtpReply, how should the JMAP server deal with multi-line 
responses?

And some nits:

   fgetMes => MessageSubmission
   succesfully => successfully
   succesful => successful

The deliveryStatus property has redundant text; the following text 
appears twice and should only appear once: "This property is server-set 
and MAY not be supported by all servers, in which case it will remain 
null. Servers that support it SHOULD update the MessageSubmission object 
each time the status of any of the recipients changes."

I believe this text:
"This is a map from the email address of each recipient of the message, 
to the most recent SMTP reply code/text known."
should read:
"This is a map from the email address of each recipient to a 
DeliveryStatus object."

I believe this should be deleted: "The isFinal property allows clients 
to know if the status may still change further."

"displayed: yes" actually means: the recipient's system claims the 
message has been displayed to the user.

		- Chris

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 28 Aug 2017 19:20:10 -0700

On 28 Aug 2017, at 10:55, Stan Kalisch wrote:
> Should this part of the spec speak to what JMAP will/will not do if a 
> "Return-Path:" header is submitted as part of a message?

I think this matter is covered sufficiently in RFC 5321 (e.g., page 58). 
If we say anything, I'd just say JMAP should comply with rules for a 
message-originating or message submission SMTP system as described in 
RFC 5321 and RFC 6409.

		- Chris

---

From: Stan Kalisch <[email-masked]>
Date: Tue, 29 Aug 2017 16:08:20 -0400

Hi Chris,

> On Aug 28, 2017, at 10:20 PM, Chris Newman <[email-masked]> wrote:
> 
> I think this matter is covered sufficiently in RFC 5321 (e.g., page 58). If we say anything, I'd just say JMAP should comply with rules for a message-originating or message submission SMTP system as described in RFC 5321 and RFC 6409.

I think the matter was covered sufficiently at the time those documents were written.  I think that, at minimum, the kind language about SMTP-compliant behavior that you describe would be helpful.

I posed my question because a rather significant amount of documents, varying in technical quality, now conflate setting the "Return-Path:" header field at time of submission (or practical equivalent) with actually setting the return path.  (You do correctly point out, of course, that this behavior runs counter to, especially, RFC 5321.). I suspect a lot of this stems from the fact that VERP is now in such massive use in bulk mail operations that we now have non-technical professionals in disciplines such as marketing interested in using the mechanism they see manifested by a "Return-Path:" header.  A lot of this documentation, consequently, reads to me like it was written with these people in mind, but also for IT personnel who have perhaps limited experience with email, and end up becoming confused by the documentation as a result.  (If one goes further into the weeds, one finds product documentation claiming variables one can use to set the "Return-Path:" header's value, directly.)

I think, in this light, the spec should say something.  Perhaps, even though it would be less elegant, rather than just refer to SMTP, it should make one or two things explicit.

---

From: Neil Jenkins <[email-masked]>
Date: Thu, 31 Aug 2017 11:24:17 +1000

On Tue, 29 Aug 2017, at 12:11 PM, Chris Newman wrote:
> On the DeliveryStatus, I don't like having "isQueued" and
> "delivered" as> separate properties because it allows a silly state (isQueued = true,> delivered = yes)

Yes, agreed.

> present in DSN message for recipient. The smtpReply may or may not

I think smtpReply is final in this status, since it has left the
local mail queues (otherwise it would be in queued status)?
Otherwise, I'm happy to go with this approach; I've pushed a change
to the pull request.
> I'm neutral on whether the other three DSN action values are included> {delayed, expanded, relayed}; they seem harmless as a client can just> treat them as synonyms for 'unknown' and helpful as clients that cared> wouldn't have to fetch & parse the DSN. But I understand there's a

Yeh, I don't feel super strongly on this, but would probably err on the
side of omitting them to keep things simpler. Anyone else got a strong
argument either way?
> The "SHOULD NOT expose ... MDN messages" is fine. The "SHOULD
> NOT expose> DSN" is a problem for failed DSNs because that means any client that
> result in silently lost mail. I think hiding of failure DSNs
> needs to be> a client-requested opt-in feature and clients MUST provide
> users with a> mechanism to indicate delivery failure if the client requests that
> feature. I'd be interested in other opinions on this point. I'm not a> fan of client opt-in features, but I don't see a better alternative in> this case.

This sounds very awkward to me; the DSNs are either messages delivered
to the inbox or they're not. Having them in the inbox except if the
client opts out sounds way more trouble than it's worth. (You could
easily add an implicit extra search term to the getMessageList filter
clients send, but there's no easy way to deal with mailbox counts and
quotas which would be affected by the messages that the clients
couldn't see.)
Whether the server should do this or not probably depends on how it
expects the user to primarily access their mail. Perhaps we should
change this all to a MAY:
*JMAP servers MAY choose not to expose DSN and MDN responses as Message
objects if they correlate to a MessageSubmission object. It SHOULD only
do this if it exposes them in the dsnBlobIds and mdnblobIds fields
instead, and expects the user to be using clients capable of fetching
and displaying delivery status via the MessageSubmission object.*
> For DeliveryStatus, I think we need some discussion about how status
> for addresses that don't exactly match one of the rcptTo addresses are
> reported. There are three cases:
>  1. there's a single DSN that has an original recipient matching a
>     rcptTo address (but for a different actual recipient).
Sorry, what do you mean "different actual recipient"?

>  1. there are multiple DSNs with different addresses that match the
>     same original recipient.
>  2. there are DSNs that don't match any original rcptTo.> I'd suggest servers may correlate for case 1, but I think case 2 & 3
> should be punted (the server doesn't correlate DSN; it just leaves it
> in the INBOX). Punting on case 1 is also ok. But the spec needs to
> advise on the correlation model to get consistent behavior.
Can you suggest some text for the spec for this? You're much more
familiar with DSNs than I am. Thanks.
> For smtpReply, how should the JMAP server deal with multi-line

Good point. I've added this:

Multi-line SMTP responses should be concatenated to a single string
as follows:
 * The hyphen following the SMTP code on all but the last line is
   replaced with a space.
 * Any prefix in common with the first line is stripped from lines after
   the first.
 * CRLF is replaced by a space.
For example:

            550-5.7.1 Our system has detected that this message is
            550 5.7.1 likely spam, sorry.

would become:

            550 5.7.1 Our system has detected that this message is
            likely spam, sorry.
> And some nits:

Thanks, fixed.

Neil.

---

From: Neil Jenkins <[email-masked]>
Date: Thu, 31 Aug 2017 11:38:46 +1000

On Wed, 30 Aug 2017, at 06:08 AM, Stan Kalisch wrote:
> I think, in this light, the spec should say something.  Perhaps, even
> though it would be less elegant, rather than just refer to SMTP, it
> should make one or two things explicit.
I'm not sure this is necessary; if we start providing advice about
correct use of SMTP the spec length will greatly increase, and it's not
directly relevant to JMAP specifically. Having said that, do you have a
suggested text you would like to see in the spec? We can then consider
that on the list.
Cheers,
Neil.

---

From: Stan Kalisch <[email-masked]>
Date: Wed, 30 Aug 2017 23:20:12 -0400

Hi Neil,

>> On Aug 30, 2017, at 9:38 PM, Neil Jenkins <[email-masked]> wrote:
>> 
> I'm not sure this is necessary; if we start providing advice about correct use of SMTP the spec length will greatly increase, and it's not directly relevant to JMAP specifically.

I personally agree that it's not directly relevant and have no qualms if that's the way you want to go.

> Having said that, do you have a suggested text you would like to see in the spec? We can then consider that on the list.

Something brief, possibly even as short as this:

> The server MUST NOT generate the *mailFrom* property from the email in the "Return-Path" header field, if present in the referenced message.

Does this work?  (I really have no opinion on "MUST NOT" vs. "SHOULD NOT" for this language.)

Thanks,
Stan

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 30 Aug 2017 22:13:03 -0700

> local mail queues (otherwise it would be in queued status)?

It could be in a remote mail queue that supports the DSN extension, so a 
subsequent notification of successful delivery via a DSN can still 
happen.

>> I'm neutral on whether the other three DSN action values are 
>> included> {delayed, expanded, relayed}; they seem harmless as a 
>> client can just> treat them as synonyms for 'unknown' and helpful as 
>> clients that cared> wouldn't have to fetch & parse the DSN. But I 
>> understand there's a
>> feature. I'd be interested in other opinions on this point. I'm not 
>> a> fan of client opt-in features, but I don't see a better 
>> alternative in> this case.
>

I can personally live with that wording, but it may make other reviewers 
uncomfortable.

> Sorry, what do you mean "different actual recipient"?

Suppose I send an email to [email-masked], I can get a DSN 
containing this:

Original-Recipient: rfc822;[email-masked]
Final-Recipient: [email-masked]
Action: delivered
Status: 250

So is that DSN left in the INBOX because the final-recipient doesn't 
match a RCPT TO address, or does it get correlated into the 
MessageSubmission DSNs field based on the original-recipient field?

>>  2. there are DSNs that don't match any original rcptTo.> I'd suggest 
>> servers may correlate for case 1, but I think case 2 & 3

Once we determine a model, I can suggest text.

>             550 5.7.1 Our system has detected that this message is

Nicely worded.

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Thu, 31 Aug 2017 16:31:27 +1000

On Thu, 31 Aug 2017, at 03:13 PM, Chris Newman wrote:
> It could be in a remote mail queue that supports the DSN
> extension, so a> subsequent notification of successful delivery via a DSN can still
> happen.

Sure, but it's not going to get a new SMTP reply code, so the
*smtpReply* property is final. The *delivered* property can still change
from "unknown" to "delivered" should a DSN arrive.
> I can personally live with that wording, but it may make other
> reviewers uncomfortable.
The other option I see is to cut the line altogether (servers may of
course do it anyway; the spec doesn't say you can't).
> MessageSubmission DSNs field based on the original-recipient field?
I would say this should be correlated based on the original-recipient
field, so this would be considered a successful delivery.
Cheers,
Neil.

---

From: Neil Jenkins <[email-masked]>
Date: Thu, 31 Aug 2017 16:34:36 +1000

On Thu, 31 Aug 2017, at 01:20 PM, Stan Kalisch wrote:
>> The server MUST NOT generate the **mailFrom** property from the email
>> in the "Return-Path" header field, if present in the referenced
>> message.> Does this work?

It seems superfluous to me, because there's already a very specific
algorithm in the spec for how to generate the mailFrom if the envelope
is not supplied, and as it doesn't reference the Return-Path header,
using this would already be a clear violation of the spec.
Neil.

---

From: "Chris Newman" <[email-masked]>
Date: Thu, 31 Aug 2017 10:56:38 -0700

> from "unknown" to "delivered" should a DSN arrive.

Actually, the "status" and "diagnostic" fields in the DSN are a new SMTP 
reply so I presumed that would replace the old one. If you just want to 
save only the SMTP submission reply, or only SMTP replies seen by the 
local ADMD the field definition needs to say that.

> field, so this would be considered a successful delivery.

Yes, but which address is used as the key in the map? Does the status 
appear under both addresses, or just the original recipient?

		- Chris

---

From: Stan Kalisch <[email-masked]>
Date: Thu, 31 Aug 2017 14:57:32 -0400

> On Aug 31, 2017, at 2:34 AM, Neil Jenkins <[email-masked]> wrote:
> 
>> Does this work?
> It seems superfluous to me, because there's already a very specific algorithm in the spec for how to generate the mailFrom if the envelope is not supplied,

You're right, and it's clear that I haven't communicated properly that I'm cognizant that what I'm trying to address isn't a technical necessity for the spec.  It seemed to me this was something feasible to incorporate without becoming a nuisance, but after rereading your text for MessageSubmission, I now agree it's something that gets in the way more than it helps.  (It also seems the issue I raised seems to be of greater concern to me than anyone else, and that's fine.)

In any event, it's not important enough of an issue to stop you from getting this section out the door and merged.  Thanks for your patience.

Stan

> and as it doesn't reference the Return-Path header, using this would already be a clear violation of the spec.
> 
> Neil.