Subject: [Jmap] DSN and MDN in JMAP

From: "Chris Newman" <[email-masked]>
Date: Tue, 18 Jul 2017 15:02:38 +0200

Adding Delivery Status RFC 3464 and Message Disposition Notification RFC 
3798 to MessageSubmission object:

Most interesting fields for a client UI:

A client wishing to show whether message has been seen by a recipient 
will look at the "Disposition-Type" per-recipient field (semantics in 
RFC 3798 section 3.2.6.2).

A client wishing to show delivery status will look at the "Action" 
per-recipient field (semantics in RFC 3464 section 2.3.3).

Thoughts on how to map data from these standards to JMAP:

1. RFC 3464 uses type;value for most items. 99% of the time only one 
type is used/needed on the Internet. I suggest we strip the "type;" when 
mapping to JSON. That makes the mapping spec slightly more complex, but 
the values easier for a client to use. Alternatively we could pass the 
values through unchanged (simpler spec, more complex client).
2. I suggest we just allow UTF-8 in all fields, including addresses. 
This gets us nearly free support for RFC 6533. If we're removing 
"type;", we also need to remove 3464/6533 quoting from addresses.
3. Probably want to convert date syntax from 5322 format to RFC 3339 
format, but numeric time zone offsets are required by RFC 3464, so we 
shouldn't break that.
4. DSNs have per-DSN fields (that apply when a report covering multiple 
recipients is sent) and per-recipient fields. I suggest storing a copy 
of the per-DSN fields with each recipient to keep the data model as 
simple as possible. An alternative would be to have an array of JSON 
objects for each DSN received and store a pointer to that record.
5. For MDN, the 'Disposition' field is a meta-field; I split it into 
three fields, it's possible to split it into 4 but I think that's not 
necessary
6. JMAP servers MAY hide DSN & MDN notification messages so they don't 
appear in the JMAP view of the mail store. If we do this and want to 
guarantee no loss of information, we might want to add a list of BLOB 
IDs to the MessageSubmission object that can be used to fetch raw DSN & 
MDN messages. *
7. An implementation of JMAP is permitted to set DSN & MDN fields for 
local deliveries without actually generating one or more DSN/MDN 
messages to transport the information, as long as the semantics of the 
fields are the same as if such a notification message was generated 
according to the DSN & MDN standards.*
8. One interesting idea... DSNs correlate using the ENVID (envelope ID, 
RFC 3461). We could require JMAP servers to set the ENVID to the ID of 
the JMAP MessageSubmission object on submission (presuming the 
submission server advertises RFC 3461). If we do this, the 
MessageSubmission ID must be restricted to ASCII and it would be best to 
avoid characters requiring xtext escaping (RFC 3461 section 4).*
9. It's possible to move most of this mapping to a separate spec, but I 
think four items belong in the JMAP Mail base spec (identified by *).

See base spec RFC 3464 & 3798 for defined semantics of the fields.

All fields are per-recipient (per #5):

{
   "Original-Envelope-Id": string,   # optional
   "Reporting-MTA": string,          # required, drop "dns;"
   "DSN-Gateway": string             # optional, drop "dns;"
   "Received-From-MTA": string       # optional, drop "dns;"
   "Arrival-Date": string            # optional, numeric time zone

   "Original-Recipient": string,
       # optional, drop address-type, allow 5321 & 6531
   "Final-Recipient": string
       # required, drop address-type, allow 5321 & 6531
   "Action": string
       # required: failed, delayed, delivered, relayed, expanded*
   "Status": string            # required: SMTP enhanced status code.
   "Remote-MTA": string        # optional, drop "dns;"
   "Diagnostic-Code": string   # optional, drop "smtp;"
   "Last-Attempt-Date": string # optional, numeric time zone required
   "Final-Log-ID": string      # optional
   "Will-Retry-Until": string  # optional, numeric time zone required

   i18n DSN (RFC 6533)
   "Localized-Diagnostic": { "lang-tag" : string, ... }

   MDN fields (per-recipient, RFC 3798)
   ------------------------------------
   "Reporting-UA": string         # optional
   "MDN-Gateway": string          # optional, drop "smtp;"
   "MDN-Original-Recipient": string   # optional
   "MDN-Final-Recipient": string      # required
   "Original-Message-ID": string  # optional
   "Disposition-Mode": string
      # required 
{manual,automatic}-action/MDN-sent-{manually,automatically}
   "Disposition-Type": string     # required {displayed,deleted}*
   "Disposition-Modifiers": string     # optional
   "Failure": string              # optional
   "Error": string                # optional
   "Warning": string              # optional
}

JMAP Clients wishing to generate MDNs (RFC 3798) will look at the 
following three headers:
   Disposition-Notification-To
   Disposition-Notification-Options
   Original-Recipient
and the $MDNSent keyword (RFC 3503)

JMAP servers MAY generate MDNs as a result of setting $Seen if they 
follow the rules in RFC 3503 and set $MDNSent keyword appropriately.

Comments?

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 19 Jul 2017 03:14:44 +1000

After discussing this with Chris, we agreed that most of this is probably not useful, and we would be better off simplifying the key information into two properties, and providing access to the raw DSN/MDN as blobs should a client wish to do more.
So I propose we add the following to the DeliveryStatus object for each recipient:
 * *delivered*: `String` This MUST be one of the following values:
   `unknown`: The delivery status is unknown. This is the initial value.
   `yes`: A DSN has been received for this recipient with Action (as per
   RFC3464 section 2.3.3) equal to "delivered". `no`: A DSN has been
   received for this recipient with Action equal to "failed".
 * *seen*: `String` This MUST be one of the following values: `unknown`:
   The seen status is unknown. This is the initial value. `yes`: An MDN
   has been received for this recipient with Disposition-Type (as per
   RFC 3798 section 3.2.6.2[1]) equal to "displayed".
 * *dsnBlobIds*: `String[]` A list of blob ids for DSNs received for
   this recipient, in order of receipt, oldest first.
 * *mdnBlobIds*: `String[]` A list of blob ids for MDNs received for
   this recipient, in order of receipt, oldest first.The other nice thing about this is that JMAP servers may be able to update the delivered/seen properties based on other mechanisms (for example when there are local recipients); the client doesn't have to care how the server knows if the message has been delivered/seen.
Neil.

Links:

  1. https://tools.ietf.org/html/rfc3798#section-3.2.6.2

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 19 Jul 2017 16:49:02 +1000

Regarding "delivered" there is a 4th state which you haven't captured here which is "definitely not yet" while the message is still in an outbound queue.  That's not quite the same as unknown.  Obviously a future-send email would also have that state.
On Wed, 19 Jul 2017, at 03:14, Neil Jenkins wrote:
> *delivered*: `String`
>  * This MUST be one of the following values:
> `yes`: A DSN has been received for this recipient with Action (as per RFC3464 section 2.3.3) equal to "delivered".
> `no`: A DSN has been received for this recipient with Action equal to "failed".
>  * *seen*: `String`
> This MUST be one of the following values:
> `unknown`: The seen status is unknown. This is the initial value.
> `yes`: An MDN has been received for this recipient with Disposition-Type (as per RFC 3798 section 3.2.6.2[1]) equal to "displayed".
>  * *dsnBlobIds*: `String[]`
> A list of blob ids for DSNs received for this recipient, in order of receipt, oldest first.
>  * *mdnBlobIds*: `String[]`
> A list of blob ids for MDNs received for this recipient, in order of receipt, oldest first.> The other nice thing about this is that JMAP servers may be able to update the delivered/seen properties based on other mechanisms (for example when there are local recipients); the client doesn't have to care how the server knows if the message has been delivered/seen.> 
> Neil.
> _________________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

Links:

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 19 Jul 2017 10:07:20 +0200

On 18 Jul 2017, at 19:14, Neil Jenkins wrote:
> After discussing this with Chris, we agreed that most of this is 
> probably not useful, and we would be better off simplifying the key 
> information into two properties, and providing access to the raw 
> DSN/MDN as blobs should a client wish to do more.
>    this recipient, in order of receipt, oldest first.The other nice 
> thing about this is that JMAP servers may be able to update the 
> delivered/seen properties based on other mechanisms (for example when 
> there are local recipients); the client doesn't have to care how the 
> server knows if the message has been delivered/seen.

I think that should be explicit in the spec, perhaps say:

JMAP servers MAY set the delivered/seen strings based on information 
other than standard DSN and MDN messages.

We could move dsnBlobIds & mdnblobIds to the MessageSubmission object. 
For the power users (or support staff) who want the details in those 
messages, JMAP-level per-recipient correlation isn't that important and 
it may be easier for a UI to put a "show technical details" feature on a 
messageSubmission object.

A couple other details I think are helpful to articulate:

JMAP servers MAY (SHOULD?) hide standard DSN and MDN messages that 
correlate to a MessageSubmission object from the normal mailbox view and 
expose them only in the dsnBlobIds and mdnblobIds fields.

When a JMAP server performs a message submission, it MAY use the same ID 
string for the RFC 3461 ENVID parameter and the JMAP MessageSubmission 
object ID. JMAP servers that do this MAY replace a client-provided value 
for ENVID with a server-provided value.

		- Chris

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Wed, 19 Jul 2017 10:46:37 +0200 (CEST)

> After discussing this with Chris, we agreed that most of this is probably not
> useful, and we would be better off simplifying the key information into two
> properties, and providing access to the raw DSN/MDN as blobs should a client
> wish to do more.

>     *
> delivered : `String`
> `yes`: A DSN has been received for this recipient with Action (as per RFC3464
> section 2.3.3) equal to "delivered". `no`: A DSN has been received for this
> recipient with Action equal to "failed".

>     *
> seen : `String`
> `yes`: An MDN has been received for this recipient with Disposition-Type (as per
> RFC 3798 section 3.2.6.2 ) equal to " displayed".

I would suggest to use 'disposed' (or another name that can be associated with message-disposition-notification) as IMHO the word 'seen' refers too much to the act of reading a message by a human being. 

/rolf

---

From: Ned Freed <[email-masked]>
Date: Wed, 19 Jul 2017 02:14:59 -0700 (PDT)

> Regarding "delivered" there is a 4th state which you haven't captured here
> which is "definitely not yet" while the message is still in an outbound queue. 
> That's not quite the same as unknown.  Obviously a future-send email would also
> have that state.

Why are we repeating prior work here? In regards to
submission-transmission-delivery, the model for all of this was worked out as
part of the message tracking work published in RFCs 3885-3888. In this
particular case, you're talking about the action field defined in RFC 3886
section 3.3.3. The specific state you're desribing is the "delayed" state.

				Ned