Subject: [Jmap] New message sending API proposal

From: Neil Jenkins <[email-masked]>
Date: Tue, 04 Jul 2017 15:39:57 +1000

Hi all,

Hopefully we can get this area resolved by the end of the Prague meeting, so I'm sending through a first draft for initial discussion. I think the proposal addresses the concerns that were raised by various people on the list, while remaining consistent with the rest of JMAP.
*Sending a message*

The core idea is the introduction of a new MessageDelivery object type. To send a message, you create a new MessageDelivery, which has the following properties:
 * *id*: String The id of the message delivery. This is server-set and
   immutable.
 * *messageId*: String The id of the message to send. This is immutable.
   The message being sent does not have to be a draft, for example when
   “redirecting” an existing message to a different email address.
 * *threadId*: String The thread id of the message to send. This is
   immutable and set by the server to the *threadId* property of the
   message referenced by the *messageId*.
 * *envelope*: Envelope|null

Information for use when sending via SMTP. This is immutable. An
*Envelope* object has the following properties:

   * *mailFrom*: String The email address to use as the return address
     in the SMTP delivery. This may be the empty string.
   * *rcptTo*: String[] An array of email addresses to send the message
     to. If the *envelope* property is null or omitted on creation, the
     server MUST generate this from the referenced message as follows:

   * *mailFrom*: The email in the *From* header, if present.
   * *rcptTo*: The deduplicated list of emails from the *To*, *Cc* and
     *Bcc* headers, if present.
 * *sendAt*: Date|Duration|null

The date to release the message for delivery. This is immutable. The
server MUST override this property to the current date-time if the value
given by the client is omitted, null, or in the past, or if the server
does not support delayed sending. If a duration is specified, the server
MUST set this property to the time of this duration from “now”.
 * *maybeCanUnsend*: Boolean If true, attempting to unsend the message
   by destroying the MessageDelivery object, *might* succeed. This is
   server set. On systems that do not support unsending, this will
   always be false. On systems that do this, it will start as true, and
   MAY transition to false when the server knows it definitely cannot
   recall the message, but MAY just remain true.
Simple SMTP proxies would not store these objects, but more advanced servers would allow you to retrieve them later, for example a client might fetch a list of all the MessageDelivery objects with maybeCanUnsend == true, to present to the user messages they can still possibly cancel.
The MessageDelivery object also seems an ideal place to be able to put delivery status for servers that support this, so clients can query for each email address the message was sent to, what happened (delivered? If not, what was the error? Has it been read?). Any thoughts on whether this is a good idea and what pieces of information should be represented?
*Unsending a message*

To unsend a message, you destroy the MessageDelivery object, using standard JMAP commands. If this succeeds, the delivery was successfully cancelled. Otherwise you get an error.
*Updating the message if the send/unsend is successful*

We really want the ability to include in the same API command an update to perform on the message if the send/unsend succeeds (i.e. the MessageDelivery object is successfully created/destroyed). For example, remove the $Draft flag and move the message to the sent folder.
Without this, you would need two round trips (the client would need to wait to see if the send was successful before updating the message), which increases the likelihood of the send succeeding but the client losing the connection for whatever reason and so never updating the message.
At the moment, this is enabled by extra onCreateUpdateMessage and onDestroyUpdateMessage arguments to the setMessageDeliveries method. They work like this:For each successful create/destroy, the server should see if there is an entry with the same id in the onCreateUpdateMessage/onDestroyUpdateMessage properties respectively. If so, add an entry to an (initially empty) *update* object, with the key being the *messageId* on the MessageDelivery object that was created/destroyed, and the value being the value from the onCreateUpdateMessage/onDestroyUpdateMessage object.After the *messagesDeliveriesSet* response is emitted, if the *update* object is not empty, an implicit call is made to *setMessages*, with the following arguments:

 * *accountId*: Same as for the call to *setMessageDeliveries*.
 * *update*: The *update* object as calculated above.
This is a little awkward, but I haven't managed to come up with a better way of representing this ability. Any thoughts? Ideas for improvements?
The full diff to the spec is available at https://github.com/jmapio/jmap/pull/99. Looking forward to getting this worked out and done!
Cheers,
Neil.

---

From: Benoit Tellier <[email-masked]>
Date: Tue, 4 Jul 2017 14:15:15 +0700

Hi Neil,

First thank you for your great proposal, and also for your pull request
on the specs.

My answers will be inlined.

Cheers,

Benoit

Le 04/07/2017 à 12:39, Neil Jenkins a écrit :
> Hi all,
>
> Hopefully we can get this area resolved by the end of the Prague
> meeting, so I'm sending through a first draft for initial discussion.
> I think the proposal addresses the concerns that were raised by
> various people on the list, while remaining consistent with the rest
> of JMAP.
>
> *Sending a message*
>
> The core idea is the introduction of a new MessageDelivery object
> type. To send a message, you create a new MessageDelivery, which has
> the following properties:
>
>  *
>     *id*: |String|
>     The id of the message delivery. This is server-set and immutable.
>  *
>     *messageId*: |String|
>     The id of the message to send. This is immutable. The message
>     being sent does not have to be a draft, for example when
>  *
>     *threadId*: |String|
>     The thread id of the message to send. This is immutable and set by
>     the server to the /threadId/ property of the message referenced by
>     the /messageId/.
>  *
>
>     *envelope*: |Envelope|null|
>
>     Information for use when sending via SMTP. This is immutable.
>
>     An *Envelope* object has the following properties:
>
>      o
>         *mailFrom*: |String|
>         The email address to use as the return address in the SMTP
>         delivery. This
>         may be the empty string.
>      o
>         *rcptTo*: |String[]|
>         An array of email addresses to send the message to.
>
>     If the /envelope/ property is |null| or omitted on creation, the
>       o *mailFrom*: The email in the /From/ header, if present.
>      o
>         *rcptTo*: The deduplicated list of emails from the /To/, /Cc/
>         and /Bcc/
>         headers, if present.
>  *
>
>     *sendAt*: |Date|Duration|null|
>
>     The date to release the message for delivery. This is immutable.
>     The server MUST override this property to the current date-time if
>     the value given by the client is omitted, |null|, or in the past,
>     or if the server does not support delayed sending. If a duration
>     is specified, the server MUST set this property to the time of
>     this duration from “now”.
>  *
>     *maybeCanUnsend*: |Boolean|
>     If |true|, attempting to unsend the message by destroying the
>     MessageDelivery object, /might/ succeed. This is server set. On
>     systems that do not support unsending, this will always be
>     |false|. On systems that do this, it will start as |true|, and MAY
>     transition to |false| when the server knows it definitely cannot
>     recall the message, but MAY just remain |true|.
>
>
> Simple SMTP proxies would not store these objects, but more advanced
> servers would allow you to retrieve them later, for example a client
> might fetch a list of all the MessageDelivery objects with
> maybeCanUnsend == true, to present to the user messages they can still
> possibly cancel.
>
> The MessageDelivery object also seems an ideal place to be able to put
> delivery status for servers that support this, so clients can query
> for each email address the message was sent to, what happened
> (delivered? If not, what was the error? Has it been read?). Any
> thoughts on whether this is a good idea and what pieces of information
> should be represented?

We need to specify in my opinion the *identity* we should use as a
sender (http://jmap.io/spec-mail.html#identities)
>
> *Unsending a message*
>
> To unsend a message, you destroy the MessageDelivery object, using
> standard JMAP commands. If this succeeds, the delivery was
> successfully cancelled. Otherwise you get an error.
>
> We really want the ability to include in the same API command an
> update to perform on the message if the send/unsend succeeds (i.e. the
> MessageDelivery object is successfully created/destroyed). For
> example, remove the $Draft flag and move the message to the sent folder.
>
> Without this, you would need two round trips (the client would need to
> wait to see if the send was successful before updating the message),
> which increases the likelihood of the send succeeding but the client
> losing the connection for whatever reason and so never updating the
> message.
>
> At the moment, this is enabled by extra onCreateUpdateMessage
> and onDestroyUpdateMessage arguments to the setMessageDeliveries
> method. They work like this:
>
> For each successful create/destroy, the server should see if there is
> an entry with the same id in the
> onCreateUpdateMessage/onDestroyUpdateMessage properties respectively.
> If so, add an entry to an (initially empty) /update/ object, with the
> key being the /messageId/ on the MessageDelivery object that was
> created/destroyed, and the value being the value from the
> onCreateUpdateMessage/onDestroyUpdateMessage object.
>
> After the /messagesDeliveriesSet/ response is emitted, if the /update/
> object is not empty, an implicit call is made to /setMessages/, with
> the following arguments:
>
>   * *accountId*: Same as for the call to /setMessageDeliveries/.
>   * *update*: The /update/ object as calculated above.
>
>
> This is a little awkward, but I haven't managed to come up with a
> better way of representing this ability. Any thoughts? Ideas for
> improvements?
IMO overall OK, we just need to be also able to expose a method allowing
the client to destroy a message sent successfully/sent fail
>
> The full diff to the spec is available
> at https://github.com/jmapio/jmap/pull/99. Looking forward to getting
> this worked out and done!
>
> Cheers,
> Neil.
>
>
> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 4 Jul 2017 09:49:28 +0100

Hi Neil,

Mostly good. A few minor comments below:

>     the server to the /threadId/ property of the message referenced by
This is not very clear to me. Are replies to earlier messages share 
threadid with them?
>       o *mailFrom*: The email in the /From/ header, if present.
I think this is the email from the Sender header field, if present. 
Falling back to the From otherwise. We probably need to say something 
about multiple From addresses here.
> thoughts on whether this is a good idea and what pieces of information 

I think it would be useful. Information that would be useful: SMTP error 
code (per each recipient or a global one for the whole message (e.g. if 
MAIL FROM was rejected), SMTP Enhanced Status codes (if available), 
Error text. Some of these can also be synthesized by the JMAP server, 
for example if a recipient's domain can't be resolved using DNS.

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 05 Jul 2017 16:18:13 +1000

On Tue, 4 Jul 2017, at 05:15 PM, Benoit Tellier wrote:
> We need to specify in my opinion the identity we should use as a sender (http://jmap.io/spec-mail.html#identities)
I think I agree, although I'm not sure whether this should be optional? At the moment, none of the properties on the Identity object affect how the server handles the message; they just let the client know what email addresses the user has, and what defaults to use for various properties in a new message from those identities. However, some servers may end up doing different things depending on the identity, and matching just based on the From address is not necessarily possible if there are multiple identities that match the address.
There's also the case of redirecting mail (although how long this will be possible for in our DMARC future I don't know). In this case you didn't write the message, and so none of the identities really make sense.
For now, I've made it optional, but interested in feedback here.

> IMO overall OK, we just need to be also able to expose a method allowing the client to destroy a message sent successfully
Yeh, this doesn't currently allow that. I agree this is going to be a thing clients want to do, so how to represent it? Bron suggested an update of the "id" property to `null` for delete, but I'm not a huge fan of the idea.
On Tue, 4 Jul 2017, at 06:49 PM, Alexey Melnikov wrote:
>>  * *threadId*: String
>> The thread id of the message to send. This is immutable and set by the server to the *threadId* property of the message referenced by the *messageId*.> This is not very clear to me. Are replies to earlier messages share threadid with them?
So the threadId is assigned by the server. When you create a message (i.e. the draft) in the mailstore, the server will assign it a message id and a thread id. If the message has reply-to/references headers set etc. this should result in the threadId being the same as the message being replied to, yes.
The property is only really needed in the MessageDelivery object because I think it will be useful for filtering when getting a MessageDeliveryList. We could omit it, but it then requires potentially more work for the client to fetch thread if presenting a view of recent sent messages and their status.
> I think this is the email from the Sender header field, if present. Falling back to the From otherwise. We probably need to say something about multiple From addresses here.
Agreed, I'll fix that. Now reads:

 * *mailFrom*: The email in the *Sender* header, if present, otherwise
   the *From* header, if present, otherwise the empty string.

If multiple addresses are present in one of these headers, or there
is more than one *Sender*/*From* header, the server SHOULD reject
the message as invalid but otherwise MUST take the first email
address in the last *Sender*/*From* header in   the [RFC5322]
version of the message.>> The MessageDelivery object also seems an ideal place to be able to put delivery status for servers that support this, so clients can query for each email address the message was sent to, what happened (delivered? If not, what was the error? Has it been read?). Any thoughts on whether this is a good idea and what pieces of information should be represented?> 
> I think it would be useful. Information that would be useful: SMTP error code (per each recipient or a global one for the whole message (e.g. if MAIL FROM was rejected), SMTP Enhanced Status codes (if available), Error text. Some of these can also be synthesized by the JMAP server, for example if a recipient's domain can't be resolved using DNS. 
Sounds reasonable. Here's a first draft; comments welcome:

*deliveryStatus*: String[DeliveryStatus]|null

 This is a map from the email address of each recipient of the message,
 to the most recent SMTP reply code/text known. The code/text MUST be
 the response to the RCPT TO stage, unless this was accepted and the
 message as a whole rejected at the end of the DATA stage, in which case
 the message rejection code and status text MUST be used instead.
 This property is server set and MAY not be supported by all servers, in which case it will remain null. Servers that support it SHOULD update the MessageDelivery object each time the status of one of the recipients changes, even if some recipients are still being retried. The *isFinal* property allows clients to know if the status may still change further.
A *DeliveryStatus* object has the following properties:
 * *code*: Number
 The SMTP reply code returned for this recipient when the server last tried to relay it. See [RFC5321], section 4.2.3 for a list of codes.

 * *text*: String The text returned by the server along with this code.
   This will normally being with an SMTP Enhanced Mail System Status
   Code, as defined in [@!RFC3463], with a registry defined in
   [RFC5248].

 * *isFinal*: Boolean This is false if the server is still trying to
   send the message to this recipient (and so the code/text could still
   change), otherwise true.Cheers,
Neil.

---

From: Benoit Tellier <[email-masked]>
Date: Wed, 5 Jul 2017 13:50:28 +0700

Again, answers inlined

Le 05/07/2017 à 13:18, Neil Jenkins a écrit :
>     We need to specify in my opinion the identity we should use as a
> I think I agree, although I'm not sure whether this should be
> optional? At the moment, none of the properties on the Identity object
> affect how the server handles the message; they just let the client
> know what email addresses the user has, and what defaults to use for
> various properties in a new message from those identities. However,
> some servers may end up doing different things depending on the
> identity, and matching just based on the From address is not
> necessarily possible if there are multiple identities that match the
> address.
>
The way I would implement that is using default mail address for the
account... It seems counter intuitive to me to add Identities and not
use them....
> There's also the case of redirecting mail (although how long this will
> be possible for in our DMARC future I don't know). In this case you
> didn't write the message, and so none of the identities really make sense.
If I forward, then I am at the origin of the actions, it seems logic to
use one of my available emails...
>> IMO overall OK, we just need to be also able to expose a method
>> allowing the client to destroy a message sent successfully
>
> Yeh, this doesn't currently allow that. I agree this is going to be a
> thing clients want to do, so how to represent it? Bron suggested an
> update of the "id" property to `null` for delete, but I'm not a huge
> fan of the idea.
IMO the more explicit choice is add **onDestroyDestroyMessages**
Set[ReferenceId].

Null should be avoided in my opinion.
>>>     The thread id of the message to send. This is immutable and set
>>>     by the server to the /threadId/ property of the message
>>>     referenced by the /messageId/.
>>>
> So the threadId is assigned by the server. When you create a message
> (i.e. the draft) in the mailstore, the server will assign it a message
> id and a thread id. If the message has reply-to/references headers set
> etc. this should result in the threadId being the same as the message
> being replied to, yes.
>
> The property is only really needed in the MessageDelivery object
> because I think it will be useful for filtering when getting a
> MessageDeliveryList. We could omit it, but it then requires
> potentially more work for the client to fetch thread if presenting a
> view of recent sent messages and their status.
>
>     *mailFrom*: The email in the /Sender/ header, if present, otherwise
>     the /From/ header, if present, otherwise the empty string.
>
>     If multiple addresses are present in one of these headers, or
>     there is more than one /Sender///From/ header, the server SHOULD
>     reject the message as invalid but otherwise MUST take the first
>     email address in the last /Sender///From/ header in the [RFC5322]
>     version of the message.
>
>>> The MessageDelivery object also seems an ideal place to be able to
>>> put delivery status for servers that support this, so clients can
>>> query for each email address the message was sent to, what happened
>>> thoughts on whether this is a good idea and what pieces of
>>> information should be represented?
>>
>> I think it would be useful. Information that would be useful: SMTP
>> error code (per each recipient or a global one for the whole message
>> (e.g. if MAIL FROM was rejected), SMTP Enhanced Status codes (if
>> available), Error text. Some of these can also be synthesized by the
>> JMAP server, for example if a recipient's domain can't be resolved
>> using DNS.
>
> *deliveryStatus*: |String[DeliveryStatus]|null|
>
> message as a whole rejected at the end of the DATA stage, in which
> case the message rejection code and status text MUST be used instead.
>
> This property is server set and MAY not be supported by all servers,
> in which case it will remain |null|. Servers that support it SHOULD
> update the MessageDelivery object each time the status of one of the
> recipients changes, even if some recipients are still being retried.
> The /isFinal/ property allows clients to know if the status may still
> change further.
>
>   * *code*: |Number|
>     The SMTP reply code returned for this recipient when the server
>     last tried to relay it. See [RFC5321], section 4.2.3 for a list of
>     codes.
>   * *text*: |String|
>     The text returned by the server along with this code. This will
>     normally being with an SMTP Enhanced Mail System Status Code, as
>     defined in [@!RFC3463], with a registry defined in [RFC5248].
>   * *isFinal*: |Boolean|
>     This is |false| if the server is still trying to send the message
>     to this
>     recipient (and so the code/text could still change), otherwise |true|.
>
> Cheers,
> Neil.
>
>

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 14 Jul 2017 14:50:56 +0200

I think this is moving in a good direction...

> the following properties:

How about calling it a MessageSubmission object? Semantically, this 
object becomes a record of a message submission (whether or not the 
message is delivered).

>      in the SMTP delivery. This may be the empty string.

We also need the envelope mail from parameters. Perhaps:

      * *mailFromParams*: Object. Mail from parameters are converted to 
a
        JSON object with the parameter name as a string key and the 
parameter
        value as a string value (with any xtext or unitext encodings 
removed,
	   (RFC 3461, 6533) and JSON string encoding applied to the value).

>      server MUST generate this from the referenced message as follows:

And we also need the Rcpt To parameters.

>  * *sendAt*: Date|Duration|null

This is not needed. If you have mail from parameters, you just use 
"holdfor" or "holduntil" mail from parameters as defined in RFC 4865.

>    recall the message, but MAY just remain true.

These semantics work for me. Good balance between simplicity for client 
and flexibility for server.

> maybeCanUnsend == true, to present to the user messages they can still 

I like that flexibility. These objects look like a client view of a 
submission log. That's useful in and of itself; definitely worth the 
effort to persist these objects.

> thoughts on whether this is a good idea and what pieces of information 

Good idea. The starting point I'd suggest is a JSON representation of:
   * RFC 3464 message/delivery-status
   * RFC 3798 message/disposition-notification
   * RFC 3886 message/tracking-status
All the semantics are standardized already. So changing the syntax isn't 
a big deal.

> successfully cancelled. Otherwise you get an error.

If the object is semantically a record of a past submission, I think 
unsend is more of a modification of the record then a deletion of the 
record. Perhaps add a mutable "unsendAttempted" (client set boolean) 
property?

> At the moment, this is enabled by extra onCreateUpdateMessage and 
> onDestroyUpdateMessage arguments to the setMessageDeliveries method. 
> They work like this:For each successful create/destroy, the server 
> should see if there is an entry with the same id in the 
> If so, add an entry to an (initially empty) *update* object, with the 
> key being the *messageId* on the MessageDelivery object that was 
> onCreateUpdateMessage/onDestroyUpdateMessage object.After the 
> *messagesDeliveriesSet* response is emitted, if the *update* object is 
> not empty, an implicit call is made to *setMessages*, with the 
> following arguments:
>
>

This is acceptable. Right now an API request always attempts all 
methods. The other way to do this would be to have an API request 
variant that executes a sequence of methods but halts the sequence on 
failure.

I also think the suggestion to default the mail from based on an 
identity reference is a good one.

> thing clients want to do, so how to represent
> it? Bron suggested an update of the "id" property to `null` for 
> delete, but I'm not a huge fan of the idea.

If we do unsend by updating an "unsendAttempted" property, then you can 
remove the record of the submission by deleting the object which seems 
clean to me.

		- Chris