Subject: [Jmap] Draft messages - mutability

From: Bron Gondwana <[email-masked]>
Date: Mon, 24 Apr 2017 11:54:59 +1000

I'm starting this thread to discuss the mutability of fields in isDraft: true messages.

In Cyrus IMAPd right now, we create the "MSGID" field as the sha1 of the "RFC822" raw message content as would be delivered over IMAP.  That's lovely for non-draft messages (except that we'll probably switch to using blake2 for new messages soon).

But it sucks for drafts, because it means that every time you edit a draft, you get a new MSGID.  Futhermore, you can't edit a draft via JMAP with ['setMessages', {update:{}}] because the fields that you want to edit are immutable, so instead you are doing ['setMessages', {create:{all the fields}, destroy: ["the old id"]}] which is a horrible way to edit drafts.

A somewhat similar discussion is happening over in CalConnect around the idea of saving draft ICALENDAR items on a server without sending out scheduling messages about them until they have finished being edited.

So the question for JMAP becomes: is a "draft message" the same datatype as a "message", or do you handle them completely differently?  Clearly you want to show them in-thread with the message they are in reply to in an interface, but how does that look over the protocol?

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 24 Apr 2017 18:02:24 -0700

On 23 Apr 2017, at 18:54, Bron Gondwana wrote:
> In Cyrus IMAPd right now, we create the "MSGID" field as the sha1 of 
> the "RFC822" raw message content as would be delivered over IMAP.  
> That's lovely for non-draft messages (except that we'll probably 
> switch to using blake2 for new messages soon).
>
> But it sucks for drafts, because it means that every time you edit a 
> draft, you get a new MSGID.  Futhermore, you can't edit a draft via 
> JMAP with ['setMessages', {update:{}}] because the fields that you 
> want to edit are immutable, so instead you are doing ['setMessages', 
> {create:{all the fields}, destroy: ["the old id"]}] which is a 
> horrible way to edit drafts.
>
> A somewhat similar discussion is happening over in CalConnect around 
> the idea of saving draft ICALENDAR items on a server without sending 
> out scheduling messages about them until they have finished being 
> edited.
>
> So the question for JMAP becomes: is a "draft message" the same 
> datatype as a "message", or do you handle them completely differently? 
>  Clearly you want to show them in-thread with the message they are in 
> reply to in an interface, but how does that look over the protocol?

I see a lot of cost and no benefit to making a draft a different 
datatype from a message. Now adding a command with semantics like 
draft-brandt-imap-replace to JMAP may be quite reasonable.

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 25 Apr 2017 11:35:34 +1000

On Tue, 25 Apr 2017, at 11:02 AM, Chris Newman wrote:
> datatype from a message.

I don't see any benefit to making it a different data type. However,
what we currently have is that, like IMAP, a message is immutable except
for flags/mailboxes. This is a pain for client authors trying to keep
track of a single draft as it is saved through multiple revisions
(especially when the user has multiple clients). One possible solution
to this is to simply say any message with the \Draft flag set is fully
mutable (while keeping the same id). I think we would also want the
server to enforce that a message cannot be changed from/to draft state
after creation. If it starts a draft it stays a draft, or vice versa.
>From a protocol perspective, since most messages won't be drafts (with a
normal user) you still get the efficiency of immutable messages (you
only have to refetch flags/mailboxes, not the whole message) for most of
the messages in the mail store. But it will be easier for MUAs to handle
drafts, and particularly allow for a much better experience when editing
the same draft message across multiple devices.
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 24 Apr 2017 22:00:05 -0400

If drafts are mutable, the implication is that synchronization based on
message ID can't happen without something like a version modifier. At which
point why not make every message versioned?

On Apr 24, 2017 9:35 PM, "Neil Jenkins" <[email-masked]> wrote:

> I don't see any benefit to making it a different data type. However, what
> we currently have is that, like IMAP, a message is immutable except for
> flags/mailboxes. This is a pain for client authors trying to keep track of
> a single draft as it is saved through multiple revisions (especially when
> the user has multiple clients). One possible solution to this is to simply
> say any message with the \Draft flag set is fully mutable (while keeping
> the same id). I think we would also want the server to enforce that a
> message cannot be changed from/to draft state after creation. If it starts
> a draft it stays a draft, or vice versa.
>
> normal user) you still get the efficiency of immutable messages (you only
> have to refetch flags/mailboxes, not the whole message) for most of the
> messages in the mail store. But it will be easier for MUAs to handle
> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap
>
>

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 24 Apr 2017 19:18:10 -0700

> after creation. If it starts a draft it stays a draft, or vice versa.

Why is it important to keep the same ID when modifying a draft? The 
approach draft-brandt-imap-replace uses is to include the new ID in the 
response to the modification operation. The replace model seems a good 
way to preserve the immutable property (which is very helpful for quick 
synchronization) while providing the client an interface that looks just 
like editing an existing message.

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 25 Apr 2017 12:18:24 +1000

On Tue, 25 Apr 2017, at 12:00 PM, Ted Lemon wrote:
> If drafts are mutable, the implication is that synchronization based
> on message ID can't happen without something like a version modifier.
> At which point why not make every message versioned?
I'm not sure I fully understand your question; every message is already
versioned (like the IMAP CONDSTORE modseq) so that you can efficiently
synchronise changes.
There is a well-defined model of data synchronisation which is used
across all data types in JMAP. A client can request the list of  ids of
a type (let's say Messages) which have changed since the client's
current state. It can then decide which of those messages it wants to
fetch, and what information it needs to fetch for each one. In the case
of non-drafts, it only needs to fetch flags/mailboxes. If drafts are
mutable, then for any draft message it has it would need to redownload
the entire message. This is the trade-off I was talking about in my
previous post.
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 24 Apr 2017 22:21:16 -0400

That seems okay as long as there isn't a ton of churn in the drafts folder.

On Apr 24, 2017 10:18 PM, "Neil Jenkins" <[email-masked]> wrote:

> There is a well-defined model of data synchronisation which is used across
> all data types in JMAP. A client can request the list of ids of a type
> (let's say Messages) which have changed since the client's current state.
> It can then decide which of those messages it wants to fetch, and what
> information it needs to fetch for each one. In the case of non-drafts, it
> only needs to fetch flags/mailboxes. If drafts are mutable, then for any
> draft message it has it would need to redownload the entire message. This
> is the trade-off I was talking about in my previous post.
>
> Neil.
>

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 25 Apr 2017 12:24:42 +1000

On Tue, 25 Apr 2017, at 12:18 PM, Chris Newman wrote:
> Why is it important to keep the same ID when modifying a draft?
>  The approach draft-brandt-imap-replace uses is to include the new
>  ID in the> response to the modification operation.

That's great if you're only using one client to edit the draft. But
consider the following scenario:
1. I start a draft on my laptop. I'm not finished, it's still open on
   the screen and I just put the laptop to sleep.2. I continue editing on my phone. Maybe I finish and send it or
   maybe I don't.3. I go back to my laptop, which immediately resyncs with the server to
   find out what's changed.
Now with immutable messages, all my laptop MUA can see is that the draft
has been deleted from the server, but this might just mean there's a
newer version of the draft. I can see the newer draft but can't know for
sure it's an update to the one I have open or not. So I can either
resave the draft that's open (which could cause a duplicate old draft to
"reappear" in the Drafts mailbox), or I can just delete it and hope for
the best (and the user can reopen the new draft if they still hadn't
finished sending).
With mutable messages, I would see there's an update to the current
message I have open. If I know the user's last changes were saved to the
server I would just fetch the update and immediately show this instead.
If the user has made other changes then I know we have a split and I can
prompt the user what they want to do. It's a much better user
experience.
Neil.

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 25 Apr 2017 10:33:25 -0700

On 24 Apr 2017, at 19:24, Neil Jenkins wrote:
>    the screen and I just put the laptop to sleep.
> 2. I continue editing on my phone. Maybe I finish and send it or
>    maybe I don't.
> 3. I go back to my laptop, which immediately resyncs with the server 
> to
>    find out what's changed.

I'd prefer not to make the common-case resync more complicated just to 
handle the rare case of multiple clients editing a draft. But if you 
think it's important to handle that case, another option is to introduce 
a draft unique identifier that's preserved across replace operations; 
then the client can know that a new message is a new version of a 
previous draft and the problem is resolved without making common-case 
resync more complicated.

		- Chris

---

From: Stu Brandt <[email-masked]>
Date: Tue, 25 Apr 2017 15:36:41 -0700

Agreed. Mutability opens the door to needing to sync only those body 
parts (or portions thereof) that changed since last sync point. It's 
also unlikely that a mutable content concept could be restricted to just 
"drafts" since there are all sorts of use cases around a user wanting to 
change the content of messages they received. In Alto, for example, once 
a user sees the photo stack the inevitable question is "how do I delete 
these old photos...I want to keep the email but without the photos".

We can totally re-envision the mail store as a versioned document store, 
which I agree would be interesting, but that seems like a significant 
scope change for JMAP to cover it. And it seems like we have enough of 
that already.

- Stuart

On 4/24/17 7:00 PM, Ted Lemon wrote:
> message ID can't happen without something like a version modifier. At
> which point why not make every message versioned?
>
> On Apr 24, 2017 9:35 PM, "Neil Jenkins" <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     __
>     what we currently have is that, like IMAP, a message is immutable
>     except for flags/mailboxes. This is a pain for client authors trying
>     to keep track of a single draft as it is saved through multiple
>     revisions (especially when the user has multiple clients). One
>     possible solution to this is to simply say any message with the
>     \Draft flag set is fully mutable (while keeping the same id). I
>     think we would also want the server to enforce that a message cannot
>     be changed from/to draft state after creation. If it starts a draft
>     it stays a draft, or vice versa.
>
>     From a protocol perspective, since most messages won't be drafts
>     (with a normal user) you still get the efficiency of immutable
>     messages (you only have to refetch flags/mailboxes, not the whole
>     message) for most of the messages in the mail store. But it will be
>     easier for MUAs to handle drafts, and particularly allow for a much
>     better experience when editing the same draft message across
>     multiple devices.
>
>     Neil.
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/jmap>
>
>
>

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 26 Apr 2017 09:02:50 +1000

Yeah, we have a strip attachments feature at FastMail. Generates a new email with same body structure but the part for each attachment replaced with a short bit of text explaining that the attachment was stripped. It's a feature that is needed less now that quotas are gigabytes rather than handful of megabytes.

The downside of the draft id is that you would need to search on it. The message object id will already have a fat index.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 11:06:16 +1000

Yeh, a persistent identifier can certainly do much of the same job (in
theory the RFC5322 Message-Id should do this if you generate it in a
sensible way). And I agree that it simplifies sync code if all messages
are the same in terms of mutability (although I don't think it would be
too hard to support).
As Bron points out, you do have more work in the client to find the
message by the draft id, but given the above, and that this is also more
easily compatible with IMAP, I think sticking with the current immutable
model is probably better.
As Chris mentioned earlier, an atomic "replace" *would* be very helpful
for handling drafts. I actually think we can model this as a standard
JMAP update, just it returns a new id. So you would have something like:
[
    [ "setMessages", {
        "update": {
            "${currentMessageId}": {
                "to": [{
                    "name": "Joe Bloggs",
                    "email": "[email-masked]"
                }],
                ... all the other properties you normally pass to
                create ...            }
        }
    }, "tag" ]
]

which returns

[
    [ "messagesSet", {
        "updated": {
            "${currentMessageId}": {
                "id": "${newMessageId}"
            }
        }
    }, "tag" ]
]

Neil.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 25 Apr 2017 22:40:05 -0400

On Apr 24, 2017, at 10:24 PM, Neil Jenkins <[email-masked]> wrote:
> Now with immutable messages, all my laptop MUA can see is that the draft has been deleted from the server, but this might just mean there's a newer version of the draft. I can see the newer draft but can't know for sure it's an update to the one I have open or not. So I can either resave the draft that's open (which could cause a duplicate old draft to "reappear" in the Drafts mailbox), or I can just delete it and hope for the best (and the user can reopen the new draft if they still hadn't finished sending).

I can see where this can happen if the client hasn't synchronized with the server, or if the message has been modified on the client before the sync happened, but in the normal case where the client is up to date, shouldn't the old version of the draft be gone from the list of drafts, and the new one present?   It's not like the _user_ is going to look at the message ID.

I suppose that if the old draft were present in a viewer, the user would see the weird behavior of the viewer disappearing, and they'd have to go click on the message again, so it would be good to have a way to discover the message that replaced it and open that one when removing the old one from the viewer.   And of course the MUA would have to notice that the old draft is no longer marked as in the draft folder, and possibly is marked as in the trash folder, and know to take some action because of that.

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 12:56:11 +1000

On Wed, 26 Apr 2017, at 12:40 PM, Ted Lemon wrote:
> I suppose that if the old draft were present in a viewer,

Yes, this is the case I was referring to: where the draft is currently
open for editing.
Neil.

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Apr 2017 15:38:20 -0700

In terms of message mutability, we've had another case for plain messages
that we find complicated, which is if the server chooses to modify the
message.

As an example case, suppose the server has decided that a message a client
has already synced is now a phishing message, and as such, wants to rewrite
all of the links in the message or make them not-linkable.  The only way to
do this with immutable messages is to remove the old message and create a
new one.[1]

If the client has already synced the message and read it, then on the next
sync, it will fail to apply the read action since the message no longer
exists, and the user will be presented with the message again as new, which
for a phishing message seems like a bad idea.

Perhaps we don't need to make them mutable, but a replace semantic which
was available in both directions would allow the client to sync state to
the new copy of the message.

[1] Of course, if you own the MUA and server, you can just say the message
is $Phishing and the MUA can modify how it displays the message, but
enterprise customers like to use their favorite clients and their security
ops folks like to protect them

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 28 Apr 2017 06:27:37 +0000

On Wed, 26 Apr 2017, at 10:38 PM, Brandon Long wrote:
> In terms of message mutability, we've had another case for plain
> messages that we find complicated, which is if the server chooses to
> modify the message.> 
> As an example case, suppose the server has decided that a message a
> client has already synced is now a phishing message, and as such,
> wants to rewrite all of the links in the message or make them not-
> linkable.
This is an interesting use case, and I don't have a good solution. Do
you have any suggestions for how this replacement might be represented
in JMAP? Happy to consider something, and whether any added complexity
is worth the benefit. I can see this could also be useful in the draft
case being discussed, for other clients to know which draft replaced a
previous one.
Neil.