Subject: Re: [Jmap] simpler future release & unsend without outbox (was Re: Best vs Good enough - adoption of JMAP)

From: "Chris Newman" <[email-masked]>
Date: Mon, 01 May 2017 14:53:18 -0700

On 27 Apr 2017, at 22:46, Neil Jenkins wrote:
> On Thu, 27 Apr 2017, at 02:06 AM, Chris Newman wrote:
>> Well, if you believe that and I don't believe that,
>
> Then clearly we need to discover where the divergence in mental models
> is occurring.

Here's my guess at the root cause:
* You're trying to mirror a particular client UI model in the mail store 
server design.
* I'm trying to minimize the necessary server infrastructure to support 
popular UI models (including yours), particularly when the server is RFC 
4865 standards-based already.

I see the following divergence as a result of these perspectives:

1. I believe there are four message types involved in this discussion:
   * draft messages
   * client-side outbox
   * future release RFC 4865 submitted
   * other submitted messages
I believe we agree the protocol should make it simple to separate and 
count these classes of messages for UI display. However, I strongly 
prefer a design where the latter two message types are treated the same 
on the server because it reduces the necessary server infrastructure 
(and, in particular, it avoids conflating mail store and mail queue 
semantics).

2. I believe that mixing client-side outbox and future release messages 
is a poor UI that will confuse users. Client-side outbox messages have 
not been submitted and will not move forward if the client is offline or 
the client removes the message from the outbox. Future release messages 
have been submitted and will move forward when the client/MUA is 
offline, even if the message is removed by the client when offline. 
These messages can only be stopped by an explicit online "unsend" MTA 
operation. This is why I believe mixing these messages will be 
confusing. You apparently do not agree with me on that point. As with 
all UI design, at some point there will be disagreements between 
reasonable people that are not resolvable. I suspect this is one of 
those cases. To me that suggests the resolution is a protocol that 
doesn't constrain the UI choices in this area.

3. I view "submit" and "unsend" as MTA actions that should not be 
confused or conflated with mail store operations. I'm opposed to 
protocol designs where a message move, delete or flag changes triggers 
MTA actions as a side-effect. Instead, I propose JMAP have explicit 
"submit" and "unsend" commands that first perform the MTA action but can 
also trigger mail store actions (like move message or flag change) if 
the MTA action succeeds.

4. I believe the "unsend" MTA operation can be applied to any submitted 
message, not just future release messages. It's more likely to succeed 
on a future release message, but there are plenty of proprietary systems 
that allow unsend to work at least until the message has been seen by 
the recipient so we shouldn't disallow that behavior in the JMAP 
protocol design.

>> * Every server that supports future release and unsend will provide
>>   the same interface to the client. No need for the "always empty
>>   outbox" alternative server behavior that you'll see with the outbox
>>   model.
>
> The server will have to advertise whether it supports scheduled send
> regardless of the mechanics underneath. For servers that do not 
> support
> scheduled send, I agree that an always-empty outbox mailbox may feel a
> little superfluous, however the point is to, as you say, always 
> provide
> the same interface to the client regardless. The client will know from
> the capabilities that it must always be empty, so will probably hide 
> it
> in the sidebar or equivalent navigation mechanism (in fact I expect
> clients will probably hide the outbox when empty regardless of 
> scheduled
> send support).

Our server already supports scheduled send via RFC 4865 and a 
proprietary 'unsend/recall' mechanism. However, with the server-side 
outbox model we'll not expose those capabilities to JMAP clients because 
our mail store has no access to mail queues and that is an intentional 
and deliberate design choice for security and scalability reasons.

I've proposed a design where existing RFC 4865 standards based servers 
can transparently expose those capabilities without constraining 
possible MUA designs.

>> * This does not unnecessarily constrain server implementation, and
>>   permits simpler server designs.
>
> I do not believe this to be the case, as I will explain below.

If an "outbox" is a mail queue (which it is, by definition, if there is 
a daemon that accesses a message from the outbox and passes it forward 
in the delivery system), then a server-side-outbox unnecessarily 
constrains server design.

>> * This preserves clean mail store and mail queue separation including
>>   the security and operational benefits that provides.
>
> This is I think where the confusion lies. The outbox really isn't a 
> mail
> queue, and no one is suggesting you merge it with your mail store.
>
> You clearly have existing infrastructure that supports holding 
> scheduled
> messages inside your mail queue and possibly cancelling them from 
> that.
> You wish to reuse this infrastructure, of course! That's fine. To do 
> so,
> when the message is moved to the outbox, you would *also* submit it 
> into
> your mail queue. The copy in the outbox would be the representation in
> JMAP land, not part of your mail queue.

This implementation approach includes properties of the following 
anti-patterns: "database-as-IPC", "race hazard" and "action at a 
distance". https://en.wikipedia.org/wiki/Anti-pattern

It raises questions like this: Should the shadow copy in the outbox have 
the received header that is present on the message in the submission 
queue? What happens if an MUA deletes the copy of the message in the 
server-side shadow outbox and the link between MUA and mail queue is 
down when that action is performed? Does the delete fail with a special 
error or does the delete succeed and then generate a new DSN-like 
message saying the attempt to unsend that message failed? Are both of 
those behaviors allowed?

Anyway, this has reinforced my point about putting unnecessary 
constraints on server design. This proposal requires complex linkages 
between the JMAP server, the mail store and the mail queue (creating a 
three-party system with object aliasing that will have unintended 
consequences).

On the flip side, my proposal was neutral on the matter of client UI. If 
your client wants to create a virtual outbox view for already-submitted 
future release messages and offer the unsend operation only on those 
messages, that's fine.

> When the message is sent to the
> next hop and can no longer be recalled,

These two things don't necessarily happen at the same time. For our 
implementation, a message can be recalled as long as the message lives 
in a queue in one of our MTAs. It can pass to the next hop MTA and still 
be unsent. Many proprietary systems allow message recall until the 
message has been seen by a recipient. JMAP's design shouldn't disallow 
that.

> you then move the JMAP message
> from outbox to sent, and remove the `\Draft` keyword.

Why require the addition of a new signaling infrastructure from MTA 
queues to mail store and add new churn to the mail store when the same 
MUA UI can be achieved without physically moving the message between 
folders at the future release time?

> Other systems may choose to implement it differently, not passing it 
> to
> the mail queue until the scheduled time is reached. As has been 
> pointed
> out already, many systems such as Gmail, FastMail etc. already have
> infrastructure to do this kind of thing since it is also used for
> deleting messages that are X days old from the trash, supporting 
> snoozed
> mail, etc.

Actually, my proposal makes this server design approach simpler. There's 
just an internal flag in the sent folder to indicate if the message has 
been passed forward. The mail queue daemon that operates on the mail 
store sent folder, can transfer the message to the next hop MTA and set 
that flag at the future release time. No need to move the message 
between folders.

> The outbox protocol is agnostic to how you wish to implement this. 
> Your
> proposal constrains you to only the option *you* already have
> implemented.

My proposal makes more server approaches possible, while being agnostic 
from the client UI side (clients can present a virtual outbox for future 
release messages or not as they choose).

>> * This makes it clear when the message enters the submission system.
>>   There's no confusion as one has with an outbox model where a 
>> message
>>   in the outbox may be submitted (if it's on the server) or maybe not
>>   (if the client is offline when the message moves to the outbox).
>
> This is clearly up to the client to distinguish, but I strongly 
> disagree
> this is a confusion. For the user, there is the outbox, which 
> represents
> the messages that are yet to be sent. If they open the outbox, I 
> expect
> most MUAs would label why they have not yet been sent (e.g. waiting 
> for
> internet connection, scheduled for time X).

See 2 above.

>> * Submission can have semantics fully compatible with existing
>>   infrastructure, making this easier/faster to deploy.
>
> As mentioned above, a lot of systems already have infrastructure for 
> "do
> action X in folder Y when time Z is reached", or you can alternatively
> use your SMTP-queue-scheduled-send to implement the outbox model, so
> this is actually *more* compatible with existing infrastructure than
> your proposal.

It's not "my SMTP-queue-scheduled-send". It's the IETF standards track 
scheduled send mechanism, RFC 4865.

>> * The "outbox" model constrains unsend to only work if the message is
>>   in the outbox. This model has no such constraint and works the same
>>   regardless of where a message that can be "unsent" lives in the
>>   infrastructure.
>
> When composing and sending a message, it goes through 2 or 3 states.
> These need to be represented in the mail store (and thus the protocol)
> so that users can understand what is going on, and what actions are
> available to them.
>
> Two of these we (I believe) agree on, since they are from IMAP:
>
> (1) The message is still being composed. It SHOULD be in the
>     "drafts" mailbox and MUST have the "\Draft" keyword. This is the
>     same as IMAP.

Agreed.

> (2) The message has been sent, cannot be recalled.

These two statements are not synonyms; see my item 4 above.

>     It SHOULD be in the
>     "sent" mailbox (although services may provide means for users to 
> ask
>     it to go somewhere else, for example we have a few users that 
> prefer
>     their sent mail to in the same mailbox as the message they were
>     replying to).
>
> The third (new) state is for "scheduled" messages. We have two options
> for how this is represented:
>
> 1. By its presence in the "outbox". Moving it out is therefore the
>    logical thing to unschedule it.

I disagree. Unsend is a separate operation with different success/fail 
consequences from the move message operation. I object to confusing 
these two operations; see my item 3 above.

> 2. By a keyword. This permits it to be in any mailbox, although we 
> would
>    probably add an outbox mailbox to our server for holding these
>    messages by default. Removing the keyword would probably be a 
> logical
>    way to unschedule it. This option is quite a bit less efficient at
>    the protocol level, as without a Mailbox object (which has a
>    precalculated total count), the client will have to issue a search
>    every single time there's a state change to see if there are any
>    scheduled messages.

I disagree. Unsend is a separate operation with different success/fail 
consequences from the change flag/keyword operation. I object to 
confusing these two operations; see my item 3 above.

My proposal is to have future release sent messages tagged with their 
future release date. The MUA can create a virtual folder for future 
release messages with a future date. Whether the server caches a count 
for the messages in that virtual folder is a server design optimization 
choice. If enough clients request that count that it helps server 
scalability to cache that count, then servers will cache that count. The 
count of future release messages is thus not relevant to the issue under 
debate.

> I'm presuming you are not advocating for option (3), which is no
> representation in the mail store, since this would be terrible for the
> user experience, and your proposal explicitly said "Add a way to
> search/select messages in the Sent folder that may be possible to
> unsend", which would indicate there must be some kind of state
> representing this status.

Per my item 3 above, I view "submit" and "unsend" as MTA actions. There 
are two ways to reflect successful MTA actions in the mail store:

A. When the submit action succeeds, the message is moved from draft 
folder to sent folder, the \draft flag is removed (if set), and an 
envelope annotation (including future release date if there is one) is 
added if not already present. When unsend succeeds, the message is moved 
from the sent folder back to the draft folder. The unsend operation can 
be applied regardless of whether the message is a future release 
message.

B. When the submit action succeeds, the \draft flag is removed, a 
$submitted flag is set and an envelope annotation (including future 
release date if there is one) is added if not already present. When the 
unsend succeeds, the $submitted flags is removed and a \draft flag is 
added.

I slightly favor option A, but have no strong opinion on this matter. 
It's even possible to design the submit & unsend actions so both of 
these are possible (always make the flag change and make the 
folder-move-on-success an optional feature of the MTA actions).

> So however you implement it, you need to trigger some state change on
> the JMAP server for each logical step so that clients can represent
> this, and update its local model as changes occur. The choice of
> representation is mainly down to what will map more easily to how
> clients wish to represent the set of scheduled messagese to users.

I disagree. The protocol design has peer goals on this matter: make 
popular UI paradigms straightforward and minimize necessary server 
infrastructure.

> UX designers seem to have overwhelmingly decided that putting these
> scheduled messages in an outbox rather than mixing them in with sent
> items or elsewhere is a clearer model for users to understand. I
> provided 5 example products in the previous email that do this, but if
> that's insufficient I noice we have representatives from Apple, Google
> and AOL on this list; perhaps they can ask their respective UI teams 
> to
> comment on what they would recommend?

I do not contest that client-side outbox is a popular UI model. I 
further agree the protocol design should allow the presentation of a 
server-side outbox UI. So I don't think there's a point of contention 
here.

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 03 May 2017 07:57:23 +0000

On Mon, 1 May 2017, at 09:53 PM, Chris Newman wrote:
> * You're trying to mirror a particular client UI model in the mail
>   store server design.
I'm trying to ensure that the API exposes the data clients need to
present common UI patterns in an efficient way, while also allowing the
most flexibility in server implementation. I think we're both aiming for
the same thing!
> 1. I believe there are four message types involved in this discussion:>   * draft messages
>   * client-side outbox

I agree, although I don't think the client-side outbox is particularly
relevant to this discussion, as it should not affect the API design. I'm
sorry I brought this up as it side-tracked the discussion somewhat. My
main point was that many existing systems assign submitted "future
release" messages to a separate mailbox on the *server*, and clients
present them to the user distinctly to those that have been released.
What I misunderstood from your proposal, is that rather than marking
messages that can be recalled/cannot be recalled, you propose a
client just look at the "release date" annotation instead. Although
you also say this:
> Actually, my proposal makes this server design approach
> simpler. There's> just an internal flag in the sent folder to indicate if the
> message has> been passed forward. The mail queue daemon that operates on the mail
> store sent folder, can transfer the message to the next hop MTA and
> set> that flag at the future release time. No need to move the message
> between folders.

So the mail queue daemon *does* perform an action on the message in the
mail store when it is sent? I previously thought you were advocating for
a model where these can be completely separate and so this is not
possible. Could you please clarify?
I think we've got various issues conflated a bit over these discussions,
so I'll clarify what I think are the separate questions:
How, if at all, are "future release" messages (that have yet to be
released) marked over the API? Is there a change in the state when they
are released? I agree this may not correspond exactly with whether a
message can be unsent, especially for internal corporate mail systems,
so a related question is: how, if at all, are messages marked that can
be unsent? (This is potentially a tristate: can (almost) definitely
unsend, can definitely not unsend, may be able to unsend).
I think if we can agree on answers to these questions, the rest will
fall out fairly easily.
> protocol designs where a message move, delete or flag changes triggers> MTA actions as a side-effect.
> Instead, I propose JMAP have explicit "submit" and "unsend" commands
> that first perform the MTA action but can> also trigger mail store actions (like move message or flag change) if
> the MTA action succeeds.
Yes, this is a perfectly fine approach; I'm not opposed to adding
explicit sendMessages/unsendMessages calls. If the call can optionally
set which mailbox(es)/keywords to set upon successful send, then this is
functionally equivalent.
> My proposal is to have future release sent messages tagged with their> future release date. The MUA can create a virtual folder for future
> release messages with a future date. Whether the server caches a count> for the messages in that virtual folder is a server design
> optimization> choice. If enough clients request that count that it helps server
> scalability to cache that count, then servers will cache that
> count. The> count of future release messages is thus not relevant to the
> issue under> debate.

What's relevant is how this count is exposed over the API. If there
is a mailbox, then the synchronisation of mailbox objects will ensure
the client is "pushed" the count; because count changes are common
and clients need to know them to present the most common UI paradigm
for mail, synchronisation of mailbox counts is optimised for in the
JMAP protocol.
One proposal that might satisfy everyone: an "outbox" or "futurerelease"
mailbox role is added (this is the IMAP SPECIAL-USE representation in
JMAP).  Servers can *optionally* have a mailbox with this role; if it
does, clients SHOULD place future-release messages in here, but like
"sent" *can* put the message in any mailbox(es). Servers that have this
mailbox MUST move messages from this mailbox to the sent mailbox at
future release time (which may just be based upon the future-release
date, or may be actually based on releasing it to the mail queue).
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 3 May 2017 12:41:49 -0400

On May 3, 2017, at 3:57 AM, Neil Jenkins <[email-masked]> wrote:
> I agree, although I don't think the client-side outbox is particularly relevant to this discussion, as it should not affect the API design. I'm sorry I brought this up as it side-tracked the discussion somewhat. My main point was that many existing systems assign submitted "future release" messages to a separate mailbox on the server, and clients present them to the user distinctly to those that have been released.

The problem with a client-side outbox is not that it affects API design, but that it creates bad UX.  Any state that is not synchronized produces surprising results: I can see somethings in one MUA that I use, but not in another.

It may be that the problem being solved here needs to be solved in a way that doesn't use the outbox as a queue, but I felt the need to reiterate this point, which I have made before: having different outboxes on different clients is a bad idea.   If the submission mechanism winds up requiring an un-synchronized client-side outbox, either explicitly or implicitly, we have made a serious mistake.

> One proposal that might satisfy everyone: an "outbox" or "futurerelease" mailbox role is added (this is the IMAP SPECIAL-USE representation in JMAP). Servers can optionally have a mailbox with this role; if it does, clients SHOULD place future-release messages in here, but like "sent" can put the message in any mailbox(es). Servers that have this mailbox MUST move messages from this mailbox to the sent mailbox at future release time (which may just be based upon the future-release date, or may be actually based on releasing it to the mail queue).

This presumes that the semantics of the "sent" mailbox are that messages that have been queued but not yet left the local queue go in "sent," not in "outbox" or "futurerelease."   That's probably correct, but it doesn't quite match the semantics implied by the name "sent."

I realize that both of these observations are a bit nit-picky, but I felt a bit of unease with both of the above quotes, so I wanted to try to figure out and express what was behind it.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 03 May 2017 14:16:37 -0700

On 3 May 2017, at 0:57, Neil Jenkins wrote:
> relevant to this discussion, as it should not affect the API design.

An important factor in protocol/API design is to avoid misleading the 
consumers. So in this respect the distinction between client-side outbox 
and server-side future release messages is important. We don't want 
protocol/API consumers to confuse those two classes of messages because 
they behave differently when the client is offline.

> I'm sorry I brought this up as it side-tracked the discussion 
> somewhat. My

I advocate an implementation model where the mail queue and mail store 
can be separate in the server implementation. I accept that some server 
implementations may choose not to do this and the protocol design 
shouldn't constrain potentially reasonable implementation choices. In 
the quote above, I was explaining one way a server implementation that 
mixes mail queue and mail store could work with the protocol model I 
proposed.

> I think if we can agree on answers to these questions, the rest will

I prefer a protocol model where it is possible for the client to easily 
identify "future release date is in the future" messages and to apply an 
"unsend" action to those messages. I further believe the model shouldn't 
limit the "unsend" action so it can only be used with that set of 
messages. And I'd prefer a model where there is no explicit state change 
in the mail store when a "future release" message is released as that 
keeps the mail queue and mail store components independent.

I'm ok if the base JMAP spec requires implementation of future release 
and unsend for future release messages. I'd prefer to leave details 
about message tracking (including whether a message can unsent) and 
unsend for other messages to a JMAP extension. Things get complicated 
for the other cases. First, it's actually a four state problem: "future 
release in the future", "unsend not possible", "unsend maybe possible", 
"no information about whether unsend is possible". And there's the 
question of whether tracking is an action (as it is for RFC 3887), or we 
require an MTA to mail store notification path for all messages to 
indicate whether a sent message can be unsent. The latter probably 
results in a better UI but I think it's too much to require in the JMAP 
base spec if we want JMAP to deploy quickly.

>> that first perform the MTA action but can> also trigger mail store 
>> actions (like move message or flag change) if
>> the MTA action succeeds.

Great, we have an approach we both support.

>> My proposal is to have future release sent messages tagged with 
>> their> future release date. The MUA can create a virtual folder for 
>> future
>> release messages with a future date. Whether the server caches a 
>> count> for the messages in that virtual folder is a server design

Virtual/smart mailboxes are a common mail UI feature. As a mail user, 
I'd like my virtual mailboxes to look the same in different mail clients 
to the extent possible. So the way to get more consistent behavior is to 
have a server-side-virtual mailbox model. I'd very much like JMAP to 
have the ability to create a virtual/smart mailbox that behaves like a 
regular mailbox in most ways (including optimized counts). That's a 
significant value-add that could be implemented in the JMAP layer. IMAP 
has primitives that can help (like RFC 5466, and RFC 5267) but nothing 
that enables smart mailbox behavior consistency across clients.

> date, or may be actually based on releasing it to the mail queue).

This is something I could implement easily as long as there's no 
requirement that the mail store verify the message was actually released 
from the mail queue at the future release time. But I'm not sure it's a 
good design as it raises some issues... Messages in the futurerelease 
mailbox have been submitted/sent, so should that mailbox also have the 
\Sent special use? And if we then end up with more than one mailbox with 
\Sent special use, will mail clients need code to prefer the one without 
the "futurerelease" special use for sent messages that aren't future 
release?

I think a futurerelease virtual mailbox (based on the sent mailbox) is a 
cleaner way to do this (and optionally a sentandreleased virtual mailbox 
for clients desiring that).

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 08 May 2017 16:00:19 +1000

On Thu, 4 May 2017, at 07:16 AM, Chris Newman wrote:
> I prefer a protocol model where it is possible for the client
> to easily> identify "future release date is in the future" messages and to
> apply an> "unsend" action to those messages. I further believe the model
> shouldn't> limit the "unsend" action so it can only be used with that set of
> messages.

I agree.

> And I'd prefer a model where there is no explicit state change
> in the mail store when a "future release" message is released as that> keeps the mail queue and mail store components independent.

I think this is the one real disagreement left, but I certainly take
your point that you would prefer not to have to do this, so we should
try to find a solution where this is optional.
> I'm ok if the base JMAP spec requires implementation of future release> and unsend for future release messages.

I'm not actually sure we should mandate servers implement future release
in order to claim base JMAP support, but we should have the interface
specified for servers that do support it.
> I'd prefer to leave details about message tracking (including whether
> a message can unsent) and unsend for other messages to a JMAP
> extension.
Agreed.

> Virtual/smart mailboxes are a common mail UI feature. As a mail user,> I'd like my virtual mailboxes to look the same in different
> mail clients> to the extent possible.

I agree these are very useful, and it would be great if they were
exposed at the protocol layer so they could be synchronised between
clients. I'm not sure whether it's a good idea to try to put them in
the base JMAP Mail spec though; depends how much we think they will
slow down implementation, and whether they're strictly necessary. If
the counts are optional then they should be pretty easy to
implement, but every extra "optional" is of course more effort to
maintain down the line.
>> One proposal that might satisfy everyone: a "futurerelease"
>> mailbox role is added (this is the IMAP SPECIAL-USE representation in>> JMAP).  Servers can **optionally** have a mailbox with this
>> role; if it>> does, clients SHOULD place future-release messages in here, but like>> "sent" **can** put the message in any mailbox(es). Servers that
>> have this>> mailbox MUST move messages from this mailbox to the sent mailbox at
>> future release time (which may just be based upon the future-release>> date, or may be actually based on releasing it to the mail queue).
> 
> requirement that the mail store verify the message was actually
> released> from the mail queue at the future release time. But I'm not
> sure it's a> good design as it raises some issues... Messages in the futurerelease> mailbox have been submitted/sent, so should that mailbox also have the> \Sent special use?

No, I'd say it would just have a "futurerelease" role. I think this is
reasonable; clients which understand this role have all the information
they need from this, and clients that don't may be confused by two
mailboxes with the "sent" role.
>  I think a futurerelease virtual mailbox (based on the sent
>  mailbox) is a> cleaner way to do this (and optionally a sentandreleased
> virtual mailbox> for clients desiring that).

The issue with this is synchronising changes with the server. The
results of a message list (a sorted query on the set of messages)
normally only change in response to some kind of state change on the
server. If futurerelease were a *virtual* mailbox, the results would
change depending on *when* you fetch the query, even if there is no
change in server state. So this would require a new mechanism to notify
the client of when the query results might have changed (presuming you
want a client to be able to be always up to date without polling
everything, which I think is a reasonable requirement).
Neil.