Subject: [Jmap] Best vs Good enough - adoption of JMAP

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 22:43:24 +0000

On the topic of best vs good enough.  I agree they are perhaps opposed, 
you could say that good enough is the enemy of best however, since once 
you have something that is good enough, your incentive to do any better 
is greatly diminished.

However, we are up against several obstacles when it comes to adoption 
of JMAP.

1. Server authors won't want to add support unless there are clients
2. Client authors won't want to add support unless there are servers
3. Client or server authors won't want to add support unless they can 
make things significantly better for mail users.
4. We are losing customers to alternate messaging systems.

If our target is just going to end up providing no benefits to users 
over what we already have with IMAP+SMTP, then nobody will bother.

Where it gets interesting for me personally at least is if we can push 
the boundary of mail experience.  In many cases this can be done in the 
MUA but in some cases it requires support from the protocols.

I hear the arguments that the more complex we make the spec, the less 
likely it will be implemented, at least correctly. I'm not a fan of 
complexity either, and neither of my suggestions do I consider to be 
particularly complex.

My argument is that if the design doesn't hold out a carrot for better 
mail experience for users, then nobody will bother, and we're wasting 
our time here, which would be a shame because we have the potential to 
achieve a lot here.

At the moment, there is a benefit on the table, mostly being the ability 
to only require configuration of access to 1 service (that's if JMAP 
does submission).

I believe we should be looking for opportunities to make mail the best 
experience possible, come up with ideas, and we may decided to defer 
some or not implement others, but we should keep them in mind so that 
perhaps we don't close off the path to some of them.  We don't need 
another protocol that is hamstrung by lack of foresight.

there are plenty of ideas around, look at the competition.

Cheers

Adrien

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 24 Apr 2017 17:39:12 -0700

On 24 Apr 2017, at 15:43, Adrien de Croy wrote:
> On the topic of best vs good enough.

This is a misleading dichotomy for an engineering discussion. My 
immediate response when you say "best" is "best for whom?" If you design 
the best possible protocol for end-user experience, it will place 
unreasonable requirements on servers and won't deploy. Good engineering 
involves tradeoffs, cost/benefit analysis, deployability analysis, 
extensibility design, etc.

> I agree they are perhaps opposed, you could say that good enough is 
> the enemy of best however, since once you have something that is good 
> enough, your incentive to do any better is greatly diminished.
>

I've never heard of a mail client author implementing a feature before 
the server exists. Server authors are the ones who have moved first in 
the email community. So the key to JMAP deployability is to make it not 
unnecessarily hostile to server developers (like myself) and server 
operators. But I agree we also want JMAP to provide significant benefits 
to client authors and their users. There are many design choices JMAP 
can make that meet all of these goals.

> the boundary of mail experience.  In many cases this can be done in 
> the MUA but in some cases it requires support from the protocols.
>
> At the moment, there is a benefit on the table, mostly being the 
> ability to only require configuration of access to 1 service (that's 
> if JMAP does submission).
>

Email is the most widely deployed multi-vendor messaging system on the 
Internet. The major value of doing JMAP as a standard is to get 
multi-vendor deployment. A walled garden will always have features that 
an open multi-vendor system can't achieve (simply for privacy reasons). 
Yes, we should look at the features that are popular in walled garden 
messaging systems and do a cost/benefit analysis. If the cost is low and 
benefit is high, let's consider it. But if the cost is high or the 
feature will involve a lot of careful design work and is outside the 
charter scope, then you'll get opposition to adding that feature to the 
JMAP base spec.

A multi-vendor standard needs to support both small and large 
deployments and support both cloud and on-premises deployments. There 
are hundreds of deployed IMAP and Submission server implementations on 
the Internet and JMAP will deploy a lot faster if it "plays well" with 
those implementations. And by "play well", I mean it can be added to an 
existing deployment without major infrastructure changes.

For example, if I'm operating a million-user deployment, it's critical 
to monitor the mail queues for problems. The proposed JMAP "outbox" 
model means I'm adding one million new mail queues to the monitoring 
system. That's a major deployment change; and thus a significant barrier 
to JMAP deployability.

Suppose we remove the "outbox" model from JMAP and replace it with a 
submission command with envelope extensions. Now suppose we want to 
require JMAP servers to support BINARYMIME submission (and/or append) as 
a boon to JMAP submission client authors? Well that's no problem from an 
infrastructure viewpoint as long as the JMAP server is permitted to 
perform the base64-encoding on binary message parts at submission/append 
time in order to remain compatible with IMAP mail stores (and 
non-BINARYMIME submission servers). Want to have the JMAP submission 
command handle both the \Sent folder copy and the Submission without 
duplicate data transmission? I think we need to do that. Want to have 
the base JMAP submission or append command include IMAP CATENATE 
functionality (assemble a message including server-stored attachments in 
order to "forward without download")? I have no problem with that.

There's quite a bit of room to improve the JMAP client author's 
experience without forcing major infrastructure changes. The WG exists 
to discuss the cost/benefit of various features and make an engineering 
judgment call. The proposals that are most likely to succeed are ones 
that can be implemented in a JMAP proxy without significant changes to 
IMAP/Submission. And proposals that would require reasonably 
straightforward or well-understood IMAP/Submission extensions are worth 
considering (e.g., if I need a per-user unique/persistent message 
identifier for IMAP, I understand the implications of that).

So the opposition you're seeing is to making fundamental architecture 
changes to the Internet email infrastructure or adding complex new 
mandatory features that have a poor cost/benefit. If you want to do 
that, go design your own walled-garden protocol. But if you want to make 
the client author experience better in a way that can deploy across 
email systems from multiple vendors, then this is the place to make and 
refine proposals that strike a deployable balance.

Sorry for the meta-discussion, but I got the impression you had 
misinterpreted opposition to infrastructure disruption as opposition to 
improving the client experience.

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Mon, 24 Apr 2017 20:54:13 -0400

On Apr 24, 2017, at 8:39 PM, Chris Newman <[email-masked]> wrote:
> I've never heard of a mail client author implementing a feature before the server exists. Server authors are the ones who have moved first in the email community. So the key to JMAP deployability is to make it not unnecessarily hostile to server developers (like myself) and server operators. But I agree we also want JMAP to provide significant benefits to client authors and their users. There are many design choices JMAP can make that meet all of these goals.

This seems like a problem, though.   If the whole thing is geared toward server vendors, then UX is out not at all visible: you don't get to find out about UX until you have an MUA.

> For example, if I'm operating a million-user deployment, it's critical to monitor the mail queues for problems. The proposed JMAP "outbox" model means I'm adding one million new mail queues to the monitoring system. That's a major deployment change; and thus a significant barrier to JMAP deployability.

This doesn't really follow.   It's perfectly possible that the way that the backend implements this is that when something is stuffed in outbox, it's put in the global mail queue, moved to sent, and the delivery status in sent is "pending."   Now the MUA can tell that the message has been queued, but no delivery attempt has been made.   Once the delivery attempt is completed, the MTA returns a status to the JMAP server indicating that the mail has been sent, or that it failed.   Or it doesn't; if it doesn't support this, then we just mark the message as "sent" as soon as it is queued.   But the facility for a richer status report is present.   In principle the JMAP server can use properly formatted and authenticated bounce messages to update delivery status if that is available.   (obviously, iterate across all the destination addresses on the message.)

> Well that's no problem from an infrastructure viewpoint as long as the JMAP server is permitted to perform the base64-encoding on binary message parts at submission/append time in order to remain compatible with IMAP mail stores (and non-BINARYMIME submission servers).

I'm probably revealing my naivete here, but _seriously_?   IMAP allows 8-bit messages, and even provides a way to require that clients support them.   Are you saying that these million-user IMAP servers are so ancient and clunky that they don't do that, or am I missing something?

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 25 Apr 2017 01:05:35 +0000

------ Original Message ------
From: "Chris Newman" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 25/04/2017 12:39:12 PM
Subject: Re: [Jmap] Best vs Good enough - adoption of JMAP

>immediate response when you say "best" is "best for whom?" If you 
>design the best possible protocol for end-user experience, it will 
>place unreasonable requirements on servers and won't deploy. Good 
>engineering involves tradeoffs, cost/benefit analysis, deployability 
>analysis, extensibility design, etc.
>
>the email community.
there are a few vendors who provide both, and they are in a special 
situation where they can really push the envelope on features because 
they have control of both sides of the conversation.

This is why Google was able to push SPDY, and why HTTP/2 is like it is.

>So the key to JMAP deployability is to make it not unnecessarily 
>hostile to server developers (like myself) and server operators. But I 
>agree we also want JMAP to provide significant benefits to client 
>authors and their users. There are many design choices JMAP can make 
>that meet all of these goals.
>
>Internet.
This one is rapidly moving.  I'd agree it's most widely deployed, but I 
don't know if it's most widely used.  Since mobiles have overtaken 
desktops as internet endpoints, I'd imagine Facebook messaging is 
getting close in terms of number of people using it, if it hasn't 
surpassed mail already.

>The major value of doing JMAP as a standard is to get multi-vendor 
>deployment. A walled garden will always have features that an open 
>multi-vendor system can't achieve (simply for privacy reasons). Yes, we 
>should look at the features that are popular in walled garden messaging 
>systems and do a cost/benefit analysis. If the cost is low and benefit 
>is high, let's consider it. But if the cost is high or the feature will 
>involve a lot of careful design work and is outside the charter scope, 
>then you'll get opposition to adding that feature to the JMAP base 
>spec.
understood.  I'm not proposing creating any more walled gardens.  
However walled gardens with open protocols, and which avoid privacy 
issues are fair game.

>existing deployment without major infrastructure changes.
yes.  It has to be deployable on all levels (e.g. meet policy and 
commercial requirements as well as technical).

>system. That's a major deployment change; and thus a significant 
>barrier to JMAP deployability.
>
>require JMAP servers to support BINARYMIME submission (and/or append) 
>as a boon to JMAP submission client authors? Well that's no problem 
>from an infrastructure viewpoint as long as the JMAP server is 
>permitted to perform the base64-encoding on binary message parts at 
>submission/append time in order to remain compatible with IMAP mail 
>stores (and non-BINARYMIME submission servers). Want to have the JMAP 
>submission command handle both the \Sent folder copy and the Submission 
>without duplicate data transmission? I think we need to do that. Want 
>to have the base JMAP submission or append command include IMAP 
>CATENATE functionality (assemble a message including server-stored 
>attachments in order to "forward without download")? I have no problem 
>with that.
Yes, I think clients need to be able to build the message on the server, 
in binary.

>experience without forcing major infrastructure changes.

I guess my point here is that I don't believe we can force major 
infrastructure changes even if we decided we wanted to.  I didn't think 
I was proposing anything that would do that.

>The WG exists to discuss the cost/benefit of various features and make 
>an engineering judgment call. The proposals that are most likely to 
>succeed are ones that can be implemented in a JMAP proxy without 
>significant changes to IMAP/Submission. And proposals that would 
>require reasonably straightforward or well-understood IMAP/Submission 
>extensions are worth considering (e.g., if I need a per-user 
>unique/persistent message identifier for IMAP, I understand the 
>implications of that).
OK, so we have an IMAP server and SMTP server.  I'm intending (if it 
works out) to implement JMAP, but I would be making whatever changes 
needed to the IMAP and SMTP servers required to provide whatever added 
benefit I can give to a user by way of JMAP.

I don't really think of JMAP as a proxy.  I think of it on an even par 
with IMAP + SUBMIT servers.  I'm not intending to proxy JMAP back via 
IMAP, we abstracted the store from IMAP in any case (so it could be 
shared with POP3), so the JMAP server would be another user of that mail 
store.

>that, go design your own walled-garden protocol. But if you want to 
>make the client author experience better in a way that can deploy 
>across email systems from multiple vendors, then this is the place to 
>make and refine proposals that strike a deployable balance.
>
No problem, I appreciate your response.

Maybe I haven't been clear in my suggestions, but I'm not proposing 
making all these suggestions mandatory, but I am keen that they remain 
possible, and not get prohibited by some design decisions now.

Being able to do something prior to submission relating to submission as 
a general concept I think has some interesting possibilities, and early 
mail verification I think has a lot more value than some are seeing.  
When you've had your umpteenth mail bounced that took you hours to 
write, you end up wishing something told you the mail address was bad at 
the beginning.

I still think as a general principle, that failing fast and early is 
preferable to slow and late.  that's where the email address 
pre-validation idea came from.

As for the other one of message notification.  I also think keeping the 
human in the loop has all manner of benefits.  With HTTP it may be 
easier to see, since people hit the refresh button over and over if the 
page doesn't load.  We built a variant of Chrome that worked with our 
HTTP status response codes during AV scanning, and the usability was 
greatly improved.

I think as a general principle, keeping people informed of what's going 
on, at a much greater level than we currently do, can save people a lot 
of wasted effort and worry.

Cheers

Adrien

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 24 Apr 2017 18:41:49 -0700

On 24 Apr 2017, at 17:54, Ted Lemon wrote:
>> I've never heard of a mail client author implementing a feature 
>> before the server exists. Server authors are the ones who have moved 
>> first in the email community. So the key to JMAP deployability is to 
>> make it not unnecessarily hostile to server developers (like myself) 
>> and server operators. But I agree we also want JMAP to provide 
>> significant benefits to client authors and their users. There are 
>> many design choices JMAP can make that meet all of these goals.
>
> This seems like a problem, though.   If the whole thing is geared 
> toward server vendors, then UX is out not at all visible: you don't 
> get to find out about UX until you have an MUA.

Yes, this has been a common problem in design of mail protocols. If you 
have any ideas on how to keep the mail client authors who think more 
about UX issues engaged in the JMAP standardization debate, I'd love to 
hear them.

>> For example, if I'm operating a million-user deployment, it's 
>> critical to monitor the mail queues for problems. The proposed JMAP 
>> "outbox" model means I'm adding one million new mail queues to the 
>> monitoring system. That's a major deployment change; and thus a 
>> significant barrier to JMAP deployability.
>
> This doesn't really follow.   It's perfectly possible that the way 
> that the backend implements this is that when something is stuffed in 
> outbox, it's put in the global mail queue, moved to sent, and the 
> delivery status in sent is "pending."   Now the MUA can tell that the 
> message has been queued, but no delivery attempt has been made.   Once 
> the delivery attempt is completed, the MTA returns a status to the 
> JMAP server indicating that the mail has been sent, or that it failed. 
>   Or it doesn't; if it doesn't support this, then we just mark the 
> message as "sent" as soon as it is queued.   But the facility for a 
> richer status report is present.   In principle the JMAP server can 
> use properly formatted and authenticated bounce messages to update 
> delivery status if that is available.   (obviously, iterate across all 
> the destination addresses on the message.)

True, if the WG finds rough consensus around the outbox model despite my 
objections, then my implementation will have an always-empty outbox as 
that's the only way to implement it without infrastructure disruption. 
I'd prefer a JMAP design that allows me to provide delivery status UX 
without infrastructure disruption.

>> Well that's no problem from an infrastructure viewpoint as long as 
>> the JMAP server is permitted to perform the base64-encoding on binary 
>> message parts at submission/append time in order to remain compatible 
>> with IMAP mail stores (and non-BINARYMIME submission servers).
>
> I'm probably revealing my naivete here, but _seriously_?   IMAP allows 
> 8-bit messages, and even provides a way to require that clients 
> support them.   Are you saying that these million-user IMAP servers 
> are so ancient and clunky that they don't do that, or am I missing 
> something?

Pretty much everything deployed is 8-bit clean in message bodies (UTF-8 
does not need to be encoded for IMAP). But a lot of the email 
infrastructure is not binary clean (e.g., a lot of mail stores will not 
store binary MIME with NUL octets). Now we do have BINARYMIME/CHUNKING 
extensions for SMTP as well as binary append and fetch extensions for 
IMAP. So I have no problem with JMAP requiring support for binary MIME 
on the wire; but it shouldn't say anything that prevents the server from 
converting to 8-bit MIME (using base64 to encode parts with NUL octets) 
for storage purposes.

		- Chris

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 25 Apr 2017 11:47:07 +1000

On Tue, 25 Apr 2017, at 11:41 AM, Chris Newman wrote:
> Yes, this has been a common problem in design of mail
> protocols. If you> have any ideas on how to keep the mail client authors who think more
> about UX issues engaged in the JMAP standardization debate, I'd love
> to> hear them.

As it happens I lead UX for FastMail, and the other proposed co-editor
of the JMAP Mail spec is also a client author. So I'm hopeful that JMAP
may do slightly better in this regards than some previous protocol
design work.
Neil.

---

From: "Chris Newman" <[email-masked]>
Date: Mon, 24 Apr 2017 19:07:17 -0700

On 24 Apr 2017, at 18:05, Adrien de Croy wrote:
> shared with POP3), so the JMAP server would be another user of that 
> mail store.

Our JMAP-like component was initially implemented as a peer access 
protocol to our IMAP & POP servers using our internal store API. At one 
point we deliberately converted it to be an IMAP proxy. That turned out 
to be a very good architecture decision for our product. It improved our 
mail store reliability and scalability by providing greater functional 
separation and a better deployment model. So I view JMAP as an 
IMAP/Submit proxy based on that experience.

POP3 is such a simple protocol that it's not worth the effort to convert 
it from an internal API consumer to a proxy, but if I started a design 
from scratch, I'd implement POP3 as a proxy to an IMAP backend.

		- Chris

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 25 Apr 2017 15:51:34 +1000

On Tue, 25 Apr 2017, at 10:54, Ted Lemon wrote:
> On Apr 24, 2017, at 8:39 PM, Chris Newman
> <[email-masked]> wrote:>> For example, if I'm operating a million-user deployment, it's
>> significant barrier to JMAP deployability.> This doesn't really follow.   It's perfectly possible that the way
All this is assuming that you have a writeback rather than writethrough
Outbox.  The implementation of Outbox in the perl JMAP proxy, for
example, is synchronous and emphemeral.  Nothing can ever actually exist
in the Outbox.  It's 79 lines of perl - currently here:
https://github.com/jmapio/jmap-perl/blob/master/JMAP/ImapDB.pm#L1116

It formulates the SMTP command, submits the content, then puts the
message in the Sent folder, all within a single DB transaction.
It's easy to create a JMAP server which doesn't hold messages in a queue
in the Outbox.  Though we also need event handling for Calendar Alerts
and other fun stuff in our server, so having those queues include
Outboxes isn't actually a huge additional management expense.  The
server will already have tooling for tracking events which are supposed
to fire for newly added messages, and will already have error reporting
for any errors.  The fact that there are a million mailboxes (FastMail's
infrastructure is also on this scale) doesn't materially affect the
amount of work once there's automation.  No matter how many mailboxes
you have, a good infrastructure will have retry ability and quite
reliable outbound queue injection.
You just don't get failures transferring from Outbox into the regular
SMTP queues.  And if you do, at least the messages are within your
system rather than in someone's client where you wind up getting reports
of failures through support channels and are sitting there on third line
support trying to debug SMTP issues on somebody's local computer with
software outside your control on some bogus network with bogus
antivirus.
I much prefer debugging message transfer between two systems that I have
full control over.
Bron.

--
  Bron Gondwana
  [email-masked]

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 25 Apr 2017 15:56:09 +1000

> without infrastructure disruption.

Ahh, I see you already replied with this.  Yeah, always-empty outbox is my first plan too as a server implementer.  It's what I did for the proxy, since it doesn't have reliable storage.

The reason to hold email in the outbox temporarily is to implement both undo-send and delayed send, both of which are popular features in new mail clients.  We'll definitely do outbox-as-queue at FastMail eventually.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 25 Apr 2017 16:01:22 -0700

On 24 Apr 2017, at 22:56, Bron Gondwana wrote:
> Ahh, I see you already replied with this.  Yeah, always-empty outbox 
> is my first plan too as a server implementer.  It's what I did for the 
> proxy, since it doesn't have reliable storage.
>
> The reason to hold email in the outbox temporarily is to implement 
> both undo-send and delayed send, both of which are popular features in 
> new mail clients.  We'll definitely do outbox-as-queue at FastMail 
> eventually.

Our infrastructure already supports FUTURERELEASE and an undo send 
capability to go with it. I'd love to expose those features to mail 
clients. So if JMAP had a submission command with envelope options and 
an undo send command that keyed off the \Sent folder; it'd be no problem 
for our implementation to provide both delayed send and undo send as 
client features. But if JMAP has the outbox model that forces mailstore 
semantics and queue semantics to be combined in one object, that's a 
huge amount of extra work on the server so I'd opt for the always-empty 
outbox approach (mostly for the improved security and scalability of a 
much simpler implementation).

So given we have at least two server implementers on the list who will 
do always-empty outbox for the first implementation, I think that begs 
the question of what's the more important feature for JMAP to offer:
A) outbox model
B) undo send and delayed send

If B is more important than A, I think we need to replace the outbox 
model with a simpler submit/undo-submit model so it's easier for servers 
to provide those features to clients.

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Tue, 25 Apr 2017 19:05:37 -0400

On Apr 25, 2017, at 7:01 PM, Chris Newman <[email-masked]> wrote:
> So given we have at least two server implementers on the list who will do always-empty outbox for the first implementation, I think that begs the question of what's the more important feature for JMAP to offer:
> A) outbox model

Obviously the cure is to combine the sent folder and the outbox...

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 09:46:14 +1000

I think there's been some misconception over the intended semantics of
the outbox proposal. The outbox was never intended to be the submission
queue, with mail queue semantics. It is intended as a standard mailbox,
with mailstore semantics, because it is where messages go **before**
they are passed on to submission. The server would validate the
message/envelope at the time of moving it to the outbox to ensure it can
be accepted for submission later of course. If you have existing
mailstore + SMTP infrastructure, implementation is not much more than
having some small demon that triggers to take the message from the
mailstore and submit it to the SMTP queue at the appropriate time.
>From a user experience point of view, this model gives us an
understandable conceptual place for messages that are *going* to be
sent but have not yet been picked up by the postman, to use a real
world analogy. Because it is a standard mailbox, we automatically have
an API for the client to fetch and show to the user all the currently
scheduled messages. I think this is important functionality as a user
if we have delayed send (and of course we also have the ability to then
cancel any of the future submissions, by just moving the messages back
out of the outbox).
With my client author hat on, I like this model because it makes it
easier to implement full offline support. If I am offline, the outbox
represents messages waiting to go out from the client. From an
implementation perspective it is much easier if my offline sync code
doesn't need a separate path for keeping track of messages to be sent
vs. normal messages moved to another folder.
The *only real difference* between a "sendMessage" command and using
"move to outbox" as the equivalent command, is that clients can get
the list of "delayed" messages waiting for submission. I think this is
valuable, and pretty much required for "scheduled send", although not
so important for "undo send". The fact that this is really the only
difference is why servers that can't or don't want to implement
delayed send or expose messages waiting to be sent can use the "empty
outbox" model.
Neil.

---

From: "Chris Newman" <[email-masked]>
Date: Tue, 25 Apr 2017 21:49:54 -0700

> mailstore and submit it to the SMTP queue at the appropriate time.

And as soon as you add that "small daemon" (also known as a mail queue 
processing daemon), you've now required the mailstore to also have mail 
queue semantics. This includes key semantics such as: the mail queue 
daemon has full read / delete access to those messages even when the 
user is offline. If the delivery doesn't happen when the user expects, 
the user is going to make a support call, so we now need all the 
monitoring, logging and robustness that comes with a mail queue. And 
addresses that were valid at the time the message entered this mail 
queue can become invalid later, so there has to be bounce-processing for 
when that happens, etc, etc.

The only way to have a server-side outbox that is not a mail queue would 
be if messages sit in the outbox indefinitely unless the user is online 
and their client decides to issue a submit command for a message in that 
fictitious outbox.

> world analogy.

This analogy only works with a desktop MUA where the "outbox" is on the 
desktop and thus owned by the "postal customer" (and incidentally, the 
outbox won't empty if the desktop is shut down). The hardware running 
JMAP is not owned but the "postal customer"; it's effectively owned by 
the postman. So an "outbox" on the JMAP server means the message has 
been given to the postman already; the postman is just waiting for the 
right time to release it to the next hop.

> Because it is a standard mailbox, we automatically have

The problem with server-side-outbox in JMAP is that it constrains the 
server implementation to three options:

1. The mail store includes a mail queue for each user.
2. The mail queue, at least for delayed messages, can be accessed with 
full JMAP semantics. Including things like body structure, mod sequence, 
flags, etc.
3. The JMAP outbox is always empty.

I really want a fourth option for server implementation:

4. The mail store and mail queue are completely separate. The mail queue 
daemon never accesses the mail store. The set of actions JMAP performs 
on objects in the mail queue is limited to only those that are strictly 
necessary to achieve a useful goal (envelope extensions like future 
release at submission time, unsend for messages still in local queue, 
and status feedback tagging on sent messages).

> vs. normal messages moved to another folder.

A client-side outbox makes sense to me. It's a server-side outbox that I 
find problematic. Let me ask you this:

Suppose JMAP implements a server-side outbox (aka mail queue). The JMAP 
client synchronizes the server-side outbox and goes offline. While the 
client is offline, the message gets submitted and the user decides they 
don't want to submit the message and delete it from the local cached 
outbox. Now what? Have you really simplified things by adding a 
server-side outbox to the mix?

> delayed send or expose messages waiting to be sent can use the "empty

My product has already implemented future release, already implemented 
unsend from local mail queue and implemented message tracking so we can 
get status of a message in the local queue (and potentially beyond 
that). I'm willing to expose these in JMAP if the protocol is designed 
in such a way that we preserve clean functional separation between mail 
store and mail queue. I'm very much not interested in all the complexity 
a server-side outbox adds on top of those three features.

I'd argue that "scheduled send" is a mail client feature and "future 
release" is a mail server feature. If there's a daemon that can move the 
message to the next hop even when the client is offline, you're talking 
about "future release" and the message has semantically been submitted 
already. So I think JMAP should put "future release" messages in the 
Sent folder to avoid confusion with a client-side outbox. I'd like JMAP 
to include a way to "unsend" future release messages so JMAP clients 
have access to the unsend feature we've already implemented.

		- Chris

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 26 Apr 2017 15:41:00 +1000

> when that happens, etc, etc.

Disagree on bounce processing.  It's no different than the next SMTP hop in that regard.

It's exactly the same as other scheduled stuff we already support in our server such as calendar alarms based on calendar events - which we happen to store in the same backend - so it already needs event scheduling.

I even blogged about our plans

https://blog.fastmail.com/2015/12/25/a-ghost-of-fastmail-future/

(complete with over-enthusiastic date predictions)

But adding the message to the Outbox also generates a full logged, replicated, replayable event in the #events mailbox which will be processed by a single queue daemon per server, and which will be monitored in the same way we monitor all our queues - by injecting messages at the start and making sure they are flowing out the other end.

> and their client decides to issue a submit command for a message in that 

Yeah, that makes delayed outbox mostly pointless.  Particularly in the multi-client world.

> right time to release it to the next hop.

We could imagine it in the case where there's a company "postman" who's still in the building for a pre-determined amount of time before they take the packages down to the central processing facility, and while it's still in the building you can waylay the postman and ask for it to removed from the postbag...

> and status feedback tagging on sent messages).

You appear to be proposing one of:

a) there is no support for delayed sending or undo
b) the user has no visibility or control of queued messages after submitting them
c) the user can control queued messages, but needs to use a different protocol than JMAP to do it
d) a new object type for queued messages needs to be defined in JMAP, with operations that allow achieving the above useful goals.

If (d), are you willing to write up the object definitions and client advice for how they would be used, so they can be tested?

> server-side outbox to the mix?

When the client re-syncs, it sees that the message it tried to move back out to Drafts no longer exists, but there's a new message in Sent.  The user notices that the send happened, oh well.

This is the same as if you hit "undo" in the Gmail interface after sending a message, but unfortunately your internet connection dropped for 30 seconds, and it was too late.

Clearly the client can't know for sure that the undo succeeded until it gets confirmation from the server.  In functional programming parlance, moving messages in and out of the "Outbox" folder is impure - it has side effects.

> have access to the unsend feature we've already implemented.

If you could write up how, over JMAP, a client could tell that a message was future release and interact with its queue, that would probably help everyone understand how what you're proposing would work.

Thanks,

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Ned Freed <[email-masked]>
Date: Wed, 26 Apr 2017 06:45:51 -0700 (PDT)

> when that happens, etc, etc.

Now take these semantics and multiply them by the number of users you have,
which can be in the many millions. And while sometimes you can treat
this as single, system-wide queue for monitoring purposes, sometimes
you'll need to lookup and probably monitor on a per-user basis.

I also assume there won't be a hardcoded name or location for this special
use mailbox. So its specialness arises from some sort of attribute. Can
that attribute be set? Reset? Can it be set on more than one mailbox?
If not why not?

It's not that I can' think of various possible ways to implement all this at
scale. I can. But none of them are easy.

> and their client decides to issue a submit command for a message in that

At which point I don't see why you're bothering with it.

> right time to release it to the next hop.

Actually, I think it's a great analogy. In the US at least, the post office
imposes specific requirements on mailboxes, including the fact that 
something put in one is now in the custody of the post office, and only
the mailbox owner can remove it.

But the part I like best about the analogy is that while the ability
to have a mailbox at every address in the country is incredibly convenient
for the customer, the cost to the postal service is absolutely staggering.
So much so that nobody other than the post office itself - who is obligated
by history and custom - provides this service at scale.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Wed, 26 Apr 2017 10:34:18 -0400

On Apr 26, 2017, at 12:49 AM, Chris Newman <[email-masked]> wrote:
> Suppose JMAP implements a server-side outbox (aka mail queue). The JMAP client synchronizes the server-side outbox and goes offline. While the client is offline, the message gets submitted and the user decides they don't want to submit the message and delete it from the local cached outbox. Now what? Have you really simplified things by adding a server-side outbox to the mix?

Why is the message in the outbox if the user doesn't want to submit it?   It should be in the drafts folder.

It's a real problem if the outbox and the drafts folder are not synchronized between clients.   Insisting that the outbox be only on the client means that every client has a different view of the state of the mail store, even when all clients are up to date.   This is confusing and inconvenient for users.

And why is it a problem for the submit service to remove stuff from the outbox when the client isn't online?   Effectively, it's just another client making changes, something that JMAP is designed to deal with very cleanly.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 26 Apr 2017 10:39:28 -0400

> you'll need to lookup and probably monitor on a per-user basis.

Of course, when the user calls in with a problem with their email, the fact that all the information about that user's email is in their unique mail store is kind of convenient.   If there is a systemic problem with mail delivery, you would need different tools to notice it.   I don't see how one or the other of these approaches is clearly superior; what is true is that the latter approach is what people are used to.

---

From: Ned Freed <[email-masked]>
Date: Wed, 26 Apr 2017 07:41:28 -0700 (PDT)

> Of course, when the user calls in with a problem with their email, the fact
> that all the information about that user's email is in their unique mail store
> is kind of convenient.

Actually, it offers no utility whatsoever. If you provide the ability to answer
such queries at high scale, you do it based on trasactional information
produced by the message transport infrastructure. This information is fed into
some kind of database that lets you lookup whatever you want for whatever
combination of sender, recipient, time window, etc. you want to support.

You most certainly don't want to do it by accessing the mailstores. They
carry enough load as it is; using them to handle a problem that's easily
separable makes no sense.

There's also a nontrivial security separation concern here. The access rights
associated with being able to look at transaction data are not the same as the
rights needed to look at message content.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Wed, 26 Apr 2017 10:56:17 -0400

> rights needed to look at message content.

These are all good points, but from this perspective then it doesn't really matter where the transaction happens as long as the database can be updated.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 26 Apr 2017 10:54:11 -0700

On 25 Apr 2017, at 22:41, Bron Gondwana wrote:
> d) a new object type for queued messages needs to be defined in JMAP, 
> with operations that allow achieving the above useful goals.
>
> If (d), are you willing to write up the object definitions and client 
> advice for how they would be used, so they can be tested?

I'll describe a model that achieves my goal (which is to provide the 
desired future release & unsend functionality without requiring the 
server to implement a storage model with combined mail store and mail 
queue semantics). I don't want to get bogged down in specification 
details yet.

Here's one way to accomplish my goal:

1. Get rid of the server-side outbox. We only have a drafts folder and a 
sent folder.

2. Add a submission command with envelope (same structure as SMTP 
envelope but JSON syntax and quoting). We pass this command a reference 
to a draft message in the drafts folder, and if the submission succeeds, 
the message is moved to the sent folder with the envelope as an 
annotation. The envelope can include the already-standard future release 
extension.

3. Add an unsend command which takes a reference to a message in the 
sent folder. If the unsend succeeds, the message can be moved back to 
the drafts folder as part of this action. How the server correlates a 
message in the sent folder with a message in the mail queue is an 
implementation detail; the spec need not specify that detail in order to 
provide the functionality with this model.

4. Add a way to search/select messages in the Sent folder that may be 
possible to unsend.

This model has a lot of advantages over the outbox model:

* Every server that supports future release and unsend will provide the 
same interface to the client. No need for the "always empty outbox" 
alternative server behavior that you'll see with the outbox model.
* This does not unnecessarily constrain server implementation, and 
permits simpler server designs.
* This preserves clean mail store and mail queue separation including 
the security and operational benefits that provides.
* This makes it clear when the message enters the submission system. 
There's no confusion as one has with an outbox model where a message in 
the outbox may be submitted (if it's on the server) or maybe not (if the 
client is offline when the message moves to the outbox).
* Submission can have semantics fully compatible with existing 
infrastructure, making this easier/faster to deploy.
* The "outbox" model constrains unsend to only work if the message is in 
the outbox. This model has no such constraint and works the same 
regardless of where a message that can be "unsent" lives in the 
infrastructure.

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Wed, 26 Apr 2017 15:09:07 -0400

On Apr 26, 2017, at 1:54 PM, Chris Newman <[email-masked]> wrote:
> I'll describe a model that achieves my goal (which is to provide the desired future release & unsend functionality without requiring the server to implement a storage model with combined mail store and mail queue semantics).
> [...]

This looks good to me, FWIW. :)

---

From: Ned Freed <[email-masked]>
Date: Wed, 26 Apr 2017 10:09:03 -0700 (PDT)

> These are all good points, but from this perspective then it doesn't really
> matter where the transaction happens as long as the database can be updated.

First, it most certainly does matter where the transaction happens, and the
fact that it isn't at all clear where submission to the transport
infrastructure occurs is a big problems with these magic folder proposals.

Second, just as it makes no sense to use the store as a means of tracking
transactions - and in that regard I'll note that transactions have to be
trackable even if the associated messages have been deleted - it also makes no
sense to use the transaction log as a queue management facility. For one thing,
general-purpose databases tend to suck at implementing anything but the
simplest of queues, and mail queues are anything but simple.

The operational constraints are also very different. The transaction log is
almost always a lower priority thing than getting mail processed. So you can
take some liberties with coherence, update speed, and even reliability and save
some $$$. This is important when you realize that it's often necessary
to retain transacton records for long periods of time. 

In contrast, the queue manager database needs to be up to date at all times,
and in the event of failure it needs to be possible to sync it up to reality
in fairly short order.

				Ned

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Apr 2017 15:23:56 -0700

> scale. I can. But none of them are easy.

I think a modern mail store already has need for this type of scheduled
events for many cases, and as such is implemented by many of them.

Off the top of my head, beyond undo send/future send, there is also
automatic emptying of the trash, remind me later, and enterprise style
retention policies.
Once you have such a model, you'll find it useful for a bunch of other
things, such as data migration or applying changes across a very large mail
store.

Brandon

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Apr 2017 15:25:43 -0700

On Mon, Apr 24, 2017 at 5:39 PM, Chris Newman <[email-masked]>
wrote:

> This is a misleading dichotomy for an engineering discussion. My immediate
> response when you say "best" is "best for whom?" If you design the best
> possible protocol for end-user experience, it will place unreasonable
> requirements on servers and won't deploy. Good engineering involves
> tradeoffs, cost/benefit analysis, deployability analysis, extensibility
> design, etc.
>
> I agree they are perhaps opposed, you could say that good enough is the
>> enemy of best however, since once you have something that is good enough,
>> your incentive to do any better is greatly diminished.
>>
>> However, we are up against several obstacles when it comes to adoption of
>> JMAP.
>>
> I've never heard of a mail client author implementing a feature before the
> server exists. Server authors are the ones who have moved first in the
> email community. So the key to JMAP deployability is to make it not
> operators. But I agree we also want JMAP to provide significant benefits to
> client authors and their users. There are many design choices JMAP can make
>> If our target is just going to end up providing no benefits to users over
>> what we already have with IMAP+SMTP, then nobody will bother.
>>
>> the boundary of mail experience.  In many cases this can be done in the MUA
>> but in some cases it requires support from the protocols.
>>
>> mail experience for users, then nobody will bother, and we're wasting our
>> time here, which would be a shame because we have the potential to achieve
>> a lot here.
>>
>> to only require configuration of access to 1 service (that's if JMAP does
>> submission).
>>
>> experience possible, come up with ideas, and we may decided to defer some
>> or not implement others, but we should keep them in mind so that perhaps we
>> don't close off the path to some of them.  We don't need another protocol
>> that is hamstrung by lack of foresight.
>>
> multi-vendor deployment. A walled garden will always have features that an
> open multi-vendor system can't achieve (simply for privacy reasons). Yes,
> we should look at the features that are popular in walled garden messaging
> systems and do a cost/benefit analysis. If the cost is low and benefit is
> high, let's consider it. But if the cost is high or the feature will
> involve a lot of careful design work and is outside the charter scope, then
> you'll get opposition to adding that feature to the JMAP base spec.
>
> A multi-vendor standard needs to support both small and large deployments
> and support both cloud and on-premises deployments. There are hundreds of
> deployed IMAP and Submission server implementations on the Internet and
> JMAP will deploy a lot faster if it "plays well" with those
> implementations. And by "play well", I mean it can be added to an existing
> deployment without major infrastructure changes.
>
> For example, if I'm operating a million-user deployment, it's critical to
> monitor the mail queues for problems. The proposed JMAP "outbox" model
> means I'm adding one million new mail queues to the monitoring system.
> That's a major deployment change; and thus a significant barrier to JMAP
> deployability.
>
> submission command with envelope extensions. Now suppose we want to require
> JMAP servers to support BINARYMIME submission (and/or append) as a boon to
> JMAP submission client authors? Well that's no problem from an
> infrastructure viewpoint as long as the JMAP server is permitted to perform
> the base64-encoding on binary message parts at submission/append time in
> order to remain compatible with IMAP mail stores (and non-BINARYMIME
> submission servers). Want to have the JMAP submission command handle both
> the \Sent folder copy and the Submission without duplicate data
> transmission? I think we need to do that. Want to have the base JMAP
> submission or append command include IMAP CATENATE functionality (assemble
> a message including server-stored attachments in order to "forward without
> download")? I have no problem with that.
>
> There's quite a bit of room to improve the JMAP client author's experience
> without forcing major infrastructure changes. The WG exists to discuss the
> cost/benefit of various features and make an engineering judgment call. The
> proposals that are most likely to succeed are ones that can be implemented
> in a JMAP proxy without significant changes to IMAP/Submission. And
> proposals that would require reasonably straightforward or well-understood
> IMAP/Submission extensions are worth considering (e.g., if I need a
> per-user unique/persistent message identifier for IMAP, I understand the
> implications of that).
>

But, if all JMAP is is a proxy of existing protocols, then it seems like an
expensive undertaking for little gain.

And does that mean any future extensions have to be also implemented in
IMAP?

Brandon

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 23:04:43 +0000

On Wed, 26 Apr 2017, at 10:25 PM, Brandon Long wrote:
> But, if all JMAP is is a proxy of existing protocols, then it seems
> like an expensive undertaking for little gain.
Agreed. JMAP is not just a proxy. We have implemented a proxy[1] to
test with, but it's not stateless (in fact quite heavyweight), as it
needs to build indexes to provide features that are not available in the
protocols it's proxying through to.
The goal with designing JMAP has been to make it possible to implement
JMAP and IMAP interfaces to the same backend mail store (so someone
could happily use both an IMAP client and a JMAP client concurrently and
see a consistent view of their data). We know the current draft achieves
this, because there's a pretty complete implementation of the current
draft in Cyrus[2]. This is a necessary property for having a migration
path, however it still gives flexibility to fix the many deficiencies of
the current standards.
Neil.

Links:

  1. https://proxy.jmap.io/
  2. http://cyrusimap.org/imap/download/installation/http/jmap.html

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 23:12:12 +0000

On Wed, 26 Apr 2017, at 05:54 PM, Chris Newman wrote:
> I'll describe a model that achieves my goal […]
> 1. Get rid of the server-side outbox. We only have a drafts folder
>    and a> sent folder.
> 
> envelope but JSON syntax and quoting). We pass this command a
> reference> to a draft message in the drafts folder, and if the submission
> succeeds,> the message is moved to the sent folder with the envelope as an
> annotation. The envelope can include the already-standard
> future release> extension.
> 
> sent folder. If the unsend succeeds, the message can be moved back to> the drafts folder as part of this action. How the server correlates a> message in the sent folder with a message in the mail queue is an
> implementation detail; the spec need not specify that detail in
> order to> provide the functionality with this model.
> 
> 4. Add a way to search/select messages in the Sent folder that may be> possible to unsend.

>From a protocol perspective, this model is basically identical to the
outbox model proposed. The only difference is that rather than put the
messages that can still be recalled in a separate "outbox" mailbox, you
put them in the sent mailbox and require some kind of search. The only
extra component your server will need is something that moves the
message from outbox to sent when it can no longer be recalled; I
seriously don't think this is a huge implementation burden. As Brandon
has pointed out, modern mailstores need this kind of "scheduled events"
plumbing for a number of user-facing features, and it has already been
implemented at scale in a number of systems.
>From a user interface perspective, I maintain it is clearer for the user
for these to be two separate mailboxes, and it also has the advantage
that mailboxes have a message count so MUAs can more easily see if there
are any messages waiting to be sent still. (I would expect most MUAs to
hide the outbox when empty.)
Neil.

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 23:27:55 +0000

And just to reinforce: users consider "sent" to be different to
"scheduled to be sent". The concept of an outbox is the common approach
that MUAs take to distinguishing this. For example:
Outlook[1]: *You can delay the delivery of an individual email message
or you can use rules to delay the delivery of all messages by having
them held in the Outbox for a specified time after you click Send.*
Boomerang for Gmail[2]: *To view all messages scheduled to be sent out,
simply go to the Boomerang-Outbox folder in your Google account.*
Mixmax for Gmail[3]: *You'll see a list of your scheduled emails in your
Outbox. You can click to Reschedule or Stop your email which moves it to
your drafts.*
MailButler for Apple Mail[4]:* Your scheduled emails will be saved in
the Outbox folder until the delivery time. […] When your email is sent
automatically, it is placed in your Sent folder, as with any other
sent email.*
I have yet to find one that puts the messages straight in your
sent folder.
Neil.

Links:

  1. https://support.office.com/en-us/article/Delay-or-schedule-sending-email-messages-026af69f-c287-490a-a72f-6c65793744ba?ui=en-US&rs=en-US&ad=US
  2. http://www.boomeranggmail.com/faq.html
  3. http://success.mixmax.com/article/35-send-later
  4. https://support.mailbutler.io/en/support/solutions/articles/6000115079-how-to-schedule-emails-

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 26 Apr 2017 23:40:41 +0000

my personal experience with the outbox folder is that it's a pain in the 
neck..

it means if I need to find a message I submitted, there are now 2 places 
to look for it instead of one, depending on its state.

This doesn't seem like a bonus to me, rather the opposite.

Adrien

------ Original Message ------
From: "Neil Jenkins" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 27/04/2017 11:12:12 AM
Subject: Re: [Jmap] simpler future release & unsend without outbox (was 
Re: Best vs Good enough - adoption of JMAP)

>>reference
>From a user interface perspective, I maintain it is clearer for the 
>user for these to be two separate mailboxes, and it also has the 
>advantage that mailboxes have a message count so MUAs can more easily 
>see if there are any messages waiting to be sent still. (I would expect 
>most MUAs to hide the outbox when empty.)
>
>Neil.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 26 Apr 2017 16:43:27 -0700

On 26 Apr 2017, at 15:25, Brandon Long wrote:
>> immediate
>> the
>> before the
>>> over
>>> the MUA
>>> ability
>>> does
>>> some
>> deployments
>> critical to
>> require
>> perform
>> experience

The JMAP charter states:

"JMAP needs to be implementable on top of an IMAP server, which also 
supports IMAP extensions specified below. The JMAP data model needs to 
remain backwards compatible with the IMAP mailstore model (where a 
message is always associated with a single mailbox), but it might 
support other underlying models (e.g. Gmail-style labels).

I largely support that constraint on the WG. Although the WG can change 
its charter to add non-controversial IMAP extensions to the list (e.g., 
draft-brandt-imap-replace-02 may fall in that category).

> And does that mean any future extensions have to be also implemented 
> in IMAP?

No. I'd implement all JMAP mail store functionality I choose to expose 
as IMAP extensions, but that's my personal implementation choice and not 
a constraint on the WG.

		- Chris

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 26 Apr 2017 23:45:52 +0000

------ Original Message ------
From: "Ted Lemon" <[email-masked]>
To: "Chris Newman" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "Bron Gondwana" <[email-masked]>
Sent: 27/04/2017 7:09:07 AM
Subject: Re: [Jmap] simpler future release & unsend without outbox (was 
Re: Best vs Good enough - adoption of JMAP)

>>queue semantics).

same

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 26 Apr 2017 23:52:54 +0000

On Wed, 26 Apr 2017, at 11:40 PM, Adrien de Croy wrote:
> my personal experience with the outbox folder is that it's a pain in
> the neck..
Ignoring the fact that this is one anecdote vs the UX design choice of a
multitude of real products, if the outbox is separate to the sent
mailbox at the protocol level it is still trivial for an MUA to present
them as a single mailbox should that be what you wish to implement. You
can easily get a message list with messages in (A OR B). Doing the
inverse is much harder.
Neil.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 26 Apr 2017 19:06:03 -0700

On 26 Apr 2017, at 16:12, Neil Jenkins wrote:

>> sent folder. If the unsend succeeds, the message can be moved back 
>> to> the drafts folder as part of this action. How the server 
>> correlates a> message in the sent folder with a message in the mail 
>> queue is an
> outbox model proposed.

Well, if you believe that and I don't believe that, then the path to 
rough consensus is to start with my proposal and refine it until you 
find it acceptable :-)

So let me refine item 4 a bit. It's my theory that JMAP should have a 
virtual folder / smart folder feature that basically behaves the same 
way as a real/physical folder to the client. I think JMAP should have 
that feature regardless of the outcome of this debate (smart/virtual 
folders is an even more dominant client UI feature than outbox). So 
let's assume we've added that feature to JMAP. Now if the client wants 
to present a server-side outbox view based on my proposal, it just 
creates an "outbox" smart folder. If the client wants to not include the 
release-in-the-future messages in the Sent folder view, it can create a 
smart folder to do that. So now the UI model you prefer is simple to 
implement and it doesn't have the deployment problems a server-side 
outbox creates.

> The only difference is that rather than put the
> seriously don't think this is a huge implementation burden.

Well, it's an implementation burden that disappears completely from the 
system with my proposal. Thus you've made an argument that my proposal 
is simpler (and thus better) than the magic-server-side-outbox model.

> As Brandon
> user

I happen to like the client-side-outbox UI model, but I believe a 
server-side-outbox is a confusing UI that violates the least 
astonishment principle whenever an offline client is used. But my 
proposal puts no constraints on the UI. If you want to express the 
server-side-outbox UI model using my proposal, we can refine my proposal 
to make that easy (I just suggested a way to do that). The 
magic-outbox-on-server puts a number of constraints on the server design 
which is one of many reasons I dislike that proposal. If you're still 
not convinced, I can enumerate more reasons why magic-outbox-on-server 
is a problem and/or further refine my proposal.

		- Chris

---

From: Ned Freed <[email-masked]>
Date: Thu, 27 Apr 2017 08:29:09 -0700 (PDT)

> From a protocol perspective, this model is basically identical to the

Maybe it is from the client side, but from the server side it's very, very
different. The key point is there's no longer a queue that has to be maintained
in a mailbox.

> put them in the sent mailbox and require some kind of search.

That's from the client perspective.

> The only

It's not even close to being the only additional component.

> implemented at scale in a number of systems.

We don't have any such plumbing in our product that's connected to folder or
messages. And I cannot think of a single case where we've been asked for a
feature that required it.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 27 Apr 2017 08:56:43 -0700 (PDT)

> Off the top of my head, beyond undo send/future send,

Done on the MTA side of things, where the queue structures don't have to map to
users and there are many, many, many orders of magnitude less data.

> there is also
> automatic emptying of the trash,

There's no need for a queue or queue management to do folder content
expiration.

> remind me later, and enterprise style retention policies.

These are simply elaborations on future release and folder content expiration.

> things, such as data migration or applying changes across a very large mail

Again, that just doesn't match our reality.

				Ned

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 28 Apr 2017 05:46:26 +0000

On Thu, 27 Apr 2017, at 02:06 AM, Chris Newman wrote:
> Well, if you believe that and I don't believe that,

Then clearly we need to discover where the divergence in mental models
is occurring.

> * Every server that supports future release and unsend will provide
>   the same interface to the client. No need for the "always empty
>   outbox" alternative server behavior that you'll see with the outbox
>   model.

The server will have to advertise whether it supports scheduled send
regardless of the mechanics underneath. For servers that do not support
scheduled send, I agree that an always-empty outbox mailbox may feel a
little superfluous, however the point is to, as you say, always provide
the same interface to the client regardless. The client will know from
the capabilities that it must always be empty, so will probably hide it
in the sidebar or equivalent navigation mechanism (in fact I expect
clients will probably hide the outbox when empty regardless of scheduled
send support).

>   permits simpler server designs.

I do not believe this to be the case, as I will explain below.

>   the security and operational benefits that provides.

This is I think where the confusion lies. The outbox really isn't a mail
queue, and no one is suggesting you merge it with your mail store.

You clearly have existing infrastructure that supports holding scheduled
messages inside your mail queue and possibly cancelling them from that.
You wish to reuse this infrastructure, of course! That's fine. To do so,
when the message is moved to the outbox, you would *also* submit it into
your mail queue. The copy in the outbox would be the representation in
JMAP land, not part of your mail queue. When the message is sent to the
next hop and can no longer be recalled, you then move the JMAP message
from outbox to sent, and remove the `\Draft` keyword.

Other systems may choose to implement it differently, not passing it to
the mail queue until the scheduled time is reached. As has been pointed
out already, many systems such as Gmail, FastMail etc. already have
infrastructure to do this kind of thing since it is also used for
deleting messages that are X days old from the trash, supporting snoozed
mail, etc.

The outbox protocol is agnostic to how you wish to implement this. Your
proposal constrains you to only the option *you* already have
implemented.

>   There's no confusion as one has with an outbox model where a message
>   in the outbox may be submitted (if it's on the server) or maybe not
>   (if the client is offline when the message moves to the outbox).

This is clearly up to the client to distinguish, but I strongly disagree
this is a confusion. For the user, there is the outbox, which represents
the messages that are yet to be sent. If they open the outbox, I expect
most MUAs would label why they have not yet been sent (e.g. waiting for
internet connection, scheduled for time X).

>   infrastructure, making this easier/faster to deploy.

As mentioned above, a lot of systems already have infrastructure for "do
action X in folder Y when time Z is reached", or you can alternatively
use your SMTP-queue-scheduled-send to implement the outbox model, so
this is actually *more* compatible with existing infrastructure than
your proposal.

> * The "outbox" model constrains unsend to only work if the message is
>   in the outbox. This model has no such constraint and works the same

When composing and sending a message, it goes through 2 or 3 states.
These need to be represented in the mail store (and thus the protocol)
so that users can understand what is going on, and what actions are
available to them.

Two of these we (I believe) agree on, since they are from IMAP:

(1) The message is still being composed. It SHOULD be in the
    "drafts" mailbox and MUST have the "\Draft" keyword. This is the
    same as IMAP.
(2) The message has been sent, cannot be recalled. It SHOULD be in the
    "sent" mailbox (although services may provide means for users to ask
    it to go somewhere else, for example we have a few users that prefer
    their sent mail to in the same mailbox as the message they were
    replying to).

The third (new) state is for "scheduled" messages. We have two options
for how this is represented:

1. By its presence in the "outbox". Moving it out is therefore the
   logical thing to unschedule it.

2. By a keyword. This permits it to be in any mailbox, although we would
   probably add an outbox mailbox to our server for holding these
   messages by default. Removing the keyword would probably be a logical
   way to unschedule it. This option is quite a bit less efficient at
   the protocol level, as without a Mailbox object (which has a
   precalculated total count), the client will have to issue a search
   every single time there's a state change to see if there are any
   scheduled messages.

I'm presuming you are not advocating for option (3), which is no
representation in the mail store, since this would be terrible for the
user experience, and your proposal explicitly said "Add a way to
search/select messages in the Sent folder that may be possible to
unsend", which would indicate there must be some kind of state
representing this status.

So however you implement it, you need to trigger some state change on
the JMAP server for each logical step so that clients can represent
this, and update its local model as changes occur. The choice of
representation is mainly down to what will map more easily to how
clients wish to represent the set of scheduled messagese to users.

UX designers seem to have overwhelmingly decided that putting these
scheduled messages in an outbox rather than mixing them in with sent
items or elsewhere is a clearer model for users to understand. I
provided 5 example products in the previous email that do this, but if
that's insufficient I noice we have representatives from Apple, Google
and AOL on this list; perhaps they can ask their respective UI teams to
comment on what they would recommend?

Neil.