Subject: Re: [Jmap] message copying and moving

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 1 Aug 2017 17:38:51 +0300

On Tue, Aug 01, 2017 at 13:38:35 +1000, Neil Jenkins wrote:
> On Mon, 31 Jul 2017, at 11:44 PM, Josef 'Jeff' Sipek wrote:
> > There appear to be three different ways in jmap mail to "copy" an email>
> > from one mailbox to another.  That is, duplicating a message but leaving
> > the> original in place (like COPY in IMAP).  They are:
> > 
> > 1. setMessages on an existing message's messageId with destination
> >   mailbox's mailboxId added to mailboxIds
> 
> Not exactly. JMAP is currently designed so it's optional for servers to
> support the concept of a single message appearing in multiple mailboxes
> ("labels" rather than "folders"). This is designated by the
> mustBeOnlyMailbox property on the mailbox objects, but I think we should
> re-evaluate this. It seems likely to be the same for every mailbox in the
> account. A simpler (and better IMO) method would be to add a
> maxMailboxesPerMessage property to the capabilities given for an account,
> which would be 1 for simple IMAP proxies and >1 for more sophisticated
> servers.

I like the idea of the server presenting a property of some sort to indicate
how many mailboxIds can be in the list for a message.  Even servers that
support label-like mailboxes likely have some upper bound.  (Should null or
absence mean 1?  unlimited?)

As far as making it a per-account capability is concerned, that seems
incompatible with common Dovecot configurations where multiple storages are
"grafted" into a single namespace.

For example, on my server running dovecot I have two storages for my email.
One to store everything but spam and one to store only spam.  This is what
it looks like from client's perspective:

A0001 LIST "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasNoChildren \UnMarked \Drafts) "/" drafts
...
* LIST (\HasNoChildren \UnMarked) "/" xfs
* LIST (\HasNoChildren \UnMarked) "/" zfs
* LIST (\Noselect \HasChildren) "/" spam
* LIST (\HasNoChildren \Marked) "/" spam/misc
...

But on the server, the emails I care about use one type of storage (sdbox at
"" on one filesystem) with certain performance characteristics, while the
spam mails can use a different type (mdbox at "spam" on a different
filesystem) with different performance characteristics.

This is just one example of how dovecot admins use multiple storages within a
namespace.  Another common configurations include: using the "imapc" storage
to graft a remote IMAP server's namespace, using mbox for INBOX while
keeping the rest of the mailboxes in maildirs, etc.

Because any combination of storages is allowed, there is no guarantee that all
mailboxes returned by LIST/getMailboxes support the same features.

I'm not really sure what this means for jmap.  I suppose the easy way out would
be to make the maxMailboxesPerMessage a per mailbox "capability", but that
doesn't really help either - just because two mailboxes support messages with
multiple locations doesn't mean that a message can be in both.

To use the above two-storage setup as an example, suppose I want to copy a
mail from INBOX to spam/misc.

With IMAP, it'd be:

	A SELECT INBOX
	B COPY 123 spam/misc

The server notices that the source of the message (INBOX) and the
destination (spam/misc) are in different storages, and does a dumb copy
instead of trying hardlink tricks.  The client doesn't care, because it
knows that the new copy of the mail will have a new UID (one that's scoped
to spam/misc).

With JMAP, it'd be (assuming that the mail being copied is initially only in
INBOX):

	setMessages: {
		"id": "mail-to-copy-id",
		"update": {
			"x": {
				"mailboxIds": [
					"INBOX-mbox-id",
					"spam/misc-mbox-id"
				]
			}
		}
	}
	...

What should the server do?  The options I see:

1. reject the update
2. do not copy, but somehow hide that from the client
3. make a physical copy, but somehow hide that from the client
4. make a physical copy, provide it a new messageId

In a way the first option is the most tempting - we can't link the message,
so let's error out and have the client try to make a real copy.  But that
seems silly - if these are the semantics that the client has to deal with,
why not just have the server do it and save a network round-trip?

The second option just doesn't work if the destination storage is something
like dovecot's "imapc" (or eventually "jmapc"), since clients connecting
directly to the remote wouldn't see the copied messages.

I'll have to mull this over some more.  I'm just thinking out loud :)

...
> > 2. importMessages with blobId referencing an existing message's blobId
> > 3. copyMessages with fromAccountId == toAccountId
> 
> While not intended for this purpose, yes these could be used to copy messages. However, 
> > The first method creates a "hardlink" copy - any keyword changes will be
> > shared by all the copies of the message.
> > The second method creates a total duplicate of the message and its
> > metadata.
> 
> Looking at the spec, this needs to be clarified. However I believe we
> definitely need to make it so that (2)/(3) MAY return an existing message
> id if the content is the same as an existing message (or alternatively an
> error; either works for me).

I'm ok with MAY return existing messageId.  Is there ever a time when
someone may want to actually create a duplicate (IMAP COPY-style) without
this sharing?  This automatic collapsing of messages could lead to some
annoyed users that want two copies but with different keywords.

> For example, the Cyrus implementation of JMAP
> uses a secure hash of the message content to generate the message id, so
> you cannot have two identical messages within the same user with different
> message ids.

Out of curiosity, when you get a COPY or setMessages with a new mailboxId
added, do you duplicate the mail itself (via a copy or a hardlink)?  Or do
you just update some indexes?  Do you store mails all in different
mailboxes?  Or did you get rid of the notion and just emulate them via some
index?

> > When it comes to moving mails from one mailbox to another (like MOVE or
> > COPY+EXPUNGE in IMAP) the situation is equally complex.  The options are:
> > 
> > 1. setMessages on an existing message's messageId with destination
> >   mailbox's mailboxId replacing the source mailbox's mailboxId in
> >   mailboxIds
> 
> Yes, this is the expected way clients would do it.

Ok.

> > 2. importMessages with blobId referencing an existing message's blobId,
> >   followed by a setMessages on the original message but removing the
> >   source mailbox's mailboxId from mailboxIds
> > 3. copyMessages with fromAccountId == toAccountId, followed by
> > setMessages on the original message but removing the source mailbox's mailboxId from mailboxIds
>
> Again, what this does depends on whether the server supports multiple
> copies of the same message content with different ids.
>
> So I think the following would help:
>  * Remove mustBeOnlyMailbox property from individual mailbox objects and
>  add a capability for maxMailboxesPerMessage instead. Add an appropriate
>  error for setMailboxes if you try to go over this limit.

As I stated above, I'm ok with mustBeOnlyMailbox being replaced by some
flavor of maxMailboxesPerMessage.

>  * Allow servers to return an error instead of copy/import if the message
>  already exists, rather than creating a duplicate with separate metadata.
>  (But if it does succeed, then it MUST have separate metadata, be a new
>  message id). 

Sounds good.

> A lot of this depends on how much flexibility we want to give servers for
> making it easier to proxy/build-on IMAP. Every extra option does make it
> harder for clients.

I agree 100% about extra options increasing complexity and pain.  However, I
don't think this is only about proxying/building-on IMAP.

IMAP makes very few demands on what features the storage must provide.  This
no doubt helped its adoption since system administrators did not have to
migrate all the users' mailboxes before exposing them via IMAP.  They just
needed to install an imap server daemon, and point it at the maildirs.  I
think this is a very valuable property.

Do we want jmap to also have this property?  Or do we want to relax it to
some degree?

The risk of not having it is making server implementers not bother to
implement jmap support anytime soon and deployments not enabling it in a
timely manner because of required storage migrations.  (Deployments with
10's of millions of user accounts take a lot of time and effort to migrate.)

> I'd be happy to forbid having two copies of the same message in the same
> user if people think this is reasonable. So something like:
>
>  * maxMailboxesPerMessage MUST be at least 10 (or greater)
>  * copyMessages fromAccount MUST be different to toAccount
>  * (import|copy)Messages MUST reject attempts to import/copy a message
>  whose content already exists.

That sounds a bit extreme.  I like that the current draft does not require
any content checking.  The messageId can be whatever makes sense for the
server.  Content hash is one reasonable value.  But adding a restriction
like this forces the server to maintain a hash of some sort of every mail -
even if it has no intention of using it for the messageIds.

> Or similar.
> 
> On Tue, 1 Aug 2017, at 12:54 AM, Ted Lemon wrote:
> > it's unlikely that the user ever wants a copy when a link is possible.
>
> Agreed. The current spec was designed for allowing the possibility that a
> link is not possible. But maybe we can be more prescriptive about what
> servers must support.

See above about grafting of storages and making migration from existing
storage systems painless.

Thanks for all the hard work!

Jeff.

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 11 Aug 2017 11:28:00 +1000

On Wed, 2 Aug 2017, at 12:38 AM, Josef 'Jeff' Sipek wrote:
> I like the idea of the server presenting a property of some sort to
> indicate how many mailboxIds can be in the list for a message.  Even
> servers that support label-like mailboxes likely have some upper
> bound.  (Should null or absence mean 1?  unlimited?)
I would say null == "unlimited" (which of course is limited by the
number of mailboxes you have in the account). I've created
https://github.com/jmapio/jmap/issues/135 regarding this issue.
> As far as making it a per-account capability is concerned, that
> seemsincompatible with common Dovecot configurations where multiple
> storages are "grafted" into a single namespace.
You can split the storage of data within a JMAP account, but you'll
probably need a single index to fulfil the requirements for an account,
such as moving a message atomically between mailboxes, having threads
span messages across all mailboxes within the account, a unique id space
etc. Looking at the Dovecot docs, it seems your dbox format already
supports something like this (Alternate storage[1]) for storing the
actual data referenced by an index at different mount points. (We do
something very similar with Cyrus at FastMail to keep newer email on SSD
and older archives on spinning rust.) This is invisible to clients.
If you can't guarantee the properties required for an account for your
different mailboxes, then they have to be exposed as different
accounts. Methods that operate between accounts (e.g. copyMessages)
don't have the same constraints. A client can still choose to show them
together of course.
> To use the above two-storage setup as an example, suppose I want to
> copy a mail from INBOX to spam/misc.> 
> With IMAP, it'd be:
> 
> A SELECT INBOX
> B COPY 123 spam/misc
> 
> destination (spam/misc) are in different storages, and does a dumb
> copy instead of trying hardlink tricks.  The client doesn't care,
> because it knows that the new copy of the mail will have a new UID
> (one that's scoped to spam/misc).
So this is a clear case where they need to be different accounts, and so
the client can only use copyMessages to move from one to the other, not
setMessages. The semantics are then very similar to IMAP.
> Is there ever a time when someone may want to actually create a
> duplicate (IMAP COPY-style) without this sharing?  This automatic
> collapsing of messages could lead to some annoyed users that want two
> copies but with different keywords.
While I don't doubt there are some people that might want this, based on
the feedback we see, the vast majority of users would be happy to only
ever have a single shared copy that can exist in multiple locations.
With JMAP as currently defined, the server is free to support either (as
mentioned, we've decided with Cyrus to not allow this).
>> uses a secure hash of the message content to generate the
>> message id, so>> you cannot have two identical messages within the same user with
>> different>> message ids.
> 
> Out of curiosity, when you get a COPY or setMessages with a new
> mailboxId added, do you duplicate the mail itself (via a copy or a
> hardlink)?
Yes, because this was the easiest coming from the IMAP world where
each mailbox was completely separate, and there was no concept of a
"user". If I were to build a new server from scratch (and we have
thought about this a lot…) I wouldn't do this though; I'd just have a
blob store of immutable objects and a per-account index, and would
just update the index.
So to get from IMAP to JMAP with Cyrus we basically did:
 * Use secure hash of message content as message id.
 * JMAP mailboxes = set of mailboxes in which that message id exists.
 * When applying keywords with JMAP, we apply to every copy.
 * When reading keywords with JMAP, you get the union of keywords on
   all copies.
>> Looking at the spec, this needs to be clarified. However I believe we>> definitely need to make it so that (2)/(3) MAY return an existing
>> message>> id if the content is the same as an existing message (or
>> alternatively an>> error; either works for me).
> 
> I'm ok with MAY return existing messageId.
> …
>> * Allow servers to return an error instead of copy/import if the
>>   message already exists, rather than creating a duplicate with
>>   separate metadata. (But if it does succeed, then it MUST have
>>   separate metadata, be a newmessage id).> Sounds good.

OK, so as you noted we shouldn't *require* the server to anything
special  because that would force it to compare content hash and there's
no real need for this. So I think the options are:
 * *May* reject (error includes existing id) if content already exists.
   But if succeeds guaranteed to be separate copy with separate
   id/keywords/mailboxes.
 * *May* return  existing id if content already exists (and applies
   keywords/mailboxes like setMailboxes to the existing copy).
We need to pick one! I've created
https://github.com/jmapio/jmap/issues/134. I think probably the first
option, but don't feel too strongly either way.
> IMAP makes very few demands on what features the storage must provide.
> This no doubt helped its adoption since system administrators did not
> have to migrate all the users' mailboxes before exposing them via
> IMAP.  They just needed to install an imap server daemon, and point it
> at the maildirs. I think this is a very valuable property.> 
> Do we want jmap to also have this property?  Or do we want to relax it
> to some degree?
I think this has to be relaxed somewhat. I presume you could reasonably
easily migrate servers from mbox to mdbox right? I would have thought
sysops wouldn't care too much if you required them to migrate, as long
as the migration was easy and handled (mostly) automatically on upgrade.
Cheers,
Neil.

Links:

  1. https://wiki2.dovecot.org/MailboxFormat/dbox#Alternate_storage

---

From: "J.B. Nicholson" <[email-masked]>
Date: Fri, 11 Aug 2017 00:27:03 -0500

Ted Lemon wrote:
> I think this is right, and is the right way to document it.   Default
> behavior on the client should be link—it's unlikely that the user ever
> wants a copy when a link is possible.   The only use case for copy is
> when the mail client is showing two mail stores on separate servers, or
> in the rare case where the user _wants_ two copies that are treated
> differently.   (I'm curious if there is general agreement on this or if
> folks think I'm crazy, though!)
I could be misreading or misunderstanding what you've written but I think I 
don't agree because I can't remember the last time I used an MUA that let 
me choose to link an email via the MUA's UI.

I also cannot recall when I would have wanted a link as a default choice 
while, say, dragging and dropping an email from one folder to another. I 
would think the default behavior on GUI MUAs would be moving or copying 
with the other option available via holding down a dead key.

I also think moving and copying are far more straightforward and familiar 
actions to non-technical users.

Hence, I think linking functionality is interesting but unlikely to be used 
much in practice. Therefore I believe copying and moving should be clear in 
the protocol.

I do understand the attraction for mail server implementers and sysadmins 
to want to use linking as much as possible to save storage space.

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 11 Aug 2017 15:31:40 +1000

On Fri, 11 Aug 2017, at 03:27 PM, J.B. Nicholson wrote:
> I also think moving and copying are far more straightforward and
> familiar actions to non-technical users.
I think you're misunderstanding; the MUA can still present move/copy
options. Moving is removing one mailbox while adding another to the
mailboxIds property of the message. Copying is adding without removing
(so you have multiple mailboxes in the mailboxIds list). Because the
copies are presented with a single id in the API, the MUA can also
expose this as "labels", but it doesn't have to.
Regards,
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 11 Aug 2017 08:37:43 -0400

Right. My point was simply that the case where the end user wants two
distinct copies of a message which can have independent state is more
unusual than the case where the end user wants a single message to appear
in two places, which at present is also somewhat unusual. There are a lot
of potential applications for the latter. I can't think of any important
applications for the former.

On Aug 11, 2017 1:31 AM, "Neil Jenkins" <[email-masked]> wrote:

> mailboxIds property of the message. Copying is adding without removing (so
> you have multiple mailboxes in the mailboxIds list). Because the copies are
> presented with a single id in the API, the MUA can also expose this as
> "labels", but it doesn't have to.
>
> Regards,
> Neil.
>
> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap
>
>