Subject: Re: [Jmap] blob API in draft-ietf-jmap-{core,mail}-01

From: Neil Jenkins <[email-masked]>
Date: Mon, 04 Sep 2017 11:39:51 +1000

Sorry for the very late response to this. I'm back on JMAP spec work at
least every Monday now, so I hope to crack through the emails and issues
in the run up to IETF 100 in Singapore.
On Thu, 27 Jul 2017, at 12:10 AM, Josef 'Jeff' Sipek wrote:
> Why is accountId required in the [download] URL?  I don't see a reason
> why a server> couldn't use blobIds that either have internal structure to correctly> determine the accountId, or to not care about accountId at all.

All ids in JMAP are scoped to a particular account, and this includes
blobIds. Depending on how your server works, you probably need to know
the account in which to look for a particular blob, so it needs to be in
the URL. You could just include it in every blobId, but why make all the
ids longer when you don't really need to?
> I'd expect the spec to say that the URL MAY contain accountId

We could make the accountId optional in the download URL; it should
pretty much make no difference to the client. Then if you don't need it
on the server you can just exclude it from the URL template.
> Should the spec provide some guidance how far into the future the
> expiration date should be?

Yes, it should. Probably 24h is a reasonable minimum.

> Should it explicitly point out that the timestamp is purely
> advisory, and> the blob may disappear at any time for a variety of other reasons?

Probably yes! I've created Issue #140[1] for these two comments.

> If identical binary content is uploaded, the same __blobId__ SHOULD be
> returned. If the same blobId is returned, should the expiration date
> be bumped as if it were the first upload of the blob?
First off, I'm going to change that SHOULD to a MAY. It's more efficient
to dedup, but easier not to. I think this is legitimately a choice for
the server based on its architecture. Regarding the expiration date
bump, probably yes again.
> Once a blobId is used by an object (e.g., via setMessages), is it
> supposed to be still accessible via the same blobId?  Or can the
> server move the> data associated with the blobId out of the "pending blob"
> storage into a> more permanent place and in the process generate a new blobId?

The former option is more efficient (and certainly what I would do if
building a new server from scratch). However, I think we need to allow
the latter for ease of implementation on existing systems; this will
require a few changes to the spec, but I don't think it's too awkward.
Basically in response to a standard create/update, the server returns
any properties that it changed that the client didn't explicitly
specify, so if the server needs to change the blobId it can return the
new one, but otherwise the client can presume it's unchanged. I've
created #141[2] for this.
> it is easy to run into a ugly scenario where a client wants to save a
> draft with two large attachments with quota enabled.> […]
> I think it would be good for jmap all requests to either succeed or
> fail in a way that tell the client what can be done to get more
> forward progress (even if it is sub-optimal) or that there is no way
> forward.  In this particular example, the issue is that the
> invalidProperties error returned from setMessages covers multiple
> situations.  At least some of them are:> 
> (1) client error: the client didn't upload the attachment before
>     trying> to reference it
> (2) blob expired: the server removed the blob because it remained
>     unreferenced for too long
> (3) blob evicted: the server removed the blob early because of over-
>     quota>     situation, etc.

Both (1) and (2) of course just mean the client has to upload the
attachment. But yes, I agree the eviction case is problematic. The way I
was thinking about it was that eviction is an edge case just specified
to allow the server to stop abusive behaviour. I was presuming your temp
space has a separate quota to your regular space, which should make your
issue very unlikely.
If you had a (say 250 MB) temp space quota (and we presume a message has
a max 50MB attachments limit), you're unlikely to get eviction unless
the client is doing something either malicious or stupid.
If I note this in the spec, does this resolve your concern? If not, do
you have another suggestion for how to provide limits that stop
malicious/broken behaviour from causing issues?
> By the way, a number of the HTTP status code descriptions
> include: "There> is no content in the response."  Is that saying that the response body> SHOULD be ignored?  Or that the responses' Content-Length MUST be 0?

I was presuming Content-Length MUST be 0. But it might be helpful
for servers to return diagnostic text here, so I'm happy to just
change it to "Clients SHOULD ignore the response body". Do you think
this is better?
Cheers,
Neil.

Links:

  1. https://github.com/jmapio/jmap/issues/140
  2. https://github.com/jmapio/jmap/issues/141

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 04 Sep 2017 16:04:27 +1000

I've created https://github.com/jmapio/jmap/pull/142 to address the
concerns raised in this thread; please review and add comments directly
on the pull request or via the mailing list.
Thanks!
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 4 Sep 2017 08:33:32 -0400

On Sep 3, 2017, at 9:39 PM, Neil Jenkins <[email-masked]> wrote:
> First off, I'm going to change that SHOULD to a MAY. It's more efficient to dedup, but easier not to. I think this is legitimately a choice for the server based on its architecture. Regarding the expiration date bump, probably yes again.

Are you sure this is a good idea?  This sounds like a gateway to the kind of inconsistency that plagues users of imap: the mua can’t depend on some useful behavior of the server without creating weird ui artifacts.  I haven’t been able to follow this in enough detail to be sure it’s a problem here, but it sounds like one. Isn’t it useful for the mua to be able to count on the id being unique?

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 04 Sep 2017 22:44:03 +1000

On Mon, 4 Sep 2017, at 22:33, Ted Lemon wrote:
> On Sep 3, 2017, at 9:39 PM, Neil Jenkins
> <[email-masked]> wrote:>> First off, I'm going to change that SHOULD to a MAY. It's more
>> efficient to dedup, but easier not to. I think this is legitimately a
>> choice for the server based on its architecture. Regarding the
>> expiration date bump, probably yes again.> 
> Are you sure this is a good idea?  This sounds like a gateway to
> the kind> of inconsistency that plagues users of imap: the mua can’t
> depend on some> useful behavior of the server without creating weird ui artifacts.  I> haven’t been able to follow this in enough detail to be sure it’s a
> problem here, but it sounds like one. Isn’t it useful for the
> mua to be> able to count on the id being unique?

The idea is that if you upload exactly the same bytes, you can get the
same ID, allowing you to cache just a single copy of the content.  It's
particularly nice in cases like with Cyrus IMAP now, where we haven't
quite got single-copy storage of attachments on the backend, but we can
give you the same ID if the bytes are identical, so if you have a copy
of something that's been attached and forwarded around, a JMAP client
could download that blob once.
I'd be happy for the wording to say MUST NOT ever use the same blobId
for different bytes.
Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 5 Sep 2017 10:39:06 +0300

On Mon, Sep 04, 2017 at 22:44:03 +1000, Bron Gondwana wrote:
...
> I'd be happy for the wording to say MUST NOT ever use the same blobId

Sounds good.

Specifically:

* if the bytes are different, the server MUST NOT use the same blobId
* if the bytes are the same, the server MAY use the same blobId

Jeff.

-- 
Research, n.:
  Consider Columbus:
    He didn't know where he was going.
    When he got there he didn't know where he was.
    When he got back he didn't know where he had been.
    And he did it all on someone else's money.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 5 Sep 2017 11:21:28 -0400

ISTM that if you do not require that the same bits have the same blob id,
you are asking for there to be subtle implementation differences that will
be inexplicable to the end user.

ISTM also that the time when you want this to happen is when you want to be
able to hang different metadata off the same bits in different contexts. So
it's not that we should choose on behavior or the other. Both are valid.
They just mean different things. Leaving this up to the implementation is
therefore asking for interop woes.

On Sep 5, 2017 3:39 AM, "Josef 'Jeff' Sipek" <[email-masked]> wrote:

> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap
>

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 06 Sep 2017 10:23:47 +1000

On Wed, 6 Sep 2017, at 01:21 AM, Ted Lemon wrote:
> ISTM that if you do not require that the same bits have the same blob
> id, you are asking for there to be subtle implementation differences
> that will be inexplicable to the end user.
I don't see how any of this is visible to the end user.

> ISTM also that the time when you want this to happen is when you want
> to be able to hang different metadata off the same bits in different
> contexts.
No. The blobId *only* refers to the binary data, not any metadata.

>  So it's not that we should choose on behavior or the other. Both are
>  valid. They just mean different things. Leaving this up to the
>  implementation is therefore asking for interop woes.
I don't think there are likely to be interop issues here. Getting the
same blobId for the same binary data can make things a little more
efficient, as you don't have to redownload a file you already have, but
other than that the client doesn't really have to care. The only
difference is it has to be able to handle the blob id potentially
changing when the blob is referenced from a new location, but there is
already a standard mechanism in JMAP for the server to let the client
know of properties that it changed during a create/update, and the
client should be able to understand this already.
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 5 Sep 2017 20:37:15 -0400

An example of how two blob IDs for the same blob would be visible to the end user would be an inexplicable excessive use of data, for example, which is one of the problems I have run into with IMAP implementations in the past.   For example, suppose you got a different blob ID per mailbox, for the same message.   If you have any sort of automatic sorting that results in messages being moved, suddenly there's going to be a lot of unnecessary data being re-transferred.   This exact failure mode is something that's made IMAP untenable when I am in a situation where I have limited data.   Granted, this would seem like a contrived example, but I was pretty surprised when IMAP did this, too.

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Wed, 6 Sep 2017 10:58:15 +0300

On Tue, Sep 05, 2017 at 20:37:15 -0400, Ted Lemon wrote:
> An example of how two blob IDs for the same blob would be visible to the
> end user would be an inexplicable excessive use of data, for example,
> which is one of the problems I have run into with IMAP implementations in
> the past.   For example, suppose you got a different blob ID per mailbox,
> for the same message.   If you have any sort of automatic sorting that
> results in messages being moved, suddenly there's going to be a lot of
> unnecessary data being re-transferred.

I'm not sure if you are talking about re-downloading or re-uploading...

Re-downloading:

The client that initiated the move will get the new blobId for the mail.
So, it can do whatever local cache update it deems necessary.  The other
clients aren't so fortunate, but I'd argue that vast majority of them will
not need (at least immediately) the whole raw message, and therefore they
will use the getMessages call to get whatever subset of the mail they need.

Note that if you do automatic sorting via sieve, the delivery happens before
any client has a chance to download the mail from the "old" mailbox, so this
isn't a concern at all.

Re-uploading:

Any jmap client that moves messages by downloading and uploading again is
broken and needs to be fixed.

> This exact failure mode is
> something that's made IMAP untenable when I am in a situation where I have
> limited data.   Granted, this would seem like a contrived example, but I
> was pretty surprised when IMAP did this, too.

The discussion here is specifically about uploading of binary data.  This
API is used for upload of attachments or importing of messages.  If a user
does either of these actions on a link with limited bandwidth, I consider it
an implicit agreement to use the bandwidth.

If, for example, a user decides to upload a 25 MB attachment, the user
should expect >=25 MB to be transferred.  After uploading the attachment,
the client can issue a setMessages call to actually consume the attachment.
The result of the setMessages call will be the new blobId (if different), so
the client can make local cache adjustments.  So, if all goes well, the
client would have uploaded about 25 MB (assuming the body of the email is
negligible in size), and downloaded approximately 0 since the responses are
rather tiny.

Editing the draft with a non-buggy client does *not* require the client to
reupload attachments.  This is one improvement over IMAP.

Finally, when sending the draft, the jmap submission API can reference the
draft email by messageId resulting in essentially 0 upload.

In other words, to compose, edit, and submit an email with a 25 MB
attachment the client can expect to upload a little over 25 MB and download
essentially nothing.  I think this is significantly better than what IMAP +
SMTP give us today (which would use 25 MB * (2 + number of edits) of
upload bandwidth).

Jeff.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 6 Sep 2017 08:47:10 -0400

> I'm not sure if you are talking about re-downloading or re-uploading...

Ah, I'm talking about when there are multiple clients.   This probably isn't an issue if you have only one, at least for the most part.   But e.g. a server-side automatic mail sorter would count as a client, and might create a great deal of activity.   A personal task manager might as well.

And yes, my concern is about downloads, not uploads.

---

From: Bron Gondwana <[email-masked]>
Date: Thu, 07 Sep 2017 00:15:46 +1000

The most common case I imagine IDs changing is when the upload creates a
temporary blobId which gets replaced by the real blobId when the
persistent message gets created.  The blobId might then be
"emailid[mimepart]" for example, which won't change again while the
message persists.
The trickier part is things like deduplication of attachments, where
your server receives two messages which have byte-for-byte identical
attachments on them.  Depending on your server architecture, you may not
want to parse the message, convert the attachments to binary and
calculate the digest to deduplicate before returning a blobId, which is
what you're talking about requiring if you require that there are never
two blobIds which result in the same bytes.
Bron.

On Wed, 6 Sep 2017, at 22:47, Ted Lemon wrote:
> On Sep 6, 2017, at 3:58 AM, Josef 'Jeff' Sipek
> <[email-masked]> wrote:>> I'm not sure if you are talking about re-downloading or re-
>> uploading...> 
> Ah, I'm talking about when there are multiple clients.   This probably
> isn't an issue if you have only one, at least for the most part.   But
> e.g. a server-side automatic mail sorter would count as a client, and
> might create a great deal of activity.   A personal task manager might
> as well.> 
> _________________________________________________
> Jmap mailing list
> [email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Wed, 6 Sep 2017 10:32:25 -0400

On Sep 6, 2017, at 10:15 AM, Bron Gondwana <[email-masked]> wrote:
> The most common case I imagine IDs changing is when the upload creates a temporary blobId which gets replaced by the real blobId when the persistent message gets created.  The blobId might then be "emailid[mimepart]" for example, which won't change again while the message persists.

Right, but isn't this actually a problem?   If it's implemented this way, the MUA is _always_ going to wind up re-downloading the attachment.

> The trickier part is things like deduplication of attachments, where your server receives two messages which have byte-for-byte identical attachments on them.  Depending on your server architecture, you may not want to parse the message, convert the attachments to binary and calculate the digest to deduplicate before returning a blobId, which is what you're talking about requiring if you require that there are never two blobIds which result in the same bytes.

It seems to me that if your server is implemented in this way, you're really arguing for lazy evaluation: don't compute the blobId until someone asks for it.   In that case, when you compute the blobId, you can do the deduplication as well.   If you haven't parsed the message, you don't even have an occasion to produce a blobId.

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 11 Sep 2017 12:00:23 +1000

On Thu, 7 Sep 2017, at 12:32 AM, Ted Lemon wrote:
> On Sep 6, 2017, at 10:15 AM, Bron Gondwana
> <[email-masked]> wrote:>> The most common case I imagine IDs changing is when the upload
>> creates a temporary blobId which gets replaced by the real blobId
>> when the persistent message gets created.  The blobId might then be
>> message persists.> Right, but isn't this actually a problem?   If it's implemented this
> way, the MUA is _always_ going to wind up re-downloading the
> attachment.
Why? The client that uploaded the file will send the original blobId in
the setMessages request and get back the new on in the messagesSet
response, so can update its internal cache without downloading it again.
Other clients cannot see it until the message is created anyway. In
either case, there's no need for the same file to be downloaded again.
Neil.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 10 Sep 2017 22:08:26 -0400

On Sep 10, 2017, at 10:00 PM, Neil Jenkins <[email-masked]> wrote:
> Why? The client that uploaded the file will send the original blobId in the setMessages request and get back the new on in the messagesSet response, so can update its internal cache without downloading it again. Other clients cannot see it until the message is created anyway. In either case, there's no need for the same file to be downloaded again.

Okay, that's a good point—if there's no way for another client to see the blobID, then we don't have that problem.   But we still do have the problem that the client has to do more than it would have to do if the blobID didn't change.   But I will admit that that is a much smaller problem.