Subject: [Jmap] hasAttachment vs. keywords

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Mon, 13 Nov 2017 07:53:27 -0500

I was thinking about keywords in jmap, and I realized that the hasAttachment
message property seems a bit special-cased and that it could be trivially
replaced by a $HasAttachment keyword.  (I don't care what exactly it is
called.)

This keyword would also nicely line up with the proposed
$HasEncryptedAttachment keyword, and it would allow IMAP servers to expose
it as well with little to no effort.

Jeff.

-- 
Research, n.:
  Consider Columbus:
    He didn't know where he was going.
    When he got there he didn't know where he was.
    When he got back he didn't know where he had been.
    And he did it all on someone else's money.

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 14 Nov 2017 10:57:22 +0800

The issue with using a keyword is that if it's not guaranteed to be set
by the server on message creation the client can't rely on it. Most
clients display an indication of whether a message has an attachment
when listing the contents of a mailbox/search, and the hasAttachment
property provides an efficient way to fetch this information without
having to download more until the user opens the message. If this
doesn't match what the user sees when they open the full message, it's a
bug. So if you can't rely on the keyword, the client would have to fetch
the full *attachments* property for each message just to show the
listing, and the keyword is irrelevant.
This feeds into the Message object format discussion which is on the
agenda for the session on Thursday. Generally this comes down to MIME
being a tree of documents, but all email clients presenting a model of
display body + attachments. Given this, I believe it makes more sense
for the Message object format to be closer to the model used in clients
rather than a simple JSON interpretation of MIME.
This still leaves open questions such as what MIME parts should be
considered "attachments" and what parts are "body" (should be displayed
immediately inline when opening the message)? I think we might want to
not prescribe a precise algorithm for this, but rather provide a
specification of intent, with guidance for implementors on what they
might want to consider. For example, at FastMail our current algorithm
for this is roughly:
 * Consider each MIME part, recursing into multipart/*
 * Ignore images < 600px in either dimension in either a
   multipart/related or multipart/alternative (because Outlook screws
   this up); they're probably an inline header/logo.
 * Otherwise, the part is an attachment if any of:
   * it's not text (according to MIME-Type)
   * it has a filename (from Content-Disposition)
   * it has Disposition-Type of "attachment"
Note, this algorithm means that if you embed a large(ish) image into an
HTML message, we still show a "has attachment" icon and include it in
searches for has:attachment. This is clearly a judgement call, but it
captures the common case of users sending each others images which
happen to be shown inline, but also may be considered a downloadable
resource by the recipient.
Neil.

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 14 Nov 2017 09:32:21 -0500

On Tue, Nov 14, 2017 at 10:57:22 +0800, Neil Jenkins wrote:
> by the server on message creation the client can't rely on it.

Agreed, but the same applies to the hasAttachment property.  The server must
set it on creation or the client can't rely on it.  (Of course the server
can be lazy about it either way.)

I was assuming that the server would set $HasAttachment/$HasNoAttachment and
$HasEncryptedAttachment/$HasNoEncryptedAttachment on creation as necessary.

Not doing it automatically, would make the keywords a bit useless and the
protocol inefficient.

> agenda for the session on Thursday.

FWIW, datatracker doesn't seem to have any info about the agenda.

> Generally this comes down to MIME

I don't think the two are mutually exclusive.

By the way, it looks like many non-IMAP clients already have to deal with
per-parsed MIME structures anyway, since the gmail API seems to provide raw
blobs and pre-parsed MIME but not a higher-level concept like "attachments".

...
> For example, at FastMail our current algorithm for this is roughly:
...
> resource by the recipient.

Given the higher-level attachment API, this seems like a reasonable
algorithm.  Assuming the attachment API remains, I'm ok with not specifying
it as required behavior, but rather as guidance.

Jeff.

-- 
If I have trouble installing Linux, something is wrong. Very wrong.
		- Linus Torvalds

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 14 Nov 2017 09:46:38 -0500

> $HasEncryptedAttachment/$HasNoEncryptedAttachment on creation as necessary.

I just saw the discussion on imapext... I suppose the server may not be able
to set $HasEncryptedAttachment/$HasNoEncryptedAttachment because of not
having the keys.  However, I don't think this changes anything about
$HasAttachment/$HasNoAttachment.

Jeff.

-- 
Ready; T=0.01/0.01 09:44:37

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 15 Nov 2017 14:59:18 +0800

On Tue, 14 Nov 2017, at 10:32 PM, Josef 'Jeff' Sipek wrote:
>> The issue with using a keyword is that if it's not guaranteed
>> to be set>> by the server on message creation the client can't rely on it.
> 
> Agreed, but the same applies to the hasAttachment property.
> The server> must set it on creation or the client can't rely on it.

It's different because (as currently specified) hasAttachment is not a
mutable property (metadata); it's a part of the parsed representation of
the (immutable) RFC5322. So by definition, it must be present if the
client tries to fetch it (I agree it could be lazily computed by the
server, but you also need to know it to be able to return the response
to getMessageList if filter =={ hasAttachment: true }, so you probably
want to do it on delivery).
> FWIW, datatracker doesn't seem to have any info about the agenda.

Sorry about that; Bron has now finally uploaded it:
https://datatracker.ietf.org/meeting/100/materials/agenda-100-jmap/
>> being a tree of documents, but all email clients presenting a
>> model of>> display body + attachments. Given this, I believe it makes more sense>> for the Message object format to be closer to the model used in
>> clients>> rather than a simple JSON interpretation of MIME.
> 

It's possibly to offer both in some way based on different properties
the client can optionally choose to fetch. That's one option we should
consider as part of the discussion on this.
Neil.

---

From: Josef Jeff Sipek <[email-masked]>
Date: Wed, 15 Nov 2017 09:48:37 -0500

On Wed, Nov 15, 2017 at 14:59:18 +0800, Neil Jenkins wrote:
> client tries to fetch it

Conceptually, I don't see a problem with the server setting keywords on
delivery.  Making the $hasAttachment keyword immutable feels a bit dirty,
but it would work.  Another option would be to leave it mutable and if a
client messes with it... well, it gets what it asked for.

...
> https://datatracker.ietf.org/meeting/100/materials/agenda-100-jmap/

Thanks.

> consider as part of the discussion on this.

Agreed.

Jeff.

-- 
Fact: 97.4% of all statistics are generated randomly.

---

From: Neil Jenkins <[email-masked]>
Date: Thu, 16 Nov 2017 10:44:59 +0800

On Wed, 15 Nov 2017, at 10:48 PM, Josef Jeff Sipek wrote:
> Conceptually, I don't see a problem with the server setting keywords
> on delivery.
There's no problem doing this (in fact at FastMail we have been setting
a $HasAttachment flag on delivery for years for this very purpose!), but
mandating it in the spec is I think trickier than it just being a
separate property which is considered part of the parsed MIME
representation.
> Making the $hasAttachment keyword immutable feels a bit dirty, but it
> would work.
Yeh, that feels really dirty to me; it's strangely different to the
other keywords then.
> Another option would be to leave it mutable and if a client messes
> with it... well, it gets what it asked for.
Yeh, although of course you might have multiple clients which could
mess it up for each other. (Consider a scenario where you try out some
client which turns out to have been badly authored and wipes all your
keywords; now the attachment icon disappears permanently on the mailbox
listing of all the other clients you use. That sounds like a support
nightmare to me.)
Neil.

---

From: Neil Jhaveri <[email-masked]>
Date: Wed, 15 Nov 2017 22:02:43 -0700

> On Nov 13, 2017, at 7:57 PM, Neil Jenkins <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Given this, I believe it makes more sense for the Message object format to be closer to the model used in clients rather than a simple JSON interpretation of MIME. 

Agreed — I think most MUAs want to deal with a single HTML body + attachments array. And I think the server needs to be doing this MIME —> (html, attachments[]) mapping, for reasons such as consistency between search results and the displayed content.

> This still leaves open questions such as what MIME parts should be considered "attachments" and what parts are "body" (should be displayed immediately inline when opening the message)? I think we might want to not prescribe a precise algorithm for this, but rather provide a specification of intent, with guidance for implementors on what they might want to consider. For example, at FastMail our current algorithm for this is roughly:
> 
> Consider each MIME part, recursing into multipart/*
> Ignore images < 600px in either dimension in either a multipart/related or multipart/alternative (because Outlook screws this up); they're probably an inline header/logo.
I don’t think I’ve encountered this one, and I’m curious to hear more!
> Otherwise, the part is an attachment if any of:
> it's not text (according to MIME-Type)
> it has a filename (from Content-Disposition)
> it has Disposition-Type of "attachment"

This roughly matches the Apple clients I worked on, although I recall a number of special-cases to fix assorted bugs, along the lines of the Outlook one you cited above. For instance, I recall a bug-fix where Windows Live Mail apparently sent text/plain MIME messages with pre-MIME-style UUencoded blob attachments, although maybe issues like that aren’t worth worrying about for JMAP. There are other more common issues like TNEF Winmail.dat attachments to also potentially consider.

I agree that a highly precise specification is probably not very practical, but I think that making some effort to document the vendor-specific cases (along with example messages) could be helpful to implementors out there. I’m not sure exactly where the best repository for something like that would be.

However, I do think we can do a highly precise specification for the mapping in the opposite direction, for the case of sending messages.

---

From: Kurt Andersen <[email-masked]>
Date: Thu, 16 Nov 2017 05:53:38 -1000

On Wed, Nov 15, 2017 at 7:02 PM, Neil Jhaveri <[email-masked]> wrote:

>
> On Nov 13, 2017, at 7:57 PM, Neil Jenkins <[email-masked]> wrote:
>
> Given this, I believe it makes more sense for the Message object format to
> be closer to the model used in clients rather than a simple JSON
> interpretation of MIME.
>
>
> Agreed — I think most MUAs want to deal with a single HTML body +
> attachments array. And I think the server needs to be doing this MIME —>
> (html, attachments[]) mapping, for reasons such as consistency between
> search results and the displayed content.
>

Why do you want to force servers to interpolate HTML (or create a body ex
nihilo?) if there isn't any?

--Kurt

---

From: Neil Jhaveri <[email-masked]>
Date: Thu, 16 Nov 2017 11:50:39 -0700

> On Nov 16, 2017, at 8:53 AM, Kurt Andersen <[email-masked]> wrote:
> 
> On Wed, Nov 15, 2017 at 7:02 PM, Neil Jhaveri <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Why do you want to force servers to interpolate HTML (or create a body ex nihilo?) if there isn't any?

I think this is a fair question, and there are some trade-offs. I think the standard for any modern client is that they need to be able to display text/html. If they support text/html, I’d imagine that clients are just up-converting plain text into HTML to simplify their view-layer. Certainly all webmail clients are in this boat.

I fully acknowledge the server doing the up-conversion, however, does come at a cost and unfortunate server involvement in presentation. For instance, quoted text may need to be up-converted to be wrapped in a <blockquote> tag, but maybe that’s not exactly what every client wants at it’s presentation layer.

I also completely sympathize with the challenges a server will face doing this kind of conversion, and that the problem may be easier to solve in the client. I think Neil Jenkins mentioned during the IETF 100 JMAP session that Fastmail handled multipart/mixed messages with multiple text/html parts by just smashing them together and letting the browser deal with the invalid HTML… and IIRC both macOS and iOS Mail also had a very similar reliance on the forgiving-parsing abilities of WebKit.

It all boils down to I think a lot of client authors are designing their UIs to basically be HTML + array of attachments (with some possibly inline), whereas the full scope of MIME is not easily mapped to that. Sigh.

I still think this is a worthwhile problem to solve on the server, so that the single primary text/html blob that the server indexes for search is exactly what the user sees, and you don’t wind up with the never-ending bug cases of “search found it, but the user can’t see it” or vice versa. But I suspect this is going to get so messy that it has to get pushed down to the clients.

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Thu, 16 Nov 2017 14:54:40 -0500

...
> I think this is a fair question, and there are some trade-offs.

Agreed.

...
> It all boils down to I think a lot of client authors are designing their
> UIs to basically be HTML + array of attachments (with some possibly
> inline), whereas the full scope of MIME is not easily mapped to that.
> Sigh.

For the "easy API" (compared to the just parse the raw blob client-side), I
would like the server to not do the concatenation of multiple parts, but
instead simply supply the client with an *array* of body parts, and the
client can concat them whichever way makes sense.  (I think it was Chris
Newman who suggested it.)

In other words, this way the whole behavior is split into two components:

- a server side "identify body parts & attachments"
- a client side "display parts & present attachments"

Yes, it does complicate the client a tiny bit, but in general it's easy
enough for a client to render more than one blob.

Note that I'm *not* talking about multipart/alternative.

> I still think this is a worthwhile problem to solve on the server, so that
> the single primary text/html blob that the server indexes for search is
> exactly what the user sees, and you don’t wind up with the never-ending
> bug cases of “search found it, but the user can’t see it” or vice versa.

The only case I can think of where this would happen is: server presents the
client with 2 body parts to concat (e.g., body and a mailing list appended
signature), the client decides to display only one.  Isn't this arguably a
client bug since it is not displaying the entire message?

Jeff.

-- 
NT is to UNIX what a doughnut is to a particle accelerator.

---

From: Raphael OUAZANA <[email-masked]>
Date: Thu, 16 Nov 2017 22:26:03 +0100

Hi,

> - a client side "display parts & present attachments"

In this case how do you compute the preview server side?

Regards,
Raphaël Ouazana.

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Thu, 16 Nov 2017 17:20:01 -0500

> In this case how do you compute the preview server side?

I don't understand... what preview are you talking about?

Jeff.

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 17 Nov 2017 10:30:15 +0800

On Fri, 17 Nov 2017, at 03:54 AM, Josef 'Jeff' Sipek wrote:
> For the "easy API" (compared to the just parse the raw blob client-
> side),> I would like the server to not do the concatenation of multiple
> parts, but> instead simply supply the client with an **array** of body
> parts, and the> client can concat them whichever way makes sense.

This I think is our preferred option. This is already what happens
internally in all of our implementations of this before being mushed
together; but the client could concatenate them just as easily, and in
many cases do a better job by rendering them separately and then
displaying the rendered parts together.
The server should index all of the parts it will return as "body", so it
shouldn't cause any issues with search/view mismatch.
> Note that I'm **not** talking about multipart/alternative.

So the client will need to be able to request whether it prefers HTML or
plain text when both are available. There is a question of whether it
should be able to ask the server to convert HTML <-> Plain if only the
non-preferred option is available. I'm not sure this is necessary these
days; clients can do the conversion themselves if they want.
Neil.

---

From: Neil Jenkins <[email-masked]>
Date: Fri, 17 Nov 2017 10:39:27 +0800

> I don't understand... what preview are you talking about?

The Message#preview[1] property I presume. The answer is the server
should take the text from the first body part of the array response. As
in the current spec, the server may choose to skip quoted sections or
salutations to return a more useful preview.
On Thu, 16 Nov 2017, at 01:02 PM, Neil Jhaveri wrote:
> I agree that a highly precise specification is probably not very
> practical, but I think that making some effort to document the vendor-
> specific cases (along with example messages) could be helpful to
> implementors out there. I’m not sure exactly where the best repository
> for something like that would be.
There is a server guide in the same repo, which I would like to expand
with this kind of thing (if we don't think it belongs in the spec
itself). This probably won't be published as an RFC (although it could
be an informational one), but we want to have a whole ecosystem of test
suite, (client) usage guide, server implementation guide etc. to help
(interoperable) adoption.
> However, I do think we can do a highly precise specification for the
> mapping in the opposite direction, for the case of sending messages.
Yep, agreed. Well volunteered. ;-)

Neil.

Links:

  1. http://jmap.io/spec-mail.html#messages