Subject: [Jmap] Attachments and composing more complex MIME structures

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 8 Aug 2017 12:23:41 +0300

Some clients (most notably iOS Mail app) often produce multipart/mixed
emails with inline text and images.  For example, if I compose a simple
email with no formatting and insert two images in the middle:

	This is a

	[inserted image]

	test of

	[another inserted image]

	iOS mail.

the email that gets produced has content-type multipart/mixed and there are
five parts (all at the top level with no nesting):

1. text/plain with "This is a"
2. image/png with the first inserted image
3. text/plain with "test of"
4. image/png with the second inserted image
5. text/plain with "iOS mail."

The image parts have inline content-disposition.

The attachment API (I'm not really sure what else to call it) as it is
currently described would work well for drafting what I think of as "email
with a bunch of attachments at the end" (that is parts with "attachment"
content disposition appended at the top-level of the MIME structure).
However, I don't think the attachment API can be used to draft one of these
inline-mime-image mails since setMessages:

1. assumes that the email text itself is one part (text/plain or text/html)
2. the attachment isInline property is meant only for HTML cid references

Did I miss something, or does this really mean that a client like this must
save drafts that are pre-composed RFC 5322 messages with all the inserted
images already embedded?  This breaks one of the exciting features of jmap -
not having to reupload all the attachments during auto-saving of drafts.

At some point on the list, Chris suggested providing a set of properties to
allow the clients to easily extract important content from the email.  Would
it make sense to extend the setMessages call to do the reverse?  That is,
assemble a RFC 5322 message from a body structure specified by the client?
I imagine each supplied body structure part to either specify text/plain or
text/html as a property, or specify the content type and disposition along
with a blob id.

The current draft has something like it, but it doesn't specify the actual
structure - just a list of attachment blobIds with some hints about their
use (i.e., the isInline property which seems to control the content
disposition, but not the placement of the part within the MIME structure).

Thanks,

Jeff.

P.S. For what it is worth, the iOS mail app produces a totally different
structure if any of the text has formatting.  (OSX Mail.app seems to always
produce it as well.)  For example, with the same content as above, but with
formatting applied to any of the text produces a multipart/alternative email
with:

1.   text/plain with the whole text without any images
2.   multipart/related
2.1. text/html with the whole text with <img> tags using cids to refer to
     the images
2.2. image/png with the first inserted image
2.3. image/png with the second inserted image

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 08 Aug 2017 20:28:37 +1000

On Tue, 8 Aug 2017, at 07:23 PM, Josef 'Jeff' Sipek wrote:
> Some clients (most notably iOS Mail app) often produce multipart/mixed> emails with inline text and images.

Although not clear yet (this bit of the spec needs a lot more details),
the intention is that a JMAP server would convert this to an htmlBody in
the Message object that combined the text and images (with cids).
> However, I don't think the attachment API can be used to draft one of> these inline-mime-image mails since setMessages:

No, but you could produce an HTML part that's going to render the same
(and be more compatible).
> Did I miss something, or does this really mean that a client like this> must save drafts that are pre-composed RFC 5322 messages with all the
> inserted> images already embedded?

If the client absolutely MUST create a multipart/mixed email like this,
then yes. The intention with the Message object format is to hide the
complexities of MIME, and try to remove alternative ways of representing
the same things. This makes it easier for client authors, at the expense
of some more complexity for server developers.
> P.S. For what it is worth, the iOS mail app produces a totally
>      different> structure if any of the text has formatting.:
> 2.1. text/html with the whole text with <img> tags using cids to
>      refer to>     the images

Doesn't sound like it would be difficult for it just to always generate
this even when no text formatting is applied then…
Neil.

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 8 Aug 2017 15:31:46 +0300

> the Message object that combined the text and images (with cids).

I guessed that was the idea there.

> > However, I don't think the attachment API can be used to draft one of
> > these inline-mime-image mails since setMessages:
> 

Is there a MIME-aware client that breaks with what the iOS client generates?
In theory the multipart/mixed approach should be *more* compatible than
HTML, since the client doesn't have to deal with HTML on top of dealing with
MIME anyway.

<personal preference>
As a mutt user that likes text/plain emails... it'd be great if these
generated emails displayed nicely with non-HTML based MUAs. :)

At the very least, the multipart/mixed emails render rather sanely in mutt.
</personal preference>

> > Did I miss something, or does this really mean that a client like this
> > must save drafts that are pre-composed RFC 5322 messages with all the
> > inserted images already embedded?
> 

I suppose the textBody/htmlBody approach should easily cover 80% of the use
cases.  The remaining 20% are possible via the importMessages call and raw
email upload for now, or an extension in the future.

Is the goal to make only the simplest of clients easy to implement?  Where
does one draw the line between simple enough to use the nice jmap interface
and too complex (IOW: just implement the whole RFC 5322/MIME/etc. stack and
use the blob API)?

> this even when no text formatting is applied then…

Agreed.

Thanks,

Jeff.

---

From: Neil Jhaveri <[email-masked]>
Date: Tue, 8 Aug 2017 11:17:41 -0700

Yeah, the Apple Mail clients have a preference to avoid sending text/html, unless strictly necessary by the content:
(A) A philosophical preference that text/plain be used whenever possible, since it results in a neater and more overall compact message, without the redundancy of including both a text/html part with a text/plain fallback alternative part.
(B) Broken implementations in the wild that caused various small support issues. The Apple clients did change once to always sending text/html if there are images, but reverted it in a software update. For instance, some Japanese cell phone carriers couldn’t properly handle messages sent to an MMS bridge e-mail address if an image wasn’t a top-level MIME part.

How do other people here feel about the preference for text/plain?

On Aug 8, 2017, at 3:28 AM, Neil Jenkins <[email-masked]> wrote:

> Although not clear yet (this bit of the spec needs a lot more details), the intention is that a JMAP server would convert this to an htmlBody in the Message object that combined the text and images (with cids).

I agree, I’d like to see if the API can remain as-is, and we detail in the spec how servers should construct MIME. I’d be happy to help with this part.

> (and be more compatible).

Can you clarify this?

There is also the uglier case of "inlined attachments" that cannot be properly referenced by HTML tags, e.g.:
	[plain text]
	[zip attachment]
	[more plain text]

I think the Apple clients are the only ones that even support this in the UI and make a multipart/mixed message out of it, but I’m not sure. Does anybody know if any other major clients offer this kind of UX?  I’ve personally never been a fan of it at all, and I think the resultant message looks confusing in some other MUAs, but it does have some occasional use. My hunch is that it only affects the Apple clients, so it doesn’t seem worth complicating matters here.

Neil

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 09 Aug 2017 18:03:31 +1000

> Is there a MIME-aware client that breaks with what the iOS client
I don't know; I haven't tested this exhaustively. I know we had bugs
with this multipart/mixed style at one point, since it was pretty
uncommon so I wouldn't be surprised if there were bugs or lack of
support elsewhere too.
More importantly, plain text is deeply problematic on mobile because of
hard wrapping. Format=flowed has zero traction[1], so you mostly end up
with hard wrapping that breaks in *really* awkward positions on many
mobile screens, as it normally presumes you can fit 80 chars in a
single line.
> In theory the multipart/mixed approach should be **more** compatible
> than HTML, since the client doesn't have to deal with HTML on top of
> dealing with MIME anyway.
Maybe, but I wouldn't be surprised if the MIME handling in some clients
extended only to "find the right part to show".
> generated emails displayed nicely with non-HTML based MUAs. :)
There's no reason why the server can't generate a reasonable textBody
component as well (it will have to do this as well as htmlBody in this
case anyway). I believe you could build an excellent plain-text only
client on JMAP if you wanted, although realistically all modern email
clients have to deal with HTML.
> Is the goal to make only the simplest of clients easy to implement?
> Where does one draw the line between simple enough to use the nice
> jmap interface and too complex (IOW: just implement the whole RFC
> 5322/MIME/etc. stack and use the blob API)?
I believe you could implement the functionality of most modern GUI email
clients without resorting to MIME parsing. This is basically the goal:
most clients never need to deal with the raw message, but some with
support for less common features can do so if they want (like for
S/MIME). For example, I believe you could implement all the
functionality in the iOS/macOS Mail apps using the JMAP Message object,
although the implementation underneath might vary in some aspects like
the one we're discussing.  (Neil, do you agree?).
Neil.

Links:

  1. https://blog.fastmail.com/2016/12/17/format-flowed/

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 09 Aug 2017 18:12:50 +1000

On Wed, 9 Aug 2017, at 04:17 AM, Neil Jhaveri wrote:
> Yeah, the Apple Mail clients have a preference to avoid sending
> text/html, unless strictly necessary by the content:
> (A) A philosophical preference that text/plain be used whenever
>     possible,> since it results in a neater and more overall compact message, without> the redundancy of including both a text/html part with a text/plain
> fallback alternative part.

Compactness is good, but with JMAP you can just upload the HTML part and
have the server generate a plain part so you're not saving over-the-wire
data on your bandwidth-constrained mobile anymore. It would really only
be for the server-to-server transmission, and I doubt that it would make
a significant difference.
> (B) Broken implementations in the wild that caused various small
>     support> issues. The Apple clients did change once to always sending
> text/html if> there are images, but reverted it in a software update. For instance,> some Japanese cell phone carriers couldn’t properly handle
> messages sent> to an MMS bridge e-mail address if an image wasn’t a top-level
> MIME part.
Interesting. I wonder whether that system is still relevant today? Given
other clients *don't* (usually) generate the multipart/mixed format,
this would presumably be seeing a lot of issues with emails sent from
e.g. Gmail.
> How do other people here feel about the preference for text/plain?

I believe that battle has been fought and lost. You can generate clean,
minimal HTML and it's the one way to get reasonable font wrapping
everywhere on mobile, including when quoting replies.
> I agree, I’d like to see if the API can remain as-is, and we detail in> the spec how servers should construct MIME. I’d be happy to help with> this part.

That would be great. :)

> There is also the uglier case of "inlined attachments" that cannot be> properly referenced by HTML tags, e.g.:
> [plain text]
> [zip attachment]
> [more plain text]
> 
> I think the Apple clients are the only ones that even support
> this in the> UI and make a multipart/mixed message out of it, but I’m not sure.

I don't believe I've seen one of them in the wild. I am now curious to
know what our client would make of it!
>  Does anybody know if any other major clients offer this kind of UX?

I'm not aware of any.

Neil.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 9 Aug 2017 11:26:57 +0100

> uncommon so I wouldn't be surprised if there were bugs or lack of 
I've spent some times implementing this in our client. We had some bugs, 
but it works nicely now. You had to implement this if you are a MIME 
capable email client.

> More importantly, plain text is deeply problematic on mobile because 
> of hard wrapping. Format=flowed has zero traction 
> <https://blog.fastmail.com/2016/12/17/format-flowed/>,
Let's not exaggerate. It is being used.

> so you mostly end up with hard wrapping that breaks in /really/ 
> awkward positions on many mobile screens, as it normally presumes you 
> can fit 80 chars in a single line.
>
> Maybe, but I wouldn't be surprised if the MIME handling in some 
> clients extended only to "find the right part to show".
>

True.
> I believe you could implement the functionality of most modern GUI 
> email clients without resorting to MIME parsing. This is basically the 
> goal: most clients never need to deal with the raw message, but some 
> with support for less common features can do so if they want (like for 
> functionality in the iOS/macOS Mail apps using the JMAP Message 
> object, although the implementation underneath might vary in some 
> aspects like the one we're discussing.  (Neil, do you agree?).
>

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Wed, 9 Aug 2017 13:45:14 +0300

On Tue, Aug 08, 2017 at 11:17:41 -0700, Neil Jhaveri wrote:
> Yeah, the Apple Mail clients have a preference to avoid sending text/html,
> unless strictly necessary by the content:
...
> How do other people here feel about the preference for text/plain?

Personally, I really like this preference for text/plain.  (Note that my
setup is strange since I use iOS on my phone and mutt on my laptop &
desktop.)

One important thing to point out about text/plain is that there are two ways
to treat it when composing the email:

1. the "unix" way, with ~80 column wrap consisting of inserted newlines, or
2. the "iOS" way, with no wrapping.

For example, if I write a paragraph of text with the iOS client, I end up
with a quoted-printable text/plain email that relies on the client to do the
wrapping since the paragraph is just one really long line.

In other words, text/plain is not inherently bad even without format=flowed.
Yes, it is not perfect, but in my opinion it works quite well even in fancy
mobile environments.

> On Aug 8, 2017, at 3:28 AM, Neil Jenkins <[email-masked]> wrote:
...
> I think the Apple clients are the only ones that even support this in the
> UI and make a multipart/mixed message out of it, but I’m not sure. Does
> anybody know if any other major clients offer this kind of UX?

IIRC, Lotus Notes supports (supported?) this as well.  It's been almost a
decade since I last used it so things may have changed since.

> I’ve personally never been a fan of it at all, and I think the resultant
> message looks confusing in some other MUAs, but it does have some
> occasional use. My hunch is that it only affects the Apple clients, so it
> doesn’t seem worth complicating matters here.

I'm not sure we can totally ignore this.  Even if Apple clients stop
generating these messages today, there are years worth of existing emails
using this kind of MIME structure.  Emails that clients will attempt to
fetch via jmap.

If we totally punt on this in jmap as part of the Message object (i.e., the
jmap server sends back some form of a message-too-complex error), any client
that wants to parse one of these mails sent by Apple clients will have to go
via the blob API and do its own MIME parsing.

Alternatively, we could make the server generate some sort of
text/{plain,html} view of the email to stuff into {text,html}Body when a
client attempts to fetch the mail.  In this case, we should probably specify
what the server should do as a bare minimum.  (E.g., the server MUST
support generating a HTML and plain text versions of arbitrary MIME
structures.)  Needless to say, this can lead to all sort of fun i18n/l10n
server-side issues.

(Now I'm curious what the FastMail client & server code does when it gets
one of these Apple client created emails...)

Actually, this makes me wonder... what should a jmap server do if the stored
message is somehow malformed?  There is a lot of broken email out there, and
the draft seems silent about how to handle them.

Jeff.

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 09 Aug 2017 21:50:23 +1000

On Wed, 9 Aug 2017, at 08:45 PM, Josef 'Jeff' Sipek wrote:
> For example, if I write a paragraph of text with the iOS client, I end
> up with a quoted-printable text/plain email that relies on the client
> to do the wrapping since the paragraph is just one really long line.
Which is fine until you’re quoting something; the wrapping (normally)
doesn’t add an extra ”>“ at the beginning of each new line, so it
becomes much harder to differentiate quoted from non-quoted text. Fine
if you’re just top-posting of course…
(Of course, the FastMail client actually does convert this to a
<blockquote> for display, so the wrapping does work fine even in quotes
as long as you never add hard line breaks, but my point is that most
clients *don't* handle this well.)
Regardless though, JMAP should happily support clients that want to use
HTML and clients that want to just do plain text.
> generating these messages today, there are years worth of existing
> emails using this kind of MIME structure.
Sorry, I perhaps should have made it clear: I don't expect this to be
ignored. I expect the server to do something sane to convert *all*
messages with the best fidelity possible to the JMAP Message format when
requested. The case of multipart mixed is actually fairly straight
forward. Suppose you have
[Text A]
[File.zip]
[Text B]

I think this should be converted to an htmlBody something like:

<div>Text A</div>
<div><br></div>
<div><a href="cid:...">File.zip</a>
<div><br></div>
<div>Text B</div>

And the textBody version very similar but without the tags.

> (Now I'm curious what the FastMail client & server code does when it
> gets one of these Apple client created emails...)
I just checked: it concatenates the text parts but doesn't insert the
mid-body links to ”attachments“ (the non-text parts are shown as
attachments to the email though of course). We handle the image case
fully though (the common case), converting this to HTML with <img> tags
in the appropriate place with a cid src so it all works.
> Actually, this makes me wonder... what should a jmap server do if the> stored message is somehow malformed?

It needs to make a best effort to represent it; I'm wary of trying to
give exact algorithms for this in the spec since there is an infinite
variety of brokenness out there and you tend to have to build up
heuristics based on the samples you see in the wild over time. Again
though, I agree the spec definitely needs to at least outline what is
expected here.
Neil.