Subject: [Jmap] Decoding of unknown headers

From: Bron Gondwana <[email-masked]>
Date: Mon, 27 Nov 2017 15:39:15 +1100

https://github.com/jmapio/jmap/issues/165

The interesting case is headers that contain arbitrary things (like raw
message-ids) that aren't necessarily encoded in RFC2047 format.
Arguments for decoding everything as it if was:
* most clients do this already for everything they display
* JSON only transports UTF-8/characters.
* you can always fetch the raw message if you need to decode
  differently.
Arguments against:
* mostly you won't be displaying unknown headers anyway, but the client
  might need to use the raw value for something.* means clients have to do their own MIME parsing if they need those
  headers, and there's no cheap way to get just that header.
A potential option is to provide rawheader and rawheader.$name fetch
options as well.  It adds to server complexity, but not greatly, since
the server will have to handle the data in that format anyway.
Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Josef 'Jeff' Sipek <[email-masked]>
Date: Tue, 28 Nov 2017 09:11:06 -0500

On Mon, Nov 27, 2017 at 15:39:15 +1100, Bron Gondwana wrote:
>   might need to use the raw value for something.
> * means clients have to do their own MIME parsing if they need those

Right now, jmap doesn't "gracefully degrade" the experience.  If a client
wants to do the boring few things, jmap rocks.  However, the moment the
client strays from just getting the subject/to/date/body for *any* reason,
the client suddenly has to implement a ton of RFCs (MIME, RFC5322, etc.) and
the experience degrades to something like IMAP.

So, I'm a bit wary of "just fetch the raw blob and parse it yourself".
(This is the same concern as with the message representation discussion.)

> the server will have to handle the data in that format anyway.

rawheader should work.  We could avoid dealing with UTF-8 by base64 encoding
the string.  (IIRC, at least one popular email API out there uses base64
encoded strings to transport potentially non-UTF-8 data to the client
through JSON.)

Jeff.

-- 
Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it.
		- Brian W. Kernighan

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 28 Nov 2017 16:00:31 +0000

On 28/11/2017 14:11, Josef 'Jeff' Sipek wrote:

> the client suddenly has to implement a ton of RFCs (MIME, RFC5322, etc.)
Exactly.
> and the experience degrades to something like IMAP.
IMAP certainly fails to degrade gracefully in a similar way. If certain 
data is not easily available, the client needs to go out of its way to 
get the date (e.g. try to do S/MIME multipart/signed validation using 
IMAP without MIME parser...). But I digress.
> rawheader should work.
I think this would be the best, as rawheader instructs the JMAP server 
not to be clever and just return a particular header field value.
>    We could avoid dealing with UTF-8 by base64 encoding
Works for me.

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 29 Nov 2017 14:58:57 +1100

Some thoughts:

We currently have the following properties which represent parsed forms
of headers: sender, from, to, cc, bcc, replyTo, subject, date. You can
also request (decoded but otherwise unparsed) headers using the
"headers" property. Perhaps what we should do is remove the special case
properties and instead have the client request header names + how to
process each one (raw base64, decoded, parsed as name/email list, parsed
as date; may be extensible in the future). The syntax would need to be
usable for setMessages too in some way.
The other thing to think about is what to do when you have multiple
headers of the same name; maybe you can explicitly request to receive
all (as an array), otherwise you get the last one of that name.
One possible syntax:

"getMessages", {
  ...
  "properties": [
    "headers.received:decoded+all"
    "headers.from:raw",
    "headers.to:emails",
    "headers.subject:decoded",
    "headers.date:date"
  ],
}

and then the Message object would look something like:

{
    "headers": {
        "received": {
            "type": "decoded+all",
            "value": [
                "from mx2 ([10.202.2... etc.",
                "from mx2.messaginge... etc."
            ]
        }
        "from": {
            "type": "raw",
            "value": "...base64..."
        },
        "to": {
            "type": "emails",
            "value": [
                {"name": "Joe Bloggs", "email": "[email-masked]"},
                {"name": "Jane Doe", "email": "[email-masked]"}
            ]
        },
        "subject": {
            "type": "decoded",
            "value": "10 år med fornøyde kunder"
        },
        "date": {
            "type": "date",
            "value": "2017-11-11T18:09:31+01:00"
        }
    }
}

This gives a lot of flexibility to clients, but is a bit  verbose. On
the other hand if you kept the "from", "to", "subject" top-level
properties as shortcuts for common header requests, most clients
wouldn't need to fallback to the verbose form so much. You could also
default to type "decoded" if no other processing explicitly requested,
and possibly even drop the {type, value} wrapper for headers with the
default decoding.
Thoughts?

Neil.