From nobody Wed Feb 7 22:11:32 2018 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BDB6312773A for ; Wed, 7 Feb 2018 22:11:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.719 X-Spam-Level: X-Spam-Status: No, score=-2.719 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=fastmailteam.com header.b=t5WJ0TzZ; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=ddwk5bGe Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qQ-euRirMLko for ; Wed, 7 Feb 2018 22:11:26 -0800 (PST) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 67AC21243F6 for ; Wed, 7 Feb 2018 22:11:26 -0800 (PST) Received: from betaweb1.internal (betaweb1.nyi.internal [10.202.2.10]) by mailout.nyi.internal (Postfix) with ESMTP id B44D920E45 for ; Thu, 8 Feb 2018 01:11:25 -0500 (EST) Received: from betaweb1 ([::ffff:10.202.2.10]) by betaweb1.internal (MEProxy); Thu, 08 Feb 2018 01:11:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= fastmailteam.com; h=content-transfer-encoding:content-type:date :from:message-id:mime-version:subject:to:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=ErqdJKOVpLZMH4i0lQ3TlI0ZwmKqHnR9SB1GH9us4 ks=; b=t5WJ0TzZPRgg9cMaZ+xQO1faoq9ie36rzMKD1f+SLHWUKZFansmPgu5oS zFUcV3d5Je+olugiGvWSvF2vH5593TMeRZhNoVgYR+8dPMZ9PVeM5QrZmNmgb6QX 6rwT/MaS+BxxMEEkg7rgSTBcjPYzMyXpebCwbhRZv05XDHlEqYBjbPrVBjwth9HM 5a0Xr6Gj701linSqy1okl4iq8fNDVjGLUNucKSUg1d/1BUort0FkMyX9vVCRXnqf TLkXwIglsTVNs9Qt+zPmzOzvyqcvWibr3WU5KgvqF2TNbJ4H407P3bqQ1uM24XNy Pw/CdW0djft7DmsLXLnoM0aZmIShg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=ErqdJKOVpLZMH4i0lQ3TlI0ZwmKqH nR9SB1GH9us4ks=; b=ddwk5bGeCQNWlf/Iuu8RUXXdukSNvIXJm2xRjuewECeeW CFC6lYFC7QBzU094LZPVgTrORbxDApl+pmpzDsFUJvMdPz1C55b5pTi3QFLkY22f D9fFrjgEGJiUHRVwSwndSnDtkCHMVDeclK1XmBZdpYj5gu0sIs/lLbfWLydb4A/x Tj5o28m1DpLIbj9JLDwuq/qnZFQPKA6Y6EGM3f5b8Nn5p6BlqBATwVSTxQMf3veS X8wdFPa9XorU3NXvRN4P8odxfQ8aWprbyVRTZUTvZUM5qiTZ5bBHe2ay0j2pvH7X v4cKGZpijNOuPSwNSr3Jue60l2q77GUQ0wre4ldlg== X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 51EDEE22A4; Thu, 8 Feb 2018 01:11:25 -0500 (EST) Message-Id: <1518070284.168650.1263633200.05C7397A@webmail.messagingengine.com> From: Neil Jenkins To: IETF JMAP Mailing List MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: multipart/alternative; boundary="_----------=_15180702851686500" X-Mailer: MessagingEngine.com Webmail Interface - ajax-2ce04827 Date: Thu, 08 Feb 2018 17:11:24 +1100 Archived-At: Subject: [Jmap] New JSON message representation X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Feb 2018 06:11:31 -0000 This is a multi-part message in MIME format. --_----------=_15180702851686500 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With an eye to closing off all remaining issues before the next IETF meeting in March, I have been looking at the issues raised with the JSON representation of an RFC5322 message. This email presents a new proposal that I think resolves the issues presented. In particular, I aim to address the following goals: * Allow access to the full MIME structure without requiring the client to parse the full message itself. * Allow access to full header field information if required. * Keep a simpler representation available for the common case. * Allow the client to only load the data required.Comments, suggestions an= d other feedback requested. Cheers! ------ Emails The *Email* object is a representation of an [@!RFC5322] message, which allows clients to avoid the complexities of MIME parsing, transport encoding and character encoding.Properties of the Email object Broadly, a message consists of two parts: a list of header fields, then a body. The body is normally a MIME-encoded set of documents in a tree structure. The JMAP Email object provides a way to access the full structure, or to use simplified properties and avoid some complexity if this is sufficient for the client application.Due to the number of properti= es involved, the set of *Email* properties is specified over the following three sub-sections.Metadata These properties represent metadata about the [@!RFC5322] message, and are not derived from parsing the message itself. * *id*: String (immutable; server-set) The id of the Email object. Note, this is the JMAP object id, NOT the [@!RFC5322] Message-Id header field value. * *blobId*: String (immutable; server-set) The id representing the raw bytes of the [@!RFC5322] message. This may be used to download the raw original message, or to attach it directly to another Email etc. * *threadId*: String (immutable; server-set) The id of the Thread to which this Email belongs. * *mailboxIds*: String[Boolean] The set of mailbox ids this email belongs to. An email MUST belong to one or more mailboxes at all times (until it is deleted). The set is represented as an object, with each key being a *Mailbox id*. The value for each key in the object MUST be true. * *keywords*: String[Boolean] (default: {}) A set of keywords that apply to the email. The set is represented as an object, with the keys being the *keywords*. The value for each key in the object MUST be true. Keywords are shared with IMAP. The six system keywords from IMAP are treated specially. The following four keywords have their first character changed from \ in IMAP to $ in JMAP and have particular semantic meaning: * $Draft: The email is a draft the user is composing. * $Seen: The email has been read. * $Flagged: The email has been flagged for urgent/special attention. * $Answered: The email has been replied to. The IMAP \Recent keyword is not exposed via JMAP. The IMAP \Deleted keyword is also not present: IMAP uses a delete+expunge model, which JMAP does not. Any message with the \Deleted keyword MUST NOT be visible via JMAP. Users may add arbitrary keywords to an email. For compatibility with IMAP, a keyword is a (case-sensitive) string of 1=E2=80=93255 characters in= the ASCII subset %x21=E2=80=93%x7e (excludes control chars and space), and MUST= NOT include any of these characters: ( ) { ] % * " \ The IANA Keyword Registry[1] as established in [@!RFC5788] assigns semantic meaning to some other keywords in common use. New keywords may be established here in the future. In particular, note: * $Forwarded: The email has been forwarded. * $Phishing: The email is highly likely to be phishing. Clients SHOULD warn users to take care when viewing this email and disable links and attachments. * $Junk: The email is definitely spam. Clients SHOULD set this flag when users report spam to help train automated spam- detection systems. * $NotJunk: The email is definitely not spam. Clients SHOULD set this flag when users indicate an email is legitimate, to help train automated spam-detection systems. * *size*: Number (immutable; server-set) The size in bytes of the whole email as counted by the server towards the user's quota. * *receivedAt*: UTCDate (immutable; default: time of creation on server) The date the email was received by the message store. This is the *internal date* in IMAP.Header fields These properties are derived from the [@!RFC5322] message header fields. Header fields may be fetched in a variety of forms: * *Decoded* The header field value with: 1. White space unfolded (as defined in [@!RFC5322] section 2.2.3) 2. The terminating CRLF at the end of the value removed 3. Any SP characters at the beginning of the value removed 4. Any [@!RFC2047] encoded sections decoded 5. Any [@!RFC6532] UTF-8 values decoded If any decodings fail, the parser SHOULD insert a unicode replacement character (U+FFFD) and attempt to continue as much as possible. * *Raw Base64* The raw bytes of the header field value from the first byte following the header field name terminating colon, up to but excluding the header field terminating CRLF, encoded as Base64 ([@!RFC4648]). (Note, this is required as the value may not be valid UTF-8 if 8-bit values are present.) * *Addresses* The header is parsed as an address-list value, as specified in [@!RFC5322] section 3.4, into the EmailAddress[] type. The *EmailAddress* object has the following properties: * *name*: String|null The *display name* of the [@!RFC5322] *mailbox* or *group*, or null if none. * *email*: String|null The *addr-spec* of the [@!RFC5322] *mailbox*, or null if a *group*. Parsing SHOULD be best-effort in the face of invalid structure to accommodate invalid messages and semi-complete drafts. EmailAddress objects MAY have an *email* property that does not conform to the *addr-spec* form (for example, may not contain an @ symbol). * *Date* The header is parsed as a date-time value, as specified in [@!RFC5322] section 3.3, into the Date type.The following *Email* proper= ties are specified for complete access to the header data of the message. Where {header-field-name} is specified within a property name, this means any series of one or more printable US-ASCII characters (i.e. characters that have values between 33 and 126, inclusive), except colon: * *headers*: EmailHeader[] (immutable) This is a list of all [@!RFC5322] header fields, in the same order they appear in the message. The values are in *Decoded* form. * *headers:asRaw*: EmailHeader[] (immutable) This is a list of all [@!RFC5322] header fields, in the same order they appear in the message. The values are in *Raw Base64* form. * *header:{header-field-name}*: String|null (immutable) The value of the *last* header field in the message with the given header field name (matched case-insensitively), or null if none. The value is in *Decoded* form. * *header:{header-field-name}:asRaw*: String|null (immutable) The value of the *last* header field in the message with the given header field name (matched case-insensitively), or null if none. The value is in *Raw Base64* form. * *header:{header-field-name}:asAddresses*: EmailAddress[]|null (immutable) The value of the *last* header field in the message with the given header field name (matched case-insensitively), or null if none. The value is in *Addresses* form. * *header:{header-field-name}:asDate*: Date|null (immutable) The value of the *last* header field in the message with the given header field name (matched case-insensitively), or null if none. The value is in *Date* form.An *EmailHeader* object has the following properties: * *name*: String The header *field name* as defined in RFC5322, with the same capitalization that it has in the message. * *value*: String The header *field value* as defined in RFC5322, in either *Decoded* or *Raw Base64* form as determined by the property being fetched.The following convenience properties are also specified fo= r the *Email* object: * *sender*: EmailAddress[]|null (immutable) The value is identical to the value of *header:Sender:asAddresses*. * *from*: EmailAddress[]|null (immutable) The value is identical to the value of *header:From:asAddresses*. * *to*: EmailAddress[]|null (immutable) The value is identical to the value of *header:To:asAddresses*. * *cc*: EmailAddress[]|null (immutable) The value is identical to the value of *header:Cc:asAddresses*. * *bcc*: EmailAddress[]|null (immutable) The value is identical to the value of *header:Bcc:asAddresses*. * *replyTo*: EmailAddress[]|null (immutable) The value is identical to the value of *header:Reply-To:asAddresses*. * *subject*: String|null (immutable) The value is identical to the value of *header:Subject*. * *sentAt*: Date (immutable; default: time of creation on server) The value is identical to the value of *header:Date:asDate*.Body parts These properties are derived from the [@!RFC5322] message body and its [@RFC2045] MIME entities.A *EmailBodyPart* object has the following propert= ies: * *partId*: String Identifies this part uniquely within the Email. This is scoped to the *emailId* and has no meaning outside of the JMAP Email object representation. * *blobId*: String The id references the raw bytes of the contents of the part after decoding any *Content-Transfer* encoding (as defined in [@!RFC2045]). Note, two parts may be transfer-encoded differently but have same the same blob id if their decoded bytes are identical and the server is using a secure hash of the data for the blob id. * *size*: Number The size, in bytes, of the raw data after content- transfer decoding (as refrenced by the *blobId*, i.e. the number of bytes in the file the user would download). * *headers*: EmailHeader[] (immutable) This is a list of all header fields in the part, in the order they appear. The values are in *Decoded* form. * *headers:asRaw*: EmailHeader[] (immutable) This is a list of all header fields in the part, in the order they appear. The values are in *Raw Base64* form. * *name*: String|null This is the [@!RFC2231] decoded *filename* parameter of the *Content-Disposition* header field, or (for compatibility with existing systems) if not present then the [@!RFC2047] decoded *name* property of the *Content-Type* header field. For compatibility with the widest array of existing systems, both forms should be set if creating a message. * *type*: String The value of the *Content-Type* header field of the part, excluding any parameters. * *charset*: String|null The value of the charset parameter of the *Conten= t- Type* header field, if present. * *disposition*: String|null The value of the *Content-Disposition* header field of the part, excluding any parameters, if present. * *cid*: String|null The value of the *Content-Id* header field of the part, if present. This may be used to reference the content from within an html body part using the cid: protocol. * *language*: String[]|null The list of language tags, as defined in [@!RFC3282], in the *Content-Language* header field of the part, if present. * *location*: String|null The URI, as defined in [@!RFC2557], in the *Content-Location* header field of the part, if present. * *width*: Number|null The width (in px) of the image, if the type is an image and the server is able to determine this information for the image. This MAY not be supported for all or any image types on some servers. * *height*: Number|null The height (in px) of the image, if the type is an image and the server is able to determine this information for the image. This MAY not be supported for all or any image types on some servers. * *subParts*: EmailBodyPart[] (optional) If type is multipart/*, this contains the body parts of each child.The following *Email* properties a= re specified for access to the body data of the message: * *bodyStructure*: EmailBodyPart[] (immutable) This is the full MIME structure of the message body, including sub parts but not recursing into message/rfc822 parts. * *bodyValues*: String[BodyValue] (immutable) This is a map of *partId* to an *EmailBodyValue* object. Which parts are included and whether the value is truncated is determined by various arguments to *Email/get* and *Email/parse*. An *EmailBodyValue* object has the following properties: * *value*: String The value of the body part after decoding *Content- Transport* encoding and decoding the *Content-Type* charset. * *isTruncated* Boolean This is true if the *value* has been truncated. * *textBody*: EmailBodyPart[] (immutable) A list of text/plain, text/html and image/* parts to display (sequentially) as the message body, with a preference for text/plain when alternative versions are available. * *htmlBody*: EmailBodyPart[] (immutable) A list of text/plain, text/html and image/* parts to display (sequentially) as the message body, with a preference for text/html when alternative versions are available. * *attachedEmails*: EmailBodyPart[] (immutable) A list of all parts of type message/rfc822. Note, this *does not* recurse, so the parts within these are not included. The attached message may be fetched using the Email/parse method and the blobId. * *attachedFiles*: EmailBodyPart[] (immutable) A list of all parts in *bodyStructure*, traversing depth-first, not of type multipart/*, and not included in any of the *textBody*, *htmlBody* or *attachedEmails* properties. Note, an HTML body part may reference image parts in attachedFiles using cid: links to reference the *Content-Id* or by referencing the *Content-Location*. * *hasAttachment*: Boolean (immutable; server-set) This is true if there are one or more parts in the message that a client UI should offer as downloadable. A server SHOULD set hasAttachment if either: * The *attachedEmails* list contains at least one item. * The *attachedFiles* list contains at least one item that does not have Content-Disposition: attachment. The server MAY ignore parts in this list that are processed automatically in some way, or are referenced as embedded images in one of the text/html parts of the message. The server MAY set hasAttachment based on implementation- defined or site configurable heuristics. * *preview*: String (immutable; server-set) Up to 256 characters of the beginning of a plain text version of the message body. This is intended to be shown as a preview line on a mailbox listing, and the server may choose to skip quoted sections or salutations to return a more useful preview.MIME structures are arbitrary nested trees of docume= nts, but the majority of email clients present a model of an email body (normally plain text or HTML), with a set of attachments. Interpreting the MIME structure to form this flat model represents considerable difficulty and causes inconsistency between clients. Therefore in addition to the *bodyStructure* property, which gives the full tree, the Email object contains 4 alternate properties with flat lists of body parts: * *textBody*/*htmlBody*: These provide a list of parts that should be rendered as the "body" of the message. This is a list rather than a single part as messages may have headers and/or footers appended/prepended as separate parts as they are transmitted, and some clients send text and images intended to be displayed inline in the body as multiple parts rather than a single HTML part with referenced images. Because MIME allows for multiple representations of the same data (using multipart/alternative), there is a textBody property (which prefers a plain text representation) and an htmlBody property (which prefers an HTML representation) to accommodate the two most common client requirements. The same part may appear in both lists where there is no alternative between the two. * *attachedEmails*/*attachedFiles*: These provide a list of parts that should be presented as "attachments" to the message. Emails are presented in a separate list so their contents may be easily fetched via a back-reference with the Email/parse method in the same request, if the client wishes to. Some images in attachedFiles may be solely there for embedding within an HTML body part; clients may wish to not present these as attachments in the user interface if they are displaying the HTML with the embedded images directly.The *bodyValues* p= roperty allows for clients to fetch the value of text parts directly without having to do a second request for the blob, and have the server handle decoding the charset into unicode. This data is in a separate property rather than on the EmailBodyPart object to avoid duplication of large amounts of data, as the same part may be included twice if the client fetches more than one of bodyStructure, textBody and htmlBody.The exact algorithm for decomposing bodyStructure into textBod= y, htmlBody, attachedEmails and attachedFiles part lists is not mandated, as this is a quality-of-service implementation issue and likely to require workarounds for malformed content discovered over time.However, we = will define a suggested algorithm for the RFC to help implementors. TODO.Email/get Standard */get* method, with the following additional arguments: * *bodyProperties*: String[] (optional) A list of properties to fetch for each EmailBodyPart returned. If omitted, this defaults to: [ "partId", "blobId", "size", "name", "type", "charset", "disposition", cid", "language", "location", "width", "height" ] * *fetchTextBodyValues*: Boolean (default: false) If true, the *bodyValues* property includes any text/* part in the textBody property. * *fetchHTMLBodyValues*: Boolean (default: false) If true, the *bodyValues* property includes any text/* part in the htmlBody property. * *fetchAllBodyValues*: Boolean (default: false) If true, the *bodyValues* property includes any text/* part in the bodyStructure property. * *maxBodyValueBytes*: Number (optional) If supplied by the client, the value MUST be a positive integer greater than 0. If a value outside of this range is given, the server MUST reject the call with an invalidArguments error. When given, the *value* property of any EmailBodyValue object returned in *bodyValues* MUST be truncated if necessary so it does not exceed this number of bytes in size. The server MUST ensure the truncation results in valid UTF-8 and does not occur mid-codepoint. If the part is of type text/html, the server SHOULD NOT truncate inside an HTML tag.If the standard *properties* argu= ment is omitted or null, the following default MUST be used instead of "all" properties: [ "id", "blobId", "threadId", "mailboxIds", "keywords", "size", "receivedAt", "sender", "from", "to", "cc", "bcc", "replyTo", "subject", "sentAt", "hasAttachment", "preview", "bodyValues", "textBody", "htmlBody", "attachedFiles", "attachedEmails" ]The following properties are= expected to be fast to fetch in a quality implementation: * id * blobId * threadId * mailboxIds * keywords * size * receivedAt * sender * from * to * cc * bcc * replyTo * subject * sentAt * hasAttachment * previewClients SHOULD take care when fetching any other properties, as t= here may be significantly longer latency in fetching and returning the data.Exam= ple Request: ["Email/get", { "ids": [ "f123u456", "f123u457" ], "properties": [ "threadId", "mailboxIds", "from", "subject", "receivedAt", "htmlBody", "bodyValues" ], "bodyProperties": [ "partId", "blobId", "size", "type" ], "fetchHTMLBodyValues": true, "maxBodyValueBytes": 256 }, "#1"]and respon= se: ["Email/get", { "accountId": "abc", "state": "41234123231", "list": [ { "id": "f123u457", "threadId": "ef1314a", "mailboxIds": { "f123": true }, "from": [{name: "Joe Bloggs", email: "joe@bloggs.com"}], "subject": "Dinner on Thursday?", "receivedAt": "2013-10-13T14:12:00Z", "htmlBody": [{ "partId": "1", "blobId": "841623871", "size": 283331, "type": "text/html" }, { "partId": "2", "blobId": "319437193", "size": 10343, "type": "text/plain" }], "bodyValues": { "1": { "isTruncated": true, "value": "

Hello ..." }, "2": { "isTruncated": false, "value": "-- \nSent by your friendly mailing list ..." } } } ], notFound: [ "f123u456" ] }, "#1"] Links: 1. https://www.iana.org/assignments/imap-keywords/imap-keywords.xhtml --_----------=_15180702851686500 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

With an eye to closing off all remaining issues before the next = IETF meeting in March, I have been looking at the issues raised with the JS= ON representation of an RFC5322 message. This email presents a new proposal= that I think resolves the issues presented. In particular, I aim to addres= s the following goals:
  • Allow access to the full MIME structure without requiring the clien= t to parse the full message itself.
  • Allow access to full header= field information if required.
  • Keep a simpler representation a= vailable for the common case.
  • Allow the client to only load the= data required.
Comments, suggestions and other feedback = requested. Cheers!

------

Emails

The Email object is a representa= tion of an [@!RFC5322] message, which allows clients to avoid the complexit= ies of MIME parsing, transport encoding and character encoding.

Properties of the Email object

Broadly, a message consists of two parts: a list of header fields, then a = body. The body is normally a MIME-encoded set of documents in a tree struct= ure. The JMAP Email object provides a way to access the full structure, or = to use simplified properties and avoid some complexity if this is sufficien= t for the client application.

Due to the number of properties inv= olved, the set of Email properties is specified over the following t= hree sub-sections.

Metadata

These pro= perties represent metadata about the [@!RFC5322] message, and are not deriv= ed from parsing the message itself.

Header fields=

These properties are derived from the [@!RFC5322] message heade= r fields. Header fields may be fetched in a variety of forms:

T= he following Email properties are specified for complete access to t= he header data of the message. Where {header-field-name} is sp= ecified within a property name, this means any series of one or more printa= ble US-ASCII characters (i.e. characters that have values between 33 and 12= 6, inclusive), except colon:

An EmailHeader obj= ect has the following properties:

The following convenience properties are also sp= ecified for the Email object: