Subject: [Jmap] Review of draft-ietf-jmap-core-00.txt and draft-ietf-jmap-mail-00.txt

From: "Chris Newman" <[email-masked]>
Date: Fri, 14 Jul 2017 15:10:12 +0200

draft-ietf-jmap-core-00

Overall: This is a high quality -00 document.

2.2 Getting an access token

You refer to "POST request" before defining what that means. I think
you need a clear statement that JMAP as described in this
specification runs on top of HTTPS, but the design does not preclude
running JMAP on top of an application protocol with semantics similar
to HTTP in the future.

2.2.1 more authorization required

SASL/GSSAPI?
  * Kerberos, SCRAM, etc

Recommend dropping yubikeyotp & u2f from base spec, but create a
registry for AuthMethod that is expert review so they can be added
later if need be.

Need to specify a minimal security mechanism that must be implemented
for interoperability of implementations. Likely https with current TLS
policy + password. It's OK to turn off the minimal mechanism as a site
policy matter if better mechanisms are available, but this is helpful
guidance for implementers. If implementer A does passwords only,
implementer B does oauth only, and implementer C does totp, these
implementations won't interoperate.

2.2.2 authentication complete

This should clarify if the server is allowed to return a different
'username' from the POST request.

> maxObjectsInGet: Number The maximum number of obje ts that the client
> may request in a single getFoos type method call.

Typo: obje ts => objects

Why base64url rather than base64?

communicate expected lifetime of accessToken & signingKey?

Recommend including a password expiration warning (either timestamp or
X days/hours).

accounts / isReadOnly: if the entire account is read-only => if the 
entire
account is read-only based on this access token. Q: does this cover
flags?

*URL => need to define minimum supported set of URI types.

2.2.3-2.2.9

Need error codes for:
* Account Disabled / Contact Admin
* Password Expired

2.2.9: maybe should be 'service temporarily unavailable'. Should
server suggest a backoff in the payload? There should probably be some
definition of "exponential backoff".

2.5.1: needs hash agility; at present time, HMAC SHA-256 MUST be
implemented, but algorithms may be added or deprecated by future best
practice documents.

3.2. API structure

This model only allows extensibility by adding methods or altering
method parameters. There's no way to attach properties to the entire
API request. There are a number of potentially interesting
request-level properties such as "fail on error" (stop on first error)
or "transaction"/"atomic" (either perform all methods successfully or
none of them). If you made the top-level an object and the array of
methods a property of the top-level object, you'd get request-level
extensibility.

3.3. Errors

"transport level" => This seems wrong because TCP is typically the
transport level for HTTP-based protocols, but returning a TCP-level
error is not desirable here. Why not just simplify this to one
sentence saying the error is returned at the HTTP level.

3.6 Concurrency

For potentially long-running operations like search and
copy/move-messages, the creates an inappropriate and unnecessary
implementation burden on the server. In general, the goal is that the
server should present a consistent view of data to the client, even
if it's slightly out-of-date by the time the method completes. It's
fine to state this and even say the ability of a server to present a
consistent view to clients is a quality-of-implementation issue for
the server. But don't force the presence of a hard locking mechanism
into the server implementation unless it's absolutely necessary as
that harms server scalability.

Normative restrictions on concurrency should be specified for those
methods that need them and be minimally restrictive. For example, a
multi-message move operation should either move all messages or none
of them, and the messages should retain their relative ordering in the
destination folder. But additional restrictions are not needed to
present a good client experience.

3.8 Date datatype

Are you sure you want to pin the timezone of timestamps to UTC? This
potentially loses information.

3.10.1 getFoos

state: need to specify scope and duration of this string. I'd guess it
is scoped to the specific getFoos request, so there's no requirement
for portability of this state token to other users or types. But
it'd be better if it's quite persistent (e.g., lasts a month or until
the client issues an update).

3.10.2 getFooUpdates

fetchRecords & fetchRecordProperties

Instead of two separate parallel properties, how about defining
fetchRecords as having the same structure as a getFoos object, but
with appropriate defaults for accountId and ids. This way,
extensions to the getFoos object naturally apply to the fetchRecords
value object.

If the cannotCalculateChanges error occurs too frequently, the server
becomes problematic for client performance. A quality server should
strive to persist state tokens for several clients per user for at
least a month.

3.10.3. setFoos

> Each create, update or destroy is considered an atomic unit.
I suggest:
Each create, update or destroy list item is considered an atomic unit.

> record, so it can substitute in the correct value if necessary in
> later method calls. The type
dangling end of paragraph.

> a separate creation id -> id map MUST be kept for each type.

I assume these must only be kept for the duration of a single API
request? It would be helpful to say that explicitly.

3.11.1. getFooList

position: it's very useful to specify position relative to the end of
the list. For mail, the following search is common: "the 50 most
recent not-deleted messages". I suggest making -50, the 50th position
from the end of the list (the way Python list slicing works). Failure
to add support like this makes RFC 5267 partial results far less
useful than it could have been. For the response, always using
non-negative positioning makes sense.

5. Uploading binary data

Can you make the header "JMAP-AccountId" instead of "X-JMAP-AccountId"
per RFC 6648?

5.1 file upload

The 'type' field is not well defined. Is this the Content-Type header
field with CFWS collapsed to SP and RFC 2231/2047 encoding removed? Or
should the content-type parameters be split out into a separate
property with an object as a value?

size: For IMAP compatibility, this needs to allow sizes up to 2^32-1,

6.2 Web hook

> The JMAP server MUST follow any redirects.

Maybe I'm a bit paranoid, but I suggest: The JMAP server
implementation MUST be capable of following at least one redirect, but
MAY choose not to do so for configured policy reasons.

6.2.1. setPushCallback

I think this needs another error code:

policyDenied: The server is not willing to connect to the specified
URL for policy reasons.

====

draft-ietf-jmap-mail-00

2. Mailboxes

Need to distinguish JMAP message id from RFC 5322 Message-ID
header. Duplicate 5322 message ids occur, so it's helpful to have a
JMAP message ID that's unique within an account.

role: I support proposal to either remove 'outbox' (or make it
optional) and replace with MessageDelivery/MessageSubmission object.
I'd also like to see this more closely aligned with RFC 6154 (at least
use the same names). We should discuss issue of multiple folders with
same role; I like the simplicity of not having that, but RFC 6154 was
written as it is for a reason; we should make sure that reason doesn't
apply to JMAP before changing that design.

sortOrder: don't hand-wave the locale-specific character order problem
as that breaks interoperability. I suggest abstracting the problem by
associating a collation (RFC 4790) with each account. If there's a
collation associated with the account, that produces a deterministic
order. Otherwise, permission has been given to be
non-deterministic. Server operators can derive the collation algorithm
name from the user's language when they register their account; then
it will work even on a client with the wrong locale configured.

mustBeOnlyMailbox: I don't understand the distinction. Is it
important?

"may*": Can we just replace this with a "myrights" per RFC 4314? If
you want to drop the rights grouping model of RFC 4314 to be simpler,
that's fine with me, but I'd prefer to keep the set of rights the same.

2.3.3. updating mailboxes

myrights (may*) isn't immutable; it's server-generated (possibly based
on an IMAP ACL from RFC 4314) so the client can't directly update
it. I think it's fine to leave ACLs as an extension to base JMAP but
having the equivalent of myrights in the base spec is a good decision.

3.1.1 filtering

This really needs to be semantically closer to IMAP. In particular,
support for extensible flags is needed.

Does this need an extensibility mechanism? A way to filter the search
that's ignored if not understood by the server (like non-critical LDAP
search controls)?

3.1.2. Sorting

We have a collation model (RFC 4790) that covers sorting. I think
servers need to implement either RFC 5051 or the Unicode Collation
Algorithm as a minimum/default. Allowing a collation to be specified
would be good for interoperabiltiy/predictability.

5. Messages

Overall, I find this to be almost as problematic as the sending model of 
the specification. I understand the goal of trying to make a minimal 
JMAP client work without having to implement 5322/MIME/2047/2231. But 
the problem with creating a data format gateway is you've got the union 
of the problems and the intersection of the functionality. And while 
having a way to fetch the raw blob is nice, that's not great for clients 
that want some of the 5322/MIME/2047/2231 functionality that's missing 
from JMAP but don't want to be forced to fetch the full message and do a 
full parse of it.

I suggest a different approach. First, define a set of properties that 
are largely un-adulterated (similar to IMAP header fetch, IMAP body part 
references, etc). Then define properties that try to 
convert/simplify/flatten that data (losing information in the process). 
Then clients can choose the level of precision they want. You'll need to 
describe the algorithms used to convert/simplify/flatten the data in 
detail. And if you're going to simplify some address fields, you need 
the ability to simplify all of them, including ones like 
Message-Disposition-To.

> or RFC 2047 encoding of non-ASCII characters, MUST be fully decoded
> into standard UTF-8.

If you say this, you need to identify the charsets that MUST be 
supported; otherwise the requirement is impossible. You'll also want to 
mention RFC 2231.

> hasAttachment: Boolean Does the message have any attachments?

You need to define what you mean by an attachment. Do you mean a MIME 
part with a Content-Disposition of "attachment", or do you mean 
something else? Be precise.

textBody & htmlBody: I assume these are converted to UTF-8? This model 
is not good. I generally want to be presented the plain text body if 
present and the HTML body if the text body is not present. Are you 
unfolding RFC 3676 for this? What if it's S/MIME signed?

date: the timezone of the sender is relevant information, so if you're 
parsing out the date, you should parse out that field as well. For 
submission you need a way to provide the timezone.

8. Vacation Response

Can this be moved to an extension? The question of how this interacts 
with Sieve, ManageSieve, etc, is tricky.

====

Missing stuff:

* email address internationalization
=> the protocol could be defined in such a way that this will mostly 
"just work" (to the degree that's possible). Needs changes to Emailer 
object, Message object, etc.

I don't have time to do it now, but I'll run through the list of IMAP 
extensions to come up with more 'missing stuff'.

		- Chris

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 19 Jul 2017 17:35:54 +1000

On Fri, 14 Jul 2017, at 23:10, Chris Newman wrote:
> 2.x [ ... ]

I haven't created any tickets for this as the discussion is mostly superseded by "split authentication out".  If we do go ahead with our own authentication document within JMAP, I'll come back to this for notes.
> request-level properties such as "fail on error" (stop on first error)> or "transaction"/"atomic" (either perform all methods successfully or> none of them). If you made the top-level an object and the array of

https://github.com/jmapio/jmap/issues/115

> sentence saying the error is returned at the HTTP level.

https://github.com/jmapio/jmap/issues/116

> implementation burden on the server. In general, the goal is that the> server should present a consistent view of data to the client, even
> of them, and the messages should retain their relative ordering in the> destination folder. But additional restrictions are not needed to

https://github.com/jmapio/jmap/issues/117

I would probably implement a lot of these things as an MVCC-style snapshot view for read operations.  Write is trickier depending how you do your state management, do you do either need to lock against changes entirely so you can give back a new state string at the end, or you wind up in cases where creating a state string is tricky.
> potentially loses information.

https://github.com/jmapio/jmap/issues/118

> 3.10.1 getFoos
> 
> state: need to specify scope and duration of this string. I'd guess it> is scoped to the specific getFoos request, so there's no requirement
> it'd be better if it's quite persistent (e.g., lasts a month or until> the client issues an update).
> 
> 3.10.2 getFooUpdates
> 
> If the cannotCalculateChanges error occurs too frequently, the server> becomes problematic for client performance. A quality server should
> Each create, update or destroy list item is considered an atomic unit.> 

https://github.com/jmapio/jmap/issues/119 (state string SHOULD persist e.g. 1 month)https://github.com/jmapio/jmap/issues/120 (setFoos and idmap duration)
https://github.com/jmapio/jmap/issues/121 (fetchRecords as getFoos with defaults)
> 3.11.1. getFooList
> 
> position: it's very useful to specify position relative to the end of> the list. For mail, the following search is common: "the 50 most
> recent not-deleted messages". I suggest making -50, the 50th position> from the end of the list (the way Python list slicing works). Failure> to add support like this makes RFC 5267 partial results far less

This already got created as:

https://github.com/jmapio/jmap/issues/114

> 5. Uploading binary data
> 
> Can you make the header "JMAP-AccountId" instead of "X-JMAP-AccountId"> per RFC 6648?

This is already covered by:

https://github.com/jmapio/jmap/issues/92

> 5.1 file upload
> 
> The 'type' field is not well defined. Is this the Content-Type header> field with CFWS collapsed to SP and RFC 2231/2047 encoding removed? Or> should the content-type parameters be split out into a separate
Size is an interesting area of discussion, what with LITERAL- and various server restrictions.  I'm feeling lazy so I just made one ticket for both of these:
https://github.com/jmapio/jmap/issues/122

> implementation MUST be capable of following at least one redirect, but> MAY choose not to do so for configured policy reasons.
> 
> 6.2.1. setPushCallback
> 

https://github.com/jmapio/jmap/issues/123

> JMAP message ID that's unique within an account.

This got made into a ticket during the meeting notes regarding naming:

https://github.com/jmapio/jmap/issues/102

> I'd also like to see this more closely aligned with RFC 6154 (at least> use the same names). We should discuss issue of multiple folders with> same role; I like the simplicity of not having that, but RFC 6154 was> written as it is for a reason; we should make sure that reason doesn't> apply to JMAP before changing that design.

Wow, for all that we've talked about this, we don't actually have a ticket for it!
https://github.com/jmapio/jmap/issues/124

> sortOrder: don't hand-wave the locale-specific character order problem> as that breaks interoperability. I suggest abstracting the problem by> associating a collation (RFC 4790) with each account. If there's a
> non-deterministic. Server operators can derive the collation algorithm> name from the user's language when they register their account; then

https://github.com/jmapio/jmap/issues/125

> mustBeOnlyMailbox: I don't understand the distinction. Is it

I haven't created anything for this, maybe Neil can just reply directly.  This is for servers which don't support having the same message in multiple mailboxes.
> you want to drop the rights grouping model of RFC 4314 to be simpler,> that's fine with me, but I'd prefer to keep the set of rights the same.> 
> myrights (may*) isn't immutable; it's server-generated (possibly based> on an IMAP ACL from RFC 4314) so the client can't directly update
https://github.com/jmapio/jmap/issues/126

> Does this need an extensibility mechanism? A way to filter the search> that's ignored if not understood by the server (like non-critical LDAP> search controls)?

Extending filtering would be done with a "using jmapesearch" or something, and that spec would define exactly how it extended filtering, so clients would opt in to search methods that the server claimed to support.
> would be good for interoperabiltiy/predictability.

updated and renamed #125 :)

> 5. Messages
> 
> Overall, I find this to be almost as problematic as the sending model of> the specification. I understand the goal of trying to make a minimal
> JMAP client work without having to implement 5322/MIME/2047/2231. But> the problem with creating a data format gateway is you've got the union> of the problems and the intersection of the functionality. And while
> having a way to fetch the raw blob is nice, that's not great for clients> that want some of the 5322/MIME/2047/2231 functionality that's missing> from JMAP but don't want to be forced to fetch the full message and do a> full parse of it.
> 
> I suggest a different approach. First, define a set of properties that> are largely un-adulterated (similar to IMAP header fetch, IMAP body part> references, etc). Then define properties that try to
> convert/simplify/flatten that data (losing information in the process).> Then clients can choose the level of precision they want. You'll need to> describe the algorithms used to convert/simplify/flatten the data in
> detail. And if you're going to simplify some address fields, you need> the ability to simplify all of them, including ones like
> Message-Disposition-To.
> 
> supported; otherwise the requirement is impossible. You'll also want to> mention RFC 2231.

This is all covered by https://github.com/jmapio/jmap/issues/104 but I've copied this text into a comment.
> You need to define what you mean by an attachment. Do you mean a MIME> part with a Content-Disposition of "attachment", or do you mean

https://github.com/jmapio/jmap/issues/127

> textBody & htmlBody: I assume these are converted to UTF-8? This model> is not good. I generally want to be presented the plain text body if

Threw this into 104.

> date: the timezone of the sender is relevant information, so if you're> parsing out the date, you should parse out that field as well. For

and this :)

> 8. Vacation Response
> 
> Can this be moved to an extension? The question of how this interacts> with Sieve, ManageSieve, etc, is tricky.

We talked this one out at dinner on Monday night.  Let me know if you still want a ticket created for it.
> "just work" (to the degree that's possible). Needs changes to Emailer> object, Message object, etc.
> 
> I don't have time to do it now, but I'll run through the list of IMAP> extensions to come up with more 'missing stuff'.

And likewise with this bit.

Do you have a github user?  Might be easier to add you to the org so you can create/edit tickets yourself!
Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]