Subject: [Jmap] Feedback on draft-ietf-jmap-core

From: Martin Thomson <[email-masked]>
Date: Wed, 5 Jul 2017 11:14:12 +1000

This is an enormous document.  I think that the working group would do
well to remember that the size of a document is inversely proportional
to the chance that it will be completed, or implemented, or even
implemented correctly, or especially implemented interoperably.  I can
see several ways in which this might be reduced in different ways.

1. Remove the bespoke authentication process.  HTTP has several
options available and many APIs don't require the use of those methods
to operate correctly.  I was shocked to see that the scheme defined in
the document even includes request signing, which has not been
successfully standardized yet.  (I can go into the long history of
these failures separately.)

2. Specify actions in terms of document formats and the semantics of
the fields.  See (for example)
https://www.mnot.net/blog/2017/05/11/status_codes for more information
on why the extensive reinterpretation of HTTP status codes that the
current draft does is inadvisable.

3. Use HTTP.  This has many aspects, but the most immediate concern is
the use of a RPC-like structure for making requests.  I was told when
this group was being chartered that this had good reasons for
existence.  I don't remember precisely, but I think that it was
something about having requests that depended on other requests and
that performance was bad.  Either way, we should discuss these
constraints more and see if there isn't a better solution.

4. Don't use EventSource.  Use HTTP/2 push if you need this sort of
functionality, or simple long polling.  Those things no longer cost
what they used to.

5. Don't use callbacks.  Client reachability isn't consistent.

Finally, I think that there is little benefit to having a "core"
protocol definition for JMAP.  Many of the functions that are provided
in the document are better provided with native HTTP functions.  For
instance, there are well established patterns for providing CRUD
semantics in HTTP.  You can look at the WebDAV or even Atompub
specifications for worked examples.

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 05 Jul 2017 14:57:30 +1000

Hi Martin

> This is an enormous document.  I think that the working group would do> well to remember that the size of a document is inversely proportional> to the chance that it will be completed, or implemented, or even
> implemented correctly

There are already several fairly complete implementations of the draft that I'm aware of. Length is not necessarily a proxy for complexity (the addition of further examples will lengthen the document, but make it quicker to grok). I believe the plan for a test suite to accompany the spec is likely to have far more relevance to correct/interoperable implementations than purely the length of the spec.
> 1. Remove the bespoke authentication process.  HTTP has several options available
The trouble is:
 * Basic auth: is essentially deprecated in this space. Gmail, FastMail, iCloud and I'm sure a growing number of others do not allow users to log in to IMAP (for example) with just a regular username/password.
 * OAuth: generally requires clients to register with each service in advance before you can have an authentication flow. This is fine if you are writing a Twitter client for the Twitter API, but impossible to do for the millions of different mail services out there, any one of which a random user expects to be able to connect to. This is possibly addressed by OAuth 2.0 Dynamic Client Registration Protocol (RFC7591[1]); I'll have to read more of the details, but I'm dubious this reduces complexity for implementors.
Ideally, I think the authentication scheme needs to meet the following criteria:
 * Is session based (creates an authentication token rather than allowing username/password directly).
 * Does not require the client writer to have manually registered with the JMAP server.
 * Allows common 2FA factors to be used (e.g. TOTP, U2F).
 * Be extensible with further authentication mechanisms in the future.
 * Doesn't require a browser in order to preform authentication.
If you have a suggestion of an existing authentication scheme that meets these criteria, we should look at that.
> I was shocked to see that the scheme defined in the document even includes request signing, which has not been> successfully standardized yet.  (I can go into the long history of these failures separately.)
This is only used to allow UAs to issue GET requests for blobs (files, attachments etc.) without setting an Authorization header. This allows you to build a full JMAP client in a browser (where for example you cannot add an Authorization header to the HTTP request resulting from setting a src attribute on an <img>), and probably simplifies other environments too.
The request signing uses JWTs as specified in RFC7519; I was not aware there is separate work going on to standardise the combination of this with URLs, which seems possibly overkill given it's fairly simple scheme.
If you believe the specified method does not provide the desired functionality, or has a security issue that needs to be addressed, then this would be valuable feedback. Making vague references about being shocked is not.
> the fields. […]
> 3. Use HTTP.  This has many aspects, but the most immediate concern is> the use of a RPC-like structure for making requests.

I explained the technical decisions behind this in the "Structure of a request/response" part of my email "JMAP Conceptual Decisions", a few weeks ago. If you missed it, a plain-text copy is available in the IETF archive at https://mailarchive.ietf.org/arch/msg/jmap/SfCe3UJPEEUdeomeso_-PAzP5tQ – please address the rationale/example there directly.
> functionality, or simple long polling.  Those things no longer cost

The EventSource part can easily be made to support long-polling instead at the option of the client (the only difference being it closes the connection after every push). Having a direct way for clients that are able to hold open connections get push notifications without requiring their own push server (see below) is still a good thing to have IMO.
> 5. Don't use callbacks.  Client reachability isn't consistent.

I presume this is referring to the Web Hook part of the push section of the spec, as this is the only place this word is mentioned. I'm not sure exactly what you mean by "client reachability isn't consistent", but the intention here (with a few header changes still required) is that the JMAP server is an RFC8030 compliant "Application Server". UAs can (with their own Push Server) use the rest of 8030 to provide push, or integrate their own mechanism with this to receive push notifications (for example Apple last time I spoke to them were not interested in using 8030 directly, but could still integrate this with their APNS system).
Regards,
Neil.

Links:

  1. https://tools.ietf.org/html/rfc7591

---

From: Martin Thomson <[email-masked]>
Date: Wed, 5 Jul 2017 18:16:40 +1000

Hi Neil,

My MTA seems to have lost your nicely decorated quoting, sorry.  Who
would have thought that that would still be a problem.

> these criteria, we should look at that.

I appreciate that HTTP authentication is (badly) challenged, but it
will depend greatly on what context you are operating in.  For
instance, if you are assuming that there can't be a user and web
browser involved in the onboarding process, that makes things pretty
challenging.

You really need to motivate that more strongly than you have: account
creation is an exceptional process and covering that makes something
arbitrarily complex.  There are many questions that you can avoid by
assuming the presence of a user with a browser: do you need CAPTCHA?
do you need to setup 2FA?  if there is a password, what are the
policies on those?

The problem with the current document is that you spend 11ish pages on
defining a new authentication scheme.  That's a huge amount of work
that distracts from your main task: making a mail access protocol.

Also, from experience with these things, 11 pages is far from enough
information to describe an authentication scheme, particularly one
with this level of complexity.  For example, I don't think that the
"totp" scheme is specified in sufficient detail to implement
interoperable implementations.

The IETF chartered a working group that concentrates on this exact
problem.  I know that you don't want to create dependencies on other
groups, but that is where this is currently headed.  I would encourage
you to talk to the folks in the OAUTH and HTTPAUTH working groups to
see if they have solutions for your problems.  There's a lot going on
there.

> I explained the technical decisions behind this in the "Structure of a
> request/response" part of my email "JMAP Conceptual Decisions", a few weeks
> ago. If you missed it, a plain-text copy is available in the IETF archive at
> https://mailarchive.ietf.org/arch/msg/jmap/SfCe3UJPEEUdeomeso_-PAzP5tQ

Since I only just now joined this list, I will reply here.  I think
that you can achieve your goals within the same framework.

As an aside, I'm accepting for the moment that your performance goals
cannot be met by the simple use of HTTP that you discredit in that
thread. I'm not an expert in this area, other than also having long
ping times to servers that many in this organization assume are close,
and I know very well the value of cutting round trips, but I'm not
convinced that the performance constraints are so severe that they
demand this much contortion.

If they are indeed that severe, even then I would prefer if you
explored alternative options that would be less opaque to HTTP.  Have
you considered creating explicit dependencies between requests just in
your request bodies?  That would allow you to send the requests
concurrently.  It's possible that a generic facility like that would
be valuable in HTTP for use with transactional APIs.

Or there is the pattern of transactional resources.  In this you
create a resource. That can be done ahead of the need to use it so
that you avoid spending the round trips on setup.  You then populate
that resource (in this case, with the actions you wish to take), then
you execute the transaction.  This model has other virtues in that the
non-idempotent requests cause minimal side-effects and the final
transaction commit can use an idempotent method.

You might have already considered these approaches and found them
wanting, or maybe you need more information before you can decide.
I'd like to hear why some time.

>> 4. Don't use EventSource.
> The EventSource part can easily be made to support long-polling instead at
> the option of the client (the only difference being it closes the connection
> after every push). Having a direct way for clients that are able to hold
> open connections get push notifications without requiring their own push
> server (see below) is still a good thing to have IMO.

The people who designed EventSource tell me that it's basically
abandoned.  It was replaced by thewebsocketprotocol, which has been
largely superceded by the facilities that HTTP/2 provides (though I
will concede that of those facilities, only long-polling is a viable
option today; push still a work in progress).

That said, you actually have *two* goals here.  One is for an
asynchronous communications channel that you can use while there are
ongoing interactions between client and server.  For that, I generally
recommend long-polling, though that can lead to generic event streams,
which can easily become the locus of a huge pile of technical debt.

The other is the need to send messages when the client is effectively
offline.  For that, you negatively want something that you manage
yourself.  It's a battery drain.  It's hard to ensure that you don't
end up with a dead connection as a result of middlebox shenanigans.
Maintaining an open connection is either prohibited or strongly
discouraged on many operating systems for those reasons.

The messages that traverse those two paths are likely different.  Push
messages are considerably more limited in what they can carry.

> I presume this is referring to the Web Hook part of the push section of the
> spec, as this is the only place this word is mentioned. I'm not sure exactly
> what you mean by "client reachability isn't consistent", but the intention
> here (with a few header changes still required) is that the JMAP server is
> an RFC8030 compliant "Application Server".

I had no idea that that was your intention from reading the draft.  If
your intent was to act as an application server and actually use the
Push API, then it needs to be explicit.

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 10 Jul 2017 16:40:15 +1000

On Wed, 5 Jul 2017, at 18:16, Martin Thomson wrote:
> On 5 July 2017 at 14:57, Neil Jenkins <[email-masked]> wrote:>> If you have a suggestion of an existing authentication scheme that meets>> these criteria, we should look at that.
> 

Some interesting places that might require authentication are standalone applications like Thunderbird, and devices without a graphical interface, like routers or smart fridges that want to be able to send email or read email.  Of course a lot of these things now pair with a phone to allow setup, but we still need a way to give them a long lived token.
> You really need to motivate that more strongly than you have: account> creation is an exceptional process and covering that makes something

It's not account creation (which is not specified in JMAP at all) - but authenticating a new device against an existing account.  You're never going to get big providers to go back to allowing users to define their own password and store it in an arbitrary number of devices which get to authenticate using just that password.  There has been too much abuse with stolen passwords to go back to that as a way of connecting third party clients.  Apple just moved iCloud over to requiring device-specific passwords, Google has required OAUTH for IMAP for a while.
> The problem with the current document is that you spend 11ish pages on> defining a new authentication scheme.  That's a huge amount of work

Do you know of any other protocol which meets the goals?  I'd love to see the spec get shorter, but not at the expense of failing to solve the problems that a client to server protocol needs to solve in 2017 to be acceptable to providers.
> interoperable implementations.

Do you want me to log a specific bug in Github about the specification of TOTP scheme?  That's my role in this group!  If you could detail how it is insufficient that would be useful.
> groups, but that is where this is currently headed.  I would encourage> you to talk to the folks in the OAUTH and HTTPAUTH working groups to
> see if they have solutions for your problems.  There's a lot going on> there.

Definitely will do.  I'll be attending ace on Monday morning:

https://datatracker.ietf.org/group/ace/about/

And oauth on Tuesday afternoon:

https://datatracker.ietf.org/group/oauth/about/

If there's anything specific of value from there that you want to suggest as a workable alternative to the current authentication scheme, we'd love to see a counter proposal.
"Your current plan sucks" isn't very helpful without a workable alternative, because what's proposed has been implemented and tested and does work.
>> I explained the technical decisions behind this in the "Structure of a>> request/response" part of my email "JMAP Conceptual Decisions", a few weeks>> ago. If you missed it, a plain-text copy is available in the IETF archive at>> https://mailarchive.ietf.org/arch/msg/jmap/SfCe3UJPEEUdeomeso_-PAzP5tQ> 
> As an aside, I'm accepting for the moment that your performance goals> cannot be met by the simple use of HTTP that you discredit in that
> ping times to servers that many in this organization assume are close,> and I know very well the value of cutting round trips, but I'm not

If you watch the video of Apple's IMAP client vs FastMail's client using a protocol based on the same request batching process as JMAP from Australia to a server in New York, you can see the difference for real.  http://jmap.io/
> explored alternative options that would be less opaque to HTTP.  Have> you considered creating explicit dependencies between requests just in> your request bodies?  That would allow you to send the requests

As a server implementer, that's a really difficult scenario.  You wind up having a huge amount of state to track for an unbounded amount of time on the server.  And you lose the ability to do backreferences (http://jmap.io/spec-core.html#setFoos).  You're asking for a ton of extra complexity in the server.  That's a massive increase in scope compared to a bounded batch of requests in a single package.
> that resource (in this case, with the actions you wish to take), then> you execute the transaction.  This model has other virtues in that the> non-idempotent requests cause minimal side-effects and the final

I'm not sure how this reduces round trips, it looks like it's even more round trips than raw REST.  And again, this is asking the server to hold more state.  You won't find any server implementation begging for this. JMAP is deliberately as lightweight on state as possible for the server - all you need to maintain between requests (apart from the data model itself) is the long-lived authentication token.  Each batch of requests stands alone.
>> The EventSource part can easily be made to support long-polling instead at>> the option of the client (the only difference being it closes the connection>> after every push). Having a direct way for clients that are able to hold>> open connections get push notifications without requiring their own push>> server (see below) is still a good thing to have IMO.
> 

This is all a really good reason not follow the latest fashions, particularly in the HTTP space where things get created and thrown away all the time.  While JMAP is defined in terms of JSON and HTTP for ease of understanding, it's really just a structured object built on maps and lists over a batch transport.  You could implement it with ASN1 over UUCP just as easily.  This is a good thing, tying it to the shiny new work in progress is not a good way to make a long-lived protocol.
> That said, you actually have **two** goals here.  One is for an
> ongoing interactions between client and server.  For that, I generally> recommend long-polling, though that can lead to generic event streams,> which can easily become the locus of a huge pile of technical debt.
> 
> The other is the need to send messages when the client is effectively> offline.  For that, you negatively want something that you manage
> The messages that traverse those two paths are likely different.  Push> messages are considerably more limited in what they can carry.

The FastMail system  sends tiny edge triggers with no real content via either channel, telling the client that it needs to do a sync.  The protocol provides very efficient methods to resync from a previous known state, so there's no benefit to having two different ways to tell the client that there are updates.  It's much simpler to always use the same resync method, regardless of the change.  This has served us very well in production for years now, which is why Neil is confident proposing the same technique for JMAP.
Cheers,

Bron.

--
  Bron Gondwana, CEO, FastMail Pty Ltd
  [email-masked]

---

From: Bron Gondwana <[email-masked]>
Date: Wed, 26 Jul 2017 12:01:13 +1000

Hi Martin,

Reading back through this email, I think we've resolved Authentication by agreeing to split it into either a separate spec or just handwave a bit "the user will be authenticated somehow" and provide an endpoint at which the list of Accounts accessible to the currently authenticated user can be fetched.  Does that resolve your concerns there?
https://github.com/jmapio/jmap/issues/105

(also a task specific to seeing if OAUTH with automatic registration could work with JMAP)https://github.com/jmapio/jmap/issues/106

Regarding push, I think we also agreed to make the wording more clear, and add encryption support:
github #108-#110.

So the main area left that isn't resolved is the use of HTTP as a transport:
https://github.com/jmapio/jmap/issues/111

and optionally having two methods for accessing every object:
1) the JMAP-method style (which would also work over websockets or similar, and is batch friendly); and2) per-Object URLs and GET / PUT / PATCH / DELETE semantics (more round trips, but maybe easier for basic integrations that don't need updates, push, etc)
https://github.com/jmapio/jmap/issues/112

Have I missed anything that you think needs a separate issue to track it?
Thanks,

Bron.

On Wed, 5 Jul 2017, at 18:16, Martin Thomson wrote:
>> I presume this is referring to the Web Hook part of the push section of the>> spec, as this is the only place this word is mentioned. I'm not sure exactly>> what you mean by "client reachability isn't consistent", but the intention>> here (with a few header changes still required) is that the JMAP server is>> an RFC8030 compliant "Application Server".
> 
> I had no idea that that was your intention from reading the draft.  If> your intent was to act as an application server and actually use the
> _________________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

---

From: Martin Thomson <[email-masked]>
Date: Wed, 26 Jul 2017 14:50:02 +1000

I think that covers it.

On 26 July 2017 at 12:01, Bron Gondwana <[email-masked]> wrote:
> Hi Martin,
>
> Reading back through this email, I think we've resolved Authentication by
> agreeing to split it into either a separate spec or just handwave a bit "the
> user will be authenticated somehow" and provide an endpoint at which the
> list of Accounts accessible to the currently authenticated user can be
> fetched.  Does that resolve your concerns there?
>
> (also a task specific to seeing if OAUTH with automatic registration could
> work with JMAP)
> https://github.com/jmapio/jmap/issues/106
>
> 1) the JMAP-method style (which would also work over websockets or similar,
> and is batch friendly); and
> 2) per-Object URLs and GET / PUT / PATCH / DELETE semantics (more round
> trips, but maybe easier for basic integrations that don't need updates,
> push, etc)
>
> _______________________________________________
> Jmap mailing list
> [email-masked]
> _______________________________________________
> Jmap mailing list
> [email-masked]

---

From: Benoit Tellier <[email-masked]>
Date: Wed, 26 Jul 2017 15:58:36 +0700

Hi,

I just opened https://github.com/jmapio/jmap/pull/129

By the following proposal, I would like to submit a small addition to
the specification to allow text search in attachments.

Attachments are the only non-searchable field, and I can hardly see why
the specification should be restrictive on this.

You will notice the *may* about text extraction. It allows a default
implementation that just index attachments with *text/** making a
minimal implementation server side working in a question of minutes,
while still allowing more clever behavior.

Cheers,

Benoit