Subject: [dispatch] Potential incoming work: web push for IMAP, CalDAV and CardDAV

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 23 Oct 2019 15:08:34 +1100

There's a couple of interesting things that have been going on in different places that I'm working on trying to get aligned and to find a home for at the IETF:

Web push for IMAP servers here:

https://github.com/coi-dev/coi-specs/blob/master/webpush-spec.md

Web push for CalDAV and CardDAV servers here:

https://datatracker.ietf.org/doc/draft-gajda-dav-push/

(there's a more recent version in the CalConnect private repositories - we haven't been pushing it just yet because the CALEXT group is already pretty busy, and we're not sure if it belongs there or in another group).

And of course JMAP already has a web push built in with the push subscription object:

https://tools.ietf.org/html/rfc8620#section-7

Anyway, I'm not sure if this is ripe for discussion at Singapore, or indeed if we've missed the deadlines for a BOF, but I figured if I post here I might see if there's any others working on similar things and if we could pull everyone together to join forces and make sure all the different push mechanisms work similarly.

Cheers,

Bron.

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 [email-masked]

---

From: Michael Toomim <[email-masked]>
Date: Tue, 22 Oct 2019 22:27:41 -0700

Hi Bron! The work on the Braid HTTP extension also provides a "web push". We are also trying to find a general approach.

The current design allows clients to specify a "Subscribe" header in the GET request, which tells the server to keep the connection open and stream all updates to the resource, as they happen. We will be publishing a new draft within the next week or two.

> On Oct 22, 2019, at 9:08 PM, Bron Gondwana <[email-masked]> wrote:
> 
> https://github.com/coi-dev/coi-specs/blob/master/webpush-spec.md <https://github.com/coi-dev/coi-specs/blob/master/webpush-spec.md>
> 
> https://datatracker.ietf.org/doc/draft-gajda-dav-push/ <https://datatracker.ietf.org/doc/draft-gajda-dav-push/>
> 
> https://tools.ietf.org/html/rfc8620#section-7 <https://tools.ietf.org/html/rfc8620#section-7>
> 
>   [email-masked] <mailto:[email-masked]>
> 
> 
> _______________________________________________
> dispatch mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/dispatch <https://www.ietf.org/mailman/listinfo/dispatch>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Oct 2019 08:50:39 +0200

On 23.10.2019 07:27, Michael Toomim wrote:
> Hi Bron! The work on the Braid HTTP extension also provides a "web
> push". We are also trying to find a general approach.
>
> The current design allows clients to specify a "Subscribe" header in the
> GET request, which tells the server to keep the connection open and
> stream all updates to the resource, as they happen. We will be
> publishing a new draft within the next week or two.
> ...

Why would you need a new header field for that?

Best regards, Julian

---

From: Michael Toomim <[email-masked]>
Date: Wed, 23 Oct 2019 14:22:19 -0700

Julian:
> Mike:
>> stream all updates to the resource, as they happen.
> 

The header is just one way to do it.  What we *need* is some way for the client to signal to the server that it wants to subscribe to changes.  Right now, a header on GET seems to be the most elegant way to signal that.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Oct 2019 05:32:14 +0200

> The header is just one way to do it.  What we *need* is some way for the client to signal to the server that it wants to subscribe to changes.  Right now, a header on GET seems to be the most elega…

My gut feeling is that you should define a media type that handles this
kind of continueing-to-update content, and then just use Accept: to tell
the server...

Best regards, Julian

---

From: Michael Toomim <[email-masked]>
Date: Wed, 23 Oct 2019 21:24:16 -0700

> kind of continueing-to-update content, and then just use Accept: to tell

Thank you, Julian!  Our spec works for any media type.  You can apply it to existing HTML, JSON, or images.  It allows CDNs to get live updates via patches, of the existing content they distribute.  CDNs will be able to support dynamic content, not just static content.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Oct 2019 08:32:43 +0200

> Thank you, Julian!  Our spec works for any media type.  You can apply it to existing HTML, JSON, or images.  It allows CDNs to get live updates via patches, of the existing content they distribute.…

Where's the spec?

Best regards, Julian

---

From: Michael Toomim <[email-masked]>
Date: Thu, 24 Oct 2019 01:38:27 -0700

>> Michael:

We're still editing it for publication, but our working area is here:

   https://github.com/braid-work/braid-spec

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Oct 2019 10:47:45 +0200

>     https://github.com/braid-work/braid-spec

Hmm. I think you'll need to work on the proper labeling of things. Even
if the client opts into this format, the Content-Type will still have to
describe the overall payload.

You're basically inventing a new payload format that can wrap multiple
payloads. But then we already have multipart/*; it really seems to me
you should be using that instead.

Best regards, Julian

---

From: Michael Toomim <[email-masked]>
Date: Thu, 24 Oct 2019 02:34:07 -0700

Thank you very much for your early comments!

> describe the overall payload.

Hm, I think I see where you're coming from.  Here's the example in my mind:

Request:
     GET /animated-braid.gif
     Subscribe

Response:
     OK 200
     Subscribe
     Cache-Control: no-cache
     Patch-Type: braid-patch(bytes, sync9(array))
     ---- Part 1
     Content-Type: image/gif
     Patch-Type: braid-patch(bytes, sync9(array))
     Content-Length: 1239

     [100:200] = <binary data>
     ---- Part 2
     Content-Type: image/gif
     Patch-Type: braid-patch(bytes, sync9(array))
     Content-Length: 62638

     [348:887] = <binary data>

I think you're saying that the Content-Type of each part should need to describe the BODY of each part.  Why is that necessary?  I would imagine that it would only impact a cache, but we rule out caches with no-cache unless they understand the patch format.

It seems to me that the Content-Type *could* conceivably describe the semantic content of the resource, even though its updates are described with a particular Patch-Type.  Similar to how Range Requests don't change the Content-Type, even though they are only sending a small patch of the object in the BODY.

> you should be using that instead.

I'm open to using multipart/*.  Are there any advantages to it?  It requires the sender to come up with a unique separator string, which means they need to scan all messages before choosing the separator to wrap them with.  It seems simpler to just use Content-Length: N to communicate how to break apart multiple messages, and as a bonus, you can stream the data and process just one message at a time.

Thank you very much for your early comments!

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Oct 2019 11:45:43 +0200

> I think you're saying that the Content-Type of each part should need to describe the BODY of each part.  Why is that necessary?  I would imagine that it would only impact a cache, but we rule out c…

Not necessarily every part...

> Thank you very much for your early comments!

And the end of the day, this is still HTTP. The response needs a media
type that describes the payload. You are defining a new payload format
that can handle multiple resources, so you need a container-like format.
multipart can do that, but I agree the mail format legacy with the
boundary string is sub optimal.

But then, sending Content-Length inside the payload is a bit weird as
well; it defeats streaming the part; you may want to consider a binary
container format instead (based on HTTP chunked, for instance).

Best regards, Julian

---

From: Michael Toomim <[email-masked]>
Date: Thu, 24 Oct 2019 02:55:38 -0700

> boundary string is sub optimal.

Ah, yes-- that *is* what we are doing.  Thank you for articulating it nicely!

> container format instead (based on HTTP chunked, for instance).

Ah, yes.  Well, you can still stream updates here by breaking up a single change into multiple patches.  Then each patch can occupy a different message.  You can make them as small as you'd like; you'll only face the overload of an extra frame header in http/2.

I'm cc'ing the braid list, in case they are interested. [Thread link] <https://mailarchive.ietf.org/arch/msg/dispatch/gU2cWDGgxc-1O1xAubdIxNeeo58>

---

From: Ranjit Avasarala <[email-masked]>
Date: Thu, 24 Oct 2019 07:32:24 -0500

why can't we use WebSockets which is based on HTTP and helps keep session
active?

On Thu, Oct 24, 2019 at 4:55 AM Michael Toomim <[email-masked]> wrote:

> Ah, yes.  Well, you can still stream updates here by breaking up a single
> change into multiple patches.  Then each patch can occupy a different
> message.  You can make them as small as you'd like; you'll only face the
> overload of an extra frame header in http/2.
>
> I'm cc'ing the braid list, in case they are interested. [Thread link]
> <https://mailarchive.ietf.org/arch/msg/dispatch/gU2cWDGgxc-1O1xAubdIxNeeo58>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch
>

---

From: Michael Toomim <[email-masked]>
Date: Thu, 24 Oct 2019 12:40:03 -0700

Ranjit:
> why can't we use WebSockets which is based on HTTP and helps keep session active? 

Thank you for the question. We certainly could use a WebSocket, and in fact, the previous version of the spec <https://tools.ietf.org/html/draft-toomim-braid-00> did use a WebSocket as transport! The question is why you'd prefer to use a WebSocket as the underlying transport, rather than a multipart-body GET request. Since we still want all the great things from HTTP— caching, URLs, idempotency, content-types, etc. —we'd either have to implement all the existing HTTP methods on top of a WebSocket, or add these new subscription features on top of HTTP. In the end, it seems to be simpler and more elegant to do the latter.

Also, the introduction mentions WebSockets—did it address your question adequately? Or is there something we should add to this:

1.  Introduction

  When a client GETs a resource over HTTP, the server provides the
  latest version, but does *not* provide updates when the resource
  changes.  This means that programmers have to implement workarounds
  whenever they need to synchronize with data that changes.

  The web originally required users to click reload when a page changed.
  In the year 2000, XMLHTTPRequest made it possible to update just part
  of a page, running a GET request behind the scenes.  But the GET
  request still could not push updates.  To work around this, web
  programmers would poll the resource, which was inefficient.
  Long-polling was invented to overcome the inefficiencies, which was
  standardized as SSE.  But SSE provides semantics of an *event-stream*,
  and although a programmer can encode a protocol within the event
  stream for updating a resource, there is still no standard way to
  update a resource.  In practice, programmers today often give up on
  using a standard semantic for "data that changes", and instead write
  their own idiosyncratic update protocol over a WebSocket, or adopt a
  web framework that does this for them.

  However, we can add support for "state that changes" into HTTP with a
  few small extensions.  Doing so transforms HTTP from a *state
  transfer* protocol into a *state synchronization* protocol, and
  extends the REST architecture into RESS: REpresentational State
  Synchronization.
https://raw.githubusercontent.com/braid-work/braid-spec/master/draft-xx-httpbis-braid-00.txt <https://raw.githubusercontent.com/braid-work/braid-spec/master/draft-xx-httpbis-braid-00.txt>
But let me say that some things actually are more elegant to implement with a WebSocket. Specifically, we are finding the request/response model of HTTP to be getting in the way. After this spec is complete, I expect us to propose a generalization of request/response that can run peer-to-peer over a QUIC channel, in which either peer can initiate or respond to GET and PUT requests, and both peers will ACK changes in state symmetrically. We sketched this in Section 4.2 of the the previous spec <https://tools.ietf.org/html/draft-toomim-braid-00#section-4.2>. It makes the protocol both simpler and more capable.

Thank you, Ranjit!

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 25 Oct 2019 13:44:25 +1100

It looks like the conversation went a long way down this path, which by my reading is "ask the server to maintain a long running connection and keep pushing updates down it" which is a slightly different use-case from the push proposals I was making, which are:

 * Client establishes a relationship with a third party push server (potentially via some platform specific, highly efficient protocol - e.g. APNs <https://en.wikipedia.org/wiki/Apple_Push_Notification_service> or GCM <https://en.wikipedia.org/wiki/Google_Cloud_Messaging>)
 * Client requests server inform the third party push server when there are changes.
 * Third party push server passes details down to client, which can then request updates from the server without having to hold a connection open or poll.

This is a general pattern which 1st party clients frequently use (including ours at Fastmail) but which doesn't have a general solution, so my impression of the work required there is defining a standard for how to request the server make that push to a third party, and what gets included in that push.

This other work of having the server hold a connection open is also interesting and potentially useful, it's just not exactly what I was suggesting!

Cheers,

Bron.

---

From: Michael Toomim <[email-masked]>
Date: Fri, 25 Oct 2019 13:14:21 -0700

Bron Gondwana:
> Client establishes a relationship with a third party push server (potentially via some platform specific, highly efficient protocol - e.g. APNs <https://en.wikipedia.org/wiki/Apple_Push_Notification_service> or GCM <https://en.wikipedia.org/wiki/Google_Cloud_Messaging>)
> Client requests server inform the third party push server when there are changes.
> Third party push server passes details down to client, which can then request updates from the server without having to hold a connection open or poll.
> 

Hi Bron, thank you for articulating this distinction, it gave me a bunch to chew on!

However, after some thought, I don't think we are actually that far off. Consider that JMAP supports both of these methods: you can subscribe to a stream of updates via a long-running GET request (section 7.3 <https://tools.ietf.org/html/rfc8620#section-7.3>) or via a third-party server (section 7.2 <https://tools.ietf.org/html/rfc8620#section-7.2>). Braid should eventually support both as well.

The choice between whether you want to use a long-running TCP connection vs. a third-party server mostly comes down to whether you're a cell phone— it saves radio power if you don't have to maintain a TCP connection to each data source you care about. But if you're a server, or you're a cell phone that is already online, it's better to connect directly.

Now, we could certainly go about defining separate web push standards for these two cases. But I think there's a good case to be made for putting them together in a single standard, with a single API for programmers to use.

I think we can gain a lot by putting this standard into basic HTTP, rather than building layers on top. By adding Synchronization to HTTP, we get the functionality of Web Push, but it can be implemented automatically, for all resources, without any extra effort from the programmer. The whole web has changed in the last 20 years— from static pages to dynamic data —and it's time we upgrade the protocol at a deep level. Getting a notification of a message on your phone shouldn't be any different from getting the actual message in the app.

---

From: "Bron Gondwana" <[email-masked]>
Date: Sat, 26 Oct 2019 10:00:31 +1100

On Sat, Oct 26, 2019, at 07:14, Michael Toomim wrote:
> I think we can gain a lot by putting this standard *into* basic HTTP, rather than building layers on top. By adding Synchronization to HTTP, we get the functionality of Web Push, but it can be implemented automatically, for all resources, without any extra effort from the programmer. The whole web has changed in the last 20 years— from static pages to dynamic data —and it's time we upgrade the protocol at a deep level. Getting a notification of a message on your phone shouldn't be any different from getting the actual message in the app.

At this point the big difference is whether you want a push of the entire resource, or (as JMAP does) an edge trigger to say "there's something new of this object type on the server - issue a query about which bits you're interested in to see if there's anything new you want to fetch".

At which point the granularity of detail becomes interesting - because you only want to be informed of changes if there's a high probability that they are things you are interested in, and even more so - if there's a high probability that you'll be able to see them! You don't want a push every time a resource on a server with a million users changes if you can only see your own resources!

We do that on our own server with pushing you every change to you own resources plus doing ACL filtering to see if any other user can see the resources and giving them an ACL push too, even if they aren't subscribed to the resources and won't notice a change.... so there could be more smarts in the push filter but it seems to balance out OK in a real life, just the occasional extra wake. And we separate data types like email and calendar into separate change pools.

Bron.

---

From: Michael Toomim <[email-masked]>
Date: Fri, 25 Oct 2019 16:18:12 -0700

> We do that on our own server with pushing you every change to you own resources plus doing ACL filtering to see if any other user can see the resources and giving them an ACL push too, even if they…

Ah, yes the JMAP approach is to inform the client of EVERY change to EVERY object, even if that object isn't being displayed right now. The client just ignores the unnecessary changes.

That works well enough for email, where all changes are bounded to the user's mailbox, and things just don't change that often. However, it doesn't scale to the general case, where a server might be serving millions of resources, and any client could conceivably want to watch any of them, but probably only a few at a time.

It is much more efficient here for a client to explicitly subscribe to the resources it wants to watch. In Braid, you subscribe to a resource by adding the Subscribe header to a GET. Then the server promises to send you updates, even if you go offline, until you explicitly give it a FORGET.

This means that the granularity of subscription is a URI, and clients are responsible for expressing which URIs they want to subscribe to. This works quite well in practice. You usually start out with just a few URIs, which means your client will do a few GETs, and then if you need finer granularity, you can always break up your objects into multiple URIs. The Linked JSON spec <https://raw.githubusercontent.com/braid-work/braid-spec/master/draft-xx-httpbis-linked-json-00.txt> makes that easy.