Subject: [Jmap] Offline / sync

From: Doug Royer <[email-masked]>
Date: Tue, 15 Nov 2016 21:12:47 -0700

On 11/15/2016 02:51 PM, Ted Lemon wrote:
> If both end maintain a versioned catalog, updating the catalog becomes a
> process of agreeing on a most recent common version, exchanging diffs
> from that, then fetching new messages or message metadata.
>
> This is a lot less chatty than IMAP.

The authoritative key is the message headers, not how the client or 
server remember the messages.

When a mailbox is opened, fetch the headers. The headers are not 
editable and do not change once received in the mailbox, compare them.

If there are some new messages, fetch what ever part of those new 
messages you want offline.

If there are missing ones, an authority already said "make this message 
go away". So delete it from the client.

Some MUAs tweak the message headers to indicate the message is read. 
Stop that and use some other handshake or other flag. That's not a 
offline issue, its a I read that message problem.

The old Netscape communicator tweaked the received time. If the unit 
digits of the time was even, it was an unread message. An odd number, 
the message had been read.

Header comparison on a shared 1MB (speed) computer was not an efficient 
method on shared 56K Internets in the early 80's. Message 'numbers' are 
an artifact of '/bin/mail[x]', the c-client derivatives, pine, ...

Its a client presentation issue in those old clients. IMAP/POP clients 
used them to number messages because a human had to type get me message 
XX by hand.  IMAP and POP protocols were an extension (almost a 
shorthand) of what the user typed on the keyboard of those shared slow 
computers from a DUMB terminal. Old school and unnecessary.

If a client wants to display message by number, that's fine. However it 
really has nothing to do with the 'key' (the headers) to the message.

If some server tweaks the headers, then how will any program ever know 
that its tweaked? It can't. It will look like the old one is deleted, 
and the new one arrived. Delete it, fetch the newly tweaked message.

Messages should not be edited or tweaked in the store.

There is much text in IMAP protocols that describe fetching some, or a 
select set of headers. Not needed any more. Get them all. its fast and a 
much better key and sync solution.

  ---Mobile---

I have a separate email account for my mobile device, because I do not 
want it downloading the tens of thousands of messages I have collected 
over the decades.

I have never met anyone that wanted to sync tens of thousands of emails 
to their mobile device. However they can.

The sync could be per server. Or could be per folder. Ether way, the 
message headers are the key and its a client issue.

--

Doug Royer - (http://DougRoyer.US)
[email-masked]
714-989-6135

---

From: Ted Lemon <[email-masked]>
Date: Wed, 16 Nov 2016 13:30:51 +0900

Doug, the fact that you have a separate email account for your mobile
device is _because_ IMAP requires you download all the headers.   IMAP
is perfectly capable of keeping a subset of all message bodies.   But
it's this precise issue that I'm talking about when I say that the way
IMAP solves this problem isn't good for constrained devices and links.
  Even my laptop can burn through an amazing amount of bandwidth
syncing when there are only two new email messages.

On Wed, Nov 16, 2016 at 1:12 PM, Doug Royer <[email-masked]> wrote:
> The authoritative key is the message headers, not how the client or server
> remember the messages.
>
> When a mailbox is opened, fetch the headers. The headers are not editable
> and do not change once received in the mailbox, compare them.
>
> If there are some new messages, fetch what ever part of those new messages
> you want offline.
>
> If there are missing ones, an authority already said "make this message go
> away". So delete it from the client.
>
> Some MUAs tweak the message headers to indicate the message is read. Stop
> that and use some other handshake or other flag. That's not a offline issue,
> its a I read that message problem.
>
> The old Netscape communicator tweaked the received time. If the unit digits
> of the time was even, it was an unread message. An odd number, the message
> had been read.
>
> method on shared 56K Internets in the early 80's. Message 'numbers' are an
> artifact of '/bin/mail[x]', the c-client derivatives, pine, ...
>
> Its a client presentation issue in those old clients. IMAP/POP clients used
> them to number messages because a human had to type get me message XX by
> hand.  IMAP and POP protocols were an extension (almost a shorthand) of what
> the user typed on the keyboard of those shared slow computers from a DUMB
> terminal. Old school and unnecessary.
>
> If some server tweaks the headers, then how will any program ever know that
> its tweaked? It can't. It will look like the old one is deleted, and the new
> one arrived. Delete it, fetch the newly tweaked message.
>
> I have a separate email account for my mobile device, because I do not want
> it downloading the tens of thousands of messages I have collected over the
> decades.
>
> I have never met anyone that wanted to sync tens of thousands of emails to
> their mobile device. However they can.
>
> The sync could be per server. Or could be per folder. Ether way, the message
> headers are the key and its a client issue.
>
> --
>
> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap
>

---

From: Doug Royer <[email-masked]>
Date: Wed, 16 Nov 2016 12:03:20 -0700

On 11/15/2016 09:30 PM, Ted Lemon wrote:
> device is _because_ IMAP requires you download all the headers.

Maybe I am confused with some if the terms. So for this email:

Offline - The MUA wants the entire MIME message at the MUA end.

Sync - I am not doing offline storage, so a protocol is needed to update 
each end with asynchronous changes at the other end.

I have a separate phone email because I don't want GIGs of email on my 
phone. The Android phone apps offline the entire account, not just named 
folders, it gets the headers and all of the messages. A flaw in the MUA 
in my opinion.

If you do not want offline storage then its a how do I get only the new 
display information debate, or selective messages - Update headers sync, 
and download. Sync.

If your talking about accessing an IMAP account from a laptop while your 
phone is also connected to the same IMAP account, and have the active 
one update the data in the passive one, then that is an update debate - 
Sync.

Offline is simple. Its a copy what ever is new. If they have the same 
file format an MUA could use rsync and not IMAP. A tweaked rsync 
protocol for offline email may be a better solution. Maybe we could 
define a virtual folder format so rsync could be used for offline.

If you going to have messages for offline usage, your going to get all 
of the headers and content anyway.

When doing offline, the server should just say. Here is a new message, 
its ZZ octets in length, and here it is, in raw over the wire MIME 
format. (perhaps compressed over the wire). Because if the folder is 
being stored offline, then the MUA has already said it wants all of the 
headers and content. So no chatter needed.

> IMAP is perfectly capable of keeping a subset of all message bodies.   But

IMAP - yes. And if the messages are large, not just because of the 
headers. A few times I have tried to debug an IMAP session and it was 
amazing how much back and forth was sent, most of which seemed 
unnecessary. Mostly I suspected a badly written implementation.

And yes, non-offline sync is more complex.

--

---

From: Ted Lemon <[email-masked]>
Date: Thu, 17 Nov 2016 05:43:19 +0900

Doug, I don't know if I'm being redundant here or if clarification is
needed, so I'll do my best to clarify, but if this still doesn't make
sense to you, we should probably drop it rather than iterating again.

First, it is a _bug_ that the MUA will only download the complete set
of messages.   It's also not true on my android phone--I have an imap
setup, and the phone doesn't download gigabytes of messages.

That said, the amount of metadata that needs to be synced to be sure
that you have a correct set of metadata is gigantic, and there is no
mechanism for eliminating redundant re-downloading of metadata.
Doing that is not hard--it just requires that the dataset be
versioned.   When you download the dataset the first time, the server
tells you what version you have.   When you connect later after having
been offline, you tell the server what version you have, and it sends
you the changes, not the whole dataset.  If you have made changes, you
send your changes.   If there are conflicts, there's a conflict
resolution process.

As a consequence, if 100 messages have arrived since you were last
connected, the maximum amount of metadata the server should ever send
you is the metadata for 100 messages, even if you have 100k messages
in your mail store.

Right now, my IMAP client on my Mac routinely downloads the entire
100k messages worth of metadata just to make sure that it has a
correct picture of what's in the mail store, because (as far as I
know, not an IMAP expert) there is no versioning process like I've
described.

So that's what I'm arguing about.   I don't know if it makes sense to
you, or if it's an operational issue that anybody here considers worth
talking about.   It's definitely derailing a bit; I only mentioned it
because the topic came up in the chartering discussion.

On Thu, Nov 17, 2016 at 4:03 AM, Doug Royer <[email-masked]> wrote:
> folders, it gets the headers and all of the messages. A flaw in the MUA in
> my opinion.
>
> display information debate, or selective messages - Update headers sync, and
> download. Sync.
>
> phone is also connected to the same IMAP account, and have the active one
> update the data in the passive one, then that is an update debate - Sync.
>
> Offline is simple. Its a copy what ever is new. If they have the same file
> format an MUA could use rsync and not IMAP. A tweaked rsync protocol for
> offline email may be a better solution. Maybe we could define a virtual
> folder format so rsync could be used for offline.
>
> If you going to have messages for offline usage, your going to get all of
> the headers and content anyway.
>
> When doing offline, the server should just say. Here is a new message, its
> ZZ octets in length, and here it is, in raw over the wire MIME format.
> (perhaps compressed over the wire). Because if the folder is being stored
> offline, then the MUA has already said it wants all of the headers and
> content. So no chatter needed.
>
> IMAP - yes. And if the messages are large, not just because of the headers.
> A few times I have tried to debug an IMAP session and it was amazing how
> much back and forth was sent, most of which seemed unnecessary. Mostly I
> suspected a badly written implementation.
>

---

From: Stu Brandt <[email-masked]>
Date: Wed, 16 Nov 2016 17:47:45 -0500

Does rfc7162 cover your point?  I believe JMAP attempts to cover this 
concept via the sinceState argument.

Cheers
- Stuart

---

From: Ted Lemon <[email-masked]>
Date: Thu, 17 Nov 2016 07:51:42 +0900

I don't think it's using the right data model.   The failure mode for
not completing the transfer of a changeset should be that the partial
changeset is discarded.   Otherwise I can't see how you can be sure
that the client has an accurate version of the mailbox state.   Of
course, I also think that mailboxes should be a separate object from
the mail store, but that's a complete departure from the IMAP model.

On Thu, Nov 17, 2016 at 7:47 AM, Stu Brandt <[email-masked]> wrote:
>>> in
>>> and
>>> file
>>> its
>>> headers.

---

From: Neil Jenkins <[email-masked]>
Date: Thu, 17 Nov 2016 11:03:49 +1100

On Wed, 16 Nov 2016, at 03:30 PM, Ted Lemon wrote:
> device is __because__ IMAP requires you download all the headers.

Right! JMAP is explicitly designed to fix this problem.

JMAP was designed with the idea that you would have a canonical data
store on the server, and all clients would want to stay in sync with
this (as opposed to POP, where different clients may fetch from the
server but are entirely independent other than this). This is how all
modern email clients work and what most people now expect. However, it
also recognises that the data set may be huge, and you may not want or
be able to download it all locally.

There are a number of levels of caching/offline support a client may
use, and JMAP has facilities to work efficiently with all of them.
(Note, I'm using the term "message list" below to describe a sorted,
filtered list of message ids, e.g. the ordered list of messages in
your inbox.)

1. Empty cache

JMAP provides methods and data structures to go from nothing to full
display of your inbox list (most email clients default view) within two
round trips and with minimal data transfer.

This is great both for webmail and first setup of any MUA, as you can
start using your email with a new client much more quickly than with
IMAP. The data structures are also such that you can present the same
view *as if you had the entire (possibly 100+ GB) dataset loaded
locally*: you know the counts in the folders, how long your message list
is (the server does sorting, thread collapsing for you), and the thread
information for messages in view.

You can jump to any part of the message list and load in just that
section, allowing the client to provide full access to your mail without
having to download it all, providing it has connectivity.

2. Disconnected partial sync

This is most common for mobile apps, but could be any app: you want to
make some of the data available offline, but not download the entire
(possibly huge) archive. With JMAP, you can easily download just a
subsection (say, first 100 ids) of the message lists for the mailboxes
you want to make available offline, and then download the message
contents themselves. I would expect most clients to get the easier and
more compact parsed representation of the message, but they could fetch
the raw RFC5322 if they prefer.

JMAP uses state strings for each data type and has commands to make it
possible to resync efficiently with the server when you come back
online. The server can tell the client exactly which messages are
new/changed or have been deleted since the last sync. Additionally, you
can sync changes to the message lists efficiently too: the server can
tell you want to insert and what to remove, without having to redownload
the full list of ids.

3. Disconnected full sync

This is really an easier case than (2) since you could just do (2) but
not limit which mailboxes and how many messages you cached. Once you
have your initial sync of everything on the server, you can just use
getMessageUpdates to very efficiently get the changes since last time.

---------

Despite living in an ever more connected world, offline support is still
very important, and while I've just given a brief overview rather than
technical deep dive, I hope this conveys that we've really thought about
the different scenarios in which this will be used, and tried to make it
as efficient as possible in each of them; certainly a significant leap
forward from IMAP.

Neil.

---

From: Doug Royer <[email-masked]>
Date: Wed, 16 Nov 2016 18:42:18 -0700

> setup, and the phone doesn't download gigabytes of messages.

I am not trying to debate, I am guessing we are not talking the same jargon.

When I set my original email account on my android phone to store-for 
OFFLINE reading mode (checked SYNC GMAIL in the gmail application), it 
filled up my android, then crashed. I had to factory reset my phone.

Then I created a 2nd gmail account to use as the phone primary email 
address.

I then added my original gmail account to my phone, and did NOT select 
store for OFFLINE (unchecked SYNC GMAIL in the gmail application), and I 
can read my original gmail account, and it does not copy the mailbox to 
the phone. And does not have those messages available for offline reading.

My new email is in offline for reading mode (gmail app calls it 'sync 
gmail').

My old email is in online only reading mode.

The 'official gmail' app: I have no idea if it uses IMAP, POP, or 
something they created. I do know if you set it for offline reading, it 
copied my entire (GIGs) account (all folders, all mailboxes for that 
account).

With Seamonkey (Spin of Thunderbird). I have "Keep messages for this 
account in this computer" checked (offline reading mode). It downloaded 
all messages with I first initialized it. It took more than a day. And 
the messages are stored in my home Linux box, AND on the gmail server.

Looking at the amount of traffic when I start seamonkey, it does not 
seem to download all headers for my gmail IMAP account. So, I am 
guessing the apple app your talking about is a inefficient implementation.

--

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 18 Nov 2016 07:15:10 +0900

Ted,

> On 17 Nov 2016, at 05:43, Ted Lemon <[email-masked]> wrote:
> 

This is covered by IMAP CONDSTORE and QRESYNC extensions. So either your server and client don't support these, or the client is badly written.

Best Regards,
Alexey

---

From: Ted Lemon <[email-masked]>
Date: Fri, 18 Nov 2016 07:02:43 +0900

Alexiy, I think the problem with CONDSTORE and QRESYNC is that the
data model isn't good, and so you can't really feel secure that you
actually have a correct picture of the whole mail store.   So every so
often you do a massive resync.   It doesn't happen every time you
check for mail, but with a better data model, it would only happen if
there were no revisions of the metadata database in common between the
two ends.

---

From: Neil Jhaveri <[email-masked]>
Date: Thu, 17 Nov 2016 22:51:23 +0000

> On Nov 17, 2016, at 2:02 PM, Ted Lemon <[email-masked]> wrote:
> 

Out of curiosity, could you elaborate on some of these issues?

Apple Mail supports both CONDSTORE and QRESYNC and, in our experience, they have worked well and been implemented well enough by servers. We admittedly do have some fallback paths to resynchronize all UIDs and flags in specific cases (e.g. syncing without CONDSTORE/QRESYNC), as we have seen some bugs where we believed the servers had not computed history exactly correctly. But we believed these to just be implementation bugs, and have encountered similar problems with Exchange Web Services, too.

Best,
- Neil

---

From: Ted Lemon <[email-masked]>
Date: Fri, 18 Nov 2016 09:01:49 +0900

Hm, okay, that's useful information.   My main complaint about the
apple client is that if it detects a blank mail store on the server,
it will erase five gigabytes of mail on the client.   But aside from
that I think I am seeing the behavior you are describing--it uses
CONDSTORE/QRESYNC most of the time, but every so often there's a
massive resync, which, if it happens over my cell phone connection,
will kill the battery.

And also if I am syncing over a lossy link, the chattiness of the
protocol means that it takes a terribly long time to sync up even when
using CONDSTORE/QRESYNC.

---

From: Doug Royer <[email-masked]>
Date: Tue, 22 Nov 2016 12:49:02 -0700

On 11/21/2016 10:38 PM, Brandon Long wrote (on ietf-smtp, imapext):

> I've said this to some client developers directly, but it's good to
> share the concept more widely:
>
> There should be no "full mirror" clients, only disconnected state
> clients which make different choices of window size.  There are Gmail
> accounts today which would exceed the ssd in many laptops, or the
> monthly bandwidth allotment of some broadband customers.  A full sync is
> useful for backups, but a regular client would be wise to make full sync
> a special case of "reasonable cache space on this device/connection
> happens to be larger than the user's account".

If your saying it should not be a requirement to mirror you account - I 
agree.

If that is not what you are saying:

The decision to mirror (offline) or not mirror your email, is a user 
need, not a protocol design issue. A protocol exists to make what is 
needed a standard. A protocol does not design user needs.

I need it, others over the years have said they need it.

And FYI, I offline my main mega-sized GMAIL account now by telling my 
MUA to keep email for offline usage. Gmail already supports it.

--

---

From: Doug Royer <[email-masked]>
Date: Tue, 22 Nov 2016 12:55:58 -0700

On 11/15/2016 05:10 AM, Tony Finch wrote (on imapext and ietf-smtp):

> Is everyone here using the term "offline mode" to mean the same thing?

No they are not.

> ...
> How much do the necessary protocol features differ between a full-fat
> disconnected mode client (a mirror on the client of the entire
> account on the server) vs a poorly-connected mobile client which needs to
> sync a smaller window on the account? Concurrent access to multiple
> mailboxes? what else?

For those that need a full copy (mirror), it could be a MUCH simpler 
protocol. Like I said earlier, you could almost use rsync.

For those that just want a fast cache of emails, that is not a mirror 
and is a much more difficult protocol.

Some POP accounts still mandate (policy) the copying of the email. Once 
you pop it, its deleted, or will be soon. If you need to keep it, you 
have to copy it. Not all email accounts are ISP's. Not all email 
providers want to be your store for your email.

--

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 23 Nov 2016 13:31:54 +1100

On 11/21/2016 10:38 PM, Brandon Long wrote (on ietf-smtp, imapext):

> clients which make different choices of window size.

I completely agree. And this is the case  JMAP is most optimised for.
Depending on the device, the size of the user's account, and the user's
preference, the client may choose a cache size such that the whole
account is mirrored. But this is just a special case of the broader
partial cache case.

Even in the full sync case, depending on bandwidth limitations it can
take a while to get to that point when adding a new account, and in the
meantime you want the user to be able to fully access their mail as
*though* it were all local, providing they are currently online. This is
again the partial cache situation that JMAP works really well for.

Neil.