Subject: [Jmap] Hacker News Discussion - getMailboxes single data set vs tree primitives or paging

From: Bron Gondwana <[email-masked]>
Date: Sun, 07 May 2017 21:06:53 +1000

https://news.ycombinator.com/item?id=14284526

It's a rather silly discussion that went off onto a tangent, that's HN for you.  The key interesting point is this:

A quick look at the protocol looks like in order to get a list of mailboxes, you have to ask for them all. In IMAP, you can say, show me top level mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous role, I was responsible for working on a mail archiving solution which had a large hierarchy of folders (into the thousands IIRC) in a single account. Access to the archive was via IMAP. Looks to me that JMAP here would require the client to potentially fetch megabytes of data just to start showing a list of folders, where the solution we had with IMAP probably took a couple of kilobytes at most to get started.

...

and the followup:

> Of course nothing stops you storing thousands of mailboxes in a non-heirarchical layout and making IMAP have large LIST fetches too.

Yes. I would issue the LIST command, and as the IMAP server started streaming the results to the client, I would immediately process each mailbox as soon as it was down, rather than waiting for them all to be retrieved. I fear that with a JSON based protocol, most server and client implementations wouldn't deal with streaming (although they technically could), and would just deal with completed blobs.

...

I think it is worth discussing the tradeoffs in terms of additional complexity to offer paging, tree filtering (parentIds: [...]) or both in the standard.  The pro side, you can access potentially millions of mailboxes without needing to know all the folder ids up front.  On the con side, it's more complexity for the server (client can always just fetch everything anyway if it doesn't care) - and most clients seem to fetch the entire tree anyway, so it just means more round trips for them.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Mike Cardwell <[email-masked]>
Date: Sun, 7 May 2017 13:39:12 +0100

> It's a rather silly discussion that went off onto a tangent, that's HN for
> you.

I am the person who is responsible for that silly discussion and also ironically
the person who said he wasn't invested in the success of JMAP enough to bring it
to this very list. Well, here I am.

> Me:
>  A quick look at the protocol looks like in order to get a list of mailboxes,
>  you have to ask for them all. In IMAP, you can say, show me top level
>  mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous
>  role, I was responsible for working on a mail archiving solution which had a
>  large hierarchy of folders (into the thousands IIRC) in a single account.
>  Access to the archive was via IMAP. Looks to me that JMAP here would require
>  the client to potentially fetch megabytes of data just to start showing a
>  list of folders, where the solution we had with IMAP probably took a couple
>  of kilobytes at most to get started.

> Bron:
>  Of course nothing stops you storing thousands of mailboxes in a
>  non-heirarchical layout and making IMAP have large LIST fetches too.

> Me:
>  Yes. I would issue the LIST command, and as the IMAP server started
>  streaming the results to the client, I would immediately process each mailbox
>  as soon as it was down, rather than waiting for them all to be retrieved. I
>  fear that with a JSON based protocol, most server and client implementations
>  wouldn't deal with streaming (although they technically could), and would just
>  deal with completed blobs.

> I think it is worth discussing the tradeoffs in terms of additional complexity
> to offer paging, tree filtering (parentIds: [...]) or both in the standard.
> The pro side, you can access potentially millions of mailboxes without needing
> to know all the folder ids up front.  On the con side, it's more complexity for
> the server (client can always just fetch everything anyway if it doesn't care)
> - and most clients seem to fetch the entire tree anyway, so it just means more
> round trips for them.

I know it's from a different time, but in the RFC for IMAP implementation
recommendations, it says the following:

https://tools.ietf.org/html/rfc2683

   Some servers present Usenet newsgroups to IMAP users.  Newsgroups,
   and other such hierarchical mailbox structures, can be very numerous
   but may have only a few entries at the top level of hierarchy.  Also,
   some servers are built against mail stores that can, unbeknownst to
   the server, have circular hierarchies - that is, it's possible for
   "a/b/c/d" to resolve to the same file structure as "a", which would
   then mean that "a/b/c/d/b" is the same as "a/b", and the hierarchy
   will never end.  The LIST response in this case will be unlimited.

   Clients that will have trouble with this are those that use

       C: 001 LIST "" *

   to determine the mailbox list.  Because of this, clients should not
   use an unqualified "*" that way in the LIST command.  A safer
   approach is to list each level of hierarchy individually, allowing
   the user to traverse the tree one limb at a time

IMAP can essentially work with folders of infinite depth when clients
don't request the full list of folders up front. It can also work
with slow backends. What if you had a giant mail store backed by
Amazon S3? Your data could be spread over multiple buckets. Why wait
for all buckets to be retrieved before you start sending data to
the client? IMAP also lets you present a dynamically generated folder
structure where each new level is calculated on demand. Perhaps
someone will want to stick a dynamic JMAP frontend in front of the
https://www.mail-archive.com/ website where the folder structure is
"List Name/Post Year/Post month". That would be a lot of folders.

I posit that allowing you to request a complete list of all folders in
one go was a mistake in IMAP, and one that shouldn't be repeated in
JMAP.

-- 
Mike Cardwell  https://www.grepular.com
OpenPGP Key    DF70 D8E5 FBD6 8519 9257  C44C 0DA6 8B1E 1801 A332
XMPP OTR Key   8924 B06A 7917 AAF3 DBB1  BF1B 295C 3C78 3EF1 46B4

---

From: Ted Lemon <[email-masked]>
Date: Sun, 7 May 2017 12:14:47 -0400

I tend to think that folders are only notionally hierarchical, so I don't love the idea of hierarchy as the mechanism for hiding folders that are no longer relevant.   Why not just have an archived flag?   Clients that care about archived folders can sync that list; clients that do not, do not have to.   The archived flag could be done automatically: "flag anything that hasn't had any new messages added to it in the past year as archived".  Or "flag anything that is hierarchically below this folder as archived" for your use case.   You could even have an "archived-when" modifier.

If there are to be bazillions of folders, the folder list could also be treated as a versioned entity and traded about using diffs.

---

From: Neil Jenkins <[email-masked]>
Date: Mon, 08 May 2017 16:19:04 +1000

When considering this before, I've tended to observe the
following points:
 * Standard mail clients generally fetch all mailboxes up front, and
   indeed often need to before they will display a (non-loading) UI.
 * The vast majority of users have very few folders (from what I
   remember last time we ran the stats at FastMail, well over 90% of
   users never created a single extra folder beyond the defaults).
So I therefore left out querying mailboxes (and so being able to fetch
them in pages) from the initial JMAP draft. The main case for this seems
to be presenting access to usenet groups or other non-strictly-email
stuff. There is always a question of whether we want to support every
use case, at the expense of making implementation more onerous.
Having said that, there's a standard mechanism in JMAP for doing
querying/sorting/paging of a record type (getFooList[1]), so if we do
decide we want this capability it is mostly obvious how it should be
specified. We'd include a filter option for *parentId* so you could
easily fetch as a hierarchy if you want, and it would be easy to add an
isArchived flag or similar later and filter on that if desired.
As long as we still allow *getMailboxes* to be called with a null *ids*
argument (therefore returning all mailboxes), clients can still just
ignore all this if they always need to fetch the whole set up front.
It's mainly therefore an extra burden on server implementations. So is
it worth the tradeoff? Thoughts?
Neil.

Links:

  1. http://jmap.io/spec-core.html#querying-data

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 08 May 2017 23:14:29 +1000

On Mon, 8 May 2017, at 16:19, Neil Jenkins wrote:
> easily fetch as a hierarchy if you want, and it would be easy to add
> an isArchived flag or similar later and filter on that if desired.> 
> As long as we still allow *getMailboxes* to be called with a null
> *ids* argument (therefore returning all mailboxes), clients can still
> just ignore all this if they always need to fetch the whole set up
> front. It's mainly therefore an extra burden on server
> implementations. So is it worth the tradeoff? Thoughts?
It makes sense to me to have a way to do paging/limiting.  It should be
easy enough for a server implementation.  We wouldn't even need
getMailboxes with a null ids, just a getMailboxList with a search/sort
that matches everything and a fetchRecords.
The tricky bit is that Mailboxes is a tree rather than a list, so the
I'm not sure how you would sort!  Simplest of course is to not do any
sorting, or to sort the entire list lexically by full path with the part
separator being a null so it always sorts by full name:
A
A/foo
A/foo/bar
A/foo bar

(that's super easy for me to do on the server, and to page into.  Can
also page just within a single parent level - which gets you both "*"
and "%" list equivalents from IMAP)
Bron.

--
  Bron Gondwana
  [email-masked]

Links:

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 08 May 2017 23:15:33 +1000

> to this very list. Well, here I am.

Hello and welcome.  Thanks for making the time to contribute :)  It is a topic worth discussing.

Cheers,

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 8 May 2017 14:31:28 +0100

Hi Neil,

On 08/05/2017 07:19, Neil Jenkins wrote:

> When considering this before, I've tended to observe the following points:
>
> them in pages) from the initial JMAP draft. The main case for this 
> seems to be presenting access to usenet groups or other 
> non-strictly-email stuff. There is always a question of whether we 
> want to support every use case, at the expense of making 
> implementation more onerous.
Isode implemented a client that listed mailboxes level by level, due to 
constraint bandwidth.

We also have several thousands of shared mailboxes (they have nothing to 
do with Usenet), I think some enterprise type deployments will be 
similar. So having a way of listing direct children and all descendants 
of a mailbox would be very useful for us.
> querying/sorting/paging of a record type (getFooList 
> <http://jmap.io/spec-core.html#querying-data>), so if we do decide we 
> want this capability it is mostly obvious how it should be specified. 
> We'd include a filter option for /parentId/ so you could easily fetch 
> as a hierarchy if you want, and it would be easy to add an isArchived 
> flag or similar later and filter on that if desired.
>
> As long as we still allow /getMailboxes/ to be called with a null 
> /ids/ argument (therefore returning all mailboxes), clients can still 
I already wrote the server code for IMAP, so not much of a burden on the 
server side.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 8 May 2017 15:14:23 +0100

It's not clear that mailboxes really need to be hierarchical, and we've 
seen that the hierarchy stuff wasn't really well implemented by IMAP 
clients.

It seems plausible that polling the complete list oftenish is unacceptable 
for at least some users, but an IMAP-like hierarchy is not the only way to 
cut bandwidth usage. Retrieving it once and then asking for updates would 
work too, like JMAP's in-mailbox updates.

Servers that don't have a global modseq or some other generation counter 
would have to fall back to sending all/many mailboxes.

Arnt

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 09 May 2017 10:29:11 +1000

> would have to fall back to sending all/many mailboxes.

So if we have just a single sort (fullpath) and an optional filter on parentId plus windowing we can handle either nested or flat folder structures very easily and handle upteen million mailboxes comfortably as well.

It seems viable to implement this even if you're doing depth-first traversal of a folder listing on a filesystem with no other smarts required, you can always use the path of the folder as the "id", though it will obviously not survive renames, so you miss that benefit.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: Neil Jenkins <[email-masked]>
Date: Tue, 09 May 2017 16:21:14 +1000

OK, there seems to be some agreement that this ability is desirable and
not going to cause significant downsides, so I'll put together a pull
request for the spec for people to review.
Neil.