From jamey@minilop.net Fri May 14 11:33:01 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5D2733A3C04 for ; Fri, 14 May 2021 11:33:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=minilop.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VmR3AP5Pd5oq for ; Fri, 14 May 2021 11:32:56 -0700 (PDT) Received: from mail-il1-x133.google.com (mail-il1-x133.google.com [IPv6:2607:f8b0:4864:20::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3F5EC3A3C00 for ; Fri, 14 May 2021 11:32:55 -0700 (PDT) Received: by mail-il1-x133.google.com with SMTP id p15so568262iln.3 for ; Fri, 14 May 2021 11:32:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=minilop.net; s=google; h=mime-version:from:date:message-id:subject:to; bh=WGBymLfv5NgvyiMkK23/jePjPNO/DPyqlmmn1tC5Y8w=; b=JT/t+df8iRHV8ocSRsWBlAkHjlsD0Rf2vThgWlSWaho5LjF2d5o958jzJdxg5OprtN ZcofH4XzcsTKcPszqMijwtWm3vEyrW3l/X9u4EBXQuRTYOz/Y6FkrKMvgsVarQC7NsI1 x7aaOlbYLdq8QaE9IkvvizN+RxJQhyXCueugg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=WGBymLfv5NgvyiMkK23/jePjPNO/DPyqlmmn1tC5Y8w=; b=q2cASyvA2E07hC++tGjzCVVYplc2mLBqZ1s41urnNjt1CiItYGfIV2ffLW0myqPV/F VJr5eRclyIFz+iX9W1eYj2QwqpemEpeg2lRo8YiskGhXPBWLJVIFVj2yksILXHgqoDGu Ld4BoM6QvsrzyV9iyJ0qUfgLZwXbgJ6fcYY/w5JGaBC6EPTFC1mM9I2I8liZe4TYvF3J ZB3A9e+NHk9ETu0krtNFJbU6di1uHG36tLi9h+nnqNeBWXrXOxbWT6bqXyMeB18EG6fx EiFxJxKeuOSNtFGs76QMD+pVh19EYYSBbY3gPHHKLM2N5Lk8PC2yFA4MhkdBZf6mSmJX qzWA== X-Gm-Message-State: AOAM5325ZQw0/KZIWYkunQ/RugXcaKTGWmrn0xSR0NmftLFX1oFOG+es xj3UKBlIFiziYDMedZQEZg+0q6oMGl9kTlpfvdZw9CHKRNk= X-Google-Smtp-Source: ABdhPJzBQjGhNS7W0mPn94bxQQ+G3INBmTtwq7bq4GhJQ//kxG0mNR36idY22SzGGLnqCO6BrrEwv8LMxs0jZ+rmn78= X-Received: by 2002:a05:6e02:671:: with SMTP id l17mr41562903ilt.267.1621017174221; Fri, 14 May 2021 11:32:54 -0700 (PDT) MIME-Version: 1.0 From: Jamey Sharp Date: Fri, 14 May 2021 11:32:42 -0700 Message-ID: To: jmap@ietf.org Content-Type: multipart/alternative; boundary="000000000000d84bfa05c24e786d" Archived-At: Subject: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 May 2021 18:35:41 -0000 --000000000000d84bfa05c24e786d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hey all: for the past few weeks I've been working on a new JMAP server with the goal of making it easy to prototype new data models. I'd love it if folks could take a look and maybe help implement or at least validate it. https://github.com/jameysharp/jterritory I went down this rabbit-hole after trying to figure out how to prototype a feed-reader data model based on JMAP. I previously wrote up rationale for that here: https://jamey.thesharps.us/2020/08/06/feed-reader-cache-coherency/ Implementing JMAP correctly is pretty complicated, and while I do think it's all necessary complexity, that makes it difficult to explore new application domains. The core idea for this project is to not worry too much about performance, but allow a developer to specify just a schema for each data type and optionally some filter and sort extensions, and get the standard methods for free. I've been confused by a few things in RFC8620. I was only going to give a couple examples but ended up with a complete list of everything I remember having trouble with, so I hope it works out to just dump it here. I have some more notes afterward on my current implementation and plans. - Should creating multiple objects allow circular references, or must it be a DAG of new objects? The former seems hard to get right if any of the creations fail. - Can result references appear in nested objects within a method's arguments, or only on the arguments directly within the top-level Invocation? - Does the client need to ensure that it never uses a creation ID of "foo" if there's some random string whose value happens to be "#foo", or does /set need to know which properties have Id type? - If the client uses the same creation ID in two method calls, despite the "SHOULD" cautioning against it, and the second create fails, should the first one's ID continue to be used? Is "the most recently created item with that id" the most recent attempt, or the most recent success? - I think I get it now, but I've been confused about how the responses from /copy relate to its three phases. There's one /copy response covering phases 1 and 2, then optionally a single /set response for phase 3, right? - What value should the position attribute in a /query response have if the requested position is past the end of the results? - If a client has a sparse array of query results and gets a /queryChanges response indicating that it should delete an object that it didn't have in the array, then I don't see how it can tell whether to shift down any of the objects it does have: the deleted object could have been anywhere. This interface only seems safe so long as clients either always keep a complete prefix of a query's results, or invalidate their cache if told to delete an object they didn't know about. - What does property immutability have to do with query result changes? Isn't the only thing that matters whether the property actually changed, rather than whether it could change? Is it just that some reasonable implementations can't compute certain changes for mutable properties? - The specification for upToId seems to imply to look up the position of the given ID in the new results, though that isn't entirely clear to me. But I think it makes more sense to find its position in the old results, and send changes that update the client's cache to the same length. That means the optimization still works even if the selected object is no longer in the results, and puts an upper bound on how much data the client might have to deal with: twice the number of items it has cached already. But overall I've found the spec pretty clear and solid so far. In the rest of this note I offer some more details on my current status and approach. I'm taking the simplifying assumption that there are no dynamically computed properties; a call to /set has to update any related objects at the same time. That lets me keep all objects in a single database table, storing simple change-sequence numbers alongside raw JSON, instead of the complexities recommended for email implementors. There are performance costs to this simplicity, but that's a deliberate trade-off I'm making. At this point I have the semantics for get, changes, set, and most of query implemented, maybe even correctly. I've also implemented result references and Core/echo, which go shockingly well together for testing. I haven't hooked anything up to HTTP yet, just writing tests that pass JSON in to the underlying functions. Currently all my tests are property-based randomized testing. I'd like to think my tests might be interesting for other server implementors. If there are test suites I might be able to reuse, I'd love to hear about them. My observation about implementing queryChanges is that you only need to be able to report changes against queries which have previously actually been run, so caching returned query results is enough to do the job generically, without understanding the semantics of the data model or filters. Of course, caching also helps with the fact that without that understanding, I also can't provide indexes to make filters perform well. Refreshing the cache can be as fast as calling /changes, as long as the cache includes all the selected sort keys for each object. Also, I think I can enable prototypes built on this framework to specify custom indexes, presumably after the developer has gotten some experience with which filters are most frequently used. I've yet to tackle the session state, blobs, or push, but those don't seem like they'll have hard questions about semantics, just some possibly tricky engineering choices. We'll see how that goes. Thanks for a nice spec and for getting to the end of this message. =F0=9F= =98=85 Jamey --000000000000d84bfa05c24e786d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hey all: for the past few weeks I've been working on = a new JMAP server with the goal of making it easy to prototype new data mod= els. I'd love it if folks could take a look and maybe help implement or= at least validate it.


I went down this rabbit-hole after trying = to figure out how to prototype a feed-reader data model based on JMAP. I pr= eviously wrote up rationale for that here:


Imple= menting JMAP correctly is pretty complicated, and while I do think it's= all necessary complexity, that makes it difficult to explore new applicati= on domains. The core idea for this project is to not worry too much about p= erformance, but allow a developer to specify just a schema for each data ty= pe and optionally some filter and sort extensions, and get the standard met= hods for free.

I've = been confused by a few things in RFC8620. I was only going to give a couple= examples but ended up with a complete list of everything I remember having= trouble with, so I hope it works out to just dump it here. I have some mor= e notes afterward on my current implementation and plans.

- Should creating multiple objects allow = circular references, or must it be a DAG of new objects? The former seems h= ard to get right if any of the creations fail.
<= br>
- Can result references appear in nested objects= within a method's arguments, or only on the arguments directly within = the top-level Invocation?

- Does the client need to ensure that it never uses a creation ID of &quo= t;foo" if there's some random string whose value happens to be &qu= ot;#foo", or does /set need to know which properties have Id type?

- If the client uses the sa= me creation ID in two method calls, despite the "SHOULD" cautioni= ng against it, and the second create fails, should the first one's ID c= ontinue to be used? Is "the most recently created item with that id&qu= ot; the most recent attempt, or the most recent success?

- I think I get it now, but I've been = confused about how the responses from /copy relate to its three phases. The= re's one /copy response covering phases 1 and 2, then optionally a sing= le /set response for phase 3, right?

- What value should the position attribute in a /query respons= e have if the requested position is past the end of the results?
<= div dir=3D"auto">
- If a client has a sparse arr= ay of query results and gets a /queryChanges response indicating that it sh= ould delete an object that it didn't have in the array, then I don'= t see how it can tell whether to shift down any of the objects it does have= : the deleted object could have been anywhere. This interface only seems sa= fe so long as clients either always keep a complete prefix of a query's= results, or invalidate their cache if told to delete an object they didn&#= 39;t know about.

- What = does property immutability have to do with query result changes? Isn't = the only thing that matters whether the property actually changed, rather t= han whether it could change? Is it just that some reasonable implementation= s can't compute certain changes for mutable properties?

- The specification for upToId seems = to imply to look up the position of the given ID in the new results, though= that isn't entirely clear to me. But I think it makes more sense to fi= nd its position in the old results, and send changes that update the client= 's cache to the same length. That means the optimization still works ev= en if the selected object is no longer in the results, and puts an upper bo= und on how much data the client might have to deal with: twice the number o= f items it has cached already.

But overall I've found the spec pretty clear and solid so far.

In the rest of this n= ote I offer some more details on my current status and approach.

I'm taking the simplifying ass= umption that there are no dynamically computed properties; a call to /set h= as to update any related objects at the same time. That lets me keep all ob= jects in a single database table, storing simple change-sequence numbers al= ongside raw JSON, instead of the complexities recommended for email impleme= ntors. There are performance costs to this simplicity, but that's a del= iberate trade-off I'm making.

At this point I have the semantics for get, changes, set, and most of q= uery implemented, maybe even correctly. I've also implemented result re= ferences and Core/echo, which go shockingly well together for testing. I ha= ven't hooked anything up to HTTP yet, just writing tests that pass JSON= in to the underlying functions.

Currently all my tests are property-based randomized testing. I&#= 39;d like to think my tests might be interesting for other server implement= ors. If there are test suites I might be able to reuse, I'd love to hea= r about them.

My observa= tion about implementing queryChanges is that you only need to be able to re= port changes against queries which have previously actually been run, so ca= ching returned query results is enough to do the job generically, without u= nderstanding the semantics of the data model or filters. Of course, caching= also helps with the fact that without that understanding, I also can't= provide indexes to make filters perform well. Refreshing the cache can be = as fast as calling /changes, as long as the cache includes all the selected= sort keys for each object. Also, I think I can enable prototypes built on = this framework to specify custom indexes, presumably after the developer ha= s gotten some experience with which filters are most frequently used.
=

I've yet to tackle the se= ssion state, blobs, or push, but those don't seem like they'll have= hard questions about semantics, just some possibly tricky engineering choi= ces. We'll see how that goes.

Thanks for a nice spec and for getting to the end of this message= . =F0=9F=98=85
Jamey
--000000000000d84bfa05c24e786d-- From nobody Tue May 18 20:06:03 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5E6663A1AF5 for ; Tue, 18 May 2021 20:06:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=fastmailteam.com header.b=h8wKQCne; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=NElayC55 Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W6-EZAztRwfB for ; Tue, 18 May 2021 20:05:55 -0700 (PDT) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id AAFD33A1AF2 for ; Tue, 18 May 2021 20:05:55 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id EF59A5C01A4 for ; Tue, 18 May 2021 23:05:53 -0400 (EDT) Received: from imap43 ([10.202.2.93]) by compute4.internal (MEProxy); Tue, 18 May 2021 23:05:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= fastmailteam.com; h=mime-version:message-id:in-reply-to :references:date:from:to:subject:content-type; s=fm2; bh=Xuvjwal 1ZTyRbPwaJvKMvFnZ5ZdVIdcgXZ6EhKleYN0=; b=h8wKQCneilOivNHD1Z6dAKE yujzIb9VFCYs0Vjp1W5Yu+4fsK0gXsjJpt3Hap9bYcNrqB1VY0fcTi4ZwbwexYoa FkZtJs+t8SVWP4jCWpsiI/blOgA6N8ESkBfAFnh8lpOtqOW+MXy4zxbS3Ll8+e/q TqBpWpg/TYLV9SKnda3xFp/eb5hpIz63ivte9mrW00DzCDeaME6RPQWHKkSIU1Ac 6uHOAhdM+sEFLRGVQkmbYMTRkVGUiVC5KmoV34lSqlN5+7Ykdvg1tObfnLm57dGe 4CDw5wD0A6w++GfPbEkzLOorP3KXNDta6RWf+GmMU2FkK9o6mowwVrKJcmLWlHg= = DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=Xuvjwa l1ZTyRbPwaJvKMvFnZ5ZdVIdcgXZ6EhKleYN0=; b=NElayC559K/r78NpYkvtX9 XFgtQYFqLyAfiR7rirr3oGdERp9HT3DXi4CArc7vdb+hKnxcLVPmn1LeNmBG+ZkS lpMUnPAd3X8z+2q919MUC4VKR2kFvPu7nRhDivEKqHMFwA3M8VPQ97TXpPWucPmz 0M8eILh+uQkrRrB885nAQ+2FbpkzHH7q5U+f/O5+P+OytSLAqZ4HVYbrl52uCae6 inTVgvCSxIrhJIsTYYk90OjZJ9jYKl9reVdcNWhJYVlcONsJU31ETkZggsy1jOvn be8mk2sIZvMgWnChJndTmNFsSkkAdd/VzcNwq6q6fWEm14bQBC+yeyNQnYvSQHHA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeikedgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreertdenucfhrhhomhepfdfpvghi lhculfgvnhhkihhnshdfuceonhgvihhljhesfhgrshhtmhgrihhlthgvrghmrdgtohhmqe enucggtffrrghtthgvrhhnpeevvdetvdduleekhfeghfetfeettdelhfehfeevffevleek uddtudffieevjeevhfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehnvghilhhjsehfrghsthhmrghilhhtvggrmhdrtghomh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id A82F918005D; Tue, 18 May 2021 23:05:53 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-701-g78bd539edf-fm-ubox-20210517.001-g78bd539e Mime-Version: 1.0 Message-Id: <978706aa-c660-451f-a119-807ec9cc1bce@beta.fastmail.com> In-Reply-To: References: Date: Wed, 19 May 2021 13:05:48 +1000 From: "Neil Jenkins" To: "IETF JMAP Mailing List" Content-Type: multipart/alternative; boundary=f90b4ab456a0490e87b9354105de3841 Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 May 2021 03:06:02 -0000 --f90b4ab456a0490e87b9354105de3841 Content-Type: text/plain Hi Jamey, > Hey all: for the past few weeks I've been working on a new JMAP server with the goal of making it easy to prototype new data models. That's awesome! Nice work. > I've been confused by a few things in RFC8620. I was only going to give a couple examples but ended up with a complete list of everything I remember having trouble with, so I hope it works out to just dump it here. I have some more notes afterward on my current implementation and plans. > > - Should creating multiple objects allow circular references, or must it be a DAG of new objects? The former seems hard to get right if any of the creations fail. The relevant bit of the spec here is: The final state MUST be valid after the "Foo/set" is finished; however, the server may have to transition through invalid intermediate states (not exposed to the client) while processing the individual create/update/destroy requests. For example, suppose there is a "name" property that must be unique. A single method call could rename an object A => B and simultaneously rename another object B => A. If the final state is valid, this is allowed. Otherwise, each creation, modification, or destruction of an object should be processed sequentially and accepted/rejected based on the current server state. I don't think we actually considered a data type that supported circular references, but the way the current spec is written I would say that if the circular reference is valid for that data type and the final state is valid, then yes you would be allowed to do that in a single /set. But I agree that's a bit ambiguous, and I think you could probably forbid it in the /set description for the particular data type. Do you have an example in mind? > - Can result references appear in nested objects within a method's arguments, or only on the arguments directly within the top-level Invocation? Only on the top-level. The *arguments object* is the top-level object passed in the *Invocation*. So this: When processing a method call, the server MUST first check the arguments object for any names beginning with "#". means just the top-level object. > - Does the client need to ensure that it never uses a creation ID of "foo" if there's some random string whose value happens to be "#foo", or does /set need to know which properties have Id type? When processing a /set you need to know which properties have Id type (well you need to know all the types really in order to be able to validate the data). > - If the client uses the same creation ID in two method calls, despite the "SHOULD" cautioning against it, and the second create fails, should the first one's ID continue to be used? Is "the most recently created item with that id" the most recent attempt, or the most recent success? Most recent success was the intention, but looking at it again I can see how you could interpret it both ways. > - I think I get it now, but I've been confused about how the responses from /copy relate to its three phases. There's one /copy response covering phases 1 and 2, then optionally a single /set response for phase 3, right? That's right. > - What value should the position attribute in a /query response have if the requested position is past the end of the results? The definition of "position" in the response is: The zero-based index of the first result in the "ids" array within the complete list of query results. So I agree it's undefined if there are no results (but I also think it's unimportant, because there are no results to position!). I would just return the requested position I think. It would probably be worth making this well-defined in a future revision. > - If a client has a sparse array of query results and gets a /queryChanges response indicating that it should delete an object that it didn't have in the array, then I don't see how it can tell whether to shift down any of the objects it does have: the deleted object could have been anywhere. This interface only seems safe so long as clients either always keep a complete prefix of a query's results, or invalidate their cache if told to delete an object they didn't know about. Close. In this situation you have to invalidate any results in your sparse query after the first gap, but you can still keep the ones before. So if I have: [A, B, -, -, D, -, -] and queryChanges tells me that E has been removed, I would have to invalidate after the first gap, so my cached query would now be: [A, B, -, -. -. -. -] > - What does property immutability have to do with query result changes? Isn't the only thing that matters whether the property actually changed, rather than whether it could change? Is it just that some reasonable implementations can't compute certain changes for mutable properties? Yes, it's that the spec was written with the idea that /queryChanges would be implemented based on knowing which records have changed from last time, not based on having the old query cached and comparing it with the new one. (This approach is hard to scale effectively.) If you only know that a record has changed, not which property on that record then you don't know what position it had in the client's current query cache (unless you have cached the old query, as you are doing), so have to tell the client to remove it an readd it to ensure it's correct. If you can return precise changes with your implementation, that's great! This is really about guidance for what to do when you can calculate usable, if slightly less concise, changes efficiently. The text could definitely be better here, it's written a bit too much with an implementation approach in mind (in general we tried to not do that, as there are multiple good ways of implementing various bits of JMAP). > - The specification for upToId seems to imply to look up the position of the given ID in the new results, though that isn't entirely clear to me. Yes, in the new results. > But I think it makes more sense to find its position in the old results, and send changes that update the client's cache to the same length. That means the optimization still works even if the selected object is no longer in the results, and puts an upper bound on how much data the client might have to deal with: twice the number of items it has cached already. I'm not quite sure what you mean by "to the same length", but yes if you have the old query cached you could optimise the case where the upToId is not in the new results a bit further. But again, the spec is written to allow implementers to calculate query changes *without* caching the old query state. > But overall I've found the spec pretty clear and solid so far. Great! This was really good feedback. I don't think any of it requires an errata (although if someone on the list disagrees, please pipe up!), but if we publish an updated RFC at some point in the future we know which points to clarify. > Currently all my tests are property-based randomized testing. I'd like to think my tests might be interesting for other server implementors. If there are test suites I might be able to reuse, I'd love to hear about them. Getting a good test suite for the JMAP spec is something we would love to have (and know from past experience is important for interoperability). I don't think anyone has published an open reusable one yet. > I've yet to tackle the session state, blobs, or push, but those don't seem like they'll have hard questions about semantics, just some possibly tricky engineering choices. We'll see how that goes. I think queryChanges is the trickiest bit to get your head around semantically so hopefully now that's done the rest is not too hard, but do keep us informed of your progress and any other spec issues you find! Cheers, Neil. --f90b4ab456a0490e87b9354105de3841 Content-Type: text/html Content-Transfer-Encoding: quoted-printable
Hi Jamey,<= br>

<= div dir=3D"auto">
Hey all: for the past few weeks I've been working = on a new JMAP server with the goal of making it easy to prototype new da= ta models.

That's awesom= e! Nice work.

I've been confused by a= few things in RFC8620. I was only going to give a couple examples but e= nded up with a complete list of everything I remember having trouble wit= h, so I hope it works out to just dump it here. I have some more notes a= fterward on my current implementation and plans.

- Should creating multiple objects allow= circular references, or must it be a DAG of new objects? The former see= ms hard to get right if any of the creations fail.

The relevant bit of the spec here is:

   The final state MUST be valid after the "Foo/set" is finished;
   however, the server may have to transition through invalid
   intermediate states (not exposed to the client) while processing the
   individual create/update/destroy requests.  For example, suppose
   there is a "name" property that must be unique.  A single method call=

   could rename an object A =3D> B and simultaneously rename another
   object B =3D> A.  If the final state is valid, this is allowed.
   Otherwise, each creation, modification, or destruction of an object
   should be processed sequentially and accepted/rejected based on the
   current server state.

I don't thi= nk we actually considered a data type that supported circular references= , but the way the current spec is written I would say that if the circul= ar reference is valid for that data type and the final state is valid, t= hen yes you would be allowed to do that in a single /set. But I agree th= at's a bit ambiguous, and I think you could probably forbid it in the /s= et description for the particular data type. Do you have an example in m= ind?

- Can result references appear in= nested objects within a method's arguments, or only on the arguments di= rectly within the top-level Invocation?

Only on the top-level. The arguments object is th= e top-level object passed in the Invocation. So this:

= When processing a method call, the server MUST first check the arguments object for any names beginning with "#".

means just the top-level object.

- Does the client need to ensure that it never uses a creation ID of = "foo" if there's some random string whose value happens to be "#foo", or= does /set need to know which properties have Id type?

When processing a /set you need to know w= hich properties have Id type (well you need to know all the types really= in order to be able to validate the data).

- If the client uses the same creation ID in two method calls, de= spite the "SHOULD" cautioning against it, and the second create fails, s= hould the first one's ID continue to be used? Is "the most recently crea= ted item with that id" the most recent attempt, or the most recent succe= ss?

Most recent success = was the intention, but looking at it again I can see how you could inter= pret it both ways.

- I think I get it n= ow, but I've been confused about how the responses from /copy relate to = its three phases. There's one /copy response covering phases 1 and 2, th= en optionally a single /set response for phase 3, right?
=

That's right.

<= blockquote type=3D"cite" id=3D"qt" style=3D"">
- What value should the position attribute in a /query respons= e have if the requested position is past the end of the results?

The definition of "position" in= the response is:

      T=
he zero-based index of the first result in the "ids" array within
      the complete list of query results.

S= o I agree it's undefined if there are no results (but I also think it's = unimportant, because there are no results to position!). I would just re= turn the requested position I think. It would probably be worth making t= his well-defined in a future revision.

- If a client has a sparse array of query results and gets a /queryCha= nges response indicating that it should delete an object that it didn't = have in the array, then I don't see how it can tell whether to shift dow= n any of the objects it does have: the deleted object could have been an= ywhere. This interface only seems safe so long as clients either always = keep a complete prefix of a query's results, or invalidate their cache i= f told to delete an object they didn't know about.

Close. In this situation you have to invalida= te any results in your sparse query after the first gap, but you can sti= ll keep the ones before. So if I have:

[A, = B, -, -, D, -, -]

and queryChanges tells me= that E has been removed, I would have to invalidate after the first gap= , so my cached query would now be:

[A, B, -= , -. -. -. -]

- What does property im= mutability have to do with query result changes? Isn't the only thing th= at matters whether the property actually changed, rather than whether it= could change? Is it just that some reasonable implementations can't com= pute certain changes for mutable properties?

Yes, it's that the spec was written with the idea t= hat /queryChanges would be implemented based on knowing which records ha= ve changed from last time, not based on having the old query cached and = comparing it with the new one. (This approach is hard to scale effective= ly.) If you only know that a record has changed, not which property on t= hat record then you don't know what position it had in the client's curr= ent query cache (unless you have cached the old query, as you are doing)= , so have to tell the client to remove it an readd it to ensure it's cor= rect.

If you can return precise changes wit= h your implementation, that's great! This is really about guidance for w= hat to do when you can calculate usable, if slightly less concise, chang= es efficiently. The text could definitely be better here, it's written a= bit too much with an implementation approach in mind (in general we tri= ed to not do that, as there are multiple good ways of implementing vario= us bits of JMAP).

- The specification = for upToId seems to imply to look up the position of the given ID in the= new results, though that isn't entirely clear to me.

Yes, in the new results.
But I think it makes more sense to find its position = in the old results, and send changes that update the client's cache to t= he same length. That means the optimization still works even if the sele= cted object is no longer in the results, and puts an upper bound on how = much data the client might have to deal with: twice the number of items = it has cached already.

I= 'm not quite sure what you mean by "to the same length", but yes if you = have the old query cached you could optimise the case where the upToId i= s not in the new results a bit further. But again, the spec is written t= o allow implementers to calculate query changes without caching t= he old query state.

But overall I've = found the spec pretty clear and solid so far.

Great! This was really good feedback. I don't thin= k any of it requires an errata (although if someone on the list disagree= s, please pipe up!), but if we publish an updated RFC at some point in t= he future we know which points to clarify.

Currently all my tests are property-based randomized testing.= I'd like to think my tests might be interesting for other server implem= entors. If there are test suites I might be able to reuse, I'd love to h= ear about them.

Ge= tting a good test suite for the JMAP spec is something we would love to = have (and know from past experience is important for interoperability). = I don't think anyone has published an open reusable one yet.

I've yet to tackle the session= state, blobs, or push, but those don't seem like they'll have hard ques= tions about semantics, just some possibly tricky engineering choices. We= 'll see how that goes.

=
I think queryChanges is the trickiest bit to get your head around s= emantically so hopefully now that's done the rest is not too hard, but d= o keep us informed of your progress and any other spec issues you find!<= br>

Cheers,
Neil.
--f90b4ab456a0490e87b9354105de3841-- From nobody Tue May 18 20:56:48 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 386B33A1C50 for ; Tue, 18 May 2021 20:56:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.096 X-Spam-Level: X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=fastmailteam.com header.b=F2U2wcwh; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=kyC96urJ Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7CHL2oCYRbvy for ; Tue, 18 May 2021 20:56:41 -0700 (PDT) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E891B3A1C3F for ; Tue, 18 May 2021 20:56:40 -0700 (PDT) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 2882C5C0194 for ; Tue, 18 May 2021 23:56:40 -0400 (EDT) Received: from imap43 ([10.202.2.93]) by compute1.internal (MEProxy); Tue, 18 May 2021 23:56:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= fastmailteam.com; h=mime-version:message-id:in-reply-to :references:date:from:to:subject:content-type; s=fm2; bh=ji5RuvN sVv3W/Wlm1rcokUVWGOzuf18weSM1Zh2JJ34=; b=F2U2wcwhhcUCJNc2br/2bCY Q/Sk0whjMoL1rmpj5zrsIl6XBV6ILQK2Ux/QagPPlNt+AXWLCdyTJ6gH2UBxhFy4 iWxcY4F5I2rLPWB4bSglvSU38Bcu365q9h9r4B6/YtAfmHilFMzV/T5lDnE6m/tm qUkFWH1CwK7mMu5u45vEEkNdbSlSkccKFxZEpp/JMuI5HD6wkAzwmKvBJThidUoF y5rME5DpBn8pZAeZzPu23HFQ+4aIJwvz9x8bUPIzMwn4ZkXFe+pNAFkboQGT5dbU hstgsDhEl9yIi56JDrXKECXi+Wl/+TO3gc3rlA5YqwLv7o4emMl2khF6quNLgug= = DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=ji5Ruv NsVv3W/Wlm1rcokUVWGOzuf18weSM1Zh2JJ34=; b=kyC96urJaEFmPkzkvU6AEi Uuj/JeS2B0wd66X9m4NRHLslS+M2hj8tFBJGL+BegqllsiNLT1cPnmnGFYqOpgeQ Wm2U57qeVuPuHQiBDw+LA3pru6ma02rm1cRMaFWqHCJ9D3KVceVS0md7VpJ5KG/K UFapWeJvdlZpEd5AbdrN4QS0YIW3oiMpUCSXDo/n4/P9EbptELzkMArhnNRljAok odIrIhaz5vi6hGbgz7kXaTxuJLOEuh8K3zvvCr3W38vkKgl7bSP+lsjimmuWhNRl nHhtx2c4I8vIa9Nrcfy9mz5dpQNmcmmr1UzvUsfET/b7V3Ew8h4PaCtVDaAlK14g == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeikedgjeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsegrtd erreerredtnecuhfhrohhmpedfuehrohhnucfiohhnugifrghnrgdfuceosghrohhnghes fhgrshhtmhgrihhlthgvrghmrdgtohhmqeenucggtffrrghtthgvrhhnpeelkefhvedvfe euieeilefftddtieefjefhfeektdeuuefftdekudetheeivdetgeenucffohhmrghinhep rhgvlhgrthhiohhnshhhihhprdhfohhonecuvehluhhsthgvrhfuihiivgeptdenucfrrg hrrghmpehmrghilhhfrhhomhepsghrohhnghesfhgrshhtmhgrihhlthgvrghmrdgtohhm X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 7219D18005D; Tue, 18 May 2021 23:56:39 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-701-g78bd539edf-fm-ubox-20210517.001-g78bd539e Mime-Version: 1.0 Message-Id: <2d6ede2e-152d-4ed6-9498-d87ecfcad9e0@dogfood.fastmail.com> In-Reply-To: <978706aa-c660-451f-a119-807ec9cc1bce@beta.fastmail.com> References: <978706aa-c660-451f-a119-807ec9cc1bce@beta.fastmail.com> Date: Wed, 19 May 2021 13:56:19 +1000 From: "Bron Gondwana" To: jmap@ietf.org Content-Type: multipart/alternative; boundary=ac47d1102b7e4224a7f27c14f509066f Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 May 2021 03:56:46 -0000 --ac47d1102b7e4224a7f27c14f509066f Content-Type: text/plain On Wed, May 19, 2021, at 13:05, Neil Jenkins wrote: > I don't think we actually considered a data type that supported circular references, but the way the current spec is written I would say that if the circular reference is valid for that data type and the final state is valid, then yes you would be allowed to do that in a single /set. But I agree that's a bit ambiguous, and I think you could probably forbid it in the /set description for the particular data type. Do you have an example in mind? I can think of one - anything double-linked - for example calendar events that have a rel=next and rel=prev relationship. [ "Foo/set", { "a" : { next: "#b" } "b" : { prev: "#a" } }, "x" ] >> - If the client uses the same creation ID in two method calls, despite the "SHOULD" cautioning against it, and the second create fails, should the first one's ID continue to be used? Is "the most recently created item with that id" the most recent attempt, or the most recent success? > > Most recent success was the intention, but looking at it again I can see how you could interpret it both ways. If you break it, you get to keep both pieces! >> But overall I've found the spec pretty clear and solid so far. > > Great! This was really good feedback. I don't think any of it requires an errata (although if someone on the list disagrees, please pipe up!), but if we publish an updated RFC at some point in the future we know which points to clarify. We should track these somewhere specific, because I for sure won't know which points to clarify in 5 years' time, and I'm unlikely to read every mail thread! Maybe as open issues on the git repo, or even in a text file in the repo? Cheers, Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd brong@fastmailteam.com --ac47d1102b7e4224a7f27c14f509066f Content-Type: text/html Content-Transfer-Encoding: quoted-printable
On Wed, May 19, 2021, at 13:05, Neil Jenkins wrote:
I don't thin= k we actually considered a data type that supported circular references,= but the way the current spec is written I would say that if the circula= r reference is valid for that data type and the final state is valid, th= en yes you would be allowed to do that in a single /set. But I agree tha= t's a bit ambiguous, and I think you could probably forbid it in the /se= t description for the particular data type. Do you have an example in mi= nd?

I can think of one - anything double-lin= ked - for example calendar events that have a rel=3Dnext and rel=3Dprev = relationship.

[ "Foo/set", {
  "a" : { next: "#b" }
  "b" : { prev: "#a" }
}, "x" ]

- If the= client uses the same creation ID in two method calls, despite the "SHOU= LD" cautioning against it, and the second create fails, should the first= one's ID continue to be used? Is "the most recently created item with t= hat id" the most recent attempt, or the most recent success?

Most recent success was the intenti= on, but looking at it again I can see how you could interpret it both wa= ys.

If you break it, you get to keep both pi= eces!

But overall I've found t= he spec pretty clear and solid so far.
=
Great! This was really good feedback. I don't think any o= f it requires an errata (although if someone on the list disagrees, plea= se pipe up!), but if we publish an updated RFC at some point in the futu= re we know which points to clarify.

We shoul= d track these somewhere specific, because I for sure won't know which po= ints to clarify in 5 years' time, and I'm unlikely to read every mail th= read!  Maybe as open issues on the git repo, or even in a text file= in the repo?

Cheers,

Bron.

<= div id=3D"sig56629417">
--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  brong@fastmailteam.com


--ac47d1102b7e4224a7f27c14f509066f-- From nobody Fri May 21 13:20:23 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4767A3A1F48 for ; Fri, 21 May 2021 13:20:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.099 X-Spam-Level: X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=minilop.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1El1UnW9UAHM for ; Fri, 21 May 2021 13:20:16 -0700 (PDT) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9A7783A1F42 for ; Fri, 21 May 2021 13:20:16 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id pi6-20020a17090b1e46b029015cec51d7cdso7773506pjb.5 for ; Fri, 21 May 2021 13:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=minilop.net; s=google; h=date:from:to:subject:message-id:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=xBIYy5lSe4na56jYq7ugNxqSpfcqVY4It230VoO+/sU=; b=ijrthNxQpmTmN2tyQ3gjj8ovYjLH4jB4HCcsvUkQkdCxxcICRoHeEUoQVM2wvSXw2E UuzG9BmjxaMevMUVcfzjzYm1UQDrZHpvqO9AdzK1jge0u0PLbEW/GDySB2ket4UCCg+w cnEN9avM0hJvY/8GTG4efcH/iOgIgHLH1dRpM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=xBIYy5lSe4na56jYq7ugNxqSpfcqVY4It230VoO+/sU=; b=QPRVbzH6PtJ4BssK0Mukt7jiRSwJroDayJxKvDNjzIGNYSY36a9OErYTNhgjrmIogU +v66RZ+hQTEk42mwbqAvhLYiYU+Q/zlQlkLwnnygj5Ctp6/EMZmjLoG5xiLNuSyXX97N pMsWETS7ZhnIAGJE/Pub86C14/CvzHxurZ6mlIq5qExxDyFzZxjzBfBdbbIADPDqStm6 YShbz2LPEabAa14cFMPUOs004fhZXgbph8iq2GrgUP4YV4kzYVqbcy/VHnR95NIe7yWl C5rw+NhAcPcbgmOQUycXRq78YDH7dYbCoELX7IZAmFWnmT8udSxChlyJWoxk4fd2qXR2 rPvQ== X-Gm-Message-State: AOAM531nEs1D/ihBk0GCYArd4tOsPh6mhQSq9owPrXVXT8KOjgZPq2I8 71qoIBhQ1zVA5B/j0HI0bWxgiO93ijhGTw== X-Google-Smtp-Source: ABdhPJzvMDKFAf55gqwQ92SM+hw5EFVONAtBCvMjebbgG3doPWlyTR5dcLe2/5tNkcdXZiaUG42K5Q== X-Received: by 2002:a17:90a:db51:: with SMTP id u17mr12462802pjx.222.1621628414681; Fri, 21 May 2021 13:20:14 -0700 (PDT) Received: from eh ([97.115.152.200]) by smtp.gmail.com with ESMTPSA id u23sm4786229pfn.106.2021.05.21.13.20.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 13:20:13 -0700 (PDT) Received: by eh (sSMTP sendmail emulation); Fri, 21 May 2021 13:20:12 -0700 Date: Fri, 21 May 2021 13:20:12 -0700 From: Jamey Sharp To: IETF JMAP Mailing List Message-ID: <20210521202012.GC7261@eh> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2d6ede2e-152d-4ed6-9498-d87ecfcad9e0@dogfood.fastmail.com> <978706aa-c660-451f-a119-807ec9cc1bce@beta.fastmail.com> Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 May 2021 20:20:21 -0000 Thanks for answering my questions! I've snipped parts where your answer fully resolved my confusion, which was most of them really… On Wed, May 19, 2021 at 01:05:48PM +1000, Neil Jenkins wrote: > > - Should creating multiple objects allow circular references, or > > must it be a DAG of new objects? The former seems hard to get > > right if any of the creations fail. > > The relevant bit of the spec here is: > > The final state MUST be valid after the "Foo/set" is finished; > however, the server may have to transition through invalid > intermediate states (not exposed to the client) while processing the > individual create/update/destroy requests. [...] > > I don't think we actually considered a data type that supported > circular references, but the way the current spec is written I would > say that if the circular reference is valid for that data type and > the final state is valid, then yes you would be allowed to do that > in a single /set. But I agree that's a bit ambiguous, and I think > you could probably forbid it in the /set description for the > particular data type. Do you have an example in mind? I don't have an example; I was actually hoping you'd say the intent was to forbid it. Since I'm writing an implementation which is agnostic to the particular data type, I need to figure out if this is a case I should allow. I haven't tested this case, but I think my current implementation fails all the creates in a cycle with invalidProperties. Looking at a different bit of the text: In the case of records with references to the same type, the server MUST order the creates and updates within a single method call so that creates happen before their creation ids are referenced by another create/update/destroy in the same call. I think this wording is a little strange because I don't think destroys can reference creation IDs at all, right? But more importantly: I think a strict reading would prohibit circular references in creates, since there is no ordering satisfying this MUST in that case. Even under that reading, it only says that a create happens before "another" create which references it, but doesn't say anything about a create referencing itself, so that's ambiguous too I think. > > - Does the client need to ensure that it never uses a creation ID > > of "foo" if there's some random string whose value happens to be > > "#foo", or does /set need to know which properties have Id type? > > When processing a /set you need to know which properties have Id > type (well you need to know all the types really in order to be able > to validate the data). Okay, sure, that's true. It's just that I wanted to substitute creation Ids first, without considering property types, and only validate the data afterward. It seems easy enough for a client to use random creation Ids which are sufficiently large that they never collide with each other or with user-provided data. So if it's intended that clients may use strings which match their own creation Ids, I might just decide this implementation doesn't interoperate with clients that do that, which seems okay in a prototyping setting. > > - If the client uses the same creation ID in two method calls, > > despite the "SHOULD" cautioning against it, and the second create > > fails, should the first one's ID continue to be used? Is "the most > > recently created item with that id" the most recent attempt, or > > the most recent success? > > Most recent success was the intention, but looking at it again I can > see how you could interpret it both ways. On Wed, May 19, 2021 at 01:56:19PM +1000, Bron Gondwana wrote: > If you break it, you get to keep both pieces! Hah, yeah. "Most recent success" means a client which violates this "SHOULD" can't predict which object it's going to be referencing, and the target could even be of a different type than expected. But after all, you did tell them they shouldn't do that… I guess I feel like that SHOULD would be better as a MUST. Getting unexpected results, but only when an error occurs in a different operation, sounds like a surefire way to suffering and support calls. On Wed, May 19, 2021 at 01:05:48PM +1000, Neil Jenkins wrote: > > - What value should the position attribute in a /query response > > have if the requested position is past the end of the results? > > The definition of "position" in the response is: > > The zero-based index of the first result in the "ids" array within > the complete list of query results. > > So I agree it's undefined if there are no results (but I also think > it's unimportant, because there are no results to position!). I > would just return the requested position I think. It would probably > be worth making this well-defined in a future revision. I agree it shouldn't matter, but I think it might be helpful to define it any of these equivalent ways: - the position immediately after the last item in the list - the length of the list - the minimum value of position for which you'd get the same response A client can use calculateTotal to request that information, or negative position to implicitly reference it. But if they've gotten into a state where they thought there'd be more results than there really are, maybe it'd be nice to just tell them? > > - If a client has a sparse array of query results and gets a > > /queryChanges response indicating that it should delete an object > > that it didn't have in the array, then I don't see how it can tell > > whether to shift down any of the objects it does have: the deleted > > object could have been anywhere. [...] > > Close. In this situation you have to invalidate any results in your > sparse query after the first gap, but you can still keep the ones > before. Ohhhh. That makes sense, thanks. The spec is wrong here though, I think: The result of this is that if the client has a cached sparse array of Foo ids corresponding to the results in the old state, then: fooIds = [ "id1", "id2", null, null, "id3", "id4", null, null, null ] If it *splices out* all ids in the removed array that it has in its cached results, then: removed = [ "id2", "id31", ... ]; fooIds => [ "id1", null, null, "id3", "id4", null, null, null ] and *splices in* (one by one in order, starting with the lowest index) all of the ids in the added array: added = [{ id: "id5", index: 0, ... }]; fooIds => [ "id5", "id1", null, null, "id3", "id4", null, null, null ] Here, "id31" is shown as removed, which wasn't in fooIds, but "id3" and "id4" come after a gap and weren't removed. > [...] the spec was written with the idea that /queryChanges would be > implemented based on knowing which records have changed from last > time, not based on having the old query cached and comparing it with > the new one. (This approach is hard to scale effectively.) Hah, definitely. Fortunately, I specifically don't care about scalability. There are some neat tricks to play with storing deltas, and updating recent queries during /set, which I think might make it actually scale. But I'm not bothering to explore them, at least for now. > I'm not quite sure what you mean by "to the same length", but yes if > you have the old query cached you could optimise the case where the > upToId is not in the new results a bit further. But again, the spec > is written to allow implementers to calculate query changes without > caching the old query state. I meant that if the client has cached the first 30 items, then upToId refers to the 30th item they knew about; a delta which gets them back to 30 cached items seems like a reasonable thing to want. I guess my feedback on this point is that it's another area that seems a little more focused on particular implementations than necessary. Which doesn't mean it needs to change, just that it's still on the list of things I'm confused about. I can't quite figure out what client UX the upToId feature supports. I can imagine trying to keep the ten threads starting at #23 visible on the screen, or keep the thread list centered around the thread which is currently being displayed, but there's no reason to expect this will let me do either of those things, right? Maybe if I understood the client side story better this would make more sense to me. > Great! This was really good feedback. I'm glad to hear it! > > Currently all my tests are property-based randomized testing. I'd > > like to think my tests might be interesting for other server > > implementors. If there are test suites I might be able to reuse, > > I'd love to hear about them. > > Getting a good test suite for the JMAP spec is something we would > love to have (and know from past experience is important for > interoperability). I don't think anyone has published an open > reusable one yet. There are a few tests which only rely on core JMAP features that I think would make a good starting point. In particular, I really like that result references can be fully tested using two (or more) Core/echo calls in the same request. > > I've yet to tackle the session state, blobs, or push, but those > > don't seem like they'll have hard questions about semantics, just > > some possibly tricky engineering choices. We'll see how that goes. > > I think queryChanges is the trickiest bit to get your head around > semantically so hopefully now that's done the rest is not too hard, > but do keep us informed of your progress and any other spec issues > you find! Will do. Thank you! Jamey From nobody Sun May 23 22:00:20 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E7EA83A1794 for ; Sun, 23 May 2021 22:00:04 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.797 X-Spam-Level: X-Spam-Status: No, score=-2.797 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=fastmailteam.com header.b=q0fjXw/V; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=m6Z/8Fur Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GFqLgTELYk24 for ; Sun, 23 May 2021 21:59:58 -0700 (PDT) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 918793A1793 for ; Sun, 23 May 2021 21:59:58 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 86B935C0112 for ; Mon, 24 May 2021 00:59:55 -0400 (EDT) Received: from imap42 ([10.202.2.92]) by compute5.internal (MEProxy); Mon, 24 May 2021 00:59:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= fastmailteam.com; h=mime-version:message-id:in-reply-to :references:date:from:to:subject:content-type; s=fm2; bh=KGnR8i9 7aHuDyzhsDwpfL296S504TqM5bomVtLs7Rww=; b=q0fjXw/VgABPiryqyKlCi1R U4FFlV2xPas5+CnPQs3IfFtmoACtORz40u7OLJbODV5w1uU2vgkSzeR3yvIxKzvj oLl29zzf3MNQ1247FvYIh4g8TwEYp94UTt6hJwYZOcBTz1p/ZEfo9/ed5GWz/dER GKBmpdjpvwIXjxvOFdcehMfbIuYbshV6lscuJYftZsf530vAo/4hgZI99MDVfen8 36v96H9XtZRfflubOSoeh3H8WbEaEnwRFkkYwM2TKutuY6pY6bXewA8XUQTsXKuc BT+MtHdQFT6rttBB5UV6x08kN353XpimP67xIG5JL/rE2AlfGnUTIHLfAok+Hvg= = DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=KGnR8i 97aHuDyzhsDwpfL296S504TqM5bomVtLs7Rww=; b=m6Z/8FurEHae/lJKo3Uiu4 utX8PXGQvTERS1WDGwe/1WfdRWc7d6mfdGZsTPOMTaGessto0H4C7rVDG4STIa8b AI/HsDFI/+yRBIrW3NGStM/fH+2MZ9OarPpL0c5yxoD9vHx4OJ0+1mjnibA53OLm IEFlaxtqrbIV1bo44e659/llvE+DI6GRB1ukM+wHZ4nEzrVpA7FBgFEJacg/8M2h Uk7yOvys6luM9LOuKbT05YZzTaodAO2MEffetp7wfdlbCpknRtPGYqScPcamGXxr a/fEeORLdycfwdHg/f18ru1yGeZ1svE7YiHsiSLkOGXn8K04xOi5dlhEclRBbdcQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdejkedgiedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreerjeenucfhrhhomhepfdfpvghi lhculfgvnhhkihhnshdfuceonhgvihhljhesfhgrshhtmhgrihhlthgvrghmrdgtohhmqe enucggtffrrghtthgvrhhnpeehuefhudejtdeiveekvdfhfffgleeflefhfeekhefhkeel kefhfeeufeevffejieenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehnvghilhhjsehfrghsthhmrghilhhtvggrmhdrtghomh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 134C6310005F; Mon, 24 May 2021 00:59:55 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-701-g78bd539edf-fm-ubox-20210517.001-g78bd539e Mime-Version: 1.0 Message-Id: <2fc30015-792a-4545-be1b-49257fec6481@beta.fastmail.com> In-Reply-To: <20210521202012.GC7261@eh> References: <20210521202012.GC7261@eh> Date: Mon, 24 May 2021 14:59:02 +1000 From: "Neil Jenkins" To: "IETF JMAP Mailing List" Content-Type: multipart/alternative; boundary=279631519b3240a58ca8bb7442d1a66b Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 May 2021 05:00:05 -0000 --279631519b3240a58ca8bb7442d1a66b Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sat, 22 May 2021, at 06:20, Jamey Sharp wrote: > Looking at a different bit of the text: >=20 > In the case of records with references to the same type, the serve= r=20 > MUST order the creates and updates within a single method call so=20= > that creates happen before their creation ids are referenced by=20= > another create/update/destroy in the same call. >=20 > I think this wording is a little strange because I don't think destroy= s=20 > can reference creation IDs at all, right? Technically it could destroy a creation in the same request (it's a weir= d edge case, but it was worded that way deliberately); I know we actuall= y came across this scenario at some point, although I can't remember why= just now. > But more importantly: I think a strict reading would prohibit circular= =20 > references in creates, since there is no ordering satisfying this MUST= =20 > in that case. Yes, I would read that the same; I think this paragraph does implicitly = rule out creating circular references directly. > It seems easy enough for a client to use random creation Ids which are= =20 > sufficiently large that they never collide with each other or with=20 > user-provided data. So if it's intended that clients may use strings=20= > which match their own creation Ids, I might just decide this=20 > implementation doesn't interoperate with clients that do that, which=20= > seems okay in a prototyping setting. Hmm, I mean, sure you can do whatever you want for prototyping, but this= is definitely not spec compliant. (My client, for example, just uses an= incrementing integer for creation ids, which avoids reuse as per spec b= ut is definitely not guaranteed to not appear by coincidence in other pr= operties). > "Most recent success" means a client which violates this=20 > "SHOULD" can't predict which object it's going to be referencing, and=20= > the target could even be of a different type than expected. But after=20= > all, you did tell them they shouldn't do that=E2=80=A6 It's a SHOULD because the server behaviour is (in theory) well defined, = and in theory you could do something crazy (e.g. Try creating three diff= erent items then set a reference to the last one that succeeded=E2=80=A6= ). > > The definition of "position" in the response is >=20 > I agree it shouldn't matter, but I think it might be helpful to define= =20 > it any of these equivalent ways: >=20 > - the position immediately after the last item in the list > - the length of the list > - the minimum value of position for which you'd get the same response Sure, I think this value is a reasonable choice. It does mean there are = a few optimisations you can't make though (because it's essentially the = same as forcing the server to calculate the total), although they are un= likely to be important ones I think. > > Close. In this situation you have to invalidate any results in you= r=20 > > sparse query after the first gap, but you can still keep the ones=20= > > before. >=20 > Ohhhh. That makes sense, thanks. The spec is wrong here though, I thin= k: >=20 > The result of this is that if the client has a cached sparse array= of > Foo ids corresponding to the results in the old state, then: >=20 > fooIds =3D [ "id1", "id2", null, null, "id3", "id4", null, null, n= ull ] >=20 > If it *splices out* all ids in the removed array that it has in it= s > cached results, then: >=20 > removed =3D [ "id2", "id31", ... ]; > fooIds =3D> [ "id1", null, null, "id3", "id4", null, null, null= ] >=20 > and *splices in* (one by one in order, starting with the lowest > index) all of the ids in the added array: >=20 > added =3D [{ id: "id5", index: 0, ... }]; > fooIds =3D> [ "id5", "id1", null, null, "id3", "id4", null, null, n= ull ] >=20 > Here, "id31" is shown as removed, which wasn't in fooIds, but "id3" an= d=20 > "id4" come after a gap and weren't removed. Hey damn, you're right! I think my brain glossed over this when proofrea= ding because the line below starts **and truncates**, and in my code the= flag that gets set when you have an unknown destroyed id is called `tru= ncateAtFirstGap`. But the spec is actually just description the final st= ep of adjusting the query results length, not removing the ids after the= gap. Would you like to submit the errata? Or I'm happy to do it. > > I'm not quite sure what you mean by "to the same length", but yes = if=20 > > you have the old query cached you could optimise the case where th= e=20 > > upToId is not in the new results a bit further. But again, the spe= c=20 > > is written to allow implementers to calculate query changes withou= t=20 > > caching the old query state. >=20 > I meant that if the client has cached the first 30 items, then upToId=20= > refers to the 30th item they knew about; a delta which gets them back = to=20 > 30 cached items seems like a reasonable thing to want. > [=E2=80=A6] > I can't quite figure out what client UX the upToId feature supports. I= =20 > can imagine trying to keep the ten threads starting at #23 visible on=20= > the screen, or keep the thread list centered around the thread which i= s=20 > currently being displayed, but there's no reason to expect this will l= et=20 > me do either of those things, right? It's purely an optimisation in both what's sent over the wire and (for c= ertain implementations) how much work the server does. In most scenarios= , a client will have just the start of a list cached. Extreme scenario, = we have the first 3 ids cached from a list of 500 results: [ id1, id2, id3, null4 =E2=80=A6 null500 ] The goal of queryChanges is to ensure that the state of the client after= applying the changes is an accurate representation of the new state on = the server, while sending a minimal delta. Suppose the last 100 items ar= e removed, and a further 50 items are inserted at the end instead. Simpl= y truncating the client's sparse array to the new length (450) means it = is now a valid representation of the new state: every id that's there is= in the correct position. This is what upToId does: instead of sending 5= 0 "added" and 100 "removed" ids you can just send the new total instead.= Cheers, Neil. --279631519b3240a58ca8bb7442d1a66b Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Sat, 22 May = 2021, at 06:20, Jamey Sharp wrote:
Looking at a different bit of the text:

    In the case of records with refe= rences to the same type, the server 
  &nbs= p; MUST order the creates and updates within a single method call so&nbs= p;
    that creates happen before their cre= ation ids are referenced by 
    anoth= er create/update/destroy in the same call.

= I think this wording is a little strange because I don't think destroys&= nbsp;
can reference creation IDs at all, right?
<= /blockquote>

Technically it could destroy a creation = in the same request (it's a weird edge case, but it was worded that way = deliberately); I know we actually came across this scenario at some poin= t, although I can't remember why just now.

But more importantly: I t= hink a strict reading would prohibit circular 
refere= nces in creates, since there is no ordering satisfying this MUST 
in that case.

Yes= , I would read that the same; I think this paragraph does implicitly rul= e out creating circular references directly.

It seems easy enough fo= r a client to use random creation Ids which are 
suff= iciently large that they never collide with each other or with 
=
user-provided data. So if it's intended that clients may use = strings 
which match their own creation Ids, I might = just decide this 
implementation doesn't interoperate= with clients that do that, which 
seems okay in a pr= ototyping setting.

Hmm, I mean= , sure you can do whatever you want for prototyping, but this is definit= ely not spec compliant. (My client, for example, just uses an incrementi= ng integer for creation ids, which avoids reuse as per spec but is defin= itely not guaranteed to not appear by coincidence in other properties).<= br>

<= div>"Most recent success" means a client which violates this 
"SHOULD" can't predict which object it's going to be referencin= g, and 
the target could even be of a different type = than expected. But after 
all, you did tell them they= shouldn't do that=E2=80=A6

It= 's a SHOULD because the server behaviour is (in theory) well defined, an= d in theory you could do something crazy (e.g. Try creating three differ= ent items then set a reference to the last one that succeeded=E2=80=A6).=

=
>   The definition of "position" in the response is

I agree it shouldn't matter, but I think it m= ight be helpful to define 
it any of these equivalent= ways:

- the position immediately after the= last item in the list
- the length of the list
<= div>- the minimum value of position for which you'd get the same respons= e

Sure, I think this value is = a reasonable choice. It does mean there are a few optimisations you can'= t make though (because it's essentially the same as forcing the server t= o calculate the total), although they are unlikely to be important ones = I think.

>   Close. In this situation you have to inval= idate any results in your 
>   sparse qu= ery after the first gap, but you can still keep the ones 
=
>   before.

Ohhhh. That m= akes sense, thanks. The spec is wrong here though, I think:

    The result of this is that if the cl= ient has a cached sparse array of
    Foo i= ds corresponding to the results in the old state, then:
    fooIds =3D [ "id1", "id2", null, null, "= id3", "id4", null, null, null ]

  = ;  If it *splices out* all ids in the removed array that it has in = its
    cached results, then:

       removed =3D [ "id2"= , "id31", ... ];
       fooI= ds =3D> [ "id1", null, null, "id3", "id4", null, null, null ]

    and *splices in* (one by one in= order, starting with the lowest
    index)= all of the ids in the added array:

 &= nbsp; added =3D [{ id: "id5", index: 0, ... }];
 &nbs= p; fooIds =3D> [ "id5", "id1", null, null, "id3", "id4", null, null, = null ]

Here, "id31" is shown as removed, wh= ich wasn't in fooIds, but "id3" and 
"id4" come after= a gap and weren't removed.

He= y damn, you're right! I think my brain glossed over this when proofreadi= ng because the line below starts and truncates, and in my = code the flag that gets set when you have an unknown destroyed id is cal= led truncateAtFirstGap. But the spec is actuall= y just description the final step of adjusting the query results length,= not removing the ids after the gap.

Would = you like to submit the errata? Or I'm happy to do it.

=
> &nbs= p; I'm not quite sure what you mean by "to the same length", but yes if&= nbsp;
>   you have the old query cached you c= ould optimise the case where the 
>   up= ToId is not in the new results a bit further. But again, the spec <= br>
>   is written to allow implementers to calcu= late query changes without 
>   caching = the old query state.

I meant that if the cl= ient has cached the first 30 items, then upToId 
refe= rs to the 30th item they knew about; a delta which gets them back to&nbs= p;
30 cached items seems like a reasonable thing to want.<= br>
[=E2=80=A6]
I can't quite figure out what cl= ient UX the upToId feature supports. I 
can imagine t= rying to keep the ten threads starting at #23 visible on 
=
the screen, or keep the thread list centered around the thread whic= h is 
currently being displayed, but there's no reaso= n to expect this will let 
me do either of those thin= gs, right?

It's purely an opti= misation in both what's sent over the wire and (for certain implementati= ons) how much work the server does. In most scenarios, a client will hav= e just the start of a list cached. Extreme scenario, we have the first 3= ids cached from a list of 500 results:

[ id1, id2, id3, null=
4 =E2=80=A6 null500 ]

The goal of queryChan= ges is to ensure that the state of the client after applying the changes= is an accurate representation of the new state on the server, while sen= ding a minimal delta. Suppose the last 100 items are removed, and a furt= her 50 items are inserted at the end instead. Simply truncating the clie= nt's sparse array to the new length (450) means it is now a valid repres= entation of the new state: every id that's there is in the correct posit= ion. This is what upToId does: instead of sending 50 "added" and 100 "re= moved" ids you can just send the new total instead.

Cheers,
Neil.
--279631519b3240a58ca8bb7442d1a66b-- From nobody Mon May 24 08:54:56 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4EF983A2DB0 for ; Mon, 24 May 2021 08:54:54 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gulbrandsen.priv.no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LqQ7p0WFmVPS for ; Mon, 24 May 2021 08:54:49 -0700 (PDT) Received: from stabil.gulbrandsen.priv.no (stabil.gulbrandsen.priv.no [IPv6:2a01:4f8:191:91a8::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 41EE73A2D8A for ; Mon, 24 May 2021 08:54:48 -0700 (PDT) Received: from stabil.gulbrandsen.priv.no (stabil.gulbrandsen.priv.no [IPv6:2a01:4f8:191:91a8::3]) by stabil.gulbrandsen.priv.no (Postfix) with ESMTP id 5E96AC0051; Mon, 24 May 2021 17:02:43 +0100 (IST) Authentication-Results: stabil.gulbrandsen.priv.no; dmarc=none (p=none dis=none) header.from=gulbrandsen.priv.no Authentication-Results: stabil.gulbrandsen.priv.no; spf=none smtp.mailfrom=arnt@gulbrandsen.priv.no DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gulbrandsen.priv.no; s=mail; t=1621872163; bh=mBlQ3DzqCaC0n71IjO8y+mbzuG0gtu3EA3aO314rLiQ=; h=From:To:Subject:Date:In-Reply-To:References:From; b=CdLi0FeYESGkz51xLP8va4d/mQck8bHwKAMAYc6STS0Nh8YGVGMXamQ+HnR6wGiqp 1aFNlg2bKgFA0qOdTyoMlrAEGmlaausDdfpuw7rUkeczcaYoxk/L2/bW/8GZvlIwcj lPrbh0Y9l/P2dO9asVSYDRH2nTbzQ8pdu/OR0xYI= Received: from arnt@gulbrandsen.priv.no by stabil.gulbrandsen.priv.no (Archiveopteryx 3.2.0) with esmtpsa id 1621872162-14925-14922/9/7; Mon, 24 May 2021 16:02:42 +0000 From: Arnt Gulbrandsen To: jmap@ietf.org Date: Mon, 24 May 2021 17:54:42 +0200 Mime-Version: 1.0 Message-Id: <9eaa7a95-3959-4c5f-afb3-508bd97acbd3@gulbrandsen.priv.no> In-Reply-To: <2fc30015-792a-4545-be1b-49257fec6481@beta.fastmail.com> References: <20210521202012.GC7261@eh> <2fc30015-792a-4545-be1b-49257fec6481@beta.fastmail.com> User-Agent: Trojita/0.7; Qt/5.11.3; xcb; Linux; Devuan GNU/Linux 3 (beowulf) Content-Type: text/plain; charset=utf-8; format=flowed Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 May 2021 15:54:55 -0000 On Monday 24 May 2021 06:59:02 CEST, Neil Jenkins wrote: > Technically it could destroy a creation in the same request > (it's a weird edge case, but it was worded that way > deliberately); I know we actually came across this scenario at > some point, although I can't remember why just now. It seems like a reasonable enough combination when a mobile device leaves airplane mode and a mobile client is playing back stored actions. Arnt From nobody Tue May 25 10:00:26 2021 Return-Path: X-Original-To: jmap@ietf.org Delivered-To: jmap@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 23B003A155A; Tue, 25 May 2021 09:58:44 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: IETF Meeting Session Request Tool To: Cc: fenton@bluepopcorn.net, jmap-chairs@ietf.org, jmap@ietf.org, superuser@gmail.com X-Test-IDTracker: no X-IETF-IDTracker: 7.30.0 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <162196192352.16223.7874095979526639167@ietfa.amsl.com> Date: Tue, 25 May 2021 09:58:44 -0700 Archived-At: Subject: [Jmap] jmap - New Meeting Session Request for IETF 111 X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 May 2021 16:58:44 -0000 A new meeting session request has just been submitted by Jim Fenton, a Chair of the jmap working group. --------------------------------------------------------- Working Group Name: JSON Mail Access Protocol Area Name: Applications and Real-Time Area Session Requester: Jim Fenton Number of Sessions: 1 Length of Session(s): 2 Hours Number of Attendees: 15 Conflicts to Avoid: Chair Conflict: calext extra dispatch Technology Overlap: uta httpbis emailcore dmarc People who must be present: Alexey Melnikov Murray Kucherawy Jim Fenton Neil Jenkins Bron Gondwana Resources Requested: Special Requests: --------------------------------------------------------- From nobody Wed May 26 15:28:48 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9D09E3A1199 for ; Wed, 26 May 2021 15:28:45 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.099 X-Spam-Level: X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=minilop.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jAf6qqSKax0M for ; Wed, 26 May 2021 15:28:41 -0700 (PDT) Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EDD893A1196 for ; Wed, 26 May 2021 15:28:40 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id f22so2148976pgb.9 for ; Wed, 26 May 2021 15:28:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=minilop.net; s=google; h=date:from:to:subject:message-id:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=NDKtihG+H6XHZ/oZgeMw16UVeSSQo3mZVFmNzIJRAtw=; b=eojeXUBYEGvJ/OvYBUZUhbJ6hL75VK5zBxiBwjZFE1HNzz55JmvOlin2AU2h5bCSzW nBgSFT5wHhUcD+JKxQlBLVliYpZypEmNsJfcuRC07jzGUHCOXFlvl/JUGb6FYS/kzVQ+ lgnNyTT+gbKY5odocfL2uJx0I26P5YB1A6mIk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=NDKtihG+H6XHZ/oZgeMw16UVeSSQo3mZVFmNzIJRAtw=; b=EVIiasjtqa0XFsp5qmxhXOqucYyNYKHdpTkD7pMObCs6xRkaJCDrqrT5zqImmBvlhq sCF8iYrHpQWcjbNkEnTdNgFkuN2a9v/Sn2mDcFfuR3dRoZjH1X5ILv7AucAIiNERiYuG SCDGkGwCNQVySkc7DQe/55j8UVOvfYJJ/muSFDkwmGEu336rDVz2HwVzVhbMfhwc74x0 paNrvHLt3RgzY5q3+OS++daUXb4xPy5+tF/My1usuYeBqQwZM2+ykMjKUpA/ozEGKuJ+ pRjzBxyXqtmZuKvY1rC8DQ+iREa8S13JRLsAClmUm7b3g0QUV1vTWXSs6qwyDbVacTqT 1yvg== X-Gm-Message-State: AOAM532es+Wan8R9TMOgJFTP5z6qvdyePCyZHpmVeN3vChESAjQ3UMNh jJPLf7hzQ0i95/q07oEhCE+frTwIVDxqgQ== X-Google-Smtp-Source: ABdhPJzy2VsQlDBQbKXZzTkcano7JJpSAnlzFDUwJ6f/oPgF6sfxqiXLgcRG5/PQL5bdhzDDnM2JTQ== X-Received: by 2002:aa7:9001:0:b029:2d4:9408:9998 with SMTP id m1-20020aa790010000b02902d494089998mr663037pfo.9.1622068118907; Wed, 26 May 2021 15:28:38 -0700 (PDT) Received: from eh ([97.115.152.200]) by smtp.gmail.com with ESMTPSA id u19sm186052pfn.158.2021.05.26.15.28.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 May 2021 15:28:38 -0700 (PDT) Received: by eh (sSMTP sendmail emulation); Wed, 26 May 2021 15:28:36 -0700 Date: Wed, 26 May 2021 15:28:36 -0700 From: Jamey Sharp To: IETF JMAP Mailing List Message-ID: <20210526222836.GA1792@eh> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <9eaa7a95-3959-4c5f-afb3-508bd97acbd3@gulbrandsen.priv.no> <2fc30015-792a-4545-be1b-49257fec6481@beta.fastmail.com> Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 May 2021 22:28:46 -0000 On Mon, May 24, 2021 at 02:59:02PM +1000, Neil Jenkins wrote: >On Sat, 22 May 2021, at 06:20, Jamey Sharp wrote: >>Looking at a different bit of the text: >>    In the case of records with references to the same type, the server  >>    MUST order the creates and updates within a single method call so  >>    that creates happen before their creation ids are referenced by  >>    another create/update/destroy in the same call. >>I think this wording is a little strange because I don't think >>destroys can reference creation IDs at all, right? > >Technically it could destroy a creation in the same request (it's a >weird edge case, but it was worded that way deliberately); I know we >actually came across this scenario at some point, although I can't >remember why just now. On Mon, May 24, 2021 at 05:54:42PM +0200, Arnt Gulbrandsen wrote: >It seems like a reasonable enough combination when a mobile device >leaves airplane mode and a mobile client is playing back stored >actions. I suppose it's not unreasonable, but I can't find any hint in the specification that the keys of `update` or the entries of `destroy` can be creation id references. The most relevant text I can find is this: Some records may hold references to other records (foreign keys). That reference may be set (via create or update) in the same request as the referenced record is created. To do this, the client refers to the new record using its creation id prefixed with a "#". My reading is that creation id references can only appear within a record, either inside the `Foo` in `create`, or inside the `PatchObject` in `update`. Am I missing something? On Mon, May 24, 2021 at 02:59:02PM +1000, Neil Jenkins wrote: >Hmm, I mean, sure you can do whatever you want for prototyping, but >this is definitely not spec compliant. Yeah, fair enough; I'll think about how to make my validation step context-sensitive. >>>Close. In this situation you have to invalidate any results in your >>>sparse query after the first gap, but you can still keep the ones >>>before. >> >>Ohhhh. That makes sense, thanks. The spec is wrong here though, I think: >>[...] >>Here, "id31" is shown as removed, which wasn't in fooIds, but "id3" and  >>"id4" come after a gap and weren't removed. > >Hey damn, you're right! I think my brain glossed over this when >proofreading because the line below starts and truncates, and in my >code the flag that gets set when you have an unknown destroyed id is >called truncateAtFirstGap. But the spec is actually just description >the final step of adjusting the query results length, not removing the >ids after the gap. Would you like to submit the errata? Or I'm happy to >do it. As much as I enjoy the idea of having my name on a contribution to IETF work for the first time, I'm not sure how to word the replacement text, so I think probably you should do it. >>I can't quite figure out what client UX the upToId feature supports. > >It's purely an optimisation in both what's sent over the wire and (for >certain implementations) how much work the server does. In most >scenarios, a client will have just the start of a list cached. Extreme >scenario, we have the first 3 ids cached from a list of 500 results: > >[ id1, id2, id3, null4 … null500 ] > >The goal of queryChanges is to ensure that the state of the client >after applying the changes is an accurate representation of the new >state on the server, while sending a minimal delta. Suppose the last >100 items are removed, and a further 50 items are inserted at the end >instead. Simply truncating the client's sparse array to the new length >(450) means it is now a valid representation of the new state: every id >that's there is in the correct position. This is what upToId does: >instead of sending 50 "added" and 100 "removed" ids you can just send >the new total instead. That's a great example for me to keep in mind but it didn't make me less confused. I've figured out what's bothering me though. Let's say the client has cached a prefix of the results, and then some of those items are destroyed. old: ["A", "B", "C", "D", ...] new: ["B", "D", ...] Say upToId is "D"; you said I should look for it in the new result, where it is at index 1. Then 'any ids that were removed but have a higher index than "upToId" SHOULD be omitted.' Which means the removed list is permitted to contain only "A", right? If so, now the client's cache says ["B", "C", "D"], which is wrong. I imagine current implementations just report all destroyed records as removed whether they were in the old query or not, which would obscure this issue. It would also come up for sorts or filters on mutable properties, but as we've discussed, nobody does that right now. What my intuition was prodding me about is that using the same index in the text of both "added" and "removed" can't be right; which led to me trying to guess client-side UX reasons why somebody might want that, instead of the minimum delta needed for correct operation. Here's what I think are the minimum necessary rules for upToId; I'd appreciate any comments on whether I've got this right: If an "upToId" is supplied and existed in the old results, any ids that were removed but had a higher index than "upToId" previously did SHOULD be omitted. All others MUST NOT be omitted. If an "upToId" is supplied and existed in the old results, the last id which was not removed, and which previously had an index no higher than "upToId" did, is the client's last cached id. Any ids which were added but now have a higher index than the client's last cached id does SHOULD be omitted. All others MUST NOT be omitted. This text might be more clear if written from the point of view of the client's requirements rather than restrictions on the server: If an "upToId" is supplied, any ids which the server is certain did not appear at or before "upToId" in the old results SHOULD be omitted. If an "upToId" is supplied, any ids which are not necessary to return the client's cache to a valid prefix of the new results, after removing the ids given in "removed", SHOULD be omitted. I think any mention of sort/filter mutability here is an unnecessary constraint on server behavior, as is requiring upToId to be present in the new results. But I believe these rules still allow every implementation the current spec allows, aside from prohibiting the situation I described above, since a server is always allowed to ignore upToId. I hope that made sense this time... Jamey From nobody Wed May 26 17:19:34 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6BF333A1787 for ; Wed, 26 May 2021 17:19:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.496 X-Spam-Level: X-Spam-Status: No, score=-1.496 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_SBL=0.5, URIBL_SBL_A=0.1] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=josephg.com header.b=6z/jibNa; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=VJZ+qek1 Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p-cc0VetZjp7 for ; Wed, 26 May 2021 17:19:27 -0700 (PDT) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 18DB43A1784 for ; Wed, 26 May 2021 17:19:26 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id C36D75C00E4 for ; Wed, 26 May 2021 20:19:24 -0400 (EDT) Received: from imap35 ([10.202.2.85]) by compute3.internal (MEProxy); Wed, 26 May 2021 20:19:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=mesmtp; bh=t3POqjGSjDbAcY8PS21mRyVEqKDT VEZbpIOqGcmvBcU=; b=6z/jibNa8Pf98HFHllfexO9s8dJeA+Ih7coEd5cAJhWr tlbt02wOyCqvbkoVChutRSHPEXwSQpL5GBxvNcIDv0E0Zpa18vGMtrhG1NWxZAJR otWFsnKwVnIvhFWmZMPEo9z/pfwYYWihZVTc+xsjrn4Ng4OXE6KA5Iiv5u71aUs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=t3POqj GSjDbAcY8PS21mRyVEqKDTVEZbpIOqGcmvBcU=; b=VJZ+qek1zVExKG9+7oScK5 XXeRu1K58JeWsLHnIYt56TpaSIt+et4Z7HoaPwem7sumHndXLgcnuNKus3jRZ4Pw q+1VypKzH1AOFh+4l7V14ynGQxMychGBVS2zfjIlVytPrglO7LGgn5eezWkjPNSA i60+FO0vGZF0voOCXiUFbrE3xPU0YO+78IzEBBmhyiuTG6J122CRyW4PUOSeqcDw hPjZjbe6ooi9/siuwvDss+K6aaMurwIyEtpCsRQXVH0oGiSeC+JAeW8/4laiGLD6 vnK1qsMs9iHCNza2wXd34eMplyo0pF3IHoamwL8pahyfxTT2/rM4sHSc01avy0TQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdekgedgfeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsegrtd erreerredtnecuhfhrohhmpedfufgvphhhucfivghnthhlvgdfuceomhgvsehjohhsvghp hhhgrdgtohhmqeenucggtffrrghtthgvrhhnpeeghffhkeeuhfevgedvudegheejhffgje fhueevkefgvdehjeeltddukeektdevudenucffohhmrghinhepghhithhhuhgsrdgtohhm necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgvse hjohhsvghphhhgrdgtohhm X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 157D515A0064; Wed, 26 May 2021 20:19:23 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-448-gae190416c7-fm-20210505.004-gae190416 Mime-Version: 1.0 Message-Id: <4dd7074a-3bfd-4bd0-949e-d62d2903e67d@www.fastmail.com> In-Reply-To: References: Date: Thu, 27 May 2021 00:19:02 +0000 From: "Seph Gentle" To: jmap@ietf.org Content-Type: multipart/alternative; boundary=fb5103d2950347eab3a68226d96882cc Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 May 2021 00:19:33 -0000 --fb5103d2950347eab3a68226d96882cc Content-Type: text/plain On Fri, May 14, 2021, at 6:32 PM, Jamey Sharp wrote: > Hey all: for the past few weeks I've been working on a new JMAP server with the goal of making it easy to prototype new data models. I'd love it if folks could take a look and maybe help implement or at least validate it. > > https://github.com/jameysharp/jterritory This is really cool. I wanted to make something like this for email a couple years ago, and got stuck parsing email in a jmap-kind of way. I ended up making this: https://github.com/josephg/mime-to-jmap This is the email envelope -> JMAP code path from cyrus, pulled out into a stand-alone library and compiled to wasm so it can be used in lots of contexts. It might take some work, but you could probably make a simple jmap email server if you want to by combining our efforts. -Seph --fb5103d2950347eab3a68226d96882cc Content-Type: text/html Content-Transfer-Encoding: quoted-printable
On Fri, May 14,= 2021, at 6:32 PM, Jamey Sharp wrote:
Hey all: for the past few w= eeks I've been working on a new JMAP server with the goal of making it e= asy to prototype new data models. I'd love it if folks could take a look= and maybe help implement or at least validate it.


This is really cool. I wanted to make something like this for email a c= ouple years ago, and got stuck parsing email in a jmap-kind of way. I en= ded up making this:


This is the email envelope -> JMAP code= path from cyrus, pulled out into a stand-alone library and compiled to = wasm so it can be used in lots of contexts. It might take some work, but= you could probably make a simple jmap email server if you want to by co= mbining our efforts.

-Seph
= --fb5103d2950347eab3a68226d96882cc-- From nobody Thu May 27 21:48:02 2021 Return-Path: X-Original-To: jmap@ietfa.amsl.com Delivered-To: jmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C8CB73A1714 for ; Thu, 27 May 2021 21:48:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.096 X-Spam-Level: X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=fastmailteam.com header.b=SQDsK/m5; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=sZ3H2CpS Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VOPuWg8LWWIp for ; Thu, 27 May 2021 21:47:56 -0700 (PDT) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C06253A1717 for ; Thu, 27 May 2021 21:47:56 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 562F85C017D for ; Fri, 28 May 2021 00:47:55 -0400 (EDT) Received: from imap43 ([10.202.2.93]) by compute3.internal (MEProxy); Fri, 28 May 2021 00:47:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= fastmailteam.com; h=mime-version:message-id:in-reply-to :references:date:from:to:subject:content-type; s=fm2; bh=gy/zOCZ 3vkGCJBxB1L6q2xlus2YqnUBTeTuKqHP5s1M=; b=SQDsK/m5EudnUIQGjQVWMkd oGnM99LZsymg3ZlOHC5spjUyWVFD/VNDp0swPix3sX0J8T+72fM31e6o+vzoCzl1 LgAaX0oMfHKqrhZ0LaMjeKxa5pHkU4glXrvIvy0ai2vSPkxcCv9YacGnjTXOzPnS SRdqsPZEJ9rFn6t9vXFPEIazu/yhqvZVnRFAjjXviYMfqazSF7J5PxfvE3eodrYF aUwIuAp+UsspwvPALwqVeE+ib3M4bRZ4FzZKXbCH8XswC5WkaNNxpPjXRYZeWZuI 4lICJG6gmStohsyynLoXkFyEy4aIXL/1gZXp2saZx1syf8EwKLZrlpCuBov1Upg= = DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=gy/zOC Z3vkGCJBxB1L6q2xlus2YqnUBTeTuKqHP5s1M=; b=sZ3H2CpSXWUhnPCkBffv+G SfGbg6ovaM6km3a9aywIdTxn6Fpq3sfcgQN6V7ucyCH67T6+qcQUCrU2ZidMhPJP gH9J7V1oQof7dX0WvJAM7EHSRnFh9VrBhUr8p2DMIzrAejQM0D7WRqTDjL9U8JEV JVdXXFQn3rWlhgBQB/p9Y3yx5VkoPj/swC+fqqiMcNgFtgw+WH7VtjNPClwjEhzV IJf6bYP8j/u42ssLMTN9OCjbdUkJMh7o5Qf5SYULrrFHV5b/dhwZJNRSBjMF6Hsw 55xSu+B+d+ALN8pXWaf1aUsvDD0ArSXMzNueiYi9RaA26E21b6xZVCWiSEeyFZGA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdekiedgkeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreerjeenucfhrhhomhepfdfpvghi lhculfgvnhhkihhnshdfuceonhgvihhljhesfhgrshhtmhgrihhlthgvrghmrdgtohhmqe enucggtffrrghtthgvrhhnpeehuefhudejtdeiveekvdfhfffgleeflefhfeekhefhkeel kefhfeeufeevffejieenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehnvghilhhjsehfrghsthhmrghilhhtvggrmhdrtghomh X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id DDF5CAC005B; Fri, 28 May 2021 00:47:54 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-701-g78bd539edf-fm-ubox-20210517.001-g78bd539e Mime-Version: 1.0 Message-Id: <590c785d-7e1a-420e-9f2a-5989e6a76c04@dogfood.fastmail.com> In-Reply-To: <20210526222836.GA1792@eh> References: <20210526222836.GA1792@eh> Date: Fri, 28 May 2021 14:47:14 +1000 From: "Neil Jenkins" To: "IETF JMAP Mailing List" Content-Type: multipart/alternative; boundary=5a85fbaa7f6d4eb5ae2e1fa04b768865 Archived-At: Subject: Re: [Jmap] new JMAP server for prototyping X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 May 2021 04:48:02 -0000 --5a85fbaa7f6d4eb5ae2e1fa04b768865 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Thu, 27 May 2021, at 08:28, Jamey Sharp wrote: > I suppose it's not unreasonable, but I can't find any hint in the=20 > specification that the keys of `update` or the entries of `destroy` ca= n=20 > be creation id references. The most relevant text I can find is this: >=20 > Some records may hold references to other records (foreign keys).=20= > That reference may be set (via create or update) in the same reque= st=20 > as the referenced record is created. To do this, the client refer= s=20 > to the new record using its creation id prefixed with a "#". >=20 > My reading is that creation id references can only appear within a=20 > record, either inside the `Foo` in `create`, or inside the `PatchObjec= t`=20 > in `update`. Am I missing something? Hmm, yes, it looks like we never made it explicit in the spec. It's a ra= re edge case, but possibly worth an errata too to be explicit. > That's a great example for me to keep in mind but it didn't make me le= ss=20 > confused. I've figured out what's bothering me though. >=20 > Let's say the client has cached a prefix of the results, and then some= =20 > of those items are destroyed. >=20 > old: ["A", "B", "C", "D", ...] > new: ["B", "D", ...] >=20 > Say upToId is "D"; you said I should look for it in the new result,=20= > where it is at index 1. Then 'any ids that were removed but have a=20 > higher index than "upToId" SHOULD be omitted.' Which means the removed= =20 > list is permitted to contain only "A", right? That was not the intention, but I agree the spec is misleading here, to = the point I think it is worth doing an errata; I'll do that too. > Here's what I think are the minimum necessary rules for upToId; I'd=20= > appreciate any comments on whether I've got this right: >=20 > If an "upToId" is supplied and existed in the old results, any id= s=20 > that were removed but had a higher index than "upToId" previously= =20 > did SHOULD be omitted. All others MUST NOT be omitted. >=20 > If an "upToId" is supplied and existed in the old results, the la= st=20 > id which was not removed, and which previously had an index no=20= > higher than "upToId" did, is the client's last cached id. Any ids= =20 > which were added but now have a higher index than the client's la= st=20 > cached id does SHOULD be omitted. All others MUST NOT be omitted.= >=20 > This text might be more clear if written from the point of view of the= =20 > client's requirements rather than restrictions on the server: >=20 > If an "upToId" is supplied, any ids which the server is certain d= id=20 > not appear at or before "upToId" in the old results SHOULD be=20 > omitted. >=20 > If an "upToId" is supplied, any ids which are not necessary to=20= > return the client's cache to a valid prefix of the new results,=20= > after removing the ids given in "removed", SHOULD be omitted. That's absolutely correct, and was the intention here. (The description in the spec is kind-of true, but only because it's grou= nded in the idea of a particular implementation where you have "tombston= e" records for the deleted items that you can sort into the new query, s= o it's the index of *those* that are lower than the upToId. It's not tha= t important, the spec is wrong and we need an errata, but I know why we = thought it made sense when it was written!) > I think any mention of sort/filter mutability here is an unnecessary=20= > constraint on server behavior, as is requiring upToId to be present in= =20 > the new results. I agree. That's particular implementation details leaking into the spec = again. Thanks again for the great feedback! As an Area Director put it to me, i= f there's no erratas it doesn't mean a spec is perfect, it just means no= one's reading it. =F0=9F=99=82 Neil. --5a85fbaa7f6d4eb5ae2e1fa04b768865 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Thu, 27= May 2021, at 08:28, Jamey Sharp wrote:
I suppose it's not unreasonable, but I can'= t find any hint in the 
specification that the keys o= f `update` or the entries of `destroy` can 
be creati= on id references. The most relevant text I can find is this:

    Some records may hold references to= other records (foreign keys). 
    Th= at reference may be set (via create or update) in the same request =
    as the referenced record is created.&n= bsp; To do this, the client refers 
   = ; to the new record using its creation id prefixed with a "#".
=

My reading is that creation id references can only a= ppear within a 
record, either inside the `Foo` in `c= reate`, or inside the `PatchObject` 
in `update`. Am = I missing something?

Hmm, yes,= it looks like we never made it explicit in the spec. It's a rare edge c= ase, but possibly worth an errata too to be explicit.

=
That's a great= example for me to keep in mind but it didn't make me less 
confused. I've figured out what's bothering me though.
<= div>
Let's say the client has cached a prefix of the resul= ts, and then some 
of those items are destroyed.
<= /div>

old: ["A", "B", "C", "D", ...]
ne= w: ["B", "D", ...]

Say upToId is "D"; you s= aid I should look for it in the new result, 
where it= is at index 1. Then 'any ids that were removed but have a 
higher index than "upToId" SHOULD be omitted.' Which means the re= moved 
list is permitted to contain only "A", right?<= br>

That was not the intention, bu= t I agree the spec is misleading here, to the point I think it is worth = doing an errata; I'll do that too.

Here's what I think are the minim= um necessary rules for upToId; I'd 
appreciate any co= mments on whether I've got this right:

&nbs= p;    If an "upToId" is supplied and existed in the old r= esults, any ids 
     that were r= emoved but had a higher index than "upToId" previously 
     did SHOULD be omitted. All others MUST NOT b= e omitted.

     If an "= upToId" is supplied and existed in the old results, the last 
     id which was not removed, and which pr= eviously had an index no 
     hi= gher than "upToId" did, is the client's last cached id. Any ids 
     which were added but now have a hig= her index than the client's last 
   &= nbsp; cached id does SHOULD be omitted. All others MUST NOT be omitted.<= br>

This text might be more clear if written fr= om the point of view of the 
client's requirements ra= ther than restrictions on the server:

 = ;    If an "upToId" is supplied, any ids which the server= is certain did 
     not appear = at or before "upToId" in the old results SHOULD be 
&= nbsp;    omitted.

 &nbs= p;   If an "upToId" is supplied, any ids which are not necessa= ry to 
     return the client's c= ache to a valid prefix of the new results, 
 &nb= sp;   after removing the ids given in "removed", SHOULD be omi= tted.

That's absolutely correc= t, and was the intention here.

(The de= scription in the spec is kind-of true, but only because it's grounded in= the idea of a particular implementation where you have "tombstone" reco= rds for the deleted items that you can sort into the new query, so it's = the index of those that are lower than the upToId. It's not that = important, the spec is wrong and we need an errata, but I know why we th= ought it made sense when it was written!)

=
I think any mention = of sort/filter mutability here is an unnecessary 
con= straint on server behavior, as is requiring upToId to be present in = ;
the new results.

I agree. That's particular implementation details leaking into the sp= ec again.

Thanks again for the great feedba= ck! As an Area Director put it to me, if there's no erratas it doesn't m= ean a spec is perfect, it just means no one's reading it. =F0=9F=99=82

Neil.
--5a85fbaa7f6d4eb5ae2e1fa04b768865-- From nobody Fri May 28 03:25:14 2021 Return-Path: X-Original-To: jmap@ietf.org Delivered-To: jmap@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 279CF3A2386; Fri, 28 May 2021 03:25:10 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: jmap@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.30.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: jmap@ietf.org Message-ID: <162219751002.24262.6378691655607035223@ietfa.amsl.com> Date: Fri, 28 May 2021 03:25:10 -0700 Archived-At: Subject: [Jmap] I-D Action: draft-ietf-jmap-jscontact-06.txt X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 May 2021 10:25:11 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the JSON Mail Access Protocol WG of the IETF. Title : JSContact: A JSON representation of contact data Authors : Robert Stepanek Mario Loffredo Filename : draft-ietf-jmap-jscontact-06.txt Pages : 22 Date : 2021-05-28 Abstract: This specification defines a data model and JSON representation of contact card information that can be used for data storage and exchange in address book or directory applications. It aims to be an alternative to the vCard data format and to be unambiguous, extendable and simple to process. In contrast to the JSON-based jCard format, it is not a direct mapping from the vCard data model and expands semantics where appropriate. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-jmap-jscontact/ There is also an HTML version available at: https://www.ietf.org/archive/id/draft-ietf-jmap-jscontact-06.html A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-jmap-jscontact-06 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Fri May 28 06:40:18 2021 Return-Path: X-Original-To: jmap@ietf.org Delivered-To: jmap@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id EDCD83A293E; Fri, 28 May 2021 06:40:15 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: jmap@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.30.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: jmap@ietf.org Message-ID: <162220921589.8666.11706079827323062461@ietfa.amsl.com> Date: Fri, 28 May 2021 06:40:15 -0700 Archived-At: Subject: [Jmap] I-D Action: draft-ietf-jmap-jscontact-vcard-04.txt X-BeenThere: jmap@ietf.org X-Mailman-Version: 2.1.29 List-Id: JSON Message Access Protocol List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 May 2021 13:40:16 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the JSON Mail Access Protocol WG of the IETF. Title : JSContact: Converting from and to vCard Authors : Mario Loffredo Robert Stepanek Filename : draft-ietf-jmap-jscontact-vcard-04.txt Pages : 41 Date : 2021-05-28 Abstract: This document defines how to convert contact information as defined in the JSContact [draft-ietf-jmap-jscontact] specification from and to vCard. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-jmap-jscontact-vcard/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-jmap-jscontact-vcard-04 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-jmap-jscontact-vcard-04 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/