Subject: [Json] JSON merge alternatives

From: Nico Williams <[email-masked]>
Date: Wed, 19 Mar 2014 18:45:51 -0500

First, IMO the WG should in fact adopt a WG item (or more) for MIME
types for use with PATCH to patch JSON texts (or entities with JSON
representations).  This is commonly known as "JSON merge", IIUC.

I work with one Rails application that has an ad-hoc JSON merge like
schema for representing updates (PUTs) to resources -- this schema is
very specific to the application in question, therefore not reusable.

I can imagine a Ruby gem that handles a JSON merge PATCH in a completely
general and reusable way.  And not just Ruby, of course.

Second, I believe the JSON merge proposals I've seen to date are no
good.

The goal of any Internet JSON merge schema should be to:

   Concisely and generally express "edits" to a JSON text, efficiently
   allowing:

    - replacement of any values
    - prepend/insert/replace/append to arrays
    - addition/removal of names from objects
    - setting null values to object names

   no matter how deeply (subject to a max) or shallowly nested these
   values be in the original, while being easy to produce and apply.
   
Note: it helps to use If-Match to make sure that the resource being
PATCHed has not been modified since it was fetched.  This allows one to
express edits with confidence.

I have two proposals: one based on augmenting a subset of the original
JSON text, the other based a sequence of paths and edit instructions for
the values found at those paths.

Proposal #1: Patch as augmented subset of the original

Follow these steps to produce a patch:

 - Start with the resource's JSON representation.

 - Remove all interior object name/value pairs that are not on the way
   to any value to be added/replaced/removed/otherwise edited.

 - Remove all interior array elements that are not on the way to any
   value to be added/replaced/removed UNLESS the element is in an array
   to be edited (more on this below).

 - Insert before each remaining array element its index.

 - To replace any value with a scalar, just write the new value where
   the old one appeared.

 - To append to any array, just append -1 and the new value to it (after
   the above removals and additions).

 - For any array you want to do more to than append -> wrap the array in
   an object with any of these named values:

    o "delete":  [<list of array indices>]
    o "prepend": [<list of values to append>]
    o "insert":  [<list of [<index>, <value>]>]
    o "set":     [<list of [<index>, <value>]>]
    o "new":     [<list of new elements to replace the old one's with>]
    o "edits":   [<list of edits (see below)>]

   To disambiguate these objects from objects that could legitimately
   appear in the resource the following named value MUST also be
   present:

    o "<URN TBD>": true

   Replacements to be performed first, then deletions, then insertions,
   then prepends, then "edits".  Edits are what would have appeared
   had there been no need to wrap the array in order to do delete,
   prepend, replace, or insert elements.

 - To add name/value pairs to objects, just add them.

 - If you want to delete values from objects -> wrap the object in an
   object with any of these names:

    o "delete": [<list of name strings>]
    o "edits":  <object containint edits>

   Edits are what would have appeared had there been no need to wrap the
   object to delete name/value pairs.

   To disambiguate these names the following named value MUST also be
   present:

    o "<URN TBD>": true

Values to be added/set appear as they should appear in the resource as
patched.

Patch application is straightforward, obvious.

Example resource:

  {
    "a": [ { "b": 1, "c": 2 }, { "d": { "e": [ 2, {} ] } } ],
    "z": [ true, false, null, null, "hello" ]
  }

Example edits:

 - Delete that empty object (path a[0].d.e[1] in jq-speak)

   { "a": [ 1, { "d": { "e": { "delete": [ 1 ],
                               "urn:ietf.org:TBD": true } } } ] }

 - Replace the array [ 2, {} ] (path a[0].d.e in jq-speak) with the
   value 0:

   { "a": [ 1, { "d": { "e": 0 } } ] }

 - Append true to the array [ 2, {} ] (path a[0].d.e in jq-speak):

   { "a": [ 1, { "d": { "e": [ true ] } } ] }

 - Delete the 1, prepend "foo", append "bar" to that same array:

   { "a": [ 1, { "d": { "e": { "delete": [ 0 ],
                               "prepend": [ "foo" ],
                               "edits": [ "bar" ],
                               "urn:ietf.org:TBD": true } } } ] }

 - Add a sibling name to "b" and "c":

   { "a": [ 0, { "f": "hello" } ] }

 - Delete "b":

   { "a": [ 0, { "delete": [ "b" ], "urn:ietf.org:TBD": true } ] }

 - Replace the "z" array with [1, 2, 3]:

   { "z": { "new": [ 1, 2, 3, ], "urn:ietf.org:TBD": true } }

Proposal #2: Use sequence of [<path>, <edit instruction>]

 - Write the path to each value to be edited as an array of path
   elements (strings for object names, numbers for array elements).

 - To replace any value with a scalar value (non-array, non-object),
   just write the path to the value and the new value as the
   instruction.

 - To add a value to an object just write the path to the object, append
   the new value's name to the path, and the instruction will be the new
   value.

 - To add (append) a value to an array, write the path to the array,
   append a -1 to the path, and the instruction will be the new value.

 - Edit instructions for arrays (other than appending to, or replacing
   the whole array with a scalar value) are any objects like:

    o "delete":  [<list of indices>] or true (if path names an array
                                        element)

    o "add":     [<list of <value>s to append]

    o "insert":  [<list of [<index>, <value>] pairs>]
    o "set":     [<list of [<index>, <value>] pairs>]
    o "prepend": [<list of <value>s to prepend]
    o "add":     [<list of <value>s to append]
    o "new":     [<new list of values to replace old ones>]

   No magic URN name is needed in this case.

 - To add an object

 - Edit instructions for objects (other than replacement with a scalar
   value) are objects with name/value pairs like:

    o "delete": [<list of name strings>]
    o "merge":  [<object whose named value pairs will replace the
                 corresponding ones of the object being edited>]

   No magic URN name is needed in this case.

The totality of the patch, then, is an array of
[ <path>, <instruction> ] elements.

Any given path could be referenced multiple times in one patch.

Proposal #2 is pretty self-explanatory.  Examples left as an exercise
for the reader.

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Mar 2014 10:56:52 +1100

Hi Nico,

  http://mailarchive.ietf.org/arch/msg/apps-discuss/mvUVvUt41tFTrPSOCfv7sfblTUM

Cheers,

On 20 Mar 2014, at 10:45 am, Nico Williams <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

--
Mark Nottingham   http://www.mnot.net/

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Mar 2014 19:01:16 -0500

>   http://mailarchive.ietf.org/arch/msg/apps-discuss/mvUVvUt41tFTrPSOCfv7sfblTUM

I.e., draft-ietf-appsawg-json-merge-patch, which has the same problems
mentioned recently on this list (e.g., using null to mean delete).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Mar 2014 19:05:07 -0500

I was responding to the question about whether the JSON WG should
re-charter to include JSON merge as a work item.

I suppose perhaps you're saying that JSON merge is an Apps area work
item and should stay in Apps.  But I can't quite tell if that's what
you meant.

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Mar 2014 11:12:55 +1100

> mentioned recently on this list (e.g., using null to mean delete).

... and it's looking for an editor. Why not just take it over and gain consensus there?

> item and should stay in Apps.  But I can't quite tell if that's what

I don't like venue shopping. json-merge-patch is still technically an active item in APPSAWG AIUI, so if you don't like it, you should discuss it over there.

Cheers,

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Mar 2014 19:21:06 -0500

> I don't like venue shopping. json-merge-patch is still technically an active item in APPSAWG AIUI, so if you don't like it, you should discuss it over there.

Thanks for clarifying.  The question of whether to have it as a JSON
WG work item was raised by someone else, and I had no idea that it
existed as an Apps area work item.  (Yes, I could and should have
noticed, but having missed it I wouldn't have been able to read your
mind either :)

As for venue, I don't care, and since it's an existing work item for
Apps, that disposes of the question as far as I'm concerned.  Others
here might disagree.

I might volunteer to take it over if there was consensus for a
proposal that I liked.  I'll resend my proposals to apps-discuss.

Thanks,

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Mar 2014 11:23:26 +1100

> proposal that I liked.  I'll resend my proposals to apps-discuss.

Cool -- I'd like to see *something* in that area progress.

Cheers,

---

From: "Manger, James" <[email-masked]>
Date: Thu, 20 Mar 2014 11:35:05 +1100

RFC 6902 "JavaScript Object Notation (JSON) Patch" looks quite close to proposal 2.
Media type: application/json-patch+json
Operations: add, remove, replace, move, copy, test

draft-ietf-appsawg-json-merge-patch "JSON Merge Patch" is a simpler subset of proposal 1. It is designed to work well with JSON objects, and objects nested in objects etc. Its main limitation (compared to proposal 1) is that it doesn't support insert/replace for individual array elements.

Adding a "merge" operation to RFC 6902 would be another way to get the best of both these approaches.

--
James Manger

-----Original Message-----
From: json [mailto:[email-masked]] On Behalf Of Nico Williams
Sent: Thursday, 20 March 2014 10:46 AM
To: [email-masked]
Subject: [Json] JSON merge alternatives

   

    o "<URN TBD>": true

    o "<URN TBD>": true

Example resource:

  {

Example edits:

 - Delete "b":

 - To add an object

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Mar 2014 19:42:19 -0500

> Operations: add, remove, replace, move, copy, test

Perfect.  Now I don't have to edit any I-Ds for this.

> draft-ietf-appsawg-json-merge-patch "JSON Merge Patch" is a simpler subset of proposal 1. It is designed to work well with JSON objects, and objects nested in objects etc. Its main limitation (comp…

Right.  Why not just drop draft-ietf-appsawg-json-merge-patch?

> Adding a "merge" operation to RFC 6902 would be another way to get the best of both these approaches.

Maybe, but I don't need it.

Thanks for the pointer to RFC6902!

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Thu, 20 Mar 2014 11:55:02 +1100

> Right.  Why not just drop draft-ietf-appsawg-json-merge-patch?

Because it is creates intuitive and succinct patches for many most common cases.

--
James Manger

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Mar 2014 21:34:24 -0500

> Because it is creates intuitive and succinct patches for many most common cases.

Well, sure, but it should be possible to take an original and a patch
written in one style and convert to the other style locally, then
PATCH using the one style.

Having just one JSON PATCH schema is more likely to interop than having two.

Of course, the alternative style might still be useful even if it were
only useful locally.

I'll leave it to someone else, methinks, at least for now.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Mar 2014 08:23:30 +0100

> item and should stay in Apps.  But I can't quite tell if that's what

For the record: I believe this work is needed, but that it should happen 
over here. This WG has the necessary JSON expertise.

Best regards, Julian

---

From: Stefan Drees <[email-masked]>
Date: Thu, 20 Mar 2014 08:32:00 +0100

On 2014-03-20 08:23 +01:00, Julian Reschke wrote:
> over here. This WG has the necessary JSON expertise. ...

+1 for necessary format expertise when fiddling with format differencing

All the best,
Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 20 Mar 2014 08:06:50 -0700

> mentioned recently on this list (e.g., using null to mean delete).

Then those problems should be discussed more. There was at least one thread about "null to delete", but I don't think it got enough focus.

As others have said: the document is already in another WG, there is no need to bring it to this WG. If you want to speak about the document, definitely join the AppsAWG mailing list at <https://www.ietf.org/mailman/listinfo/apps-discuss>. Also, if you want to volunteer to edit the document, saying so there would be great.

Do note that that WG already has lots of relevant readers: people who write applications protocols that use JSON. Having both them and JSON partisans in the discussion would be a good thing.

Matt and I are not looking at empire-building and see no reason to take this document from an active WG just to put it here. One of the great things about the IETF is that many people participate on (or at least lazily follow) many mailing lists.

--Paul Hoffman