Subject: Re: [Netconf] Fwd: I-D Action: draft-bierman-netconf-restconf-03.txt

From: Wojciech Dec <[email-masked]>
Date: Mon, 6 Jan 2014 12:33:11 +0100

Hi Andy, all,

are the issues leading to this draft  documented somewhere? The IETF
88 minutes only talk about the yang patch aspect.

Anyway, I took a read through the latest document and the change to
have all Yang data-nodes be resources. Am I correct in interpreting it
that now  every leaf node  effectively becomes a resource with a
separate URI? Could the authors provide some more insight regarding
this change?

An unfortunate characteristic, introduced in -02, that still persists
in -03 is the configuration and operational split. But we discussed
that on a separate thread....

Regards,
Wojciech.

On 21 December 2013 21:26, Andy Bierman <[email-masked]> wrote:
> FYI,
>
> This version of RESTCONF addresses many of the issues raised at the last
> IETF.
> See the change log for details.
>
>
> Andy
>
>
> ---------- Forwarded message ----------
> From: <[email-masked]>
> Date: Sat, Dec 21, 2013 at 12:23 PM
> Subject: I-D Action: draft-bierman-netconf-restconf-03.txt
> To: [email-masked]
>
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
>
>
>         Title           : RESTCONF Protocol
>         Authors         : Andy Bierman
>                           Martin Bjorklund
>                           Kent Watsen
>                           Rex Fernando
>         Filename        : draft-bierman-netconf-restconf-03.txt
>         Pages           : 95
>         Date            : 2013-12-21
>
> Abstract:
>    This document describes a RESTful protocol that provides a
>    programmatic interface over HTTP for accessing data defined in YANG,
>    using the datastores defined in NETCONF.
>
>
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-bierman-netconf-restconf/
>
> There's also a htmlized version available at:
> http://tools.ietf.org/html/draft-bierman-netconf-restconf-03
>
> A diff from the previous version is available at:
> http://www.ietf.org/rfcdiff?url2=draft-bierman-netconf-restconf-03
>
>
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>
> _______________________________________________
> I-D-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i-d-announce
> Internet-Draft directories: http://www.ietf.org/shadow.html
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 6 Jan 2014 03:49:49 -0800

> separate URI? Could the authors provide some more insight regarding

Since YANG Patch is now optional, there is no way to delete an optional leaf
or leaf-list otherwise, except to copy the entire resource, and then replace
the entire resource (minus the optional leaf).

> that on a separate thread....

The split is removed. The content query parameter is used to pick
config, non-config or both.

Andy

---

From: Wojciech Dec <[email-masked]>
Date: Wed, 8 Jan 2014 11:35:17 +0100

> the entire resource (minus the optional leaf).

I am still not able to get the full rationale for the change.  Can the
authors or chairs provide that?

Anyway, it now appears that every single data leaf is a resource,
instead of an attribute, and the spec doesn't specify a distinction
between handling parent resources and its sub-resources, e.g. At the
very least POST/PUT operations to sub resources need to be constrained
by their parent resource, and leaving that up to the implementation is
kind of a step backwards for the spec as a whole besides being IMO a
major complication for client or server, and likely both e.g how
should a change to a sub-resource that doesn't meet some condition of
the parent be handled? For a single parent resource, how should
multiple sub-resource changes be coordinated (the parent resource
needs to be consistent)? Etc.

If this current development was driven by questions/problems in the
support of HTTP Patch operation (incl. JSON patch), the solution
appears to be possibly worse than the supposed problem. That's why it
would be good to understand the rationale some more.

> config, non-config or both.
Ah, yes, see that now. Glad that email discussion helped.

Thanks.

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 08 Jan 2014 12:29:39 +0100 (CET)

Wojciech Dec <[email-masked]> wrote:
> instead of an attribute

Yes, every leaf is a subresource to its parent list or container.
This just means that you can GET/POST/DELETE the leaf directly, w/o
having to PATCH the parent.

> and the spec doesn't specify a distinction

I am afraid I do not understand your concern.  Could you provide an
example (data model and requests) that you feel is problematic or
unclear?

> would be good to understand the rationale some more.

If the "yang patch" media type is opitonal, there is no other way to
delete optional leafs.  If the optional leaf is a resource it can be
removed with DELETE.

/martin

---

From: Wojciech Dec <[email-masked]>
Date: Fri, 10 Jan 2014 10:37:59 +0100

Hi Martin,

> example (data model and requests) that you feel is problematic or

A simple example:

    container book {

                leaf price {
                     type string;
                 }
                leaf tax-amount {
                     type string;
                 }

     }

Price and taxt are typically related.
A (better) REST API design would seek to minimize transactional
effects to the client while protecting the consistency/sanity of the
data: To update the resource, a POST operation to foo/book would carry
in the envelope both a new price and the tax amount. A (worse) REST
API design would expose both price and tax-amount as separate
resources, accept POST to both foo/book/price and foo/book/tax-amount
and hope-for-the-best that the client succeeds and all. Several non
trivial failuire scenarios come up here too.

The key is that REST API design is very much about determining what is
a resource,  its representation by a URI, and what are the attributes
of a resource. In draft -03, everything is now a resource, and
everything is also attribute. This IMO ultimately complicates and
bloats code (on client, server, and likely both), and will lead to
brittle API and poor user experience.

Another quick example:

    container book {

                list page {
                    key page-nr;
                    leaf page-nr {type string;}
                    leaf text {type string;}
                 }
     }

The RESTConf URI for the above would <root stuff here>/book/page/{page-nr}/text

In general with REST APIs it is important that we do NOT expose the
dependent sub-resources directly thus allowing someone to create (POST
or PUT) to <root stuff here>/book/page/{page-nr}  without a book, or
text without a page, or other cases that do not make sense, and in
general requiring a feast of error cases.
Requiring the text POST/PUT  handler to also do book creation,
validation, etc, is not a great design. It complicates code and
creates ugly code dependencies, should the model change, etc.

> delete optional leafs.  If the optional leaf is a resource it can be

Yes, but the current "solution" is IMO a lot worse than a) mandating
PATCH, and I mean plain JSON/XML PATCH  or b) handling (specifying in
the draft how-to do) updates via POST. Thus bringing in back the
simple patch would be a good move. And the Yang patch is something
that can go into another spec, I agree.

Cheers,
Wojciech.

---

From: Andy Bierman <[email-masked]>
Date: Fri, 10 Jan 2014 02:37:54 -0800

> that can go into another spec, I agree.

Plain PATCH does not allow an optional leaf to be deleted.
JSON patch does not work well with named data like YANG.
Ignoring YANG naming and using integer indices that renumber
every time a list is changed is a non-starter.  The only way to
delete an optional leaf is to GET the entire parent resource,
edit it offline (remove the leaf) and PUT the parent resource.
This is operationally disruptive and expensive to implement
in the server.  Optional leafs make up most configuration data,
so a CM solution that doesn't work well for optional leafs is a bad idea.

Andy

---

From: Wojciech Dec <[email-masked]>
Date: Fri, 10 Jan 2014 14:39:16 +0100

> so a CM solution that doesn't work well for optional leafs is a bad idea.

I should have been clearer: Media Type discussion aside, I meant JSON
patch as in "json patch for json reresented yang data", ie likely a
subset of pure 'application/json-patch+json', with yang-patch being in
some other doc. . The fact that JSON patch does not work for xml is,
for me at least, not an issue. A simple answer can thus be that the
patch is for json only.
In general, I don't quite follow why you say that json patch doesn't
allow for leaf deletion; the "remove" op seems to be pretty much
intended for that. Gee, the json (yang) patch could even be just
limited to just this task.

That said, a solution to the " how to delete an optional leaf" problem
that ends up treating every data item in the system to a resource,
with no way for the designer to override, appears also as a bad idea.
Given that we have a full API spec here, even adding a custom
parameterized operation to the URI may be a better solution. Also,
evidence from the REST API world out there also indicates that for
better or worse, GET and PUT do actually work at scale. I would still
see this still as easier to deal with and implement than having every
single (sub-resource) data item as a resource....

So, in summary, "conservative" options, all leading to getting the
spec back from resource extremes as I see can be:
1. Stay with GET and PUT (I suspect that this is what will get most use anyway).
2. Introduce JSON patching, in the form of say
"application/yang-json-patch" to the spec (happy to work up some text)
3. Get back the "full" yang patch from draft -02
4. Introduce a new action/operation invoked via a parameter.

Thoughts?

Cheers.

---

From: Wojciech Dec <[email-masked]>
Date: Fri, 10 Jan 2014 16:13:04 +0100

> 4. Introduce a new action/operation invoked via a parameter.

Forgot to add one more:
5. Introduce meta-data to handle the partial update.

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 10 Jan 2014 16:25:29 +0100 (CET)

> trivial failuire scenarios come up here too.

Note that with the current design, you get both, and the client can
choose to use the resource that fits his needs best.  Specifically,
book is still a resource, so if the client wants to update both leafs
in one transaction, it would POST to the book resource.

> bloats code (on client, server, and likely both)

In our server we actually had special code to handle the case that
leafs were not resources.  So in our case the code is now simpler.

On the client side I do not know.  But again note that the client can
choose to treat all leafs as attributes if it wants to.

> and will lead to
> text without a page, or other cases that do not make sense

Without a book doesn't work since a page is defined under the book.
In general, such constraints should be expressed in the data model
(with proper containment and/or must / unique / mandatory etc
constraints).  This separates the semantic correctness of the instance
data from the specific API details used to change the underyling
data.  This is especially true of we want to support multiple
protocols (which is what we're doing with RESTCONF).

> , and in
> validation, etc, is not a great design.

Agreed, but that is not the intention.

> It complicates code and

I agree that we should have some mandatory way of patching that also
can remove optional leafs.  It is important that we can make arbitrary
changes in one transaction.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Fri, 10 Jan 2014 08:30:50 -0800

On Fri, Jan 10, 2014 at 5:39 AM, Wojciech Dec <[email-masked]> wrote:

> >> >> >> have all Yang data-nodes be resources. Am I correct in
> interpreting
> >> >> >> it
> >> >> > or leaf-list otherwise, except to copy the entire resource, and
> then
> >> >> > replace
JSON Patch does not work for NETCONF datastores very well
because they contain data named with YANG list keys.  It has
nothing to do with XML.  It has to do with the problems related
to editing foo[i] instead of foo[test1][row2][step3], where "i" is
constantly changing for a given resource.

> single (sub-resource) data item as a resource....
As Martin pointed out, your app can choose not to treat leafs as
resources and everything still works.  What will break in your
application code if you choose to treat leafs as attributes instead
of resources?

IMO, an optional leaf or leaf-list is a sub-resource because it can be
created
and deleted after the parent resource is created.

> 4. Introduce a new action/operation invoked via a parameter.
I don't understand this list, especially 2 & 3.

Andy

---

From: Wojciech Dec <[email-masked]>
Date: Fri, 10 Jan 2014 19:42:00 +0100

Hi Martin,

> in one transaction, it would POST to the book resource.

Well, ok, but how can the server assume what he client chooses to do,
other than supporting every possibility. Or alternatively, how can we
tell the client: "Although RESTCONF spec + YANG model might indicate
otherwise, leafs are not resources in our implementation"?

> protocols (which is what we're doing with RESTCONF).

Yeah, but how would one do that...

> Agreed, but that is not the intention.

... without ending up with the above, or assuming a very very specific
data store ? Or perhaps, that's the thing I'm not understanding: what
is the responsibility of RESTconf layer and what is the responsibility
of the data store in terms of the yang schema?

Cheers,
Wojciech.

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 10 Jan 2014 19:48:18 +0100 (CET)

> otherwise, leafs are not resources in our implementation"?

The server still has to support the relevant operations on the leaf
resources, but the client can choose to not use them if it wants to.

See also below.

> of the data store in terms of the yang schema?

If the server gets a request from a client to set the title of a book:

  PUT /.../book/{book-name}/page/{page-nr}/text

and the book doesn't exist, the server signal an error.  It will not
automatically create the book.

(this example made the book a list, not a container)

/martin

---

From: Wojciech Dec <[email-masked]>
Date: Sat, 11 Jan 2014 13:03:47 +0100

> automatically create the book.

Sure, I know that this should happen, but what is the responsibility
of RESTconf layer and what is the responsibility
of the data store in terms of the yang schema?
Currently it appears that Restconf assumes that the whole resource
containment (model enforcement) is assumed to be done by the data
store.

/Wojciech.

---

From: Martin Bjorklund <[email-masked]>
Date: Sat, 11 Jan 2014 13:09:35 +0100 (CET)

Wojciech Dec <[email-masked]> wrote:
> >> >> >> > Since YANG Patch is now optional, there is no way to delete an
> >> >> >> > optional
> >> >> >> > leaf

This is up to the implementation.  In a multi-protocol implementation
one might think that the underlying datastore takes care of the "model
enforcement", but nothing prevents you from doing everything in the
RESTCONF layer.

/martin