Subject: [Netconf] RESTCONF Resource Model (was Re: Fwd: I-D Action: draft-bierman-netconf-restconf-03.txt)

From: Andy Bierman <[email-masked]>
Date: Fri, 10 Jan 2014 10:03:33 -0800

Hi,

The resource management model is a fairly important part of
the RESTCONF architecture.  Perhaps all attempts so far have
been too simplistic.

The YANG node type seems to be irrelevant to answering
the question "what is a sub-resource?"  The set of mandatory[1]
descendant nodes are not sub-resources of a given parent.
They must exist and cannot be created and deleted
independently of the parent resource. So a sub-resource
is really any descendant sub-tree that can be created and
deleted independent of the parent resource.

IMO there should not be a complex set of rules for restricting URIs
and HTTP methods, based on mandatory vs. optional nodes.
This is a conceptual distinction which does not need to be
enforced somehow in the protocol.

It seems arbitrary to say that a resource target URI can point
at a container/list but not a leaf/leaf-list.  The server is going
to use the YANG validation rules to determine if an operation
is allowed.  DELETE will fail for a mandatory container, same
as a leaf.

[1] mandatory can be complex:
   static: mandatory, min-elements
   dynamic: if-feature, when + <static>

Andy

On Fri, Jan 10, 2014 at 7:13 AM, Wojciech Dec <[email-masked]> wrote:

> On 10 January 2014 14:39, Wojciech Dec <[email-masked]> wrote:
> > On 10 January 2014 11:37, Andy Bierman <[email-masked]> wrote:
> >>
> >>
> >>
> >> On Fri, Jan 10, 2014 at 1:37 AM, Wojciech Dec <[email-masked]>
> wrote:
> >>>
> >>> Hi Martin,
> >>>
> >>> On 8 January 2014 12:29, Martin Bjorklund <[email-masked]> wrote:
> >>> > Wojciech Dec <[email-masked]> wrote:
> >>> >> On 6 January 2014 12:49, Andy Bierman <[email-masked]> wrote:
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Jan 6, 2014 at 3:33 AM, Wojciech Dec <[email-masked]
> >
> >>> >> > wrote:
> >>> >> >>
> >>> >> >> Hi Andy, all,
> >>> >> >>
> >>> >> >> are the issues leading to this draft  documented somewhere? The
> IETF
> >>> >> >> 88 minutes only talk about the yang patch aspect.
> >>> >> >>
> >>> >> >> Anyway, I took a read through the latest document and the change
> to
> >>> >> >> have all Yang data-nodes be resources. Am I correct in
> interpreting
> >>> >> >> it
> >>> >> >> that now  every leaf node  effectively becomes a resource with a
> >>> >> >> separate URI? Could the authors provide some more insight
> regarding
> >>> >> >> this change?
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > Since YANG Patch is now optional, there is no way to delete an
> >>> >> > optional leaf
> >>> >> > or leaf-list otherwise, except to copy the entire resource, and
> then
> >>> >> > replace
> >>> >> > the entire resource (minus the optional leaf).
> >>> >>
> >>> >>
> >>> >> I am still not able to get the full rationale for the change.  Can
> the
> >>> >> authors or chairs provide that?
> >>> >>
> >>> >> Anyway, it now appears that every single data leaf is a resource,
> >>> >> instead of an attribute
> >>> >
> >>> > Yes, every leaf is a subresource to its parent list or container.
> >>> > This just means that you can GET/POST/DELETE the leaf directly, w/o
> >>> > having to PATCH the parent.
> >>> >
> >>> >> and the spec doesn't specify a distinction
> >>> >> between handling parent resources and its sub-resources, e.g. At the
> >>> >> very least POST/PUT operations to sub resources need to be
> constrained
> >>> >> by their parent resource, and leaving that up to the implementation
> is
> >>> >> kind of a step backwards for the spec as a whole besides being IMO a
> >>> >> major complication for client or server, and likely both e.g how
> >>> >> should a change to a sub-resource that doesn't meet some condition
> of
> >>> >> the parent be handled? For a single parent resource, how should
> >>> >> multiple sub-resource changes be coordinated (the parent resource
> >>> >> needs to be consistent)? Etc.
> >>> >
> >>> > I am afraid I do not understand your concern.  Could you provide an
> >>> > example (data model and requests) that you feel is problematic or
> >>> > unclear?
> >>>
> >>>
> >>> A simple example:
> >>>
> >>>     container book {
> >>>
> >>>                 leaf price {
> >>>                      type string;
> >>>                  }
> >>>                 leaf tax-amount {
> >>>                      type string;
> >>>                  }
> >>>
> >>>      }
> >>>
> >>> Price and taxt are typically related.
> >>> A (better) REST API design would seek to minimize transactional
> >>> effects to the client while protecting the consistency/sanity of the
> >>> data: To update the resource, a POST operation to foo/book would carry
> >>> in the envelope both a new price and the tax amount. A (worse) REST
> >>> API design would expose both price and tax-amount as separate
> >>> resources, accept POST to both foo/book/price and foo/book/tax-amount
> >>> and hope-for-the-best that the client succeeds and all. Several non
> >>> trivial failuire scenarios come up here too.
> >>>
> >>> The key is that REST API design is very much about determining what is
> >>> a resource,  its representation by a URI, and what are the attributes
> >>> of a resource. In draft -03, everything is now a resource, and
> >>> everything is also attribute. This IMO ultimately complicates and
> >>> bloats code (on client, server, and likely both), and will lead to
> >>> brittle API and poor user experience.
> >>>
> >>> Another quick example:
> >>>
> >>>
> >>>     container book {
> >>>
> >>>                 list page {
> >>>                     key page-nr;
> >>>                     leaf page-nr {type string;}
> >>>                     leaf text {type string;}
> >>>                  }
> >>>      }
> >>>
> >>> The RESTConf URI for the above would <root stuff
> >>> here>/book/page/{page-nr}/text
> >>>
> >>> In general with REST APIs it is important that we do NOT expose the
> >>> dependent sub-resources directly thus allowing someone to create (POST
> >>> or PUT) to <root stuff here>/book/page/{page-nr}  without a book, or
> >>> text without a page, or other cases that do not make sense, and in
> >>> general requiring a feast of error cases.
> >>> Requiring the text POST/PUT  handler to also do book creation,
> >>> validation, etc, is not a great design. It complicates code and
> >>> creates ugly code dependencies, should the model change, etc.
> >>>
> >>>
> >>> >
> >>> >> If this current development was driven by questions/problems in the
> >>> >> support of HTTP Patch operation (incl. JSON patch), the solution
> >>> >> appears to be possibly worse than the supposed problem. That's why
> it
> >>> >> would be good to understand the rationale some more.
> >>> >
> >>> > If the "yang patch" media type is opitonal, there is no other way to
> >>> > delete optional leafs.  If the optional leaf is a resource it can be
> >>> > removed with DELETE.
> >>>
> >>> Yes, but the current "solution" is IMO a lot worse than a) mandating
> >>> PATCH, and I mean plain JSON/XML PATCH  or b) handling (specifying in
> >>> the draft how-to do) updates via POST. Thus bringing in back the
> >>> simple patch would be a good move. And the Yang patch is something
> >>> that can go into another spec, I agree.
> >>>
> >>
> >>
> >> Plain PATCH does not allow an optional leaf to be deleted.
> >> JSON patch does not work well with named data like YANG.
> >> Ignoring YANG naming and using integer indices that renumber
> >> every time a list is changed is a non-starter.  The only way to
> >> delete an optional leaf is to GET the entire parent resource,
> >> edit it offline (remove the leaf) and PUT the parent resource.
> >> This is operationally disruptive and expensive to implement
> >> in the server.  Optional leafs make up most configuration data,
> >> so a CM solution that doesn't work well for optional leafs is a bad
> idea.
> >
> > I should have been clearer: Media Type discussion aside, I meant JSON
> > patch as in "json patch for json reresented yang data", ie likely a
> > subset of pure 'application/json-patch+json', with yang-patch being in
> > some other doc. . The fact that JSON patch does not work for xml is,
> > for me at least, not an issue. A simple answer can thus be that the
> > patch is for json only.
> > In general, I don't quite follow why you say that json patch doesn't
> > allow for leaf deletion; the "remove" op seems to be pretty much
> > intended for that. Gee, the json (yang) patch could even be just
> > limited to just this task.
> >
> > That said, a solution to the " how to delete an optional leaf" problem
> > that ends up treating every data item in the system to a resource,
> > with no way for the designer to override, appears also as a bad idea.
> > Given that we have a full API spec here, even adding a custom
> > parameterized operation to the URI may be a better solution. Also,
> > evidence from the REST API world out there also indicates that for
> > better or worse, GET and PUT do actually work at scale. I would still
> > see this still as easier to deal with and implement than having every
> > single (sub-resource) data item as a resource....
> >
> > So, in summary, "conservative" options, all leading to getting the
> > spec back from resource extremes as I see can be:
> > 1. Stay with GET and PUT (I suspect that this is what will get most use
> anyway).
> > 2. Introduce JSON patching, in the form of say
> > "application/yang-json-patch" to the spec (happy to work up some text)
> > 3. Get back the "full" yang patch from draft -02
> > 4. Introduce a new action/operation invoked via a parameter.
>
> Forgot to add one more:
> 5. Introduce meta-data to handle the partial update.
>
> >
> > Thoughts?
> >
> > Cheers.
> >>
> >>
> >>>
> >>> Cheers,
> >>> Wojciech.
> >>
> >>
> >>
> >> Andy
> >>
> >>>
> >>> >
> >>> > /martin
> >>
> >>
>

---

From: Wojciech Dec <[email-masked]>
Date: Tue, 14 Jan 2014 21:13:26 +0100

> enforced somehow in the protocol.

The problem is that in designing an API, it's generally crucial to
allow the designer to control how the API will be consumed. The -03
Restconf spec removes most of such controls by not distinguishing
between a resource and its sub-resources: every possible data item is
a URI resource possibly exposed to the HTTP verbs, and likewise every
resource is a data item of its parent resource. Relying on the client
in "knowing" the "right way" in which to consume an API will lead to
poor API experience(s) and breakages like leading to corrupt data,
besides likely gross deviation from REST principles. An Example of the
corruption problem was/is presented below in the original text in the
"update tax-amount along with price" example. Relying on the client to
"do the right thing" is problematic, if the history of API usage is
anything to go by.

The way I see it: Draft -02 did not have this issue, at least not as a
major issue. Now, the issue was seemingly introduced in draft -03 to
address another problem, the deletion of an optional leaf/data-item,
for which there are other reasonable solutions that do not lead to the
issue in draft -03. Summarized, the solutions to these partial deletes
are:

1. Stay with GET and PUT as the means to update (even if sub-optimal
from a data size transfer - they work for most APIs)
2. Introduce a subset of regular JSON patching, in the form of a new
media-type, that provides specifically the "delete" operation
3. Indicate a "full" yang patch, as was done in draft -02 (or
whichever draft that text lives in now).
4. Introduce a new action/operation resource (eg
test.com/restconf/jukebox/songs/delete)
5. Define a new meta-data header for the partial delete operation.

That said, there is another solution, which however involves Yang: 6.
Define in Yang a meta-parameter which designates what containers or
lists are to be URIs or have their end-leaf resources "directly"
exposed.
Surely, some other options can also come into play, but in essence, I
think at least the above options need to be looked at.
An eventual solution to these problems would be for implementations to
"manually" mark up their code, in effect steering or staying away from
Restconf, or leaving it confined to some very narrow usage/audience
which would be a shame.

Regards,
Wojciech.

>> >>> >> > On Mon, Jan 6, 2014 at 3:33 AM, Wojciech Dec
>> >>> >> > <[email-masked]>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi Andy, all,
>> >>> >> >>

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 14 Jan 2014 22:33:29 +0100 (CET)

Wojciech Dec <[email-masked]> wrote:
> allow the designer to control how the API will be consumed.

Agreed.

> The -03
> "update tax-amount along with price" example.

As I already wrote, the proper YANG-way to handle this case is to
provide constraint in the data model, in the form of must
expressions.  Just requiring the client to always send both values
doesn't guarantee anything.

> Relying on the client to

Right, so this is why constraint enforcement in the server is
important.

> from a data size transfer - they work for most APIs)

This means that if you want to delete two optional leafs in two
different resources, you have to do two PUTs.  This means two
different transactions.

> whichever draft that text lives in now).

Yes this might be an option.

> test.com/restconf/jukebox/songs/delete)

This has the same problem as 1.

> 5. Define a new meta-data header for the partial delete operation.

Is this restful / httpful?

> lists are to be URIs or have their end-leaf resources "directly"

This is something we have discussed from the very start as well.  So
far we haven't seen any user requirements for it.  However, it doesn't
solve the delete problem.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 Jan 2014 14:19:12 -0800

> doesn't guarantee anything.

I think a client could be developed that viewed all YANG terminals
as attributes and all lists and containers as resources.  I do not
see the value in this approach, because in order to edit a resource
the app really needs to know the YANG definition of the resource.

A generic "create_resource" approach based on containers/leafs, etc.
may not work because the must/min-elements/mandatory type of
constraints will be enforced by the server, even if they are ignored
by the client.

> > issue in draft -03. Summarized, the solutions to these partial deletes

I still do not understand the issue.
Declaring containers and lists to be resources and leaf, leaf-list
to be attributes does not account for the YANG constraints. I don't
see how preventing the server from accepting a DELETE on
a leaf or leaf-list breaks any clients.  If your client doesn't
want to use DELETE and prefers to use YANG Patch or PUT instead,
that still works.

Andy

> > >> >> On Fri, Jan 10, 2014 at 1:37 AM, Wojciech Dec <[email-masked]
> >
> > >> >> wrote:
> > >> >>>
> > >> >>> Hi Martin,
> > >> >>>
> > >> >>> >> between handling parent resources and its sub-resources, e.g.
> At
> > >> >>> >> the
> > >> >>> >> kind of a step backwards for the spec as a whole besides being
> IMO
> > >> >>> >> a

---

From: Wojciech Dec <[email-masked]>
Date: Wed, 15 Jan 2014 10:34:59 +0100

> the app really needs to know the YANG definition of the resource.

Yes, given our previous discussion on URI's, etc, I see that in the
generic case apps would require to also know the YANG definition too,
but whether the app chooses to update a composite resource going
sub-resource by sub-resource or addressing the resource is something
that the YANG model doesn't say. As mentioned, typically the API
designer has the levers to guide this sort of behavior.

> want to use DELETE and prefers to use YANG Patch or PUT instead,

Well, that's because it is not the client that breaks, but rather,
actions by the client can (and actually case will, coz we've seen this
in other REST APIs) corrupt the state of the data-store/lead to
inconsistent data on the server. Since the sub resources are directly
accessible, they could get updated directly and asynchronously
irrespective of any related sub-resources. Should a client fail after
doing the "PUT/update price" but before "PUT/update tax-amount", the
store will be in trouble.
A side problem in this is also what should clients accessing the
parent resource see while these "sub transactions" are happening.
With REST, this problem is much easier to deal with when the
PUT/update operation is done at the parent level.

This type of problem is sometimes "solved" in some (supposedly)
RESTful services by locking the resource (ie trying to do a
synchronous change), locking related resources, etc.

Regards,
Wojciech.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Jan 2014 02:20:53 -0800

> designer has the levers to guide this sort of behavior.

   container my-resource {
       leaf a { type string; }
       list b {
          min-elements 1;
          key x;
          leaf x { type string; }
       }
   }

Here is an example that breaks the simple "lists are sub-resources" design.
The client code cannot assume that it always has to provide child leafs and
never
has to provide child lists. The server will reject requests to create
my-resource
unless it has at least 1 instance of list b.  The client cannot ignore the
YANG constraints.

> store will be in trouble.

YANG defines what needs to be present for the resource to be valid.
Nothing short of understanding the YANG constraints is going to
prevent an app from sending invalid requests.

Andy

> >> > >> >>> >> >> are the issues leading to this draft  documented
> somewhere?
> >> > >> >>> >> >> The
> >> > >> >>> >> >> IETF
> >> > >> >>> >> > Since YANG Patch is now optional, there is no way to
> delete
> >> > >> >>> >> > an
> >> > >> >>> >> > optional leaf
> >> > >> >>> >> > or leaf-list otherwise, except to copy the entire
> resource,
> >> > >> >>> >> > and
> >> > >> >>> >> > then
> >> > >> >>> >> > replace
> >> > >> >>> >> I am still not able to get the full rationale for the
> change.
> >> > >> >>> >> Can
> >> > >> >>> >> the
> >> > >> >>> > This just means that you can GET/POST/DELETE the leaf
> directly,
> >> > >> >>> > w/o
> >> > >> >>> > example (data model and requests) that you feel is
> problematic
> >> > >> >>> > or
> >> > >> >>> > unclear?
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> A simple example:
> >> > >> >>>
> >> > >> >>>     container book {
> >> > >> >>>
> >> > >> >>>                 leaf price {
> >> > >> >>>                      type string;
> >> > >> >>>                  }
> >> > >> >>>                 leaf tax-amount {
> >> > >> >>>                      type string;
> >> > >> >>>                  }
> >> > >> >>>
> >> > >> >>>      }
> >> > >> >>>
> >> > >> >>> effects to the client while protecting the consistency/sanity
> of
> >> > >> >>> the
> >> > >> >>> data: To update the resource, a POST operation to foo/book
> would
> >> > >> >>> carry
> >> > >> >>> dependent sub-resources directly thus allowing someone to
> create
> >> > >> >>> (POST
> >> > >> >>> or PUT) to <root stuff here>/book/page/{page-nr}  without a
> book,
> >> > >> >>> or
> >> > >> >>> >> appears to be possibly worse than the supposed problem.
> That's
> >> > >> >>> >> why
> >> > >> >>> >> it
> >> > >> > subset of pure 'application/json-patch+json', with yang-patch
> being
> >> > >> > in
> >> > >> > 1. Stay with GET and PUT (I suspect that this is what will get
> most
> >> > >> > use
> >> > >> > anyway).

---

From: Kent Watsen <[email-masked]>
Date: Wed, 15 Jan 2014 17:57:27 +0000

Hi Wojciech,

As I see it, the access-model defined in the current RESTCONF proposal is
very similar NETCONF.  Both provide a hierarchical tree of data that
clients can get/edit/delete at any level.  In NETCONF, the "verbs" are
<get-config> (w/ subtree filtering) and <edit-config> (with operations),
while in HTTP its POST/GET/PUT/DELETE and an optional PATCH.  In both
cases the client needs to know what it's doing and the server enforces
constraints defined by the YANG model.   Is there a significant difference
I'm missing?

Thanks,
Kent

---

From: Wojciech Dec <[email-masked]>
Date: Thu, 16 Jan 2014 14:30:13 +0100

Hi Kent,

well, NETCONF provides a good deal more than is typically encountered,
or sensible in RESTful designs, eg the "rollback-on-error".
Candidly, I don't think that RESTConf is or should be an HTTP based
equivalent of Netconf. If it is to be so then a title like HTTPCONF
may be better.

Regards,
Wojciech.

---

From: Wojciech Dec <[email-masked]>
Date: Thu, 16 Jan 2014 14:41:29 +0100

> prevent an app from sending invalid requests.

Think you're answering a different question . The case/problem I
highlighted was a problem of API design, whereby the Restconf Spec
doesn't give any control to the API designer.
A (well) designed API for the simplistic "update book price &
tax-amount" use-case would see (by design) the user  send in a
PUT/update book {price: $10, tax-amount: $2}, rather than allow all
three combinations of updating that/those resource(s). I.e. the price
is not a free resource, but a characteristic (sub-resource) of the
book.

Cheers

>> >> > >> >>> >> >> Anyway, I took a read through the latest document and
>> >> > >> >>> >> >> the
>> >> > >> >>> >> >> change
>> >> > >> >>> >> >> to
>> >> > >> >>> >> >> that now  every leaf node  effectively becomes a
>> >> > >> >>> >> >> resource
>> >> > >> >>> >> >> with a
>> >> > >> >>> >> major complication for client or server, and likely both
>> >> > >> >>> >> e.g
>> >> > >> >>> >> how
>> >> > >> >>> and hope-for-the-best that the client succeeds and all.
>> >> > >> >>> Several
>> >> > >> >>> non
>> >> > >> >>> bloats code (on client, server, and likely both), and will
>> >> > >> >>> lead
>> >> > >> >>> to
>> >> > >> >>> In general with REST APIs it is important that we do NOT
>> >> > >> >>> expose
>> >> > >> >>> the
>> >> > >> >>> text without a page, or other cases that do not make sense,
>> >> > >> >>> and
>> >> > >> >>> in
>> >> > >> >>> >> If this current development was driven by
>> >> > >> >>> >> questions/problems
>> >> > >> >>> >> in the
>> >> > >> >>> > If the "yang patch" media type is opitonal, there is no
>> >> > >> >>> > other
>> >> > >> >>> > way to
>> >> > >> >>> > delete optional leafs.  If the optional leaf is a resource
>> >> > >> >>> > it
>> >> > >> >>> > can be
>> >> > >> >>> > removed with DELETE.
>> >> > >> >>>

---

From: Kent Watsen <[email-masked]>
Date: Thu, 16 Jan 2014 18:40:56 +0000

>or sensible in RESTful designs, eg the "rollback-on-error".

That doesn't address the point I was trying to make, let me try again.
You're having concern with an API strategy and I pointed out that it's
essentially the same as what we've been doing with NETCONF for some time
now.  It seems that if the issue was significant, it would've been raised
before, but that isn't the case, which suggests this is a non-issue,
unless there is a significant difference between the two.  Can you
identify one?

FWIW, RESTCONF doesn't support rollback-on-error, since it relies on
YANG-based enforcement, a luxury NETCONF didn't start with.

Thanks,
Kent

---

From: Wojciech Dec <[email-masked]>
Date: Fri, 17 Jan 2014 13:23:36 +0100

> unless there is a significant difference between the two.  Can you

Quoting from previous mail. Consider the following

     container book {

                 leaf price {
                      type string;
                 }
                 leaf tax-amount {
                      type string;
                 }

     }

Price and tax-amount are related, and both exist only because of the book.
A (proper) REST API design  would seek to minimize transactional (ill)
effects to the client while protecting the consistency/sanity of the
data: To update the book resource, a POST operation to foo/book would
carry in the envelope both a new price and the tax amount. A (worse)
REST API design would expose both price and tax-amount as separate
resources, subject to separate POST operations, which is what  the
changes in draft -03 lead to (and draft -01, -02 didn't)... These can
fail and leave inconsistent data around, etc.

> YANG-based enforcement, a luxury NETCONF didn't start with.

??
NETCONF provides a protocol for elaborate CRUD operations, including
"luxuries" (as you put it) of roll-back, which quite definitely would
be usable in the above update case should one want to do it with
Netconf.
As stated previously, the change in the API design in -03 introduced
this issue to address a problem that can be dealt with through other
means.

Cheers.

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 17 Jan 2014 13:28:07 +0100 (CET)

Wojciech Dec <[email-masked]> wrote:
> carry in the envelope both a new price and the tax amount.

The current RESTCONF document does this.

> A (worse)
> resources, subject to separate POST operations

The current RESTCONF document does this as well.

The client can choose the method it wants to use.

I think we're going in circles...

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Jan 2014 13:30:48 +0100

> Price and tax-amount are related, and both exist only because of the book.

Sorry, this particular example doesn’t work for me. In my country, VAT rate changes every 6 months on the average, so for our bookstore owners it might make a very good sense to be able to adjust tax separately.

Lada

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C