Subject: Re: [Netconf] In an update, when is a delete a delete?

From: Kent Watsen <[email-masked]>
Date: Thu, 18 May 2017 21:15:48 +0000

I'm unsure if this is a good example, but assuming you have data model snippet:

  list foo {
    key name;
    leaf name { type string;}
  }

and the on-change subscription snippet:

   /foo[name="x]

and the following occurs in time:

  T0: there are no 'foo' entries
  T1: foo entry w/ name=="y" is created
  T2: foo entry w/ name=="x" is created
  T3: foo entry "x" is renamed to "z"
  T4: foo entry "y" is renamed to "x"
  T5: foo entry "x" is deleted

I'd expect:

  T0: nothing
  T1: nothing
  T2: "x" created notification
  T3: "x" modified-out notification
  T4: "x" modified-in notification
  T5: "x" deleted notification

I think that this is your #2.

Kent  // as a contributor

On 5/17/17, 5:29 PM, "Netconf on behalf of Alexander Clemm" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Hello all,

In updating the YANG-Push document (draft-ietf-netconf-yang-push), we have come across one issue that we wanted to raise with the working group.

As part of an on-change subscription, update records reflect the type of change (e.g. whether the value of an object has changed, or whether an object was created or deleted); a subscription allows also to specify whether interested only in specific types of changes (for example, only creates and deleted but no value changes).

At the same time, a subscription filter specifies which objects to include as part of a subscription and which not.  (Really, it is not so much of a “filter” on a stream that is generated independently of the filter, than it is a policy of which objects to include as part of subscribed update records.)   However, a subscription filter (such as XPath) can be used to also specify a value filter, which will include or exclude objects based on their current value. This makes it possible to e.g. subscribe to an object “foo” but only if its value is 5.

Now, this means that the same object could be included in one update, but excluded in another update, due to its value no longer meeting the filter criteria.  For example, if foo’s value changes from 5 to 3 in one cycle, a periodic subscription will no longer include foo in its next update.  The question now concerns how to properly handle this in the case of an on-change subscription.

One possibility concerns reporting the fact that “foo” no longer meets the subscription criteria and is no longer included in the update record as a “delete” event.  If foo’s value again becomes “5” at a later point in time, that would be reported as a “create” event.  If foo’s value changes again from 5 at a later point in time and then changes back to 3  before the time of the update (perhaps because the value changed during the dampening interval), it would be reported as another “delete” event (without ever reporting a create event).  On the other hand, if foo’s value changed from 3 to 6 and back again, nothing would be reported because it did not meet the filter criteria at any point in time.

From the perspective of the receiver this may make sense if it is synching its copy of the state.  However, from the perspective of the publisher, the object was never created or deleted – only its value changed, and the case when the object was truly created or deleted can no longer be distinguished from the case when its value changed.  A “create” simply means “an object now meets a filter criteria, that was not reported in the previous cycle” (which does not mean that the object was actually created – it may have been created, or it may have simply undergone a value change).

An alternative (let’s call it alternative 2) is therefore to make a distinction between whether an object was created or deleted, or whether its value fell in or out of a filter range.  This appears semantically cleaner.  However, it will require modifying the encoding to allow for distinction between those cases (currently, just plain patch encoding is used).

A third alternative is to let filters select only data nodes to subscribe to, and separate out the value filter (or disallow it as a feature altogether).  This alternative has the drawback of being less conceptually powerful, even if it may be easier to implement.

Thoughts?  Any preferences between 1, 2, and 3?
--- Alex

---

From: Andy Bierman <[email-masked]>
Date: Thu, 18 May 2017 14:52:51 -0700

>   T4: foo entry "y" is renamed to "x"

Do you mean some proprietary non-standard rename operation?
Because NETCONF and RESTCONF have only delete(x) and create(y)

>   T4: "x" modified-in notification

T3 and T4 events are not possible in NETCONF or RESTCONF.
There is no way to rename a data node. These protocols will
produce delete(x) and a create(y) events

> I think that this is your #2.

Andy

>
>
> On 5/17/17, 5:29 PM, "Netconf on behalf of Alexander Clemm" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>
>
> Hello all,
>
>
>
> In updating the YANG-Push document (draft-ietf-netconf-yang-push), we
> have come across one issue that we wanted to raise with the working group.
>
>
>
> As part of an on-change subscription, update records reflect the type of
> change (e.g. whether the value of an object has changed, or whether an
> object was created or deleted); a subscription allows also to specify
> whether interested only in specific types of changes (for example, only
> creates and deleted but no value changes).
>
>
>
> At the same time, a subscription filter specifies which objects to include
> as part of a subscription and which not.  (Really, it is not so much of a
> “filter” on a stream that is generated independently of the filter, than it
> is a policy of which objects to include as part of subscribed update
> records.)   However, a subscription filter (such as XPath) can be used to
> also specify a value filter, which will include or exclude objects based on
> their current value. This makes it possible to e.g. subscribe to an object
> “foo” but only if its value is 5.
>
>
>
> Now, this means that the same object could be included in one update, but
> excluded in another update, due to its value no longer meeting the filter
> criteria.  For example, if foo’s value changes from 5 to 3 in one cycle, a
> periodic subscription will no longer include foo in its next update.  The
> question now concerns how to properly handle this in the case of an
> on-change subscription.
>
>
>
> One possibility concerns reporting the fact that “foo” no longer meets the
> subscription criteria and is no longer included in the update record as a
> “delete” event.  If foo’s value again becomes “5” at a later point in time,
> that would be reported as a “create” event.  If foo’s value changes again
> from 5 at a later point in time and then changes back to 3  before the time
> of the update (perhaps because the value changed during the dampening
> interval), it would be reported as another “delete” event (without ever
> reporting a create event).  On the other hand, if foo’s value changed from
> 3 to 6 and back again, nothing would be reported because it did not meet
> the filter criteria at any point in time.
>
>
>
> From the perspective of the receiver this may make sense if it is synching
> its copy of the state.  However, from the perspective of the publisher, the
> object was never created or deleted – only its value changed, and the case
> when the object was truly created or deleted can no longer be distinguished
> from the case when its value changed.  A “create” simply means “an object
> now meets a filter criteria, that was not reported in the previous cycle”
> (which does not mean that the object was actually created – it may have
> been created, or it may have simply undergone a value change).
>
>
>
> An alternative (let’s call it alternative 2) is therefore to make a
> distinction between whether an object was created or deleted, or whether
> its value fell in or out of a filter range.  This appears semantically
> cleaner.  However, it will require modifying the encoding to allow for
> distinction between those cases (currently, just plain patch encoding is
> used).
>
>
>
> A third alternative is to let filters select only data nodes to subscribe
> to, and separate out the value filter (or disallow it as a feature
> altogether).  This alternative has the drawback of being less conceptually
> powerful, even if it may be easier to implement.
>
>
>
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>
>

---

From: Alexander Clemm <[email-masked]>
Date: Fri, 19 May 2017 00:48:25 +0000

So, I think so far the consensus is #1 is not a good idea.  So we can probably drop #1 and should choose between #2 and #3.

Personally, I would tend to agree with leaving RMON-type stuff separate, i.e. use a filter to select data nodes, not apply additional comparators for values.  However, if a filter allows to refer to state/values, then really the subscription could cover that.  I am not sure if a general warning would be sufficient to not issue such requests.  A client might issue an according request, but a publisher could feel free to reject a subscription request if the filter construct requests something that is not supported.  However, if the publisher could accept the request, then we run into the encoding issue that we need to distinguish why an object was omitted (or included): because it was deleted (created), or because its value no longer matches (or does match) the filter.

I interpret the “modified-in” and “modified-out” suggested by Kent precisely as a distinction between those cases.  “Modified-out” simply means its value changed so it no longer matches the filter (but the object still exists).  Whether we call it modified-in/out or something else is another question.  Of course, the issue is that this means this is a new case that is not covered by the patch syntax, which would require an extension of the encoding.

I find #3 simpler and more straightforward, but I can see applications for #2 and it results in the more general and potentially powerful mechanism.  Perhaps #2 then?

--- Alex

From: Andy Bierman [mailto:[email-masked]]
Sent: Thursday, May 18, 2017 2:53 PM
To: Kent Watsen <[email-masked]>
Cc: Alexander Clemm <[email-masked]>; Netconf <[email-masked]>
Subject: Re: [Netconf] In an update, when is a delete a delete?

On Thu, May 18, 2017 at 2:15 PM, Kent Watsen <[email-masked]<mailto:[email-masked]>> wrote:
I'm unsure if this is a good example, but assuming you have data model snippet:

  list foo {
    key name;

   /foo[name="x]

I'd expect:

  T0: nothing
  T1: nothing

Kent  // as a contributor

Andy

Hello all,

_______________________________________________
Netconf mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netconf

---

From: Andy Bierman <[email-masked]>
Date: Thu, 18 May 2017 18:33:58 -0700

On Thu, May 18, 2017 at 5:48 PM, Alexander Clemm <[email-masked]
> wrote:

> So, I think so far the consensus is #1 is not a good idea.  So we can
> probably drop #1 and should choose between #2 and #3.
>
>
>
> Personally, I would tend to agree with leaving RMON-type stuff separate,
> i.e. use a filter to select data nodes, not apply additional comparators
> for values.  However, if a filter allows to refer to state/values, then
> really the subscription could cover that.  I am not sure if a general
> warning would be sufficient to not issue such requests.  A client might
> issue an according request, but a publisher could feel free to reject a
> subscription request if the filter construct requests something that is not
> supported.  However, if the publisher could accept the request, then we run
> into the encoding issue that we need to distinguish why an object was
> omitted (or included): because it was deleted (created), or because its
> value no longer matches (or does match) the filter.
>
>
>
> I interpret the “modified-in” and “modified-out” suggested by Kent
> precisely as a distinction between those cases.  “Modified-out” simply
> means its value changed so it no longer matches the filter (but the object
> still exists).  Whether we call it modified-in/out or something else is
> another question.  Of course, the issue is that this means this is a new
> case that is not covered by the patch syntax, which would require an
> extension of the encoding.
>
>
>

What breaks if the client just sees data nodes appear or disappear from 1
patch to the next?
Why does the client need to know the data node still exists, but the foo==5
filter changed.
The client decided that only state for foo==5 needs to be watched, so from
its POV
the extra metadata should not be needed.

I find #3 simpler and more straightforward, but I can see applications for
> #2 and it results in the more general and potentially powerful mechanism.
> Perhaps #2 then?
>
>
>
> --- Alex
>

Andy

>
>
> *From:* Andy Bierman [mailto:[email-masked]]
> *Sent:* Thursday, May 18, 2017 2:53 PM
> *To:* Kent Watsen <[email-masked]>
> *Cc:* Alexander Clemm <[email-masked]>; Netconf <
> [email-masked]>
> *Subject:* Re: [Netconf] In an update, when is a delete a delete?
>
>
>
>
>
>
>

---

From: Alexander Clemm <[email-masked]>
Date: Fri, 19 May 2017 01:51:01 +0000

Hi Andy,
Below
--- Alex

From: Andy Bierman [mailto:[email-masked]]
Sent: Thursday, May 18, 2017 6:34 PM
To: Alexander Clemm <[email-masked]>
Cc: Kent Watsen <[email-masked]>; Eric Voit (evoit) <[email-masked]>; Netconf <[email-masked]>
Subject: Re: [Netconf] In an update, when is a delete a delete?

……………………
<snip>

What breaks if the client just sees data nodes appear or disappear from 1 patch to the next?
Why does the client need to know the data node still exists, but the foo==5 filter changed.
The client decided that only state for foo==5 needs to be watched, so from its POV
the extra metadata should not be needed.

<Alex>
For periodic, this is what happens.
In the case of on-change, we would indicate changes.  If a data node is not included in an update, how would we be able to distinguish between the case of (not included because no change) and (not included because no longer meets the filter)?  We do need to indicate to the subscriber that there has been a change in that the data node no longer meets the criteria.  Presumably we want to distinguish this from the case that the data node ceased to exist.  (Not distinguishing those cases would put us back to alternative #1, which I think the consensus is that we don’t want that)
</ALEX>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 18 May 2017 20:49:57 -0700

Hi Alex,

On Thu, May 18, 2017 at 6:51 PM, Alexander Clemm <[email-masked]

> *Sent:* Thursday, May 18, 2017 6:34 PM
> *To:* Alexander Clemm <[email-masked]>
> *Cc:* Kent Watsen <[email-masked]>; Eric Voit (evoit) <
> [email-masked]>; Netconf <[email-masked]>
> In the case of on-change, we would indicate changes.  If a data node is
> not included in an update, how would we be able to distinguish between the
> case of (not included because no change) and (not included because no
> longer meets the filter)?  We do need to indicate to the subscriber that
> there has been a change in that the data node no longer meets the
> criteria.  Presumably we want to distinguish this from the case that the
> data node ceased to exist.  (Not distinguishing those cases would put us
> back to alternative #1, which I think the consensus is that we don’t want
> that)
>
> </ALEX>
>
>
>

Regarding not included because of no-change:

How can happen/?
Not if the client is requesting non-overlapping subtrees /foo and /bar.
These are safe to mix in 1 subscription. The client should not assume
that an update for /foo means /bar was deleted.

Are you referring to overlapping subtrees in 1 subscription?
How would this be configured? Seems complicated to manage.
The semantics of "on-change" should be easy to manage on both client and
server.

Andy

---

From: Alexander Clemm <[email-masked]>
Date: Fri, 19 May 2017 21:30:54 +0000

Hi Andy,

I am not sure I understand your question below.  I am not referring to mix-and-match between subscription – not to bundling or anything like that.  Merely to an on-change subscription.

If we do send updates regarding changes, and a subscription allows to apply a “value filter”, there are cases when a change is caused by a node being deleted since the last update cycle, and there are cases when a change is caused by a node changing its value so it no longer matches the value of the filter (and create case accordingly).  This can clearly occur.  Given this, I think we are faced with the alternative of either being able to properly distinguish the two (option #2 in the earlier thread), or disallowing filters based on data node value (option #3).  I was under the impression you were arguing for #3 earlier?

--- Alex

Sent: Thursday, May 18, 2017 8:50 PM

Hi Alex,

On Thu, May 18, 2017 at 6:51 PM, Alexander Clemm <[email-masked]<mailto:[email-masked]>> wrote:
Hi Andy,
Below
--- Alex

From: Andy Bierman [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, May 18, 2017 6:34 PM
To: Alexander Clemm <[email-masked]<mailto:[email-masked]>>
Cc: Kent Watsen <[email-masked]<mailto:[email-masked]>>; Eric Voit (evoit) <[email-masked]<mailto:[email-masked]>>; Netconf <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Netconf] In an update, when is a delete a delete?

……………………
<snip>

<Alex>

How can happen/?

Are you referring to overlapping subtrees in 1 subscription?
How would this be configured? Seems complicated to manage.
The semantics of "on-change" should be easy to manage on both client and server.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 19 May 2017 14:36:44 -0700

On Fri, May 19, 2017 at 2:30 PM, Alexander Clemm <[email-masked]

> Hi Andy,
>
>
>
> I am not sure I understand your question below.  I am not referring to
> mix-and-match between subscription – not to bundling or anything like
> that.  Merely to an on-change subscription.
>
>
>
> If we do send updates regarding changes, and a subscription allows to
> apply a “value filter”, there are cases when a change is caused by a node
> being deleted since the last update cycle, and there are cases when a
> change is caused by a node changing its value so it no longer matches the
> value of the filter (and create case accordingly).  This can clearly
> occur.  Given this, I think we are faced with the alternative of either
> being able to properly distinguish the two (option #2 in the earlier
> thread), or disallowing filters based on data node value (option #3).  I
> was under the impression you were arguing for #3 earlier?
>
>
>

I guess I like option #4 (allow data filters on static data, say SHOULD NOT
use on dynamic data)
The solution seems too complicated if there is a need for "I am still here,
but I am not being reported" .
Especially since that means "still here in your custom view of things", not
"still here in the device".

--- Alex

Andy

> *Sent:* Thursday, May 18, 2017 8:50 PM
> On Thu, May 18, 2017 at 6:51 PM, Alexander Clemm <
> [email-masked]> wrote:
>
> Hi Andy,
>
> Below
>
> --- Alex
>
>
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 20 May 2017 08:45:14 +0200

> "still here in the device".

What is 'static data'?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Sat, 20 May 2017 07:17:27 -0700

> > "still here in the device".

Data you do not expect to change, like if:type.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 20 May 2017 16:28:07 +0200

> Data you do not expect to change, like if:type.

Until I swap the line card...

Anyway, if you propose 'allow data filters on static data', then there
needs to be some reasonably precise definition of 'static data'.

/js

---

From: Andy Bierman <[email-masked]>
Date: Sat, 20 May 2017 10:06:29 -0700

On Sat, May 20, 2017 at 7:28 AM, Juergen Schoenwaelder <
[email-masked]> wrote:

> > > > I guess I like option #4 (allow data filters on static data, say
> SHOULD
> > > NOT
> > > > use on dynamic data)
> > > > Especially since that means "still here in your custom view of
> things",
> > > not
That may be a deployment choice, so it is hard to define.
I do not object to option (2). I just prefer (4).

/js

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 21 May 2017 08:53:49 +0200

> I do not object to option (2). I just prefer (4).

(4) is underdefined as far as I can tell.

/js