Subject: [Netconf] In an update, when is a delete a delete?

From: Alexander Clemm <[email-masked]>
Date: Wed, 17 May 2017 21:29:40 +0000

Hello all,

In updating the YANG-Push document (draft-ietf-netconf-yang-push), we have come across one issue that we wanted to raise with the working group.

As part of an on-change subscription, update records reflect the type of change (e.g. whether the value of an object has changed, or whether an object was created or deleted); a subscription allows also to specify whether interested only in specific types of changes (for example, only creates and deleted but no value changes).

At the same time, a subscription filter specifies which objects to include as part of a subscription and which not.  (Really, it is not so much of a "filter" on a stream that is generated independently of the filter, than it is a policy of which objects to include as part of subscribed update records.)   However, a subscription filter (such as XPath) can be used to also specify a value filter, which will include or exclude objects based on their current value. This makes it possible to e.g. subscribe to an object "foo" but only if its value is 5.

Now, this means that the same object could be included in one update, but excluded in another update, due to its value no longer meeting the filter criteria.  For example, if foo's value changes from 5 to 3 in one cycle, a periodic subscription will no longer include foo in its next update.  The question now concerns how to properly handle this in the case of an on-change subscription.

One possibility concerns reporting the fact that "foo" no longer meets the subscription criteria and is no longer included in the update record as a "delete" event.  If foo's value again becomes "5" at a later point in time, that would be reported as a "create" event.  If foo's value changes again from 5 at a later point in time and then changes back to 3  before the time of the update (perhaps because the value changed during the dampening interval), it would be reported as another "delete" event (without ever reporting a create event).  On the other hand, if foo's value changed from 3 to 6 and back again, nothing would be reported because it did not meet the filter criteria at any point in time.

>From the perspective of the receiver this may make sense if it is synching its copy of the state.  However, from the perspective of the publisher, the object was never created or deleted - only its value changed, and the case when the object was truly created or deleted can no longer be distinguished from the case when its value changed.  A "create" simply means "an object now meets a filter criteria, that was not reported in the previous cycle" (which does not mean that the object was actually created - it may have been created, or it may have simply undergone a value change).

An alternative (let's call it alternative 2) is therefore to make a distinction between whether an object was created or deleted, or whether its value fell in or out of a filter range.  This appears semantically cleaner.  However, it will require modifying the encoding to allow for distinction between those cases (currently, just plain patch encoding is used).

A third alternative is to let filters select only data nodes to subscribe to, and separate out the value filter (or disallow it as a feature altogether).  This alternative has the drawback of being less conceptually powerful, even if it may be easier to implement.

Thoughts?  Any preferences between 1, 2, and 3?
--- Alex

---

From: Igor Bryskin <[email-masked]>
Date: Thu, 18 May 2017 01:35:14 +0000

Hi Alex,

My preference is option 2. I like events Created /Deleted associated with objects actually installed /removed  in/from the data store and events ModifiedIn/ModifiedOut for the object values changed into/out of subscription filter specifications.

Cheers,
Igor

From: Netconf [mailto:[email-masked]] On Behalf Of Alexander Clemm
Sent: Wednesday, May 17, 2017 5:30 PM
To: Netconf
Subject: [Netconf] In an update, when is a delete a delete?

Hello all,

---

From: Randy Presuhn <[email-masked]>
Date: Wed, 17 May 2017 20:01:54 -0700

Hi -

> Thoughts?  Any preferences between 1, 2, and 3?

1 is simply awful.
2 sounds like the right general direction to me.
3 sounds klugey, but perhaps I misunderstood the proposal

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 18 May 2017 08:49:38 +0200 (CEST)

Hi,

Alexander Clemm <[email-masked]> wrote:
> Hello all,
> 
> In updating the YANG-Push document (draft-ietf-netconf-yang-push), we
> have come across one issue that we wanted to raise with the working
> group.
> 
> As part of an on-change subscription, update records reflect the type
> of change (e.g. whether the value of an object has changed, or whether
> an object was created or deleted); a subscription allows also to
> specify whether interested only in specific types of changes (for
> example, only creates and deleted but no value changes).
> 
> At the same time, a subscription filter specifies which objects to
> include as part of a subscription and which not.

Hmm, which filter are you talking about?  The only XPath filter I find
in the current set of documents is the
"ietf-subscribed-notifications:xpath" filter type (which btw is sorely
underspecified).  Section 2.2 of
draft-ietf-netconf-subscribed-notifications-02 says:

   Events which evaluate to "true" as a
   result of the evaluation by the filter must traverse the filter in
   their entirety.

It's not clear what this means, but my guess is that this is supposed
to work like the old RFC 5277 filters, where the filter expression is
evaluated on the notification contents, and if the expression returns
"true" (for XPath filters this means converting the results to a
boolean), then the notification is sent, otherwise not.

But it seems you are referring to some other filter which would be
used to select a node set for which changes are reported?

I would like to understand which filter mechanism you mean before
having an opinion in this matter.

/martin

> (Really, it is not
> so much of a "filter" on a stream that is generated independently of
> the filter, than it is a policy of which objects to include as part of
> subscribed update records.)  However, a subscription filter (such as
> XPath) can be used to also specify a value filter, which will include
> or exclude objects based on their current value. This makes it
> possible to e.g. subscribe to an object "foo" but only if its value is
> 5.
> 
> Now, this means that the same object could be included in one update,
> but excluded in another update, due to its value no longer meeting the
> filter criteria.  For example, if foo's value changes from 5 to 3 in
> one cycle, a periodic subscription will no longer include foo in its
> next update.  The question now concerns how to properly handle this in
> the case of an on-change subscription.
> 
> One possibility concerns reporting the fact that "foo" no longer meets
> the subscription criteria and is no longer included in the update
> record as a "delete" event.  If foo's value again becomes "5" at a
> later point in time, that would be reported as a "create" event.  If
> foo's value changes again from 5 at a later point in time and then
> changes back to 3 before the time of the update (perhaps because the
> value changed during the dampening interval), it would be reported as
> another "delete" event (without ever reporting a create event).  On
> the other hand, if foo's value changed from 3 to 6 and back again,
> nothing would be reported because it did not meet the filter criteria
> at any point in time.
> 
> >From the perspective of the receiver this may make sense if it is
> >synching its copy of the state.  However, from the perspective of the
> >publisher, the object was never created or deleted - only its value
> >changed, and the case when the object was truly created or deleted can
> >no longer be distinguished from the case when its value changed.  A
> >"create" simply means "an object now meets a filter criteria, that was
> >not reported in the previous cycle" (which does not mean that the
> >object was actually created - it may have been created, or it may have
> >simply undergone a value change).
> 
> An alternative (let's call it alternative 2) is therefore to make a
> distinction between whether an object was created or deleted, or
> whether its value fell in or out of a filter range.  This appears
> semantically cleaner.  However, it will require modifying the encoding
> to allow for distinction between those cases (currently, just plain
> patch encoding is used).
> 
> A third alternative is to let filters select only data nodes to
> subscribe to, and separate out the value filter (or disallow it as a
> feature altogether).  This alternative has the drawback of being less
> conceptually powerful, even if it may be easier to implement.
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 18 May 2017 09:29:35 -0700

Hi,

I wonder why we really need the client to choose to subscribe to
"micro-events".
It makes sense to me for subscriptions to instance data using keys and
static leafs like if:type.
Subscribing to "foo=5" does not seem that useful for replicating state or
anything else.
YANG Push should not include RMON Alarms & Events type of functionality.

The trick to good standards writing is to support the use-cases in a way
that
does not introduce massive implementation complexity for functionality
outside those use-cases.

So I would pick (3) and add a warning about avoiding filters for volatile
data nodes
such as counters.

Andy

On Wed, May 17, 2017 at 2:29 PM, Alexander Clemm <[email-masked]
> wrote:

> have come across one issue that we wanted to raise with the working group.
>
>
>
> As part of an on-change subscription, update records reflect the type of
> change (e.g. whether the value of an object has changed, or whether an
> object was created or deleted); a subscription allows also to specify
> whether interested only in specific types of changes (for example, only
> creates and deleted but no value changes).
>
>
>
> At the same time, a subscription filter specifies which objects to include
> as part of a subscription and which not.  (Really, it is not so much of a
> “filter” on a stream that is generated independently of the filter, than it
> is a policy of which objects to include as part of subscribed update
> records.)   However, a subscription filter (such as XPath) can be used to
> also specify a value filter, which will include or exclude objects based on
> their current value. This makes it possible to e.g. subscribe to an object
> “foo” but only if its value is 5.
>
>
>
> Now, this means that the same object could be included in one update, but
> excluded in another update, due to its value no longer meeting the filter
> criteria.  For example, if foo’s value changes from 5 to 3 in one cycle, a
> periodic subscription will no longer include foo in its next update.  The
> question now concerns how to properly handle this in the case of an
> on-change subscription.
>
>
>
> One possibility concerns reporting the fact that “foo” no longer meets the
> subscription criteria and is no longer included in the update record as a
> “delete” event.  If foo’s value again becomes “5” at a later point in time,
> that would be reported as a “create” event.  If foo’s value changes again
> from 5 at a later point in time and then changes back to 3  before the time
> of the update (perhaps because the value changed during the dampening
> interval), it would be reported as another “delete” event (without ever
> reporting a create event).  On the other hand, if foo’s value changed from
> 3 to 6 and back again, nothing would be reported because it did not meet
> the filter criteria at any point in time.
>
>
>
> From the perspective of the receiver this may make sense if it is synching
> its copy of the state.  However, from the perspective of the publisher, the
> object was never created or deleted – only its value changed, and the case
> when the object was truly created or deleted can no longer be distinguished
> from the case when its value changed.  A “create” simply means “an object
> now meets a filter criteria, that was not reported in the previous cycle”
> (which does not mean that the object was actually created – it may have
> been created, or it may have simply undergone a value change).
>
>
>
> An alternative (let’s call it alternative 2) is therefore to make a
> distinction between whether an object was created or deleted, or whether
> its value fell in or out of a filter range.  This appears semantically
> cleaner.  However, it will require modifying the encoding to allow for
> distinction between those cases (currently, just plain patch encoding is
> used).
>
>
>
> A third alternative is to let filters select only data nodes to subscribe
> to, and separate out the value filter (or disallow it as a feature
> altogether).  This alternative has the drawback of being less conceptually
> powerful, even if it may be easier to implement.
>
>
>
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>
>

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 18 May 2017 20:18:15 +0000

My belief is that we should enable the same capabilities with subscribe that a platform would support with a get.  And the filter syntax supported need not diverge between the two.  Based on that, Option 2 is my preference.

As we have not been specifying filter syntax command ‘MUST’, property/value filtering would become something vendors could choose to support or not.  (I.e., codifying an explicit “MAY NOT” for filtering seems premature.)

Eric

From: Netconf, May 18, 2017 12:30 PM

Hi,

I wonder why we really need the client to choose to subscribe to "micro-events".
It makes sense to me for subscriptions to instance data using keys and static leafs like if:type.
Subscribing to "foo=5" does not seem that useful for replicating state or anything else.
YANG Push should not include RMON Alarms & Events type of functionality.

The trick to good standards writing is to support the use-cases in a way that
does not introduce massive implementation complexity for functionality
outside those use-cases.

So I would pick (3) and add a warning about avoiding filters for volatile data nodes
such as counters.

Andy

On Wed, May 17, 2017 at 2:29 PM, Alexander Clemm <[email-masked]<mailto:[email-masked]>> wrote:
Hello all,

At the same time, a subscription filter specifies which objects to include as part of a subscription and which not.  (Really, it is not so much of a “filter” on a stream that is generated independently of the filter, than it is a policy of which objects to include as part of subscribed update records.)   However, a subscription filter (such as XPath) can be used to also specify a value filter, which will include or exclude objects based on their current value. This makes it possible to e.g. subscribe to an object “foo” but only if its value is 5.

Now, this means that the same object could be included in one update, but excluded in another update, due to its value no longer meeting the filter criteria.  For example, if foo’s value changes from 5 to 3 in one cycle, a periodic subscription will no longer include foo in its next update.  The question now concerns how to properly handle this in the case of an on-change subscription.

One possibility concerns reporting the fact that “foo” no longer meets the subscription criteria and is no longer included in the update record as a “delete” event.  If foo’s value again becomes “5” at a later point in time, that would be reported as a “create” event.  If foo’s value changes again from 5 at a later point in time and then changes back to 3  before the time of the update (perhaps because the value changed during the dampening interval), it would be reported as another “delete” event (without ever reporting a create event).  On the other hand, if foo’s value changed from 3 to 6 and back again, nothing would be reported because it did not meet the filter criteria at any point in time.

From the perspective of the receiver this may make sense if it is synching its copy of the state.  However, from the perspective of the publisher, the object was never created or deleted – only its value changed, and the case when the object was truly created or deleted can no longer be distinguished from the case when its value changed.  A “create” simply means “an object now meets a filter criteria, that was not reported in the previous cycle” (which does not mean that the object was actually created – it may have been created, or it may have simply undergone a value change).

An alternative (let’s call it alternative 2) is therefore to make a distinction between whether an object was created or deleted, or whether its value fell in or out of a filter range.  This appears semantically cleaner.  However, it will require modifying the encoding to allow for distinction between those cases (currently, just plain patch encoding is used).

_______________________________________________
Netconf mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netconf

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 18 May 2017 20:33:01 +0000

> From: Martin Bjorklund, May 18, 2017 2:50 AM
> 
> Hi,
> 
> Hmm, which filter are you talking about?  The only XPath filter I find in the
> current set of documents is the "ietf-subscribed-notifications:xpath" filter
> type (which btw is sorely underspecified).  Section 2.2 of
> It's not clear what this means, but my guess is that this is supposed to work
> like the old RFC 5277 filters, where the filter expression is evaluated on the
> notification contents, and if the expression returns "true" (for XPath filters this
> means converting the results to a boolean), then the notification is sent,
> otherwise not.

This is the case for events.  For these, the filter should result in a yes/no for any event in a stream.
 
> But it seems you are referring to some other filter which would be used to
> select a node set for which changes are reported?

For datastores, I suspect the RFC6241 selection filter should be the dominant one.   But it is conceivable that an XPATH selection filter (i.e., not a boolean result) could be provided. 

Eric

> I would like to understand which filter mechanism you mean before having an
> opinion in this matter.
> 
> 
> /martin
> 
> 
> 
> 
> > (Really, it is not
> > >changed, and the case when the object was truly created or deleted
> > >can no longer be distinguished from the case when its value changed.
> > >A "create" simply means "an object now meets a filter criteria, that
> > >was not reported in the previous cycle" (which does not mean that the
> > >object was actually created - it may have been created, or it may
> > >have simply undergone a value change).
> >

---

From: Alexander Clemm <[email-masked]>
Date: Mon, 22 May 2017 17:27:33 +0000

Hi Martin,

Almost overlooked your question below.  What is meant by the filter is specified in section 3.5 of the YANG-Push document.  

"Only a single filter can be applied to a subscription at a time.  The  following filter types are included in the yang-push data model: [subtree] [xpath]"

In effect, the filter specifies which data nodes to consider when sending updates.  

---Alex

-----Original Message-----
From: Martin Bjorklund [mailto:[email-masked]] 
Sent: Wednesday, May 17, 2017 11:50 PM
To: Alexander Clemm <[email-masked]>
Cc: [email-masked]
Subject: Re: [Netconf] In an update, when is a delete a delete?

Hi,

> include as part of a subscription and which not.

Hmm, which filter are you talking about?  The only XPath filter I find in the current set of documents is the "ietf-subscribed-notifications:xpath" filter type (which btw is sorely underspecified).  Section 2.2 of
draft-ietf-netconf-subscribed-notifications-02 says:

It's not clear what this means, but my guess is that this is supposed to work like the old RFC 5277 filters, where the filter expression is evaluated on the notification contents, and if the expression returns "true" (for XPath filters this means converting the results to a boolean), then the notification is sent, otherwise not.

But it seems you are referring to some other filter which would be used to select a node set for which changes are reported?

I would like to understand which filter mechanism you mean before having an opinion in this matter.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 22 May 2017 21:29:46 +0200 (CEST)

Alexander Clemm <[email-masked]> wrote:
> Hi Martin,
> 
> Almost overlooked your question below.  What is meant by the filter is
> specified in section 3.5 of the YANG-Push document.
> 
> "Only a single filter can be applied to a subscription at a time.  The
> following filter types are included in the yang-push data model:
> [subtree] [xpath]"

Actually, only "subtree" is defined in yang-push, "xpath" is defined
in subscribed-notifications.  [Side note - I think this is wrong,
subscribed-notifications should also define "subtree".]

But these filters are used by the server to decide if a certain
notification that has been generated will be sent to the client or
not. 

If you want to define filters to specifify which nodes to
subscribe to, I think you need to define new filters, not try to
reuese these notification filters.

As for your question, I think such a filter should be defined to
return a node-set to which the client subscribe to changes.  If any
node (or subnode to) in this node-set changes, the notif will be
sent.  Then the question about value comparision is not relevant
anymore.

/martin

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 23 May 2017 16:03:59 +0000

> Martin Bjorklund, May 22, 2017 3:30 PM
> 
> Actually, only "subtree" is defined in yang-push, "xpath" is defined in
> subscribed-notifications.

At the top of yang-push page 7, xpath selection is described.  Is there something you feel missing?

>  [Side note - I think this is wrong, subscribed-notifications should also define "subtree".]
>
> But these filters are used by the server to decide if a certain notification that
> has been generated will be sent to the client or not.

Yes, the filters in subscribed-notification are supposed to give a boolean indication as to whether a specific event should traverse the filter in its entirety. RFC6241 section 6 subtree filters are written to provide a subset of content.   I suppose it would be possible to define an event-based subtree-filter-type where a non-null result of the subtree filter means that a particular event should traverse that filter.  Is this what you are suggesting?

> If you want to define filters to specifify which nodes to subscribe to, I think
> you need to define new filters, not try to reuese these notification filters.

Filtering syntax is hard, so we have been trying to adopt whatever is available for GET.  This way we don't have to educate users on a new universe of what is possible.  I fully expect that lots of learnings are going to come in the industry here over time, and this will be revisited in the future.

> As for your question, I think such a filter should be defined to return a node-
> set to which the client subscribe to changes.  If any node (or subnode to) in
> this node-set changes, the notif will be sent.  Then the question about value
> comparision is not relevant anymore.

Excellent, on-change should only send an update if the results of the subscription filter have changed since the previous push.   It is quite possible that an object has been created and then deleted since the last push.  Representing this was the genesis of Alex's question.

Eric

> > > In updating the YANG-Push document (draft-ietf-netconf-yang-push),
> > > we have come across one issue that we wanted to raise with the
> > > working group.
> > >
> > > As part of an on-change subscription, update records reflect the
> > > type of change (e.g. whether the value of an object has changed, or
> > > whether an object was created or deleted); a subscription allows
> > > also to specify whether interested only in specific types of changes
> > > (for example, only creates and deleted but no value changes).
> > >
> > > the filter, than it is a policy of which objects to include as part
> > > of subscribed update records.)  However, a subscription filter (such
> > > as
> > > XPath) can be used to also specify a value filter, which will
> > > include or exclude objects based on their current value. This makes
> > > it possible to e.g. subscribe to an object "foo" but only if its
> > > value is 5.
> > >
> > > Now, this means that the same object could be included in one
> > > update, but excluded in another update, due to its value no longer
> > > meeting the filter criteria.  For example, if foo's value changes
> > > from 5 to 3 in one cycle, a periodic subscription will no longer
> > > include foo in its next update.  The question now concerns how to
> > > properly handle this in the case of an on-change subscription.
> > >
> > > One possibility concerns reporting the fact that "foo" no longer
> > > meets the subscription criteria and is no longer included in the
> > > update record as a "delete" event.  If foo's value again becomes "5"
> > > at a later point in time, that would be reported as a "create"
> > > event.  If foo's value changes again from 5 at a later point in time
> > > and then changes back to 3 before the time of the update (perhaps
> > > because the value changed during the dampening interval), it would
> > > be reported as another "delete" event (without ever reporting a
> > > create event).  On the other hand, if foo's value changed from 3 to
> > > 6 and back again, nothing would be reported because it did not meet
> > > >synching its copy of the state.  However, from the perspective of
> > > >the publisher, the object was never created or deleted - only its
> > > >value changed, and the case when the object was truly created or
> > > >deleted can no longer be distinguished from the case when its value
> changed.
> > > >A "create" simply means "an object now meets a filter criteria,
> > > >that was not reported in the previous cycle" (which does not mean
> > > >that the object was actually created - it may have been created, or
> > > >it may have simply undergone a value change).
> > >
> > > semantically cleaner.  However, it will require modifying the
> > > encoding to allow for distinction between those cases (currently,
> > > just plain patch encoding is used).
> > >
> > > feature altogether).  This alternative has the drawback of being
> > > less conceptually powerful, even if it may be easier to implement.
> > >

---

From: Alexander Clemm <[email-masked]>
Date: Tue, 23 May 2017 17:56:37 +0000

Hi Martin, 
Thank you for your response.  
Two more comments inline, <ALEX> (including a response to Eric)
--- Alex

-----Original Message-----
From: Eric Voit (evoit) [mailto:[email-masked]] 
Sent: Tuesday, May 23, 2017 9:04 AM
To: Martin Bjorklund <[email-masked]>; Alexander Clemm <[email-masked]>
Cc: [email-masked]
Subject: RE: [Netconf] In an update, when is a delete a delete?

> > Almost overlooked your question below.  What is meant by the filter 
> > is specified in section 3.5 of the YANG-Push document.
> >
> > "Only a single filter can be applied to a subscription at a time.  
> > The following filter types are included in the yang-push data model:
> > [subtree] [xpath]"
> 
> in subscribed-notifications.

> notification that has been generated will be sent to the client or not.

> If you want to define filters to specifify which nodes to subscribe 
> to, I think you need to define new filters, not try to reuese these notification filters.

<ALEX> I think the difference in semantics between a filter that filters on the contents that is included in a notification (and applies to the notification as a whole), and a filter that specifies updates of which data nodes to include in an update notification, is actually key.  (Arguably, the second filter defines which content to generate in the first place.)  

We will review the text to make sure the distinction is clear.  That said, we do want to use the same filter syntax constructs for both, even if they are used slightly differently when used as a notification filter vs a filter that specifies which updates to include. 
</ALEX>

> return a node- set to which the client subscribe to changes.  If any 
> node (or subnode to) in this node-set changes, the notif will be sent.  
> Then the question about value comparision is not relevant anymore.

<ALEX>I interpret the statement differently:  It is not about whether the results of the subscription filter have changed, but if any nodes in the node-set that had been subscribed to changed.   In other words, I take it as an endorsement of option 3, not of option 1.  
--- Alex
</ALEX>

Eric

> > > type of change (e.g. whether the value of an object has changed, 
> > > or whether an object was created or deleted); a subscription 
> > > allows also to specify whether interested only in specific types 
> > > of changes (for example, only creates and deleted but no value changes).
> > >
> > Hmm, which filter are you talking about?  The only XPath filter I 
> > find in the current set of documents is the 
> > "ietf-subscribed-notifications:xpath" filter type (which btw is 
> > sorely underspecified).  Section 2.2 of
> > It's not clear what this means, but my guess is that this is 
> > supposed to work like the old RFC 5277 filters, where the filter 
> > expression is evaluated on the notification contents, and if the 
> > expression returns "true" (for XPath filters this means converting 
> > the results to a boolean), then the notification is sent, otherwise not.
> >
> > > so much of a "filter" on a stream that is generated independently 
> > > of the filter, than it is a policy of which objects to include as 
> > > part of subscribed update records.)  However, a subscription 
> > > filter (such as
> > > include or exclude objects based on their current value. This 
> > > makes it possible to e.g. subscribe to an object "foo" but only if 
> > > its value is 5.
> > >
> > > event.  If foo's value changes again from 5 at a later point in 
> > > time and then changes back to 3 before the time of the update 
> > > (perhaps because the value changed during the dampening interval), 
> > > it would be reported as another "delete" event (without ever 
> > > reporting a create event).  On the other hand, if foo's value 
> > > changed from 3 to
> > > 6 and back again, nothing would be reported because it did not 
> > > meet the filter criteria at any point in time.
> > >
> > > >that the object was actually created - it may have been created, 
> > > >or it may have simply undergone a value change).
> > >
> > > An alternative (let's call it alternative 2) is therefore to make 
> > > a distinction between whether an object was created or deleted, or 
> > > subscribe to, and separate out the value filter (or disallow it as 
> > > a feature altogether).  This alternative has the drawback of being

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 23 May 2017 19:57:20 +0200 (CEST)

> there something you feel missing?

Yes.  First of all, the YANG module defines an identity called
"xpath", based on "sn:filter".  So this filter has nothing to do with
selecting nodes in a datastore; this filter is used to match against a
generated notification record.

Second, the XPath filter is sorely underspecified.  The XPath context
is not described, the expected result data type is not defined, and
it is not described how the result is supposed to be used.

> > notification that
> Yes, the filters in subscribed-notification are supposed to give a
> boolean indication as to whether a specific event should traverse the
> filter in its entirety. RFC6241 section 6 subtree filters are written
> to provide a subset of content.  I suppose it would be possible to
> define an event-based subtree-filter-type where a non-null result of
> the subtree filter means that a particular event should traverse that
> filter.  Is this what you are suggesting?

Yes.  Note that this is already provided by RFC 5277, and I have
always assumed that this new work will provide at least the same
functions as RFC 5277 (and more).  (But note that the XPath filter is
underspecified also in RFC 5277...)

> > to, I think
> Filtering syntax is hard, so we have been trying to adopt whatever is
> available for GET.  This way we don't have to educate users on a new
> universe of what is possible.  I fully expect that lots of learnings
> are going to come in the industry here over time, and this will be
> revisited in the future.

?

I am not proposing any new filter syntax.  I am saying that the
current filter nodes as defined in subscribed-notification cannot be
used to select nodes to subscribe to for changes.

> > return a node-
> Excellent, on-change should only send an update if the results of the
> subscription filter have changed since the previous push.  It is quite
> possible that an object has been created and then deleted since the
> last push.

I don't understand what you're trying to say with these sentences (but
since the first word was "Excellent" maybe it's ok ;)

/martin

> Representing this was the genesis of Alex's question.
> 
> Eric
> 
> > /martin
> > 
> > 
> > 
> > 
> > 
> > >

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 23 May 2017 22:16:36 +0000

> From: Martin Bjorklund, May 23, 2017 1:57 PM
> 
> > > > Almost overlooked your question below.  What is meant by the
> > > > filter is specified in section 3.5 of the YANG-Push document.
> > > >
> Yes.  First of all, the YANG module defines an identity called "xpath", based on
> "sn:filter".  So this filter has nothing to do with selecting nodes in a datastore;
> this filter is used to match against a generated notification record.

To address this we could split the xpath identity into two types: "xpath-selection" and  "xpath-boolean".  These would have different definitions, but both reference http://www.w3.org/TR/1999/REC-xpath-19991116  .  The difference is that one results in a xpath node-set expression, and the other an xpath Boolean expression.  Make sense?

> Second, the XPath filter is sorely underspecified.  The XPath context is not
> described, 

I understand and agree on your comment about the context.  The intent here is to provide equivalent capabilities of a GET.  As it would be a huge undertaking to try to consolidate an industry-wide view of the minimal xpath syntax and capabilities in networking, I am hoping this doesn't fall under the umbrella of YANG subscription.  I would be glad to support someone who wishes to take this up though.

> the expected result data type is not defined, and it is not described
> how the result is supposed to be used.

As for the result, the anydata output should be provided to the subscriber (with appropriate security applied).  They can determine how to use it.   The preferred embodiment would be to maintain a local extract of the Publisher's datastore (as defined by the filter).

> Yes.  Note that this is already provided by RFC 5277, and I have always
> assumed that this new work will provide at least the same functions as RFC
> 5277 (and more).  (But note that the XPath filter is underspecified also in RFC
> 5277...)

I also want to make sure that a non-null result from a filter allows the event to pass.  I suspect that an xpath Boolean filter could be designed to do this, but I will tweak the subscribed-notifications text so that unnecessary filtering expression complexity is not artificially required.

> I am not proposing any new filter syntax.  I am saying that the current filter
> nodes as defined in subscribed-notification cannot be used to select nodes to
> subscribe to for changes.

Understand.  Hopefully with the "xpath-selection" change proposed above, this will be covered.

Eric

> > > node (or subnode
> > > to) in
> > > this node-set changes, the notif will be sent.  Then the question
> > > about value comparision is not relevant anymore.
> >
> I don't understand what you're trying to say with these sentences (but since
> the first word was "Excellent" maybe it's ok ;)
> 
> 
> /martin
> 
> 
> > > > > In updating the YANG-Push document
> > > > > (draft-ietf-netconf-yang-push), we have come across one issue
> > > > > that we wanted to raise with the working group.
> > > > >
> > > > > At the same time, a subscription filter specifies which objects
> > > > > to include as part of a subscription and which not.
> > > >
> > > > > so much of a "filter" on a stream that is generated
> > > > > independently of the filter, than it is a policy of which
> > > > > objects to include as part of subscribed update records.)
> > > > > However, a subscription filter (such as
> > > > > makes it possible to e.g. subscribe to an object "foo" but only
> > > > > if its value is 5.
> > > > >
> > > > > update, but excluded in another update, due to its value no
> > > > > longer meeting the filter criteria.  For example, if foo's value
> > > > > changes from 5 to 3 in one cycle, a periodic subscription will
> > > > > no longer include foo in its next update.  The question now
> > > > > concerns how to properly handle this in the case of an on-change
> subscription.
> > > > >
> > > > > (perhaps because the value changed during the dampening
> > > > > interval), it would be reported as another "delete" event
> > > > > (without ever reporting a create event).  On the other hand, if
> > > > > foo's value changed from 3 to
> > > > > >From the perspective of the receiver this may make sense if it
> > > > > >is synching its copy of the state.  However, from the
> > > > > >perspective of the publisher, the object was never created or
> > > > > >deleted - only its value changed, and the case when the object
> > > > > >was truly created or deleted can no longer be distinguished
> > > > > >from the case when its value
> > > changed.
> > > > > >that was not reported in the previous cycle" (which does not
> > > > > >mean that the object was actually created - it may have been
> > > > > >created, or it may have simply undergone a value change).
> > > > >
> > > > > An alternative (let's call it alternative 2) is therefore to
> > > > > make a distinction between whether an object was created or
> > > > > deleted, or whether its value fell in or out of a filter range.
> > > > > This appears semantically cleaner.  However, it will require
> > > > > modifying the encoding to allow for distinction between those
> > > > > cases (currently, just plain patch encoding is used).
> > > > >
> > > > > subscribe to, and separate out the value filter (or disallow it
> > > > > as a feature altogether).  This alternative has the drawback of
> > > > > being less conceptually powerful, even if it may be easier to
> implement.
> > > > >

---

From: Andy Bierman <[email-masked]>
Date: Tue, 23 May 2017 15:47:58 -0700

On Tue, May 23, 2017 at 3:16 PM, Eric Voit (evoit) <[email-masked]> wrote:

> To address this we could split the xpath identity into two types:
> "xpath-selection" and  "xpath-boolean".  These would have different
> definitions, but both reference http://www.w3.org/TR/1999/REC-
> xpath-19991116  .  The difference is that one results in a xpath node-set
> expression, and the other an xpath Boolean expression.  Make sense?
>
> I understand and agree on your comment about the context.  The intent here
> is to provide equivalent capabilities of a GET.  As it would be a huge
> undertaking to try to consolidate an industry-wide view of the minimal
> xpath syntax and capabilities in networking, I am hoping this doesn't fall
> under the umbrella of YANG subscription.  I would be glad to support
> someone who wishes to take this up though.
>
>

subtree and XPath filtering are implemented in various ways which could all
be considered valid.
The node-set result is not the same across server implementations.

For example:

   <filter>
     <foo />
     <foo>
       <bar>1</bar>
     </foo>
  </filter>

Does the server return the entire /foo subtree or return the entire /foo
subtree
followed by another subtree for /foo[bar=1]?  (yes. that's the problem)
XPath has many more ways to overlap node-sets.

Should we even document all the under-implemented servers that do not
support the
complete XPath syntax? Probably not, but it would be worthwhile to have an
"XPath-core"
that is widely supported.

I agree that YANG Push is not the correct document for this sort of work.

Perhaps YANG Push filtering should be 2 phases: (1) select + (2) test),
similar to how XSLT works.

(1) select can be simple and static, like a node-instance-identifier
expression from NACM
(2) test can be an XPath boolean expression run against each node selected
in (1),
      using the selected node as the context node

Andy

> As for the result, the anydata output should be provided to the subscriber
> (with appropriate security applied).  They can determine how to use it.
>  The preferred embodiment would be to maintain a local extract of the
> Publisher's datastore (as defined by the filter).
>
> I also want to make sure that a non-null result from a filter allows the
> event to pass.  I suspect that an xpath Boolean filter could be designed to
> do this, but I will tweak the subscribed-notifications text so that
> unnecessary filtering expression complexity is not artificially required.
>
> since

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 24 May 2017 09:35:45 +0200 (CEST)

"Eric Voit (evoit)" <[email-masked]> wrote:
> > "xpath", based on
> definitions, but both reference
> http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference is that
> one results in a xpath node-set expression, and the other an xpath
> Boolean expression.  Make sense?

Not really.  In subscribed-notifications you have defined a generic
filter mechanism, that is used to block/allow the sending of generated
notifications to a subscriber.  This filter mechanism cannot be used
to select nodes to subscribe to for changes in a datastore.  You need
to define a separate mechansim for that in yang-push.  (Maybe not
even call it "filter", but perhaps "selection").

Some comments on subscribed-notifications: This generic mechanism
allows various filter syntaxes.  This generic mechanism needs to
explain what is required by a filter syntax definition (an identity,
evaluation rules).  In section 2.2 the document says that two filter
syntaxes are supported, but only one is defined (xpath).  It needs to
explain that a filter is supposed to return true or false (this part
RFC 5277 got right, see section 3.6).  Also, the current module has
the filter in an anyxml node; it is not clear how an XPath expression
is encoded in anyxml.

> > is not
> I understand and agree on your comment about the context.  The intent
> here is to provide equivalent capabilities of a GET.

I understand that.

> As it would be a
> huge undertaking to try to consolidate an industry-wide view of the
> minimal xpath syntax and capabilities in networking

Ehh... yes?  What does this have to do with specifiying the XPath
context?

> , I am hoping this
> doesn't fall under the umbrella of YANG subscription.  I would be glad
> to support someone who wishes to take this up though.
> 
> As for the result, the anydata output should be provided to the
> subscriber (with appropriate security applied).

I was referring to the output of the filter evaluation.

> They can determine
> how to use it.  The preferred embodiment would be to maintain a local
> extract of the Publisher's datastore (as defined by the filter).
> 
> > always
> I also want to make sure that a non-null result from a filter allows
> the event to pass.  I suspect that an xpath Boolean filter could be
> designed to do this, but I will tweak the subscribed-notifications
> text so that unnecessary filtering expression complexity is not
> artificially required.

Please make sure you understand how subtree filters and XPath filters
work in RFC 5277.  There is nothing wrong with that functionality.

> > current filter

No, see above.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 24 May 2017 09:42:35 +0200 (CEST)

> XPath has many more ways to overlap node-sets.

Actually, for XPath the situation is better, since the XPath
evaluation will return a node set (which does not contain duplicates,
regardless of the expression), and the resulting node set is returned
(in the case of get).

As for subtree filter, RFC 6241 says:

   Specific
   data instances are not duplicated in the response in the event that
   the request contains multiple filter subtree expressions that select
   the same data.

>       using the selected node as the context node

I wonder if (2) is even needed...

/martin

---

From: Alexander Clemm <[email-masked]>
Date: Wed, 24 May 2017 22:25:26 +0000

Let me briefly summarize where I think we are. The thread has clearly moved on from the initial issue of how to distinguish between causes for updates to no longer include a given data node (because it was deleted, or because it changed its value to no longer match a filter) to other issues.  

For those other issues, I think there are in fact two separate items that we are trying to discuss at the same time, which are really orthogonal to one another:

- The first item concerns the concept of a "filter" vs a "selector".  A filter is what gets specified for any subscription to notifications, which defines which notifications a subscriber wants to receive.  The filter is applied to the notification as a whole, i.e. either the notification is delivered or it is not.  It does not apply to subsets of contents within the notification.  A selector, on the other hand, is used to specify updates of which data nodes to include in a YANG-push subscription.  The same update notification could include updates of several data nodes, hence a filter applied to the notification as a whole would be inappropriate here - the semantics is slightly different: as a subscriber, "of  which data nodes would you like to receive updates", not "which notifications would you like to receive".  

Filters and selectors can be specified using the same syntax.  We are faced with a choice between specifying a single construct as part of a subscription, which is treated as a filter (in case of a "regular" subscription for notification updates) or as a selector (in case of a subscription to datastore updates) , or having separate objects, i.e. adding a separate "selector" construct for YANG-push.  Using a single object amounts to overloading.  It is more compact but with arguably a more complex semantics.  Using separate object results in a model that is more verbose, but has arguably simpler semantics.  

- The second item concerns the issue of whether the filter/selector construct has a dynamic type or a static type.  This is where the issue of identity vs case statement etc comes in.  In case of a dynamic type, we use two objects:  One object of a generic type (anydata) holds the filter/selector construct itself, the second object specifies how to interpret it, i.e. which type it is.  (In case of "overloading, we can also make explicit the distinction whether it is a filter or a selector).  That second object is an identityref, referencing one of the identities that designates the filter/selector type.  In case of a static type, we use a case statement (to distinguish which specific type it is).  

The argument that Eric is making is that we should have a single overloaded object that can serve as a filter or a selector depending on the context and whether it is used in a notification subscription or a YANG-push subscription, and that we use a dynamic type including an identityref that indicates whether the object serves as a selector or a filter.  From my perspective, I feel that not overloading may be conceptually a bit "cleaner", but at the end of the day I am fine either way.  And I am not entirely sure, Martin, what you are proposing.  Either way, we should document the issue and our choice clearly.  

--- Alex

-----Original Message-----
From: Martin Bjorklund [mailto:[email-masked]] 
Sent: Wednesday, May 24, 2017 12:36 AM
To: [email-masked]
Cc: Alexander Clemm <[email-masked]>; [email-masked]
Subject: Re: [Netconf] In an update, when is a delete a delete?

"Eric Voit (evoit)" <[email-masked]> wrote:
> > > > Actually, only "subtree" is defined in yang-push, "xpath" is 
> > > > defined in subscribed-notifications.
> > >
> > "xpath", based on "sn:filter".  So this filter has nothing to do 
> > with selecting nodes in a datastore; this filter is used to match 
> > against a generated notification record.
> 

Not really.  In subscribed-notifications you have defined a generic filter mechanism, that is used to block/allow the sending of generated notifications to a subscriber.  This filter mechanism cannot be used to select nodes to subscribe to for changes in a datastore.  You need to define a separate mechansim for that in yang-push.  (Maybe not even call it "filter", but perhaps "selection").

Some comments on subscribed-notifications: This generic mechanism allows various filter syntaxes.  This generic mechanism needs to explain what is required by a filter syntax definition (an identity, evaluation rules).  In section 2.2 the document says that two filter syntaxes are supported, but only one is defined (xpath).  It needs to explain that a filter is supposed to return true or false (this part RFC 5277 got right, see section 3.6).  Also, the current module has the filter in an anyxml node; it is not clear how an XPath expression is encoded in anyxml.

> > Second, the XPath filter is sorely underspecified.  The XPath 
> > context is not described,
> 

I understand that.

> minimal xpath syntax and capabilities in networking

Ehh... yes?  What does this have to do with specifiying the XPath context?

> > the expected result data type is not defined, and it is not 
> > described how the result is supposed to be used.
> 

> > > boolean indication as to whether a specific event should traverse 
> > > the filter in its entirety. RFC6241 section 6 subtree filters are 
> > > written to provide a subset of content.  I suppose it would be 
> > > possible to define an event-based subtree-filter-type where a 
> > > non-null result of the subtree filter means that a particular 
> > > event should traverse that filter.  Is this what you are suggesting?
> > 
> > functions as RFC

Please make sure you understand how subtree filters and XPath filters work in RFC 5277.  There is nothing wrong with that functionality.

> > > Filtering syntax is hard, so we have been trying to adopt whatever 
> > > is available for GET.  This way we don't have to educate users on 
> > > a new universe of what is possible.  I fully expect that lots of 
> > > learnings are going to come in the industry here over time, and 
> > > this will be revisited in the future.
> > 
> > ?
> > 

No, see above.

/martin

> > > > return a node- set to which the client subscribe to changes.  If 
> > > > any node (or subnode
> > > > to) in
> > > > this node-set changes, the notif will be sent.  Then the 
> > > > question about value comparision is not relevant anymore.
> > >
> > > Excellent, on-change should only send an update if the results of 
> > > the subscription filter have changed since the previous push.  It 
> > > is quite possible that an object has been created and then deleted 
> > > since the last push.
> > 
> > I don't understand what you're trying to say with these sentences 
> > (but since the first word was "Excellent" maybe it's ok ;)
> > 
> > 
> > /martin
> > 
> > 
> > > > > > (draft-ietf-netconf-yang-push), we have come across one 
> > > > > > issue that we wanted to raise with the working group.
> > > > > >
> > > > > > As part of an on-change subscription, update records reflect 
> > > > > > the type of change (e.g. whether the value of an object has 
> > > > > > changed, or whether an object was created or deleted); a 
> > > > > > subscription allows also to specify whether interested only 
> > > > > > in specific types of changes (for example, only creates and 
> > > > > > deleted but no value
> > changes).
> > > > > >
> > > > > > At the same time, a subscription filter specifies which 
> > > > > > objects to include as part of a subscription and which not.
> > > > >
> > > > > Hmm, which filter are you talking about?  The only XPath 
> > > > > filter I find in the current set of documents is the 
> > > > > "ietf-subscribed-notifications:xpath" filter type (which btw 
> > > > > is sorely underspecified).  Section 2.2 of
> > > > > supposed to work like the old RFC 5277 filters, where the 
> > > > > filter expression is evaluated on the notification contents, 
> > > > > and if the expression returns "true" (for XPath filters this 
> > > > > means converting the results to a boolean), then the 
> > > > > notification is sent, otherwise not.
> > > > >
> > > > > But it seems you are referring to some other filter which 
> > > > > would be used to select a node set for which changes are reported?
> > > > >
> > > > > > XPath) can be used to also specify a value filter, which 
> > > > > > will include or exclude objects based on their current 
> > > > > > value. This makes it possible to e.g. subscribe to an object 
> > > > > > "foo" but only if its value is 5.
> > > > > >
> > > > > > Now, this means that the same object could be included in 
> > > > > > one update, but excluded in another update, due to its value 
> > > > > > no longer meeting the filter criteria.  For example, if 
> > > > > > foo's value changes from 5 to 3 in one cycle, a periodic 
> > > > > > subscription will no longer include foo in its next update.  
> > > > > > The question now concerns how to properly handle this in the 
> > > > > > case of an on-change
> > subscription.
> > > > > >
> > > > > > One possibility concerns reporting the fact that "foo" no 
> > > > > > longer meets the subscription criteria and is no longer 
> > > > > > included in the update record as a "delete" event.  If foo's value again becomes "5"
> > > > > > event.  If foo's value changes again from 5 at a later point 
> > > > > > in time and then changes back to 3 before the time of the 
> > > > > > update (perhaps because the value changed during the 
> > > > > > dampening interval), it would be reported as another 
> > > > > > "delete" event (without ever reporting a create event).  On 
> > > > > > the other hand, if foo's value changed from 3 to
> > > > > > 6 and back again, nothing would be reported because it did 
> > > > > > not meet the filter criteria at any point in time.
> > > > > >
> > > > > > >From the perspective of the receiver this may make sense if 
> > > > > > >it is synching its copy of the state.  However, from the 
> > > > > > >perspective of the publisher, the object was never created 
> > > > > > >or deleted - only its value changed, and the case when the 
> > > > > > >object was truly created or deleted can no longer be 
> > > > > > >distinguished from the case when its value
> > > > changed.
> > > > > > >A "create" simply means "an object now meets a filter 
> > > > > > >criteria, that was not reported in the previous cycle" 
> > > > > > >(which does not mean that the object was actually created - 
> > > > > > >it may have been created, or it may have simply undergone a value change).
> > > > > >
> > > > > > modifying the encoding to allow for distinction between 
> > > > > > those cases (currently, just plain patch encoding is used).
> > > > > >
> > > > > > A third alternative is to let filters select only data nodes 
> > > > > > to subscribe to, and separate out the value filter (or 
> > > > > > disallow it as a feature altogether).  This alternative has 
> > > > > > the drawback of being less conceptually powerful, even if it 
> > > > > > may be easier to
> > implement.
> > > > > >

---

From: Andy Bierman <[email-masked]>
Date: Wed, 24 May 2017 15:55:09 -0700

Hi,

Let's compare to what 5277 already has defined.
I agree with Martin that new functionality should be defined with
new terms and new specs.

First, RFC 5277 allows the event-type to be filtered.
This is basic functionality every server should support.
(e.g., select netconf-config-change, ignore netconf-session-start)
This event type test is not separate from the event content filter.
Both are boolean tests.

Notification filters for YANG Push need to select the event-types used,
such as push-change-update.  It would be nice if this was handled
automatically,
instead of listing them in the <establish-subscription>.

Traditional notification filters would require that the structure of an
event such
as push-change-update be known to the developer in full detail.
IMO this is a bad approach.  A new filter type that selects data nodes
would be much better.  The current proposal is easy to under-specify
and (IMO) nearly impossible for operators to use.

Andy

On Wed, May 24, 2017 at 3:25 PM, Alexander Clemm <[email-masked]

> Let me briefly summarize where I think we are. The thread has clearly
> moved on from the initial issue of how to distinguish between causes for
> updates to no longer include a given data node (because it was deleted, or
> because it changed its value to no longer match a filter) to other issues.
>
> For those other issues, I think there are in fact two separate items that
> we are trying to discuss at the same time, which are really orthogonal to
> one another:
>
> - The first item concerns the concept of a "filter" vs a "selector".  A
> filter is what gets specified for any subscription to notifications, which
> defines which notifications a subscriber wants to receive.  The filter is
> applied to the notification as a whole, i.e. either the notification is
> delivered or it is not.  It does not apply to subsets of contents within
> the notification.  A selector, on the other hand, is used to specify
> updates of which data nodes to include in a YANG-push subscription.  The
> same update notification could include updates of several data nodes, hence
> a filter applied to the notification as a whole would be inappropriate here
> - the semantics is slightly different: as a subscriber, "of  which data
> nodes would you like to receive updates", not "which notifications would
> you like to receive".
>
> Filters and selectors can be specified using the same syntax.  We are
> faced with a choice between specifying a single construct as part of a
> subscription, which is treated as a filter (in case of a "regular"
> subscription for notification updates) or as a selector (in case of a
> subscription to datastore updates) , or having separate objects, i.e.
> adding a separate "selector" construct for YANG-push.  Using a single
> object amounts to overloading.  It is more compact but with arguably a more
> complex semantics.  Using separate object results in a model that is more
> verbose, but has arguably simpler semantics.
>
> - The second item concerns the issue of whether the filter/selector
> construct has a dynamic type or a static type.  This is where the issue of
> identity vs case statement etc comes in.  In case of a dynamic type, we use
> two objects:  One object of a generic type (anydata) holds the
> filter/selector construct itself, the second object specifies how to
> interpret it, i.e. which type it is.  (In case of "overloading, we can also
> make explicit the distinction whether it is a filter or a selector).  That
> second object is an identityref, referencing one of the identities that
> designates the filter/selector type.  In case of a static type, we use a
> case statement (to distinguish which specific type it is).
>
> The argument that Eric is making is that we should have a single
> overloaded object that can serve as a filter or a selector depending on the
> context and whether it is used in a notification subscription or a
> YANG-push subscription, and that we use a dynamic type including an
> identityref that indicates whether the object serves as a selector or a
> filter.  From my perspective, I feel that not overloading may be
> conceptually a bit "cleaner", but at the end of the day I am fine either
> way.  And I am not entirely sure, Martin, what you are proposing.  Either
> way, we should document the issue and our choice clearly.
>
> --- Alex
>
>
> Not really.  In subscribed-notifications you have defined a generic filter
> mechanism, that is used to block/allow the sending of generated
> notifications to a subscriber.  This filter mechanism cannot be used to
> select nodes to subscribe to for changes in a datastore.  You need to
> define a separate mechansim for that in yang-push.  (Maybe not even call it
> "filter", but perhaps "selection").
>
> Some comments on subscribed-notifications: This generic mechanism allows
> various filter syntaxes.  This generic mechanism needs to explain what is
> required by a filter syntax definition (an identity, evaluation rules).  In
> section 2.2 the document says that two filter syntaxes are supported, but
> only one is defined (xpath).  It needs to explain that a filter is supposed
> to return true or false (this part RFC 5277 got right, see section 3.6).
> Also, the current module has the filter in an anyxml node; it is not clear
> how an XPath expression is encoded in anyxml.
>
> Please make sure you understand how subtree filters and XPath filters work
> in RFC 5277.  There is nothing wrong with that functionality.
>

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 25 May 2017 00:40:09 +0000

Hi Andy, Alex, & Martin,

From: Andy Bierman, May 24, 2017 6:55 PM

Hi,

<eric> Agree that event tests of both boolean and selection are needed (like RFC-5277), and that both may allow events to pass.  I have a proposal for some text to cover this below (see green).

Notification filters for YANG Push need to select the event-types used,
such as push-change-update.  It would be nice if this was handled automatically,
instead of listing them in the <establish-subscription>.

<evoit> Either you provide a stream or a datastore as a target for the filter.  Even then, a filter-type remains needed to decode the syntax of the filter anyxml.

Traditional notification filters would require that the structure of an event such

<evoit> Agree that a subtree filter for events is needed.

Andy

On Wed, May 24, 2017 at 3:25 PM, Alexander Clemm <[email-masked]<mailto:[email-masked]>> wrote:
Let me briefly summarize where I think we are. The thread has clearly moved on from the initial issue of how to distinguish between causes for updates to no longer include a given data node (because it was deleted, or because it changed its value to no longer match a filter) to other issues.

<evoit> fully agree.

<evoit> I don’t think the semantics are simpler.  In all cases, you already need a filter-type to be able to interpret the filter syntax (e.g., xpath vs. subtree). With this as a given, the current draft’s overloading does have a few benefits:

·        Prohibits population of both an event and a selection filter – there is only filter object.

·        Simple: 1:1 mapping from RPC filter to the single yang data node filter object.

·        It turns out that moving to filter identities from a strict explicit-case based hierarchy simplified the model groupings.  This removed a full page from the YANG model from the previous version.  This is goodness.

·        Nicely extensible via identity hierarchies (rather than also adding a parallel case structure which needs independent treatment).

I can live with it if people want to add an explicit subtyping based on ‘selection’ and ‘test’.  But it seems to add complexity and rigidity to me.

<evoit> There are so many variations of filter types, having two independent, intersecting constructs doesn’t seem beneficial to me.

<eric> Yes.  If we do go with the overloading, here is how I would construct it…:

(a) In subscribed-notifications, update the identity filters so the following are covered:

Identity filter;
    Identity test-filter {base filter;}
       Identity xpath-test-filter {base  test-filter;}
       Identity  subtree-test-filter {base test-filter;}

(b) In YANG Push, update the identity filters so the following are covered:

Identity node-selection-criteria {base sn:filter;}
    Identity subtree-node-selection-criteria {base node-selection-criteria;}
    Identity xpath-node-selection-criteria {base node-selection-criteria;}

(c) in subscribed-notifications, improve the definition of  “anyxml filter” to
"Evaluation criteria encoded in a syntax of a filter-type.  If the filter is applied against an event stream and there is a non-empty or positive result, the event is passed along.  If the filter is applied against a datastore for periodic extracts, the resulting node-set result is passed along. If the filter is applied against a datastore looking for changes, deltas from the last update in the form of a patch result are passed along."

--- Alex

-----Original Message-----
From: Martin Bjorklund [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, May 24, 2017 12:36 AM
To: [email-masked]<mailto:[email-masked]>
Cc: Alexander Clemm <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Netconf] In an update, when is a delete a delete?

"Eric Voit (evoit)" <[email-masked]<mailto:[email-masked]>> wrote:
> > "Eric Voit (evoit)" <[email-masked]<mailto:[email-masked]>> wrote:
> > > > Alexander Clemm <[email-masked]<mailto:[email-masked]>> wrote:
> > > > > Hi Martin,
> > > > >

<evoit> see above.  Selection is now part of the filter identify hierarchy.

I am hoping to leave xpath expression encoding up to the vendor.  I suspect that attempting to come up with some cross-vendor parseable encoding mechanism is unlikely to make the xpath filter definitions themselves cross-vendor anyway.  We should wait for someone who wants to pick up general filtering as a topic rather than addressing this piecemeal.

I understand that.

<evoit> 100% agree.  I will make sure the text covers this.

Eric

No, see above.

/martin

> > > > > From: Martin Bjorklund [mailto:[email-masked]<mailto:[email-masked]>]
> > > > > To: Alexander Clemm <[email-masked]<mailto:[email-masked]>>
> > > > > Cc: [email-masked]<mailto:[email-masked]>
> > > > > Alexander Clemm <[email-masked]<mailto:[email-masked]>> wrote:
> > > > > > Hello all,
> > > > > >

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 25 May 2017 11:09:50 +0200 (CEST)

Hi,

Thank you Alex for a well-written summary.

Alexander Clemm <[email-masked]> wrote:
> Let me briefly summarize where I think we are. The thread has
> clearly moved on from the initial issue of how to distinguish
> between causes for updates to no longer include a given data node
> (because it was deleted, or because it changed its value to no
> longer match a filter) to other issues.
> 
> For those other issues, I think there are in fact two separate items
> that we are trying to discuss at the same time, which are really
> orthogonal to one another: 
> 
> - The first item concerns the concept of a "filter" vs a "selector".
>   A filter is what gets specified for any subscription to
>   notifications, which defines which notifications a subscriber
>   wants to receive.  The filter is applied to the notification as a
>   whole, i.e. either the notification is delivered or it is not.  It
>   does not apply to subsets of contents within the notification.  A
>   selector, on the other hand, is used to specify updates of which
>   data nodes to include in a YANG-push subscription.  The same
>   update notification could include updates of several data nodes,
>   hence a filter applied to the notification as a whole would be
>   inappropriate here - the semantics is slightly different: as a
>   subscriber, "of which data nodes would you like to receive
>   updates", not "which notifications would you like to receive".
> 
> Filters and selectors can be specified using the same syntax.  We
> are faced with a choice between specifying a single construct as
> part of a subscription, which is treated as a filter (in case of a
> "regular" subscription for notification updates) or as a selector
> (in case of a subscription to datastore updates) , or having
> separate objects, i.e. adding a separate "selector" construct for
> YANG-push.  Using a single object amounts to overloading.  It is
> more compact but with arguably a more complex semantics.  Using
> separate object results in a model that is more verbose, but has
> arguably simpler semantics.
> 
>   construct has a dynamic type or a static type.  This is where the
>   issue of identity vs case statement etc comes in.  In case of a
>   dynamic type, we use two objects: One object of a generic type
>   (anydata) holds the filter/selector construct itself, the second
>   object specifies how to interpret it, i.e. which type it is.  (In
>   case of "overloading, we can also make explicit the distinction
>   whether it is a filter or a selector).  That second object is an
>   identityref, referencing one of the identities that designates the
>   filter/selector type.  In case of a static type, we use a case
>   statement (to distinguish which specific type it is).

If you can make the identity-based solution work, that's fine with
me.  But the current text needs to work out the details.  It's not
clear at all what what the contents of the "anyxml" node is supposed
to be.

That said, even the "choice"-based solution is exensible, since it can
be augmented.  Unless it turns out that the identity-based solution
can be made to work well, I would prefer the "choice"-based solution.

> overloaded object that can serve as a filter or a selector depending
> on the context and whether it is used in a notification subscription
> or a YANG-push subscription, and that we use a dynamic type
> including an identityref that indicates whether the object serves as
> a selector or a filter.  From my perspective, I feel that not
> overloading may be conceptually a bit "cleaner", but at the end of
> the day I am fine either way.  And I am not entirely sure, Martin,
> what you are proposing.

I have a strong preference for clear semantics in separate objects.  I
think overloading the semantics in this particular example would be a
huge mistake.  Just to mention one detail, the context for the
filter and selector are completely different - the filter works on the
contents of a notification and the selector works on the contents of a
datastore.  The filter returns a boolean and the selector a node set.
(that was two details :)

/martin

> Either way, we should document the issue
> and our choice clearly.
> 
> --- Alex
> 
>

---

From: Robert Wilton <[email-masked]>
Date: Thu, 25 May 2017 11:15:17 +0100

Hi,

> - The first item concerns the concept of a "filter" vs a "selector".  A filter is what gets specified for any subscription to notifications, which defines which notifications a subscriber wants to…
If you look at this problem in a slightly different way, then I'm not 
sure that "selectors" are necessarily different from "filters".

One could see all subscribers of periodic and on-change YANG Push 
subscriptions to really be recipients of the same single conceptual 
stream of data node change (and periodic) updates from a datastore. This 
conceptual stream of changes can be regarded as an unbounded sequence of 
notifications, where each notification contains a change for one or more 
data nodes.  Under this description, it seems that a YANG Push 
"selector" then acts like a Notification "filter" in that it chooses 
which of those messages in that conceptual stream get sent on to each 
subscriber.

The only difference I see between "selectors" and "filters" is that the 
YANG push messages acted on by a selector COULD be pruned to only 
include the nodes of interest.  But perhaps this distinction is a 
actually a property of the type of notification that is being 
generated.  I.e. whether a notification can be pruned by a 
"filter/selector", or whether it is a binary decision to send or drop 
the notification message (like RFC 5277).

For notifications that could be pruned, then I think that the choice as 
to whether to prune the notification data should be an implementation 
choice, and that it should be acceptable to also send messages that 
include extra nodes that are not of interest, and perhaps in the extreme 
case to send some notifications that don't actually contain any nodes of 
interest at all!

Rob

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 25 May 2017 13:33:21 +0200 (CEST)

Robert Wilton <[email-masked]> wrote:
> > moved on from the initial issue of how to distinguish between causes
> > for updates to no longer include a given data node (because it was
> > deleted, or because it changed its value to no longer match a filter)
> > to other issues.
> >
> > - filter is what gets specified for any subscription to notifications,
> > - which defines which notifications a subscriber wants to receive.  The
> > - filter is applied to the notification as a whole, i.e. either the
> > - notification is delivered or it is not.  It does not apply to subsets
> > - of contents within the notification.  A selector, on the other hand,
> > - is used to specify updates of which data nodes to include in a
> > - YANG-push subscription.  The same update notification could include
> > - updates of several data nodes, hence a filter applied to the
> > - notification as a whole would be inappropriate here - the semantics is
> > - slightly different: as a subscriber, "of which data nodes would you
> > - like to receive updates", not "which notifications would you like to
> > - receive".
> stream of data node change (and periodic) updates from a
> datastore. This conceptual stream of changes can be regarded as an
> unbounded sequence of notifications, where each notification contains
> a change for one or more data nodes.  Under this description, it seems
> that a YANG Push "selector" then acts like a Notification "filter" in
> that it chooses which of those messages in that conceptual stream get
> sent on to each subscriber.

This has been discussed before.  I think it is a convoluted hack.  The
notifications would have to conceptually contain the entire datastore
(entire opertational state for example), and then you'd have to
handwave a bit to produce a pruned notification with just the nodes
the subscriber is interested in.   What's the point anyway when
there's already a clean, simple solution available?

/martin

> The only difference I see between "selectors" and "filters" is that
> the YANG push messages acted on by a selector COULD be pruned to only
> For notifications that could be pruned, then I think that the choice
> as to whether to prune the notification data should be an
> implementation choice, and that it should be acceptable to also send
> messages that include extra nodes that are not of interest, and
> perhaps in the extreme case to send some notifications that don't
> actually contain any nodes of interest at all!
> 
> Rob
>

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 25 May 2017 13:06:21 +0000

> From: Martin Bjorklund, May 25, 2017 5:10 AM
> 
> Hi,
> 
> > object amounts to overloading.  It is more compact but with arguably a
> > more complex semantics.  Using separate object results in a model that
> > is more verbose, but has arguably simpler semantics.
> >
> If you can make the identity-based solution work, that's fine with me.  But the
> current text needs to work out the details.  It's not clear at all what what the
> contents of the "anyxml" node is supposed to be.
> 
> That said, even the "choice"-based solution is exensible, since it can be
> augmented.  Unless it turns out that the identity-based solution can be made
> to work well, I would prefer the "choice"-based solution.
> 
> > or a YANG-push subscription, and that we use a dynamic type including
> > an identityref that indicates whether the object serves as a selector
> > or a filter.  From my perspective, I feel that not overloading may be
> > conceptually a bit "cleaner", but at the end of the day I am fine
> > either way.  And I am not entirely sure, Martin, what you are
> > proposing.
> 
> I have a strong preference for clear semantics in separate objects.  I think
> overloading the semantics in this particular example would be a huge mistake.
> Just to mention one detail, the context for the filter and selector are
> completely different - the filter works on the contents of a notification and the
> selector works on the contents of a datastore.  The filter returns a boolean
> and the selector a node set.
> (that was two details :)

Ok.  Unless anyone objects, let's go with separate objects then.  

We can still retain the identity hierarchy.  The result to would be something like:

In subscribed-notifications...:

Identity filter;

"anyxml filter"
Description "Event stream evaluation criteria encoded in a syntax of a supported type of test-filter.  If the filter is applied against an event stream and there is a non-empty or positive result, the event is passed along."

In YANG Push...:

"anyxml selector"
Description "YANG datastore query encoded in the syntax of a supported type of node-selection-criteria.  If the selector is applied against a datastore for periodic extracts, the resulting node-set result is passed along. If the selector is applied against a datastore looking for changes, deltas from the last update in the form of a patch result are passed along."

Eric

> /martin
> 
> 
> > Either way, we should document the issue and our choice clearly.
> >
> > --- Alex
> >
> >
> > > http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference is
> > > that one results in a xpath node-set expression, and the other an
> > > xpath Boolean expression.  Make sense?
> >
> mechanism, that is used to block/allow the sending of generated notifications
> to a subscriber.  This filter mechanism cannot be used to select nodes to
> subscribe to for changes in a datastore.  You need to define a separate
> mechansim for that in yang-push.  (Maybe not even call it "filter", but perhaps
> "selection").
> >
> allows various filter syntaxes.  This generic mechanism needs to explain what
> is required by a filter syntax definition (an identity, evaluation rules).  In
> section 2.2 the document says that two filter syntaxes are supported, but only
> one is defined (xpath).  It needs to explain that a filter is supposed to return
> true or false (this part RFC 5277 got right, see section 3.6).  Also, the current
> module has the filter in an anyxml node; it is not clear how an XPath
> expression is encoded in anyxml.
> >
> > > I understand and agree on your comment about the context.  The
> > > intent here is to provide equivalent capabilities of a GET.
> >
> > I understand that.
> >
> > > As it would be a
> > > doesn't fall under the umbrella of YANG subscription.  I would be
> > > glad to support someone who wishes to take this up though.
> > >
> > > how to use it.  The preferred embodiment would be to maintain a
> > > local extract of the Publisher's datastore (as defined by the filter).
> > >
> > > > > > But these filters are used by the server to decide if a
> > > > > > certain notification that has been generated will be sent to
> > > > > > the client or not.
> > > > >
> > > > > Yes, the filters in subscribed-notification are supposed to give
> > > > > a boolean indication as to whether a specific event should
> > > > > traverse the filter in its entirety. RFC6241 section 6 subtree
> > > > > filters are written to provide a subset of content.  I suppose
> > > > > it would be possible to define an event-based
> > > > > subtree-filter-type where a non-null result of the subtree
> > > > > filter means that a particular event should traverse that filter.  Is this
> what you are suggesting?
> > > >
> > > > > > subscribe to, I think you need to define new filters, not try
> > > > > > to reuese these notification filters.
> > > > >
> > > > > Filtering syntax is hard, so we have been trying to adopt
> > > > > whatever is available for GET.  This way we don't have to
> > > > > educate users on a new universe of what is possible.  I fully
> > > > > expect that lots of learnings are going to come in the industry
> > > > > here over time, and this will be revisited in the future.
> > > >
> > > > ?
> > > >
> > > > current filter nodes as defined in subscribed-notification cannot
> > > > be used to select nodes to subscribe to for changes.
> > >
> > > > > > As for your question, I think such a filter should be defined
> > > > > > to return a node- set to which the client subscribe to
> > > > > > changes.  If any node (or subnode
> > > > > > to) in
> > > > > Excellent, on-change should only send an update if the results
> > > > > of the subscription filter have changed since the previous push.
> > > > > It is quite possible that an object has been created and then
> > > > > deleted since the last push.
> > > >
> > > > > > > > As part of an on-change subscription, update records
> > > > > > > > reflect the type of change (e.g. whether the value of an
> > > > > > > > object has changed, or whether an object was created or
> > > > > > > > deleted); a subscription allows also to specify whether
> > > > > > > > interested only in specific types of changes (for example,
> > > > > > > > only creates and deleted but no value
> > > > changes).
> > > > > > > >
> > > > > > > > value. This makes it possible to e.g. subscribe to an
> > > > > > > > object "foo" but only if its value is 5.
> > > > > > > >
> > > > > > > > one update, but excluded in another update, due to its
> > > > > > > > value no longer meeting the filter criteria.  For example,
> > > > > > > > if foo's value changes from 5 to 3 in one cycle, a
> > > > > > > > periodic subscription will no longer include foo in its next update.
> > > > > > > > The question now concerns how to properly handle this in
> > > > > > > > the case of an on-change
> > > > subscription.
> > > > > > > >
> > > > > > > > event.  If foo's value changes again from 5 at a later
> > > > > > > > point in time and then changes back to 3 before the time
> > > > > > > > of the update (perhaps because the value changed during
> > > > > > > > the dampening interval), it would be reported as another
> > > > > > > > "delete" event (without ever reporting a create event).
> > > > > > > > On the other hand, if foo's value changed from 3 to
> > > > > > > > >From the perspective of the receiver this may make sense
> > > > > > > > >if it is synching its copy of the state.  However, from
> > > > > > > > >the perspective of the publisher, the object was never
> > > > > > > > >created or deleted - only its value changed, and the case
> > > > > > > > >when the object was truly created or deleted can no
> > > > > > > > >longer be distinguished from the case when its value
> > > > > > changed.
> > > > > > > > >(which does not mean that the object was actually created
> > > > > > > > >- it may have been created, or it may have simply undergone a
> value change).
> > > > > > > >
> > > > > > > > An alternative (let's call it alternative 2) is therefore
> > > > > > > > to make a distinction between whether an object was
> > > > > > > > created or deleted, or whether its value fell in or out of a filter
> range.
> > > > > > > > This appears semantically cleaner.  However, it will
> > > > > > > > require modifying the encoding to allow for distinction
> > > > > > > > between those cases (currently, just plain patch encoding is used).
> > > > > > > >
> > > > > > > > A third alternative is to let filters select only data
> > > > > > > > nodes to subscribe to, and separate out the value filter
> > > > > > > > (or disallow it as a feature altogether).  This
> > > > > > > > alternative has the drawback of being less conceptually

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 26 May 2017 10:10:57 +0200 (CEST)

"Eric Voit (evoit)" <[email-masked]> wrote:
> > be
> > think

I'm not sure why you would have this hierarchy.

I would define a base identity "filter", and describe that a filter
works on the notification content, and returns a boolean if the filter
passed or not.  Also explain how the filter input data is supposed to
be defined (i.e., explain your current anyxml approach (I think you
will see that it doesn't quite works, and you'll end up with specific
per-filter-type parameters - see below)).  Then I'd define "xpath" and
"subtree" based on "filter", and use text from RFC 5277 to explain how
such filters are evaluated (+ more text on XPath, as noted earlier in
this thread.)  This would be done in subscribed-notifications.

In addition to this, in yang-push, I'd define similar identities for
"datastore-selector".

> "anyxml filter"
> Description "Event stream evaluation criteria encoded in a syntax of a
> supported type of test-filter.  If the filter is applied against an
> event stream and there is a non-empty or positive result, the event is
> passed along."
> 
> In YANG Push...:
> 
> Description "YANG datastore query encoded in the syntax of a supported
> type of node-selection-criteria.  If the selector is applied against a
> datastore for periodic extracts, the resulting node-set result is
> passed along. If the selector is applied against a datastore looking
> for changes, deltas from the last update in the form of a patch result
> are passed along."

So you're suggesting that a filter that might take some parameters
would not specify these parameters in YANG, but instead in plain text
-- Suppose I want to define a new regexp-based filter, which takes two
input parameters "dialect" and "expression".  How would this be done
if it has to go into an "anyxml" node?

/martin

> > > filter
> > notifications
> > explain what
> > > work
> > > > > > > > > included in the update record as a "delete" event.  If foo's
> > > > > > > > > value
> > again becomes "5"

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Fri, 26 May 2017 13:39:47 +0000

Hi Martin,

There are two differences in what we are describing:
(#1) Do we have a parent identity which spans both the test filter types and the selection filter types
(#2) Should we expose additional filter parameters within the YANG model

For (#1), I see no harm in providing a parent identity.  Perhaps your concern is documentation compatibility with what was previously driven by 5277?  If that is the case, we could rename the top level filter to "evaluation-criteria".  This would free up "test-filter" to be renamed to "filter".  If this is ok, below is what is in the might be included in subscribed-notifications.  I believe matches to what you are suggesting, with the exception of the top level evaluation-criteria identity. 

  identity evaluation-criteria {
    description
      "Base identity for representing types of filtering or selection syntaxes which may be applied against some target.";
  }
  identity filter {
    base evaluation-criteria;
    description
      "Evaluation criteria used as a pass/fail test against events.  A boolean true, or a non-null result of a node 
       selection will allow events to traverse the filter.";
  }  
  identity subtree-filter {
    base filter;
    description
      "A RFC-6241 based filter which attempts to select nodes within an event.  A successful 
       selection of any nodes results in a positive test result.";
    reference "RFC-6241, #5.1";
  }  
  identity xpath-filter {
    base filter;
    description
      "A filter applied to an event which follows the syntax specified in yang:xpath1.0. Success 
       is indicated by either a positive boolean result, or a non-null node selection.";
    reference "XPATH: http://www.w3.org/TR/1999/REC-xpath-19991116";
  }

For (#2), within the YANG model we expose a target for the evaluation-criteria (i.e., a stream or datastore) and the anydata.  I don't think we should try for more.  The subscription specifications are complex enough already.  

If we try to break out additional parameters from the opaque anydata (either selector or filter), there is going to be a desire to break out even more parameters.  And then there will be attempts to codify valid interplays with these parameters.  There are many lifetimes of complexity here.  Your example of regexp is an excellent one, look at:
https://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines 
it is hard for even a wiki page to maintain links to the variations.

If someone has a proposal for how to parameterize these filtering criteria simply, I would love to listen.  Note: Such a proposal may not disrupt other vendor syntaxes which can be transparently supported via the anyxml.   Absent such a proposal, we should leave the specifics up to implementations for now.  

> From: Martin Bjorklund, May 26, 2017 4:11 AM
> 
> > > > For those other issues, I think there are in fact two separate
> > > > items that we are trying to discuss at the same time, which are
> > > > really orthogonal to one another:
> > > >
> > > > part of a subscription, which is treated as a filter (in case of a "regular"
> > > > subscription for notification updates) or as a selector (in case
> > > > of a subscription to datastore updates) , or having separate objects, i.e.
> > > > adding a separate "selector" construct for YANG-push.  Using a
> > > > single object amounts to overloading.  It is more compact but with
> > > > arguably a more complex semantics.  Using separate object results
> > > > in a model that is more verbose, but has arguably simpler semantics.
> > > >
> > > current text needs to work out the details.  It's not clear at all
> > > what what the contents of the "anyxml" node is supposed to be.
> > >
> > > That said, even the "choice"-based solution is exensible, since it
> > > can be augmented.  Unless it turns out that the identity-based
> > > solution can be made to work well, I would prefer the "choice"-based
> > > solution.
> > >
> > > > overloaded object that can serve as a filter or a selector
> > > > depending on the context and whether it is used in a notification
> > > > subscription or a YANG-push subscription, and that we use a
> > > > dynamic type including an identityref that indicates whether the
> > > > object serves as a selector or a filter.  From my perspective, I
> > > > feel that not overloading may be conceptually a bit "cleaner", but
> > > > at the end of the day I am fine either way.  And I am not entirely
> > > > sure, Martin, what you are proposing.
> > >
> > > I have a strong preference for clear semantics in separate objects.
> > > I think overloading the semantics in this particular example would
> > > be a huge mistake.
> > > Just to mention one detail, the context for the filter and selector
> > > are completely different - the filter works on the contents of a
> > > notification and the selector works on the contents of a datastore.
> > > The filter returns a boolean and the selector a node set.
> > > (that was two details :)
> >
> I would define a base identity "filter", and describe that a filter works on the
> notification content, and returns a boolean if the filter passed or not.  Also
> explain how the filter input data is supposed to be defined (i.e., explain your
> current anyxml approach (I think you will see that it doesn't quite works, and
> you'll end up with specific per-filter-type parameters - see below)).  

Per above, opaque data interpreted by implementations is the plan, unless someone has an alternative proposal. 

> Then I'd
> define "xpath" and "subtree" based on "filter", and use text from RFC 5277 to
> explain how such filters are evaluated (+ more text on XPath, as noted earlier in

Yes, this is the intent.

> In addition to this, in yang-push, I'd define similar identities for "datastore-
> selector".

Yes, this is the intent.
 
> So you're suggesting that a filter that might take some parameters would not
> specify these parameters in YANG, but instead in plain text
> -- Suppose I want to define a new regexp-based filter, which takes two input
> parameters "dialect" and "expression".  How would this be done if it has to go
> into an "anyxml" node?

Vendors will have to specify valid examples of what will be formatted within their anyxml.

Eric

> > > > > > > > > Almost overlooked your question below.  What is meant by
> > > > > > > > > the filter is specified in section 3.5 of the YANG-Push document.
> > > > > > > > >
> > > > > > > > > The following filter types are included in the yang-push
> > > > > > > > > data
> > > > > > > > > model:
> > > > > > > > > [subtree] [xpath]"
> > > > > > > >
> > > > > > > > Actually, only "subtree" is defined in yang-push, "xpath"
> > > > > > > > is defined in subscribed-notifications.
> > > > > > >
> > > > > > "xpath", based on "sn:filter".  So this filter has nothing to
> > > > > > do with selecting nodes in a datastore; this filter is used to
> > > > > > match against a generated notification record.
> > > > >
> > > > > "xpath-selection" and "xpath-boolean".  These would have
> > > > > different definitions, but both reference
> > > > > http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference
> > > > > is that one results in a xpath node-set expression, and the
> > > > > other an xpath Boolean expression.  Make sense?
> > > >
> > > to select nodes to subscribe to for changes in a datastore.  You
> > > need to define a separate mechansim for that in yang-push.  (Maybe
> > > not even call it "filter", but perhaps "selection").
> > > >
> > > syntaxes are supported, but only one is defined (xpath).  It needs
> > > to explain that a filter is supposed to return true or false (this
> > > part RFC 5277 got right, see section 3.6).  Also, the current module
> > > has the filter in an anyxml node; it is not clear how an XPath
> > > > > huge undertaking to try to consolidate an industry-wide view of
> > > > > the minimal xpath syntax and capabilities in networking
> > > >
> > > > > doesn't fall under the umbrella of YANG subscription.  I would
> > > > > be glad to support someone who wishes to take this up though.
> > > > >
> > > > > > > >  [Side note - I think this is wrong,
> > > > > > > > certain notification that has been generated will be sent
> > > > > > > > to the client or not.
> > > > > > >
> > > > > > > Yes, the filters in subscribed-notification are supposed to
> > > > > > > give a boolean indication as to whether a specific event
> > > > > > > should traverse the filter in its entirety. RFC6241 section
> > > > > > > 6 subtree filters are written to provide a subset of
> > > > > > > content.  I suppose it would be possible to define an
> > > > > > > event-based subtree-filter-type where a non-null result of
> > > > > > > the subtree filter means that a particular event should
> > > > > > > traverse that filter.  Is this
> > > what you are suggesting?
> > > > > >
> > > > > > Yes.  Note that this is already provided by RFC 5277, and I
> > > > > > have always assumed that this new work will provide at least
> > > > > > the same functions as RFC
> > > > > I also want to make sure that a non-null result from a filter
> > > > > allows the event to pass.  I suspect that an xpath Boolean
> > > > > filter could be designed to do this, but I will tweak the
> > > > > subscribed-notifications text so that unnecessary filtering
> > > > > expression complexity is not artificially required.
> > > >
> > > > > > > > subscribe to, I think you need to define new filters, not
> > > > > > > > try to reuese these notification filters.
> > > > > > >
> > > > > > > educate users on a new universe of what is possible.  I
> > > > > > > fully expect that lots of learnings are going to come in the
> > > > > > > industry here over time, and this will be revisited in the future.
> > > > > >
> > > > > > ?
> > > > > >
> > > > > > I am not proposing any new filter syntax.  I am saying that
> > > > > > the current filter nodes as defined in subscribed-notification
> > > > > > cannot be used to select nodes to subscribe to for changes.
> > > > >
> > > > > > > > As for your question, I think such a filter should be
> > > > > > > > defined to return a node- set to which the client
> > > > > > > > subscribe to changes.  If any node (or subnode
> > > > > > > > to) in
> > > > > > > Excellent, on-change should only send an update if the
> > > > > > > results of the subscription filter have changed since the previous
> push.
> > > > > > > It is quite possible that an object has been created and
> > > > > > > then deleted since the last push.
> > > > > >
> > > > > > I don't understand what you're trying to say with these
> > > > > > sentences (but since the first word was "Excellent" maybe it's
> > > > > > ok ;)
> > > > > >
> > > > > >
> > > > > > /martin
> > > > > >
> > > > > >
> > > > > > > > > > (draft-ietf-netconf-yang-push), we have come across
> > > > > > > > > > one issue that we wanted to raise with the working group.
> > > > > > > > > >
> > > > > > > > > > reflect the type of change (e.g. whether the value of
> > > > > > > > > > an object has changed, or whether an object was
> > > > > > > > > > created or deleted); a subscription allows also to
> > > > > > > > > > specify whether interested only in specific types of
> > > > > > > > > > changes (for example, only creates and deleted but no
> > > > > > > > > > value
> > > > > > changes).
> > > > > > > > > >
> > > > > > > > > > At the same time, a subscription filter specifies
> > > > > > > > > > which objects to include as part of a subscription and which
> not.
> > > > > > > > >
> > > > > > > > > "ietf-subscribed-notifications:xpath" filter type (which
> > > > > > > > > btw is sorely underspecified).  Section 2.2 of
> > > > > > > > > It's not clear what this means, but my guess is that
> > > > > > > > > this is supposed to work like the old RFC 5277 filters,
> > > > > > > > > where the filter expression is evaluated on the
> > > > > > > > > notification contents, and if the expression returns
> > > > > > > > > "true" (for XPath filters this means converting the
> > > > > > > > > results to a boolean), then the notification is sent, otherwise not.
> > > > > > > > >
> > > > > > > > > But it seems you are referring to some other filter
> > > > > > > > > which would be used to select a node set for which
> > > > > > > > > changes are reported?
> > > > > > > > >
> > > > > > > > > I would like to understand which filter mechanism you
> > > > > > > > > mean before having an opinion in this matter.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > /martin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > (Really, it is not
> > > > > > > > > > independently of the filter, than it is a policy of
> > > > > > > > > > which objects to include as part of subscribed update
> > > > > > > > > > records.) However, a subscription filter (such as
> > > > > > > > > > XPath) can be used to also specify a value filter,
> > > > > > > > > > which will include or exclude objects based on their
> > > > > > > > > > current value. This makes it possible to e.g.
> > > > > > > > > > subscribe to an object "foo" but only if its value is 5.
> > > > > > > > > >
> > > > > > > > > > Now, this means that the same object could be included
> > > > > > > > > > in one update, but excluded in another update, due to
> > > > > > > > > > its value no longer meeting the filter criteria.  For
> > > > > > > > > > example, if foo's value changes from 5 to 3 in one
> > > > > > > > > > cycle, a periodic subscription will no longer include
> > > > > > > > > > foo in its next update.
> > > > > > > > > > The question now concerns how to properly handle this
> > > > > > > > > > in the case of an on-change
> > > > > > subscription.
> > > > > > > > > >
> > > > > > > > > > One possibility concerns reporting the fact that "foo"
> > > > > > > > > > no longer meets the subscription criteria and is no
> > > > > > > > > > longer included in the update record as a "delete"
> > > > > > > > > > event.  If foo's value
> > > again becomes "5"
> > > > > > > > > > point in time and then changes back to 3 before the
> > > > > > > > > > time of the update (perhaps because the value changed
> > > > > > > > > > during the dampening interval), it would be reported
> > > > > > > > > > as another "delete" event (without ever reporting a create
> event).
> > > > > > > > > > 6 and back again, nothing would be reported because it
> > > > > > > > > > did not meet the filter criteria at any point in time.
> > > > > > > > > >
> > > > > > > > > > >From the perspective of the receiver this may make
> > > > > > > > > > >sense if it is synching its copy of the state.
> > > > > > > > > > >However, from the perspective of the publisher, the
> > > > > > > > > > >object was never created or deleted - only its value
> > > > > > > > > > >changed, and the case when the object was truly
> > > > > > > > > > >created or deleted can no longer be distinguished
> > > > > > > > > > An alternative (let's call it alternative 2) is
> > > > > > > > > > therefore to make a distinction between whether an
> > > > > > > > > > object was created or deleted, or whether its value
> > > > > > > > > > fell in or out of a filter
> > > range.
> > > > > > > > > > require modifying the encoding to allow for
> > > > > > > > > > distinction between those cases (currently, just plain
> > > > > > > > > > patch encoding is used).
> > > > > > > > > >
> > > > > > > > > > nodes to subscribe to, and separate out the value
> > > > > > > > > > filter (or disallow it as a feature altogether).  This
> > > > > > > > > > alternative has the drawback of being less

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 26 May 2017 16:22:30 +0200 (CEST)

Hi,

"Eric Voit (evoit)" <[email-masked]> wrote:
> For (#1), I see no harm in providing a parent identity.

I do, since there is no benefit in a common parent identity.  It just
adds complexity to what's really a simple model.

> Perhaps your
> concern is documentation compatibility with what was previously driven
> by 5277?

No.

> If that is the case, we could rename the top level filter to
> "evaluation-criteria".  This would free up "test-filter" to be renamed
> to "filter".  If this is ok, below is what is in the might be included
> in subscribed-notifications.  I believe matches to what you are
> suggesting, with the exception of the top level evaluation-criteria
> identity.
> 
>       "Base identity for representing types of filtering or selection
>       syntaxes which may be applied against some target.";
>   }
>   identity filter {
>       "Evaluation criteria used as a pass/fail test against events.  A
>       boolean true, or a non-null result of a node
> For (#2), within the YANG model we expose a target for the
> evaluation-criteria (i.e., a stream or datastore) and the anydata.  I
> don't think we should try for more.  The subscription specifications
> are complex enough already.
> 
> If we try to break out additional parameters from the opaque anydata
> (either selector or filter), there is going to be a desire to break
> out even more parameters.  And then there will be attempts to codify
> valid interplays with these parameters.

I don't understand what this means.  Maybe you can provide an example
by using my proposal below?

> There are many lifetimes of
> complexity here.  Your example of regexp is an excellent one, look at:

This is completely missing my point, which has to do with how filter
parameters are specified for new filters; not that there are many
regexp dialects.

> If someone has a proposal for how to parameterize these filtering
> criteria simply, I would love to listen.  Note: Such a proposal may
> not disrupt other vendor syntaxes which can be transparently supported
> via the anyxml.  Absent such a proposal, we should leave the specifics
> up to implementations for now.

[...]

> Vendors will have to specify valid examples of what will be formatted

Specification by example is not very appealing IMO.   Ok, here's a
proposal for how to do this:

  leaf filter-type {
    mandatory true;
    type identityref {
      base filter;
    }
  }
  choice filter-parameters {
    mandatory true;
    case subtree-parameters {
      when 'derived-from-or-self(../filter-type,
                                 "sn:subtree-filter")';
      anydata subtree {
        description "<copy text from 5277>";
      }
    }
    case xpath-parameters {
      when 'derived-from-or-self(../filter-type,
                                 "sn:xpath-filter")';
      leaf xpath {
        type yang:xpath1.0;
        description
          "<explain the XPath context, copy text from 5277>";
      }
    }
  }

Now, if a vendor/sdo wants to add a new filter type, it can be done
like this:

  identity regexp-filter {
    base "sn:filter";
    description
      "...";
  }
  augment "/sn:filters/sn:filter/sn:filter-parameters" {
    when 'derived-from-or-self(../filter-type,
                               "rx:xpath-filter")';
    leaf regexp-dialect {
      type enumeration {
        enum "xsd";
        enum "posix";
        ...
      }
    }
    leaf regexp-expression {
      type string;
    }
  }

Once we have this, we can simplify by removing the identity; it isn't
really necessary.  (IIRC this was suggested by Juergen earlier in this
thread.)

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 26 May 2017 17:29:36 +0200 (CEST)

Hi,

See below for a correction.

>     when 'derived-from-or-self(../filter-type,

Here there should be a case:

  case regexp-parameters {

  }

/martin

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Fri, 26 May 2017 20:15:30 +0000

Hi Martin,

Good discussion.  Some thoughts in-line...

> From: Martin Bjorklund, May 26, 2017 11:30 AM
> 
> Hi,
> 

There are three options for traversing the filter criteria subtyping:
(a) filter sub-typing defined by a multi-tier identity hierarchy
(b) explicit subtyping for selection vs. test, with independent identity sets
(c) explicit subtyping for all current and future types of filters

If we ever use (a) in any form, there is value in a parent identity.  As the current working draft is based (b), I will pull it out.  We can always re-add in parallel with (b) if someone expresses a business need.

> > > Perhaps your
> > > concern is documentation compatibility with what was previously
> > > driven by 5277?
> >
> > No.
> >
> > > "evaluation-criteria".  This would free up "test-filter" to be
> > > renamed to "filter".  If this is ok, below is what is in the might
> > > be included in subscribed-notifications.  I believe matches to what
> > > you are suggesting, with the exception of the top level
> > > evaluation-criteria identity.
> > >
> > > evaluation-criteria (i.e., a stream or datastore) and the anydata.
> > > I don't think we should try for more.  The subscription
> > > specifications are complex enough already.
> > >

Looking at your regexp example, I now see that by 'parameters', you also meant that you wanted a different yang object type for each type of supported filter.  I.e., you want (c).  My earlier interpretation was that you wanted (b) with additional parameters to break down the contents and interworkings of a common anydata event filter object.

The explicit subtyping with different object types that you have below close to what we had up through yang-push draft -v04 (minus your addition of the case regexp & dialect parameter).   Draft -v05 went the other way to (a) by minimizing the number and potential growth of filter object types via the addition of the filter-type.  Working draft -v06 is of (b), and is attempting to balance these needs.  The cost of both (a) and (b) is that we have a new object type, and the application must validate the filter-type syntax -- foregoing datatype enforcement by the model.  But as the application will need to do its own syntax checks for the enforcement for platform specific filtering capabilities anyway, this doesn't seem to be a bad tradeoff.

BTW: I also would have also leaned towards explicit subtyping of option (c) if it were definitively possible to know whether a platform could support a specific filter back at the subscriber.  But all filters will still need publisher validation, so yang data-type enforcement provides only minimal protections.   

So with (b) vs (c), it feels like we are coming a model style preference.  I have a preference for (b): the one with fewer object types, and the one where it is the identities which might grow over time.  If we remain with (b), the interface specification should be less verbose.

> > > https://en.wikipedia.org/wiki/Comparison_of_regular_expression_engin
> > > es it is hard for even a wiki page to maintain links to the
> > > variations.
> >
> > > not disrupt other vendor syntaxes which can be transparently
> > > supported via the anyxml.  Absent such a proposal, we should leave
> > > the specifics up to implementations for now.
> >
> > [...]
> >

This is helpful.  I now understand what you are proposing.

> Here there should be a case:

I like this augmentation structure.  With this, it is possible for vendor specific augmented objects to be added the model, versus parameters being embedded in the anydata.  Augmentations like this can be done with (b) too should people agree on filter parameters needing to be exposed.  Hopefully type of augmentation this can hold us until the time someone picks up filters holistically.

Eric

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 29 May 2017 16:02:16 +0200 (CEST)

Hi Eric,

See comments inline.

> (c) explicit subtyping for all current and future types of filters

It would help if you could provide valid YANG snippets of these three
options.  I *think* I know what you mean with (a) and (b), but not
(c).

> If we ever use (a) in any form, there is value in a parent identity.

What does this mean?

> As the current working draft is based (b), I will pull it out.

Hmm, are you saying that you will _not_ use (b)?

> We can
> always re-add in parallel with (b) if someone expresses a business
> need.

What does this mean?

> > > >       "A filter applied to an event which follows the syntax specified
> > > >       in
> > > >       yang:xpath1.0. Success
> > > >        is indicated by either a positive boolean result, or a non-null
> > > >        node
> > > >        selection.";
> Looking at your regexp example, I now see that by 'parameters', you
> also meant that you wanted a different yang object type for each type
> of supported filter.  I.e., you want (c).  My earlier interpretation
> was that you wanted (b) with additional parameters to break down the
> contents and interworkings of a common anydata event filter object.

Hmmm.  I now realize that I probably do not understand what (a) and
(b) are.  I thought my proposal was (b)...

> The explicit subtyping with different object types that you have below
> close to what we had up through yang-push draft -v04 (minus your
> addition of the case regexp & dialect parameter).  Draft -v05 went the
> other way to (a) by minimizing the number and potential growth of
> filter object types via the addition of the filter-type.  Working
> draft -v06 is of (b), and is attempting to balance these needs.  The
> cost of both (a) and (b) is that we have a new object type, and the
> application must validate the filter-type syntax -- foregoing datatype
> enforcement by the model.

Can you explain what this last sentence means?

> But as the application will need to do its
> own syntax checks for the enforcement for platform specific filtering
> capabilities anyway, this doesn't seem to be a bad tradeoff.
> 
> BTW: I also would have also leaned towards explicit subtyping of
> option (c) if it were definitively possible to know whether a platform
> could support a specific filter back at the subscriber.

Can you explain the problem in more details?  There might be a
solution to this.

/martin

> But all
> filters will still need publisher validation, so yang data-type
> enforcement provides only minimal protections.
> 
> So with (b) vs (c), it feels like we are coming a model style
> preference.  I have a preference for (b): the one with fewer object
> types, and the one where it is the identities which might grow over
> time.  If we remain with (b), the interface specification should be
> less verbose.
> 
> I like this augmentation structure.  With this, it is possible for
> vendor specific augmented objects to be added the model, versus
> parameters being embedded in the anydata.  Augmentations like this can
> be done with (b) too should people agree on filter parameters needing
> to be exposed.  Hopefully type of augmentation this can hold us until
> the time someone picks up filters holistically.
> 
> Eric  
> 
> > >   }
> > >
> > 
> > 
> > /martin
> > 
> > 
> > 
> > >

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 30 May 2017 16:37:41 +0000

Hi Martin,

I have updated the yang models for:

Subscribed notifications:
https://github.com/netconf-wg/rfc5277bis/blob/master/ietf-subscribed-notifications%402017-05-25.yang
Yang-push: 
https://github.com/netconf-wg/yang-push/blob/master/ietf-yang-push%402017-05-26.yang

To provide details.  Some more thoughts in-line.

> From: Martin Bjorklund, May 29, 2017 10:02 AM
> 
> Hi Eric,
> 
> See comments inline.
> 
> > > > > (#1) Do we have a parent identity which spans both the test
> > > > > filter types and the selection filter types
> > > > I do, since there is no benefit in a common parent identity.  It
> > > > just adds complexity to what's really a simple model.
> >
> It would help if you could provide valid YANG snippets of these three options.  I
> *think* I know what you mean with (a) and (b), but not (c).
> 

See two comments below.
 
> Hmm, are you saying that you will _not_ use (b)?

No, the current models embody (b).   
 
> > always re-add in parallel with (b) if someone expresses a business

We could have a common identity spanning the set of event and datastore filter identities.  This could be useful if anyone wanted to use this abstraction.  But this is not in the current models.  So we can ignore this for now.

> > > > > renamed to "filter".  If this is ok, below is what is in the
> > > > > might be included in subscribed-notifications.  I believe
> > > > > matches to what you are suggesting, with the exception of the
> > > > > top level evaluation-criteria identity.
> > > > >
> > > > > If we try to break out additional parameters from the opaque
> > > > > anydata (either selector or filter), there is going to be a
> > > > > desire to break out even more parameters.  And then there will
> > > > > be attempts to codify valid interplays with these parameters.
> > > >
> > > > I don't understand what this means.  Maybe you can provide an
> > > > example by using my proposal below?
> >

Your proposal looked like (c) to me.  This is because you have explicit subtyping for different types of event filters: i.e., anydata subtree and leaf xpath (xpath1.0).  See linked yang files above for an example of (b).

> Can you explain what this last sentence means?

The yang model itself won't constrain to things like "type yang:xpath1.0" for individual filter types.

> > But as the application will need to do its own syntax checks for the
> > enforcement for platform specific filtering capabilities anyway, this
> > doesn't seem to be a bad tradeoff.
> >
> Can you explain the problem in more details?  There might be a solution to this.

There are many checks which need to be made on a subscription filter before it is known to be acceptable.  For example: does it match the overall syntax of the filter type, does it match to the subset of the syntax supported by a platform, is the nesting, or number of parameters too complex to be supported by a platform, is the result of a selection filter too large to be handled by a platform, are the total resources taken by the filter too high relative to available capacity, etc.   Only the first of these can be addressed by strong yang typing.

Eric

> > > > > https://en.wikipedia.org/wiki/Comparison_of_regular_expression_e
> > > > > ngin es it is hard for even a wiki page to maintain links to the
> > > > > variations.
> > > >
> > > > This is completely missing my point, which has to do with how
> > > > filter parameters are specified for new filters; not that there
> > > > are many regexp dialects.
> > > >
> > > > > If someone has a proposal for how to parameterize these
> > > > > filtering criteria simply, I would love to listen.  Note: Such a
> > > > > proposal may not disrupt other vendor syntaxes which can be
> > > > > transparently supported via the anyxml.  Absent such a proposal,
> > > > > we should leave the specifics up to implementations for now.
> > > >
> > > > [...]
> > > >
> > > > Now, if a vendor/sdo wants to add a new filter type, it can be
> > > > done like this:
> > > >
> > > >   identity regexp-filter {
> > > >     base "sn:filter";
> > > >     description
> > > >       "...";
> > > >   }
> > > > Once we have this, we can simplify by removing the identity; it
> > > > isn't really necessary.  (IIRC this was suggested by Juergen
> > > > earlier in this
> > > > thread.)
> > > >
> > > >
> > > >
> > > > /martin
> > > >
> > > >
> > > >
> > > >