Subject: RE: sub-tree filtering proposals

From: "Glenn Waters" <[email-masked]>
Date: Fri, 4 Jun 2004 16:23:08 -0400

I guess the point around the XPath proposal has always been why don't we
line up with an existing standard, even if we only select a subset of that
standard. That way we get the best of both worlds -- we get just the
features we want and we don't have to reinvent the wheel. Further, if at
some point down the road we decide we want more powerful filtering/selection
then we can go back to the trough and sip a bit more of the XPath juice.

With the approach I mention above the opportunity at least exists to attempt
to use some existing tools and open source.

Regards, /gww 

> -----Original Message-----
> From: Andy Bierman [mailto:[email-masked]]
...
[lots of comments by Andy, Wes, Jeurgen, et. al.]

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 Jun 2004 13:34:55 -0700

At 01:23 PM 6/4/2004, Glenn Waters wrote:

>I guess the point around the XPath proposal has always been why don't we line up with an existing standard, even if we only select a subset of that standard. That way we get the best of both worlds -- we get just the features we want and we don't have to reinvent the wheel. Further, if at some point down the road we decide we want more powerful filtering/selection then we can go back to the trough and sip a bit more of the XPath juice.

There are two problems with this approach:
1) It might not be easy to get the WG to agree on a minimal subset
2) An "arbitrary" subset of Xpath isn't likely to be easily supported
   by off-the-shelf Xpath tools

>With the approach I mention above the opportunity at least exists to attempt to use some existing tools and open source.
>
>Regards, /gww 

Andy

>> From: Andy Bierman [<mailto:[email-masked]>mailto:[email-masked]] 
>... 

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Wes Hardaker <[email-masked]>
Date: Fri, 04 Jun 2004 14:01:40 -0700

>>>>> On Fri, 04 Jun 2004 13:34:55 -0700, Andy Bierman <[email-masked]> said:

Andy> 2) An "arbitrary" subset of Xpath isn't likely to be easily supported
Andy> by off-the-shelf Xpath tools

It's likely most xpath implementations will support the subset you
need (as opposed to the worse case of trying to get all xpath
implementations to support all the xpath functionality in an
interoperable way).

vendors that want to only include the minimal part of xpath to reduce
size, which I suspect is your point, could quite possibly find the
sections of code they don't want and remove it [requires code access
or a vendor willing to do the work for you of course].  In general,
It's a heck of a lot easier to remove something from something you
have too much of than create something for something you don't yet
have or don't yet have all of.

-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

---

From: "David T. Perkins" <[email-masked]>
Date: Fri, 04 Jun 2004 14:28:45 -0700

Hi,

On the subseting, you just need to make sure that there is an error code
that says that the XPATH might be valid, but the agent implementation
doesn't support it. 

(And, of course, there will be people that will say that subsetting
must not be allowed, since memory, CPU, and network bandwidth is
free, and, thus, subsetting require more work to be done in figuring
out how to take a "perfectly reasonable XPATH specification"
and splitting it into several XPATH specifications, and combining
the results.)

>have or don't yet have all of.
Regards,
/david t. perkins

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 Jun 2004 15:15:28 -0700

>and splitting it into several XPATH specifications, and combining

How about if you, Glenn, and Wes write up a detailed specification 
of the exact Xpath subset a NETCONF agent must support, with
rationale and examples?  We need concrete proposals to consider
this approach any further.  If nobody's willing to write up a
solution proposal, then the issue is moot.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 5 Jun 2004 09:27:25 +0200

>    by off-the-shelf Xpath tools

Off-the-shelf XPATH tools will most likely support more than the subset
so I fail to see why this is a problem. In fact, I would like to support
full XPATH in an implementation since we have XPATH anyway (since we use
libxml). Having to support another subtree filtering mechanism is just
added code and complexity for no gain. So hopefully the subtree filtering
maps to an XPATH subset (in which case we can just map and use XPATH
for everything). (But if it maps to an XPATH subset, then of course the
question is why we do subtree filtering in the first place...)

/js

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen, Germany

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 5 Jun 2004 13:46:11 +0200

> 1) It might not be easy to get the WG to agree on a minimal subset

So here is a proposal. Netconf implementations must support filter
expressions that are the following subset of XPATH expressions. The
subset consists of the following features:

1) Absolute and relative paths (where path elements are element 
   names):

   //users/user/name
   /users/user/name
   //name

2) Element value selectors:

   //users/user/name
   //users/user[name="fred"]
   //users/user[name="fred"]/name

3) Attributes selectors:

   //interface[@ifName="eth0"]

4) Expressions may contain multiple path expressions, separated by 
   a | symbol

   //users/user[name="root"]/type | //interface[@ifName="eth0"]

This set of features covers all the examples Andy posted and thus seems 
to be sufficient. ;-) What it boils down to is to select stuff by path,
element values and attribute values.

Sure, this needs to be written down more formally and in a way that is
consistent with the XPATH specification. I am willing to give that a
try if people can agree that this is indeed a reasonable XPATH subset
that is easy to implement and serves our purpose.

/js

---

From: Andy Bierman <[email-masked]>
Date: Mon, 07 Jun 2004 10:23:17 -0700

>On Fri, Jun 04, 2004 at 01:34:55PM -0700, Andy Bierman wrote:

I have asked the XML Directorate to consider the subtree proposal
and this Xpath subset proposal, and give their opinion.
If they think we should use this Xpath subset, then I don't
mind choosing Xpath subset over subtree filtering.

>   //interface[@ifName="eth0"]

How can multiple element and/or attribute selectors be combined?

>element values and attribute values.

I think this is a good minimal subset.  It's actually way more
powerful than what we have now with SNMP or CLI.  

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 7 Jun 2004 23:03:05 +0200

> How can multiple element and/or attribute selectors be combined?

As Joel M. Halpern already pointed out in private email, you can
have multiple [] selections, for example:

//interface[type="ethernet"][connector="present"]

I am assuming here an interface representation such as:

    <interface>
      <name>eth0</name>
      <type>ethernet</type>
      <connector>present</connector>
      <!-- lots of other stuff -->
    </interface>

The same works for attributes and things can even be nested. Assuming
an SNMP message in XML format, a valid expression would be:

//snmp[version="1"][community="public"]/pdu[@type="getnext"]

This would match a document such as the following:

    <snmp length="42">
      <version>1</version>
      <community>public</community>
      <pdu type="getnext">
        <request-id>1804289385</request-id>
        <error-status>0</error-status>
        <error-index>0</error-index>
        <varbind>
          <name>1.3.6.1.2.1.1.3</name>
          <value/>
        </varbind>
      </pdu>
    </snmp>

[Don't worry about the strange example - this is simply copied from some
 stuff I am working on today and by no means represents any reasonable 
 data model for netconf.]

/js

---

From: Frank Strauß <[email-masked]>
Date: Tue, 08 Jun 2004 12:36:23 +0200

>> How can multiple element and/or attribute selectors be combined?

I'm not sure, whether the question is about XPath in general or a proposed
specific subset/derivation functionality. In case of true XPath, I'd like
to add this information...

> //interface[type="ethernet"][connector="present"]

This kind of operation first builds the set of all <interface>s which is
then limited to those with type="ethernet" and this subset is then limited
to those with connector="present". This means a sequence of predicates is
not commutative, in contrast to the logical AND operator. An example where
this would be signifcant is this:
//interface[type="ethernet"][position()=last()] would address the last
ethernet interface, while
//interface[position()=last()][type="ethernet"] would address the last
interface, if it is an ethernet interface, otherwise an empty node set.

Xpath predicates (the parts in brackets) may contain boolean operators
("expr1 and expr2","expr1 or expr2", "not(expr)").

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 9 Jun 2004 12:54:00 +0200

> ("expr1 and expr2","expr1 or expr2", "not(expr)").

True in general. 

However, the proposed required minimal subset does not include things 
like last() or position() - only simple comparisons of element/attribute
values (no other operators). Do you think the fact that sequences of 
predicates are not commutative is still important in this special 
restricted case?

/js

---

From: Frank Strauß <[email-masked]>
Date: Wed, 09 Jun 2004 13:44:30 +0200

> predicates are not commutative is still important in this special 

I'm sorry, I did not follow the discussion and reasoning on the proposed XPath
subset. From the operator's point of view I would deeply regret if netconf
would come up with such a derivation from the standard. It reminds me on a
phrase from the (earlier) SMI specs: "...using an adapted subset..." which
caused many misunderstandings in the past and made it impossible to use
existing tool chains. I hope we have learned enough from those problems,
especially since probably the most important advantage of XML based protocols
is the ease of application and broad knowledge on its base specs.

In general, a service might (be required to) supply *more* than a standard, and
a client implementation might make use of less than the standard. But if a
service is allowed to support *less* than the standard, it is simply not
supporting that standard and causes trouble to client implementors.

To (kind of) answer the question: In order to (a) make the expressions look
more like what people who are familiar with XPath would expect, and to (b)
support also an "or" operator (and a "not" operator), I would prefer the
notation with the according operators over the sequence of predicates. If only
an "and" operator would be required, I wouldn't care, since it would not be
XPath, anyway.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 09 Jun 2004 08:42:04 -0700

>is the ease of application and broad knowledge on its base specs.

Juergen is not proposing an adapted subset.  It is a pure subset.
The main argument for Xpath subset here is that it will be easier
to adapt existing tools than to build new ones from scratch.
Eventually, more platforms will be able to support full Xpath,
so the minimum subset may become a non-factor over time. 

Clearly, every NETCONF developer would need to know about the Xpath
subset.  Limiting usage to the NETCONF subset is not an issue 
for developers generating Xpath expressions themselves or writing
them manually.  If an NMS developer (using an application to generate
Xpath expressions) limits the selection criteria to the subset
that NETCONF supports, then what is the likelihood that the
application will generate expressions outside the subset?

Obviously, an application that supports full Xpath will be
capable of offering more sophisticated selection features
than NETCONF supports.  In theory, if the developer avoids
those features, the application should inter-operate with
the NETCONF agent.  In practice, it's not going to be this simple.

>an "and" operator would be required, I wouldn't care, since it would not be

We do not have WG consensus that all devices can support,
and all applications need, full Xpath.  I think we have
consensus for a limited set of features:
  - select all nodes matching specified attribute or simple content values
  - select a specific instance of an object, in a manner independent
    of any instance naming scheme
  - select specific nodes by name
  - select by any combination of the above

This is the feature set we have been assuming via the subtree filtering
examples all along.  If some people think we need more powerful
selection criteria as the minimum feature set, then speak up.

Juergen's Xpath subset seems to be enough to support this feature set.
I think the subtree filtering supports it as well.

My biggest concern is that we forget that the target platforms
for NETCONF agent are embedded operating systems within network 
devices, and small devices don't need the same complex features 
that large devices may need.

IMO, we should do the following (yet another proposal :-)

  - mandatory-to-implement subtree node selection
  - optional-to-implement "full Xpath 1.0" node selection
    - #xpath capability set if this is supported

The XML Directorate consensus seems to be leaning to partial
Xpath, but there are strong concerns that an Xpath subset will be 
incompatible with standard tools, and therefore pointless.

The advantages of the proposal above are:
  - subtree filtering is easy to implement on agents;
    only XML parser needed, not XML and Xpath parsers
  - subtree expressions are valid XML which look identical
    to the data models (XML content) that operators will
    need to know anyways.  Xpath should not be mandatory-to-know
    in order to use NETCONF.
  - Vendors are encouraged to also support full Xpath 1.0, if it
    is appropriate for the platform and data model size
  - We avoid definition and deployment of a NETCONF-only Xpath subset

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 8 Jun 2004 09:26:51 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "Frank Strauß" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, June 09, 2004 8:42 AM
> Subject: Re: sub-tree filtering proposals
...
...

I think this helps a lot, since it permits agent developers to use
off-the-shelf code with little or no modification.  The downside
is that management applications would probably be written to
work with the minimum subset, and not take advantage of
the additional features.

Would it make sense for there to be a capability that indicates
whether the filter support is the basic subset or full Xpath?
(If absent, the assumption would be "basic subset.")

Randy

---

From: Sandeep Adwankar <[email-masked]>
Date: Wed, 09 Jun 2004 11:45:25 -0500

> to adapt existing tools than to build new ones from scratch.

The argument to adapt existing tools is weak.
It means that I need to rebuild xerces parser to support this
subset while still struggling to keep it within the device memory limits.
I rather prefer using xerces as it is or not to use it at all.

>   - select by any combination of the above

I wonder if (future) NetConf Data model will support all these features,
or should it be part of the protocol specification. Won't these be
requirements for NetConf data model?

>     - #xpath capability set if this is supported

I like second part: #xpath capability set that supports full XPath
for devices that can house it.

Thanks
Sandeep

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 8 Jun 2004 09:53:08 -0700

Hi -

> From: "Randy Presuhn" <[email-masked]>
> To: <[email-masked]>
> Sent: Tuesday, June 08, 2004 9:26 AM
...
...

Doh. That's what Andy proposed a couple of lines earlier.
They say short-term memory is the first to go.  :-)

Randy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 9 Jun 2004 18:40:53 +0200

> (If absent, the assumption would be "basic subset.")

This is what was under discussion as far as I remember.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 9 Jun 2004 15:42:22 +0200

> is the ease of application and broad knowledge on its base specs.

First, I believe a subset of XPATH (not an adapted subset) is still better 
than something homegrown. (The most vocal members of the WG do not seem
to buy full XPATH as a requirement.) Personally, I believe that the
operators using netconf will ask for full XPATH anyway, so the subset
requirement will likely not be a real issue in practice.
 
> In general, a service might (be required to) supply *more* than a standard, 
> and a client implementation might make use of less than the standard. But 
> if a service is allowed to support *less* than the standard, it is simply 
> not supporting that standard and causes trouble to client implementors.

I think the general idea is that boxes announce whether they do support full
XPATH or just a reduced subset. I do not see that there is anything wrong
with this.

The alternative, a home grow filter format which implementations are 
required to support, seems worse to me. Bottom line is that either 
(a) the netconf WG makes a decision to require XPATH full stop, or
(b) the WG makes a decision to support XPATH and require a subset of it, or
(c) the WG makes a decision to support XPATH and to require another 
    home-grown filter format.

Since (a) does not seem to be getting WG concensus, I believe (b) is the
best we can do.

> notation with the according operators over the sequence of predicates. If 
> only an "and" operator would be required, I wouldn't care, since it would 
> not be XPath, anyway.

Point taken. I can live either way. I just do not want to see (c)
happen.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 9 Jun 2004 18:22:47 +0200

>     - #xpath capability set if this is supported

So what is "subtree node selection" compared to "subtree filtering"?
I am confused now. Do you refer to your original selection proposal
now?

>     only XML parser needed, not XML and Xpath parsers

Come on, parsing Xpath expression is really not that complicated. I am
really wondering on which basis you calculate how much you save on
embedded boxes...

>     need to know anyways.  Xpath should not be mandatory-to-know

In case "subtree expressions" means just expressions of the form
"/foo/bar", then we are back in SNMP stone age where everything
you want to select on has to be part of the name (OID).

>   - We avoid definition and deployment of a NETCONF-only Xpath subset

At the price of creating another totally incompatible beast.

/js

---

From: Frank Strauß <[email-masked]>
Date: Wed, 9 Jun 2004 22:34:52 +0200

On 09. Jun 2004, at 15:42, Juergen Schoenwaelder wrote:

> [...]
> than something homegrown.

I agree.

> (The most vocal members of the WG do not seem
> to buy full XPATH as a requirement.)

Maybe the reason is, that there are still not many operators and 
developers
of manager software raising their voice. (I'm just guessing.)

>  Personally, I believe that the

However, a netconf standard should have the purpose to come to a common
sense on what agents support and what managers can rely on. If we have
reasons to assume that operators and others will ask for full XPath (and
I do think so, too) we should address this need *now* (and BTW not in a
subsequent revision as long as we did not even find rough consensus for
the initial revision).

> XPATH or just a reduced subset. I do not see that there is anything 

In general, I agree.

> Since (a) does not seem to be getting WG consensus, I believe (b) is 
> the
> best we can do.

First to say, my preference would be (a). If there are reasons against 
(a)
(I guess, I missed a discussion on this), I would also favor (b) over 
(c).

> really wondering on which basis you calculate how much you save on

I would also like to see a clear reasoning, not just on the pure code
size. These aspects come to my mind:

- We should take into account that many XPath implementations exist.
   My expectation would be that deriving a subset implementation would
   need some time (not much, but more that just picking an existing
   full implementation) just to shrink the code size for a small
   percentage.
- An XPath subset would most probably *not* reduce CPU load compared
   to full XPath.
- Building a new XPath (subset) implementation from scratch could be
   required by some vendors' policies. A full implementation would fit
   in the idea of reusability, while a netconf subset would be limited
   netconf only.
- Same is true for a home-grown filtering mechanism.
- A less powerful filtering mechanism would cause more network traffic
   and thus increase network and CPU load, since managers would have
   to retrieve more data than required if the required filtering cannot
   be done on the agent side.

Furthermore, nowadays even the cheapest APs, switches and other devices
come with an HTTP server, SSH, SNMP, syslog, and much more stuff. 
Firmware
resides in relatively large, however cheap enough flash memory. The ease
of usability and manageability through such features often makes it an
attractive and successful product. This generally weighs more than the
costs to develop and build such features. At least, this is my 
impression.

---

From: "David Harrington" <[email-masked]>
Date: Wed, 9 Jun 2004 23:25:21 -0400

Hi,

From a vendor-neutral application development standpoint (and presumably
from a vendor-neutral operations standpoint), a combination of
mandatory/optional filtering features will need to be supported to manage a
heterogeneuos network of devices, and it will be much simpler to support
full Xpath and a well-defined subset of Xpath than it would be to support
full Xpath plus another filtering mechanism. 

Would operators prefer to have two mechanisms to learn, or just one
mechanism with two well-defined variations?

Dave Harrington
[email-masked]

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Frank Strauß
> Sent: Wednesday, June 09, 2004 4:35 PM
> To: [email-masked]
> Cc: [email-masked]
> > First, I believe a subset of XPATH (not an adapted subset) is still 
> > better than something homegrown.
> 
> I agree.
> 
> > (The most vocal members of the WG do not seem to buy full 
> XPATH as a 
> > requirement.)
> 
> Maybe the reason is, that there are still not many operators 
> and developers of manager software raising their voice. (I'm 
> just guessing.)
> 
> However, a netconf standard should have the purpose to come 
> to a common sense on what agents support and what managers 
> can rely on. If we have reasons to assume that operators and 
> others will ask for full XPath (and I do think so, too) we 
> should address this need *now* (and BTW not in a subsequent 
> revision as long as we did not even find rough consensus for 
> the initial revision).
> 
> > I think the general idea is that boxes announce whether they do 
> > support full XPATH or just a reduced subset. I do not see 
> that there 
> > is anything wrong with this.
> 
> In general, I agree.
> 
> > (b) the WG makes a decision to support XPATH and require a 
> subset of 
> > it, or
> > the best we can do.
> (I guess, I missed a discussion on this), I would also favor 
> (b) over (c).
> 
> 
> I would also like to see a clear reasoning, not just on the 
> pure code size. These aspects come to my mind:
> 
> Furthermore, nowadays even the cheapest APs, switches and 
> other devices come with an HTTP server, SSH, SNMP, syslog, 
> and much more stuff. 
> Firmware
> resides in relatively large, however cheap enough flash 
> memory. The ease of usability and manageability through such 
> features often makes it an attractive and successful product. 
> This generally weighs more than the costs to develop and 
> build such features. At least, this is my impression.
> 
> 
> --