Subject: Re: sub-tree filtering proposals

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 3 Jun 2004 16:43:25 +0200

On Fri, May 28, 2004 at 02:56:05PM -0700, Andy Bierman wrote:
> 
> A) Issue 14.1: Subset specification
> 
> A.1) Extensibility
> 
> The <get> and <get-config> operations should be modified to 
> allow for the utilization of different types of standard, 
> proprietary, and experimental filter mechanisms in an extensible 
> way.  
>   - An additional containment layer (called <filter>) is needed 
>     above the user data model content in the get operation <rpc>
>     message.
>   - The <filter> element can appear zero or one times
>   - There is one attribute defined, called 'filter-type', which 
>     identifies the type of filter processing requested by the manager
>   - The content type is 'any' (any element outside the netconf
>     namespace)

Any specific reason why you want to put the <filter/> element into
the <data/> element? Why not just do the following:

   <rpc message-id="101" xmlns="netconf-uri">
     <get>
       <filter filter-type="subtree">
         <foo xmlns="http://example.com/schema-1"/>
       </filter>
     </get>
   </rpc>

   <rpc-reply message-id="101" xmlns="netconf-uri">
     <data>
       <foo xmlns="http://example.com/schema-1">7</foo>
     </data>
   </rpc-reply>

> B) Issue 14.1.2: Subtree Filtering
> ----------------------------------

[...]

Out of curiosity, I am wondering how this subtree matching relates to 
XPATH filters or a subset thereof. So I started a little exercise to
write XPATH expressions for your examples. In fact, it might be useful
if the subtree filtering actually can be translated into XPATH
expressions automatically on boxes that are capable to do XPATH
filtering (so you just have one piece of filtering logic).

Some open issues are listed at the end. Perhaps someone with more XPATH
experience can answer some of them.
 
> C) Examples
> -----------
>  
> C.1: No filter
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       ... entire set of data models returned ...
>     </data>
>   </rpc-reply>
> 
> Notes:
> 
>  - Since the sub-tree filter is inclusive, and the start state
>    is the empty set, the best way to get everything on the
>    agent is to leave out the filter.  Not sure the best way
>    to fit this into <get> and <get-config> operations though.

The equivalent XPATH expression might be "//*".
 
> C.2: Empty filter
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter type="subtree">
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>     </data>
>   </rpc-reply>
> 
> Notes:
> 
>  - Nothing is selected because no content match or selection
>    nodes are present.  This is not an error.

I think there are several XPATH expressions that produce an empty result 
- not sure yet which is the nicest (not that this is really important).
 
> C.3) Select the entire <users> sub-tree
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter filter-type="subtree">
>           <users xmlns="http://example.com/schema-1"/>
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       <users xmlns="http://example.com/schema-1">
>         <user>
>           <name>root</name>
>           <type>superuser</type>
>           <full-name>Charlie Root</full-name>
>           <company-info>
>             <dept>1</dept>
>             <id>1</id>
>           </company-info>
>         </user>
>         <user>
>           <name>fred</name>
>           <type>admin</type>
>           <full-name>Fred Flintstone</full-name>
>           <company-info>
>             <dept>2</dept>
>             <id>2</id>
>           </company-info>
>         </user>
>         <user>
>           <name>barney</name>
>           <type>admin</type>
>           <full-name>Barney Rubble</full-name>
>           <company-info>
>             <dept>2</dept>
>             <id>3</id>
>           </company-info>
>         </user>
>       </users>
>     </data>
>   </rpc-reply>
>       
> Notes:
> 
>  - The filter contains one selection node (<users>), so just
>    that sub-tree is selected by the filter
>  - This example represents the fully-populated <users> data model 
>    in most of the filter examples that follow.  In a real
>    data model, the 'company-info' would not likely be returned
>    with the list of users for a particular host or network.

The equivalent XPATH expression might be "//users" or "/users"
(not sure yet what precisely your semantics are here).

>  - The following filter request would have produced the same
>    result, but only because the container <users> defines one
>    child element (<user>)
> 
>     <rpc message-id="101" xmlns="netconf-uri">
>       <get>
>         <data>
>           <filter filter-type="subtree">
>             <users xmlns="http://example.com/schema-1">
>               <user/>
>             </users>
>           </filter>
>         </data>
>       </get>
>     </rpc>

The equivalent XPATH expression might be "//users/user" or "/users/user"
(not sure yet what precisely your semantics are here).
 
> C.4) Select all 'name' elements within the 'users' sub-tree
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter filter-type="subtree">
>           <users xmlns="http://example.com/schema-1">
>             <user>
>               <name/>
>             </user>
>           </users>
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       <users xmlns="http://example.com/schema-1">
>         <user>
>           <name>root</name>
>         </user>
>         <user>
>           <name>fred</name>
>         </user>
>         <user>
>           <name>barney</name>
>         </user>
>       </users>
>     </data>
>   </rpc-reply>
> 
> Notes:
> 
>  - This filter contains two containment nodes (users, user),
>    and one selector node (name).
>  - All instances of the 'name' element in the same sibling set 
>    are selected in the filter output.
>  - The manager may need to know that 'name' is used as an
>    instance identifier in this particular data structure,
>    but the agent does not need to know that meta-data in 
>    order to process the request.

The equivalent XPATH expression might be "//users/user/name".
 
> C.5) One specific <user> entry
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter filter-type="subtree">
>           <users xmlns="http://example.com/schema-1">
>             <user>
>               <name>fred</name>
>             </user>
>           </users>
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       <users xmlns="http://example.com/schema-1">
>         <user>
>           <name>fred</name>
>           <type>admin</type>
>           <full-name>Fred Flintstone</full-name>
>           <company-info>
>             <dept>2</dept>
>             <id>2</id>
>           </company-info>
>         </user>
>       </users>
>     </data>
>   </rpc-reply>
>       
> Notes:
> 
>  - This filter contains two containment nodes (users, user)
>    and one content match node (name).  
>  - All instances of the sibling set containing 'name',
>    for which the value of 'name' equals "fred", are selected
>    in the filter output.

The equivalent XPATH expression might be "//users/user[name="fred"]".
 
> C.6) Specific elements from a specific <user> entry
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter filter-type="subtree">
>           <users xmlns="http://example.com/schema-1">
>             <user>
>               <name>fred</name>
>               <type/>
>               <full-name/>
>             </user>
>           </users>
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       <users xmlns="http://example.com/schema-1">
>         <user>
>           <name>fred</name>
>           <type>admin</type>
>           <full-name>Fred Flintstone</full-name>
>         </user>
>       </users>
>     </data>
>   </rpc-reply>
>       
> Notes:
> 
>  - This filter contains two containment nodes (users, user),
>    one content match node (name), and two selector nodes
>    (type, full-name).
>  - All instances of the 'type' and 'full-name' elements in
>    the same siibling set containing 'name', for which the value 
>    of 'name' equals "fred", are selected in the filter output.
>    The 'company-info' element is not included because the
>    sibling set contains selection nodes.

The XPATH expression "//users/user[name="fred"]/name" will just return
the name element - not sure how one can select multiple elements with
XPATH in this case (but I am an XPATH beginner - perhaps someone out
there knows the answer).

> C.7) Multiple sub-trees
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter filter-type="subtree">
>           <users xmlns="http://example.com/schema-1"/>
>             <user>
>               <name>root</name>
>               <company-info/>
>             </user>
>             <user>
>               <name>fred</name>
>               <company-info>
>                 <id/>
>               </company-info>
>             </user>
>             <user>
>               <name>barney</name>
>               <type>superuser</type>
>               <company-info>
>                 <dept/>
>               </company-info>
>             </user>
>           </users>
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       <users xmlns="http://example.com/schema-1"/>
>         <user>
>           <name>root</name>
>           <company-info>
>             <dept>1</dept>
>             <id>1</id>
>           </company-info>
>         </user>
>         <user>
>           <name>fred</name>
>           <company-info>
>             <id>2</id>
>           </company-info>
>         </user>
>       </users>
>     </data>
>   </rpc-reply>
> 
> Notes:
>  
>  - This filter contains three sub-trees (name=root, fred, barney)
>  - The "root" sub-tree filter contains two containment nodes (users, 
>    user), one content match node (name), and one selector node 
>    (company-info).  The sub-tree selection criteria is met, and just 
>    the company-info sub-tree for "root" is selected in the filter 
>    output.

//users/user[name="root"]/company-info

>  - The "fred" sub-tree filter contains three containment nodes 
>    (users, user, company-info), one content match node (name), and 
>    one selector node (id). The sub-tree selection criteria is met, 
>    and just the 'id' element within the company-info sub-tree for 
>    "fred" is selected in the filter output.

//users/user[name="fred"]/company-info/id

>  - The "barney" sub-tree filter contains three containment nodes (users, 
>    user, company-info), two content match nodes (name, type), and one 
>    selector node (dept). The sub-tree selection criteria is not met
>    because user "barney" is not a "superuser", and the entire sub-tree 
>    for "barney" (including its parent 'user' entry) is excluded from
>    the filter output.

//users/user[name="barney" and type="superuser"]/company-info

You can combine these these expressions using a pipe character:

//users/user[name="root"]/company-info |
//users/user[name="fred"]/company-info/id |
//users/user[name="barney" and type="superuser"]/company-info

> C.8) Table with attribute naming
> 
> Request:
> 
>   <rpc message-id="101" xmlns="netconf-uri">
>     <get>
>       <data>
>         <filter filter-type="subtree">
>           <t:interfaces xmlns:t="http://example.com/schema-2"/>
>             <t:interface t:ifName="eth0"/>
>           </t:interfaces>
>         </filter>
>       </data>
>     </get>
>   </rpc>
> 
> Reply:
> 
>   <rpc-reply message-id="101" xmlns="netconf-uri">
>     <data>
>       <t:interfaces xmlns:t="http://example.com/schema-2">
>         <t:interface t:ifName="eth0">
>           <t:ifInOctets>45621</t:ifInOctets>
>           <t:ifOutOctets>774344</t:ifOutOctets>
>         </t:interface>
>       </t:interfaces>
>     </data>
>   </rpc-reply>
>       
> Notes:
> 
>  - This filter contains one containment node (interfaces), one 
>    attribute match expression (ifName), and one selector node
>    (interface).
>  - All instances of the 'interface' sub-tree which have an
>    ifName attribute equal to "eth0" are selected in the
>    filter output.
>  - The filter data elements and attributes must be qualified 
>    because the 'ifName' attribute will not be considered part 
>    of the 'schema-2' namespace if it is unqualified.
>  - If 'ifName' were a child node instead of an attribute, then
>    the following request would produce the same results:
> 
>     <rpc message-id="101" xmlns="netconf-uri">
>       <get>
>         <data>
>           <filter filter-type="subtree">
>             <interfaces xmlns="http://example.com/schema-2">
>               <interface>
>                 <ifName>eth0</ifName>
>               </interface>
>             </interfaces>
>           </filter>
>         </data>
>       </get>
>     </rpc>

interfaces/interface/[@ifName="eth0"]

OPEN ISSUES:

- I have ignored namespaces so far. I am sure XPATH will have a way 
  to deal with them. Not sure though how that looks like in concrete
  syntax (and how ugly it will be).
- I do not yet know how to select several elements (kind of doing a
  projection on elements) (see example C.6 above).

/js

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen, Germany

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/netconf/>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 03 Jun 2004 15:31:23 -0700

>       <foo xmlns="http://example.com/schema-1">7</foo>

I guess this would be fine.
You still need to "get everything", but that can be if 
filter is missing, then get everything.

>filtering (so you just have one piece of filtering logic).

The problem with XPath (as we went through before)
is that we are not going to specify a NETCONF-only subset
of XPath.  We are not going to require every box to
implement XPath.  This subtree filtering is intentionally
very limited so it's easy to implement.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 4 Jun 2004 00:45:15 +0200

> very limited so it's easy to implement.

You do not want to subset XPATH to reduce the implementation costs and
at the same time you propose a different filter mechanism to reduce the 
implementation cost (which at the end might map to a subset of XPATH).

OK, lets not restart this discussion without a beer. Lets see whether 
there are any other technical comments on your proposal. It seems this
mailing list is way too quiet.

/js

---

From: Andy Bierman <[email-masked]>
Date: Thu, 03 Jun 2004 17:18:45 -0700

>mailing list is way too quiet.

I agree the list is too quiet!

Note that sub-tree filtering has been in the spec all along, 
but it hasn't been documented yet (except in examples).  I tried
to document the behavior the document examples have shown
but never explained in detail.

I would be in favor of standardizing both subtree and Xpath
filtering at this time.  The only extra work:

  - #xpath capability: if set, the agent supports filter-type="xpath"
    for any operation that takes the <filter> parameter
  - <filter filter-type="xpath">
      ... xpath expression ...
    </filter>

I don't want to make Xpath mandatory, but I think we should still
use it for complex filtering.

>/js

Andy

---

From: Wes Hardaker <[email-masked]>
Date: Thu, 03 Jun 2004 22:32:36 -0700

>>>>> On Fri, 4 Jun 2004 00:45:15 +0200, Juergen Schoenwaelder <[email-masked]> said:

Juergen> You do not want to subset XPATH to reduce the implementation
Juergen> costs and at the same time you propose a different filter
Juergen> mechanism to reduce the implementation cost (which at the end
Juergen> might map to a subset of XPATH).

Juergen has a perfect point here.  You're creating something new,
which will take new code to implement something that doesn't yet exist
because you don't want to use something else which is potentially more
complex but yet code exists for it.  Don't you think it will take
significantly less work for implementations to implement partial XPath
for which there is a plethora of pre-existing code than it will be to
implement (and get right) something entirely new?  The whole point
behind using XML in the first place for this initiative was to make
use of existing tools, but yet for this one you're throwing out the
existing tools and libraries. Seems odd. 

-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 Jun 2004 07:04:21 -0700

>Juergen has a perfect point here.  You're creating something new,

I'm documenting something that has been in the spec since
it was called XMLCONF.  

>complex but yet code exists for it. 

I am extremely confident that implementation of this
subtree filtering will take less code space and dev-effort
than a full Xpath implementation.

You can't just point to the equivalent Xpath for subtree filters
and ignore the rest of the Xpath feature set.  

> Don't you think it will take

Repetition is the key to learning...
There will be no NETCONF standard subset of Xpath.
Compare subtree to full Xpath implementation only.
The WG already decided that implementation of full Xpath 
for NETCONF conformance is not appropriate for most devices,
and will not be required.

>existing tools and libraries. Seems odd. 

See the previous thread on Xpath, especially Phil's "closing
arguments".  IMO, we would keep NETCONF out of most network
devices for years because the agent implementation cost and
runtime resource requirements would be too high.

What is it about subtree filtering that is so hard to implement?
Clearly, all existing embedded implementations can handle
containment nodes and leaf nodes with simple content.  There
are implementations from multiple vendors of all of these
features because they are simple and intuitive:
   -- retrieve only the subtrees that exactly match the 
      criteria presented in the search
   -- return a subtree starting at an empty leaf node (e.g. <users/>)

These are the fundamental filter operations that we need
for NETCONF.  IMO, Xpath is overkill for these fundamental
needs.  

Andy

---

From: Wes Hardaker <[email-masked]>
Date: Fri, 04 Jun 2004 08:12:55 -0700

>>>>> On Fri, 04 Jun 2004 07:04:21 -0700, Andy Bierman <[email-masked]> said:

Andy> I'm documenting something that has been in the spec since
Andy> it was called XMLCONF.  

No one was debating what you were doing.

Andy> What is it about subtree filtering that is so hard to implement?

I never said it would be hard to implement, but you do have to
implement it.  Heck, XPath isn't hard to implement it just takes
time.  So would this.

Anyway, you don't need to respond to this as you've already "declared
it so", so continuing this would probably not be worth your time.

---

From: Andy Bierman <[email-masked]>
Date: Fri, 04 Jun 2004 08:31:23 -0700

>implement it.  Heck, XPath isn't hard to implement it just takes

It's not just the implementation time and runtime resources.
As Rob pointed out, the operators would need to learn how
to use Xpath.  It's debatable whether operators will find
Xpath or this subtree filtering more complicated.

>it so", so continuing this would probably not be worth your time.

Sorry for over-reacting.
I got upset at the prospect of rehashing the entire "Xpath subset"
thread over again.  

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 5 Jun 2004 09:21:27 +0200

> Xpath or this subtree filtering more complicated.

I bet that several operators know XPATH better than we do. I am talking
about the operators that actually automate their processes. The other
crowd will continue using the CLI anyway.

/js