Subject: Re: [netmod] XPath errors in ietf-snmp.yang

From: Jernej Tuljak <[email-masked]>
Date: Wed, 01 Aug 2012 09:22:02 +0200

Dne 31.7.2012 16:21, piše Martin Bjorklund:
> Hi,
>
> Jernej Tuljak<[email-masked]>  wrote:
>> Dne 29.7.2012 1:34, piše Andy Bierman:
>>> Hi,
>>>
>>> I am getting lots of XPath errors reported from yangdump:
>>>
>>>
>>>
>>> *** Generated by yangdump 5.0.da08cd7
>>> *** Copyright (c) 2008-2012, Andy Bierman, All Rights Reserved.
>>>
>>> Warning: no child nodes found in XPath expr '../type = read or ../type = write'
>>> ietf-snmp-proxy.yang:114.24: warning(432): no child node available
>>>
>>> Warning: no child nodes found in XPath expr '../type = read or ../type = write'
>>> ietf-snmp-proxy.yang:114.42: warning(432): no child node available
>>>
>>> Warning: no child nodes found in XPath expr '../type = trap or ../type = inform'
>>> ietf-snmp-proxy.yang:123.24: warning(432): no child node available
>>>
>>> Warning: no child nodes found in XPath expr '../type = trap or ../type = inform'
>>> ietf-snmp-proxy.yang:123.42: warning(432): no child node available
>>>
>>> These errors in ietf-snmp-proxy.yang are because the strings need to
>>> be quoted.  When unquoted, they are XPath node names, not strings.
>>>
>>> e.g.:
>>>
>>>     ../type = 'trap' or ../type = 'inform'
>> I agree. These should be quoted.
> Fixed.
>
>>> Warning: no child nodes found in XPath expr 'snmp:params/snmp:v1 or
>>> snmp:params/snmp:v2c'
>>> ietf-snmp-community.yang:196.12: warning(432): no child node available
>>>
>>> Warning: no child nodes found in XPath expr 'snmp:params/snmp:v1 or
>>> snmp:params/snmp:v2c'
>>> ietf-snmp-community.yang:196.35: warning(432): no child node available
>>>
>>>
>>> There is a missing '/' so XPath this is supposed to be a child node
>>>
>>> e.g.
>>>      /snmp:params/snmp:v1
>> No. There is no toplevel params node in this module. These should probably be
>> "../snmp:v1 or ../snmp:v2c". They appear to be refering to nodes of
>>> the params choice inside /snmp:snmp/snmp:target.
> Yes.  I fixed this by moving the when stmt up one level:
>
>    augment /snmp:snmp/snmp:target {
>      when "snmp:params/snmp:v1 or snmp:params/snmp:v2c";
>      leaf mms {
>      ...

Hmm.. I don't think you can refer to snmp:params in an XPath expression. 
It's a YANG choice statement and will never exist  in an instance 
document when the expression is evaluated. This will always result to 
"false". The context node of the above when statement is 
/snmp:snmp/snmp:target, which will either have snmp:v1 or snmp:v2c as a 
child element but no snmp:params. Unless I'm missing something.

This is my first time seeing a conditional augment in action. Does this 
mean that such augmentations should be done dynamically based on the 
current instance document (from a YANG implementation's perspective)? 
This is bad. I've been under the impression that a YANG data model is 
"static" where the entire schema is known before hand, even conditional 
leaves and branches. But this means that the model may not be known 
until and actual instance is present - the schema may change under 
certain conditions.

Or does a conditional augment mean that the augmentation will always be 
done but all of the augmented nodes "inherit" the augment's when 
conditions making all of them conditional? This would make much more 
sense to me. Would this be legal as a potential workaround to 
dynamically changing a module's schema? What was the group's intention 
regarding this matter?

Jernej

>>> Warning: no child nodes found in XPath expr
>>> '/snmp/usm/remote[engine-id=current()]/user[name=current()/../usm/user-name]'
>>> ietf-snmp-usm.yang:194.13: warning(432): no child node available
>>>
>>> Warning: no child nodes found in XPath expr
>>> '/snmp/usm/remote[engine-id=current()]/user[name=current()/../usm/user-name]'
>>> ietf-snmp-usm.yang:194.29: warning(432): no child node available
>>>
>>> Warning: no child nodes found in XPath expr
>>> '/snmp/usm/remote[engine-id=current()]/user[name=current()/../usm/user-name]'
>>> ietf-snmp-usm.yang:194.55: warning(432): no child node available
>>>
>>>
>>> Not sure if the 3 errors above are yangdump problems or due to
>>> restrictions on what can be
>>> referenced within an augment-stmt.  The XPath and combination of uses
>>> and augments
>>> is complicated to follow, so not sure if it is correct yet.
>> I don't think these are errors.
> Neither do I.
>
>>> Warning: no child nodes found in XPath expr '../map-type = snmp:specified'
>>> ietf-snmp-tls.yang:174.30: warning(432): no child node available
>>>
>>> Not sure what snmp:specified is -- XPath thinks it is a node name because this
>>> is an unquoted string.  There is no child 'specified' under this leaf.
>> snmp:specified appears to be an identity in ietf-snmp-tls (base
>>> cert-to-tm-security-name). This should also be quoted, it's a
>>> QName value.
> Fixed.
>
>
>
> /martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 1 Aug 2012 00:54:12 -0700

On Wed, Aug 1, 2012 at 12:22 AM, Jernej Tuljak <[email-masked]> wrote:
> It's a YANG choice statement and will never exist  in an instance document
> when the expression is evaluated. This will always result to "false". The
> context node of the above when statement is /snmp:snmp/snmp:target, which
> will either have snmp:v1 or snmp:v2c as a child element but no snmp:params.
> Unless I'm missing something.

You are right.
Good catch -- it isn't obvious but the syntax for the augment target path
is not the same as an XPath expression.  Choice and case nodes cannot
ever appear in an XPath path-expr.  These nodes do not exist in the
data tree.

The ABNF is clear:

   uses-augment-arg-str = < a string that matches the rule
                            uses-augment-arg >

   uses-augment-arg    = descendant-schema-nodeid

  augment-arg-str     = < a string that matches the rule
                           augment-arg >

   augment-arg         = absolute-schema-nodeid

  when-stmt           = when-keyword sep string optsep ...

>
> This is my first time seeing a conditional augment in action. Does this mean
> that such augmentations should be done dynamically based on the current
> instance document (from a YANG implementation's perspective)? This is bad.
> I've been under the impression that a YANG data model is "static" where the
> entire schema is known before hand, even conditional leaves and branches.
> But this means that the model may not be known until and actual instance is
> present - the schema may change under certain conditions.

The complexity involved in implementing when-stmt in an automated fashion
is probably an order of magnitude more complicated than any other YANG
statement.  There are many ways they can be inherited, and the context node
is not always straight-forward.  Sometimes the context node doesn't
even exist, so evaluating the when-stmt actually requires that the
data tree be modified in order to test it.  Since no value is provided,
the server needs to pick an arbitrary value, which is completely broken.

This often needs to be done to decide whether a
min-elements of mandatory-stmt is really true (because when=false
overrides mandatory=true).  Since when-stmts interact, they may need to
be reevaluated (A depends on B, B depends on C, and C depends on A).
Delete 1 and they maybe they all get deleted.  The order the when-stmts
are evaluated can change the result.

>
> Or does a conditional augment mean that the augmentation will always be done
> but all of the augmented nodes "inherit" the augment's when conditions
> making all of them conditional? This would make much more sense to me. Would
> this be legal as a potential workaround to dynamically changing a module's
> schema? What was the group's intention regarding this matter?
>

The schema is not static.  When-stmt causes nodes to magically appear
or disappear instantly --- it is applied right away, not at validation time.
The intent was to support conditional nodes.  The result is complicated.

Since features are not required to be static either, if-feature-stmt can
also cause nodes to appear or disappear.

> Jernej

Andy

---

From: Jernej Tuljak <[email-masked]>
Date: Wed, 01 Aug 2012 10:46:52 +0200

Dne 1.8.2012 9:54, piše Andy Bierman:
> On Wed, Aug 1, 2012 at 12:22 AM, Jernej Tuljak<[email-masked]>  wrote:
>     uses-augment-arg-str =<  a string that matches the rule
>                              uses-augment-arg>
>
>    augment-arg-str     =<  a string that matches the rule
>                             augment-arg>
>

I know about if-feature statements since I implemented them for our XML 
editor. Selecting a set of enabled features may truncate entire subtrees 
of the schema. But this is not the same as how I described conditional 
augmentation. For if-features the user decides before hand (before 
writing an instance document) what the schema should be. This is a one 
time only thing as features of a server do not change that often. When 
the user chooses another set of features, she is willingly changing the 
schema. But conditional augments may cue in at any time during the 
creation of an instance document, since it decides what the schema shall 
be. This is not what I expected. Am I the only one? I mean, this would 
probably lead to chicken or the egg dilemmas, would it not?

These are also the only cases where schema nodes may appear and 
disappear at runtime that I know of as of today. In all other cases the 
schema is static. All the schema nodes are there, but there may be 
constraints specified on them which later constrain nodes in an instance 
document such as a datastore. I think you are mixing constraints for the 
schema and constrains for instance nodes.

So I disagree that schema nodes may appear at disappear just like that. 
They should not. That's why when-stmt exists - so that data based on 
these schema nodes may be constrained without changing the schema itself 
(the structure of leafs, lists, etc.). What I'm trying to say is that 
when-stmts are part of a static schema.

May I have an example of how when-stmt may magically remove/add schema 
nodes (which is not a conditional augment)?

Jernej

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 01 Aug 2012 06:36:49 -0700 (PDT)

Jernej Tuljak <[email-masked]> wrote:
> Hmm.. I don't think you can refer to snmp:params in an XPath expression. It's
> a YANG choice statement and will never exist in an instance document when the
> expression is evaluated. This will always result to "false". The context node
> of the above when statement is /snmp:snmp/snmp:target, which will either have
> snmp:v1 or snmp:v2c as a child element but no snmp:params. Unless I'm missing
> something.

Right, this should be 

  augment /snmp:snmp/snmp:target {
    when "snmp:v1 or snmp:v2c";

Sorry about that.

> instance document (from a YANG implementation's perspective)? This is
> bad. I've been under the impression that a YANG data model is "static" where
> the entire schema is known before hand, even conditional leaves and
> branches. But this means that the model may not be known until and actual
> instance is present - the schema may change under certain conditions.
> 
> but all of the augmented nodes "inherit" the augment's when conditions making
> all of them conditional? This would make much more sense to me.

I am not sure I understand the distinction.  Let's try an example:

Given:

  container a {
    leaf b { ... }
  }

Then

  augment a {
    when "b = 1";
    leaf c {
      ...
    }
  }

and

  augment a {
    leaf c {
      when "../b = 1";
      ...
    }
  }

produces the same end result.  So I guess you can say that the when is
inherited in the augment.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 1 Aug 2012 06:48:01 -0700

On Wed, Aug 1, 2012 at 1:46 AM, Jernej Tuljak <[email-masked]> wrote:
>>> document
>>> mean
>>> bad.
>>> done
> editor. Selecting a set of enabled features may truncate entire subtrees of
> the schema. But this is not the same as how I described conditional
> augmentation. For if-features the user decides before hand (before writing
> an instance document) what the schema should be. This is a one time only
> thing as features of a server do not change that often. When the user
> chooses another set of features, she is willingly changing the schema. But
> conditional augments may cue in at any time during the creation of an
> instance document, since it decides what the schema shall be. This is not
> what I expected. Am I the only one? I mean, this would probably lead to
> chicken or the egg dilemmas, would it not?
>
> These are also the only cases where schema nodes may appear and disappear at
> runtime that I know of as of today. In all other cases the schema is static.
> All the schema nodes are there, but there may be constraints specified on
> them which later constrain nodes in an instance document such as a
> datastore. I think you are mixing constraints for the schema and constrains
> for instance nodes.
>
> So I disagree that schema nodes may appear at disappear just like that. They
> should not. That's why when-stmt exists - so that data based on these schema
> nodes may be constrained without changing the schema itself (the structure
> of leafs, lists, etc.). What I'm trying to say is that when-stmts are part
> of a static schema.

Features do not change that often but they can change, so if you write code
that assumes they do not change, it can easily beak.  When the schema node
disappears, so do all the instance nodes.
>
> May I have an example of how when-stmt may magically remove/add schema nodes
> (which is not a conditional augment)?

look at the ABNF.  When-stmt is allowed in uses-stmt, choice-stmt,
case-stmt, etc.

  choice foo {
     when "blah";
     case A {
        when "blah-blah";
         uses bar {
             when "blah-blah-blah";
             augment bar1/bar2 {
                when "etc.";
                 leaf X {
                     when "/on and /on";
                      type string;
                 }
              }
          }
       }
    }

Andy

---

From: Jernej Tuljak <[email-masked]>
Date: Wed, 01 Aug 2012 15:49:53 +0200

> inherited in the augment.

Thank you. That's exactly what I wanted to hear. This makes conditional 
augments non-dynamic and the entire schema known before hand. Simply 
put, the part of schema nodes under your augment will always be 
augmented to the target, but will only make sense if there's a b leaf 
with a value of 1. That doesn't mean that this part of the schema does 
not exist.

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 1 Aug 2012 18:31:32 -0700

On Aug 1, 2012, at 12:22 AM, Jernej Tuljak wrote:

> This is my first time seeing a conditional augment in action. Does this mean that such augmentations should be done dynamically based on the current instance document (from a YANG implementation's perspective)? This is bad. I've been under the impression that a YANG data model is "static" where the entire schema is known before hand, even conditional leaves and branches. But this means that the model may not be known until and actual instance is present - the schema may change under certain conditions.

I argued several times exactly as you do, see for example this thread:

http://www.ietf.org/mail-archive/web/netmod/current/msg04686.html

But I was a lonely voice in a desert. So now we are two, good.

Lada

> 
> Or does a conditional augment mean that the augmentation will always be done but all of the augmented nodes "inherit" the augment's when conditions making all of them conditional? This would make much more sense to me. Would this be legal as a potential workaround to dynamically changing a module's schema? What was the group's intention regarding this matter?

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 1 Aug 2012 18:47:18 -0700

> But I was a lonely voice in a desert. So now we are two, good.

I think Jernej was right when he wrote that the schema is always static,
but the instance document determines if values are allowed or not.
Both uses-stmt and augment-stmt behave this way (if when-stmt used in them).
It is not easy to implement.

But I can see how this is problematic.
Good thing we overloaded the unknown-element error-tag ;-)
It is used for unknown and unexpected elements.
If /foo/bar only exists if when=true, then for when=false we would have to
decide if that node is unknown by the server, vs. an instance
of that node is unexpected for the request.

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 1 Aug 2012 19:05:32 -0700

> It is not easy to implement.

If the schema were static, then

leaf foo {
   mandatory true;
   when X;
   …
}

would lead to an error if "foo" is not present (provided its parent is), no matter what the value of X.

Actually, DSDL validation would signal this error because it is done in two steps: schema validation (RELAX NG) and semantic validation (Schematron).

But I don't think there is agreement that this is the correct interpretation.

Lada

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 02 Aug 2012 08:53:02 +0200

Dne 2.8.2012 4:05, piše Ladislav Lhotka:
>> On Wed, Aug 1, 2012 at 6:31 PM, Ladislav Lhotka<[email-masked]>  wrote:

This is a bug actually. RFC6020 (8.2) is quite clear on this. 
Mandatory-stmt conditon will only be enforced if when-stmt condition 
evaluates to false.  Perhaps such problems should be delegated only to 
Schematron. That is by always making foo rng:optional, and transforming 
mandatory-stmt into an XPath expression and moving it together with when 
condition so they must be satisfied on foo's parent (parent becomes the 
context node):

when foo-parent is present in the instance document
-> when X condition for foo is false AND foo does not exist then error 
for missing mandatory foo
-> when X condition for foo is false AND foo exists then error for 
unsatisfied constraint on foo

It doesn't matter how the error is generated. But it should be properly 
generated.

Jernej

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 02 Aug 2012 09:00:04 +0200

Dne 2.8.2012 8:53, piše Jernej Tuljak:
>>>>> This is my first time seeing a conditional augment in action. Does 
>>>>> this mean that such augmentations should be done dynamically based 
>>>>> on the current instance document (from a YANG implementation's 
>>>>> perspective)? This is bad. I've been under the impression that a 
>>>>> YANG data model is "static" where the entire schema is known 
>>>>> before hand, even conditional leaves and branches. But this means 
>>>>> that the model may not be known until and actual instance is 
>> Actually, DSDL validation would signal this error because it is done 
>> in two steps: schema validation (RELAX NG) and semantic validation 
>> (Schematron).
>>
> Schematron. That is by always making foo rng:optional, and 
> transforming mandatory-stmt into an XPath expression and moving it 
> together with when condition so they must be satisfied on foo's parent 
> (parent becomes the context node):

No. I misread RFC6020. This would stand if the when was defined for 
parent-foo. Hierarchy of constraints for Ladislav's example appears to 
be undefined.

Jernej

> It doesn't matter how the error is generated. But it should be 
> properly generated.
>
> Jernej
>
>> Lada
>>
>>>>> Or does a conditional augment mean that the augmentation will 
>>>>> always be done but all of the augmented nodes "inherit" the 
>>>>> augment's when conditions making all of them conditional? This 
>>>>> would make much more sense to me. Would this be legal as a 
>>>>> potential workaround to dynamically changing a module's schema? 
>>>>> What was the group's intention regarding this matter?
>>>> --

---

From: Andy Bierman <[email-masked]>
Date: Thu, 2 Aug 2012 00:39:44 -0700

On Wed, Aug 1, 2012 at 11:53 PM, Jernej Tuljak <[email-masked]> wrote:
>>>>> mean that such augmentations should be done dynamically based on the current
>> Actually, DSDL validation would signal this error because it is done in
>> two steps: schema validation (RELAX NG) and semantic validation
>> (Schematron).
>>
> This is a bug actually. RFC6020 (8.2) is quite clear on this. Mandatory-stmt
> conditon will only be enforced if when-stmt condition evaluates to false.
> Perhaps such problems should be delegated only to Schematron. That is by
> always making foo rng:optional, and transforming mandatory-stmt into an
> XPath expression and moving it together with when condition so they must be
> satisfied on foo's parent (parent becomes the context node):
>
> -> when X condition for foo is false AND foo does not exist then error for
> missing mandatory foo

I think you are wrong.
when=false is not a constraint error.
It causes the node to be immediately deleted.
When-stmt = false means that node is not part of the
required child nodes if the parent is present.

augment /foo {
   when "/bar=5";
   leaf X { type string; mandatory true; }
}

vs:

container foo {
   leaf X {
      when "/bar = 5";
      type string;
      mandatory true;
   }
}

IMO, these should be conceptually equivalent.
I don't think the mandatory error is ever triggered
if the node is deleted due to when=false.

Andy

>>>>> done but all of the augmented nodes "inherit" the augment's when conditions

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 02 Aug 2012 11:38:20 +0200

Dne 2.8.2012 9:39, piše Andy Bierman:
> On Wed, Aug 1, 2012 at 11:53 PM, Jernej Tuljak<[email-masked]>  wrote:

I think this is an implementation detail. The correct errors would be 
reported either way, expressing the intention of the model. Only my way 
is surely easier to implement and would also put less strain on the 
device that has to implement it. On second thought, my error may provide 
more detail, explaining that an instance node cannot appear, because 
it's when condition is not satisfied, as opposed to saying "unexpected 
node" which is the only thing you can do.

You still seem to be under the impression that schema nodes MUST 
magically appear and disappear at XPath will (Ladislav referred to this 
as "run-time modifications to the schema"). That schema may have to be 
"re-generated" to add missing/remove unnecessary schema nodes according 
to what the content of an instance document is, which is way too 
complicated to be useful or to implement. Instead, the way I see it, 
such nodes are always present but are used only when certain conditions 
are met, conditions that are governed by instance documents, making YANG 
a static schema which may be pruned via if-feature statements in a 
controlled manner.

I agree that if-feature statements are different because they actually 
eliminate parts of a schema before hand the way you described, upon 
"hello message being received"/"a server being implemented". Those 
schema nodes temporarily cease to exist (or even permanently if those 
parts of a module are never implemented on the server). But these are 
not run-time modifications. At least not until "hot-plugging" of 
features is supported by NETCONF, but this would simply imply that 
pluggable parts of the schema should always be present on the server.

I think that the idea to completely decouple the schema from instances 
based on it is the way to go. Your schema is not such a schema, since it 
needs an instance to determine it's final form (the instance becomes a 
part of the schema). It's form must be determined after every 
modification to the instance. Constantly. Not entirely sure how you do it.

One other thing. Our generic client application displays the entire 
schema in a tree graphic component and even offers queries to be made 
from it. How could it possibly do that if the tree's final form cannot 
be determined?

Also this would make RFC6110 next to useless since the assumption that 
YANG schema is static is the foundation it builds upon. Generating DSDL 
schema is an expensive operation...and it would have to be done after 
every modification to an instance document.

The when-stmt only makes its parent data definition statement 
conditional. There's nothing written in RFC6020 that this would imply 
modifications to the schema (can't find it). Although multiple 
paragraphs can be found which serve as workarounds for non-static 
implementations of YANG schema. The point in 8.3.1 Payload Parsing which 
relates to when-stmt is an example.

Sorry, but I'm completely with Ladislav on this one. Non-static schema 
or run-time changes to a schema are bad and should never have been 
allowed to become a part of YANG. I cannot argue that they are not a 
part of it.

Jernej

---

From: Andy Bierman <[email-masked]>
Date: Thu, 2 Aug 2012 03:12:07 -0700

I agree that all of this is just implementation details.

This 2nd bullet in 8.3.2 says that false when-stmt nodes are deleted
immediately:

 During <edit-config> processing:

   o  If the NETCONF operation creates data nodes under a "choice", any
      existing nodes from other "case" branches are deleted by the
      server.

   o  If the NETCONF operation modifies a data node such that any node's
      "when" expression becomes false, then the node with the "when"
      expression is deleted by the server.

This text in 7.19.5 says that the statement containing the when-stmt
is conditional, i.e., does not apply if when=false:

   The "when" statement makes its parent data definition statement
   conditional.  The node defined by the parent data definition
   statement is only valid when the condition specified by the "when"
   statement is satisfied.

It doesn't matter how an implementation figures out what to do.

  leaf X {
     type string;
     mandatory true;
     when "blah";
  }

The text is clear that the mandatory-stmt is not enforced if when=false.

Andy

On Thu, Aug 2, 2012 at 2:38 AM, Jernej Tuljak <[email-masked]> wrote:
>>>>>>> this
>>>>>>> current
>>> Mandatory-stmt
>>> for
> reported either way, expressing the intention of the model. Only my way is
> surely easier to implement and would also put less strain on the device that
> has to implement it. On second thought, my error may provide more detail,
> explaining that an instance node cannot appear, because it's when condition
> is not satisfied, as opposed to saying "unexpected node" which is the only
> thing you can do.
>
> You still seem to be under the impression that schema nodes MUST magically
> appear and disappear at XPath will (Ladislav referred to this as "run-time
> modifications to the schema"). That schema may have to be "re-generated" to
> add missing/remove unnecessary schema nodes according to what the content of
> an instance document is, which is way too complicated to be useful or to
> implement. Instead, the way I see it, such nodes are always present but are
> used only when certain conditions are met, conditions that are governed by
> instance documents, making YANG a static schema which may be pruned via
> if-feature statements in a controlled manner.
>
> eliminate parts of a schema before hand the way you described, upon "hello
> message being received"/"a server being implemented". Those schema nodes
> temporarily cease to exist (or even permanently if those parts of a module
> are never implemented on the server). But these are not run-time
> modifications. At least not until "hot-plugging" of features is supported by
> NETCONF, but this would simply imply that pluggable parts of the schema
> should always be present on the server.
>
> I think that the idea to completely decouple the schema from instances based
> on it is the way to go. Your schema is not such a schema, since it needs an
> instance to determine it's final form (the instance becomes a part of the
> schema). It's form must be determined after every modification to the
> instance. Constantly. Not entirely sure how you do it.
>
> One other thing. Our generic client application displays the entire schema
> in a tree graphic component and even offers queries to be made from it. How
> could it possibly do that if the tree's final form cannot be determined?
>
> Also this would make RFC6110 next to useless since the assumption that YANG
> schema is static is the foundation it builds upon. Generating DSDL schema is
> an expensive operation...and it would have to be done after every
> modification to an instance document.
>
> The when-stmt only makes its parent data definition statement conditional.
> There's nothing written in RFC6020 that this would imply modifications to
> the schema (can't find it). Although multiple paragraphs can be found which
> serve as workarounds for non-static implementations of YANG schema. The
> point in 8.3.1 Payload Parsing which relates to when-stmt is an example.
>
> Sorry, but I'm completely with Ladislav on this one. Non-static schema or
> run-time changes to a schema are bad and should never have been allowed to
> become a part of YANG. I cannot argue that they are not a part of it.
>
> Jernej
>
>>
>>
>>>>>>> conditions

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 02 Aug 2012 17:02:01 -0700

> modification to the instance. Constantly. Not entirely sure how you do it.

In fact the solution can already be found in RFC 6110: make 'when' statement essentially equivalent to 'must', except for a different error message.

> every modification to an instance document.

Absolutely. It would be possible to forget about RELAX NG and implement all validation in Schematron. However, such an approach would ignore the lessons learnt by the XML community and embodied in DSDL: Schema-based validation is a very robust procedure that practically cannot be fooled and provides guarantees that the instance document is syntactically correct so that it can be safely parsed. Validation based on checking XPath expressions is much more fragile and is best used only for "business rules" (semantic constrainst) AFTER schema-based validation has been successfully finished.

> allowed to become a part of YANG. I cannot argue that they are not a 

Agreed.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 2 Aug 2012 17:14:03 -0700

Hi,

Sorry but YANG does not work the way you think it does for the sake of
your static test tool.  BTW, pyang does not validate the XPath statement.
It validates instance documents against the XPath that it finds.
This is hardly the same as verifying the correctness of the YANG statement.

A false when-stmt is not an error unless it occurs in the RPC input
parameters.  (We are insulated from <config> content because it
is declared as an 'anyxml' so from a RPC POV, the when-stmts
within the data are not relevant to RPC input validation.)

The XPath statements are conceptual.  There is no requirement
by the server to use an XPath validator to implement them.
When-stmt is applied right away, not applied at validation
time like must-stmt.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 2 Aug 2012 20:43:16 -0700

Hi -

> From: "Jernej Tuljak" <[email-masked]>
> To: "Andy Bierman" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Thursday, August 02, 2012 2:38 AM
> Subject: Re: [netmod] XPath errors in ietf-snmp.yang
...
> allowed to become a part of YANG. I cannot argue that they are not a 

Perhaps, but they satisfy real needs in open systems, systems
with hot-swap and hot-insertion components or subsystems, and
application management, to name a few.  They are straightforward to
support in legacy protocols like CMIP and SNMP and their corresponding
data model languages, and whether one likes it or not, coping with
schema changes is an important part of configuration management.
Perhaps they're not needed for the Netconf space, but recognize that
excluding them as possibilities would make Netconf even more limited
than what it was supposed to supplant.

Randy

---

From: Jernej Tuljak <[email-masked]>
Date: Fri, 03 Aug 2012 09:21:32 +0200

Hi,

that's just the beauty of it. It is not a static test tool. It adapts to 
whatever changes you make to the modules. It adapts when you choose 
enabled features and it adapts when you change them. It tells you what 
you did wrong while you write entire datastores, rpc operations, even 
when dealing with incomplete XML documents. And it does so in quite a 
simple way. The way of RFC6110, which extracts the entire schema from 
available modules and with just a few XSLT transformations you are set 
to go. Forever. It's quite genius really (credit to Ladislav and others 
involved with RFC6110 here). I don't understand why anyone would bother 
reinventing the wheel trying to implement his own way of validating 
NETCONF content when it's all already out there.

I never claimed that pyang checks the validity of XPath expressions. It 
cannot do so, because XPath expressions cannot be evaluated without 
context and context can only be found within instance documents, a thing 
that _never_ exists during YANG module validation. It only exists during 
NETCONF content validation, when you have an actual instance document to 
work with. Therefore when-stmt cannot be applied at any other time than 
during content validation. It represents a semantic constraint on data 
nodes, not a constraint on the schema nodes. And pyang applies these 
constraints within pyang.dsdl plugin only when it is validating an XML 
document against a YANG schema. There is no other way.

I don't understand the need to merge YANG validation with NETCONF 
content validation into a single validation process. It does not make 
sense. As I understand it, this is exactly what you are doing. Sometimes 
it even feels like you are saying that a schema node and an instance 
node are the same thing. It should be clear that a schema node is not an 
instance node, the former only represents the structure and conditions 
of appearance of the latter. Schema nodes never get deleted, instance 
nodes do.

Let's return to Ladislav's example:

   leaf X {
      type string;
      mandatory true;
      when ". = ../foo";
   }

For when-stmt condition to be evaluated _at all_ an instance of X must 
exist first, meaning that the mandatory constraint is always satisfied 
if an evaluation occurred. After all X is the condition expression's 
context node. You cannot evaluate without context. You can however 
validate it's syntax at any time - provided that you are using a static 
schema, which contains all the schema nodes for you to search.

If when is ever true or false, one can be sure that the mandatory 
constraint has been satisfied, eliminating the need to enforce it.

The above example represents an instance node, which has to exist and 
fulfill the condition in the when-stmt at the same time. Anything else 
is an error. It's actually mandatory which takes precedence here, so 
what Ladislav currently does (structure first, then semantics) is the 
correct interpretation, IMO.

There's a reason why when-stmt's argument is an XPath expression and not 
a YANG construct, like a schema-nodeid.

Jernej

Dne 3.8.2012 2:14, piše Andy Bierman:
> parameters.  (We are insulated from<config>  content because it
> On Thu, Aug 2, 2012 at 5:02 PM, Ladislav Lhotka<[email-masked]>  wrote:
>> Jernej Tuljak<[email-masked]>  writes:
>>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 3 Aug 2012 01:02:30 -0700

On Fri, Aug 3, 2012 at 12:21 AM, Jernej Tuljak <[email-masked]> wrote:
> whatever changes you make to the modules. It adapts when you choose enabled
> features and it adapts when you change them. It tells you what you did wrong
> while you write entire datastores, rpc operations, even when dealing with
> incomplete XML documents. And it does so in quite a simple way. The way of
> RFC6110, which extracts the entire schema from available modules and with
> just a few XSLT transformations you are set to go. Forever. It's quite
> genius really (credit to Ladislav and others involved with RFC6110 here). I
> don't understand why anyone would bother reinventing the wheel trying to
> implement his own way of validating NETCONF content when it's all already
> out there.
>
> cannot do so, because XPath expressions cannot be evaluated without context
> and context can only be found within instance documents, a thing that
> _never_ exists during YANG module validation. It only exists during NETCONF
> content validation, when you have an actual instance document to work with.
> Therefore when-stmt cannot be applied at any other time than during content
> validation. It represents a semantic constraint on data nodes, not a
> constraint on the schema nodes. And pyang applies these constraints within
> pyang.dsdl plugin only when it is validating an XML document against a YANG
> schema. There is no other way.

The YANG must and when statements can be checked to make sure
the XPath is well-formed and all the schema nodes are actually defined.
There is really no reason for an YANG must or when stmt to reference
garbage nodes that do not exist in the data model.

>
> I don't understand the need to merge YANG validation with NETCONF content
> validation into a single validation process. It does not make sense. As I
> understand it, this is exactly what you are doing. Sometimes it even feels
> like you are saying that a schema node and an instance node are the same
> thing. It should be clear that a schema node is not an instance node, the
> former only represents the structure and conditions of appearance of the
> latter. Schema nodes never get deleted, instance nodes do.
>

Implementation details do not matter.
The server must send the correct responses and alter its
internal state correctly.  There are many ways this could
be done.

Some servers allow YANG modules to be loaded and unloaded
at run-time.  The schema tree is static, meaning once a node
is placed in the tree it doesn't move, but the tree can change at run-time.
YANG deviations also affect the schema tree, which are allowed to
be loaded at any time.

> Let's return to Ladislav's example:

Yes, there are some pathological cases that make no
semantic sense that cannot be implemented correctly.

  leaf Y {
     type int32;
     mandatory true;
     when "not(.)";
  }

>
> For when-stmt condition to be evaluated _at all_ an instance of X must exist
> first, meaning that the mandatory constraint is always satisfied if an
> evaluation occurred. After all X is the condition expression's context node.
> You cannot evaluate without context. You can however validate it's syntax at
> any time - provided that you are using a static schema, which contains all
> the schema nodes for you to search.
>
> If when is ever true or false, one can be sure that the mandatory constraint
> has been satisfied, eliminating the need to enforce it.
>
> fulfill the condition in the when-stmt at the same time. Anything else is an
> error. It's actually mandatory which takes precedence here, so what Ladislav
> currently does (structure first, then semantics) is the correct
> interpretation, IMO.
>
> There's a reason why when-stmt's argument is an XPath expression and not a
> YANG construct, like a schema-nodeid.

Yes, so they generate a boolean result.
A schema-nodeid is not relevant here.
That is used for augment and path targets.

Andy

>>> In fact the solution can already be found in RFC 6110: make 'when'
>>> statement essentially equivalent to 'must', except for a different error
>>> message.
>>>
>>> Absolutely. It would be possible to forget about RELAX NG and implement
>>> all validation in Schematron. However, such an approach would ignore the
>>> lessons learnt by the XML community and embodied in DSDL: Schema-based
>>> validation is a very robust procedure that practically cannot be fooled and
>>> provides guarantees that the instance document is syntactically correct so
>>> that it can be safely parsed. Validation based on checking XPath expressions
>>> is much more fragile and is best used only for "business rules" (semantic
>>> constrainst) AFTER schema-based validation has been successfully finished.
>>>
>>>>>> properly
>>>>>>>> If /foo/bar only exists if when=true, then for when=false we would
>>>>>>>> have
>>>>>>>> to
>>>>>>>>>> always be

---

From: Jernej Tuljak <[email-masked]>
Date: Fri, 03 Aug 2012 12:17:01 +0200

Dne 3.8.2012 10:02, piše Andy Bierman:
> On Fri, Aug 3, 2012 at 12:21 AM, Jernej Tuljak<[email-masked]>  wrote:
>> Hi,
>>

Of course. That's what makes YANG great. That it's extensible the way 
any other schema is. A each set of modules may provide a different 
schema. A single additional module may change the existing schema tree 
significantly making it another schema. If-features are also great. But 
do note that these are not the "runtime modifications to the schema 
tree" me and Ladislav were talking about. What I've been saying all 
along is that if we have a module set of size one (a single module) it's 
schema is always the same. A schema node count will always be the same 
for it. The only way to modify the count is via features or by adding 
new modules to the set, which does happen, but not often. That's what I 
mean when I say that YANG is a static schema. An extensible but static 
schema. Adding a module with deviations, augments, etc. to the set 
creates a new but also static schema.

But none of this should have anything to do with NETCONF content 
validation. Allowing the schema count to be modified during validation 
with when-stmt's is just...well, bad. RFC6020 should have never allowed 
it. Separation of YANG validation/compiling which results in a read-only 
schema tree and NETCONF content validation, which only uses this schema 
tree to validate an XML document, should have been enforced. I don't see 
any merit in this being a single process.

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 3 Aug 2012 07:25:45 -0700

> than what it was supposed to supplant.

This is of course still possible - 'when' conditions are checked as semantic constraints at the same time as 'must'. In order to have a static schema (i.e. independent of the instance document it is supposed to check), one only needs to acknowledge that validation is split into two phases - syntactic (grammar + datatypes) and semantic (rules involving instance values).

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 3 Aug 2012 07:31:26 -0700

I don't think there is much point to discussing implementation details
on the WG mailing list.

Andy

On Fri, Aug 3, 2012 at 3:17 AM, Jernej Tuljak <[email-masked]> wrote:
>>> enabled
>>> context
> Of course. That's what makes YANG great. That it's extensible the way any
> other schema is. A each set of modules may provide a different schema. A
> single additional module may change the existing schema tree significantly
> making it another schema. If-features are also great. But do note that these
> are not the "runtime modifications to the schema tree" me and Ladislav were
> talking about. What I've been saying all along is that if we have a module
> set of size one (a single module) it's schema is always the same. A schema
> node count will always be the same for it. The only way to modify the count
> is via features or by adding new modules to the set, which does happen, but
> not often. That's what I mean when I say that YANG is a static schema. An
> extensible but static schema. Adding a module with deviations, augments,
> etc. to the set creates a new but also static schema.
>
> But none of this should have anything to do with NETCONF content validation.
> Allowing the schema count to be modified during validation with when-stmt's
> is just...well, bad. RFC6020 should have never allowed it. Separation of
> YANG validation/compiling which results in a read-only schema tree and
> NETCONF content validation, which only uses this schema tree to validate an
> XML document, should have been enforced. I don't see any merit in this being
> a single process.
>
> Jernej
>
>>
>>> exist
>>> constraint
>>>>>>>>>>>> making all of them conditional? This would make much more sense
>>>>>>>>>>>> to
>>>>>>>>>>>> me. Would
>>>>>>>>>>>> this be legal as a potential workaround to dynamically changing
>>>>>>>>>>>> a
>>>>>>>>>>>> module's

---

From: Jonathan Hansford <[email-masked]>
Date: Fri, 03 Aug 2012 16:59:37 +0100

Protocol Independent Network Management Data Modeling Languages -
Lessons Learned from the SMIng Project (draft-schoenw-sming-lessons-01),
a document I believe helped inform the development of YANG, states
"preference should be give to solutions that simplify the task of
readers. Reduction of the efforts required by writers is of secondary
importance and the reduction of the efforts required by tool developers
is of least important preference." 

As someone who is still getting
up-to-speed on YANG and therefore considers himself primarily a reader
for now, I hope the issues that have been discussed in this thread only
generate this level of uncertainty for tool developers and the task of
readers remains simplified. How esoteric are these discussions? Are we
looking at corner cases, or are these issues at the core of YANG module
development?

Jonathan

On 03.08.2012 15:31, Andy Bierman wrote: 

> I
don't think there is much point to discussing implementation details
on the WG mailing list.
> 
> Andy
> 
> On Fri, Aug 3, 2012 at 3:17 AM,
Jernej Tuljak <[email-masked]> wrote:
> 
>> Dne 3.8.2012
10:02, piše Andy Bierman: 
>> 
>>> On Fri, Aug 3, 2012 at 12:21 AM,
Jernej Tuljak<[email-masked] [1]> wrote: 
>>>> Hi, that's
just the beauty of it. It is not a static test tool. It adapts to
whatever changes you make to the modules. It adapts when you choose
enabled features and it adapts when you change them. It tells you what
you did wrong while you write entire datastores, rpc operations, even
when dealing with incomplete XML documents. And it does so in quite a
simple way. The way of RFC6110, which extracts the entire schema from
available modules and with just a few XSLT transformations you are set
to go. Forever. It's quite genius really (credit to Ladislav and others
involved with RFC6110 here). I don't understand why anyone would bother
reinventing the wheel trying to implement his own way of validating
NETCONF content when it's all already out there. I never claimed that
pyang checks the validity of XPath expressions. It cannot do so, because
XPath expressions cannot be evaluated without context and context can
only be found within instance documents, a thing that _never_ exists
during YANG module validation. It only exists during NETCONF content
validation, when you have an actual instance document to work with.
Therefore when-stmt cannot be applied at any other time than during
content validation. It represents a semantic constraint on data nodes,
not a constraint on the schema nodes. And pyang applies these
constraints within pyang.dsdl plugin only when it is validating an XML
document against a YANG schema. There is no other way.
>>> The YANG must
and when statements can be checked to make sure the XPath is well-formed
and all the schema nodes are actually defined. There is really no reason
for an YANG must or when stmt to reference garbage nodes that do not
exist in the data model. 
>>> 
>>>> I don't understand the need to merge
YANG validation with NETCONF content validation into a single validation
process. It does not make sense. As I understand it, this is exactly
what you are doing. Sometimes it even feels like you are saying that a
schema node and an instance node are the same thing. It should be clear
that a schema node is not an instance node, the former only represents
the structure and conditions of appearance of the latter. Schema nodes
never get deleted, instance nodes do.
>>> Implementation details do not
matter. The server must send the correct responses and alter its
internal state correctly. There are many ways this could be done. Some
servers allow YANG modules to be loaded and unloaded at run-time. The
schema tree is static, meaning once a node is placed in the tree it
doesn't move, but the tree can change at run-time. YANG deviations also
affect the schema tree, which are allowed to be loaded at any time.
Of course. That's what makes YANG great. That it's extensible the way
any other schema is. A each set of modules may provide a different
schema. A single additional module may change the existing schema tree
significantly making it another schema. If-features are also great. But
do note that these are not the "runtime modifications to the schema
tree" me and Ladislav were talking about. What I've been saying all
along is that if we have a module set of size one (a single module) it's
schema is always the same. A schema node count will always be the same
for it. The only way to modify the count is via features or by adding
new modules to the set, which does happen, but not often. That's what I
mean when I say that YANG is a static schema. An extensible but static
schema. Adding a module with deviations, augments, etc. to the set
creates a new but also static schema. But none of this should have
anything to do with NETCONF content validation. Allowing the schema
count to be modified during validation with when-stmt's is just...well,
bad. RFC6020 should have never allowed it. Separation of YANG
validation/compiling which results in a read-only schema tree and
NETCONF content validation, which only uses this schema tree to validate
an XML document, should have been enforced. I don't see any merit in
this being a single process. Jernej 
>> 
>>>> Let's return to Ladislav's
example: leaf X { type string; mandatory true; when ". = ../foo"; }
Yes, there are some pathological cases that make no semantic sense that
cannot be implemented correctly. leaf Y { type int32; mandatory true;
when "not(.)"; } 
>>> 
>>>> For when-stmt condition to be evaluated _at
all_ an instance of X must exist first, meaning that the mandatory
constraint is always satisfied if an evaluation occurred. After all X is
the condition expression's context node. You cannot evaluate without
context. You can however validate it's syntax at any time - provided
that you are using a static schema, which contains all the schema nodes
for you to search. If when is ever true or false, one can be sure that
the mandatory constraint has been satisfied, eliminating the need to
enforce it. The above example represents an instance node, which has to
exist and fulfill the condition in the when-stmt at the same time.
Anything else is an error. It's actually mandatory which takes
precedence here, so what Ladislav currently does (structure first, then
semantics) is the correct interpretation, IMO. There's a reason why
when-stmt's argument is an XPath expression and not a YANG construct,
like a schema-nodeid.
>>> Yes, so they generate a boolean result. A
schema-nodeid is not relevant here. That is used for augment and path
targets. Dne 3.8.2012 2:14, piše Andy Bierman:
[email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

Links:
------
[1] mailto:[email-masked]
[2]
mailto:[email-masked]
[3]
https://www.ietf.org/mailman/listinfo/netmod
[4]
mailto:[email-masked]
[5] mailto:[email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 3 Aug 2012 18:09:12 +0200

> looking at corner cases, or are these issues at the core of YANG module

My take is that these discussions, as important they might be for
implementors, are for most users a non issue, especially readers.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 3 Aug 2012 10:01:14 -0700

> I don't think there is much point to discussing implementation details

It is not an implementation detail! What Jernej and I are trying to communicate is that the current handling of 'when' constraints and the related "auto-deletion" of nodes is not only hard-to-implement but also ill-defined and broken. But as long as you dismiss any evidence of this plain fact as "corner cases" we can hardly make our point.

The thing is: With increasing complexity of our data models, we must be sure that our foundation is rock solid. With a complex model, such a "corner case" can lead to mysterious behaviour or outright failures. Let me paraphrase a Czech saying: A hundred corner cases are able to plague a donkey.

Anyway, here is another corner case that we already had in our private discussion:

container foo {
    when "bar != 42";
    leaf bar {
        type uint8;
        mandatory "true";
    }
}

1. The "bar" leaf is set to 42 by the client.
2. The "foo" container is auto-deleted.
3. The XPath expression in 'when' loses its context node, so the expression is not properly defined and cannot be used 
   for any argumentation about validity.
4. But then there is no reason why the mandatory leaf and container are missing.
5. So what?

Lada

>>>> content

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 3 Aug 2012 10:08:24 -0700

> implementors, are for most users a non issue, especially readers.

This is an IETF WG mailing list, not "yang-readers".

Lada

---

From: YumaWorks <[email-masked]>
Date: Fri, 3 Aug 2012 11:57:27 -0700

Sent from my iPhone

> Anyway, here is another corner case that we already had in our private discussion:

But this is a nonsense example
A real server would not ever do this

These are just implementation details for your tool.  YANG does not need to change. 

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 3 Aug 2012 13:07:50 -0700

Sent from my iPhone

> It is not an implementation detail! What Jernej and I are trying to communicate is that the current handling of 'when' constraints and the related "auto-deletion" of nodes is not only hard-to-imple…

It is not ill-defined and broken. 
I understand you would like to validate requests with an offline tool but that is not the purpose of the protocol. 

Even for a simple 'create' operation the request cannot be validated without knowing if the data already exists or not.  The current state of the system cannot be ignored. 

If the real when or must statements in real modules were not implementable  then I would be concerned but that is not the case.  

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sat, 4 Aug 2012 10:34:41 -0700

> implementors, are for most users a non issue, especially readers.

There are advanced YANG usage issues for YANG module writers.
Here are a few:

  * Never reference descendant-or-self nodes in a when-stmt
  * Should refer to descendant-or-self nodes in a must-stmt
    (Choose easy to understand stmt placement)
  * Use when-stmt in choice, case, uses with extreme care.
    Be careful to identify the correct context node
  * Make sure there are never any circular dependencies across
    multiple when-stmts; best to refer only to static objects
    in when-stmt (like the keys and mandatory nodes)
  * For must/when, be careful of referencing nodes in
    different YANG features than the context node

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 5 Aug 2012 13:14:18 +0200

> A real server would not ever do this

It is valid YANG, so at least it is a flaw in language definition.

Moreover, it doesn't really matter what the XPath expression is in my example above, the problem is still the same: if "foo" is not present, then the context node for the XPath expression doesn't exist end the expression is therefore undefined. The 'when' statements thus cannot be applied to make the mandatory property of "foo" conditional.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 5 Aug 2012 13:54:50 +0200

>    Be careful to identify the correct context node

The only case when 'when' statement doesn't have the problem with missing context node is for 'augment'. It is also the only case where 'when' cannot be replaced with 'must'.

IMO, an elegant solution to all these issues therefore would be to allow 'when' only for making conditional augments, 'must' should suffice everywhere else. All the caveats that you listed then wouldn't be necessary.

>    in when-stmt (like the keys and mandatory nodes)

Right, the "corner case" here is, e.g.

container a {
    when "../b/bar = 1";
    leaf foo {
        type uint8;
        mandatory true;
    }
}
container b {
    when "../a/foo = 1";
    leaf bar {
        type uint8;
        mandatory true;
    }
}

Both 'when' statements refer to a mandatory node, so your advice doesn't really help.

Such circular dependencies may actually happen and would be hard to detect, especially in deeper hierarchies.

>    different YANG features than the context node

I don't think this can ever be a problem for 'must'. Do you have an example?

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sun, 5 Aug 2012 05:01:49 -0700

...
> Moreover, it doesn't really matter what the XPath expression is in my example above, the problem is still the same: if "foo" is not present, then the context node for the XPath expression doesn't e…

This is a well-known issue that only matters if the when-expr
contains any descendant-or-self node references.  This makes
no semantic sense and probably should be made illegal in YANG.
As long as the path-expr nodes point outside the subtree, it is OK
to make a dummy (empty) context node to test for when-stmt.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sun, 5 Aug 2012 05:06:16 -0700

> The only case when 'when' statement doesn't have the problem with missing context node is for 'augment'. It is also the only case where 'when' cannot be replaced with 'must'.

This is wrong.

It occurs in all cases where min-elements or mandatory=tree may
be applied to missing instances, and the schema node has when-stmts
tied to it.  Inherited when-stmts from uses/choice/case are real fun
to implement.

> IMO, an elegant solution to all these issues therefore would be to allow 'when' only for making conditional augments, 'must' should suffice everywhere else. All the caveats that you listed then wou…

No -- when-stmt is too useful.
Some lines of code are harder than others to write.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 5 Aug 2012 15:06:51 +0200

> to make a dummy (empty) context node to test for when-stmt.

This is of course non-standard and other implementations may decide to do something else, potentially with different results. Standard XPath implementations are not able to do this, and that's why RFC 6110 is not compatible with this kludge and indeed its validation results are different for this special case.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 5 Aug 2012 15:25:31 +0200

> tied to it.  Inherited when-stmts from uses/choice/case are real fun

No, 'when' under 'augment' doesn't suffer from these problems because

1. 'augment' cannot add mandatory contents,

2. the context node (augment target) is not removed if the XPath expression in 'when' evaluates to false.

> No -- when-stmt is too useful.

Can you be more specific? What are the cases where 'when' gives you more than 'must'? FWIW, in the routing model I systematically replaced all non-augment whens with musts and it doesn't seem to matter (except that Martin complained about it in his review).

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sun, 5 Aug 2012 06:53:32 -0700

> This is of course non-standard and other implementations may decide to do something else, potentially with different results. Standard XPath implementations are not able to do this, and that's why…

What do you mean it is non-standard?
However the server produces the correct answer is fine.
These are conceptual statements anyway -- it says so right
in the RFC.  The server does not have to use XPath to
implement the constraint.

I do not agree that we need to change the ABNF for YANG to
make when-stmt simpler to implement.  It has been pointed out
that there are many ways to achieve the same resulting nodes
in YANG (i.e., inline vs. augment vs. grouping/uses).  There is
nothing special about augment wrt/ when-stmt processing.
(It's just as hard as the other corner-cases ;-)

I objected to the DSDL draft several times because the normative
behavior for a NETCONF server can only be determined by the
NETCONF server, and not some static message on the wire approach.

There are limitations to offline validation and the correct solution
is to document those limitations, not change the server to match them.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 5 Aug 2012 20:09:18 +0200

> implement the constraint.

I am not aware of any text about adding dummy nodes in 6020. The adjective "conceptually" is clearly explained in sec. 7.19.5: "This means that an implementation does not have to use an XPath evaluator on the device." Nothing about tinkering with the instance document.

> (It's just as hard as the other corner-cases ;-)

Maybe it's not necessary to change the ABNF. I actually propose the following compromise:

1. For validation purposes, 'when' is essentially equivalent to 'must'. In terms of RFC 6020 it only means to remove two words '"when" or' in the first paragraph of sec. 8.2, and perhaps explicitly say that "when" conditions do not override mandatory properties of nodes as defined in sec. 3.1.

BTW, features are really different because their presence or absence is known beforehand and they don't depend on instance documents.

2. Operationally, if the server finds out that the value of a "when" condition on an existing data node changes from false to true and the node is not mandatory (as per sec. 3.1), it will remove that node and its descendants.

I believe that your caveats for YANG writers would then be no more necessary and no changes to existing modules would be needed. 

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 05 Aug 2012 21:38:25 +0200 (CEST)

Hi,

I agree fully with Andy on this issue.  The main problem seems to be
that the DSDL mapping cannot handle "when" properly when it is
combined with mandatory true.  But this is a not a problem with 6020.
 
The evaluation mechanism is (conceptually) as follows.  If a node with
a when stmt exists, the when expression is evaluated with the node as
context node.  If the expression evaluates to false, the node is
deleted.  If a node with a when expression is about to be created, it
is added to the tree (I think this is what Andy refers to as the dummy
node), and the when expression is evaluated with the node as context
node.  If the expression is evaluated to false, the error specified in
8.3.1 is returned.

With this model, it is even clear what happens if a when expression
refers to descendant-or-self (Andy's recommendation still applies -
don't do this b/c it is probably not what you want).  The only
problematic case is if there is a loop (a refers to b and b refers to
a).  This could have been clarified in the text (e.g., by stating that
the behavior in such a pathological case is implementation dependent).

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 05 Aug 2012 21:47:16 +0200 (CEST)

> node.  If the expression is evaluated to false, the error specified in

I forgot to add that if the when expression is false, the mandatory  /
min-elements stmts are not checked.

Also, the when expression does *not* dynamically change the schema
tree.  It only has implications on the data tree.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Sun, 5 Aug 2012 22:45:03 +0200

> combined with mandatory true.  But this is a not a problem with 6020.

Sorry, I cannot agree. Sec. 7.19.5 defines the context node for 'when' XPath expression as follows:

   o  If the "when" statement is a child of any other data definition
      statement, the context node is the data definition's node in the
      data tree.

It doesn't say what happens if "data definition's node" is absent in the data tree. And without a context node, the XPath expression is not well-defined.   

> node.  If the expression is evaluated to false, the error specified in

OK, but what about this:

leaf enable-foo {
    type boolean;
}
container foo {
    when "../enable-foo = 'true'";
    leaf bar {
        type uint8;
        mandatory true;
    }
}

Now, consider an instance document containing <enable-foo>false</enable-foo> but not <foo>, i.e. the XPath context node is missing. Simple question: is this a valid document? If your answer is yes, could you explain how the validity follows from 6020?

I think this is where Andy needs to temporarily insert <foo> as his dummy element, which is however not "about to be created" as you write, at least not as a result of client's action. This has no support in 6020, the alone XPath.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sun, 5 Aug 2012 14:58:25 -0700

> I think this is where Andy needs to temporarily insert <foo> as his dummy element, which is however not "about to be created" as you write, at least not as a result of client's action. This has no…

It is just an implementation choice to use a data tree at all.
A temporary node that only exists during some internal
server state and is never visible to the client is not a standards issue.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 06 Aug 2012 22:17:55 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> It doesn't say what happens if "data definition's node" is absent in the data
> tree. And without a context node, the XPath expression is not well-defined.
> 
> 
> > 
> Now, consider an instance document containing <enable-foo>false</enable-foo>
> but not <foo>, i.e. the XPath context node is missing. Simple question: is
> this a valid document? If your answer is yes, could you explain how the
> validity follows from 6020?
> 
> I think this is where Andy needs to temporarily insert <foo> as his dummy
> element, which is however not "about to be created" as you write, at least not
> as a result of client's action. This has no support in 6020, the alone XPath.

Ok, I see your point.  But this doesn't seem to be related to the
original issue in this thread - that "when" somehow modifies the
*schema tree*.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 6 Aug 2012 13:36:34 -0700

> original issue in this thread - that "when" somehow modifies the

I agree.  I was not clear when I referred to schema nodes appearing
and disappearing
out of the schema tree.  Technically, the implementation does not add or remove
any schema nodes (independently of load and unload module operations).

Schema nodes have some saved state, and when the API functions that find
schema nodes are used, the disabled nodes are skipped.  So the yuma schema
tree is static.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 8 Aug 2012 18:18:12 +0200

> original issue in this thread - that "when" somehow modifies the

Not schema tree but schema, with which one can check the structure of an XML document without looking into element values of the validated document (think of a RELAX NG or XSD schema).

The 'mandatory' statement and the rules of section 3.1 in RFC 6020 result in schema constraints, but 'when' is a semantic constraint which should have no influence on the schema.

Anyway, if we can agree that the instance document in my example above is *not* valid, I will be satisfied.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 8 Aug 2012 09:49:29 -0700

> Anyway, if we can agree that the instance document in my example above is *not* valid, I will be satisfied.

Not sure what instance document you mean, but no, I do
not agree that an offline validation tool can ignore
the nodes that are currently in the target datastore
for every possible NETCONF request.  The 'when-stmt'
is a useful tool that allows many database tasks to be automated.

The protocol operations are written for a server to implement,
and they often require that the current values in the target datastore
be examined to process the client <rpc> request correctly.

The use of edit-config as a 'patch' tool is a feature, not a bug.
To do complete validation, the offline validation tool needs a correct
copy of the current target datastore.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 8 Aug 2012 10:47:50 -0700

.....
>>>> validity follows from 6020?

I assume these are the contents of the <config> node in an edit-config
operation.  The edit-config operation and the when statement are very clear
about when-stmts that are applied to the <rpc> context node vs.
those which are applied to a conceptual datastore + state document.

In your example, setting enable-foo=false or deleting it will cause the
/foo container to be deleted by the server.  The YANG statements
like when/must apply to conceptual datastores, not to NETCONF
messages.

>> Not schema tree but schema, with which one can check the structure of an XML document without looking into element values of the validated document (think of a RELAX NG or XSD schema).

A static off-line validation tool is limited in its inherent capabilities.
The protocol clearly defines the operations in terms of the
behavior on the server and the target datastore.  An offline tool
cannot provide datastore validation without knowing what is in it.

....

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 8 Aug 2012 21:50:59 +0200

> Not sure what instance document you mean, but no, I do

The document that contains <enable-foo>false</enable-foo> but not <foo>.

> for every possible NETCONF request.  The 'when-stmt'

I don't understand.

> be examined to process the client <rpc> request correctly.

Are these server actions documented in any standard?

> The use of edit-config as a 'patch' tool is a feature, not a bug.

I have nothing against it, provided that the resulting datastore content is valid.

> copy of the current target datastore.

Is this correct copy different from what a client obtains after sending a <get-config> without a filter?

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 8 Aug 2012 22:04:40 +0200

> I assume these are the contents of the <config> node in an edit-config

To keep things simple, it could be the content of a <copy-config>.

> like when/must apply to conceptual datastores, not to NETCONF

If the server does this, the resulting datastore content is invalid because from sec. 3.1 in 6020 it follows that the "foo" container is mandatory. The 'when' statement cannot change this because its XPath expression has no context node and is therefore undefined.

Lada

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 9 Aug 2012 00:49:54 +0200

Are you sure that you used the correct example? Because it does not seem
consistent with what you are saying. Container foo is not mandatory.. And
there's nothing wrong with an instance of enable-foo being present in a
datastore without an instance of foo. There's no such constraint on the
enable-foo leaf. It's just a plain old leaf.

What you claim would hold if mandatory-stmt and when-stmt were used on the
same leaf. A server would delete such a leaf instance if it's when
evaluated to false..but doing so would be deleting a mandatory data node -
leaving the datastore in an invalid state. Is this what you are saying?

Jernej
Dne 8. avg. 2012 22:04 je "Ladislav Lhotka" <[email-masked]> napisal/-a:

> If the server does this, the resulting datastore content is invalid
> because from sec. 3.1 in 6020 it follows that the "foo" container is
> mandatory. The 'when' statement cannot change this because its XPath
> expression has no context node and is therefore undefined.
>
>
> Lada
>
> >
> >
> >
> >>> Not schema tree but schema, with which one can check the structure of
> an XML document without looking into element values of the validated
> document (think of a RELAX NG or XSD schema).
> >>>
> >

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 9 Aug 2012 06:49:58 +0200

On Aug 9, 2012, at 12:49 AM, Jernej Tuljak wrote:

> Are you sure that you used the correct example? Because it does not seem consistent with what you are saying. Container foo is not mandatory.. And there's nothing wrong with an instance of enable-foo being present in a datastore without an instance of foo. There's no such constraint on the enable-foo leaf. It's just a plain old leaf.

Container "foo" is mandatory due to Sec. 3.1 in RFC 6020 (third bullet).

Lada

> 
> What you claim would hold if mandatory-stmt and when-stmt were used on the same leaf. A server would delete such a leaf instance if it's when evaluated to false..but doing so would be deleting a mandatory data node - leaving the datastore in an invalid state. Is this what you are saying?
> 
> Jernej
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 9 Aug 2012 06:56:54 -0700

Hi,

Why are we debating an implementation detail?
There are lots of combinations where the when-stmt
needs to be conceptually evaluated on a missing node
to determine if that node is triggering an error or not.
So what?  We have the same situation with conceptual
NP containers that contain default leafs:

   leaf if-foobar {
       when "../foobar";
       type string;
   }

   container foobar {
       leaf def {
          type int32;
          default 42;
       }
   }

The when-stmt is always true, whether the server
has a /foobar node or not.  It is an implementation
detail.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sat, 11 Aug 2012 07:41:09 -0700

Hi,

I think the scope of offline validation should be limited to YANG rpc input
parameters.  All of the <config> parameters in ietf-netconf.yang
are declared as 'anyxml'.  From the rpc-statement validation POV,
anyxml is treated as a leaf.  There are no YANG child nodes
specified, so therefore there are no constraints on child nodes of the
anyxml node.

The internal processing of the anyxml child nodes is part of the
datastore editing behavior for the edit-config or copy-config
implementation.  It is not part of the <rpc> input parameter validation.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Sat, 11 Aug 2012 19:26:59 +0200

> Why are we debating an implementation detail?

I have never mentioned any implementation in this thread.
 
> has a /foobar node or not.  It is an implementation

This case is not interesting because sec. 7.19.5 in RFC 6020 makes it very clear that the accessible tree for 'when' evaluation contains all leafs with default values.

Defaults interact with 'when' in another problematic way though:

leaf enable-foo {
  type boolean;
}
leaf foo {

Consider this scenario:

1. Client A sets "enable-foo" to "false".
2. Server deletes the leaf "foo".
3. Client B issues a <get-config>, asking for "foo".
4. The response will contain no data.

Now, can client B assume that "foo" effectively contains the default value (42)? If not, why?

The datastore schema is an important part of the contract between the server and client. If it was so, as you wrote, that the current schema can only be determined on the server, the client couldn't sometimes correctly interpret data obtained from the server. Therefore, I think that after finishing <hello> exchange *both* the server and client should be able to construct the schema and it should be identical on both sides of the connection.

So it is not about offline validation but rather about the contract and its interpretation. I don't know why the client couldn't validate the contents of a reply to <get-config> or <get>.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Sat, 11 Aug 2012 12:51:57 -0700

The <filter> element is also defined in YANG as anyxml.
Offline validation applies to <rpc> input parameters.
Those can always be validated offline
because the instance document for the when-stmt
evaluation is the <rpc> request message.

You are trying to apply the RPC input validation beyond
what YANG defines.  The NETCONF protocol operations
require the input parameters to be conceptually applied
to the current target datastore.

The DSDL mapping works for detecting errors in RPC input and output.
It does not work for validating the entire server operation if
the current state of the target datastore needs to be known to
support that operation.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 13 Aug 2012 10:10:23 +0200

> the current state of the target datastore needs to be known to

This discussion doesn't lead anywhere because you keep repeating this offline validation thing.

RFC 6020 provides, for the most part, validation rules for datastore contents, and I am discussing these rules rather than DSDL or offline validation.

Specifically, I am trying to point to gaps and inconsistencies in the definition of 'when'.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Mon, 13 Aug 2012 07:54:18 -0700

> Specifically, I am trying to point to gaps and inconsistencies in the definition of 'when'.

I think the RFCs are quite clear.
If the when-stmt applies to RPC input parameters, then the
complete <rpc> instance document is used for validation.

If the when-stmt applies to a data node within a NETCONF datastore,
then the complete conceptual datastore (plus all state) is used for validation.
Any tool that has access to this information can implement NETCONF.

Martin pointed out that even if descendant-or-self nodes are referenced
in a when-stmt, the server knows there are no instances of them,
so even these XPath expressions can be conceptually evaluated.

Please define the gaps and why a NETCONF server cannot be
implemented because of these gaps.

Andy