Subject: [netmod] Query about augmenting module from submodule in YANG 1.0

From: "Ivory, William" <[email-masked]>
Date: Thu, 3 Aug 2017 09:22:57 +0000

Hi,

We're trying to solve a modularity problem with a YANG module by splitting it into submodules and augmenting the parent module from each submodule.  However, despite the wording below in YANG 1.0 section 7.15, we've found a couple of threads online with comments suggesting it's only allowed in YANG 1.1?  Would appreciate clarification.

RFC 6020 section 7.15 suggests it is allowed:

'
   The "augment" statement allows a module or submodule to add to the
   schema tree defined in an external module, or the current module and
   its submodules, and to add to the nodes from a grouping in a "uses"
   statement.
'

Versus online comments here: https://www.ietf.org/mail-archive/web/netmod/current/msg15418.html

'> On 01 Mar 2016, at 10:38, Anton Tkáčik <anton.tkacik at pantheon.tech> wrote:

> Hi,

> Noticed other issue with example set,

> In https://github.com/mbj4668/pyang/issues/194 Lada stated that in YANG 1.0 submodule can not augment nodes

> defined in parent model.

> Is that correct that submodule can not augment definition defined in parent module?

This isn't possible in YANG 1.0 but will be possible in 1.1. However, in the present case the definition being augmented from the submodule is arguably in a different module.

Lada
'

Thanks,

William

---

From: Jan Lindblad <[email-masked]>
Date: Mon, 7 Aug 2017 15:27:49 +0200

The submodule concept in YANG 1.0 is, well, not very useful, and even less intuitive. That's why it saw major rework in YANG 1.1.

A YANG 1.0 submodule cannot reference the module that includes it, directly or indirectly. This is because in YANG 1.0 the symbols in other submodules of the same namespace are invisible to the submodule unless they are explicitly included. And parent modules can't be included by a submodule because that would lead to an inclusion loop. It is possible to reference (augment, etc) other sibling submodules, though. So if you split your modules cleverly, you might be able to resolve your referential constraints anyway. 

If you really want to take the submodule path, I'd recommend moving to YANG 1.1. In the interest of preserving the hair tone of IT-architects.

/jan

> We’re trying to solve a modularity problem with a YANG module by splitting it into submodules and augmenting the parent module from each submodule.  However, despite the wording below in YANG 1.0 section 7.15, we’ve found a couple of threads online with comments suggesting it’s only allowed in YANG 1.1?  Would appreciate clarification.
>  
> ‘
> ‘
> Versus online comments here: https://www.ietf.org/mail-archive/web/netmod/current/msg15418.html <https://www.ietf.org/mail-archive/web/netmod/current/msg15418.html>
>  
> ‘> On 01 Mar 2016, at 10:38, Anton Tkáčik <anton.tkacik at pantheon.tech> wrote:
> > 
> > Hi,
> > In https://github.com/mbj4668/pyang/issues/194 <https://github.com/mbj4668/pyang/issues/194> Lada stated that in YANG 1.0 submodule can not augment nodes
> > defined in parent model.
> > 
> ‘
> _______________________________________________
> netmod mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/netmod <https://www.ietf.org/mailman/listinfo/netmod>

---

From: "Ivory, William" <[email-masked]>
Date: Mon, 7 Aug 2017 14:37:04 +0000

Hi Jan,

Thanks – we’ll look at trying to put everything into submodules in this case.

Regards,

William

From: Jan Lindblad [mailto:[email-masked]]
Sent: 07 August 2017 14:28
To: Ivory, William <[email-masked]>
Cc: [email-masked]
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

/jan

‘

Versus online comments here: https://www.ietf.org/mail-archive/web/netmod/current/msg15418.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail-2Darchive_web_netmod_current_msg15418.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=OxxQRDucETBaDPn4KGNWcLlu4e8AMSfuyJJjrklp3R0&e=>

‘> On 01 Mar 2016, at 10:38, Anton Tkáčik <anton.tkacik at pantheon.tech> wrote:

> In https://github.com/mbj4668/pyang/issues/194<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mbj4668_pyang_issues_194&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=bkakKJEZzCBq3BkP5NzW-wDX6KOZHpOnT0u-ySg8rS0&e=> Lada stated that in YANG 1.0 submodule can not augment nodes

Lada
‘

Thanks,

William

_______________________________________________
netmod mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netmod<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=x7sK1jWYtSsQJr8r6G7FjWR5gAoMtgv6zRwxT4bzMGQ&e=>

---

From: Vladimir Vassilev <[email-masked]>
Date: Mon, 7 Aug 2017 21:30:32 +0200

Hello,

IMO "submodule"s  are a striking example of redundant complexity in an 
otherwise very close to perfection YANG (regardless if it is YANG 1.0 or 
1.1).

Modules and submodules have been around for a while however the ratio of 
the currently published modules compared with the submodules is about 40 
modules to 1 submodule (if one ignores all the modules and submodules 
from  particular networking hardware manufacturer that is particularly 
keen on using submodules). As a far but still relevant analogy Java has 
a limitation of 1 file per class and this atomicity has proven to be an 
advantage especially in terms of enforcing modularity. IMO there is 
nothing that can be done with the help of submodules that can not be 
done without them.

For the sake of the argument can you provide a synthesized description 
of the problem that lead you to a drastic solution like "we’ll look at 
trying to put everything into submodules in this case."?

Vladimir

On 08/07/2017 04:37 PM, Ivory, William wrote:
> *From:*Jan Lindblad [mailto:[email-masked]]
> *Sent:* 07 August 2017 14:28
> *To:* Ivory, William <[email-masked]>
> *Cc:* [email-masked]
> *Subject:* Re: [netmod] Query about augmenting module from submodule 
> in YANG 1.0
>
> The submodule concept in YANG 1.0 is, well, not very useful, and even 
> less intuitive. That's why it saw major rework in YANG 1.1.
>
> A YANG 1.0 submodule cannot reference the module that includes it, 
> directly or indirectly. This is because in YANG 1.0 the symbols in 
> other submodules of the same namespace are invisible to the submodule 
> unless they are explicitly included. And parent modules can't be 
> included by a submodule because that would lead to an inclusion loop. 
> It is possible to reference (augment, etc) other sibling submodules, 
> though. So if you split your modules cleverly, you might be able to 
> resolve your referential constraints anyway.
>
> If you really want to take the submodule path, I'd recommend moving to 
> YANG 1.1. In the interest of preserving the hair tone of IT-architects.
>
> /jan
>
>     We’re trying to solve a modularity problem with a YANG module by
>     splitting it into submodules and augmenting the parent module from
>     each submodule.  However, despite the wording below in YANG 1.0
>     section 7.15, we’ve found a couple of threads online with comments
>     suggesting it’s only allowed in YANG 1.1?  Would appreciate
>     clarification.
>
>     Versus online comments
>     here:https://www.ietf.org/mail-archive/web/netmod/current/msg15418.html
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail-2Darchive_web_netmod_current_msg15418.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=OxxQRDucETBaDPn4KGNWcLlu4e8AMSfuyJJjrklp3R0&e=>
>
>     > Inhttps://github.com/mbj4668/pyang/issues/194
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mbj4668_pyang_issues_194&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=bkakKJEZzCBq3BkP5NzW-wDX6KOZHpOnT0u-ySg8rS0&e=>  Lada stated that in YANG 1.0 submodule can not augment nodes
>
>     > defined in parent model.
>
>     >
>
>     https://www.ietf.org/mailman/listinfo/netmod
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=x7sK1jWYtSsQJr8r6G7FjWR5gAoMtgv6zRwxT4bzMGQ&e=>
>
>
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: "Ivory, William" <[email-masked]>
Date: Tue, 8 Aug 2017 08:15:32 +0000

Hi Vladimir,

We have one YANG file that represents multiple components in the system.  Currently they are bundled together, so having a single YANG file is ok.  In future we’d like to be able to break this down into multiple daemons each dealing with a subset of the YANG.  However, we don’t wish to change the namespace of the YANG as that would not be backwards compatible.  So, submodules looked to be a good way to do this.  I wouldn’t call it drastic – it is one YANG file we are talking about breaking up into parts.

Regards,

William

From: Vladimir Vassilev [mailto:[email-masked]]
Sent: 07 August 2017 20:31
To: Ivory, William <[email-masked]>
Cc: [email-masked]' <[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

Hello,
IMO "submodule"s  are a striking example of redundant complexity in an otherwise very close to perfection YANG (regardless if it is YANG 1.0 or 1.1).

Modules and submodules have been around for a while however the ratio of the currently published modules compared with the submodules is about 40 modules to 1 submodule (if one ignores all the modules and submodules from  particular networking hardware manufacturer that is particularly keen on using submodules). As a far but still relevant analogy Java has a limitation of 1 file per class and this atomicity has proven to be an advantage especially in terms of enforcing modularity. IMO there is nothing that can be done with the help of submodules that can not be done without them.

For the sake of the argument can you provide a synthesized description of the problem that lead you to a drastic solution like "we’ll look at trying to put everything into submodules in this case."?

Vladimir

Regards,

William

From: Jan Lindblad [mailto:[email-masked]]
Sent: 07 August 2017 14:28
To: Ivory, William <[email-masked]><mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

/jan

‘

‘> On 01 Mar 2016, at 10:38, Anton Tkáčik <anton.tkacik at pantheon.tech> wrote:

Lada
‘

Thanks,

William

netmod mailing list

https://www.ietf.org/mailman/listinfo/netmod<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=M7t8vTUb71XRWW7ZfSHTMlFEaAhzOdmQuBmw2ah-uGc&s=NFJL1RjYNxNMcnPhhm--ECwdEdyUXHGEVEq4fsjzruk&e=>

---

From: Vladimir Vassilev <[email-masked]>
Date: Wed, 9 Aug 2017 16:20:25 +0200

On 08/08/2017 10:15 AM, Ivory, William wrote:
>
> Hi Vladimir,
>
> We have one YANG file that represents multiple components in the 
> system.  Currently they are bundled together, so having a single YANG 
> file is ok.  In future we’d like to be able to break this down into 
> multiple daemons each dealing with a subset of the YANG.  However, we 
> don’t wish to change the namespace of the YANG as that would not be 
> backwards compatible.  So, submodules looked to be a good way to do 
> this.  I wouldn’t call it drastic – it is one YANG file we are talking 
> about breaking up into parts.
>
I see your point. IMO the only real justification for people designing 
using submodules instead of modules is when they are limited to single 
namespace and need a workaround solution like in your case.
I was hoping your problem could be something that can convince me that 
submodules existence in YANG can be justified with more then its 
function as a workaround replacement for modules in this particular case.

My grudge against submodules is not only based on the significant 
implementation and support effort they require for something that is 
used very rarely. A completely separate source file quantum for YANG 
that lacks the key property of a module at the YANG level - modularity. 
For submodules are both non-reusable and interdependent. Very few 
organizations publish submodule based designs probably for the same 
reasons I avoid them. Submodules are great if you want to publish 
non-reusable YANG though.

IETF went once for design with submodules in ietf-snmp.yang. Even in 
that case (well organized YANG module) I would have preferred a single 
file with some of the exotic features modularized in separate modules 
instead. Dynamic compilers still need to go through all submodules even 
in device that supports only the base SNMP functionality before features 
can be evaluated. As a result instead of the 10KB of actually 
implemented schema 60KB of additional YANG has to be retrieved in the 
worst case and compiled.

Both submodules and alternative datastores are examples of how 
complexity is introduced with innocent intentions and how it eventually 
multiplies (ref. draft-nmdsdt-netconf-rfc7895bis-01).

The biggest problem I have with submodules is they break the atomicity 
of the module concept. There is something that is not right with that. 
Worse than the unjustified implementation and standardization effort. A 
compromise that should have been avoided.

IMO If you absolutely don't need submodules it is best to stay away from 
them.

Vladimir

> Regards,
>
> William
>
> *From:*Vladimir Vassilev [mailto:[email-masked]]
> *Sent:* 07 August 2017 20:31
> *To:* Ivory, William <[email-masked]>
> *Cc:* [email-masked]' <[email-masked]>
> otherwise very close to perfection YANG (regardless if it is YANG 1.0 
> or 1.1).
>
> Modules and submodules have been around for a while however the ratio 
> of the currently published modules compared with the submodules is 
> about 40 modules to 1 submodule (if one ignores all the modules and 
> submodules from  particular networking hardware manufacturer that is 
> particularly keen on using submodules). As a far but still relevant 
> analogy Java has a limitation of 1 file per class and this atomicity 
> has proven to be an advantage especially in terms of enforcing 
> modularity. IMO there is nothing that can be done with the help of 
> submodules that can not be done without them.
>
>     <mailto:[email-masked]>
>     *Cc:* [email-masked] <mailto:[email-masked]>
>     *Subject:* Re: [netmod] Query about augmenting module from
>     submodule in YANG 1.0
>
>     The submodule concept in YANG 1.0 is, well, not very useful, and
>     even less intuitive. That's why it saw major rework in YANG 1.1.
>
>     other submodules of the same namespace are invisible to the
>     submodule unless they are explicitly included. And parent modules
>     can't be included by a submodule because that would lead to an
>     inclusion loop. It is possible to reference (augment, etc) other
>     sibling submodules, though. So if you split your modules cleverly,
>     you might be able to resolve your referential constraints anyway.
>
>     If you really want to take the submodule path, I'd recommend
>     moving to YANG 1.1. In the interest of preserving the hair tone of
>     IT-architects.
>
>     /jan
>
>         We’re trying to solve a modularity problem with a YANG module
>         by splitting it into submodules and augmenting the parent
>         module from each submodule.  However, despite the wording
>         below in YANG 1.0 section 7.15, we’ve found a couple of
>         threads online with comments suggesting it’s only allowed in
>         YANG 1.1?  Would appreciate clarification.
>
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=M7t8vTUb71XRWW7ZfSHTMlFEaAhzOdmQuBmw2ah-uGc&s=NFJL1RjYNxNMcnPhhm--ECwdEdyUXHGEVEq4fsjzruk&e=>
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 09 Aug 2017 17:01:09 +0200

> implemented schema 60KB of additional YANG has to be retrieved in the 

I agree. I have seen quite a few bugs (both in my and somebody else's code)
directly caused by submodules, e.g. in definition lookup. You are right that in
YANG 1.0 the "include" logic was really horrible but still we may be better off
with no submodules at all.

I remember that in early stages of YANG there was some irrational fear of
introducing too many namespaces, and submodules may be a consequence of it. As
you write, submodules provide no benefits whatsoever in terms of modularity, but
the overhead in terms of metadata, IANA registration etc. is pretty much the
same as for modules.

Lada

> >         <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_m
> > ail-2Darchive_web_netmod_current_msg15418.html&d=DwMFaQ&c=LFYZ-
> > o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-
> > 26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=OxxQRDucETBaDPn4KGN
> > WcLlu4e8AMSfuyJJjrklp3R0&e=>
> > 
> >         <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mbj
> > 4668_pyang_issues_194&d=DwMFaQ&c=LFYZ-
> > o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-
> > 26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=bkakKJEZzCBq3BkP5Nz
> > W-wDX6KOZHpOnT0u-ySg8rS0&e=>  Lada stated that in YANG 1.0 submodule can not
> > augment nodes
> > 
> >         > defined in parent model.
> > 
> >         >
> > 
> >         This isn't possible in YANG 1.0 but will be possible in 1.1.
> > However, in the present case the definition being augmented from the
> > submodule is arguably in a different module.
> > 
> >           
> > 
> >         Lada
> > 
> >         ‘
> > 
> >         Thanks,
> > 
> >         William
> > 
> >         <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_m
> > ailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-
> > o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-
> > 26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=x7sK1jWYtSsQJr8r6G7
> > FjWR5gAoMtgv6zRwxT4bzMGQ&e=>
> > 
> > 
> > 
> > 
> >     <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailm
> > an_listinfo_netmod&d=DwMFaQ&c=LFYZ-
> > o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-
> > 26_Qa9vw&m=M7t8vTUb71XRWW7ZfSHTMlFEaAhzOdmQuBmw2ah-uGc&s=NFJL1RjYNxNMcnPhhm-
> > -ECwdEdyUXHGEVEq4fsjzruk&e=>
> > 
> 
-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 9 Aug 2017 17:13:12 +0200

On Wed, Aug 09, 2017 at 05:01:09PM +0200, Ladislav Lhotka wrote:
> 
> I remember that in early stages of YANG there was some irrational
> fear of introducing too many namespaces, and submodules may be a
> consequence of it. As you write, submodules provide no benefits
> whatsoever in terms of modularity, but the overhead in terms of
> metadata, IANA registration etc. is pretty much the same as for
> modules.

In case YANG 2.0 is ever done, I suggest someone files a proposal to
remove submodules if the cost/benefit ratio is at odds. There is
nothing wrong with removing stuff that has been found problematic.

The motivation for submodules was that organizations maintaining large
modules with multiple people can do so without having to mess around
with tools like m4 scripts to produce a single module from 'snippets'
and to avoid integration surprises. But perhaps using m4 scripts and
decent version control systems (that can integrate and compile on
checkin) is indeed cheaper than having submodules part of the YANG
language itself.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Wed, 9 Aug 2017 09:20:27 -0700

On Wed, Aug 9, 2017 at 7:20 AM, Vladimir Vassilev <[email-masked]>
wrote:

>> We have one YANG file that represents multiple components in the system.
>> Currently they are bundled together, so having a single YANG file is ok.
>> In future we’d like to be able to break this down into multiple daemons
>> each dealing with a subset of the YANG.  However, we don’t wish to change
>> the namespace of the YANG as that would not be backwards compatible.  So,
>> submodules looked to be a good way to do this.  I wouldn’t call it drastic
>> – it is one YANG file we are talking about breaking up into parts.
>>
> submodules existence in YANG can be justified with more then its function
> as a workaround replacement for modules in this particular case.
>
> implementation and support effort they require for something that is used
> very rarely. A completely separate source file quantum for YANG that lacks
> the key property of a module at the YANG level - modularity. For submodules
> are both non-reusable and interdependent. Very few organizations publish
> submodule based designs probably for the same reasons I avoid them.
> Submodules are great if you want to publish non-reusable YANG though.
>
> IETF went once for design with submodules in ietf-snmp.yang. Even in that
> case (well organized YANG module) I would have preferred a single file with
> some of the exotic features modularized in separate modules instead.
> Dynamic compilers still need to go through all submodules even in device
> that supports only the base SNMP functionality before features can be
> evaluated. As a result instead of the 10KB of actually implemented schema
> 60KB of additional YANG has to be retrieved in the worst case and compiled.
>
> Both submodules and alternative datastores are examples of how complexity
> is introduced with innocent intentions and how it eventually multiplies
> (ref. draft-nmdsdt-netconf-rfc7895bis-01).
>
> The biggest problem I have with submodules is they break the atomicity of
> the module concept. There is something that is not right with that. Worse
> than the unjustified implementation and standardization effort. A

I argued against submodules from the start.
They add a lot of complexity for implementors and module readers.
Too much cost for the 1 benefit of reusing a namespace.
Good summary about why they are quite rigid and not really modular at all.

They work even worse when include-without-revision is used.
In this most common case, the actual submodule revisions used
are very implementation-dependent.  There is no way to cross-check
the main module revision against the submodule revisions.
(i.e., no belongs-to foo@2017-01-01, just belongs-to foo).

I have seen companies start to use submodules, then undo it when they
figure out
that a monolithic YANG-ball is not as workable as a module-set.

> Vladimir

Andy

>> *Subject:* Re: [netmod] Query about augmenting module from submodule in
>> YANG 1.0
>>
>> Hello,
>>
>> modules to 1 submodule (if one ignores all the modules and submodules from
>> particular networking hardware manufacturer that is particularly keen on
>> using submodules). As a far but still relevant analogy Java has a
>> limitation of 1 file per class and this atomicity has proven to be an
>> advantage especially in terms of enforcing modularity. IMO there is nothing
>> that can be done with the help of submodules that can not be done without
>> them.
>>
>> For the sake of the argument can you provide a synthesized description of
>> the problem that lead you to a drastic solution like "we’ll look at trying
>> to put everything into submodules in this case."?
>>
>> Vladimir
>>
>>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
>> ietf.org_mail-2Darchive_web_netmod_current_msg15418.html&
>> d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FmJP9CH54z5mG
>> 3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_q
>> dR2aM3uOFWpZYtgF122Ec&s=OxxQRDucETBaDPn4KGNWcLlu4e8AMSfuyJJjrklp3R0&e=>
>>
>>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__
>> github.com_mbj4668_pyang_issues_194&d=DwMFaQ&c=LFYZ-o9_HUMeM
>> TSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_Qa9vw&m=YC
>> 4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=bkakKJEZzCBq3BkP
>> 5NzW-wDX6KOZHpOnT0u-ySg8rS0&e=>  Lada stated that in YANG 1.0 submodule
>> can not augment nodes
>>
>>         > defined in parent model.
>>
>>         >
>>
>>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
>> ietf.org_mailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-o9_
>> HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_
>> Qa9vw&m=YC4w6Zi9KhBp0MnnvA42_qdR2aM3uOFWpZYtgF122Ec&s=x7sK1j
>> WYtSsQJr8r6G7FjWR5gAoMtgv6zRwxT4bzMGQ&e=>
>>
>>
>>
>>
>>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
>> ietf.org_mailman_listinfo_netmod&d=DwMFaQ&c=LFYZ-o9_
>> HUMeMTSQicvjIg&r=FmJP9CH54z5mG3DFGBdc_9q1TLpYQ31-TQ-26_
>> Qa9vw&m=M7t8vTUb71XRWW7ZfSHTMlFEaAhzOdmQuBmw2ah-uGc&s=NFJL1R
>> jYNxNMcnPhhm--ECwdEdyUXHGEVEq4fsjzruk&e=>
>>
>>

---

From: Robert Wilton <[email-masked]>
Date: Mon, 21 Aug 2017 11:23:47 +0100

> nothing wrong with removing stuff that has been found problematic.
I agree.

I've added https://github.com/netmod-wg/yang-next/issues/26

Rob

---

From: "Ivory, William" <[email-masked]>
Date: Mon, 21 Aug 2017 13:44:20 +0000

Hi Rob,

That would make it very hard to update existing 1.x YANG models to use new features in YANG 2.x if they used submodules.  Maybe that's something that no one would ever consider doing anyway, or maybe YANG 1.1 already has similar differences to 1.0?  I had (perhaps naively) assumed that you could migrate a namespace / model from YANG 1.0 to 2.0?

Regards,

William

-----Original Message-----
From: netmod [mailto:[email-masked]] On Behalf Of Robert Wilton
Sent: 21 August 2017 11:24
To: [email-masked]
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

I agree.

I've added https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netmod-2Dwg_yang-2Dnext_issues_26&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlAT7owI&s=-kR4fUtXArQy0RwWb32DpT1bP4X_cNqt2zJVoC0JiX8&e= 

Rob

_______________________________________________
netmod mailing list
[email-masked]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_netmod&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlAT7owI&s=t7vGIH8ABuAm00e-bkSowD9eawModGq0N2OkjANtpYI&e=

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Mon, 21 Aug 2017 14:01:45 +0000

Hi William, Rob, Andy,

Given their limited usefulness and the detriments, perhaps we should
discourage the creation of new submodules in RFC6087Bis.

Thanks,
Acee

On 8/21/17, 9:44 AM, "netmod on behalf of Ivory, William"
<[email-masked] on behalf of [email-masked]> wrote:

>Hi Rob,
>
>That would make it very hard to update existing 1.x YANG models to use
>new features in YANG 2.x if they used submodules.  Maybe that's something
>that no one would ever consider doing anyway, or maybe YANG 1.1 already
>has similar differences to 1.0?  I had (perhaps naively) assumed that you
>could migrate a namespace / model from YANG 1.0 to 2.0?
>
>Regards,
>
>William
>
>I've added 
>https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netmod-2Dw
>g_yang-2Dnext_issues_26&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ
>2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlAT7ow
>I&s=-kR4fUtXArQy0RwWb32DpT1bP4X_cNqt2zJVoC0JiX8&e=
>
>Rob
>
>>
>https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_
>listinfo_netmod&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ2ZPGqwky
>XmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlAT7owI&s=t7vG
>IH8ABuAm00e-bkSowD9eawModGq0N2OkjANtpYI&e=
>

---

From: Robert Wilton <[email-masked]>
Date: Mon, 21 Aug 2017 15:04:04 +0100

Hi William,

The github issue is really only to track this point for further 
discussion and to the avoid losing the useful discussion on the alias.

I think that there is often useful discussion on the NETMOD alias that 
ends up at least with a partial resolution, but that never gets 
documented anywhere.  In a few cases these end up as errata, but often 
they either get deferred to a future version, or are treated as 
collective wisdom from the WG participants.  For the latter case, I'm 
hoping that we can store the summary of some of these discussions on a 
community maintained wiki.

YANG 1/1.1 modules using sub-modules could be upgraded into a single 
YANG 2.0 module.  This is an allowed backwards compatible change with no 
changes to the node paths or namespaces.

The pertinent text is in RFC 7950, sec 11:

    o  A module may be split into a set of submodules or a submodule may
       be removed, provided the definitions in the module do not change
       in any way other than those allowed here.

Thanks,
Rob

On 21/08/2017 14:44, Ivory, William wrote:
> .

---

From: Robert Wilton <[email-masked]>
Date: Mon, 21 Aug 2017 16:14:50 +0100

Hi Acee,

That makes sense.

The other thing that I think that we have got wrong is modelling regex 
pattern statements.  I think that it would be much better if these were 
written to be less exhaustive and much simpler.

E.g. the "route distinguisher" pattern in 
draft-ietf-rtgwg-routing-types-09 is defined as this:

          pattern
            '(0:(6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|'
          +     '6[0-4][0-9]{3}|'
          +     '[0-5]?[0-9]{0,3}[0-9]):(429496729[0-5]|'
          +     '42949672[0-8][0-9]|'
          +     '4294967[01][0-9]{2}|429496[0-6][0-9]{3}|'
          +     '42949[0-5][0-9]{4}|'
          +     '4294[0-8][0-9]{5}|429[0-3][0-9]{6}|'
          +     '42[0-8][0-9]{7}|4[01][0-9]{8}|'
          +     '[0-3]?[0-9]{0,8}[0-9]))|'
          + '(1:((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|'
          +     '25[0-5])\.){3}([0-9]|[1-9][0-9]|'
          +     '1[0-9]{2}|2[0-4][0-9]|25[0-5])):(6553[0-5]|'
          +     '655[0-2][0-9]|'
          +     '65[0-4][0-9]{2}|6[0-4][0-9]{3}|'
          +     '[0-5]?[0-9]{0,3}[0-9]))|'
          + '(2:(429496729[0-5]|42949672[0-8][0-9]|'
          +     '4294967[01][0-9]{2}|'
          +     '429496[0-6][0-9]{3}|42949[0-5][0-9]{4}|'
          +     '4294[0-8][0-9]{5}|'
          +     '429[0-3][0-9]{6}|42[0-8][0-9]{7}|4[01][0-9]{8}|'
          +     '[0-3]?[0-9]{0,8}[0-9]):'
          +     '(6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|'
          +     '6[0-4][0-9]{3}|'
          +     '[0-5]?[0-9]{0,3}[0-9]))|'
          + '(6(:[a-fA-F0-9]{2}){6})|'
          + '(([3-57-9a-fA-F]|[1-9a-fA-F][0-9a-fA-F]{1,3}):'
          +     '[0-9a-fA-F]{1,12})';
        }

But I think that it would be much easier to read, and quite possibly 
more performant to execute, if the pattern regex was written something 
like the following:

  pattern:
     '(0:[0-9]{1,5}:[0-9]{1,10})|
      (1:([0-9]{1,3}\.){4}:[0-9]{1,5})|
      (2:[0-9]{1,10}:0:[0-9]{1,5})|
      (6(:[a-fA-F0-9]{2}){6})';

Of course, this would allow more invalid values, but most servers would 
be expected to reject those when it converts them into an internal 
binary format any way.

What do you, and others, think?

Thanks,
Rob

---

From: Lou Berger <[email-masked]>
Date: Tue, 22 Aug 2017 14:38:07 -0400

Acee, Rob,

    Can you propose a specific change/addition to 6087Bis?

Thanks,
Lou

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Tue, 22 Aug 2017 22:21:23 +0000

Hi Rob,
I’m fine with simplifying (even if the resultant pattern is much less impressive ;^). I’ve copied the BESS WG to see if there are any objections.
Thanks,
Acee

From: "Robert Wilton -X (rwilton - ENSOFT LIMITED at Cisco)" <[email-masked]<mailto:[email-masked]>>
Date: Monday, August 21, 2017 at 11:14 AM
To: Acee Lindem <[email-masked]<mailto:[email-masked]>>, "Ivory, William" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>'" <[email-masked]<mailto:[email-masked]>>, Andy Bierman <[email-masked]<mailto:[email-masked]>>
Subject: Pattern statements [was Re: [netmod] Query about augmenting module from submodule in YANG 1.0]

Hi Acee,

That makes sense.

The other thing that I think that we have got wrong is modelling regex pattern statements.  I think that it would be much better if these were written to be less exhaustive and much simpler.

E.g. the "route distinguisher" pattern in draft-ietf-rtgwg-routing-types-09 is defined as this:

         pattern

But I think that it would be much easier to read, and quite possibly more performant to execute, if the pattern regex was written something like the following:

 pattern:

Of course, this would allow more invalid values, but most servers would be expected to reject those when it converts them into an internal binary format any way.

Thanks,
Rob

Hi William, Rob, Andy,

Thanks,
Acee

On 8/21/17, 9:44 AM, "netmod on behalf of Ivory, William"
<[email-masked] on behalf of [email-masked]><mailto:[email-masked]@intl.att.com> wrote:

Hi Rob,

Regards,

William

-----Original Message-----
From: netmod [mailto:[email-masked]] On Behalf Of Robert Wilton
Sent: 21 August 2017 11:24
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in
YANG 1.0

I agree.

I've added

Rob

/js

[email-masked]<mailto:[email-masked]>https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_

_______________________________________________
netmod mailing list
[email-masked]<mailto:[email-masked]>https://www.ietf.org/mailman/listinfo/netmod

---

From: Alex Campbell <[email-masked]>
Date: Tue, 22 Aug 2017 22:27:57 +0000

Hi,

I'm not Rob, but my understanding is that if a module author wanted to migrate to YANG 2.0, they could merge their submodules back into the main module - which is not a difficult procedure and does not break compatibility with clients.

Alex
________________________________________
From: netmod <[email-masked]> on behalf of Ivory, William <[email-masked]>
Sent: Tuesday, 22 August 2017 1:44 a.m.
To: 'Robert Wilton'; [email-masked]'
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

Hi Rob,

Regards,

William

-----Original Message-----
From: netmod [mailto:[email-masked]] On Behalf Of Robert Wilton
Sent: 21 August 2017 11:24
To: [email-masked]
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

I agree.

Rob

---

From: "Ivory, William" <[email-masked]>
Date: Wed, 23 Aug 2017 07:08:10 +0000

...  except that if the whole reason for splitting into submodules was to allow the submodules to belong to different packages in our system, combining them back again is not possible.  I wouldn't be splitting them unless I needed to for good reason.

William

-----Original Message-----
From: Alex Campbell [mailto:[email-masked]] 
Sent: 22 August 2017 23:28
To: Ivory, William <[email-masked]>; 'Robert Wilton' <[email-masked]>; [email-masked]' <[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

Hi,

Alex

Hi Rob,

Regards,

William

-----Original Message-----
From: netmod [mailto:[email-masked]] On Behalf Of Robert Wilton
Sent: 21 August 2017 11:24
To: [email-masked]
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

I agree.

Rob

_______________________________________________
netmod mailing list
[email-masked]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_netmod&d=DwIFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=esi8GPSc1xVjTt9SKxqzNHRDXT2P1h01a-UebnST-Yo&s=PctKy3ij6W0TQs1NFp18SX8MQtYKeG9RxADh3cphcxU&e=

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 23 Aug 2017 09:24:35 +0200

What are packages? I think submodules declare to which module they
belong, no? Perhaps you are doing something that submodules do not
even support.

/js

---

From: "Ivory, William" <[email-masked]>
Date: Wed, 23 Aug 2017 08:24:29 +0000

Sorry - meant Debian packages.  We have a large YANG module that really ought to be handled by multiple daemons, so plan to use submodules so we can identify the different parts and process them in the right place.  Recombining into a single module would lose that granularity.

William

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: 23 August 2017 08:25
To: Ivory, William <[email-masked]>
Cc: 'Alex Campbell' <[email-masked]>; 'Robert Wilton' <[email-masked]>; [email-masked]' <[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

What are packages? I think submodules declare to which module they belong, no? Perhaps you are doing something that submodules do not even support.

/js

On Wed, Aug 23, 2017 at 07:08:10AM +0000, Ivory, William wrote:
> To: Ivory, William <[email-masked]>; 'Robert Wilton' 
> <[email-masked]>; [email-masked]' <[email-masked]>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netmod
> -2Dwg_yang-2Dnext_issues_26&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK
> 3u4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU6
> 1xSHgPlAT7owI&s=-kR4fUtXArQy0RwWb32DpT1bP4X_cNqt2zJVoC0JiX8&e=
> 
> Rob
> 
> >
> > The motivation for submodules was that organizations maintaining 
> > large modules with multiple people can do so without having to mess 
> > around with tools like m4 scripts to produce a single module from 'snippets'
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> man_listinfo_netmod&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ
> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlA
> T7owI&s=t7vGIH8ABuAm00e-bkSowD9eawModGq0N2OkjANtpYI&e=
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> man_listinfo_netmod&d=DwIFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ
> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=esi8GPSc1xVjTt9SKxqzNHRDXT2P1h01a-Uebn
> ST-Yo&s=PctKy3ij6W0TQs1NFp18SX8MQtYKeG9RxADh3cphcxU&e=
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail
> man_listinfo_netmod&d=DwIBAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ
> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=RH3zvD2B8_s4uw1PXm_ka37vgz9_q2Rc87tD8K
> fZ9jA&s=XydU0vXE0AEg2FDE-kx_Ae6rOOAh5koxEeJ2cefgDNA&e=

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.jacobs-2Duniversity.de_&d=DwIBAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=RH3zvD2B8_s4uw1PXm_ka37vgz9_q2Rc87tD8KfZ9jA&s=OW5FOHEum17LXiaKGUQjFyrIcEJmvkc2yXhwsTsCe-Y&e= >

---

From: Robert Wilton <[email-masked]>
Date: Wed, 23 Aug 2017 10:46:15 +0100

Hi William,

I think that there might be some downsides to your proposed solution of 
using submodules - which you may have already considered:

1) All of the debian packages will have to be installed together to 
allow the YANG module to be built, or otherwise there will be missing 
submodules, and the module will fail to compile (since the top level 
module must list all included sub-modules).
2) It might end up with your requiring tight versioning of all of your 
debian packages together with the same version number, or otherwise you 
may need greater care over how the sub-modules are updated and 
dependencies are handled.

If your YANG was being designed from scratch then using separate YANG 
modules may allow for a cleaner solution - but I appreciate that may not 
help you with where you are now.

Thanks,
Rob

---

From: "t.petch" <[email-masked]>
Date: Wed, 23 Aug 2017 10:45:51 +0100

----- Original Message -----
From: "Robert Wilton" <[email-masked]>
Sent: Monday, August 21, 2017 4:14 PM

> draft-ietf-rtgwg-routing-types-09 is defined as this:

Simplify!

Bear in mind that the regex for an IPv6 address was wrong for a long
time in base YANG before anyone noticed - it was just too complex.

And ABNF learnt long ago that just because something could be expressed
in code does not mean that it is a good idea to do so.  If a simple
English statement replaces many lines of ABNF, then that is a good
tradeoff.

Pragmatically I am not sure what the cutoff for complexity should be but
it should be less than we have now.

Paradoxically, given the original thread, the time when large
expressions may work ok is when they have a 'sub-module' like structure,
when I can look at a group of lines in isolation and form a view of what
it does then move on to the next group and so on, building up an overall
picture piece by piece.

Tom Petch

> >>> nothing wrong with removing stuff that has been found problematic.
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netmod-2
Dw
g_yang-2Dnext_issues_26&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYi
aQ
2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlAT7
ow
> >>> The motivation for submodules was that organizations maintaining
large
> >>> checkin) is indeed cheaper than having submodules part of the YANG
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailma
n_
listinfo_netmod&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ2ZPGqw
ky
XmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPlAT7owI&s=t7
vG
> >> IH8ABuAm00e-bkSowD9eawModGq0N2OkjANtpYI&e=

------------------------------------------------------------------------
--------

---

From: "Ivory, William" <[email-masked]>
Date: Wed, 23 Aug 2017 10:05:52 +0000

Personally I hate regexps as anything other than the simplest tend to be too cryptic to understand easily (or to test).  At least from YANG 1.1 we have the ability to have 'exclude' patterns (IIRC), but it's still not ideal.

What I'd like to see instead is bitwise operators (&, |, << and >>) along with 'for' loops and possibly local variables allowed in the version of XPATH  that we use.  Would likely need to define YANG-specific functions to extend XPATH, but we already have some (current() etc) so that wouldn't be a problem.  We have quite a few cases where we have to resort to calling out to scripts to provide local validation that would not be necessary if we had such operators in XPATH, and would allow NETCONF clients to perform validation off-box.

It would also be fun to implement - I rather enjoyed implementing our XPATH parser - but will concede that's probably not a priority when determining feature sets for future versions (-:

William

-----Original Message-----
From: t.petch [mailto:[email-masked]] 
Sent: 23 August 2017 10:46
To: Acee Lindem (acee) <[email-masked]>; Ivory, William <[email-masked]>; [email-masked]; Andy Bierman <[email-masked]>; Robert Wilton <[email-masked]>
Subject: Re: [netmod] Pattern statements [was Re: Query about augmenting module from submodule in YANG 1.0]

----- Original Message -----
From: "Robert Wilton" <[email-masked]>
Sent: Monday, August 21, 2017 4:14 PM

Simplify!

Bear in mind that the regex for an IPv6 address was wrong for a long time in base YANG before anyone noticed - it was just too complex.

And ABNF learnt long ago that just because something could be expressed in code does not mean that it is a good idea to do so.  If a simple English statement replaces many lines of ABNF, then that is a good tradeoff.

Pragmatically I am not sure what the cutoff for complexity should be but it should be less than we have now.

Paradoxically, given the original thread, the time when large expressions may work ok is when they have a 'sub-module' like structure, when I can look at a group of lines in isolation and form a view of what it does then move on to the next group and so on, building up an overall picture piece by piece.

Tom Petch

large
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_m
> >> ailman_listinfo_netmod&d=DwICaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u
> >> 4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=V3Xd09IeBbuwHtIPd4YIt9kQBsxSD
> >> K7zldFfmT12OO8&s=Tkq4GWOUmeznETu_UVxLpOom3UXEiI5I2mLe7tpDoZw&e=
>
>

> man_listinfo_netmod&d=DwICaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYiaQ
> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=V3Xd09IeBbuwHtIPd4YIt9kQBsxSDK7zldFfmT
> 12OO8&s=Tkq4GWOUmeznETu_UVxLpOom3UXEiI5I2mLe7tpDoZw&e=
>

---

From: "Ivory, William" <[email-masked]>
Date: Wed, 23 Aug 2017 10:09:27 +0000

Hi Robert,

Probably shouldn't have mentioned packages as that's really a diversion from the main point.  What submodules provide is a way to differentiate YANG nodes in a single namespace so that while still belonging to that single namespace, subsets may be handled by different daemons simply by routing according to submodule id.  That keeps all the identifying logic within YANG.

Obviously if we could start afresh, we'd use separate modules from day one.  However, we have existing YANG that needs to remain to be backwards-compatible, so that isn't an option.

Regards,

William

-----Original Message-----
From: Robert Wilton [mailto:[email-masked]] 
Sent: 23 August 2017 10:46
To: Ivory, William <[email-masked]>; 'Juergen Schoenwaelder' <[email-masked]>
Cc: 'Alex Campbell' <[email-masked]>; [email-masked]' <[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

Hi William,

I think that there might be some downsides to your proposed solution of using submodules - which you may have already considered:

1) All of the debian packages will have to be installed together to allow the YANG module to be built, or otherwise there will be missing submodules, and the module will fail to compile (since the top level module must list all included sub-modules).
2) It might end up with your requiring tight versioning of all of your debian packages together with the same version number, or otherwise you may need greater care over how the sub-modules are updated and dependencies are handled.

If your YANG was being designed from scratch then using separate YANG modules may allow for a cleaner solution - but I appreciate that may not help you with where you are now.

Thanks,
Rob

On 23/08/2017 09:24, Ivory, William wrote:
> From: Juergen Schoenwaelder 
> [mailto:[email-masked]]
> Cc: 'Alex Campbell' <[email-masked]>; 'Robert Wilton' 
>> 3u4ZYiaQ2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU
>> 6 1xSHgPlAT7owI&s=-kR4fUtXArQy0RwWb32DpT1bP4X_cNqt2zJVoC0JiX8&e=
>>
>> Rob
>>
>> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgPl
>> A T7owI&s=t7vGIH8ABuAm00e-bkSowD9eawModGq0N2OkjANtpYI&e=
>>
>> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=esi8GPSc1xVjTt9SKxqzNHRDXT2P1h01a-Ueb
>> n ST-Yo&s=PctKy3ij6W0TQs1NFp18SX8MQtYKeG9RxADh3cphcxU&e=
>>
>> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=RH3zvD2B8_s4uw1PXm_ka37vgz9_q2Rc87tD8
>> K fZ9jA&s=XydU0vXE0AEg2FDE-kx_Ae6rOOAh5koxEeJ2cefgDNA&e=

---

From: Robert Wilton <[email-masked]>
Date: Wed, 23 Aug 2017 11:25:41 +0100

> Obviously if we could start afresh, we'd use separate modules from day one.  However, we have existing YANG that needs to remain to be backwards-compatible, so that isn't an option.
Another potential solution that doesn't use sub modules could be to 
define a bespoke YANG extension to annotate your YANG schema tree with 
any necessary routing information for it to be handled by the 
appropriate daemon.

Thanks,
Rob

---

From: "Ivory, William" <[email-masked]>
Date: Wed, 23 Aug 2017 10:27:17 +0000

Yes, we'll consider that.

Thanks,

William

-----Original Message-----
From: Robert Wilton [mailto:[email-masked]] 
Sent: 23 August 2017 11:26
To: Ivory, William <[email-masked]>; 'Juergen Schoenwaelder' <[email-masked]>
Cc: 'Alex Campbell' <[email-masked]>; [email-masked]' <[email-masked]>
Subject: Re: [netmod] Query about augmenting module from submodule in YANG 1.0

> Obviously if we could start afresh, we'd use separate modules from day one.  However, we have existing YANG that needs to remain to be backwards-compatible, so that isn't an option.
Another potential solution that doesn't use sub modules could be to define a bespoke YANG extension to annotate your YANG schema tree with any necessary routing information for it to be handled by the appropriate daemon.

Thanks,
Rob

> To: Ivory, William <[email-masked]>; 'Juergen 
> Schoenwaelder' <[email-masked]>
>>>> In case YANG 2.0 is ever done, I suggest someone files a proposal 
>>>> to remove submodules if the cost/benefit ratio is at odds. There is 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netm
>>> o
>>> d
>>> -2Dwg_yang-2Dnext_issues_26&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8ky
>>> e
>>> K
>>>> and to avoid integration surprises. But perhaps using m4 scripts 
>>>> and decent version control systems (that can integrate and compile 
>>>> on
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_ma
>>> i
>>> l
>>> man_listinfo_netmod&d=DwICAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYi
>>> a
>>> Q
>>> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=l7c4IPL049A2bVVO14fyBMly211xU61xSHgP
>>> l A T7owI&s=t7vGIH8ABuAm00e-bkSowD9eawModGq0N2OkjANtpYI&e=
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_ma
>>> i
>>> l
>>> man_listinfo_netmod&d=DwIFAw&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYi
>>> a
>>> Q
>>> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=esi8GPSc1xVjTt9SKxqzNHRDXT2P1h01a-Ue
>>> b n ST-Yo&s=PctKy3ij6W0TQs1NFp18SX8MQtYKeG9RxADh3cphcxU&e=
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_ma
>>> i
>>> l
>>> man_listinfo_netmod&d=DwIBAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=p8kyeK3u4ZYi
>>> a
>>> Q
>>> 2ZPGqwkyXmQgBH6r5jpYiYWzhqJ48&m=RH3zvD2B8_s4uw1PXm_ka37vgz9_q2Rc87tD
>>> 8 K fZ9jA&s=XydU0vXE0AEg2FDE-kx_Ae6rOOAh5koxEeJ2cefgDNA&e=

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Aug 2017 12:53:33 +0200

> time in base YANG before anyone noticed - it was just too complex.

Why was it wrong? Just because it was too complex?

> English statement replaces many lines of ABNF, then that is a good

Well, YANG models are also intended to be read by tools that so far
don't understand English statements. Concerning human users, the easiest
thing might be to refer to a corresponding RFC, which the descriptions
already do.

> it does then move on to the next group and so on, building up an overall

Of course, regular expression languages are notorically human
unfriendly, no matter what flavour we take. The ability to build them
step by step from reusable pieces, e.g. using non-terminals, would
certainly help.

Lada

> to
> and

---

From: Vladimir Vassilev <[email-masked]>
Date: Wed, 23 Aug 2017 13:52:13 +0200

On 08/21/2017 05:14 PM, Robert Wilton wrote:

> pattern statements.  I think that it would be much better if these 
> were written to be less exhaustive and much simpler.
>
> Of course, this would allow more invalid values, but most servers 
> would be expected to reject those when it converts them into an 
> internal binary format any way.
>
You still need the 
|(([3-57-9a-fA-F]|[1-9a-fA-F][0-9a-fA-F]{1,3}):[0-9a-fA-F]{1,12}) in the 
end to not reject valid values though.

IMO a pattern statement has value if it absolutely defines the set of 
valid strings. In general I do not see the benefit of pattern statements 
that do not reject all invalid string instances. I prefer the original 
pattern or none at all.

Vladimir

---

From: Robert Wilton <[email-masked]>
Date: Wed, 23 Aug 2017 14:23:12 +0100

On 23/08/2017 12:52, Vladimir Vassilev wrote:
>> The other thing that I think that we have got wrong is modelling 
>> regex pattern statements.  I think that it would be much better if 
>> these were written to be less exhaustive and much simpler.
>>
>> more performant to execute, if the pattern regex was written 
>> something like the following:
>>
>>  pattern:
> |(([3-57-9a-fA-F]|[1-9a-fA-F][0-9a-fA-F]{1,3}):[0-9a-fA-F]{1,12}) in 
> the end to not reject valid values though.
Sure, OK.

> valid strings.
It still has value if it also performs some simple checks and removes 
obvious mistakes.

But even if a value passes the regex filter, it still doesn't guarantee 
that is the value is correct.  Someone could put a typo in there, or 
perhaps configure a multicast IP address where only unicast addresses 
are allowed, or put the same IP address on two separate interfaces, or 
use a IP address that they don't own, etc ...

> In general I do not see the benefit of pattern statements that do not 
> reject all invalid string instances. I prefer the original pattern or 
> none at all.
OK, so some potential counter examples:
1) Email address.  I understand that the full regex to validate all 
email addresses is very complex, but checking that it at least contains 
an @ symbol still has benefit.  It would seem that a short imperfect 
regex is better than a complete perfect regex.
2) A list of VLAN ranges, e.g. want to allow strings that look like 
this: "1-10,20-400,600,2000-3000", but only with non overlapping values 
in ascending order.  It is easy to write a regex to check that the 
structure is right, but AFAIK it is hard (impossible?) to write a regex 
that ensures that the ranges don't overlap and are specified in 
ascending order.

So, I propose that we use regexes for checking that the string is 
structurally correct, but don't use regexes to perform numerical range 
checks of string encoded numbers, since it makes the regexes hard to 
read/verify, and doesn't improve the readability of the YANG file either.

Thanks,
Rob

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 23 Aug 2017 15:36:57 +0200

On Wed, Aug 23, 2017 at 02:23:12PM +0100, Robert Wilton wrote:
> 
> 1) Email address.  I understand that the full regex to validate all email
> addresses is very complex, but checking that it at least contains an @
> symbol still has benefit.  It would seem that a short imperfect regex is
> better than a complete perfect regex.

What is your definition of 'better'? A stricter pattern catches more
errors. An imperfect pattern is better than none.

> 2) A list of VLAN ranges, e.g. want to allow strings that look like this:
> "1-10,20-400,600,2000-3000", but only with non overlapping values in
> ascending order.  It is easy to write a regex to check that the structure is
> right, but AFAIK it is hard (impossible?) to write a regex that ensures that
> the ranges don't overlap and are specified in ascending order.

So what. Does this provide a helpful argument whether patterns should
be strict or imperfect?

> read/verify, and doesn't improve the readability of the YANG file either.

So here is the point I think:

   It is desirable that regexes are as strict as they can be.
   However, if regexes become so complicated that they become a
   verification and maintenance problem by themself, then less strict
   regexes may be a better choice.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Aug 2017 16:22:05 +0200

>    regexes may be a better choice.

Either way, the regex must not produce false negatives, i.e. reject valid
values. For some regexes this is what makes them complicated.

Also, I don't see any need for replacing existing patterns unless they are
wrong. We have descriptions to tell human readers about the permitted value set.

Lada

---

From: "t.petch" <[email-masked]>
Date: Wed, 23 Aug 2017 17:28:49 +0100

----- Original Message -----
From: "Ladislav Lhotka" <[email-masked]>
Sent: Wednesday, August 23, 2017 11:53 AM

> > Sent: Monday, August 21, 2017 4:14 PM
<snip>
> > would

No; it contained a definite error.

This was probably in yang-types and probably around 2012, quite late in
the day, and it stuck in my mind that so many had looked at it and
failed to spot that it was wrong, not just that it did not cater for
some aspects such as interface I-D.  I have used it before as an example
of over complexity

I will have the e-mail filed, along with several thousand other NETMOD
ones so I will find it later rather than sooner.

Tom Petch

> >> >> That would make it very hard to update existing 1.x YANG models
to
> > use
> >> >> From: netmod [mailto:[email-masked]] On Behalf Of
Robert
> > Wilton
> >> >> Subject: Re: [netmod] Query about augmenting module from
submodule
> > in

---

From: Robert Wilton <[email-masked]>
Date: Wed, 23 Aug 2017 17:55:59 +0100

> errors. An imperfect pattern is better than none.
My definition of 'better' is:
- is relatively easy for a human to read/review.
- doesn't exclude any valid values.
- doesn't check numerical ranges, only the number of digits.
- is simple enough to trivially work with most normal regex engines.
- otherwise the pattern is as strict as possible given the constraints 
above.

> So what. Does this provide a helpful argument whether patterns should
My point is only that it is very easy to produce invalid config that 
passes whatever regex is used so it is better to produce a simpler and 
more human readable regex rather than expand it to perform range 
checking of numerical values as well.

>     regexes may be a better choice.
I partly agree, but the line of where I would define a regex as being 
too complicated may be different from you ;-).

Thanks,
Rob

---

From: Robert Wilton <[email-masked]>
Date: Wed, 23 Aug 2017 17:58:41 +0100

I agree.

> wrong.
I also agree.  I'm not trying to retroactively change existing regexes, 
just tweak the guidance when new regexes are being constructed so that 
they end up being a bit simpler.

Thanks,
Rob

>   We have descriptions to tell human readers about the permitted value set.
>
> Lada
>
>> /js
>>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 23 Aug 2017 19:47:35 +0200

> - is relatively easy for a human to read/review.

likely subjective

> - doesn't exclude any valid values.

obviously

> - doesn't check numerical ranges, only the number of digits.

seems arbitrary, why are numerical ranges special?

> - is simple enough to trivially work with most normal regex engines.

and I thought we rely on a standard...

> I partly agree, but the line of where I would define a regex as being too
> complicated may be different from you ;-).

Very likely and as frustrating as it is, there likely is no precise
guideline that does not leave room for interpretation. For me, the
more complex the pattern gets (and my bar for 'complex' is very low),
the more I am interested in test cases (valid data that must be
accepted by the pattern and invalid data that must be rejected by the
pattern). But yes, all this is somewhat subjective.

/js

---

From: Vladimir Vassilev <[email-masked]>
Date: Wed, 23 Aug 2017 20:09:57 +0200

> errors. An imperfect pattern is better than none.
In the case of e-mail address string pattern complying without 
exceptions to rfc5322#section-3.4 specification a ~500 characters of 
pattern expression is needed. There is value in defining data types that 
can be automatically verified by YANG. IMO in this case a strict pattern 
is justified.
> So what. Does this provide a helpful argument whether patterns should
IMO loading strings with semantics instead of using YANG is example of 
bad model design. There is no RFC (like in the e-mail example) that 
mandates use of string with complex format instead of YANG to model 
range configuration. Complex strings can and should be avoided in cases 
like this one.
> c.
I agree. But it is the enthropy of having configuration that can not be 
strictly validated by model aware automation tools that has been the 
reason for introduction of leafref, must, when, range, pattern etc. So 
giving up the goal of handling validation in automated model driven way 
to the advantage of readability is a tough compromise.

I agree with Rob that the pattern in draft-ietf-rtgwg-routing-types-09 
is complicated to read. I have the feeling the design can be improved 
without compromising the strictnes of the data type. Redefinig the 
typedef as a union of 5 string based types with seperate strict patterns 
is one suggestion for moving logic from the string type to YANG.

Vladimir

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Aug 2017 21:45:41 +0200

> ones so I will find it later rather than sooner.

I believe you mean the case when it was realized that "ipv4-address" and "ipv6-
address" permit also zone indices: 

https://www.ietf.org/mail-archive/web/netmod/current/msg07456.html

We can hardly blaim the pattern expression for this though.

Other than that, I am not aware of any reported problem concerning the "ipv6-
address" type. In fact, I even don't think the two ANDed regexs are excessively
complex. Just compare them to ABNF productions proposed for the same purpose in
RFC 3986, Appendix A.

Lada

PS. Yes, it's my child, so I do have a reason to feel offended. :-)

> 
> Tom Petch
> 
> 
> 
> > > 
> > > And ABNF learnt long ago that just because something could be
> 
> expressed
> > don't understand English statements. Concerning human users, the
> 
> easiest
> > > Pragmatically I am not sure what the cutoff for complexity should be
> 
> but
> > > expressions may work ok is when they have a 'sub-module' like
> 
> structure,
> > > when I can look at a group of lines in isolation and form a view of
> 
> what
> > > it does then move on to the next group and so on, building up an
> 
> overall
> > > picture piece by piece.
> > 
> > > > > Given their limited usefulness and the detriments, perhaps we
> 
> should
> > > > > <[email-masked] on behalf of [email-masked]>
> > > 
> > > wrote:
> > > > > 
> > > > > > Hi Rob,
> > > > > > 
> > > > > > new features in YANG 2.x if they used submodules.  Maybe that's
> > > 
> > > something
> > > > > > that no one would ever consider doing anyway, or maybe YANG 1.1
> > > 
> > > already
> > > > > > has similar differences to 1.0?  I had (perhaps naively) assumed
> > > 
> > > that you
> > > > > > > On Wed, Aug 09, 2017 at 05:01:09PM +0200, Ladislav Lhotka
> 
> wrote:
> > > > > > > > I remember that in early stages of YANG there was some
> 
> irrational
> > > > > > > > fear of introducing too many namespaces, and submodules may be
> 
> a
> > > > > > > > consequence of it. As you write, submodules provide no
> 
> benefits
> > > > > > > > whatsoever in terms of modularity, but the overhead in terms
> 
> of
> > > > > > > > metadata, IANA registration etc. is pretty much the same as
> 
> for
> > > > > > > > modules.
> > > > > > > 
> > > > > > > In case YANG 2.0 is ever done, I suggest someone files a
> 
> proposal
> > > to
> > > > > > > remove submodules if the cost/benefit ratio is at odds. There
> 
> is
> > > > > > > nothing wrong with removing stuff that has been found
> 
> problematic.
> > > > > > I agree.
> > > > > > 
> > > > > > I've added
> > > > > > 
> 
> > > > > > > The motivation for submodules was that organizations
> 
> maintaining
> > > large
> > > > > > > modules with multiple people can do so without having to mess
> > > 
> > > around
> > > > > > > with tools like m4 scripts to produce a single module from
> > > 
> > > 'snippets'
> > > > > > > and to avoid integration surprises. But perhaps using m4
> 
> scripts
> > > and
> > > > > > > decent version control systems (that can integrate and compile
> 
> on
> > > > > > > checkin) is indeed cheaper than having submodules part of the
> 
> YANG
> > > > > > > language itself.
> > > > > > > 
> > > > > > > /js
> > > > > > > 
> > > > > > 
> > ----------------------------------------------------------------------
> 
> --
> > > --------
> > > 
> > >

---

From: Per Hedeland <[email-masked]>
Date: Wed, 23 Aug 2017 22:03:32 +0200

On 2017-08-23 20:09, Vladimir Vassilev wrote:
> In the case of e-mail address string pattern complying without exceptions to rfc5322#section-3.4 specification a ~500 characters of pattern expression is needed. There is value in defining data types 
> that can be automatically verified by YANG. IMO in this case a strict pattern is justified.
> IMO loading strings with semantics instead of using YANG is example of bad model design. There is no RFC (like in the e-mail example) that mandates use of string with complex format instead of YANG to 
> model range configuration. Complex strings can and should be avoided in cases like this one.
> I agree. But it is the enthropy of having configuration that can not be strictly validated by model aware automation tools that has been the reason for introduction of leafref, must, when, range, 
> pattern etc. So giving up the goal of handling validation in automated model driven way to the advantage of readability is a tough compromise.

If I may offer a more "philosophical" view: in my mind, a YANG module is
not an instruction for a server (or client) implementation as to what
validation to perform (and I say that with the bias of working for a
company that uses YANG modules precisely for that, and with the specific
components that translate them into such instructions) - it is a
specification of the data model. If the pattern statement allows values
that are actually invalid, the specification is incorrect.

I.e. I'm with your original comment - if you can't get it right, leave
it out. "Under-specification" will always exist, and is acceptable in my
opinion. Specification that gives the appearance of being complete
(admittedly this is also subjective), but actually isn't, is not (in my
opinion).

--Per

> I agree with Rob that the pattern in draft-ietf-rtgwg-routing-types-09 is complicated to read. I have the feeling the design can be improved without compromising the strictnes of the data type. 
> Redefinig the typedef as a union of 5 string based types with seperate strict patterns is one suggestion for moving logic from the string type to YANG.
> 
> Vladimir
>> /js
>>
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Aug 2017 22:04:11 +0200

> they end up being a bit simpler.

This is really subjective, and I am certainly on the side of making the regexes
as strict as possible because they guard not only against typos but also against
intentionally invalid values, i.e. attacks.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 23 Aug 2017 22:37:06 +0200

> (admittedly this is also subjective), but actually isn't, is not (in my

I completely agree.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 24 Aug 2017 09:07:43 +0200 (CEST)

> > > "t.petch" <[email-masked]> writes:

[...]

> PS. Yes, it's my child, so I do have a reason to feel offended. :-)

Note that we had a different pattern to start with.

The new pattern was added 2009-03-27, and it is still the same as in
RFC 6991.

The new pattern was a simplified version using AND; the old version
was a single pattern.

/martin