Subject: [netmod] smi2yang containment hierarchy

From: David Reid <[email-masked]>
Date: Thu, 05 Jan 2012 17:20:51 -0500

I'm looking at implementing the current smi to yang conversion rules in
the netconf server. More specifically, I'm trying to use the converted 
yang module as the data model for netconf, snmp, cli, web, etc. I'm a 
little concerned about the current containment hierarchy. Maybe this isn't 
a problem, but let me try to explain my concern.

I know that our charter only specifies a data modeling language for netconf,
but I think there is a vision to use yang for more than just netconf and I 
know that this is happening in practice. What I would like to see in the 
future are documents kind of like RFC 3781 (SMIng mappings to SNMP) to 
define YANG mappings other protocols.  One such mapping I would like to see 
is YANG to an EOS-like (Evolution of SNMP) version of SNMP. It might also 
include a mapping from a subset of yang to SNMPv3, allowing someone to write 
the model in yang (using the defined subset) and produce a MIB for SNMPv3
retrieval in addition to netconf. I'm not suggesting we do that now, but 
the current smi to yang mapping provides a very good foundation for that 
type of document. The only problem I see is that if the yang containment 
in the converted module conflicts with the OID hierarchy, a yang mapping to 
SNMP will be more difficult, and I don't want to do anything now that would 
make such a mapping document difficult or impossible in the future.

It seems to me that allowing the yang containment to not match the OID 
hierarchy will make a yang mapping to SNMP more difficult. I'm not sure the 
best solution. It seems like we would need to flatten scalars (like the -01 
draft) and remove the top level container. The problem with the top-level
container is that it has no OID and thus does not fit in the SNMP world. The
unflattened scalars make containers like "interfaces" contain different data
depending on if it is retrieved in netconf or retrieved in SNMP.

Of course I have all the OIDs in the converted yang module, so I can extract
what I need for SNMP. But it seems wrong to have the data organized differently
if retrieved through SNMP than through netconf. And I'm not sure which way 
to organize it when retrieved through the web interface or cli.

I'm not sure if I stated my concern clearly, but I'm curious as to what some 
of the rest of you think.

-David Reid

---

From: Andy Bierman <[email-masked]>
Date: Thu, 05 Jan 2012 14:47:15 -0800

On 01/05/2012 02:20 PM, David Reid wrote:
> I'm looking at implementing

That's all I had to read to decide you're probably right.

the current smi to yang conversion rules in
> a problem, but let me try to explain my concern.

Preaching to the choir here.

The vision has always been to decouple each of the layers (transport, operation, and content).

It should be possible for you to write a new RFC to support new protocols.

But I don't agree the SMI mapping needs to be flattened.

The same problem exists in NETCONF tools.
The yuma server solves this problem with the 'ncx:root' extension.

    container config {
      ncx:root;
    }

For XML, XPath and other processing, this serves as the root node '/'.
Is the problem that the valid YANG path '/' has no translation in SMIv2?
(If so, make one up.)

I don't like arbitrary containment as a general rule.
I prefer to have actual SMIv2 siblings to be YANG siblings as well,
but there should be 1 top-level container for the module, plus 0 or more
notifications at the top level.

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>
>

---

From: Andy Bierman <[email-masked]>
Date: Thu, 05 Jan 2012 15:18:01 -0800

> a problem, but let me try to explain my concern.

I guess I am agreeing with you that an extra container for scalars should be removed.
It just makes tools harder to implement and use because it creates a special corner-case
when there is no need for one.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 6 Jan 2012 08:32:04 +0100

> of the rest of you think.

I think I do not understand your message. 

- Stuff that does not have an OID does not exist in the SNMP
  world. Where is the problem?

- We put scalars into containers for a specific reason. Is this not
  true anymore? And why is this problematic?

- The OID tree and the NETCONF containment tree are different things.
  Just look at how augmentations work. For example, the ifXTable sits
  somewhere pretty unrelated to the ifTable in the OID tree. Has this
  ever been a serious problem?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Fri, 06 Jan 2012 03:33:52 -0800

On 01/05/2012 11:32 PM, Juergen Schoenwaelder wrote:
...
>    true anymore? And why is this problematic?

This text on page 8 has huge implications:

       A top-level YANG container is generated.  The container's name is
       the SMIv2 module name and the container MUST be config false.  The
       generation of the top-level container MAY be skipped if the SMIv2
       module does not define any objects that go into the top-level
       container (e.g., an SMIv2 module only defining textual
       conventions).

It seems that there MUST be a top-level container, and it MUST be config true.
Otherwise it will be too difficult to use the translations, and they can't even be
augmented in the future with config objects.

I do not agree with you that we can simply generate config true and ignore
RowStatus, StorageType, read-write MAX-ACCESS, writable OID nodes, etc.
But blinding forcing all OBJECT-TYPE macros to be read-only is not going to
be compatible with anything. (Consider how useful a converted RMON-MIB would
be if all the control tables were read-only, and the app could not create
any alarms/events, host, matrix, or other collections.)

As for scalars:

As long as the translation algorithm is deterministic, tools can deal with it.
IMO, one top-level container is OK, and won't be as optimal as hand-converted,
but easy to remember and construct XPath:

      augment "/if-mib:IF-MIB/if-mib:ifTable/if-mib:ifEntry" { ... }

      leaf dumb-example {
         type leafref { path "/if-mib:IF-MIB/if-mib:ifNumber"; }
      }

I prefer to have all the top-level MIB objects be child nodes of the NP container
according to the module name.  In the MIB module, ifNumber and ifTable are siblings,
so they should be at the same level in the YANG translation.

I don't see the value of preserving fooTable.fooEntry everywhere either.
The translated objects could really boil down to a sequence of leaf or list statements.

module if-mib {

   container IF-MIB {
       leaf ifNumber { config false; ... }
       list ifEntry { ... }
   }
   notification linkUp { ... }
   notification linkDown { ... }

}

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 06 Jan 2012 03:58:14 -0800

....

3 more comments:

1) Why not just use exact-case, and the same string for module name and top-level container?

OLD:

    module if-mib {
       container IF-MIB {}
    }

NEW:

    module IF-MIB {
       container IF-MIB {}
    }

2) We should think very carefully about how SMIv2-converted YANG is going to
    coexist with "new YANG", either vendor or SDO created.  IMO, we don't
    have a good solution for integrated data organization yet.

3) We cannot rely on assumed translated output as the canonical standard
    YANG version of an SMIv2 module, if the translation is non-deterministic and
    implementations are free to choose important details like config true/false.
    We want to be able to reuse standard MIB modules without writing new RFCs for
    each one.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 6 Jan 2012 14:14:11 +0100

>   notification linkDown { ... }

Having all scalars flat was considered to be a bad idea. Please
re-read the discussions we had about this. 

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 6 Jan 2012 14:18:35 +0100

> 1) Why not just use exact-case, and the same string for module name and top-level container?

Which problem is being solved?
 
>    have a good solution for integrated data organization yet.

How do we determine that we are done thinking carefully?

> 3) We cannot rely on assumed translated output as the canonical
>    standard YANG version of an SMIv2 module, if the translation is
>    non-deterministic and implementations are free to choose
>    important details like config true/false.  We want to be able to
>    reuse standard MIB modules without writing new RFCs for each one.

The standard says it is read-only. As long as you assume just that, we
are fine. And if an implementation implements something as a config
object, it should announce this with proper deviations.

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 06 Jan 2012 06:49:43 -0800

> Which problem is being solved?

IMO this is simpler -- less special cases.

> How do we determine that we are done thinking carefully?

I guess we just see if anybody uses converted SMI modules for anything.
Those that really care about SNMP and NETCONF integration will figure it out.

> object, it should announce this with proper deviations.

OK -- there aren't really enough writable SMIv2 objects to make that
set of problems worth solving.  People should use new YANG modules if any writable objects
are needed.

Andy

---

From: David Reid <[email-masked]>
Date: Fri, 06 Jan 2012 09:51:40 -0500

>   ever been a serious problem?

Augments has been one of the biggest pains to implement simply because of the
difference in containment. (There is also a small performance hit, but I think
it's negligible). I guess that's my problem with unflattened scalars.
I implemented scalars based on the 01 draft and it was very easy. Now I'm 
looking to update to the 03 draft and it's going to be more challenging.
Maybe implementation cost shouldn't be a big concern. It's not unimplementable,
just harder.

I think we put scalars into containers so that scalars in yang modules would
have the same logical grouping that the MIB module has, which is a reasonable
goal. But it conflicts with the desire to have the yang containment match the
OID tree.

-David Reid

---

From: David Reid <[email-masked]>
Date: Fri, 06 Jan 2012 10:04:57 -0500

> IMO this is simpler -- less special cases.

I'm confused. The current draft specifies the "NEW:" case above, correct?
What is the proposed change?

-David Reid

---

From: Andy Bierman <[email-masked]>
Date: Fri, 06 Jan 2012 09:09:35 -0800

> What is the proposed change?

None I guess.
I was confused too -- the prefixes are still lower-case, even though the module name is now
upper-case. I suppose forcing lower-case prefixes is some sort of feature in somebody's mind.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 06 Jan 2012 21:30:23 +0100 (CET)

Andy,

Andy Bierman <[email-masked]> wrote:
> I suppose forcing lower-case prefixes is some sort of
> feature in somebody's mind. 

What is this forcing of lower case prefix you are talking about?

If you read the latest document (-03), you will see that an
implementation is free to pick any prefix it wants.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 10 Jan 2012 15:31:57 +0100

On Fri, Jan 06, 2012 at 09:09:35AM -0800, Andy Bierman wrote:

> I was confused too -- the prefixes are still lower-case, even though
> the module name is now upper-case. I suppose forcing lower-case
> prefixes is some sort of feature in somebody's mind.

The prefixes are not forced to be lower-case anymore in the current
I-D.

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 10 Jan 2012 15:35:44 +0100

On Fri, Jan 06, 2012 at 09:51:40AM -0500, David Reid wrote:
> 
> Augments has been one of the biggest pains to implement simply
> because of the difference in containment. (There is also a small
> performance hit, but I think it's negligible). I guess that's my
> problem with unflattened scalars.  I implemented scalars based on
> the 01 draft and it was very easy. Now I'm looking to update to the
> 03 draft and it's going to be more challenging.  Maybe
> implementation cost shouldn't be a big concern. It's not
> unimplementable, just harder.
> 
> I think we put scalars into containers so that scalars in yang
> modules would have the same logical grouping that the MIB module
> has, which is a reasonable goal. 

And somehow I felt the WG was willing to pay the implementation costs
for this instead of facing some long lists of collections of scalars.

> But it conflicts with the desire to have the yang containment match
> the OID tree.

This is where I stop following you. The scalars are not put in
arbitrary containers - the containers are derived from the
registration tree.

/js

---

From: David Reid <[email-masked]>
Date: Tue, 10 Jan 2012 17:23:24 -0500

> arbitrary containers - the containers are derived from the

An SNMP subtree walk on interfaces would return the scalars and ifTable. A
netconf get on interfaces would only return the scalars.

-David Reid

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 11 Jan 2012 08:47:42 +0100

> netconf get on interfaces would only return the scalars.

OK, now I see what you mean. The question is whether this matters.

SNMP walks are in general not the same as NETCONF gets. And as
mentioned earlier, ifTable and ifXTable are quite far away in the SNMP
OID tree while the YANG augment puts things together. With the YANG
models derived from SNMP MIB modules, you actually get all interface
augments when you retrieve ifTable unless you put specific filters in
place.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 11 Jan 2012 15:58:38 +0100 (CET)

> augments when you retrieve ifTable unless you put specific filters in

I prefer to keep things as they are.  Incidentally, the result of all
this moving scalars around is that the current draft exactly matches
the structure we've been using the last five years :)

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 11 Jan 2012 16:37:35 +0100

> the structure we've been using the last five years :)

Just in case this has not been clear, I also prefer to stop moving
things around since I feel we go in circles.

/js

---

From: David Reid <[email-masked]>
Date: Wed, 11 Jan 2012 15:31:24 -0500

> augments when you retrieve ifTable unless you put specific filters in

We had to add special hacks to handle augments since SNMP and NETCONF are so
different in this respect. We'll have to do similar things in this case if the
SNMP and NETCONF containment is different. It can be done, it's just more work.
I seem to be in the minority on this, so I'm OK leaving it as it is.

-David Reid