Subject: [netmod] Filenames, module names and import/include by revision

From: <[email-masked]>
Date: Wed, 22 Feb 2012 10:46:18 -0000

Hi,

I'm relatively new to NETCONF and YANG and am trying to understand the
relationships between filenames, module names and import/include by
revision. I have been digging around for some time looking for guidance
and finally came across the "import-by-revision guidelines"
<http://www.ietf.org/mail-archive/web/netmod/current/msg02128.html>
thread starting 22/02/2009, which stated (with correction for second row
in table):

> I plan to put a section in the YANG usage guidelines draft
> about import-by-revision, but hopefully there will be
> consensus on the details first.
> 
> Of course, standards-track modules MUST use revision dates,
> but there are a lot of considerations for other types of
> YANG modules.
> 
> The following applies to submodules/include as well.
> There are 3 independent boolean variables:
> 
> importer: module with the import-stmt
>   yes:
>     import X { prefix x; revision 2009-01-01; }
>   no:
>     import X { prefix x; }
> 
> 
> imported: module being imported
>   yes:
>     revision 2009-01-01 { description ""; }
>   no:  // no revision-stmts
> 
> filename:
>   yes:
>      X.2009-01-01.yang
>   no:
>      X.yang
> 
>    +---------------------------------+--------+
>    |  importer | imported | filename | result |
>    +-----------+----------+----------+--------+
>    |    no     |   no     |   no     |  (1)   |
>    +-----------+----------+----------+--------+
>    |    no     |   no     |   yes    |  (2)   |
>    +-----------+----------+----------+--------+
>    |    no     |   yes    |   no     |  (1)   |
>    +-----------+----------+----------+--------+
>    |    no     |   yes    |   yes    |  (2)   |
>    +-----------+----------+----------+--------+
>    |    yes    |   no     |   no     |  (3)   |
>    +-----------+----------+----------+--------+
>    |    yes    |   no     |   yes    |  (3)   |
>    +-----------+----------+----------+--------+
>    |    yes    |   yes    |   no     |  (4)   |
>    +-----------+----------+----------+--------+
>    |    yes    |   yes    |   yes    |  (5)   |
>    +-----------+----------+----------+--------+
> 
> 
> (1) Always accepted if any instance of module X found
> (2) Search first for X.yang, which fails;
>     Then search for X.*.yang and return any instance
>     of module X found (very implementation dependent)
> (3) Search fails, requested version not found
> (4) If any X.yang found, it is accepted only if its highest
>     valued revision date matches the 'importer' value.
> (5) If any X.2009-01-01.yang found, it is accepted only
>     if its highest valued revision date matches the
>     'importer' value.

Can someone advise whether this guidance is still valid? It does not
appear to have made it into any of the various updates to the YANG usage
guidelines, which is a pity.

One problem I have with this table is that it doesn't explain the search
order (other than for result 2). Any implementation would have no
understanding of whether available files included the revision in their
name or not. Result 2 implies files without the revision should be
searched for first, though this appears to be at variance with the Mode
1 example on page 32 of the Yuma User Manual (v2.2). Without guidance,
different implementations could have different search orders. Might this
be problematic? If so, it might be better to remove the filename column
from the table and explain the search order in the results. If not, that
too could do with being explained.

Finally, according to RFC6020 Section 5.2, the file SHOULD be of the
form:

module-or-submodule-name ['@' revision-date] ( '.yang' / '.yin' )

Presumably that means references to X.*.yang and X.2009-01-01.yang in
the guidance above should now have the first DOT replaced with an AT
symbol. That also confirms (for me at least) that the import and include
statements always use the module name not the filename (since the
identifier cannot include an AT symbol). This, together with the
recommendation in RFC6020 Section 5.2 replicated above, suggests a
one-to-one mapping between files and modules.

Any advice gratefully received. Thanks,

Jonathan Hansford

This email and any files attached are intended for the addressee and may contain information of a confidential nature. If you are not the intended recipient, be aware that this email was sent to you in error and you should not disclose, distribute, print, copy or make other use of this email or its attachments. Such actions, in fact, may be unlawful. In compliance with the various Regulations and Acts, General Dynamics United Kingdom Limited reserves the right to monitor (and examine for viruses) all emails and email attachments, both inbound and outbound. Email communications and their attachments may not be secure or error- or virus-free and the company does not accept liability or responsibility for such matters or the consequences thereof. General Dynamics United Kingdom Limited, Registered Office: 21 Holborn Viaduct, London EC1A 2DY. Registered in England and Wales No: 1911653.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 22 Feb 2012 09:27:01 -0800

On 02/22/2012 02:46 AM, [email-masked] wrote:
> Hi,
>
> I’m relatively new to NETCONF and YANG and am trying to understand the relationships between filenames, module names and import/include by revision. I have been digging around for some time looking
> for guidance and finally came across the “import-by-revision guidelines” <http://www.ietf.org/mail-archive/web/netmod/current/msg02128.html> thread starting 22/02/2009, which stated (with correction
> for second row in table):
> ....
(thought that text looked familiar :-)
>
> Can someone advise whether this guidance is still valid? It does not appear to have made it into any of the various updates to the YANG usage guidelines, which is a pity.
>

No it didn't make it into RFC 6087 because it wasn't really relevant to
YANG module writers or reviewers. (Just implementors ;-)

> One problem I have with this table is that it doesn’t explain the search order (other than for result 2). Any implementation would have no understanding of whether available files included the
> revision in their name or not. Result 2 implies files without the revision should be searched for first, though this appears to be at variance with the Mode 1 example on page 32 of the Yuma User
> Manual (v2.2). Without guidance, different implementations could have different search orders. Might this be problematic? If so, it might be better to remove the filename column from the table and
> explain the search order in the results. If not, that too could do with being explained.
>

There is no search order that a tool must implement.  There is no RFC
that addresses a collection of YANG modules, other than the tool will
magically figure out how to find the module from the imports/include statements.

I have advocated for a "YANG Library" RFC in the past, but haven't really
written any text (except in yuma documentation).  There are lots of ways
to implement this (e.g., search for all YANG first, then YIN, or search
each subdir in the module path for both? What if you find a submodule
when you're looking for an import? Keep looking? Error out?  Do you
search for all module revisions and use the newest, or use the first
that you find?  etc.)

> Finally, according to RFC6020 Section 5.2, the file SHOULD be of the form:
>
> module-or-submodule-name [’@’ revision-date] ( ’.yang’ / ’.yin’ )
>
> Presumably that means references to X.*.yang and X.2009-01-01.yang in the guidance above should now have the first DOT replaced with an AT symbol. That also confirms (for me at least) that the import
> and include statements always use the module name not the filename (since the identifier cannot include an AT symbol). This, together with the recommendation in RFC6020 Section 5.2 replicated above,
> suggests a one-to-one mapping between files and modules.
>

Yes -- this was changed a long time ago when we realized (duh!) that
legal module name characters cannot be the separator char between fields.

> Any advice gratefully received. Thanks,

Andy

---

From: <[email-masked]>
Date: Wed, 22 Feb 2012 21:52:18 -0000

Unfortunately it appears the YANG Editor plugin for Eclipse
<http://atnog.av.it.pt/~ptavares/yangplugin/>  was implemented without
access to these guidelines as I have downloaded a number of modules from
NETCONF Central, each of which include their revision in the filename
(e.g. [email-masked]) and a revision statement. When the
modules are then imported into other modules (e.g. import iana-if-type
{...}, without a revision statement, as they are in various RFCs) the
YANG Editor cannot find the files.

Might it be worth revisiting the idea of a "YANG Library" RFC? I can see
similar problems occurring as more implementations are developed unless
the import/include guidelines are detailed somewhere.

Jonathan Hansford

---

From: Andy Bierman <[email-masked]>
Date: Wed, 22 Feb 2012 14:01:54 -0800

On 02/22/2012 01:52 PM, [email-masked] wrote:
> Unfortunately it appears the YANG Editor plugin for Eclipse <http://atnog.av.it.pt/~ptavares/yangplugin/> was implemented without access to these guidelines as I have downloaded a number of modules
> from NETCONF Central, each of which include their revision in the filename (e.g. [email-masked]) and a revision statement. When the modules are then imported into other modules (e.g.
> import iana-if-type {…}, without a revision statement, as they are in various RFCs) the YANG Editor cannot find the files.
>
> Might it be worth revisiting the idea of a "YANG Library" RFC? I can see similar problems occurring as more implementations are developed unless the import/include guidelines are detailed somewhere.
>

Perhaps -- if you install yuma the modules will be under /usr/share/yuma/modules
without any dates in the names.  Maybe that will help.

http://sourceforge.net/projects/yuma/

Andy

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 12:07:31 -0000

For product vendors, the import/include search order is an
implementation detail which has little impact. Even if they produce
their own client, its development will probably be closely tied to the
development of the product server, maybe even by the same engineers.

However, for those developing a system management client for use with
multiple vendors' products, any potential for ambiguity leading to
conflict is no mere detail. This is further compounded if the client is
a generic client that can be used to manage different systems using
products, in any combination, from a wide range of vendors. Ambiguity is
the curse of the system integrator and that integration takes place, at
least in part, in the system management client.

RFC3535 "Overview of the 2002 IAB Network Management Workshop", Section
3 "Operator Requirements", requirement 9 states:

... It is desirable to extract, document, and standardize the common
parts of these network wide configuration database schemas.

If the search order threatens the standardisation of configuration
database schemas then isn't one of the underpinnings of the NETCONF/YANG
work threatened? And we then have weaknesses in our solutions that are
similar to some for CLIs:

- ... It is very well possible that the same command on different
devices, even from the same vendor, behaves differently.

- ... can not be used efficiently to automate processes in an
environment with a heterogenous set of devices.

As I said at the start of the thread, I am new to NETCONF and YANG and,
if someone can convince me the search order is irrelevant (rather than
an implementation detail), I will willingly climb down off my soapbox.
Anything for an easy life!

Jonathan Hansford

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 23 Feb 2012 14:21:49 +0100 (CET)

Hi,

<[email-masked]> wrote:
> products, in any combination, from a wide range of vendors.

IMO, how a client finds modules really is an implementation issue.

One client might use the filesystem and a search path.

Some other client might store all modules in some kind of database.

A third client might use the <get-schema> operation defined in RFC
6022 to dynamically download modules from devices, and just store them
internally.

In the last two examples there are no filenames involved.

It is important to remember that if a module uses import/incluce by
revision, there is no amiguity about which version of the (sub)module
it refers to.  However, if a client says that it supports a module A
that imports module B w/o revision, a client needs to check which
version of B the device advertises.

/martin

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 13:35:57 -0000

But should filenames with the revision be searched for before or after
those without? As stated at the start of this thread, Yuma appears to
search in a different order to that suggested in the original
"import-by-revision guidelines" thread. When the import statement
includes a revision statement, as you say, there is no ambiguity. But
when the import statement doesn't include a revision statement (as is
the case with the YANG modules defined in RFCs), do you search all
search paths for all files with and without revisions? And do you take
the first file found, or the file with the latest revision?

Jonathan Hansford

-----Original Message-----
From: Martin Bjorklund [mailto:[email-masked]] 
Sent: 23 February 2012 13:22
To: Jonathan Hansford
Cc: [email-masked]; [email-masked]
Subject: Re: [netmod] Filenames, module names and import/include by
revision

Hi,

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 23 Feb 2012 14:47:54 +0100 (CET)

> the first file found, or the file with the latest revision?

If you know which revision you are looking for, either b/c import by
revision was used, or the server advertises the revision it
implements, then there is no problem, right?  And per 6020, the server
MUST advertise the revision, so you actually know the revision to look
for in your generic client.

/martin

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 13:52:12 -0000

Thanks. I'll climb down off my soapbox for now. I'm almost convinced!

I obviously need to go away and do some more reading.

Jonathan Hansford

-----Original Message-----
From: Martin Bjorklund [mailto:[email-masked]] 
Sent: 23 February 2012 13:48
To: Jonathan Hansford
Cc: [email-masked]; [email-masked]
Subject: Re: [netmod] Filenames, module names and import/include by
revision

/martin

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 14:43:19 -0000

Apologies, but I'm just going to be back up on my soapbox, very briefly.

This thread started, not because I'm implementing a client of server. I
haven't got that far yet. I have been trying to use a free YANG editor
to get up-to-speed. Unfortunately it doesn't find YANG files with the
revision in the filename, whether the import includes a revision
statement or not. 6020 is of no help for a YANG editor as there is no
server to advertise the revision. Obviously the editor's developer was
not aware of the need to search for files both with and without the
revision. 

How can they and other developers become aware of this issue without
some guidance? I only became aware of the original thread from 2009
because I don't give up easily (as you've probably noticed).

Jonathan Hansford

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 23 Feb 2012 15:50:58 +0100

> because I don't give up easily (as you've probably noticed).

It might be reasonable to write some implementation guidelines
document at some point in time. However, import by revision and the
file name layout is defined in RFC 6020 section 5.2.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 23 Feb 2012 15:54:11 +0100 (CET)

> not aware of the need to search for files both with and without the

I think we did what we could in 6020 when we stated:

   YANG modules and submodules are typically stored in files, one module
   or submodule per file.  The name of the file SHOULD be of the form:

   YANG compilers can find imported modules and included submodules via
   this convention.

The reason we did write this is that for MIBs, different tools have
different conventions, for example no file suffix, ".mib", ".my",
".txt".  With YANG we wanted to at least recommend a common file name
convention.

/martin

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 14:56:23 -0000

It might be reasonable to infer from 6020 that an include statement with
a revision statement should look for files with the revision in the
filename. But is it also reasonable to infer that it should look for
files without the revision in the filename? Or to infer that an include
statement without a revision statement should look for files with the
revision in the filename? And those are all inferences that someone
could easily overlook.

Jonathan Hansford

-----Original Message-----
From: Juergen Schoenwaelder
[mailto:[email-masked]] 
Sent: 23 February 2012 14:51
To: Jonathan Hansford
Cc: [email-masked]; [email-masked]
Subject: Re: [netmod] Filenames, module names and import/include by
revision

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 23 Feb 2012 16:04:24 +0100

Jonathan,

if you want to start a draft on implementation guidelines, by all
means, please do so. I am not saying everything is totally obvious
from the specs. (And I guess this is even more true for NETCONF than
it is for YANG.) Having a NETCONF / YANG implementation guidelines
document at some point in time might really be useful.

/js

---

From: <[email-masked]>
Date: Thu, 23 Feb 2012 15:12:49 -0000

As I've said, I'm new to both NETCONF and YANG, and haven't implemented
anything yet. I'd be willing to help, but you'd be waiting a long time
(and get an awful lot of emails from me) before I would even know where
to start, let alone have anything to deliver. And I'm sure there are
other guidelines buried in the NETMOD Discussion Archive that I haven't
unearthed yet.

Jonathan Hansford

-----Original Message-----
Sent: 23 February 2012 15:04

Jonathan,

/js

On Thu, Feb 23, 2012 at 02:56:23PM +0000,
[email-masked] wrote:
> > This thread started, not because I'm implementing a client of
server.
> I
> This email and any files attached are intended for the addressee and
may contain information of a confidential nature. If you are not the
intended recipient, be aware that this email was sent to you in error
and you should not disclose, distribute, print, copy or make other use
of this email or its attachments. Such actions, in fact, may be
unlawful. In compliance with the various Regulations and Acts, General
Dynamics United Kingdom Limited reserves the right to monitor (and
examine for viruses) all emails and email attachments, both inbound and
outbound. Email communications and their attachments may not be secure
or error- or virus-free and the company does not accept liability or
responsibility for such matters or the consequences thereof. General
Dynamics United Kingdom Limited, Registered Office: 21 Holborn Viaduct,
London EC1A 2DY. Registered in England and Wales No: 1911653.

---

From: Andy Bierman <[email-masked]>
Date: Thu, 23 Feb 2012 09:27:35 -0800

> the first file found, or the file with the latest revision?

Sounds like the editor you are using does not support revisions at all.

If the user requests a specific revision, then the revision-date
must be found in the module, and it must match the requested date.

Not sure why RFC 6020 says this is a SHOULD instead of a MUST.
Sec. 5.2 offers no guidance wrt/ why a compliant implementation
can choose not to support it.

Andy

> A third client might use the<get-schema>  operation defined in RFC

---

From: <[email-masked]>
Date: Fri, 24 Feb 2012 09:42:54 -0000

To be fair to the editor, I meant "with and without revisions in the
filename", not with and without revision statements. 

Looking at the code for the emacs and vim editors at YANG Central, they
don't appear to parse import/include statements unlike the editor I've
been testing. It seeks to validate the source file and generates an
error because it doesn't include filenames with revision dates in the
search path.

I guess if I want to go for a full YANG-aware editor I'll have to get
one of the commercial offerings.

Jonathan Hansford

> From: Andy Bierman [mailto:[email-masked]]
> Sent: 23 February 2012 17:28
> To: Jonathan Hansford

---

From: Martin Bjorklund <[email-masked]>
Date: Fri, 24 Feb 2012 11:17:34 +0100 (CET)

> one of the commercial offerings.

If you like emacs, you can combine the yang-mode with flymake,
configured to use pyang or yangdump as the syntax checker.  It doesn't
give you all the bells and whistles that the commercial editors give
you, but it might be useful anyway.

For pyang, put this in your .emacs:

(require 'flymake)

(defun flymake-yang-init ()
  (let* ((temp-file (flymake-init-create-temp-buffer-copy
		     'flymake-create-temp-inplace))
	 (local-file (file-relative-name
		      temp-file
		      (file-name-directory buffer-file-name))))
    (list "pyang" (list local-file))))
; try this to enforce the canonical form
;    (list "pyang" (list "--canonical " local-file))))

(setq flymake-allowed-file-name-masks
      (cons '(".+\\.yang$"
	      flymake-yang-init
	      flymake-simple-cleanup
	      flymake-get-real-file-name)
	    flymake-allowed-file-name-masks))

Make sure you have pyang in your path, fire up emacs, and open a .yang
file.  Hoover over the highlighed lines to see the error / warning
message.  If you try this you will notice that pyang complains about
the flymake-generated filename.  That should be easy to fix.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 24 Feb 2012 13:00:48 +0100

> you, but it might be useful anyway.

Actually, if you don't get sick when seeing XML, you can use emacs + nxml mode as a really full YANG-aware editor, with context-sensitive autocompletion of YANG commands and on-the-fly validation. Of course, the flip side is that you have to write modules in the YIN format. (I do it all the time.)

Just use the RELAX NG schema for YIN which is included in pyang distribution.

Lada
 
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C