Subject: Re: [Netconf] zerotouch/19: How to encode YANG-modeled artifacts?

From: Kent Watsen <[email-masked]>
Date: Fri, 13 Jan 2017 18:00:29 +0000

Somewhat answering my own question, but by no means coming to a complete solution, I realized that RESTCONF does something similar with its yang-data extension and YANG data template:

   o  yang-data extension: A YANG external statement that conforms to
      the "yang-data" extension statement found in Section 8.  The yang-
      data extension is used to define YANG data structures that are
      meant to be used as YANG data templates.  These data structures
      are not intended to be implemented as part of a configuration
      datastore or as operational state within the server, so normal
      YANG data definition statements cannot be used.

   o  YANG data template: a schema for modeling protocol message
      components as conceptual data structure using YANG.  This allows
      the messages to be defined in an encoding-independent manner.
      Each YANG data template is defined with the "yang-data" extension,
      found in Section 8.  Representations of instances conforming to a
      particular YANG data template can be defined for YANG.  The XML
      representation is defined in YANG version 1.1 [RFC7950], and
      supported with the "application/yang-data+xml" media type.  The
      JSON representation is defined in JSON Encoding of Data Modeled
      with YANG [RFC7951], and supported with the "application/
      yang-data+json" media type.

This looks good on the surface, but:

  1) there shouldn’t be a need to reference the RESTCONF RFC

  2) such definitions should be available outside RESTCONF protocol
     operations (currently they seem bound to when using the 
     "application/yang-data+xml" media type)

  3) I didn’t know that extensions can be used to define instance
     data, as I’ve always only seen them to define YANG statements,
     with no instance encoding.  Is the “yang-data extension” term
     above defining this behavior, above and beyond what’s provided
     by RFC 7950?

  4) this seems like an overly complicated way to achieve what I
     believe should be simple thing.

Thoughts?

Kent

---

From: Andy Bierman <[email-masked]>
Date: Fri, 13 Jan 2017 10:34:01 -0800

On Fri, Jan 13, 2017 at 10:00 AM, Kent Watsen <[email-masked]> wrote:

>
> Somewhat answering my own question, but by no means coming to a complete
> solution, I realized that RESTCONF does something similar with its
> yang-data extension and YANG data template:
>

We had a RESTCONF issue on a similar topic
https://github.com/netconf-wg/restconf/issues/68

I raised the issue; nobody cared; we didn't change anything;
Just reference RESTCONF was the WG consensus.

> Kent

Andy

>
>
>
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 13 Jan 2017 20:41:48 +0100

> > yang-data extension and YANG data template:

[...]

> Just reference RESTCONF was the WG consensus.

And hopefully some future version of YANG provides proper support for
this (and specifications that use the RESTCONF workaround can ideally
eventually retire this dependency on RESTCONF in the future).

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 16 Jan 2017 13:35:12 +0100

>      believe should be simple thing.

This would be another merit of my proposal to remove any dependencies on
datastores and protocol(s) from YANG spec: it should only deal with
deciding whether a given data tree is valid (perhaps for several levels
of validity, e.g. grammar + datatypes, semantic rules, references). The
notion of a (conceptual) data tree is already implicitly present in most
of the 7950 text, and everything else can IMO be specified in other
documents.

If we don't do this, then I believe we will see YANG being "misused"
again and again as in your examples. This just undermines the value of
the YANG standard.

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 16 Jan 2017 14:14:05 +0100 (CET)

Kent Watsen <[email-masked]> wrote:
>      data

I'm not sure I understand what you mean.  The "yang-data" extension
doesn't define instance data.  It defines the *schema* for instance
data.

> , as I’ve always only seen them to define YANG statements,
>      with no instance encoding.

There is nothing in RFC 7950 that would make this illegal.

> Is the “yang-data extension” term

Yes, I would like to see a new core statement "structure" or something
similar, which could be used to define the schema for instances that
are not tied to any datastores, operations, or notifications.  The
semantics of these structures must be defined in the description
statement.  The core statement "augment" should be extended so that it
can be used to augment such structures.

But this requires a new version of YANG.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Tue, 17 Jan 2017 20:24:47 +0000

> Yes, I would like to see a new core statement "structure" or 
> something similar, which could be used to define the schema 
> for instances that are not tied to any datastores, operations,
> or notifications.  The semantics of these structures must be
> defined in the description statement.  The core statement 
> "augment" should be extended so that it can be used to augment
> such structures.

I’m not sure if a core statement is best, but this can be hashed
out later...

> But this requires a new version of YANG.

Indeed, and thus I just added an entry for YANG-next:
  https://github.com/netmod-wg/yang-next/issues/8

But the zerotouch draft can’t wait that long, what should it 
do in the meanwhile?   Would it be acceptable to state in the
draft and also in the YANG module itself that the YANG module
MUST NOT ever be implemented by a NC/RC server?

Thanks,
Kent

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 Jan 2017 13:50:27 -0800

> MUST NOT ever be implemented by a NC/RC server?

The term MUST NOT has requirements for use which are not met here.
IMO it is sufficient to say the module is not intended for implementation
in a NC/RC server.

If the data-structure extension is used then the module looks empty to a
NC/RC server
so not sure why this is even an issue.

Andy

---

From: Kent Watsen <[email-masked]>
Date: Tue, 17 Jan 2017 23:01:06 +0000

On 1/17/17, 4:50 PM, "Andy Bierman" <[email-masked]> wrote:

> The term MUST NOT has requirements for use which are not met 
> here. IMO it is sufficient to say the module is not intended
> for implementation in a NC/RC server.

Okay

> If the data-structure extension is used then the module looks
> empty to a NC/RC server so not sure why this is even an issue.

Because the data-structure extension (assuming we finalize on
that) doesn’t exist yet, so I need (I guess) an interim solution.

Another option is to “quickly” run a draft through the NETMOD WG
to “update” RFC 7950.  We could try, but no matter how fast we go,
it would surely delay the zerotouch some...

Kent

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 18 Jan 2017 09:53:14 +0100

> On 17 Jan 2017, at 22:50, Andy Bierman <[email-masked]> wrote:
> 
> 
> 

It depends on the purpose that the module is used for. It doesn't work very well if it is intended to be normative ("some data MUST be valid according to this module") because validity, XPath context and other things are defined in terms of specific datastores. It is also often impossible to decide whether it describes configuration or state data (some YANG rules depend on it).

So using YANG in this way is not really supported by the spec and it is not guaranteed that everybody extrapolates the rules in the same way. Creativity in interpretation of standards IMO isn't very good.

Lada

> 
> If the data-structure extension is used then the module looks empty to a NC/RC server

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 18 Jan 2017 16:30:08 +0100 (CET)

> > I’m not sure if a core statement is best, but this can be hashed

If it is not a core statement, "augment" won't work.  But of course,
we can define extensions "structure" and "augment-structure".

> > > But this requires a new version of YANG.

Yes.

> It depends on the purpose that the module is used for. It doesn't work
> very well if it is intended to be normative ("some data MUST be valid
> according to this module") because validity, XPath context and other
> things are defined in terms of specific datastores. It is also often
> impossible to decide whether it describes configuration or state data
> (some YANG rules depend on it).
> 
> So using YANG in this way is not really supported by the spec and it
> is not guaranteed that everybody extrapolates the rules in the same
> way. Creativity in interpretation of standards IMO isn't very good.

+1

Two alternatives:

  o  Define yet another one-off solution for zerotouch, i.e., an
     extension called  'zerotouch-artifact' or whatever.

  o  Define a generic extension "structure" in a separate draft,
     maybe with an "augment-structure" extension.  Document that
     this is intended as an interim solution until YANG is updated.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Wed, 18 Jan 2017 18:27:04 +0000

Hi Martin,

>>> Andy writes: 
>>> IMO it is sufficient to say the module is not intended for
>>> implementation in a NC/RC server.
>> 
>> Lada writes:
> Martin writes:

When you say “alternatives”, are these alternatives to Andy’s 
suggestion above as well (e.g., a simple note).  Yes, Lada’s
comments are valid, but they’re equally valid even when using
your two suggestions too, right?

Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 18 Jan 2017 20:19:28 +0100

> your two suggestions too, right?

What is wrong with reusing the RESTCONF hack?

/js

---

From: Kent Watsen <[email-masked]>
Date: Wed, 18 Jan 2017 19:52:00 +0000

Hi Juergen,

> What is wrong with reusing the RESTCONF hack?

1) copy/paste or reference?  if reference, then:

   a) it’s not too bad for the zerotouch draft, as it already
      has a normative reference to the RESTCONF draft.

   b) but it’s not good for the anima-voucher draft, as it 
      currently has no reference to the RESTCONF draft.

2) it appears that the definitions in the RESTCONF draft
   are bound to when using the "application/yang-data+xml"
   media type.  Even if not technically true, it’s still
   going to confuse some people.

3) I’m not sure if the RESTCONF solution is even valid.
   That is, I’ve never known extensions to be used to define
   the schema for instance data; I’ve only seen them to
   define YANG statements.

Thanks,
Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 18 Jan 2017 21:46:33 +0100

>       currently has no reference to the RESTCONF draft.

So what? There is always going to be a references, unless you define
something yourself to reinvent the wheel.
 
>    going to confuse some people.

I do not see this in the description of the yang-data extension
definition.

>    the schema for instance data; I’ve only seen them to

If this is true, you have to stop and wait until YANG has built-in
support for this.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 18 Jan 2017 22:41:44 +0100 (CET)

Kent Watsen <[email-masked]> wrote:
> suggestion above as well (e.g., a simple note).

No, sorry if I wasn't clear.  My +1 was to Lada's statement that a
simple note is not really supported.  I propose two alternatives
instead.

> Yes, Lada’s

No.  There are no restrictions on the semantics of an extension
statement specified in RFC 7950.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 18 Jan 2017 22:50:35 +0100 (CET)

> What is wrong with reusing the RESTCONF hack?

Yes this is also a possibility.  It just happens to be defined in a
somewhat weird place...

I had to re-read the definiiton of rc:yang-data, I thought it was more
dependent on restconf than it really is.  In fact, except for one
sentence (*), it is very generic and can be used as-is for the
zerotouch artifacts.

(*) It says:

  A specification using this extension MUST specify the
  message encoding rules, including the content media type.

It seems this means that a content media type is *required* for *every*
usage of this extension.  Maybe we should have written:

  A specification using this extension MUST specify the
  message encoding rules, including the content media type, if one is
  required.

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 18 Jan 2017 23:29:00 +0100

>   message encoding rules, including the content media type, if one is

I think zerotouch somewhere needs to specify the 'message encoding
rules'. In case zerotouch supports multiple encodings, it needs to
specify a way (aka 'media type') to distinguish encodings or in case
there is only one encoding allowed, the specification needs to say
that the content media type is a constant.

Perhaps the wording is not optimal, but I do not consider the wording
problematic enough to justify a copy and paste action with a subtle
wording change. I am concerned about having to deal with N variations
of this in the future.

/js

---

From: Kent Watsen <[email-masked]>
Date: Wed, 18 Jan 2017 22:52:20 +0000

Martin, Juergen,

> wording change. I am concerned about having to deal with N variations

Okay, I’ll use/ref the RC draft.  I’ll also be sure to add a note for
why it is knowingly awkward (per the discussion above).

FWIW, the zerotouch draft does allow multiple encodings, for all its
sources of bootstrapping data (e.g., removable storage, DNS server,
DHCP server, bootstrap server, etc.) but, in each of these cases, it
also defines how the encoding is identified.  So, in a way, it does
support “content media type”, just not *HTTP* media types.  As an 
egregious example, the removable storage solution says:

      Information Type:  Mapped to a file containing a standard YANG
         encoding for the YANG modeled data described in Section 4.1.  A
         filenaming convention SHOULD be used to indicate data encoding
         (e.g., boot-info.[xml|json]).

Other than maybe it should be a MUST, do you think something like this
is in the spirit of what’s being asked for?

Thanks,
Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 19 Jan 2017 00:05:12 +0100

> is in the spirit of what’s being asked for?

I think it carries the idea. But then the wording is somewhat vague or
soft (a convention SHOULD be used followed by 'e.g.' with an example
leaves some room for interpretation) - can I come up with my own
convention for XML and JSON encoded content as I see fit?

/js

---

From: Kent Watsen <[email-masked]>
Date: Thu, 19 Jan 2017 00:51:19 +0000

Hi Juergen,

> convention for XML and JSON encoded content as I see fit?

Understood, and we can tighten up the language some.  The main takeaway is that the gist of this is okay, given the specific language in the RESTCONF draft.  Thanks for the confirmation!

Kent

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 19 Jan 2017 08:44:01 +0100 (CET)

> wording change. I am concerned about having to deal with N variations

I fully agree with you.

Until we have something similar built-in (if ever), rc:yang-data
should be used.

/martin