Subject: [Netconf] Data Plane notifications within subscribed notifications (was RE: WG adoption poll draft-voit-netconf-notification-messages-01)

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 29 Aug 2017 22:51:17 +0000

Hi Martin,

Moving the discussion to a new thread since as it mostly deals with Q&A related to the subscribed-notifications draft...

It is worth looking back at how the set of work here in NETCONF progressed...  
(1) The original requirements were for yang-push...
(2) ..which the WG requested be generalized to cover the 5277 event-notification control plane   
(3)   ..which then needed to cover the event-notification data plane (for 5277 obsolescence)
(4)     ..which then needed an update to 7950 & 8040 to handle common & extensible message headers

The addition of draft-voit-netconf-notification-messages is the only proposal I know of which begins to fill step (4).  But per Alex' point on a parallel thread, this need not delay yang-push.   And having these all be in a massive bundle of drafts for concurrent RFC publication is unwieldy.   We need to avoid setting dependencies throughout the entire YANG ecosystem just to get yang-push done.

Based on this I am suggesting that we delay the declaration of RFC-5277 obsolescence until we *also* have draft-voit-netconf-notification-messages (or some other adopted RFC) fill the role of next-generation yang-notification-message in the data plane.

Doing this is not so hard, what we would need to do is:

(a) Reframe existing data plane content from Section 6 of subscribed-notifications to temporarily reference only <xs:element name="notification" type="NotificationType"/> from 5277.  Note that this temporary path which can support neither subscription-id, nor augmentable headers.  But this at least completes the control plane definition.

(b) Not declare 5277 obsolete until we have a new, extensible, transport protocol independent yang-notification-message approved which can use the capabilities of the control plane.  
Note: this path will *not* slow yang-push, as different data plane messages are used here.

More in-line...

> From: Martin Bjorklund, August 28, 2017 4:17 AM
> 
> "Eric Voit (evoit)" <[email-masked]> wrote:
> > Hi Martin,
> >
> > Thanks for your thoughts.
> >
> > We had two threads on notifications and draft structure issues in January.
> > (1) nested notifications
> > (2) New Notification and Subscription Features WASFW: 3 Options for
> > Subscription & Event Notification draft structure
> >
> > My read on the outcome was that that there was a desire have a specification
> which would be a standalone enhancement to RFC-7950.  But the document
> could be later incorporated into a YANG 1.2.
> >
> > One dialog between Kent & Andy also focused on the exact issue of this
> thread. See:
> > https://mailarchive.ietf.org/arch/msg/netconf/NXK4eWlOE32vEn_7X0yHC_8u
> > 4Mk
> >
> > Andy has continued his thinking and as you point out has included some of his
> suggested concepts into draft-bierman-netmod-yang-data-ext-00.
> >
> > Your very useful solution proposal below synchs nicely with that
> > thinking.   And an integration of these proposals should be included
> > within this draft.  My suggested next step is to collapse that work
> > into the next version.
> 
> Agreed!  I would like to see this introdcuced before adopting the document.

Attached yang file is what I plan to post with next revision of draft-voit-netconf-notification-messages.  

Reflected in the yang file is the change to rc:yang-data for the definition of notification messages

These are in a single, separate yang file for the following reasons:

a) Segmentation: allow decoupling from control plane capabilities per thread above.  

b) Removes all non-RFC dependencies: i.e., allows the evolution of draft-bierman-netmod-yang-data-ext and extension augment-yang-data independently of this draft.  (Note: We do need Andy's capability to extend the proposed header with additional fields.  But current drafts need not do this augmenting, and can depend fully on existing RFCs.)

c) Simpler: single place to look for the definitions of individual & bundled yang-notification-messages

> > As a side note: we did consider including much of this within
> > draft-ietf-netconf-subscribed-notifications.   But it was not
> > necessary to delay that draft (which focuses on the control plane)
> > with something that fixes gaps in yang 1.1 notifications & data plane.
> > So what we did in section 6 is to define the minimum information which
> > must be transported in a data plane message, and provide two header
> > objects which need to be included in an existing message which would
> > then make the data plane compliant.  This is a minimal way to get
> > people started, and can easily work with existing
> > 5277 data plane parsing.
> 
> But then draft-ietf-netconf-subscribed-notifications
> (+event-notif) cannot obsolete RFC 5277, as it does currently.

I agree we should not assert full RFC 5277 obsolesce until there is a replacement for the one-way notification element of RFC-5277, Section 4.   

> And I don't understand how I can implement draft-ietf-netconf-subscribed-
> notifications with RFC5277, when draft-ietf-netconf-subscribed-notifications
> mandates a "subscription-id" in the header, when no such element is defined in
> RFC 5277.  It seems to me that draft-ietf-netconf-subscribed-notifications really
> requires draft-voit-netconf-notification-messages.

I agree having a standards based data plane definition (perhaps based from the attached yang file) is preferable to a non-standards based answer for the data plane.  But I would rather see a delay in the subscribed notifications dataplane rather than a delay yang-push. 

> I would prefer a complete solution, or rather, that we build from the bottom
> and up, rather than the other way around.

See note above on evolution at the start of this email on how yang-push is driving the closure of parallel questions.

> Note that you're trying to replace several mechanisms:
> 
> 1)  The "notification" message in NETCONF's Message Layer (see section
>     1.2 of RFC 6241), and corresponding in RESTCONF.
> 
>     This is currently defined in Section 2.2 and 4 in RFC 5277
>     (NETCONF) and section 6.4 in RFC 8040 (RESTCONF).
> 
> 2)  How YANG notifications are encoded in XML and JSON.
> 
>     This is currently defined in 7.16 in RFC 7950 (XML) and section
>     6.4 in RFC 8040 (JSON).
> 
> 3)  How notification subscriptions are handled.
> 
>     This is currently defined in section 2.1 and 3 in RFC 5277
>     (NETCONF) and section 6 in RFC 8040 (RESTCONF).
> 
> 
> As a minimum, it must be clear how these updates / replacements are done in
> the new set of documents.

Agree with above.   This is exactly why we need to partition/phase here. 

Eric
 
> 
> 
> /martin
> 
> 
> 
> The rest of section 6 then points to this new work in preparation for the time
> yang notifications are updated (as per the thread above).
> >
> > Eric
> >
> > > -----Original Message-----
> > > From: Martin Bjorklund [mailto:[email-masked]]
> > > Sent: Friday, August 25, 2017 3:12 AM
> > > To: [email-masked]
> > > Cc: Eric Voit (evoit) <[email-masked]>; [email-masked]
> > > Subject: Re: [Netconf] WG adoption poll
> > > draft-voit-netconf-notification-
> > > messages-01
> > >
> > > Hi,
> > >
> > > Alexander Clemm <[email-masked]> wrote:
> > > > Hi,
> > > >
> > > > Couple of thoughts inline, <ALEX>
> > > >
> > > > --- Alex
> > > >
> > > > -----Original Message-----
> > > > From: Netconf [mailto:[email-masked]] On Behalf Of
> > > > Martin Bjorklund
> > > > Sent: Thursday, August 24, 2017 11:44 AM
> > > > To: [email-masked]
> > > > Cc: [email-masked]
> > > > Subject: Re: [Netconf] WG adoption poll
> > > > draft-voit-netconf-notification-messages-01
> > > >
> > > > Eric,
> > > >
> > > > [warning: email quoting messed up below]
> > > >
> > > >
> > > > "Eric Voit (evoit)" <[email-masked]> wrote:
> > > > > From: Andy Bierman, August 24, 2017 12:05 PM
> > > > >
> > > > > On Thu, Aug 24, 2017 at 5:24 AM, Martin Bjorklund <mbj@tail-
> > > f.com<mailto:[email-masked]>> wrote:
> > > > > Hi,
> > > > >
> > > > > I support the problem statement in this draft, but I have
> > > > > concerns with the proposed solution.
> > > > >
> > > > > Currently, the notification message is defined in RFC 5277, for
> > > > > both NETCONF and RESTCONF.  It has a non-exensible header, and I
> > > > > agree that it would be good to fix that.
> > > > >
> > > > > We have a set of notfication-related documents that are supposed
> > > > > to replace RFC 5277, to make it more flexible.  I think that one
> > > > > of the documents in this set (not sure which one) needs to
> > > > > properly define a new notification message, and introduce an
> > > > > extensibility mechanism for the header.
> > > > > <Eric> Draft-ietf-netconf-subscribed-notifications, section
> > > 6<https://tools.ietf.org/html/draft-ietf-netconf-subscribed-notifica
> > > tions- 03#section-6> defines minimal object additions as well as
> > > backwards compliance with YANG 1.1 Notifications (RFC-7950 section
> > > 7.16.3)<https://tools.ietf.org/html/rfc7950#section-7.16.3> .
> > > >
> > > > Actually, you point to one of my main concerns with the set of
> > > > notification
> > > drafts.  The section you refer *should* provide a clear
> > > specification for what the notification message is, and a clear
> > > specification of how the header can be extended.  But it doesn't; it
> > > talks about things that ought to go into a notification, and shows an
> example.
> > > >
> > > > I have similar concerns for how filters and streams are defined.
> > > >
> > > > > It is this backwards compliance part with existing YANG
> > > > > notifications which
> > > is critical as we don’t want to require a new notification element
> > > for the existing event subscription work.
> > > > > draft-voit-netconf-notification-messages is the first attempt at
> > > > > an
> > > extensibility mechanism and format for a new notification message.   It is
> > > because these extensible headers are not backwards compatible that a
> > > separate new draft is being proposed.
> > > > > Then a followup document (like
> > > > > draft-voit-netconf-notification-messages) can use this
> > > > > extensibility mechanism to define more common header fields.
> > > > > <Eric> It seems that a small number of known headers would be
> > > > > useful to
> > > put in the same draft as the extensibility mechanism.  We could
> > > break these into two drafts, but I think that would be more work for a
> reader.
> > > >
> > > > This is also fine with me.  Then I assume that this draft is not
> > > > really needed at
> > > this time?
> > > >
> > > > <ALEX> What I am taking away from this is the request to make the
> > > > definition
> > > of the <notification> message more formal, i.e. explicit.  Sure,
> > > let's look at what edits are required.  What I do not follow is the
> > > conclusion why the draft would not be needed?  Clearly, there are
> > > new elements being defined in draft-voit-
> > > netconf-notification-messages, including the option to bundle multiple
> notification records.
> > >
> > > I read Eric's message above that he preferred to define small set of
> > > headers in the same doc as the extensibility mechanism itself, and
> > > that he didn't want two documents.  Personally I'm fine with two
> > > documents, as long as the base document with the extensibility mechanism
> is clear.
> > >
> > > > </ALEX>
> > > >
> > > > And BTW, the mechanism that this draft "extends" the notification
> > > > header is
> > > by defining a YANG notification.  This simply doesn't work of
> > > course, since we need to define how any YANG notification is encoded
> > > in this new notfication message that the base documents will define.
> > > >
> > > >
> > > > <ALEX> I would consider the fact that we are using YANG a feature,
> > > > not a bug.
> > >
> > > If you misuse a well-defined construct (YANG's notification
> > > statement), how can that be a feature?
> > >
> > > > Now, it is true that really we are dealing with two tiers of
> > > > information - the
> > > notification record itself, and the construct that it is contained /
> > > encapsulated in (the bundle/housekeeping stuff).  And yes, an
> > > independently defined notification should be transported
> > > "encapsulated" within the new header we are defining, not separately from
> it.
> > > >
> > > > Really what we should be doing is update RFC 7950.  I think I
> > > > vaguely
> > > remember an earlier discussion thread that stated we could state in
> > > a new RFC
> > > "Updates: RFC 7950", with text that explains how YANG-defined
> > > notifications are encoded.  This is probably what we need to do here.
> > >
> > > Agreed.  BUT, it needs to be done properly.  Here's a quick sketch
> > > of a solution
> > > proposal:
> > >
> > > In the base document, define the new message layout / the
> > > encapsulation of notifications.  This needs to be in a new XML
> > > namespace in order to differentiate from RFC 5277.  Use yang-data for this
> purpose:
> > >
> > >   module ietf-yang-notification-message {
> > >     namespace "urn:ietf:params:xml:ns:netconf:notification:2.0";
> > >               // or maybe even a new namespace w/o 'netconf' in it
> > >     prefix ynm;
> > >
> > >     import ietf-restconf {
> > >       prefix rc;
> > >     }
> > >
> > >     rc:yang-data notification {
> > >       container header {
> > >         leaf event-time { ... }
> > >         leaf another-common-leaf { ... }
> > >       }
> > >       // notification content goes here, or possibly:
> > >       //  container data { // ... here }
> > >     }
> > >   }
> > >
> > > The base document should update RFC 7950 and state that all YANG
> > > defined notifications are encapsulated in this new structure,
> > > regardless of protocol.  It also needs to update RFC 8040 which currently
> says that RFC 5277 is used.
> > >
> > >
> > > To extend this, use Andy's proposed yang-data-augment, and make use
> > > of features and modules to avoid the kitchen-sink syndrome:
> > >
> > >   module notifi-header-extensions-for-foobar {
> > >     ...
> > >     import ietf-yang-notification-message {
> > >       prefix ynm;
> > >     }
> > >     import yang-data-ext {
> > >       prefix ydx;
> > >     }
> > >
> > >     feature foobar-extra { ... }
> > >
> > >     ydx:augment-yang-data /ynm:notification/ynm:header {
> > >       leaf foobar-id { ...}
> > >       leaf foobar-extra {
> > >         if-feature foobar-extra;
> > >         ...
> > >       }
> > >       ...
> > >     }
> > >   }
> > >
> > > Details about bundling messages TDB in the base spec.
> > >
> > >
> > > /martin
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > </ALEX>
> > > >
> > > >
> > > > > On another level, I'm also worried that this becomes a
> > > > > kitchen-sink of various headers that may or may not useful.  I
> > > > > would prefer an initially small set of well thought-through
> > > > > headers, and work from there.
> > > > > <Eric> When developing v01, it seemed to me more useful to
> > > > > include
> > > potential header objects which could later be removed.  The real intention
> in
> > > their inclusion is to spur others’ thinking.   I have zero issue in dropping or
> > > morphing proposed headers during draft development.     (Note that it was
> > > entries like observation-domain-id which allowed Benoit to make a
> > > 1:1 conceptual mapping with corresponding concepts in IPFIX at IETF
> > > 98.)
> > > > >
> > > > > I would like to see these concerns addressed before the WG
> > > > > adopts this document.
> > > > >
> > > > > I agree these issues need to be addressed.
> > > > >
> > > > > <Eric> Hopefully the thoughts above sufficient for now.   Any alternative
> > > approaches/proposals would be great to hear.
> > > >
> > > > My proposal would be to make sure the base is solid before adding
> > > extensions.
> > > >
> > > >
> > > > /martin
> > > >
> > > >
> > > >
> > > > >
> > > > > Many other protocols have these properties:
> > > > >   1) ability to add headers over time in separate RFCs
> > > > >   2) ability to identify mandatory vs. optional property (even
> > > > > if nothing else
> > > understood)
> > > > >   3) ability to skip over unknown optional headers
> > > > >
> > > > > <Eric> These three are viable in the current draft.
> > > > >
> > > > > Eric
> > > > >
> > > > >
> > > > > /martin
> > > > >
> > > > > Andy
> > > > >
> > > > >
> > > > > Kent Watsen <[email-masked]<mailto:[email-masked]>>
> wrote:
> > > > > >
> > > > > > All,
> > > > > >
> > > > > > This is start of a two-week poll on making the following draft
> > > > > > a NETCONF working group document:
> > > > > >
> > > > > >   draft-voit-netconf-notification-messages-01 [1]
> > > > > >
> > > > > > Please send email to the list indicating "yes/support" or
> > > > > > "no/do not support".  If indicating no, please state your
> > > > > > reservations with the document.  If yes, please also feel free
> > > > > > to provide comments you'd like to see addressed once the
> > > > > > document is a WG
> > > document.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > > https://tools.ietf.org/id/draft-voit-netconf-notification-mess
> > > > > > ages
> > > > > > -0
> > > > > > 1
> > > > > >
> > > > > >
> > > > > > Thank you,
> > > > > > Kent (and Mahesh)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Netconf mailing list
> > > > > > [email-masked]<mailto:[email-masked]>
> > > > > > https://www.ietf.org/mailman/listinfo/netconf
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Netconf mailing list
> > > > > [email-masked]<mailto:[email-masked]>
> > > > > https://www.ietf.org/mailman/listinfo/netconf
> > > > >
> > > > <
> > > > _______________________________________________
> > > > Netconf mailing list
> > > > [email-masked]
> > > > https://www.ietf.org/mailman/listinfo/netconf