Subject: [Netconf] Subscribed-notifications, replay objects

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 11 Jul 2017 21:55:25 +0000

In an email earlier today, Martin pointed out replay was under-defined.  To fix this, below is suggested text and yang objects.  The goal of this is keep this as close to RFC-5277 as possible.

So following would replace the text in Section 4.1.1

Replay provides the ability to establish an event subscription which is also capable of passing along recently generated events.  In other words, as the subscription initializes itself, it sends any previously generated notifications for the target event stream which meet the filter and timeframe criteria.  These historical events would then be followed by events generated after the subscription has been established.  All events will be delivered in the order generated.  Replay is only viable for dynamic subscriptions.  Replay is an optional feature.   Replay is dependent on a notification stream supporting some form of notification logging, although it puts no restrictions on the size or form of the log, or where it resides within the device.

The presence of a replay-start-time within an establish-subscription RPC is the way a replay may be been requested.  The provided start time must be earlier than the current time.  If the start time points earlier than the maintained history of Publisher's event buffer, then the subscription must be rejected.  In this case the error response to the <establish-subscription> request should include a start time supportable by the Publisher. An end time may be specified using the optional stop-time parameter, which may also be earlier than the current time.  If no stop-time is present, notifications will continue to be sent until the subscription is terminated.

Not all streams will support replay.  Those that do must include the <replay-supported> flag.  In addition, a notification stream that does support replay is not expected to have an unlimited supply of saved notifications available to accommodate any given replay request.  Subscribers may do a get on <replay-log-creation-time> and <replay-log-aged-time> to assess the availability of notifications for replay.  The actual number of stored notifications available for retrieval at any given time is a publisher specific matter.   Control parameters for this aspect of the feature are outside the scope of this document.

The updated data model tree in Section 3 will includes:.
    +--ro streams
       +--ro stream* [stream]
          +--ro stream
          +--ro description
          +--ro replay-support?
          +--ro replay-log-creation-time?
          +--ro replay-log-aged-time?

And for the YANG model in Section 9, we have:
(Note: all definitions taken from RFC-5277 section 3.4)

  container streams {
    config false;
    description
      "This container contains information on the built-in streams
      provided by the publisher.";
    list stream {
      key "stream";
      description
        "Identifies the built-in streams that are supported by the
         publisher.";
      leaf stream {
        type stream;
        description
          "A handle for a sequential set of events, each of which
           is characterized by its own domain and semantics.
           In case configurable custom streams are supported,
           as indicated by the custom-stream identity, the configuration
           of those custom streams is provided separately.";
      }
      leaf description {
        type string;
        mandatory true;
        description
          "A description of the event stream, including such information
           as the type of events that are sent over this stream.";
      }
      leaf replay-support {
        type empty;
        description
          "Indicates that event replay is available on this stream.";
      }
      leaf replay-log-creation-time {
        type yang:date-and-time;
        description
          "The timestamp of the creation of the log used to support the
          replay function on this stream. Note that this might be
          earlier then the earliest available notification in the log.
          This object is updated if the log resets for some reason.
          This object MUST be present if replay is supported.";
      }
      leaf replay-log-aged-time {
        type yang:date-and-time;
        description
          "The timestamp of the last notification aged out of the log.
          This object MUST be present if replay is supported and any
          notifications have been aged out of the log.";
      }
    }
  }

As always, thoughts and comments welcome.

Eric

---

From: Rohit R Ranade <[email-masked]>
Date: Wed, 12 Jul 2017 08:49:38 +0000

Hi Eric,

"Replay is only viable for dynamic subscriptions.".
If we have configured subscriptions, but due to some reason the session goes down and during this time events occur, then how to get the notifications for these events when the session connects again ?
Also it is not clear, why it is viable only for dynamic subscriptions.

From: Netconf [mailto:[email-masked]] On Behalf Of Eric Voit (evoit)
Sent: 12 July 2017 03:25
To: [email-masked]
Subject: [Netconf] Subscribed-notifications, replay objects

  container streams {
    config false;
    description

Eric

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Wed, 12 Jul 2017 11:51:25 +0000

Hi Rohit,

Thoughts in-line...

From: Rohit R Ranade, July 12, 2017 4:50 AM

Hi Eric,

If a transport session goes down, it shouldn't be considered replay if events are queued and awaiting transmission.  (In fact, for connectionless transport, this might be common.)

If a configured subscription crashes and is restored in a way where the Publisher fully recognizes the gap in events, this is indistinguishable from the previous case.  And in fact, a new subscription-started notification should not be sent.

If the Publisher loses context (e.g. via a reboot), it will have no idea which previous events it must send.  It could send duplicates if it even makes such an attempt.  So by sending a subscription-started it notifies the subscriber of a discontinuity in transmission.  And if the subscriber wishes, it could attempt a replay via a different (dynamic) subscription to fill any Subscriber recognized time-gap in events.

The most common case here will be a reboot.  If event logs, are retained how far back in the past can/should you send events?  Perhaps we could cover a few corner-cases with more specification, but having a targets dynamic subscription replay seems a simpler way to address this.

Thanks,
Eric

From: Netconf [mailto:[email-masked]] On Behalf Of Eric Voit (evoit)
Sent: 12 July 2017 03:25
To: [email-masked]<mailto:[email-masked]>
Subject: [Netconf] Subscribed-notifications, replay objects

  container streams {
    config false;
    description

Eric

---

From: Rohit R Ranade <[email-masked]>
Date: Thu, 13 Jul 2017 05:56:34 +0000

Hi Eric,

I was going through the draft and have a few  questions.
   <notification
          xmlns=" urn:ietf:params:xml:ns:netconf:notification:1.0">
           <eventTime>2007-09-01T10:00:00Z</eventTime>
       <subscription-id>500</subscription-id>
       <link-failure xmlns="http://acme.example.com/system">
           <if-name>so-1/2/3.0</if-name>
           <if-admin-status>up</if-admin-status>
           <if-oper-status>down</if-oper-status>
       </link-failure>
   </notification>

ð  Why is the subscription-id required in notification ? In which scenario, Client needs to know this notification belongs to which subscription-id  ?

In section 5.1:

In this case, when there is something to transport

   for an active subscription, transport protocol specific call-home

   operations will be used to establish the connection.
What if the server does not support call-home how to handle this scenario ? Whether call-home feature is mandatory to implement this draft ? Please clarify.

I feel the concept of replay and dynamic vs configured subscription should not be mixed.
The configured subscription ensures that it is persistent across transport disconnect / reboot. I feel, IF there is no mechanism to transport the notifications from server to client, then for events generated during this time, their corresponding notifications should be dropped.
If it is logged for sending as soon as transport is available, then in NETCONF case as soon <hello> is exchanged there will be a stream of notifications that will be sent by the Server to the Client, for which the client may not be ready.
Can we have a separate rpc for starting the replay for configured subscription ?

Also can you please add a flow diagram which shows how the notifications(control & data)  flow between Client and Server for configured subscription in normal and across-reboot scenario.

With Regards,
Rohit

From: Eric Voit (evoit) [mailto:[email-masked]]
Sent: 12 July 2017 17:21
To: Rohit R Ranade; [email-masked]
Subject: RE: Subscribed-notifications, replay objects

Hi Rohit,

Thoughts in-line...

Thanks,
Eric

  container streams {
    config false;
    description

Eric

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 13 Jul 2017 14:03:11 +0000

Hi Rohit,

From: Rohit R Ranade, July 13, 2017 1:57 AM

Hi Eric,

<eric> A few scenarios:

(1) If the Subscriber is a controller serving multiple applications, it show to which application the information should be sent.  This may also be important for different dampening intervals on commonly subscribed objects.
(2) Easier debug

In section 5.1:

<eric> The call home operations are tied to the transport draft.  For NETCONF this is:
draft-ietf-netconf-netconf-event-notifications
For example, see section 3.4.3.

<eric> I think what you are asking for is that the Subscriber is capable of recognizing a discontinuity in event passing, as identified by the receipt of a new <subscription-started> state change notification for a subscription-id which it already recognizes.  This RPC would be allowed on configured and dynamic subscriptions.  In this case the RPC would provide the subscription id, and the start & stop times only.

If this is what you are suggesting, I do see one issue.  There will be an interleaving of old and new events within the same subscription's event stream. I.e., the ordering of the events in the stream would not be sequential in time.

To me it seems safer and simpler if a replay goes to a separate subscription-id.  I would be very interested to get others' opinions on this.

Also can you please add a flow diagram which shows how the notifications(control & data)  flow between Client and Server for configured subscription in normal

<eric>  As these can vary slightly by transport, we put them in the Transport drafts.  Does Section 3.4 of
draft-ietf-netconf-netconf-event-notifications
meet your needs?

and across-reboot scenario.

<evoit> If there is a reboot, we had not been thinking of a difference in flows.   If based on the request for opinions above, other WG members also want to enable replay on configured subscriptions we could certainly add a flow here.

Thanks for the review,
Eric

With Regards,
Rohit

From: Eric Voit (evoit) [mailto:[email-masked]]
Sent: 12 July 2017 17:21
To: Rohit R Ranade; [email-masked]<mailto:[email-masked]>
Subject: RE: Subscribed-notifications, replay objects

Hi Rohit,

Thoughts in-line...

Thanks,
Eric

  container streams {
    config false;
    description

Eric

---

From: Rohit R Ranade <[email-masked]>
Date: Mon, 17 Jul 2017 07:47:48 +0000

Hi Eric,

""

To me it seems safer and simpler if a replay goes to a separate subscription-id.  I would be very interested to get others' opinions on this.""
[Rohit R Ranade]  The client can know that it was down from Time T1 to T2. So when his transport is ready, the client will  want to know what notifications were missed between T1 - T2. It does not depend on whether Client knows it was missed or not.

About the subscription-id, a few questions.

1.       How is the client expected to maintain this subscription-id ? Each APP on the client side to maintain a range of ids ? 1-10 : APP1, 2-20: APP2 etc ?  Is it not better to expose it as a subscription-name as this name can be meaningful to them rather than asking client to maintain a ID-to-Name mapping. Secondly if new APPs are added, they will need to adjust the subscription-id range given for each APP which maybe cumbersome. If use a name, Client can impose rules about how to set a subscription-name.

2.       About the error scenarios it is not clear, if Session1 use <rpc> to generate a subscription-id(say X)  and if Session2 uses same "X" for configuring the subscription, then the configuration will fail ? Whether persistent data and operation-data should be used to recognize duplicacy ?

With Regards,
Rohit R
Sent: 13 July 2017 19:33

Hi Rohit,

=>  Why is the subscription-id required in notification ? In which scenario, Client needs to know this notification belongs to which subscription-id  ?

<eric> A few scenarios:

In section 5.1:

[Rohit R Ranade]

Thanks for the review,
Eric

With Regards,
Rohit

Hi Rohit,

Thoughts in-line...

Thanks,
Eric

  container streams {
    config false;
    description

Eric

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Mon, 17 Jul 2017 20:24:59 +0000

Hi Rohit,

From: Rohit R Ranade, July 17, 2017 3:48 AM

Hi Eric,

""

<Eric2> I think your intent is to make the reboot transparent from the perspective of the receiver application.  And this is a worthy goal, but it will be complex to achieve on a reboot.  Some thoughts:

(1) If that can be accomplished without the Receiver involvement, does anything new need to be specified in standards at all?   All the configured subscription receiver would see is a delay in Notification delivery.

(2) To implement this transparency seamlessly,  the Publisher would have maintain and persistently save the last notification which was successfully sent on a subscription prior to T1.

·       Without being knowing the last one sent, you might get duplicates, or you might unknowingly miss events.   Also events can be bundled and sent as a group.  Therefore event-time should not be relied upon to be definitive on whether a particular event was sent or not.

·       If nothing identifies that a configured subscription was previously active, the first time a subscription initializes, how do you know if a subscription was active before?  Should any events be replayed?

(3) The current specification has a notification for subscription-started.  This alerts the receiver that some discontinuity occurred.  And as a result, the receiver has the option of trying to reconstruct missing events, should they choose.

I don't see any reason why some combination of the above cannot be implemented by a platform without any need to change the specification at all.  Would you concur?

1.       How is the client expected to maintain this subscription-id ? Each APP on the client side to maintain a range of ids ? 1-10 : APP1, 2-20: APP2 etc ?  Is it not better to expose it as a subscription-name as this name can be meaningful to them rather than asking client to maintain a ID-to-Name mapping.
<Eric2> Configured subscription id is set on the Publisher.  Out-of-band logic will be needed to bond the application to the subscription-id.   Dynamic subscriptions won't have this issue.

Secondly if new APPs are added, they will need to adjust the subscription-id range given for each APP which maybe cumbersome. If use a name, Client can impose rules about how to set a subscription-name.
<Eric> For configured subscriptions, it would be up to some management system to maintain these ranges.  For dynamic subscriptions, the subscription-id's are returned at establishment time.   It is possible that some future protocol might be viable to partition these ranges dynamically to assist in subscription-group handling.  But there is nothing attempted at this point.

<Eric> I think you are asking about the potential for collisions in the subscription-id space.   Section 10.1 of configured-subscriptions talks about the splitting of the address ranges to avoid such collisions.

Eric

With Regards,
Rohit R

Hi Rohit,

<eric> A few scenarios:

In section 5.1:

Thanks for the review,
Eric

With Regards,
Rohit

Hi Rohit,

Thoughts in-line...

Thanks,
Eric

  container streams {
    config false;
    description

Eric

---

From: Balazs Lengyel <[email-masked]>
Date: Thu, 20 Jul 2017 15:40:07 +0200

Hello Eric,

We have been using replay for on-change notifications in our own
ericsson solution. I would not like to loose that possibility. I
agree that replay is not that useful on periodic subscriptions,
but it is usable for on-change. We use on-change notifications for
configuration mirroring. If we loose the notification session for
some time, it is much cheaper to replay the lost notifications
then to get a full config update.

regards Balazs

On 2017-07-11 23:55, Eric Voit (evoit)
wrote:

In an email earlier today, Martin pointed
out replay was under-defined.  To fix this, below is suggested
text and yang objects.  The goal of this is keep this as close
to RFC-5277 as possible.

So following would replace the text in
Section 4.1.1

Replay provides the ability to establish
an event subscription which is also capable of passing along
recently generated events.  In other words, as the
subscription initializes itself, it sends any previously
generated notifications for the target event stream which meet
the filter and timeframe criteria.  These historical events
would then be followed by events generated after the
subscription has been established.  All events will be
delivered in the order generated.  Replay is only viable for
dynamic subscriptions.  Replay is an optional feature.
Replay is dependent on a notification stream supporting some
form of notification logging, although it puts no restrictions
on the size or form of the log, or where it resides within the
device.

The presence of a replay-start-time
within an establish-subscription RPC is the way a replay may
be been requested.  The provided start time must be earlier
than the current time.  If the start time points earlier than
the maintained history of Publisher's event buffer, then the
subscription must be rejected.  In this case the error
response to the <establish-subscription> request should
include a start time supportable by the Publisher. An end time
may be specified using the optional stop-time parameter, which
may also be earlier than the current time.  If no stop-time is
present, notifications will continue to be sent until the
subscription is terminated.

Not all streams will support replay.
Those that do must include the <replay-supported> flag.
In addition, a notification stream that does support replay is
not expected to have an unlimited supply of saved
notifications available to accommodate any given replay
request.  Subscribers may do a get on
<replay-log-creation-time> and
<replay-log-aged-time> to assess the availability of
notifications for replay.  The actual number of stored
notifications available for retrieval at any given time is a
publisher specific matter.   Control parameters for this
aspect of the feature are outside the scope of this document.

The updated data model tree in Section 3
will includes:.

+--ro streams

+--ro stream* [stream]

+--ro stream

+--ro description

+--ro replay-support?

And for the YANG model in Section 9, we
have:

(Note: all definitions taken from RFC-5277
section 3.4)

container streams {

config false;

description

"This container contains information
on the built-in streams

list stream {

key "stream";

description

"Identifies the built-in streams
that are supported by the

publisher.";

leaf stream {

type stream;

description

"A handle for a sequential set of
events, each of which

is characterized by its own
domain and semantics.

In case configurable custom
streams are supported,

as indicated by the
custom-stream identity, the configuration

of those custom streams is
provided separately.";

}

leaf description {

type string;

mandatory true;

description

"A description of the event
stream, including such information

as the type of events that are
sent over this stream.";

}

leaf replay-support {

type empty;

description

"Indicates that event replay is
available on this stream.";

}

type yang:date-and-time;

description

"The timestamp of the creation of
the log used to support the

replay function on this stream.
Note that this might be

earlier then the earliest
available notification in the log.

This object is updated if the log
resets for some reason.

This object MUST be present if
replay is supported.";

}

type yang:date-and-time;

description

"The timestamp of the last
notification aged out of the log.

This object MUST be present if
replay is supported and any

notifications have been aged out
of the log.";

}

}

}

Eric

_______________________________________________
Netconf mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netconf

--
Balazs Lengyel Ericsson Hungary Ltd.
Senior Specialist
Mobile: +36-70-330-7909 email: [email-masked]

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 20 Jul 2017 16:09:57 +0000

Hi Balazs,

There is value of being able to resend/replay lost on-change updates.  Let's figure out the best way to frame this.

First let's establish the scope.  As datastore changes need to be applied in sequence, patching a datastore with replayed data could require that you first back out any subsequent changes which were applied on the receiver's objects (if any).  Is Ericsson considering the backed-out scenario as part of replay scope?   At this point I am assuming no, but we don't want to define any technology mechanisms which might preclude this in the future.

Per the discussion during today's NETCONF session.  It seems risky to ask for on-change replay based on timestamps (at least for generalized implementations).  There are two issues for me here:

(1) Which timestamps should be used, & what about notification delay? A pure time based replay as part of a generic on-change capability could have loss/duplication, particularly as change-updates can be dampened and batched.

*       As you noted in your parallel thread: "Many systems don't differentiate between notification time and record-time."

*       With dampening, you need to know when the update actually got sent.  I.e., you need to look at the notification time rather than the time a change was made to a datastore.

*       When replay state time & notification time timestamps are really close, maintaining the integrity or the receiver's datastore extract would need to be done by looking at whether a notification was sent or not.

(2) Replay from what?

*       An attempt to replay on-change notifications means that there was a previous dynamic subscription with identical parameters previously in place.  In addition, it must be possible to reconstruct deltas from that previous dynamic subscription. (This is not an issue for configured subscriptions, and therefore it is not an issue for the multiple-receiver case.)

*       There is not a meaningful difference in providing one patch operation representing all changes, and incremental patch operations.

Based on this, it would seem to me that on-change replay should be done based on an augmentation to the establish-subscription RPC.  This augmentation would allow an the addition of the input object "previous-subscriptions-last-notification-id".  Having that parameter tightly focuses the viable use and security of this type of replay.  As notification-id is defined in draft-voit-netconf-notification-messages, it seems like we need this nail this down in conjunction with that draft.  Perhaps the RPC augmentation could be added here?

BTW: In the original version of draft-voit-netmod-yang-notifications2, we allowed random access in case receivers wanted to get past information.  This random access was seen as overkill by some.  If we want a more robust replay capability than above, we should revisit this.

Eric

From: Balazs Lengyel [mailto:[email-masked]]
Sent: Thursday, July 20, 2017 9:40 AM
To: Eric Voit (evoit) <[email-masked]>; [email-masked]
Subject: Re: [Netconf] Subscribed-notifications, replay objects

Hello Eric,

We have been using replay for on-change notifications in our own ericsson solution. I would not like to loose that possibility. I agree that replay is not that useful on periodic subscriptions, but it is usable for on-change. We use on-change notifications for configuration mirroring. If we loose the notification session for some time, it is much cheaper to replay the lost notifications then to get a full config update.

regards Balazs

On 2017-07-11 23:55, Eric Voit (evoit) wrote:
In an email earlier today, Martin pointed out replay was under-defined.  To fix this, below is suggested text and yang objects.  The goal of this is keep this as close to RFC-5277 as possible.

  container streams {
    config false;
    description

Eric

Netconf mailing list

[email-masked]<mailto:[email-masked]>

--

Senior Specialist

Mobile: +36-70-330-7909              email: [email-masked]<mailto:[email-masked]>