Subject: [IPFIX] new monitoring interval information elements

From: Paul Aitken <[email-masked]>
Date: Sat, 16 Jun 2012 15:56:11 +0100

Dear IPFIX experts,

FYI, I'm requesting two new IPFIX Information Elements from IANA to 
record the start and end of a monitoring interval, which is a period of 
time during which the Metering Process is running.

The existing IPFIX Information Elements provide flow based timestamps, 
system init, and observation times. There's no indication of the 
Metering Process, whichmay be running for a period of time during which 
no flows of interest or relevanceare observed.

In fact, many existing elements are described as "The total number of X 
since the Metering Process (re-)initialization". However, there's 
currently no indication of when that (re-)initialization was. These new 
Information Elements enable an IPFIX device to indicate when the 
Metering Process started and ended.

These new elements are particularly useful when the Metering Process is 
windowed - ie, it runs for a fixed interval, exporting all its 
observations only at the end of that interval, before optionally 
starting over. With these new Information Elements, an IPFIX device can 
readily indicate the start (and end) of each interval.

Our present need is only for milliseconds resolution. At this time we 
don't need the equivalent seconds, microseconds, or nanoseconds 
elements, although these could be created in future if needs be.

Also note the subtle distinction between "monitoring" and "metering": a 
device could be monitoring traffic for some time, without detecting 
anything of interest or relevance to be metered. The metering interval 
could almost be expressed by minFlowStart* to maxFlowEnd*. We've chosen 
to name these elements "monitoring interval" rather than "metering 
interval" to clearly express this difference.

Name:         monitoringIntervalStartMilliSeconds
Data Type:    dateTimeMilliseconds
Sematics:     -
Description: The absolute timestamp at which the monitoring interval 
started.
               A Monitoring interval is the period of time during which 
the Metering Process is running.
Units:        milliseconds

Name:         monitoringIntervalEndMilliSeconds
Data Type:    dateTimeMilliseconds
Sematics:     -
Description: The absolute timestamp at which the monitoring interval ended.
               A Monitoring interval is the period of time during which 
the Metering Process is running.
Units:        milliseconds

Thanks,
P.

---

From: Gerhard Muenz <[email-masked]>
Date: Sun, 17 Jun 2012 19:17:31 +0200

Paul,

Please check if the proposed IEs are compatible with 
ipfixMeteringProcessCacheDiscontinuityTime in IPFIX-MIB and 
cacheDiscontinuityTime in IPFIX-CONFIG.

Parameters which exist in IPFIX-MIB, IPFIX-CONFIG and as IE should 
follow the same naming scheme and semantic, if possible.

Thanks,
Gerhard

On 16.06.2012 16:56, Paul Aitken wrote:
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Paul Aitken <[email-masked]>
Date: Mon, 25 Jun 2012 16:38:58 +0100

Gerhard,

It's not entirely clear what a "discontinuity" is since it's not clearly 
defined. However, inferring from section 5.8 of RFC 5815, 
ipfixMeteringProcessCacheDiscontinuityTime and cacheDiscontinuityTime 
are intended to indicate when the Metering Process was running but 
wasn't able to meter correctly.

The new Information Elements which I'm proposing below aren't related to 
discontinuities. They address a different set of problems. In this case 
the Metering Process is functioning correctly, so there are no 
discontinuities. If there were, they can be expressed with 
ipfixMeteringProcessCacheDiscontinuityTime and cacheDiscontinuityTime. 
However, they are not relevant to the discussion here.

Looking in IANA's IPFIX Information Element registry, we can see many 
Information Elements (about 15) defined as "The number of X since the 
Metering Process (re-)initialization" - so we need to know when that was 
in order to be able to use these Information Elements properly.

These definitions suggest an expectation that the Metering Process will 
start and may subsequently restart. That could be for many reasons; 
there's no suggestion that it's an error.

The new Information Elements proposed below serve to indicate when the 
Metering Process (re)started, and when it ended. As far as I know, there 
are currently no Information Elements for this.

Reviewing RFC 5101, I noticed that the "Metering Process Statistics 
Option Template" contains several fields defined as, "The total number X 
since the Exporting Process re-initialization", so it seems that an 
additional Information Element is required for that too, in the case of 
connectionless export transports.

Thanks,
P.

On 17/06/12 18:17, Gerhard Muenz wrote:
>> ended.

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 25 Jun 2012 18:50:38 +0200

Paul,

Ok, I see the difference to discontinuity time.

Some further thoughts:

The common understanding is that the "time since re-initialization" 
means sysUpTime. For example, have a look at the description of 
flowEndSysUpTime:
"The relative timestamp of the last packet of this Flow. It indicates 
the number of milliseconds since the last (re-)initialization of the 
IPFIX Device (sysUpTime)."

Your proposed IEs do not mention neither (re-)initialization nor 
sysUpTime. They rather seem to specify a time interval which is located 
somewhere within the sysuptime interval. Hence, the interval does not 
work to define the time frame of exportedMessageTotalCount, for example. 
You would need to define a new IE exportedMessageInMonitoringInterval.

The different IE descriptions suggest that the time of re-initialization 
(i.e. sysuptime) can be different for Metering Process, Exporting 
Process etc.
It seems that we need IE to export sysUpTime for each process.

Thanks
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Mon, 25 Jun 2012 18:08:42 +0100

Gerhard,

> IPFIX Device (sysUpTime)."

Yes, sysUpTime (so specifically, flowStartSysUpTime and 
flowEndSysUpTime), are times "since the last (re-)initialization of the 
IPFIX Device".

These are not necessarily related to the metering process.

Taking some other examples: octetTotalCount, packetTotalCount, 
droppedOctetTotalCount, droppedPacketTotalCount, observedFlowTotalCount, 
ignoredPacketTotalCount, ignoredOctetTotalCount, 
postMCastOctetTotalCount, ... are all of the form:

     "The total number of X since the Metering Process 
(re-)initialization for this Observation Point."

We could suppose that the Metering Process initialised at system uptime. 
However, that may not be the case: it could have started any amount of 
time later. eg, a user could configure a Metering Process at any time. 
Or, an automated system could start and stop Metering Processes 
according to different times of day or different network conditions. 
Without timestamps to tell us, we simply do not know.

> sysUpTime. They rather seem to specify a time interval which is 
> located somewhere within the sysuptime interval. Hence, the interval 
> does not work to define the time frame of exportedMessageTotalCount, 
> for example. You would need to define a new IE 
> exportedMessageInMonitoringInterval.

Indeed, because the IEs which I'm specifying here are particular to the 
Monitoring Process, whereas exportedMessageTotalCount seems to require a 
time reference for the Exporting Process. It's a related, though 
different, issue.

(ie, the Exporting Process may start from system init, or at any time 
since then. It may have been started before, or after, the Metering 
Process. Without a suitable timestamp IE, we simply do not know.)

> The different IE descriptions suggest that the time of 
> re-initialization (i.e. sysuptime) can be different for Metering 
> Process, Exporting Process etc.

Indeed this is so. There is no necessity for these to be the same.

> It seems that we need IE to export sysUpTime for each process.

We agree.

So in principal, you understand what I'm trying to do. However, you 
would like clearer definitions of the Information Elements?

Thanks,
P.

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 25 Jun 2012 19:24:48 +0200

Paul,

It depends on what kind of interval you want to specify.

If you want to specify the (re-)initialization time of the Metering 
Process, why not name the IE meteringProcessSysUpTime or (if you do not 
like sysup) meteringProcessInitializationTime?
And mention (re-)initialization in the description?
Dito for exportingProcessInitializationTime etc.

If we need the stop time, you can use meteringProcessStopTime or so.

If you want to define a new interval which is not starting at 
(re-)initialization time but later, then your definitions are ok. But 
this means that you need to specify new counter elements for this interval.

Thanks,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 29 Jun 2012 00:25:31 +0100

Hi Gerhard

Thanks for your inputs on this thread.  It seems to me that the idea of "re-initialising" the Metering Process is pretty confusing, in light of what we're actually doing.  I think my mental mode of what is happening is a bit different from Paul's, so let me start at the beginning, show how I think of the behaviour, and see what we can come up with to solve our problem.

We would like to split the monitored packet streams into discrete buckets, based on time.

    t0 -----> t1 -----> t2 -----> t3 -----> t4 -----> t5 ----->

The packet stream is processed by the Metering Process as usual, and flows form in the cache in the usual manner.  However, observations from one interval won't be merged with observations from a different interval.  Ideally, at the end of each interval, all flows are exported and flushed from the cache.

There are several ways to think about this.  One way is to explicitly flush the cache at the end of each interval (i.e. reset / reinitialise the Metering Process).  The way I see it though, we could just make an identifier of the current interval a key field and apply the optimisation of allowing flows from outside the current interval to be flushed from the cache ASAP, since we know they won't be updated again.

Hopefully that gives a bit of a bigger picture.  What we are looking for are two new fields that we can add to the flows, that identify the beginning at end of the interval they were observed in.  In some circumstances the device may not fully stick to the provided interval window, so we can't rely on the end of the interval really being the intervalStart + intervalSize.

I was thinking of:

  observationWindowStart  dateTimeMilliseconds  The absolute timestamp of the beginning of the current Observation Window.
  observationWindowEnd    dateTimeMilliseconds  The absolute timestamp of the end of the current Observation Window.

However, Paul rightly pointed out to me that IPFIX doesn't have any concept of an Observation Window, and we're not sure how we can define that in description of the IE.  The idea of re-initialising the Metering Process may fit in with the current IPFIX model, but I don't think it properly conveys the behaviour we're after.

Kind regards,

Andrew

On 25 Jun 2012, at 18:24, Gerhard Muenz wrote:
> If you want to specify the (re-)initialization time of the Metering Process, why not name the IE meteringProcessSysUpTime or (if you do not like sysup) meteringProcessInitializationTime?
> If you want to define a new interval which is not starting at (re-)initialization time but later, then your definitions are ok. But this means that you need to specify new counter elements for this interval.
> 
> Thanks,
> Gerhard
> 
>

---

From: Gerhard Muenz <[email-masked]>
Date: Sat, 30 Jun 2012 11:59:53 +0200

Andrew,

As I pointed out in my previous mail, the window IEs do not seem to be 
compatible with the descriptions of existing *TotalCount IEs because 
these refer to the time since (re-)initialization. You could use 
*DeltaCount, though.

Gerhard