Subject: Re: [IPFIX] new monitoring interval information elements

From: Nevil Brownlee <[email-masked]>
Date: Fri, 17 Aug 2012 16:33:46 +1200

Hi all:

Some time back Paul Aitken asked IANA for two new IEs to
describe "monitoringInterval."  There was some discussion of these
in June~July, but no clear consensus.  The last email on this was
from Andrew Johnson (below).

To me Andrew's definitions seem reasonable; can we go ahead with
them, or can anyone offer further improvements?

Cheers, Nevil

On 7/07/12 1:42 AM, Andrew Johnson wrote:
> Hi Gerhard
>
> There is more than one way to present the model I describe.  It would seem to be valid to suggest that the Metering Process is re-initialising every interval and you could use delta or total counts, but you would also have to honour all the other aspects of re-initialising the Metering Process, like resetting any Metering Process statistics being gathered.  I think this model would add a requirement on any Collecting Process to understand new IEs that describe the Metering Process re-initialisation in order to properly understand the total counts.
>
> I would like to model this using the idea that the interval is a common property (i.e. key field) of the flow.  Again, delta counts or total counts *could* be used, because once a certain window has passed the new flows would have a different set of keys.  Delta counts make more sense though, because then the collector doesn't have to be configured to understand that the interval is the key.  The collector would be free to aggregate as it needs, with the ability to:
>    * show the aggregate view of flows, i.e. aggregate each flow across intervals
>    * show the whole network over time, i.e. aggregate all flows per interval
>    * show how a flow evolves over time, i.e. filter out a flow and display the per-interval info
>
>
> The problem we are having, is to accurately define IEs that can describe the beginning and end of this interval.  I was hoping that by laying out the model, we could get some consensus on the model and the IE definitions.  My initial suggestion was:
>
>    observationWindowStart  dateTimeMilliseconds  The absolute timestamp of the beginning of the current Observation Window.
>    observationWindowEnd    dateTimeMilliseconds  The absolute timestamp of the end of the current Observation Window.
>
>
> but Observation Window is undefined.  I'm open to suggestions on improving these.  Perhaps:
>
>    meteringIntervalStart   dateTimeMilliseconds  The absolute timestamp of beginning of the Metering
>                                                  Interval the observation occurred in.  Where a
>                                                  series of Metering Intervals separates time into a
>                                                  a set of discrete buckets of a configurable length.
>    meteringIntervalEnd     dateTimeMilliseconds  The absolute timestamp of the end of the Metering
>                                                  Interval the observation occurred in.  See
>                                                  meteringIntervalStart.
>
>
> Cheers, Andrew
>
>
> On 30 Jun 2012, at 10:59, Gerhard Muenz wrote:
>> Andrew,
>>
>> As I pointed out in my previous mail, the window IEs do not seem to be compatible with the descriptions of existing *TotalCount IEs because these refer to the time since (re-)initialization. You could use *DeltaCount, though.
>>
>> Gerhard
>>
>>
>> On 29.06.2012 01:25, Andrew Johnson wrote:
>>> Hi Gerhard
>>>
>>> Thanks for your inputs on this thread.  It seems to me that the idea of "re-initialising" the Metering Process is pretty confusing, in light of what we're actually doing.  I think my mental mode of what is happening is a bit different from Paul's, so let me start at the beginning, show how I think of the behaviour, and see what we can come up with to solve our problem.
>>>
>>> We would like to split the monitored packet streams into discrete buckets, based on time.
>>>
>>>      t0 -----> t1 -----> t2 -----> t3 -----> t4 -----> t5 ----->
>>>
>>>
>>> The packet stream is processed by the Metering Process as usual, and flows form in the cache in the usual manner.  However, observations from one interval won't be merged with observations from a different interval.  Ideally, at the end of each interval, all flows are exported and flushed from the cache.
>>>
>>> There are several ways to think about this.  One way is to explicitly flush the cache at the end of each interval (i.e. reset / reinitialise the Metering Process).  The way I see it though, we could just make an identifier of the current interval a key field and apply the optimisation of allowing flows from outside the current interval to be flushed from the cache ASAP, since we know they won't be updated again.
>>>
>>> Hopefully that gives a bit of a bigger picture.  What we are looking for are two new fields that we can add to the flows, that identify the beginning at end of the interval they were observed in.  In some circumstances the device may not fully stick to the provided interval window, so we can't rely on the end of the interval really being the intervalStart + intervalSize.
>>>
>>> I was thinking of:
>>>
>>>    observationWindowStart  dateTimeMilliseconds  The absolute timestamp of the beginning of the current Observation Window.
>>>    observationWindowEnd    dateTimeMilliseconds  The absolute timestamp of the end of the current Observation Window.
>>>
>>>
>>> However, Paul rightly pointed out to me that IPFIX doesn't have any concept of an Observation Window, and we're not sure how we can define that in description of the IE.  The idea of re-initialising the Metering Process may fit in with the current IPFIX model, but I don't think it properly conveys the behaviour we're after.
>>>
>>>
>>> Kind regards,
>>>
>>> Andrew
>>>
>>>
>>> On 25 Jun 2012, at 18:24, Gerhard Muenz wrote:
>>>> Paul,
>>>>
>>>> It depends on what kind of interval you want to specify.
>>>>
>>>> If you want to specify the (re-)initialization time of the Metering Process, why not name the IE meteringProcessSysUpTime or (if you do not like sysup) meteringProcessInitializationTime?
>>>> And mention (re-)initialization in the description?
>>>> Dito for exportingProcessInitializationTime etc.
>>>>
>>>> If we need the stop time, you can use meteringProcessStopTime or so.
>>>>
>>>> If you want to define a new interval which is not starting at (re-)initialization time but later, then your definitions are ok. But this means that you need to specify new counter elements for this interval.
>>>>
>>>> Thanks,
>>>> Gerhard
>>>>
>>>>
>>>> On 25.06.2012 19:08, Paul Aitken wrote:
>>>>> Gerhard,
>>>>>
>>>>>> Ok, I see the difference to discontinuity time.
>>>>>>
>>>>>> Some further thoughts:
>>>>>>
>>>>>> The common understanding is that the "time since re-initialization"
>>>>>> means sysUpTime. For example, have a look at the description of
>>>>>> flowEndSysUpTime:
>>>>>> "The relative timestamp of the last packet of this Flow. It indicates
>>>>>> the number of milliseconds since the last (re-)initialization of the
>>>>>> IPFIX Device (sysUpTime)."
>>>>>
>>>>> Yes, sysUpTime (so specifically, flowStartSysUpTime and
>>>>> flowEndSysUpTime), are times "since the last (re-)initialization of the
>>>>> IPFIX Device".
>>>>>
>>>>> These are not necessarily related to the metering process.
>>>>>
>>>>> Taking some other examples: octetTotalCount, packetTotalCount,
>>>>> droppedOctetTotalCount, droppedPacketTotalCount, observedFlowTotalCount,
>>>>> ignoredPacketTotalCount, ignoredOctetTotalCount,
>>>>> postMCastOctetTotalCount, ... are all of the form:
>>>>>
>>>>>       "The total number of X since the Metering Process
>>>>> (re-)initialization for this Observation Point."
>>>>>
>>>>> We could suppose that the Metering Process initialised at system uptime.
>>>>> However, that may not be the case: it could have started any amount of
>>>>> time later. eg, a user could configure a Metering Process at any time.
>>>>> Or, an automated system could start and stop Metering Processes
>>>>> according to different times of day or different network conditions.
>>>>> Without timestamps to tell us, we simply do not know.
>>>>>
>>>>>
>>>>>
>>>>>> Your proposed IEs do not mention neither (re-)initialization nor
>>>>>> sysUpTime. They rather seem to specify a time interval which is
>>>>>> located somewhere within the sysuptime interval. Hence, the interval
>>>>>> does not work to define the time frame of exportedMessageTotalCount,
>>>>>> for example. You would need to define a new IE
>>>>>> exportedMessageInMonitoringInterval.
>>>>>
>>>>> Indeed, because the IEs which I'm specifying here are particular to the
>>>>> Monitoring Process, whereas exportedMessageTotalCount seems to require a
>>>>> time reference for the Exporting Process. It's a related, though
>>>>> different, issue.
>>>>>
>>>>> (ie, the Exporting Process may start from system init, or at any time
>>>>> since then. It may have been started before, or after, the Metering
>>>>> Process. Without a suitable timestamp IE, we simply do not know.)
>>>>>
>>>>>
>>>>>> The different IE descriptions suggest that the time of
>>>>>> re-initialization (i.e. sysuptime) can be different for Metering
>>>>>> Process, Exporting Process etc.
>>>>>
>>>>> Indeed this is so. There is no necessity for these to be the same.
>>>>>
>>>>>
>>>>>> It seems that we need IE to export sysUpTime for each process.
>>>>>
>>>>> We agree.
>>>>>
>>>>> So in principal, you understand what I'm trying to do. However, you
>>>>> would like clearer definitions of the Information Elements?
>>>>>
>>>>> Thanks,
>>>>> P.
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> IPFIX mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/ipfix
>>>
>>
>
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix
>

-- 
---------------------------------------------------------------------
  Nevil Brownlee                    Computer Science Department | ITS
  Phone: +64 9 373 7599 x88941             The University of Auckland
  FAX: +64 9 373 7453   Private Bag 92019, Auckland 1142, New Zealand

---

From: Brian Trammell <[email-masked]>
Date: Mon, 20 Aug 2012 12:53:30 +0200

Hi Nevil, all,

This seems reasonable; however, I'd ask what the semantic difference is between these IEs and those defined for the same purpose in RFC5655 (minFlowStart* and maxFlowEnd*, IEs 261, 265, 268-273).

Cheers,

Brian