Subject: [IPFIX] Fw: Export of long lived flow information

From: John Court <[email-masked]>
Date: Thu, 1 Nov 2012 08:01:20 +1000

Damn,

Sorry about the previous message, when I re-read it the third time I 
realised that the positive allows the confusion of failing to clearly 
indicate what "isn't" included.

I am not sure if this is better but what about :

If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record MUST account for ALL packets observed after/before 
this timestamp.

John Court

----- Forwarded by John Court/Australia/IBM on 01/11/2012 07:56 -----

From:   John Court/Australia/IBM
To:     Brian Trammell <[email-masked]>, 
Cc:     [email-masked], Gerhard Muenz <[email-masked]>, Paul Aitken 
<[email-masked]>
Date:   01/11/2012 07:37
Subject:        Re: [IPFIX] Export of long lived flow information

Personally I always find negative logic requires a double read to make 
sure I understand it.  Whats wrong with the positive logic equivalent ?

>>If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record do not refer to packets observed before/after this 
>>timestamp.

Becomes :

If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record ONLY refer to packets observed after/before this 
timestamp.

Thanks

John Court

From:   Brian Trammell <[email-masked]>
To:     Gerhard Muenz <[email-masked]>, 
Cc:     Paul Aitken <[email-masked]>, John Court/Australia/IBM@IBMAU, 
<[email-masked]>
Date:   31/10/2012 18:17
Subject:        Re: [IPFIX] Export of long lived flow information

Hi, Gerhard, all,

Thought about this a bit too, and I can't come up anything better than the 
negative clarification. I might consider changing "preceding" and 
"succeeding" to "before" and "after" to simplify the language for 
non-native speakers, and stick an "observed" in there to make clear we're 
talking about observation time:

If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record do not refer to packets observed before/after this 
timestamp.

It either case, it does make things a lot more explicitly clear, so I'd 
definitely add it to the timestamp/duration IEs.

Thanks, cheers,

Brian

On Oct 31, 2012, at 8:52 AM, Gerhard Muenz wrote:

> 
> Hi all,
> 
> I scanned through the IANA registry and briefly checked occurrences of 
"this Flow". I do not think that we need change them in general, except 
for those IEs which define a measurement interval.
> 
> So, we have:
> - flowStart* (including flowStartDeltaMicroseconds)
> - flowEnd* (including flowEndDeltaMicroseconds)
> - flowDuration*
> 
> My first shot was the following change for absolute flowStart*/flowEnd* 
timestamps:
> 
> OLD:
> The absolute timestamp of the first|last packet of this Flow.
> NEW:
> The absolute timestamp of the first|last packet accounted in this Flow 
Record.
> 
> We could add a sentence clarifying that the Flow properties reported in 
this Flow Record refer to the given measurement interval.
> At the moment, I only find a good "negative" way to describe this:
> "If this Information Element is present in a Flow Record, Flow 
properties in this Flow Record do not refer to packets 
preceding|succeeding this timestamp."
> 
> Do you have better suggestions?
> Or do you think that this clarification is not necessary?
> 
> Regards,
> Gerhard
> 
> 
> On 30.10.2012 13:38, Brian Trammell wrote:
>> Hi, Paul, all,
>> 
>> If someone throws together a quick summary of the position for me I'd
>> be happy make a couple of slides and do the presentation in person in
>> Atlanta.
>> 
>> From where I sit, it looks like we just go ask IANA to make the
>> registry change. Wearing my IE-Doctors-author hat (I can't wear an IE
>> Doctor hat, there aren't any yet. :) ), I'd say this revision would be
>> covered (and permissible) under point 2 in section 5.2 of ie-doctors.
>> 
>> Best regards,
>> 
>> Brian
>> 
>> 
>> On 30 Oct 2012, at 13:21 , Paul Aitken wrote:
>> 
>>> Gerhard,
>>> 
>>> I agree with your definitions. Thanks for clarifying.
>>> 
>>> So what's the next step?
>>> 
>>>   * Update the IANA definitions?
>>>   * Add clarifications to the WG documents?
>>> 
>>> 
>>> Do we need a short presentation at the upcoming WG meeting?
>>> 
>>> P.
>>> 
>>> 
>>> On 30/10/12 11:51, Gerhard Muenz wrote:
>>>> Paul,
>>>> 
>>>> On 26.10.2012 20:20, Paul Aitken wrote:
>>>>> Andrew, Gerhard,
>>>>> 
>>>>>>> My understanding is that both, deltaCounts and totalCounts contain 
the number of packets or octets observed in the indicated time interval. 
So, for identical flowStart* and flowEnd* timestamps, the values are the 
same.
>>>>>> This is my understanding as well.
>>>>> 
>>>>> There has to be a difference between delta and total counts, else we
>>>>> wouldn't have both of them!
>>>>> 
>>>>> Suppose we have a permanent cache, so the cache entries never 
expire.
>>>>> 
>>>>> For a new flow starting at t0 with a first export at t1, the
>>>>> timestamps, delta, and total counts are the same.
>>>>> 
>>>>> However with the second export at t2, the total and delta counts are
>>>>> different although their timestamps match (they'll both say, "t0 to
>>>>> t2").
>>>> 
>>>> No, this would contradict the new definition of flowStart* we are 
just discussing.
>>>> If delta counts are exported for the interval (t1,t2), then 
flowStart* is t1.
>>>> If delta counts are exported for the interval (t0,t2), then 
flowStart* is t0.
>>>> If total counts are exported, flowStart is always t0.
>>>> These statements hold regardless of which type of cache is used by 
the Metering Process. In general, the information model does not care 
about how the cache is implemented. The exported information just must 
follow the IE definition.
>>>> 
>>>>> 
>>>>> With the traditional (non-permanent) cache, the entry would probably
>>>>> have been removed at t1 and re-created on a subsequent packet, so at
>>>>> t2 the delta and total counts would both be equal. However it'd be
>>>>> incorrect to report the total count, because that's defined as the
>>>>> total number of packets or bytes ..."since the Metering Process
>>>>> (re-)initialization for this Observation Point".
>>>> 
>>>> You must not export total counters in this case because you reset 
counters before re-initialization of the Metering Process.
>>>> 
>>>> Thanks,
>>>> Gerhard
>>>> 
>>>>> 
>>>>> 
>>>>>>> However, the description of totalCounts says that you report the 
number of packets or octets observed for this Flow since 
re-initialization. So, you must never reset the counter for this Flow, 
even after observing a FIN or RST.
>>>>>>> If you reset flow counters, or if you remove Flows from your 
Cache, you cannot use totalCounts any more unless you re-initialize the 
Metering Process (e.g. after flushing the entire permanent Cache).
>>>>>> 
>>>>>> I can try some tests later, but from what I have seen (and been 
told) many totals being exported are in fact just a delta sent once at the 
end of the flow.  If a later flow had the same IPs, protocol, and ports as 
an earlier flow I'm pretty sure a new start time will be sent rather than 
the the first time that flow was seen since reinitializing the metering 
process.
>>>>> 
>>>>> So the MP uses a traditional (non-permanent) cache. In RFC 6728
>>>>> terms, a TimeoutCache or NaturalCache rather than a PermanentCache.
>>>>> 
>>>>> 
>>>>>> Or to put it an other way I think deltas are being sent, but called 
totals by the implementation because it seemed like the right thing to do 
for a value being sent once at the end of the flow.
>>>>> 
>>>>> The collector could be aggregating deltas to keep running totals.
>>>>> 
>>>>> 
>>>>>> I suspect that totals reporting on the export process (eg 
exportedOctetTotalCount, exportedMessageTotalCount) are, however, reported 
with a start time that is only reset on reinitialization.
>>>>> 
>>>>> Definitely, because these are defined as "The total number of X that
>>>>> the Exporting Process has sent since the Exporting Process
>>>>> (re-)initialization ...".
>>>>> 
>>>>> P.
>>> 
>>> _______________________________________________
>>> IPFIX mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Thu, 1 Nov 2012 07:22:37 +0100

Hi, John,

Thanks, but neither of these work, really. The former must make reference to both timestamps, and the latter doesn't account for the possibility that an MP may either fail to observe a packet, decide not to include a packet for its own reasons, or be dealing with observations that aren't really packets at all.

Cheers,

Brian

On Oct 31, 2012, at 11:01 PM, John Court wrote:

> Damn, 
> 
> Sorry about the previous message, when I re-read it the third time I realised that the positive allows the confusion of failing to clearly indicate what "isn't" included. 
> 
> If this Information Element is present in a Flow Record, Flow properties in this Flow Record MUST account for ALL packets observed after/before this timestamp. 
> 
> 
> John Court
> 
> 
> Cc:        [email-masked], Gerhard Muenz <[email-masked]>, Paul Aitken <[email-masked]> 
> Date:        01/11/2012 07:37 
> Personally I always find negative logic requires a double read to make sure I understand it.  Whats wrong with the positive logic equivalent ? 
> 
> >>If this Information Element is present in a Flow Record, Flow properties in this Flow Record do not refer to packets observed before/after this >>timestamp. 
> 
> Becomes : 
> 
> If this Information Element is present in a Flow Record, Flow properties in this Flow Record ONLY refer to packets observed after/before this timestamp. 
> 
> Thanks 
> 
> 
> John Court
> 
> 
> 
> 
> 
> Cc:        Paul Aitken <[email-masked]>, John Court/Australia/IBM@IBMAU, <[email-masked]> 
> Date:        31/10/2012 18:17 
> Thought about this a bit too, and I can't come up anything better than the negative clarification. I might consider changing "preceding" and "succeeding" to "before" and "after" to simplify the language for non-native speakers, and stick an "observed" in there to make clear we're talking about observation time:
> 
> If this Information Element is present in a Flow Record, Flow properties in this Flow Record do not refer to packets observed before/after this timestamp.
> 
> It either case, it does make things a lot more explicitly clear, so I'd definitely add it to the timestamp/duration IEs.
> 
> Thanks, cheers,
> 
> Brian
> 
> > I scanned through the IANA registry and briefly checked occurrences of "this Flow". I do not think that we need change them in general, except for those IEs which define a measurement interval.
> > 
> > So, we have:
> > My first shot was the following change for absolute flowStart*/flowEnd* timestamps:
> > 
> > OLD:
> > The absolute timestamp of the first|last packet accounted in this Flow Record.
> > 
> > We could add a sentence clarifying that the Flow properties reported in this Flow Record refer to the given measurement interval.
> > "If this Information Element is present in a Flow Record, Flow properties in this Flow Record do not refer to packets preceding|succeeding this timestamp."
> > 
> >>>>>>> My understanding is that both, deltaCounts and totalCounts contain the number of packets or octets observed in the indicated time interval. So, for identical flowStart* and flowEnd* timestamps, the values are the same.
> >>>>> Suppose we have a permanent cache, so the cache entries never expire.
> >>>>> 
> >>>> No, this would contradict the new definition of flowStart* we are just discussing.
> >>>> If delta counts are exported for the interval (t1,t2), then flowStart* is t1.
> >>>> If delta counts are exported for the interval (t0,t2), then flowStart* is t0.
> >>>> These statements hold regardless of which type of cache is used by the Metering Process. In general, the information model does not care about how the cache is implemented. The exported information just must follow the IE definition.
> >>>> 
> >>>>> 
> >>>> You must not export total counters in this case because you reset counters before re-initialization of the Metering Process.
> >>>> 
> >>>> Thanks,
> >>>> Gerhard
> >>>> 
> >>>>> 
> >>>>> 
> >>>>>>> However, the description of totalCounts says that you report the number of packets or octets observed for this Flow since re-initialization. So, you must never reset the counter for this Flow, even after observing a FIN or RST.
> >>>>>>> If you reset flow counters, or if you remove Flows from your Cache, you cannot use totalCounts any more unless you re-initialize the Metering Process (e.g. after flushing the entire permanent Cache).
> >>>>>> 
> >>>>>> I can try some tests later, but from what I have seen (and been told) many totals being exported are in fact just a delta sent once at the end of the flow.  If a later flow had the same IPs, protocol, and ports as an earlier flow I'm pretty sure a new start time will be sent rather than the the first time that flow was seen since reinitializing the metering process.
> >>>>> 
> >>>>>> Or to put it an other way I think deltas are being sent, but called totals by the implementation because it seemed like the right thing to do for a value being sent once at the end of the flow.
> >>>>> 
> >>>>>> I suspect that totals reporting on the export process (eg exportedOctetTotalCount, exportedMessageTotalCount) are, however, reported with a start time that is only reset on reinitialization.
> >>>>>