Subject: Re: [IPFIX] Export of long lived flow information

From: John Court <[email-masked]>
Date: Wed, 24 Oct 2012 07:59:32 +1000

Thanks for the interest in resolving this,

Andrew, I understand your persistent cache argument.  The reason I don't 
personally use "Total" and DO use "Delta"  is more around what the 
collector does should it not see some of the updates and suddenly gets one 
that shows large counter values.  This could mistakenly result in showing 
huge traffic over a short period incorrectly.  This is particularly true 
if you do as Brian suggested and are setting the flowStartTime based only 
on the current record view.

Brian, thanks for your detailed explanation.  Everything with the 
exception of flowStartTime was as I am currently doing.  I had perhaps 
mistakenly taken the approach that keeping the flowStartTime as the 
"conceptual" start rather than for this reporting period would make it 
easier for the collector to understand what was happening.  Not actually 
sure how I will co-ordinate reseting that time on export yet :-)

I would suggest that this sort of situation and Brians explanation be 
added as an example perhaps in 5101 or even 5472 as it would help with 
Collector interoperability I am sure :-)

Thanks again.

John Court
Software Engineer
IBM Security Systems Division
IBM Australia Development Laboratory
Office:  +61 7 5552 4014
Mobile: +61 430 841328

---

From: Brian Trammell <[email-masked]>
Date: Wed, 24 Oct 2012 07:19:34 +0200

Hi, John,

I'll say here that I've never really understood the arguments for export from persistent caches, and don't recommend the approach unless you have other overriding application requirements. But that's merely my opinion, and other people seem to like persistent caches and running total export (especially router people, who need to keep the cache around for other reasons anyway), and I guess it's just a sort of point of view thing.

My problem with export from persistent caches with updating totals is that if you want to do anything with the data other than drive per-flow or per-aggregate displays (a la rrdtool or MRTG), you need to do the same amount of postprocessing as you would with nonpersistent export with Ta equal to your desired export interval in order to get "real flow data", and you lose packet and byte counters per active export interval. When each subpart of the flow has accurate first/last timestamps, you get a time series which can give you some idea of the evolution of the packet and byte rates (is it regular i.e. bulk transfer? is its rate slowly variable i.e. bulk transfer under congestion control? is it a low rate highly variable flow? and so on). With persistent cache export you kind of have the same thing, but (1) you have to subtract to get it and (2) you have no accurate midpoint timestamp; you can only deduce it from the export time and your best guess about Tde and Tdc.

I'm not sure what you mean about "how to coordinate resetting (start time) on export"... if you're using non-persistent export, you expire the flow completely out of the cache on active timeout (with flowEndReason activeTimeout) and start a new flow record when you see the first (continuation) packet. If you want to keep ancillary information about the flow (e.g. deduced AS information if you're not a packet-forwarding device, other private labeling information that is expensive to calculate or requires examination of leading payload), you can keep it around in a secondary cache to be resurrected by flow key and associated with the new flow record when the continuation packet comes.

As 5101bis is still open, I'll see if I can come up with some suitable language on the meaning of timestamps therefor (without the editorializing on persistent export), and propose to the list.

Best regards,

Brian

On Oct 23, 2012, at 11:59 PM, John Court <[email-masked]> wrote:

> Andrew, I understand your persistent cache argument.  The reason I don't personally use "Total" and DO use "Delta"  is more around what the collector does should it not see some of the updates and suddenly gets one that shows large counter values.  This could mistakenly result in showing huge traffic over a short period incorrectly.  This is particularly true if you do as Brian suggested and are setting the flowStartTime based only on the current record view. 
> 
> Brian, thanks for your detailed explanation.  Everything with the exception of flowStartTime was as I am currently doing.  I had perhaps mistakenly taken the approach that keeping the flowStartTime as the "conceptual" start rather than for this reporting period would make it easier for the collector to understand what was happening.  Not actually sure how I will co-ordinate reseting that time on export yet :-) 
> 
> I would suggest that this sort of situation and Brians explanation be added as an example perhaps in 5101 or even 5472 as it would help with Collector interoperability I am sure :-) 
> 
> Thanks again. 
> 
> 
> John Court
> Software Engineer
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: John Court <[email-masked]>
Date: Wed, 24 Oct 2012 16:00:58 +1000

Hi Brian,

I suspect I have been mis-interpreting your concept of "persistent 
caches".  From the way you describe it, I would categorise what I export 
from as a persistent cache.  The reason being that for packet processing 
efficiency reasons, all the "connection" concepts are kept in one place 
and that includes the accounting (i.e. packets, octets).  I only get to 
see these periodically or on connection close. In IPFIX terminology the 
meter isn't resetting the flowStartTime or doing actual flow termination 
outside of detection of "connection" termination.  Perhaps it could but 
that time may be used in other processing, I will have to see. 

I am not arguing with any of your points.  I see the logic behind the 
operation, I just need to adjust some outputs for long lived connections. 

Thanks

John Court

From:   Brian Trammell <[email-masked]>
To:     John Court/Australia/IBM@IBMAU, 
Cc:     [email-masked]
Date:   24/10/2012 15:19
Subject:        Re: [IPFIX] Export of long lived flow information

Hi, John,

I'll say here that I've never really understood the arguments for export 
from persistent caches, and don't recommend the approach unless you have 
other overriding application requirements. But that's merely my opinion, 
and other people seem to like persistent caches and running total export 
(especially router people, who need to keep the cache around for other 
reasons anyway), and I guess it's just a sort of point of view thing.

My problem with export from persistent caches with updating totals is that 
if you want to do anything with the data other than drive per-flow or 
per-aggregate displays (a la rrdtool or MRTG), you need to do the same 
amount of postprocessing as you would with nonpersistent export with Ta 
equal to your desired export interval in order to get "real flow data", 
and you lose packet and byte counters per active export interval. When 
each subpart of the flow has accurate first/last timestamps, you get a 
time series which can give you some idea of the evolution of the packet 
and byte rates (is it regular i.e. bulk transfer? is its rate slowly 
variable i.e. bulk transfer under congestion control? is it a low rate 
highly variable flow? and so on). With persistent cache export you kind of 
have the same thing, but (1) you have to subtract to get it and (2) you 
have no accurate midpoint timestamp; you can only deduce it from the 
export time and your best guess about Tde and Tdc.

I'm not sure what you mean about "how to coordinate resetting (start time) 
on export"... if you're using non-persistent export, you expire the flow 
completely out of the cache on active timeout (with flowEndReason 
activeTimeout) and start a new flow record when you see the first 
(continuation) packet. If you want to keep ancillary information about the 
flow (e.g. deduced AS information if you're not a packet-forwarding 
device, other private labeling information that is expensive to calculate 
or requires examination of leading payload), you can keep it around in a 
secondary cache to be resurrected by flow key and associated with the new 
flow record when the continuation packet comes.

As 5101bis is still open, I'll see if I can come up with some suitable 
language on the meaning of timestamps therefor (without the editorializing 
on persistent export), and propose to the list.

Best regards,

Brian

Thanks again. 

John Court
Software Engineer

---

From: Brian Trammell <[email-masked]>
Date: Wed, 24 Oct 2012 09:13:20 +0200

Hi, John,

Okay, I understand now... I wasn't really responding directly to you on this point, just to the general thread on timestamps in MPs with persistent-cache export, and delta versus total counters.

In your case, if you've _already got_ a persistent cache for other reasons, then by all means use it. :) In that case you just need an extra timestamp, to keep track of the time of the first packet observed after the last export, in addition to the first "real" packet, if you want to export "complete" (delta) flows.

Best regards,

Brian

On Oct 24, 2012, at 8:00 AM, John Court <[email-masked]> wrote:

> Hi Brian, 
> 
> I suspect I have been mis-interpreting your concept of "persistent caches".  From the way you describe it, I would categorise what I export from as a persistent cache.  The reason being that for packet processing efficiency reasons, all the "connection" concepts are kept in one place and that includes the accounting (i.e. packets, octets).  I only get to see these periodically or on connection close. In IPFIX terminology the meter isn't resetting the flowStartTime or doing actual flow termination outside of detection of "connection" termination.  Perhaps it could but that time may be used in other processing, I will have to see.   
> 
> I am not arguing with any of your points.  I see the logic behind the operation, I just need to adjust some outputs for long lived connections.   
> 
> Thanks 
> 
> John Court
> 
> 
> 
>

---

From: Paul Aitken <[email-masked]>
Date: Wed, 24 Oct 2012 10:39:45 +0100

Brian,

> In your case, if you've _already got_ a persistent cache for other 
> reasons, then by all means use it. :) In that case you just need an 
> extra timestamp, to keep track of the time of the first packet 
> observed after the last export, in addition to the first "real" 
> packet, if you want to export "complete" (delta) flows.

A collector can obtain delta counts from a series of total counts.

It may be a little inaccurate if observations were not evenly 
distributed throughout the interval between total count reports. However 
this may not matter if the collector is simply reporting aggregates or 
averages within an interval.

eg, if the collector produces hourly reports, it can simply calculate 
the delta between the current packetTotalCount and the packetTotalCount 
an hour ago, without being concerned whether the packets arrived at the 
beginning of the hour, throughout the hour, or at the end of the hour.

P.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 24 Oct 2012 10:48:27 +0100

John,

> caches".

In a normal cache, the entries are eventually removed - because they've 
ended, or they've not seen traffic for an amount of time, or they're 
just too old, or there's simply not enough room in the cache.

Whereas in a permanent cache, the entries are never removed.

P.

---

From: John Court <[email-masked]>
Date: Thu, 25 Oct 2012 06:28:43 +1000

Just to be crystal clear on this point of persistent caches.  Even when 
sending "totalCount" fields, the flowStartTime is still relative to the 
current flow record, it doesn't represent the "original" first packet ever 
seen for the flow key in the cache ?  I just want to make sure of the 
semantics of flowStartTime in all cases.

Thanks again for the comments and clarifications
 
John Court
Software Engineer

From:   Paul Aitken <[email-masked]>
To:     John Court/Australia/IBM@IBMAU, 
Cc:     Brian Trammell <[email-masked]>, [email-masked]
Date:   24/10/2012 19:49
Subject:        Re: [IPFIX] Export of long lived flow information

John,

In a normal cache, the entries are eventually removed - because they've 
ended, or they've not seen traffic for an amount of time, or they're just 
too old, or there's simply not enough room in the cache.

P.

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 24 Oct 2012 22:35:03 +0200

Hi,

flowStartTime is the time of the first packet you are reporting on in 
the given record. So, including totalCount fields implicitly means that 
flowStartTime is the time of the very first packet ever observed for 
this flow.

Now, you can continue discussing what happens if both totalCounts and 
deltaCounts are included in the same record :)

Regards,
Gerhard

On 24.10.2012 22:28, John Court wrote:
> current flow record, it doesn't represent the "original" first packet
> ever seen for the flow key in the cache ?  I just want to make sure of
> the semantics of flowStartTime in all cases.
>
> ------------------------------------------------------------------------
>
>
>
> John,
>
> I suspect I have been mis-interpreting your concept of "persistent caches".
>

---

From: Paul Aitken <[email-masked]>
Date: Wed, 24 Oct 2012 22:19:05 +0100

John, Gerhard,

Looking at the definitions:

octetTotalCount:

          The total number of octets in incoming packets
          for this Flow at the Observation Point since the Metering
          Process (re-)initialization for this Observation Point.

packetTotalCount:

          The total number of incoming packets for this Flow
          at the Observation Point since the Metering Process
          (re-)initialization for this Observation Point.

So even if there's a long gap in the traffic, the intention is that a MP 
which is reporting totalCount fields remembers that it saw the flow before.

So +1 to what Gerhard says: "flowStartTime is the time of the very first 
packet ever observed for this flow." (ie, the "original" first packet).

Whereas when reporting deltaCount fields, the MP may "forget" about the 
flow (by purging the cache entry) - so after the traffic gap the flow 
must start over as if it's entirely new, since the MP retains no 
history. Therefore the flowStartTime must be the first packet in the new 
delta - though, the collector might choose to aggregate this with 
previous deltas, setting the flowStartTime to the earliest reported and 
the flowEndTime to the latest.

This is undoubtedly something that we should capture in one of our 
updated docs.

Thanks for the great question John!

P.

On 24/10/12 21:35, Gerhard Muenz wrote:
> the given record. So, including totalCount fields implicitly means 
> that flowStartTime is the time of the very first packet ever observed 
> for this flow.
>

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 24 Oct 2012 22:45:42 +0100

Hi all

Paul, thanks for the terminology correction, I did mean "total" counters and was confusing people with "updates".

I was thinking that a mechanism that allowed a non-permanent flow to be exported multiple time would be useful.  For example, security applications generally want to know about a new flow ASAP, so they can act on the information, but a short active timeout values lead to using more export bandwidth.  I was thinking we could do something like export a report of the flow after the first packet, and then export the final version of the flow once the normal timeouts had decided it was over.

I had in mind something like using a delta count, followed by a total count.  Reading the below definition of Total counts though, I'm not sure that will work, but I think it depends on how we interpret the definition of "Flow".  If two records have matching key fields but different starting timestamps, are they the same Flow?

I would argue that a single Flow can't have two flowStartTimes, so maybe not.  This would mean that we shouldn't reset the flowStartTimes between sending reports for the same permanent Flow.

Cheers, Andrew

On 24 Oct 2012, at 22:19, Paul Aitken wrote:
> So even if there's a long gap in the traffic, the intention is that a MP which is reporting totalCount fields remembers that it saw the flow before.
> 
> So +1 to what Gerhard says: "flowStartTime is the time of the very first packet ever observed for this flow." (ie, the "original" first packet).
> 
> 
> Whereas when reporting deltaCount fields, the MP may "forget" about the flow (by purging the cache entry) - so after the traffic gap the flow must start over as if it's entirely new, since the MP retains no history. Therefore the flowStartTime must be the first packet in the new delta - though, the collector might choose to aggregate this with previous deltas, setting the flowStartTime to the earliest reported and the flowEndTime to the latest.
> 
> This is undoubtedly something that we should capture in one of our updated docs.
> 
>> flowStartTime is the time of the first packet you are reporting on in the given record. So, including totalCount fields implicitly means that flowStartTime is the time of the very first packet ever observed for this flow.
>> 
>> Now, you can continue discussing what happens if both totalCounts and deltaCounts are included in the same record :)
>> 
>> Regards,
>> Gerhard
>> 
>>

---

From: Paul Aitken <[email-masked]>
Date: Wed, 24 Oct 2012 23:06:39 +0100

Andrew,

> I was thinking that a mechanism that allowed a non-permanent flow to be exported multiple time would be useful.  For example, security applications generally want to know about a new flow ASAP, so…

I have in the past discussed the idea of exporting a "new flow alert" 
using zero-valued counters in order to make the collector aware that 
we've started monitoring it - so I'm claiming prior art on that.

> I had in mind something like using a delta count, followed by a total count.  Reading the below definition of Total counts though, I'm not sure that will work, but I think it depends on how we inte…

5101 defines:

       A Flow is defined as a set of IP packets passing an Observation
       Point in the network during a certain time interval.

- so it's all about the timestamps :-)

> I would argue that a single Flow can't have two flowStartTimes, so maybe not.

However, two flows with different flowStartTimes can be merged into one 
flow.

> This would mean that we shouldn't reset the flowStartTimes between sending reports for the same permanent Flow.

Definitely. If it's a permanent flow and you're exporting totalCount 
fields - which are measured "since the Metering Process 
(re-)initialization for this Observation Point" - then the flowStartTime 
must surely be the time the first ever packet was observed.

P.

---

From: John Court <[email-masked]>
Date: Thu, 25 Oct 2012 11:04:17 +1000

Yep I think everyone is starting to see the ambiguity that needs to be 
cleared up :-)

Paul,

If you take that literally shouldn't that be interpreted to mean that the 
totalCount continues into the next time a connection is up between the 
same flow key ?  Even if a flowEndReason of :

 0x03: end of Flow detected
The Flow was terminated because the Metering Process
detected signals indicating the end of the Flow, for
example, the TCP FIN flag.

That clearly wouldn't be of much use IMO and makes it difficult to see 
what the flowEndReason field semantics mean in that context.  Just 
pointing out that taking that definition literally doesn't give a useful 
answer on its own either :-).  Although maybe that does make sense in a 
router context ?  Can you clarify this some more, perhaps you never intend 
using the flowEndReason IE in your case ?

Thanks

From:   Paul Aitken <[email-masked]>
To:     Andrew Johnson <[email-masked]>, 
Cc:     Gerhard Muenz <[email-masked]>, John 
Court/Australia/IBM@IBMAU, [email-masked]
Date:   25/10/2012 08:07
Subject:        Re: [IPFIX] Export of long lived flow information

Andrew,

> I was thinking that a mechanism that allowed a non-permanent flow to be 
exported multiple time would be useful.  For example, security 
applications generally want to know about a new flow ASAP, so they can act 
on the information, but a short active timeout values lead to using more 
export bandwidth.  I was thinking we could do something like export a 
report of the flow after the first packet, and then export the final 
version of the flow once the normal timeouts had decided it was over.

> I had in mind something like using a delta count, followed by a total 
count.  Reading the below definition of Total counts though, I'm not sure 
that will work, but I think it depends on how we interpret the definition 
of "Flow".  If two records have matching key fields but different starting 
timestamps, are they the same Flow?

5101 defines:

> This would mean that we shouldn't reset the flowStartTimes between 
sending reports for the same permanent Flow.

P.

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 25 Oct 2012 20:26:28 +0200

Hi,

It seems that some Information Element descriptions need clarification. 
For example, I think that the description of the Information Elements 
flowStart* and flowEnd* should be clarified:

OLD:
The absolute timestamp of the first|last packet of this Flow.
NEW:
The absolute timestamp of the first|last packet accounted in this Flow 
Record.

I assume that this is how it is implemented in most existing 
implementations. Also, I assume that this is the intended meaning.

My understanding is that both, deltaCounts and totalCounts contain the 
number of packets or octets observed in the indicated time interval. So, 
for identical flowStart* and flowEnd* timestamps, the values are the same.

However, the description of totalCounts says that you report the number 
of packets or octets observed for this Flow since re-initialization. So, 
you must never reset the counter for this Flow, even after observing a 
FIN or RST.
If you reset flow counters, or if you remove Flows from your Cache, you 
cannot use totalCounts any more unless you re-initialize the Metering 
Process (e.g. after flushing the entire permanent Cache).

Using totalCounts, the flowStart* timestamp is identical in all Flow 
Records of the same Flow. Also, the collector knows that - for all but 
the first Flow Record of a Flow - the totalCount values include packets 
which were already reported in earlier Flow Records for the same Flow. 
Hence, each new Flow Record of the same Flow is an update of the 
previous ones. Summing up totalCount values in these Flow Records 
results in duplicate counts.
On the other hand, with deltaCounts, the Flow Records refer to distinct 
time intervals. So, you can sum up counters without having duplicates.

Although these subtle differences are not very obvious, the Information 
Element descriptions are quite clear. flowEndReason can be used to 
report some extra information but is not needed to understand the 
meaning of the Flow Records.

Thanks,
Gerhard

On 25.10.2012 03:04, John Court wrote:
> *Definitely. If it's a permanent flow and you're exporting totalCount
> must surely be the time the first ever packet was observed.*
>
> If you take that literally shouldn't that be interpreted to mean that
> the totalCount continues into the next time a connection is up between
> the same flow key ?  Even if a flowEndReason of :
>
> router context ?  Can you clarify this some more, perhaps you never
> intend using the flowEndReason IE in your case ?
>
> Thanks
>
>
>
>
>
> Cc: Gerhard Muenz <[email-masked]>, John Court/Australia/IBM@IBMAU,
> [email-masked]
> Date: 25/10/2012 08:07
>  > I was thinking that a mechanism that allowed a non-permanent flow to
> be exported multiple time would be useful.  For example, security
> applications generally want to know about a new flow ASAP, so they can
> act on the information, but a short active timeout values lead to using
> more export bandwidth.  I was thinking we could do something like export
> a report of the flow after the first packet, and then export the final
> count.  Reading the below definition of Total counts though, I'm not
> sure that will work, but I think it depends on how we interpret the
> definition of "Flow".  If two records have matching key fields but
> different starting timestamps, are they the same Flow?
>
> 5101 defines:
>

---

From: John Court <[email-masked]>
Date: Fri, 26 Oct 2012 07:40:56 +1000

>>The absolute timestamp of the first|last packet accounted in this Flow 

For some reason that does make it much clearer to me although that could 
just be because I have had the benefit of all the preceding discussion.

I think that clears up exactly what the statistics represent in a flow 
record.  The only nastiness is that we use terminology of flowEnd* when 
really if you are using totals, conceptually the flow really ISN'T ending 
its just being "Reported".  I am not suggesting changes to anything just 
hopefully making it clear why the mis-interpretations may occur.

For my own purposes its clear I can only use deltas as my work is at the 
application level and only bridges and routers would seem to make sense 
for the "total" counters concept given your definition. For me the 
aggregation into "totals" is up to the collector.

Again thanks for every ones patience it has certainly helped further my 
education and hopefully improve future interoperability of IPFIX devices 
:-)

John Court

From:   Gerhard Muenz <[email-masked]>
To:     John Court/Australia/IBM@IBMAU, 
Cc:     Paul Aitken <[email-masked]>, Andrew Johnson 
<[email-masked]>, [email-masked]
Date:   26/10/2012 04:27
Subject:        Re: [IPFIX] Export of long lived flow information

Hi,

OLD:

Thanks,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Thu, 25 Oct 2012 23:56:14 +0100

Gerhard,

If everyone's happy with this clarification, then we should ask IANA to 
update the descriptions in the IPFIX registry.

P.

On 25/10/12 19:26, Gerhard Muenz wrote:
>
> Hi,
>
> It seems that some Information Element descriptions need 
> clarification. For example, I think that the description of the 
> Information Elements flowStart* and flowEnd* should be clarified:
>
> OLD:

---

From: Andrew Feren <[email-masked]>
Date: Thu, 25 Oct 2012 20:15:50 -0400

+1
I have tripped over that wording a few times in the past.  The change from
Flow to  Flow Record is an improvement.

-Andrew

---

From: Brian Trammell <[email-masked]>
Date: Fri, 26 Oct 2012 09:27:22 +0200

Another +1.

Cheers, B

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 26 Oct 2012 15:24:42 +0100

Hi folks

A flow is defined as a set of packets which all have some common properties, and the original idea of a flow was based on the common 5-tuple of IP address, IP protocol and ports.  A Flow Record is formed from any observation of packets belonging to a Flow (not all packets within the flow are necessarily observed).

If my PC opens an HTTP connection to some server and downloads a simple web page, then we'd expect to see one traditional flows (per direction, but ignore that for now).  If I open a new HTTP connection to the same server, and coincidentally use the same source port, weeks later, is that the same flow?

I had always thought of these as two flows, and the flowEndReason implies it, but that would introduce some sort of time property as one of the common property shared by the packets that make up a flow.  A collector might aggregate the two reports, but removing the time property is much like any other form of aggregation.

It seems to me that we're using the timeout values of the cache as a sort of ill-defined common property, but things get confusing when we export total counts, or because the cache is low on resources, etc.  Ideally, we'd want to be able to send more than one Flow Record for the same Flow and provide enough information for the Collector to reconstruct what the Monitoring Process is using to define a Flow.

Cheers, Andrew

On 25 Oct 2012, at 22:40, John Court wrote:
> For some reason that does make it much clearer to me although that could just be because I have had the benefit of all the preceding discussion. 
> 
> I think that clears up exactly what the statistics represent in a flow record.  The only nastiness is that we use terminology of flowEnd* when really if you are using totals, conceptually the flow really ISN'T ending its just being "Reported".  I am not suggesting changes to anything just hopefully making it clear why the mis-interpretations may occur. 
> 
> For my own purposes its clear I can only use deltas as my work is at the application level and only bridges and routers would seem to make sense for the "total" counters concept given your definition. For me the aggregation into "totals" is up to the collector. 
> 
> Again thanks for every ones patience it has certainly helped further my education and hopefully improve future interoperability of IPFIX devices :-) 
> 
> John Court 
> 
> 
> 
> 
> Cc:        Paul Aitken <[email-masked]>, Andrew Johnson <[email-masked]>, [email-masked] 
> Date:        26/10/2012 04:27

---

From: Andrew Feren <[email-masked]>
Date: Fri, 26 Oct 2012 12:23:23 -0400

Hi Gerhard,

> Hi,
[ snip ]
> number of packets or octets observed in the indicated time interval. 
> So, for identical flowStart* and flowEnd* timestamps, the values are 
> the same.
This is my understanding as well.
>
> However, the description of totalCounts says that you report the 
> number of packets or octets observed for this Flow since 
> re-initialization. So, you must never reset the counter for this Flow, 
> even after observing a FIN or RST.
> If you reset flow counters, or if you remove Flows from your Cache, 
> you cannot use totalCounts any more unless you re-initialize the 
> Metering Process (e.g. after flushing the entire permanent Cache).

I can try some tests later, but from what I have seen (and been told) 
many totals being exported are in fact just a delta sent once at the end 
of the flow.  If a later flow had the same IPs, protocol, and ports as 
an earlier flow I'm pretty sure a new start time will be sent rather 
than the the first time that flow was seen since reinitializing the 
metering process.  Or to put it an other way I think deltas are being 
sent, but called totals by the implementation because it seemed like the 
right thing to do for a value being sent once at the end of the flow.

I suspect that totals reporting on the export process (eg 
exportedOctetTotalCount, exportedMessageTotalCount) are, however, 
reported with a start time that is only reset on reinitialization.

-Andrew

---

From: Paul Aitken <[email-masked]>
Date: Fri, 26 Oct 2012 20:20:48 +0100

Andrew, Gerhard,

>> My understanding is that both, deltaCounts and totalCounts contain 
>> the number of packets or octets observed in the indicated time 
>> interval. So, for identical flowStart* and flowEnd* timestamps, the 
>> values are the same.

There has to be a difference between delta and total counts, else we 
wouldn't have both of them!

Suppose we have a permanent cache, so the cache entries never expire.

For a new flow starting at t0 with a first export at t1, the timestamps, 
delta, and total counts are the same.

However with the second export at t2, the total and delta counts are 
different although their timestamps match (they'll both say, "t0 to t2").

With the traditional (non-permanent) cache, the entry would probably 
have been removed at t1 and re-created on a subsequent packet, so at t2 
the delta and total counts would both be equal. However it'd be 
incorrect to report the total count, because that's defined as the total 
number of packets or bytes ..."since the Metering Process 
(re-)initialization for this Observation Point".

>> re-initialization. So, you must never reset the counter for this 
>> Flow, even after observing a FIN or RST.
> many totals being exported are in fact just a delta sent once at the 
> end of the flow.  If a later flow had the same IPs, protocol, and 
> ports as an earlier flow I'm pretty sure a new start time will be sent 
> rather than the the first time that flow was seen since reinitializing 
> the metering process.

So the MP uses a traditional (non-permanent) cache. In RFC 6728 terms, a 
TimeoutCache or NaturalCache rather than a PermanentCache.

> Or to put it an other way I think deltas are being sent, but called 
> totals by the implementation because it seemed like the right thing to 
> do for a value being sent once at the end of the flow.

The collector could be aggregating deltas to keep running totals.

> reported with a start time that is only reset on reinitialization.

Definitely, because these are defined as "The total number of X that the 
Exporting Process has sent since the Exporting Process 
(re-)initialization ...".

P.

---

From: Paul Aitken <[email-masked]>
Date: Fri, 26 Oct 2012 21:21:17 +0100

Andrew,

> A flow is defined as a set of packets which all have some common 
> properties, and the original idea of a flow was based on the common 
> 5-tuple of IP address, IP protocol and ports.  A Flow Record is formed 
> from any observation of packets belonging to a Flow (not all packets 
> within the flow are necessarily observed).
>
> If my PC opens an HTTP connection to some server and downloads a 
> simple web page, then we'd expect to see one traditional flows (per 
> direction, but ignore that for now).  If I open a new HTTP connection 
> to the same server, and coincidentally use the same source port, weeks 
> later, is that the same flow?

Per RFC 5101:

So if you define different time intervals (eg, "last week", "this week", 
and "next week") then yes, you've got two different flows there.

Whereas, if you define a interval which encompasses both of your 
connections, then you've only got one flow. And if your flow is going to 
last for weeks, then I hope you're reporting intermediate values.

> I had always thought of these as two flows, and the flowEndReason 
> implies it, but that would introduce some sort of time property as one 
> of the common property shared by the packets that make up a flow.  A 
> collector might aggregate the two reports, but removing the time 
> property is much like any other form of aggregation.

The time property is built in to the RFC 5101 definition.

P.

> It seems to me that we're using the timeout values of the cache as a 
> sort of ill-defined common property, but things get confusing when we 
> export total counts, or because the cache is low on resources, etc. 
>  Ideally, we'd want to be able to send more than one Flow Record for 
> the same Flow and provide enough information for the Collector to 
> reconstruct what the Monitoring Process is using to define a Flow.
>
>
> Cheers, Andrew

---

From: Paul Aitken <[email-masked]>
Date: Mon, 29 Oct 2012 11:56:34 +0000

Should we change "this Flow" to "accounted in this Flow Record" 
throughout the registry?

I counted 67 instances. eg:

octetDeltaCount

          The number of octets since the previous report (if any)
          in incoming packets for this Flow at the Observation Point.

tcpControlBits

          TCP control bits observed for packets of this Flow.
          The information is encoded in a set of bit fields.
          For each TCP control bit, there is a bit in this
          set.  A bit is set to 1 if any observed packet of this
          Flow has the corresponding TCP control bit set to 1.
          A value of 0 for a bit indicates that the corresponding
          bit was not set in any of the observed packets
of this Flow.

flowDurationMilliseconds

          The difference in time between the first observed packet
of this Flow and the last observed packet of this Flow.

ingressInterface

            The index of the IP interface where packets of this Flow
            are being received.

egressInterface

            The index of the IP interface where packets of
            this Flow are being sent.

P.

---

From: Paul Aitken <[email-masked]>
Date: Mon, 29 Oct 2012 12:16:59 +0000

John,

> the same flow key ?

Yes, that's what it says.

> Even if a flowEndReason of :
>

The definitions of *totalCount are independent of flowEndReason.

> pointing out that taking that definition literally doesn't give a 
> useful answer on its own either :-).  Although maybe that does make 
> sense in a router context ?  Can you clarify this some more, perhaps 
> you never intend using the flowEndReason IE in your case ?

We do use flowEndReason. However, flowEndReason doesn't logically apply 
to permanent flows: they're permanent, therefore, they don't end.

P.

---

From: Andrew Feren <[email-masked]>
Date: Mon, 29 Oct 2012 09:08:40 -0400

Hi Paul,

Interesting observation.   The use of flow in these has never tripped me 
up the same way as the text in the original question.

>          in incoming packets for this Flow at the Observation Point.

For this I think Flow is correct.  I also think this highlights the 
distinction being made by people between Flow and Flow Record.  If a 
Flow is bounded the same way (has the same start and end time) as a Flow 
Record then there can't have been a previous report for this Flow.  The 
Flow and Flow Record would be the same.

>          of this Flow and the last observed packet of this Flow.

Eep.  Thinking about this is making me rethink my initial +1, but I do 
still think some rewording is needed.  Using Flow Record works for 
deltas, but not for totals.  I'm not sure yet what the right wording is.

>            this Flow are being sent.

I think these last two are typically part of what defines a flow and 
won't change with out changing the Flow.  So I guess Flow is as good as 
Flow Record.

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 30 Oct 2012 12:51:41 +0100

Paul,

On 26.10.2012 20:20, Paul Aitken wrote:
> For a new flow starting at t0 with a first export at t1, the
> timestamps, delta, and total counts are the same.
>

No, this would contradict the new definition of flowStart* we are just 
discussing.
If delta counts are exported for the interval (t1,t2), then flowStart* 
is t1.
If delta counts are exported for the interval (t0,t2), then flowStart* 
is t0.
If total counts are exported, flowStart is always t0.
These statements hold regardless of which type of cache is used by the 
Metering Process. In general, the information model does not care about 
how the cache is implemented. The exported information just must follow 
the IE definition.

> have been removed at t1 and re-created on a subsequent packet, so at
> t2 the delta and total counts would both be equal. However it'd be
> incorrect to report the total count, because that's defined as the
> total number of packets or bytes ..."since the Metering Process

You must not export total counters in this case because you reset 
counters before re-initialization of the Metering Process.

Thanks,
Gerhard

>> I can try some tests later, but from what I have seen (and been 
>> told) many totals being exported are in fact just a delta sent once at 
>> the end of the flow.  If a later flow had the same IPs, protocol, and 
> So the MP uses a traditional (non-permanent) cache. In RFC 6728
> terms, a TimeoutCache or NaturalCache rather than a PermanentCache.
>
>
> Definitely, because these are defined as "The total number of X that
> the Exporting Process has sent since the Exporting Process

---

From: Paul Aitken <[email-masked]>
Date: Tue, 30 Oct 2012 12:21:33 +0000

Gerhard,

I agree with your definitions. Thanks for clarifying.

So what's the next step?

     * Update the IANA definitions?
     * Add clarifications to the WG documents?

Do we need a short presentation at the upcoming WG meeting?

P.

On 30/10/12 11:51, Gerhard Muenz wrote:
> Metering Process. In general, the information model does not care 
> about how the cache is implemented. The exported information just must 
> follow the IE definition.
>
>>
>>> told) many totals being exported are in fact just a delta sent once 
>>> at the end of the flow.  If a later flow had the same IPs, protocol, 
>>> and ports as an earlier flow I'm pretty sure a new start time will 
>>> be sent rather than the the first time that flow was seen since 
>>> reinitializing the metering process.
>>
>>> totals by the implementation because it seemed like the right thing 
>>> to do for a value being sent once at the end of the flow.
>>

---

From: Brian Trammell <[email-masked]>
Date: Tue, 30 Oct 2012 13:38:54 +0100

Hi, Paul, all,

If someone throws together a quick summary of the position for me I'd be happy make a couple of slides and do the presentation in person in Atlanta.

From where I sit, it looks like we just go ask IANA to make the registry change. Wearing my IE-Doctors-author hat (I can't wear an IE Doctor hat, there aren't any yet. :) ), I'd say this revision would be covered (and permissible) under point 2 in section 5.2 of ie-doctors.

Best regards,

Brian

On 30 Oct 2012, at 13:21 , Paul Aitken wrote:

>>>>> My understanding is that both, deltaCounts and totalCounts contain the number of packets or octets observed in the indicated time interval. So, for identical flowStart* and flowEnd* timestamps, the values are the same.
>> No, this would contradict the new definition of flowStart* we are just discussing.
>> If delta counts are exported for the interval (t1,t2), then flowStart* is t1.
>> If delta counts are exported for the interval (t0,t2), then flowStart* is t0.
>> These statements hold regardless of which type of cache is used by the Metering Process. In general, the information model does not care about how the cache is implemented. The exported information just must follow the IE definition.
>> 
>>> 
>> You must not export total counters in this case because you reset counters before re-initialization of the Metering Process.
>> 
>> Thanks,
>> Gerhard
>> 
>>> 
>>> 
>>>>> However, the description of totalCounts says that you report the number of packets or octets observed for this Flow since re-initialization. So, you must never reset the counter for this Flow, even after observing a FIN or RST.
>>>>> If you reset flow counters, or if you remove Flows from your Cache, you cannot use totalCounts any more unless you re-initialize the Metering Process (e.g. after flushing the entire permanent Cache).
>>>> 
>>>> I can try some tests later, but from what I have seen (and been told) many totals being exported are in fact just a delta sent once at the end of the flow.  If a later flow had the same IPs, protocol, and ports as an earlier flow I'm pretty sure a new start time will be sent rather than the the first time that flow was seen since reinitializing the metering process.
>>> 
>>>> Or to put it an other way I think deltas are being sent, but called totals by the implementation because it seemed like the right thing to do for a value being sent once at the end of the flow.
>>> 
>>>> I suspect that totals reporting on the export process (eg exportedOctetTotalCount, exportedMessageTotalCount) are, however, reported with a start time that is only reset on reinitialization.
>>>

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 31 Oct 2012 08:52:11 +0100

Hi all,

I scanned through the IANA registry and briefly checked occurrences of 
"this Flow". I do not think that we need change them in general, except 
for those IEs which define a measurement interval.

So, we have:
- flowStart* (including flowStartDeltaMicroseconds)
- flowEnd* (including flowEndDeltaMicroseconds)
- flowDuration*

My first shot was the following change for absolute flowStart*/flowEnd* 
timestamps:

OLD:

We could add a sentence clarifying that the Flow properties reported in 
this Flow Record refer to the given measurement interval.
At the moment, I only find a good "negative" way to describe this:
"If this Information Element is present in a Flow Record, Flow 
properties in this Flow Record do not refer to packets 
preceding|succeeding this timestamp."

Do you have better suggestions?
Or do you think that this clarification is not necessary?

Regards,
Gerhard

On 30.10.2012 13:38, Brian Trammell wrote:
> Hi, Paul, all,
>
> If someone throws together a quick summary of the position for me I'd
> be happy make a couple of slides and do the presentation in person in
> Atlanta.
>
> From where I sit, it looks like we just go ask IANA to make the
> registry change. Wearing my IE-Doctors-author hat (I can't wear an IE
> Doctor hat, there aren't any yet. :) ), I'd say this revision would 
> be
> covered (and permissible) under point 2 in section 5.2 of ie-doctors.
>
> Best regards,
>
> Brian
>
>
>>>>>> My understanding is that both, deltaCounts and totalCounts 
>>>>>> contain the number of packets or octets observed in the indicated 
>>>>>> time interval. So, for identical flowStart* and flowEnd* 
>>>>>> timestamps, the values are the same.
>>>> different although their timestamps match (they'll both say, "t0 
>>>> to
>>>> t2").
>>>
>>> These statements hold regardless of which type of cache is used by 
>>> the Metering Process. In general, the information model does not care 
>>> about how the cache is implemented. The exported information just 
>>> must follow the IE definition.
>>>
>>>>
>>>>>> If you reset flow counters, or if you remove Flows from your 
>>>>>> Cache, you cannot use totalCounts any more unless you 
>>>>>> re-initialize the Metering Process (e.g. after flushing the entire 
>>>>>> permanent Cache).
>>>>>
>>>>> at the end of the flow.  If a later flow had the same IPs, 
>>>>> protocol, and ports as an earlier flow I'm pretty sure a new start 
>>>>> time will be sent rather than the the first time that flow was seen 
>>>>> since reinitializing the metering process.
>>>>
>>>>> Or to put it an other way I think deltas are being sent, but 
>>>>> called totals by the implementation because it seemed like the

---

From: Brian Trammell <[email-masked]>
Date: Wed, 31 Oct 2012 09:16:28 +0100

Hi, Gerhard, all,

Thought about this a bit too, and I can't come up anything better than the negative clarification. I might consider changing "preceding" and "succeeding" to "before" and "after" to simplify the language for non-native speakers, and stick an "observed" in there to make clear we're talking about observation time:

If this Information Element is present in a Flow Record, Flow properties in this Flow Record do not refer to packets observed before/after this timestamp.

It either case, it does make things a lot more explicitly clear, so I'd definitely add it to the timestamp/duration IEs.

Thanks, cheers,

Brian

On Oct 31, 2012, at 8:52 AM, Gerhard Muenz wrote:

> 
> Hi all,
> 
> I scanned through the IANA registry and briefly checked occurrences of "this Flow". I do not think that we need change them in general, except for those IEs which define a measurement interval.
> 
> So, we have:
> My first shot was the following change for absolute flowStart*/flowEnd* timestamps:
> 
> OLD:
> The absolute timestamp of the first|last packet accounted in this Flow Record.
> 
> We could add a sentence clarifying that the Flow properties reported in this Flow Record refer to the given measurement interval.
> "If this Information Element is present in a Flow Record, Flow properties in this Flow Record do not refer to packets preceding|succeeding this timestamp."
> 
>> Doctor hat, there aren't any yet. :) ), I'd say this revision would be

---

From: John Court <[email-masked]>
Date: Thu, 1 Nov 2012 07:37:17 +1000

Personally I always find negative logic requires a double read to make 
sure I understand it.  Whats wrong with the positive logic equivalent ?

>>If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record do not refer to packets observed before/after this 
>>timestamp.

Becomes :

If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record ONLY refer to packets observed after/before this 
timestamp.

Thanks

John Court

From:   Brian Trammell <[email-masked]>
To:     Gerhard Muenz <[email-masked]>, 
Cc:     Paul Aitken <[email-masked]>, John Court/Australia/IBM@IBMAU, 
<[email-masked]>
Date:   31/10/2012 18:17
Subject:        Re: [IPFIX] Export of long lived flow information

Hi, Gerhard, all,

Thought about this a bit too, and I can't come up anything better than the 
negative clarification. I might consider changing "preceding" and 
"succeeding" to "before" and "after" to simplify the language for 
non-native speakers, and stick an "observed" in there to make clear we're 
talking about observation time:

If this Information Element is present in a Flow Record, Flow properties 
in this Flow Record do not refer to packets observed before/after this 
timestamp.

It either case, it does make things a lot more explicitly clear, so I'd 
definitely add it to the timestamp/duration IEs.

Thanks, cheers,

Brian

Record.
>>>>>>> My understanding is that both, deltaCounts and totalCounts contain 
the number of packets or octets observed in the indicated time interval. 
So, for identical flowStart* and flowEnd* timestamps, the values are the 
same.
>>>>>>> If you reset flow counters, or if you remove Flows from your 
Cache, you cannot use totalCounts any more unless you re-initialize the 
Metering Process (e.g. after flushing the entire permanent Cache).
>>>>>> I can try some tests later, but from what I have seen (and been 
told) many totals being exported are in fact just a delta sent once at the 
end of the flow.  If a later flow had the same IPs, protocol, and ports as 
an earlier flow I'm pretty sure a new start time will be sent rather than 
the the first time that flow was seen since reinitializing the metering 
process.
>>>>>> Or to put it an other way I think deltas are being sent, but called 
totals by the implementation because it seemed like the right thing to do 
for a value being sent once at the end of the flow.
>>>>>> I suspect that totals reporting on the export process (eg 
exportedOctetTotalCount, exportedMessageTotalCount) are, however, reported 
with a start time that is only reset on reinitialization.