Subject: RE: [ipfix] bytes and packet counters, tstamp of last report

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 8 Oct 2003 15:45:44 -0400

Hi,

  In some use cases counters may have desirable properties as Tal
has described.  But like many things, counters also have their 
limitations and downsides.

  In the basic Flow Export scenario which is exemplified by existing
Netflow implementations, a flow record indicates a summary of
observed behavior.

  Consider two flow events:

      sourceAddress=1.2.3.4
      destinationAddress=1.2.3.5
      packetCount=1000
      byteCount=100000
      flowCreationTime=2003-10-08T10:00:00Z
      flowEndTime=2003-10-08T10:00:05Z
      TcpControlBits=0x13

      sourceAddress=1.2.3.4
      destinationAddress=1.2.3.6
      packetCount=1000
      byteCount=200000
      flowCreationTime=2003-10-08T10:00:02Z
      flowEndTime=2003-10-08T10:00:06Z
      TcpControlBits=0x13

  Each of these completely describes a conversation, as it includes both
  the fin and syn TCP flags.  Start time, end time and absolute counts.

  Given the large percentage of flows which when exported, describe a
  complete conversation, I don't see the advantage of using counters.

  I'm not even sure how one would propose the use of counters as an
  alternative.  Would you require two records be sent for each
  flow (one with byteCounter and packetCounter=0 and another with
  the final value?)

  Since most flows can be represented by a single record, doubling
  the number of records to accomodate counters seems a bit odd.
  Counters also require the holding of state, which produces
  a much greater burden on the collector.  Since the necessary
  state must be held by the observer, why double the workload?
  I.e. a single   counter is pretty useless.  I need to delta it
  against a previously observed matching counter to arrive at a
  quantity.

  For information models such as PSAMP, there may be a compelling
  use case.

  So I imagine adding an explicit annotation in the information
  model for counters would be worthwile.  However, I think that
  counter behavior such as when it is zeroed out and its rollover
  behavior need to be appropriately documented and thought out.
  (e.g. on restart, the counters presumably reset to zero, so
  here is additional state a collector may need to monitor to
  reconcile (if possible) existing observations).

  For IPFIX's base information model, I would strongly discourage
  "fixing" something which isn't broken.

Regards,

  Jeff Meyer

_________________________________________________________________
Get MSN 8 Dial-up Internet Service FREE for one month.  Limited time offer--

sign up now!   http://join.msn.com/?page=dept/dialup

> -----Original Message-----
> From: majordomo listserver 
> [mailto:[email-masked]]On Behalf
> Of Tal Givoly
> Sent: Wednesday, October 08, 2003 9:14 AM
> To: [email-masked]; Maurizio Molina
> Cc: [email-masked]
> Subject: RE: [ipfix] bytes and packet counters, tstamp of last report
> 
> 
> Both counters have value (running and delta). Obviously, they 
> must be kept
> discrete from an information model perspective as they 
> represent different
> information. Devices sometimes maintain one, sometimes the other, and
> sometimes both. Just as an example, as far as I recall, 
> NetFlow v5-8 doesn't
> maintain running counters (counter to your suggestion) EVEN THOUGH it
> operates over a non-reliable and non-congestion-aware 
> transport. Our probe
> product emits both for various different purposes.
> 
> Tal
> 
> -----Original Message-----
> From: majordomo listserver 
> [mailto:[email-masked]]On Behalf
> Of Stewart Bryant
> Sent: Wednesday, October 08, 2003 8:09 AM
> To: Maurizio Molina
> Cc: [email-masked]'
> Subject: Re: [ipfix] bytes and packet counters, tstamp of last report
> 
> 
> 
> 
> Maurizio Molina wrote:
> 
> > Hi,
> > the IPFIX info model states (sec. 6.10) that
> >
> >    The packet count can be a running counter and is the 
> count from the
> >   beginning of the flow establishment.
> >
> >   The packet count can be a delta counter and is the count since the
> >   last report for this flow.
> >
> 
> I would prefer that we only supported running counters because:
> 
> a) Supporting two types is more scope for non-interworking, and
>     you can always convert from one to the other at the collector.
> 
> b) Running counters also work over an unreliable transport
>     whereas delta counters mandate the use of a reliable transport.
> 
> c) Running counters are most likely what the hardware is keeping
>     anyway, therefore delta counters are more work and more storage
>     at the exporter.
> 
> If we decide that we need both then we have to represent them
> as two different information elements because the info model does
> not support sub-typing.
> 
> Stewart
> 
> > (There's the same statement for byte counts in 6.11).
> >
> > To me, this doesn't clarify wheter it is both...and  (2 counters) or
> > either .... or (1 counter).
> >
> > Moreover, there is currently no room for a field containing the
> > timestamp of the last report of a flow. I think that keeping this
> > timestamp is helpful for many applications and the info model should
> > support it.
> > Regards,
> > Maurizio
> >
> >
> >
> > --
> > Help        mailto:[email-masked] and say 
> "help" in message
> > body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> >
> >
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message
> body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Maurizio Molina <[email-masked]>
Date: Thu, 09 Oct 2003 11:28:40 +0200

Jeff,
I posted my original message because I think that the interpretation of 
an information model should be univoque, and therefore a statement like 
"this counter can be this or that..." doesn't make sense (and in this 
respect, I think that the info model is actually broken).
My guess is that this "ambiguity" was put on purpose waiting for an 
agreement about what the counters should really reperesent (absolute or 
delta).
 From the replies on the ML, it seems that this is still a subject of 
discussion.
Therefore, I think the correct way to do is to provide support, in the 
information model, for both type of counters as separate information 
elements (because sub-typing s not allowed, as Stewart pointed out). 
Then the discussion on the merits of  absolute and delta counters should 
turn out in indications (in the protocol specification) about whether 
these counters SHOULD, MAY, MUST, etc.... be exported.
Ah, BTW, there's also the same issue for the droppedPacketCount (6.22) 
and droppedByteCount (6.26).
Regards,
Maurizio

---

From: Benoit Claise <[email-masked]>
Date: Fri, 10 Oct 2003 13:10:54 +0200

Hi,

>  "fixing" something which isn't broken.
I fully agree with Jeff here regarding the flow records but there are 
actually 2 aspects to this problem.

1.
The flow record.
There is an expiration process for the flow records.
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-protocol-00.txt

4.1 Flow Expiration 
    
   A Flow is considered to be inactive if no packets belonging to the 
   Flow have been observed at the Observation Point for a given timeout 
   interval otherwise it is considered as an active flow.  
   A Flow can be exported under the following conditions: 
    
      1. If the Metering Process can detect the end of a Flow, it 
      SHOULD export the Flow Records at the end of the Flow. For 
      example, a Flow generated by TCP [3] type of traffic where the 
      FIN or RST bits indicate the end of the Flow. 
       
      2. If the Flow has been inactive for a certain period of time. 
      This inactivity timeout SHOULD be configurable, with a minimum 
      value of 0 for an immediate expiration. For example, a Flow 
      generated by UDP [2] type of traffic. 
       
      3. For long-lasting Flows, the Metering Process SHOULD export the 
      Flow Records on a regular basis. This periodicity SHOULD be 
      configurable. 
       
      4. If the Metering Process experiences internal constraints, a 
      Flow MAY be forced to expire prematurely (for example, counters 
      wrapping or low memory). 

What the IPFIX information model tries to say is that: a flow expires and we send the packet/byte count for the flow duration. So this is actually a running counter for the flow duration.
Note: a long last flow is actually composed a several flows (see condition 3 before). 
Ok, I understand that this could be better explained in the IPFIX information model draft:
   The packet count can be a running counter and is the count from the
   beginning of the flow establishment.
   The packet count can be a delta counter and is the count since the
   last report for this flow.

http://www.ietf.org/internet-drafts/draft-claise-netflow-9-05.txt speaks of:
                                            Incoming counter with  
    IN_BYTES                     1    N     length N x 8 bits for bytes  
                                            associated with an IP Flow    
 
                                            Incoming counter with  
    IN_PKTS                      2    N     length N x 8 bits for  
                                            packets associated 
                                            with an IP Flow  

2. 
The second aspect is actually total counters.
Something that needs to added to the IPFIX protocol draft is described in its "open issues" section:
   - The proposal on the table is to send a IPFIX Sync (this would be 
   an Options Data Records) message periodically (periodicity is 
   configurable), with the following information (aside the standard 
   IPFix header) 
           * Number of flow records sent (for each template?)  
           * Packets and bytes sent (for each template?) 

So we need to keep a few running counters.

In http://www.ietf.org/internet-drafts/draft-claise-netflow-9-05.txt, there are explicitely described:
                                            Counter with length   
                                            N x 8 bits for bytes 
    TOTAL_BYTES_EXP              40   N     for the number of bytes  
                                            exported by the Observation 
                                            Domain 
 
                                            Counter with length  
                                            N x 8 bits for bytes  
    TOTAL_EXP_PKTS_SENT          41   N     for the number of packets  
                                            exported by the Observation  
                                            Domain  
 
                                            Counter with length  
                                            N x 8 bits for bytes  
    TOTAL_FLOWS_EXP              42   N     for the number of Flows  
                                            exported by the Observation  
                                            Domain  

I propose to also explicitely described them in the IPFIX information 
model draft.

Regards, Benoit.