Subject: RE: [ipfix] drawing the line between protocol and data model docu ment

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 11 Jun 2003 13:00:13 -0400

Hi,
 
  From a practical perspective, simply saying that a field is numeric and
may be encoded as 1 or 100 bytes seems like it puts an unnecessary and
unjustified burden on the collector, or any other app which might want to
handle this data.
 
  Is there some particular reason why saying a specific information item,
e.g. numBytes, is a 32-bit integer or a 64-bit integer is considered
undesireable.  It certainly seems like the developer of a consuming
application would benefit from knowing how to target variables in their
application for given information items.  I guess one could assume that
everything will fit in a 64-bit integer and code your app that way, but
again, this seems like an unnecessary burden for unclear gains.
 
  I don't think anything needs to change in the protocol.  Simply a more
constrained approach to dealing with numeric fields with explicit types.
 
Regards,

  Jeff Meyer

-----Original Message-----
From: Benoit Claise [mailto:[email-masked]]
Sent: Wednesday, June 11, 2003 4:31 AM
To: Benoit Claise
Cc: Mark Fullmer; [email-masked]; [email-masked]
Subject: Re: [ipfix] drawing the line between protocol and data model
document

Dear all,

Mark, 

On Thu, May 01, 2003 at 10:51:49AM -0400,  [email-masked]
<mailto:[email-masked]>  wrote:

<snip>

The richness of the protocol encoding specification is another

matter. The protocol would reference each element in the data

model and define exaclty how that element is encoded. In the above 

example, the protocol is free to encode that as an unsigned byte 

if the value is < 256. 

The ramifications of doing this with a template based system are a bit

ugly.  Consider a flow with just 2 counters -- octets, packets.

Each could have a 1 to 8 byte encoding.  That's a potential of 64

different templates for representing 1 type of flow.

This (at first glance) looks like it adds considerable cycles to the

encode / decode process to save a few bytes in the transport.

The current v9 spec appears to allow this, but only for IN_BYTES and

IN_PKTS.

                                          counter with length

   IN_BYTES                 1       N     N x 8 bits for bytes

                                          associated with an IP Flow

Is this true?  Benoit?

I'm trying to catch up with my IPIFX emails; I know I'm late on this thread,
which seems to have reached consensus as far as I can tell, but as this is a
direct question to me, let me answer.
Actually, there are:
BYTES            1             4             field type for the 32 bit bytes
counter
BYTES_64      23           8             field type for the 64 bit bytes
counter

Let me reply to my own email because, after speaking to different people, I
have to correct what I said.
We actually do:

                                          Incoming counter with length

   IN_BYTES                 1       N     N x 8 bits for bytes

                                          associated with an IP Flow

                                          Incoming counter with length

   IN_PKTS		    2       N     N x 8 bits for packets

                                          associated with an IP Flow

And no BYTES_64 is defined.

Regards, Benoit.

So the draft that you refer to needs a quick update.
<!--[endif]-->
Regards, Benoit

mark

--

Help         mailto:[email-masked]
<mailto:[email-masked]>  and say "help " in message body
<inmessagebodyUnsubscribemailto:[email-masked]> 

Unsubscribe mailto:[email-masked] and say

"unsubscribe ipfix" in message body

Archive      http://ipfix.doit.wisc.edu/archive/
<http://ipfix.doit.wisc.edu/archive/>

---

From: "Tal Givoly" <[email-masked]>
Date: Wed, 11 Jun 2003 21:49:55 -0700

Benoit,

I agree with Jeff. I would add that, no more than 2 options should be
available for bytes/packet counters (32/64-bit unsigned integers).

But I would consider offering only a 64-bit value for byte counters. From
SNMP we learned that 32-bit byte counters cause countless problems. Since
IPFIX byte counters may be a result of aggregates, with today's networks it
is very feasible to overflow the 32-bit integer container.

Tal
  -----Original Message-----
  From: majordomo listserver [mailto:[email-masked]]On Behalf
Of MEYER,JEFFREY D (HP-Cupertino,ex1)
  Sent: Wednesday, June 11, 2003 10:00 AM
  To: 'Benoit Claise'
  Cc: Mark Fullmer; [email-masked]; [email-masked]
  Subject: RE: [ipfix] drawing the line between protocol and data model
document

  Hi,

  Regards,

    Jeff Meyer
    -----Original Message-----
    From: Benoit Claise [mailto:[email-masked]]
    Sent: Wednesday, June 11, 2003 4:31 AM
    To: Benoit Claise
    Cc: Mark Fullmer; [email-masked]; [email-masked]
    Subject: Re: [ipfix] drawing the line between protocol and data model
document

    Dear all,

      Mark,
On Thu, May 01, 2003 at 10:51:49AM -0400, [email-masked] wrote:

<snip>

                                          counter with length

Is this true?  Benoit?

      I'm trying to catch up with my IPIFX emails; I know I'm late on this
thread, which seems to have reached consensus as far as I can tell, but as
this is a direct question to me, let me answer.
      Actually, there are:
      BYTES            1             4             field type for the 32 bit
bytes counter
      BYTES_64      23           8             field type for the 64 bit
bytes counter
    Let me reply to my own email because, after speaking to different
people, I have to correct what I said.
    We actually do:

Regards, Benoit.

mark

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Thu, 12 Jun 2003 14:47:11 +0200

Tal, Jeff,

[let me reply to both of you in a single email]

> But I would consider offering *only* a 64-bit value for byte counters. 
> From SNMP we learned that 32-bit byte counters cause countless 
> problems. Since IPFIX byte counters may be a result of aggregates, 
> with today's networks it is very feasible to overflow the 32-bit 
> integer container.

How often do you poll the SNMP variables like interface counters? 5 minutes?
This is order of magnitude higher than the flow expiration, in most of 
the cases.

http://www.ietf.org/internet-drafts/draft-claise-netflow-9-02.txt
    -> 3.2 Flow Expiration

   A Flow is considered to be inactive if no packets belonging to the 
   Flow have been observed at the Observation Point for a given timeout 
   interval otherwise it is considered as an active flow.  
   A Flow can be exported under the following conditions: 
    
      1. If the Exporter can detect the end of a Flow, it SHOULD export 
      the Flow Records at the end of the Flow. For example, a Flow 
      generated by TCP [3] type of traffic where the FIN or RST bits 
      indicate the end of the Flow. 
       
      2. If the Flow has been inactive for a certain period of time. 
      This inactivity timeout SHOULD be configurable, with a minimum 
      value of 0 for an immediate expiration. For example, a Flow 
      generated by UDP [2] type of traffic. 
       
      3. For long-lasting Flows, the Exporter SHOULD export the Flow 
      Records on a regular basis. This periodicity SHOULD be 
      configurable. 
       
      4. If the Exporter experiences internal constraints, a Flow MAY 
      be forced to expire prematurely (for example, counters wrapping 
      or low memory). 

So it's only in case 3 (so potentially an aggregation, as you described) that we could potentially needs a higher counter.
And in that case again, a simple solution is to export the flow records just before it reaches the max value of counter32 (in case the length for counter32 is choosen).

But if you don't want that, there is also another solution: when this case 3. occurs, you can define a new template with the length corresponding to counter64.
And then you have 2 templates: one with counter32 (used most of the time) and one with counter64 (used in a few cases).

Another solution, nothing prevents you to export flow data records always with counter64

See below...

>     *From:* majordomo listserver
>     [mailto:[email-masked]]*On Behalf Of *MEYER,JEFFREY D
>     (HP-Cupertino,ex1)
>     *Sent:* Wednesday, June 11, 2003 10:00 AM
>     *To:* 'Benoit Claise'
>     *Cc:* Mark Fullmer; [email-masked]; [email-masked]
>     *Subject:* RE: [ipfix] drawing the line between protocol and data
>     model document
>
>     Hi,
>      
>       >From a practical perspective, simply saying that a field is
>     numeric and may be encoded as 1 or 100 bytes seems like it puts an
>     unnecessary and unjustified burden on the collector, or any other
>     app which might want to handle this data.
>
Am I correct that the burden would only be when you decode the template 
records?
Because when you receive a Data FlowSet, the collecting process must 
read the FlowSet ID and automatically know the data type length.

>      
>       Is there some particular reason why saying a specific
>     information item, e.g. numBytes, is a 32-bit integer or a 64-bit
>     integer is considered undesireable.
>
But then, why to limit to 32-bit and 64-bit integer? What about other 
possibilities? Where to stop? Then we end up with a long information 
model because this principle of 32-bit, 64-bit integer should be then 
applied to all data type that are counters.

>     It certainly seems like the developer of a consuming application
>     would benefit from knowing how to target variables in their
>     application for given information items.  I guess one could assume
>     that everything will fit in a 64-bit integer and code your app
>     that way, but again, this seems like an unnecessary burden for
>     unclear gains.
>
The gain, as I see it, is in the flexibility with a simple information model

Regards, Benoit.

>      
>       I don't think anything needs to change in the protocol.  Simply
>     a more constrained approach to dealing with numeric fields with
>     explicit types.
>      
>     Regards,
>
>       Jeff Meyer
>
>         *From:* Benoit Claise [mailto:[email-masked]]
>         *Sent:* Wednesday, June 11, 2003 4:31 AM
>         *To:* Benoit Claise
>         *Cc:* Mark Fullmer; [email-masked];
>         [email-masked]
>         *Subject:* Re: [ipfix] drawing the line between protocol and
>         data model document
>
>         Dear all,
>
>>         Mark,
>>
>>         I'm trying to catch up with my IPIFX emails; I know I'm late
>>         on this thread, which seems to have reached consensus as far
>>         as I can tell, but as this is a direct question to me, let me
>>         answer.
>>         Actually, there are:
>>         BYTES            1             4             field type for
>>         the 32 bit bytes counter
>>         BYTES_64      23           8             field type for the
>>         64 bit bytes counter
>
>         Let me reply to my own email because, after speaking to
>         different people, I have to correct what I said.
>         We actually do:
>
>>>Help        mailto:[email-masked] and say "help" in message body