Subject: RE: [ipfix] CUrrent issues, Draft deadlines

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 14 Oct 2003 11:45:07 -0400

Hi,

  I would second Carter's opinion on counters v. integers
for the basic IPFIX export information elements.

  A few other points are that:
    a. The current behavior of NFvX is to use integers not 
      counters.
    b. Since information is being exported and not polled, 
      counters provide less benefit.  Especially when most
      flows are exported exactly once.
    c. There needs to be some strategy on the collector to
      do delta calculations with flows.  Since the number of
      unique flow records would be huge, searching for 
      previous flows to do the delta can be costly.
 
  One alternative to address c. with counters would be to have
explicit indications of whether a flow record is the "last"
one.  If it is last, then it does not need to be preserved
for subsequent delta calculations.  But this alternative sounds
a little clunky to me.

Regards,

  Jeff Meyer

> -----Original Message-----
> From: majordomo listserver 
> [mailto:[email-masked]]On Behalf
> Of Carter Bullard
> Sent: Monday, October 13, 2003 7:55 PM
> To: 'Nevil Brownlee'; [email-masked]
> Subject: RE: [ipfix] CUrrent issues, Draft deadlines
> 
> 
> Hey Nevil,
>    With regard to counters vs integers.  I would suggest that
> the better default choice would be integers.  Counters
> would require that originators of IPFIX always support the
> largest data type possible, where an integers strategy can
> be used by a probe to limit the size of the reported metrics.
> This would provide the opportunity to control the
> memory demands of a probe, and minimize the amount of
> data on the wire.
> 
>    Because counters have roll-over behaviors, regardless
> of the size of the counters you support, the counter strategy
> is fundamentally the same as integers, except that with
> counters you throw away some intermediate records and keep
> others.  This introduces possible uncertainty.  The reader is
> throwing away data, did it throw away the right data?  I
> would rather like to avoid having any IPFIX component
> throw away data as a part of its basic function.
> 
> Carter
> 
> 
> > -----Original Message-----
> > From: majordomo listserver
> > [mailto:[email-masked]] On Behalf Of Nevil Brownlee
> > Sent: Monday, October 13, 2003 6:37 PM
> > To: [email-masked]
> > Subject: [ipfix] CUrrent issues, Draft deadlines
> >
> >
> >
> > Hello all:
> >
> > Reminder for all the IPFIX draft editors:  deadline for IETF 58, the
> > Minneapolis meeting (9-14 Nov 03) is 0900 ET on Monday 27 October.
> > That's not very far away now, we need to focus on getting the next
> > revisions done - which means we need to get closer to consensus on
> > the (not very many) remaining issues.
> >
> > Here's my current list.  It's not by any means complete (I 
> presume the
> > draft editors are keeping their own 'issues' lists), it's just the
> > ones which have been prominent on the mailing list recently.  I've
> > indicated my own opinions inside <wg chair hat off> brackets!
> >
> > Proto-1:  Default transport protocol.
> >    Note that there's no question of choosing a single 
> protocol, there
> >    never was.  The issue is that we need to pick one 'mandatory-to-
> >    implement' transport, so that users will be able to talk to any
> >    IPFIX device when they plug it onto their network.
> >
> >    At the moment (following the list discussion) it seems
> > that TCP would
> >    be better for IPFIX collectors, since it's easily 
> available at user
> >    level on every host.  But SCTP seems to offer 
> advantages, the most
> >    obvious being the ability to carry partially reliable data with
> >    congestion control.
> >
> >    <wg chair hat off> could we agree on TCP 'mandatory to 
> implement,'
> >    and assume that vendors who see technical advantages in SCTP will
> >    also implement SCTP and encourage their customers to use it?
> >    </wg chair hat off>
> >
> > IM-1: Counters vs Integers
> >    We seem to acknowledge that counters (running totals) 
> and Integers
> >    (reset to zero when the exporter sends out data about a flow) can
> >    both be useful, so the Information Model needs to have 
> both Counter
> >    and Integer types.
> >
> >    My experience with RTFM is that counters have the advantage that
> >    it doesn't matter if you loose a few intermediate counts for any
> >    reason.  If that happens you loose some time granularity, but you
> >    still have most, or at least some, of the packet and byte counts
> >    for the flow.
> >
> >    I'm not sure that - at least for packet and byte counts - we need
> >    both.  <wg chair hat off> could we agree to only have counters?
> >    </wg chair hat off>
> >
> > General-2: Variable-size objects
> >    There probably are a few objects which really do need
> > variable-length
> >    encodings; they need to be identified, and the Information Model
> >    must support them.
> >
> >    However, much of the discussion on this has centred on 
> the need to
> >    minimise the number of bytes to carry objects, especially small
> >    integers for packet and byte counts.  Almost sounds to me like
> >    an argument for ASN.1 encodings ..!
> >
> >    <wg chair hat off> I liked the suggestion that this is really an
> >    encoding issue.  The objects should be defined in the Information
> >    Model with their maximum sizes (64-bit), and the 
> Protocol document
> >    should specify how they are encoded into various numbers 
> of bytes.
> >    That encoding could be fixed numbers of bytes (specified in the
> >    template), or a variable-length encoding. </wg chair hat off>
> >
> > General-3: Timestamps
> >    I think we have a consensus that timestamps should be UTC-based
> >    rather than meter-uptime-based - any dissenting opinion on that?
> >
> >    We also have consensus that microsecond precision is needed.
> >
> >    I haven't seen any strong statements for (seconds, microseconds)
> >    vs (NTP-standard 32.32-bit fixed binary).  <wg chair hat off>
> >    I find NTP-standard easier to work with, but we need a WG
> > consensus.
> >    Lets see some opinions on this, please. </wg chair hat off>
> >
> > Those are my four 'current' IPSIF issues.  Let's try to get
> > to consensus
> > on these in the next week or so.  Also please read the
> > drafts, and post
> > text to the list for any sections you think need improvement.  Don't
> > just sit back and assume the draft editors will do it - after all,
> > they need your contributed text to edit!
> >
> > Cheers, Nevil
> >
> > --------------------------------------------------------------
> > ---------
> >    Nevil Brownlee                   Director, Technology Development
> >    Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
> >    FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand
> >
> >
> > -------------------------------------------------
> > This mail sent through University of Auckland
> > http://www.auckland.ac.nz/
> >
> > --
> > Help        mailto:[email-masked] and say "help"
> > in message body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> >
> 
> 
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Fri, 17 Oct 2003 14:22:19 -0400

Maurizio,

  I agree with your sentiments around requiring both.  I disagree
with your statement that counters benefit charging.  Again, most
flows are single export records which you either receive or you
don't.  State management for counters, make them most useful if
you have a relatively constrained space (i.e. it may be OK for
exporter based aggregation schemes such as NFv8), but in the situation
where each flow has a unique src/dstIP and src/dstPort we're talking
about millions of possible counters to maintain.

  Integers are how deployed Netflow works today, and I would
not recommend changing something which works.

Regards,

  Jeff Meyer

> Of Maurizio Molina
> Sent: Friday, October 17, 2003 10:59 AM
> To: [email-masked]
> Cc: 'Nevil Brownlee'; [email-masked]
> Subject: Re: [ipfix] CUrrent issues, Draft deadlines
> 
> 
> Nevil, Carter,
> my position would be to support both Counters and Integers in the 
> information model, and to NOT make any default choice, but rather 
> request with a MUST that the IPFIX devices support both.
> 
> In fact, on the  ML we had arguments pro/against Integers and 
> arguments 
> pro/against Counters, and all of them are probably  correct given the 
> application that had in mind who wrote them.
> But that's exacly the point: in theory ANY application receiving flow 
> records and making something out of them could work both with 
> Integers 
> and Counters (provided it knows what it is receiving...). Of course 
> there are applications that care most about variations (e.g. load 
> monitoring) and therefore are happy with Integers, but can also work 
> with Counters (they just need more processing). 
> Simmetrically, there are 
> applications that need the total amount of traffic of a flow (e.g. 
> Charging) and therefore are happy with Counters, but can also 
> work with 
> Integers (they just need more processing).
> But I can envisage that there can be "poor" or legacy 
> applications that 
> can only work with either Integers
> or Counters (not both), and if the probe cannot support the 
> data in the 
> format they need they cannot work at all.
> 
> Of  course, making a default choice  gives more chance for market 
> differentiation ("good" probes will support both...) but 
> opens the door 
> to interoperability problems.
> 
> Maurizio
> 
> Carter Bullard wrote:
> 
> >Hey Nevil,

---

From: "Carter Bullard" <[email-masked]>
Date: Fri, 17 Oct 2003 14:38:23 -0400

Hey Maurizio,
   I completely agree with your arguments, but I'd rather not
put a requirement on the probe that it MUST support counters.
My reason for choosing integers is based solely from the
perspective of the probe, with a desire to help them be
inexpensive and perform well in demanding environments.

   I think this perspective is appropriate because an
application can adjust, regardless of what strategy the data
originator uses.  But my experience is that it is easier
for the application to deal with integers over counters
because it reduces the states that the application must hold.

   Consider the situation where the probe is a high performance
probe in a demanding environment, say supporting 10M+
simultaneous flows.  The probe wants to support small metrics, say
16-bit ints for the packet and byte values, because most flows
don't really live very long.   For the small percentage of long
lived flows, a counter method quickly reduces to a complex integer
strategy, because the roll-over condition occurs very frequently,
forcing the reader to recover the counter strategy with a
large metric size, say 32-bit ints.

   It would be inefficient for the probe to keep 64-bit counters
for every metric in order to give an application a counter view
for a very small percentage of the traffic.

Carter

> From: Maurizio Molina [mailto:[email-masked]]
> Sent: Friday, October 17, 2003 12:59 PM
> To: [email-masked]