Subject: RE: [ipfix] Variant Field Types.

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Thu, 2 Oct 2003 15:58:56 -0400

Stewart,

  Counters are an interesting (and irritating) beast.  Thankfully I 
am not aware of any currently defined in the context of IPFIX.

  If we do encounter these, then there should be an appropriate 
annotation in the IM indicating this property.  My leaning would
be to avoid them.  The observer in the IPFIX model seems capable
of delivering accurate summaries of an observed flow, vs. relying
on SNMP style counters.

  In the case of explicit numeric quantities, i.e. "on this flow I
saw this many of this thing", clearly the exporter needs to ensure
that the size of its export field has sufficient capacity to accomodate
the largest value it may observe.

  From the perspective of a collector, the maximal size should 
reflect the largest accumulation of this field which is expected.
This would seem to lean towards int64 for most quantities.

  If the quantities are expected to accumulate beyond this amount,
I would recommend the use of float or double.  I'm not aware of
much consistent support for integers > 2^64.

Regards,

  Jeff Meyer

> -----Original Message-----
> From: Stewart Bryant [mailto:[email-masked]]
> Sent: Thursday, October 02, 2003 12:03 PM
> To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> Cc: 'Ganesh Sadasivan'; Mark Fullmer; [email-masked]
> Subject: Re: [ipfix] Variant Field Types.
> 
> 
> Jeff
> 
> Whether the size reduction fits in the IM or in the protocol draft
> depends on how closely you see the drafts mapping onto the
> structure of the implementation.
> 
> Ultimately the exporter and the collector have to have the same
> shared IM, including knowledge of the object size that the exporter
> is actually using.
> 
> One obvious example is counter wrapping (there may be others).
> Lets say IM defines a 16 bit counter object, and the collector
> uses this to interpret the data it receives from the exporter. The
> exporter on the other hand reduces this to 8 bits. The sequence
> 7,8,0 means quite different things to each party. In the case
> above the exporter saw 9 object and the collector thought
> there had been 17. The collector can only correctly determine
> what ocured if it knows that the exporter is using the abbreviated
> size.
> 
> Now you could regard the use of a different size as a protocol matter
> i.e. compression is being applied to the data, but to get the correct
> shared semantics, the protocol engine now has to be aware of the
> true semantics of IEs rather than just passing them up as abstract
> objects.
> 
> Therefore I would think that the part of the system that understands
> the IEs needs to be aware of the size that the exporter is using,
> and this leads me to conclude that we might as well describe the
> size behavior in the IM.
> 
> Stewart
> 
> 
> 
> 
> 
> MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> 
> > Hi,
> >  
> >   I think the details of using smaller encoding in an 
> implementation 
> > belong in the protocol specification vs. the information 
> model.  The 
> > information model already references the protocol 
> specification for how 
> > fields of a given type are encoded.
> >  
> >   For specific fields, there is still a decision on what 
> would be the 
> > maximal type.  E.g. do ports need to be 32-bit or 16?  Do 
> AS numbers 
> > need to be 32 or 64?  Given the unit of measure for 
> SampleInterval, what 
> > is its limit?  These details would go in the information model.
> >  
> > Regards,
> >  
> >   Jeff Meyer
> > 
> >     -----Original Message-----
> >     From: Ganesh Sadasivan [mailto:[email-masked]]
> >     Sent: Wednesday, October 01, 2003 11:35 AM
> >     To: Mark Fullmer
> >     Cc: MEYER,JEFFREY D (HP-Cupertino,ex1); [email-masked]
> >     Subject: Re: [ipfix] Variant Field Types.
> > 
> >     Mark,
> > 
> >     Mark Fullmer wrote:
> > 
> >>I'm in favor of setting upper bounds, for example counters don't go
> > 
> > 
> >     I agree that a field even if variable in length is limited by an
> >     upper bound and this is limit is set on a
> >     case by case basis. The whole point of my first e-mail is rather
> >     than nailing a field to an explicit
> >     type like uint64_t, to let implementors know that  they can
> >     send/receive these fields in smaller
> >     sizes too.
> > 
> >>past 64 bits.  If you don't do this, what's a collector to 
> do with say
> >>a 72 bit counter?  You can't store it in a uint_64_t 
> anymore so collectors
> >>would have to implement arbitrary precision arithmetic to 
> handle potential
> >>cases.  I don't think this is reasonable to impose on the 
> collectors.
> >>
> >>The issue I have with the variable length fields as 
> currently specified
> >>is the potential for template explosion.  If a template has 8
> >>counters, each of which could be 1..8 byte then you have
> >>8^8 potential templates for that one stream of flows.
> > 
> >     Yes a separate template is needed for varying length. Template
> >     explosion because if this would be
> >     an implementation choice rather than that imposed by 
> the definition
> >     itself. Also the same would
> >     have happened if we explicitly define field types for 
> the counters
> >     of varying length.
> > 
> >     Thanks
> >     Ganesh
> > 
> >>I was hoping for the sake of the collectors we could keep the active
> >>templates down to something < 64K so a template lookup could be done
> >>with an index instead of a hash.
> >>
> >>mark
> >>
> >>On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D 
> (HP-Cupertino,ex1) wrote:
> >>  
> >>
> >>>Ganesh,
> >>> 
> >>>  I'm not clear on how this benefits the collector.  Tal 
> and myself produce
> >>>different implementations of carrier deployed products 
> which collect from
> >>>100's of devices and dozens of protocols, perhaps our 
> experience here is
> >>>worth something, and it tells us variant is undesireable 
> on the collector...
> >>> 
> >>>  Regarding your other point:
> >>> 
> >>>One way to solve this problem is if the collector assumes 
> the largest size
> >>>for 
> >>>computations?
> >>> 
> >>>  This was my point in the e-mail to Stewart.  Have the 
> information model be
> >>>EXPLICIT about what is the largest size which is being 
> modeled for a data
> >>>item and declare it.  Allow the exporter to declare a 
> smaller size in the
> >>>template, if the exporter knows that an individual record 
> will always be
> >>>accomodated (based on its implementation) in the smaller size.
> >>> 
> >>>  So for instance this might say that byteCount should be 
> declared in the
> >>>information model as a 64-bit int, while AS is probably 
> sufficient at
> >>>32-bits.
> >>> 
> >>>  This accomodates the:
> >>>      exporter - can use smaller size if known
> >>>      n/w - can reduce BW if know a smaller size is used
> >>>      info model - explicit is better than "vague"
> >>>      collector - addresses persistence and aggregation issues
> >>> 
> >>>  Can we call this a compromise?
> >>> 
> >>>Regards,
> >>> 
> >>>  Jeff Meyer
> >>>
> >>>-----Original Message-----
> >>>From: Ganesh Sadasivan [mailto:[email-masked]]
> >>>Sent: Tuesday, September 30, 2003 10:45 AM
> >>>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> >>>Cc: [email-masked]
> >>>Subject: Re: [ipfix] Variant Field Types.
> >>>
> >>>
> >>>
> >>>
> >>>MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> >>>
> >>>
> >>>Hi,
> >>> 
> >>>  Although the use of variant types may seem convenient 
> from the perspective
> >>>of an
> >>>exporter, I think the various consumers of the data would 
> benefit from
> >>>sticking with
> >>>an explicit type system.
> >>>
> >>>It would benefit the exporter, the n/w, flexibility of 
> interpretation of
> >>>fields in the 
> >>>information model  and the collector.
> >>>
> >>>
> >>> 
> >>>  Consider operations which involve summing fields, 
> storing them to a DB,
> >>>etc.
> >>>
> >>>One way to solve this problem is if the collector assumes 
> the largest size
> >>>for 
> >>>computations?
> >>>
> >>>
> >>> 
> >>>  Explicit typing ensures that these consuming 
> applications can handle these
> >>>values
> >>>appropriately and also makes the information modeller 
> think about how these 
> >>>types are to be used.
> >>> 
> >>>  The examples you cite:
> >>> 
> >>>- ingressPort 
> >>>- egressPort 
> >>>- packetCount 
> >>>- byteCount 
> >>>- sourceAS 
> >>>- destinationAS
> >>>- nextHopAS 
> >>>- droppedPacketCount
> >>>- samplingInterval
> >>>- droppedByteCount
> >>> 
> >>>  fall into a few categories, identifiers (such as ports 
> and AS's), counts
> >>>and intervals.
> >>> 
> >>>  For the identifiers, I believe the size is fixed by the 
> relevant RFC's
> >>>(i.e. why would I 
> >>>have a 1 byte Port?).  For counts, the use of 32 vs. 64 
> bits is a valid
> >>>question, however
> >>>these can be handled by making them explicit (i.e. the 
> information model
> >>>says if
> >>>I get this value it is a 32-bit quantity or a 64-bit 
> quantity and the code
> >>>will address
> >>>it accordingly).  For intervals, the resolution would 
> determine which size
> >>>is most
> >>>appropriate.
> >>> 
> >>>
> >>>For AS the size could be 2 or 4 bytes.
> >>>You may prefer to use a 1 byte port if you are a small 
> system as against a 4
> >>>byte for 
> >>>a gigantic system. Both have their own implementation reasons.
> >>>
> >>>
> >>>  The option always exist to define variations on these data items
> >>>explicitly.  E.g.
> >>>byteCount32 and byteCount64.
> >>> 
> >>>
> >>>Yes. But is'nt this not more of an overhead to introduce 
> new field type as
> >>>the requirements
> >>>come in and then make sure that it gets deployed in 
> exporter and collector?
> >>>
> >>>
> >>>
> >>>  If there is a need to vary the sizes for various fields 
> I'd much rather
> >>>see the rationale
> >>>for each discrete sizing made explicit vs. forcing all the 
> downstream
> >>>consumers of this \
> >>>information have to deal with special cases and ambiguity.
> >>> 
> >>> 
> >>>
> >>>This is not a special case handling as you can see that 
> the list above
> >>>covers about 
> >>>40% of the fields defined today in the information model. 
> It is way to
> >>>handle 
> >>>ambiguity arising out of implementation differences.
> >>>
> >>>Thanks
> >>>Ganesh
> >>>
> >>>
> >>>Regards,
> >>> 
> >>>  Jeff Meyer
> >>>
> >>>
> >>>  
> >>>
> >>>-----Original Message-----
> >>>From: Ganesh Sadasivan [ mailto:[email-masked]
> >>><mailto:[email-masked]> ]
> >>>Sent: Monday, September 29, 2003 6:00 PM
> >>>To: [email-masked] <mailto:[email-masked]> 
> >>>Subject: [ipfix] Variant Field Types.
> >>>
> >>>
> >>>Hi,
> >>>
> >>>
> >>>
> >>>The information model <draft-ietf-ipfix-info-01.txt> only
> >>>
> >>>has types for fixed length objects and variable length 
> >>>
> >>>string like objects.
> >>>
> >>>
> >>>
> >>>There are a number of information element types (listed below)
> >>>
> >>>that would be better defined in terms of fixed-size run-time
> >>>
> >>>specified objects. The type variation comes due to difference
> >>>
> >>>in the size of the objects used because of an implementation,
> >>>
> >>>as demanded by the network or otherwise.
> >>>
> >>>
> >>>
> >>>The way to make this work is to use the template
> >>>
> >>>length field to determine the exact sub-type:
> >>>
> >>>
> >>>
> >>>The definitions of a variant field type can be stated as:
> >>>
> >>>
> >>>
> >>>type varUnsignedInt
> >>>
> >>>    if (element.length == 1)
> >>>
> >>>         type = unsignedByte
> >>>
> >>>    else if (element.length <= 2)
> >>>
> >>>         type = unsignedShort
> >>>
> >>>    else if (element.length <= 4)
> >>>
> >>>         type = unsignedInt
> >>>
> >>>    else if (element.length <= 8)
> >>>
> >>>         type = unsignedLong
> >>>
> >>>   else type = undefined
> >>>
> >>>
> >>>
> >>>type varInt
> >>>
> >>>    if (element.length == 1)
> >>>
> >>>         type = byte
> >>>
> >>>    else if (element.length <= 2)
> >>>
> >>>         type = short
> >>>
> >>>    else if (element.length <= 4)
> >>>
> >>>         type = int
> >>>
> >>>    else if (element.length <= 8)
> >>>
> >>>         type = long
> >>>
> >>>   else type = undefined
> >>>
> >>>
> >>>
> >>>The candidates that fit into the description above are (I may
> >>>
> >>>have missed some also):
> >>>
> >>>- ingressPort 
> >>>
> >>>- egressPort 
> >>>
> >>>- packetCount 
> >>>
> >>>- byteCount 
> >>>
> >>>- sourceAS 
> >>>
> >>>- destinationAS
> >>>
> >>>- nextHopAS 
> >>>
> >>>- droppedPacketCount
> >>>
> >>>- samplingInterval
> >>>
> >>>- droppedByteCount
> >>>
> >>>
> >>>
> >>>Please comment.
> >>>
> >>>
> >>>
> >>>Thanks
> >>>
> >>>Ganesh
> >>>
> >>>    
> >>>
> >>>
> >>>    
> >>
> >>
> >>--
> >>Help        mailto:[email-masked] and say 
> "help" in message body
> >>Unsubscribe mailto:[email-masked] and say
> >>"unsubscribe ipfix" in message body
> >>Archive     http://ipfix.doit.wisc.edu/archive/
> >>
> >>  
> > 
> > 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 03 Oct 2003 09:28:12 +0100

> am not aware of any currently defined in the context of IPFIX.

packetCount and byteCount are not counters?

In any case we are designing a general purpose system here,
and future applications designers should be able to use any
basic primitive data object and a wrapping counter is such
an object. We also need to be able to scele the system without
exploding the number of IEs. Including the variant type
allows us to do this.

> annotation in the IM indicating this property.  

That is what defining them as a variant type is doing.

> My leaning would be to avoid them.  

Sure, but why deny the flexibility to the designer that does need them.
We have an existance proof in Netflow v9 that there is utility in this
type of definition, and I am sure that many more will appear over time.

> The observer in the IPFIX model seems capable

in the normal case...

> This would seem to lean towards int64 for most quantities.

I am not so worried about the collector (although I am worried
about an aggregator), I am however worried about the observer/exporter
which is much more likely to be memory constrained.

> much consistent support for integers > 2^64.

I agree that in most case 64 bits works, my concern is at the other
end of the system where the information is gleaned from the flows.

Stewart

---

From: Mark Fullmer <[email-masked]>
Date: Sun, 5 Oct 2003 00:19:07 -0400

> much consistent support for integers > 2^64.

Some implementation of the observation point will be running in kernel
space where floating point arithmetic is not available.

mark