Subject: Re: [ipfix] Variant Field Types.

From: Mark Fullmer <[email-masked]>
Date: Wed, 1 Oct 2003 11:15:55 -0400

I'm in favor of setting upper bounds, for example counters don't go
past 64 bits.  If you don't do this, what's a collector to do with say
a 72 bit counter?  You can't store it in a uint_64_t anymore so collectors
would have to implement arbitrary precision arithmetic to handle potential
cases.  I don't think this is reasonable to impose on the collectors.

The issue I have with the variable length fields as currently specified
is the potential for template explosion.  If a template has 8
counters, each of which could be 1..8 byte then you have
8^8 potential templates for that one stream of flows.

I was hoping for the sake of the collectors we could keep the active
templates down to something < 64K so a template lookup could be done
with an index instead of a hash.

mark

On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> Ganesh,
>  
>   I'm not clear on how this benefits the collector.  Tal and myself produce
> different implementations of carrier deployed products which collect from
> 100's of devices and dozens of protocols, perhaps our experience here is
> worth something, and it tells us variant is undesireable on the collector...
>  
>   Regarding your other point:
>  
> One way to solve this problem is if the collector assumes the largest size
> for 
> computations?
>  
>   This was my point in the e-mail to Stewart.  Have the information model be
> EXPLICIT about what is the largest size which is being modeled for a data
> item and declare it.  Allow the exporter to declare a smaller size in the
> template, if the exporter knows that an individual record will always be
> accomodated (based on its implementation) in the smaller size.
>  
>   So for instance this might say that byteCount should be declared in the
> information model as a 64-bit int, while AS is probably sufficient at
> 32-bits.
>  
>   This accomodates the:
>       exporter - can use smaller size if known
>       n/w - can reduce BW if know a smaller size is used
>       info model - explicit is better than "vague"
>       collector - addresses persistence and aggregation issues
>  
>   Can we call this a compromise?
>  
> Regards,
>  
>   Jeff Meyer
> 
> -----Original Message-----
> From: Ganesh Sadasivan [mailto:[email-masked]]
> Sent: Tuesday, September 30, 2003 10:45 AM
> To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> Cc: [email-masked]
> Subject: Re: [ipfix] Variant Field Types.
> 
> 
> 
> 
> MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> 
> 
> Hi,
>  
>   Although the use of variant types may seem convenient from the perspective
> of an
> exporter, I think the various consumers of the data would benefit from
> sticking with
> an explicit type system.
> 
> It would benefit the exporter, the n/w, flexibility of interpretation of
> fields in the 
> information model  and the collector.
> 
> 
>  
>   Consider operations which involve summing fields, storing them to a DB,
> etc.
> 
> One way to solve this problem is if the collector assumes the largest size
> for 
> computations?
> 
> 
>  
>   Explicit typing ensures that these consuming applications can handle these
> values
> appropriately and also makes the information modeller think about how these 
> types are to be used.
>  
>   The examples you cite:
>  
> - ingressPort 
> - egressPort 
> - packetCount 
> - byteCount 
> - sourceAS 
> - destinationAS
> - nextHopAS 
> - droppedPacketCount
> - samplingInterval
> - droppedByteCount
>  
>   fall into a few categories, identifiers (such as ports and AS's), counts
> and intervals.
>  
>   For the identifiers, I believe the size is fixed by the relevant RFC's
> (i.e. why would I 
> have a 1 byte Port?).  For counts, the use of 32 vs. 64 bits is a valid
> question, however
> these can be handled by making them explicit (i.e. the information model
> says if
> I get this value it is a 32-bit quantity or a 64-bit quantity and the code
> will address
> it accordingly).  For intervals, the resolution would determine which size
> is most
> appropriate.
>  
> 
> For AS the size could be 2 or 4 bytes.
> You may prefer to use a 1 byte port if you are a small system as against a 4
> byte for 
> a gigantic system. Both have their own implementation reasons.
> 
> 
>   The option always exist to define variations on these data items
> explicitly.  E.g.
> byteCount32 and byteCount64.
>  
> 
> Yes. But is'nt this not more of an overhead to introduce new field type as
> the requirements
> come in and then make sure that it gets deployed in exporter and collector?
> 
> 
> 
>   If there is a need to vary the sizes for various fields I'd much rather
> see the rationale
> for each discrete sizing made explicit vs. forcing all the downstream
> consumers of this \
> information have to deal with special cases and ambiguity.
>  
>  
> 
> This is not a special case handling as you can see that the list above
> covers about 
> 40% of the fields defined today in the information model. It is way to
> handle 
> ambiguity arising out of implementation differences.
> 
> Thanks
> Ganesh
> 
> 
> Regards,
>  
>   Jeff Meyer
> 
> 
>   
> 
> -----Original Message-----
> From: Ganesh Sadasivan [ mailto:[email-masked]
> <mailto:[email-masked]> ]
> Sent: Monday, September 29, 2003 6:00 PM
> To: [email-masked] <mailto:[email-masked]> 
> Subject: [ipfix] Variant Field Types.
> 
> 
> Hi,
> 
> 
> 
> The information model <draft-ietf-ipfix-info-01.txt> only
> 
> has types for fixed length objects and variable length 
> 
> string like objects.
> 
> 
> 
> There are a number of information element types (listed below)
> 
> that would be better defined in terms of fixed-size run-time
> 
> specified objects. The type variation comes due to difference
> 
> in the size of the objects used because of an implementation,
> 
> as demanded by the network or otherwise.
> 
> 
> 
> The way to make this work is to use the template
> 
> length field to determine the exact sub-type:
> 
> 
> 
> The definitions of a variant field type can be stated as:
> 
> 
> 
> type varUnsignedInt
> 
>     if (element.length == 1)
> 
>          type = unsignedByte
> 
>     else if (element.length <= 2)
> 
>          type = unsignedShort
> 
>     else if (element.length <= 4)
> 
>          type = unsignedInt
> 
>     else if (element.length <= 8)
> 
>          type = unsignedLong
> 
>    else type = undefined
> 
> 
> 
> type varInt
> 
>     if (element.length == 1)
> 
>          type = byte
> 
>     else if (element.length <= 2)
> 
>          type = short
> 
>     else if (element.length <= 4)
> 
>          type = int
> 
>     else if (element.length <= 8)
> 
>          type = long
> 
>    else type = undefined
> 
> 
> 
> The candidates that fit into the description above are (I may
> 
> have missed some also):
> 
> - ingressPort 
> 
> - egressPort 
> 
> - packetCount 
> 
> - byteCount 
> 
> - sourceAS 
> 
> - destinationAS
> 
> - nextHopAS 
> 
> - droppedPacketCount
> 
> - samplingInterval
> 
> - droppedByteCount
> 
> 
> 
> Please comment.
> 
> 
> 
> Thanks
> 
> Ganesh
> 
>     
> 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Wed, 01 Oct 2003 11:34:56 -0700

Mark,

>I'm in favor of setting upper bounds, for example counters don't go

I agree that a field even if variable in length is limited by an upper 
bound and this is limit is set on a
case by case basis. The whole point of my first e-mail is rather than 
nailing a field to an explicit
type like uint64_t, to let implementors know that  they can send/receive 
these fields in smaller
sizes too.

>8^8 potential templates for that one stream of flows.
Yes a separate template is needed for varying length. Template explosion 
because if this would be
an implementation choice rather than that imposed by the definition 
itself. Also the same would
have happened if we explicitly define field types for the counters of 
varying length.

Thanks
Ganesh