Subject: Re: [ipfix] Variant Field Types.

From: Stewart Bryant <[email-masked]>
Date: Wed, 01 Oct 2003 07:47:55 +0100

We need to consider IEs on a case by case basis, but I
can see that the collector designer would have difficulty
with completely open-ended IE sizes.

One approach would be for ipfix to declare a recommended
maximum for an object, but allow the exporter to exceed that.
The other is, as Jeff proposes, to set a hard limit. Either
way I think that we agree that the exporter can choose to
use a smaller IE size.

In the soft limit case, presumably the exporter would only
exceed the recommended limit for good reason, and a collector
unable to operate with an IE that large would have to flag
an error.

The difference between a hard and a soft upper bound it how
far out we set the upper limit. The twin problems of
undersizing and backwards compatibility are notorious in
this industry leading to all sorts of work-around. A hard
limit design would therefore need to set the limit way
above what currently seemed reasonable. Would this then
lead to unreasonable storage requirements in the collector?

Although we could perhaps regard the collector storage
capacity as approximating infinity, what about an aggregator?

Stewart

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> Hi,
> 
>   One possible way to address Stewart's concern, which seems 
> primarily around exporters not wanting to fill in 64-bit fields,
> is to separate the concern of the information model and the
> encoding.
> 
>   Specifically, since the current protocol specification 
> explicitly states lengths used in templates, rather than the
> type, an exporter could chose to send quantities in 4 bytes
> or 2 bytes or whatever.  The consumer would know that the
> quantity was actually targeted at some specified size and
> process it accordingly.
> 
>   Obviously this only works for "downsizing" data items, not
> upsizing them.  But since the arguments seem around "constrained"
> systems, then downsizing would appear to be the use case.
> 
>   Changes in things like AS number from 16 to 32-bytes are
> changes in the information model, I'd prefer to see these
> reflected with explicit updates.
> 
>   If explicit types are good enough for C, Java, and DB's I
> imagine they're good enough for IPFIX.
> 
> -- Jeff
> 
> 
>>-----Original Message-----
>>From: Stewart Bryant [mailto:[email-masked]]
>>Sent: Tuesday, September 30, 2003 10:06 AM
>>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
>>Cc: [email-masked]
>>Subject: Re: [ipfix] Variant Field Types.
>>
>>
>>
>>
>>MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
>>
>>>Hi,
>>> 
>>>  Although the use of variant types may seem convenient from the 
>>>perspective of an
>>>exporter, I think the various consumers of the data would 
>>
>>benefit from 
>>
>>>sticking with
>>>an explicit type system.
>>> 
>>
>>Variant types allow better use of resources and the ability to upgrade
>>fields without the need to firstly get IETF approval, and secondly
>>synchronise deployment of the exporter and collector upgrade.
>>
>>Anything that counts is a good example. To cope with the high end
>>systems we really need to make them all 64bits. That means that
>>tiny system that are memory and export limited and which may never
>>wrap a 16 or 32 bit counter need to devote resources to 64 bits.
>>
>>If we were designing this say 5 years ago we would probably not have
>>considered 32 bit AS numbers, now we need to. There are bound to be
>>other IE that need to be upgraded, and for each and every one we need
>>to synchronise both exporter and collector.
>>
>>By introducing this flexibility we get greater scope for how we handle
>>these upgrades.
>>
>>
>>>  Consider operations which involve summing fields, storing 
>>
>>them to a 
>>
>>>DB, etc.
>>> 
>>
>>The collector storage is less likely to be the long pole in the tent
>>than the exporter or the transmission bandwidth to the collector. The
>>collector can therefore more readily be implemented to accept the
>>largest reasonable value than can the exporter.
>>
>>
>>>  Explicit typing ensures that these consuming applications 
>>
>>can handle 
>>
>>>these values
>>>appropriately and also makes the information modeller think 
>>
>>about how these
>>
>>>types are to be used.
>>
>>But they have to think worst case during deployment, and my contention
>>is that this places undue stress on the exporter and on the network
>>bandwidth.
>>
>>
>>> 
>>>  The examples you cite:
>>> 
>>>- ingressPort
>>>- egressPort
>>>- packetCount
>>>- byteCount
>>>- sourceAS
>>>- destinationAS
>>>- nextHopAS
>>>- droppedPacketCount
>>>- samplingInterval
>>>- droppedByteCount
>>> 
>>>  fall into a few categories, identifiers (such as ports and AS's), 
>>>counts and intervals.
>>> 
>>>  For the identifiers, I believe the size is fixed by the 
>>
>>relevant RFC's 
>>
>>>(i.e. why would I
>>>have a 1 byte Port?).  
>>
>>Clearly but not long ago you would have said "why would I 
>>have a 4 byte
>>AS number"
>>
>>Also you are thinking in terms of the "examples" we have in 
>>the current
>>RFCs. We need to design this both for future RFCs and for 
>>vendor specific
>>IEs.
>>
>>
>>>For counts, the use of 32 vs. 64 bits is a valid 
>>>question, however
>>>these can be handled by making them explicit (i.e. the 
>>
>>information model 
>>
>>>says if
>>>I get this value it is a 32-bit quantity or a 64-bit 
>>
>>quantity and the 
>>
>>>code will address
>>>it accordingly).  For intervals, the resolution would 
>>
>>determine which 
>>
>>>size is most
>>>appropriate.
>>
>>But this then makes it harder for tiny systems or cramps 
>>large systems.
>>
>>
>>> 
>>>  The option always exist to define variations on these data items 
>>>explicitly.  E.g.
>>>byteCount32 and byteCount64.
>>
>>Thay leads to two (or more) varients that the collector has 
>>to support.
>>The alternative is for the collector to support the larger, 
>>and the exporter
>>to support what ever size it wants. An alternative way to 
>>think of this
>>is as a data compression technology.
>>
>>
>>> 
>>>  If there is a need to vary the sizes for various fields I'd much 
>>>rather see the rationale
>>>for each discrete sizing made explicit vs. forcing all the 
>>
>>downstream 
>>
>>>consumers of this \
>>>information have to deal with special cases and ambiguity.
>>> 
>>
>>That requires too much (unnecessary) synchronisation between 
>>exporter and collector.
>>
>>Regards
>>
>>Stewart
>>
>>
>>> 
>>>Regards,
>>> 
>>>  Jeff Meyer
>>>
>>> 
>>>
>>>    -----Original Message-----
>>>    From: Ganesh Sadasivan [mailto:[email-masked]]
>>>    Sent: Monday, September 29, 2003 6:00 PM
>>>    To: [email-masked]
>>>    Subject: [ipfix] Variant Field Types.
>>>
>>>Hi,
>>>
>>>The information model <draft-ietf-ipfix-info-01.txt> only
>>>has types for fixed length objects and variable length 
>>>string like objects.
>>>
>>>There are a number of information element types (listed below)
>>>that would be better defined in terms of fixed-size run-time
>>>specified objects. The type variation comes due to difference
>>>in the size of the objects used because of an implementation,
>>>as demanded by the network or otherwise.
>>>
>>>The way to make this work is to use the template
>>>length field to determine the exact sub-type:
>>>
>>>The definitions of a variant field type can be stated as:
>>>
>>>type varUnsignedInt
>>>    if (element.length == 1)
>>>         type = unsignedByte
>>>    else if (element.length <= 2)
>>>         type = unsignedShort
>>>    else if (element.length <= 4)
>>>         type = unsignedInt
>>>    else if (element.length <= 8)
>>>         type = unsignedLong
>>>   else type = undefined
>>>
>>>type varInt
>>>    if (element.length == 1)
>>>         type = byte
>>>    else if (element.length <= 2)
>>>         type = short
>>>    else if (element.length <= 4)
>>>         type = int
>>>    else if (element.length <= 8)
>>>         type = long
>>>   else type = undefined
>>>
>>>The candidates that fit into the description above are (I may
>>>have missed some also):
>>>- ingressPort 
>>>- egressPort 
>>>- packetCount 
>>>- byteCount 
>>>- sourceAS 
>>>- destinationAS
>>>- nextHopAS 
>>>- droppedPacketCount
>>>- samplingInterval
>>>- droppedByteCount
>>>
>>>Please comment.
>>>
>>>Thanks
>>>Ganesh
>>
> 
> --
> Help        mailto:[email-masked] and say "help" in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/