Subject: RE: [ipfix] Variant Field Types.

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 1 Oct 2003 14:59:45 -0400

Hi,
 
  I think the details of using smaller encoding in an implementation belong
in the protocol specification vs. the information model.  The information
model already references the protocol specification for how fields of a
given type are encoded.
 
  For specific fields, there is still a decision on what would be the
maximal type.  E.g. do ports need to be 32-bit or 16?  Do AS numbers need to
be 32 or 64?  Given the unit of measure for SampleInterval, what is its
limit?  These details would go in the information model.
 
Regards,
 
  Jeff Meyer

-----Original Message-----
From: Ganesh Sadasivan [mailto:[email-masked]]
Sent: Wednesday, October 01, 2003 11:35 AM
To: Mark Fullmer
Cc: MEYER,JEFFREY D (HP-Cupertino,ex1); [email-masked]
Subject: Re: [ipfix] Variant Field Types.

Mark,

Mark Fullmer wrote:

I'm in favor of setting upper bounds, for example counters don't go

I agree that a field even if variable in length is limited by an upper bound
and this is limit is set on a
case by case basis. The whole point of my first e-mail is rather than
nailing a field to an explicit
type like uint64_t, to let implementors know that  they can send/receive
these fields in smaller
sizes too.

past 64 bits.  If you don't do this, what's a collector to do with say

a 72 bit counter?  You can't store it in a uint_64_t anymore so collectors

would have to implement arbitrary precision arithmetic to handle potential

cases.  I don't think this is reasonable to impose on the collectors.

The issue I have with the variable length fields as currently specified

is the potential for template explosion.  If a template has 8

counters, each of which could be 1..8 byte then you have

8^8 potential templates for that one stream of flows.

Yes a separate template is needed for varying length. Template explosion
because if this would be
an implementation choice rather than that imposed by the definition itself.
Also the same would 
have happened if we explicitly define field types for the counters of
varying length. 

Thanks
Ganesh

I was hoping for the sake of the collectors we could keep the active

templates down to something < 64K so a template lookup could be done

with an index instead of a hash.

mark

On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D (HP-Cupertino,ex1)
wrote:

Ganesh,

  I'm not clear on how this benefits the collector.  Tal and myself produce

different implementations of carrier deployed products which collect from

100's of devices and dozens of protocols, perhaps our experience here is

worth something, and it tells us variant is undesireable on the collector...

  Regarding your other point:

One way to solve this problem is if the collector assumes the largest size

for 

computations?

  This was my point in the e-mail to Stewart.  Have the information model be

EXPLICIT about what is the largest size which is being modeled for a data

item and declare it.  Allow the exporter to declare a smaller size in the

template, if the exporter knows that an individual record will always be

accomodated (based on its implementation) in the smaller size.

  So for instance this might say that byteCount should be declared in the

information model as a 64-bit int, while AS is probably sufficient at

32-bits.

  This accomodates the:

      exporter - can use smaller size if known

      n/w - can reduce BW if know a smaller size is used

      info model - explicit is better than "vague"

      collector - addresses persistence and aggregation issues

  Can we call this a compromise?

Regards,

  Jeff Meyer

-----Original Message-----

From: Ganesh Sadasivan [ mailto:[email-masked]
<mailto:[email-masked]> ]

Sent: Tuesday, September 30, 2003 10:45 AM

To: MEYER,JEFFREY D (HP-Cupertino,ex1)

Cc:  [email-masked] <mailto:[email-masked]> 

Subject: Re: [ipfix] Variant Field Types.

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

Hi,

  Although the use of variant types may seem convenient from the perspective

of an

exporter, I think the various consumers of the data would benefit from

sticking with

an explicit type system.

It would benefit the exporter, the n/w, flexibility of interpretation of

fields in the 

information model  and the collector.

  Consider operations which involve summing fields, storing them to a DB,

etc.

One way to solve this problem is if the collector assumes the largest size

for 

computations?

  Explicit typing ensures that these consuming applications can handle these

values

appropriately and also makes the information modeller think about how these 

types are to be used.

  The examples you cite:

- ingressPort 

- egressPort 

- packetCount 

- byteCount 

- sourceAS 

- destinationAS

- nextHopAS 

- droppedPacketCount

- samplingInterval

- droppedByteCount

  fall into a few categories, identifiers (such as ports and AS's), counts

and intervals.

  For the identifiers, I believe the size is fixed by the relevant RFC's

(i.e. why would I 

have a 1 byte Port?).  For counts, the use of 32 vs. 64 bits is a valid

question, however

these can be handled by making them explicit (i.e. the information model

says if

I get this value it is a 32-bit quantity or a 64-bit quantity and the code

will address

it accordingly).  For intervals, the resolution would determine which size

is most

appropriate.

For AS the size could be 2 or 4 bytes.

You may prefer to use a 1 byte port if you are a small system as against a 4

byte for 

a gigantic system. Both have their own implementation reasons.

  The option always exist to define variations on these data items

explicitly.  E.g.

byteCount32 and byteCount64.

Yes. But is'nt this not more of an overhead to introduce new field type as

the requirements

come in and then make sure that it gets deployed in exporter and collector?

  If there is a need to vary the sizes for various fields I'd much rather

see the rationale

for each discrete sizing made explicit vs. forcing all the downstream

consumers of this \

information have to deal with special cases and ambiguity.

This is not a special case handling as you can see that the list above

covers about 

40% of the fields defined today in the information model. It is way to

handle 

ambiguity arising out of implementation differences.

Thanks

Ganesh

Regards,

  Jeff Meyer

-----Original Message-----

From: Ganesh Sadasivan [  mailto:[email-masked]
<mailto:[email-masked]> 

 <mailto:[email-masked]> <mailto:[email-masked]> ]

Sent: Monday, September 29, 2003 6:00 PM

To:  [email-masked] <mailto:[email-masked]>
<mailto:[email-masked]> <mailto:[email-masked]> 

Subject: [ipfix] Variant Field Types.

Hi,

The information model <draft-ietf-ipfix-info-01.txt> only

has types for fixed length objects and variable length 

string like objects.

There are a number of information element types (listed below)

that would be better defined in terms of fixed-size run-time

specified objects. The type variation comes due to difference

in the size of the objects used because of an implementation,

as demanded by the network or otherwise.

The way to make this work is to use the template

length field to determine the exact sub-type:

The definitions of a variant field type can be stated as:

type varUnsignedInt

    if (element.length == 1)

         type = unsignedByte

    else if (element.length <= 2)

         type = unsignedShort

    else if (element.length <= 4)

         type = unsignedInt

    else if (element.length <= 8)

         type = unsignedLong

   else type = undefined

type varInt

    if (element.length == 1)

         type = byte

    else if (element.length <= 2)

         type = short

    else if (element.length <= 4)

         type = int

    else if (element.length <= 8)

         type = long

   else type = undefined

The candidates that fit into the description above are (I may

have missed some also):

- ingressPort 

- egressPort 

- packetCount 

- byteCount 

- sourceAS 

- destinationAS

- nextHopAS 

- droppedPacketCount

- samplingInterval

- droppedByteCount

Please comment.

Thanks

Ganesh

    

--

Help         mailto:[email-masked]
<mailto:[email-masked]>  and say "help" in message body

Unsubscribe  mailto:[email-masked]
<mailto:[email-masked]>  and say

"unsubscribe ipfix" in message body

Archive      http://ipfix.doit.wisc.edu/archive/
<http://ipfix.doit.wisc.edu/archive/>

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 02 Oct 2003 20:03:12 +0100

Jeff

Whether the size reduction fits in the IM or in the protocol draft
depends on how closely you see the drafts mapping onto the
structure of the implementation.

Ultimately the exporter and the collector have to have the same
shared IM, including knowledge of the object size that the exporter
is actually using.

One obvious example is counter wrapping (there may be others).
Lets say IM defines a 16 bit counter object, and the collector
uses this to interpret the data it receives from the exporter. The
exporter on the other hand reduces this to 8 bits. The sequence
7,8,0 means quite different things to each party. In the case
above the exporter saw 9 object and the collector thought
there had been 17. The collector can only correctly determine
what ocured if it knows that the exporter is using the abbreviated
size.

Now you could regard the use of a different size as a protocol matter
i.e. compression is being applied to the data, but to get the correct
shared semantics, the protocol engine now has to be aware of the
true semantics of IEs rather than just passing them up as abstract
objects.

Therefore I would think that the part of the system that understands
the IEs needs to be aware of the size that the exporter is using,
and this leads me to conclude that we might as well describe the
size behavior in the IM.

Stewart

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

> Hi,
>  
>   I think the details of using smaller encoding in an implementation 
> belong in the protocol specification vs. the information model.  The 
> information model already references the protocol specification for how 
> fields of a given type are encoded.
>  
> maximal type.  E.g. do ports need to be 32-bit or 16?  Do AS numbers 
> need to be 32 or 64?  Given the unit of measure for SampleInterval, what 
> is its limit?  These details would go in the information model.
>  
> Regards,
>  
>   Jeff Meyer
> 
>     I agree that a field even if variable in length is limited by an
>     upper bound and this is limit is set on a
>     case by case basis. The whole point of my first e-mail is rather
>     than nailing a field to an explicit
>     type like uint64_t, to let implementors know that  they can
>     send/receive these fields in smaller
>     sizes too.
> 
>     Yes a separate template is needed for varying length. Template
>     explosion because if this would be
>     an implementation choice rather than that imposed by the definition
>     itself. Also the same would
>     have happened if we explicitly define field types for the counters
>     of varying length.
> 
>     Thanks
>     Ganesh
> 
>>On Tue, Sep 30, 2003 at 04:20:07PM -0400, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
>>  
>>
>>>Ganesh,
>>> 
>>>Cc: [email-masked]
>>Help        mailto:[email-masked] and say "help" in message body
>>Unsubscribe mailto:[email-masked] and say

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/