Subject: RE: [ipfix] Variable length fields data type

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 16 Apr 2003 16:22:17 -0700

Hi,

Obviously you need to fix the type codes up front.  Ethernet address
would be contained in an octetString.  Or we decide up front that MACAddr
is a special type (6 bytes) in order to save 2 bytes.

I suppose if you think that there's going to be lots of new and important
fixed length fields which aren't basic numeric quantities, and you're 
concerned with the 2 byte overhead, you could continue with the NFv9
approach.  However, if you think the majority of the data items you'll
have in the future fall into the existing categories, you use octetString,
as your back door for any subsequent extensions.  No ambiguity, things 
decode fine...

jeff

> -----Original Message-----
> From: Mark Fullmer [mailto:[email-masked]]
> Sent: Wednesday, April 16, 2003 4:10 PM
> To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> Cc: [email-masked]
> Subject: Re: [ipfix] Variable length fields data type
> 
> 
> If the encoding length is not specified in the template, an 
> unrecognized
> "type code" received by the collector would result in a template that
> can not be processed -- more correctly the data records could not be
> decoded.  With the existing Type and Length style encoding 
> unrecognized
> fields can be ignored and skipped over.
> 
> For example, if EthernetAddress was used by the exporter and 
> unknown to
> the collector, the offset of the next field could not be computed 
> because the collector doesn't know how many bytes an 
> EthernetAddress is.
> 
> mark
> 
> On Wed, Apr 16, 2003 at 02:10:47PM -0700, MEYER,JEFFREY D 
> (HP-Cupertino,ex1) wrote:
> > Hi,
> >  
> >   I'd suggest an alternate approach with regards to the templates.
> >  
> >   Today the templates carry two pieces of information for 
> each attribute in
> > a flow record:  attribute identifier, an integer value from a single
> > namespace, and length.
> >  
> >   One can still send two factors in the template, but convey more
> > information by sending:  attribute identifier, an integer 
> value and a "type
> > code."  The type space I would recommend is the following:
> >  
> >      boolean
> >      int8
> >      uint8
> >      int16
> >      uint16
> >      int32
> >      uint32
> >      int64
> >      uint64  ? debateable
> >      real32
> >      real64
> >      utf8
> >      octetString
> >      ipV4Addr
> >      ipV6Addr
> >  
> >  Each type would have a unique (two octet) identifier, and 
> the lengths are
> > assumed fixed for each type, with the exception of utf8 and 
> octetString.  So
> > the encoding would not change in the flow record, and the 
> encoding for utf8
> > and octetString would use 2 octets (or 4) length field.
> >  
> >  This helps address extension, where a collector 
> implementation may not
> > recognize the attribute id, but would at least be able to 
> render the field
> > in a more intelligible manner.  I.e. just having a length 
> doesn't tell me if
> > I have a signed or unsigned integer or an ipV4Addr.
> >  
> >   Distinguishing between arbitrary byte sequences (octetString) and
> > printable strings (utf8) would also aid in presentation.
> >  
> >   For boolean I would encode this as an octet with a value 
> of 0 or 1, to
> > avoid bit level alignment.
> >  
> >   This would also help the data model.  I.e. you define how 
> each of the
> > various primitive types are encoded once and each attribute 
> simply indicates
> > what type it is.  Today's document has lot's of repeating 
> text describing
> > how each attribute is encoded.  Good engineering practice 
> says that lots of
> > copy paste should be addressed through indirection to a 
> central definition.
> >  
> > Regards,
> >  
> >   Jeff Meyer
> > 
> > -----Original Message-----
> > From: Benoit Claise [mailto:[email-masked]]
> > Sent: Wednesday, April 16, 2003 7:35 AM
> > To: [email-masked]
> > Subject: [ipfix] Variable length fields data type
> > 
> > 
> > Dear all,
> > 
> > During the last IETF, in both IPFIX and PSAMP WGs, it was 
> correctly noted
> > that the current basis protocol for IPFIX has got a 
> limitation regarding the
> > export of variable length data types. So for example for 
> the export of a
> > text or for the export of packet sample (whose length can 
> vary in PSAMP). 
> > 
> > Here is a proposal: in the template Flowset (template 
> definition), let's put
> > FFFF for the length of the variable length data types.
> > 
> > Then, the actual length is encoded in data Flowset (flow records) as
> > follows:
> > 
> > - if length is < 255 bytes, it stored in 1 byte
> > 
> > - if length is >=255 bytes, 255 is stored in the first 
> byte, and the actual
> > 
> >   16 bit length is stored in the next 2 bytes.
> > 
> > 
> > 
> > The cases of length of >= 255 will be very rear. Even PSAMP 
> packet fragments
> > should not have a length above 255 bytes, otherwise we 
> would report just too
> > much paylod and we would have privacy issues.
> > 
> > And even in these cases, one extra byte will represent <= 
> 1/255-th part of
> > data record.
> > 
> > 
> > 
> > Why FFFF and not 0?
> > 
> > The reason for this is that FFFF is a truely invalid value 
> which should be
> > 
> > caught in collectors, by good implementations, today. For 0 
> this isn't
> > 
> > necessary the case as it's meaning is "just" pointless, but 
> not a course
> > 
> > of concern for the collectors. 
> > 
> > Basically FFFF should make the collector (current 
> implementations) discard
> > the template as being invalid.
> > 
> > Note also that some option templates data types could 
> potentially have a
> > length of 0.
> > 
> > 
> > 
> > Do we need a length coded on more than 16 bits?
> > 
> > I don't think so!
> > 
> > Even if PSAMP would decide to report the full packet (not 
> permitted by RFC
> > 2804 btw), the packet length will never need more than 16 
> bits to be coded
> > as the length in an IP packet is
> > 
> > coded on 16 bits. Note also that the Length field in the 
> Data FlowSet
> > 
> > Format is coded on 16 bits; so can't exceed a variable 
> length of 64 k -
> > 
> > 8 bytes (In the Data FlowSet, the FlowSet ID + the FlowSet 
> Length + 1 byte
> > for 255 + the
> > 
> > 16 bits variable length)
> > 
> > 
> > 
> > So the advantages of this proposal are:
> > 
> > - code the length on 1 byte for most of the cases
> > 
> > - use the 8 bits to code the length, so with a max of 254 bytes
> > 
> >   And not using the Most Significant Bit for the length:
> > 
> >     first bit reserved (if 0, then length max of 7 bits = 127 bytes)
> > 
> >     first bit is 1, then the length is coded on 14 bits 
> max; 7 bits from
> > 
> >     the 1st bytes and 7 last bits from the second bytes. So 
> max length = 16k
> > for 2 bytes
> > Note: with flow records containing variable length fields, 
> the collecting
> > process will have to decode each flow in the FlowSet before 
> determining how
> > many flow records are part of the Data Flowset ... due to 
> the fact that
> > there is a length field in the data flowsetBut, with flow 
> records not
> > containing variable length fields, the collecting process 
> can deduce the
> > number of  the number of flow records in  a Data Flowset 
> (by dividing the
> > flow set length by record length) at the time of the packet 
> reception. 
> > 
> > Any feedback?
> > 
> > Regards, Benoit
> > 
> > 
> > 
> > 
> > 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 17 Apr 2003 11:42:42 -0400

IMHO, the current v9 Type and Length templates should stay as is.

mark