Subject: RE: [ipfix] Variable length fields data type

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 16 Apr 2003 14:10:47 -0700

Hi,
 
  I'd suggest an alternate approach with regards to the templates.
 
  Today the templates carry two pieces of information for each attribute in
a flow record:  attribute identifier, an integer value from a single
namespace, and length.
 
  One can still send two factors in the template, but convey more
information by sending:  attribute identifier, an integer value and a "type
code."  The type space I would recommend is the following:
 
     boolean
     int8
     uint8
     int16
     uint16
     int32
     uint32
     int64
     uint64  ? debateable
     real32
     real64
     utf8
     octetString
     ipV4Addr
     ipV6Addr
 
 Each type would have a unique (two octet) identifier, and the lengths are
assumed fixed for each type, with the exception of utf8 and octetString.  So
the encoding would not change in the flow record, and the encoding for utf8
and octetString would use 2 octets (or 4) length field.
 
 This helps address extension, where a collector implementation may not
recognize the attribute id, but would at least be able to render the field
in a more intelligible manner.  I.e. just having a length doesn't tell me if
I have a signed or unsigned integer or an ipV4Addr.
 
  Distinguishing between arbitrary byte sequences (octetString) and
printable strings (utf8) would also aid in presentation.
 
  For boolean I would encode this as an octet with a value of 0 or 1, to
avoid bit level alignment.
 
  This would also help the data model.  I.e. you define how each of the
various primitive types are encoded once and each attribute simply indicates
what type it is.  Today's document has lot's of repeating text describing
how each attribute is encoded.  Good engineering practice says that lots of
copy paste should be addressed through indirection to a central definition.
 
Regards,
 
  Jeff Meyer

-----Original Message-----
From: Benoit Claise [mailto:[email-masked]]
Sent: Wednesday, April 16, 2003 7:35 AM
To: [email-masked]
Subject: [ipfix] Variable length fields data type

Dear all,

During the last IETF, in both IPFIX and PSAMP WGs, it was correctly noted
that the current basis protocol for IPFIX has got a limitation regarding the
export of variable length data types. So for example for the export of a
text or for the export of packet sample (whose length can vary in PSAMP). 

Here is a proposal: in the template Flowset (template definition), let's put
FFFF for the length of the variable length data types.

Then, the actual length is encoded in data Flowset (flow records) as
follows:

- if length is < 255 bytes, it stored in 1 byte

- if length is >=255 bytes, 255 is stored in the first byte, and the actual

  16 bit length is stored in the next 2 bytes.

The cases of length of >= 255 will be very rear. Even PSAMP packet fragments
should not have a length above 255 bytes, otherwise we would report just too
much paylod and we would have privacy issues.

And even in these cases, one extra byte will represent <= 1/255-th part of
data record.

Why FFFF and not 0?

The reason for this is that FFFF is a truely invalid value which should be

caught in collectors, by good implementations, today. For 0 this isn't

necessary the case as it's meaning is "just" pointless, but not a course

of concern for the collectors. 

Basically FFFF should make the collector (current implementations) discard
the template as being invalid.

Note also that some option templates data types could potentially have a
length of 0.

Do we need a length coded on more than 16 bits?

I don't think so!

Even if PSAMP would decide to report the full packet (not permitted by RFC
2804 btw), the packet length will never need more than 16 bits to be coded
as the length in an IP packet is

coded on 16 bits. Note also that the Length field in the Data FlowSet

Format is coded on 16 bits; so can't exceed a variable length of 64 k -

8 bytes (In the Data FlowSet, the FlowSet ID + the FlowSet Length + 1 byte
for 255 + the

16 bits variable length)

So the advantages of this proposal are:

- code the length on 1 byte for most of the cases

- use the 8 bits to code the length, so with a max of 254 bytes

  And not using the Most Significant Bit for the length:

    first bit reserved (if 0, then length max of 7 bits = 127 bytes)

    first bit is 1, then the length is coded on 14 bits max; 7 bits from

    the 1st bytes and 7 last bits from the second bytes. So max length = 16k
for 2 bytes
Note: with flow records containing variable length fields, the collecting
process will have to decode each flow in the FlowSet before determining how
many flow records are part of the Data Flowset ... due to the fact that
there is a length field in the data flowsetBut, with flow records not
containing variable length fields, the collecting process can deduce the
number of  the number of flow records in  a Data Flowset (by dividing the
flow set length by record length) at the time of the packet reception. 

Any feedback?

Regards, Benoit

---

From: Mark Fullmer <[email-masked]>
Date: Wed, 16 Apr 2003 19:10:21 -0400

If the encoding length is not specified in the template, an unrecognized
"type code" received by the collector would result in a template that
can not be processed -- more correctly the data records could not be
decoded.  With the existing Type and Length style encoding unrecognized
fields can be ignored and skipped over.

For example, if EthernetAddress was used by the exporter and unknown to
the collector, the offset of the next field could not be computed 
because the collector doesn't know how many bytes an EthernetAddress is.

mark

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/