Subject: [ipfix] Variable length fields data type

From: Benoit Claise <[email-masked]>
Date: Wed, 16 Apr 2003 16:35:04 +0200

Dear all,

During the last IETF, in both IPFIX and PSAMP WGs, it was correctly 
noted that the current basis protocol for IPFIX has got a limitation 
regarding the export of variable length data types. So for example for 
the export of a text or for the export of packet sample (whose length 
can vary in PSAMP).

Here is a proposal: in the template Flowset (template definition), let's put FFFF for the length of the variable length data types.
Then, the actual length is encoded in data Flowset (flow records) as follows:
- if length is < 255 bytes, it stored in 1 byte
- if length is >=255 bytes, 255 is stored in the first byte, and the actual
  16 bit length is stored in the next 2 bytes.

The cases of length of >= 255 will be very rear. Even PSAMP packet fragments should not have a length above 255 bytes, otherwise we would report just too much paylod and we would have privacy issues.
And even in these cases, one extra byte will represent <= 1/255-th part of data record.

Why FFFF and not 0?
The reason for this is that FFFF is a truely invalid value which should be
caught in collectors, by good implementations, today. For 0 this isn't
necessary the case as it's meaning is "just" pointless, but not a course
of concern for the collectors. 
Basically FFFF should make the collector (current implementations) discard the template as being invalid.
Note also that some option templates data types could potentially have a length of 0.

Do we need a length coded on more than 16 bits?
I don't think so!
Even if PSAMP would decide to report the full packet (not permitted by RFC 2804 btw), the packet length will never need more than 16 bits to be coded as the length in an IP packet is
coded on 16 bits. Note also that the Length field in the Data FlowSet
Format is coded on 16 bits; so can't exceed a variable length of 64 k -
8 bytes (In the Data FlowSet, the FlowSet ID + the FlowSet Length + 1 byte for 255 + the
16 bits variable length)

So the advantages of this proposal are:
- code the length on 1 byte for most of the cases
- use the 8 bits to code the length, so with a max of 254 bytes
  And not using the Most Significant Bit for the length:
    first bit reserved (if 0, then length max of 7 bits = 127 bytes)
    first bit is 1, then the length is coded on 14 bits max; 7 bits from
    the 1st bytes and 7 last bits from the second bytes. So max length = 16k for 2 bytes

Note: with flow records containing variable length fields, the 
collecting process will have to decode each flow in the FlowSet before 
determining how many flow records are part of the Data Flowset ... due 
to the fact that there is a length field in the data flowsetBut, with 
flow records not containing variable length fields, the collecting 
process can deduce the number of  the number of flow records in  a Data 
Flowset (by dividing the flow set length by record length) at the time 
of the packet reception.

Any feedback?

Regards, Benoit
_
_

---

From: Mark Fullmer <[email-masked]>
Date: Wed, 16 Apr 2003 16:49:00 -0400

It's MTU barriers that need to be avoided.  Is there anything that can
move more than a 9000 byte GigE Jumbo frame?

I don't think privacy issues should play a role in defending the 16K
limit.  An implementation would artifically limit the snap length to
something much much smaller than 16K if privacy were an issue.

The work done to save one byte when the field length is < 254 seems
a bit excessive...It probably doesn't matter one way or the other.

mark

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Thu, 17 Apr 2003 12:02:30 +0200

Mark,

>It's MTU barriers that need to be avoided.  
>
Why? Why not let IP fragmentation do its job?

>Is there anything that can
What about 16 Mbit/s Token Ring, 17914 bytes?

I don't think this is a good idea to limit the max. size of a variable length data type assuming a layer 2 technology.
What technologiy/MTU will we have in the future?

>something much much smaller than 16K if privacy were an issue.
Agreed.

>a bit excessive...It probably doesn't matter one way or the other.
Well, it comes from the experience that having a full metering process 
(non sampled) could produce a lot of data!

Regards, Benoit.

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 17 Apr 2003 11:33:36 -0400

> Why? Why not let IP fragmentation do its job?

The maximum size packet that can be sampled will be largest frame that
can be put on the wire, not the maximum sized IP packet, which would
be fragmented.

> What about 16 Mbit/s Token Ring, 17914 bytes?

Isn't this a good argument that the 16K limit is too small then?

> What technologiy/MTU will we have in the future?

My point was that layer2 MTU's are much less than the the 64K limit of an
IP datagram and that a device will be sampling from layer2 where
IP datagrams > MTU would be fragmented to something smaller.

> (non sampled) could produce a lot of data!

It's just 8 bits!

mark

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 29 Apr 2003 11:07:58 +0100

> Even if PSAMP would decide to report the full packet
 > (not permitted by RFC 2804 btw), the packet length will
 > never need more than 16 bits to be coded as the length
 > in an IP packet is coded on 16 bits.

I hope that this does not degenerate into a rat-hole, but I
don't think that RFC2804 actually says this. RFC2804 just says
that the IETF will not standardise wiretapping (which has
all sorts of legal complexities).

Stewart