Subject: RE: [ipfix] export packet length

From: "Reinaldo Penno" <[email-masked]>
Date: Fri, 25 Apr 2003 09:24:51 -0700

That's a good idea.

Other protocols like HTTP and SIP make it mandatory to include the
Content-Length header (which tell the length of the payload) when carried
over TCP. It helps a lot when dealing with fragmentation.

Regards,

> -----Original Message-----
> From: Maurizio Molina [mailto:[email-masked]] 
> Sent: Friday, April 25, 2003 11:50 AM
> To: [email-masked]
> Subject: [ipfix] export packet length
> 
> 
> Hi all,
> I hope not to rise an issue already discussed in the WG. If 
> yes, I apologise in advance. If an IPFIX export packet is 
> carried over TCP, it's not unusual that it gets split across 
> multiple TCP segments. At the collector side, on the 
> contrary, the implementation of the parsing is much simpler 
> if only full export packet are parsed. If the information 
> about the WHOLE export packet were contained in the header, 
> it would be simple to wait until all the bytes composing an 
> export packets are read from the TCP socket before beginning 
> the parsing. Unfortunately, this information isn't currently 
> contained in  Netflow 9 packet header (I suppose, due to 
> Netflow's 9 "bias" towards UDP...). Therefore, a parsing 
> implementation having to deal with this "fragentation" 
> problem without this information needs to be much more 
> complicated (e.g. getting step by step the lenght of each 
> flow set, see if it's all there, parse it and go on....). In 
> summary: if IPFIX has to be carried over tcp, having the 
> length of the export packet in the packet header would be 
> beneficial. Regards, Maurizio
> 
> 
> --
> Maurizio Molina
> Research Staff member
> Network Laboratories Heidelberg
> NEC Europe Ltd.
> Kurfuersten-Anlage 36, 69115 Heidelberg, Germany.
> Tel. (49)6221 90511-18 Fax: (49)6221 90511-55
> e-mail: [email-masked]
> Web: www.ccrle.nec.de
> 
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say 
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
>

---

From: Peter Ludemann <[email-masked]>
Date: Sun, 27 Apr 2003 11:20:36 -0700

Reinaldo/Maurizio:

Benoit's scheme tries to save bytes, so it would be counter-productive 
to also have a length for the entire packet when it is not needed. Here 
are some implementation details ...

If you're using Java, DataInputStream.readFully() will do what you want 
for each flowSet:
    // the calls to "new" can be avoided for efficiency
    byte [] lengthBytes = new byte[2];
    stream.ReadFully(lengthByte2);
    int length = ((lengthByte[0]&0xff) << 8) + (lengthByte[1]&0xff);
    byte [] flowSet = new byte[length];
    stream.ReadFully(flowset, 0, length); // blocks until all bytes for 
flowSet are available

There are C++ libraries that have similar capabilities. (By the way, 
don't just copy this code; there are some subtletities that require more 
complexity; but the same subtleties exist if you have a length for the 
entire NetFlow packet.)

If you're using UDP, instead of depending on IP fragmentation, you might 
want to use something like IBM's "variable block spanned" scheme for 
long records split over shorter blocks (basically, it splits a string 
into as many segments as needed, with a length for each segment and a 
flag indicating whether it's the last segment or not). Again, this could 
be done on a per-flowSet basis. Of course, this doesn't work properly if 
a UDP packet gets dropped; but neither would IP-fragments.

(If your conclusion from all this is that UDP is more trouble than it's 
worth, I'd agree with you.)

- peter

PS: Benoit's 14-bit encoding of string lengths isn't very complicated to 
handle; and it does save a byte!:
        int strLen = buf[pos++]&0xff;
        if (strLen > 0x7f) { strLen = ((strLen&0x7f) << 7) + (buf[pos++] 
& 0x7f); }

Reinaldo Penno wrote:

> Content-Length header (which tell the length of the payload) when 
> carried over TCP. It helps a lot when dealing with fragmentation.
>
> Regards,
>

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Sun, 27 Apr 2003 22:40:34 -0400

> are some implementation details ...

There are other ways to save bytes.

SysUpTime and Unix Secs which are currently sent with every export PDU can
go away when a connection oriented protocol like TCP or SCTP are used.

The SysUptime : Unix_Secs mapping, which should be
SysUpTime : {Unix_Secs,Unix_Msecs} only needs to be sent once per
Source ID per connection.

The current Count header field could be changed to Length....

mark

---

From: Maurizio Molina <[email-masked]>
Date: Mon, 28 Apr 2003 16:47:39 +0200

> The current Count header field could be changed to Length....

Mark,
being a 16 bit counter it would limit the export packet size to 65536 bytes.
Don't you see it as a constraint?
Maurizio

---

From: Maurizio Molina <[email-masked]>
Date: Mon, 28 Apr 2003 17:07:46 +0200

Peter Ludemann wrote:

> There are C++ libraries that have similar capabilities.

Also in a standard C implemenatation you can still live with the information
currently contained in the header (basically using the number of bytes
returned from the socket reading function). It's just more complex and prone
to errors.
I was suggesting that 4 bytes more per each export packet would make life
easier for implementations over TCP.
Regards,
Maurizio

> (By the way,

---

From: Lev Walkin <[email-masked]>
Date: Mon, 28 Apr 2003 13:26:41 -0700

> returned from the socket reading function). It's just more complex and prone

By the way, some stacks (notably, some versions of Linux) could silently
truncate the large _outgoing_ UDP packet. On the remote end of
communication one would receive a perfectly good UDP data, but with the
contents effectively less then expected. The default credit for the
UDP packets with zero checksum (they're allowed to pass as the correct
ones in the majority of the IP stacks) doesn't really help anything.

> easier for implementations over TCP.

As in the DNS protocol (1035): it has a plain structure over UDP and
an extended one (with the length prepended) when transferred over TCP.

-- 
Lev Walkin
[email-masked]

---

From: Mark Fullmer <[email-masked]>
Date: Tue, 29 Apr 2003 11:34:43 -0400

> Don't you see it as a constraint?

It limits the PDU size to 64K.  How this translates to packets depends
on the transport.  I don't think this is an issue.

mark