Subject: [ipfix] Lost Flow Records - Synch Message - Summary up to now

From: "Reinaldo Penno" <[email-masked]>
Date: Thu, 15 May 2003 06:10:20 -0700

I read the whole thread again and got the following points that I would like
to flesh out.

*Collector vs exporter initiated. 

If Exporter initiated the collector will know how many flow records (overall
or per template) were lost associated with the number of lost packets based
on the sequence number.
	Collector send sync message every O(minutes)

If collector initiated, the collector would let the exporter know how many
packets it missed. 
	This might impact the failover process since the exporter wouldn't
be sure of which last sequence number was 		actually received.

Collector initiated whenever it detect gaps in the sequence numbers, export
answers with a synch message.

* Contents of the sync message

	*Flow record counters per template or number of flow records is
enough? The total number of flow records 

	*Lost of template definitions.

---

From: Mark Thibodeau <[email-masked]>
Date: Thu, 15 May 2003 09:59:37 -0400

Exporter initiated synch messages make sense.  If the collector
initiates the synch, there can be ambiguity in the state of the
collector as you mentioned below.

NetflowV9 already has the concept of "Option" templates and data flows.
These option messages can be used to send the total number of flow
records exported.  The Option data can be sent at some configurable
interval.

For content, the number of flow records exported per template may be
required.  Some templates may be aggregated.  Losing an aggregated flow
record can be worse than losing a micro-flow record.  Using only a
global counter does not allow you to figure out how many aggregated
flows have been lost.

As for lost template definitions, this should not happen.  Since IPFIX
will be using TCP, the exporter will know whether or not the template
was received successfully.

/Mark

Reinaldo Penno wrote:

>
>
> I read the whole thread again and got the following points that I
> would like to flesh out.
>
> If Exporter initiated the collector will know how many flow records
> (overall or per template) were lost associated with the number of lost
> packets based on the sequence number.
>
> If collector initiated, the collector would let the exporter know how
> many packets it missed.
>         This might impact the failover process since the exporter
> wouldn't be sure of which last sequence number was            actually
> received.
>
> Collector initiated whenever it detect gaps in the sequence numbers,
> export answers with a synch message.
>
>         *Flow record counters per template or number of flow records
> is enough? The total number of flow records
>

--

****************************************************
* Mark Thibodeau, P Eng.
* 670 RSP Development - Firmware Designer
* Alcatel CID - (613) 784-5375
* Fax - (613) 599-3696
* [email-masked]
****************************************************

---

From: Nevil Brownlee <[email-masked]>
Date: Fri, 16 May 2003 03:11:21 +1200

> collector as you mentioned below.

<WG chair hat OFF>
I agree, exporter sync makes sense to me.

> records exported.  The Option data can be sent at some configurable

Ditto.
 
> was received successfully.

There's been some apparent confusion on the list about transport for IPFIX.
Consensus from the last IPFIX meeting was that the 1.0 version of IPFIX
could use TCP transport [so as to make it easier for developers to implement
and test], but that version 1.0 would NOT rely on two-way messages between
exporter and collector [so that it would be usable in a minimal environment].
So Mark's statement above is not a sufficient answer.
</WG chair hat OFF>

Cheers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: David Moore <[email-masked]>
Date: Thu, 15 May 2003 09:24:22 -0700

> So Mark's statement above is not a sufficient answer.

I'm probably missing something, but how does just using TCP ensure that
the exporter knows that it was received successfully?  (especially
in environments where you can't reach into the guts of TCP).

If the process hands a large buffer to the kernel/OS, it will take
it and let the process continue.  Some or all of that data may be
successfully transmitted via TCP to the receiver.  If the connection
fails, the kernel/OS knows how much of it made it (to the other
kernel, not necessarily application), but it doesn't tell that
information to the process.  All that the process knows is that it
is later told that the connection is gone.  How does it determine
from this whether it was successful or unsuccessful with the
transfer?  Particularly in the case when it is writing additional
data after the template being queued up in the kernel/OS.

-- david

---

From: Peter Ludemann <[email-masked]>
Date: Thu, 15 May 2003 10:21:40 -0700 (PDT)

--- David Moore <[email-masked]> wrote:

[snip]

> If the process hands a large buffer to the kernel/OS, it
> will take it and let the process continue.  Some or all of 
> that data may be successfully transmitted via TCP to the 
> receiver.  If the connection fails, the kernel/OS knows 
> how much of it made it (to the other kernel, not 
> necessarily application), but it doesn't tell that
> information to the process.

[snip]

Which is why the various competitors to NetFlow had their own
ACKs. BTW, even if the TCP implementation allowed access to
the TCP-level ACKs, you'd still want IPFIX-protocol level
ACKs because you need *application* level acknowledgment of
receipt and processing.

Even if you don't want ACKs in the data transmission (despite
the the overhead being low), you do want ACKs in the template
transmission, for the reasons already given by various
people. But this is tricky because you need to ensure that
all the receivers agree (you may have multiple receivers for
fail-over). CRANE was criticised for its fairly elaborate
template negotiation; but even without negotiation, there
remains a fair bit of complexity with ACKing template
information from receivers.

An alternative is to design the protocol so that (a) a
receiver can request template information [and be assured it
receives ALL the information] and (b) the messages can be
handled even if a template is not known (e.g., a template ID,
a version ID, and a message length, so the message can be
skipped). Some of the rejected protocols had this
capabilities already, so you might want to revisit them
before redesign-by-committee.

- peter

---

From: Mark Thibodeau <[email-masked]>
Date: Thu, 15 May 2003 13:39:23 -0400

I agree with both Nevil and David.  The TCP ACK is not enough information to
determine if the Template made it to the Server or not.

I was originallly thinking of an application where the exporter has access to the
guts of the TCP stack.  If the TCP ACK never came for the Template, the exporter
would resend it until it had positive acknowledgement.

But, David is right.  Most applications will probably just use a standard stack and
forget about it.  The information about what was ACKed is not propagated to the
exporter.

I'm not sure about Nevil's point though.  From the IPFIX meeting minutes:

     The evaluation team has recommended NetFlow v9 as a basis for the IPFIX
     protocol.......Then, specify the initial IPFIX transport be TCP.....

Nevil's email seems to say that we will accept either TCP or UDP ("to make it
usable in a minimal environment").  What are we saying?  IPFIX recommends TCP, but
if an application wants to still use UDP then that's OK.  It seems like we haven't
made a decision at all on the transport protocol.  When IPFIX becomes a standard,
what will the transport protocol be?  If an application does not use that transport
protocol, then I assume that it does not comply with the standard.

Or am I totally out to lunch.  :-()

Cheers,
Mark

--

---

From: Alex Audu <[email-masked]>
Date: Thu, 15 May 2003 13:29:45 -0500

Mark,

IPFIX only recommends that the transport protocol be congestion aware.
So,..even
DCCP (a congestion aware variant of UDP) qualifies in this case as a
viable option.

Regards,
Alex.

Mark Thibodeau wrote:

> I agree with both Nevil and David.  The TCP ACK is not enough
> information to determine if the Template made it to the Server or not.
>
> I was originallly thinking of an application where the exporter has
> access to the guts of the TCP stack.  If the TCP ACK never came for
> the Template, the exporter would resend it until it had positive
> acknowledgement.
>
> But, David is right.  Most applications will probably just use a
> standard stack and forget about it.  The information about what was
> ACKed is not propagated to the exporter.
>
> Nevil's email seems to say that we will accept either TCP or UDP ("to
> make it usable in a minimal environment").  What are we saying?  IPFIX
> recommends TCP, but if an application wants to still use UDP then
> that's OK.  It seems like we haven't made a decision at all on the
> transport protocol.  When IPFIX becomes a standard, what will the
> transport protocol be?  If an application does not use that transport

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 15 May 2003 21:03:50 -0400

> determine if the Template made it to the Server or not.

A periodic application level ACK is only needed to ensure that flow
records in flight (in the exporter TCP/IP stack, on the wire, in the
collector TCP/IP stack) are not lost when switching from a primary
to backup collector, or potentially re-establishing a lost connection
to a (primary) collector.

Collector ACK's only need to be done if the exporter has the resources to
resend the window of flows between ACK's, ie it's optional.

I'm assuming for simplicity that an exporter will always send
a template before sending data records that use it, I don't think
this is an unreasonable assumption.  This restriction could probably
be lifted if the collector only ACK'd flows it could decode (ie it
had the template).

If you believe otherwise please provide a concrete example.

mark

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Sun, 25 May 2003 18:12:08 +0200

On torsdag, maj 15, 2003, at 17:11 Europe/Stockholm, Nevil Brownlee 

> So Mark's statement above is not a sufficient answer.

Ok, so I am way behind on email and I am not following this list very 
closely. And I am getting a déja vu feeling on this question...

One thing I am struggling (still) to understand is the use of TCP over 
UDP. I asked this question in San Fransisco and I think the answer was 
that reliable transport was needed. I am not sure I agree (still) with 
this. Isnt't it better to loose some data instead of in worst case 
congestion control coming into play?

- kurtis -

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Sun, 25 May 2003 18:19:50 +0200

> ACKs because you need *application* level acknowledgment of

What if you do this over a bad line, let's say a Ethernet that have 
full/half duplex misconfigured or other problems so that you start 
loosing large amounts of data?

- kurtis -

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Sun, 25 May 2003 18:22:08 +0200

congestion aware for the template only?

What do you do when congestion happens?

- kurtis -

---

From: Peter Ludemann <[email-masked]>
Date: Sun, 25 May 2003 10:01:21 -0700 (PDT)

--- Kurt Erik Lindqvist <[email-masked]> wrote:
> > Which is why the various competitors to NetFlow had their
> > own ACKs. BTW, even if the TCP implementation allowed 
> > access to the TCP-level ACKs, you'd still want IPFIX-
> > protocol level ACKs because you need *application* level 
> > acknowledgment of receipt and processing.
> >
> 
> What if you do this over a bad line, let's say a Ethernet
> that have full/half duplex misconfigured or other problems 
> so that you start loosing large amounts of data?
> 
> - kurtis -

In this case, you'll have a lot of retransmits. Most TCP
implementations provide some method of monitoring the
retransmits: for example, "netstat -s" on Solaris,
/proc/net/* on Linux. [NB: if you were trying to be as
reliable as possible with UDP, you'd also have lots of
retransmits.]

If there is congestion, the application (IPFIX) layer will
see its transmit queue growing (this is separate from the
kernel buffer for TCP). It can make a decision, for example
to break the connection and reestablish it, to fail-over to
another receiver, to throw away old records, throw away new
records, use a higher level of aggregation, etc.

I don't think that these behaviours should be specified in
the protocol; they should simply be allowed. And the protocol
should be general enough to allow informing the receiver of
the behaviour (e.g., a template with summary of
non-transmitted data).

- peter

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Sun, 25 May 2003 22:04:35 +0200

> the behaviour (e.g., a template with summary of

Well, this is what worries me though. If this is data from high speed 
interfaces like 10G, you will produce quite a bit of data in worst 
case. If what we are doing is handling congestion, then we also need to 
queue the data somewhere. For high speed interfaces, this will be _a 
lot_ of data.

I would think that we for most of this would be better to drop the data 
but give the receiver/collector a mechanism to detect that it have lost 
some (sampled) data. Maybe even an indication of how much data.

- kurtis -

---

From: Peter Ludemann <[email-masked]>
Date: Sun, 25 May 2003 15:41:29 -0700 (PDT)

Let's try an actual situation:

  500 byte records
  10,000 records/sec (5 Mbs)
  10 second time-out to start fail-over
  10 seconds to do fail-over

Then, the transmit buffer would need to be 100MB to ensure no
data loss. For safety and to allow for "catch-up"
transmission on fail-over, double that: 200MB. I can buy
256MB of high quality ECC memory for less than US$50, so I
don't see any reason for concern. (If you're sending more
than 10,000 rec/sec from your device, you should probably
reconsider how you're aggregating.)

The TCP buffer is irrelevant to these calculations
(typically, it's 64K-256K) because we cannot remove from our
transmit buffer until we get an ACK from the other end.

If we cannot do a fail-over within the time given (e.g., 20
sec), then we will have to throw away information. Whether we
throw away the oldest, newest, or random: that's a matter for
the implementation.

If we want to guard against longer network failures (e.g., 10
minutes), then we must have more memory for our transmit
buffer. It doesn't matter what transport we are using ... if
we want reliability, we must provide a buffer of this size.

A separate set of data (template) can contain information
such as time period information, total number of bytes sent,
number of bytes acknowledged, etc. This can be used to
communicate data loss information. But this is an issue of
the data model, not of the transport (except that it should
probably be kept in a separate buffer, or sent on a separate
channel, to ensure that it isn't lost).

- peter

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Mon, 26 May 2003 08:34:40 +0200

> reconsider how you're aggregating.)

If this is a router, 256MB is a lot, and it will cost you way much more 
than $50.

> we want reliability, we must provide a buffer of this size.

What I am questioning is if you really want that level of reliability. 
At least in all cases. The costs of doing this are quite high.

- kurtis -

---

From: "Tal Givoly" <[email-masked]>
Date: Mon, 26 May 2003 09:57:21 -0700

I should remind you that records may be compactly encoded in 50-100 bytes.
It doesn't fundamentally change the requirement for buffering, however, it
changes the requirement, in the example from 256 MB to 12.5-25.6 MB. This is
not insignificant, but is definitely more feasible.

Kurt, I agree completely that many applications do not justify this level of
reliability (and associated costs). On the other hand, billing and
accounting in many environments may. For aggregate records, typically not
exposed at 10K rec/sec, it is very much desired as each record may describe
much more cost.

This level of reliability is not a must for the protocol - as was agreed,
the protocol must be extensible enough to support extensions that provide
this level of reliability.

Tal

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Kurt Erik Lindqvist
Sent: Sunday, May 25, 2003 11:35 PM
To: Peter Ludemann
Cc: [email-masked]
Subject: Re: [ipfix] Summary up to now

- kurtis -

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Tue, 27 May 2003 12:42:12 +0200

> not insignificant, but is definitely more feasible.

Way more so.

> this level of reliability.

I am sure there are situations where this level of reliability is 
wished for, but let those that want it pay that price. From what I was 
reading the suggestion was to make TCP the default behavior and I think 
already there we have a problem. Made optional, fine by me.

- kurtis -