Subject: RE: [ipfix] STCP as a default transport not

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 8 Oct 2003 10:23:03 -0700

Hi,

  I guess there is a concern of whether we are approaching this from
the "optimize for the usual case" or "optimize for the worst case".

  The former is usually preferred.

  The general concern seems to be appropriate "capacity" on the 
collector side so that little to no back pressure occurs on the
sender.

  UDP's fire and forget model makes this trivial, but its lack of
congestion aware behavior is apparently a no no.  From the discussion
it sounds like PR_SCTP has similar properties on overflow, where
things just get tossed, like the behavior of the input UDP socket
buffer when the collector is falling behind.

  However, in a mode where the collector is sufficiently endowed
to consume over TCP, which would hopefully be the majority of
situations, taking advantage of TCP's ubiquity and reliability
properties makes a lot of sense to me.

  I'm still dubious of whether disallowing UDP (which happens to 
be the only current concrete binding of NFv9) in favor of SCTP-PR
is really the best approach.

  Personally I'd recommend:

     o bindings over TCP for reliable communication while accomodating for 
       some failover mechanism if the collector becomes congested or
disabled; 
     o bindings of UDP for best effort (i.e. no exporter back pressure) in

       configurations where exporter and collector are coresident and
congestion 
       aware behavior is not a concern; 
     o binding of PR_SCTP where congestion aware behavior and best effort
are 
       desired.

Regards,

  Jeff Meyer

> -----Original Message-----
> From: Stewart Bryant [mailto:[email-masked]]
> Sent: Wednesday, October 08, 2003 6:09 AM
> To: [email-masked]
> Cc: [email-masked]; 'Reinaldo Penno'; 'MEYER,JEFFREY D
> (HP-Cupertino,ex1)'; [email-masked];
> [email-masked]
> Subject: Re: [ipfix] STCP as a default transport not
> 
> 
> 
> 
> Carter Bullard wrote:
> 
> > Hey Stewart,
> > 
> >>We should spend some time thinking about the behavior we want in the
> >>exporter when the network is congested, perhaps due to an
> >>attack. PR-SCTP
> >>gives us the option of running a best effort data collection
> >>to gleen what
> >>is going on, when a TCP based exporter would otherwise 
> collapse due to
> >>backlog on the exporter.
> > 
> > 
> > This is a real engineering issue to talk about.  Issues in 
> producer/consumer
> > dynamics are real problems for IPFIX, especially when the 
> flow record
> > source is generating 1M records per second, which my probes 
> are capable
> > of doing.  In my experience, the reader/collector not keeping up is
> > never a network congestion problem,  it is almost always a 
> reader/collector
> > resource problem, slow disks, not enough memory, competing 
> processes,
> > whatever.  
> 
> The critical issue is which side of TCP does it occur. If TCP 
> has accepted
> the data then the collector application had better do some 
> intelligent.
> My concern is with the exporter or probe having a backlog for 
> any reason.
> 
> > It happens in SMP multi-threaded environments where the flow
> > generator and its collector are on the same machine, it 
> happens when the
> > consumer is on the other end of a dedicated ATM, hyper-channel or
> > infiniband links.  Its not an issue of reliable vs 
> unreliable transport,
> > its more an issue of what does the probe do when it finds 
> it has 1M records
> > that aren't being consumed, and another 1M records are 
> coming very soon.
> 
> On this point we absolutely agree.
> 
> > Does it store them, drop them with an indication?  
> 
> I think that we have to assume that it does not have the storage.
> 
>  From the PR_SCTP draft:
> 
>     One example of partially reliable service is specified in this
>     document, namely a "timed reliability" service.  This 
> service allows
>     the service user to indicate a limit on the duration of 
> time that the
>     sender should try to transmit/retransmit the message (this is a
>     natural extension of the "lifetime" parameter already in the base
>     protocol).
> 
> This seems to have the right properties
> 
> > Does it proactively seek
> > an alternate association, the same association or does it 
> wait for the
> > multiple readers to come back when they are ready.  Are 
> there going to be
> > multiple readers?
> > 
> > Usually the monitor is being attacked, or its generating 
> twice the load
> > because something else is being attacked in the network.  
> No congestion,
> > just a lot of records.  
> 
> Anything that means that the exporter has a bunch of data wedged in
> its output queues waiting for an ACK looks like congestion to 
> the exporter.
> 
> > I don't think PR-SCTP touches these problems.
> > 
> > If you want the record generator to have a proactive role in
> > failover and recover, maybe RTP/RTCP would be a better choice.
> > Explicit feedback control channel provides enough to decide if
> > it should fail over.
> 
> As I recall RTP requires the application to define its own approach to
> congestion, and we do have to have a congestion strategy. It would
> be better to use a congestion aware transport that has a build-in
> strategy to bail-out if there is a problem, and PR-SCTP has just those
> properties.
> 
> Stewart
> 
> 
> > 
> > Carter
> > 
> > 
> >>Stewart
> >>
> >>
> >>
> >>
> >>--
> >>Help        mailto:[email-masked] and say "help"
> >>in message body
> >>Unsubscribe mailto:[email-masked] and say
> >>"unsubscribe ipfix" in message body
> >>Archive     http://ipfix.doit.wisc.edu/archive/
> >>
> > 
> > 
> > 
> > 
> > 
> > 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Thu, 09 Oct 2003 18:15:15 +0200

Jeff,

I would like to comment on your recommendations.

>       some failover mechanism if the collector becomes congested or
But if there is no congestion, I assume that SCTP_PR would do the same 
good job as TCP in term of reliability.
And we know that TCP would have too big memory requirements in case of 
congestion. Hence you propose to failover mechanism.
So why have TCP at all?

>       aware behavior is not a concern; 
We tried very hard this path. It's a NO from the area director, even for 
coresident exporter/collector.

>     o binding of PR_SCTP where congestion aware behavior and best effort
So I conclude that there is only solution left that makes sense! PR_SCTP.

Regards, Benoit.

---

From: Peter Ludemann <[email-masked]>
Date: Thu, 9 Oct 2003 09:47:06 -0700 (PDT)

I want to correct one misunderstanding about TCP/IP ...

Benoit Claise wrote:
> And we know that TCP would have too big memory requirements
> in case of congestion. Hence you propose to failover
> mechanism.
> So why have TCP at all?

The "too big memory requirements" has nothing to do with
TCP/IP and is solely a function of the desired over-all
reliability and granularity of the data transfer.

In most implementations of TCP, you can do blocking or
non-blocking I/O.

In the blocking case, your write is blocked as soon as the
TCP buffer is full (typically, the buffer is 256K or smaller
but you can make it as small as 8K or so if you want). So,
you probably won't do things this way (you can handle this
properly with multi-threading; but the net result is very
similar to non-blocking I/O [next paragraph]).

In the non-blocking case, you get a return code that the
buffer is full. You can then decide whether or not to queue
the data or do something else with it (you have to implement
this queuing yourself). So, the "big memory requirements" has
*nothing* to do with TCP and is only to do with how granular
and reliable you want the information transmission. If you
want to throw data away when the network is congested, TCP
will cooperate very nicely with you. The difference from
using UDP is that the transmitter actually knows what is
being thrown away.

Alternatively, TCP/IP (or another reliable transport) allows
the data transmitter to observe congestion or network outage
and then aggregating statistics until the congestion or
outage goes away.

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Sat, 11 Oct 2003 18:53:31 -0500

>     o binding of PR_SCTP where congestion aware behavior and best effort
Jeff:

But you miss a big point in this... Why not

o bind over SCTP for the reliabile data
o and send PR-SCTP for the case where you have records you are
   willing to give up in the crisis

SCTP in combination with PR-SCTP will:

1) Deliver all reliable information for you JUST LIKE TCP
2) During normal times all the records you are prepared to drop in
     a distress situation will get across TOTALLY reliably.
3) When you hit the rare situation .. the sender will begin to drop
     protecting your network..

So why would you not take the best of both worlds.. which is what SCTP 
can offer
you with PR-SCTP... they work on the same association..

R

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Sat, 11 Oct 2003 18:54:59 -0500

> congestion. Hence you propose to failover mechanism.

Exactly .. in times of no congestion or reader overload SCTP_PR would 
deliver the data JUST LIKE SCTP i.e.
fully reliable... thus you get your cake and eat it too :>

R

> We tried very hard this path. It's a NO from the area director, even 
> for coresident exporter/collector.
>