Subject: RE: [ipfix] Congestion management

From: "Carter Bullard" <[email-masked]>
Date: Mon, 13 Oct 2003 12:22:59 -0400

Hey Peter,
   The issue is that PR-SCTP is being advocated as the
protocol for both reliable and un-reliable transport
for IPFIX.  In my due diligence in trying to understand
if PR-SCTP is appropriate for the unreliable transport
requirements of IPFIX, I have to get past the statements
of enthusiastic support, which are always optimistic and
never revealing, and get to the reality of the protocol.

   First and foremost, and this has been reiterated by
all so far, PR-SCTP is SCTP with only minor changes.
When running in its most unreliable mode of operation
PR-SCTP imposes states for the sender and receiver,
there are memory requirements and protocol interfaces
that a switch/router will have to support for SCTP's
congestion control, which cannot be turned off.  These
are resources that a router currently does not have to
spend when transporting Netflow over UDP.

   In most installations of IPFIX (> 90%), congestion control
is not a requirement, as the collector is directly connected
to the switch/router.  In these cases, PR-SCTP may not
be an appropriate protocol.  It is very important that
everyone realize that PR-SCTP is not being advocated because
it can be as efficient as Netflow over UDP when it is
operating as an unreliable transport.

   But for those 10% that do need congestion control, where
the data is traversing multiple hops, and PR-SCTP may be
appropriate, you run into issues like NAT and you have
new requirements for security.  These are issues that are
real problems for some people that do need to be mentioned
and discussed.

Carter

> -----Original Message-----
> From: Peter Lei [mailto:[email-masked]]
> Sent: Monday, October 13, 2003 9:49 AM
> To: [email-masked]
> Cc: [email-masked]; 'David Moore'; 'Nevil Brownlee'
> Subject: Re: [ipfix] Congestion management
>
>
> Carter Bullard wrote:
> > Hey Peter,
> >    You have to incur SCTP-PR functions regardless if you
> > use them or not.  The draft is very clear on this fact.
> > SCTP-PR doesn't modify the wire protocol, so -PR doesn't
> > reduce any overhead states or messages.  It simply modifies
> > the API and some sender semantics.  And, of course, you don't
> > get some required features such as congestion detection,
> > without turning on some rather constrained behavior.  For
> > instance you can't do any kind of loss detection with
> > SCTP-PR unless you use in order delivery (no "U" bits set).
> > That can't be too efficient.  And the header is huge, what
> > is it? a common header, a chunk header and a data
> > header, we're talking, what, 32 bytes?
>
> I'm not sure what you are getting at here... of course the
> functions have to be there in order for you to get the
> functionality...
>
> Please re-read the draft... PR-SCTP does indeed extend the
> on-the-wire protocol; it is NOT just an API change.
>
> The setting of the "U" bits does NOT "disable" loss detection;
> the whole association is congestion controlled.  And that is
> the point.  I'd like to have IPFIX and provide reliable and
> un-reliable service while being congestion controlled; and I
> can do this with one association between a probe and collector.
> This is why IMO PR-SCTP fits so nicely.
>
>
> > One problem I see is that IPFIX will use IPSec for its security.
> > Without RFC 3554, "On the Use of Stream Control Transmission
> > Protocol (SCTP) with Ipsec", which is not widely implemented,
> > IPSec isn't compatible with SCTP features like multi-homing,
> > basically making the fail-over mechanisms unuseable, and
> > of course NAT doesn't work with multi-homing.
>
> Yes, without RFC 3554 one cannot do multi-homed SCTP w/IPsec
> efficiently.  I would argue this is no different than the TCP
> case today, i.e. you need multiple associations.  However, at
> least SCTP gives you the *possibility* of doing this with one
> association once IPSec implementations catchup w/3554.
>
> > I'm sure you feel that SCTP-PR is the most efficient mechanism
> > for transporting bulk records at 100 Mbps across a single hop
> > dedicated link, but does anyone have any real experience to
> > back this position?
>
> Well, it is probably not the "most efficient"... but this isn't
> the general case we're designing IPFIX for, are we?  If so, we
> can forget the NAT arguments.  And yes, we definitely have done
> high rate bulk data transfers with SCTP/PR-SCTP.
>
> regards,
> --peter
>
> > Carter
> >
> >
> >
> >
> > -----Original Message-----
> > From: Peter Lei [mailto:[email-masked]]
> > Sent: Friday, October 10, 2003 12:06 AM
> > To: [email-masked]
> > Cc: [email-masked]; 'David Moore'; 'Nevil Brownlee'
> > Subject: Re: [ipfix] Congestion management
> >
> >
> > Carter Bullard wrote:
> >
> >>Hey David,
> >>   SCTP-PR is partial reliable, not unreliable.  You
> >>can't seem to turn off the failure detection and
> >>protection capabilities of normal reliable SCTP, nor can
> >>you turn off the congestion avoidance logic of SCTP, so
> >>it is definitely more complex and more expensive compared
> >>to say RTP/UDP, and probably TCP.
> >
> >
> > PR-SCTP can definitely be "unreliable" if you so choose;
> > have the service definition be "try this only once please"
> > and the stack will only trasmit each send() call once.
> >
> > If the transport choice is to be "congestion controlled",
> > why would NOT do congestion avoidance?!  The feedback loop
> > is exactly what allows SCTP to have TCP-friendly congetion
> > control.
> >
> > I see failure detection as a benefit, not a detriment...
> > It certainly would be nice to know that the transport has
> > "gone away" so I can maybe failover to an alternate
> > collector.  Plus, I don't want to send packets needlessly
> > if the collector has somehow crashed (e.g. might cause
> > congestion?)
> >
> >
> >>   The key is to find a transition technology for the
> >>1000's of existing IP flow monitors that are purely
> >>UDP based today.  A simple transitional technology
> >>should be appropriate, and I think RTP/UDP is a good
> >>one, especially since the IESG has historically accepted
> >>it as passing the congestion awareness criteria.
> >>
> >>   I'm personally concerned that the only mentioned
> >>transport candidates are connection-oriented and unicast
> >>only.  There are serious advantages to connection-less
> >>transports and multicast support, and I'm interested in
> >>ensuring that IPFIX can explore those advantages.  Neither
> >>SCTP-PR or TCP can support multicast, so I'm an advocate
> >>of a three transport model.
> >
> >  >
> >
> >>   But Nevil is right, the real issue is what should be the
> >>default transport for IPFIX, and here it seems to me that
> >>SCTP-PR should NOT be the default, although there is enough
> >>consensus for it to be a supported transport.  TCP seems to
> >>be the only real candidate for the default, simply from an
> >>interoperability standpoint.  If you've only got one shot
> >>at attaching to an IP flow probe, are you going to use SCTP-PR
> >>or TCP, if the probe could support either, but not both?
> >>I think TCP will be the higher probability choice, which
> >>would make it the default, by default.
> >
> >
> > I disagree... picking TCP as the default tranport for reliable
> > data then requires you to use RTP/UDP for the unreliable data
> > (unless you want to send it ALL over TCP)...
> >
> > SCTP w/PR-SCTP extension allows both (over the same association
> > even, if desired) and IMO seems to provide all the transport
> > services that IPFIX requires.
> >
> > regards,
> > --peter
> >
> >
> >>Carter
> >>
> >>
> >>
> >>
> >>
> >>>-----Original Message-----
> >>>From: David Moore [mailto:[email-masked]]
> >>>Sent: Thursday, October 09, 2003 7:37 PM
> >>>To: Carter Bullard
> >>>Cc: 'Nevil Brownlee'; [email-masked]
> >>>Subject: Re: [ipfix] Congestion management
> >>>
> >>>
> >>>On Thu, Oct 09, 2003 at 07:04:39PM -0400, Carter Bullard wrote:
> >>>
> >>>
> >>>>  When you're dealing with 1M+ records per second, the
> >>>>concepts of record loss detection and selective retransmission
> >>>>are just not practical.  With these types of monitors, if
> >>>>there is packet loss, there really isn't any way that the probes
> >>>>can slow down, so, like RFC 3497, the concept would have to be
> >>>>to have some type of feedback signal so the probe can stop.
> >>>>So RTP/RTCP sounds good.  I'd rather IPFIX send the message,
> >>>>but the concept of IPFIX being congestion aware itself seems
> >>>>to have dropped through a fissure somewhere.
> >>>
> >>>With SCTP-PR, afaik, allows you to tune whether you want any
> >>>particular pieces of sent data to be reliable or unreliable.  By
> >>>setting every piece to be unreliable, then you have UDP in terms
> >>>of reliability.  However, SCTP-PR has the feedback signal which
> >>>already provides TCP-friendly congestion avoidance.
> >>>
> >>>Is there some performance cost to SCTP-PR when all data is marked
> >>>unreliable that you feel is too high, to require building
> a different
> >>>feedback mechanism on top of UDP to also have congestion avoidance?
> >>>
> >>>I understand the desire for a UDP transport which has no congestion
> >>>avoidance.  But if the goal is something UDP-like (unreliable) but
> >>>which does have congestion avoidance, IPFIX should use something
> >>>already a standard (or about to be a standard) rather than
> inventing
> >>>its own.
> >>>
> >>>
> >>>-- david
> >>>
> >>
> >>
> >>
> >>
> >>
> >>--
> >>Help        mailto:[email-masked] and say
> "help" in message
> >
> > body
> >
> >>Unsubscribe mailto:[email-masked] and say
> >>"unsubscribe ipfix" in message body
> >>Archive     http://ipfix.doit.wisc.edu/archive/
> >>
> >
> >
> >
> >
> >
> > --
> > Help        mailto:[email-masked] and say
> "help" in message body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> >
>
>

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Peter Lei <[email-masked]>
Date: Mon, 13 Oct 2003 14:15:54 -0500

> never revealing, and get to the reality of the protocol.

Nor would I argue that we shouldn't do due diligence here...

> spend when transporting Netflow over UDP.

Yes, as I mentioned before, you DO need to have those memory/
protocol requirements there... however, they would already
exist if you are doing reliable export with SCTP.  Unless you
are going to mandate multiple transports (like SCTP + x/UDP,
or TCP + x/UDP), I don't believe you will get the efficiency
that you want from a single default transport without doing
something like SCTP + PR-SCTP.

As mentioned in another thread in this list, one can constrain
the memory utilization of the PR-SCTP stack.

> operating as an unreliable transport.

We already know we aren't going to get a transport protocol as
efficient as UDP if it must be congestion controlled.  And as
mentioned several times on this list, congestion control IS a
hard requirement.

Given that IPFIX must have congestion control anyways with the
default transport, that's why SCTP+PR-SCTP is being advocated.
If there wasn't that requirement, folks would still be using
UDP for flow export... or maybe that's what you are proposing??

I haven't seen an explicit recommendation for an alternative
so far, beyond UDP or RTP/UDP.

> real problems for some people that do need to be mentioned

Agreed... and at the worst case, I don't believe these are
any more difficult than the TCP case w/multi-homing.  You can
always "fall back" to having singly homeed associations that
look like TCP connections-- in that scenario, SCTP will work
fine with exisiting IPSec and NAT implementations.

--peter