Subject: Re: [Tsvwg] sctp performance

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 7 Oct 2004 06:11:29 -0600

Salvatore,

Performance tests on a clean network do not exercise SCTP's strong
points.  SCTP was not really intended to outperform TCP on a clean
network with a single interface: it was intended to outperform TCP on a
real network with noise, failures and reconfiguration.  If you look at
even single stream tests under noise SCTP rapidly outperforms TCP in
dirty networks.

The SOCK_SEQPACKET test deviation from theoretical is probably more of a
software glitch in bundling, Nagle'ing and Minshall'ing in the OpenSS7
stack more than a problem with the protocol (that was noted in the
results).

The SOCK_STREAM tests on a single interface did outperform TCP, however,
note that the tests saturated the network and both are running very
close to theoretical maximums.

The interesting thing about the OpenSS7 tests were the multiple
interface tests on SOCK_STREAM, that showed that max throughput was a
multiple of the number of available interfaces.  Once TCP saturates
a single interface it has nowhere else to go.  SCTP can load multiple
interfaces to saturation.

These were throughput tests.  Latency tests show SCTP has comparable
results to TCP (with slight improvements) on clean networks, and
SCTP exhibits significant reduced latencies to TCP under even as little
as 5% noise.  On distrupted networks (50% interface failure on a 1 or
2 second duty cycle) TCP stops and SCTP continues.

There are a lot of good papers out there testing more real environments
as well as NS-2 simulations that demonstrate the benefits of SCTP over
TCP for its original intended application, telephony signalling.  Search
on "SCTP Performance" and you can read for the better part of a day.

--brian

On Thu, 07 Oct 2004, Salvatore Loreto wrote:

> 
>    Hi all,
> 
> 
> 
>    reading the experiments on SCTP performance
> 
> 
> 
>    LKSCTP
> 
>    [1]http://datatag.web.cern.ch/datatag/WP3/sctp/tests.htm
> 
>    OPENSS7
>    [2]http://www.openss7.org/sctp_perf.html
> 
> 
> 
>    If  we  ignore  the  gap  underline between  TCP  and  SCTP in LKSCTP,
>    becose due   to   implementation   problem   ( and  I  think  now  the
>    situation is  better,  isn't  it? does someone have up to date results
>    about?)
> 
>    and use just the result underline on OPENSSS7
> 
> 
> 
>    I've some questions/doubts and I'd like know you opinion about:
> 
> 
> 
>    1)  the  test  SCTP with 1 interface shows a substanzial equivalent to
>    TCP.
> 
>    Is  it  possible SCTP with 1 interface is TCP, or if you want if there
>    is    not HoL block there are not difference between two protocols due
>    to the same congestion algorithm.
> 
> 
> 
>    2)  SCTP  with  2  interface has a better performance just for message
>    size greater than 64-byte.
> 
>    But  SCTP was born for Signaling transport, and signaling messages are
>    yes lager than 64-bye but not very large.
> 
> 
> 
>    when in the experiment is used SOCK_SEQPACKET
> 
>    3)  the  real  start up with 1 or 2 interfaces are both worse than the
>    Theoretical  one.  The  strange  is the reale one with 2 interfaces is
>    worse than the reale one with just one interface.
> 
>    4)  similar  considerations  ( done above ) could be done also for the
>    other phases . So SOCK_SEQPACKET seems not so good.
> 
> 
> 
> 
> 
>    regards
> 
>    sal
> 
> 
> 
> 
>      _________________________________________________________________
> 
>    [3]Yahoo!  Mail  ti  protegge  gratis  dalla  posta indesiderata e dai
>    virus!
> 
> References
> 
>    1. http://datatag.web.cern.ch/datatag/WP3/sctp/tests.htm
>    2. http://www.openss7.org/sctp_perf.html
>    3. http://it.rd.yahoo.com/mail/taglines/*http://it.mail.yahoo.com

> _______________________________________________
> tsvwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/tsvwg

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart <[email-masked]>
Date: Thu, 07 Oct 2004 08:50:02 -0400

Brian/Salvatore:

Just a couple of comments...

> stack more than a problem with the protocol (that was noted in the

I would tend to agree with Brian here.. SOCK_SEQPACKET is really
an implementation shade.. In KAME other than controlling a
few minor things all the same code runs... so it sould make
very little difference....

> on "SCTP Performance" and you can read for the better part of a day.

A couple of other things:

a) The CERN tests were done on an OLD OLD OLD version of lk-sctp which
    was NOT performance tuned at all.. I am not even sure if they have
    completed a pass yet of tuning.. I know its on their to-do list.

b) IPERF ... which is used on both sites.. to me.. is NOT a very good
    benchmark. SCTP sends "messages" from one end to the other. TCP sends
    a byte stream from one end to the other. You then take and use IPERF
    to test the number of bytes streaming from one end to the other.
    This puts SCTP at a dis-advantage. Especially considering that MOST
    ALL applications send messages NOT streams of bytes. So if you added
    some additional overhead to TCP.. say framing messages at the app
    layer.. this would slow TCP down I would wager.

    I have always wanted to develop a variable message sender that could
    send a sequence of variable messages and get a reading of messages
    per second for various sizes....

c) CRC32c is extremely expensive (in overall terms of CPU use) on the
    TCP side they generally have an off-load engine they can use to
    not even do the c-sum... so in such a case when you begin to
    get CPU limited SCTP will lag behind TCP... just on the checksum
    basis alone. Bottom line is once you get CRC32c offload SCTP will
    be able to do a bit better :-D

Regards

R

-- 
Randall Stewart
ITD
803-345-0369 <or> 815-342-5222

---

From: Pawel Hadam <[email-masked]>
Date: Thu, 07 Oct 2004 15:25:56 +0200

Hi

>    completed a pass yet of tuning.. I know its on their to-do list.

I was working with linux kernel 2.6.7 with lkSCTP 1.0.1 and I obtained 
much better results.

>    per second for various sizes....

Sounds like my own testing apps. I wrote a sender and receiver that 
accept some customization parameters (message size, buffers).

I wrote a raport about my measurements and experiments with lkSCTP on 
1Gb network, but:

1) It is now being read by my supervisors.
2) It is in french :) - I really am sorry, it was written for a french 
project.

Once the raport finished, I will, I hope, try to publish something in 
this subject.

If anybody wants the french version, I may send him the PDF (there are a 
lot of pictures :))). Any comments are always welcome.

Regards
Pawel

---

From: Pawel Hadam <[email-masked]>
Date: Thu, 07 Oct 2004 16:03:33 +0200

(I am sorry for my previous, signed e-mail)

Hi

Regards
Pawel

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 7 Oct 2004 09:21:46 -0600

Pawel,

Please send me a copy of the report when it is available.

Thank you.

--brian

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 7 Oct 2004 11:41:03 -0400 (EDT)

Hi,

Just adding another pointer -

Sourabh Ladha had some (throughput) results on the KAME stack on FreeBSD
4.6 (TCP NewReno):
http://www.cis.udel.edu/~amer/PEL/poc/pdf/IPCCC2004-ladha.FTP.over.SCTP.pdf

regards,
jana

> lot of pictures :))). Any comments are always welcome.

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 7 Oct 2004 09:55:56 -0600

Janardhan,

Interesting: at 0% probability of loss, single homed KAME SCTP performed
poorer than TCP.  It looks like SCTP implementors still have something to
learn from TCP implementors in some cases.  SCTP wasn't really aimed
exclusively at greater than 3% message loss networks...  Signalling networks
normally have far better than 1ppm BER.

--brian

---

From: Pawel Hadam <[email-masked]>
Date: Thu, 07 Oct 2004 18:49:35 +0200

Hi

> OPENSS7
> http://www.openss7.org/sctp_perf.html

I am interested in a case with 2 interfaces.
How did they do the load-balancing between 2 interfaces?
It was on SCTP or application (iperf ???) level ???

Does anybody know?

Pawel

---

From: Randall Stewart <[email-masked]>
Date: Thu, 07 Oct 2004 13:17:15 -0400

> normally have far better than 1ppm BER.

Hmm..

The results are interesting.. but I think your 3% figure is a
bit off..

Thats only one of the many graphs present...

Some considerations of these results:

a) Netbsd does NOT apply maxburst in TCP. So this will
    definetly influence small transfers... and if you
    note the larger the transfer the less the difference
    between the graphs.. in fact the 1Meg transfer in the
    later pages looks like possibly SCTP in all cases did
    better instead of in 2 out of 3 cases as it did in all others.

b) Note that at ALL times if you use all of sctp .. multi-streaming
    etc. it does better.. it is only one of the 3 graphs that is slower
    in small transfers.

c) 2002 was about the time I was performance optimizing KAME.. which
    actually needs to have a second pass... so I am not really sure
    which version Sourabh was actually using...

d) Considering the very very little difference between TCP and SCTP
    even in these cases .. thats pretty darn good.. especially when
    you consider that the netbsd stack as been tweaked and tuned for
    at least 10 years... and KAME was about 1 - 1.5 years when Sourabh
    did this work... all in all I think the results quite positive :-D

Note to Salvatore: Yes KAME does have H-S TCP in it.. its a configurable
option when you compile it. And no I have no papers on any results since
my focus as been in other places then that :-0

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 7 Oct 2004 20:06:09 -0600

Pawel,

The load balancing is performed in the SCTP implementation.  It's the same
algorithm that allowed us to acheive 20ms nominal fail-over times between
interfaces in lmco's testing.

--brian

---

From: Janardhan Iyengar <[email-masked]>
Date: Tue, 12 Oct 2004 10:30:11 -0400 (EDT)

Hi Brian/all,

Sorry about the late reply - I was on a paper deadline.

My thoughts have been voiced by Randy and Prof. Amer. The experiments were
performed about 2 years ago, and performance tuning has happened since
then in the KAME stack.

> It looks like SCTP implementors still have something to learn from TCP
> implementors in some cases.

I'm not sure much else can replace performance tuning that happens with
time and usage. That a new SCTP could match up to an aged TCP stack is
quite good, I think.

regards,
jana

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Wed, 13 Oct 2004 14:26:09 -0400 (EDT)

Sorry for reviving an old thread, but I'm just getting caught up on this
list.

> interfaces to saturation.

I am curious about SCTP is able to increase throughput as the number of
interfaces increase. Jana has showed that simply using all paths with SCTP
is counter-productive. The transport has to be "smart" about getting
around different issues when doing Concurrent Multipath Transfer (CMT).
Jana has found several issues with "naive" CMT, but has done a nice
job of addressing them one by one. He's still in the process of tackling
more of these issues.

http://www.cis.udel.edu/~iyengar/publications/2004.spects.iyengar.pdf
http://www.cis.udel.edu/~iyengar/publications/2004.icon.iyengar.pdf

~armando

0--                                              --0
| Armando L. Caro Jr.  |  Protocol Engineering Lab |
| www.armandocaro.net  |    University of Delaware |
0--                                              --0

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 13 Oct 2004 15:24:14 -0600

Armando,

As I'm sure you know, the OpenSS7 SCTP implementation has been using a
weighted version of the algorithms from the second paper since 2001.  It
also uses a fast restoration technique for restoring suspect
destinations since 2001.  The implementation also uses a (patentable) FR
algorithm that is far simpler than the split-FR presented in the first
paper.  These allow the implementation to exhibit multiplicative
throughput and single RTO fail-over simultaneously.  Neither algorithm
violates AIMD, FR or max burst rules.  They are in fact more
conservative.

I believe I sent you this weighted algorithm some years ago and said it
was worthy of attention. ;)

Perhaps more interesting than just the pure multiplicative increase is
the way that the implementation smoothly adapts and moves out of the way
of a single-interface TCP load resulting in maximization of both
throughputs.  Unfortunately, those results haven't been presented yet.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 14 Oct 2004 05:33:10 -0600

Randall,

Armando listed two papers:

1st paper:  http://www.cis.udel.edu/~iyengar/publications/2004.spects.iyengar.pdf
2nd paper:  http://www.cis.udel.edu/~iyengar/publications/2004.icon.iyengar.pdf

--brian

On Thu, 14 Oct 2004, Randall Stewart wrote:

> > weighted version of the algorithms from the second paper since 2001. 
> 
> Question....
> 
> What is the "second paper" ... are you refering to some of the
> work Armando and Jana have been doing at the U-D or some other
> paper?
> 
> R
> --
> Randall Stewart

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Thu, 14 Oct 2004 12:36:47 -0400 (EDT)

> was worthy of attention. ;)

To be honest, I didn't remember you sending me an algorithm. I dug through
my old mails and found a private discussion we had in Nov 2001. I found
the weighted algorithm that you're talking about, but the algorithm you
sent me was for selecting a destination to retransmit to and failover to.
Since the email was sent privately, I don't want to post the algorithm in
this public forum without your permission. But in any case, the
algorithm/selection process looks reasonable, but I don't see how it
applies to Concurrent Multipath Transfer (CMT).

Maybe I am misunerstanding you, and you are not doing CMT? Are you saying
that additional interfaces multiplies the throughput performance because
your failover algorithm is so agressive? You are advocating using PMR=0,
RTO.Min=0-10ms, etc. I have also done work that comes to the conclusion
that PMR=0 provides the best performance
(http://www.armandocaro.net/papers/2004.TR2005-04.acaro.pdf), but an
RTO.Min=0-10ms would only work on special case networks (eg, SS7)  and not
the Internet.

~armando

0--                                              --0