Subject: KAME SCTP on FreeBSD 5.4/SMP

From: Stephan Baucke <[email-masked]>
Date: Fri, 19 May 2006 16:43:40 +0200

Hi,

I'm currently doing some SCTP benchmarking using various SCTP stacks. 
I've also tried KAME SCTP with FreeBSD 5.4. I'm using Opteron 275 
dual-core machines, so I have compiled the kernel with SMP support.

When I run a simple Netperf test ("netperf -H <host> -t SCTP_STREAM"), 
the kernel reproducably locks up completely (the same test runs just 
fine e.g. on LKSCTP). Using another homebrew benchmark, a few packets 
get through, then the program hangs with ps showing "L+" ("waiting for a 
lock") as the process state.

Are there any known problems with KAME and FreeBSD, particularly with 
SMP? Has anybody successfully run Netperf with KAME? I'd appreciate any 
hints.

BR,
Stephan.

---

From: "Peter Lei (peterlei)" <[email-masked]>
Date: Fri, 19 May 2006 11:38:25 -0500

I'm fairly certain we don't have all the SMP issues worked out
in the FreeBSD 5.4 + KAME snap.  If you can move to FreeBSD 6.0
or FreeBSD 6.1 (directly), you'll enjoy much better success.

I'm not sure if Randy has put up a new tarball up on sctp.org
yet or not; the latest code includes a script to modify and link
the appropriate files in depending on your FreeBSD version (6.0,
6.1, 7.0-current).

--peter

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 19 May 2006 19:22:54 +0200

Hi Stephan,

the KAME SCTP is very old and probably does not work with 5.4. The  
code has
been improved a lot in the meantime. It is currently only 'available'  
via
Randalls CVS server and supports also FreeBSD 6.1.

For some benchmarks you might be interested in looking at the results at
http://sctp.fh-muenster.de
which compared FreeBSD, Linux and Solaris.

Please note that Sun provided an improved version. We are redoing the
measurements but need an even more improved version from Sun.

Best regards
Michael

On May 19, 2006, at 4:43 PM, Stephan Baucke wrote:

> Hi,
>
> I'm currently doing some SCTP benchmarking using various SCTP  
> stacks. I've also tried KAME SCTP with FreeBSD 5.4. I'm using  
> Opteron 275 dual-core machines, so I have compiled the kernel with  
> SMP support.
>
> When I run a simple Netperf test ("netperf -H <host> -t  
> SCTP_STREAM"), the kernel reproducably locks up completely (the  
> same test runs just fine e.g. on LKSCTP). Using another homebrew  
> benchmark, a few packets get through, then the program hangs with  
> ps showing "L+" ("waiting for a lock") as the process state.
>
> Are there any known problems with KAME and FreeBSD, particularly  
> with SMP? Has anybody successfully run Netperf with KAME? I'd  
> appreciate any hints.
>
> BR,
> Stephan.
>

---

From: Stephan Baucke <[email-masked]>
Date: Sun, 21 May 2006 12:28:59 +0200

Thanks Peter. I suspected that the SCTP code in the latest KAME snap 
wasn't current from the modification dates of the source files. I 
wouldn't mind moving to FreeBSD 6.1, but as far as I can tell there is 
no updated code publicly available for that.

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Sun, 21 May 2006 12:43:18 +0200

Hi Michael,

your benchmark results are interesting and have in fact inspired me to 
do some digging myself. :) I found that there are a lot of unexpected 
pitfalls when benchmarking SCTP, partly because of immature 
implementations, partly because of undocumented features (for example I 
noticed more or less accidentally that LKSCTP turns off the Nagle 
algorithm by default, which has a significant impact on throughput when 
sending small messages). Also interesting were comparative benchmarks 
with TCP: Still none of the SCTP stacks that I tried can compete with a 
good TCP implementation in terms of raw throughput.

But anyway, I would really like to try a more recent version of KAME ...

BR,
Stephan.

Michael Tuexen wrote:
> Hi Stephan,
> 
> the KAME SCTP is very old and probably does not work with 5.4. The code has
> been improved a lot in the meantime. It is currently only 'available' via
>> get through, then the program hangs with ps showing "L+" ("waiting for 
>> a lock") as the process state.
>>
>> SMP? Has anybody successfully run Netperf with KAME? I'd appreciate 
>> any hints.
>>
>> BR,
>> Stephan.
>>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 21 May 2006 06:04:13 -0600

Stephan,

On Sun, 21 May 2006, Stephan Baucke wrote:

> Still none of the SCTP stacks that I tried can compete with a 

Then you haven't tried one ;)  See

    http://www.openss7.org/sctp_perf.html

where OpenSS7 SCTP outperformed TCP on Linux even for 1 interface on
a clean network (and devistates it with 2 interfaces).

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Stephan Baucke <[email-masked]>
Date: Mon, 22 May 2006 11:05:50 +0200

Brian,

have you tried these tests with Gbit Ethernet? It looks like both 
protocols where mostly limited by the 100BaseT link. For reference, 
Linux TCP (2.6.16) can saturate GigE (at about 940 Mbps) using 2.2 GHz 
Opteron CPUs.

BR,
Stephan.

---

From: Randall Stewart <[email-masked]>
Date: Mon, 22 May 2006 05:19:57 -0400

Stephan:

Its very hard to imagine an SCTP implementation doing
this with CRC32c done in software...

Most TCP stacks that run over Gig E have a TCP-Checksum
offload that is used as well... If you disable this
you would find TCP hard pressed to fill a Gig pipe.. :-0

And note, I will get the 6.0/6.1 code for FreeBSd up soon..
6.0 is ready but I have not gotten the 6.1 release code
in place yet (we are currently using a 6.1RC1 I think)...
Sometime in the next week or two (I hope)..

R

Stephan Baucke wrote:
>>> Still none of the SCTP stacks that I tried can compete with a good 
>>> TCP implementation in terms of raw throughput.
>>
>>

-- 
Randall Stewart
ITD - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 03:44:44 -0600

Stephan,

I will be performing some GiGE testing soon, but in the tests cited the
2.57GHz P4 processor was not even hitting 10% saturating 2 100Mbps
interfaces, so saturating GiGE will be easy, I have a 3.0GHz 630 HT here
that should be able to saturate 2 GiGE interfaces.

I suppose the real test is not what can be saturated (because a fast
enough processor and bus can saturate anything), a bigger question is
what the CPU utilization is at saturation.  Another question is the
latency through the stack.

I have started using netperf in my tests instead of iperf.  Netperf can
perform Request/Response tests that are much more indicative of stack
latency and scheduling issues than unidirectional stream throughput
tests.  Netperf also supports CPU utilization measurements.

Netperf can be made to support XTI (STREAMS) interface as well and
recent testing with UDP using netperf (STREAMS vs Sockets) has indicated
that Linux Fast-STREAMS outperforms Linux NET4 BSD Sockets by factors
ranging from 2 to 20, particularly on RR tests.  (Don't believe what
you read about STREAMS in the LKML FAQ.)  It will be interesting to see
if OpenSS7 STREAMS SCTP further outperforms the Sockets version thus
providing significant performance gains over NET4 TCP.

Also, the tests performed with UDP and STREAMS tend to indicate that
largest bottleneck on recent Linux kernels is not the protocol stack
implementation, but inefficiencies of the Linux Sockets implementation
(poor locality of code, a mess of non-BSD functions strewn across many
objects, code bloat: resulting in poor cache and pipelining
performance).

I hope to be finished SCTP performance testing well before the August
interop.

--brian

---

From: Stephan Baucke <[email-masked]>
Date: Mon, 22 May 2006 12:51:49 +0200

Brian F. G. Bidulock wrote:

> I will be performing some GiGE testing soon

Maybe I'll beat you to it. :) I was planning to try OpenSS7, but first 
have to dig out a Linux with a 2.4 kernel ...

> but in the tests cited the
> enough processor and bus can saturate anything)

That is what I had hoped, but e.g. LKSCTP reaches only about 540 Mbps on 
my machines with close to 100% CPU load in the best case (using 1400 
byte messages). That is about half of TCP's performance, which then uses 
~85% CPU. The delta gets worse with smaller message sizes (which leads 
me to believe there is more behind it than just TCP's weaker checksum or 
hardware offload).

> tests.  Netperf also supports CPU utilization measurements.

Yes, it is very versatile and easy to use. Be careful with the CPU 
utilization though, because the results sometimes vary significantly 
depending on the measurement method used by Netperf.

> providing significant performance gains over NET4 TCP.

This will indeed be interesting.

> objects, code bloat: resulting in poor cache and pipelining

Could be, but then TCP should suffer from it too, no?

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Mon, 22 May 2006 12:55:34 +0200

> you would find TCP hard pressed to fill a Gig pipe.. :-0

True. My network driver indeed supports checksum offload. I will try 
disabling it, but don't believe that's the whole story ...

> Sometime in the next week or two (I hope)..

That would be great.

Stephan.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:11:15 -0600

Stephan,

> have to dig out a Linux with a 2.4 kernel ...

The STREAMS version runs on 2.6.  I'm sorry but I haven't moved the
sockets version to 2.6 (primarily because all our SIGTRAN and SS7 stuff
uses STREAMS).

> me to believe there is more behind it than just TCP's weaker checksum or 

LKSCTP seems to perform dismally in both the DAST tests and Michael's
tests as well.

> depending on the measurement method used by Netperf.

It performs fairly reliably on Linux when the processor is pinned.  It is
the resulting CPU real-time per transaction info that I find useful.

> Could be, but then TCP should suffer from it too, no?

Yes, I believe it does.  I have even considered a full STREAMS TCP
implementation for comparison.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:12:32 -0600

Stephan,

> disabling it, but don't believe that's the whole story ...

Another way is to try testing on the loopback interface (most stacks
disable checksum altogether on the loopback interface).

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:44:36 -0600

Michael,

On Mon, 22 May 2006, Michael Tuexen wrote:

> KAME and Solaris disable the checksum on Loopback. I suggested
> that also to the LKSCTP guys, but they do not want that.
> 

Strange.  TCP and UDP on Linux disable checksum on loopback.  (The
loopback interface is marked checksum unnecessary.)

--brian

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 22 May 2006 13:37:16 +0200

The bad throughput of LKSCTP for small messages sizes is due
to some bugs in LKSCTP. They have been reported, but I do not
know if they have been fixed. I got no real feedback...

I'm currently retesting the KAME and Solaris stuff and will
retest the LKSCTP stuff after that...

Best regards
Michael

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 22 May 2006 05:20:31 -0700

> disable checksum altogether on the loopback interface).

Perhaps the most appropriate test would be one that timed delivery
of payload that required at least CRC32 protection, and supplied it
at the application layer when the transport did not.

After all, it is unlikely that any real application that requires high
bandwidth could really live with only TCP's 16-bit protection.
If you have a lot of data, then it needs protecting. If you don't
then you are probably more interested in measuring latency.

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 22 May 2006 14:08:44 +0200

Hi Brian,

thanks for letting me know. I'll re-suggest that after the
next round of measurements. The 'do it like TCP and UDP'
argument might help.

Best regards
Michael

---

From: Randall Stewart <[email-masked]>
Date: Mon, 22 May 2006 10:24:20 -0400

> disabling it, but don't believe that's the whole story ...

My machines (2.4 Gig Xeon and 2.8 Gig Xeon) were getting
about 640Meg or so in some testing I was doing during
SMP locking testing..

Still have some work to go.. since I had bad cards at the
time and I am not sure if the problem was the cards or the
locking..

I was getting 20% idle yet.. and still not saturating the
link.. but I think I was seeing errors :-0

The cards were failing the em0 card checksum too.. I now
have new "good" cards.. but have not had time to re-test :-0

>> Sometime in the next week or two (I hope)..

I should have it sometime next I am out of the country
right now so I will have time next week to get this
sorted out...

As to the disabling of the assisted c-sum.. its tricky
you have to go in and play with the TCP stack to disable
it.. either that or just add a in_chksum() (or whatever
the function is) call even when its driver supported..

Lots of fun.. I did it a while ago.. have not tried
it with the recent stack though..

R

---

From: Randall Stewart <[email-masked]>
Date: Mon, 22 May 2006 10:29:01 -0400

> then you are probably more interested in measuring latency.
That is actually a great idea..

Write a modifyed netperf (or whatever Brian said he was using) that
adds a CRC32C to it for TCP.. and does not for SCTP.. this would
give you a pretty good comparison.. given the framing and the crc32c in
tcp against sctp's built in framing and crc32c...

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 09:09:55 -0600

Randall,

> tcp against sctp's built in framing and crc32c...

Well, no, that would not be fair: CRC32C performed during copy from user
in SCTP is cheaper than the user performing it before the send in TCP.

--brian

---

From: Randall Stewart <[email-masked]>
Date: Mon, 22 May 2006 11:40:00 -0400

> in SCTP is cheaper than the user performing it before the send in TCP.
Hmm.. I guess so.. it would not matter that much though..

On the other hand.. just forcing even the TCP checksum on
the interface (and not using the CRC32c at all) though
biased against SCTP.. would give one an idea on how
well the implementation was working..

As to Caitlin's idea.. in some ways it is fair.. since, as Caitlin
points out, if you are doing highspeed data RDMA/iSCSI etc.. you WILL
be doing it in user land and loose this advantage..

Either test would be interesting to me :-D

R

---

From: "Sridhar Samudrala" <[email-masked]>
Date: Mon, 22 May 2006 10:56:29 -0700

> that also to the LKSCTP guys, but they do not want that.

I don't remember saying that we do not want to disable checksum on
loopback. It is currently implemented that way. We could easily fix it so that
checksum is not done on loopback. I will try to get this done pretty soon.

> know if they have been fixed. I got no real feedback...

Unfortunately i am not able spend much time on LKSCTP to improve its
performance. Any help from community to improve LKSCTP performance
is more than welcome.

Thanks
Sridhar

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 22 May 2006 20:14:19 +0200

Hi Sridhar,

see my comments in-line.

Best regards
Michael

> I don't remember saying that we do not want to disable checksum on
I think someone said that this should not be done on Linux... Do not
know why.
> checksum is not done on loopback. I will try to get this done  
That is great. This helps a lot on loopback.
> performance. Any help from community to improve LKSCTP performance
I'll retest Linux in combination with Solaris and FreeBSD soon, since  
both
have changed a lot in the meantime.
I'll let you know about problems, however I can not provide fixes...

---

From: Stephan Baucke <[email-masked]>
Date: Mon, 22 May 2006 12:51:49 +0200

Stephan.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:12:32 -0600

Stephan,

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:12:32 -0600

Stephan,

--brian

---

From: Stephan Baucke <[email-masked]>
Date: Mon, 22 May 2006 12:51:49 +0200

Stephan.

---

From: Stephan Baucke <[email-masked]>
Date: Mon, 22 May 2006 12:55:34 +0200

That would be great.

Stephan.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:11:15 -0600

Stephan,

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 22 May 2006 05:44:36 -0600

Michael,

--brian