Subject: aggregating bursts of messages

From: CRV§ADER//KY <[email-masked]>
Date: Sat, 01 Jul 2006 14:43:33 +0200

I wrote a simple SCTP client under linux that

1)sets STCP_NODELAY to 0
2)connects to a a stcp_darn server on localhost
3)sends 3 "hello world" messages using 3 stcp_send() calls
4)sleep(3)
5)sends other 3 "hello world" messages.

I sniffed the traffic and got 14 packets!
INIT
INIT_ACK
COOKIE_ECHO
COOKIE_ACK
DATA
SACK
DATA
SACK
DATA
SACK
(3 seconds later)
DATA
SACK
DATA DATA
SACK

That's incredibly inefficient!!! My target, since I'm sending 3 messages
at a time, is to send just **6** packets (correct me if I'm wrong, but
packets containing COOKIE_ECHO or COOKIE_ACK can carry data too):

INIT
INIT_ACK
COOKIE_ECHO DATA DATA DATA
COOKIE_ACK SACK
(3 seconds later)
DATA DATA DATA
SACK

or, in the worst case, it's really important not to send more than 8:
(4-way handshake)
DATA DATA DATA
SACK
(3 seconds later)
DATA DATA DATA
SACK

In other words, I either need something like
send(int msg_count, void **messages, int *msg_len)

or something like
send_enable(0);
send();
send();
send();
send_enable(1);

to send more than one message at a time, and I really don't think mine
is a particularly exotic request.

Are there any existing solutions I missed?

---

From: Randall Stewart <[email-masked]>
Date: Thu, 06 Jul 2006 06:48:03 -0400

Let me see if I understand the basic's of
what you want to do.. I can't comment on the linux
implementation since I do not know how they support
the Nagle algorithm..

Setting SCTP_NODELAY to 0 means you want Nagle on... this
should be the default.. but setting it on does not hurt.

Then you would like to send 3 messages a pause and then
3 more messages... getting the most bundling you can.

There are two socket models to use this in..

If you use the one-to-one model (which I think you are
doing).. you should see

INIT-->
<--INIT-ACK
---COOKIE-ECHO-->
<--Cookie-Ack
----DATA---->
<---SACK----
----DATA+DATA---->
<---SACK----
<time>
-----DATA--->
<---SACK----
-----DATA+DATA--->
<---SACK----

Assuming of course some reasonable delay talking to
the peer... say a 1ms RTT... if you are talking to the
loopback this may skew things since the response can
be so fast that there is no time for any data to get
written before a response shows up... depending on
the scenarios..

The other model would be one-to-many.. in this case
you should see:

----INIT---->
<---INIT-ACK---
----COOKIE-ECHO+DATA+DATA+DATA--->
<---COOKIE-ACK+SACK-----
<time>
----DATA---->
<---SACK----
----DATA+DATA---->
<---SACK-----

Now the above of course our ideal situation and
things may vary based on RTT time and other factors.

If you cannot see this when the two machines
are seperated by at least a 1ms RTT time.. then there
might possibly be a bug in the implementation of Nagle...

R

-- 
Randall Stewart
NSSTG - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 6 Jul 2006 16:47:57 +0200

Dear all,

as Randy already pointed out the socket style determines if you
bundle user messages with the COOKIE-ECHO.

I tested sending 3 messages in a loop on Mac OS X which results in
INIT
INIT-ACK
COOKIE-ECHO
COOKIE-ACK
DATA
SACK
DATA
SACK
DATA
SACK
SHUTDOWN
SHUTDOWN-ACK
SHUTDOWN-COMPETE

This is over the loopback interface with Nagle enabled. Why?
I guess that you call send(), context switch to the kernel,
send DATA, receive DATA, process DATA, send SACK, receive SACK,
context switch to the sender or receiver, I do not know, but
when it comes to the sender, call send(), context switch,
nothing is outstanding, send DATA, and so on.

Things will be different on a non-loopback interface.

Regarding your
send_enable(0)
send()
send()
send()
send_enable(1)
suggestion:

A student of mine, Martin Becke, has implemented a  
sctp_send_multiple_messages()
cal, which can be used to transfer multiple user message to the  
kernel with
one system call. This will make the above possible and also reduces  
the number
of systems calls. He also has implemented a corresponding recv call.  
The code will
be available on FreeBSD and Mac OS X soon. You can really increase  
the performance
of the stack for small messages by using these new functions.

Best regards
Michael

---

From: CRV§ADER//KY <[email-masked]>
Date: Sat, 08 Jul 2006 13:09:39 +0200

> cal, which can be used to transfer multiple user message to the kernel with
> one system call. This will make the above possible and also reduces the
> number
> of systems calls. He also has implemented a corresponding recv call. The
> code will
> be available on FreeBSD and Mac OS X soon. You can really increase the
> performance

such a function would be extremely useful and I think it should be in
included in the API RFC.
best-effort systems would save bandwidth AND cpu, and it could be the
only acceptable solution for real-time (also HARD real-time!) systems
that would set SCTP_NODELAY=1.

I just designed a hard real-time network based on RTNet and I was forced
to choose UDP (with application-level stream interleaving and message
size limitation -_-) because this feature was not available.

(to be honest, none of the actual Linux API works with the current
implementation of RTNet, but that is beyond the point).

---

From: Randall Stewart <[email-masked]>
Date: Sat, 08 Jul 2006 10:44:37 -0400

Michael:

Lets discuss this at the IETF next week.. maybe we can
get it in the next pass of the API doc :-D

R