Subject: Question regarding the SCTP-kame implementation

From: Torbjörn Andersson <[email-masked]>
Date: Wed, 10 Dec 2003 10:01:06 +0100

I'm testing the SCTP-kame implementation and I have some specific
questions about the SCTP send buffer in Kame. I want to know how the
send buffer behaves. The reason to why I want to know this is that I'm
studying the total transmission time for individual messages. I guess
someone reading this mailing list know this.

How big is it? 
Is it tuned to the cwnd?
Is the size of the send buffer dependent of the congestion state,
slowstart/fast recovery/congestion avoidance etc?
Is it chunk or byte limited?

Best Regards, Torbjörn

--------------------------------------------------------------
| Torbjörn Andersson  phone: +46 54 700 11 61
| Computer Science    fax: +46 54 700 18 28
| Karlstad University URL: http://www.cs.kau.se/~tora
---------------------------------------------------------------

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 10 Dec 2003 06:35:08 -0600

Torbjorn:

Yes, there are a couple of us that know... in fact I am
behind on getting a couple of patches in too :-<

But as to your questions...

>someone reading this mailing list know this.
The current send buffer and receive window are
defaulted to the maximum socket buffer minus a bit... the
actual code is in sctp_init() in sctp_usrreq.c and the lines
look like:

    sb_max_adj = (u_long)((u_quad_t)(SB_MAX) * MCLBYTES / (MSIZE + 
MCLBYTES));
    sctp_sendspace = min((min(SB_MAX, sb_max_adj)),
                 ((nmbclusters/2) * SCTP_DEFAULT_MAXSEGMENT));
    /* Now for the recv window, should we take the same
     * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX min
     * above. For now I will just copy.
     */
    sctp_recvspace = sctp_sendspace;

SB_MAX is set in sys/socketvar.h

In order to get real high performance (GigE type speed) you will need
to set this up higher from the shipped default (256*1024). My machine
that I do high bandwidth testing on has its set to (2048 * 1024)

You also can change this by doing sysctl since both the sendspace and 
recvspace
are sysctl variables (at least in freebsd).

>Is it tuned to the cwnd?
cwnd is tied to the network and may get limited by your send
buffer and peers rwnd but it will never influence the value
of your send space.

>slowstart/fast recovery/congestion avoidance etc?
Nope... these are  unrelated items .. cwnd deals with
the associations view of the network, sendspace deals
with how much outstanding data a app can write down
the socket buffer...

You can also change the value down or up by doing a
setsockopt() on the SND_BUF...

>Is it chunk or byte limited?
There is also an additional limit placed on an implementation
as well.. number of chunks... This is tuned so that if you
do a WHOLE bunch of 1 byte sends for example:

while(1)
   send(1 byte);

You will block when you hit a threshold of chunks.. otherwise
you would quickly exhaust the number of mbufs in the system
which will have dire consequences...

This value is setup to either be the max of
(SCTP_ASOC_MAX_CHUNKS_ON_QUEUE, nmbclusters)

nmbclusters is tuneable in the config file.. .while the chunks on queue
is a hardcoded #define  currently set to 512

Hope that all helps..

R

---

From: Torbjörn Andersson <[email-masked]>
Date: Wed, 10 Dec 2003 15:21:04 +0100

Thanks for your quick answer. Your answer has moved my focus from the
send buffer to the blocking behavior of the socket.

When does SCTP-block?
	Is it dependent of the state of the association?

> -----Ursprungligt meddelande-----
> Från: Randall Stewart (cisco) [mailto:[email-masked]]
> Skickat: den 10 december 2003 13:35
> Till: Torbjörn Andersson
> Kopia: [email-masked]
> Ämne: Re: Question regarding the SCTP-kame implementation
> 
> Torbjorn:
>

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 10 Dec 2003 09:19:03 -0600

It depends on a couple of things you need to answer for me...

1) Do you have blocking IO on? FNBIO I think is the socket opt
2) What model are you using 1-1 or 1-M?

Generally the 1-1 model will block at connect()/accept() and
when the send buffer is full... if blocking IO is on (I think the
default)... If you do non-blocking IO then thats a different
story.. oh and one of the bugs that is still in patch prepartion as
to do with blocking io.. there is a problem in the 1-1 model
when you do non-blocking io.. connect will fail.. I fixed this
a while ago but just kept forgetting to get the right
cvs diffs to kame.. its in the in_proto.c and in6_proto.c files
(the 1-1 model needs PR_CONNREQUIRED)

R

Torbjörn Andersson wrote:

>>>send buffer behaves. The reason to why I want to know this is that
>>>      
>>>
>I'm
>  
>
>>nmbclusters is tuneable in the config file.. .while the chunks on
>>    
>>
>queue
>  
>

---

From: Torbjörn Andersson <[email-masked]>
Date: Wed, 10 Dec 2003 17:38:16 +0100

I append a file to this message, which contain the results of an
experiment I have performed. 
The experiment measured individual ULP message times, which include the
time of getting through the send buffer, router buffer (12 segements)
and network (25ms). Further retransmissions and reordering at the
receiver can delay the messages. The experiment does not have any
background traffic on the test network. 

Each message contains 500 bytes of data and they are sent as fast as
possible. A 1-1 SCTP socket is used at the sender and the socket is
configured as blocking (default).

Now, one can see in the figure that the message times increase
differently in four different phases. I was actually expecting one maybe
two different phases until the buffer of the SCTP association becomes
dominant and everything levels out. 
But it does not look like that and I believed first that there was some
intelligent buffer management at the sender. But the send buffer is
statically configured. 
The next thing I though could explain the different message increase
speeds was the blocking algorithm.

Does anyone have any input to this?

> Skickat: den 10 december 2003 16:19
> Till: Torbjörn Andersson
> Ämne: Re: SV: Question regarding the SCTP-kame implementation
>

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Thu, 11 Dec 2003 06:54:17 -0600

>Does anyone have any input to this?
I am not sure what would cause a 4 stage change.. I would
expect two stages like you...hmmm could loss have entered in
here.. that would cause a jump down from slow start to ca...

Might I also make one suggestion...

If you enable

SCTP_STAT_LOGGING

you can also enable:
SCTP_CWND_LOGGING
SCTP_BLK_LOGGING
SCTP_FR_LOGGING
SCTP_MAP_LOGGING

(note you must enable stat logging to get any of
  the other ones to compile )...

So.. what does this gain you?

CWND_LOGGING will log the congestion window events
BLK_LOGGING will log when you block due to buffer full conditions
FR_LOGGING will log retransmissions (I think also timeouts)
MAP_LOGGING is probably least interesting .. it logs
                           mapping array movement .. the receiver side
                           bit map manipulations.

Now You can enable one or any combination of the logging..

either manually add it to

opt_sctp.h

or I think the changes have gone in with the last kame patch that
let you add it directly to the config file... i.e.
options SCTP_STAT_LOGGING
options SCTP_CWND_LOGGING

will turn on the cwnd logging... if you want to do it in the opt_sctp.h
file

#define SCTP_STAT_LOGGING 1
#define SCTP_CWND_LOGGING 1

.

The log defaults to 80,000 events. You can expand this larger
if that is not enough for the transfer you are doing by
changing SCTP_STAT_LOG_SIZE in sctp_constants.h

If you are interested in doing this let me know ... I have a
log pulling/printing set of utilities I can send you that
can then pull and display the log information. This logging
is quite low overhead being a array of ints that gets manipulated
to generate the log... I have used it for debugging quite effectively
since doing a
SCTP_DEBUG
generates a lot more information and slows things down...

Let me know if you want the utilities...

R

>>>>>questions about the SCTP send buffer in Kame. I want to know how
>>>>>          
>>>>>
>the
>  
>
>>>>>studying the total transmission time for individual messages. I
>>>>>          
>>>>>
>guess
>  
>
>>>>    * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX
>>>>        
>>>>
>min
>  
>
>>>>In order to get real high performance (GigE type speed) you will
>>>>        
>>>>
>need
>  
>
>>>>to set this up higher from the shipped default (256*1024). My
>>>>        
>>>>
>machine
>  
>
>>>>You also can change this by doing sysctl since both the sendspace
>>>>        
>>>>
>and
>  
>
>>>>recvspace