Subject: congestion control questions

From: Torbjörn Andersson <[email-masked]>
Date: Tue, 21 Sep 2004 09:48:26 +0200

I have previously done studies on SCTP with ULPs that have a fixed
traffic patterns. Now I'm testing SCTP with dynamic traffic patterns,
where burst of messages are generated by the ULP at a specified
interval. My goal is to study how the idle time between bursts affects
the cwnd. 

According to RFC 2960:

Page 88: (Slow-Start)
"When the endpoint does not transmit data on a given transport
address, the cwnd of the transport address should be adjusted to
max(cwnd/2, 2*MTU) per RTO."

Page 89: (Congestion avoidance)
"Same as in the slow start, when the sender does not transmit DATA
on a given transport address, the cwnd of the transport address
should be adjusted to max(cwnd / 2, 2*MTU) per RTO."

According to implementers guide:
---------
1:   New text: (Section 6.1)
     ---------
     D) When the time comes for the sender to transmit new DATA chunks,
the
        protocol parameter Max.Burst SHOULD be used to limit the number
of
        packets sent. The limit MAY be applied by adjusting cwnd as
follows:

        if((flightsize + Max.Burst*MTU) < cwnd)
           cwnd = flightsize + Max.Burst*MTU

        Or it MAY be applied by strictly limiting the number of packets
        emitted by the output routine.

     E) Then, the sender can send out as many new DATA chunks as Rule A
        and Rule B above allow.

My findings:

I have found that is seems that the "kame" implementation on SCTP does
not follow the RFC specification and cut down the cwnd during my idle
periods, which sometimes are up to 3s in my tests. 
The RTT value of my network is 150ms and the RTO estimate is likely to
be smaller than 3s. The RTO_MIN values is not altered by my ULP so if
the "kame" implementation has its RTO_MIN set to 1 s then the cwnd would
have been cut down at least two times.

The MAX.BURST rule from the impl-guide seems not to be implemented in
"kame". My bursts are 40 messages long, 40 * 500 bytes, and therefore
should my ULP be affected by this rule.

Question:

What are your thoughts about all this? Will the "kame" implementation in
the future implement these rules/algorithm?

--------------------------------------------------------------
| Torbjörn Andersson  phone: +46 54 700 11 61
| Computer Science    fax: +46 54 700 18 28
| Karlstad University URL: http://www.cs.kau.se/~tora
---------------------------------------------------------------

---

From: Randall Stewart <[email-masked]>
Date: Tue, 21 Sep 2004 06:40:17 -0400

> periods, which sometimes are up to 3s in my tests. 
Hmm.. this I know is true.. if one DOES NOT  turn off the
max.burst feature during compile, then it will NOT do
cwnd degrade..

The reason being is that I feel that having a max.burst of
4 packets removes the need for the cwnd degrade since you
won't be able to send more than 4 packets at a time
anyway... which is now what things resort to at idle
time.... so why have the degrade?

> should my ULP be affected by this rule.

Hmm... so you have 20000 bytes queued in burst of say 3 seconds
or about 14 packets of 1500 byte MTUs...

1) Which style of max.burst is compiled?
2) What amount is max.burst set to? -- a 'sysctl -a | grep
                    and look at net.inet.sctp.maxburst
                    will tell.

This could be a bug.. depending on style..

To tell style.. the default is to count packets but the
alternate is gained by defining
SCTP_USE_ALLMAN_BURST
when you compile (or in a .h).  This gets the formula
version.

This most likely is a bug... since the 20000 bytes
should be queued right away (removing the opps for sending
from the app) and then from there its just a matter
of SACK's triggering the 4pkts at a time...

I have not tested it in the way you are doing so (with
small packets).. so it would not surprise me if the
default method had a problem.. the
ALLMAN version would not since it whacks the cwnd..

R

-- 
Randall Stewart
ITD
803-345-0369 <or> 815-342-5222

---

From: Torbjörn Andersson <[email-masked]>
Date: Wed, 22 Sep 2004 09:36:18 +0200

Thank you Randall for the answers. I have some more questions/comments
below, please read them.

> -----Ursprungligt meddelande-----
> Från: Randall Stewart [mailto:[email-masked]]
> Skickat: den 21 september 2004 12:40
> Till: Torbjörn Andersson
> Kopia: [email-masked]
> Ämne: Re: congestion control questions
> 

I have not turned off the max.burst feature when I compiled "Kame".
However I see sadly no traces of the max.burst feature in my results
(tcpdump,ULP times, etc). Maybe is there a bug that is affecting me.

> time.... so why have the degrade?

Is the RFC statement for cwnd degrading optional? The RFC uses the
wording "should" and not "SHOULD". Is "should" optional?

When I study the max.burst algorithm I find it affecting applications
that have idle periods greater than RTO_MIN, => flightsize == 0. I guess
there is a motivation for this algorithm somewhere. Is there an article
describing this somewhere?

> of SACK's triggering the 4pkts at a time...

Sadly I'm not seeing this.

---

From: Randall Stewart <[email-masked]>
Date: Wed, 22 Sep 2004 17:02:41 -0400

Torbjörn Andersson wrote:
> Thank you Randall for the answers. I have some more
> questions/comments below, please read them.
> 
> 
>> -----Ursprungligt meddelande----- Från: Randall Stewart
>> [mailto:[email-masked]] Skickat: den 21 september 2004 12:40 Till:
>> Torbjörn Andersson Kopia: [email-masked] Ämne: Re:
>> congestion control questions
>> 
>>> I have previously done studies on SCTP with ULPs that have a
>>> fixed traffic patterns. Now I'm testing SCTP with dynamic traffic
>>> 
> 
> patterns,
> 
>>> interval. My goal is to study how the idle time between bursts
> 
> affects
> 
>>> the cwnd.
>>> 
>>> According to RFC 2960:
>>> 
>>> Page 88: (Slow-Start) "When the endpoint does not transmit data
>>> should be adjusted to max(cwnd/2, 2*MTU) per RTO."
>>> 
>>> Page 89: (Congestion avoidance) "Same as in the slow start, when
>>> the sender does not transmit DATA on a given transport address,
>>> the cwnd of the transport address should be adjusted to max(cwnd
>>> / 2, 2*MTU) per RTO."
>>> 
>>> 
>>> According to implementers guide: --------- 1:   New text:
>>> (Section 6.1) --------- D) When the time comes for the sender to
>>> transmit new DATA
> 
> chunks,
> 
>>> the protocol parameter Max.Burst SHOULD be used to limit the
> 
> number
> 
>>> of packets sent. The limit MAY be applied by adjusting cwnd as 
>>> follows:
>>> 
>>> 
>>> if((flightsize + Max.Burst*MTU) < cwnd) cwnd = flightsize +
>>> Max.Burst*MTU
>>> 
>>> 
>>> Or it MAY be applied by strictly limiting the number of
> 
> packets
> 
>>> E) Then, the sender can send out as many new DATA chunks as
> 
> Rule A
> 
>>> and Rule B above allow.
>>> 
>>> 
>>> My findings:
>>> 
>>> I have found that is seems that the "kame" implementation on SCTP
>>> 
> 
> does
> 
>>> not follow the RFC specification and cut down the cwnd during my
> 
> idle
> 
>> Hmm.. this I know is true.. if one DOES NOT  turn off the max.burst
>> feature during compile, then it will NOT do cwnd degrade..
> 
> 

Try defining the ALLMAN method by
putting in sctp_constants.h

#define SCTP_USE_ALLMAN_BURST 1

And then recompiling all..

This should I think fix your problem ... at least it slamms
the cwnd... but of course thats in bytes not packets
so you could see more than 4 packets occasionally (with small
messages)....

Is ONDELAY on or off?

> 
> 
>> The reason being is that I feel that having a max.burst of 4
>> packets removes the need for the cwnd degrade since you won't be
>> able to send more than 4 packets at a time anyway... which is now
>> what things resort to at idle time.... so why have the degrade?
> 
> 

Hmm.. I will have to go look at it.. but in any event SHOULD or should
(it probably should be in caps)... SHOULD is NOT a MUST.. if you
must have a really GOOD reason NOT to degrade... and I do
believe I do (if max burst is working right.. which I think
the allman version will... thinking on small packets and the
current code I can see a bug)...

Try the Allman version and let me know how it works...
>  that have idle periods greater than RTO_MIN, => flightsize == 0. I
> guess there is a motivation for this algorithm somewhere. Is there an
> article describing this somewhere?

Not sure.. Mark Allman might know.. the idea is you don't want to
make massive bursts into the network.. otherwise you end up
overrunning router queues..

I have tested this in a local network and your performance can
go in the dump if you burst out to much...

R
> 
> 
>>> The RTT value of my network is 150ms and the RTO estimate is
>>> likely
> 
> to
> 
>>> be smaller than 3s. The RTO_MIN values is not altered by my ULP
>>> so
> 
> if
> 
>>> the "kame" implementation has its RTO_MIN set to 1 s then the
>>> cwnd
> 
> would
> 
>>> The MAX.BURST rule from the impl-guide seems not to be
>>> implemented
> 
> in
> 
>>> "kame". My bursts are 40 messages long, 40 * 500 bytes, and
> 
> therefore
> 
>> Hmm... so you have 20000 bytes queued in burst of say 3 seconds or
>> about 14 packets of 1500 byte MTUs...
>> 
>> 1) Which style of max.burst is compiled? 2) What amount is
>> max.burst set to? -- a 'sysctl -a | grep and look at
>> net.inet.sctp.maxburst will tell.
>> 
>> To tell style.. the default is to count packets but the alternate
>> is gained by defining
> 
> 
> 
>> SCTP_USE_ALLMAN_BURST when you compile (or in a .h).  This gets the
>> formula version.
>> 
>> 
>> This most likely is a bug... since the 20000 bytes should be queued
>> right away (removing the opps for sending from the app) and then
>> from there its just a matter of SACK's triggering the 4pkts at a
>> time...
> 
> 
>> I have not tested it in the way you are doing so (with small
>> packets).. so it would not surprise me if the default method had a
>> problem.. the ALLMAN version would not since it whacks the cwnd..
>> 
>> R
>> 
>>> Question:
>>> 
>>> What are your thoughts about all this? Will the "kame"
> 
> implementation in
> 
>>> -------------------------------------------------------------- |
>>> Torbjörn Andersson  phone: +46 54 700 11 61 | Computer Science
>>> fax: +46 54 700 18 28 | Karlstad University URL:
>>> http://www.cs.kau.se/~tora 
>> -- Randall Stewart ITD 803-345-0369 <or> 815-342-5222
> 
> 
>