Subject: Considering Satellite and other Long RTT time communications

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 05 Mar 2003 15:59:26 -0600

Hi all:

As some of you may be aware I have been doing some interesting
work at cisco with both TCP and SCTP in a satellite networks.
In these networks you have a HIGH RTT time, and possibly HIGH bit
error rates..

There are a couple of things implementors can do that I have
found make a vast performance improvment and it is something
newer SCTP stacks should considering when selecting default
parameters. Two key parameters that you might want to contemplate
is:

1) Intial Rwnd
and
2) Intial cwnd.
and
3) max.burst

For 1) the higher you can make this the better... since the rwnd will
directly limit how much you can have out per RTT... and thus in
a high RTT situation it directly limits your throughput even on
a zero error channel... Many have done studies with this
in TCP.. same is true for SCTP (see papers by Mark Allman and
others http://roland.grc.nasa.gov/~mallman/  has a start)

I realize that some implementations may not have the memory flexibility
to bump this initial value very high..but for general purpose type
processors, where you have the flexibility, I would strongly recommend
setting this by default to at least 128k... if not higher.. (my bsd 
implementation
will now default to 128k).

For 2) this may be a bit controversial  but.. currently, in many TCP 
implementations,
if the implementation detects that the destination is on the local lan 
segment it will
set the initial cwnd to 4 segments instead of 2. I would argue that if 
you detect in
the INIT<->INIT-ACK RTT to be over 400ms setting the initial cwnd to 4 might
be a good idea as well. Especially when you consider you will not get 
feedback back
from your initial send for close to (or more than) 1/2 second...

Sending 4 segments can also enable future algorithms to be applied using 
packet-pair
type technology as well (Mark as a few papers on this on his site...).

For 3) Currently max.burst  is recommended to be set at 4, for networks 
with a
high RTT.. I think it may be better to have a max.burst in the range of 
6-8...mine
goes to 8 when the sat.network flag gets turned on... (I keep this flag 
in the TCB
and validate that it stays on at every RTT measurment.. if it swtiches off
I penalize the peer that was spoofing me :>)

Of course for 2 & 3 you must use the initial-rtt to figure out what type of
rules to apply... and you may even want to periodically validate that the
peer was not pulling the wool over your eyes  as I do :>

At minimum number 1 above is a very safe and easy thing to do to
assure SCTP will run a bit more optimally over sat networks..

I will also be floating a draft fairly soon on how to address
the other property I mentioned :>

R

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 05 Mar 2003 16:48:20 -0600

Nivedita Singhvi wrote:

> Randall Stewart (cisco) wrote:
>
>> Hi all:
>>
> Very cool! I was trying to solve performance issues
> at a customer site who was doing TCP over satellite
> and getting poor performance..
>
> I didnt find too much helpful documentation out there,
> frankly, and very few real world studies with data..

Most of the work I saw out there was studies and simulations..
Mark as done a lot...

>
> If you could point me to anything that would help,
> I'd appreciate it very much..
>
> I found it critical to have large socket buffers,
> of course..but also, had problems maintaining a
> large enough window to fill the pipe..congestion
> window never got above 10 segments..
>
> I played around with hacking (evil) up cwnd for TCP,
> and yes, I did find upto 10% improvement in just
> allowing cwnd to be in the range of 4-6 to begin with..

Yep.. and I don't think this is un-reasonable (bumping to 4).. since
its usually a long time until you get feedback and a
normal network citizen in 500ms is going to send a lot
more than 4 packets...

Another thing that will hold you down is the errors.
The Error rate is very very bursty in a satellite. And
when it bursts you start to either lose packets or
you may get a delay...

>
> However, I found that rtt was very bursty - ave
> rtt was 800ms, but every 10th or so packet (including
> pings, so this isnt TCP)..would go upto 1600ms, 2500ms,
> even 3400ms..seemed to be multiple of rtt times..and
> this was a constant feature of the satellite connection..
> I wasnt able to acquire any info regarding why this was
> occurring..but it can have nasty effects on TCP - because
> you inevitably drop packets, shrink window..and it repeats
> up and down..killing performance..
>
> Is there anything you can suggest regarding this problem,
> short of altering the rtt adjustment code? (I didnt try
> anything that severe, although they were in favor of it)..

Cisco will be putting out a product (soon) that has our solution in
it for TCP.. and even some additional things for SCTP. I will
be putting out a draft for SCTP soon as well.. and if you
do what the draft says with the technology  it makes a
TREMENDOUS difference even with error rates of
6% :>

> I'd be very interested in learning what you suggest for
> SCTP - will you be generating anything equivalent for TCP?

Soon I wil be posting a draft as I said.. for TCP there are very
limtied things that can be done.. and the cisco product will address
a lot of these.. the other thing is windows.. get window scaling on and
raise your rwnd up in TCP too.. this will help (as you noted).. And of
course your send window had better be big for your app to.. otherwise
it will never be able to keep the pipe full...

R

> thanks,
> Nivedita

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 05 Mar 2003 19:10:14 -0600

Qiaobing Xie wrote:

>"Randall Stewart (cisco)" wrote:
>...
>  
>
>Maybe this can benefit an over-the-air link in mobile phone networks
>too. It has the same properties, plus limited bandwidth. 
>
>-Qiaobing
>
>  
>
Yep,

The technology we are working on should work for mobile systems
as well as satellite :>

R

---

From: Nivedita Singhvi <[email-masked]>
Date: Wed, 05 Mar 2003 14:30:36 -0800

thanks,
Nivedita