Subject: strange results with high latency, no loss

From: Brad Penoff <[email-masked]>
Date: Tue, 1 Nov 2005 22:07:21 -0800 (PST)

Greetings,

We have developed a version of the Message Passing Interface (MPI) to run 
atop SCTP (google "SCTP MPI").  We have an 8 node cluster that runs 
FreeBSD 5.3 with an SCTP implementation obtained directly from Randall and 
colleagues in mid May (thanks!).  As the title reads, we're seeing some 
strange results when running on emulated networks with high latency and no 
loss that are explained below.

We enabled Dummynet and use it quite frequently.  Most of our 
experiments were working with the loss settings in Dummynet.  There, as 
expected, we found SCTP to outperform TCP.  Loss makes the RTTs vary but 
SCTP's improved congestion control and multistreaming features help 
flatten these.

Recently, we've been working on tests with increased latency meaning 
we set the delay in Dummynet, with no loss.  For some applications, we're 
seeing very strange results; as latency increases SCTP starts to lose to 
TCP (under only loss, SCTP edges out TCP).  Regardless of the input 
parameters to the application, we see this same general trend.  For 
example, for one setting we had these wallclock times in seconds:

Results A:
         0%0ms  0%40ms   0%80ms
SCTP	0.05	15.72	31.75
TCP	0.05	8.38	16.85

So you can see the hit that SCTP is taking.  We also obtained an 
application from Randall that lets us clear and print out the kernel 
"pegs" which are essentially various counters within the kernel.  For this 
data, they are in table A below.  We don't know what most of the output is 
so we were wondering if anyone here could give me a hand explaining what 
is causing this performance drop.  Either it's SCTP's error or Dummynet...

When we focused on only 0%80ms, we saw some additional strangeness 
depending on the actual input parameters.  To explain, first we'll 
describe our application as if it were using the SCTP API directly (rather 
than MPI implemented using SCTP; we have seen the same results in non-MPI 
apps we ran).

The application on client A does this:
for(int i=0; i<loop; i++) {
  sctp_sendmsg(env_size bytes)
  sctp_recvmsg(env_size bytes)
  sctp_sendmsg(msg_size bytes)
}

..and client B reciprocates.

The results in A were ran with loop=1000, env_size=48 and msg_size=12024.

Is all that were seeing poor small message performance?  As we'll show 
later, it doesn't seem to be that simple...  Also, why does performance 
get affected so drastically with increased latencies?

When focusing on ONLY 0% loss and 80ms, we varied the parameters and saw 
some other strangeness.

Results B:
                    parameters
loop           100      100       100
env_size        48    12024     16024
msg_size     12024    12024     16024
-------------------------------------- 
SCTP          31.75    32.58     63.74
TCP           16.85    20.32     22.32

So it doesn't seem to be a small message problem here because INCREASING 
the size of both env_size and msg_size from 12024 to 16024 causes TCP to 
increase 2 seconds but SCTP 31 seconds.  Again, the "pegs" are below but 
in table B.

Generally, we're confused; can anyone offer any advice on why we're seeing 
this?

Thanks ahead of time,
brad

Table A
---------
0%	0ms	SCTP
loop	100	msg_size 12024	env_size	48

sack_rcv	0x000001f5	sack_snt	0x00000066	tsns_snt 
0x000003ea	tsns_rcv	0x00000066
pkt_sent	0x0000044e	pkt_rcvd	0x0000025b	tsns_ret 
0x00000000	dup_tsns	0x00000000
hbs__rcv	0x00000000	hbackrcv	0x00000000	htb__snt 
0x00000000	win_prbe	0x00000000
pktswdat	0x00000066	t3-timeo	0x00000000	dsack-to 
0x00000000	hb_timer	0x00000000
fst_rxts	0x00000000	timerexp	0x00000000	fr_inwin 
0x00000000	blk_rwnd	0x00000000
blk_cwnd	0x00000004	rwnd_drp	0x00000000	bad_strm 
0x00000000	bad_ssnw	0x00000000
drpnomem	0x00000000	drpfragm	0x00000000	badvtags 
0x00000000	badcsumv	0x00000000
packetin	0x0000025b	mcastrcv	0x00000000	hdrdrops 
0x00000000	no_portn	0x00000000
cwnd_nf	0x00000004	co_snds	0x000002c0	co_nodat	0x000001f3 
cw_nu_ss	0x00000191
max_brst	0x000000c2	expr_del	0x00000063	no_cp_in 
0x00000000	cach_src	0x00000453
cw_nocum	0x00000000	cw_incss	0x00000065	cw_incca 
0x00000000	cw_skip	0x00000000
cw_nu_ca	0x00000000	cw_maxcw	0x000034bc	diff_ss 
0x00000000	diff_ca	0x00002ed0
tqs@ss	0x00000000	sqs@ss	0x00000000	tqs@ca	0x00000000 
sqq@ca	0x00000000
lmtu_mov	0x00002ee0	lcnt_mov	0x00000001	sndqctss 
0x00000000	sndqctca	0x00000000
movemax	0x000000ca	move_equ	0x00000064	nagle_qo 
0x00000000	nagle_of	0x00000067
out_fr_s	0x00000067	sostrnos	0x00000000	nostrnos 
0x000001f3	sosnqnos	0x00000000
nosnqnos	0x000001f3	intoperr	0x00000000	dupssnrc 
0x00000000	multi-fr	0x00000000
vtag-exp	0x00000259	vtag-bog	0x00000000	t3-safeg 
0x00000000	pd--mbox	0x00000000
pd-ehost	0x00000000	pdmb_wda	0x00000000	pdmb_ctl 
0x00000000	pdmb_bwr	0x00000000
pd_corup	0x00000000	pd_nedat	0x00000000	pd_errpd 
0x00000000	fst_prep	0x00000000
pd_daNFo	0x00000000	pd_dIWin	0x00000000	pd_dIZrw 
0x00000000	pd_BadDa	0x00000000
pd_dMark	0x00000000	ecne_rcv	0x00000000	cwr_perf 
0x00000000	ecne_snt	0x00000000
chun_drp	0x00000000	nollsndq	0x00000000	ll_c_err 
0x00000000	nollcwnd	0x00000063
nollbrst	0x00000000	no_ifqsp	0x00000000	rcvnotcb 
0x00000000	rcvhadtc	0x00000066
pd_InRcv	0x00000000	pdhad2wa	0x00000000	pdhad2rc 
0x00000000	pdnostok	0x00000000
entersor	0x00000066	resv1	0x00000000	resv2	0x00000000 
resv3	0x00000000

0%	40ms
loop	100	msg_size 12024	env_size	24

sack_rcv	0x000001f5	sack_snt	0x00000066	tsns_snt 
0x000003ea	tsns_rcv	0x00000066
pkt_sent	0x0000044e	pkt_rcvd	0x0000025b	tsns_ret 
0x00000000	dup_tsns	0x00000000
hbs__rcv	0x00000000	hbackrcv	0x00000000	htb__snt 
0x00000000	win_prbe	0x00000000
pktswdat	0x00000066	t3-timeo	0x00000000	dsack-to 
0x00000000	hb_timer	0x00000000
fst_rxts	0x00000000	timerexp	0x00000001	fr_inwin 
0x00000000	blk_rwnd	0x00000000
blk_cwnd	0x00000004	rwnd_drp	0x00000000	bad_strm 
0x00000000	bad_ssnw	0x00000000
drpnomem	0x00000000	drpfragm	0x00000000	badvtags 
0x00000000	badcsumv	0x00000000
packetin	0x0000025b	mcastrcv	0x00000000	hdrdrops 
0x00000000	no_portn	0x00000000
cwnd_nf	0x00000005	co_snds	0x000002c1	co_nodat	0x000001f4 
cw_nu_ss	0x00000191
max_brst	0x000000c2	expr_del	0x00000063	no_cp_in 
0x00000000	cach_src	0x00000453
cw_nocum	0x00000000	cw_incss	0x00000065	cw_incca 
0x00000000	cw_skip	0x00000000
cw_nu_ca	0x00000000	cw_maxcw	0x000034bc	diff_ss 
0x00000000	diff_ca	0x00002ed0
tqs@ss	0x00000000	sqs@ss	0x00000000	tqs@ca	0x00000000 
sqq@ca	0x00000000
lmtu_mov	0x00002ee0	lcnt_mov	0x00000001	sndqctss 
0x00000000	sndqctca	0x00000000
movemax	0x000000ca	move_equ	0x00000064	nagle_qo 
0x00000000	nagle_of	0x00000067
out_fr_s	0x00000067	sostrnos	0x00000001	nostrnos 
0x000001f3	sosnqnos	0x00000000
nosnqnos	0x000001f4	intoperr	0x00000000	dupssnrc 
0x00000000	multi-fr	0x00000000
vtag-exp	0x00000259	vtag-bog	0x00000000	t3-safeg 
0x00000000	pd--mbox	0x00000000
pd-ehost	0x00000000	pdmb_wda	0x00000000	pdmb_ctl 
0x00000000	pdmb_bwr	0x00000000
pd_corup	0x00000000	pd_nedat	0x00000000	pd_errpd 
0x00000000	fst_prep	0x00000000
pd_daNFo	0x00000000	pd_dIWin	0x00000000	pd_dIZrw 
0x00000000	pd_BadDa	0x00000000
pd_dMark	0x00000000	ecne_rcv	0x00000000	cwr_perf 
0x00000000	ecne_snt	0x00000000
chun_drp	0x00000000	nollsndq	0x00000000	ll_c_err 
0x00000000	nollcwnd	0x00000063
nollbrst	0x00000000	no_ifqsp	0x00000000	rcvnotcb 
0x00000000	rcvhadtc	0x00000066
pd_InRcv	0x00000000	pdhad2wa	0x00000000	pdhad2rc 
0x00000000	pdnostok	0x00000000
entersor	0x00000066	resv1	0x00000000	resv2	0x00000000 
resv3	0x00000000

0%	80ms
loop	100	msg_size 12024	env_size	48

sack_rcv	0x000001f5	sack_snt	0x00000066	tsns_snt 
0x000003ea	tsns_rcv	0x00000066
pkt_sent	0x0000044e	pkt_rcvd	0x0000025b	tsns_ret 
0x00000000	dup_tsns	0x00000000
hbs__rcv	0x00000000	hbackrcv	0x00000000	htb__snt 
0x00000000	win_prbe	0x00000000
pktswdat	0x00000066	t3-timeo	0x00000000	dsack-to 
0x00000000	hb_timer	0x00000000
fst_rxts	0x00000000	timerexp	0x00000001	fr_inwin 
0x00000000	blk_rwnd	0x00000000
blk_cwnd	0x00000005	rwnd_drp	0x00000000	bad_strm 
0x00000000	bad_ssnw	0x00000000
drpnomem	0x00000000	drpfragm	0x00000000	badvtags 
0x00000000	badcsumv	0x00000000
packetin	0x0000025b	mcastrcv	0x00000000	hdrdrops 
0x00000000	no_portn	0x00000000
cwnd_nf	0x00000005	co_snds	0x000002c1	co_nodat	0x000001f4 
cw_nu_ss	0x00000191
max_brst	0x000000c2	expr_del	0x00000063	no_cp_in 
0x00000000	cach_src	0x00000453
cw_nocum	0x00000000	cw_incss	0x00000065	cw_incca 
0x00000000	cw_skip	0x00000000
cw_nu_ca	0x00000000	cw_maxcw	0x000034bc	diff_ss 
0x00000000	diff_ca	0x00002ed0
tqs@ss	0x00000000	sqs@ss	0x00000000	tqs@ca	0x00000000 
sqq@ca	0x00000000
lmtu_mov	0x00002ee0	lcnt_mov	0x00000001	sndqctss 
0x00000000	sndqctca	0x00000000
movemax	0x000000ca	move_equ	0x00000064	nagle_qo 
0x00000000	nagle_of	0x00000067
out_fr_s	0x00000067	sostrnos	0x00000001	nostrnos 
0x000001f3	sosnqnos	0x00000001
nosnqnos	0x000001f3	intoperr	0x00000000	dupssnrc 
0x00000000	multi-fr	0x00000000
vtag-exp	0x00000259	vtag-bog	0x00000000	t3-safeg 
0x00000000	pd--mbox	0x00000000
pd-ehost	0x00000000	pdmb_wda	0x00000000	pdmb_ctl 
0x00000000	pdmb_bwr	0x00000000
pd_corup	0x00000000	pd_nedat	0x00000000	pd_errpd 
0x00000000	fst_prep	0x00000000
pd_daNFo	0x00000000	pd_dIWin	0x00000000	pd_dIZrw 
0x00000000	pd_BadDa	0x00000000
pd_dMark	0x00000000	ecne_rcv	0x00000000	cwr_perf 
0x00000000	ecne_snt	0x00000000
chun_drp	0x00000000	nollsndq	0x00000000	ll_c_err 
0x00000000	nollcwnd	0x00000063
nollbrst	0x00000000	no_ifqsp	0x00000000	rcvnotcb 
0x00000000	rcvhadtc	0x00000066
pd_InRcv	0x00000000	pdhad2wa	0x00000000	pdhad2rc 
0x00000000	pdnostok	0x00000000
entersor	0x00000066	resv1	0x00000000	resv2	0x00000000 
resv3	0x00000000

Table B
---------
0%	80ms
loop	100	msg_size 12024	env_size	48

sack_rcv	0x000001f5	sack_snt	0x00000066	tsns_snt 
0x000003ea	tsns_rcv	0x00000066
pkt_sent	0x0000044e	pkt_rcvd	0x0000025b	tsns_ret 
0x00000000	dup_tsns	0x00000000
hbs__rcv	0x00000000	hbackrcv	0x00000000	htb__snt 
0x00000000	win_prbe	0x00000000
pktswdat	0x00000066	t3-timeo	0x00000000	dsack-to 
0x00000000	hb_timer	0x00000000
fst_rxts	0x00000000	timerexp	0x00000001	fr_inwin 
0x00000000	blk_rwnd	0x00000000
blk_cwnd	0x00000005	rwnd_drp	0x00000000	bad_strm 
0x00000000	bad_ssnw	0x00000000
drpnomem	0x00000000	drpfragm	0x00000000	badvtags 
0x00000000	badcsumv	0x00000000
packetin	0x0000025b	mcastrcv	0x00000000	hdrdrops 
0x00000000	no_portn	0x00000000
cwnd_nf	0x00000005	co_snds	0x000002c1	co_nodat	0x000001f4 
cw_nu_ss	0x00000191
max_brst	0x000000c2	expr_del	0x00000063	no_cp_in 
0x00000000	cach_src	0x00000453
cw_nocum	0x00000000	cw_incss	0x00000065	cw_incca 
0x00000000	cw_skip	0x00000000
cw_nu_ca	0x00000000	cw_maxcw	0x000034bc	diff_ss 
0x00000000	diff_ca	0x00002ed0
tqs@ss	0x00000000	sqs@ss	0x00000000	tqs@ca	0x00000000 
sqq@ca	0x00000000
lmtu_mov	0x00002ee0	lcnt_mov	0x00000001	sndqctss 
0x00000000	sndqctca	0x00000000
movemax	0x000000ca	move_equ	0x00000064	nagle_qo 
0x00000000	nagle_of	0x00000067
out_fr_s	0x00000067	sostrnos	0x00000001	nostrnos 
0x000001f3	sosnqnos	0x00000001
nosnqnos	0x000001f3	intoperr	0x00000000	dupssnrc 
0x00000000	multi-fr	0x00000000
vtag-exp	0x00000259	vtag-bog	0x00000000	t3-safeg 
0x00000000	pd--mbox	0x00000000
pd-ehost	0x00000000	pdmb_wda	0x00000000	pdmb_ctl 
0x00000000	pdmb_bwr	0x00000000
pd_corup	0x00000000	pd_nedat	0x00000000	pd_errpd 
0x00000000	fst_prep	0x00000000
pd_daNFo	0x00000000	pd_dIWin	0x00000000	pd_dIZrw 
0x00000000	pd_BadDa	0x00000000
pd_dMark	0x00000000	ecne_rcv	0x00000000	cwr_perf 
0x00000000	ecne_snt	0x00000000
chun_drp	0x00000000	nollsndq	0x00000000	ll_c_err 
0x00000000	nollcwnd	0x00000063
nollbrst	0x00000000	no_ifqsp	0x00000000	rcvnotcb 
0x00000000	rcvhadtc	0x00000066
pd_InRcv	0x00000000	pdhad2wa	0x00000000	pdhad2rc 
0x00000000	pdnostok	0x00000000
entersor	0x00000066	resv1	0x00000000	resv2	0x00000000 
resv3	0x00000000

0%	80ms
loop	100	msg_size 12024	env_size	12024

sack_rcv	0x00000385	sack_snt	0x000001f6	tsns_snt 
0x0000070a	tsns_rcv	0x00000386
pkt_sent	0x00000901	pkt_rcvd	0x0000070e	tsns_ret 
0x00000000	dup_tsns	0x00000000
hbs__rcv	0x00000000	hbackrcv	0x00000000	htb__snt 
0x00000000	win_prbe	0x00000000
pktswdat	0x00000386	t3-timeo	0x00000000	dsack-to 
0x00000000	hb_timer	0x00000000
fst_rxts	0x00000000	timerexp	0x00000001	fr_inwin 
0x00000000	blk_rwnd	0x00000000
blk_cwnd	0x000000ca	rwnd_drp	0x00000000	bad_strm 
0x00000000	bad_ssnw	0x00000000
drpnomem	0x00000000	drpfragm	0x00000000	badvtags 
0x00000000	badcsumv	0x00000000
packetin	0x0000070e	mcastrcv	0x00000000	hdrdrops 
0x00000000	no_portn	0x00000000
cwnd_nf	0x000000ca	co_snds	0x00000774	co_nodat	0x0000044f 
cw_nu_ss	0x00000259
max_brst	0x000000c8	expr_del	0x00000000	no_cp_in 
0x00000000	cach_src	0x00000903
cw_nocum	0x00000000	cw_incss	0x0000012d	cw_incca 
0x00000000	cw_skip	0x00000000
cw_nu_ca	0x00000000	cw_maxcw	0x00004074	diff_ss 
0x00000000	diff_ca	0x00002ed0
tqs@ss	0x00000000	sqs@ss	0x00000000	tqs@ca	0x00000000 
sqq@ca	0x00000000
lmtu_mov	0x00002ee0	lcnt_mov	0x00000001	sndqctss 
0x00000000	sndqctca	0x00000000
movemax	0x000000ca	move_equ	0x000000c8	nagle_qo 
0x00000000	nagle_of	0x00000067
out_fr_s	0x00000067	sostrnos	0x00000000	nostrnos 
0x0000044f	sosnqnos	0x00000000
nosnqnos	0x0000044f	intoperr	0x00000000	dupssnrc 
0x00000000	multi-fr	0x00000000
vtag-exp	0x0000070c	vtag-bog	0x00000000	t3-safeg 
0x00000000	pd--mbox	0x00000000
pd-ehost	0x00000000	pdmb_wda	0x00000000	pdmb_ctl 
0x00000000	pdmb_bwr	0x00000000
pd_corup	0x00000000	pd_nedat	0x00000000	pd_errpd 
0x00000000	fst_prep	0x00000000
pd_daNFo	0x00000000	pd_dIWin	0x00000000	pd_dIZrw 
0x00000000	pd_BadDa	0x00000000
pd_dMark	0x00000000	ecne_rcv	0x00000000	cwr_perf 
0x00000000	ecne_snt	0x00000000
chun_drp	0x00000000	nollsndq	0x00000000	ll_c_err 
0x00000000	nollcwnd	0x00000063
nollbrst	0x00000000	no_ifqsp	0x00000000	rcvnotcb 
0x00000000	rcvhadtc	0x00000066
pd_InRcv	0x00000000	pdhad2wa	0x00000000	pdhad2rc 
0x00000000	pdnostok	0x00000000
entersor	0x00000066	resv1	0x00000000	resv2	0x00000000 
resv3	0x00000000

0%	80ms
loop	100	msg_size 16024	env_size	16024

sack_rcv	0x000004b2	sack_snt	0x0000025a	tsns_snt 
0x00000962	tsns_rcv	0x000004b2
pkt_sent	0x00000bc0	pkt_rcvd	0x00000967	tsns_ret 
0x00000000	dup_tsns	0x00000000
hbs__rcv	0x00000000	hbackrcv	0x00000000	htb__snt 
0x00000000	win_prbe	0x00000000
pktswdat	0x000004b2	t3-timeo	0x00000000	dsack-to 
0x00000000	hb_timer	0x00000000
fst_rxts	0x00000000	timerexp	0x00000002	fr_inwin 
0x00000000	blk_rwnd	0x00000000
blk_cwnd	0x000001f5	rwnd_drp	0x00000000	bad_strm 
0x00000000	bad_ssnw	0x00000000
drpnomem	0x00000000	drpfragm	0x00000000	badvtags 
0x00000000	badcsumv	0x00000000
packetin	0x00000969	mcastrcv	0x00000000	hdrdrops 
0x00000000	no_portn	0x00000002
cwnd_nf	0x000001f5	co_snds	0x000009ce	co_nodat	0x00000519 
cw_nu_ss	0x000002bf
max_brst	0x0000018c	expr_del	0x00000000	no_cp_in 
0x00000000	cach_src	0x00000bbf
cw_nocum	0x00000000	cw_incss	0x000001f3	cw_incca 
0x00000000	cw_skip	0x00000000
cw_nu_ca	0x00000000	cw_maxcw	0x00004c2c	diff_ss 
0x00000000	diff_ca	0x00002ec0
tqs@ss	0x000005e0	sqs@ss	0x00000002	tqs@ca	0x00000000 
sqq@ca	0x00000000
lmtu_mov	0x00002ee0	lcnt_mov	0x00000001	sndqctss 
0x00000000	sndqctca	0x00000000
movemax	0x000000ca	move_equ	0x000000c8	nagle_qo 
0x00000000	nagle_of	0x00000067
out_fr_s	0x00000067	sostrnos	0x00000001	nostrnos 
0x00000518	sosnqnos	0x00000002
nosnqnos	0x00000517	intoperr	0x00000000	dupssnrc 
0x00000000	multi-fr	0x00000000
vtag-exp	0x00000965	vtag-bog	0x00000000	t3-safeg 
0x00000000	pd--mbox	0x00000000
pd-ehost	0x00000000	pdmb_wda	0x00000000	pdmb_ctl 
0x00000000	pdmb_bwr	0x00000000
pd_corup	0x00000000	pd_nedat	0x00000000	pd_errpd 
0x00000000	fst_prep	0x00000000
pd_daNFo	0x00000000	pd_dIWin	0x00000000	pd_dIZrw 
0x00000000	pd_BadDa	0x00000000
pd_dMark	0x00000000	ecne_rcv	0x00000000	cwr_perf 
0x00000000	ecne_snt	0x00000000
chun_drp	0x00000000	nollsndq	0x00000000	ll_c_err 
0x00000000	nollcwnd	0x00000063
nollbrst	0x00000000	no_ifqsp	0x00000000	rcvnotcb 
0x00000000	rcvhadtc	0x00000066
pd_InRcv	0x00000000	pdhad2wa	0x00000000	pdhad2rc 
0x00000000	pdnostok	0x00000000
entersor	0x00000066	resv1	0x00000000	resv2	0x00000000 
resv3	0x00000000

---

From: Brad Penoff <[email-masked]>
Date: Wed, 2 Nov 2005 03:13:19 -0800 (PST)

hey Ryan,

Thanks for the reply.  I'll try to clarify what you adressed inline.

brad

On Tue, 1 Nov 2005, Ryan Bickhart wrote:

> Hi Brad,
>
> You mentioned that you are seeing the strange results for some of the
> applications...does this mean that for other applications SCTP is
> performing as expected, even at high latency?

The class of applications that were showing these performance patterns all 
follow the pseudo code.  As for those that performed as expected... <see 
below>

> If this is the case, could
> you explain the differences in behavior (in terms of running over the
> SCTP API directly, like you did below) for the applications that do and
> don't exhibit the strangeness?
>

The applications that do not exhibit these performance patterns under high 
latency only basically had one client doing all sends and the other one 
doing receives; here SCTP and TCP performed nearly identical.  Adding loss 
at any point tipped the scales towards SCTP.

> Also, I'm just curious how your dummynet setup works. Do you have
> dummynet running on each of the 8 client machines, or do you have the
> clients running normally (i.e. no dummynet) and some dedicated dummynet
> box sitting between the clients to add the latency?

Good question.  I forgot to say that on each node, the dummynet and the 
application share the same CPU (so we have no dedicated box for dummynet). 
We also used 'noerror' to set dummynet.

Hope this helps clarify,
brad

>
> Ryan
>
>
> -----Original Message-----
> From: Brad Penoff [mailto:[email-masked]]
> Sent: Tuesday, November 01, 2005 10:07 PM
> To: [email-masked]
> Cc: Humaira Kamal; Mike Yao Chen Tsai; Edith Vong; Alan Wagner
> Subject: strange results with high latency, no loss
>
> Greetings,
>
> We have developed a version of the Message Passing Interface (MPI) to
> run atop SCTP (google "SCTP MPI").  We have an 8 node cluster that runs
> FreeBSD 5.3 with an SCTP implementation obtained directly from Randall
> and colleagues in mid May (thanks!).  As the title reads, we're seeing
> some strange results when running on emulated networks with high latency
> and no loss that are explained below.
>
> Recently, we've been working on tests with increased latency meaning we
> set the delay in Dummynet, with no loss.  For some applications, we're
> "pegs" which are essentially various counters within the kernel.  For
> this data, they are in table A below.  We don't know what most of the
> output is so we were wondering if anyone here could give me a hand
> explaining what is causing this performance drop.  Either it's SCTP's
> error or Dummynet...
>
> describe our application as if it were using the SCTP API directly
> (rather than MPI implemented using SCTP; we have seen the same results
> in non-MPI apps we ran).
>
> Generally, we're confused; can anyone offer any advice on why we're
> seeing this?
>
> Thanks ahead of time,
> brad
>
> ...
>

---

From: Randall Stewart <[email-masked]>
Date: Thu, 10 Nov 2005 10:42:14 -0500

Ryan/Brad/All:

Just to post a little update to this "problem" for those
interested... we visited UBC yesterday and got to the
bottom of it..

SCTP implements the max burst by default ..
When doing this type of testing max.burst on a
dedicated network gets in the way.. which is exactly
what is happening.. and TCP does NOT implement this.
So to make a fair comparison I had Ryan change his
sysctl.conf to change max.burst to 255... i.e.
net.inet.sctp.maxburst=255

or

sysctl -w "net.inet.sctp.maxburst=255"

from a root command line..

And now he sees no unusal results... what it was is
TCP keeps blasting away and when the cwnd gets as big
as the size of the data he is sending.. where SCTP would
stop.. adding an extra round trip... by changing max.burst
for a private network like this... it makes SCTP and TCP
the same.. and now they seem to be getting the results
they expect :-D

R

Brad Penoff wrote:
> The class of applications that were showing these performance patterns 
> all follow the pseudo code.  As for those that performed as expected... 
> <see below>
> 
> The applications that do not exhibit these performance patterns under 
> high latency only basically had one client doing all sends and the other 
> one doing receives; here SCTP and TCP performed nearly identical.  
> Adding loss at any point tipped the scales towards SCTP.
> 
> application share the same CPU (so we have no dedicated box for 
> dummynet). We also used 'noerror' to set dummynet.
> 
> Hope this helps clarify,
> brad
> 
> 
>>
>> Ryan
>>
>>

-- 
Randall Stewart
ITD - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 10 Nov 2005 11:25:35 -0700

Randall,

I thought they were testing high-latency, no loss networks:
what makes their situation so different from real-world
satellite communication?

If Max.burst disadvantages SCTP in comparison to TCP, should
it not be scrapped?

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Thu, 10 Nov 2005 14:54:49 -0500

Well, it is settable using sysctl. Plus, some TCP implementations may
implement max burst. In fact, I remember the last time I looked at
Linux's code (both 2.4 and 2.6), I was surprised to find that they
implement a max burst feature.

--
Armando
www.armandocaro.net

---

From: Randall Stewart <[email-masked]>
Date: Thu, 10 Nov 2005 18:44:48 -0500

> If Max.burst disadvantages SCTP in comparison to TCP, should

Max.burst has to do with routers in the path..

They are testing in a simple

Host |------------------------| Host

over a switch. There are no other
nodes involved. Thus there is no danger
of inducing congestion in the network.. because
there really is not a network other than a E-net.

If they would setup a routed network with real routers
and leave max.burst set to the default level I think
they would find that they do a LOT better than tcp
since tcp drives the routers to drop packets quicker..

In fact this is what we have seen in some testing in a
real satellite network...

Note that there RTT times are not anywhere near a sat network
either 40ms and 80ms is not even close to a sat link..

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 10 Nov 2005 18:48:54 -0500

> implement a max burst feature.
So does sun according to Kacheong... if I remember right.

Of course we should be able to get the code and look for
ourselves with the Open Solaris stuff :-D

R

---

From: Janardhan Iyengar <[email-masked]>
Date: Fri, 11 Nov 2005 15:57:17 -0500 (EST)

Brian,

Adding to Randy's comments:

max.burst is primarily meaningful in a large network environment like the 
Internet, where a large cwnd may allow a sender to get LARGE instantaneous 
throughput.

If a sender grows its cwnd to a large value, and then goes idle for a 
little while, and then starts sending again, it should not be allowed to 
send cwnd amount of data because its ack clock is lost. cwnd reflects the 
amount of data that can be outstanding over the interval of an RTT; in the 
general case, cwnd SHOULD NOT allow a sender to burst out cwnd amount of 
data at line-rate, or within a fraction of an RTT. Such a burst can cause 
sudden congestion and losses downstream.

The idea of max burst has been around for a while, but may not necessarily 
be implemented in all OSes (some do). But it is largely a _good thing_. As 
far as I know, SCTP is the first place where this idea was written in 
stone.

- jana

> either 40ms and 80ms is not even close to a sat link..

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 11 Nov 2005 14:11:00 -0700

Janardhan,

Yes, I agree, it is functionally misplaced.  As it is applicable to
all transports and really only affects throttling and queuing of
packets to less than line rate, it should be placed in the IP layer
and removed from the transport layer.

--brian

---

From: Janardhan Iyengar <[email-masked]>
Date: Fri, 11 Nov 2005 17:10:36 -0500 (EST)

Brian,

I don't see how max burst functionality can be put in the IP layer.

Yes, it throttles sending rate, but the pacing is decided by an ack clock 
which is at the transport layer. The ack clock "tick" will be different 
for different end-to-end connections, and cannot be generalized across 
all transport layer connections.

- jana

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 11 Nov 2005 17:40:07 -0700

Janardhan,

I though you made reference to line-rate.  Ack rate an line-rate are
two different things.  The number of bytes transmitted in a given RTT
(ack interval) are already limited by cwnd.  The transport has no idea
of line rate.  The IP layer does.

--brian

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 11 Nov 2005 20:42:56 -0500

Brian,

cwnd is meant to limit the amount of data outstanding, but that does
not mean that the sender can burst that amount out at line rate.
That's what Jana was referring to. The sender should spread the cwnd
amount of data across an RTT. That's why slow start exists... to
gradually get the flow going. Once the cwnd is large and the flow goes
idle, the sender needs to be careful about getting the flow going
again.

Imagine cwnd=50 packets... it's fine to have 50 outstanding, meaning
that some are in flight to the receiver, and some have acks in flight
back to the sender. However, it is not a good idea to send 50 packets
at once. The sender is likely to be hurting itself, because one of the
upstream routers might not be able to support such a large burst.
(Disclaimer: 50 pkts is just a number out of thin air. I'm not saying
that routers can't absorb a burst of 50 pkts)

--
Armando
www.armandocaro.net

---

From: Janardhan Iyengar <[email-masked]>
Date: Fri, 11 Nov 2005 20:59:02 -0500 (EST)

Brian,

Ack-rate decides the rate at which new data gets clocked out. And this 
ack-rate is specific to a transport connection. The reason being that each 
tranport connection may have a different RTT, a different bottleneck, adn 
different bottleneck and network path properties. All of these things 
decide ack clock rate (or the space between acks), which in turn paces
data packets. Note that cwnd amount of data is sent, but this data is 
paced over an RTT with the help of the ack clock.

I assume that the ack rate is lesser than line rate at the end host, which 
is a fair assumption for a lot of folks using the Internet.

I mentioned line rate because in the absence of max burst, an idle sender 
with a large cwnd is limited only by line rate, which may be much higher 
than the acceptable rate downstream. When active data flow existed, the 
sender could use the ack clock to figure out what the acceptable rate
downstream is, but without the ack clock, the sender is limited only by 
line rate - which may be unacceptable (assuming that line rate is higher 
than acceptable rate downstream somewhere).

Max burst has nothing to do with line rate per se. It controls sending 
when the ack clock is being lost. The IP layer is not aware of ack clocks 
per transport connection, and can therefore not do anything.

- jana

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 12 Nov 2005 02:28:39 -0700

Armando,

Yes, I understand, Armando, however, a transport protocol can only
control how fast it delivers packets for transmission to the IP layer.
The IP layer, and ultimately, the interface driver determines when, and
at what rate, they are transmitted on the line.

For example, with queuing and routing policies in the IP layer(just
local to the sending host), it is possible that the transport could
spread these 50 packets out over the RTT and yet the IP layer queues
them and then passes them all to the interface at the same time to be
burst on the line at line rate: particularly during periods of
congestion.  Just about any mechanism for priority queueing of IP
packets at the IP layer could cause this effect (i.e. the lower priority
packets get queued and ultimately sent in a burst).

Just mention of the term line-rate should alert one to the fact that
it cannot be a transport-layer-only solution.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 12 Nov 2005 03:22:34 -0700

Janardhan,

On Fri, 11 Nov 2005, Janardhan Iyengar wrote:

> with a large cwnd is limited only by line rate,

Here I lose you.  The transport is not responsible for placing packets on
the line.  I would say that the idle sender with a large cwnd is limited
by the queueing delays and disciplines presented by the local IP layer.

> which may be much higher 
> line rate

And here again.

> - which may be unacceptable (assuming that line rate is higher 

Well that might be the intention, however, the way that the procedures
are currently worded don't appear to achieve that.   Also, the current
procedures drastically degrade cwnd instead of just, as you intend,
throttling transmissions with the ack clock.

If the mechanism achieved, as you intend, to simply throttle the rate of
the sender to that rate at which it would have sent data while normally
fully achieving its cwnd, then it should not have caused a problem in the
simple host-to-host scenario that Randall described.

I think that your purpose could be achieved by degrading cwnd on the
basis of RTO rather than on the rather arbitrary basis currently
described.

Also, not all networks are barbells, some are footballs, even in the
Internet.  I think that this type of throttling is currently the
responsibility of the router at bottleneck (if a bottleneck even exists)
and that the normal response of packet dropping (or ECN or something
similar at the IP level) should suffice.  Any of these responses will
cause cwnd to degrade exponentially rather than the sender attempting
to be psychic and throttling itself in advance of any congestion.

There are many situations in which even a burst of a fully developed
cwnd can easily be handled by a network, even through a bottleneck.
The current mechanism certainly cannot accurately predict what the
prevailing conditions at any bottleneck will be when the burst (if the
local IP layer even delivers a burst) arrives.

--brian