Subject: question regarding retransmission

From: Daisy Chang <[email-masked]>
Date: Tue, 22 Oct 2002 14:27:11 -0500

Hi,
    I was looking to fix a problem in our SCTP retransmission code and
had a confusion on the following scenario. I would like to clear it up
and make sure that we will do it right.  The confusion is about
retransmission under the circumstances of outstanding chunks eligible
for retransmit (residuals from previous T3rtx timeout) and data on
multiple transports in an association.

    The scenario:
    If there are 3 transports - t1, t2, t3. Where t1 is the primary
path, and t2 is the first choice for doing any retransmission with.
Assuming that chunks with TSN #1, #2, and #3 have been transmitted thru
t1, and the next chunk, #4, was sent with sendmsg() MSG_ADDR_OVER flag
on and destined to be sent thru t3.   Also, assuming that t1, t2, t3
have the same MTU sizes, and that each chunk would fit in the MTU just
right.  So,
  Transmitted:
    t1: #1, #2, #3
    t2: nothing
    t3: #4

With this, if t1's T3-rtx timer expired, #1, #2, #3 would be marked for
retramission but only #1 would be sent out due to the MTU constraint
(RFC 2960 6.3 E3).  So we will end up with
    Transmitted:
    t1: nothing
    t2: #1
    t3: #4
    marked for retransmission: #2, #3

Next, if t3's T3-rtx timer pops, a retransmission needs to be made again
thru t2, which chunk should be sent out?   Is it #1, or #2 or even #4?
In other words, which one of the following would be the result of this
timeout? Is it

    Transmitted:
    t1: nothing
    t2: #1 (again)
    t3: nothing
    marked for retransmission: #2, #3. #4
or
    Transmitted:
    t1: nothing
    t2: #1, #2
    t3: nothing
    marked for retransmission:  #3. #4
or
    Transmitted:
    t1: nothing
    t2: #1, #4
    t3: nothing
    marked for retransmission:  #2, #3

--

Thanks,
Daisy

+--------------------------------------------------------------------------+
Daisy Chang                     IBM Linux Technology Center
e-mail: [email-masked]
Tel: 512-838-4194  T/L: 678-4194  Fax: 512-838-4663
11400 Burnet Road, M/S 9812     Austin, TX 78758-3493

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 22 Oct 2002 16:07:45 -0500

Daisy,

Admittedly, I haven't read through some late rtx rules enhancements in
impl guide, but my intuition is that you are over-engineering over a
simple problem. See my comments below...

Daisy Chang wrote:
> Assuming that chunks with TSN #1, #2, and #3 have been transmitted
> thru t1, and the next chunk, #4, was sent with sendmsg() MSG_ADDR_OVER
> flag on and destined to be sent thru t3.   Also, assuming that t1, t2,
> t3 have the same MTU sizes, and that each chunk would fit in the MTU
> just right.  So,
>   Transmitted:
>     t1: #1, #2, #3
>     t2: nothing
>     t3: #4
> 
> With this, if t1's T3-rtx timer expired, #1, #2, #3 would be marked
> for retramission but only #1 would be sent out due to the MTU
> constraint (RFC 2960 6.3 E3).  So we will end up with
>     Transmitted:
>     t1: nothing
>     t2: #1
>     t3: #4
> Next, if t3's T3-rtx timer pops, a retransmission needs to be made
> again thru t2, which chunk should be sent out?   

sequence of events should be straightforward here:
 1) collapse t3 cwind to 1 MTU: done.
 2) mark #4 for re-trans: done.
 3) see if there is an alternate dest that is still up: yes (t1 and t2)

Now the spec doesn't say how to select btw t1 or t2, but it doesn't
matter that much I think. If you pick t1 and re-send #4 thru it
(remember you still have 1 chunk/1 MTU cwnd quota available on t1), I
think it's perfectly legal and nothing terrible can happen. If you pick
t2 (as you seem to prefer), you will have to queue #4 behind #2 and #3,
since you don't have any quato left on t2 since t2 is still under the
"max one chunk of upto MTU in flight allowed" mode. But either way you
would be just fine. 

-Qiaobing

> Is it #1, or #2 or
> even #4?  In other words, which one of the following would be the
> result of this timeout? Is it
> 
>     Transmitted:
>     t1: nothing
>     t2: #1 (again)
>     t3: nothing

---

From: Daisy Chang <[email-masked]>
Date: Tue, 22 Oct 2002 17:50:25 -0500

> "max one chunk of upto MTU in flight allowed" mode. But either way you

Thanks, Qiaobing.  This is very good information for me.

Based on your answer, may I say that, different chunks might be retransmitted at
different transport depends on the choices of the outgoing transport at the time,
and since the choices of the outgoing transport for retransmission is up to the
implementation, there is really not a definite sequence of chunks being mandated
by the RFC for retransmission?  I am not implying anything like retransmitting
chunks in random order, on the contrary, my case is that we are considering
retransmitting chunks always in the same order, regardless the choice of the
transport. That is, back to  the example, at the time of t3 timeout, we will
queue #4 behind #2 and #3, and possibly send out #2 either on t1 or t2,
regardless which we end up picking.  This is OK, isn't it?

One more thing, in your description,  you said "if you pick t2 (as you seem to
prefer), you will have to queue #4 behind #2 and #3, since you don't have any
quato left on t2 since t2 is still under the 'max one chunk of upto MTU in flight
allowed' mode", could you explain how t2 got into this mode?  If we assume that,
this example begins with all three transports' cwnd up to, let's say, 4 * MTU at
the moment,  do you think that t2 would still end up in this mode given the
sequence of events?

Best Regards
Daisy

--

Thanks,
Daisy

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 22 Oct 2002 18:50:29 -0500

....
> Based on your answer, may I say that, different chunks might be
> retransmitted at different transport depends on the choices of the
> outgoing transport at the time, and since the choices of the outgoing
> transport for retransmission is up to the implementation, there is
> really not a definite sequence of chunks being mandated by the RFC for
> retransmission? I am not implying anything like retransmitting chunks
> in random order, on the contrary, my case is that we are considering
> retransmitting chunks always in the same order, regardless the choice
> of the transport.

What your scenario depicts is a case of multiple time-outs. I think it
would be hard (and may not worth the effort) to still try to keep the tx
order of all the affected chunks under such circumstance (and it would
also be fruitless even if you try, since it won't mean much to the
arrival order at the receiver side). IMO, it'd be nice to maintain the
order in such a case. but if it's too much effort for the impl to do
that, then why bother...

> That is, back to  the example, at the time of t3
> timeout, we will queue #4 behind #2 and #3, and possibly send out #2
> either on t1 or t2, regardless which we end up picking.  This is OK,
> isn't it?

I believe so, as far as you always obey the cwnd rules. Also see below.

> 
> One more thing, in your description,  you said "if you pick t2 (as you
> seem to prefer), you will have to queue #4 behind #2 and #3, since you
> don't have any quato left on t2 since t2 is still under the 'max one
> chunk of upto MTU in flight allowed' mode", could you explain how t2
> got into this mode?  

My mistake, when I wrote that I was thinking wrongly that t2 was the one
that got timed-out. But the fact that you didn't rtx #2 and #3 along
with #1 indicates that you ran out of cwnd on t2. So, if you insist to
picking t2 for rtx #4, you have to queue it after #2 and #3.

You basically have a case where you have more data marked for rtx than
the cwnd of one dest addr allows and at the same time you have other
dest addresses in the association that still have cwnd room available.
My question to you is why keeping yourself from using the available cwnd
on those alternate dest addresses to re-send the rest of the data? I
don't think the spec prohibit that. It's up to the impl.

> If we assume that, this example begins with all
> three transports' cwnd up to, let's say, 4 * MTU at the moment,  do
> you think that t2 would still end up in this mode given the sequence
> of events?
> 
> Best Regards
> Daisy
> 
> >
> > > Is it #1, or #2 or

---

From: Daisy Chang <[email-masked]>
Date: Wed, 23 Oct 2002 11:18:32 -0500

> don't think the spec prohibit that. It's up to the impl.

Ah, OK.  I think that, the limitation of one MTU  (single packet) came from RFC
6.3.3 (E3).   Simply reading those bullets in 6.3.3, I would say that regardless
the cwnd size of t2 at the time of t3 timeout, as long as it is bigger than, or
equal to, 1 MTU, one and only one packet will be sent out with the retransmitted
data.  This is not a limitation from anything on t2, but rules for "Handle T3-rtx
expiration".

But apparently you were suggesting that we should send retransmission up to what is
allowed by cwnd on t2.  I gathered that you might be referring to the following
paragraph in the end of 6.3.3,

    "Note: Any DATA chunks that were sent to the address for which the
    T3-rtx timer expired but did not fit in one MTU (rule E3 above),
   should be marked for retransmission and sent as soon as cwnd allows
   (normally when a SACK arrives)."

Is that right? So, without SACK in the picture (not in our example), I should
interprete this paragraph such that, the retransmission size at the T3-rtx timeout
is subject to the cwnd size of the destination transport address to which the
retransmission is being sent (this may be different from the address for which the
timer expires).

If this is true, I would agree that there is no reason why we couldn't send #2 and
#3 on t2, right after #1, at the same time.  The spec didn't prohibit that, but can
be more clear, I think. :-)

Thanks
Daisy

--

Thanks,
Daisy

---

From: Qiaobing Xie <[email-masked]>
Date: Wed, 23 Oct 2002 12:20:09 -0500

...
> Ah, OK.  I think that, the limitation of one MTU  (single packet) came
> from RFC 6.3.3 (E3).   Simply reading those bullets in 6.3.3, I would
> say that regardless the cwnd size of t2 at the time of t3 timeout, as
> long as it is bigger than, or equal to, 1 MTU, one and only one packet
> will be sent out with the retransmitted data.  This is not a
> limitation from anything on t2, but rules for "Handle T3-rtx
> expiration".

Remember SCTP congestion control is on a per dest addr basis (except for
a shared rwnd for obvious reason). Time-out on one dest addr should not
affect the CC behavior of any other dest addr. E.g., if you get a t-o on
t3, you would not collapse cwnd of t2, would you? Some wording in the
spec is made more for a single-homed peer (I think somewhere in the RFC
Lixia made a comment on this). 

> 
> But apparently you were suggesting that we should send retransmission
> up to what is allowed by cwnd on t2.  I gathered that you might be
> referring to the following paragraph in the end of 6.3.3,
> 

If the peer is single-homed, then you have to wait for a SACK to free up
your cwnd. But in your case, you have 3 dest addrs and thus 3 cwnds and
you still have room left in some of those 3 cwnds, right? Your impl does
not have to take this advantage, but you can.

> Is that right? So, without SACK in the picture (not in our example), I
> should interprete this paragraph such that, the retransmission size at
> the T3-rtx timeout is subject to the cwnd size of the destination
> transport address to which the retransmission is being sent (this may
> be different from the address for which the timer expires).

You are almost right here. The exception is the dest addr which timed
out.. that guy has one additional constraint - you can not have more
than one packet in flight.

> 
> If this is true, I would agree that there is no reason why we couldn't
> send #2 and #3 on t2, right after #1, at the same time.  The spec
> didn't prohibit that, but can be more clear, I think. :-)

The intention in the spec is that we discourage people from using more
than one cwnd simultaneously if the peer is multi-homed, since otherwise
an SCTP asoc would gain a unfair bandwidth advantage over a TCP
connection. So we say in the spec "don't round-robin dest addresses".
But when dealing with a t-o, I think it's reasonable to allow one take
full advantage of the available cwnds. Of cause, an impl can choose not
to...

---

From: Daisy Chang <[email-masked]>
Date: Wed, 23 Oct 2002 13:59:40 -0500

Qiaobing,
    Your explanation really cleared up quite some confusions that I have in the
past couple of days.  Thank you so much.

Regards,
Daisy

--

Thanks,
Daisy