Subject: Question about the TSN order of retransmissions

From: Karoline Malmkjær <[email-masked]>
Date: Fri, 10 Feb 2006 10:58:33 +0100

Hi list.

In RFC2960, section 6.3.2, rule E3 states that one packet-full of
"the earliest (i.e., lowest TSN) outstanding DATA chunks for the
address for which the T3-rtx has expired" should be
retransmitted.  

Does this hold even if there are other chunks, with lower TSNs,
already available for retransmission?  This might happen if the
timer on another address had expired and more than one
packet-full of chunks were outstanding on that other address.

It seems to me that it would be more efficient at the receiver to
always retransmit the lowest TSNs, independently of their
previous destination.

Thanks in advance for any replies!

  Karoline

-- 
Karoline Malmkjær, System Developer
Ericsson Telebit
DK-8260 Viby J, Denmark

---

From: Mark Butler <[email-masked]>
Date: Fri, 10 Feb 2006 21:03:42 -0700

Karoline Malmkjær wrote:

>already available for retransmission?  
>
Yes.

>This might happen if the
T3-rtx based retransmission is always done per destination address 
because expiration of the timer on one address is not a good indication 
that chunks were not received on other addresses, particularly if the 
RTT varies significantly between them.  So the assumption is that data 
transmitted to other addresses is fine until the T3-rtx timers specific 
to those addresses expire or they qualify for fast retransmit.  
Hopefully only one path fails at a time - T3-rtx expiration is a 
preliminary indication of path failure, so only those chunks are 
re-transmitted to a new address.

 - Mark

---

From: Mark Butler <[email-masked]>
Date: Fri, 10 Feb 2006 22:24:49 -0700

Mark Butler wrote:

> because expiration of the timer on one address is not a good 
> indication that chunks were not received on other addresses, 
> particularly if the RTT varies significantly between them.  So the 
> assumption is that data transmitted to other addresses is fine until 
> the T3-rtx timers specific to those addresses expire or they qualify 
> for fast retransmit.  Hopefully only one path fails at a time - T3-rtx 
> expiration is a preliminary indication of path failure, so only those 
> chunks are re-transmitted to a new address.

Let me try that again....

 I think strictly speaking you are not allowed to do what you suggest, 
but it sounds like a good idea to me.  Retransmitting retransmit-pending 
chunks with earlier TSNs first in this case shouldn't cause any problems 
and should help quite a bit in some situtations.

 - Mark

---

From: Randall Stewart <[email-masked]>
Date: Sat, 11 Feb 2006 06:45:54 -0500

> packet-full of chunks were outstanding on that other address.

So you are saying something like:

---TSN0--->dest-A
---TSN1--->dest-A
---TSN2--->dest-B
---TSN3--->dest-B

T3-rxt on Dest-A
---TSN0--->dest-B (to alt assuming cwnd space)
TSN1-Ready to rtx (to dest-B)

T3-rxt on Dest-B
---TSN2--->dest-A (to alt assuming cwnd space)

Can you instead do

---TSN1--->dest-A?

This is an interesting question...

First of all it means you would not be retransmitting to the
alternate destination.. which some of Armando's research shows
can be a good thing..

Second why were you sending to both? Unless of coure
the user specified to send to the non-primary address...

Third even the slightest difference in path RTT's will
make things dodgy at best (especially for FR)..

Fourth, in this scenario when you go to retransmit would
you have cwnd space to dest-A? I guess you might.. since
the timer expired and left the cwnd at 1MTU, and nothing
is outstanding (assuming that the user has not sent more).

Fifth.. even more strange.. when B's timer expires
TSN0 is now the lowest possible TSN sent to B.. even
though it was just retransmitted, and probably has
not had a chance to get a SACK back.  This is the
dodgy bits that happen when one starts dong
CMT (conncurrent multipath transfer).. and things
that one has to take into account... Jana as been
looking at this :-D

In this particular state (if I have it right) it seems
to me you could do either retransmission.. yes
I know what the RFC states.. but the RFC was
written assuming you are sending to one primary
and when the timer goes off you move to the
alternate which as been idle...

I think I would:

1) Not mark TSN0 (I keep a timestamp on each
    outgoing chunk so I can tell when this
    sort of thing happens).

2) Would send to A any that fit (lowest TSN's)

3) Would at that same time send to B any
    that fit (lowest TSN's)

4) Still move things to the alternate before
    making my bundling decisions...

All of which (2 and 3) taking into account the
time things were sent so that I don't do something
silly like retransit TSN0 (which has not had
a chance to be acked).

R

-- 
Randall Stewart
ITD - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: Karoline Malmkjær <[email-masked]>
Date: Sat, 11 Feb 2006 23:53:24 +0100

Randall Stewart <[email-masked]> writes:

>> retransmitted.  Does this hold even if there are other chunks,
>> with lower TSNs,

Thank you for your comments!

In fact, I was rather thinking of three addresses:

---TSN0--->dest-A
---TSN1--->dest-A
---TSN2--->dest-B
---TSN3--->dest-B

T3-rxt on Dest-A
---TSN0--->dest-C (C is unused, so we're good on cwnd space)
TSN1-Ready to rtx (to dest-C)

T3-rxt on Dest-B
---TSN2--->dest-C (we're still good on cwnd space)

This seems unfortunate, since TSN2 gets sent to C before TSN1, so
I would rather do

---TSN1--->dest-C

I was only thinking of one timer per destination, so if there
were only two addresses, I would restart the timer on dest-B when
retransmitting (in order not to retransmit TSN-0, indeed).  So I
guess that two addresses are not enough to exhibit the behaviour
with that timer-setup.

> that one has to take into account... Jana as been

If we have three or more addresses and user continues sending new
data (which will trigger a send to primary), I think the
situation gets very close to CMT issues no matter what.  cwnd on
primary does get reduced to 1 MTU, but since we retransmit to
alternate, the cwnd on primary is unused.  So 1 MTU of any data
marked for retransmission gets sent to primary on the next user
send():

---TSN0--->dest-A
---TSN1--->dest-A
---TSN2--->dest-B
---TSN3--->dest-B

T3-rxt on Dest-A
---TSN0--->dest-C (C is unused, so we're good on cwnd space)
TSN1-Ready to rtx (to dest-C)

T3-rxt on Dest-B
---TSN2--->dest-C (we're still good on cwnd space)

User send(msg4)
---TSN1--->dest-A (queuing up TSN4)

Or should sending on user send() events be blocked while
re-transmitting?

> alternate which as been idle...

Yes, that's what I suspected.  So maybe the nit-picking question
is whether the phrasing in the RFC is to be read as
"implementation advice" or "requirement".  It doesn't say "MUST",
so one could argue for the first, as long as the general
behaviour wrt congestion control, liveness, etc. is maintained.

>    making my bundling decisions...

So you would send off one MTU-A and one MTU-B?  Or just prepare
them for sending?

> silly like retransit TSN0 (which has not had

OK

  Karoline

-- 
Karoline Malmkjær, M.Sc., PhD    System Developer
Ericsson Telebit (LMD/R)         Tel: +45 89 38 51 00
Skanderborgvej 232               Fax: +45 89 38 51 01
DK-8260 Viby J, Denmark          Email: [email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 11 Feb 2006 17:12:26 -0700

Karoline,

On Sat, 11 Feb 2006, Karoline Malmkj?r wrote:

--X--snip--X--
> ---TSN2--->dest-C (we're still good on cwnd space)

How can you be good on cwnd space for TSN2 and not TSN1?  Either
TSN0 is still in flight to C or it was acked.  If it is still in
flight (and the chunks are about the same size) there is no cwnd
available for TSN2 (otherwise TSN1 would have been sent with
TSN0), so I assume that TSN0 was acked.  When an ack was
received for TSN0, it should trigger sending TSN1.

So I don't see that your scenario can play out the way you
describe.

(If TSN2 is much smaller than TSN1 and there is still cwnd
available, count yourself lucky that it was retransmitted ahead
of TSN1.)

--brian

> guess that two addresses are not enough to exhibit the behaviour
--X--snip--X--

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Karoline Malmkjær <[email-masked]>
Date: Sun, 12 Feb 2006 11:58:15 +0100

> Karoline,

Hi Brian,

> So I don't see that your scenario can play out the way you

For simplicity and without loss of generality, lets assume that
only one chunk fits in each MTU (and that MTU is about the same
for all destinations).

When the retx timer goes off for dest-A, dest-C is unused, so
cwnd-C is 2*MTU (or min(4*MTU, max (2*MTU, 4380 bytes)).  But
since it is a timeout-triggered retransmission, only one MTU is
sent.  So there is space left in cwnd-C.

If the next event is that the retx timer goes off for dest-B,
again only one MTU is sent, and it only contains chunks with
original destination dest-B (by rule E3), so TSN-1 is not a
candidate.  So by the RFC, TSN-2 gets sent to dest-C.

  Karoline

---

From: Randall Stewart <[email-masked]>
Date: Sun, 12 Feb 2006 07:30:41 -0500

A thought below...

> candidate.  So by the RFC, TSN-2 gets sent to dest-C.

But thinking about this.. when you go to do the
retransmission..

you
a) mark all outstanding to dest-B as needing to be
    retransmitted (I also add the bit about checking
    times)

b) select and mark them to an alternate address (in your
    example dest-C)

c) pick the earliest TSN and retransmit it.

Now, by definition, NOTHING is outstanding to
dest-B.. so at this point <c> refers to dest-C.

What is the earliest outstanding TSN headed to
dest-C... its not 2 its TSN-1..

R

---

From: Karoline Malmkjær <[email-masked]>
Date: Tue, 14 Feb 2006 11:34:04 +0100

> dest-C... its not 2 its TSN-1..

Yes, that's more or less what I would like to do, and I think I
see how to do it.

I don't see it causing any inter-op problems, either, but a very
nit-picking compliance test might consider it a violation (once
such tools get on the market for SCTP).

So I just want to make sure that such a behaviour is "legal",
i.e., that it can be considered RFC-compliant.

Could one argue that the RFC2960 phrasing "the address for which
the T3-rtx has expired" really only covers the situation where
only one timer has expired?

Then, there are two ways to generalize to the situation where
several timers expire immediately after each other:

1) "any address (or chunk if these have separate timers) for
   which the T3-rtx has expired and not been restarted"

2) "the exact address (or chunks) for which the T3-rtx has just
   expired".

Both of these would then be legal, and it would depend on the
data-structures of each implementation which one is the easiest
to handle.

Or is that too much of a "creative license" with the text?

Thanks!

  Karoline

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 14 Feb 2006 12:42:09 +0100

Hi Karoline,

see my comments in-line.

Best regards
Michael

> such tools get on the market for SCTP).
You can have a look at the ETSI conformance tests and the implementation
available at sctp.fh-muenster.de. If I remember it correctly there
are only very limited tests on retransmissions.

---

From: Randall Stewart <[email-masked]>
Date: Tue, 14 Feb 2006 11:27:00 -0500

My comments in line...

> such tools get on the market for SCTP).

If such a thing occurs.. then we need to get the
builder of such a tool to fix it.. since this
would be WRONG on the tool builders part IMO..

> i.e., that it can be considered RFC-compliant.

I consider it legal ;-D
> only one timer has expired?
Yep... that is all I believe we wrote it in the context
of.. If one did not limit the scope someway one
could write 100's of more pages .. and that
would be very boring to the reader and to the
writer :-D

> data-structures of each implementation which one is the easiest
Yep...

> Or is that too much of a "creative license" with the text?

I think this is not "creative license" but implementation details..
Which is always allowed some flexibility.. as long as
it does not cause inter-operability issues.. which this one
should not..

R