Subject: [tcpm] TLP pto scheduling

From: hiren panchasara <[email-masked]>
Date: Wed, 11 Oct 2017 14:50:11 -0700

Looking at TLP draft and RACK draft,

   Open state: the sender has so far received in-sequence ACKs with no
   SACK blocks, and no other indications (such as retransmission
   timeout) that a loss may have occurred.

So, loss probes are not scheduled after the first SACK block seen on the
connection? Shouldn't they be scheduled as long as the connection is
not in recovery? Or am I missing something?

Any clarification would be great.

Cheers,
Hiren

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 11 Oct 2017 22:40:36 +0000

On Wed, Oct 11, 2017 at 9:50 PM, hiren panchasara
<[email-masked]> wrote:
> connection?

Right.

> Shouldn't they be scheduled as long as the connection is

If there is a SACK block seen, then the sender should be scheduling a
RACK timer, which would fire faster than the TLP timer anyway.

Basically, if there is a SACK block then you have info about how long
it should have taken for the previous packets to arrive. A TLP is a
probe, intended for when there is no feedback at all arriving at the
sender.

cheers,
neal

---

From: hiren panchasara <[email-masked]>
Date: Wed, 11 Oct 2017 16:24:32 -0700

> probe, intended for when there is no feedback at all arriving at the

So, I think the idea is that a SACK block is all RACK needs to start
marking previous packets as lost.

I am still confused by how these 2 conditions are different wrt TLP:
1) slow-start before receiving any SACK 
2) Congestion avoidance after a loss-recovery episode is finished and
connection is humming along

Now if acks stop arriving, we send loss probe in case of 1) but not in
case of 2).

How would RACK detect those "tail" packets lost in that situation for
2)?

Probably I am missing something here. :-)

Thanks for taking time and responding.
Cheers,
Hiren

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 11 Oct 2017 23:38:07 +0000

> marking previous packets as lost.

Yes.

> connection is humming along

These are not different with respect to TLP. Those are 2 different
states in the congestion control algorithm state machine. But TLP does
not care about the congestion control algorithm state machine. TLP
only cares about the loss recovery state machine. We used the term
"Open state" in the draft to refer to the state in a loss recovery
state machine when there are no SACKed sequence ranges in the SACK
scoreboard, and we are not in the middle of fast recovery or timeout
recovery (the terminology is borrowed from the Linux TCP loss recovery
state machine's TCP_CA_Open state).

> Now if acks stop arriving, we send loss probe in case of 1) but not in

Nope, both (1) and (2) are in the "Open state" in the loss recovery
state machine, so we would send a TLP in either case.

I think the sentence in the draft is ambiguous. It seems you
interpreted "so far" to mean "so far in the lifetime of the
connection", when the draft is trying to convey "so far in the current
loss recovery epoch" (or something like that). :-)

We will try to clarify that in the next rev of the draft.

thanks,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 11 Oct 2017 23:43:53 +0000

> We will try to clarify that in the next rev of the draft.

How's this:

  Open state: the sender's loss recovery state machine is in
  its normal, default state: there are no SACKed sequence
  ranges in the SACK scoreboard, and neither fast recovery,
  timeout-based recovery, nor ECN-based cwnd reduction
  are underway.

neal

---

From: hiren panchasara <[email-masked]>
Date: Wed, 11 Oct 2017 16:49:49 -0700

> loss recovery epoch" (or something like that). :-)

Awesome!
Thanks a ton for clearing my doubts.
> We will try to clarify that in the next rev of the draft.

I guess RACK draft is going to be the source of truth for TLP or the TLP
draft is going to be maintained?

Cheers,
Hiren

---

From: hiren panchasara <[email-masked]>
Date: Wed, 11 Oct 2017 16:50:59 -0700

>   timeout-based recovery, nor ECN-based cwnd reduction

Sounds good! Thank you.

Cheers,
Hiren

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 11 Oct 2017 23:53:17 +0000

> draft is going to be maintained?

Yes, the RACK draft will continue to be the official latest source for
the TLP mechanism.

neal

---

From: Michael Welzl <[email-masked]>
Date: Thu, 12 Oct 2017 07:43:44 +0200

Hi,

This was an interesting dialogue to follow, thanks!

> On Oct 12, 2017, at 1:53 AM, Neal Cardwell <[email-masked]> wrote:
> 

Good to know!

While we’re at it - I have a question about RACK and TLP too:

Say, we have a cwnd of 100 segments and hit heavy congestion: all packets are lost.
ssthresh becomes 50, or 70 in case of Cubic. A TLP timer would fire, and let’s say that this one single packet would make it. Because an ACK arrives, the sender would stay in FR, and from the resulting SACK info, RACK would know that this was much later than all the previous packets, and hence would immediately send out a burst (maybe paced, but that’s another story) of 50 or 70 packets, right?

Have I understood this correctly? If so, isn’t this a pretty aggressive behavior?  I mean, we have one RTT of losing everything, a probe timer later we get one packet through, and then we have so many packets again, straight away after this long pause… this may or may not be the right thing to do. As you know I like the RACK idea, but if it can turn out to be as aggressive as that, I would appreciate evaluation results showing that this doesn’t become a problem (and if it does become a problem, some limitations could be put in place). I don’t think we’ve seen this type of evaluation so far.

Cheers,
Michael

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 12 Oct 2017 06:33:33 +0000

> Have I understood this correctly? If so, isn’t this a pretty aggressive behavior?  I mean, we have one RTT of losing everything, a probe timer later we get one packet through, and then we have so m…

Good question. We cover that kind of question (with an example very
close to yours) in section 7.5, "Interaction with congestion control",
in:
  https://tools.ietf.org/html/draft-ietf-tcpm-rack-02

At a high level, we feel that what the sender does when it gets a
small number of packets SACKed that indicate a large number of packet
lost is a congestion control decision, which we feel should be
considered to be decoupled (as much as possible) from the loss
detection logic in RACK.

That said, as we mention in the draft, we recommend combining RACK
with PRR [RFC6937]. After receiving the SACK for the TLP probe packet,
at which point RACK marks the whole rest of the flight lost, PRR would
not send a burst, but rather would slow-start, starting from 1 packet
(1, 2, 4, 8....).

neal

---

From: Michael Welzl <[email-masked]>
Date: Thu, 12 Oct 2017 08:55:35 +0200

Hi,

> On Oct 12, 2017, at 8:33 AM, Neal Cardwell <[email-masked]> wrote:
> 
>  https://tools.ietf.org/html/draft-ietf-tcpm-rack-02 <https://tools.ietf.org/html/draft-ietf-tcpm-rack-02>

Ah, right - sorry for missing this  (I had read this section once, but forgot about it).

> considered to be decoupled (as much as possible) from the loss

That makes sense to me - although I think we do need a concrete default recommendation on what to do (in the absence of any better solutions, e.g. pacing or whatever, ..).

> not send a burst, but rather would slow-start, starting from 1 packet

Yes, this is nicely explained in section 7.5, and it seems to me to be a reasonable thing to do.

The recommendation to use PRR is kinda there, but IMO a bit too weak:
***
A packet marked lost by RACK SHOULD NOT be
   retransmitted until congestion control deems this appropriate (e.g.
   using [RFC6937]).
***

What about, e.g.:
***
A packet marked lost by RACK SHOULD NOT be
   retransmitted until congestion control deems this appropriate.
Specifically, Proportional Rate Reduction [RFC6937] SHOULD be
enabled when using RACK.
***

Cheers,
Michael

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 12 Oct 2017 07:20:54 +0000

> Specifically, Proportional Rate Reduction [RFC6937] SHOULD be

Thanks, Michael. That sounds good to me. CC-ing my co-authors to get
their sense.

BTW, this has been the behavior of TLP-initiated recovery in Linux TCP
since our team added it in March of 2013. (Back then Linux TCP was
using TLP + FACK + PRR; now it's with TLP + RACK + PRR.)

neal

---

From: hiren panchasara <[email-masked]>
Date: Thu, 12 Oct 2017 08:18:16 -0700

> Hi,
[snip]
> Specifically, Proportional Rate Reduction [RFC6937] SHOULD be

Yes please. I ran into the exact same problem when Neal/Yuchung pointed
me to PRR and implementing that fixed the issue of bursting out
retransmissions too fast during recovery. Even without pacing it does a
very nice job so yeah, it should be marked as 'necessary' with rack.

Cheers,
Hiren