Subject: [tcpm] Review of draft-fairhurst-tcpm-newcwv-03

From: Yuchung Cheng <[email-masked]>
Date: Mon, 16 Jul 2012 15:56:04 -0700

Summary:

First of all the problem this draft is trying to solve is important:
AFAIK servers and data-centers disable slow-start after idle because
it simply hurts latency too badly. Modern TCP-smart applications all
use persistent connections to amortize handshake and slow-start
overhead and the RFC2861 standard nullifies that to certain extent.
While the applications don't want to create big burst to cause
congestion, they have no better choices (low latency vs occasional
burst losses: choose one). This draft proposes a middle road. Thanks
for starting this!

Details:
Section 1
I found the term "variable-rate" application little awkward: for
example some constant-rate (CBR) app may still go through slow-start,
overshoots, and experiences non-validated phases. But it's nit.

Section 4.1
Since the draft already requires SACK, I recommend to use RFC3517 pipe
to better measure the amount of outstanding data in the network.

Section 4.2

a) Justification of 2/3 and 1/3 is much appreciated.
b) During the validated phase when FS >= 2/3cwnd, are ACKs used to
compute new cwnd. E.g., in slow-start with cwnd=30 and FS=20, will the
next 10 ACKs increase the cwnd to 40? also is this phase check applied
to both transmitting new data and receiving acks?

Section 4.3.1
"  An application that remains in the non-validated phase for a period
   greater than five minutes is required to adjust its congestion
   control state.  At the end of the non-validated phase, the sender
   MUST update cwnd:

           cwnd = max(FlightSize*2, IW)."
Some justification to set cwnd to double the FlightSize? it seems with
2/3 rule in the non-validated phase, cwnd can grow to as big as 4/3 of
original value at the end of the non-validated phase.

btw, it flows better if 4.3.1 and 4.3.2 are swapped.

Section 4.3.2
"A sender that detects a packet-drop or receives an ECN marked packet
 MUST calculate a safe cwnd, based on the volume of acknowledged data:

        cwnd = FlightSize - R.

 Where, R is the volume of data that was reported as unacknowledged by
 the SACK information.  This follows the method proposed for Jump
 Start [[Liu07]."
so is R 0 or 1 of of an ECN marked packet?

Some justification to your approach compared to RFC5861?
i.e. why (FlightSize - R)/2 vs FlightSize/2?

After thougths:
Since this RFC attempts to replace RFC2861, maybe it can also address
this performance handicap in the introduction of RFC2861?

"We propose that the TCP sender should not increase the congestion window
   when the TCP sender has been application-limited (and therefore has
   not fully used the current congestion window).  We have explored
   these algorithms both with simulations and with experiments from an
   implementation in FreeBSD."

This happens frequently today because many application pauses are
within RTOs. For example, the acks of smaller HTTP responses after
some big one will now not used to compute cwnd. They are used for RTO
estimation only.

I think we need a general solution to cover when cwnd > pipe. So far
we have laminar and your draft. But a unified solution would be great
but we are making progress :)

Thanks.

---

From: Joe Touch <[email-masked]>
Date: Mon, 16 Jul 2012 16:10:18 -0700

On 7/16/2012 3:56 PM, Yuchung Cheng wrote:
> it simply hurts latency too badly.

If they're mostly doing HTTP, it shouldn't matter at all. HTTP's 
transaction pattern defeats slow-start after idle anyway.

Joe

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 18 Jul 2012 19:55:14 +0200

Hi Gorry,

I believe this draft addresses a very important problem which exists and need 
to be solved. But I'm not too sure about the solution that is proposed. But I 
also don't know what the right/perfect solution would be.

My biggest concern is regarding potential bursts you might introduce. Maybe 
I'm too conservative but I believe whatever you gone chance in TCP, one thing 
to avoid is the chance to cause large bursts of packets. Recently, it was 
actually recognized that the block sending behavior of youTube (eventough it 
is using normal standard-conform TCP) leads to bursts and large loss 
probabilities. Actually that is a problem that exists in today's TCP and 
needs to be addressed too.

So I guess what I propose is to limit the burst size. This is not completely 
though through yet but you could allow some slow start like behavior in CA if 
the flow has been application limited. So maybe your mechanism could be 
extended to allow only bursts of 2 (or 3) packets. That would mean that you 
would be able to increase your sending rate form 1*flightsize to 2*(or 
3*)flightsize in one RTT if the flow was application limited before. Don't 
you think that might be already sufficient? 
In case of idle times this would (unfortunately) lead back to normal slow 
start behavior but the IW10 draft also specifies a restart window of 10. This 
might help the problem already a lot...?
Btw. in Linux (and potentially in some RFC?) there is already a maximum burst 
size of 3 implemented but this is only used to limited the congestion window 
to flightsize + maxburstsize.

I will send some more detailed comments on the draft tomorrow (don't have my 
notes with me at the moment). But one more point are the values you have 
chosen for the several thresholds. I know there is a whole section reasoning 
why 5 minutes have been chosen but that didn't really convince me. 
One more though on my proposal above (which is still not though through): If 
you would track the max cwnd since the last congestion notification event and 
use this as a threshold upto which the kind of slow-start-behavior in CA (as 
explained above) is allow, I don't think that there is any time limit needed 
at all...

I guess this whole idea leads to something similar as proposed by TCP Laminar. 
As Yuchung mentioned those two proposals should be regarded together.

Some more detailed comments tomorrow...

Mirja

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 19 Jul 2012 12:01:37 +0200

Hi Gorry,

here as promised the more detailed comments:

- section 4: Please add a reference to the later chapter that the value of 5 
minutes will be further discussed.

- section 4.2: Please explain where the 2/3*cwnd comes from. I would have 
though validated phase is when flightsize >= cwnd or maybe >= cwnd - 
maxburstsize (as in today's implementation the cwnd will be limited to 
flightsize+maxburstsize anyway)

- section 4.3: Should the case that the 5 minutes are over here be mentioned 
as well...?

- section 4.3: in 2. bulletpoint the reference is wrong; should be to section 
4.3.2.

- section 4.3: I would merge the last paragraph into there first bulletpoint 
as I was asking me exactly that question when reading the first bulletpoint

- section 4.3.1: "by setting a burst size limit, using a pacing algorithm, or 
some other method." -> I think there should be one (or more) mechanisms 
specified here as part of the algorithm

- section 4.3.1: "cwnd = max(FlightSize*2, IW)" -> Why flightsize*2? This can 
still be a very large not validated window. I would go for 
flightsize+maxburstsize (at least have static offset here). 

- section 4.3.1: "cwnd = max(FlightSize*2, IW)" -> If you keep this, it should 
be  "cwnd = min(max(FlightSize*2, IW),cwnd)"

- section 4.3.1: "ssthresh = max(ssthresh, 3*cwnd/4)" -> Is this the new or 
the old cwnd? Why are you taking a different value than usually?

- section 4.3.2: Why is the reaction to congestion different than what should 
be done when leaving the nonvalidated phase?

- section 4.3.2: "cwnd = ((FlightSize - R)/2)" -> I wouldn't wanne specify the 
concrete congestion response in this document. There might be congestion 
control algorithms like DCTCP that so something different than halving (at 
least for ECN). The document should just say something like "perform the 
normal congestion response specified by the used congestion control algorithm 
after the adaption of the congestion window". You might even think about 
having two window variables here. Will the congestion control only controls 
the cwnd, your second window, which than determines the allowed packets in 
flight, might be larger. Moreover, you have to make sure that the congestion 
control will not raise the cwnd anyfurther while being application-limited!

- section 4.4: I would make this an own section 5

- section 4.4: Are there references available on what the "idle intervals of 
common applications" and "period for which the capacity of an Internet path 
may commonly be regarded as stable" are?

- section 4.4: 3. paragraph: Maybe be even more strict and say something 
like "If rapid changes in the path characteristics are detected, the new 
method SHOULD not be used anymore". Maybe even require to monitor the RTT's 
and give a threshold when to disable this method.

- section 4.4: Also refer the respective TCP mechanisms that use a 5 minutes 
timer.

Mirja

On Wednesday 18 July 2012 19:55:14 Mirja Kühlewind wrote:
> Hi Gorry,
>
> I believe this draft addresses a very important problem which exists and
> need to be solved. But I'm not too sure about the solution that is
> proposed. But I also don't know what the right/perfect solution would be.
>
> I'm too conservative but I believe whatever you gone chance in TCP, one
> thing to avoid is the chance to cause large bursts of packets. Recently, it
> was actually recognized that the block sending behavior of youTube
> (eventough it is using normal standard-conform TCP) leads to bursts and
> large loss probabilities. Actually that is a problem that exists in today's
> TCP and needs to be addressed too.
>
> So I guess what I propose is to limit the burst size. This is not
> completely though through yet but you could allow some slow start like
> behavior in CA if the flow has been application limited. So maybe your
> mechanism could be extended to allow only bursts of 2 (or 3) packets. That
> would mean that you would be able to increase your sending rate form
> 1*flightsize to 2*(or 3*)flightsize in one RTT if the flow was application
> limited before. Don't you think that might be already sufficient?
> start behavior but the IW10 draft also specifies a restart window of 10.
> This might help the problem already a lot...?
> Btw. in Linux (and potentially in some RFC?) there is already a maximum
> burst size of 3 implemented but this is only used to limited the congestion
> window to flightsize + maxburstsize.
>
> I will send some more detailed comments on the draft tomorrow (don't have
> my notes with me at the moment). But one more point are the values you have
> chosen for the several thresholds. I know there is a whole section
> reasoning why 5 minutes have been chosen but that didn't really convince
> me. One more though on my proposal above (which is still not though
> through): If you would track the max cwnd since the last congestion
> notification event and use this as a threshold upto which the kind of
> slow-start-behavior in CA (as explained above) is allow, I don't think that
> there is any time limit needed at all...
>
> I guess this whole idea leads to something similar as proposed by TCP
> Laminar. As Yuchung mentioned those two proposals should be regarded
> together.
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Joe Touch <[email-masked]>
Date: Thu, 19 Jul 2012 08:05:07 -0700

> transaction pattern defeats slow-start after idle anyway.

PS - we did try to deal with this issue a decade ago, but the effort was 
not take up by the WG:

http://www.isi.edu/touch/pubs/draft-hughes-restart-00.txt

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 19 Jul 2012 10:26:21 -0700

>> transaction pattern defeats slow-start after idle anyway.
Why not if the interval between two HTTP responses are large, say two minutes?
Your draft below suggested receiving HTTP requests have some effects but
I didn't see that in RFC 2861 (and I couldn't find the mailing list discussion
cited either).

Could you provide an example? Thanks.

>>
>> Joe
>
>
> PS - we did try to deal with this issue a decade ago, but the effort was not
> take up by the WG:
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 19 Jul 2012 10:37:36 -0700

> I didn't see that in RFC 2861 (and I couldn't find the mailing list discussion

Let's say the connection goes idle for 2 minutes. TCP doesn't do 
anything when a connection goes idle; it only does things when something 
is sent (in general).

The most likely reason the server would have something new to send is 
that the client sends a new request. The client request is short enough 
to fit in 1-2 segments, so shouldn't be an issue (and Nagle should be 
off, since it's interactive with multibyte exchanges).

The server receives the request, and then sends a response. The server 
has a response that's typically larger - thus the burst. The server 
decides "how long has it been since I *received* anything? Since that 
time is very short, there's no slow-start restart, and the burst goes out.

The only time the server would timeout and do slowstart restart would be 
if it decides to send something a while after anything has been received 
- that might occur for auto-refresh pages, but not much else.

The issue is explained in detail here: J. Heidemann, K. Obraczka, J. 
Touch, “Modeling the Performance of HTTP Over Several Transport 
Protocols,” IEEE/ACM Transactions on Networking, V5, N5, Oct. 1997, 
pp.616-630.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 20 Jul 2012 07:37:47 -0700

On Thu, Jul 19, 2012 at 10:37 AM, Joe Touch <[email-masked]> wrote:
> Let's say the connection goes idle for 2 minutes. TCP doesn't do anything
> when a connection goes idle; it only does things when something is sent (in
> general).
>
> The most likely reason the server would have something new to send is that
> the client sends a new request. The client request is short enough to fit in
> 1-2 segments, so shouldn't be an issue (and Nagle should be off, since it's
> interactive with multibyte exchanges).
>
> The server receives the request, and then sends a response. The server has a
> response that's typically larger - thus the burst. The server decides "how
> long has it been since I *received* anything? Since that time is very short,

but RFC 2861 does not use the most recently received timing? not in their
main algorithm (section 3.2) at least.

> there's no slow-start restart, and the burst goes out.
>
> The only time the server would timeout and do slowstart restart would be if
> it decides to send something a while after anything has been received - that
> might occur for auto-refresh pages, but not much else.
>
> The issue is explained in detail here: J. Heidemann, K. Obraczka, J. Touch,
> “Modeling the Performance of HTTP Over Several Transport Protocols,”
> IEEE/ACM Transactions on Networking, V5, N5, Oct. 1997, pp.616-630.

But why does receiving something recently justify the sending cwnd is safe to
 not cause burst-induced losses? Sorry but I can't find the explanation in
the paper.

---

From: Joe Touch <[email-masked]>
Date: Fri, 20 Jul 2012 08:25:31 -0700

On 7/20/2012 7:37 AM, Yuchung Cheng wrote:
...
> main algorithm (section 3.2) at least.

I'm referring to the algorithm  as per Standard TCP [RFC5681] which, 
resets CWND to the restart window after the app goes idle (as described 
in this doc). However, RFC  2681 failed to note that the restart 
algorithm was widely deployed at the time, and was based on erroneous 
logic in a footnote of an unpublished extension of Van Jacobson's 1988 
congestion control paper (ax explained in detail in 
draft-hughes-restart-00.txt).

>   not cause burst-induced losses? Sorry but I can't find the explanation in

It doesn't justify it - it was an incorrect conclusion. The goal is 
"restart if you haven't SENT in a while", but Van's footnote said that 
most implementations at that time had a 'time since last received' 
timer, but not a 'time since last sent', and that because TCP is 
symmetric you can use the receive timer instead of the send one. That 
logic was false, and persistent HTTP turned out to have exactly that 
pattern.

However, the mechanism proposed in this doc still allows most HTTP 
exchanges to burst an entire window - because it would be very likely 
that users will click on URLs within a received page within the 5 minute 
timeout.

I.e., I don't think section 4 of this doc is appropriate (it still 
amounts to a send timer), and still favor the "burst-or-lose" style 
approach as recommended in draft-hughes-restart-00.txt.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 20 Jul 2012 14:43:02 -0700

On Fri, Jul 20, 2012 at 8:25 AM, Joe Touch <[email-masked]> wrote:
>>> has a
> I'm referring to the algorithm  as per Standard TCP [RFC5681] which, resets
> CWND to the restart window after the app goes idle (as described in this
> doc). However, RFC  2681 failed to note that the restart algorithm was
> widely deployed at the time, and was based on erroneous logic in a footnote
> of an unpublished extension of Van Jacobson's 1988 congestion control paper
> (ax explained in detail in draft-hughes-restart-00.txt).
>
>
>>> if
>>> Touch,
> It doesn't justify it - it was an incorrect conclusion. The goal is "restart
> if you haven't SENT in a while", but Van's footnote said that most
> implementations at that time had a 'time since last received' timer, but not
> a 'time since last sent', and that because TCP is symmetric you can use the
> receive timer instead of the send one. That logic was false, and persistent
> HTTP turned out to have exactly that pattern.
>
> However, the mechanism proposed in this doc still allows most HTTP exchanges
> to burst an entire window - because it would be very likely that users will
> click on URLs within a received page within the 5 minute timeout.
>
> I.e., I don't think section 4 of this doc is appropriate (it still amounts
> to a send timer), and still favor the "burst-or-lose" style approach as
> recommended in draft-hughes-restart-00.txt.
I enjoy reading this draft. Very nice summary of all the mechanisms proposed!
I also like burst-or-lose due to its simplicity (and is interested in
rate-pacing as well).

In burst-or-lose, do the acks of the N packets increment the cwnd if
FlightSize < cwnd?

I'd recommend newcwv draft to cite and discuss hugest-restart draft.

---

From: Joe Touch <[email-masked]>
Date: Fri, 27 Jul 2012 15:43:35 -0700

Hi,

> In burst-or-lose, do the acks of the N packets increment the cwnd if

Not sure - it's been a long time since I cared about this ;-) The doc 
should explain what you need to figure it out...

Joe