Subject: Re: [quicwg/base-drafts] When the sender is using standard Reno congestion control, ack every ~2 packets (#1428)

From: Christian Huitema <[email-masked]>
Date: Sat, 01 Sep 2018 10:58:36 -0700

I commented on #1715 already. I don't think this works very well, because whatever byte or packet limit we use depend on the network bandwidth, which is going to vary during the course of the connection. IMHO, that's why specifications like "ACK at least every 1/4th of the min RTT" are better, because they do not depend on the bandwidth.

The reason for frequent ACK in TCP Reno is not congestion control but "ACK clocking". The received ACK is a signal to send new packets. Van Jacobson explains that in his original design: he basically wanted to keep the number of packet in transit constant, and ACK clocking achieves that -- modulo changes in the congestion window of course. If the sender is not using some form of rate control, the infrequent ACK will cause ACK clocking to degenerate. The ACK will open the window for a large number of packets, which will be sent in a single batch. This can cause spikes in network usage, stress buffers, etc. That's the main argument for sending ACK at east every 2 packets.

However, even if receivers send ACK every 2 packets, ACK clocking can still fail. For example, queues on the path from receiver to sender can cause ACK to bunch. This will have a cumulative effect, as bunched ACKs will cause packets to be sent in bunches, causing more bunches of ACKs, etc. So in practice senders who want to achieve high rates of transmission have to implement some form of rate control.

So my preference would be to not specify anything but a simple min RTT based limit on ACK delay.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-417876864

---

From: MikkelFJ <[email-masked]>
Date: Sat, 01 Sep 2018 11:06:56 -0700

Thanks for informative background.
If it is only for clocking, then frequent ACK's looks an awful lot like a huge hammer. We already have flow control and we could add rules for rate limitiing that does involve very frequent ACKs that can be expensive, for example IoT power use and also context switching on server dealing with many connections.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-417877310

---

From: ianswett <[email-masked]>
Date: Tue, 11 Sep 2018 15:54:13 -0700

Thanks Christian, specifying the preferred peer max ack delay as a fraction of RTT is what gQUIC does today, and works well for BBR and other rate-based congestion controllers.  However, at the moment, there's gQUIC also sends an ACK every 10 packets, but some experiments have been done to raise that to 20, and no degradation was observed.

My concern is that I never got fraction of RTT based acking to work as well as every 2 packets for TCP Cubic, but I never tried a value lower than 1/8RTT.

In general, I think this suggestion is good and easy to implement, but I'm a bit concerned about regressions vs the "ACK every 2 packets" approach with Reno or Cubic.

We could have a 'status quo' knob that if the ack delay isn't specified, just ack every two packets with the recommended max ack delay?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-420453247

---

From: Christian Huitema <[email-masked]>
Date: Tue, 11 Sep 2018 21:52:38 -0700

Ian, you probably have vastly more data that I do. When we are concerned with Reno and Cubic, is that a theoretical concern based on simulation, or a practical issue visible in your telemetry? I say that because ACK compression and ACK suppression does happen a lot to TCP "in the wild". Cable modems do it, for example. If I was shipping a TCP implementation on a major platform, I would do something about it, so I suppose that most of the big releases have.

In any case, you have to go back to the reason why implementations want fewer ACKs -- first to reduce CPU load caused by packet-sent interrupts, and second to ease potential bottlenecks on the reverse path. So there no point being dogmatic. If ACKs are spaced "enough", be it in packets or in milliseconds, you win already; going further will only improve your CPU cost a little, but it might risk a lot.

I suspect that is how gQUIC came to adopt a 10 packet cap. Or maybe a 10 ms cap as well.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-420511395

---

From: ianswett <[email-masked]>
Date: Wed, 12 Sep 2018 07:21:49 -0700

The data was from experiments in the wild looking at YouTube and Search metrics.  The results weren't awful, but there were small regressions in some metrics, which appeared to indicate Cubic wasn't achieving as much bandwidth with 1/4 RTT decimation enabled.  One possible issue with the experiments is that QUIC's Cubic implementation currently compares bytes_in_flight to CWND to determine if it's app-limited(https://cs.chromium.org/chromium/src/net/third_party/quic/core/congestion_control/tcp_cubic_sender_bytes.cc?q=cubic_sen&sq=package:chromium&g=0&l=369), and if it's app-limited it doesn't increase the CWND.  I wouldn't expect less frequent ACKs to cause that to be more likely, but if it did, it could explain the behavior.

In traces, it was clear that this experiment was further increasing gaps between ACKs vs acking every 2 packets, at least for some users.  It's very possible this is a Wifi artifact and sending more ACKs caused the Wifi access point to release ACKs a bit more often.  The other potential issue is that QUIC paces at 1.25*CWND/RTT in congestion avoidance for Cubic, because that's what the Linux kernel does.  This pretty much guarantees that a flow will be CWND limited when a new ACK arrives.  We discussed lowering this to 1*CWND/RTT, but never ran an experiment with that.

That brings up another point.  If one doesn't have pacing enabled(which I wouldn't advise, but I know pacing isn't always enabled at this point) then being ACK-clocked should smooth out traffic some.

The 10 packet cap was a random number I came up with, and there was no fundamental reasoning behind it.  I have no evidence it's valuable or necessary.

I'd like to be in a situation where multiple implementations have data to share on this, in case my results were a result of an implementation bug, but we're not there yet.  So my goal is to provide some knobs to allow experimentation, and we need to decide what those should be.

Related to this is the explicit max ack delay, which feeds into the TLP and RTO timeouts.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-420665538

---

From: ianswett <[email-masked]>
Date: Thu, 20 Sep 2018 12:24:23 -0700

The decision in New York is to ACK every two packets, except for the recvmmsg case when multiple may be received on a connection at once.

Another decision is we want to communicate explicit Max Ack Delay and stop trying to use a max filter to track Max Ack Delay for TLP and RTO.

As with TCP, any fraction of RTT approach besides the "no more than an RTT of delay" will be via experimental extension only.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-423302870

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Sep 2018 15:47:58 -0700

Closed #1428.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#event-1864363973

---

From: Martin Thomson <[email-masked]>
Date: Mon, 24 Sep 2018 15:48:00 -0700

Fixed in #1781.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1428#issuecomment-424150569