Subject: [quicwg/base-drafts] Changing the Default QUIC ACK Policy (#3529)

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 18 Mar 2020 00:53:16 -0700

As currently specified, the default QUIC ACK policy models TCP’s ACK policy, but consumes significantly more return path capacity. This can impact resource usage of the return channel, can constrain forward throughput, and can impact the performance of other sessions sharing an uplink (draft-fairhurst-quic-ack-scaling). This was not discussed in Zurich - where discussion moved instead to the separate topic of draft-iyengar-quic-delayed-ack.

In many current network segments (e.g., cable networks, mobile cellular, WiFi and other radio tech), the effects of asymetry are presently mitigated by network devices RFC3449 that rely upon observing TCP headers.  This results in ossification problems, but does practically have beneit. It reduces the TCP ACK rate over the constrained link, it reduces consumption of "radio" transmission opportunities, and reduces traffic volume from ACKs. These mitigations are not possible, nor desirable for QUIC. However, the default performance of QUIC  is therefore much worse than for TCP.

In QUIC, the default ACK Ratio has to set by the endpoints implementing the protocol  because the sender and receiver have no information about the "cost" of sending an ACK using a link along an asymmetric path.  These endpoints can also choose a non-default value to enable a new CC, scale to higher rates, etc (e.g. draft-iyengar-quic-delayed-ack).

We have shown there can be significant benefit from an ACK Ratio of 1:10 and this is expected to alos have benefit for higher levels of asymmetry. The proposed change from 1:2 to 1:10 is predicated on  QUIC's design decisions: starting a connection with an Initial Window of 10 is acceptable, QUIC uses PTO as the loss detection method (a predicate to issue #3357), and QUIC has some form of pacing at the sender, at least to mitigate IW bursts. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529

---

From: martinduke <[email-masked]>
Date: Thu, 19 Mar 2020 12:37:15 -0700

Even though we've avoided straying this far from the TCP spec, I think allowing this would be good for the internet. It should probably be documented for TCP, too.

How does ack stretching of this nature work for BBR bandwidth estimation? It's harder to use the ack pacing, no?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-601378758

---

From: ianswett <[email-masked]>
Date: Sun, 22 Mar 2020 09:17:50 -0700

> As currently specified, the default QUIC ACK policy models TCP’s ACK policy, but consumes significantly more return path capacity. This can impact resource usage of the return channel, can constrai…

I don't believe QUIC ACKs are significantly larger.  In fact, I believe the ACK frame itself is typically smaller than TCP ACKs, but the savings there are used on QUIC's 16-byte auth hash and likely a non-zero-length CID.  Do you have a size analysis written up somewhere?

> In many current network segments (e.g., cable networks, mobile cellular, WiFi and other radio tech), the effects of asymetry are presently mitigated by network devices RFC3449 that rely upon observ…

I think this is the core issue.  It's not that QUIC ACKs are really consuming significantly more capacity in terms of bytes on the wire, it's that middleboxes can't decide to thin the ACKs, because they don't know which packets are ACK-only packets.

> We have shown there can be significant benefit from an ACK Ratio of 1:10 and this is expected to alos have benefit for higher levels of asymmetry. The proposed change from 1:2 to 1:10 is predicated…

Did you implement the optimization already described in the recovery draft?: "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response.  In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets."

@larseggert did and I believe found it hugely reduce the number of ACKs, sometimes too much.

As I've said before, I would really prefer not specify an approach in QUIC transport that is a heuristic like the PR attached to this issue does.  In particular, on lower bandwidth networks, 10 packets may be the entire congestion window and is too large a value.  On higher bandwidth networks, every 20 packets has shown to be slightly better than 10, at least when using BBR.  I wrote #3518 to add some editorial text about why every 2 packets is a SHOULD, not a MUST.  Chrome did not to default enable the ACK strategy #3501 recommends until BBR was launched, because there was a performance regression with Cubic.  I'd rather not assume everyone is using BBR or a BBR-like congestion controller.

@martinduke BBR does fine with stretch ACKs.  Cubic and Reno suffer in some cases, however.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-602233642

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 24 Mar 2020 01:12:44 -0700

QUIC ACKs are larger!
The smallest IPv4 QUIC ACK packet is 51 bytes. The ACK frame is 4 bytes, but is sent using a QUIC packet. For IPv4 this is: 20 (IP) + 8 (UDP) +3 (1+1+1 QUIC header ... assuming a 1 B CID) + 4 (minimum ACK frame size)+ 16 (crypto). Other types of frames may also be sent in the same packet, increasing this size. That size difference matters.

The point is that QUIC ACKs are consuming more capacity in terms of bytes on the wire,  and often also important: in more transmission opportunities at the physical layer. That makes QUIC perform more poorly than it needs to across some paths. 

The text "As an optimization, a receiver MAY process multiple packets before sending any ACK frames in response. In this case the receiver can determine whether an immediate or delayed acknowledgement should be generated after processing incoming packets." Is permissive to do something different - alas the effect of ordering of the processing depends on rate, timing, etc - The present text is not a recommendation to do something appropriate, which I think it needs to. If this were to describe that a receiver SHOULD reduce the ACK to data packet ratio to 1:10, then that would help. 

On low bandwidth links, then I agree that waiting for 10 packets would not be suitable, but then wouldn't the rule 4 ACKs/RTT still cover this case?

(I have no problems believing every an ACK 20 packets can be slightly better than 10 in some cases, but then we have chosen IW at 10 and that relates to bustiness and responsiveness of the basic spec, so I think 1:10 makes more sense as the recommendation. If the CC and path information suggest 1:20 - perhaps as the rate increases, then a sender can "tune" as per the transport parameter proposal. That I STILL assert that this is a different issue from the expected default.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-603091832

---

From: mirjak <[email-masked]>
Date: Tue, 24 Mar 2020 04:05:25 -0700

Would it help to give a recommendation like, if the congestion window is large, the ACK radio should be reduced? Not sure if we can easily give a threshold for the window but to align with the 4 ACKs per RTT rule we could say 40 or to be more conservation maybe 100?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-603174537

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 24 Mar 2020 05:28:43 -0700

If the receiver knew the cwnd... we could add that.

The current proposal we made is here (along with some of the arguments about why QUIC and TCP are slightly different): draft-fairhurst-quic-ack-scaling

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-603210051

---

From: ianswett <[email-masked]>
Date: Tue, 24 Mar 2020 05:44:50 -0700

So the min TCP ack is 40 bytes vs QUIC at 50(0 byte CIDs are common, so I'll assume that), 25% larger in the minimum case.

However, with a few SACK blocks, TCP ACKs are likely to be larger than QUIC ACKs, since SACK consumes 10 bytes for the first block and 8 for each additional block.  By the time the second block is added, there's a good chance QUIC ACKs are smaller and it's almost certainly true with 3 SACK blocks(which I think is a typical number).

So I'll go back to my previous statement that the real issue is that the network can't thin QUIC ACKs effectively.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-603216970

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 24 Mar 2020 05:52:17 -0700

May be "yes", may be "no" on a larger size after loss - since QUIC ACK ranges are often larger - but that depends on loss pattern, etc. In our case, we saw more bytes. 
 
But "yes" on me saying there is a need for QUIC to make good default decisions about the ACK rate, because QUIC becomes responsible for transport working across heterogeneous paths and we need to get good defaults rather than rely on "ACK-Thining" and other network modification.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-603220242

---

From: IngJohEricsson <[email-masked]>
Date: Sat, 28 Mar 2020 03:04:19 -0700

Anybody has a gut feeling how very low latency congestion control fares with sparse ACKs?. I have experienced that DCTCP is more picky with ACK aggregation than e.g Reno. I suspect that ACK-thinning can have the give the issue as well. A too sparse ACK spacing can also give problems. Another aspect is how e.g. TCP Prague with paced chirps work with sparse ACKs ?. To me the RTT/4 rule makes sense but I feel that the best alternative depends a  lot on both the congestion control and the source material. Large file transfers are more trivial than sources that are occasionally idle. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-605423225

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 28 Mar 2020 03:16:52 -0700

I think this isn't the correct question. This is about the default policy. 
The default policy is about what a receiver does by "default" - the current text uses a value of ACK every other packet, 1:2. TCP specified this when transmission rates were low, and TCP depended on individual ACKs to clock data (no pacing, IW=2), detect loss (no PTO), increase cwnd (per ACK packet), etc. In response, TCP causes ACK congestion and strange interactions with radio access. Performance over WiFi, DOCSIS etc suffers unless the network tries to address this. 
I don't see why QUIC, which doesn't rely on any of these, needs to set a DEFAULT of 1:2. This is literally a few lines of code difference, and we could use 1:10.
These CC's you mention may have different needs and that needs to be handled by telling the remote that your CC has these specific needs - which I think is what Jana's extension is about. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-605424655

---

From: IngJohEricsson <[email-masked]>
Date: Sun, 29 Mar 2020 23:31:03 -0700

I am kind of donkey in the middle... 
ACK thinning in networks should be discouraged. Standardization of ACK thinning mechanisms was proposed in 3GPP RAN2, this request was turned down mainly with reference to that it can harm transport protocol performance and does not work safely. Nothing stops UE (terminal and modem) vendors from implement ACK-thinning but I certainly hope that 5G UE vendors don't attempt to thin QUIC-ACKs as this is a slippery slope.   
High ACK rates is a 5G problem mainly for high band NR (=5G at e.g 28GHz). It is mainly a coverage issue as UEs have a limited output power due to practical and regulatory requirements. At the same time one need to address that end users don't only download movies today and even VoD streaming implies uplink traffic in the form of chunk requests. And then we have an increasing amount of people doing uplink streaming or upload an infinite amount of pictures of food plates and cats :-) 
Operators and 5G vendors have to deal with this and solve the problem with deployment of 5G access in various frequency bands to get the best downlink and uplink performance.
Reduced ACK rates is  of course welcome but I don't currently see a problem that the initial ACK ratio is 1:2 , to increase to 1:? as the congestion window grows, bounded by e.g an RTT/4 rule. But here I need to admit that I at least I have not studied this in detail.
Ergo.. I am not convinced that it is necessary to handwire the QUIC spec to 1:10 is needed but I also cannot say that it should not be done.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-605808722