Subject: Re: [quicwg/base-drafts] Changing the Default QUIC ACK Policy (#3529)

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 01 May 2020 08:02:03 -0700

These results/article on computational efficiency are really 
interesting. Please see below.

On 01/05/2020 05:40, Kazuho Oku wrote:
> Hi, Gorry
>
> Thank you for sharing your experiments. I'm delighted to see results 
> showing that use of ack10 gives us comparable throughput to ack2.
>
> Recently, I have been looking into how ack-frequency affects the 
> computational efficiency of the server [1]. Using the (almost) same 
> setup, I have today run a benchmark comparing three ACK strategies: 
> ack2, ack10, ack 1/8cwnd.
>
> ack2: 334Mbps
> ack10: 414Mbps
> ack 1/8cwnd: 487Mbps
>
> As can be seen, ack10 is better than ack2, but lags behind ack1/8cwnd. 
> That's because there are still frequent ACKs. They take considerable 
> amount of CPU cycles.
>
This is useful, we see here there's a clear benefit here in using a 
larger ACK Ratio also in terms of server load.  We can see already the 
benefit at 1:10 and GSO helps as expected.
> If we want to see QUIC being as lightweight as TLS over TCP, it is 
> critically important to reduce ACK frequency as much as possible. This 
> is because the cost of processing ACK in QUIC is much more expensive 
> than TCP, due QUIC being a userspace transport.
>
We think that this is also really true. QUIC ACKs are also larger than 
TCP ACKs (see https://erg.abdn.ac.uk/users/gorry/ietf/QUIC/).
> Based on this, if we are to change what we have in the base drafts, 
> I'd much prefer adopting what we have in the proposed delayed ack 
> extension [2], rather than just changing the default ack policy from 
> ack2 to ack10. By adopting what we have in the delayed ack extension, 
> we can expect better performance, and the endpoints would have the 
> freedom to change the behavior when the default ack policy is deemed 
> insufficient (regardless of the default being ack2 or ack10).
>
> [1] 
> https://www.fastly.com/blog/measuring-quic-vs-tcp-computational-efficiency
> [2] https://tools.ietf.org/html/draft-iyengar-quic-delayed-ack-00
>
I don't think the two approaches conflict.

The difference between the two approaches is that the small change to 
the base specification to use an ACK Ratio 1:10 ensures the default 
return path is at least as good as TCP for common traffic (even with 
common TCP ACK Thinning). Without this change, the end user could 
experience significant differences in performance between TCP and 
between different QUIC versions over asymmetric paths. 

Whereas to me, the transport parameter extension looks like an exciting 
way to enable many more things, but there are also many ways this could 
usefully evolve based on CC, offload, pacing etec and I think this will 
take much more to work-out what is needed, safe and best ... then to 
reach consensus. We still also need to specify a default ACK Ratio in 
this case as well.

Best wishes,

Gorry 
> 2020年4月28日(火) 20:37 Gorry Fairhurst <[email-masked] 
> <mailto:[email-masked]>>:
> >
> > On 27/04/2020 17:19, Lars Eggert wrote:
> >
> > Folks who care about a change here need to prioritize progressing 
> this issue. We're close to being able to last-call the specs, and 
> would like this resolved in time..
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub, or unsubscribe.
> >
> > We’ve completed a round of updated analysis of the proposed change 
> to the QUIC spec, using quicly (changing to a 1:10 policy), and 
> chromium (which currently does 1:10 by default). Our hope is that this 
> change to the spec,  to actually make standard end-to-end QUIC work 
> better than TCP over asymmetric paths!
> >
> > This is a short slide deck about the performance of the proposed change:
> > https://erg.abdn.ac.uk/users/gorry/ietf/QUIC/QUIC-ack10-24-april-00.pdf
> >
> > There have been a number of other change made by the QUIC ID editors 
> relating to ACK processing, so we wanted to be sure these results were 
> up to date. (These results do not consider additional ACKs after 
> re-ordering - reflecting the latest text removing that.)
> >
> > We also really hope this will motivate a change the practice of 
> forcing of QUIC fallback to TCP (with PEP/Thining) over highly 
> asymmetric paths. People may also be interested in work for L4S to 
> evaluate the impact of ACK Thining for TCP-Prague.
> >
> > Please do provide feedback, or any questions on this - as Lars 
> notes, we think the time is running out for considering new changes, 
> even relatively simple ones like this.
> >
> > Gorry
> >
> > P.S. First email: 
> https://mailarchive.ietf.org/arch/msg/quic/NZBbuY88v0lrVspQE2A5br83asA/
>
>
>
> --
> Kazuho Oku

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-622423228

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 02 May 2020 09:50:19 -0700

I'm greatly in favour of draft-iyengar-quic-delayed-ack-00 (BTW, where would you prefer me to give review comments about that?)

I am much less keen on any default delayed ACK policy, whether 1:2 as now in quic-transport, or the 2-10/100 numbers proposed in draft-fairhurst-quic-ack-scaling-02

My concerns would melt away if the default policy was explicitly /implementation-dependent/ and the draft instead gave /guidance/ to implementers.
* The draft could still give the same 2-10/100 policy, but as /guidance/ for the current public Internet. 
* The draft should also state the conditions under which this guidance was determined (e.g. Internet paths with BDP between x and y, or whatever).

What's the difference? Stating a default policy implies this is a matter of interoperability. It's not. If we change to giving guidance, I'm sure most implementers will just use the guidance. However, implementers will be free to evolve their default as they gain experience. 

This reduces the possibility that other behaviours will evolve around any default like 2-10/100. For instance, network monitoring using it as a signature. Or congestion controls optimizing around it. In turn, this will reduce the chance of ossification around the first numbers we thought of. 

Senders can always use the ACK frequency frame to ask a receiver to change from its default. Then receivers will gradually learn what senders prefer them to do, and change their default accordingly. IMO, a continually improving default is important for short flows, so senders can avoid requesting a different ACK frequency right from the start.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-622982264

---

From: IngJohEricsson <[email-masked]>
Date: Sun, 03 May 2020 23:49:27 -0700

Hi

Need to admit that I have not had too much time to follow the QUIC work lately. 

I too believe that draft-iyengar-quic-delayed-ack-00 is a good long term solution that allows endpoint to set a desired ACK frequency that is not more than necessary. As experiments show there are good incentives to keep the ACK frequency as low as possible in terms of client/server load and asymmetric links limitations. An ACK-FREQUENCY frame  will allow to find the proper balance between the requirements given by the congestion control, the traffic type and the above given limitations.

I am not too concerned if it takes an extra development/standards cycle to have that in place in the standard and in running code as long as the QUIC community shows other SDOs that the challenge with QUIC ACKs is being addressed. 

/Ingemar

From: Bob Briscoe <[email-masked]> 
Sent: den 2 maj 2020 18:50
To: quicwg/base-drafts <[email-masked]>
Cc: Ingemar Johansson S <[email-masked]>; Comment <[email-masked]>
Subject: Re: [quicwg/base-drafts] Changing the Default QUIC ACK Policy (#3529)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <https://protect2.fireeye.com/v1/url?k=e0af5b33-be0fe15d-e0af1ba8-86b1886cfa64-01ac033f188a7bed&q=1&e=8abe6920-7020-473e-a5d8-baa4399162a3&u=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fissues%2F3529%23issuecomment-622982264> , or unsubscribe <https://protect2.fireeye.com/v1/url?k=e5ce6fc1-bb6ed5af-e5ce2f5a-86b1886cfa64-3428571a7702a6a4&q=1&e=8abe6920-7020-473e-a5d8-baa4399162a3&u=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACRZ2GBG4MMKB4D2RGHJF53RPRFMBANCNFSM4LOJ4RQA> .  <https://github.com/notifications/beacon/ACRZ2GDOHHO3WOKEELVMNPTRPRFMBA5CNFSM4LOJ4RQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEUQ7I6A.gif> 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-623289368

---

From: Lars Eggert <[email-masked]>
Date: Mon, 04 May 2020 06:28:01 -0700

@gorryfair @IngJohEricsson could you please trim the quoted text when you reply by email?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-623463301

---

From: Lars Eggert <[email-masked]>
Date: Mon, 04 May 2020 06:31:52 -0700

(I trimmed the comments, but forwarding inline replies to email messages doesn't work well.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-623465406

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 25 May 2020 22:39:03 -0700

Given that this is an area where there is still obviously experimentation and refinement, @bbriscoe's comment seems on point; baking a specific algorithm as requirements into the specifications doesn't seem like a good idea (although putting constraints on endpoints might make sense).

Can we focus on general guidance rather than documenting a specific algorithm? What would that PR look like?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-633817331

---

From: Lars Eggert <[email-masked]>
Date: Tue, 26 May 2020 23:48:30 -0700

+1 to @mnot's suggestion. If we want to make changes for v, they need to happen now, and they must be in a form that can let us get consensus.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-634464079

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 May 2020 02:31:47 -0700

To be clear, when I said that having a default policy is not a matter of 
interoperability, that's only strictly true if QUIC supports something 
like draft-iyengar-quic-delayed-ack for the sender to ask the receiver 
to use a certain policy.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-634544299

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 27 May 2020 20:07:01 -0700

I've proposed more context and text around our current guidance in the draft, see #3706.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635068015

---

From: IngJohEricsson <[email-masked]>
Date: Thu, 28 May 2020 04:26:16 -0700

I believe that the text around asymmetric links look fine, not really necessary to write up much more. 
What can be good to know that for the 5G case with limited UL (uplink = modem->base station transmission) , the pain that a flow with an unnecessary high ACK rate causes is on the flow itself or other flows sharing the same access.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635281465

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 28 May 2020 07:20:05 -0700

There are some different points of pain here, the default and the need 
to adapt to the endpoint:

An ACK Ratio (AR 1:1): is the finest granuality of feedback. A QUIC AR 
of 1:1 would consume return path capacity proportional to about 6% of 
the forward traffic (assuming no loss, etc). This is more than TCP 
(about 3%).

An ACK Ratio (AR 1:2): For a return path bottleneck, QUIC with an AR of 
1:2 will still have more impact on sharing flows than TCP would with the 
same AR 1:2 (because QUIC ACKs are larger, and because networks do not 
currently thin QUIC ACKs).

So QUIC needs to choose the default  - is it 1:2, 1:4, 1:8, 1:10?

A QUIC AR 1:10 results in an ACK traffic volume of about 1% of the 
forward traffic. To me, that's a small proportion, which is why I argued 
that 1:10 is a good recommendation. The default is important when: the 
endpoints have no knowledge of the asymetry or the impact they have on a 
link's layer 2 loading, or the endpoints need to reduce the collateral 
damage to other IP flows sharing the return path capacity.

Negotiation: The AR can be driven by the application or endpoint - 
e.g.,  a higher rate connection may well have a benefit from an AR of 
1:10, 1:20, 1:100,  etc - but such ARs are seldom chosen to reduce the 
impact on sharing a return bottleneck, more likely they are chosen to 
benefit the performance of an application or endpoint. That 
consideration is different, and likely depends on the application or 
ednpoint - which to me is where the flexibility offered by a transport 
parameter is so important. draft-iyengar-quic-delayed-ack will still 
need guidance on the "default".

I argued that the default policy impacts the congestion behaviour when 
there is appreciable asymmetry this causes at layer 2 or the collateral 
damage (congestion impact on sharing flows). Sure, the choice of 1% is 
just a number I said was reasonable, but I've seldom encountered L2 
technologies where less than 1% of forward traffic would be 
significantly better.

Gorry

On 27/05/2020 10:31, Bob Briscoe wrote:
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
> <https://github.com/quicwg/base-drafts/issues/3529#issuecomment-634544299>, 
> or unsubscribe 
> <https://github.com/notifications/unsubscribe-auth/ABYLLERFGWEHGO2FSSOAQFDRTTMXTANCNFSM4LOJ4RQA>.
>
-- 
G. Fairhurst, School of Engineering

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635379450

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 28 May 2020 23:41:27 -0700

@gorryfair : I understand your position. While I agree that ack thinning happens with TCP, it is not what I expect to find on a common network path. And as far as I know, the endpoints still implement the RFC 5681 recommendation of acking every other packet.

What you are arguing is that _with ack thinners in the network_ TCP's behavior on _those_ network paths is different. However, I do not believe we understand the performance effects of these middleboxes, that is, how they might reduce connection throughput when the asymmetry is not a problem. Importantly,  I don't think we should be specifying as default what some middleboxes might be doing without fully understanding the consequences of doing this. Please bear in mind that Chrome's ack policy of 1:10 works with BBR, not with Cubic or Reno. 

I do not believe there is a magic number at the moment, and as @kazuho noted, even 1:10 is not a small enough ratio under some conditions. Given this, and given that we know 1:2 is used by TCP endpoints, it makes the most sense to specify that as the suggested guidance. The new text proposed in #3706 describes the tradeoff involved, and the rationale for using 1:2. I am strongly against doing anything substantially different without _overwhelming information_.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635790986

---

From: Lars Eggert <[email-masked]>
Date: Fri, 29 May 2020 00:41:24 -0700

Agree with @janaiyengar. 1:2 seems a reasonable default, and there seems to be considerable interest in draft-iyengar-quic-delayed-ack, which has a better proposal for stacks that care.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635815718

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 29 May 2020 02:40:43 -0700

I think the new ACK text is improving the specification, but I'd like to 
be sure we have thought about this decision. Let me try to carefully 
respond on-list here, to see where we agree/disagree:

On 29/05/2020 07:41, Jana Iyengar wrote:
>
> @gorryfair <https://github.com/gorryfair> : I understand your 
> position. While I agree that ack thinning happens with TCP, it is not 
> what I expect to find on a common network path. And as far as I know, 
> the endpoints still implement the RFC 5681 recommendation of acking 
> every other packet.
>
I disagree, I think we shouldn't be perpetuating a myth that a sender 
receives a TCP ACK for every other packet. Even in the 90's 
implementations TCP stacks often generated stretch-ACKs for 3 segments. 
Since then, limited byte counting was introduced and significantly 
improved the sender's response to stretch ACKs, and that has been good 
for everyone.

Senders still rely on byte-counting in TCP to handle Stretch-ACKs, and 
this need is not decreasing: new network cards reduce per-packet receive 
processing using Large Receive Offload (LRO) or Generic Receiver Offload 
(GRO).
>
> What you are arguing is that /with ack thinners in the network/ TCP's 
> behavior on /those/ network paths is different.
>
I am saying many TCP senders actually do today see stretch-ACKs. There 
are papers that have shown significant presence of stretch ACKs, e.g., 
(H. Ding and M. Rabinovich. TCP stretch acknowledgements and timestamps: 
findings and implications for passive RTT measurement. ACM SIGCOMM CCR, 
45(3):20–27, 2015.). A TCP Stretch-ACK of 3 or 4 was common in their 
datasets (about 10% of cases). In some networks, the proportion of 
Stretch-ACKs  will be much much higher, since ACK Thining is now widely 
deployed in WiFi drivers as well as cable, satellite and other access 
technologies - often reducing ACK size/rate by a factor of two.

> However, I do not believe we understand the performance effects of 
> these middleboxes, that is, how they might reduce connection 
> throughput when the asymmetry is not a problem.
>
I am intrigued and would like to know more, in case I missed something? 
Typical algorithms I am aware of track a queue at the "bottleneck" and 
then make a decision based on upon the contents of the queue. If there 
is no asymmtery, there isn't normally a queue and the algorithm won't 
hunt to discard an "ACK".

> Importantly, I don't think we should be specifying as default what 
> some middleboxes might be doing without fully understanding the 
> consequences of doing this.
>
But, maybe, there is some truth in being wary: Encrypted flows can also 
build return path queues (as does QUIC traffic). Without an ability to 
interpret the transport data (which nobody, including me, wants for 
QUIC), a bottleneck router is forced to either use another rule to drop 
packets from the queue (such as the smallest packets) or to allow a 
queue to grow, limiting forward direction throughput.

I would say neither of these outcomes are great because the router is 
trying to fix the transport's asymetry. However, if QUIC causes this 
problem, I suggest some method will proliferate as QUIC traffic increases.

> Please bear in mind that Chrome's ack policy of 1:10 works with BBR, 
> not with Cubic or Reno.
>
I didn't find something in QUIC spec. that was a major concern. It 
looked to me like QUIC had learned from TCP how to handle stretch-ACKs.

After that, we changed quicly to use 1:10 with Reno, and things worked 
well. Didn't @kazuho <https://github.com/kazuho> also use Reno?

> I do not believe there is a magic number at the moment,
>
Sure, the deisgn of TCP recognised that AR 1:1 generated too much 
traffic, AR 1:2 for TCP resulted in an Forward:Return traffic ratio of 
~1.5%, and QUIC increases this  ~3%.

However, as network link speeds increased, this proved too low for many 
network links and so, TCP ACK thinning came into being. This often 
reduces the TCP AR to around 1:4 (<1%). If  QUIC were to use an AR 1:10 
it would be ~0.5%, of course if QUIC specified a default AR 1:4, it 
would also help in many cases.

> and as @kazuho <https://github.com/kazuho> noted, even 1:10 is not a 
> small enough ratio under some conditions.
>
I totally agree, an endpoint might wish to change the AR. However, I 
don't see many **paths** where subnetwork link asymmetry drives that use 
case (spending ~10% of transmission bursts on ~1% of traffic seems like 
something that most links will be designed/dimensioned to work with). On 
the other hand, endpoint application/stack considerations (such as a 
different CC, e.g. the BBR case you note) will likely benefit from other 
ratios/behaviours. I agree this motivates your ID, and that a transport 
parameter can be of great value to synchronise the endpoint behaviours.

However, that was not my comment: The endpoints know little-to-nothing 
about layer 2 congestion... and if we wish to discourage mitigations 
such as thinning or small packet discard, which I really would like to 
disincentivise), then we shouldn't make the QUIC default worse than TCP!

> Given this, and given that we know 1:2 is used by TCP endpoints, it 
> makes the most sense to specify that as the suggested guidance.
>
That is a myth wrt to the sender, which is why I would like this 
discussed. Even in 2010, RFC 5690 seemed to ignore the presence of ACK 
Thinning, let's not do this again.
>
> The new text proposed in #3706 
> <https://github.com/quicwg/base-drafts/pull/3706> describes the 
> tradeoff involved, and the rationale for using 1:2. I am strongly 
> against doing anything substantially different without /overwhelming 
> information/.
>
> —
> <https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635790986>, 
> or unsubscribe 
> <https://github.com/notifications/unsubscribe-auth/ABYLLERPAQNQCY5NVBUIDRLRT5KI3ANCNFSM4LOJ4RQA>.
>
What do we agree upon?

Gorry

P.S. I'll certainly review #3706 
<https://github.com/quicwg/base-drafts/pull/3706> and will do this 
without bias, whatever the outcome is.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-635878980

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 31 May 2020 19:53:15 -0700

@gorryfair 
>> Please bear in mind that Chrome's ack policy of 1:10 works with BBR, not with Cubic or Reno.
> 
> I didn't find something in QUIC spec. that was a major concern. It looked to me like QUIC had learned from TCP how to handle stretch-ACKs.
> 
> After that, we changed quicly to use 1:10 with Reno, and things worked well. Didn't @kazuho <https://github.com/kazuho> also use Reno?

Just to clarify, in our experiment, we reduced the ACK frequency from the default (2) to `max(1/8 cwnd, 3 MTU)` after leaving slow start, then updated it every 4 PTO. I _think_ that would work fine on a high-bandwidth, stable network. But I am not sure if it would work as well as ack2 on all network conditions.

Anyways, I'd explain our experiment as an optimization of ack 2, as it was reduction of ack rate after leaving slow start. I think that that was a sensible choice, because IIUC having frequent acks during slow start is essential.

Assuming that we'd prefer using ack 2 during slow start, the question is if we need to hard-code in the base draft how to _optimize_ after leaving slow start. While I agree that continuing to use ack 2 after slow start is concerning for certain networks, I am reluctant to spending time on looking for such a fixed set of values across diverse networks (including much slower links or lossy ones).

Rather, my preference goes to shipping the base-draft as-is, then moving on to finalize the ack-frequency frame. I think we can assume most if not all of the popular clients to support the ack-frequency frame. Then, it becomes less of a question what the default in the base-draft is.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3529#issuecomment-636585635