Subject: Re: [quicwg/base-drafts] ACK generation recommendation (#3304)

From: mjoras <[email-masked]>
Date: Mon, 06 Jan 2020 10:10:38 -0800

I'll do some tests with mvfst this week or next and post numbers. We have tests for both single stream throughput and an http server.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3304#issuecomment-571246474

---

From: mirjak <[email-masked]>
Date: Tue, 07 Jan 2020 02:54:04 -0800

I think the algorithm proposed here are actually protocol independent and should not be specified in the any quic draft but in a separate draft (in tsvwg...?).

I actually think that the SHOULD we currently have is fine because it really just means if you implement this protocol and you don't know any better, we recommend you to use this value.

However, we could of course add one more sentence and explain the tradeoffs. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3304#issuecomment-571537565

---

From: mjoras <[email-masked]>
Date: Tue, 14 Jan 2020 15:19:30 -0800

Some tests, as promised, cc @janaiyengar @dtikhonov.

The first test is using an "iperf-style" single stream, single conn, single thread on client and server. The server uses BBR as the congestion controller. The server is an infinite source and the client is an infinite sink and the test runs for 60s and measures the average throughput.

The first situation is using the loopback interface on Linux, standard MTU size, and no introduced delay or loss. Changing the ACK generation interval from 10 -> 2, we observe a **20% relative decrease in throughput**.

Using the same test but with a 15ms netem delay with mild loss and the results are more dramatic. The ACK ranges expand considerably, which ends up being a significant cost for both the client and the server which results in a **50% relative decrease in throughput**.

These tests don't really reflect how most people plan to use QUIC (on the internet, not with multi-Gbps sustained transfers), but I believe they are illustrative of the costs we're dealing with. Note that the vast majority of the profiled stacks for mvfst in these tests are spent in sendmsg, recvmsg, crypto, and serializing the QUIC frames. There are still some opportunities for us to micro-optimize our implementation, but relative to the "fixed" costs most implementations are paying, it is likely only a few percentage reductions here and there. ACK handling, for example, which is very implementation-dependent, ends up being **less costly to the server** than writing ACK frames when the ACK interval is 2 versus 10.

We also have a way to test this using a real reverse proxy with synthetic traffic. This particular set up uses a **real** transatlantic backbone link with minimal loss. The link has an RTT of ~100ms, and the server is using BBR as the congestion controller. Two tests are of interest, one is generally not CPU bound for the server while the other is. The one typically not CPU bound is many clients each requesting one 1MB resource. The test which typically becomes CPU bound is many clients each requesting ten 1MB resources.

For one 1MB resource per client the ACK interval from 2 -> 10 actually **increases RPS by about 20%**. I think this is largely due to congestion controller benefits from a higher ack frequency (note that this does **does not** start out by acking every other, it starts out acking every 10).

For the ten 1MB resources per client case we see the dramatic results again, where the increased ACK frequency causes a **60-70% drop** in RPS. In this case it seems that the client is the one that's really causing the regression by having to ACK much more frequently. 

All of this is to say, I think there are dragons lurking here. We've had good success with the heuristic of ACKing every 10 combined with 1/4 or 1/8 RTT. After some thought I don't think including this (and the "after 100" optimization) as the default recommendation is problematic, as long as we have some nice language conveying the basics of the tradeoffs at play here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3304#issuecomment-574419847

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 29 Jan 2020 09:07:29 -0800

To me the problem is in two parts ... The first part is "what is the default?" - I still think we shouldn't be designing a QUIC default with significantly worse performance than TCP - simply because TCP can take advantage of an in-network device to Thin ACKs - we should think carefully about setting an appropriate default. To avoid a long "issue" we wrote a draft on this and can present results if there is interest, with a proposal to change the default in the spec:
https://datatracker.ietf.org/doc/draft-fairhurst-quic-ack-scaling/

draft-iyengar-quic-delayed-ack provides opportunities to tune the ACK policy for a server and CC algorithm.  I don't see the two drafts in competition. Both could proceed?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3304#issuecomment-579859660