Subject: Re: [quicwg/base-drafts] QUIC PTO is too conservative, causing a measurable regression in tail latency (#3526)

From: ianswett <[email-masked]>
Date: Tue, 12 May 2020 07:48:06 -0700

The most recent set of experiments indicate that my above suggestion of arming for max(left_edge + SRTT+4*RTTVar, right_edge + 1.5*SRTT) is a very slight improvement for some metrics, but it looks to be in the noise for most metrics.

We did find SRTT+2*RTTVar is as good or better than SRTT+4*RTTVar for all application metrics, with a very small difference in retransmit rates(probably because PTO can send new data).

I'd be very curious to see what @mjoras finds in his testing, but I'm increasingly inclined to close this with no action.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-627392539

---

From: mjoras <[email-masked]>
Date: Wed, 20 May 2020 10:31:09 -0700

I will have data on this in a few days.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-631617976

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 May 2020 15:42:02 -0700

The editors discussed this and concluded that it would be best to mark this QUICv2.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-634318637

---

From: mjoras <[email-masked]>
Date: Wed, 27 May 2020 10:33:40 -0700

Following up, I ran a test for 4 days on a major country. The results are mostly in line with what @ianswett saw. 

1. Tail response latency (p95) saw a measurable but small decrease of about 1 percent.
2. Various application-level outlier metrics (e.g. time to load images, time to load a feed, etc.) showed slight improvements that also correspond with the change.
3. There were no regressions in any other metrics.

Based on this I think the strategy of arming the first PTO based on max(left_edge + SRTT+4RTTVar, right_edge + 1.5SRTT) is indeed an improvement, but it is not huge.

Based on this I would agree with punting work on this to v2, since we plausibly will have time to come up with different strategies here that may be a larger improvement.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-634822736

---

From: ianswett <[email-masked]>
Date: Wed, 27 May 2020 12:46:16 -0700

Thanks for the data @mjoras !

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-634901504

---

From: Martin Thomson <[email-masked]>
Date: Wed, 27 May 2020 18:23:26 -0700

> I ran a test for 4 days on a major country

This probably shouldn't make me nervous, but it still does.  But I still greatly appreciate the data.

For my own peace of mind (though you will never alleviate the nervousness), can you confirm that you track both observed loss rates AND the rate of spurious retransmission?  It strikes me that there would be high correlation between improvements in latency and real loss, so the loss rates probably don't change except where the faster transmission coincides with true congestion (which I would expect to be undetectable), and the same is likely also true for spurious retransmission, but I just wanted to confirm.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-635035382

---

From: mjoras <[email-masked]>
Date: Fri, 29 May 2020 12:29:51 -0700

> This probably shouldn't make me nervous, but it still does. But I still greatly appreciate the data.

@martinthomson For what it's worth, I did extensive testing of this change on smaller scale first, e.g. A/B tests on groups of individual hosts in different locales to ensure that nothing was fundamentally problematic. But indeed, there is good reason to be nervous when experimenting with recovery on the Internet 🙂 

> can you confirm that you track both observed loss rates AND the rate of spurious retransmission? It strikes me that there would be high correlation between improvements in latency and real loss, so the loss rates probably don't change except where the faster transmission coincides with true congestion (which I would expect to be undetectable), and the same is likely also true for spurious retransmission, but I just wanted to confirm.

One thing I'll note here is that the vast majority of the newly-fired PTOs in this test (i.e. PTOs that previously would have fired later) were not retransmissions at all but rather PTOs using fresh application data. As such the retransmission rate, spurious or otherwise, was pretty much constant when compared to the control.

I think your intuition about the loss rate is totally warranted; in this test though there was not a statistically significant change in the observed loss rate, either the average or in the p95+ or p99 tail.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3526#issuecomment-636147424