Subject: [quicwg/base-drafts] Time based loss recommendation not sensitive to paths with high RTT variability (#2207)

From: MikkelFJ <[email-masked]>
Date: Tue, 18 Dec 2018 16:10:04 -0800

9/8 RTT for loss detection might be to aggressive in some cases, unless the intention is to retransmit defensively.

On occasion Denmark->Beijing ping runs stable at 211ms RTT in 7 out of 10 packets with the rest begin 280+ms, in other cases timing is all over the place, like below

```
ping -c10 xiaomi.cn
PING xiaomi.cn (58.83.160.156): 56 data bytes
64 bytes from 58.83.160.156: icmp_seq=0 ttl=44 time=211.440 ms
64 bytes from 58.83.160.156: icmp_seq=1 ttl=44 time=446.901 ms
64 bytes from 58.83.160.156: icmp_seq=2 ttl=44 time=368.392 ms
64 bytes from 58.83.160.156: icmp_seq=3 ttl=44 time=211.281 ms
64 bytes from 58.83.160.156: icmp_seq=4 ttl=44 time=511.985 ms
64 bytes from 58.83.160.156: icmp_seq=5 ttl=44 time=213.076 ms
64 bytes from 58.83.160.156: icmp_seq=6 ttl=44 time=320.433 ms
64 bytes from 58.83.160.156: icmp_seq=7 ttl=44 time=572.960 ms
64 bytes from 58.83.160.156: icmp_seq=8 ttl=44 time=493.316 ms
64 bytes from 58.83.160.156: icmp_seq=9 ttl=44 time=412.027 ms

--- xiaomi.cn ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 211.281/376.181/572.960/126.994 ms
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207

---

From: ianswett <[email-masked]>
Date: Tue, 18 Dec 2018 18:27:11 -0800

It's recommended as 9/8 * max(SRTT, latest_RTT).  So this means a packet sent more than 1/8 RTT prior to the current packet must be acknowledged.

I agree that RTT can be very noisy, but that tends to be from queueing/wifi/etc delays more than reordering.  If you have some data indicating reordering is substantially more common than the current draft can deal with, I'd be very interested.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448446617

---

From: MikkelFJ <[email-masked]>
Date: Wed, 19 Dec 2018 01:37:57 -0800

Regarding reordering I don't have any data I'd suspect there will be with those timing variations. I'll keep it in mind to mention, if a find any data.

My data is from Wifi, but it is no way near that variation.

Here are ping data the same 5GHz wifi source to a danish site.nearby site in Denmark. Physically short distance but more hops than I would have hoped for.

Here the variation is actually possibly larger than to China because of the short overall time and router noise (and potentially wifi interference).

 ping -c10 dtu.dk
PING dtu.dk (192.38.91.25): 56 data bytes
64 bytes from 192.38.91.25: icmp_seq=0 ttl=53 time=5.848 ms
64 bytes from 192.38.91.25: icmp_seq=1 ttl=53 time=7.265 ms
64 bytes from 192.38.91.25: icmp_seq=2 ttl=53 time=5.769 ms
64 bytes from 192.38.91.25: icmp_seq=3 ttl=53 time=6.761 ms
64 bytes from 192.38.91.25: icmp_seq=4 ttl=53 time=5.488 ms
64 bytes from 192.38.91.25: icmp_seq=5 ttl=53 time=9.453 ms
64 bytes from 192.38.91.25: icmp_seq=6 ttl=53 time=3.579 ms
64 bytes from 192.38.91.25: icmp_seq=7 ttl=53 time=5.642 ms
64 bytes from 192.38.91.25: icmp_seq=8 ttl=53 time=7.297 ms
64 bytes from 192.38.91.25: icmp_seq=9 ttl=53 time=5.114 ms

--- dtu.dk ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.579/6.222/9.453/1.497 ms

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448530911

---

From: MikkelFJ <[email-masked]>
Date: Wed, 19 Dec 2018 09:42:40 +0000 (UTC)

And here are ping times from Denmark to a New Jersey datacenter. This is a direct submarine cable and a few hundred km over land.

Here the variation is much lower, and fits the 9/8 recommendations.

ping -c10 23.226.128.10
PING 23.226.128.10 (23.226.128.10): 56 data bytes
64 bytes from 23.226.128.10: icmp_seq=0 ttl=53 time=100.651 ms
64 bytes from 23.226.128.10: icmp_seq=1 ttl=53 time=102.843 ms
64 bytes from 23.226.128.10: icmp_seq=2 ttl=53 time=102.355 ms
64 bytes from 23.226.128.10: icmp_seq=3 ttl=53 time=104.239 ms
64 bytes from 23.226.128.10: icmp_seq=4 ttl=53 time=104.736 ms
64 bytes from 23.226.128.10: icmp_seq=5 ttl=53 time=102.465 ms
64 bytes from 23.226.128.10: icmp_seq=6 ttl=53 time=103.996 ms
64 bytes from 23.226.128.10: icmp_seq=7 ttl=53 time=102.295 ms
64 bytes from 23.226.128.10: icmp_seq=8 ttl=53 time=101.644 ms
64 bytes from 23.226.128.10: icmp_seq=9 ttl=53 time=102.435 ms

--- 23.226.128.10 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 100.651/102.766/104.736/1.179 ms

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448532342

---

From: MikkelFJ <[email-masked]>
Date: Wed, 19 Dec 2018 01:59:46 -0800

@ianswett's point is that even if there is variation it probably will not change fast between two packets, rather it changes over longer time frames such is the interval between PING packets. I'm not sure this is always true since a 250ms buffering is unlikely.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448537453

---

From: Loganaden Velvindron <[email-masked]>
Date: Wed, 19 Dec 2018 02:05:39 -0800

>From .mu to .dk:

PING dtu.dk (192.38.91.25) 56(84) bytes of data.
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=1 ttl=44 time=243 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=2 ttl=44 time=277 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=3 ttl=44 time=278 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=4 ttl=44 time=280 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=5 ttl=44 time=240 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=6 ttl=44 time=284 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=7 ttl=44 time=286 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=8 ttl=44 time=288 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=9 ttl=44 time=290 ms
64 bytes from redirector.ait.dtu.dk (192.38.91.25): icmp_seq=10 ttl=44 time=293 ms

--- dtu.dk ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 240.406/276.264/293.286/17.763 ms

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448539188

---

From: Loganaden Velvindron <[email-masked]>
Date: Wed, 19 Dec 2018 05:58:35 -0800

@dtaht unders latency problems with wifi very well.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448605309

---

From: Dave Täht <[email-masked]>
Date: Wed, 19 Dec 2018 06:28:07 -0800

Heh. One of my hopes is that everybody *runs* to deploy the fq_codel for wifi code ( https://www.usenix.org/system/files/conference/atc17/atc17-hoiland-jorgensen.pdf ) and then you won't have to deal with current level of (up to 10 second) insanity on wifi in quic. But... oh wait... early tests of 5G are showing 1.6seconds of latency on some tests... great use for that 1ms mac, eh?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448614506

---

From: janaiyengar <[email-masked]>
Date: Wed, 19 Dec 2018 12:58:33 -0800

The threshold in time-based loss detection is a reordering threshold, not an absolute delay threshold. The path has to have both reordering and high delay variability... I'll grant that that is certainly possible. The experience we have so far is with this particular algorithm, which does not incorporate variance.

I'm not opposed to incorporating the variance into the delay metric, but I don't think we know what the right answer there is.  We could add a note that discusses variance and that we need more data on how to do this right.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448741572

---

From: MikkelFJ <[email-masked]>
Date: Wed, 19 Dec 2018 14:09:33 -0800

I think such a note would be good because it encourages experimentation and also adds awareness that the default might not always work as intended in all scenarios. We cannot just randomly introduce a square and/or a square root into the formula and hope for the best.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448761697

---

From: ianswett <[email-masked]>
Date: Wed, 19 Dec 2018 19:06:43 -0800

It currently says
"Implementers MAY experiment with using other reordering thresholds, including absolute thresholds, bearing in mind that a lower multiplier reduces reordering resilience and increases spurious retransmissions, and a higher multiplier increases loss detection delay."

I would be hesitant to add anything else, particularly about variance, unless someone can provide some evidence RTT var and reordering in time are correlated.  I can imagine building a network where they were correlated, but I don't think that's how existing networks work.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448842634

---

From: janaiyengar <[email-masked]>
Date: Wed, 19 Dec 2018 19:47:40 -0800

@ianswett : My suggestion is to note that variance is not included in the time-threshold measurement, and that implementers MAY experiment with it as well (in addition to the others). 

I don't think the long list of RTT measurements on this issue are making the necessary point -- high RTT variability has nothing to do with reordering. That said, it's definitely possible for a network to have high variability _and_ reordering. They don't need to be correlated, they just need to happen together.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#issuecomment-448853795

---

From: ianswett <[email-masked]>
Date: Thu, 20 Dec 2018 13:29:22 -0800

Closed #2207 via #2213.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2207#event-2039447011