Subject: [quicwg/base-drafts] kInitialRtt of 100 msec is too aggressive (#2184)

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 13 Dec 2018 22:42:54 -0800

This value will cause unnecessary timeouts for CI and 0-RTT data on many Internet paths. And pretty much always on satellite links. If we want to pick a magic number then let's pick 200 msec or 300 msec (the historical minrto values for Linux and Windows respectively).

Should we add text that implementations MAY use 2 * cached path RTT if platform support is available? Even QUIC itself could maintain such a cache per path. IMO this is the right way to make Initial RTO more aggressive. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184

---

From: Martin Thomson <[email-masked]>
Date: Thu, 13 Dec 2018 22:53:04 -0800

I don't understand why this is a problem.  What harm, aside from retransmissing the Initial packet, does this do?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447232934

---

From: MikkelFJ <[email-masked]>
Date: Fri, 14 Dec 2018 03:33:09 -0800

Summary from other issue: Denmark to US East Cost is just above 100ms min and avg. roundtrip. China is > 200 min and > 300 on average, and New Zealand worse than that. Denmark to Netherlands is 4ms.

So having a timeout of 100ms basically guarantees aggressive retransmission for anything but regional traffic. Maybe that is acceptable, but you can feed a lot of data on a fat pipe in 100ms, especially if you use 0-RTT and if you can tolerate replay attacks. And for space it might be worse because the link cost could be longer and bandwidth much more expensive.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447298946

---

From: Marten Seemann <[email-masked]>
Date: Fri, 14 Dec 2018 06:54:29 -0800

We discussed this in https://github.com/quicwg/base-drafts/issues/752 and decided to err on the side of lower latency. 
@RyanAtGoogle shared some numbers there that might be more interesting than a single RTT measurement from Denmark.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447348658

---

From: MikkelFJ <[email-masked]>
Date: Fri, 14 Dec 2018 06:59:17 -0800

Denmark happens to be my location, but it isn't entirely irrelevant with multiple data centers from Apple, Google and Facebook being constructed, and multiple new sea cables from said companies landing here. That said, no single location can be representative.

https://www.youtube.com/watch?v=iMAThVcqzuk

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447350117

---

From: ianswett <[email-masked]>
Date: Fri, 14 Dec 2018 08:51:49 -0800

The text says "Resumed connections over the same network SHOULD use the previous connection's final smoothed RTT value as the resumed connection's initial RTT."

Did you want to broaden this to include non-resumed connections where you estimate the path RTT some other way?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447384118

---

From: MikkelFJ <[email-masked]>
Date: Fri, 14 Dec 2018 09:12:31 -0800

> The text says "Resumed connections over the same network SHOULD use the previous connection's final smoothed RTT value as the resumed connection's initial RTT."

That at least resolves the 0-RTT concern.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447390192

---

From: MikkelFJ <[email-masked]>
Date: Fri, 14 Dec 2018 10:02:18 -0800

With 0-RTT in check, I think the early RTO shouldn't be viewed as a failure recovery mechanism but more like a sample mechanism for discovering uncharted territory. In that light, it could go down to 10ms as long as it is only until first contact. However, because the Initial is not small (1200+ bytes), a too small original RTO would be wasteful.

So the question becomes: what is acceptable latency, what is the likely error rate, and how much bandwidth is wasted by aggressive probing.

If, for example, some common scenarios has a 66% packet loss and RTO is 100ms, you have to wait about 200ms extra to connect, even if the RTT is small, like 4ms (say noisy WLAN).

On the other hand, if the latency is high, like 500ms and the loss is small, you would be sending 5x the necessary data to connect, so 5x1200 bytes, but if you loose a packet, connectivity starts a 600ms instead of 500ms.

You could also imagine a very conservative initial RTO like 1000ms but at the same time send out small probe packets every 10ms until a valid RTT estimate is achieved, but that has its own problems in complexity and trust.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447404494

---

From: janaiyengar <[email-masked]>
Date: Fri, 14 Dec 2018 13:24:11 -0800

We can recommend whatever we want here, implementations will play around with this number. As @martinthomson notes, the cost is limited to potential spurious retransmissions of the Initial, which isn't a high cost.

Google's deployment numbers are skewed to Google's network, but they do suggest that 100ms is reasonable.  Based on those numbers, and based on the discussion in #752, I'd argue for keeping it at 100ms.

I like @pravb and @ianswett 's idea of adding something that says this can be set based on a cached value even on non-resumed connections if the path's RTT is known by other means (cached from a previous TCP connection in the platform, for instance.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447483488

---

From: ianswett <[email-masked]>
Date: Fri, 14 Dec 2018 17:53:05 -0800

I've been wanting to experiment with an initial RTT based on other non-resumed connections for a while, but haven't gotten around to it in gQUIC.  It certainly seems like it'll be better than a fixed value, so I'm happy to encourage that in the spec.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447527220

---

From: Christian Huitema <[email-masked]>
Date: Sun, 16 Dec 2018 19:45:27 -0800

Half-way around the world is 20Km. The speed of light through optical fiber is about 100ms. Thus, halfway around the world is 100ms, and the corresponding round trip 200ms. Using 200ms by default makes sense. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447714243

---

From: Martin Thomson <[email-masked]>
Date: Sun, 16 Dec 2018 20:56:43 -0800

So we optimize for the worst case?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447722659

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 16 Dec 2018 20:57:06 -0800

I think kInitialRTT of 100ms is conservative enough (at least for Initial packets), because a client would be retransmitting at most one packet, even when the server is at the opposite side of the world.

QUIC has 0-RTT packets that allows an endpoint to transmit several number of packets before seeing a response from the server (see also #2135), because the server "probably" knows the key to decrypt the packets. Considering the fact that we allow that, I think that retransmitting an Initial packet when it has "probably" been lost is also fine.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447722703

---

From: Christian Huitema <[email-masked]>
Date: Sun, 16 Dec 2018 22:23:10 -0800

It depends how many times a client is going to retransmit. It is all a matter of tradeoffs. Suppose that we transmit 4 times at 100 ms interval; on a geo stationary satellite link the response will come after 600ms. That means all connection attempts will fail. If the client does exponential back off, it will transmit at 0, 100, 300, 700. Then the satellite connection will work most of the time. But the client will only certify a failure after 1100 ms. 

So no, we don't need to optimize for the worst case. But we need to make sure that the worst case does not always fail. If nodes want optimization, they can remember the RTT of commonly used destinations and use that instead of the default.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447735381

---

From: Martin Thomson <[email-masked]>
Date: Sun, 16 Dec 2018 22:46:25 -0800

Do we have a maximum?  I didn't think that was the subject of the discussion.  Timeouts always need to look at the tail.  Though having more packets in the pipe might allow that tail to be brought in a little more than otherwise.

As for remembering RTTs being the solution here, strengthening our reliance on mechanisms that reinforce incumbancy is the best outcome.  I certainly expect people to do that, but if that means that a site is only "fast" on return visits, that's going to disadvantage new players more.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447739531

---

From: Marten Seemann <[email-masked]>
Date: Sun, 16 Dec 2018 22:46:45 -0800

The initial crypto retransmission timeout is reiche the initial RTT, so the first retransmission would happen after 200ms, not after 100ms. Every further retransmission will be exponentially backed off.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447739586

---

From: Christian Huitema <[email-masked]>
Date: Sun, 16 Dec 2018 22:55:56 -0800

@martinthomson I don't know what the practical maximum is, I don't have access to telemetry data for that. I would pick 1 second by default, or maybe 1.2 sec for 2 geo-sat hops. I don't particularly worry about having too many packets in flight, I just want to ensure that in the worst case the node has a chance to receive the response to at least the first retransmission before giving up. So I guess if the total wait time is about 1.5 sec we can be fine. Based on what @marten-seemann said, the 100 initial RTT results in a schedule of retransmissions at 200, 600, 1400 and 3000 ms, giving up 6.2 sec after the initial attempt. This seems like plenty. A bit slow, actually.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447741398

---

From: Marten Seemann <[email-masked]>
Date: Sun, 16 Dec 2018 23:11:24 -0800

@huitema: Are you suggesting that 100ms is not aggressive enough then, especially considering Ryan's statistics in https://github.com/quicwg/base-drafts/issues/752#issuecomment-325998995?
Where does your suggestion to retransmit exactly 4 times come from?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447744436

---

From: MikkelFJ <[email-masked]>
Date: Mon, 17 Dec 2018 09:37:34 +0000 (UTC)

I'd suggest increasing slightly above 100ms because US - Europe will hover between 101 - 150 ms, unless we want to almost always retransmit the Initial packet defensively on non regional links, which kind of makes sense.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447779943

---

From: MikkelFJ <[email-masked]>
Date: Mon, 17 Dec 2018 10:06:44 +0000 (UTC)

I don't think Ryans estimates are particularly convincing except to ensure that QUIC also works well for that case. Any non-Google user couldn't care less about how many data centers Google use to distribute queries when deploying a QUIC application. You need to ensure QUIC works optimally well in the sub 40ms range, but the outliers needs to handled, as @huitema points out.

One option is to retransmit Initial packets 4 times in fixed intervals of 80 ms, or larger if bandwidth is expensive until at least one packet has been received by peer. The 4x ensures that one packet gets through event at loss rates up to 75% and in the typical cases at most one or two packets are sent. If there is no response after the 4th packet, the burst is repeated at increasingly larger intervals.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447789198

---

From: ianswett <[email-masked]>
Date: Mon, 17 Dec 2018 08:22:05 -0800

Chrome currently uses a 4 second timeout on the client if it hasn't received anything back.  That's fairly long , but 1 second is certainly to short based on what I know.  That being said, RTTs >1s are only really common on 2g, so one could use a much shorter timeout if you knew more about the network.

@mikkelfj As Marten pointed out, the current retransmission schedule is 200, 600, 1400 and 3000 ms, etc, so US/Europe connections only incur a spurious retransmit if the RTT is >200ms or there's loss.

So like Jana, I think 100ms is reasonable, but I think we should add some more text about predicting RTT based on past network measurements, even if they're not on resumed connections.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2184#issuecomment-447904643