Subject: [quicwg/base-drafts] Idle timeout interaction with RTO (#1429)

From: Subodh Iyengar <[email-masked]>
Date: Thu, 07 Jun 2018 06:23:50 -0700

When RTO timer is active, idle timeout only fires when idle timeout goes over RTO. This is because we specify that idle timeout is reset whenever a packet is sent or received. RTO will trigger packet transmission. We should do something better about this.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429

---

From: Nick Banks <[email-masked]>
Date: Thu, 07 Jun 2018 06:40:05 -0700

If RTO is active that means you are actively trying to send something, and therefore, in my mind, means you are not idle. I'd never expect the idle timeout to fire while in this state.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-395423804

---

From: Mike Bishop <[email-masked]>
Date: Thu, 07 Jun 2018 15:58:26 +0000 (UTC)

I seem to recall an algorithm for detecting a peer was gone, to close the connection. I also recall an assertion that either that timer or the idle timer, but never both, would be running. Did that ever turn into text?

But yes, I agree with Nick - if you're actively retransmitting, you're not idle and this timer shouldn't be running. If you're retransmitting and it's not getting through, the connection might still time out, but it won't be an idle timeout. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-395473778

---

From: Martin Thomson <[email-masked]>
Date: Mon, 11 Jun 2018 09:29:17 -0700

We don't actually define idleness.  Big surprise.

@ianswett had a more concise answer and I can't remember it right now.

It might be reasonable to define it as not including TLP or RTO (for sending).  Then RTO has a hard limit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396303809

---

From: ianswett <[email-masked]>
Date: Mon, 11 Jun 2018 09:55:43 -0700

One fairly good definition of the start of idleness is the max(time of last received packet, time of last sent packet after a packet was received).  The latter is there to ensure you don't idle timeout immediately after sending a request.  

Though if you're RTOing packets you're not technically idle, there's no proof the connection is still alive, so extending the idle timeout when sending timer based retransmissions is quite dangerous.

There is some inherent asymmetry between when you should no longer send new data vs when you should stop receiving data.  Ideally you'd continue processing incoming data for at least an RTT longer than you would send new data.

There are definitely some potential benefits in timing out the connection earlier when you believe it's dead, such as after 5RTOs or another criteria, but I'd be hesitant to recommend a uniform approach to dead connection detection for all applications and circumstances.  I think H2 commonly bundles a ping with new requests and then there's a subsequent timeout?  In QUIC, the PING wouldn't be necessary, since the request gets acknowledged.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396311683

---

From: Martin Thomson <[email-masked]>
Date: Mon, 11 Jun 2018 14:09:21 -0700

You mean the first packet you sent after the last received packet?  I think that works reasonably well.

It does fail to capture the request you might have spontaneously sent a millisecond ago, but I think that is a good thing.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396388119

---

From: ianswett <[email-masked]>
Date: Mon, 11 Jun 2018 15:04:46 -0700

@RyanAtGoogle may remember some details I'm forgetting, but yes, first packet you sent after the last received packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396402139

---

From: Martin Thomson <[email-masked]>
Date: Mon, 11 Jun 2018 15:12:11 -0700

How does that work when a client receives an ACK (which doesn't generate any response), then has a long quiescent period after which it sends a request?  That doesn't seem right.  Is there a limit on the time between receipt and sending?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396403848

---

From: ianswett <[email-masked]>
Date: Mon, 11 Jun 2018 17:32:17 -0700

There's no limit in the existing Chromium code, except the idle timeout.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396428356

---

From: janaiyengar <[email-masked]>
Date: Mon, 11 Jun 2018 17:33:02 -0700

@ianswett: It seems like that second part (time of last sent packet after a packet was received) captures the RTO cases, so that you don't keep restarting this timer if the sender keeps RTOing with no ACKs being received. 

@martinthomson: I think this works ok in your example right? The client receives an ACK which causes it to count the next packet, whenever it is sent, as a new start of the idle timeout period.

That said, idle timeout should not be used when the endpoint is RTOing into the void. That is a failure, and ought to be captured by a different constant, since it's about how long we're willing to keep trying before giving up on the connection, and will be f(RTO). Idle timeout does not have anything to do with the RTO period.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396428459

---

From: janaiyengar <[email-masked]>
Date: Mon, 11 Jun 2018 17:33:59 -0700

@ianswett : The Chromium code is incomplete currently, IIRC. It doesn't have this notion of a network timeout as distinct from application idle timeout. We need to separate those timeouts.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396428607

---

From: Martin Thomson <[email-masked]>
Date: Mon, 11 Jun 2018 17:47:40 -0700

Here's the example: Idle timeout is 10s.  You send a packet and subsequently receive an ACK at t=0.  You send a new message at T=10-sRTT.  You are now no longer idle, but the message you sent is going to arrive at the peer too late with ~50% probability of arriving before the idle timeout fires on their end.  During that last sRTT (and maybe a little before), you have a very good chance of not keeping the connection alive, but you will persist nonetheless and what will happen is that requests will fail.  The worst thing is that requests will fail in unrecoverable ways.

It's very clear that you don't want to count an RTO here.  A dead path is a common reason that you are hitting an RTO and restarting the idle timer just makes detecting the dead path that much harder (if not impossible as Subodh discovered).

I suspect that you want proof of life - that is, packets inbound - and to only rely on that.  You probably also need some advice about avoiding idle timeout by doing a liveness check once you reach `idle_timeout-sRTT-n*rtt_var` for some value of `n` that you tune based on your tolerance toward losing the messages you are about to send (for an idempotent message, `n` might be zero, for instance).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396430649

---

From: ianswett <[email-masked]>
Date: Mon, 11 Jun 2018 17:56:48 -0700

Chromium now has them separated(see: the PathDegradingAlarm and the RetransmittableOnWireAlarm https://cs.chromium.org/chromium/src/net/third_party/quic/core/quic_connection.cc?sq=package:chromium&g=0&l=281)  Path degrading is there solely for connection migration at the moment, and RetransmittableOnWireAlarm really should be able to be combined with the existing ping alarm.

You're right that separating idle timeout from dead path detection is a good choice in the long term.  We might only want to wait 15 seconds before declaring the path dead when we're retransmitting data, but a connection may have a 5 or 10 minute idle timeout.

My concern about dead path detection is that I don't have enough data to support a particular choice of when to consider a path dead.  5RTOs worked fairly well in the one experiment, but I have no comparison data with 4 or 6RTOs, and all of these are magic numbers.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-396431948

---

From: Mike Bishop <[email-masked]>
Date: Wed, 13 Jun 2018 14:28:32 -0700

The definition of idle given in the Seattle slides (which note that spec text is needed) is the time since any of the following:

- Last *new* app data sent or received
- Last PING sent, retransmissions excluded
- Last PING received

The definition of path timeout, separately, is the time since the first unacked packet for which an ACK is expected (i.e. not ACK-only) *and* whose packet number is greater than the largest ever acked.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-397093223

---

From: Martin Thomson <[email-masked]>
Date: Wed, 13 Jun 2018 19:08:49 -0700

The requirement to distinguish new vs. retransmitted is hard.  I think that last received is simple and effective enough - any data you send will be acknowledged, and retransmitted if it is not, so the start of the timer is close enough to the point that activity ceases.

The other thing to consider here is whether we allow the idle timeout to be asymmetric.  That is, it is in effect a statement of "if I don't see a packet from you in N seconds, I will drop state for this connection."  On face value, that doesn't help, because the peer with the shorter timer ultimately determines when the connection goes away, but it could allow - for example - a client to make a request without having to worry about a server's idle timeout.  Say the client has a 10s timeout and the server has a 30s timeout.  Rather than having to probe to see if the connection is still live when the connection gets close to 10s of idleness, the client can just send a request because it knows that the server will still be there (the network path, less so, but it's wise to keep that separate from this).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-397143976

---

From: ianswett <[email-masked]>
Date: Thu, 14 Jun 2018 07:39:19 -0700

Your last point about asymmetric-ness is important.  The Chromium implementation does something like what you described with asymmetric timeouts.

I would prefer to document a more general approach I briefly mentioned above, where you can't send new data within some time of the idle timeout, but you will continue to process and respond to incoming data until the timeout is hit, or maybe even a bit longer.  I think this works well for non-HTTP applications, whereas picking a shorter timeout on the client side only really works well for client driven apps like HTTP.

A simple approach might be: "New data must be sent with at least one RTO timeout remaining, and data should be received, processed and responded to for at least one RTO timeout after the specified idle timeout expires."?  Receiving and processing a new packet resets the idle timeout, so there's no special case necessary for sending in response to a packet received at the very end of the connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-397319588

---

From: Martin Thomson <[email-masked]>
Date: Thu, 14 Jun 2018 09:55:08 -0700

That seems reasonable.  I'm not inclined to go with MUST here, but instead note that sending within an RTO of the timeout is risky.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-397365067

---

From: ianswett <[email-masked]>
Date: Thu, 14 Jun 2018 10:00:31 -0700

Agreed, I think a MUST is too strong.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1429#issuecomment-397366644