Subject: [quicwg/base-drafts] Desirable behavior when it takes time to derive the traffic keys for the next PN space (#3821)

From: Kazuho Oku <[email-masked]>
Date: Sat, 04 Jul 2020 22:08:46 -0700

During the handshake, there are moments when an endpoint receives a packet from its peer and calculates the traffic keys of the next PN space. To give some examples, a server receives an Initial packet containing ClientHello, and derives the Handshake keys. Or, a client receives the Handshake packets, verifies the handshake transcript (including the certificate chain) and derives the 1-RTT keys.

While deriving the traffic keys of the next PN space, an endpoint might receive packets protected belonging to the next PN space. Note that the process of deriving the traffic keys might take some time.

I see Chrome running on a PC spending ~100ms for verifying the certificate chain, when it is launched from the command line along with the URL to be fetched. I presume that such launch-to-fetch-URL behavior would not be uncommon on PCs and handheld devices. Another (more) extreme case would be when OCSP is involved.

In these cases, the time required for deriving the new keys could take more than a handful of RTTs.

Hence the following questions, that might be worth clarifying in the specifications:

* Q1. How should an endpoint behave when it takes time to derive the traffic keys of the next PN space?
  * Q1.1. Should the endpoint continue ACKing packets of the current PN space?
  * Q1.2. Can the endpoint discard packets of the next PN space, as it can expect those packets to be retransmitted?
* Q2. How should the peer behave?

Re Q1.1, I think that the answer is yes, it should. Such an endpoint should calculate the next set of traffic keys asynchronously. It should continue running the QUIC transport for the existing PN space. To give an example, while the client verifies the certificate chain, it should ACK and continue ACKing the Handshake packets that it receives.

Re Q1.2, I think the answer is no. An endpoint SHOULD buffer packets of the next PN space while it calculates the traffic keys for that PN space. This is because the peer cannot distinguish the endpoint dropping packets vs. packets being dropped by the network. Dropping the packets lead to the peer's congestion controller exiting the startup phase. That is going to have a negative effect on performance.

Re Q2, I would argue that, ideally, the peer SHOULD refrain from arming the PTO timer for the packets of the next PN space, when it knows that the endpoint has all the information to derive the traffic keys of the next PN space, and knows that the endpoint would send something once those traffic keys are derived. To give an example, a server SHOULD NOT arm the PTO timer for 0.5 RTT data until it receives ClientFinished. At the very least, we should caution the server to not send Handshake(PING) packets alongside the 1-RTT probe, as ACKs for Handshake(PING) packets would reset the PTO timer.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821

---

From: Martin Thomson <[email-masked]>
Date: Sun, 05 Jul 2020 21:47:57 -0700

We encountered a problem similar to the one described here in our stack.  The latter part of a coalesced packet was being dropped because we had to spend time validating the certificate.  We ultimately added a small amount of buffering to address this.  And I have debugged a case where handshakes were failing due to the interaction between PTO timers in different packet number spaces.  For that, I don't have a good solution yet; I'm not convinced that deferring PTO is the right answer.

But I don't think that this requires anything in specifications, aside from perhaps a caution, noting that we already have a caution in [Section 5.7 of the TLS draft](https://quicwg.org/base-drafts/draft-ietf-quic-tls.html#pre-hs-protected).  That already suggests and permits (though doesn't recommend) buffering.

We could add a note that failing to buffer legitimate packets could lead to marking delivered packets as lost unnecessarily and performance degradation

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-654012206

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 06 Jul 2020 01:53:45 -0700

> And I have debugged a case where handshakes were failing due to the interaction between PTO timers in different packet number spaces. For that, I don't have a good solution yet; I'm not convinced that deferring PTO is the right answer.

I would argue that a probe packet is expected to send when an endpoint does not receive a response that it expects. That's necessary because it is the only way to guarantee forward progress (and therefore [we use a MUST](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.2.4-1)). But in this particular case that we are discussing, the ball is on the side of the peer. While I am happy to discuss the nuance, I do think that it needs to be something other than "MUST send a probe."

Note also that an endpoint sending PTOs in this case would in turn require the peer to have a larger buffer for storing undecryptable packets in order to avoid buffer overflow (that leads to performance degradation).

> But I don't think that this requires anything in specifications, aside from perhaps a caution, noting that we already have a caution in [Section 5.7 of the TLS draft](https://quicwg.org/base-drafts…

Aside from relaxing the MUST pointed out above, I think I'd be fine with just adding notes. Though I might argue that it is more important than the existing note that discusses about an "exceptional" case; some endpoints would almost always need time to calculate the next keys.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-654104851

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 07 Jul 2020 16:34:13 -0700

FWIW, [section 4.1.4 of the TLS draft](https://quicwg.org/base-drafts/draft-ietf-quic-tls.html#name-encryption-level-changes) states that: _as keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption level are available. These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes._

Part of the problem is that this is untrue. TLS stacks sometimes take time in providing the net set of traffic keys. And QUIC stacks do not even pretend that TLS stacks are acting in a synchronous manner. Instead, they act intentionally send ACKs for Handshake packets that are later than the 1-RTT packets they buffer.

This is a behavior that is observable from the peer, and leads to the exit of slow start depending on how you design the loss recovery. I think we might want to use normative words stating what the expectations are.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-655193479

---

From: ianswett <[email-masked]>
Date: Tue, 14 Jul 2020 09:23:58 -0700

Thanks for filing this.  I think I mostly understand the issue, and I think this is mostly a correctness issue with Chrome.  That being said, we may want to add some caution that this is a potential issue and to avoid handshake operations(ie: cert verification on the client and remote handshaking on the server) from excessively delaying acknowledgements.

I've seen lots of cases when the cert is spread over 3 packets and Chrome ACKs the first two packets before the cert is verified, so this doesn't seem to be a case of key availability.  Google servers don't have cert compression enabled yet, so many certs don't fit into 1 or 2 packets.

The recovery draft already says that peers don't arm for ApplicationData(including 0.5RTT) before the handshake is confirmed.  Knowing the peer has the information to derive the keys isn't really enough, since you'd also have to know the peer has received a packet of the next encryption level.  As a result, I can't think of any changes to this part of the PTO logic I'd want to make.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658277641

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 14 Jul 2020 10:20:53 -0700

@ianswett I agree that the there is no design changes necessary to the recovery draft.

The crux of the problem is the TLS draft. It currently states:
> As keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption level are available. These events are not asynchronous; __they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes__. ([section 4.1.4](https://quicwg.org/base-drafts/draft-ietf-quic-tls.html#section-4.1.4-1))

However, contrary to what the TLS draft says, Chrome (and IIUC what @martinthomson says, Firefox also) processes the TLS handshake messages containing the certificate chain in an asynchronous manner.

Consider the following example, where a web browser receives three packets:
* Handshake (PN=1, EncryptedExtensions)
* Handshake (PN=2, Certificate, CertificateVerify, Finished)
* 1-RTT (PN=3, NewSessionTicket)
* (30ms later)
* receives Handshake (PN=4, PTO)

The TLS draft suggests that client would be processing PN=3 before PN=4, because the browser would process the TLS handshake transcript synchronously and therefore it would have the 1-RTT keys when it recognizes PN=3 for the first time.

But the reality is that, if the browser takes 100ms to verify the certificate chain, the client buffers PN=3, acking PN=4 before PN=3, even though the interval between PN=3 and PN=4 is greater than max_ack_delay.

As can be seen, this is an observable behavior from the server-side.

Therefore, we need to fix the TLS draft at least (assuming that we agree that browsers are doing the correct thing).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658306606

---

From: ianswett <[email-masked]>
Date: Tue, 14 Jul 2020 10:46:49 -0700

One more note is that the RTT samples can be greatly inflated if a packet(ie: the Initial) is lost and the PTO fires to retransmit the data, and that causes the previously buffered Handshake packets to be acknowledged.  Coalescing an ack-eliciting Handshake packet with the Initial typically mitigates this, as long as it's processed before the buffered packets.  Since coalescing is already an optimization in the draft, it may be worth a note that it can prevent inflated RTT samples.

We've observed some cases where this causes the handshake to timeout because the server's PTO is so long, and our handshake timeout is fairly short(9 or 10 seconds).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658319612

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 14 Jul 2020 11:31:42 -0700

The discussion about RTT estimate is interesting. I'm not sure about the Initial vs. Handshake case, but when the client spends X milliseconds (e.g. X=100) verifying the certificate chain at the same time buffering 0.5 RTT data it receives, then the server would obtain an RTT sample of `X - max_ack_delay` milliseconds (e.g., 75 ms). That's not great.

One way of resolving the issue would be to not cap ack_delay using max_ack_delay for ApplicationData packets that were sent during the handshake. The rationale for not activating PTO for 1-RTT packets during the handshake is that the peer might not immediately have the necessary keys to process those packets. We assume the peer to buffer such packets. When the sender has such an assumption, it does not really make sense to cap the ack_delay.

I think doing something like that would be fine, even though I would not push on recommending such a strategy.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658341794

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 14 Jul 2020 18:57:34 +0000 (UTC)

@ianswett raises a good point about inflated RTTs. @kazuho's point about max_ack_delay actually exposes a grey area in the current semantics of that parameter. It is expected to be the maximum delay that the endpoint introduces. Other delay is supposed to be considered to be part of the "path". This is a situation where that assumption doesn't hold well. Separately, reporting this delay as part of ack_delay assumes that the implementation timestamps the packet when received at the endpoint. Meaning that if the endpoint actually does the cert verification synchronously, and does not read from the socket until that is complete, then it will need to read kernel receive times. (This also assumes that an endpoint is expected to timestamp the packet on receipt, not on decryption, which is very much how most stacks will do it, but there's not much difference at the moment in the common case between these two times.)

I'm tempted to say that we don't try to solve this in the protocol yet. Let's experiment with strategies that we think might work in implementations, share notes, and come back to the spec when we have a good answer. For the time being, we expect that the delays at the clients should not be super common, and even when it is the case, a couple of RTT samples should not significantly sway the RTT. I'm ok with that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658354136

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 14 Jul 2020 16:20:38 -0700

@janaiyengar 
> Separately, reporting this delay as part of ack_delay assumes that the implementation timestamps the packet when received at the endpoint. Meaning that if the endpoint actually does the cert verification synchronously, and does not read from the socket until that is complete, then it will need to read kernel receive times.

I think we do not need require endpoints doing that. We do not even require endpoints to verify certificate chain asynchronously, meaning that we allow endpoints to simplify their implementation at the risk of performance penalty.

I'd argue that not capping ack_delay during the handshake is strictly better than capping the value all the time, assuming that the client's clock is not drifting. That is because the server would have a better RTT estimate when the client reports the correct numbers. If the client fails to do so, it'd be the same as status quo, where RTT estimate would become larger than the actual value.

All that said, I am totally fine with leaving this an open area that endpoints can experiment.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658459501

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 14 Jul 2020 19:25:47 -0700

Summarizing my thoughts after discussing this with @ianswett and thinking about this with @kazuho. Apologies for being verbose, but I'm trying to summarize from the top.

**Problem**:

At a high level, the problem we are discussing here is that an endpoint buffers packets, because it isn't able to read them or write acks for them, and it might take a while to dequeue the buffer. As a result, the acks can be sent much later than when the packets were received at the endpoint, causing the ack_delay to potentially be much higher than max_ack_delay. This inflates the smoothed_rtt and rtt_variance, inflating the PTO, which can cause handshakes to fail (if a "handshake timeout" period is employed).

The problem is that the receiver endpoint can take arbitrarily long to respond. The ack_delay variable is meant to signal precisely this -- we tout it as an important distinction from TCP -- and it is useful here, except that it is limited to max_ack_delay in the various RTT estimations. This was done deliberately, allowing for any additional delays to become part of the path delay, with the expectation that such additional delays are outside of the control of the QUIC endpoint and are likely to be recurring. While that remains true, the issue that we are discussing here is an extreme case that occurs during the handshake and is not expected to be recurring.

**Proposed Solution:**

We use the ack delay signal during the handshake, and acknowledge that we might need to treat ack delays reported during the handshake period as _slightly_ special. Specifically, we can ignore the max_ack_delay, but we still limit the RTT sample to >= min_rtt. An endpoint does this until the handshake is _confirmed_. That is, until the endpoint is certain that the peer has 1RTT read and write keys and therefore does not have to buffer any more packets for lack of keys.

**Also:**

Separately, the recovery spec currently says don't arm PTO on ApplicationData until `handshake complete`. This should be `handshake confirmed`. This is because handshake completes at a client when it sends a CFIN, after which the client can send 1-RTT data. The CFIN however might get lost and the server might buffer the 1-RTT data until it receives a retransmission of the CFIN. Changing this to `handshake confirmed` makes it so that no PTO is armed on ApplicationData until there's no Initial or Handshake data left in flight.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658507628

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 14 Jul 2020 21:51:15 -0700

Regarding @janaiyengar's last point, I've changed the moment that the client starts arming PTO timer for ApplicationData PN space from handshake complete to handshake comfirmed. As Jana states, that is the correct thing to do based on the rationale that we have been using to explain why it should not be armed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658544030

---

From: Martin Thomson <[email-masked]>
Date: Wed, 15 Jul 2020 00:34:27 -0700

FWIW, I was *really* concerned about that last change.  Aside from it making this a non-editorial change, the way that we avoid handshake deadlock depends very much on HANDSHAKE_DONE being followed by probes.  But as the server confirms and completes the handshake at the same time, I've convinced myself that this is OK.  

(That this is what our stack did, largely by accident, is vindication of sorts...)

I would encourage people to sit down and run the analysis on that change to convince themselves that this is not reintroducing the handshake deadlock.

The other part of the proposed solution needs to be slightly different.  @janaiyengar proposes that we ignore `max_ack_delay` prior to confirming the handshake.  I think that's good, but it is not completely correct, and nor would it be sufficient.

We need to ignore `max_ack_delay` for acknowledgements of packets that were sent prior to handshake confirmation.  Those are the ones that might have been buffered until keys were ready.

In order for that to work, we have two new requirements on packet processing for buffered packets:

1. endpoints MUST process and acknowledge (or discard) any packets that they might have buffered during the handshake within `max_ack_delay` of completing the handshake

2. endpoints MUST report an ACK Delay that is based on when a packet is received, not when it was processed

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658599289

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 15 Jul 2020 01:29:22 -0700

Regarding `max_ack_delay`, I think @martinthomson's comment is accurate with possibly one slight modification.

> 2. endpoints MUST report an ACK Delay that is based on when a packet is received, not when it was processed

We might want to say that ACK Delay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally.

The observation here is that any unintentional delay (e.g., decryption time, payload processing time) should be considered part of RTT, because that is unavoidable.

And the added bonus of defining this way is that then there would be no change required for QUIC stacks that synchronously process the TLS handshake messages.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658626211

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 15 Jul 2020 05:06:40 -0700

Just FWIW, I might point out that we are actually making the design streamlined, by improving consistency in how we handle ack delay. The new rule is:
* The receiver reports the sum of the controlled delays by using the delay field of an ACK frame.
* When the sender knows what the maximum of that controlled delay is, the sender ...
    * uses that value (i.e. max_ack_delay) to arm the PTO timer, and
    * uses that value (i.e. max_ack_delay) to calculate the RTT, because if the value of the delay field is greater than the known maximum, the difference is considered as uncontrolled delay and therefore has to be accounted as part of the path delay.
* When the sender does *not* know what the maximum of that controlled delay is, the sender ...
    * does not arm the PTO timer, because it does not know when it can expect an ACK to arrive, and
    * calculates the RTT without the capability of detecting uncontrolled delay.

It's simply that up until now, we have misused max-ack-delay in the last bullet point case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658728956

---

From: Martin Thomson <[email-masked]>
Date: Wed, 15 Jul 2020 16:20:54 -0700

This is a good outcome, and I agree with Kazuho.

However, I am going to request that someone (not me preferably) take this to the mailing list to confirm with a wider group.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-659064210

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 17 Jul 2020 00:13:32 -0700

@martinthomson : I agree with what you've spelled out above ([here](https://github.com/quicwg/base-drafts/issues/3821#issuecomment-658599289)). It is close to what I had in mind, but I didn't do a good job of even spelling out fully what I was thinking.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-659913581

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 28 Jul 2020 19:27:44 -0700

Posting the email I sent out to the list here, FTR. This got no pushback and support, FWIW. Note that this resolution makes this issue a design one, so I'm removing the editorial tag.

```
Problems:

During the handshake, an endpoint can receive packets (carrying data or acknowledgements) that it cannot decrypt because the requisite keys are not yet available. This can happen because the crypto machinery at the endpoint is waiting for earlier packets to be received to yield the keys. Similarly, a client might not be able to read 1-RTT packets received from the server because it is waiting for certificate validation to complete.

1. The assumption in the TLS specification is that when an endpoint is busy doing something (certificate validation or key generation for instance), it won't be able to respond to received packets with acknowledgements (Section 4.1.4).

Contrary to the assumption in the spec, Chrome and Firefox both process the certificate chain asynchronously, and therefore acks handshake packets immediately, even the ones that arrive after they have started verifying the certificate chain. But they buffer 1-RTT packets, and therefore the issue exists for 1-RTT packets.

As a result, there's potential delay, sometimes significant, in responding to some packets during the handshake. There are two side effects of this.

S1. First, this means that an endpoint that runs a PTO timer and retransmits data for 0-RTT and 1-RTT packets while the handshake is still in progress might be doing this quite unnecessarily. The peer may well have received this data and have queued it, waiting to be able to decrypt it. 

S2. Second, acks for these packets that are buffered and then later decrypted and read can be sent much later than when the packets were received at the peer, causing the ack_delay to potentially be much higher than the advertised max_ack_delay. The endpoint ignores ack_delay larger than max_ack_delay, and this inflates the smoothed_rtt and rtt_variance, potentially inflating the PTO significantly (which can cause handshakes to fail if the endpoint employs a "handshake timeout" period).

Proposed Solutions:

S1. To solve the first problem, we already have a condition in the recovery draft to _not_ arm the PTO on 1-RTT data until the handshake is complete (Section 6.2.1). This is not quite adequate, since it still allows a client to arm a PTO timer on 1-RTT data on sending a CFIN and 1-RTT data, before ensuring that the server has received the CFIN. The proposed solution here is to change this condition from "handshake complete" to "handshake confirmed". Meaning that an endpoint only arms a PTO timer on 0-RTT or 1-RTT data when it knows that the peer has the keys necessary to read and write in 1-RTT, and can therefore respond with an ACK.

We think this holds up, but we'd like to get this under everyone's eyes, to ensure that the discussion isn't missing subtle deadlock issues.

S2. To solve the second problem, the proposal is to use ack_delay as reported by the endpoint that buffers the data appropriately. ack_delay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. For this, we need the following:

- endpoints MUST report an ack_delay that is the sum of the controlled delay (i.e. the duration of the data and the ack being buffered)

- endpoints ignore max_ack_delay for acknowledgements of packets that were sent prior to handshake confirmation. Those are the ones that might have been buffered at the peer until keys were ready.

In addition, we explicitly recommend endpoints that asynchronously process handshake messages to buffer 1-RTT packets, instead of dropping them, as dropping packets has significant impact on congestion control. Note that this matches what those implementations already do.
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-665393532

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 28 Jul 2020 19:34:54 -0700

@martinthomson noted:
```
> - endpoints ignore max_ack_delay for acknowledgements of packets that
> were sent prior to handshake confirmation. Those are the ones that
> might have been buffered at the peer until keys were ready.

For this, endpoints need to retain an additional value: the highest packet number
that was sent prior to confirmation (or the lowest packet number sent after confirmation).
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-665395450

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 28 Jul 2020 19:36:32 -0700

This should be marked design.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3821#issuecomment-665395868