Subject: [quicwg/base-drafts] Handshake failure (infinite ping-pong) when path MTU is asymmetric (#4183)

From: Kazuho Oku <[email-masked]>
Date: Mon, 05 Oct 2020 22:09:39 -0700

Consider the following scenario, assuming that the MTU from the client to the server is 1200 bytes, and the MTU from the server to client is 1000 bytes.
* step 1. Client sends:
  * datagram (1200 bytes), carrying Initial (Hello).
* step 2. In response, the server sends:
  * datagram (1200 bytes), carrying Initial (Hello) and a Handshake packet that contains the first half of handshake-level TLS messages (i.e. EncryptedExtensions, Certificate, CertificateVerify, Finished).
  * datagram (500 bytes), carrying Initial (PING) and a Handshake packet that contains the latter half of the handshake-level TLS messages.

The first datagram gets lost, the second datagram reaches the client.

* step 3. In response to the datagram being received, the client sends:
  * datagram (tiny), carrying Initial (ACK).
* step 4. When the server receives the packet, it learns the RTT, and arms the time-based loss detection timer. In a fraction of RTT, that timer fires, and the server sends what have been sent in step 2.

The first of the two datagrams sent in step 4 gets lost, the second one reaches the client.

* step 5. In response to the datagram being received, the client sends an ACK, as it did in step 3.

As we can see, we have an infinite loop that cannot make progress.

This example might sound a little strange, as the server arms the time-based loss detection timer for Handshake packets when receiving an ACK for the Initial packet. Though I would point out that the specification is lax on how an endpoint might use an Initial ACK to drive loss recovery of other packet number spaces; see [the third paragraph of section 6.2.3 of the recovery draft](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.2.3-3).

We might pretend that the MAY in [the second paragraph of section 6.2.3](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.2.3-2) restricts alternative loss signals (i.e. duplicate CRYPTO frames, receipt of Handshake packets before obtaining Handshake keys) from being applied to anything other than Initial PN space. But even in that case, I think we still have a failure mode. If the size of ServerHello is slightly above the MTU, if the server is designed to embed ACK in the first Initial packet for each send event, and if the client skips packet numbers upon observing possible loss, the size of the server-sent ACK frame increases two for every Initial packet that the client sends. This means that for each round-trip, the space that can be used for sending CRYPTO data will decrease by 2 bytes for the first Initial packets that the server sends in each iteration. These 2 bytes will be sent as tiny packets, and they will reach the client, while the full-size packet gets consistently dropped. In other words, the failure mode in this case is that there would be a ping-pong between the endpoints, and only 2 bytes of data can be exchanged per each round-trip.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 05 Oct 2020 22:14:24 -0700

Regarding possible solution, I think the most straightforward fix would be to requires servers to always send full-sized datagrams, until the path is validated. We can be limit that requirements to endpoints that use something other than ACK or time threshold for retransmission.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704031814

---

From: Martin Thomson <[email-masked]>
Date: Mon, 05 Oct 2020 22:35:54 -0700

> step 3. In response to the datagram being received, the client sends:
>    datagram (tiny), carrying Initial (ACK).

Minor correction here: the client will send a 1200 byte Initial packet.  Though this doesn't stop the problem from persisting.

I think that the proposed remedy is out of proportion to the problem.  This is not a situation that we would expect to be common (this is an IPv4 only path, for starters).  Do you have evidence of a path with an asymmetric, sub-1200 byte MTU?

You could say that this is a problem if the initial state is 1400/1200 path MTU and the first Initial from the client is 1400, but the ACK is 1200.  But then endpoints are allowed to send less than 1400, so the correct response there is "don't do that".  It's just that we have a minimum packet size of 1200, so the correct reaction is "don't do QUIC" on that path.  The problem being that you don't necessarily get to that point easily.

(I'm not following the 2 bytes per round trip thing fully, but it seems like it's 2 useless bytes.  If I'm write, you assume that the server doesn't trim its ACK frames.  Ours will.  Even without ACK from the client, it will trim *before* the size of the ACK increases by another byte: after 32 round trips when the largest acknowledged reaches 64.  But the client is sending ACK frames, so the server can stop sending ACK ranges.  So we likely stop making progress well before the two bytes per round trip get anywhere. This is probably a distraction from the main topic though...)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704038608

---

From: Marten Seemann <[email-masked]>
Date: Mon, 05 Oct 2020 22:41:11 -0700

In quic-go, I implemented a handshake timeout (that’s independent of the idle timeout) and defaults to 10s. If the handshake doesn’t complete, for whatever reason, during that time, the connection gets killed. Maybe that’s the easier solution?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704040217

---

From: Martin Thomson <[email-masked]>
Date: Mon, 05 Oct 2020 22:43:25 -0700

I think that the handshake timeout is a common response that ties up a bunch of these problems neatly.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704040980

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 05 Oct 2020 22:53:39 -0700

Handshake timeout is the last resort. The problem here is that the endpoints would exchange packets rapidly until the handshake times out. As an example, if the handshake timeout is 10 seconds and the RTT is 10ms, a server might transmit 1700 datagrams (10 seconds / 10 ms * 2 datagrams * 9/8; where 9/8 is the loss recovery threshold).

> Minor correction here: the client will send a 1200 byte Initial packet. Though this doesn't stop the problem from persisting.

Thank you for the correction.

> I think that the proposed remedy is out of proportion to the problem. This is not a situation that we would expect to be common (this is an IPv4 only path, for starters). Do you have evidence of a…

This is a fair point; I'll see if I can get some information.

> (I'm not following the 2 bytes per round trip thing fully, but it seems like it's 2 useless bytes. If I'm write, you assume that the server doesn't trim its ACK frames. Ours will. Even without ACK from the client, it will trim before the size of the ACK increases by another byte: after 32 round trips when the largest acknowledged reaches 64. But the client is sending ACK frames, so the server can stop sending ACK ranges. So we likely stop making progress well before the two bytes per round trip get anywhere. This is probably a distraction from the main topic though...)

(The server's ACK never reaches the client, as it is always coalesced with the CRYPTO frame. Therefore, a client cannot ACK the ACK. Eventually, the server will fail to build the ACK frame as the number of gaps become too large, but until then, there would be a ping-pong. But if we think that the behavior explained in the main topic is conformant to the specification, this is just a distraction as you point out.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704044396

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 05 Oct 2020 22:58:14 -0700

@martinthomson: We sadly have evidence of path MTU being set (for some ASes) to < 1200 bytes. It's a small enough fraction and this ought to be fixed, but it exists. Basically we should end up not doing QUIC under these conditions.

The core of the problem is that the failure mode here is a slow death. This handshake is ultimately going to fail, but potentially after a lot of wasted packets, depending on implementation details. @kazuho's proposed fix is to drive this partial failure to deliver packets to complete failure, which is really what ought to happen.

The proposed solution has the server effectively doing PMTU validation during the first RTT along with path validation by forcing all packets to be max size, ensuring that there are no small packets don't reach the client. Commonly, this should end up only padding the last packet in the first flight from the server, and if you're not careful, the packet carrying the ACK of the client's Initial.

I see this as an optimization of a failure case that exists but is uncommon. Some text that draws caution to this point in the PMTU section and suggests what a server can do might be useful.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704046255

---

From: David Schinazi <[email-masked]>
Date: Tue, 06 Oct 2020 09:18:55 -0700

FWIW, Google's implementation avoids this problem by adding padding to ensure that all UDP datagrams containing Initial packets that we generate are at least 1200 bytes long - on both client and server. I would personally be in favor of making that a MUST in the spec, since verifying the MTU during the handshake is one of the important advantages of QUIC over TCP. If we do not solve this issue normatively, then users will be burdened with a noticeable timeout when they access servers that don't do this on networks with low server-to-client MTUs, which exist.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704391873

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 06 Oct 2020 15:01:08 -0700

FWIW, I think there's another failure mode that might be more disastrous.

What if the client has a TLS session ticket, and resumption succeeds? In that case, the server's first flights carrying Initial and Handshake packets might not be a full-sized datagram. The first full-sized datagrams can be 1-RTT packets.

I think that means that if a client tries to resume a QUIC connection by using a TLS session ticket from a address for which the return path has a MTU less than 1200 bytes, we might see the connection being stuck. IIUC, this would happen for clients doing happy eyeball too, as the handshake for HTTP/3 will complete before HTTP/2.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704576344

---

From: mjoras <[email-masked]>
Date: Tue, 06 Oct 2020 15:47:06 -0700

> FWIW, Google's implementation avoids this problem by adding padding to ensure that all UDP datagrams containing Initial packets that we generate are at least 1200 bytes long - on both client and server. 

We do this too. This ensures clients on weird networks simply fallback to TCP. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704592726

---

From: Martin Thomson <[email-masked]>
Date: Tue, 06 Oct 2020 17:38:08 -0700

This discussion suggests two safeguards:

1. Clients should have a handshake timer that causes them to abandon a connection that has not reached the "confirmed" state.  Clients should abandon a connection at this time even if it has made progress toward that goal.  If they do not, any hole in the specification or implementation that allows for liminal states will result in that state being persistent.

2. Servers should ensure that all packets they send are at least 1200 bytes until address validation is complete.  This will ensure that the path supports the minimum MTU that QUIC requires.  Other mechanisms can then kick in to ensure that QUIC isn't used on that path.

Both of these can be enacted separately in terms of pull requests on the specification.  I'm reluctant - at this stage - to make these carry normative force in the sense that we use the M-word, but both recommendations seem likely to be adopted.

I will also point out that this is an IPv4-specific problem unless we allow for unbounded use of IP header extensions.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704625739

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 06 Oct 2020 18:16:33 -0700

@martinthomson Regarding the two safeguards, I am in favor of having the latter, but I am not sure if we need the former. The former does not prevent a failure mode that I pointed out in https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704576344, that prevents the connection from making progress _after_ the handshake is complete.

So I think just having the latter is fine. Regarding how strong the safeguard should be, I think my preference goes to using a SHOULD. While I think every server should be sending full-size packets until the path is validated, I am not sure if there's any benefit in allowing the client to verify that the server is doing that. At the same time, there is no reason to prohibit the server that thinks it knows the path MTU from sending smaller packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704635563

---

From: mjoras <[email-masked]>
Date: Tue, 06 Oct 2020 18:20:06 -0700

I agree with @kazuho. We've previously decided we aren't really in the business of suggesting "higher level" behaviors like this. If we did we should probably also codify things like TCP fallbacks. Instead we trust people will do those on their own. I think a "handshake timeout" falls in a similar bucket of an obvious idea people will implement in practice (and there are obvious analogs in the TCP world in the form of connection timeouts, which may or may not include TLS).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-704636496

---

From: Mike Bishop <[email-masked]>
Date: Thu, 08 Oct 2020 07:20:20 -0700

It sounds like "until address validation is complete" might not be sufficient.  The Initial token provides validation, but doesn't exercise the path MTU.  We should clarify that even if the address is validated by a NEW_TOKEN token (i.e. the anti-amplification limit is waived), the server should send 1200-byte datagrams until some in-connection address validation has also succeeded.

However, that feels like a new concept that I'm a little hesitant to try introducing. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705603255

---

From: David Schinazi <[email-masked]>
Date: Thu, 08 Oct 2020 07:53:13 -0700

"until address validation is complete" sounds complex and error-prone, why not simply apply the requirement to initial packets like we do on the client?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705624836

---

From: Martin Thomson <[email-masked]>
Date: Thu, 08 Oct 2020 15:58:04 -0700

To continue the discussion from the PR...

@marten-seemann points out that the handshake never moves on if you pad the Initial.  @kazuho then points out that we currently recommend that clients re-send their Initial if they see a Handshake packet that they can't decrypt (I keep forgetting that because we haven't implemented that bit yet, ...).  I think that perhaps that we might take Marten's suggestion and tweak the recommendation about re-sending Initials at the client.  If the client limits the number of Initial packets it sends this way, we avoid an infinite cycle.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705865435

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 08 Oct 2020 17:12:48 -0700

@MikeBishop I share your concern and the hesitation. And I think we can stick to discussing path validation.

My point would be that we do not say how endpoints should act when the MTU of a validated path goes below 1200 bytes mid-connection. Then, why do we need to talk about revalidating path MTU when the client reconnects with a valid token? Note that the lifetime of a QUIC connection can be longer than the lifetime of a NEW_TOKEN token.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705892972

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 08 Oct 2020 17:20:03 -0700

@martinthomson I would not oppose to tweaking the MAY that allows client to retransmit ClientHello when receiving a Handshake or a 1-RTT packet for which the client does not have keys. If we do that, I agree that the server-side recommendation can be reduced to Initial packets only.

That said, if we are fine with changing the specification on both the client- and server-side, my preference would be to recommend both endpoints padding all datagrams up to 1200 bytes until the path is validated (or there is a guarantee that the peer has validated), as @martinthomson suggested in https://github.com/quicwg/base-drafts/pull/4188#discussion_r501450489. Doing so eliminates the chance of having ping-pong, regardless of the recovery algorithm adopted by endpoints.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705894927

---

From: Martin Thomson <[email-masked]>
Date: Thu, 08 Oct 2020 17:24:33 -0700

The reason I prefer the Initial-only padding is that it is principled: it says that you can't start a connection with small packets, from either side.  And the cost is only tweaking an optional optimization.  Padding every packet might be better, and I don't think we should look to prohibit it, but I don't think that it is necessary to even recommend it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705896324

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 08 Oct 2020 18:20:14 -0700

Fair enough. I've updated the PR, please take a look.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705910239

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 08 Oct 2020 18:30:12 -0700

Ugh apologies, I've been having this discussion on the PR. I would argue that a different principle is in order here: that if the problem is an MTU problem, then you want complete failure, not partial failures. Sending packets that are < 1200 bytes risks partial failures of all kinds.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705914800

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 08 Oct 2020 19:21:58 -0700

@janaiyengar I tend to share your concern, though, I might point out that an endpoint ought to be prepared to receive peer's small-sized packets, unless we use a MUST. Do we want to state that both sides MUST start with sending full-sized packets until it receives an ACK from the peer?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705928349

---

From: Marten Seemann <[email-masked]>
Date: Thu, 08 Oct 2020 19:53:41 -0700

That optimization doesn't seem unreasonable, so I'd prefer to account for it.

Why don't we tie it to dropping of Handshake keys on the server side? This might be slightly inefficient in the case of HRR, but we don't have to introduce another flavor of path validation.

We do have a similar issue for connection migration, don't we? You could probe the path using a small packet, switch to that path, and then discover that the MTU is too low to support 1280 byte packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705937826

---

From: Martin Thomson <[email-masked]>
Date: Thu, 08 Oct 2020 20:23:35 -0700

If this is something we want to address with "get an ACK, drop the padding", then I could live with that.  But I can't pad 1-RTT packets the way I want to there.

I'd still prefer to say Initial-only, but I could live with the ACK-based arrangement.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705945224

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 08 Oct 2020 20:26:52 -0700

I tend to agree with what @martinthomson suggested earlier; limit how many times an endpoint can retransmit packets before PTO timer expires. That's what the PR says now.

Sending something before PTO is fine only when packet loss is rare. If we repeatedly see packets being dropped, then the sender has to reduce the send rate.

@marten-seemann Do you have any issue with that?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705945991

---

From: Marten Seemann <[email-masked]>
Date: Thu, 08 Oct 2020 21:01:38 -0700

> @marten-seemann Do you have any issue with that?

In fact, I do. Implementing a limit for the number times you can retransmit data seems like a lot of additional complexity (as you have to lift that limit at some point to achieve reliable data delivery).

Unrelated to that: I noticed an interesting interplay with the amplification limit here: We might end up in a situation where the server has less than a full packet size worth of credit in its amplification limit, and normally would send a smaller packet. It seems like this situation will occur exactly when we're worried about asymmetric MTUs, so in practice the amplification limit will not be 3, but, 2.x.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705954682

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 08 Oct 2020 22:01:51 -0700

@marten-seemann 
> In fact, I do. Implementing a limit for the number times you can retransmit data seems like a lot of additional complexity (as you have to lift that limit at some point to achieve reliable data del…

Could you elaborate a bit more please, because I'm not sure how it would be?

When sending the first flight, the server would have arm PTO timer (or would have postponed arming the timer when it had reached the amplification limit). Receipt of a duplicate CRYPTO data is a MAY that allows the server to have a special path for triggering what would have happened when PTO timer should have fired. I do not think the complexity changes a lot if we add a counter that restricts the invocation of that special path (in practice, I think that the counter can be _one_, meaning that it is a boolean flag for each connection).

> Unrelated to that: I noticed an interesting interplay with the amplification limit here: We might end up in a situation where the server has less than a full packet size worth of credit in its ampl…

I would not call this a 2.x situation. Rather, I would say that the server is guaranteed to have a send window of 3600 bytes. If the client sends Initial packets greater than 1200 bytes, then the server might have more room which it cannot use, when the server decides to stick to 1200 bytes. But that's fine, because servers should not be built under the assumption that the client would always be sending Initial packets greater than 1200 bytes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705968892

---

From: Marten Seemann <[email-masked]>
Date: Thu, 08 Oct 2020 22:10:51 -0700

> When sending the first flight, the server would have arm PTO timer (or would have postponed arming the timer when it had reached the amplification limit). Receipt of a duplicate CRYPTO data is a MAY that allows the server to have a special path for triggering what would have happened when PTO timer should have fired. I do not think the complexity changes a lot if we add a counter that restricts the invocation of that special path (in practice, I think that the counter can be one, meaning that it is a boolean flag for each connection).

That sounds complex to me. I'd prefer to have a solution that doesn't require use to modify the retransmission logic. I feel like this is a symptom of using the wrong requirement for padding packets.

> I would not call this a 2.x situation. Rather, I would say that the server is guaranteed to have a send window of 3600 bytes. If the client sends Initial packets greater than 1200 bytes, then the s…

Now let's say the server sends 1280 packets. It can send two of those packets, and then is left with 1040 bytes. The MTU probing requirement now says that a full-size packet has to be sent, whereas the amplification limit requires to not send a full-size packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-705971213

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 11 Oct 2020 14:38:11 -0700

@marten-seemann 
> That sounds complex to me. I'd prefer to have a solution that doesn't require use to modify the retransmission logic. I feel like this is a symptom of using the wrong requirement for padding packet…

It seems to me that you are seeking for something impossible, or am I missing something?

Are you suggesting that all packets, regardless of the packet type, should be padded up to at least 1200 bytes until the endpoint receives an ACK? IIUC that's the only way to retain the existing retransmission rules (e.g., endpoint MAY retransmit CRYPTO data when receiving a packet for which it does not have keys).

But doing so would have negative consequences on what a server can send. At the moment, the PR says _pad Initial packets_. That means that in practice, a server can send datagrams containing Handshake packets of any size. Therefore, the server can fully utilize the 3x window even when there is a mismatch between the endpoints' MTU, if the server is fine with _not_ prepending Initial packets to every Handshake packet it sends.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-706771848

---

From: martinduke <[email-masked]>
Date: Mon, 12 Oct 2020 10:45:41 -0700

Why is the server putting an Initial PING in its second packet? While legal, this is unnecessary behavior that allows the infinite loop to occur. That seems like another "don't do that" kind of thing. Without that PING, both sides will time out quickly.

Now, if the server hello spills into two packets, that's a different thing. But I think we might have enough time before ServerHello gets that big for the any paths with this problem to be fixed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707259231

---

From: martinduke <[email-masked]>
Date: Mon, 12 Oct 2020 10:53:54 -0700

I would be comfortable with saying "any datagram that contains an Initial Packet MUST be at least 1200 Bytes unless they contain a CONNECTION_CLOSE frame." This just makes client and server requirements symmetric, and will have little practical impact on servers that aren't doing weird things.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707263115

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Oct 2020 14:09:44 -0700

> Why is the server putting an Initial PING in its second packet? While legal, this is unnecessary behavior that allows the infinite loop to occur. That seems like another "don't do that" kind of thing.

We explicitly suggest that such optimization makes sense, because it gives the server a RTT estimate and to run time-based loss detection; please refer to the [third paragraph of section 6.2.3 of the recovery draft](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.2.3-3).

> I would be comfortable with saying "any datagram that contains an Initial Packet MUST be at least 1200 Bytes unless they contain a CONNECTION_CLOSE frame."

That's certainly an option, though I see at least two problems:
* A MUST is going to be a breaking change, because it implies that the peer can test and reject packets that do not conform to the requirement. I'm not sure if we want to make a breaking change that is not strictly necessary, at this moment.
* Padding all Initial packets hurt performance, because the server's send window is restricted to 3x of what it has received from the client. For non-coalescing servers, padding Initial packets to full size would mean that it would have ~900 bytes less space to send the certificate chain in the first three packets (assuming that ServerHello is around 2x0 bytes). Or servers would lose one MTU of send window when sending an ACK-only Initial packet. I tend to prefer giving servers freedom regarding how (and if) they would implement defense against paths with MTUs being too small.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707347106

---

From: martinduke <[email-masked]>
Date: Mon, 12 Oct 2020 14:21:37 -0700

@kazuho my proposed text wouldn't require most servers to pad Initial Packets. In general it will coalesce the Initial Packet with a Handshake to get to 1200.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707351763

---

From: martinduke <[email-masked]>
Date: Mon, 12 Oct 2020 14:22:23 -0700

As for MUST vs SHOULD, I don't have a strong opinion but I'm attracted to the idea of the client and server requirements being symmetric, although in practice the client will have to pad and the server won't.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707352070

---

From: David Schinazi <[email-masked]>
Date: Mon, 12 Oct 2020 14:56:10 -0700

I like @martinduke's proposal of `any datagram that contains an Initial Packet MUST be at least 1200 Bytes unless they contain a CONNECTION_CLOSE frame.`.

To respond to @kazuho:
- I think a MUST isn't going to be problematic here, we can tell clients to not enforce this MUST for now. Getting the final spec in the right shape is more important than what implementors will do before we ship the spec
- I don't think we should take into consideration the performance of servers that don't coalesce - coalescing is pretty much required for performance
- If a server is about to send an INITIAL-ACK-only packet, it should coalesce as much of its full first flight with that as possible! The requirement for 1200 bytes doesn't mean they have to be wasted PADDING bytes

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707365600

---

From: ianswett <[email-masked]>
Date: Mon, 12 Oct 2020 15:08:24 -0700

Apologies, I'm catching up on this.  I support @martinduke suggestion and I believe it's either exactly what we do today or what we intend to do.  It seems this aligns with @mjoras implementation experience as well.

In terms of the fast Initial retransmission optimization, I thought there was already text that caused that to only be exercised once per connection, to avoid various ping-pong issues?  If not, I'd support adding that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707369846

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Oct 2020 16:20:20 -0700

I would respond to @DavidSchinazi as I think his comments breaks ups the point in a very good way.

> * I think a MUST isn't going to be problematic here, we can tell clients to not enforce this MUST for now. Getting the final spec in the right shape is more important than what implementors will do before we ship the spec

If we are going to make the specification in the right shape, I actually (slightly) prefer requiring servers (or both endpoints) to always use full-sized packets until they receive an ACK (or until some later point; e.g., path being validated). Doing so gives us the guarantee that none of the packets would pass through a path that cannot handle QUIC traffic.

Compared to that, just requiring Initial to be padded is a half-baked solution, especially when we consider the fact that we suggest the receipt of Handshake packets implies the loss of Initial packets.

> * I don't think we should take into consideration the performance of servers that don't coalesce - coalescing is pretty much required for performance

My understanding has been that non-coalescing endpoints are considered first-class citizens; I would rather not revisit the requirements (or what we consider as reasonable) at this stage.

> * If a server is about to send an INITIAL-ACK-only packet, it should coalesce as much of its full first flight with that as possible! The requirement for 1200 bytes doesn't mean they have to be wasted PADDING bytes

The problem with ACK-only packet is not in the first flight. It is in the second flight.

The problem of mandating all Initial packets to be padded is that, when loss of ServerHello is involved, we might start seeing servers spending 3 RTTs to complete the handshake rather than 2 RTTs.

Consider the case where the size of the server's certificate chain is between 2 MTU and 3 MTU. When receiving the client's first Initial packet, the server sends a three datagrams: Initial+Handshake, Handshake, Handshake. Let's say that at least the first datagram was lost.

Assuming that the endpoints do not implement the MAY in [section 6.2.3](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#name-speeding-up-handshake-compl), the client retransmits the Initial packet when its PTO timer expires.

At 50% probability, this packet will be delivered to the server before the server's PTO timer expires for the server's first flight. That means that the server will be sending an ACK-only packet at 50% probability, in which case, there would be only 2 MTU worth of window when the server's PTO timer expires, meaning that the server will be incapable of sending all the TLS handshake messages in its second flight.

I'm not sure if this is the only problem that we have.

Anyways, to summarize, my hesitation against a "MUST pad all Initial packets" is that it is likely going to have unnecessary negative consequences on the handshake latency; my preference goes to minimizing the impact of the change at a moment when we are rolling out QUIC.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707392731

---

From: ianswett <[email-masked]>
Date: Mon, 12 Oct 2020 16:48:30 -0700

Google has always been padding all Initial packets to full size, so if this causes major issues, it'd be helpful to see an A/B experiment that demonstrated the problem.

To me, the issues you're outlining are related to the amplification factor, which I agree is fairly conservative and it would not be surprising if a larger number(ie: 5 or 10) is accepted as the norm in server deployments.

As you stated, if servers do implement the MAY, this is not a significant issue.

I'm unclear on why "MUST pad all Initial packets" could be an issue.  Typically Initial packets are coalesced with Handshake packets, so the padding is either nonexistent or not that significant.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707400549

---

From: David Schinazi <[email-masked]>
Date: Mon, 12 Oct 2020 16:56:32 -0700

> Compared to that, just requiring Initial to be padded is a half-baked solution, especially when we consider the fact that we suggest the receipt of Handshake packets implies the loss of Initial pac…

I don't think I understand this. Putting the requirement on INITIAL packets is a complete solution - it's not possible to decrypt a packet from any other packet number space without having first successfully decrypted and parsed at least one INITIAL packet. Am I missing something?

> My understanding has been that non-coalescing endpoints are considered first-class citizens; I would rather not revisit the requirements (or what we consider as reasonable) at this stage.

I'll agree with that, but non-coalescing endpoints aren't getting the performance they could get with coalescing regardless of this issue, so if they care about performance they can implement it.

> At 50% probability, this packet will be delivered to the server before the server's PTO timer expires for the server's first flight. That means that the server will be sending an ACK-only packet at…

Can you elaborate? If the server receives a retransmission of the ClientHello, and this server cares about the number of round trips involved in handshakes, then it should retransmit its ServerHello with this ACK because that's a pretty good signal that the ServerHello was lost.

> I'm not sure if this is the only problem that we have.

That's totally fair, but I have the same intuition with any ACK-based or path-validation-based solution because they're way harder to reason about.

> Anyways, to summarize, my hesitation against a "MUST pad all Initial packets" is that it is likely going to have unnecessary negative consequences on the handshake latency; my preference goes to mi…

Given that multiple widely deployed stacks operate this way today, I'm not convinced about these negative consequences being a problem in reality.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707403912

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Oct 2020 17:28:21 -0700

@ianswett @DavidSchinazi I understand that you are not having issues with your deployments (and I am glad to hear that). But that does not necessary mean that the specification is giving good advice. In fact, the draft is very lux about when you can retransmit during the handshake; what you have as an evidence is that _one_ pair of endpoints implementing particular strategy works fine. That's very different from making sure that any pair of endpoints implementing various "reasonable" strategies work fine.

> I don't think I understand this. Putting the requirement on INITIAL packets is a complete solution - it's not possible to decrypt a packet from any other packet number space without having first su…

Consider the case where all the following stands:
* server sends full-sized Initial packets, that gets dropped
* server transmits half-sized Handshake packets, that is not dropped
* client retransmits CRYPTO data when receiving a Handshake packet before obtaining Handshake keys (*)
* server retransmits Initial and Handshake packets when it receives duplicate CRYPTO data (**)

This set of behavior is permitted in status quo, and that causes ping-pong. This is one of the issue we are trying to resolve.

We can provide guidelines that would not cause ping-pong. But that's fragile when endpoints are incentivized to use signals other than ACKs to trigger retransmission. A more reliable way of preventing ping-pong is to prevent not only Initial packets but also Handshake packets and 1-RTT packets from passing through.

> If the server receives a retransmission of the ClientHello, and this server cares about the number of round trips involved in handshakes, then it should retransmit its ServerHello with this ACK because that's a pretty good signal that the ServerHello was lost.

I'm happy to point out that we already have that advice in the recovery draft. However, it is a MAY. I'm not sure if we can assume that all the QUIC servers to implement such strategy.

What I do not understand is the benefit of using a MUST rather than "SHOULD pad Initial packets," when using a MUST can hurt the performance of endpoints that obey to the specification.

Are you implying that some clients consider the server as alive when it receives the first packet that is decryptable, rather than when receiving the first packet that is bound to the connection (e.g., an undecryptable Handshake packet)?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707413125

---

From: Martin Thomson <[email-masked]>
Date: Mon, 12 Oct 2020 17:37:59 -0700

Aside from Kazuho, who seems to want *more* packets to be padded, it seems like the current trend is toward more strongly mandating padding of Initial to >=1200 bytes.  If we go with MUST, then I think we need to consider enforcement options at the client.  I was happy with SHOULD on the understanding that this is a niche problem, but if we want to engage with the enforcement question, then I can live with MUST.

The tweaks I'm seeing open are:

1. An exception for CONNECTION_CLOSE.  This is good.

2. An exception for non-ack-eliciting Initial packets.  This seems less necessary.  I can see how it is appealing, but all these exclusions are mounting up and it's getting complicated.

For David's benefit, the ACK-only carveout would allow a server to receive an Initial, send an ACK, then later send its own flight without having that ACK contribute too much toward the amplification limit.  If the server has to go elsewhere to get certificates or signatures, that early ACK might be useful to a client; it gets min_rtt primed with a good value.  The alternative is suppressing the ACK to avoid spending the 1200 bytes.

Maybe we could unify these if we designated CONNECTION_CLOSE as non-ack-eliciting.  Then we have an exception, and cause not to have the client enforce a minimum size on Initial-carrying datagrams.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707415698

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Oct 2020 17:44:47 -0700

@martinthomson FWIW, my first preference goes to "SHOULD pad Initial", as that causes least amount of changes to existing stacks.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707417400

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 12 Oct 2020 19:21:04 -0700

I think we might be losing the core problem here; I'll try to restate it, with the hope of driving towards a simpler resolution.

We currently require the client to confirm that the path from it to the server supports a PMTU of at least 1200 bytes (here):
```
A client MUST expand the payload of all UDP datagrams carrying Initial packets
to at least the smallest allowed maximum packet size (1200 bytes)
```

There is however no requirement in the spec that the server confirm that the path from it to the client be at least 1200 bytes. This can lead to a situation where asymmetric path MTUs can cause strange partial failures. This is not theoretical; we are seeing this at our servers with production traffic, where some paths (predominantly in Europe) in practice have a lower than 1200-byte MTU.

At a minimum, we need to add a requirement that ensures that the handshake fails under these conditions. This allows a client to fall back to TCP. This is a MUST-level design change if we don't want the connection to fail later or perform miserably later.

I propose the following two options:

1. Partial failures are more painful and problematic than complete failures. So, ideally, if the server's max packet size is not supported by the path, I would like to have _no_ packet be received by a client than some packets be received. A server that is concerned about performance with padding ACK-only packets from the server in the first flight can choose to never send an ACK-only packet until the path is validated -- the client cannot make forward progress until it receives some data from the server anyways, so receiving the ACK does not do much (yes, it reduces the number of retransmissions from the client, but that is a small side effect.) A concern here is that the change might be too disruptive, this late in the game.

2. We can simply state that our requirement is that the handshake MUST fail -- that the server MUST confirm that the path supports the minimum 1200-byte MTU, and cause the handshake to fail if not. We can then suggest ways in which the server can accomplish this: by padding all ack-eliciting Initial packets, by padding all packets until path validation, etc. This leaves room for implementers to do different things (as they might already be doing right now), and therefore will be a less disruptive change. This can cause wastage of packets, for example when implementers only pad Initial packets (we've seen this), but that's fine, since the handshake does fail eventually.

At this point, I'd like to go with option (2) above. Thoughts?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707443155

---

From: David Schinazi <[email-masked]>
Date: Tue, 13 Oct 2020 09:07:53 -0700

@kazuho thank you for the latest detailed explanation - I think I now understand what you're saying: padding INITIALs will successfully ensure that the handshake fails if the path doesn't support 1200 byte packets in both directions, but your concern is that a client that implements "always retransmit INITIAL CRYPTO when receiving undecryptable HANDSHAKE" will send quite a few retransmissions before giving up. I was confused because I thought there was a way the handshake could succeed, but now rereading the thread it all makes way more sense :) So in effect, we have two problems on our hands with the spec as it is today. On a path where server-to-client doesn't support 1200 byte packets
1. the handshake could succeed and then application data will fail in odd ways
2. many packets can be wasted during the handshake before it inevitably fails

I'm personally much more concerned about (1) but I agree that we should do something about (2). Because (1) can impact correctness (browsers cancel their TCP fallback when the handshake succeeds so we need to ensure that handshake success means requests can flow) and (2) only impacts performance, I'd be inclined to solve (1) with a MUST and (2) with a SHOULD.

Therefore, we can solve (1) by saying servers MUST pad INITALs (with @martinduke's exception for CONNECTION_CLOSE and @martinthomson's second exception for non-ACK-eliciting). Then we solve (2) by tweaking [recovery 6.2.3](https://tools.ietf.org/html/draft-ietf-quic-recovery-31#section-6.2.3) to say that endpoints SHOULD limit they number of times they use the MAY there per connection.

@janaiyengar I conceptually agree with your principled approach (option 2), but I do worry about how that would look in practice. Saying that the server MUST ensure that the path supports 1200-byte packets is non-enforceable, so it reduces to saying the server SHOULD pad INITIAL (with exceptions) and doesn't address @kazuho's ping-pong.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707846998

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 13 Oct 2020 13:40:53 -0700

@DavidSchinazi: Yes -- we have two issues.

I agree that `MUST ensure that the path supports 1200-byte packets` is unenforceable, but that's fine IMO, since a server not following this MUST is hurting itself. That said, in the interest of convergence, I'm ok with saying `MUST pad INITIAL packets unless it contains a CONNECTION_CLOSE frame`.

My goal was to avoid making a specific carve-out for ACK packets, since this is a performance-related carve-out as well. I'm also not confident that we understand how different client implementations might react to receiving ACKs but not receiving any other packet. I'm not sure how helpful the suggestion of "limit the number of times clients use the MAY" is without something more specific.

For the performance issue, I'm inclined to suggest that servers pad all packets (except those carrying CONNECTION_CLOSE) and avoid sending ACK-only packets until path validation is complete. The only issue here is delaying of ACKs, but (i) max_ack_delay does not apply to these ACKs, which makes any consequent large ack delay usable, and (ii) a client cannot make any forward progress with just the ACK anyway.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707996334

---

From: David Schinazi <[email-masked]>
Date: Tue, 13 Oct 2020 16:05:34 -0700

I don't feel too strongly about the carveout for non-ACK-eliciting packets, so I'm OK with us landing the new "MUST pad INITIAL except CONNECTION_CLOSE" with or without the ACK carveout

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708056367

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 13 Oct 2020 16:21:47 -0700

@DavidSchinazi 
> I don't feel too strongly about the carveout for non-ACK-eliciting packets, so I'm OK with us landing the new "MUST pad INITIAL except CONNECTION_CLOSE" with or without the ACK carveout

I would be fine with that _with ACK being included_ as one of the exceptions. Maybe the text would be something like "to prevent the handshake from succeeding when the MTU of the return path is below 1200 bytes, a server MUST pad Initial packets that carry CRYPTO data." I think in practice that'd be equivalent to "MUST pad Initial expect CONNECTION_CLOSE or ACK-only."

I think that [@janaiyengar's comment](https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707443155) was a good summary regarding the options we have: 1) prevent any packets from passing through 2) prevent the handshake from succeeding even though some packets could pass through.

It seems to be that we are converging on the latter option.

The downside of the latter option, as people have pointed out, is that we need to deal with ping-pong. I think there is an emerging cencensus to limit the number of times an endpoint can send probes before PTO expiration, the only remaining debate seems to be if that number should be one between @ianswett and @martinthomson.

My argument against "MUST pad all Initial packets" has been that it is more than what we need for preventing the handshake to succeed. In particular, as I have stated previously, the need to pad ACK-only Initial packets has been the concern. While it is true that endpoints can always coalesce (by withholding ACKs as @janaiyengar points out), I do not think there is reason to sacrifice performance of those who do not.

OTOH, endpoints have to prevent CRYPTO data from reaching the peer if it wants to prohibit the handshake from succeeding. That means that they have to _at least_ pad the packets that carry CRYPTO data. I think everybody will be fine with that being a requirement on Initial packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708061598

---

From: martinduke <[email-masked]>
Date: Tue, 13 Oct 2020 16:49:23 -0700

> My argument against "MUST pad all Initial packets" has been that it is more than what we need for preventing the handshake to succeed. In particular, as I have stated previously, the need to pad AC…

Part of the value of me is that we simplify the protocol by making the client and server requirements identical. As I argue in the PR reviews, I think the performance penalty is actually quite small.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708069734

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 13 Oct 2020 17:21:48 -0700

@martinduke 
> Part of the value of me is that we simplify the protocol by making the client and server requirements identical.

I'm not sure if the requirements are going to be identical when we try to allow servers send CONNECTION_CLOSE without padding.

That inevitably changes how the receiver would validate the requirements. On the server-side, validation happens before the server decrypts the Initial packets. On the client-side, to handle the frame-level exception, validation has to happen after decryption.

For the sender, people who prefer symmetry can have that, regardless of the outcome of this discussion.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708079375

---

From: martinduke <[email-masked]>
Date: Wed, 14 Oct 2020 10:26:34 -0700

The more I think about it, if there's no padding requirement for servers, there is no effective probing of the server->client PMTU at all. For example, my server implementation could send a packet each time it has an ACK to send or a single message from TLS. The server flight could be a stream of datagrams of a few hundred bytes. If they MUST pad initials, then there's at least one packet that's 1200B.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708547419

---

From: martinduke <[email-masked]>
Date: Wed, 14 Oct 2020 11:14:29 -0700

@kazuho If the padding requirement is a SHOULD, there is a requirement to decrypt anyway that can only be ignored on the first packet a server receives in a connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708574331

---

From: Martin Thomson <[email-masked]>
Date: Wed, 14 Oct 2020 14:50:44 -0700

s/Initial packets containing CRYPTO frames/ack-eliciting Initial packets/

That is simpler to my mind, and easier to implement.  It excludes ACK and CONNECTION_CLOSE, but includes CRYPTO and PING.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708679529

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 14 Oct 2020 15:00:47 -0700

@martinthomson 
> That is simpler to my mind, and easier to implement. It excludes ACK and CONNECTION_CLOSE, but includes CRYPTO and PING.

I can live with that, though I do not think it would make things simpler, assuming that endpoints are not going to validate the packet size during frame processing (would you?).

The downside is the all the Handshake packets carrying messages up to ServerFinished have to be padded, when the sender wants to coalesce a Initial(PING) to elicit an Initial ACK from the client (the optimization discussed in [the third paragraph of section 6.2.3 of the recovery draft](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.2.3-3)).

All that said, I would not oppose to using "ack-eliciting Initial packets," because that's what quicly does as a sender :-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708683413

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 14 Oct 2020 16:12:19 -0700

Anyways I've changed the PR (#4188) to "MUST pad ack-eliciting Initial packets." PTAL.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708708364

---

From: Marten Seemann <[email-masked]>
Date: Wed, 14 Oct 2020 20:41:15 -0700

I don't think it's ok to not pad ACK-only packets.
Recovery says:
> That is, the client MUST set the probe timer if the client has not received an acknowledgement for one of its Handshake packets and the handshake is not confirmed (see Section 4.1.2 of [QUIC-TLS]), even if there are no packets in flight. 

Thus, the client could receive an ACK for its ClientHello (in a packet that was not padded), and it will keep retransmitting the ClientHello indefinitely.

>From my understanding, the condition should be the following:
The server MUST pad *all* packets until:
* it has received an ACK for any Initial packet, OR
* until the Initial packet number space has been dropped.

Lazy implementations could only implement the latter condition.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708876280

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 14 Oct 2020 21:15:41 -0700

@marten-seemann Oh that's a very good point, and I actually think that you have found a problem that is unrelated to path MTU validation.

Let's think of two possible cases:
1.  A client might not be able to send ClientHello in one flight. It has to wait for the server to ACK the first flight to send the rest of the ClientHello. Post-quantum is a good example.
2. A server that spends time in generating ServerHello. It could be due to the server being slow (consider IoT), or because the server has to fetch a new per-SNI configuration from a different server.

To support case 1, servers have to send ACKs before generating ServerHello. To support case 2, servers have to postpone sending ACKs while generating ServerHello. Otherwise there would be a ping-pong.

Am I correct that this issue exists unrelated to the PMTU problem?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-708886097

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 15 Oct 2020 06:25:43 -0700

It looks like the main issue here is being resolved. I hope to such a degree that the changes can be included in the draft version going for IETF LC. I would appreciate if people can review the latest version of the  PR (#4188) so that we can get a good sense if this has agreement among the most active. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709322810

---

From: ianswett <[email-masked]>
Date: Thu, 15 Oct 2020 07:38:59 -0700

Based on @marten-seemann comment, I'm now favoring requiring both the client AND server to pad all Initial packets except CONNECTION_CLOSE.

However, I don't think the point raised is that problematic.  The client will keep PTO-ing a full-sized Initial containing a PING, but with exponential backoff.  This isn't that different from the situation when a connection is black-holed, though if the RTT sample was very small, this would be more aggressive than a new connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709370625

---

From: Lars Eggert <[email-masked]>
Date: Thu, 15 Oct 2020 07:46:29 -0700

Marking this as "design" based on the discussion and on #4188. (If we end up changing our minds about what we want to do, we can change the label.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709375648

---

From: David Schinazi <[email-masked]>
Date: Thu, 15 Oct 2020 14:43:02 -0700

> Thus, the client could receive an ACK for its ClientHello (in a packet that was not padded), and it will keep retransmitting the ClientHello indefinitely.

This doesn't make sense to me. If the client has received an ACK for its ClientHello, why is it retransmitting it? The fact that the packet containing the initial ACK from the server was padded or not doesn't change anything.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709606337

---

From: ianswett <[email-masked]>
Date: Thu, 15 Oct 2020 15:08:11 -0700

@DavidSchinazi I believe that comment is an error.  It should be "Thus, the client could receive and ACK for its PING sent in Initial ...."

As you said, once the ClientHello is acknowledged, a peer would not retransmit it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709615780

---

From: David Schinazi <[email-masked]>
Date: Thu, 15 Oct 2020 15:55:56 -0700

Ah, thanks @ianswett .

Going back to the principled approach I described earlier (use MUST to makes sure the handshake doesn't complete on bad asymmetric links; use SHOULD to prevent wasted bytes for handshakes that won't ever succeed), I'd say this can be solved by saying that servers SHOULD pad ACK-only initial packets.

That said, saying that the server MUST pad all initials except for ones that only contain CONNECTION_CLOSE frames would also be fine by me. Relatedly, that's what our implementation does.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709632078

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 15 Oct 2020 16:15:07 -0700

@ianswett 
> However, I don't think the point raised is that problematic. The client will keep PTO-ing a full-sized Initial containing a PING, but with exponential backoff. This isn't that different from the si…

Thanks to @ianswett for pointing out that we already have exponential backoff as a defense. As he pointed out in the other thread, our consensus that was established in #3546 was that PTO back off counter must _not_ be reset under these circumstances, because resetting the counter leads to a ping pong when the server takes time in generating ServerHello.

Based on that prior consensus, I think there are two ways forward:

* 1) Make no changes to #4188, ACK-only Initials do not have to be padded, because we can rely on that defense.
* 2) Reopen #3546 as the defense is too fragile, consider other methods, and also discuss if ACK-only Initials have to be padded.

My preference goes to option 1, though personally I would not oppose to reopening #3546. I do not want to see sudden spike of traffic when a server fails to instantly load a SNI-specific configuration.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709638022

---

From: ianswett <[email-masked]>
Date: Thu, 15 Oct 2020 16:29:04 -0700

Padding ACK-only Initial packets does make this ping-pong with exponential backoff much less problematic, so I still think padding all Initials to full size except CONNECTION_CLOSE is likely the option with the smallest number of potential issues.

I think the only way to reopen #3546 is to revisit how we prevent servers from being blocked by the amplification factor, ie #3162. That could make things simpler, but I don't believe the WG is up for it and we're pretty late in the game to be making major changes, so I'd advise against it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709642057

---

From: Martin Thomson <[email-masked]>
Date: Thu, 15 Oct 2020 16:34:24 -0700

Closed #4183 via #4188.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#event-3883965667

---

From: Martin Thomson <[email-masked]>
Date: Thu, 15 Oct 2020 22:02:02 -0700

Reopened #4183.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#event-3884564009

---

From: Martin Thomson <[email-masked]>
Date: Thu, 15 Oct 2020 22:10:46 -0700

Reopening this.  My mistake for not reading all of the discussion here.

Like others, I believe that #4188 is sufficient.  ACK frames will get through.  The client will send another packet on PTO (as the timer remains armed), but the client doesn't reset the PTO count, so it will back off to nothing.  As Ian says, it sends a little faster if the RTT is lower than its initial estimate, but it still continues.

The only other difference I see here is that the small packets containing ACK frames will tickle the idle timer on the client.  For clients that rely on the idle timeout alone to end connection attempts they will hang around until the exponential backoff exceeds the idle timeout.  (Our test client is like this, because we expect applications to set a timer for connection establishment.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-709784901

---

From: ianswett <[email-masked]>
Date: Fri, 16 Oct 2020 11:21:34 -0700

That note about the idle timer is important in practice, since it means implementations that use either should always race TCP and QUIC and/or set an application level timeout.

However, when sending 0-RTT requests, it's preferable to avoid replaying them if at all possible, so its preferable not to race QUIC and TCP until one is fairly sure the QUIC connection will fail.  Tickling the idle timer would mean the application level timeout is more likely to be hit, and those are usually fairly conservative.

FWIW, our implementation pads Initial ACKs and I don't intend to change that(nor does the PR require any change).

To me, there would be a nice simplicity changing the text to "You MUST pad all Initial packets except those containing CONNECTION_CLOSE.", but one could also document this potential issue with unpadded ACKs tickling the idle timer and let implementations choose what they want to do.  I do think it's worth doing one of these two actions, though.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-710369210

---

From: martinduke <[email-masked]>
Date: Fri, 16 Oct 2020 12:05:44 -0700

So we've merged #4188. Is the remaining issue whether or not to require padding of ACKs?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-710442243

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 16 Oct 2020 14:39:04 -0700

@ianswett 
> However, when sending 0-RTT requests, it's preferable to avoid replaying them if at all possible, so its preferable not to race QUIC and TCP until one is fairly sure the QUIC connection will fail.…

I'm not sure if I agree with that rationale.

By padding just Initial ACKs without padding Handshake ACKs, there's still chance that the client would receive packets.

As @janaiyengar noted [here](https://github.com/quicwg/base-drafts/issues/4183#issuecomment-707443155), if we want to prevent clients from seeing anything from the server when a sub-1200-byte path is being used, the requirement has to be to pad *all long header packets*.

But our consensus is not do that, and let some packets pass through (e.g., Handshake packets).

That means that a client, that wishes to run a handshake timer, has to be prepared for doing some packet processing *then* determine if it should extend the handshake timeout. With the current state of #4188 being merged, that determination can be made by seeing if you have received any CRYPTO data from the server.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-710665649

---

From: ianswett <[email-masked]>
Date: Fri, 16 Oct 2020 15:00:57 -0700

The idle timer is only extended if one can _process_ a packet, and Handshake packets can't be processed without receiving all the Initial CRYPTO data, so I don't believe there's an issue.  Additionally, the client won't receive Handshake ACKs if it hasn't sent Handshake packets, so that should not be an issue.

>From section 10.1 of transport-31:
> An endpoint restarts its idle timer when a packet from its peer is
   received and processed successfully.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-710672641

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 16 Oct 2020 16:50:13 -0700

@ianswett Thank you for elaborating. The timeout design that you explained is different from what I had in mind, but I agree that it's sensible.

Though, taking a step back, I am actually wondering if the use-case you described still exists (or has to exist).

In the early days of QUIC, before TLS 1.3 was being standardized, it did make sense to not happy-eyeball HTTP requests, because you do not want to let servers process both of the requests that have been happy-eyeballed.

But with the standardization of TLS 1.3 and 425 status code (RFC 8470), the situation has changed. A server is now required to ask clients replay non-idempotent requests.

I think we can, and should, rely on 425, considering that HTTP/3 is something new. Server administrators that enable HTTP/3 can make sure (and should make sure) that they would not receive replays of non-idempotent request, rather than relying on the browser making probablistical guesses.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-710700294

---

From: ianswett <[email-masked]>
Date: Sun, 18 Oct 2020 11:12:18 -0700

Based on people I've talked to, 425 may not be widely implemented.  Additionally, there are scenarios where identifying replay is quite difficult.  But even so, the risk is not overly high in most cases.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-711348732

---

From: ianswett <[email-masked]>
Date: Mon, 19 Oct 2020 09:03:27 -0700

I realized that clients could mitigate the issue of unpadded ACKs extending the idle timeout by not extending it for unpadded Initial packets, at least until a full-sized packet has been received.  If we stick with the current text, I think our implementation will likely do just that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-712267879

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 19 Oct 2020 17:27:48 -0700

We ought to be able to rely on exponential backoff of PTO and idle timeout for correctness, which we agreed as the goal here. Is there a reason what we have now will not work?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-712514468

---

From: ianswett <[email-masked]>
Date: Mon, 19 Oct 2020 17:41:29 -0700

Doubling the amount of time for idle timeout to occur is a bummer, so I think it's worth documenting the potential issue if we're not going to change the normative text?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-712518377

---

From: Christian Huitema <[email-masked]>
Date: Tue, 20 Oct 2020 12:07:12 -0700

Could someone restate the "emerging consensus" ? I am reading the thread, and under the impression that the consensus is "PR #4188 is sufficient". Is that the actual emerging consensus?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-713076969

---

From: ianswett <[email-masked]>
Date: Tue, 20 Oct 2020 12:12:42 -0700

Yes @huitema, that is the emerging consensus.

I wrote #4252 to document a potentially non-obvious effect of not padding ACK only packets, but it doesn't seem like people think that's necessary, so I should probably just close it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-713079861

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 20 Oct 2020 13:27:39 -0700

We merged https://github.com/quicwg/base-drafts/pull/4188 and as per Lar's email on the list, we'll use IETF last call to judge things. So I'm closing this.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#issuecomment-713119513

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 20 Oct 2020 13:27:39 -0700

Closed #4183.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/4183#event-3900651648