Subject: Re: [quicwg/base-drafts] loss of only two packets can lead to an unrecoverable situation (#2267)

From: Christian Huitema <[email-masked]>
Date: Sun, 06 Jan 2019 23:16:46 -0800

I think this issue is an indirect justification for implicit ACK. The server may have dropped the handshake key, but it can send and receive 1-RTT packets. So can the client. If the client sends an 1-RTT packet after sending the CFIN, it should receive an ACK from the server. At that point, it knows there is no point in retransmitting the CFIN. If neither the 1-RTT not the CFIN handshake are acked, both can be repeated. If the handshake CFIN was received but the ACK lost, as @marten-seemann envisages, the server may not be able to process handshake packets anymore, but it can receive and ACK the 1-RTT packet.

Of course, that falls apart if the handshake and 1-RTT packets are coalesced. Maybe we need a warning there... 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451842773

---

From: Christian Huitema <[email-masked]>
Date: Sun, 06 Jan 2019 23:23:46 -0800

@martinthomson: MSL in TCP is determine by the practical observations of how long packets can reside in networks. There are measurement papers describing that. At least 30 seconds has been observed in practice, for example due to routers freezing and then restarting, or other bugs of that nature. Since TCP has by default no crypto, the very old packet could be accepted by chance in a new connection and cause havoc. Hence the use of the MSL timer, especially in the FIN-WAIT state.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451843958

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 06 Jan 2019 23:24:07 -0800

@huitema 
> I think this issue is an indirect justification for implicit ACK.

I am not sure if it is. To me, it seems that the fact that we have an unrecoverable situation for lower epochs and if we need to shutdown the connection in such a situation are different.

Can't the endpoints simply continue communicating using higher epochs, regardless of if there was something not ACKed when they drop the context of lower epochs?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451844018

---

From: Christian Huitema <[email-masked]>
Date: Sun, 06 Jan 2019 23:26:51 -0800

@kazuho, I think we agree. If the endpoints can continue communicating at epoch=4, there is no reason to drop a connection because of an ACK or timer issue at epoch=3.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451844505

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 07 Jan 2019 00:00:47 -0800

Then, I'd assume that there is nothing we need to address?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451851230

---

From: Marten Seemann <[email-masked]>
Date: Mon, 07 Jan 2019 00:05:38 -0800

> I think this issue is an indirect justification for implicit ACK. The server may have dropped the handshake key, but it can send and receive 1-RTT packets. So can the client. If the client sends an 1-RTT packet after sending the CFIN, it should receive an ACK from the server. At that point, it knows there is no point in retransmitting the CFIN.

I agree, but it's better if the server send an ack-eliciting 1-RTT packet right after the handshake completes, since there's no uncertainty on the server's side if the client already has 1-RTT keys. In the stream-0 design team we once had a proposal fo a HANDSHAKE_DONE frame (a 1-byte ack-eliciting frame that is delivered reliably). I consider this a much cleaner solution than relying on timers which might or might not be chosen appropriately to work with a particular loss pattern.

Dropping handshake keys, and moving forward from the crypto mode in the loss recovery (so we can use the PTO timer instead of the crypto timer), is then really straightforward.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451852213

---

From: ianswett <[email-masked]>
Date: Tue, 15 Jan 2019 14:33:47 -0800

@kazuho I think we should do something, because in this case the CFIN would be in-flight, and that would cause only the handshake timer to be active in the recovery draft, even if the CFIN is no longer ACK-able.

Even without HANDSHAKE_DONE, anytime you receive a 1-RTT ACK, you can assume all Handshake data has been received and no longer needs retransmission, so you can drop the Handshake send keys.  That's a unilateral action that the peer doesn't need to be aware of I believe, as long as it's only the send keys being dropped and not receive.

We could advise coalescing a 1-RTT PING with a Handshake timeout retransmission if 1-RTT send keys are available and also advise to consider all Handshake data delivered once a 1-RTT ACK is received and processed?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-454578037

---

From: Christian Huitema <[email-masked]>
Date: Tue, 15 Jan 2019 17:56:33 -0800

@ianswett I don't think that you can drop the Handshake send keys unilaterally just because you received a 1RTT ACK. You can safely mark all the locally sent handshake packets as acknowledged, and you know that they will never need to be retransmitted. But if you receive a repeated handshake packet from the peer, it means that the peer is not implementing implicit ACK, and also is expecting an ACK on the repeated handshake packet. If you don't send the ACK, the peer could repeat the handshake packet N times and drop the connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-454621847

---

From: Christian Huitema <[email-masked]>
Date: Tue, 15 Jan 2019 23:13:53 -0800

If we want to drop handshake keys, it has to be coordinated with the peer in a standard way. For example, we could update the spec and say that handshake keys are expected to be dropped after the first 1RTT ack has been received (client side) or the first 1RTT-protected packet has been received (server side). That will work because it compels both peers to do implicit acks. But without stating that in the standard, no, you cannot do it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-454676071

---

From: ianswett <[email-masked]>
Date: Wed, 16 Jan 2019 07:56:18 -0800

@huitema Thanks for the correction, I keep forgetting about sending ACKs.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-454831297

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 16 Jan 2019 20:17:55 -0800

FWIW, current TLS draft states: _After all CRYPTO frames for a given encryption level have been sent and all expected CRYPTO frames received, and all the corresponding acknowledgments have been received or sent, an endpoint starts a timer._

To me this sounds safe and that there is nothing we need to fix.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455036849

---

From: ianswett <[email-masked]>
Date: Thu, 17 Jan 2019 19:56:17 +0000 (UTC)

@kazuho I believe in the case Marten is concerned about "After all CRYPTO frames for a given encryption level have been sent " is not true, because the acknowledgements are not received?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455310438

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 17 Jan 2019 15:36:03 -0800

@ianswett Seems to me that that is the case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455372082

---

From: David Schinazi <[email-masked]>
Date: Fri, 18 Jan 2019 15:50:03 -0800

Having thought about this some more, I think that the HANDSHAKE_DONE proposal might be our best way forward.

Knowing when to stop sending ClientFinished/Ack is a variant of the two generals problem, meaning it's actually impossible to know when to stop sending over a lossy network without out-of-band information.

There are two ways to solve this. One is timeouts, similar to what TCP does on graceful connection close. However timeouts aren't great, because they're by definition always too short or too long. In this case, we can do better by using an explicit event that would trigger key discarding. We even have a suitable stream of out-of-band information: our 1-RTT protected packets.

One could consider having the client treat the receipt of any valid 1-RTT packet as confirmation that the ClientFinished was received, but that doesn't work because the server is allowed to send non-sensitive application data before validating the ClientFinished.

Having the server send a retransmittable HANDSHAKE_DONE frame as soon as it receives ClientFinished solves this issue. The server discards its handshake keys when it sends HANDSHAKE_DONE and the client discards them when it receives it.

I personally find this preferable to a short timeout (which can cause an infinite send) and a long timeout (which costs resources).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455723789

---

From: Martin Thomson <[email-masked]>
Date: Sun, 20 Jan 2019 20:16:52 -0800

Having thought about this some more myself, I'm pretty happy with what we have.  We don't throw away keys until they aren't needed (as @kazuho points out), plus a little extra.  So I think that this is a case of "phew, false alarm".

If anything, we should be discussing whether the timer is needed at all.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455943537

---

From: Marten Seemann <[email-masked]>
Date: Sun, 20 Jan 2019 20:33:49 -0800

I don’t think this is the case. In the scenario I described, the server sent and received all CRYPTO frames, and it sent acknowledgments for all the packets. This satisfies the condition for dropping the keys (since it says *all the corresponding acknowledgments have been received OR sent*). 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455945590

---

From: Christian Huitema <[email-masked]>
Date: Sun, 27 Jan 2019 18:37:53 -0800

I just completed a PR in Picoquic to fix this issue. What it does:

1) Client side, implicit ACK of the first Initial Packet if the handshake key is received. After that, apply a special parser for Initial frames (see point 4 below)
2) Server side, move to a "ready" state when the 1st 1-RTT packet from client is received; as part of that move, implicit ACK of Initial and Handshake packets sent by the server.
3) Client side, move to the "ready" state when first ACK of 1-RTT packet is received from server. This triggers implicit ACK of Initial and Handshake packets sent by the client.
4) In Ready state, use a specific parser for Initial and Handshake message. Effectively ignore all frames in these packets, but if there is an ACK eliciting frame, schedule an ACK.

This removes a bunch of corner cases, such as spurious retransmit of Initial packets in the middle of a migration. It also closes an avenue for DOS against established connections, since Initial packets cannot anymore change the state of the connection once the handshake is complete.

Well, not quite all attacks: Initial packets can still trigger an ACK that might cause an allergic reaction at the peer. But then, the attacker could also send spoofed ACK in Initial packets directly to the peer, so this is not exactly a new attack. Plus, if the peer adopts the defenses outlined above, the spoofed ACK will have no side effect.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-457982113

---

From: Marten Seemann <[email-masked]>
Date: Sun, 27 Jan 2019 18:44:05 -0800

> Server side, move to a "ready" state when the 1st 1-RTT packet from client is received; as part of that move, implicit ACK of Initial and Handshake packets sent by the server.

This will only work if the application protocol actually makes the client send any data after establishing a new connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-457982877

---

From: Christian Huitema <[email-masked]>
Date: Sun, 27 Jan 2019 19:25:20 -0800

@marten-seemann yes, that's a potential issue. The clients that I see in test seem to always send something in practice, but that's not guaranteed. "ACK of the CFIN" would do it, which I guess is also what Kazuho suggested. Suggested text?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-457988235

---

From: Marten Seemann <[email-masked]>
Date: Sun, 27 Jan 2019 20:41:54 -0800

The issue is that the ACK for the CFIN is lost.

I think the cleanest solution here is making the server send a HANDSHAKE_DONE frame in 1-RTT once the handshake is complete. At this point, it can immediately drop the Handshake keys. The HANDSHAKE_DONE is retransmitted according to normal 1-RTT retransmission rules, and as soon as the client receives it, it also drops Handshake keys and switches loss recovery to 1-RTT mode.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-457997260

---

From: Christian Huitema <[email-masked]>
Date: Sun, 27 Jan 2019 20:53:08 -0800

Yes, adding an handshake done frame would be a very good idea

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-457998594

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 27 Jan 2019 21:03:33 -0800

@marten-seemann 
> I think the cleanest solution here is making the server send a HANDSHAKE_DONE frame in 1-RTT once the handshake is complete. At this point, it can immediately drop the Handshake keys.

That's much more aggressive than what is required now.

At the moment, we keep the handshake keys 3 PTO after the client learns that all the handshake data have been received by the server (see 
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455036849).

I think it makes sense to use a 3 PTO timer to avoid having negative side-effects to the congestion controller, and I do not understand why we need an explicit signal (Handshake_Done).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-457999885

---

From: Nick Banks <[email-masked]>
Date: Sun, 27 Jan 2019 21:08:29 -0800

My vote would be to have an explicit signal instead of a timer based solution. The loss recovery / congestion controller could use the exact same signal to fix any necessary state. I like the Handshake Done solution.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458000516

---

From: Marten Seemann <[email-masked]>
Date: Sun, 27 Jan 2019 21:35:45 -0800

@kazuho 

> At the moment, we keep the handshake keys 3 PTO after the client learns that all the handshake data have been received by the server (see #2267 (comment)).

See https://github.com/quicwg/base-drafts/issues/2267#issuecomment-455945590 why that doesn't work.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458004079

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 27 Jan 2019 22:48:16 -0800

@marten-seemann I disagree, as I have stated in 
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-451844018.

Current design guarantees that both endpoints will have 1-RTT read and write keys 3-PTO before the handshake keys are dropped (I think we can agree on this).

It seems to me that you consider loss of an ACK for a Handshake packet should lead to the termination of the connection. Would you mind clarifying why? Because I do not think that's necessary.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458015764

---

From: Marten Seemann <[email-masked]>
Date: Sun, 27 Jan 2019 23:00:51 -0800

> Current design guarantees that both endpoints will have 1-RTT read and write keys 3-PTO before the handshake keys are dropped (I think we can agree on this).

Both endpoints have the 1-RTT keys, but the client doesn't know that the server has them, since the ACK for the CFIN was lost. For the client, this situation is indistinguishable from the CFIN itself being lost, so it has to keep retransmitting it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458018063

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Mon, 28 Jan 2019 13:17:51 +0000 (UTC)

I encountered this issue while testing quinn during interop.  client kept retransmitting Handshake but server had already dropped handshake key.  The problem is that server starts timer too soon; client has no way to know that handshake is received by server and it started timer.
My preference is to have a explicit signal from server.  It is much better to doing something complicated using ACKs.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458127755

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 28 Jan 2019 23:53:39 -0800

@marten-seemann 
> Both endpoints have the 1-RTT keys, but the client doesn't know that the server has them, since the ACK for the CFIN was lost. For the client, this situation is indistinguishable from the CFIN itse…

That's true, however I am not sure if introducing a new frame (that needs to be delivered reliably) is the correct solution. At least for my implementation, consulting if an ACK for 1-RTT packet has been received each time a Handshake packet is deemed lost is easier than reliably transmitting a HANDSHAKE_DONE frame and using it's receipt as a signal.

Having that said, _if_ we are going to depart from the current approach, possibly by introducing a HANDSHAKE_DONE frame, I think it's worth reconsidering when we drop the Initial keys.

Current text requires endpoints to drop Initial packets as soon as they know that the peer knows the Handshake keys. The downside of this approach is delivery of Initial packets and their RTT is sometimes lost.

If we are to have an explicit signal for indicating the end of the handshake, we could delay dropping not only the Handshake keys but also the Initial keys until that signal is being emitted.

The design would be:
* each endpoint emits HANDSHAKE_DONE packet when all the CRYPTO data sent using Initial and Handshake packets are acknowledged
* an endpoint would retain it's Initial and Handshake keys until it sends HANDSHAKE_DONE frame and also receives HANDSHAKE_DONE frame

This is simpler than the current design. There's no fear of key being dropped having negative impact on loss recovery / congestion control. There's only one event for dropping the keys (instead of two). And the logic can be exactly the same on the server and the client.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458441125

---

From: Nick Banks <[email-masked]>
Date: Wed, 30 Jan 2019 00:19:09 -0800

Perhaps another way to do this, similar to HANDSHAKE_DONE, would be to have a KEY_RETIRE frame that be used to throw the Initial and Handshake keys away individually. This might be slightly better than the proposed HANDSHAKE_DONE method proposed above because the Initial key could be thrown away earlier than the end of the handshake.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458852679

---

From: Martin Thomson <[email-masked]>
Date: Wed, 30 Jan 2019 18:24:11 -0800

Tokyo conclusion: we will explore the addition of the proposal @nibanks had in https://github.com/quicwg/base-drafts/issues/2267#issuecomment-458852679.  This will allow us to address the multiple key update issue, and make the implicit acknowledgment mess transitioning from Initial to Handshake (i.e., #1951) together with the one solution.

Tentatively, the signal we're looking for is a frame that says "I have emplaced read and write keys for epoch N".  Once a peer has both sent and received this frame, then they can discard keys for earlier epochs (and indeed SHOULD for Initial).

We might also use this as a signal to drive the solution to #2309.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2267#issuecomment-459190108