Subject: [quicwg/base-drafts] Flow control for post-handshake CRYPTO messages (#1834)

From: Victor Vasiliev <[email-masked]>
Date: Thu, 04 Oct 2018 16:47:29 -0700

Consider the following situation.  The client sends a 0-RTT handshake.  The server accepts the handshake, installs the 1-RTT keys and sends its ServerHello...Finished flight.  The client can now send a bunch of post-handshake messages to the server which the server cannot process until it receives ClientFinished, so it has to buffer them.  We should bound that buffer somehow.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834

---

From: David Benjamin <[email-masked]>
Date: Thu, 04 Oct 2018 16:55:20 -0700

(Client post-handshake messages don't really exist today, but they might in the future, e.g., with https://tools.ietf.org/html/draft-wood-tls-ticketrequests-00.)

Another incarnation of this problem is if the client receives a huge window of CRYPTO frames into the future but not the first byte, so it can't process anything yet. With STREAM frames, we have flow control. With CRYPTO frames during the handshake, we know the handshake itself is bounded. Post-handshake, we don't know a priori the server won't send lots and lots of tickets.

Of course, realistically, it won't send that many in a row, so one could get away with just picking a bound. But this is much more of an ad-hoc bound than the handshake. It also may send lots of tickets over the course of a connection, so the bound is in how far ahead it can get, rather than the handshake which can impose a stronger total bound. This then starts looking an awful lot like flow control...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-427205433

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 04 Oct 2018 19:14:57 -0700

> The client can now send a bunch of post-handshake messages to the server which the server cannot process until it receives ClientFinished, so it has to buffer them. We should bound that buffer somehow.

Isn't the expected behavior of the server to drop the 1-RTT packets containing the post-handshake messages until it receives the ClientFinished (that activates the 1-RTT read key)?

To put it another way, my understanding is that you _can_ have any size of buffer you want to better cope with packet reordering, but that happens per-packet rather than per-frame (because it's not just the post-handshake messages that cannot be trusted prior to receiving ClientFinished), and that the buffer can be arbitrary sized because you can drop it to trigger a retransmit.

Am I missing something?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-427225638

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 04 Oct 2018 19:30:08 -0700

@davidben 
> Another incarnation of this problem is if the client receives a huge window of CRYPTO frames into the future but not the first byte, so it can't process anything yet. With STREAM frames, we have fl…

I agree that the problem exists.

However, in my view, the memory exhaustion issue does not get resolved just by having a window for the CRYTO frames. My understanding that most TLS stacks buffers the octets of a handshake message until it receives a complete one. That means that the TLS stack is expected to buffer at most 16MB of junk even if we introduce window to the CRYPTO frames.

Considering that, I think it would be preferable to at first discuss introducing bounds for the TLS messages in the TLS working group, then consider how to apply that to QUIC. Until then, each QUIC stacks and TLS stacks can (continue to) have their own internal limits.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-427227807

---

From: David Benjamin <[email-masked]>
Date: Thu, 04 Oct 2018 19:37:06 -0700

I don't think that's an accurate reduction.

TLS stacks usually impose ad-hoc limits tighter than the maximum 2^24, sometimes as a function of the maximum certificate size the caller configures. It's unfortunate that it's ad-hoc but different consumers do indeed tolerate different certificate sizes. It's plausible to do that analysis per-message and to do it per-flight since the set of messages in a flight is different.

Post-handshake is very different. The pattern of messages is no longer fixed. Rather it is a property of how the application uses TLS. Indeed the QUIC spec even calls out applications choosing to send tickets at various points during the connection. Thus TLS-level limits, ad-hoc or spec'd, are not sufficient in this case. QUIC needs to do its share of the work.

(There's a reason I dislike post-handshake messages so much. :-) People always forget this complexity.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-427228821

---

From: David Benjamin <[email-masked]>
Date: Thu, 04 Oct 2018 19:45:20 -0700

Re the client Finished case, I guess that pulls in the mess of stuff around there. The spec is a little unclear what a server does in between server and client Finished. In particular, if doing 0-RTT the server needs to at least be willing to send half-RTT data. This text suggests that, since there's a binder, processing out of order is okay. (Unless I'm reading it wrong?)
https://tools.ietf.org/html/draft-ietf-quic-tls-15#section-5.7

I agree that, if you believe the server just shouldn't process 1RTT packets before client Finished at all, the buffering issue is solved. Though it does leave it unable to read the ACKs to half-RTT data which is differently weird. :-)

I think the general post-handshake issue is probably the more isolated instance. They're the same problem in terms of buffering, but one has a confounding factor around whether it's actually an issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-427230016

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 04 Oct 2018 20:16:36 -0700

@davidben 
> https://tools.ietf.org/html/draft-ietf-quic-tls-15#section-5.7

Oh I missed it. Thank you for pointing that out.

> I agree that, if you believe the server just shouldn't process 1RTT packets before client Finished at all, the buffering issue is solved.

I had assumed that TLS stacks always expose the 1-RTT read key to the QUIC stacks acting as a server when it receives ClientFinished. I'd be curious to know if there are TLS stacks that takes an alternative approach.

> Though it does leave it unable to read the ACKs to half-RTT data which is differently weird. :-)

I assume that issue is unrelated to 0-RTT, right?

IIUC the advise to the QUIC stacks is that until you know that the peer can process a packet, you might want to coalesce the packet you send with another packet that uses a lower-epoch key; then you can have a _weak_ confirmation that the packet has reached the peer. Having such signal is beneficial to not give false information to the congestion controller though the loss recovery code needs to assume that the packet might have been lost, because a stack is allowed to drop a packet that is was unable to decrypt.

> I think the general post-handshake issue is probably the more isolated instance. They're the same problem in terms of buffering, but one has a confounding factor around whether it's actually an iss…

I agree. I think it is preferable to fix the issue.

Maybe I should rephrase my argument to that there should be limits in both layers (i.e. limit on the post-handshake message size in the TLS layer, and the window size in the QUIC layer, because having limit only on one layer does not prevent excessive buffering on the other layer. The two limits could either be a shared single value, two values shared using one extension, or two values shared using different mechanisms.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-427234124

---

From: Martin Thomson <[email-masked]>
Date: Wed, 24 Oct 2018 21:37:02 -0700

I'm very much inclined to leave this "unfixed", but to leave a warning that sending large post-handshake messages risks connection failure.

> I had assumed that TLS stacks always expose the 1-RTT read key to the QUIC stacks acting as a server when it receives ClientFinished. I'd be curious to know if there are TLS stacks that takes an al…

That was what I had assumed too, until I thought through the nasty interactions with partial Handshake flights and 0-RTT.  That is, the TLS stack makes secrets available at the point that it would normally be installing them for its own use.  In that way, a server wouldn't install the 1-RTT read keys until it had processed the finished from the client, exactly as you say.

My current patch (which is still under review) releases secrets as soon as possible to avoid the problem where the server can't read Handshake acknowledgements because it is receiving 0-RTT.  The main consequence of that is that you need a separate API where the TLS stack tells the QUIC stack what it is willing to send and receive (well, I had a ton of fun fixing tests as well, but that's just because we were a little lazy with certain things in the past).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-432911920

---

From: David Benjamin <[email-masked]>
Date: Thu, 25 Oct 2018 16:32:31 -0700

Right, this issue originally came up as we were thinking through key availability vs key install times. The problem is, as you note, it's weird to allow sending data you cannot read ACKs for. (Conversely, it's weird, perhaps fatally so, to allow receiving data you cannot send ACKs for.) Though it's not the only scenario. The other side may send a burst of tickets.

> I'm very much inclined to leave this "unfixed", but to leave a warning that sending large post-handshake messages risks connection failure.

The point is that's not a sufficient warning. This is either a problem or a nitpick depending on the solution.

Sending an overly large message, in general, risks connection failure. TLS stacks bound the size there based on some ad-hoc analysis that no sane NewSessionTicket (or other message) will ever exceed such-and-such bytes based on what goes in it. TLS stacks typically buffer up a message before processing, so it makes sense that the TLS stack is responsible for buffering it.

But that's distinct from sending *many* post-handshake messages. TLS over QUIC must also account for that due to the lack of flow control. TLS over TCP does not have this problem because TCP provides flow control. If I try to send a million 100-byte TLS messages over TCP, I'll quickly run into TCP's flow control.

QUIC, however, doesn't have such flow control for CRYPTO frames. Instead it relies on a limit from the crypto protocol:
https://tools.ietf.org/html/draft-ietf-quic-transport-15#section-10.4

Although the TLS stack natively bounds *a single message* rather than *total data*, it is easy to compute a total limit during the handshake because there are only a fixed handful of messages. Thus the advice there works.... *except* for post-handshake messages. Thus the issue.

Realistically, it's probably fine to handwaive and say:
* The bound is not total data ever sent at the encryption level, but amount of data ahead of the read position.
* Pick some ad-hoc limit for post-handshake stuff, enough to fit a bunch of NSTs.
* Yeah, don't send too many NSTs in a row. You can send a bunch of them over the lifetime of the connection, but space them out a bit so you're sure the receiver isn't waiting on the first byte, having buffered the rest up.
* Probably you won't end up in a situation where the first NST packet keeps getting lost and as you queue up more and more NSTs over the lifetime of the connection. That would be strange.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-433239181

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 25 Oct 2018 19:02:22 -0700

> That was what I had assumed too, until I thought through the nasty interactions with partial Handshake flights and 0-RTT. That is, the TLS stack makes secrets available at the point that it would n…

Thank you for pointing that out. I forgot about that exception. So while that is the only exception picotls has in terms of key installation timing, I agree that it's merely about having one exception or two...

> But it also smells weird, so perhaps proper flow control (toss post-hs stuff onto stream 0?) would be better instead?

I am totally fine with handwaving, but in case we are to have flow control, I'd prefer having flow control for CRYPTO streams in all epochs. They could all use a fixed-size window that would be defined in the specification. I prefer that because IMO having flow control for every stream (i.e. all the ordinary streams + all the 4 CRYPTO streams) is a simplification.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-433261852

---

From: Subodh Iyengar <[email-masked]>
Date: Mon, 29 Oct 2018 22:54:38 -0700

FWIW we had a similar discussion about stream 0 flow control in https://github.com/quicwg/base-drafts/issues/1252

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1834#issuecomment-434180857