Subject: [quicwg/base-drafts] Single Stream of Bytes for CRYPTO Frames (#1592)

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 10:31:25 -0700

Currently, the spec states:

> There is a separate flow of cryptographic handshake data in each encryption level, each of which starts at an offset of 0. This implies that each encryption level is treated as a separate CRYPTO stream of data.

This means you must track set of bytes in each encryption level (Initial, 0-RTT, Handshake and 1-RTT) as a separate stream and manage them accordingly; separate outstanding sent byte ranges, recovery ranges, acked ranges.

Additionally, it makes it a little unclear where one 'stream' stops and another 'starts'. For instance, in the 0-RTT case, the client (is supposed to) sends EOED with 0-RTT key, immediately followed by (ideally in the same UDP datagram, but not required) FIN with the Handshake key. If the 0-RTT EOED is lost, but the FIN isn't it could look like the client just skipped EOED. Instead, if they were all in the same stream, with FIN's offset starting where EOED left off, it would be easier to track this.

For these reasons, I propose each CRYPTO stream in each encryption level's offset starts where the previous level stopped.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592

---

From: Marten Seemann <[email-masked]>
Date: Fri, 20 Jul 2018 11:53:15 -0700

Wouldn’t that be equivalent to going back to a single CRYPTO stream, which changes encryption levels at various offsets?
If I remember correctly, this was one of the main pain points with the old design, so I’d be unhappy to revert this change.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406695557

---

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 11:55:29 -0700

I wouldn't say we had exactly that, as now we are much more explicit with encryption level (as there are 4 different ones). Why do you say that was a pain point? I feel that is a much cleaner and explicit design to implement.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406696243

---

From: ianswett <[email-masked]>
Date: Fri, 20 Jul 2018 12:06:46 -0700

I have the single stream approach implemented today, and it's ugly on the send side, because I need a very weird datastructure to determine what was sent at what encryption level and make sure there's no cross-encryption level bundling/etc.

With a stream per level, it's much clearer what's going on, and I can re-use most of the standard stream code.  I need to actually implement this, so maybe there are some issues I'm not anticipating, but I know the single stream approach is ugly.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406699167

---

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 12:10:51 -0700

Using a standard stream for this will bring in 4 times the overhead. What is your FC limit for each stream? What tracking variables do you store for each? Having one stream with 4 offsets (Initial is implicitly offset 0) is a much more efficient design, IMO. I also have the single stream approach implemented (because that's what I thought  we were supposed to do, before I reread the spec) and I have a much simpler implementation that using 4 different streams.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406700137

---

From: Marten Seemann <[email-masked]>
Date: Fri, 20 Jul 2018 12:26:07 -0700

IMHO, being able to reuse the existing stream code (including STREAM frame reassembly etc.) is the one of most important advantage of this change. Special-casing stream 0 to work with changing encryption levels required a significant amount of additional code.
I’m not worried about the overhead of this at all, after all, it’s just 4 streams, and you can remove them after the handshake completes (some of them even earlier, if you want to take optimizations to the extreme). 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406703559

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 20 Jul 2018 12:27:11 -0700

@marten-seemann, +1. I, too, think that the overhead is small and temporary.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406703820

---

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 12:29:04 -0700

You are assuming an attacker completes the handshake. Your server might use 4 times the memory for the same attack, if you use 4 different streams. As for code re-use, it's all implementation dependent. My stream code and reassembly code are already separate, so crypto can already reuse the same reassembly code.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406704254

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 20 Jul 2018 12:35:17 -0700

What is the attack, though?  Resource exhaustion?  So a handshake uses four (by the way -- why four? I thought there were three encryption levels) streams.  Once handshake is complete, the client can open a hundred streams with a single packet!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406705561

---

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 12:39:04 -0700

Yes, the attack in this case would be resource exhaustion. There are 3 packet number spaces, but 4 encryption levels (Initial, 0-RTT, Handshake, 1-RTT). Each encryption level can be used to send CRYPTO frames (EOED uses 0-RTT). As for the number of streams a client can open, that is controllable by the server. It can choose to give less while under attack. Additionally, completing the handshake to get those extra streams requires additional resources on the attacker side, which might make the attack less attractive.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406706348

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 20 Jul 2018 12:46:16 -0700

Perhaps the recommendation, then, is not to allocate a lot of resources until the handshake is completed?  There are ways to minimize memory usage.  The server has other state in addition to the stream state, so the memory increase is not fourfold.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406708035

---

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 12:53:59 -0700

I agree this isn't the only memory being allocated, but it is one that is directly controllable/influenced by the attacker. If they attack your flow control for each 'stream' they can attempt to maximize your memory usage and then move on to a new connection to do the same.

With a single stream (and associated FC window/buffer) it seems a lot easier to manage this attack. It is still doable with 4 streams, but requires some cross stream logic, which I doubt anyone already has.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406709818

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 20 Jul 2018 13:03:30 -0700

@nibanks, you've mentioned your own implementation twice, but one implementation's preference is a dubious argument for a specification change.

I don't claim to understand all intricacies of the CRYPTO frame design (we haven't gotten around to implementing draft-13).  Nevertheless, the current designed has been "Design Teamed" and hummed upon.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406712082

---

From: Nick Banks <[email-masked]>
Date: Fri, 20 Jul 2018 13:08:30 -0700

I know the design team suggested the CRYPTO frame design. I was part of the design team. At the time, in the theoretical sense it all made sense. After implementing it, I have seen some issues. Also, I agree one implementation experience isn't enough to change things outright, but it's enough to open an issue and have the discussion.

I'm not arguing to undo everything. I am just trying to consolidate things I see as bloated and overly complex. I very much like the design of multiple encryption levels. All I am arguing for is a change in how each starts its offset.

If folks still generally believe the current way we have offsets is better, then so be it. That is not my experience, so that's why I opened the issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406713278

---

From: ekr <[email-masked]>
Date: Sat, 21 Jul 2018 06:59:37 -0700

I prefer the "starts at zero" structure. I found it much easier to just
have separate flows and then not have to worry about data coming on one
encryption level that had offsets appropriate to another encryption level.

On Fri, Jul 20, 2018 at 1:08 PM, Nick Banks <[email-masked]>
wrote:

> I know the design team suggested the CRYPTO frame design. I was part of
> the design team. At the time, in the theoretical sense it all made sense.
> After implementing it, I have seen some issues. Also, I agree one
> implementation experience isn't enough to change things outright, but it's
> enough to open an issue and have the discussion.
>
> I'm not arguing to undo everything. I am just trying to consolidate things
> I see as bloated and overly complex. I very much like the design of
> multiple encryption levels. All I am arguing for is a change in how each
> starts its offset.
>
> If folks still generally believe the current way we have offsets is
> better, then so be it. That is not my experience, so that's why I opened
> the issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406713278>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABD1ofVNBO5BCXpt1c95QpAKPlm4r5Jqks5uIji8gaJpZM4VYkas>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406798567

---

From: Nick Banks <[email-masked]>
Date: Sat, 21 Jul 2018 08:09:45 -0700

Closed #1592.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#event-1746113313

---

From: Nick Banks <[email-masked]>
Date: Sat, 21 Jul 2018 08:09:45 -0700

Well, it seems like everyone else likes the current design, so I will leave it be.

FYI, I still plan to implement it as a single stream with offsets, but I will just add/subtract offsets when I receive/send the CRYPTO frames. That way I don't need to duplicate all the send tracking state, reassembly buffers/flow control. It's not as clean, but still gets me to the memory usage I wanted.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1592#issuecomment-406802801