Subject: Re: [quicwg/base-drafts] Length-prefixes and flow control (#1432)

From: Martin Thomson <[email-masked]>
Date: Wed, 01 Aug 2018 18:34:27 -0700

(Just an aside.)  HPACK never needed EOS.  The loss of efficiency is tiny (one bit for some never-used symbol), but it should have been possible to use any sequence that is longer than the number of remaining bits.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-409776607

---

From: Ryan Hamilton <[email-masked]>
Date: Thu, 02 Aug 2018 13:10:53 +0000 (UTC)

If you want to process an entire block at once, you must have the entire block in memory, obviously. Since no application wants to buffer an unlimited amount of data, the application will have some limit on the size of the block that it accepts. If it reads the length and the length is greater than the limit, it can abort. Otherwise, I would argue, it should read the block out of the receive buffer and into an application buffer. This avoids deadlock and also prevents unlimited buffering.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-409920483

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 02 Aug 2018 06:34:06 -0700

@martinthomson, @MikeBishop -- surely one can stream _HEADERS_. It's just a matter of programming.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-409927661

---

From: Martin Thomson <[email-masked]>
Date: Thu, 02 Aug 2018 18:59:39 -0700

This isn't a question of a particular construct; I'm more concerned about what advice we might give generally.  Ryan's advice is only really part of that (though it's pretty solid and pragmatic advice).

There are probably other things to say about what a sender might do that help.  For instance, if you think that something might be parsed as a single unit, then it's probably not a good idea to only send part of it.  Waiting until flow control credit is available - while it might risk a stall in the event that credits never arrive - might be a good idea.

It's also probably necessary to say something about inter-stream dependencies along the lines of: "If you have a dependency between the content of two streams, then you need to ensure that the dependent information is not sent until the information it depends on can be sent.  In particular, the necessary flow control credit needs to have been consumed."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410121095

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 02 Aug 2018 19:37:00 -0700

I think this issue should be tagged `-http` instead of `-transport`.  This logic is something for HQ to figure out.

> It's also probably necessary to say something about inter-stream dependencies along the lines of: "If you have a dependency between the content of two streams, then you need to ensure that the depe…

IIRC, this was one of the conclusions reached by the Header Compression Design Team.  Nevertheless, I don't see any verbiage addressing this either HTTP or QPACK drafts.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410126622

---

From: Martin Thomson <[email-masked]>
Date: Thu, 02 Aug 2018 21:25:10 -0700

We have #1420 for the specific advice, but I had hoped to gather some more general advice that that could reference.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410140692

---

From: Ryan Hamilton <[email-masked]>
Date: Fri, 03 Aug 2018 14:19:29 +0000 (UTC)

It seems like the general advice should be something along the lines of, "applications should not use a stream's receive buffer to store data while they wait for more data to arrive, as this can potentially lead to deadlock. instead they should read data out of the buffers expeditiously."

Really receive buffers should only be used to store data when the application is temporarily blocked from processing data and should be read as soon as the application is unblocked. What do you think?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410264806

---

From: ianswett <[email-masked]>
Date: Fri, 03 Aug 2018 08:48:03 -0700

It's worth a note about in-order data as well, since typically the main purpose of the receive buffer is to re-assemble data in order.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410295084

---

From: Ryan Hamilton <[email-masked]>
Date: Fri, 03 Aug 2018 12:51:22 -0700

That's a good point. The receive buffer *exists* to re-assemble data in order. The receive buffer is *sized* to allow the pipe to be filled by the sender in the face of packet loss and receiver application jitter. (If a full CWND worth of data is sent, but the first packet is lost, we don't want to have to wait for the retransmission of that data to be received before we can send new data. We also don't want to have to stop sending after a full CWND just because the app got swapped out temporarily, or is waiting for a seek, for example). It does not exist as an alternative to application data storage and any attempt to treat it as such is fraught with peril. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410358827

---

From: mirjak <[email-masked]>
Date: Mon, 06 Aug 2018 02:31:52 -0700

I guess it could be a good idea to give some more general advise (not only HTTP-specific) in the applicability statement. There is a section on "Use of Streams" and we could add another subsection to explain how this deadlock can happen and given some guidance inline with what @RyanAtGoogle and @martinthomson said. Anybody willing to send a PR directly or should I just open an ops-draft issue for now?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410647505

---

From: Dmitri Tikhonov <[email-masked]>
Date: Mon, 06 Aug 2018 14:00:44 +0000 (UTC)

I'd like to make another comment related to the EOS symbol.  The QPACK encoding process produces two outputs: the encoder stream instructions and the header block.  Now that the encoder framing has been removed, the encoder instructions can be written directly to the stream, while the associated header block must be buffered because we don't know its length [1,2].

We can avoid buffering of the header block by introducing an _End of the Header Block_ instruction at the cost of one byte.  The header block has just a few instructions and can easily accommodate another one.  This would also force our hand with regard to the Base Index: we'd have to pick it immediately.  On the other hand, it can make the design simpler: just set it to the same value as the Largest Reference.  This will offset the cost of the new EOS instructions, as Base Reference no longer has to be transmitted.

For this to work, we'd have to bring back _CONTINUATION_ frames, which seem to have had been designed exactly with this use case in mind.  Without them, we still have to buffer to calculate the size of the _HEADERS_ frame.  For background, see issue #76 and the [httpbis PR #548](https://github.com/http2/http2-spec/pull/548).  Needless to say, I do not agree with the rationale behind removing CONTINUATION frames from QUIC.

Footnotes:

1. I am aware of the deadlock hazard brought on by writing the header block and the encoder stream concurrently.  The assumption is that an implementation that writes to packets directly has the logic to do this right.
1. One alternative is to pre-calculate the encoded length of the header block.  This is unworkable because it a) essentially doubles the work; and b) causes buffering of unencoded header fields.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410717845

---

From: Dmitri Tikhonov <[email-masked]>
Date: Mon, 06 Aug 2018 18:34:06 -0700

> We can avoid buffering of the header block by introducing an _End of the Header Block_ instruction at the cost of one byte. [...]

This won't work -- the Largest Reference must still precede the header block. \*sigh\* I guess we must buffer....

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-410904061

---

From: Mike Bishop <[email-masked]>
Date: Tue, 07 Aug 2018 13:35:06 -0700

Actually, CONTINUATION frames were designed because the maximum supported size of a header set in some implementations was potentially much larger than the maximum size of a single HTTP/2 frame.  HQ eliminates them because our maximum frame size is much larger.  Sending chunks of headers at a time was a later proposal to effectively grease support for CONTINUATION, since they'd be relatively rare otherwise.

HTTP/2 still requires the entire compressed header set to be sent at once -- HEADERS+CONTINUATION effectively functions as a single jumbo frame, and didn't permit any other frame (including from other streams) to come between them.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1432#issuecomment-411193124