Subject: [quicwg/base-drafts] H3 GOAWAY should be symmetric and cover bidi and uni streams (#2632)

From: afrind <[email-masked]>
Date: Wed, 17 Apr 2019 23:52:20 +0000 (UTC)

The current spec only allows GOAWAY to be sent from the server to the client and contains only the last processed bidirectional stream ID.  This makes sense for the vanilla HTTP use case, but may not play well with extensions.  Unidirectional streams are a flexible extension point, and it may be useful for each endpoint to communicate to the peer the last processed unidirectional stream id.  An HTTP extension may also define a use for server initiated bidirectional streams, and may want the GOAWAY signal for these also.

True if it's an extension then the extension author can also extend GOAWAY to cover these cases or introduce another mechanism, but it would be a shame if many extensions had to cover the same ground.

I suggest we allow GOAWAY to be sent by either endpoint, and that it should contain both the last processed peer initiated bidirectional and unidirectional stream.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 17 Apr 2019 17:35:11 -0700

I was compelled by Alan's discussion about this.

It sounds like you are suggesting having two fields in the GOAWAY frame, one for each type. 

My thought was to mirror the way MAX_STREAMS works and have independent GOAWAY frames.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484311419

---

From: ianswett <[email-masked]>
Date: Thu, 18 Apr 2019 06:27:43 -0700

I think this argument makes sense, but this feels increasingly like a transport feature and not an HTTP feature.

If you add a unidirectional limit, I'd prefer a single frame, unless @LPardue has a reason he thinks separate frames are likely to be useful in practice.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484508094

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 18 Apr 2019 06:55:47 -0700

> I think this argument makes sense, but this feels increasingly like a transport feature and not an HTTP feature.

I don't have a strong opinion here. For reference, the original issue about moving GOAWAY out of transport is https://github.com/quicwg/base-drafts/issues/696

For the separate frames topic:
HTTP/3 currently lacks explanation about how GOAWAY works with MAX_STREAMS (see #2629). So my suggestion pre-empts that without much thought put into the final fix. Therefore it might be wrong. Some weak arguments: 

Before sending a GOAWAY, there is no explicit limit set by the server. The stream ID limit is implicit and real but more importantly, you don't invoke the requirement `a connection SHOULD send an initial GOAWAY frame with the last Stream ID set to the current value of QUIC’s MAX_STREAM_ID and SHOULD NOT increase the MAX_STREAM_ID thereafter.`. Imagination required on how we fix this language up for MAX_STREAMS.

>From Alan's problem statement description, I imagined GOAWAY to operate more like a drain state trigger. So having separate frames would allow the draining of unidirectional traffic and bidirectional traffic independently. E.g. a client declares "I'm intending to leave soon, and I don't want to accept new things from you (pushes) but I want the ability to create new requests, so please continue to give me bidi stream credits".

On a more practical level, a frame that includes both fields when only one type wants to be drained will likely end up with a set to max_int. So we'd waste bytes to express "don't do anything". Modelling this by making fields optional adds complexity for little gain. 

Finally, we a separate GOAWAY for unidirectional streams opens the door for it to be an extension frame. However, I think this would be the wrong thing to do as it would make managing extensions very difficult.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484520163

---

From: ianswett <[email-masked]>
Date: Thu, 18 Apr 2019 07:09:14 -0700

The salient comment from #696 for me is "I think that the GOAWAY frame needs to identify requests, not streams"

Making GOAWAY bidirectional and cover unidirectional and bidirectional streams seems to run counter to that statement, which is why it was starting to feel more transport-y.

But, I don't feel that strongly about this, unless it becomes an implementation hassle of some sort.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484526011

---

From: ianswett <[email-masked]>
Date: Thu, 18 Apr 2019 07:13:42 -0700

I think the use case I'm not sure it compelling is the one where we only want to drain one type of stream, but no the other.  My model of GOAWAY is that it's used for graceful shutdown, and I agree that could be useful on either side(moreso for non-HTTP applications, but maybe with WebTransport?).  I have a hard time thinking of an application where graceful shutdown requires continued creation of one of the stream types, but they may exist.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484527998

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 18 Apr 2019 07:26:00 -0700

> I have a hard time thinking of an application where graceful shutdown requires continued creation of one of the stream types, but they may exist.

I think this is the pragmatic argument for making a single frame. You don't want to make it easy for implementations to get into semi-draining states unintentionally. Sending GOAWAY with a limit of both forces that decision, which is good.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484533284

---

From: Mike Bishop <[email-masked]>
Date: Thu, 18 Apr 2019 16:27:19 +0000 (UTC)

I had thought the H2 requirement was that after sending you will not accept *or generate* new streams, but I can't actually find that text in 7540.  If we're concerned about being semi-draining, this seems like a restriction I'd want to have as well.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-484582112

---

From: Martin Thomson <[email-masked]>
Date: Tue, 23 Apr 2019 20:01:36 -0700

I don't think that we need to make special accommodations for possible future extensions, where the requirements for graceful shutdown are unpredictable.

> I have a hard time thinking of an application where graceful shutdown requires continued creation of one of the stream types, but they may exist.

Strange.  There's an example in h3.  In order to answer a request, a server might create multiple push streams after the point that shutdown is initiated.

The model for GOAWAY in h3 is based on the realization that the primary purpose of HTTP is to provide answers to requests.  Stop the requests and you eventually run out of work.  For the client, no signaling is required.  For the server, it just has to have a way to block incoming requests.  To Ian's point about identifying requests, I believe that it does (if the text doesn't support that, that's just a bug).

Extensions can change anything.  But we can't really say what the graceful shutdown model would be without understanding the extension.  Assuming that once shutdown is initiated there will no longer be any need for new streams is not safe.

I can easily think of cases where new streams are needed to fulfill existing commitments.  Any protocol that treats streams as messages with an 1:N or N:M relationship between the message that initiates a transaction and the messages required to complete that transaction is open to this.

Take a SQL or similar exchange where a transaction is opened, some queries and updates performed, and the transaction is committed or rolled back.  A graceful shutdown might trigger rollback or it needs to allow for existing transactions to complete.  If you want to allow transactions to complete, then you might have multiple streams created after the shutdown order is issued.

And the SQL case is still simple, because it is still fundamentally transactional.  If the protocol is truly bidirectional, with either peer initiating activity that might take time, then you need a more sophisticated shutdown process.  The quanta used to measure what is accepted or rejected during shutdown might not be a discrete linear sequence as it is for simple transactional protocols; it could be multi-dimensional.  Indeed, the reason for this issue is the possibility that h3 with extensions could end up being multi-dimensional.  The stuff people are doing with bidirectional transport use could end up like that, especially where it is intended to be multiplexed with h3.

Expecting things to line up around streams is probably reasonable, but when streams are used for multiple purposes, things get messy.  You need to be identifying the unit.  We get to use streams for GOAWAY because client-initiated bidirectional streams are exclusively used for requests, but we use stream identifiers only as means of identifying requests there (again, if the text implies otherwise, we need to fix that).

So my view is that GOAWAY is good enough for shutdown of requests.  But that is all we can realistically provide for at this layer.  If an extension needs to be shutdown gracefully, then the extension that adds that thing will need to provide signaling in support of that shutdown.  Practically, that means multiple frames involved in shutdown, but that's not terrible, just wait for all shutdown-y things to complete shutdown before tearing down the connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486048895

---

From: ianswett <[email-masked]>
Date: Tue, 23 Apr 2019 20:08:32 -0700

I think I'm fine with GOAWAY as is, but PUSH is never a required feature, it's a server-side optimization/pessimization.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486050117

---

From: Martin Thomson <[email-masked]>
Date: Tue, 23 Apr 2019 21:08:50 -0700

This wasn't making a value-judgment of server push, just a note that there are cases that don't fit the model proposed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486059837

---

From: ianswett <[email-masked]>
Date: Wed, 24 Apr 2019 07:15:01 -0700

To clarify, I was trying to note push is optional, and not required.

In practice, I don't think PUSH would work in this case, at least in browsers.  If the client receives a GOAWAY, then I wouldn't expect any new requests to go to that connection, including those which may be matched with results from the push cache.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486260062

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 24 Apr 2019 15:40:42 -0700

That's true, but URLs may have already been promised (not yet pushed) and there may already be requests waiting to rendezvous with the pushed stream. WDYT?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486454382

---

From: Mike Bishop <[email-masked]>
Date: Wed, 24 Apr 2019 16:41:18 -0700

That's also a detail of one particular push implementation, not an inherent way that push works.  I *think* the Push implementation in WinInet would still work, because the push cache is at the level of a request group rather than an individual connection.  (Somewhat moot for deployment use, but existence proof that there's more than one possible implementation of that.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486466564

---

From: ianswett <[email-masked]>
Date: Wed, 24 Apr 2019 16:50:37 -0700

I currently see 4 options:
 1) Do nothing and assume there are some implementation specific edge cases that are going to go fairly sideways, but they're unlikely to cause user visible issues.
 2) Move GOAWAY to transport and have it include uni and bidi.
 3) Add a max push id to the existing GOAWAY
 4) Add a new frame for push.

I'm ok with 1 and 2, and would be hesitant to pursue 3 or 4.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486468323

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 24 Apr 2019 17:10:30 -0700

I prefer option 1. 

I dont see what 3 provides. I think we could achieve the same by adding text that says something like "the server commits to resolving all existing promises sent on streams up to and including the stream ID indicated in GOAWAY. Pushes may be cancelled or reset as normal".

My 2p. GOAWAY is starting to sound like more of a STOP_CREATING (you have credit but it's a waste to use it in the current state) and PROCESSED (endpoint processed all entities up to ID, it is safe to retry higher IDs on a different connection).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486472094

---

From: ianswett <[email-masked]>
Date: Wed, 24 Apr 2019 17:40:15 -0700

@afrind besides the theoretical interest of how GOAWAY works with extensions, do you have any HTTP/3 extension in mind where the current GOAWAY breaks badly?  And if so, would it be feasible to add another GOAWAY_EXTENSION frame for that use case?

ie: Can we just not fix this, or make some editorial tweaks?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-486477070

---

From: afrind <[email-masked]>
Date: Mon, 29 Apr 2019 18:05:29 +0000 (UTC)

The extension we are developing allows complete bidirectionality to HTTP, so a server can initiate a request/response to the client.  The H2 mapping of this extension uses a client-sent GOAWAY to clue the server in to stop doing this.  I don't know that missing this will break-badly; it probably depends on what application developers do with the bidirectionality.  It's asymmetric with H2, seemingly because we think we got it wrong in H2 and are trying to correct it?  I'm trying to say there is some value in keeping it symmetric.

I think Lucas correctly identified the semantics of what GOAWAY is - STOP_CREATING and PROCESSED.

As mentioned in the original issue, anyone writing an extension can also extend the GOAWAY frame under that extension that does what I proposed, so doing nothing is of course viable. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2632#issuecomment-487683538