Subject: [quicwg/base-drafts] Forwarding upstream error to downstream (#3300)

From: Kazuho Oku <[email-masked]>
Date: Thu, 12 Dec 2019 19:54:50 -0800

In HTTP/2, it has been easy for a proxy to forward upstream errors to client.

When the upstream abruptly closes a H1 connection (i.e., closed before serving the correct number of bytes as suggested by the content-length header, or before trailers when chunked encoding is used), the HTTP/2 terminator could forward all the content that it received from upstream, then send an RST_STREAM frame. Because HTTP/2 uses TCP, which guarantees the transmission order, the client receives the bytes that the origin has sent, before receiving a reset.

In contrast, HTTP/3 does not have such guarantee. When a HTTP/3 proxy resets a request stream mid-response, the QUIC stack is allowed to cease transmitting data. Depending on when the HTTP/3 proxy receives input from upstream, and depending on how the responses are prioritized, there is fair chance that the HTTP/3 proxy would be resetting the request stream with HTTP_REQUEST_CANCELED, without sending any response at all.

Are we fine with this behavior change?

We could argue that the root cause is the upstream failing to send the correct response. But the change of the failure mode might be an issue, because it would worsen the user-experience when the upstream misbehaves. For example, when an upstream server abruptly terminates the transfer of a JPEG image, a user connected through an HTTP/2 proxy would receive a partial image, whereas a user connected through an HTTP/3 proxy would see nothing (due to the reasons stated above).

Related to: #2426

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 12 Dec 2019 20:47:52 -0800

PS. Note that HTTP expects intermediaries to forward these errors. If an intermediary erases this error, downstream might cache and reuse the corrupt response.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-565298869

---

From: ianswett <[email-masked]>
Date: Mon, 16 Dec 2019 08:42:11 -0800

If the TCP connection is closed with a reset, there's no guarantee how many bytes the intermediary has read from the kernel buffer before the reset is received, so I don't think the behavior is likely to be that different.

If there's an error at the HTTP layer, then I think there would commonly be a change in what the client sees, but that seems like a buggy backend, so I'm not sure it's worth attempting to deal with more gracefully?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566141520

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 16 Dec 2019 12:19:22 -0800

@ianswett's point is right: there's no guarantee what the proxy has read out of a connection that has received a TCP RST. Additionally, there's no guarantee that the H1 sender's pending data has been sent out of its TCP buffers yet either.

@kazuho: Do you think think this is somehow different?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566225931

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Dec 2019 16:02:25 -0800

@ianswett @janaiyengar Thank you for your responses.

The issue is NOT related with TCP resets.

HTTP/1.1 does not distinguish between TCP FIN and RST, but it has it's own framing; see [RFC 7230 section 3.3.3](https://httpwg.org/specs/rfc7230.html#rfc.section.3.3.3). HTTP/2 does not (need to) distinguish between the two, it relies on HTTP/2 frames to see if an HTTP message has been transferred completely.

My point is that a HTTP/1.1 or HTTP/2 intermediary acting as a "tunnel" (as defined in  [RFC 7230 section 2.3](https://httpwg.org/specs/rfc7230.html#rfc.section.2.3)), is able to forward unexpected termination of an HTTP message that it received from upstream to downstream. In HTTP/1.1, what a tunnel can do is abruptly terminate a chunked encoding (by closing the connection normally). In HTTP/2, a tunnel sends a RST_STREAM.

In either case, the close signal never overtakes the partial payload.

I am not sure what others do, but at least H2O have been strict about forwarding partial payload and error close as they were received from upstream; see https://github.com/h2o/h2o/pull/1490. The intention is to provide the same response to the user even when a server running behind returns a corrupt response.

But in HTTP/3, because a stream reset can overtake the transmission of application data, it becomes very hard to act as a tunnel in such a way.

The worst case is as follows: consider the case where an origin always returns an HTTP response (containing an image) that terminates abruptly (at the HTTP/1.1 or HTTP/2 layer). Note that the delivery of partial, progressive JPEG image might not have any negative effects on user experience. Or the origin might in fact be returning an entire image, but failing to terminate the chunked encoding correctly.

When the connection between the tunnel and the client is HTTP/1.1 or HTTP/2, the tunnel can (and likely will) forward that partial response. But when the connection between the tunnel and the client is HTTP/3, it is very hard for the tunnel to deliver the partial response before the stream reset. Assuming that the partial response is of low priority, it is likely that the client would see nothing (rather than an image), as the stream reset signal would precede delivery of any stream data.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566306169

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Dec 2019 17:09:05 -0800

One possible solution to this issue is to allow a tunnel to intentionally send a malformed response to downstream, rather than resetting the stream. I've created #3303 that specifies such behavior.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566324188

---

From: Martin Thomson <[email-masked]>
Date: Mon, 16 Dec 2019 17:30:00 -0800

I don't think that this is a good response to this condition.  It creates a situation where an intermediary purposefully violates the protocol.  If the connection is broken, then let it be broken.  The intermediary can forward precisely what it received, ensure that that data was received, then reset the stream if it truly cares about fidelity.

Note that the intermediary might also have to deal with gaps toward the tail of such a broken stream as well, and then they can't propagate that information with the same accuracy.

We shouldn't permit a condition that the recipient can't distinguish from badness.  The result of a change like this is to force clients to accept this junk as something other than a connection-level error.  That means we can't catch badness that looks similar to this.  For this particular kind of behaviour, I would expect connection errors in most implementations.

I realize that HTTP/1.1 implementations do this all the time and that CDNs often have to patch up these situations, but this isn't patching it up, it's propagating the badness to the rest of the system.  You are right the the badness is sometimes constrained.  As you say, the trailing part of a progressive image often can be lost with no visible impact.  This is an area where browsers might need to be tolerant of resets and retain at least what has already been displayed.  You might want to be able to rely on that happening too, so maybe this is an area in which we pursue solutions in other areas (like Fetch).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566338140

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Dec 2019 18:23:44 -0800

@martinthomson 
> I don't think that this is a good response to this condition. It creates a situation where an intermediary purposefully violates the protocol. If the connection is broken, then let it be broken. Th…

The problem here is that it is complex to ensure that the data was received before transmitting a reset. That's something that the QUIC transport does _not_ have a concept, and I'm not sure if we can except QUIC stacks to expose a signal indicating if data has been acked.

We might argue that in-order delivery of stream resets was a feature of HTTP/2, and that we need it in HTTP/3. As you state, I think we'd better define a way to patch up these situations, otherwise people might suffer from worse user experience when using HTTP/3.

Am I correct in assuming that your distaste is against the solution proposed in #3303 rather than against this issue itself? If so, would you be more open to using a HTTP/3 frame for indicating abrupt termination?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566350354

---

From: Martin Thomson <[email-masked]>
Date: Mon, 16 Dec 2019 19:46:09 -0800

Look, I recognize the difficulty here, but I don't think that this problem suggests anything other than a (maybe) a new stream reset error code.  I realize that pushes for widening of the HTTP/3-to-transport interface, but I can't see an alternative that does not create horrible externalities for this use case.  Important as it might be for CDNs and the web, I don't think that it is a good idea to have the rest of the system bear the cost of dealing with the problem.

I am very much opposed to the proposal in #3303 for previously-stated reasons.

Enabling in-order delivery of stream resets would be a massive disruption to the protocol, even if we only did it at the HTTP/3 layer.  I don't concretely know how to do that, but it implies a significant shift (or bifurcation) in the stream model if we did it at the transport layer.

Adding in-order resets at the application layer for HTTP/3 is not trivial either.  A frame on the same stream is not going to work because it prevents an intermediary from forwarding without reframing at that layer.  Otherwise, they would not be able to guarantee that they can send any frame on the stream.  That requires a big shift in how people approach building of intermediaries.  Something on a control stream might work, but then you have the potential for ambiguity about placement relative to the target stream unless you effectively replicate the RESET_STREAM design at the application layer.  That would make me very uncomfortable.

It seems to me that you *can* address this without changing the protocol, even if it requires some awkward crossing of protocol layers.  That's my preference.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566367359

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Dec 2019 20:07:43 -0800

@martinthomson Thank you for sharing your views. I am not sure if it correct to frame this issue as specific to CDNs, I might argue that it is a problem for intermediaries in general.

I agree that we shouldn't try to introduce in-order delivery of stream resets at the transport layer. If we are to introducing that, it should be a new signal within HTTP/3.

> Adding in-order resets at the application layer for HTTP/3 is not trivial either. A frame on the same stream is not going to work because it prevents an intermediary from forwarding without reframing at that layer. Otherwise, they would not be able to guarantee that they can send any frame on the stream. That requires a big shift in how people approach building of intermediaries.

I think this is what I disagree. I _think_ that many if not most of the intermediaries to be doing reframing. In case of HTTP/2, intermediaries _should_ have been doing reframing by itself, otherwise it cannot prioritize the responses, I'd assume that proxies would be doing the same for HTTP/3 (though I might be wrong).

If that assumption is correct, the fix would be as trivial as introducing a HTTP/3 frame indicating the error, which is sent on the request stream. It's ugly to have two ways of signaling error (one at QUIC layer and one at HTTP layer), but it'd help us provide same expectations across different versions of HTTP.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566371284

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Dec 2019 20:17:00 -0800

PS. Or, if we are to consider H3 tunnels that does not do reframing as something within the scope of the specification, I think we need to have something like #3303.

Consider the case of a H3-H3 intermediary that parses the request, and binds HTTP requests to different origins, transmitting the *response* as-is without dealing with frames. Such a tunnel would work as expected, assuming that QPACK dynamic table is not used.

In this type of design, a stream-level error (i.e. abrupt termination of a stream) does not indicate a connection-level error. Therefore, we need to state that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566373198

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 16 Dec 2019 20:30:37 -0800

I think I agree with @martinthomson here. Adding in-order stream reset sounds extremely complicated at either the HTTP/3 or QUIC layers. The particular use case here does not seem compelling enough to warrant this complexity. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566375791

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Dec 2019 17:59:14 -0800

@RyanAtGoogle 
> Adding in-order stream reset sounds extremely complicated at either the HTTP/3 or QUIC layers. The particular use case here does not seem compelling enough to warrant this complexity.

As stated above, if it ought to be a complicated mechanism at the HTTP/3 layer depends on the how the HTTP/3 tunnel frames the response. As we've discussed previously in #1885, an HTTP/3 intermediaries can and will have different framing strategies. When the strategy employed by an intermediary is to create a new DATA frame every time it emits something, sending a HTTP frame to indicate an error after sending all the HTTP response it has is trivial.

All that said, I think that we might not need a new frame after all.

We already allow a HTTP tunnel to send a malformed response (see [the third paragraph of section 4.1.3](https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.3-3)). That tunnel might be coalescing HTTP messages going to different endpoints. That means that an endpoint should not handle a stream that contains invalid DATA frames as a connection-level issue.

I've changed #3303 to be a clarification of this requirement, which has been implicit until now.

WDYT?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566832026

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 17 Dec 2019 18:51:13 -0800

Ah! This sounds much better. No additional complexity in HTTP/3 or QUIC? SGTM.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566843489

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 17 Dec 2019 20:03:06 -0800

Thanks, @kazuho -- this re-framing (pun alert!) helps quite a bit. What you now have in the PR makes sense. Do you mind changing the title and description of this issue to reflect that?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566857672

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Dec 2019 21:29:50 -0800

@janaiyengar Changed the title. Though I'm not sure if you like the new one. I think the only difference is that we are now talking about errors in the headers too.

@RyanAtGoogle 
> No additional complexity in HTTP/3 or QUIC? SGTM.

Yes. That true for what we have now in #3303.

Though, unfortunately, I now realize that we have the [following provision in section 7.1](https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-7.1-6): When a stream terminates cleanly, if the last frame on the stream was truncated, this MUST be treated as a connection error of type H3_FRAME_ERROR._

If we consider that a tunnel should be allowed to send DATA frames (or HTTP/1 chunks) that it receives from upstream as is, forwarding them frame-by-frame to downstream, and expect things to work, then we need change this provision to so that incomplete (DATA) frames at the end of the stream would be considered as a malformed response instead of a connection-level error.

I would prefer making such a change, though I would anticipate hearing objections. Admittedly, it is an unnecessary complexity for clients, and for server deployments who control the origin. To this end, I wonder what other proxy vendors think.

@MikeBishop @LPardue WDYT?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-566874094

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 18 Dec 2019 10:40:18 -0800

I wasn't following this discussion so closely, so IIUC the problem is that the stream cannot end abruptly or else there are no guarantess from QUIC layer that the DATA will arrive at the HTTP client. Closing cleanly would ensure DATA is delivered but triggers h3-24 section 7.1.

> If we consider that a tunnel should be allowed to send DATA frames (or HTTP/1 chunks) that it receives from upstream as is,

To invoke the problem, an intermediary would have needed to committed a DATA frame of length `N`, to stream offset `O`, then attempt to fin the stream at offset `F`, which is less than `O+N`. I can see several scenarios that this might occur in an intermediary and it would be quite unfortunate to cause a connection close.

Since request streams are the only core streams that can be cleanly closed without blowing up the connection I think the middle ground might be to state something like: an endpoint that receives truncated leading HEADERS or PUSH_PROMISE should treat it as a connection error. Truncation of DATA and non-leading HEADERS is ok. Extension frames or streams need to consider truncation and specify the behaviour.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-567158031

---

From: Mike Bishop <[email-masked]>
Date: Thu, 19 Dec 2019 08:22:17 -0800

> In either case, the close signal never overtakes the partial payload.

This is not true, or at least not universal.  It's entirely legitimate for a TCP stack to dump anything that's in buffers (unread by the application) and immediately surface the TCP RST.  The reset can and does overtake partial payload in places.

> We already allow a HTTP tunnel to send a malformed response (see [the third paragraph of section 4.1.3](https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.3-3)). That tunnel might…

That's not my understanding of that paragraph.  It says that if an intermediary receives a malformed response, it MUST NOT forward it and MUST treat receipt of the malformed response as a stream error.  This is unclear in two ways we need to correct:

- First, it doesn't say whether the specified error should be sent upstream or downstream.  I believe this intends to say it resets the upstream stream that sent it a malformed response.
- Second, assuming the first answer, it doesn't specify what to send downstream _instead_ of forwarding the malformed response.

I would propose that we need YAEC to signal an aborted response upstream.

The suggestion that we send as much as we can, then cleanly terminate the stream disturbs me.  I'm inclined to take a hard line here and keep the text that intermediaries shouldn't be forwarding known broken responses.  That way lies consistency bugs.  You forward data as long as it's valid; if it becomes invalid, then you forward a RESET_STREAM that indicates the upstream is no longer talking sense.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-567559242

---

From: Mike Bishop <[email-masked]>
Date: Thu, 19 Dec 2019 11:33:29 -0800

Ah, I see -- you're keying off the language that says an intermediary might not process the message and therefore might not notice that the message is malformed.  I agree that an intermediary that deliberately doesn't check for message validity wouldn't trigger this requirement for things like missing pseudo-headers.

I'm less sanguine about saying the tunnel could silently pass through things like truncated frames, unknown frame types, etc.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-567631045

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 19 Dec 2019 21:40:23 -0800

@LPardue @MikeBishop Thank you for sharing your thoughts.

I can see the hesitation against requiring endpoints (clients) to consider partial frame at the end of stream as stream-level errors (rather than connection-level problem).

Maybe what we want to discuss first is:

Q1. Do intermediary developers (especially of those that talk to servers owned by different organizational entities) want to deliver the partial response that they've received from upstream to downstream, before tearing down the stream?

Q2. Do we need a mechanism for signaling such behavior? That mechanism can be a partial frame at the end of the stream, or could be a new HTTP/3 frame. Or if the answer to this question is no, then, people wanting to do Q1 need to send the partial response body, wait for all the ACKs, then send a stream reset.

My answer to Q1 is a strong yes, and Q2 is a weaker yes. I'd prefer having a simple signaling mechanism (partly because I think everybody should do Q1), but for myself, I can resort to the complex way of delaying the emission of stream reset.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-567796332

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 20 Dec 2019 17:22:45 -0800

> I'm less sanguine about saying the tunnel could silently pass through things like truncated frames, unknown frame types, etc.

It seems to me that we don't need to specify what tunnels _ought to_ do but recognize what tunnels _might_ do. The way I read it, Kazuho's proposed recommendation makes sense when a tunnel passes a partial response through, as H2O does and others might as well.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-568139395

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 21 Dec 2019 02:23:16 -0800

I am a yes for Q1. For Q2, if we wanted to pursue another frame type I don't (yet?) understand how it would work, won't the usefulness of the frame be subject to the size of DATA frames that people chose to use?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-568169967

---

From: ianswett <[email-masked]>
Date: Mon, 23 Dec 2019 07:01:11 -0800

With TCP, typically the reset will arrive after the payload, so the typically the partial payload can be processed.

How about using a similar recommendation for QUIC/H3: You CAN/SHOULD send all the data before transmitting the reset, but there's no need to guarantee it all arrives before sending the reset?  If an implementation really wants to guarantee the payload arrives before the stream is reset, it can, but this seems like best effort.

I don't want to add a new mechanism for this, but I agree it'd be nice to avoid the H3 connection being torn down in this case.  I do think we should scope this case a bit tighter and be more specific than the current PR, since it currently says any incorrectly formatted response MUST NOT cause a connection close and I'm realizing how wide that might be.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-568497686

---

From: Mike Bishop <[email-masked]>
Date: Fri, 27 Dec 2019 07:50:57 -0800

I think there's a larger idea here:  We took a series of PRs to make any protocol misbehavior connection-fatal.  However, when there's an intermediary, it's not necessarily clear whether the misbehavior is on the part of the intermediary or the origin.  And it's not necessarily clear to the client whether there's an intermediary, so it generally will have to assume one or the other all the time.

The takeaway for me is that we overreached -- if the trigger for an error could be attributed to the origin rather than the intermediary, then the error should not be connection-fatal.  If there are things in this class that we believe need to be connection-fatal anyway, the corollary is that the intermediary MUST check for and sanitize these things.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3300#issuecomment-569295073