Subject: [quicwg/base-drafts] Remap STOPPING to something other than zero (#1804)

From: Kazuho Oku <[email-masked]>
Date: Wed, 26 Sep 2018 01:18:00 -0700

It is common practice to use error code of zero as a non-error. We follow the practice in the transport draft.

However, we enforce the application protocol bindings to adopt a different practice by stating that application error code of zero is mapped to STOPPING, which indicates that a stream has been reset in response to a STOP_SENDING frame.

This is IMO counterintuitive and would be a source of confusion. I proposed to renumber the error code for STOPPING to something other than zero (possibly to one).

PS. The other way is to let stacks send zero (indicating non-error) in a RST_STREAM frame in response to STOP_SENDING. However, I think that is not the right solution because reset is an abnormal way of closing a stream. In other words, zero should be reserved for streams being shutdown using FIN.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804

---

From: MikkelFJ <[email-masked]>
Date: Wed, 26 Sep 2018 01:40:55 -0700

Yes, and at the same time reserve 0 to mean OK.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424634340

---

From: ianswett <[email-masked]>
Date: Wed, 26 Sep 2018 13:39:21 +0000 (UTC)

Good suggestion, it would make sense if 0 always meant NO_ERROR or something equivalent.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424718574

---

From: Nick Banks <[email-masked]>
Date: Wed, 26 Sep 2018 06:40:51 -0700

Why does the transport need to know what the stream error codes mean at all? If we get a STOP_SENDING, we just need to RST_STREAM. Why does the error code meaning even matter? I'd personally like to not have any assumptions for these values.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424719089

---

From: MikkelFJ <[email-masked]>
Date: Wed, 26 Sep 2018 06:41:41 -0700

It doesn't, but establishing a convention makes it much easier to write generic API's that do not know about application logic.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424719368

---

From: Nick Banks <[email-masked]>
Date: Wed, 26 Sep 2018 07:00:05 -0700

For the API, the error code should be purely passthrough, IMO. The app layer on one side shuts the stream down with an error code and the app layer on the other side is delivered the error code. No need at all to interpret it at a lower layer.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424725842

---

From: MikkelFJ <[email-masked]>
Date: Wed, 26 Sep 2018 07:24:25 -0700

Unless you want to represent the error code as an exception, or as non-nil in go, or conventially as nonzero in C.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424734624

---

From: Nick Banks <[email-masked]>
Date: Wed, 26 Sep 2018 07:44:37 -0700

The app might have many 'success' error codes or use some bit field or anything... The API shouldn't make any assumptions. IMO, the app should just get an indication that the peer reset the stream, and here is the error code they provided. If you go the exception route, any time the reset happens you throw an exception that just contains the raw error code. You don't want your API not to support some protocol that was initially written on one API that allows for any value, because you made some assumption that gets broken.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424741891

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 26 Sep 2018 08:52:18 -0700

@nibanks Am I correct in assuming that you are *only* opposed to defining in the transport draft that the app_error_code of zero means NO_ERROR, and *not* opposed to changing `STOPPING` to some value other than zero?

IMO we can certainly leave the former up to each application protocol.

However the latter needs to be defined in the transport spec. due to the fact being that `STOP_SENDING` is an transport feature. We need to define an error code that indicates that a stream has been reset due to that frame being received. And considering the fact that HQ is going to have a NO_ERROR code, it is better to leave zero for NO_ERROR (note also that HTTP/2 defines NO_ERROR as zero).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424764689

---

From: Nick Banks <[email-masked]>
Date: Wed, 26 Sep 2018 09:14:26 -0700

Ideally, I'd leave the entire space for the application layer. I just looked through the spec and it receiver of a RST_STREAM with STOPPING error code doesn't seem to do anything special, compared to receiving a RST_STREAM with any other error code. The goal of the STOP_SENDING frame is just to get your peer to RST_STREAM the stream. The best I could come up with is that if you get a STOPPING error code, you don't pass that up to the app layer directly, and internally just treat it as a confirmation of your STOP_SENDING. But, at least with my implementation, if the app layer has already shutdown it's receive direction on a stream, and I receive any RST_STREAM, I never indicate the event up, because it was shutdown locally first. With that design, the actual error code becomes irrelevant.

That being said, if someone comes up with a valid reason to keep a specific STOPPING error code, I'd prefer to use 0xFFFF instead of 0, 1 or any other value.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424773773

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 26 Sep 2018 09:51:55 -0700

@nibanks Thank you for the clarification.

> That being said, if someone comes up with a valid reason to keep a specific STOPPING error code, I'd prefer to use 0xFFFF instead of 0, 1 or any other value.

Regarding the need for STOPPING as an error code, my preference is to use the HQ draft as an example.

In HQ, we need to distinguish between an ordinary close (i.e. closed by FIN) and a reset, because only the former is cacheable. We also need to transmit the reason of a reset (i.e. app_error_code of RST_STREAM). Also, it is understanding is that it is not uncommon for operators to log how the streams were closed, for debugging and statistical analysis of the service quality.

Considering these aspects, the following three ways of closing a response need to be distinguishable:
* a) ordinary close
* b) reset by client (i.e. STOP_SENDING)
* c) reset by server (has many error codes)

In practice, it would be appreciated if we could use a single integer to indicate all three ways of closing. We can use `NO_ERROR` (which we already have in the HQ spec) for _a_. We have various error codes for _c_. Considering that, it would be natural to have a special error code within the 16-bit integer space for _b_, which is the `STOPPING` error code.

Regarding code point, my strong preference goes to using some value other than zero (hence this issue). I oppose to using 0xffxx, because the range is define as for private use in the transport error codes, and becasue I prefer not to have contradicting conventions in the transport and application binding. But any other value is fine for me.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424790318

---

From: Nick Banks <[email-masked]>
Date: Wed, 26 Sep 2018 17:17:51 +0000 (UTC)

> I oppose to using 0xffxx, because the range is define as for private use in the transport error codes

Where is that indiciated. Is that HQ specific? Here is the blurb from the transport spec about application error codes:

```
   There is no restriction on the use of the 16-bit error code space for
   application protocols.  However, QUIC reserves the error code with a
   value of 0 to mean STOPPING.  The application error code of STOPPING
   (0) is used by the transport to cancel a stream in response to
   receipt of a STOP_SENDING frame.
```
In generally, I feel that taking one single value from an otherwise completely free number space for just one particular scenario feels bad to me. There are plenty of other ways to achieve the same thing as STOPPING.

- You could have a different frame type (STOP_COMPLETE).
- You could just accept any RST_STREAM in response to a STOP_SENDING as confirmation.
- You could have a special bit/flag in the RST_STREAM to indicate there is no application error code.

IMO, reserving one value is the worst way to achieve this. Someone will always not be happy whatever value we chose, and the application layer will not understand why it can't use one value. How do you enforce that at an API layer? Wait until they happen to try to send that (in some particular, edge case scenario?) and then error out? What is the chance they don't see that until they deploy?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424799049

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 26 Sep 2018 10:29:14 -0700

> Where is that indiciated. Is that HQ specific?

[Section 13.3](https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#rfc.section.13.3): _"The “QUIC Transport Error Codes” registry governs a 16-bit space. This space is split into two spaces that are governed by different policies. Values with the first byte in the range 0x00 to 0xfe (in hexadecimal) are assigned via the Specification Required policy [RFC8126]. Values with the first byte 0xff are reserved for Private Use [RFC8126]."_

> IMO, reserving one value is the worst way to achieve this. Someone will always not be happy whatever value we chose, and the application layer will not understand why it can't use one value.

I agree that it is not a beautiful. But it is practical. It is beneficial to have an error code that covers all the three ways of closing responses, because that gives us a standard way of indicating how it was closed using just one number.

When tracking down an issue in the real world, it would often be the case that multiple HQ stacks are involved. It is highly preferable if we could just talk about the error codes rather than referring to the *frame type and error code* when trying to nail down issues.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424802605

---

From: Nick Banks <[email-masked]>
Date: Wed, 26 Sep 2018 11:52:02 -0700

The RST_STREAM error codes are not transport error codes, but application error codes. So that section you listed doesn't apply. Only the one I posted above.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424829432

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 26 Sep 2018 17:24:51 -0700

> The RST_STREAM error codes are not transport error codes, but application error codes. 

Yes. What I pointed out in 
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424790318 is that it would be not a good idea to require application bindings to assign different code points  for private use than what the transport draft does.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-424913341

---

From: Mike Bishop <[email-masked]>
Date: Thu, 27 Sep 2018 10:59:16 -0700

The value in having a STOPPING error code is understanding the source.  Because this is a transport feature, STOPPING permits the recipient to understand that the stream is being aborted because it sent a STOP_SENDING (i.e. its own action), not because the peer encountered a fatal error during the intervening RTT.  From another viewpoint, the "source" of the RST is the remote transport, not the remote application.

(I would envision STOP_SENDING surfacing to the remote API as a failure to write because the RST_STREAM has already been sent, not as a signal to which the application responds by triggering a RST_STREAM.)

I don't feel strongly opposed to having, say, a STOP_COMPLETE frame, but it would have identical practical effect to RST_STREAM.  That feels like a cosmetic change, and I'm not sure the difference is critical.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425186886

---

From: Nick Banks <[email-masked]>
Date: Thu, 27 Sep 2018 11:14:28 -0700

@MikeBishop The biggest difference with using a STOPPING error code vs any of the other options is how the an API ensures/communicates that the App layer can't use that value.

Additionally, if the App layer has already closed/shutdown their write pipe/direction, and then the transport receives a STOP_SENDING, I would assume the transport would silently process it, and not indicate to the App layer, since it has already discarded the pipe.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425192019

---

From: Mike Bishop <[email-masked]>
Date: Thu, 27 Sep 2018 18:24:56 +0000 (UTC)

It would likely not indicate to the app layer, but might still generate a RST_STREAM if the stream closed normally and retransmissions are still outstanding.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425195277

---

From: Nick Banks <[email-masked]>
Date: Thu, 27 Sep 2018 12:02:53 -0700

I'm imagining the logic for this stuff something like this:
```
recv_stop_sending():
	if (already_sent_rst_stream())
		do_nothing()
	else
		indicate_stop_sending_to_app()
		send_rst_stream(STOPPING)
		
recv_rst_stream():
	if (already_sent_stop_sending())
		do_nothing()
	else
		indicate_abortive_shutdown_to_app()
```
The way I see it, when you have sent a STOP_SENDING, you really don't care how the peer closes/resets the stream as long as it does and stops sending. No matter what, you wouldn't indicate it up to the app layer. Requiring the use of an explicit STOPPING error code still seems pointless to me. If you are worried about differentiating and indicating up that the peer reset their stream right **after** you told them to stop, I don't see the point. It's going to be a race condition. Simplest just to follow the logic above.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425207032

---

From: Martin Thomson <[email-masked]>
Date: Fri, 28 Sep 2018 12:18:38 -0700

Isn't STOPPING essentially the no error case?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425539323

---

From: Nick Banks <[email-masked]>
Date: Fri, 28 Sep 2018 12:22:32 -0700

The more I was thinking, if you agree that after you send a STOP_SENDING frame out, you won't indicate anything up to the app layer when you receive any RST_STREAM frame, then the actual error code value doesn't really matter. At most it gets logged locally, and that's it.

We can say that the entire number space is open to the application layer; but we only indicate it up if we haven't sent a STOP_SENDING (i.e. the app layer hasn't closed/shutdown that pipe already).

So, if we haven't sent a STOP_SENDING, then the received RST_STREAM must contain an app error code. If we have sent a STOP_SENDING, then the RST_STREAM might contain either an app error code or a STOPPING value (whatever that is...) but we don't really care, because we will just ignore it.

Then, we can either leave STOPPING as is or just get rid of it entirely and let the implementation send whatever it wants, since we will just ignore it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425540297

---

From: Martin Thomson <[email-masked]>
Date: Fri, 28 Sep 2018 12:33:00 -0700

That would work.  It messes with our pattern of specifying error codes for these conditions, but I think that I would be happier with that than having this odd carve-out.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425542825

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 28 Sep 2018 13:19:19 -0700

@nibanks @martinthomson +1

I was a bit worried about the interaction with QPACK, but it seems like a needless fear. [Section 5.3.3](https://quicwg.org/base-drafts/draft-ietf-quic-qpack.html#rfc.section.5.3.3) seems to already talk about abandoning a stream.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425554299

---

From: Mike Bishop <[email-masked]>
Date: Fri, 28 Sep 2018 13:31:09 -0700

The difficulty with that is, if the transport implementation sends "whatever it wants," then it's stomping on / making assumptions about the code in the application's own space.  It's better to have one carve-out defined in the spec than each implementation creating its own uncoordinated carve-out.

I think the transport doc either needs to specify a single carve-out so implementations are coordinated, or make the error code optional and specify that it's omitted in response to a RST_STREAM.  (Which is equivalent to a STOPPING frame.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425557306

---

From: Nick Banks <[email-masked]>
Date: Fri, 28 Sep 2018 13:33:00 -0700

@MikeBishop I disagree for the reasons I stated above. When you have sent a STOP_SENDING, the error code you receive in RST_STREAM is irrelevant; because you won't do anything with it. It's not stomping on the application layer's error codes because it doesn't get delivered to them.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425557714

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 28 Sep 2018 13:42:03 -0700

Yeah if we agree that some QUIC stacks **may** ignore the error code of a RST_STREAM frame received after sending STOP_SENDING, then the value we assign for STOPPED is irrelevant for such stacks. OTOH, the value needs to be defined if we agree to allow the existence of stacks that do **not** ignore, and for such stacks, we might argue that assigning a non-zero value makes sense.

The question is, do we want to allow both ways of implementing QUIC?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425559847

---

From: Nick Banks <[email-masked]>
Date: Fri, 28 Sep 2018 13:45:46 -0700

@kazuho that's a good way to put it. Personally, I see very little value in not ignoring the error code. For one, I imagine most implementations wouldn't even have a good way to indicate it up, since the app layer has already closed it. So my vote would be to not allow both ways and just always ignore it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425560789

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 28 Sep 2018 16:36:56 -0700

@nibanks Yeah. Assuming that we have a rough consensus to ignore the value, I think that we could for example state that an endpoint MUST send an RST_STREAM frame with an error code of zero when resetting a stream in response to a STOP_SENDING frame, however that does not restrict application binging for using that error code for other purposes because the peers that send STOP_SENDING is not expected to use the error code in any ways.

I think that stating as such has two benefits: 1) no change at the wire-level 2) we have a well-defined behavior (as @MikeBishop suggested).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425594324

---

From: MikkelFJ <[email-masked]>
Date: Fri, 28 Sep 2018 23:52:11 -0700

Okay, en esoteric example, but

what if there was another error before STOP_SENDING was received and, even if STOP_SENDING means loss of interest, it might not be without interest that something went wrong earlier. For example STOP_SENDING could mean: thanks, I now have enough audio samples for my AI to decide that the dog is barking, or not, but it still wants to know if the entire decision should be cancelled for some reason.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425621686

---

From: Mike Bishop <[email-masked]>
Date: Fri, 28 Sep 2018 23:58:20 -0700

Not that esoteric, actually.  The HTTP equivalent would be the server deciding it doesn't need to see the end of the request body around the same time the client decides it doesn't need to see the response.  The answer is that the client needs not only to RST_STREAM the request, but also STOP_SENDING on the response.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425622104

---

From: Mike Bishop <[email-masked]>
Date: Sat, 29 Sep 2018 00:02:25 -0700

@kazuho, that's a good clarification.  I don't particularly see it as a problem that applications can *also* use the error code used for STOPPING, but it's oddly structured that the transport MUST send a value chosen out of a range whose semantics it cannot know.  (Even if, yes, that value is basically only ever going to be seen by debug logs.)  If we required 0 to be NO_ERROR for all application protocols and specified that was the error code sent in response to a STOP_SENDING, would that address the concern?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425622313

---

From: MikkelFJ <[email-masked]>
Date: Sat, 29 Sep 2018 00:03:28 -0700

Alternatively STOP_SENDING could include the error code it wishes to receive.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425622353

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 29 Sep 2018 01:51:54 -0700

@MikeBishop 
> If we required 0 to be NO_ERROR for all application protocols and specified that was the error code sent in response to a STOP_SENDING, would that address the concern?

No opposition from me. The existence of an RST_STREAM frame indicating NO_ERROR makes me a bit sad, because then we cannot use the 16-bit space to indicate both ordinary close (via FIN) _and_ a reset. But that is the same with the other approach (that requires all the QUIC stacks to ignore RST_STREAM after sending STOP_SENDING).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425628690

---

From: Nick Banks <[email-masked]>
Date: Sat, 29 Sep 2018 10:44:04 -0700

Personally, I like that graceful and abortive stream closure are separate concepts. To me, it makes them more explicitly different, so they can't get confused.

As far as what to put in the RST_STREAM (sent in response to a STOP_SENDING), I am fine with saying always set error code to 0 OR saying that the any value may be used and the receiver should ignore the value if it sent a STOP_SENDING. If I could choose, I'd prefer the latter.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425663044

---

From: Igor Lubashev <[email-masked]>
Date: Sun, 30 Sep 2018 09:08:05 -0700

Sorry for jumping in a bit late, but I think a different frame is preferable here.

We already made two frames: APPLICATION_CLOSE and CONNECTION_CLOSE precisely for this same reason.  Namely, even though the transport behavior is identical, the causes of sending the frames are different -- one is triggered by an application and the other by the transport.  Likewise, RST_STREAM is sent due to an application condition, while STOPPED_STREAM/STREAM_STOPPED/STREAM_STOP_COMPLETE/whatever frame is caused by the transport.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1804#issuecomment-425731596