Subject: [quicwg/base-drafts] Add pacing note to flow control and other edits (#3301)

From: Jana Iyengar <[email-masked]>
Date: Fri, 13 Dec 2019 18:26:13 -0800

Addresses feedback from
- @gorryfair,  to add a note about pacing to handle bursts resulting from flow control credit;
- @kazuho, that advice on when to send *_DATA frames (at least 2 RTTs before sender is blocked) is too specific.

I&#39;ve also done some editorializing to fix flow.
You can view, comment on, or merge this pull request online at:

  https://github.com/quicwg/base-drafts/pull/3301

-- Commit Summary --

  * add note on pacing and editorialze

-- File Changes --

    M draft-ietf-quic-transport.md (55)

-- Patch Links --

https://github.com/quicwg/base-drafts/pull/3301.patch
https://github.com/quicwg/base-drafts/pull/3301.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301

---

From: Marten Seemann <[email-masked]>
Date: Fri, 13 Dec 2019 20:15:52 -0800

Is this the right place to discuss pacing? In my mind, we’re talking about different layers here. Bursty sending could happen due to many reasons, only one of them is due to increased flow control credit. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#issuecomment-565680857

---

From: Jana Iyengar <[email-masked]>
Date: Sat, 14 Dec 2019 00:09:09 -0800

Yeah I was wondering that too.  We do talk about application limited in the
recovery doc, so maybe that's the place to clarify this. I'll see (after
the weekend) if it makes sense to add anything there.

On Fri, Dec 13, 2019, 8:15 PM Marten Seemann <[email-masked]>
wrote:

> Is this the right place to discuss pacing? In my mind, we’re talking about
> different layers here. Bursty sending could happen due to many reasons,
> only one of them is due to increased flow control credit.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/pull/3301?email_source=notifications&email_token=ACUOBVAKHV35WISVUFEYPJTQYRMXBA5CNFSM4J2YHBAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG3ZVWI#issuecomment-565680857>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACUOBVHIJVMTCFUFMKF75NDQYRMXBANCNFSM4J2YHBAA>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#issuecomment-565694181

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 14 Dec 2019 09:00:41 -0800

I think we should discuss what to do in one place, but given the sort of traffic bursts that can arise if you happen to get this wrong, I am still keen to see a couple of sentences here that explain the pathology when a sender has cwnd and no credit, and then suddenly get lots of credit. We see the enormous burst, and it is REALLY not nice if you don't have a pacer to fix it (or figure out the receiver should have sent the credit earlier). 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#issuecomment-565734340

---

From: Jana Iyengar <[email-masked]>
Date: Sat, 14 Dec 2019 15:13:36 -0800

@janaiyengar pushed 1 commit.

bb2988130cf981e813f5378b9f908c3b09f134ca  remove pacing recommendation

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301/files/94754815fe5d11140f77cebe979ff954e8598574..bb2988130cf981e813f5378b9f908c3b09f134ca

---

From: Jana Iyengar <[email-masked]>
Date: Sat, 14 Dec 2019 15:16:52 -0800

@gorryfair: [Section 6.9](https://quicwg.org/base-drafts/draft-ietf-quic-recovery.html#section-6.9) in the recovery draft covers the general case of an under-utilized cwnd, which applies here. The same bursting can happen with a bursty app as well, but we've said it all in one place -- in that section of the recovery draft. I think that section could use some editing work (which I'll take on separately), but we've already said there that it applies to flow control. I think that ought to cover the pacing concern. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#issuecomment-565761543

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 14 Dec 2019 23:12:06 -0800

gorryfair commented on this pull request.

> @@ -833,11 +833,6 @@ implementations.  As an optimization, an endpoint could send frames related to
 flow control only when there are other frames to send or when a peer is blocked,
 ensuring that flow control does not cause extra packets to be sent.
 
-When a sender receives credit after being blocked, it might send a large amount
-of data in response. As is recommended in {{QUIC-RECOVERY}}, implementations
-should pace this data to avoid sending it in a burst and causing short-term

I think this cross-reference is needed - people implementing flow control need to ensure this part is correctly implemented also.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-332246780

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 16 Dec 2019 21:01:46 -0800

6bea25dc982beacbecf4576ff2945a8e89ede4f7  add a reference to sec 6.9 without recommendation

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301/files/bb2988130cf981e813f5378b9f908c3b09f134ca..6bea25dc982beacbecf4576ff2945a8e89ede4f7

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 16 Dec 2019 21:02:08 -0800

janaiyengar commented on this pull request.

I've added a reference in but with better text I think.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r358597149

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 16 Dec 2019 22:26:04 -0800

kazuho commented on this pull request.

LGTM modulo the point below.

> @@ -801,33 +801,41 @@ limit is increased.
 
 ## Flow Credit Increments {#fc-credit}
 
-This document leaves when and how many bytes to advertise in a MAX_STREAM_DATA
-or MAX_DATA frame to implementations, but offers a few considerations.  These
-frames contribute to connection overhead.  Therefore frequently sending frames
-with small changes is undesirable.  At the same time, larger increments to
-limits are necessary to avoid blocking if updates are less frequent, requiring
-larger resource commitments at the receiver.  Thus there is a trade-off between
-resource commitment and overhead when determining how large a limit is
-advertised.
+Implementations decide when and how many bytes to advertise in MAX_STREAM_DATA
+and MAX_DATA frames. This section describes one requirement and offers a few
+considerations.
+
+A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before

I understand that we are moving existing text, but "MUST NOT" seems a bit odd to me, as it could be interpreted as a prohibition against the receiver intentionally blocking the stream. Maybe "SHOULD NOT" is better.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-333054360

---

From: ianswett <[email-masked]>
Date: Tue, 17 Dec 2019 09:12:47 -0800

ianswett commented on this pull request.

Some editorial suggestions

 
+sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a

```suggestion
sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so means a sender will
```

 
+sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a
+sender could be blocked for the rest of the connection if the peer chooses to

```suggestion
be blocked for the rest of the connection if the peer chooses not to
```

-or MAX_DATA frame to implementations, but offers a few considerations.  These
-frames contribute to connection overhead.  Therefore frequently sending frames
-with small changes is undesirable.  At the same time, larger increments to
-limits are necessary to avoid blocking if updates are less frequent, requiring
-larger resource commitments at the receiver.  Thus there is a trade-off between
-resource commitment and overhead when determining how large a limit is
-advertised.
+Implementations decide when and how many bytes to advertise in MAX_STREAM_DATA
+and MAX_DATA frames. This section describes one requirement and offers a few
+considerations.
+
+A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
+sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a
+sender could be blocked for the rest of the connection if the peer chooses to
+not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames. Even if the peer sent these

```suggestion
send STREAM_DATA_BLOCKED or DATA_BLOCKED frames. Even if the peer sent these
```

+considerations.
+
+A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
+sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a
+sender could be blocked for the rest of the connection if the peer chooses to
+not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames. Even if the peer sent these
+frames, waiting for them means that a sender will be blocked for at least an
+entire round trip.
+
+A sender that runs out of flow control credit will be unable to send new data
+and is considered blocked, resulting in degraded performance for the
+connection. To avoid blocking a sender and to reasonably account for the
+possibility of loss, a receiver can send a MAX_STREAM_DATA or MAX_DATA frame
+multiple times within a round trip or send it early enough to allow for recovery
+from potential loss.

```suggestion
from loss of the frame.
```

-advertised.
+Implementations decide when and how many bytes to advertise in MAX_STREAM_DATA
+and MAX_DATA frames. This section describes one requirement and offers a few
+considerations.
+
+A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
+sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a
+sender could be blocked for the rest of the connection if the peer chooses to
+not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames. Even if the peer sent these
+frames, waiting for them means that a sender will be blocked for at least an
+entire round trip.
+
+A sender that runs out of flow control credit will be unable to send new data
+and is considered blocked, resulting in degraded performance for the
+connection. To avoid blocking a sender and to reasonably account for the

```suggestion
connection. To avoid blocking a sender,
```

+Implementations decide when and how many bytes to advertise in MAX_STREAM_DATA
+and MAX_DATA frames. This section describes one requirement and offers a few
+considerations.
+
+A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
+sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a
+sender could be blocked for the rest of the connection if the peer chooses to
+not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames. Even if the peer sent these
+frames, waiting for them means that a sender will be blocked for at least an
+entire round trip.
+
+A sender that runs out of flow control credit will be unable to send new data
+and is considered blocked, resulting in degraded performance for the
+connection. To avoid blocking a sender and to reasonably account for the
+possibility of loss, a receiver can send a MAX_STREAM_DATA or MAX_DATA frame

```suggestion
a receiver can send a MAX_STREAM_DATA or MAX_DATA frame
```

> +sending a MAX_STREAM_DATA or MAX_DATA frame, since doing so will mean that a
+sender could be blocked for the rest of the connection if the peer chooses to
+not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames. Even if the peer sent these
+frames, waiting for them means that a sender will be blocked for at least an
+entire round trip.
+
+A sender that runs out of flow control credit will be unable to send new data
+and is considered blocked, resulting in degraded performance for the
+connection. To avoid blocking a sender and to reasonably account for the
+possibility of loss, a receiver can send a MAX_STREAM_DATA or MAX_DATA frame
+multiple times within a round trip or send it early enough to allow for recovery
+from potential loss.
+
+Control frames contribute to connection overhead. Therefore, frequently sending
+MAX_STREAM_DATA and MAX_DATA frames with small changes is undesirable.  At the
+same time, larger increments to limits are necessary to avoid blocking if

I'd switch this around, ie:
"At the same time, less frequent updates require larger increments to the limits to avoid blocking."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-333436130

---

From: ianswett <[email-masked]>
Date: Tue, 17 Dec 2019 09:14:49 -0800

-If a sender runs out of flow control credit, it will be unable to send new data
-and is considered blocked.  It is generally considered best to not let the
-sender become blocked.  To avoid blocking a sender, and to reasonably account
-for the possibility of loss, a receiver should send a MAX_DATA or
-MAX_STREAM_DATA frame at least two round trips before it expects the sender to
-get blocked.
-
-A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
-sending MAX_STREAM_DATA or MAX_DATA, since doing so will mean that a sender will
-be blocked for at least an entire round trip, and potentially for longer if the
-peer chooses to not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames.
+When a sender receives credit after being blocked, it can send a large amount of
+data in response resulting in short-term congestion; see Section 6.9 in

```suggestion
data in response, resulting in short-term congestion; see Section 6.9 in
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-333441889

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 17 Dec 2019 21:42:18 -0800

Yes, it seems unnecessary. Let's do this separately in a design issue though, letting this one be editorial.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r359160308

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 17 Dec 2019 21:47:54 -0800

b361a8bef74a08a3f29392432648d3450a028037  Ian's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301/files/6bea25dc982beacbecf4576ff2945a8e89ede4f7..b361a8bef74a08a3f29392432648d3450a028037

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 17 Dec 2019 21:55:08 -0800

Ah, I forgot that I had thought about this and decided to leave it as it is :-) We do not require a sender to send a BLOCKED frame, and we therefore need to require receivers to not wait for this frame since it may never arrive. That's a deadlock (of a strange sort).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r359162790

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 17 Dec 2019 22:31:07 -0800

Ah that's a good point. Rereading the entire paragraph, I think I'm less concerned about readers possibly misinterpreting the text (though, admittedly, I was a bit confused).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r359170269

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 18 Dec 2019 19:07:36 -0800

906c77f51c88d60d0de9f98fed0b9591192923ad  rephrase and slight shuffle

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301/files/b361a8bef74a08a3f29392432648d3450a028037..906c77f51c88d60d0de9f98fed0b9591192923ad

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 18 Dec 2019 19:07:59 -0800

I've rephrased and moved the text around a bit, so hopefully it's clearer now.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r359668548

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 18 Dec 2019 19:44:21 -0800

Indeed it is. Thank you for the change.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r359674956

---

From: ianswett <[email-masked]>
Date: Thu, 19 Dec 2019 14:17:51 -0800

A few more suggestions, but this looks close.

> @@ -801,33 +801,37 @@ limit is increased.

How about "Managing Flow Control"?

-and is considered blocked.  It is generally considered best to not let the
-sender become blocked.  To avoid blocking a sender, and to reasonably account
-for the possibility of loss, a receiver should send a MAX_DATA or
-MAX_STREAM_DATA frame at least two round trips before it expects the sender to
-get blocked.
-
-A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
-sending MAX_STREAM_DATA or MAX_DATA, since doing so will mean that a sender will
-be blocked for at least an entire round trip, and potentially for longer if the
-peer chooses to not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames.
+implementations.  As an optimization, an endpoint could send frames related to
+flow control only when there are other frames to send or when a peer is blocked,
+ensuring that flow control does not cause extra packets to be sent.
+
+A sender that is blocked could choose to not send STREAM_DATA_BLOCKED or

```suggestion
A blocked sender is not required to send STREAM_DATA_BLOCKED or
```

-get blocked.
-
-A receiver MUST NOT wait for a STREAM_DATA_BLOCKED or DATA_BLOCKED frame before
-sending MAX_STREAM_DATA or MAX_DATA, since doing so will mean that a sender will
-be blocked for at least an entire round trip, and potentially for longer if the
-peer chooses to not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames.
+implementations.  As an optimization, an endpoint could send frames related to
+flow control only when there are other frames to send or when a peer is blocked,
+ensuring that flow control does not cause extra packets to be sent.
+
+A sender that is blocked could choose to not send STREAM_DATA_BLOCKED or
+DATA_BLOCKED frames. Therefore, a receiver MUST NOT wait for a
+STREAM_DATA_BLOCKED or DATA_BLOCKED frame before sending a MAX_STREAM_DATA or
+MAX_DATA frame; doing so could result in the sender being blocked for the rest
+of the connection. Even if the sender sent these frames, waiting for them will

```suggestion
of the connection. Even if the sender sends these frames, waiting for them will
```

-sending MAX_STREAM_DATA or MAX_DATA, since doing so will mean that a sender will
-be blocked for at least an entire round trip, and potentially for longer if the
-peer chooses to not send STREAM_DATA_BLOCKED or DATA_BLOCKED frames.
+implementations.  As an optimization, an endpoint could send frames related to
+flow control only when there are other frames to send or when a peer is blocked,
+ensuring that flow control does not cause extra packets to be sent.
+
+A sender that is blocked could choose to not send STREAM_DATA_BLOCKED or
+DATA_BLOCKED frames. Therefore, a receiver MUST NOT wait for a
+STREAM_DATA_BLOCKED or DATA_BLOCKED frame before sending a MAX_STREAM_DATA or
+MAX_DATA frame; doing so could result in the sender being blocked for the rest
+of the connection. Even if the sender sent these frames, waiting for them will
+result in the sender being blocked for at least an entire round trip.
+
+When a sender receives credit after being blocked, it can send a large amount of

```suggestion
When a sender receives credit after being blocked, it may be able to send a large amount of
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-334967789

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 19 Dec 2019 19:52:33 -0800

That seems too general. I'm not in love with what we have, but I prefer it because it's more specific.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#discussion_r360218670

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 19 Dec 2019 19:54:00 -0800

d626db6f961f3e5a48b6e4891ff5fa1af2827fec  Ian's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301/files/906c77f51c88d60d0de9f98fed0b9591192923ad..d626db6f961f3e5a48b6e4891ff5fa1af2827fec

---

From: ianswett <[email-masked]>
Date: Fri, 20 Dec 2019 06:44:56 -0800

ianswett approved this pull request.

I can see that.  These flow control sections are a bit too long for my liking, but it's good enough for now.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-335296026

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 20 Dec 2019 07:08:18 -0800

kazuho approved this pull request.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/3301#pullrequestreview-335309809