Subject: Re: [quicwg/base-drafts] Clarify ACK of ACKs and bundling a PING (#2794)

From: Igor Lubashev <[email-masked]>
Date: Mon, 01 Jul 2019 00:09:06 -0700

igorlord commented on this pull request.

>  To limit ACK Ranges (see {{ack-ranges}}) to those that have not yet been
 received by the sender, the receiver SHOULD track which ACK frames have been
 acknowledged by its peer. The receiver SHOULD exclude already acknowledged
 packets from future ACK frames whenever these packets would unnecessarily
-contribute to the ACK frame size.
-
-Because ACK frames are not sent in response to ACK-only packets, a receiver that
-is only sending ACK frames will only receive acknowledgements for its packets if
-the sender includes them in packets with non-ACK frames.  A sender SHOULD bundle
-ACK frames with other frames when possible.
+contribute to the ACK frame size.  When the receiver is only sending
+non-ACK-eliciting packets, it can bundle a PING with a fraction of them, such

By the way, the PING has an additional behavior requirement that it is ACKed ASAP, unlike other ACK-eliciting frames.  Bundling a PING forces the other endpoint to be slightly less efficient in its packet sending.  There are other alternatives/hacks like MAX_DATA(repeat latest value) that elicits ACKs but does not imply immediacy (and since it is a repeat, it does not require retransmission on loss, just like PING).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-256163087

---

From: MikkelFJ <[email-masked]>
Date: Mon, 01 Jul 2019 00:25:31 -0700

mikkelfj commented on this pull request.

> -received packet that contains frames other than ACK and PADDING frames.
-An endpoint MUST NOT send a packet containing only an ACK frame in response
-to a packet containing only ACK or PADDING frames, even if there are packet
-gaps which precede the received packet. This prevents an indefinite feedback
-loop of ACKs. The endpoint MUST however acknowledge packets containing only
-ACK or PADDING frames when sending ACK frames in response to other packets.
+Packets containing only ACK frames are not congestion controlled, so there are
+limits on how frequently they can be sent.  An endpoint MUST NOT send more than
+one packet containing only an ACK frame per received ACK-eliciting packet
+(one containing frames other than ACK and/or PADDING).  An endpoint MUST NOT
+send a packet containing only an ACK frame in response to a non-ACK-eliciting
+packet (one containing only ACK and/or PADDING frames), even if there are
+packet gaps which precede the received packet. This prevents an indefinite
+feedback loop of acknowledgements, which may prevent the connection from ever
+becoming idle. The endpoint MUST however acknowledge non-ACK-eliciting packets
+when sending ACK frames in response to other packets.

I think so - it may not be technically required, but it would be unclear to a new reader whether non-ack-eliciting packets should be acked, and if so, how.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r298911104

---

From: ianswett <[email-masked]>
Date: Mon, 01 Jul 2019 08:01:13 -0700

@ianswett pushed 1 commit.

81a9886a84216209e2f3a8b4db475a5428b0e58e  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/09ee3ebc938c58016d7c81eaec22dc2f1b8cb454..81a9886a84216209e2f3a8b4db475a5428b0e58e

---

From: ianswett <[email-masked]>
Date: Mon, 01 Jul 2019 08:02:02 -0700

ianswett commented on this pull request.

Thanks Martin, I believe I got all your suggestions and I created a sub-section for limiting ACK ranges, since this section was getting very long.

> @@ -2859,13 +2859,16 @@ valid frames? -->
 
 ### Sending ACK Frames
 
-An endpoint MUST NOT send more than one packet containing only an ACK frame per

Good point, this PR is turning into a rewrite of this section, but it needed it.

> @@ -2859,13 +2859,16 @@ valid frames? -->
 
 ### Sending ACK Frames
 
-An endpoint MUST NOT send more than one packet containing only an ACK frame per

Given ACK frames aren't congestion controlled, I think this is better as a MUST.  Also, this PR adds text about not bundling a PING or other ack-eliciting packet with every ACK frame.

I just checked both the transport and recovery drafts and I don't see any indication of a PING causing an immediate ACK.

> @@ -2883,16 +2886,21 @@ needing acknowledgement are received.  The sender can use the receiver's
 Strategies and implications of the frequency of generating acknowledgments are
 discussed in more detail in {{QUIC-RECOVERY}}.
 
+An endpoint that is only sending acknowledgements will not receive
+acknowledgments from its peer unless those acknowledgements are included in
+packets with ACK-eliciting frames.  A sender SHOULD bundle ACK frames with
+other frames when possible.

To clarify, you're thinking of the case when you're sending ACK-eliciting packets but not receiving any?  Good question, I'll add more text highlighting this and improving this paragraph.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-256379963

---

From: ianswett <[email-masked]>
Date: Mon, 01 Jul 2019 08:04:54 -0700

b4f756770b96c402a2b90123c2d3a9378806ed9a  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/81a9886a84216209e2f3a8b4db475a5428b0e58e..b4f756770b96c402a2b90123c2d3a9378806ed9a

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 01 Jul 2019 08:30:13 -0700

You are correct. I remember we had a discussion about doing so, but I also do not see it landing anywhere.  So this PING as an example of what can be done sounds good.  But "MUST NOT bundle a PING" in the following sentence looks like normative language, and PING is not longer just an example there.

Maybe you want to write: "MUST NOT make every otherwise non-ACK-eliciting packet ACK-eliciting, in order to ..." or "MUST NOT add an ACK-eliciting frame to every otherwise non-ACK-eliciting packet, in order to ...".

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299100230

---

From: David Schinazi <[email-masked]>
Date: Mon, 01 Jul 2019 12:50:05 -0700

DavidSchinazi commented on this pull request.

> -ACK or PADDING frames when sending ACK frames in response to other packets.
+An endpoint sends ACK frames to acknowledge packets it has received and
+processed. Sending ACK frames is the primary mechanism for advancing the state
+of a connection.
+
+Packets containing only ACK frames are not congestion controlled, so there are
+limits on how frequently they can be sent.  An endpoint MUST NOT send more than
+one packet containing only an ACK frame per received ACK-eliciting packet
+(one containing frames other than ACK and/or PADDING).  An endpoint MUST NOT
+send a packet containing only an ACK frame in response to a non-ACK-eliciting
+packet (one containing only ACK and/or PADDING frames), even if there are
+packet gaps which precede the received packet. Limiting the sending of ACK
+frames avoids creating an indefinite feedback loop of acknowledgements,
+which could prevent the connection from ever becoming idle. The endpoint MUST
+however acknowledge non-ACK-eliciting packets when sending ACK frames in
+response to Ack-eliciting packets.

s/Ack-eliciting/ACK-eliciting/

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-256529083

---

From: ianswett <[email-masked]>
Date: Mon, 01 Jul 2019 13:08:15 -0700

5d90a56f7104a2377ea015a718752ef8700dce4a  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/b4f756770b96c402a2b90123c2d3a9378806ed9a..5d90a56f7104a2377ea015a718752ef8700dce4a

---

From: Martin Thomson <[email-masked]>
Date: Mon, 01 Jul 2019 19:24:12 -0700

martinthomson commented on this pull request.

Yes, that is always the problem case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299281274

---

From: Martin Thomson <[email-masked]>
Date: Mon, 01 Jul 2019 19:24:43 -0700

martinthomson approved this pull request.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-256638140

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 01 Jul 2019 20:18:49 -0700

### Sending ACK Frames

The text about not bundling PING (& friends) with every ACK is for the sender of only-ACKs.

The question here is about the receiver of only-ACKs.  Presumably such endpoint has just sent a train of packets, and when it received an ACK for all the packets, it has nothing more to send.  At that point, it may wish to ACK the last ACK w/o waiting for the receiver to send it ACK+PING.  Would that be sane behavior?  I think so.  But if the sender bundles ACKs for ACKs in packets with data, the receiver will hold at most 1-rtt worth of state, so it is not that bad I guess.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299289179

---

From: MikkelFJ <[email-masked]>
Date: Mon, 01 Jul 2019 23:52:54 -0700

### Sending ACK Frames

What if change the requirement to simply not ACK'ing ACK's too often. That would also do away with PING feedback loops. It seems that we are just moving the problem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299326239

---

From: MikkelFJ <[email-masked]>
Date: Mon, 01 Jul 2019 23:54:04 -0700

Oh, I think I suggested that before and @ianswett said the peer might want to control when to receive ACK's.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299326569

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 15:17:34 -0700

We can say it differently, but it's important to say somewhere that non-ACK-eliciting packets are ACKed when an ACK is sent.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299707473

---

From: MikkelFJ <[email-masked]>
Date: Tue, 02 Jul 2019 15:22:14 -0700

that Oh, ... comment landet the wrong place - I responded elsewhere

> What if change the requirement to simply not ACK'ing ACK's too often. That would also do away with PING feedback loops. It seems that we are just moving the problem.

and this is what the comment was an update to.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#discussion_r299708727

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 02 Jul 2019 15:55:57 -0700

janaiyengar commented on this pull request.

> @@ -2859,13 +2859,21 @@ valid frames? -->
 
 ### Sending ACK Frames

This second sentence seems unnecessary.

> @@ -2859,13 +2859,21 @@ valid frames? -->
 
 ### Sending ACK Frames

```suggestion
one ACK-frame-only packet in response to receiving an ACK-eliciting packet
```

```suggestion
packet gaps which precede the received packet. Limiting ACK
```

```suggestion
frames avoids an infinite feedback loop of acknowledgements,
```

I don't think you need this MUST here. How about "The endpoint however includes acknowledgements for any non-ACK-eliciting packets as normal when it receives and acknowledges an ACK-eliciting packet."

> @@ -2874,6 +2882,18 @@ sender to become limited by the congestion controller (as described in
 receiver. Therefore, a sender SHOULD ensure that other frames are sent in
 addition to PADDING frames to elicit acknowledgments from the receiver.

```suggestion
An endpoint that is only sending ACK frames will not receive
```

> @@ -2874,6 +2882,18 @@ sender to become limited by the congestion controller (as described in
 receiver. Therefore, a sender SHOULD ensure that other frames are sent in
 addition to PADDING frames to elicit acknowledgments from the receiver.

```suggestion
acknowledgments from its peer.
```

> @@ -2874,6 +2882,18 @@ sender to become limited by the congestion controller (as described in
 receiver. Therefore, a sender SHOULD ensure that other frames are sent in
 addition to PADDING frames to elicit acknowledgments from the receiver.

```suggestion
An endpoint SHOULD bundle ACK frames with
```

> @@ -2874,6 +2882,18 @@ sender to become limited by the congestion controller (as described in
 receiver. Therefore, a sender SHOULD ensure that other frames are sent in
 addition to PADDING frames to elicit acknowledgments from the receiver.
 
+An endpoint that is only sending acknowledgements will not receive
+acknowledgments from its peer unless those acknowledgements are included in
+packets with ACK-eliciting frames.  A sender SHOULD bundle ACK frames with
+other frames when there are new ACK-eliciting packets to acknowledge.
+When only non-ACK-eliciting packets need to be acknowledged, the sender MAY

```suggestion
When only non-ACK-eliciting packets need to be acknowledged, an endpoint MAY
```

+as one per round trip, to enable dropping unnecessary ACK ranges and any state

```suggestion
as once per round trip, to enable dropping unnecessary ACK ranges and any state
```

 received by the sender, the receiver SHOULD track which ACK frames have been
 acknowledged by its peer. The receiver SHOULD exclude already acknowledged
 packets from future ACK frames whenever these packets would unnecessarily
-contribute to the ACK frame size.
-
-Because ACK frames are not sent in response to ACK-only packets, a receiver that
-is only sending ACK frames will only receive acknowledgements for its packets if
-the sender includes them in packets with non-ACK frames.  A sender SHOULD bundle
-ACK frames with other frames when possible.
+contribute to the ACK frame size.  When the receiver is only sending
+non-ACK-eliciting packets, it can bundle a PING with a fraction of them, such
+as one per round trip, to enable dropping unnecessary ACK ranges and any state
+for previously sent packets.  The receiver MUST NOT bundle a PING with all
+packets that would otherwise not be ACK-eliciting, in order to avoid an
+indefinite feedback loop of acknowledgements.

```suggestion
infinite feedback loop of acknowledgements.
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-257183127

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 15:57:37 -0700

9f9ad4757f0d5247aa40b2250dfaa9e29c7d94b1  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/5d90a56f7104a2377ea015a718752ef8700dce4a..9f9ad4757f0d5247aa40b2250dfaa9e29c7d94b1

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 15:58:25 -0700

e9c0357b7416ad5d14684731b05a3a02fc42e539  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/9f9ad4757f0d5247aa40b2250dfaa9e29c7d94b1..e9c0357b7416ad5d14684731b05a3a02fc42e539

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 15:59:15 -0700

d6b8f0f1b276e409186910c9ca6c0a1f7c490af3  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/e9c0357b7416ad5d14684731b05a3a02fc42e539..d6b8f0f1b276e409186910c9ca6c0a1f7c490af3

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 15:59:43 -0700

e02a4e8f026b786d1a6b900787a8ea2a4d5cd4ec  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/d6b8f0f1b276e409186910c9ca6c0a1f7c490af3..e02a4e8f026b786d1a6b900787a8ea2a4d5cd4ec

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:00:06 -0700

3525929ad18f4b68aa23a8d89b0dc00dc945ea91  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/e02a4e8f026b786d1a6b900787a8ea2a4d5cd4ec..3525929ad18f4b68aa23a8d89b0dc00dc945ea91

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:07:04 -0700

e4296ef4b5dbe54de69fdbd477f9ccb7a01ff818  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/3525929ad18f4b68aa23a8d89b0dc00dc945ea91..e4296ef4b5dbe54de69fdbd477f9ccb7a01ff818

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:07:20 -0700

ec6a311f399ea4a1868d03066cf432872ff7e666  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/e4296ef4b5dbe54de69fdbd477f9ccb7a01ff818..ec6a311f399ea4a1868d03066cf432872ff7e666

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:07:55 -0700

e32333c304c4333bf3c986ae619e08532652a7d4  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/ec6a311f399ea4a1868d03066cf432872ff7e666..e32333c304c4333bf3c986ae619e08532652a7d4

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:09:55 -0700

42540ef2946775787f457841486975bf5d4d8445  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/e32333c304c4333bf3c986ae619e08532652a7d4..42540ef2946775787f457841486975bf5d4d8445

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:13:36 -0700

200841fa1a5175571e6a7c6c6ee01c8726bbb866  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/42540ef2946775787f457841486975bf5d4d8445..200841fa1a5175571e6a7c6c6ee01c8726bbb866

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 16:14:54 -0700

Thanks Jana, I accepted or integrated variations on all your suggestions except one.

 ### Sending ACK Frames

Agreed, removed.

Isn't the existing text more accurate(minus the sender/endpoint change)?

 ### Sending ACK Frames

The existing text follows what TCP does as closely as I can manage, which I think is the best prior art we can follow here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-257189796

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 18:49:30 -0700

6c786bdf032d81c8bf8169d77e4c647b1c595b76  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/200841fa1a5175571e6a7c6c6ee01c8726bbb866..6c786bdf032d81c8bf8169d77e4c647b1c595b76

---

From: Martin Thomson <[email-masked]>
Date: Tue, 02 Jul 2019 20:29:26 -0700

So many comments, so much opportunity for this to get worse if we don't stop.

> @@ -2883,16 +2901,20 @@ needing acknowledgement are received.  The sender can use the receiver's
 
+
+### Limiting ACK ranges

```suggestion
### Limiting ACK Ranges
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#pullrequestreview-257244661

---

From: ianswett <[email-masked]>
Date: Tue, 02 Jul 2019 21:22:41 -0700

5e43503dd4265966177e7f20e54cd576f845f33d  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794/files/6c786bdf032d81c8bf8169d77e4c647b1c595b76..5e43503dd4265966177e7f20e54cd576f845f33d

---

From: Martin Thomson <[email-masked]>
Date: Sun, 07 Jul 2019 18:51:09 -0700

Merged #2794 into master.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2794#event-2464711909