Subject: [quicwg/base-drafts] Should we allow ACK-only packets to be declared lost? (#3451)

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Feb 2020 15:23:38 -0800

After some discussion, the "in-flight" condition here leads to an odd requirement.

> A packet is declared lost if it meets all the following conditions:
> * The packet is unacknowledged, in-flight, and was sent prior to an acknowledged packet.
> * [other stuff]

This says that lost ACK-only packets cannot be declared lost.  This buries a potential signal that might drive a congestion controller.

This comes from TCP.  In TCP, detecting loss of an ACK is impossible, but in QUIC, we can.  But this requirement essentially forbids that.  Implementations don't seem to be consistent on this point.

@ianswett says that if we change this, we should also mandate the acknowledgment of ACK-only packets when sending an ACK for other ACK-eliciting packets.

We could simply say that this is good and recognize that this requirement is consistent with not requiring ACKs for these.  Or we could take the signal and declare them lost.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 11 Feb 2020 17:33:41 -0800

I do not think we need to allow ACK-only packets to be declared lost in the specification. That's something an implementation _might_ do.

At the same time, I'd point out that QUIC is designed on the premise that "acknowledgment of ACK-only packets when sending an ACK for other ACK-eliciting packets." The most prominent example is [section 13.2.4](https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#name-limiting-ack-ranges); the entire section is based on the assumption that an endpoint behaves that way. Therefore, we should be clear about the requirement, if we aren't already clear about that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-584972551

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Feb 2020 19:46:56 -0800

We should at least recognize that ACK-only packets can be declared lost.  One interpretation of the current text is that ACK-only packets MUST NOT be declared lost.

The reason that you might not want to declare these as lost that is that the requirements for acknowledging ACK-only packets are (necessarily) more loose than ACK-eliciting packets.  But I believe that it is simpler to allow code to declare them lost.  Correct me if I am wrong, but I believe that you have one fewer condition if you ignore whether a packet is in flight or not when doing loss detection.  It is also likely better to mark lost ACKs as lost as it means that you don't ignore a potential congestion signal.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585013482

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 11 Feb 2020 20:15:25 -0800

The problem is that a loss event has impact on CC, which applies to the amount of data the endpoint sends.

IIUC, the basis of CC is that an endpoint throttles the bandwidth *it* uses based on the loss rate of data *it* sent. However, generation of ACKs is not controlled by the endpoint. It's controlled by the peer.

Consider the case of in-flight wifi that uses a satellite link. The client (web browser) could be sending far more ACK-only packets than ACK-eliciting packets. In such case, I am concerned about the impact of taking loss of ACK-only packets into calculation.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585019747

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Feb 2020 21:06:13 -0800

I don't follow that argument.

There's a simpler one: if sending packets causes congestion such that loss (or ECN-CE marking) occurs, then ignoring that signal means that you will likely create more congestion.

I don't see how the relationship to inbound packets is relevant here.  There are many ways in which the rate of outbound packets might be determined by inbound packets.  Servers answers requests, after all.  And if ACK-only packets are the dominant use of a link, then deliberately ignoring signals from them likely only drives send rates up to unsustainable levels; and that's on top of not counting those packets toward limits (something that is necessary).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585031589

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 11 Feb 2020 21:33:57 -0800

Let my paraphrase.

> There's a simpler one: if sending packets causes congestion such that loss (or ECN-CE marking) occurs, then ignoring that signal means that you will likely create more congestion.

Are you arguing that an endpoint SHOULD throttle the rate in which it sends ACK-only packets, when some of those ACK-only packets are being lost?

I would assume that your answer is no, and that is because send-rate of ACKs are not controlled by the endpoint. It is controlled by the peer that is sending data.

Up until now, CC is per-direction. An endpoint sends data at a rate controlled by ACKs sent by peer in response. The other side does the same thing, individually.

While I do not dispute the fact that *ideally* we should take the loss of ACK-only packets into consideration, my argument is that we are not at that point (yet), because we do not control the send-rate of ACK-only packets.

Using the loss of ACK-only packets without controlling the send-rate of ACK-only packets based on their loss essentially means that we are going to prioritize emission of ACKs above data. This would be a departure from the per-direction CC design that we have now.

As I stated in my previous comment, I am concerned about the negative impacts that the change in design might cause.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585037874

---

From: Marten Seemann <[email-masked]>
Date: Tue, 11 Feb 2020 22:21:30 -0800

> Using the loss of ACK-only packets without controlling the send-rate of ACK-only packets based on their loss essentially means that we are going to prioritize emission of ACKs above data. This woul…

Can't find the text right now, but I thought that ack-only packets are not blocked by the congestion controller. At least that's what I implemented in quic-go.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585049508

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Feb 2020 22:58:23 -0800

@marten-seemann is right.  You don't allow ACK-only packets to be limited by the congestion controller.  I believe that don't is "can't" in practice.

And we should prioritize acknowledgements over data.  There's a point where you can't do anything but ACK, but that's the case no matter what we decide here.

All I'm saying is that loss of an ACK-only packet is a signal that we are being encouraged to ignore right now, for no good reason.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585059284

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 12 Feb 2020 05:27:50 -0800

> @marten-seemann is right. You don't allow ACK-only packets to be limited by the congestion controller. I believe that don't is "can't" in practice.

Agreed.

> All I'm saying is that loss of an ACK-only packet is a signal that we are being encouraged to ignore right now, for no good reason.

Yeah. I might agree that there is "no good reason" to agree. But my argument is that simply saying that an endpoint MAY take loss of ACK-only packets into consideration does not fix the problem.

Please correct me if I am wrong, but CC is designed as a feedback loop between send-rate adjustment and loss observation. Including loss of ACK-only packets to the latter, without adjusting the send-rate of ACK-only packets is going to change the equation.

Let's consider an example. Consider the case of a satellite link that has 1000ms RTT, 1Mb/s downstream bandwidth. When the downstream path is fully utilized, there would be ~100 downstream packets per second (or per RTT), which means that there would be ~50 ACKs being sent to upstream.

Let's assume that the loss rate of packets sent to upstream is 2%, regardless of the packet size. Under the status quo, CWND to upstream would be ~8 packets, because on average one loss would be observed per every 50 ACK-eliciting packets being sent. However, if we are to take into consideration the loss of ACK-only packets, the outcome would be that packet loss would be observed at a probability of 70% for each RTT. I think that would lead us to a CWND size of 2 packets or below.

One might argue that the loss rate is actually proportional to the size of packets, rather than the number of packets. So let's assume that the size of an ACK-only packet is typically 100 bytes. That does move a needle a bit, but I think the problem still exists. Assume that MTU is 1400 bytes, loss rate being 10% / MTU, ACK-eliciting packet are always full-sized, and that ACK-only packets are 50 bytes long. In status quo, I think that CWND would be somewhere between 2 MTU to 3 MTU (because the loss rate is 10% / MTU). But once we take the loss of ACK-only packets into consideration, the observed loss rate per RTT becomes 1.5x~2x, because the amount of data being sent per RTT increases by 50*50=2500 bytes. That would lead to a noticeable reduction of CWND size.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585204444

---

From: ianswett <[email-masked]>
Date: Wed, 12 Feb 2020 07:14:14 -0800

I'd prefer to keep this as is, because I think ensuring everything works correctly with an updated design is going to be a bit complex and potentially error prone, and I don't see any value in the change.

For example, when an ACK is lost, if the CWND is reduced, that doesn't reduce the rate of ACKs, it just reduces the rate of sending future data.  Because ACKs do not count towards bytes in flight, the congestion window would never increase if a peer is only sending ACKs, so we'd be putting peers in a situation that the CWND could decrease, but never increase.  That alone seems sub-optimal.

FWIW, I do believe we discussed this at length a few years ago and ended up where we did because of (at least) three points:
1) ACKs are not required to be acknowledged, even once.
2) This is very similar to what TCP does.
3) Congestion controlling ACKs is a research topic.

@martinthomson We track lost packets for a while in case they were declared lost too early, so the existing design is quite natural and a bit simpler.

If in QUICv2/etc we think it's worth making ACK-only packets congestion controlled and count towards bytes in flight, I think that would be a more holistically appealing design.  I also think it's quite a bit of work and research to get right.

But I do think we should make this point a bit clearer in recovery and articulate why this choice was made.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585253162

---

From: mirjak <[email-masked]>
Date: Wed, 12 Feb 2020 07:42:59 -0800

I agree with Martin, there is no reason to ignore loss of ACKs as a signal. What you do with this information is a different question. There have been proposals for "ACK congestion control" in the past which basically means that you may adjust your ACK rate (e.g. if you send an ACK for every second packet you could easily reduce that rate if possible by your indicated max ack delay). I'm not saying we should specify any "ACK congestion control" but we should definitely make it possible to enable future experimentation on this.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585267457

---

From: ianswett <[email-masked]>
Date: Wed, 12 Feb 2020 07:48:42 -0800

I'd be happy to add text that says:
"You MAY detect the loss of ACK-only packets and use that information to adjust the congestion controller or the rate of ACK-only packets being sent, but this document does not describe a mechanism for doing so."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585270360

---

From: Martin Thomson <[email-masked]>
Date: Wed, 12 Feb 2020 12:50:23 -0800

> or the rate of ACK-only packets being sent

I wasn't suggesting that we do this bit.  Just the other piece.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585411119

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 12 Feb 2020 15:54:36 -0800

Thank you, @kazuho, for raising a point that I had completely forgotten! I remember now why I had not wanted ACK loss to drive the congestion controller. I think @kazuho and @ianswett are saying similar things, but the short of it is this: the congestion controller assumes that cwnd amount of data is in flight when it reacts to loss, reducing the cwnd. However, the cwnd has nothing to do with the amount of ACKs that are in flight, and as a result, the reduction of the cwnd in response makes no sense in this case.

(It does make an increasing amount of sense to use as the traffic in this direction is increasingly dominated by ack-eliciting data, but that also makes it increasingly likely that a congestive loss will be of an ack-eliciting packet.)

We should not change the design here. I'm fine with @ianswett's proposed addition, but I don't think we should use the loss of ACKs as a congestion signal that affects the congestion window.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585478025

---

From: Christian Huitema <[email-masked]>
Date: Wed, 12 Feb 2020 20:57:59 -0800

Are there any interop issues with this? Implementations keep tab of what is going on in all kind of ways. Many use "ack of ack" mechanisms to notice which ack have been acknowledged by the peer and which ack ranges can be safely pruned. Some do what Mirja described. Some don't care. Please do not try to overspecify this. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-585550723

---

From: Igor Lubashev <[email-masked]>
Date: Sun, 16 Feb 2020 11:43:56 -0800

The text is slightly ambiguous on the topic of whether pure-ACK packets are to be acknowledged.

Section "13.2.  Generating Acknowledgements" implies that they are ACKed later:

> Packets that are not ack-eliciting are only acknowledged when an ACK frame is sent for other reasons.

Section "13.2.1.  Sending ACK Frames" implies that they SHOULD but not MUST be ACKed:

> Every packet SHOULD be acknowledged at least once

and

> [all] packets marked with the ECN Congestion Experienced (CE) codepoint in the IP header SHOULD be acknowledged immediately

It seems strange NOT to acknowledge all packets when sending an ACK frame. What benefit is there to NOT doing so? And if there is no benefit, why not require the protocol implementations to be more predictable?

What specific response a specific congestion controller has as a reaction to a lost / EC-marked pure-ACK packet is an implementation matter, of course.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-586744717

---

From: MikkelFJ <[email-masked]>
Date: Fri, 21 Feb 2020 05:17:22 -0800

@igorlord I'm not sure there is a contradiction here. You want to acknowledge all packets eventually because as you say, it is odd to have to bypass an ACK frame. But you don't want certain frames to actively initiate ACKS. I think this is covered in the three sentences that you quote.

The odd special case is when the connection terminates. What if there are pending ACKs that you don't want to acknowledge?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-589648936

---

From: ianswett <[email-masked]>
Date: Fri, 21 Feb 2020 06:09:39 -0800

@igorlord The reason it's a SHOULD is both because it's not designed to be required for QUIC to function correctly and because of the case when no ACK frame is sent prior to a connection close.

The only advantages I can think of would arise in extremely simple implementations, such as those which only track 1 ack range.  In that case, a receiver which received an ACK-only packet and then an ack-eliciting packet with a gap between the two would want to only acknowledge the ack-eliciting packet.  

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-589668170

---

From: Igor Lubashev <[email-masked]>
Date: Fri, 21 Feb 2020 10:14:13 -0800

@mikkelfj, @ianswett Thanks, the SHOULD in "Section "13.2.1. Sending ACK Frames" being a SHOULD due to connection termination makes sense, but it would benefit for being explicit that it is the only case when packets may never get ACKed.

@ianswett Some implementations might not care about loss of ACK-only packets, while others may care about such losses. For the latter ones, knowing of such losses is as required for correct functioning as knowing of loss of packets with DATAGRAMs (what bytes count toward a congestion window is not the only implementation concern).

I do not think that there is any burden on even the simplest implementation. If the simplest implementation detects a gap, it can send a packet with an ACK. If that gap was detected due to a receipt of an non-ack-eliciting packet, a PING frame could be included with the resulting ACK (to satisfy "_An endpoint MUST NOT send a non-ack-eliciting packet in response to a non-ack-eliciting packet, even if there are packet gaps which precede the received packet._"

The bottom line is that building in a "loss ambiguity" into the protocol without a great reason is a mistake. TCP has a retransmission ambiguity, and we know it is a problem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-589770391

---

From: ianswett <[email-masked]>
Date: Mon, 24 Feb 2020 06:55:16 -0800

> I do not think that there is any burden on even the simplest implementation. If the simplest implementation detects a gap, it can send a packet with an ACK. If that gap was detected due to a receipt of an non-ack-eliciting packet, a PING frame could be included with the resulting ACK (to satisfy "An endpoint MUST NOT send a non-ack-eliciting packet in response to a non-ack-eliciting packet, even if there are packet gaps which precede the received packet."

Your solution very much goes against what I intended for that text, so I filed #3480 to clarify what I meant and add some more text about what can be expected of ACKs.  I think it makes sense to continue discussion on that issue, if there is further discussion, because this issue is about whether ACK-only packets can be declared lost, not about acknowledgement reliability.

In particular, acknowledgements are not guaranteed to be received.  Almost all packets get acknowledged, but a sender can never rely on all packets being acknowledged, even of ACK-eliciting packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3451#issuecomment-590360731