Subject: Re: [quicwg/base-drafts] Reduce restrictions on valid RTT samples (#2568)

From: ianswett <[email-masked]>
Date: Tue, 02 Apr 2019 12:03:02 -0700

@nibanks In both cases, I believe the current algorithm of limiting ack_delay to max_ack_delay makes perfect sense.  In the case of CPU load, RTT should show up in SRTT, RTTVar, or both.

In the second case, the text also advises including any expected timer delay in your advertised max_ack_delay.

This issue is about what happens if the largest acked is an ACK-only packet(not ack-eliciting).  In this case, we do ignore the RTT sample, and I do think that could turn out to be problematic, particularly in very asymmetric flows.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479150237

---

From: Nick Banks <[email-masked]>
Date: Tue, 02 Apr 2019 20:36:41 +0000 (UTC)

@ianswett my comments were in reference to @janaiyengar's last sentence:
```
Basically, I propose that we use the RTT sample if ack_delay < max_ack_delay.
```
I am arguing that we shouldn't throw away an RTT sample all together if ack_delay is greater than max_ack_delay.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479190035

---

From: ianswett <[email-masked]>
Date: Tue, 02 Apr 2019 13:51:04 -0700

Even if it is not ack-eliciting?  Doing that could subject calculations to huge and unpredictable RTT spikes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479196010

---

From: Nick Banks <[email-masked]>
Date: Tue, 02 Apr 2019 14:04:51 -0700

No. I thought Jana's comment was even for ack-eliciting packets. It wasn't clear.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479201784

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 02 Apr 2019 16:42:14 -0700

Apologies for any confusion. @nibanks I was referring to non-ack-eliciting packets in that second half of my description, where I made that proposal.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479257161

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 03 Apr 2019 12:38:27 -0700

I've been doing more thinking, and here's where I've landed. (Thanks @kazuho and @ianswett for conversations.)

The higher order principle is this: max_ack_delay suggests a contract between this endpoint and its peer. The peer is supposed to send ACKs for ack-eliciting frames within max_ack_delay amount of time. Effectively, the peer _wants_ to wait for no more than max_ack_delay amount of time, but if the actual received ack_delay is larger (due to ack loss or scheduler latency), the RTT should be increased to compensate for this additional delay beyond max_ack_delay, because it's not within the peer's control and we might consider it part of "path" delay.

With this principle in place, I think it still makes sense to make the first change I proposed: that an ACK should be used for RTT measurement if any packet it newly acks is ack-eliciting. The RTT measurement is still on the largest_acked (since ack-delay is reported for the largest_acked).

It is difficult to reason about ACKs that only ack non-ack-eliciting packets, since the max_ack_delay contract does not hold. It is specifically difficult to reason about the reported ack_delays, since the peer can wait arbitrarily long and report arbitrarily long ack_delays. It's incorrect to add any additional delays (beyond max_ack_delay) into the RTT since the increased delay was not due to uncontrollable circumstances. Simply using the ack_delay, however large it may be, allows for the peer to simply game the RTT.  Using only the samples that have ack_delay < max_ack_delay might introduce a bias. It gets complicated quickly.

So, I conclude that we should not use ACKs of non-ack-eliciting packets for RTT measurement. This is what we do right now, so that's good. I'll add some text explaining why.

We should also recommend that an endpoint that is only sending non-ack-eliciting packets should piggyback a PING frame along with an ACK roughly once an RTT so that its RTT estimate is reasonably current.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479629846

---

From: MikkelFJ <[email-masked]>
Date: Wed, 03 Apr 2019 12:46:01 -0700

> We should also recommend that an endpoint that is only sending non-ack-eliciting packets should piggyback a PING frame along with an ACK roughly once an RTT so that its RTT estimate is reasonably c…

The sounds like a complication - and extra timer check and keeping track of whether ACK eliciting packets were sent. Also, it could lead to a lot of unnecessary PONG traffic. Is it really that important to update RTT that often, and if so, wouldn't it be better to revisit the idea of measuring RTT for non-ack eliciting packets dispite the potential downfalls?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479632252

---

From: ianswett <[email-masked]>
Date: Wed, 03 Apr 2019 19:49:53 +0000 (UTC)

+1 to @janaiyengar 's last comment.  I look forward to a PR.

In regards to the last paragraph, we already have some text in transport about this, and I'm not sure how specific I want to be, given it's only really important as a mechanism for the sender to drop state if they're waiting for acks of acks.

 "Because ACK frames are not sent in response to ACK-only packets, a
   receiver that is only sending ACK frames will only receive
   acknowledgements for its packets if the sender includes them in
   packets with non-ACK frames.  "

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479633547

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 03 Apr 2019 15:36:54 -0700

My +1 as well.

I think it's reasonable to recommend endpoints to send PINGs, if it wants to calculate RTT when it is not sending data.

First of all, this is _not_ a departure from the design we have in TCP; in TCP, an endpoint that is not sending data has no way of inferring RTT from ACKs.

Second, requiring the peer to send ACKs for ACK-only packets promptly makes the specification complex. Because such a mechanism cannot be solely based on a timer (because ACK-of-ACK cannot be sent without piggybacking), even though it needs to be based on a timer (otherwise the endpoint cannot receive an ACK in a timely manner).

Considering these two aspects, I think using PINGs is simpler. Also, endpoints can choose different strategies on how frequent they would send PING; some might send a PING every one second (or 1 RTT), or a download-only application might not send PINGs at all.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-479683624

---

From: Marten Seemann <[email-masked]>
Date: Sun, 07 Apr 2019 19:49:02 -0700

> With this principle in place, I think it still makes sense to make the first change I proposed: that an ACK should be used for RTT measurement if any packet it newly acks is ack-eliciting. The RTT…

The problem with this change is that it requires a sender to keep state for non-ack-eliciting. Currently, in my implementation, I only save a sent packet (including its metadata like the time sent and the packet size) if that packet is ack-eliciting.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-480664952

---

From: ianswett <[email-masked]>
Date: Mon, 08 Apr 2019 06:23:47 -0700

@marten-seemann That seems like a sensible point.  Possibly this should be a SHOULD, not a MUST when the largest acked is not ack-eliciting?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-480829688

---

From: Marten Seemann <[email-masked]>
Date: Mon, 08 Apr 2019 07:00:03 -0700

We could also require (or at least recommend?) the peer to not send acknowledgements for non-ack-eliciting packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-480843696

---

From: Nick Banks <[email-masked]>
Date: Mon, 08 Apr 2019 07:56:57 -0700

I disagree that we shouldn't be acknowledging non-ack-eliciting packets as the largest_ack. We should always be sending the most up to date information available? @marten-seemann why don't you track the non-ack-eliciting packets? They shouldn't be very high in number or really structurally different from other packets? It seems like an unnecessary optimization that would just cause you more problems in this scenario.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-480867509

---

From: ianswett <[email-masked]>
Date: Mon, 08 Apr 2019 15:02:59 +0000 (UTC)

I agree with Nick that we should keep sending the most up-to-date information.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-480870139

---

From: MikkelFJ <[email-masked]>
Date: Mon, 08 Apr 2019 08:11:04 -0700

It's been a while since I worked actively on a QUIC impl., but in my implementation, buffers would be retained until all pending packets in a block have been acked or lost, then that block is flushed. Having an unnecessary complication to this system would be painful. If it is only the leading block, i.e. one of a few recently sent non ack eliciting packets, it won't impact the overall design as much.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-480873359

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 08 Apr 2019 15:14:38 -0700

@marten-seemann: I agree with @nibanks that tracking non-ack-eliciting packets shouldn't be much additional load, but I don't think that's necessary.

If an ACK acknowledges packets that you don't know about, they've been acked or they're non-ack-eliciting. Either ways, this ACK should not generate a valid RTT sample.

I'll also note that you are not required to use every ACK that acks any ack-eliciting packets for an RTT sample.  If you're not tracking a packet, you can't use it's ack for an RTT measurement.  Ignoring these ACKs is the only way forward, and the text will allow that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481028084

---

From: Marten Seemann <[email-masked]>
Date: Mon, 08 Apr 2019 18:08:49 -0700

It might not be a lot of additional load, but a lot of additional complexity. For example, what happens if the peer never acks an ack-eliciting packet (either because it was actually lost, or as part of an attack)? You then have to somehow clear that packet from your history. If you use the normal loss detection mechanism for it, you have to be careful not to take any congestion controller action upon loss of that packet. You also don't want to set any timers based on non-ack-eliciting packets, etc.

I never suggested sending out-of-date information, that would indeed be a bad idea. There's just no need to send an ACK at all if largest_acked is not ack-eliciting, and I'm suggesting that we recommend (mandate?) that peers don't do it. They should just wait until they receive the next ack-eliciting packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481062758

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 08 Apr 2019 18:32:59 -0700

@marten-seemann : Agreed, which is why I said that it's not necessary to track non-ack-eliciting packets.  We already say  in the transport draft that an endpoint not send an ACK if all it's received is ACKs.  Is that what you meant?

The problem here is that a peer might decide to piggyback an ACK frame with some other stuff it is about to send (MAX_STREAM_DATA perhaps). That could result in an ACK frame that acks only non-ack-eliciting packets. It seems unnecessary to put the burden on a peer to not send this ack. It's just as easy to ignore such acks when received.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481067112

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 08 Apr 2019 18:34:43 -0700

@marten-seemann : FWIW, I did suggest a way forward -- that an endpoint ignore ACK frames (for RTT sampling) when the ACK frame doesn't newly acknowledge ack-eliciting packets that the endpoint is tracking. Does this work for you?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481067417

---

From: Marten Seemann <[email-masked]>
Date: Mon, 08 Apr 2019 18:53:21 -0700

I just realized that there's a pathological traffic pattern which would prevent a peer from ever getting a valid RTT sample (e.g. if all ACKs are sent on the max_ack_delay timer, and if after every ack-eliciting packet a non-ack-eliciting packet is received). It might be safe to ignore this edge case and assume that it won't happen a lot in practice.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481070700

---

From: Nick Banks <[email-masked]>
Date: Mon, 08 Apr 2019 19:33:45 -0700

I don't want to have a design that has a certain scenario where things break down, but just ignores it. Also, I don't want to have a design where the receiver has to decide if I can send an ACK with my current state or not. Generally, I expect receivers to only keep track of the time they received the largest packet number; and just use that when they frame an ACK.

@marten-seemann as to your issues with keeping track of ACK-only packets I'd argue they are almost entirely the same as normal packets. The ONLY difference is congestion control. As far as the peer trying to attack you by explicitly not ACK'ing the packet, that's no different for any packet. The same logic applies to all packets you send. Eventually, you age out lost packets.

IMO, the simplest and most straight forward design is to always send ACK frames for your complete, current state, independent of packet contents. If the receiver of the ACK frame decides the RTT measurement shouldn't be used, that seems to be trivial enough logic to add in that specific ACK processing code. But I still argue that so long as the ACK wasn't delayed too long (~max_ack_delay) then the RTT measurement should be used.

You don't want the worst case traffic pattern of each flight/burst of packets ends with an ACK-only packet to cause you never to accept RTT measurements from the correspond ACK you get back.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481078137

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 08 Apr 2019 19:41:51 -0700

@marten-seemann : This pathological case is exactly why I'm proposing a change to "any acked packet was ack-eliciting".

@nibanks: To be clear, if you do not have any state for packets that are being acked, you have no choice but to ignore the ack. This is not a new rule, it just falls out of standard ack processing as we have it now.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481079528

---

From: MikkelFJ <[email-masked]>
Date: Mon, 08 Apr 2019 22:15:32 -0700

> The problem here is that a peer might decide to piggyback an ACK frame with some other stuff it is about to send (MAX_STREAM_DATA perhaps). That could result in an ACK frame that acks only non-ack-…

There is the interesting derived effect that if you do not ACK non-ack-eliciting packets, this could be used as optimistic ack attack mitigation. This means that you can avoid gaps in the packet number space. You could periodically coalesce non-ack-eliciting packets with other packets as necessary.

The problem is of course that the receiver needs to track those gaps, but I'm not sure that is much different than what it already needs to track.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-481105223

---

From: ianswett <[email-masked]>
Date: Mon, 15 Apr 2019 15:25:55 +0000 (UTC)

I believe this was closed by #2592 

@janaiyengar I'll let you confirm that and close this.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-483298929

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 15 Apr 2019 13:07:50 -0700

Closed by #2592 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#issuecomment-483399047

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 15 Apr 2019 13:07:53 -0700

Closed #2568.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2568#event-2277726041