Subject: [quicwg/base-drafts] recovery: Clarifying the priority of Probe packet (#3780)

From: Junho Choi <[email-masked]>
Date: Thu, 18 Jun 2020 09:51:46 -0700

Currently "Sending Probe Packets" define the priority what to be in Probe packet as follows:

- A probe packet SHOULD carry new data when possible.
- A probe packet MAY carry retransmitted unacknowledged data when new data is unavailable.

And also says "Implementations MAY use alternative strategies for determining the content of probe packets, including sending new or retransmitted data based on the application's priorities."

When there is no new data, we need to choose which one to retransmit when there is a multiple data loss in the sender. Since QUIC's Probe packet approach is a combined experience of TCP TLP and RTO and I believe Probe packet is more similar to TLP. RACK draft says (https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rack-08#section-7.3) "The probe data segment can either be new, previously unsent data, or a retransmission of previously sent data just below SND.NXT" so it's pretty clear which one to send among the lost data (newest one).

Is there any reason that it's up to the implementers when it need to choose which one to send from lost data? It's better to clarify which one to send among retransmitted unacknowledged data, from oldest or newest when there is multiple data packet lost.

I think it's better to send newest (unacked) one first to follow the reasoning of TLP (new to the old), but there may be some reason why the current draft doesn't define it clearly.

https://github.com/cloudflare/quiche/pull/562#discussion_r441864077 has some background and my comment.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780

---

From: Christian Huitema <[email-masked]>
Date: Sun, 21 Jun 2020 23:53:04 -0700

A lot of the reasoning behind TLP comes from TCP counting bytes and not counting packets. In TCP, new data means a new sequence number, thus a higher chance to trigger ACKs or SACKS. But for Quic, any new packet will have a new sequence number, so from a transport engine point of view it does not matter very much what data is in that packet. Logically it should be the most urgent data among all the available streams. So first pick the most urgent stream with something to send; if that something is old data presumed lost, send it; if there is no data presumed lost, and there is new data available, send that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647323761

---

From: Junho Choi <[email-masked]>
Date: Mon, 22 Jun 2020 10:23:42 -0700

> Logically it should be the most urgent data among all the available streams.

I think this is the right answer but need some clarification on the draft text.

I think there is 2 different approach for the content of Probe packet depending on the role of Probe packet: 1) if the tail loss is most concerning (what's what I understand from TLP) then we would send a new data if available, or newest lost to move the right edge and update ACK ranges. 2) If HoL is most concerning then we would send oldest lost data (or highest priority data) to move the left edge (but it may be delivered already, because we may lost some acks)

However the draft says we always try to send a new data first regardless of lost data, so I assume 1). But if the priority is simply on the urgent data, we don't need to send a new data first.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647667407

---

From: Vidhi Goel <[email-masked]>
Date: Mon, 22 Jun 2020 20:46:12 -0700

This is a great question and I incline towards sending the newest (tail) unacknowledged data first (same as TLP in TCP). Lets say we sent packets 1,2,3,4 & 5 at the same time and assume 1 & 5 were lost. Now, PTO fires.

1. Lets say, on PTO expiry, we retransmit the oldest packet i.e. 1. When the PTO timer fires a second time, only then we would send pn=5.
2. OTOH, if we retransmit the tail first, i.e. pn=5, it provides faster recovery, because an arriving ACK with largest pn=4 will mark pn=1 as lost and we will retransmit that immediately.

The unacknowledged packets at the head have the advantage of subsequent packets allowing them to recover via ACKs marking them as lost. Not so much for the tail packets.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647890042

---

From: Marten Seemann <[email-masked]>
Date: Mon, 22 Jun 2020 20:50:01 -0700

> 2. OTOH, if we retransmit the tail first, i.e. pn=5, it provides faster recovery, because an arriving ACK with largest pn=4 will mark pn=1 as lost and we will retransmit that immediately.

@goelvidhi I'm not sure I understand the argument here. Why would the content of the retransmission matter for declaring packets lost?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647890906

---

From: Vidhi Goel <[email-masked]>
Date: Mon, 22 Jun 2020 21:00:42 -0700

We declare packets lost when 

> A packet is declared lost if it meets all the following conditions:

> The packet is unacknowledged, in-flight, and was sent prior to an acknowledged packet.
> Either its packet number is kPacketThreshold smaller than an acknowledged packet (Section  6.1.1), or it was sent long enough in the past (Section 6.1.2).

So, a tail packet (with no packets after it) wouldn't be marked lost until after some earlier packets were retransmitted.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647893273

---

From: Martin Thomson <[email-masked]>
Date: Mon, 22 Jun 2020 21:02:44 -0700

I don't understand why people think that it is important to standardize this.  Do we have research that points more clearly to one choice being better than another?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647893739

---

From: Marten Seemann <[email-masked]>
Date: Mon, 22 Jun 2020 21:15:59 -0700

> So, a tail packet (with no packets after it) wouldn't be marked lost until after some earlier packets were retransmitted.

That's correct, but a retransmission always has a higher packet number than earlier sent packets, so the contents of the packet don't matter, right? Or am I misunderstanding your argument?

I agree with @martinthomson. We already have some advice about retransmitting packets in section 13.3:
> Endpoints SHOULD prioritize retransmission of data over sending new data, unless priorities specified by the application indicate otherwise; see Section 2.3.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647896665

---

From: Eric Kinnear <[email-masked]>
Date: Mon, 22 Jun 2020 21:24:08 -0700

> That's correct, but a retransmission always has a higher packet number than earlier sent packets, so the contents of the packet don't matter, right?

This seems right to me; we're retransmitting frames, not packets.

> Do we have research that points more clearly to one choice being better than another?

If folks get a chance to experiment with this and publish their results, that would be super cool!

But it seems like that's somewhat orthogonal to getting this spec finished, since so far we've settled with "doing the default thing should get you functional QUIC". 

It seems like all options here work well enough that we don't need further text unless either we have some data or we have a principle of why one option is deterministically better?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647898608

---

From: Vidhi Goel <[email-masked]>
Date: Mon, 22 Jun 2020 21:41:40 -0700

> That's correct, but a retransmission always has a higher packet number than earlier sent packets, so the contents of the packet don't matter, right? Or am I misunderstanding your argument?

I am not saying that the contents matter. What I am saying is if we send a PTO probe for the tail packet, we would recover faster as it doesn't have to wait for the retransmissions for earlier packets (1 as 6). With this approach we would be able to retransmit both tail (via PTO) and head (via ACK for pn 4) faster than the other approach.

That being said, I am neutral towards making any change to the draft and leaving it to implementors is a fine option.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647903349

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 22 Jun 2020 22:14:03 -0700

I also think that we do not need to standardize this.

If I understand correctly, we do not define how data should be sent initially. I think the only point we make is that _a QUIC implementation SHOULD provide ways in which an application can indicate the relative priority of streams_ (https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-2.3). I do not think that here is a baked-in assumption that the receiving application would be reading the stream from the beginning.

Assuming that my understanding is correct (it could be wrong), I do not think we need to talk any more about what should be done to transmit data the second time.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-647912394

---

From: Martin Thomson <[email-masked]>
Date: Tue, 23 Jun 2020 02:13:48 -0700

> This seems right to me; we're retransmitting frames, not packets.

Well... We retransmit information, not frames or packets. :)  Those people who do retransmit frames are not strictly *wrong*, but they are also not going to get the best results either.

Kazuho has provided what I think is the definitive response here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-648016355

---

From: Junho Choi <[email-masked]>
Date: Tue, 23 Jun 2020 02:53:30 -0700

My point is what to send in Probe packet, not a usual retransmission detected by loss detection algorithm. 

recovery 6.3.4 has the following sentence:

   A) When the PTO timer expires, and there is new or previously sent
   unacknowledged data, it MUST be sent.  A probe packet SHOULD carry
   new data when possible.

   B) A probe packet MAY carry retransmitted
   unacknowledged data when new data is unavailable, when flow control
   does not permit new data to be sent, or to opportunistically reduce
   loss recovery delay.

   C) Implementations MAY use alternative strategies
   for determining the content of probe packets, including sending new
   or retransmitted data based on the application's priorities.

While B) can be a research topic which one is better to retransmit if there is multiple loss reported (oldest or newest), I wonder if A) and C) seems conflicting because that new data MUST be sent limits C) to pick up a strategy to send which data to retransmit.

I think there is some history why A) is in the first place and I believe it's from TLP definition. As many people pointed out, QUIC's monotonic packet number property will make better ACK ranges report coming back when we send a Probe packet.

This makes less importance on sending a new data as MUST. If so, I think we don't need to have a new data transmission as MUST.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-648037106

---

From: Christian Huitema <[email-masked]>
Date: Tue, 23 Jun 2020 15:45:00 -0700

I guess @junhochoi has a point. There is probably a tad too much text in the recovery draft, and the MUST there would be better replaced by an RFC 6919 variant.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-648466426

---

From: Martin Thomson <[email-masked]>
Date: Fri, 26 Jun 2020 00:16:41 -0700

Having re-read these, the inconsistency isn't that inconsistent.  But it could still be improved.

The rules require that you send unacknowledged data, whether it was sent before or not.  Then they recommend unsent data, and allow retransmissions (the latter of which is somewhat redundant).  Then (C) is something about not really meaning the first MUST.

If anything, this should be tuned down to a recommendation to send unacknowledged data and to steer clear of any proscription regarding new transmissions vs. retransmissions.

However, we should try to seek consensus on a change because it would seem like any change here is a change to normative language.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3780#issuecomment-650019832