Subject: [quicwg/base-drafts] How many probed paths should a server remember? (#3489)

From: Kazuho Oku <[email-masked]>
Date: Mon, 02 Mar 2020 05:45:09 -0800

At the moment, it is my understanding that we do not give any advice in particular. But to me it seems that there would be issues if the number of probed paths that the server remembers is less than the number of spare connection IDs it issues.

Consider the case of a server that advertises three spare connection IDs, but retains information of one probing path (i.e. remember one set of 5-tuple that has been probed, and the connection ID being associated to that tuple).

If a client connecting to this server probes three paths concurrently, and if it keeps on sending PATH_CHALLENGE packets periodically on that path (in order to keep the holes in NAT open), two issues arise:
* The connection ID pool of the server would run out quickly, as the server would use a new connection ID for each probing packet it sends in response (because each probing packet sent by the client would have a different source address than the previous packet). This is an issue, as we state that _an endpoint MAY limit the frequency or the total number of connection IDs issued for each connection to avoid the risk of running out of connection IDs_ [section 5.1.1](https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#section-5.1.1-5).
* The purpose of path probing is to prepare an alternate path beforehand. That benefit diminishes if the server remembers only one (or few) of the paths being probed.

Instead of recommending servers to remember paths as much as the number of spare connection IDs that they issue, we can introduce a transport parameter that caps the number paths that a client may probe. But IMO having a new TP seems unnecessarily complex, because I think that servers can simply associate path information to each active connection ID it maintains.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489

---

From: ianswett <[email-masked]>
Date: Mon, 02 Mar 2020 05:54:49 -0800

Given QUIC v1 is chartered to support connection migration and not multipath, we could also say clients shouldn't probe more than one extra path at once.

I do not want to add another TP for this use case, because I don't think it's sufficiently valuable.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-593413292

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 02 Mar 2020 07:53:58 -0800

> Given QUIC v1 is chartered to support connection migration and not multipath, we could also say clients shouldn't probe more than one extra path at once.

I'm fine with having a hard-coded limitation, though if we are to set the maximum to one, then we need to decide whether or not migration due to SPA is counted. It would be simpler to include it, but the potential downside would be that probing using a different network interface would be forbidden until migration to the preferred address concludes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-593471801

---

From: Martin Thomson <[email-masked]>
Date: Wed, 11 Mar 2020 22:45:12 -0700

So a server doesn't need to probe in response to a client that probes a path, so it doesn't really have any state to attribute to probed paths.  A server can just respond to probes.  The cost to it, if it doesn't, is that it has to probe after a migration and that could have performance implications.

I would be happy saying that an endpoint ideally both probes and remembers the status of those probes for as many paths as it has issued, active connection IDs.  Probing paths and remembering the status of probes partly mitigates the reduction in CWND that follows migration, so there is a potential performance gain from keeping this state.  But the window is still likely reset, so the performance hit is essentially unavoidable either way.

The total number of paths a server might have to remember is limited by the active_connection_id_limit of the client, but it can be less (as you already noted), so the server has the ability to limit its exposure.

Thus, a server can probe and remember the status of as many paths as it likes.  I'd say that this is a fine piece of advice, but that's all it needs to be.  I agree that yet another transport parameter is not a great reaction to this relatively minor issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598017306

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 11 Mar 2020 22:54:36 -0700

@martinthomson 
> So a server doesn't need to probe in response to a client that probes a path, so it doesn't really have any state to attribute to probed paths. A server can just respond to probes. The cost to it,…

I think I disagree with the observation.

The problem of responding to a probe without retaining state for that path is that the server burns a CID for each probing packet. As stated in the [original comment](https://github.com/quicwg/base-drafts/issues/3489#issue-573984694), the worst case scenario is that CIDs will be consumed at a high rate when a client probes multiple paths simultaneously.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598019503

---

From: ianswett <[email-masked]>
Date: Thu, 12 Mar 2020 06:21:43 -0700

> The problem of responding to a probe without retaining state for that path (in addition to this performance implication) is that the server burns a CID for each probing packet. As stated in the [original comment](https://github.com/quicwg/base-drafts/issues/3489#issue-573984694), the worst case scenario is that CIDs will be consumed at a high rate when a client probes multiple paths simultaneously.

Personally, I don't think this worst case scenario is very bad at all.  But I'd support adding some cautionary/explanatory text.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598182246

---

From: Eric Kinnear <[email-masked]>
Date: Thu, 12 Mar 2020 12:33:09 -0700

I don't think migration vs. multipath is currently the distinction for how many paths a client probes, it's the restriction on how many paths the client _uses_ at any one time, and I think that's appropriate. There are many cases where I'd want to be able to probe more than one path, and assuming that a device has a single network attachment (where we need to include IPv4 and IPv6 as more than one given that you need a new CID for each local address) doesn't make sense either.

The number of paths that a client can probe simultaneously is limited by how many CIDs the server is willing to issue to that client. 

> Instead of recommending servers to remember paths as much as the number of spare connection IDs that they issue, we can introduce a transport parameter that caps the number paths that a client may…

If adding that TP is unnecessarily complex (and I agree that it probably is), what's wrong with recommending servers to remember paths up to as many as the number of spare CIDs that they issue? The whole point of issuing more than one spare CID is that the other endpoint may need to probe additional paths or those probes may be lost, but if you aren't willing to deal with things coming in from 100 places, don't issue 100 CIDs. Obviously that's an exaggeration, but 2/4/6/8 for each client puts a nice bound and is roughly what I'd expect to be sending in a TP anyways.

There's already a mechanism where the person doing the probing can "retire" that CID and signal to you that you no longer need to have that state around, and that's tied directly into their ability to generate new state that you'd need to keep around.

In essence, I think I agree with @martinthomson's statement above, if a server needs to reduce its exposure further it can do so, at the potential cost of performance or burning CIDs faster (as @kazuho points out). 

Enhancing the text around the existing sentence in Section 5.1.1 (that cautions against issuing too many CIDs) to also say that a server needs to maintain this state and should consider that when issuing new CIDs could be reasonable, but I wouldn't consider that time for a new TP?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598376105

---

From: MikkelFJ <[email-masked]>
Date: Thu, 12 Mar 2020 14:13:57 -0700

I tend to agree with Eric, but I think there are some aspects to consider:

You don't want to limit yourself to one extra path as Ian suggests because you might have 3 or 4 paths forward, e.g. sittig on Wlan 2.5GHz with optional 5GHz if the first becomes congested, along with a 4G and 5G modem path. You need to monitor these and choose the best path forward once the preferred path fails. Maybe this is a bit extreme, but just to illustrate path migration isn't linear.

At the same time, the peer need to issue enough IDs to not stall progress in addition to accommodating the number of paths forward a typical client might need. I'd say having double the number of CIDs in spare is probably OK, but maybe there isn't a need for a spare CID for every path. So it could be number of paths + 2 or something.

Thus, a recommendation could be that a client SHOULD not track paths beying having 2 spare CIDs and a server SHOULD be prepared to track the number of issued CIDs minus two, as a general rule. Exceeding that may require delayed probing which could lead to temporary delays.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598420935

---

From: MikkelFJ <[email-masked]>
Date: Thu, 12 Mar 2020 14:19:48 -0700

BTW: a related migration issue: a client might have to track separate root certificates for separate paths because one path could be on a corporate WLAN with substituted root certificates for traffic monitoring while the other path happens over 4G or 5G. This is probably not something that can be supported on a single connection neither in QUIC v1 or v2, but it is actually a concern in a real use case I'm aware of.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598423047

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 12 Mar 2020 14:28:33 -0700

I tend to agree with what @erickinnear says.

If I haven't been clear, I share the view that probably the best solution to this issue is to recommend servers to remember at most N paths, where N is the maximum number of active connection IDs it provides to a peer at any given moment.

I think @mikkelfj is correct in pointing out that the client might not use all the spare CIDs for probing, but I would argue that that can be a local decision of each client implementation. In terms of operational complexity on the server-side, the recommendation being N or N - 1 or N / 2 does not matter at all. Let's simply recommend N for the servers, and be silent about what clients should do.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-598426118

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 17 Mar 2020 17:25:38 -0700

I think it makes sense to simply advise that servers should remember as many paths as the additional CIDs they hand out, as @erickinnear suggests. @erickinnear, can you write up a PR for this?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-600363904

---

From: Eric Kinnear <[email-masked]>
Date: Tue, 17 Mar 2020 17:28:40 -0700

Sure, will do! I'm thinking this will live right next to the issuing CIDs bit: as we note that you may want to rate limit, we can say it would be a good idea to recognize the state that may be associated with each CID, in this case, the path validation status (which is needed for the validating paths section anyways, since we say you might have remembered it from previously) and any other per-path state that you might keep.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3489#issuecomment-600364586