Subject: [quicwg/base-drafts] disable_active_migration with SPA (#3765)

From: Igor Lubashev <[email-masked]>
Date: Mon, 15 Jun 2020 09:49:59 -0700

Issue #3608 by @MikeBishop was about deployments where migration with the handshake address was problematic, but migration with server preffered_address (SPA) was ok.

However, there are deployments where servers still have a preferred address, but client migration using either server address is not likely to work well,

For example, a server is a backend in front of a load balancer (and hence wants to use a prefered address that does not go via the load balancer for improved performance and reliability).  However, the server is in a network location that is unlikely to be accessible or be a logical choice when changing to a different network attachment provider.

So it is not a matter of "disambiguating connections using CIDs" (_as posed by @kazuho in response to @DavidSchinazi in https://github.com/quicwg/base-drafts/issues/3608#issuecomment-631869936_), but it is a matter of server's network location.

With #3670 PR, how is such server able to provide a SPA and still request that clients do not attempt an active migration after switching to SPA (occasional NAT rebidings are fine; it is changes to another network attachment provider -- active migrations -- that are not ok)?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 15 Jun 2020 10:18:48 -0700

@igorlord 
> However, there are deployments where servers still have a preferred address, but client migration using either server address is not likely to work well,

I think the question is how such an endpoint running the preferred address distinguish between the connections that migrate to that preferred address?

When a client migrates to the preferred address, the client-side tuple is allowed to change. Even if the client implementation continues to use the same port, the tuple could still change due to a NAT in the middle. Assuming that is the case, a server providing SPA is, in practice, required to be capable of using the connection ID to determine the connection. 

To paraphrase, the requirement for supporting SPA is equivalent to supporting migration. Both require CID-based routing. Hence no reason to have a knob to disable migration when SPA is being used.

> For example, a server is a backend in front of a load balancer (and hence wants to use a preferred address that does not go via the load balancer for improved performance and reliability). However, the server is in a network location that is unlikely to be accessible or be a logical choice when changing to a different network attachment provider.

I wonder if such deployment would have worked reliably even before draft-28. How can a server be sure that the client would not connect through a different network attachment provider? As stated above, it is my understanding that a client is allowed (and have to be allowed) to use any source tuple when migrating to SPA.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644263033

---

From: ianswett <[email-masked]>
Date: Mon, 15 Jun 2020 12:12:51 -0700

@igorlord it appears you're describing a case when policy precludes migrating with the SPA, even though it is likely to work from a technical perspective.

In this case, I believe you need to send a stateless reset to any clients that change IPs to ones which are not supported.  You'll want to do that anyway, but that does cost a round trip.

The design change would be to add a disable migration feature for every server IP, but I'm not sure that's warranted to save a round trip in this specific case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644324972

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 15 Jun 2020 12:57:22 -0700

There is no problem in such deployment to do the right thing regarding Connection ID tracking, etc.  The problem that a change in network attachment provider is unlikely to work - you could either blackhole, get ICMP unreachable, or your performance would be slow.

If the client honors "disable_active_migration", they would open a new connection from the new network attachment provider (and maybe get a different deployment).  If the client ignores disable_active_migration, they are on their own.  (After path verification, the HTTP3 server would likely send GoAway or close the connection.)

>  it is my understanding that a client is allowed (and have to be allowed) to use any source tuple when migrating to SPA.

This is the consequence of #3608 (#3670), which breaks the deployment scenario in this issue.  (It used to be handled ok before.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644346096

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 15 Jun 2020 14:29:10 -0700

> This is the consequence of #3608 (#3670), which breaks the deployment scenario in this issue. (It used to be handled ok before.)

I disagree. Since the earlier versions of the draft, a client has been allowed to use a different address, when migrating to SPA. A prominent example is a client switching to a different IP protocol (i.e. from v4 to v6, or vice versa).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644400199

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 15 Jun 2020 14:53:21 -0700

@ianswett Policy could be one reason.  Technical network topology is a very likely other reason.

Think of Netflix or YouTube or CDN caches deep inside a provider network.  It is not just a matter of policy that things would not work great, if you migrated to a different provider.  This is a very common deployment model on the Internet today.  Pre- #3608 (#3670),  such deployments would work fine.  The new language that exempts SPA from "disable_active_migration" breaks these deployments.

Maybe there is a need to distinguish "disable_network_migration" (original meaning of "disable_active_migration") from "preferred_address_migration_only" (the new concept introduced by #3670).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644409753

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 15 Jun 2020 15:05:49 -0700

@kazuho 

> Since the earlier versions of the draft, a client has been allowed to use a different address, when migrating to SPA. A prominent example is a client switching to a different IP protocol (i.e. from v4 to v6, or vice versa).

Of course, by default the client can use any address when migrating.  A Carrier NAT may switch source IPs automatically anyway.  The question is interaction of _disable_active_migration_ and _SPA_ features.  We've previously been treating disable_active_migration as "do not switch client addresses deliberately", while SPA is a request to "switch server address".

There are important use cases for both types of behaviours -- ok for clients to switch networks when using preferred address and not ok for clients to switch networks when using preferred address.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644414377

---

From: Mike Bishop <[email-masked]>
Date: Tue, 16 Jun 2020 15:16:04 +0000 (UTC)

The discussion on the previous issue was that if this scenario was needed, an extension could be defined that gives the same guidance for the SPAs.  That offers better flexibility, and clients would be motivated to implement if it saves them a round trip discovering a broken connection when connections will break anyway.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644829920

---

From: Igor Lubashev <[email-masked]>
Date: Tue, 16 Jun 2020 09:27:59 -0700

@MikeBishop I would think the new behaviour is better delegated to an extension.

The new language (w/o an extension) irreparably causes performance problems for the kind of very common deployments I've indicated -- if they use disable_active_migration w/ SPA, migrating clients will be affected after switching to SPA; if they use disable_active_migration w/o SPA, clients cannot use a more performant/reliable preferred address.

However, if you have a server deployment where client migration w/ SPA is fine, just provide SPA and do not set disable_active_migration.  The purpose of SPA is to tell the client "you will get better service using that other address".  So clients will just switch to SPA as soon as they can, and everyone will be happy.

The new behaviour is pandering to not-very-well-behaved clients that do not obey SPA at the expense of well-behaved ones.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644871386

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 17 Jun 2020 02:22:40 -0700

@igorlord To me the description of the problem that you stated in https://github.com/quicwg/base-drafts/issues/3765#issuecomment-644346096 helped me better understand what you are concerned about. Thank you for providing them.

Though I tend to wonder if we'd really want to use SPA in such case.

The rationale behind SPA is that providing a *preferred* address improves reachability. In the past, we considered two cases. The first one is when an anycast address is used as the handshake address. Then, switching to a unicast address is preferable, as it improves reachability. The second case that we considered is when a legacy load balancer that can route packets only by looking at the 4-tuple is being used. In such case, letting each backend server provide its own address as the preferred address improves reachability.

But in the scenario that you describe, I am not sure if the preferred address would provide better reachability than the original address. If that is the case, wouldn't it be sufficient to continue using the original address throughout the lifetime of the connection?

Or is your case specifically about using SPA to bypass the legacy load balancer in an environment that has reachability issues with the preferred address too? If that is the case, I can understand the concern, and I would not oppose to having a new signal, either within QUICv1 or as an extension. Though I would oppose to changing the current definition of `disable_active_migration`, as it sounds to me like a corner-case of a corner-case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-645260401

---

From: Igor Lubashev <[email-masked]>
Date: Wed, 17 Jun 2020 16:23:45 +0000 (UTC)

Thanks for the analysis, @kazuho.  The case I described could, indeed, involve a simple 4-tuple legacy load balancer in front of QUIC-supporting backends.  But it could also involve a QUIC-aware load balancer and still benefit from SPA for improved reliability and performance (removal of one more system with its failure modes and removal of some internal forwarding step within a deployment).

And then there is still another case -- using an anycast address as the handshake address and landing on a server deep inside your provider network.  You will want to use SPA to avoid a risk of anycast route change during a long-lived connection.  But you do not want the client with a secondary connection to another provider trying to migrate.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-645459858

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 17 Jun 2020 16:42:09 -0700

> And then there is still another case -- using an anycast address as the handshake address and landing on a server deep inside your provider network. You will want to use SPA to avoid a risk of anycast route change during a long-lived connection.

Could you please elaborate why it is beneficial to migrate from an anycast address to SPA in this particular case?

It is my understanding that this is about a server located in an ISP network, hosting an anycast address and a preferred address. I would assume that that server is only reachable from client within that ISP, regardless of the address being the anycast address or the preferred address.

Assuming that is the case, migrating to a preferred address does not improve the chance of *each connection* surviving across unintentional client address migrations. Regardless of if the client uses the original address or the preferred address to designate the server, the connection survives while the client stays in the ISP network. If it goes out, then the connection dies regardless of the server address being used.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-645682509

---

From: Igor Lubashev <[email-masked]>
Date: Wed, 17 Jun 2020 19:19:23 -0700

I had in mind a large ISP spanning a continent. There can be dozens of POPs inside that ISP announcing an anycast address. A change in routing inside that network can cause a different POP being selected. You want to move off anycast address, because anycast is inherently unstable.

This is what I had in mind for the anycast case.  The other case, a desire to stop directing the flow via the load balancer and pointing the client at the server directly, is a distinct case. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-645728456

---

From: Mike Bishop <[email-masked]>
Date: Mon, 22 Jun 2020 10:39:28 -0700

@kazuho, you're correct that the session would not survive handover regardless.  At that point, it's about the performance improvement of telling the client that it's better to spin up a new connection instead of making any attempt to migrate.  If you lose your network address, the connection is dead; recover as quickly as you can.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3765#issuecomment-647675109