Subject: [quicwg/base-drafts] QUIC connection migration and IPv6 only NAT64/DNS64 Networks (#2122)

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 06:39:10 -0800

Here's one issue that I believe should be addressed either in the final version.

Client device has both cellular and wifi interfaces.
Server has dual stack deployment.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122

---

From: Mike Bishop <[email-masked]>
Date: Wed, 12 Dec 2018 06:47:27 -0800

I doubt this is feasible in our current architecture.  It would effectively require the server not only to have multiple addresses from the client's perspective, but require the server to have addresses that it can't tell the client about because they're local to the client (NAT64).  I suspect the best we'll get for v1 is that migration fails, but the client successfully opens a new connection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446612834

---

From: Mike Bishop <[email-masked]>
Date: Wed, 12 Dec 2018 06:49:56 -0800

This sounds much more like a variant of full multipath, which is already explicitly out-of-scope for v1.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446613731

---

From: MikkelFJ <[email-masked]>
Date: Wed, 12 Dec 2018 14:56:36 +0000 (UTC)

Although not strictly related to this problem, there are also concerns about DNS query privacy and host name exposure in TLS handshake. So a version x might look into the broader DNS / cert auth problem along with multiple server paths.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446616025

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 15:00:14 +0000 (UTC)

"It would effectively require the server not only to have multiple addresses from the client's perspective"

Not necessarily. The issue is making existing deployments work with the clients.
For now most server deployments are IPv4 only or Dual stack.
If servers are IPv4 only, they can remain IPv4 and a client side solution can work even when one interface has IPv4 only and the other one has IPv6 only connectivity.

For servers that are IPv4 only servers, one could begin on WiFi and then synthesize IPv6 address when WiFi goes away and the connection is being migrated to cellular with IPv6 only NAT64/DNS64 connectivity.
For the other way around, if one gets synthesized AAAA from DNS64, one could try and parse out the last 32 bits and attempt a connection over WiFi with v4 only configuration.

===

The problem that needs more than just client side work-around is for making things work with dual stack sever deployments which I would become more and more common going forward.

"but require the server to have addresses that it can't tell the client about because they're local to the client (NAT64)."

I didn't get this. Can you please elaborate? Server must only communicate their public addresses that are global in scope.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446617371

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 15:02:44 +0000 (UTC)

> Although not strictly related to this problem, there are also concerns about DNS query privacy and host name exposure in TLS handshake. So a version x might look into the broader DNS / cert auth pr…

I was implying these frames to be sent post handshake, encrypted and protected and not exposing anything in the TLS handshake itself.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446618306

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 15:06:01 +0000 (UTC)

> This sounds much more like a variant of full multipath
This is only about sharing endpoint information. The use could be limited to connection migration if the endpoints don't support full fledged multi-path.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446619556

---

From: MikkelFJ <[email-masked]>
Date: Wed, 12 Dec 2018 07:11:01 -0800

> I was implying these frames to be sent post handshake, encrypted and protected and not exposing anything in the TLS handshake itself.

Not arguing on that, just suggesting that address lookup can have a much broader scope and when a single QUIC endpoint starts acting a bit like a DNS perhaps that scope should be taken into account. But certainly not for v1.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446621471

---

From: MikkelFJ <[email-masked]>
Date: Wed, 12 Dec 2018 07:19:46 -0800

Note that generally an endpoint does not know its own public address because it can sit behind load balancers and multiple layers of NAT devices. So having the endpoint publish alternative addresses will not the solve the problem in general.

In fact the scenario could be reformulated to load balancer on multiple stacks instead of the server endpoint. You end up needing some DNS service that can deal with it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446624696

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 16:41:40 +0000 (UTC)

I see the point about load-balancers and NATs.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446655676

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 12 Dec 2018 09:13:03 -0800

I agree that having a way for the server to signal its IP-addresses to the client is very important. Even in the v1 of Quic.

Otherwise, Quic migration will hardly be effective.

To easily deploy Quic behind a load-balancer one simply assigns unicast IPs to the servers and lets Quic announce these (v4 and v6) IPs to the client as "preferred address". Then the client can migrate by using the preferred address. There needs to be a way to announce the preferred address in both v4 *and* v6.

This should address Scenario 1 and Scenario 2 that @plakhera is describing.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446667137

---

From: MikkelFJ <[email-masked]>
Date: Wed, 12 Dec 2018 09:43:51 -0800

> To easily deploy Quic behind a load-balancer one simply assigns unicast IPs to the servers and lets Quic announce these (v4 and v6) IPs to the client as "preferred address".

There is nothing easy about assigning IPs. It can be a lot more manageable to update a DNS entry than to updating all server endpoints in cluster, but it obviously depends on use case and environment.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446677534

---

From: Mike Bishop <[email-masked]>
Date: Wed, 12 Dec 2018 17:55:08 +0000 (UTC)

@plakhera, my comment about addresses local to the client was thinking more about a server which is v4-only won't know how to construct a corresponding IPv6 address for the client network's NAT64.  The client would expect to discover those via DNS64 and needn't know that there's a relationship between the addresses.

Is there precedent for the NAT64 address mapping being (un)done on the client?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446681393

---

From: janaiyengar <[email-masked]>
Date: Wed, 12 Dec 2018 17:46:22 -0800

So, preferred_address has slightly different semantics than what you're asking for. It currently means that the server prefers that the client use the indicated address. What you want is an _alternate_ address, something that the client can use if it migrates to a network on a different address family.

I think we can get the low-hanging fruit here by having alternate_address_v4 and alternate_address_v6 transport parameters. That said, I'm not convinced of how important this is to solve in v1. This affects only migrations, and then only those that are from networks on one address family to a different one.

I'll note that an explicit ADD/DELETE_ADDR doesn't work because of NATs (as @mikkelfj notes).  That is fine; in the spec, we've basically used receiving a packet from a new address as an implicit ADD_ADDR signal. The problem you're running into right now is that we don't support server migration in v1, which doesn't allow a server to send a packet from an unknown address to a client.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446811780

---

From: ianswett <[email-masked]>
Date: Wed, 12 Dec 2018 20:12:21 -0800

To echo @cpaasch point, this could be a large problem for connection migration.

Without a way to communicate an alternate address to the peer, it seems like v6 to v4-only will consistently fail?

If connection migration is best effort(ie: expected to fail some user-visible fraction of the time), this may be Ok.  But if it's expected to work ~100% of the time when an alternate network is available, this is a real problem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446836642

---

From: Martin Thomson <[email-masked]>
Date: Wed, 12 Dec 2018 20:23:31 -0800

I always expected this to work on a best-effort basis.  The problem with doing something stronger here is that it blows out the scope.  Badly.  Migration is hard enough when only one endpoint moves, but this would require a lot more complexity and machinery.

Mike labelled this `quicv2`, which is entirely appropriate.  Though it might be OK to define an extension that parallels preferred_address that provides a v4 fallback address.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446838279

---

From: janaiyengar <[email-masked]>
Date: Wed, 12 Dec 2018 22:02:41 -0800

This is a problem for migration between different address families. It will always fail under those conditions. This is something we should fix, but I fear that without some experimentation, we're not going to get it right. 

I agree with Martin -- having an extension define the alternate v4/v6 address in transport params may be the way to sensibly do this experiment.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446852478

---

From: erickinnear <[email-masked]>
Date: Wed, 12 Dec 2018 22:28:09 -0800

To the point made by @MikeBishop, the NAT64 synthesis can generally be done on device for some of the problem cases here. 

It seems like between (a) resolving both address families and potentially preferring the one matching the address family available on your other interface(s) and (b) address synthesis on-device, many of the cases are covered already. 

Beyond that, I was also under the impression that this was best effort. To that end, I'm not sure how much this needs to be addressed in v1, as I don't know if significant added complexity is the right tradeoff -- an extension sounds like a great place to experiment with this. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446857058

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 22:41:53 -0800

> Is there precedent for the NAT64 address mapping being (un)done on the client?

Server need not know about the DNS64 prefix in client's network. It would only needs to share its addressing information with client. it's the responsibility of the client to synthesize IPv6 address from it if needed (post PLAT discovery).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446859554

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 22:58:46 -0800

> I think we can get the low-hanging fruit here by having alternate_address_v4 and alternate_address_v6 transport parameters. 

+1. Just that bit by itself (either having two new TPs alternate_address_v4 or  alternate_address_v6 or splitting preferred_address to _v4 and v6) would help in most cases.

> That said, I'm not convinced of how important this is to solve in v1. This affects only migrations, and then only those that are from networks on one address family to a different one.

It is increasingly becoming common for cellular carriers to deploy IPv6 only NAT64/DNS64 networks, even in countries where ISPs have stayed IPv4 only on the CPE, mostly because of differences in how connectivity is achieved in two scenarios and the use-cases.
It won't be far fetched to think that this disparity would be widened by end of next year.

The current limitation introduces legitimate concerns of deploying dual stack configuration for services that still desire transport protocol level connection migration support.

Also we should probably try our best to have version one be address family agnostic. The fact that there is only one preferred_address family parameter doesn't help.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446862799

---

From: plakhera <[email-masked]>
Date: Wed, 12 Dec 2018 23:04:16 -0800

@ekr 

> It seems like between (a) resolving both address families and potentially preferring the one matching the address family available on your other interface(s) and (b) address synthesis on-device, ma…

Most combinations of server configurations and network configurations work.
What does not is dual stack server deployment and starting on an interface that has ipv6 only Nat64/dns64 connectivity and moving to ipv4 only interface.

DNS may be configured for load-balancing itself, so to assume that A record points to the same node and AAAA record points to may not be correct.
Also, as you may be aware, some devices may suppress sending A query altogether on an interface that is connected to IPv6 only network (even with NAT64/DNS64).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446863947

---

From: MikkelFJ <[email-masked]>
Date: Thu, 13 Dec 2018 02:32:57 -0800

Also note that a client with multiple addresses may have a problem with 0-RTT tokens that are fixed to a single path, i.e. client gets a token for IPv4 and but later uses it on IPv6.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-446920154

---

From: David Schinazi <[email-masked]>
Date: Thu, 13 Dec 2018 09:59:12 -0800

I agree with @plakhera ,  this is an important issue, and will be increasingly common as more cellular carriers switch to IPv6-only. At Google we've been seeing some number of as-yet unexplained connection migration failures, and our working hypothesis is address family mismatch. I think this does work with the concept of `preferred_address`. If a QUIC server deployment uses anycast IPs for initial packets then asks the client to switch to an address that uniquely identifies this server, why constrain the client to one address family? If the client used a variant of Happy Eyeballs, it could have ended up connecting using IPv4 even though it prefers IPv6. Sending both address families in `preferred_address` allows the client to switch back to the best address family.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-447062473

---

From: Mike Bishop <[email-masked]>
Date: Thu, 13 Dec 2018 12:02:41 -0800

As @janaiyengar noted, the current semantics of `preferred_address` are not quite what you're looking for, though -- this is a set of other addresses the server could be reached over at need, not a server's request to attempt an immediate switch to one of them.

I could see changing the structure and semantics such that the server sends a list of "equivalent" addresses and perhaps indicates one of them as preferred.  I'm hesitant to try pulling this into v1 at this point, but I'll also acknowledge that `preferred_address` in particular hasn't seen a lot of deployment experience.  A change driven by issues with doing that in the real world might be a reasonable argument.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/2122#issuecomment-447101809