Subject: [IPsec] Can IPSec (RFC 5996) support tunnels with end point being (virtual) CPEs which has a set of workload attached (say Virtual Machines) all having virtual IP addresses?

From: Linda Dunbar <[email-masked]>
Date: Thu, 13 Apr 2017 16:49:16 +0000

IPsec experts:

As more enterprises are moving workload to 3rd party data centers (like Amazon AWS), the common way is to use IPSec to establish secure tunnels among branch offices and workload hosted by the cloud data centers.
Today, it is assumed that IPSec tunnel is between Data Center (say Amazon AWS) gateway and branch office CPE.

But many enterprises are worried that their workload traffic are exposed behind the data center gateway (e.g. in AWS data center). It would make enterprises more comfortable to have the secure tunnels go all the way to the workload (say servers, or VMs) within the 3rd party DC.

-        To support this, one end point of the tunnel will be (virtual) CPEs which has a set of workload attached (say Virtual Machines).  All having virtual IP addresses. Can IPSec (RFC 5996) support this combination? How?
-         Does the enterprise who lease virtual resource from 3rd party data center have to have a public routable IP address to establish IPsec tunnel between branch offices and the (virtual) gateway in the 3rd party DC?
-        Is there any limitation (i.e. the maximum number of IPSec sessions) that a server can support?
-        Is there any advantages of IPsec Comparing with using TLS? Does TLS provide Tunnel mode?
-        I2NSF has a proposal on using Controller to manager all the IPSec tunnels: https://datatracker.ietf.org/doc/html/draft-abad-i2nsf-sdn-ipsec-flow-protection. What kind of issues do you see with the proposed approach?

Thanks, Linda Dunbar

---

From: Michael Richardson <[email-masked]>
Date: Thu, 13 Apr 2017 15:48:13 -0400

Linda Dunbar <[email-masked]> wrote:
    > - To support this, one end point of the tunnel will be (virtual) CPEs
    > which has a set of workload attached (say Virtual Machines). All having
    > virtual IP addresses. Can IPSec (RFC 5996) support this combination?
    > How?

As long the branch office CPE has a public IP address, or one is willing to
establish a hub system with a public IP address, all the VMs can just be
considered to behind NAPT.   This is done in the field regularly.
Given that Amazon's IPsec tunnel end points cost extra, and sometimes have
significant compatibility problems, this is often the preferred solutions
regardless.

    > - Does the enterprise who lease virtual resource from 3rd party data
    > center have to have a public routable IP address to establish IPsec
    > tunnel between branch offices and the (virtual) gateway in the 3rd
    > party DC?

Not always, no.

    > that a server can support?

Depends upon ram and network speed.
In general, the limits are quite high.
We have configured systems with 3000 tunnels from multiply-NAT'ed IoT devices
in the field.  3000 was chosen as a soft limit per gateway machine due to
anticipated traffic load vs speed for virtual devices.

It's WAY easier with IKEv2 and putting IPv6 inside, but many customers are
afraid of this because they have little experience with IPv6.

    > - Is there any advantages of IPsec Comparing with using TLS? Does TLS

There is no IETF standard TLS-based VPN.  There are dozens of proprietary
stacks, including the very popular OpenVPN.

    > - I2NSF has a proposal on using Controller to manager all the IPSec
    > tunnels:
    > https://datatracker.ietf.org/doc/html/draft-abad-i2nsf-sdn-ipsec-flow-protection.
    > What kind of issues do you see with the proposed approach?

I didn't read it.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Yoav Nir <[email-masked]>
Date: Fri, 14 Apr 2017 00:42:11 +0300

Hi

What Michael said.  Additional comments within.

>> virtual IP addresses. Can IPSec (RFC 5996) support this combination?

To clarify, RFC 7296 (and previously 5996) are for IKEv2, the key exchange and authentication protocol. RFC 4301 defines IPsec.

> significant compatibility problems, this is often the preferred solutions

Yeah. AWS (and Azure as well) want you to have a virtual server room in their cloud, and you connect your IPsec to an AWS gateway in tunnel mode. They also prefer to forego IKE’s ability to negotiate tunnel selectors (the set of addresses and protocols being tunneled) and instead negotiate universal tunnels and choose what to tunnel using configuration. To add confusion, they call this “route-based VPN” even if you don’t use a routing protocol at all.

There are other options with cloud providers where you can install whatever IPsec software you want on your virtual server, but that costs extra.

> afraid of this because they have little experience with IPv6.

Yeah. Gateways routinely manage thousands or tens of thousands of tunnels on commodity hardware. Its usually the amount of IPsec traffic that becomes the bottleneck.

> stacks, including the very popular OpenVPN.

And all of those proprietary stacks are used primarily for the remote-access / road-warrior scenario. Hardly anyone uses them for site-to-site.

>> What kind of issues do you see with the proposed approach?

I did. They have two cases. In case #1 the controller provisions credentials for IKE and entries in PAD and SPD. In case #2 you forego IKE and have the controller provision the SAs (including keys).

I especially didn’t like case #2. Sharing a secret key among three entities is a bad idea. A shared authentication credential can also be misused, but that’s a hard attack to mount. A shared traffic key makes the controller a very attractive target.

More in general, SDN was born in the data center. In a data center an all-knowing controller makes sense. This is true for routing as it is for NSFs such as firewalls, IDPs and IDSs. VPN extends the reach of the private network to all corners of the Internet. Think of a store chain with a CPE in every one of thousands of branches. Or a bank. The problem there is that there is no central administrative function. Local branches may switch ISP and renumber their network without bothering to tell the IT people. So the model where the controller knows everything is tough to deploy in practice.

It is probably necessary to have two-way communications, where the CPE tells the controller about its topology (how it partitions the Internet to “in” vs “out”) so that the controller can set up the appropriate SPDs.

There have been several attempts at this. RFC 7018 describes requirements, but the WG ultimately failed to publish a solution document.  There are also more recent commercial solutions sold today under the marketing name of “SD-WAN”, which is sort of like SDN if you squint hard enough. All of these have some interaction between CPE and controllers (or hubs) which draft-abad does not.

Yoav

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 13 Apr 2017 22:06:35 +0000

Michael,

Thank you very much for the detailed reply.

You said:
      "As long the branch office CPE has a public IP address, or one is willing to establish a hub system with a public IP address, all the VMs can just be considered to behind NAPT. "

Is the "INTERNAL_IP4_ADDRESS" in RFC5996 intended for establishing IPSec tunnel between remote VMs behind NAPT (all VMs have the virtual IP address)?
Does the CPE (with Public IP address) still need to know the NAT public IP address (even though the IPsec tunnel goes through the NAT to the VMs behind the NAT)?

Possible to have one IPSec tunnel with multiple VMs end points? (i.e. 1<-> N tunnel: A tunnel with one CPE  on one end and many VMs on the other end)?

Thank you.

Linda

-----Original Message-----
From: Michael Richardson [mailto:[email-masked]]
Sent: Thursday, April 13, 2017 2:48 PM
To: [email-masked]
Cc: Linda Dunbar <[email-masked]>
Subject: Re: [IPsec] Can IPSec (RFC 5996) support tunnels with end point being (virtual) CPEs which has a set of workload attached (say Virtual Machines) all having virtual IP addresses?

    > virtual IP addresses. Can IPSec (RFC 5996) support this combination?

As long the branch office CPE has a public IP address, or one is willing to establish a hub system with a public IP address, all the VMs can just be
considered to behind NAPT.   This is done in the field regularly.
Given that Amazon's IPsec tunnel end points cost extra, and sometimes have significant compatibility problems, this is often the preferred solutions regardless.

Not always, no.

Depends upon ram and network speed.
In general, the limits are quite high.
We have configured systems with 3000 tunnels from multiply-NAT'ed IoT devices in the field.  3000 was chosen as a soft limit per gateway machine due to anticipated traffic load vs speed for virtual devices.

It's WAY easier with IKEv2 and putting IPv6 inside, but many customers are afraid of this because they have little experience with IPv6.

    > - Is there any advantages of IPsec Comparing with using TLS? Does TLS

There is no IETF standard TLS-based VPN.  There are dozens of proprietary stacks, including the very popular OpenVPN.

I didn't read it.

--
Michael Richardson <[email-masked]<mailto:[email-masked]>>, Sandelman Software Works  -= IPv6 IoT consulting =-

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 13 Apr 2017 22:26:23 +0000

Yoav,

Thank you very much for the reply. And the valuable comments to draft-abad-i2nsf-sdn-ipsec-flow-protection.

You said:

What you described is the common practice today, i.e.  Establishing a tunnel (IPSec Tunnel) between AWS gateway and Client CPE. This approach not only cost extra, but not really secure, because your traffic behind AWS gateway can exploited by 3rd party. Correct?

Is there an option in IPSec that allows me to establish a tunnel between my CPE and my virtual resources leased from AWS (i.e. transparently by-pass AWS gateway)?

You also said:

The software is controlled by the client, correct? If the client owns this software, it doesn't really cost extra, does it?

Linda
-----Original Message-----
From: Yoav Nir [mailto:[email-masked]]
Sent: Thursday, April 13, 2017 4:42 PM
To: Linda Dunbar <[email-masked]>
Cc: [email-masked]; Michael Richardson <[email-masked]>
Subject: Re: [IPsec] Can IPSec (RFC 5996) support tunnels with end point being (virtual) CPEs which has a set of workload attached (say Virtual Machines) all having virtual IP addresses?

Hi

On 13 Apr 2017, at 22:48, Michael Richardson <[email-masked]<mailto:[email-masked]>> wrote:
>> which has a set of workload attached (say Virtual Machines). All
>> having virtual IP addresses. Can IPSec (RFC 5996) support this combination?
>> How?

> As long the branch office CPE has a public IP address, or one is
> willing to establish a hub system with a public IP address, all the VMs can just be
> Given that Amazon's IPsec tunnel end points cost extra, and sometimes
> have significant compatibility problems, this is often the preferred
> solutions regardless.

> We have configured systems with 3000 tunnels from multiply-NAT'ed IoT
> devices in the field.  3000 was chosen as a soft limit per gateway
> machine due to anticipated traffic load vs speed for virtual devices.
>
> It's WAY easier with IKEv2 and putting IPv6 inside, but many customers
> are afraid of this because they have little experience with IPv6.

> There is no IETF standard TLS-based VPN.  There are dozens of
> proprietary stacks, including the very popular OpenVPN.

Yoav

---

From: Yoav Nir <[email-masked]>
Date: Fri, 14 Apr 2017 07:03:20 +0300

> On 14 Apr 2017, at 1:26, Linda Dunbar <[email-masked]> wrote:
> 
> Yoav,
> 
>  <>

Yes, the cloud provider in this case has access to the plain-text.

> Is there an option in IPSec that allows me to establish a tunnel between my CPE and my virtual resources leased from AWS (i.e. transparently by-pass AWS gateway)?

Sure. IPsec can be established between any two nodes on the Internet as long as there’s reachability between them. If there’s only one-way reachability (say, if one is behind a NAPT) then the tunnel has to be established one way, but after that traffic can flow in both directions. If there’s no reachability (if both are behind different NAPT) then they can’t communicate directly, but a hub-and-spoke scenario where they’re traffic passes through a third gateway that decrypts and re-encrypts is still possible. It’s all a matter of configuration.

> The software is controlled by the client, correct? If the client owns this software, it doesn't really cost extra, does it?

I’ve never actually bought cloud service, but as far as I know getting “just a web server” or “a web server and a database” is cheaper (and scales better) then getting full access to a Linux VM where you can install stuff. There might also be a firewall preventing some sorts of traffic to the VM, but I don’t know much about it.

Yoav

> From: Yoav Nir [mailto:[email-masked] <mailto:[email-masked]>]
> On 13 Apr 2017, at 22:48, Michael Richardson <[email-masked] <mailto:[email-masked]>> wrote:
> >
> >
> > Linda Dunbar <[email-masked] <mailto:[email-masked]>> wrote:
> >> https://datatracker.ietf.org/doc/html/draft-abad-i2nsf-sdn-ipsec-flow-protection <https://datatracker.ietf.org/doc/html/draft-abad-i2nsf-sdn-ipsec-flow-protection>.

---

From: Michael Richardson <[email-masked]>
Date: Fri, 14 Apr 2017 13:05:55 -0400

mcr> As long the branch office CPE has a public IP address, or one is willing
mcr> to establish a hub system with a public IP address, all the VMs can just
mcr> be considered to behind NAPT.

linda> Is the "INTERNAL_IP4_ADDRESS" in RFC5996 intended for establishing
linda> IPSec tunnel between remote VMs behind NAPT (all VMs have the virtual
linda> IP address)?

This is used when transport mode is used through a NAPT.
It doesn't apply to tunnel mode.

linda> Does the CPE (with Public IP address) still need to know the NAT
linda> public IP address (even though the IPsec tunnel goes through the NAT
linda> to the VMs behind the NAT)?

If I understood the question, the answer is No.

linda> Possible to have one IPSec tunnel with multiple VMs end points?
linda> (i.e. 1<-> N tunnel: A tunnel with one CPE on one end and many VMs on
linda> the other end)?

A single IPsec (4301) tunnel can service traffic between two subnets.
In IKEv2, we use negotiate ranges (which is a super concept) rather than
subnets.

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 18 Apr 2017 15:03:34 +0300

> It doesn't apply to tunnel mode.

Actually not completely accurate. It is used when the road warrior
connects to the enterprise network, and wants to have IP-address
assigned to it so it can have internal ip-address it can use to
communicate with devices in that network.

I.e., when you connect to your laptop to enterprise SGW, the SGW can
assign you INTERNAL_IP4_ADDRESS of 10.0.1.22, and then you use that
address as your source IP when you send packets to the enterprise
network, and the SGW will take care that return packets sent to that
address is deliver back to you.

In that case you still use tunnel mode IPsec, as the outer IP headers
of the Child SA is your outer public IP-address (or public address of
your NAT box, if you are behind NAT), and the SGWs address. 

> In IKEv2, we use negotiate ranges (which is a super concept) rather than

One IPsec tunnel goes between two hosts, so connection would be
between SGW and the one VM. Inside that tunnel there can be multiple
Child SAs each carrying traffic inside them as specified by the
traffic selector (which could be subnet to subnet or single IP to
single IP, whatever is needed).

You cannot have one IPsec tunnel to be going to two different VM
devices on the cloud provides side. You need to different IPsec
tunnels, one for each host. 
-- 
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Tue, 18 Apr 2017 09:22:23 -0400

Tero Kivinen <[email-masked]> wrote:
    linda> Possible to have one IPSec tunnel with multiple VMs end points?
    linda> (i.e. 1<-> N tunnel: A tunnel with one CPE on one end and many VMs
    linda> on the other end)?

    >> In IKEv2, we use negotiate ranges (which is a super concept) rather
    >> than subnets.

    > One IPsec tunnel goes between two hosts, so connection would be between
    > SGW and the one VM. Inside that tunnel there can be multiple Child SAs
    > each carrying traffic inside them as specified by the traffic selector
    > (which could be subnet to subnet or single IP to single IP, whatever is
    > needed).

    > tunnels, one for each host.

Good point, but that's not, I think, what she was asking for.
Perhaps Linda can clarify her question.

---

From: Gabriel Lopez <[email-masked]>
Date: Fri, 21 Apr 2017 13:29:50 +0200

Hi,

Thanks for reading the doc, comments in line ...

> El 13 abr 2017, a las 23:42, Yoav Nir <[email-masked]> escribió:
> 
>> 
> 
> 

Well, we are talking about key distribution, not key sharing. The controller is acting a trusted key distribution center. The same problem arises in KDCs or IdPs.
In any case, we propose two cases to cover different scenarios, if case#2 does not fit, for whatever reason, in this particular scenario, case#1 could be used.

> There have been several attempts at this. RFC 7018 describes requirements, but the WG ultimately failed to publish a solution document.  There are also more recent commercial solutions sold today u…

The doc does not preclude this communication between the CPE and the Controller in order to allow the latter to be aware of configuration details about the former. In fact, Netconf/yang (proposed as an example of southbound interface in the draft) or SNMP could be used for that. We should clarify this point in the next version (we are currently working on it).

Thanks again for you comments.

Regards, Gabi.

> 
> Yoav
> 
> _______________________________________________
> IPsec mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/ipsec <https://www.ietf.org/mailman/listinfo/ipsec>

-----------------------------------------------------------
Gabriel López Millán
Departamento de Ingeniería de la Información y las Comunicaciones
University of Murcia
Spain
Tel: +34 868888504
Fax: +34 868884151
email: [email-masked] <mailto:[email-masked]>

---

From: Rafa Marin Lopez <[email-masked]>
Date: Sun, 23 Apr 2017 11:34:26 +0200

Hi Yoav:

> I especially didn’t like case #2. Sharing a secret key among three entities is a bad idea.

[Rafa] In my opinion, distributing key material from a trusted entity should not be considered as a bad idea since it has been done in the past.

> A shared authentication credential can also be misused, but that’s a hard attack to mount. A shared traffic key makes the controller a very attractive target.

In SDN paradigm a SDN is always attractive to attack, even without considering key distribution. If the SDN controller is attacked it will make the network inoperable. Thus, it is not a special issue in the SDN-based key management. Moreover, as I mentioned in a previous e-mail some time ago:

"[Rafa] Two comments here. 1) I see the controller as a trusted entity, otherwise it should not be trusted to any operation (not only IPsec-related). Thus, providing keying material should not be a problem. Even in case the NSFs negotiate keys, IKE credentials may be provided by the controller. Moreover, with the right permissions, the controller could access to the SAD state and observe keys there. 2) Distributing key material will be performed over a secure channel between the controller and the NSF, so no secret keys in clear are transmitted over the network.”

Moreover the SDN controller may remove the keys immediately after they have been installed in the NSF.

> 
> More in general, SDN was born in the data center. In a data center an all-knowing controller makes sense.

[Rafa] Therefore, case 2 would make sense there. 

> This is true for routing as it is for NSFs such as firewalls, IDPs and IDSs. VPN extends the reach of the private network to all corners of the Internet. Think of a store chain with a CPE in every one of thousands of branches. Or a bank. The problem there is that there is no central administrative function. Local branches may switch ISP and renumber their network without bothering to tell the IT people. So the model where the controller knows everything is tough to deploy in practice.

[Rafa] I guess, it will depend on the scenario. I remember an e-mail to I2NSF (https://www.ietf.org/mail-archive/web/i2nsf/current/msg01111.html) mentioning that they were doing sort of case 2 and they would like to have a standard way of doing it. Precisely the effort behind our I-D is trying to provide an standard interface standard not only for case 1 (with IKE in the NSF) but also for case 2 (no IKE in the NSF). 
> It is probably necessary to have two-way communications, where the CPE tells the controller about its topology (how it partitions the Internet to “in” vs “out”) so that the controller can set up th…

[Rafa] I do not see a problem here. In fact, that is the assumption when we wrote the I-D. Perhaps, we should state it in a clear form. Moreover, between CPEs and SDN controller that manages them a security association is required and, moreover, it is typical communication between them to carry out this management.

> There have been several attempts at this. RFC 7018 describes requirements, but the WG ultimately failed to publish a solution document.  There are also more recent commercial solutions sold today u…

[Rafa] It is not reflected explicitly (but assumed) in the draft-abad because that happens all the time if you assume the SDN paradigm (the discovery), so not only in the specific case of IPsec-management. The SDN controller needs to know what CPEs are under its control. For example in openflow you configure the device with a master SDN controller'IP and credentials to connect to the SDN controller. In any case, we can explicitly mention this in the I-D, no major problem.

Best Regards.
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec

-----------------------------------------------------------
Rafael Marin Lopez, PhD
Dept. Information and Communications Engineering (DIIC)
Faculty of Computer Science-University of Murcia
30100 Murcia - Spain
Telf: +34868888501 Fax: +34868884151 e-mail: [email-masked]
----------------------------------------------------------

---

From: Paul Wouters <[email-masked]>
Date: Sun, 23 Apr 2017 13:42:03 -0400

Sent from my iPhone
> On Apr 23, 2017, at 05:34, Rafa Marin Lopez <[email-masked]> wrote:
> 

>> [Rafa] I guess, it will depend on the scenario. I remember an e-mail to I2NSF (https://www.ietf.org/mail-archive/web/i2nsf/current/msg01111.html) mentioning that they were doing sort of case 2 and…

Don't. You will need to builtin there same security features as IKE has. Focus instead of implementing "minimal ikev2"

https://tools.ietf.org/html/draft-kivinen-ipsecme-ikev2-minimal-01

For example, most modern ciphers require timely rekeying and must never reuse IV/counters with the same private key. Synching that across independent hosts is impossible.

Paul

---

From: Rafa Marin Lopez <[email-masked]>
Date: Fri, 28 Apr 2017 00:26:55 +0200

Hi Paul:

Thank you for your comments. Please, see mine inline.

> Don't. You will need to builtin there same security features as IKE has.

[Rafa] Yes, but in case 2 all those features (e.g. key management operations) are in the controller but not in the NSFs. The NSFs only implements IPsec stack and deploys, for example, a NETFCONF server for the communication with the controller. Thus, all the key management operations are performed in the controller in case 2 and the resulting IPsec SAs are installed in the NSFs using, for example, NETCONF. 

> Focus instead of implementing "minimal ikev2"
> 

[Rafa] Minimal IKEv2 implementation in the NSF would be case 1. 
Nevertheless, implementing minimal ikev2 does not solve, for example, NAT-T for case 1 so controller should do something anyway.

> 
> 
> For example, most modern ciphers require timely rekeying and must never reuse IV/counters with the same private key.

[Rafa] Why do you imply that IV/counters will be the same with same private key? That is not the assumption in case 2 where the controller always generates fresh IPsec SAs.

> Synching that across independent hosts is impossible.

[Rafa] I do not see how it is impossible when a controller that have access to both NSFs provides that information. If it is not impossible when using IKEv2, it should not be impossible for case 2. 

The point is that the result of running IKEv2 is the IPsec SAs in the IPsec kernel of both NSFs. In case 2, that state is coming from the controller that is performing the key management operations (generate fresh key material, IV, etc… per each IPsec SA). The controller has a  connection with both NSFs using, for example, NETCONF. Moreover, as I commented , you can read this e-mail (https://www.ietf.org/mail-archive/web/i2nsf/current/msg01111.html). So there is proof that it is working.

Thus, it is more difficult to manage, yes, (in fact, we admit that in our I-D), but not impossible. 
In fact, I think it would be worthy if you can provide a concrete example of that use case where both NSFs are under the same controller. Maybe we can help you to address your concern and explain how the operation would be in that specific case you have in mind.

Best Regards and many thanks for your comments.