Subject: [dc] IRTF datacenter research group discussion list

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 16 Dec 2011 08:23:43 +0000

Hi,

I wanted to make you all aware of the IRTF's dcrg-interest mailing list, which was set up following a face-to-face meeting at SIGCOMM this year to discuss a possible IRTF research group on datacenter networking: http://irtf.org/mailman/listinfo/dcrg-interest

There has not been much activity towards the formation of an IRTF RG since SIGCOMM, but I am hopeful that the high energy level demonstrated in the various Taipei meetings on the topic will inject some energy here - several of the topics that may be too early for the IETF to standardize around would fit an IRTF RG very well. I'm certainly supportive of this.

Lars
IRTF Chair

-- 
Mobile number during December:    +358 46 5215582
Mobile number starting January:  +49 151 12055791

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 22 Dec 2011 08:12:06 +0000

Hi all,

There has been a lot of L2 over L3 solutions or proposals for data center network and data center interconnect till now. However, it seems recently there are increasing interests on L3 over L3 (e.g., IP over IP) solutions for data center network and data center interconnect, please see the following text quoted from IETF82 L2VPN minutes ( https://www.ietf.org/proceedings/82/minutes/l2vpn.txt). It's a general belief that L3 is more scalable than L2. Especially when considering the data center interconnection case, the L3 over L3 solutions can bring DC operators more benefits compared to the L2 over L3 solutions, such as path optimization, active-active data centers, MAC table reduction on DC switches and broadcast flood suppression etc . 

Although Layer 2 connectivity is still required for some high-availability clusters which use non-IP and link-local multicast for communication, more and more cluster vendors are either already able to support cluster services at Layer3 or will support it in the near future. In addition, the GSLB mechanism (e.g., DNS based GSLB) works very well in most cases, is geo-cluster service still a common requirement for data center interconnect? If not, could we spend any time on exploring the possibility of using L3 over L3 solution for the most data center interconnection scenarios where geo-clustering, especially non-IP based geo-clustering is not needed?

Best regards,
Xiaohu

**************************************************************************************************************************
Kireeti: I keep seeing ISID, PBB, VLANs.  We have to stop conceding to layer 2 and start moving to layer 3 (applause) and lots of problems will go away.

Florin: We should put VPN on the same page to modify the priorities. You need to expand so can look at L3 over L3 as well as L2 over L3. For item number 2 we need to look at L3 transport as well as Ethernet and MPLS.  And for number 3 we need to work on this as there's a lot of expertise in L2, L3 and other WGs.

Marc: The problem statement needs to include more of the L3 issues around sending L2 over L3, as the L2 traffic already contains L3.  Issue is just framing.

Eric Nordmark; "Yes, Yes and Yes".  We need to focus from the DC prospective and not from the VPN view.  don't just think about Ethernet over IP, but also IP over IP.  Hypervisor may get decoupled over time.
**************************************************************************************************************************
> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Eggert, Lars
> 发送时间: 2011年12月16日 16:24
> 收件人: [email-masked]
> 主题: [dc] IRTF datacenter research group discussion list
> 
> Hi,
> 
> I wanted to make you all aware of the IRTF's dcrg-interest mailing list, which
> was set up following a face-to-face meeting at SIGCOMM this year to discuss a
> possible IRTF research group on datacenter networking:
> http://irtf.org/mailman/listinfo/dcrg-interest
> 
> There has not been much activity towards the formation of an IRTF RG since
> SIGCOMM, but I am hopeful that the high energy level demonstrated in the
> various Taipei meetings on the topic will inject some energy here - several of
> the topics that may be too early for the IETF to standardize around would fit an
> IRTF RG very well. I'm certainly supportive of this.
> 
> Lars
> IRTF Chair
> 
> 
> --

---

From: Ping Pan <[email-masked]>
Date: Thu, 22 Dec 2011 05:49:54 -0800

2011/12/22 Xuxiaohu <[email-masked]>

> Hi all,
>
> There has been a lot of L2 over L3 solutions or proposals for data center
> network and data center interconnect till now. However, it seems recently
> there are increasing interests on L3 over L3 (e.g., IP over IP) solutions
> for data center network and data center interconnect, please see the
> following text quoted from IETF82 L2VPN minutes (
> https://www.ietf.org/proceedings/82/minutes/l2vpn.txt). It's a general
> belief that L3 is more scalable than L2. Especially when considering the
> data center interconnection case, the L3 over L3 solutions can bring DC
> operators more benefits compared to the L2 over L3 solutions, such as path
> optimization, active-active data centers, MAC table reduction on DC
> switches and broadcast flood suppression etc .
>
> Although Layer 2 connectivity is still required for some high-availability
> clusters which use non-IP and link-local multicast for communication, more
> and more cluster vendors are either already able to support cluster
> services at Layer3 or will support it in the near future. In addition, the
> GSLB mechanism (e.g., DNS based GSLB) works very well in most cases, is
> geo-cluster service still a common requirement for data center
> interconnect? If not, could we spend any time on exploring the possibility
> of using L3 over L3 solution for the most data center interconnection
> scenarios where geo-clustering, especially non-IP based geo-clustering is
> not needed?
>
>
This makes perfect sense!!

Ping

> Kireeti: I keep seeing ISID, PBB, VLANs.  We have to stop conceding to
> layer 2 and start moving to layer 3 (applause) and lots of problems will go
> away.
>
> Florin: We should put VPN on the same page to modify the priorities. You
> need to expand so can look at L3 over L3 as well as L2 over L3. For item
> number 2 we need to look at L3 transport as well as Ethernet and MPLS.  And
> for number 3 we need to work on this as there's a lot of expertise in L2,
> L3 and other WGs.
>
>
> Marc: The problem statement needs to include more of the L3 issues around
> sending L2 over L3, as the L2 traffic already contains L3.  Issue is just
> framing.
>
> Eric Nordmark; "Yes, Yes and Yes".  We need to focus from the DC
> prospective and not from the VPN view.  don't just think about Ethernet
> over IP, but also IP over IP.  Hypervisor may get decoupled over time.
>
> _______________________________________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dc
>

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 23 Dec 2011 06:38:40 +0000

> -----邮件原件-----
> 发件人: Xuxiaohu
> 发送时间: 2011年12月22日 16:28
> 收件人: 'Eggert, Lars'; [email-masked]; [email-masked].'
> 主题: IP over IP solution for data center interconnect
> 
> Hi all,
> 
> there are increasing interests on L3 over L3 (e.g., IP over IP) solutions for data
> center network and data center interconnect, please see the following text
> quoted from IETF82 L2VPN minutes
> ( https://www.ietf.org/proceedings/82/minutes/l2vpn.txt). It's a general belief
> that L3 is more scalable than L2. Especially when considering the data center
> interconnection case, the L3 over L3 solutions can bring DC operators more
> benefits compared to the L2 over L3 solutions, such as path optimization,
> active-active data centers, MAC table reduction on DC switches and broadcast
> flood suppression etc .

By the way, the ARP table scaling issue on DC gateways, which is deemed by the ARMD WG as the only worthy ARP problem in data center networks, could also be solved with the IP over IP solution.

Best regards,
Xiaohu
 
> 
> Although Layer 2 connectivity is still required for some high-availability clusters
> which use non-IP and link-local multicast for communication, more and more
> cluster vendors are either already able to support cluster services at Layer3 or
> will support it in the near future. In addition, the GSLB mechanism (e.g., DNS
> based GSLB) works very well in most cases, is geo-cluster service still a
> common requirement for data center interconnect? If not, could we spend any
> time on exploring the possibility of using L3 over L3 solution for the most data
> center interconnection scenarios where geo-clustering, especially non-IP based
> geo-clustering is not needed?
> 
> Best regards,
> Xiaohu
> 
> ****************************************************************
> **********************************************************
> Kireeti: I keep seeing ISID, PBB, VLANs.  We have to stop conceding to layer 2
> and start moving to layer 3 (applause) and lots of problems will go away.
> 
> Florin: We should put VPN on the same page to modify the priorities. You need
> to expand so can look at L3 over L3 as well as L2 over L3. For item number 2 we
> need to look at L3 transport as well as Ethernet and MPLS.  And for number 3
> we need to work on this as there's a lot of expertise in L2, L3 and other WGs.
> 
> 
> sending L2 over L3, as the L2 traffic already contains L3.  Issue is just framing.
> 
> Eric Nordmark; "Yes, Yes and Yes".  We need to focus from the DC prospective
> and not from the VPN view.  don't just think about Ethernet over IP, but also IP
> over IP.  Hypervisor may get decoupled over time.
> ****************************************************************
> **********************************************************
> > -----邮件原件-----

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 23 Dec 2011 11:44:48 -0600

> By the way, the ARP table scaling issue on DC gateways, which is deemed by the ARMD WG as the only worthy ARP problem in data center networks, could also be solved with the IP over IP solution.

Two comments:

First, if anybody is interested in exploring additional address resolution issues in the datacenter, please speak up now on the ARMD mailing list. We are attempting to finish our Problem Statement soon. The Problem Statement will only include those issues that are described by WG contributors. So if you think we should deem other issues to be worthy problems, please contribute to the discussion.

Second, at a high-level I think that a number of ARMD participants will agree with your comment about ARP scale issues being solved by "IP over IP" solutions. In fact, it's not just IP-over-IP that can help. Any map-and-encap scheme, including L2-over-IP, can help by shifting the burden to a mapping mechanism. Local ARP/ND proxy functions (e.g. in the VM v-switch, TOR, etc) could take advantage of the mapping mechanism to facilitate address resolution.

In my personal opinion: The real problem is figuring out how to do the mapping, what the right mapping approach is for a datacenter, etc.  This is the topic of the NVO3 effort that was discussed during L2VPN in Taipei.

Cheers,
-Benson

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 24 Dec 2011 01:27:09 +0000

> -----邮件原件-----
> 发件人: Benson Schliesser [mailto:[email-masked]]
> 发送时间: 2011年12月24日 1:45
> 收件人: Xuxiaohu
> 抄送: Eggert, Lars; [email-masked]; [email-masked]; [email-masked]
> 主题: Re: [dcrg-interest] IP over IP solution for data center interconnect
> 
> 
> > By the way, the ARP table scaling issue on DC gateways, which is deemed by
> the ARMD WG as the only worthy ARP problem in data center networks, could
> also be solved with the IP over IP solution.
> 
> Two comments:
> 
> First, if anybody is interested in exploring additional address resolution issues in
> the datacenter, please speak up now on the ARMD mailing list. We are
> attempting to finish our Problem Statement soon. The Problem Statement will
> only include those issues that are described by WG contributors. So if you think
> we should deem other issues to be worthy problems, please contribute to the
> discussion.
> 
> Second, at a high-level I think that a number of ARMD participants will agree
> with your comment about ARP scale issues being solved by "IP over IP"
> solutions. In fact, it's not just IP-over-IP that can help. Any map-and-encap
> scheme, including L2-over-IP, can help by shifting the burden to a mapping
> mechanism. Local ARP/ND proxy functions (e.g. in the VM v-switch, TOR, etc)
> could take advantage of the mapping mechanism to facilitate address
> resolution.

As for L2 over IP solutions, I agree that the ARP broadcast flood impact on the servers/routers and networks could be alleviated by using the ARP cache mechanism (tools.ietf.org/pdf/draft-shah-armd-arp-reduction-02.txt) or some ARP mapping system.

However, since the DC gateways usually need to forward traffic to so many hosts at most time, they need a very large ARP cache table to contain so many ARP entries. Otherwise, the ARP cache aging-out timer should be set short enough. In the way, the DC gateways would have to send ARP requests more frequently, in addition, the gateways are required to have a much large packet cache to cache the received packets destined for the hosts before the ARP responses for them are returned, otherwise the received packets would have to be discarded when the cache is overflowed. In a word, I don't know how the mapping system could help on this ARP table scaling issue.

Best regards,
Xiaohu

> In my personal opinion: The real problem is figuring out how to do the mapping,
> what the right mapping approach is for a datacenter, etc.  This is the topic of
> the NVO3 effort that was discussed during L2VPN in Taipei.
> 
> Cheers,
> -Benson
>

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 27 Dec 2011 11:43:15 +0530

For all the proponents of L3, here is a thought.

The biggest concern in cloud is data protection. That is just make or break. Data can't move when the VM moves. It's just too expensive and time consuming. Further, to reduce disk wastage you have to put data on network storage. That's just easy for data backup, and cheap in terms of total disk space you need. There is a lot going on here in terms of data de-duplication.

Now given network storage, you have two options - NAS and SAN.

On the cloud each VM administrator has root privileges. And IP and MAC addresses can be duplicated (if you are root just configure any IP and MAC on a virtual interface of choice). So, you have to protect the network storage from multiple tenants who can have the same IP address and MAC address. NAS assumes your id on the UNIX is globally unique and your permissions are unique - not true anymore. So, you have IP, MAC and ID duplication.

That means that if you have to protect data, you have to know somehow which host is allowed to see which disks. Doing this end-to-end using TCP/IP is problematic because the NAS controller doesn't know about tenant id, and whether the MAC and IP are being duplicated across customers. To make the storage aware of segmentation, you have to carry some segment (GRE, MPLS, VLAN, etc.) into the storage box. That's not true today.

Network level isolation happens in SAN, but not in NAS. NAS just runs end-to-end over TCP. In SAN you can say which host will see which disk. The FC security is high because MAC and FC-ID is assigned by the network and not in the control of the host. The network knows which port is which host/MAC. So, if you only know which port is which customer, you know which MAC can see which disk. Binding a port to a customer is not hard. But mapping all of that to data security and disk visibility is not trivial. E.g. you have to do GRE tunnel mapping to a set of disk permissions. Who needs that?

SAN does not use IP. It uses Fiber Channel. With convergence, we are moving to FC over Ethernet (FCoE). FCIP and iSCSI have not worked very well. Security for data comes from FC and lowering of cost comes from use of FCoE. That needs native Ethernet (no IP). 

When you have site-to-site VM mobility, data is not moving across sites. E.g. an application server will connect to a set of load-balanced set of database servers within a VLAN. The application and database servers can be different sites, and they connect over a VLAN. Alternately, the application and database servers can be in the same location and the DB server accesses the SAN over the Internet. Even if data is replicated across many sites, load-balancing across application or db servers uses VLANs. In HPC as well, there is a lot of broadcast and discovery, and this may use IP or not, the fact is that it depends on broadcast. The HPC case externalizes the memory on a host to other hosts through RDMA. Everything to do with Big Data will rely on this, and besides disk externalization we will also see increase in memory externalization. I just went to a IEEE HPC cloud conference :-)

L3 assumes that all data (disk and memory) is inside the host. That assumption changed a decade ago. Cloud just increases the distribution. Just as compute is optimized by muxing the VMs on a physical machine, storage is optimized by consolidating data separately on storage devices and memory by hosting it in different hosts. Network has to deal with all types of traffic and security/isolation (imagine someone changing memory over the network). Storage and memory traffic uses L2. To discover each other, hosts use L2. These L2 domains need to span across sites and within the site.

This are not corner cases. This is just central to what cloud means in the long run. 

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Friday, December 23, 2011 12:09 PM
To: Eggert, Lars; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [armd] IP over IP solution for data center interconnect

Best regards,
Xiaohu

_______________________________________________
armd mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/armd

---

From: Pedro Marques <[email-masked]>
Date: Mon, 26 Dec 2011 23:01:48 -0800

> For all the proponents of L3, here is a thought.

summary of the original email: multi-tenant storage solutions require
FiberChannel; an L3 service model doesn't support FiberChannel.

There are multiple storage solutions that support multi-tenancy that
work on top of IP making the argument moot (1). At large scale running
a separate network for data and for storage imposes very significant
costs. I'd venture that the proponents of L3 have very little interest
in FiberChannel.

The same goes for the argument that load-balancing requires L2. There
is existence proof of the contrary.

There is a wide range of technology choices available in this space.
I'm sure there is interest in L2 based solutions but the claim that L3
based solutions are not feasible has already been disproved by
deployed solutions.

1 - Example: When a storage solution presents a block device to the VM
(e.g. Amazon EBS) it is the host OS/hypervisor that maps the guest OS
operations into network operations; doing so in the infrastructure
network real rather than in the tenant's network realm.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 27 Dec 2011 13:19:34 +0530

There is InfiniBand besides FiberChannel. And we are talking about converging not separating them. The question is converge over L2 or L3. I think you did not read me correctly. IP based solutions have been defined before Ethernet solutions but not used widely. We can discuss why.

I would be interested to know how the storage multi-tenancy solutions deal with private IP, MAC and ID in the public domain, which can be duplicated. 

The argument was not around load-balancing, it is around discovery.

Your example of the hypervisor is interesting. I'm aware that Amazon does things in the hypervisor. They provide IP mobility through NAT. Security through a hypervisor firewall. The consequence is that you pay extra if you wanted a mobile VM, and you manage your own firewall (at least the last time I checked). If tenant isolation is all the way into the host, then network QoS, bandwidth etc is also moot. 

Thanks, Ashish

---

From: <[email-masked]>
Date: Tue, 27 Dec 2011 16:26:26 -0500

Ashish,

> I would be interested to know how the storage multi-tenancy solutions deal with private IP, MAC and ID
> in the public domain, which can be duplicated.

A common current technique is to use per-tenant VLANs (a tenant may have multiple VLANs) and use NAT (as needed) between the VLANs and the public Internet in order to separate VLAN addressing from public Internet addressing, effectively resulting in a (virtual) NAT instance per VLAN.  A 12-bit VLAN tag is not large enough to scale up to the desired large-scale data center scope, so both VxLAN and NVGRE use a 24-bit virtual network identifier to replace the 12-bit VLAN tag; VxLAN calls this a VNI (VxLAN [or Virtual] Network Identifier), and NVGRE calls it a TNI (Tenant Network Identifier).  VNI and TNI encap/decap can be handled by the hypervisor and are limited to the datacenter; in this structure, VMs can't use a VNI or TNI directly, and the (virtual) NAT handles encap/decap for inbound/outbound traffic so that trying to send encapsulated (VxLAN or NGRE) traffic in from the outside results in double encapsulation with no effect on tenant isolation.

Before one of the L2VPN folks comes after me for that "limited to the datacenter" comment ;-) ... an L2VPN gateway would probably translate between a 24-bit VNI/TNI and a 24-bit VPN ID - end-to-end use of a single identifier for the L2VPN and the virtual networks on both ends is possible, but (IMHO) difficult when there are multiple administrative domains involved in managing the identifiers.

Use of VLANs to scope filesystem visibility on NAS fileservers is a common deployed practice in datacenters (ditto for VLANs to limit iSCSI storage visibility), thereby dealing with the following:

> So, you have to
> protect the network storage from multiple tenants who can have the same IP address and MAC address.
> NAS assumes your id on the UNIX is globally unique and your permissions are unique - not true anymore.
> So, you have IP, MAC and ID duplication.

Duplication of IP, MAC and ID on separate VLANs works fine - this model is carried over to virtual networks based on VNI or TNI.  The sentence containing "globally unique" (incorrectly) assumes a single fileserver; instead, consider a single NAS system presenting what is effectively a virtual fileserver per VLAN (or VNI/TNI) and similarly for iSCSI - that is what sorts out this duplication, including use of different identity services/serves per VLAN (or VNI/TNI).

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ashish Dalela (adalela)
> Sent: Tuesday, December 27, 2011 2:50 AM
> To: Pedro Marques
> Cc: [email-masked]
> There is InfiniBand besides FiberChannel. And we are talking about converging not separating them. The
> question is converge over L2 or L3. I think you did not read me correctly. IP based solutions have
> been defined before Ethernet solutions but not used widely. We can discuss why.
> 
> I would be interested to know how the storage multi-tenancy solutions deal with private IP, MAC and ID
> in the public domain, which can be duplicated.
> 
> Your example of the hypervisor is interesting. I'm aware that Amazon does things in the hypervisor.
> They provide IP mobility through NAT. Security through a hypervisor firewall. The consequence is that
> you pay extra if you wanted a mobile VM, and you manage your own firewall (at least the last time I
> checked). If tenant isolation is all the way into the host, then network QoS, bandwidth etc is also
> moot.
> 
> Thanks, Ashish
> 
> 
> > The biggest concern in cloud is data protection. That is just make or break. Data can't move when
> the VM moves. It's just too expensive and time consuming. Further, to reduce disk wastage you have to
> put data on network storage. That's just easy for data backup, and cheap in terms of total disk space
> you need. There is a lot going on here in terms of data de-duplication.
> >
> > On the cloud each VM administrator has root privileges. And IP and MAC addresses can be duplicated
> (if you are root just configure any IP and MAC on a virtual interface of choice). So, you have to
> protect the network storage from multiple tenants who can have the same IP address and MAC address.
> NAS assumes your id on the UNIX is globally unique and your permissions are unique - not true anymore.
> So, you have IP, MAC and ID duplication.
> >
> > That means that if you have to protect data, you have to know somehow which host is allowed to see
> which disks. Doing this end-to-end using TCP/IP is problematic because the NAS controller doesn't know
> about tenant id, and whether the MAC and IP are being duplicated across customers. To make the storage
> aware of segmentation, you have to carry some segment (GRE, MPLS, VLAN, etc.) into the storage box.
> That's not true today.
> >
> > Network level isolation happens in SAN, but not in NAS. NAS just runs end-to-end over TCP. In SAN
> you can say which host will see which disk. The FC security is high because MAC and FC-ID is assigned
> by the network and not in the control of the host. The network knows which port is which host/MAC. So,
> if you only know which port is which customer, you know which MAC can see which disk. Binding a port
> to a customer is not hard. But mapping all of that to data security and disk visibility is not
> trivial. E.g. you have to do GRE tunnel mapping to a set of disk permissions. Who needs that?
> >
> > SAN does not use IP. It uses Fiber Channel. With convergence, we are moving to FC over Ethernet
> (FCoE). FCIP and iSCSI have not worked very well. Security for data comes from FC and lowering of cost
> comes from use of FCoE. That needs native Ethernet (no IP).
> >
> > When you have site-to-site VM mobility, data is not moving across sites. E.g. an application server
> will connect to a set of load-balanced set of database servers within a VLAN. The application and
> database servers can be different sites, and they connect over a VLAN. Alternately, the application
> and database servers can be in the same location and the DB server accesses the SAN over the Internet.
> Even if data is replicated across many sites, load-balancing across application or db servers uses
> VLANs. In HPC as well, there is a lot of broadcast and discovery, and this may use IP or not, the fact
> is that it depends on broadcast. The HPC case externalizes the memory on a host to other hosts through
> RDMA. Everything to do with Big Data will rely on this, and besides disk externalization we will also
> see increase in memory externalization. I just went to a IEEE HPC cloud conference :-)
> >
> > L3 assumes that all data (disk and memory) is inside the host. That assumption changed a decade ago.
> Cloud just increases the distribution. Just as compute is optimized by muxing the VMs on a physical
> machine, storage is optimized by consolidating data separately on storage devices and memory by
> hosting it in different hosts. Network has to deal with all types of traffic and security/isolation
> (imagine someone changing memory over the network). Storage and memory traffic uses L2. To discover
> each other, hosts use L2. These L2 domains need to span across sites and within the site.
> >
> > By the way, the ARP table scaling issue on DC gateways, which is deemed by the ARMD WG as the only
> worthy ARP problem in data center networks, could also be solved with the IP over IP solution.
> >
> > Best regards,
> > Xiaohu
> >
> >>

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Wed, 28 Dec 2011 09:37:34 +0530

David,

Thanks for the explanation. So, it is possible to pull network abstractions (VLAN etc) into the storage controller. The scheme is also extensible to other things like VXLAN or NVGRE. Then you tie the file system permissions to the VLAN. Am I reading this correctly?

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, December 28, 2011 2:56 AM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: RE: [dc] [armd] IP over IP solution for data center interconnect

Ashish,

Thanks,
--David

---

From: <[email-masked]>
Date: Tue, 27 Dec 2011 23:28:54 -0500

Ashish,

> Thanks for the explanation. So, it is possible to pull network abstractions (VLAN etc) into the
> storage controller.

Sure - not only is it possible, there's also plenty of deployed storage equipment that does this,
today.  Also, please avoid use of the phrase "storage controller" in this discussion, as that's
often a component of a much larger storage system, storage array, or fileserver.

> The scheme is also extensible to other things like VXLAN or NVGRE.

Yes.

> Then you tie the file system permissions to the VLAN. Am I reading this correctly?

Not exactly ;-).  While one could do that, the result can be "interesting" if there is ID
duplication among tenants.  It's better to use separate filesystems per tenant (with each
filesystem tied to the identity service for the corresponding tenant), and tie the list or
set of exported filesystems to the VLAN (or virtual network based on VNI/TNI).  The result
is that if Tenant A is on a VLAN (or virtual network) that isn't supposed to have access to
Tenant B filesystems, it doesn't matter what IDs or addresses are duplicated or forged on
Tenant A's VLAN (or virtual network) - Tenant B's filesystems are inaccessible (e.g., can't
be mounted).  For iSCSI, LUN mapping/masking is the analogous functionality, and that
functionality is widely implemented in storage systems.

Thanks,
--David

> Sent: Tuesday, December 27, 2011 11:08 PM
> To: Black, David
> Cc: [email-masked]
> Thanks for the explanation. So, it is possible to pull network abstractions (VLAN etc) into the
> storage controller. The scheme is also extensible to other things like VXLAN or NVGRE. Then you tie
> the file system permissions to the VLAN. Am I reading this correctly?
>
> Thanks, Ashish
>
> A common current technique is to use per-tenant VLANs (a tenant may have multiple VLANs) and use NAT
> (as needed) between the VLANs and the public Internet in order to separate VLAN addressing from public
> Internet addressing, effectively resulting in a (virtual) NAT instance per VLAN.  A 12-bit VLAN tag is
> not large enough to scale up to the desired large-scale data center scope, so both VxLAN and NVGRE use
> a 24-bit virtual network identifier to replace the 12-bit VLAN tag; VxLAN calls this a VNI (VxLAN [or
> Virtual] Network Identifier), and NVGRE calls it a TNI (Tenant Network Identifier).  VNI and TNI
> encap/decap can be handled by the hypervisor and are limited to the datacenter; in this structure, VMs
> can't use a VNI or TNI directly, and the (virtual) NAT handles encap/decap for inbound/outbound
> traffic so that trying to send encapsulated (VxLAN or NGRE) traffic in from the outside results in
> double encapsulation with no effect on tenant isolation.
>
> Before one of the L2VPN folks comes after me for that "limited to the datacenter" comment ;-) ... an
> L2VPN gateway would probably translate between a 24-bit VNI/TNI and a 24-bit VPN ID - end-to-end use
> of a single identifier for the L2VPN and the virtual networks on both ends is possible, but (IMHO)
> difficult when there are multiple administrative domains involved in managing the identifiers.
>
> Use of VLANs to scope filesystem visibility on NAS fileservers is a common deployed practice in
> datacenters (ditto for VLANs to limit iSCSI storage visibility), thereby dealing with the following:
>
> > So, you have to
> Duplication of IP, MAC and ID on separate VLANs works fine - this model is carried over to virtual
> networks based on VNI or TNI.  The sentence containing "globally unique" (incorrectly) assumes a
> single fileserver; instead, consider a single NAS system presenting what is effectively a virtual
> fileserver per VLAN (or VNI/TNI) and similarly for iSCSI - that is what sorts out this duplication,
> including use of different identity services/servers per VLAN (or VNI/TNI).
>
> Thanks,
> --David

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Wed, 28 Dec 2011 10:08:24 +0530

David,

Thanks again. So what is your view of the L2/L3 debate? Is the direction to use more of iSCSI and NAS or to converge FC over Ethernet? I understand that there may be very few FCoE deployments today, but I'm asking in terms of your perception of the direction. What trends we perceive vis-à-vis cloud?

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, December 28, 2011 9:59 AM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: RE: [dc] [armd] IP over IP solution for data center interconnect

Ashish,

Yes.

Thanks,
--David

---

From: <[email-masked]>
Date: Wed, 28 Dec 2011 22:43:03 -0500

I've changed the title of this thread in order to summarize the interaction of
storage networking protocols with the sort of large scale L2 and L3 data center
networking that's under discussion here.  The quick summary of the thread below
is that storage systems (both SAN and NAS) are capable of multi-tenant isolation
based on VLANs, and if one views encapsulation-based virtual networking as an
extension to VLANs, then those multi-tenant isolation properties can be extended
to such encapsulation-based approaches, both L2-in-L3 [MAC-in-IP], e.g., VxLAN,
NVGRE, and L2-in-L2 [MAC-in-MAC], e.g., TRILL with 24-bit identifiers.

The major data center storage networking protocols that run over IP and/or
Ethernet networks are:

- NAS (file): NFS and CIFS.  These are (or should be) run over TCP/IP, and hence
	are agnostic as to whether the network service is L2 vs. L3.
- SAN (block): iSCSI and FCoE
	o iSCSI uses TCP, and hence is also agnostic as to whether the networking
		service is L2 vs. L3.
	o FCoE does not use TCP or IP.  Further FCoE effectively requires DCB as part
		of the Ethernet (L2) service in order to avoid frame drops.  Fibre Channel
		(the FC part of FCoE) reacts very poorly to drops, to put it mildly.

For FCoE, the effective requirement for DCB Ethernet is much more important (IMHO)
than the fact that FCoE requires an L2 network service (the E in FCoE is Ethernet).
From what I've seen, DCB support tends not to be part of the initial designs for
either L2-in-L3 or L2-in-L2 encapsulations.  IMHO, DCB Ethernet functionality should
be a possible enhancement to network virtualization for data centers, but should not
be a baseline requirement.  When DCB Ethernet is not present, iSCSI can be a
reasonable alternative to FCoE for open systems storage access.

DCB Ethernet is the combination of PFC (Priority Flow Control), ETS (Enhanced
Transmission Selection) and DCBX (Data Center Bridging eXchange).  If these are new
concepts, this link may be useful for background info:
	http://en.wikipedia.org/wiki/Data_center_bridging

OTOH, if storage networking itself is a new concept, this link from the IETF Prague
proceedings - Transport Area meeting (March 2011) may be useful for background info:
	http://www.ietf.org/proceedings/80/slides/tsvarea-2.pdf

Thanks,
--David (NB: I'm a co-chair of the IETF storm WG that is currently finishing up a
	revision to the iSCSI specifications).

> From: Ashish Dalela (adalela) [mailto:[email-masked]]
> Sent: Tuesday, December 27, 2011 11:38 PM
> To: Black, David
> Cc: [email-masked]
> Thanks again. So what is your view of the L2/L3 debate? Is the direction to use more of iSCSI and NAS
> or to converge FC over Ethernet? I understand that there may be very few FCoE deployments today, but
> I'm asking in terms of your perception of the direction. What trends we perceive vis-à-vis cloud?
> 
> Thanks, Ashish
> 
>  
> > > I would be interested to know how the storage multi-tenancy solutions deal with private IP, MAC
> > > and ID  in the public domain, which can be duplicated.
> >
> > (as needed) between the VLANs and the public Internet in order to separate VLAN addressing from
> > public Internet addressing, effectively resulting in a (virtual) NAT instance per VLAN.  A 12-bit
> > VLAN tag is not large enough to scale up to the desired large-scale data center scope, so both
> > VxLAN and NVGRE use a 24-bit virtual network identifier to replace the 12-bit VLAN tag; VxLAN
> > calls this a VNI (VxLAN [or Virtual] Network Identifier), and NVGRE calls it a TNI (Tenant Network
> > Identifier).  VNI and TNI encap/decap can be handled by the hypervisor and are limited to the
> > datacenter; in this structure, VMs can't use a VNI or TNI directly, and the (virtual) NAT handles
> > encap/decap for inbound/outbound traffic so that trying to send encapsulated (VxLAN or NGRE) traffic
> > in from the outside results in double encapsulation with no effect on tenant isolation.
> >
> > > protect the network storage from multiple tenants who can have the same IP address and MAC
> > > address. NAS assumes your id on the UNIX is globally unique and your permissions are unique -
> > > not true anymore. So, you have IP, MAC and ID duplication.
> >

[... rest of thread snipped ...]

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 29 Dec 2011 04:08:34 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表
> [email-masked]
> 发送时间: 2011年12月29日 11:43
> 收件人: [email-masked]
> 抄送: [email-masked]; [email-masked]
> 主题: [dc] Storage networking, L2 and L3 (was: IP over IP solution for data
> center interconnect)
> 

Fully agree that the support for FCoE should not be a baseline requirement. In addition, IMHO, the most popular and practical application of FCoE is one-hop FCoE, rather than multi-hop FCoE. One-hop FCoE could be supported in the IP over IP solution, e.g., L3VPN.  By the way, FCoE would not be used in the DCI scenario SINCE lossless can't be ensured here.

Best regards,
Xiaohu

---

From: Russ White <[email-masked]>
Date: Thu, 29 Dec 2011 11:25:30 -0500

> That means that if you have to protect data, you have to know somehow which host is allowed to see which disks. Doing this end-to-end using TCP/IP is problematic because the NAS controller doesn't…

Maybe I'm dense, but wouldn't it be just as easy to add the capability
to differentiate based on IP address as it would be to design the entire
network and all protocols around what the current software capabilities
on the hypervisor are?

:-)

Russ