Subject: Re: [dc] [armd] IP over IP solution for data center interconnect

From: <[email-masked]>
Date: Sun, 1 Jan 2012 14:10:08 -0500

> > That means that if you have to protect data, you have to know somehow which host is allowed to see
> which disks. Doing this end-to-end using TCP/IP is problematic because the NAS controller doesn't know
> about tenant id, and whether the MAC and IP are being duplicated across customers. To make the storage
> aware of segmentation, you have to carry some segment (GRE, MPLS, VLAN, etc.) into the storage box.
> That's not true today.
> 
> Maybe I'm dense, but wouldn't it be just as easy to add the capability
> to differentiate based on IP address as it would be to design the entire
> network and all protocols around what the current software capabilities
> on the hypervisor are?

As previously noted, Ashish's original statement about what storage boxes cannot do was just
plain wrong, as there's plenty of VLAN support on current storage boxes.  FWIW, differentiation
based on IP address is not difficult, although one has to think carefully about scope of IP
address as differentiating on source IP address assumes that the source IP address isn't forged
(VLANs are easier to work with than IP blocks for this concern).

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Mon, 2 Jan 2012 09:28:32 +0530

David,

On a L3 network, VLAN will help you contain broadcasts but not enough security. E.g. If a tenant has more than one VLAN then you can route across the VLANs. Now, you need IP to identify. This now opens up to issues like IP duplication. 

VLANs will work if one customer equals one VLAN. The other thing I find interesting is that this conversation happened in the context of L3 vs. L2 and you mentioned that TCP/IP can be used for storage as a way to avoid L2. But, now, we need VLAN for segmentation. Now when you do DCI, then you need MAC over IP (or EoMPLS or something else), but not IP over IP (the entire context of this discussion).

http://tools.ietf.org/html/draft-dalela-dc-approaches-00 analyzes many of these issues collectively. I believe the goal of this group should be to take the problem set as a whole - and we know there are many possible approaches that already exist.

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, January 02, 2012 12:40 AM
To: [email-masked]; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center interconnect

Thanks,
--David

_______________________________________________
dc mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dc

---

From: Pedro Marques <[email-masked]>
Date: Sun, 1 Jan 2012 20:55:47 -0800

Ashish,
Please see inline.

On Sun, Jan 1, 2012 at 7:58 PM, Ashish Dalela (adalela)
<[email-masked]> wrote:
> VLANs will work if one customer equals one VLAN.

Clearly not an useful assumption. A customer must be able to user
multiple 'segments'/'security groups' and have traffic be exchanged
among them.

As for the 'issues like IP duplication'. It would be preferable to
assume in the conversation  basic security features such as the
verification of the inner and outer addresses, whether this is IP in
mac or IP in IP.

> The other thing I find interesting is that this conversation happened in the context of L3 vs. L2 and you mentioned that TCP/IP can be used for storage as a way to avoid L2. But, now, we need VLAN for segmentation.

Why do you need VLANs segmentation ? An exactly what is the scope of
the VLANs that you need for segmentation ? interface-scope or
data-center scope ?

  Pedro.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Mon, 2 Jan 2012 10:55:26 +0530

Pedro,

>> It would be preferable to assume in the conversation  basic security
features such as the verification of the inner and outer addresses,
whether this is IP in mac or IP in IP.

That's another very important issue that all network services
(firewalls, load-balancers, wan optimizer, ..) need to take into account
the change in the header structure (mentioned in the "approaches"
draft). Changes to these structures break network services. So, if there
is no generally agreed upon way of doing this, then there are risks.
Also, how does a service know how many headers to look at?

I believe this datacenter scope. Most of the reasons are historical.
Broadcast discovery (including ARP resolve), simpler configuration.

Thanks, Ashish

-----Original Message-----
From: Pedro Marques [mailto:[email-masked]] 
Sent: Monday, January 02, 2012 10:26 AM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,
Please see inline.

On Sun, Jan 1, 2012 at 7:58 PM, Ashish Dalela (adalela)
<[email-masked]> wrote:
>
> David,
>
> On a L3 network, VLAN will help you contain broadcasts but not enough
security. E.g. If a tenant has more than one VLAN then you can route
across the VLANs. Now, you need IP to identify. This now opens up to
issues like IP duplication.

> The other thing I find interesting is that this conversation happened
in the context of L3 vs. L2 and you mentioned that TCP/IP can be used
for storage as a way to avoid L2. But, now, we need VLAN for
segmentation.

  Pedro.

---

From: Pedro Marques <[email-masked]>
Date: Mon, 2 Jan 2012 09:40:52 -0800

Ashish,
The characteristics that historically made L2 the lowest cost solution
for data-center access and aggregation become strong drawbacks one you
assume that a member of the 'segment' can be anywhere in the
data-center.

One of the requirements that is important to note is that tenants need
multiple network 'segments' (i.e. a logical VLAN) and that these
segments must be able to exchange traffic with a control similar to
today's vlan-to-vlan routed ACLs.

In traditional L2/L3 designs the VLAN interfaces sit at the
aggregation layer, separating downstream L2 vlans. Once the members of
the "VLAN" can be anywhere in the DC, the inter-vlan traffic exchange
capability must now also be supported taking that into account.

Mobile IP-style home agent solutions are not acceptable given that
they imply traffic crossing the DC network twice.

If you want to follow a traditional 3 tier design, you end up with
having all the aggregation switches being members of all the vlans.
Creating very very large broadcast domains with typically a small
number of end-systems.

VM are also not managed the way that traditional servers are. The
traditional advantages of a mixed L2/L3 design not longer apply here.
When a VM is instantiated the management system that does instantiate
it knows its network address(es). The traditional L2 "plug-and-play"
simplicity is now irrelevant because at any reasonable scale VMs are
not "plug-and-play"... they are explicitly managed.

There is the need in the IETF to standardize solutions for "larger
broadcast domains". They are applicable to small data-centers and
there is a large number of people that just want to extend the
traditional model since that is what they are familiar with.

But there is also interest in standardizing solutions that do not use
broadcast domains. There are enough people that believe the later are
cheaper to operate at large scale.

  Pedro.

---

From: <[email-masked]>
Date: Mon, 2 Jan 2012 17:33:43 -0500

Ashish,

> On a L3 network, VLAN will help you contain broadcasts but not enough security. E.g. If a tenant has
> more than one VLAN then you can route across the VLANs. Now, you need IP to identify. This now opens
> up to issues like IP duplication.

That's not correct - IP duplication across tenants is already an issue with one VLAN per tenant (e.g., multiple tenants use 10.0.0.0/8 addresses), and allowing multiple VLANs per tenant doesn't change that situation significantly *as long as* the ability to "route across the VLANs" is scoped by tenant - that routing is simply not set up to allow forwarding across VLANs for different tenants, ditto for the per-tenant VPN interfaces (e.g., to connect to tenant data centers for hybrid cloud).  This approach extends to the virtual networks for NVGRE and VxLAN.

> The other thing I find interesting is that this
> conversation happened in the context of L3 vs. L2 and you mentioned that TCP/IP can be used for
> storage as a way to avoid L2.

Use of TCP/IP in storage protocols allows L3 routing, but does not require use of L3 routing.  VLANs for segmentation are also not incompatible with L3 routing; see above.

> But, now, we need VLAN for segmentation. Now when you do DCI, then you
> need MAC over IP (or EoMPLS or something else), but not IP over IP (the entire context of this
> discussion).

In the context of storage provided by the cloud provider as a service to tenants, I strongly disagree; neither iSCSI nor NFS requires L2 connectivity between the storage systems and servers (physical or virtual [e.g., VMs]) that use the storage.

Beyond that, I observe that in current data centers, physical and virtual servers/machines most commonly consume an L2 networking service, and hence I'm skeptical of approaches that start by declaring that only an L3 service will be offered on the grounds that such approaches may be disruptive.  That causes me to prefer MAC-over-IP data-center-interconnect approaches to IP-over-IP - I don't object to IP-over-IP in principle, rather I'm skeptical that it effectively solves the entire problem for virtual machine mobility across data centers.

Thanks,
--David

> From: Ashish Dalela (adalela) [mailto:[email-masked]]
> Sent: Sunday, January 01, 2012 10:59 PM
> To: Black, David; [email-masked]; [email-masked]
> Subject: RE: [dc] [armd] IP over IP solution for data center interconnect
> 
> 
> David,
> 
> On a L3 network, VLAN will help you contain broadcasts but not enough security. E.g. If a tenant has
> more than one VLAN then you can route across the VLANs. Now, you need IP to identify. This now opens
> up to issues like IP duplication.
> 
> VLANs will work if one customer equals one VLAN. The other thing I find interesting is that this
> conversation happened in the context of L3 vs. L2 and you mentioned that TCP/IP can be used for
> storage as a way to avoid L2. But, now, we need VLAN for segmentation. Now when you do DCI, then you
> need MAC over IP (or EoMPLS or something else), but not IP over IP (the entire context of this
> discussion).
> 
> http://tools.ietf.org/html/draft-dalela-dc-approaches-00 analyzes many of these issues collectively. I
> believe the goal of this group should be to take the problem set as a whole - and we know there are
> many possible approaches that already exist.
> 
> Thanks, Ashish
>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 02 Jan 2012 23:39:45 +0100

Hi David,

 > I don't object to IP-over-IP in principle, rather I'm skeptical that
 > it effectively solves the entire problem for virtual machine mobility
 > across data centers.

Would you care to elaborate which part of the VM mobility could not be 
solved in your opinion by IP in IP ?

Many thx,
R.

---

From: <[email-masked]>
Date: Mon, 2 Jan 2012 21:54:59 -0500

Hi Robert,

> solved in your opinion by IP in IP ?

Here's a start:

I'm interested in an IaaS public or hybrid cloud where the tenants bring
their own VMs.

For VMs that are using L2 networking service, changing a VM MAC address as
part of live migration of the VM is problematic, because traffic to the VM
is likely to be dropped until the new MAC propagates (e.g., have to wait
for ARP caches to update).  As part of this discussion, I'm assuming
no ability to modify the IP stacks in the VMs (e.g., the VMs may be based
on standard OS images).  In essence, it takes too long to rebind an IP to
a new MAC, and the result can be disruptive to a live VM migration.  There
are probably examples of VMs that don't care about this sort of disruption
(e.g., where it's ok to expect a user to occasionally hit Reload in the
browser), but there are plenty of examples where that disruption is not
acceptable.

If one is prepared to use only L3 networking service with VMs, I believe
IP in IP is workable. An example of what I mean by an L3 service is that the
subnet consists of one link (i.e., no L2 bridging), which allows reuse of L2
addresses on the link (e.g., the VM's MAC doesn't change even though it's on
a new link, and its ARP cache is still valid).  As already noted on this list,
a significant concern with this sort of approach is the number of host routes
needed at scale.

The idea of larger L2 subnets connected by IP-in-IP across data centers
seems interesting, but one has to somehow propagate the gratuitous ARP that
occurs after a live VM migration.  I haven't thought much about this - 
it may be feasible to somehow preserve the MAC across the VM migration,
keep ARP working across the IP-in-IP connection, etc. without reconstructing
a full MAC-in-IP connection.  If anyone has details on something useful
that's been done here, please share.

I don't want to say that people can't or shouldn't use L3 IP-in-IP across
data centers, but I do want to ensure that we can develop solutions that
support live VM migration without breaking (or requiring massive changes to)
the L2 networking service that that is commonly used by VMs.

If one is prepared to modify the IP stack, e.g., by inserting some sort of
locator/identifier separation protocol, then the rebinding problem changes
from IP-to-new-MAC to ID-to-new-LOC.  That's plausible if one has complete
control of that level of infrastructure, but I don't think it's a good
starting assumption for the IaaS scenarios that I'm interested in.

Does that help?

Thanks,
--David

> From: Robert Raszuk [mailto:[email-masked]]
> Sent: Monday, January 02, 2012 5:40 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 17:08:49 +0530

Pedro,

I'm not advocating classic L2. If you have the seen the "approaches" draft I forwarded to this alias, I have listed multiple deficiencies of L2, including map-encap L2. Also discussed is the hierarchical MAC approach that scales better than all approaches known so far. 

L2 forwarding does not necessarily mean a broadcast domain. E.g. you can forward packets based on hierarchical MAC without a VLAN and hence never broadcast. The packets will be forwarded just like static IP routing but based on a MAC prefix. 

As you have correctly mentioned, some people need broadcasts and some don't. So far L2 = broadcast and L3 = non-broadcast. Hierarchical MAC doesn't have that assumption. I would like to converge TCP/IP and non-TCP/IP, broadcast and non-broadcast, traffic into a single forwarding approach that will scale. If you have VLAN you get broadcast, multicast and unicast. If you don't have VLAN, you just get unicast and multicast. Shift from one to another should require only a configuration of a VLAN tag on the port, not a change of the entire infrastructure. Agree?

The thing to bear in mind is that there will be customers who want either, and a provider has to use the same infra to support both. If you lock a provider into a exclusively L2 or L3, then they can't use the same infrastructure for supporting either. 

I don't want to split the requirements into multiple use-cases because then this DC group will be many groups - one doing L2 and another doing L3. That I think you will agree is not optimal for anyone. So, I'm looking for an approach that can converge all requirements into a single forwarding plane, on the same infrastructure. The issues of scale you mentioned don't exist in Hierarchical MACs, which scale better than anything we know of.

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Pedro Marques
Sent: Monday, January 02, 2012 11:11 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center interconnect

Ashish,

  Pedro.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 14:35:20 +0100

Ashish,

> The issues of scale you mentioned don't exist in Hierarchical MACs,
> which scale better than anything we know of.

So you are advocating solution which is based on encapsulation - that is
fine.

However how could you ever arrive at the conclusion that HMACs would
scale better then "anything we know". Well I don't know about you, but I
know that the key to scaling is ability to aggregate. And it is not that 
huge mystery that MACs aggregate rather poorly while there are quite 
well deployed protocols (be it IPv4 or IPv6) which aggregate natively.

For inter-dc this is IMHO a must. A must even if you build it using
traditional routers or OF enabled switches - does not matter.

> I don't want to split the requirements into multiple use-cases
> because then this DC group will be many groups - one doing L2 and
> another doing L3. That I think you will agree is not optimal for
> anyone

Why MAC-in-IP does not solve it for everyone ? And there are deployed 
solutions already ..

IMHO what this group should accomplish is not to try to reinvent the 
world, but perhaps as example discuss where is the right boundary of 
encapsulation, how should we communicate between network and hosts, what 
kind of DC instrumentation should be IETF blessed for easy integration 
(ie min subset of functionality it should possess etc .... )

R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 14:51:50 +0100

Hello David,

Much thx for the below comments.

IaaS is indeed very good example where you don't have control over 
customer VMs .. you are not essentially able to provision such VMs. 
However I am assuming that VM mobility is within your network boundaries 
or boundary interfaces such VMs may connect to.

 > As part of this discussion, I'm assuming no ability to modify the IP
 > stacks in the VMs (e.g., the VMs may be based on standard OS images).

You know that for locator/identifier separation protocol there are both 
host based ILNP and network based proposals LISP. Only ILNP requires 
changes to the VM OS (which IMHO is not that big of a deal and provide 
OS images your customers could use in their VMs which support ILNP). But 
let's not too far on that.

--

So for scalable IaaS at the L2 level what seem to be missing is 
effectively an ARP servers which would provide VM MAC resolution plane 
to IP address used for it's encapsulation.

Just like DNS resolves char names to IP address or LISP ALT resolves IDs 
to locators we now need a way to within your domain (IaaS boundary) or 
maybe even one day beyond resolve MAC address to IP.

Do we have such proposal already on the table ?

Disclaimer: This is only for those cases where native L3 or ID/Loc 
schemes could not be used VM to VM.

Thx,
R.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 19:30:10 +0530

Robert,

>> So you are advocating solution which is based on encapsulation - that
is fine.

No, I'm not. Did you read the draft I had mentioned? 
Hierarchical MAC is not encapsulation. It is one 48 bit address.

>> well deployed protocols (be it IPv4 or IPv6) which aggregate natively

You are hitting the issue on the nail. So, read the draft I mentioned. 
Hierarchical MAC is higher bits "network prefix" and lower bits "host
id". 
That's summarizable like IP address and aggregated. 
It has 46 bits to modify so larger than IPv4 internet.

I won't comment on the rest, because you have made an assumption about
encapsulation.

I refer to this -
http://tools.ietf.org/html/draft-dalela-dc-approaches-00. 

Thanks, Ashish

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 7:05 PM
To: Ashish Dalela (adalela)
Cc: Pedro Marques; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,

R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 15:54:01 +0100

Ashish,

OK let's just discuss what is in your draft on Hierarchical Addressing.

1. You have 48 bits 32 go for host remaining 16 goes for switches. How 
do you aggregate at the TOR or AGGR switch boundary ? Are you assuming 
single HOST - SWITCH with max 65K flat macs ?

2. Can you deploy this on existing VMs and existing switches ?

3. What new protocol you envision to use to distribute those new MACs ?

4. What is the advantage of using this vs ILNP if we assume that hosts 
should be modified ?

5. The proposal does not support aggregation .. even the draft says it :)

"The total number of hardware entries anywhere in the network equals the 
total number of switches and remains agnostic of VM mobility."

So if I have 100K switches I can not do any aggregation and need to 
"route" 100K MAC addresses.

6. Who provides me the mapping between switch mac and host/vm mac behind 
such switch ? Do switches proxy arp globally within your domain ?

Thx,
R.

---

From: DIEGO LOPEZ GARCIA <[email-masked]>
Date: Tue, 03 Jan 2012 16:03:24 +0100

> capability must now also be supported taking that into account.

Agreed.

> they imply traffic crossing the DC network twice.

Sure for the steady state, but I guess it could be an acceptable solution in transient situations, when a migration is taking place.

> not "plug-and-play"... they are explicitly managed.

Agreed. Anyway, you'll agree that "sensible default values" would be useful...

> cheaper to operate at large scale.

You can count me in…

Be goode,

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D

e-mail: [email-masked]
Tel:      +34 913 129 041
Mobile: +34 682 051 091
-----------------------------------------

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at.
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

---

From: DIEGO LOPEZ GARCIA <[email-masked]>
Date: Tue, 03 Jan 2012 16:41:27 +0100

Hi,

> Do we have such proposal already on the table ?

I am probably missing some detail, but could not this resolution be done at the virtual network manager in the hypervisor? That would require to only change host hypervisors and not guest VM OSs.

Be goode,

Dr Diego R. Lopez
Telefonica I+D

e-mail: [email-masked]
Tel:      +34 913 129 041
Mobile: +34 682 051 091

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 16:46:30 +0100

Hi Diego,

The point is to avoid storing such mapping everywhere (read in every 
host's hypervisor). Only those mapping which are necessary for a given 
host should be stored locally if at all necessary.

Analogy ... Do you store all DNS entries locally on your pc/mac ?

Best,
R.

---

From: DIEGO LOPEZ GARCIA <[email-masked]>
Date: Tue, 03 Jan 2012 17:12:01 +0100

Hi Robert,

> Analogy ... Do you store all DNS entries locally on your pc/mac ?

I cannot agree more. You store data in some common (distributed) service, but make the hypervisor do the resolution. Much like resolvers and servers in DNS as well.

Be goode,

Dr Diego R. Lopez
Telefonica I+D

e-mail: [email-masked]
Tel:      +34 913 129 041
Mobile: +34 682 051 091

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 21:45:09 +0530

Robert,

Please see inline.

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 8:24 PM
To: Ashish Dalela (adalela)
Cc: Pedro Marques; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,

[AD] The higher bits identify a switch - it's a switch-id. The hosts are
dynamically assigned a host-id under that switch. Let's assume 23 bits
are for switch-id and 23 bits for host-id. To forward a packet to the
host, you only have to look at the first 23 bits. That's a MAC prefix to
route against.

[AD] You can have 2^23 switches in a network and 2^23 hosts under each
switch.

[AD] What do you mean by this? Any VM can be configured with any MAC.
Any physical host can be configured with any MAC on any logical
interface. Configuration standpoint this is possible. Forwarding
standpoint, that's another question.

[AD] IS-IS extensions. It can be TRILL extensions.

[AD] I'm not familiar with the ILNP work, but I'm assuming you are
talking about Loc-Id separation. If not, correct me. If yes, each Loc-Id
binding can be a host route, with mobility. These host-routes are a
scaling problem. Traditional IP packet have IP as ID and MAC as LOC. We
are just extending this LOC to make it actually location aware rather
than a flat address which is fixed regardless of where the location is.

5. The proposal does not support aggregation .. even the draft says it
:)

[AD] For 1 million VM, and 25 VM per host, you need 40,000 hosts. With
48 port access switches, you need 833 switches. That's the routing table
size for any switch in the datacenter - core, aggregation, access.
Contrast this with host-routes, if each VM talks to 100 VMs, then each
access switch needs 48 * 25 * 100 = 120,000 host routes. Just because
the network prefix is 23 bits does not mean we have to store 10^23
prefixes. We have to store only as many switches as there are in the
network. Ratio between VM : switch is 1000 : 1 (today, assuming 48 port
access and 20 VM per port). That means instead of storing host-routes
which will grow proportional to VM growth, we store switch-id, which
will grow at 1000 times slower rate. As VM density increases, this
growth rate is further slowed down. There are other techniques to
further reduce the rate of growth. But in any case, 1000 times slower is
a lot slow.

[AD] I don't know how you came to that conclusion. Think of HMAC as an
IP address. Instead of 32 bits it is 46 bits. You route by prefixes in
L3, and you are routing by the same prefixes here. Just as you aggregate
IP, same way you aggregate MAC. It's not different.

[AD] Variation of the same question. Above should answer it.

Thx,
R.

> So you are advocating solution which is based on encapsulation - that
is

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 17:37:30 +0100

Ashish,

> host, you only have to look at the first 23 bits. That's a MAC prefix to

I don't think we are at the point where the biggest worry is how to 
optimize forwarding to be on as little bits as necessary. I think quite 
a few real hardware can forward today at line rate even by 128 bit lookup.

> standpoint, that's another question.

I mean forwarding and ARP IP to new MAC resolution changes required.

> scaling problem.

Can you provide any substantial evidence that DNS/DNSSec can not scale 
with host routes ?

> Traditional IP packet have IP as ID and MAC as LOC.

Wrong. Traditional IP address is both locator and ID today. Hint: 
Locators and IDs must be able to be resolved globally for system to work 
globally.

> We

That is what Loc/ID split is all about. Perhaps your proposal could be 
discussed in RRG or better in LISP WGs ....

> size for any switch in the datacenter - core, aggregation, access.

So if I move a VM from one switch to the other I need to change it's MAC 
address. Here I already have issue ... if I just change the switch 
portion or entire address .. how are my TCP sessions are going to stay 
up ? I am assuming IP address v4 or v6 did not change - correct ?

Could you provide walk step by step the sequence of events for VM 
mobility in your solution ?

> prefixes.

You are persistently stuck with thinking flat. Think DNS or any other 
mapping plane. There is zero requirement that any entity should store 
all entries. All it needs to story is sufficient information to know how 
to get information where needed. That is significant difference.

> We have to store only as many switches as there are in the

Nope. I am not storing host routes. In fact in the previous email I 
explicitly commented that flooding and storing host routes flat is not 
scalable.

R.

---

From: <[email-masked]>
Date: Tue, 3 Jan 2012 11:41:36 -0500

Ashish,

> > [AD] The higher bits identify a switch - it's a switch-id.

That breaks VM migration across switches by forcing a MAC change.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Ashish Dalela (adalela)
> Sent: Tuesday, January 03, 2012 11:15 AM
> To: [email-masked]

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 22:31:58 +0530

Robert,

optimize forwarding to be on as little bits as necessary. I think quite 
a few real hardware can forward today at line rate even by 128 bit
lookup

What's the point of saying this? I'm just describing what a HMAC is. It
will be at most 48 bits.

>> I mean forwarding and ARP IP to new MAC resolution changes required.

NO changes to ARP. Changes to forwarding yes. Instead of looking up 48
bits of a MAC, we look up fewer bits. CAM vs. TCAM.

with host routes ?

What does this have to do with DNS? You have an ID and you need a LOC,
you need a binding. That binding is a host-route. I'm talking about
hardware entries. 

globally.

You are probably a routing guy :-) Think of switching. You unplug a host
from one port and plug into another. The switch learns the new location
based on the MAC. There is no separate locator assigned in switching.
MAC is the locator.

Today on VM mobility MAC can change (it is implementation dependant).
Host sends a Grat ARP announcing its new IP-MAC binding. IP never
changes.

1. VM is tied to switch S1, and has a host id H1. Its MAC is S1:H1
2. VM is moved to switch S2, and gets a host id H2. Its MAC is S2:H2.
3. VM sends a Grat ARP for S2:H2.
4. Network installs a temporary redirect at S1 to redirect all packets
sent to S1:H1 to S2:H2.
5. In 15-30 seconds every host ARPs for the moved VM, and gets S2:H2.
6. Every host starts sending packet to S2:H2 in 15-30 seconds.
7. Temporary redirect installed at step 4 ages in 60 seconds.

DNS is control plane. I'm talking about host-route entries installed in
hardware. This is used for forwarding.

We've been thru this discussion any number of times in the past. Look at
any map-encap for mobility and it needs to store host routes. You might
want to look at the older emails.

Thanks, Ashish

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 10:08 PM
To: Ashish Dalela (adalela)
Cc: Pedro Marques; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,

I don't think we are at the point where the biggest worry is how to 
optimize forwarding to be on as little bits as necessary. I think quite 
a few real hardware can forward today at line rate even by 128 bit
lookup.

globally.

R.

---

From: <[email-masked]>
Date: Tue, 3 Jan 2012 12:02:37 -0500

Just saw this:

> If the VM can be moved, then all you need to do is install a temporary
> redirect of packets to the new location. Each host will refresh the MAC
> after 15-30 seconds. If the packets are redirected from old to new
> location for these 30 seconds, the redirect can be aged automatically.
> This happens all the time in mobile networks in what is called a "fast
> handoff" where you redirect the packets until handoff is completed.

"all you need to do" should incur a $5 fine for the network version of
"a simple matter of programming" :-).

I see a bunch of problems here:

Mobile networks aren't running VMs.  ARP caches matter - a little bit of
Googling turns up multiple minute timeouts for some older OS versions and 4
hours for some routers.

In addition, some VMs need static MACs (e.g., software license key or
encryption key may be bound to the MAC).  There will be other reasons why
a VM's MAC cannot be changed when it is live migrated.

This also doesn't cover physical host usage of the burnt-in hardware MAC
on the NIC.  I assume that those need to continue to work as-is.  For
46-bits, I'm guessing that the Hierarchical MAC approach takes all the
locally administered MACs.  If any of those MACs are already in use,
that'll be a problem.

It also looks like it's seriously disruptive to existing L2 network management.

I see no mention of TRILL, SPB or PBB in the draft that advocates Hierarchical
MACs - those are three existing approaches to this problem that can aggregate
without forcing MAC changes on live VM migration.  At the very least, a solid
comparison to existing technology is in order.

Finally, this sort of MAC hierarchy work probably belongs in IEEE, not IETF.

Thanks,
--David

> Sent: Tuesday, January 03, 2012 11:42 AM
> To: [email-masked]
> Cc: [email-masked]

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 22:40:01 +0530

David,

It doesn't have to. A VM will send a Grat ARP. It's informing about its
MAC address to IP binding. If you shut the interface before move and
unshut it after move, and use the unshut to get a new MAC it can be
done. It's implementation that we have to talk about. We are inventing
so many host based solutions - can't we discuss this change?

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 10:12 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: RE: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,

Thanks,
--David

interconnect
> >>> know that the key to scaling is ability to aggregate. And it is
not
> > that
> > scale better then "anything we know". Well I don't know about you,
but
> I

---

From: Pedro Marques <[email-masked]>
Date: Tue, 3 Jan 2012 09:12:33 -0800

Ashish,

On Tue, Jan 3, 2012 at 3:38 AM, Ashish Dalela (adalela)
<[email-masked]> wrote:
>
> Pedro,
>
> I'm not advocating classic L2.

> L2 forwarding does not necessarily mean a broadcast domain.

The forwarding algorithm is perhaps an implementation detail. It would
be great to understand what is the service model that you propose. A
IEEE 802 service model does imply a broadcast domain.

>. E.g. you can forward packets based on hierarchical MAC without a VLAN and hence never broadcast. The packets will be forwarded just like static IP routing but based on a MAC prefix.

If you service model only provide IP transport, why do you care about
the MAC address ? It seems totally irrelevant to the service being
provided. On the other hand to implement an IEEE 802 compatible
service, MAC addresses are required.

> So far L2 = broadcast and L3 = non-broadcast.

It would be best to speak in terms of the service functionality to applications.

>Hierarchical MAC doesn't have that assumption. I would like to converge TCP/IP and non-TCP/IP, broadcast and non-broadcast, traffic into a single forwarding approach that will scale.

As long as that is free it sounds good... is however important to keep
in mind that there is a class of deployments for which non-IP and
broadcast traffic is irrelevant and for which cost is the single most
important factor.

> If you have VLAN you get broadcast, multicast and unicast. If you don't have VLAN, you just get unicast and multicast. Shift from one to another should require only a configuration of a VLAN tag on the port, not a change of the entire infrastructure. Agree?
>

I'm sorry but i'm not able to understand what you mean.

> The thing to bear in mind is that there will be customers who want either, and a provider has to use the same infra to support both. If you lock a provider into a exclusively L2 or L3, then they ca…

That is very confusing to me. Why would multiple infrastructures be
necessary to support different services ? For instance in the carrier
space, L3 overlays and L2 overlays are supported on top of the same
infrastructure every day.

> I don't want to split the requirements into multiple use-cases because then this DC group will be many groups - one doing L2 and another doing L3.

If it would split in many groups each solving a specific focused
problem then progress could actually be made. For instance both an L2
and L3 groups could actually converge on requirements.

That sounds a much better option than to search for the holly grail of
the solution that provides an IEEE 802 service at no extra cost.

> That I think you will agree is not optimal for anyone.

It is not clear what you are trying to optimize.

> So, I'm looking for an approach that can converge all requirements into a single forwarding plane, on the same infrastructure.

The infrastructure and the service of the overlay network to VMs are
different layers. Conflicting the two is not useful.

  Pedro.

---

From: <[email-masked]>
Date: Tue, 3 Jan 2012 12:14:37 -0500

It's not compatible with widely deployed VM live migration mechanisms that don't
change the MAC.  As noted in my other follow-up email, things will break, and I'd
prefer to look at solutions that don't break things that currently work.

IMHO, this thread is also well off-track, as you're advocating a solution as
opposed to coming up with a concise problem statement.

Thanks,
--David

> Sent: Tuesday, January 03, 2012 12:10 PM
> To: Black, David
> Cc: [email-masked]
> > > but I

---

From: Pedro Marques <[email-masked]>
Date: Tue, 3 Jan 2012 09:16:22 -0800

That assumes that the MAC has relevance in the network. It is possible
to build solutions such that packets are forwarded based on their IP
addresses rather than their MACs.

  Pedro.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 23:01:20 +0530

I had sent a problem statement earlier, but did not see comments.
http://www.ietf.org/id/draft-dalela-dc-requirements-00.txt

Let's see which ones are solved, and which aren't. 

>> As noted in my other follow-up email, things will break, and I'd
>> prefer to look at solutions that don't break things that currently
work

Scale will break the network while it preserves the VM functionality. 

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 10:45 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: RE: [dc] [armd] IP over IP solution for data center
interconnect

It's not compatible with widely deployed VM live migration mechanisms
that don't
change the MAC.  As noted in my other follow-up email, things will
break, and I'd
prefer to look at solutions that don't break things that currently work.

IMHO, this thread is also well off-track, as you're advocating a
solution as
opposed to coming up with a concise problem statement.

Thanks,
--David

interconnect
> > host, you only have to look at the first 23 bits. That's a MAC
prefix
> to
> > 3. What new protocol you envision to use to distribute those new
MACs
> ?
> > further reduce the rate of growth. But in any case, 1000 times
slower
not
> > > encapsulation, how should we communicate between network and
hosts,
> > what

---

From: Pedro Marques <[email-masked]>
Date: Tue, 3 Jan 2012 09:31:48 -0800

David,

> prefer to look at solutions that don't break things that currently work.

Whether the VMs change the MAC or not is not necessarily relevant
either. What is the service model to the VM ? Is it an IEEE compatible
LAN or an ethernet (point-to-point) interface with the ability to
carry IP traffic ?. In the later case the MAC is not relevant.

  Pedro.

---

From: <[email-masked]>
Date: Tue, 3 Jan 2012 12:35:17 -0500

> carry IP traffic ?. In the later case the MAC is not relevant.

The widely deployed service model for VMs that I'm thinking of is
an IEEE-compatible LAN (L2 network service, for which the MAC matters).

Thanks,
--David

> Sent: Tuesday, January 03, 2012 12:32 PM
> To: Black, David
> Cc: [email-masked]

---

From: Pedro Marques <[email-masked]>
Date: Tue, 3 Jan 2012 09:42:37 -0800

David,
I propose the following as a problem statement. The IETF goal should
be to standardize:
   1) an IEEE-compatible LAN service (which implies broadcast support).
   2) an IP unicast service.

Both services should be able to run on top of a IP infrastructure
network as an overlay. Progress on both service models should be made
independently.

  Pedro.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 23:15:25 +0530

Suppose you have an IP solution.

To support mobility you need IP-in-IP encapsulation. 

As VM density increases, as VM-to-VM conversation grows, as interfaces
per VM increase, the host routes increase.

These host routes are in addition to network routes, local host-port
bindings, ACLs, etc. That means in addition to everything that existed
so far. 

Eventually, you hit a limit on the access, and you have to reduce size
of network, reduce VM mobility, reduce VM density per server, reduce
application spread. 

The alternative is to constantly increase network hardware table sizes
at access, which increases costs and energy.

We have to realize that IP encapsulations put network and compute at
opposite sides of the cost trend. Compute cost reduces slowly as size
grows. Network cost grows rapidly as size grows. 

Thanks,
Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Pedro Marques
Sent: Tuesday, January 03, 2012 10:46 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

  Pedro.

interconnect
not
but

---

From: Pedro Marques <[email-masked]>
Date: Tue, 3 Jan 2012 09:59:32 -0800

Ashish,

> bindings, ACLs, etc. That means in addition to everything that existed

Whether you call the information "host routes" or "mac addresses" does
not change the fact that you have identities associated with the
end-points (VMs).

> of network, reduce VM mobility, reduce VM density per server, reduce

I'm sorry but that was just a huge leap in the reasoning... i'm not
able to understand  how you reached these conclusions.

> grows. Network cost grows rapidly as size grows.

When you refer to "IP encapsulations" in the sentence above... do you
mean X on top of IP or IP on top of X ?

Either way the claim seems to be rather unsubstantiated.

  Pedro.

---

From: Steven Blake <[email-masked]>
Date: Tue, 3 Jan 2012 17:23:12 -0500

> host, you only have to look at the first 23 bits. That's a MAC prefix to

[snip]

FYI, MOOSE is a detailed proposal for hierarchical MAC addressing,
resolution, and forwarding:

http://www.cl.cam.ac.uk/~mas90/MOOSE/

Regards,

/////////////////////////////////////////////
Steven Blake       [email-masked]
Extreme Networks              +1 919-884-3211

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Wed, 4 Jan 2012 06:37:55 +0530

Yes, there is also PORTLAND. http://cseweb.ucsd.edu/~vahdat/papers/portland-sigcomm09.pdf
Thanks, Ashish

-----Original Message-----
From: Steven Blake [mailto:[email-masked]] 
Sent: Wednesday, January 04, 2012 3:53 AM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center interconnect

[snip]

Regards,

---

From: Warren Kumari <[email-masked]>
Date: Thu, 5 Jan 2012 10:36:29 -0500

> Suppose you have an IP solution.

Sure.

> To support mobility you need IP-in-IP encapsulation. 

And if you do an overly you always do an IP encapsulation (to cover GRE, IPIP, sit, IPSec, PPP, etc). 

> per VM increase, the host routes increase.

No.

The only thing that the network needs to know is the routes to the hypervisors / physical machines -- this is a solved problem.
The VM addresses and routes are only visible to the [gateways, hypervisors with VMs in that overlay, other VMs in the same overlay, mapping server].

For a really old overview: http://tools.ietf.org/html/draft-wkumari-dcops-l3-vmmobility-00

> bindings, ACLs, etc. That means in addition to everything that existed

No.

> of network, reduce VM mobility, reduce VM density per server, reduce

No.

> at access, which increases costs and energy.

No.

> grows. Network cost grows rapidly as size grows. 

No.

> Ashish Dalela (adalela)

---

From: AshwoodsmithPeter <[email-masked]>
Date: Thu, 5 Jan 2012 15:56:27 +0000

One thing that I've not seen discussed here is multicast state.

Clearly we can encapsualte unicast X in unicast Y and forward only on Y in the core thus eliminating knowledge of X and considerable savings. We've been doing so for years for various X's and Y's.

However if we wish to deal with multicast X in an efficient manner it complicates the problem and requires knowledge of or mapping of X multicast into Y.

What are people's thoughts on the requirement for efficient multicast (i.e. state in Y) v.s. emulated (serial unicast hence no state in Y) multicast solutions?

Peter Ashwood-Smith

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Thu, 5 Jan 2012 21:30:32 +0530

Hi Warren,

You are prescribing to a hypervisor based solutions and table scaling
issues don't arise there (or at least are not obvious) because it all
happens in software. However, there are other issues. For example, how
do you implement broadcast and multicast? The standard mechanism today
is to map a broadcast domain to a multicast group in L3. Now, what
happens if some rogue user sends an IGMP join to that group - what was
on the VLAN is now accessible to everyone through an IGMP join. For user
level multicast, there are other issues. Assume I'm doing a VDI cloud,
where users need to join multicast video conferencing. The group is user
determined, not admin determined. How do we know that the user is not
joining a VLAN mapped multicast group?

We need to keep a complete set of goals in mind. Otherwise, we can solve
an issue and miss a goal. E.g. multicast and broadcast. 

Thanks, Ashish

-----Original Message-----
From: Warren Kumari [mailto:[email-masked]] 
Sent: Thursday, January 05, 2012 9:06 PM
To: Ashish Dalela (adalela)
Cc: Warren Kumari; Pedro Marques; [email-masked]; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Sure.

> To support mobility you need IP-in-IP encapsulation. 

And if you do an overly you always do an IP encapsulation (to cover GRE,
IPIP, sit, IPSec, PPP, etc). 

No.

The only thing that the network needs to know is the routes to the
hypervisors / physical machines -- this is a solved problem.
The VM addresses and routes are only visible to the [gateways,
hypervisors with VMs in that overlay, other VMs in the same overlay,
mapping server].

For a really old overview:
http://tools.ietf.org/html/draft-wkumari-dcops-l3-vmmobility-00

No.

No.

No.

No.

prefix
MACs
slower

---

From: <[email-masked]>
Date: Thu, 5 Jan 2012 11:29:40 -0500

Ashish,

For IP-based overlays, this starts by carefully distinguishing the inner (user) and
outer (provider/infrastructure) IP address blocks, and completely controlling access
to the outer IP address blocks via encap/decap.  The result is that an IGMP join
from a user gets encapsulated and can't be processed with respect to those outer
addresses.  That creates a requirement to provide an IP multicast service to the 
users over the encapsulation.

Thanks,
--David

> Sent: Thursday, January 05, 2012 11:01 AM
> To: Warren Kumari
> Cc: Pedro Marques; Black, David; [email-masked]

---

From: Thomas Narten <[email-masked]>
Date: Thu, 05 Jan 2012 11:41:06 -0500

> an IEEE-compatible LAN (L2 network service, for which the MAC matters).

And specifically, within one VLAN. I.e., there is no notion of trying
to emulate multiple VLANs in such a service.

Right?

Thomas

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Thu, 5 Jan 2012 23:08:59 +0530

David,

So, where does the multicast packet get replicated, since the network is
not aware of user-level multicast groups? How are those distribution
trees to be built over an overlay?

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, January 05, 2012 10:00 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: RE: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,

For IP-based overlays, this starts by carefully distinguishing the inner
(user) and
outer (provider/infrastructure) IP address blocks, and completely
controlling access
to the outer IP address blocks via encap/decap.  The result is that an
IGMP join
from a user gets encapsulated and can't be processed with respect to
those outer
addresses.  That creates a requirement to provide an IP multicast
service to the 
users over the encapsulation.

Thanks,
--David

Ashish Dalela (adalela)
interconnect
> >>> [AD] The higher bits identify a switch - it's a switch-id. The
hosts
> > are
> >>> scaling problem. Traditional IP packet have IP as ID and MAC as
LOC.
> >>> than a flat address which is fixed regardless of where the
location
> > is.
> >>>
> >>> 5. The proposal does not support aggregation .. even the draft
says
> > it
> >>> :)
> >>>
> >>> "The total number of hardware entries anywhere in the network
equals
> > the
> >>> [AD] I don't know how you came to that conclusion. Think of HMAC
as
> > an
> >>> IP address. Instead of 32 bits it is 46 bits. You route by
prefixes
> > in
> >>>>>> So you are advocating solution which is based on encapsulation
-
not
hosts,

---

From: <[email-masked]>
Date: Thu, 5 Jan 2012 13:09:09 -0500

Ashish,

> trees to be built over an overlay?

The answer to this question starts from a realization that there are two
networks, and hence two address spaces, involved - the user-visible overlay
network and the underlying data center network that carries the overlay
network via encap/decap.  The user VMs (or physical servers) only see the
user-visible overlay network and its addresses, so the replication of user
multicast packets is a service of that user-visible overlay network
(specifically L3 nodes in that network that operate on the user-visible
IP addresses), not the underlying network.

To make this specific, suppose the user VMs are using 192.168.0.0/16 addresses,
and these are encapsulated over an underlying data center network that is
using 10.0.0.0/8 addresses.  If a user VM (e.g., VDI desktop) sends a join
request for a 10.0.0.0/8 address, *nothing* should happen, because those
10. addresses aren't visible in the overlay network.  Instead, the overlay
network is responsible for providing IP multicast service for the 192.168.0.0/16
addresses - the default IP gateway for that overlay network is one possible
location.

> How are those distribution trees to be built over an overlay?

The same way that they're build in a non-overlay network - the L3 multicast
functionality is the same, only the IP addresses are different.  This is
relatively straightforward for a MAC-in-IP (L2-in-L3) overlay when the service
provided by the L2 overlay includes multicast.  When there's no multicast
provided by the overlay's L2 service or no L2 service (IP-in-IP/L3-in-L3
approach), multicast for the overlay network may involve explicit copying of
packets at L3 nodes in the overlay network (not the underlying network),
and is part of the overlay design.

Thanks,
--David

> Sent: Thursday, January 05, 2012 12:39 PM
> To: Black, David
> Cc: [email-masked]

---

From: <[email-masked]>
Date: Thu, 5 Jan 2012 13:19:02 -0500

> to emulate multiple VLANs in such a service.

Right.  Multiple VLANs result in multiple separate instances of the L2 service
(e.g., accessed via multiple network interfaces [vNICs] in a VM that needs
access to multiple VLANs).

Thanks,
--David

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Thu, 5 Jan 2012 23:54:37 +0530

David,

When you have a default GW replicating packets, how do you get
multipath?

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Thursday, January 05, 2012 11:39 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Ashish,

The answer to this question starts from a realization that there are two
networks, and hence two address spaces, involved - the user-visible
overlay
network and the underlying data center network that carries the overlay
network via encap/decap.  The user VMs (or physical servers) only see
the
user-visible overlay network and its addresses, so the replication of
user
multicast packets is a service of that user-visible overlay network
(specifically L3 nodes in that network that operate on the user-visible
IP addresses), not the underlying network.

To make this specific, suppose the user VMs are using 192.168.0.0/16
addresses,
and these are encapsulated over an underlying data center network that
is
using 10.0.0.0/8 addresses.  If a user VM (e.g., VDI desktop) sends a
join
request for a 10.0.0.0/8 address, *nothing* should happen, because those
10. addresses aren't visible in the overlay network.  Instead, the
overlay
network is responsible for providing IP multicast service for the
192.168.0.0/16
addresses - the default IP gateway for that overlay network is one
possible
location.

> How are those distribution trees to be built over an overlay?

The same way that they're build in a non-overlay network - the L3
multicast
functionality is the same, only the IP addresses are different.  This is
relatively straightforward for a MAC-in-IP (L2-in-L3) overlay when the
service
provided by the L2 overlay includes multicast.  When there's no
multicast
provided by the overlay's L2 service or no L2 service (IP-in-IP/L3-in-L3
approach), multicast for the overlay network may involve explicit
copying of
packets at L3 nodes in the overlay network (not the underlying network),
and is part of the overlay design.

Thanks,
--David

interconnect
> > >>> 1. You have 48 bits 32 go for host remaining 16 goes for
switches.
> > > How
> > >>> dynamically assigned a host-id under that switch. Let's assume
23
> > > bits
> > >>> are for switch-id and 23 bits for host-id. To forward a packet
to
> > >>> [AD] You can have 2^23 switches in a network and 2^23 hosts
under
> > > each
> > >>> talking about Loc-Id separation. If not, correct me. If yes,
each
> > > Loc-Id
> > >>> binding can be a host route, with mobility. These host-routes
are
> a
> > >>> [AD] For 1 million VM, and 25 VM per host, you need 40,000
hosts.
> > > With
> > >>> 48 port access switches, you need 833 switches. That's the
routing
> > > table
> > >>> Contrast this with host-routes, if each VM talks to 100 VMs,
then
> > > each
> > >>> network. Ratio between VM : switch is 1000 : 1 (today, assuming
48
> > > port
> > >>> 6. Who provides me the mapping between switch mac and host/vm
mac
> > > behind
> > >>>>>> However how could you ever arrive at the conclusion that
HMACs
> > > would
> > >>>>>> huge mystery that MACs aggregate rather poorly while there
are
> > > quite
> > >>>> I won't comment on the rest, because you have made an
assumption
> > > about
-
> > >>>> IMHO what this group should accomplish is not to try to
reinvent
> > > the
> > >>>> world, but perhaps as example discuss where is the right
boundary
> > > of

---

From: "Stiliadis, Dimitrios (Dimitri)" <[email-masked]>
Date: Thu, 5 Jan 2012 13:28:17 -0600

David:

> provided by the L2 overlay includes multicast.

I am not sure how "straightforward" this is. Yes, it is simple, if you
are talking about broadcast traffic, and there is one L3 multicast group
associated with each L2 overlay. But if you are talking about a
"multicast" service in the L2 domain, one would need multiple L3 multicast groups
(one for each multicast-group of the L2 overlay). The core would
need to support a very large number of multicast groups and someone
(NMS ?) would need to manage the L2 group to L3 group associations.

Dimitri

> Sent: Thursday, January 05, 2012 10:25 AM
> To: [email-masked]
> Cc: [email-masked]
> > > >>> From: [email-masked] [mailto:[email-masked]] On
> Behalf
> > Of
> > > > Ashish Dalela (adalela)
> > > >>> [AD] What do you mean by this? Any VM can be configured with
> any
> > > MAC.
> > > >>> 4. What is the advantage of using this vs ILNP if we assume
> that
> > > > hosts
> > > >>> should be modified ?
> > > >>>
> > > >>> prefixes. We have to store only as many switches as there are
> in
> > the
> > > >>> So if I have 100K switches I can not do any aggregation and
> need
> > to
> > > >>>>>> scale better then "anything we know". Well I don't know
> about
> > > > you,
> > > >>>> but I
> > > >>>> Hierarchical MAC is higher bits "network prefix" and lower
> bits
> > > > "host
> > > >>>> id".

---

From: <[email-masked]>
Date: Thu, 5 Jan 2012 14:45:49 -0500

Same answer as for a non-overlay network - pick whatever multipath technology/design
that you want to use for L3 multicast (instead of just doing it at the default
gateway), and apply that design to L3 in the overlay network.

The overlay network really is a network and really offers actual networking services ...

Thanks,
--David

> Sent: Thursday, January 05, 2012 1:25 PM
> To: Black, David
> Cc: [email-masked]

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 5 Jan 2012 22:37:01 -0500

> For a really old overview: http://tools.ietf.org/html/draft-wkumari-dcops-l3-vmmobility-00

Is the issue that needs to be resolved for overlays only regarding the number of ARPs that need to be managed by the gateway?  Could this issue not be resolved operationally by say having more gateways?  

What other issues exist and need to be resolved for overlays, besides deciding on a tunneling encapsulation?  Is the IETF expected to standardize control protocols for overlays?

There is clearly a need for server-based virtual networks as well as a need for scalable network-based Ethernet virtual networks for DC.  Shouldn't these be separate conversations?

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Fri, 6 Jan 2012 10:33:13 +0530

Aldrin,

I like the way you say - "the only thing the network needs to know" -
:-) The same could be said of the hypervisor as well :-)

The problem is that once you start pushing the intelligence into the
server stack, you will have to keep pushing more and more  - e.g.
firewalls, multicast, broadcast, packet inspection, flow mgmt, etc. You
will find that you are re-inventing on the overlay everything that
exists on the underlay. The challenges are stacked up against the
overlay, not the underlay.

You will also find that you need to communicate between the overlay and
the underlay to get the desired bandwidth, QoS, flow mgmt, multipathing,
tree optimization, and that is never going to be easy. You will also
find that hardware accelerated networks perform better, deliver the
requisite high-availability, and consume less energy, as compared to
doing the same thing in the hypervisor.

Having said that, I recognize there are two models of virtualization
that we know of. The "overlay" model is like a network hypervisor in
which individual customers are like VMs. The "multiplexed" model is like
a multi-user OS. Both multi-user OS and the multi-VM hypervisor are
isolated equally well. But they have different use-cases.

The main use-case for the VM hypervisor model is to multiplex multiple
different OS/tools/application "environments" into the same HW, letting
each VM the ability to shut itself down, reboot or whatever it wants to
do. In case of network, that "environment" is a set of protocols. So, if
you have one tenant running OSPF another one running IS-IS and yet
another running BGP, and they want to keep playing with their network
configuration, it makes sense to run these in the overlay or the VM
mode. 

If everyone has the same environment (i.e. runs the same protocols, and
expects common controls), it makes more sense to run them in the
multi-user (multiplexed) mode rather than multi-VM (overlay) mode. The
multi-VM model delegates administration to the VM owner. The multi-user
model owns the administration while letting a "tenant" use the network.
This is a conscious choice - is cloud going to open up the configuration
of network like they open up the administration of VM? From what I know,
the answer is "no".

These architectural models exist outside the networking domain, and I
would reach out into another domain to borrow the intuitions, use-cases
and challenges. Architecturally, you will find the same challenges in
managing the overlay model as exist in the multi-VM model. If there are
no benefits to be gained from that, you are better off in the
multiplexed mode.

But then again - beauty lies in the eyes of the beholder.

Thanks, Ashish

From: Aldrin Isaac [mailto:[email-masked]] 
Sent: Friday, January 06, 2012 9:07 AM
Cc: Ashish Dalela (adalela); Pedro Marques; [email-masked];
[email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Is the issue that needs to be resolved for overlays only regarding the
number of ARPs that need to be managed by the gateway?  Could this issue
not be resolved operationally by say having more gateways?  

What other issues exist and need to be resolved for overlays, besides
deciding on a tunneling encapsulation?  Is the IETF expected to
standardize control protocols for overlays?

There is clearly a need for server-based virtual networks as well as a
need for scalable network-based Ethernet virtual networks for DC.
Shouldn't these be separate conversations?

---

From: Brad Hedlund <[email-masked]>
Date: Fri, 6 Jan 2012 11:58:30 -0800

"So, if you have one tenant running OSPF another one running IS-IS and yet another running BGP, and they want to keep playing with their network configuration, it makes sense to run these in the overlay or the VM mode. "
This didn't make much sense to me. Why would a customer of an overal model be managing or configuring a routing protocol?  The overlay simplifies the customer's topology view to that of a single logical segment.

Cheers,
Brad

________________________________
From: [email-masked] [[email-masked]] On Behalf Of Ashish Dalela (adalela) [[email-masked]]
Sent: Thursday, January 05, 2012 11:03 PM
To: Aldrin Isaac
Cc: Pedro Marques; [email-masked]; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center interconnect

Aldrin,

I like the way you say – “the only thing the network needs to know” - :-) The same could be said of the hypervisor as well :-)

The problem is that once you start pushing the intelligence into the server stack, you will have to keep pushing more and more  - e.g. firewalls, multicast, broadcast, packet inspection, flow mgmt, etc. You will find that you are re-inventing on the overlay everything that exists on the underlay. The challenges are stacked up against the overlay, not the underlay.

You will also find that you need to communicate between the overlay and the underlay to get the desired bandwidth, QoS, flow mgmt, multipathing, tree optimization, and that is never going to be easy. You will also find that hardware accelerated networks perform better, deliver the requisite high-availability, and consume less energy, as compared to doing the same thing in the hypervisor.

Having said that, I recognize there are two models of virtualization that we know of. The “overlay” model is like a network hypervisor in which individual customers are like VMs. The “multiplexed” model is like a multi-user OS. Both multi-user OS and the multi-VM hypervisor are isolated equally well. But they have different use-cases.

The main use-case for the VM hypervisor model is to multiplex multiple different OS/tools/application “environments” into the same HW, letting each VM the ability to shut itself down, reboot or whatever it wants to do. In case of network, that “environment” is a set of protocols. So, if you have one tenant running OSPF another one running IS-IS and yet another running BGP, and they want to keep playing with their network configuration, it makes sense to run these in the overlay or the VM mode.

If everyone has the same environment (i.e. runs the same protocols, and expects common controls), it makes more sense to run them in the multi-user (multiplexed) mode rather than multi-VM (overlay) mode. The multi-VM model delegates administration to the VM owner. The multi-user model owns the administration while letting a “tenant” use the network. This is a conscious choice – is cloud going to open up the configuration of network like they open up the administration of VM? From what I know, the answer is “no”.

These architectural models exist outside the networking domain, and I would reach out into another domain to borrow the intuitions, use-cases and challenges. Architecturally, you will find the same challenges in managing the overlay model as exist in the multi-VM model. If there are no benefits to be gained from that, you are better off in the multiplexed mode.

But then again – beauty lies in the eyes of the beholder.

Thanks, Ashish

From: Aldrin Isaac [mailto:[email-masked]]
Sent: Friday, January 06, 2012 9:07 AM
Cc: Ashish Dalela (adalela); Pedro Marques; [email-masked]; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center interconnect

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 6 Jan 2012 20:23:48 -0500

Hi Ashish,

The first three sentences are by Warren Kumari -- I seem to have lost the leading ">" quote character in my response.  

My questions to the folks on this list are:

Is the issue that needs to be resolved for server-based overlays only regarding the number of ARPs that need to be managed by the gateway?  Could this issue not be resolved operationally by say having more gateways?  
 
What other issues exist and need to be resolved for server-based overlays, besides deciding on a tunneling encapsulation?  Is the IETF expected to standardize control protocols for server-based overlays?
 
There is clearly a need for server-oriented virtual networks as well as a need for scalable network-oriented Ethernet virtual networks for DC.  Shouldn't these be separate conversations?

Cheers -- aldrin

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Sat, 7 Jan 2012 19:36:17 +0530

Brad,

I'm not saying it makes a sense, but here are some use-cases. One of the
things that SDN proposed as part of network virtualization was the
ability to run multiple protocols in the same network. Their goals were
experimental, but there could be other goals as well. E.g. if a cloud
customer is running OSPF in their private network but the cloud provider
was running IS-IS, an overlay could (hypothetically) want to run OSPF in
their "segment". I'm not saying it makes a lot of sense, but that is an
example of when the "overlay" VM style model of the network makes sense
for virtualization. Other overlay models IMO are trying to fix basic
issues with IP/Ethernet - namely lack of location awareness - which can
be solved in other ways without an overlay. So, if the only reason to
run an overlay is to solve the location independence, then that's not
justification enough. But if there are other reasons, that make sense to
me. I haven't heard any other reasons.

Thanks, Ashish

From: Brad Hedlund [mailto:[email-masked]] 
Sent: Saturday, January 07, 2012 1:29 AM
To: Ashish Dalela (adalela); Aldrin Isaac
Cc: Pedro Marques; [email-masked]; [email-masked]
Subject: RE: [dc] [armd] IP over IP solution for data center
interconnect

	"So, if you have one tenant running OSPF another one running
IS-IS and yet another running BGP, and they want to keep playing with
their network configuration, it makes sense to run these in the overlay
or the VM mode. "

This didn't make much sense to me. Why would a customer of an overal
model be managing or configuring a routing protocol?  The overlay
simplifies the customer's topology view to that of a single logical
segment.

Cheers,

Brad

From: [email-masked] [[email-masked]] On Behalf Of Ashish
Dalela (adalela) [[email-masked]]
Sent: Thursday, January 05, 2012 11:03 PM
To: Aldrin Isaac
Cc: Pedro Marques; [email-masked]; [email-masked]
Subject: Re: [dc] [armd] IP over IP solution for data center
interconnect

Aldrin,

Thanks, Ashish