Subject: Re: [dc] new drafts

From: Aldrin Isaac <[email-masked]>
Date: Mon, 2 Jan 2012 11:23:17 -0500

Hi Ashish, 

Hierarchical MAC should be considered for addition to the Ethernet toolkit for "legacy" Ethernet.  However, I would want to see it combined with end-station and uPE UNI that allows for *network-assigned* MAC/MAC-prefix, and also support for MAC "default route" announcement.  As you advocate, an ARP registry could further enhance scaling for ARP (if there are no show stopping limitations), but hierarchical MAC and ARP registry should operate independently.  Support for virtual network contexts would be required for both UNI and registry.  I am a big proponent of VRFs and virtual LANs so a solution that takes these off the table would be off my table too.

Best -- aldrin

On Dec 30, 2011, at 8:54 PM, Ashish Dalela (adalela) wrote:

> Hi Aldrin,
>  
> >> How would you address gratuitous ARP when using the hierarchical MAC addressing with a registry to store MAC-IP bindings?
>  
> We should ideally have a protocol that maps ARP messages into the registry and vice versa. Why new protocol? Greater security and reliability plus avoiding ARP broadcasts. Gratuitous ARP doesn’t have an acknowledge, so if you want to perform some actions based on this, you need to get an “acknowledge” for it. Gratuitous ARP can get dropped.
>  
> The other issue is that a user can misuse it do MAC hijacking. But if you have a hierarchical MAC you can’t hijack because network knows your location. Things like dot1x which solved these issues in the campus space are not used in datacenter.
>  
> >> Also, when using MAC prefixes in complex L2VPN topologies how could we address the risk of a hub forwarding into the wrong spoke context on the leaf switch?  Each egress port on the leaf switch may be in a different context (ex: EVI in EVPN).
>  
> We should avoid building any kind of VRF at the control plane, because that’s just not scalable. With hierarchical MAC, you have global L2 addressing so from a forwarding perspective it looks like static L3 routing and anyone can send packets anywhere. When a VM moves, this doesn’t change the global forwarding table (unless a virtual switch moves – which is also possible). To segment customers we tag packets with some tenant id, and it will be dropped at destination. The tags will be present both at ingress and egress ports.
>  
> There are only two possible approaches – drop at source or drop at destination. If we try to drop at source, we need a mapping to which destinations are allowed (too many, and the scale is unpredictable). If we drop at destination, we still need to know which destinations are allowed, but that’s just comprised of local host entries (no new entries).
>  
> I’m not totally certain if I answered your question, so let me know.
>  
> Thanks, Ashish
>  
> From: Aldrin Isaac [mailto:[email-masked]] 
> Sent: Saturday, December 31, 2011 4:07 AM
> To: Ashish Dalela (adalela)
> Cc: [email-masked]
> Subject: Re: [dc] new drafts
>  
> Hi Ashish,
>  
> How would you address gratuitous ARP when using the hierarchical MAC addressing with a registry to store MAC-IP bindings?
>  
> Also, when using MAC prefixes in complex L2VPN topologies how could we address the risk of a hub forwarding into the wrong spoke context on the leaf switch?  Each egress port on the leaf switch may be in a different context (ex: EVI in EVPN).
>  
> Thanks -- aldrin
>  
>  
> On Dec 30, 2011, at 6:00 AM, Ashish Dalela (adalela) wrote:
> 
> 
>  
> Folks,
>  
> I have posted 2 drafts for this group to review.
>  
> http://www.ietf.org/id/draft-dalela-dc-requirements-00.txt
> http://www.ietf.org/id/draft-dalela-dc-approaches-00.txt
>  
> The first draft captures 10 requirements / problems to be addressed in the datacenter space. These were described over the email earlier.
>  
> The second draft discusses various approaches to addressing these requirements. This draft analyzes the scaling properties of various approaches and makes recommendations at the end for future work that can be taken by this group.
>  
> Request your feedback and discussion.
>  
> Thanks, Ashish
> _______________________________________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dc

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Mon, 2 Jan 2012 22:19:50 +0530

Hi Aldrin,

I am 100% in agreement with everything you said below. What I meant by
VRF below was dynamically pushing host routes with every VM
creation/deletion over the control plane. That will cause convergence
issues and lead to instability. With large number of tenants, and
something or other always going on, we don't want the control plane in
flux always. We need a VRF-like segment that separates tenants (without
overlapping with VLANs because users need VLANs too) without loading the
control plane. 

I was trying to nail down this "approach" through the draft, by showing
that this is an approach that addresses all the problems in the most
scalable way. We can do L2 (Ethernet) or L3 (TCP/IP) using a common
forwarding plane, and that we remain compatible with legacy L2/L3
networks - they will always exist.

Support/feedback to this approach probably needs to be discussed and
agreed upon. Solutions based on this are not very hard to achieve (I can
bring the proposals).

Thanks, Ashish

From: Aldrin Isaac [mailto:[email-masked]] 
Sent: Monday, January 02, 2012 9:53 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: Re: [dc] new drafts

Hi Ashish, 

Hierarchical MAC should be considered for addition to the Ethernet
toolkit for "legacy" Ethernet.  However, I would want to see it combined
with end-station and uPE UNI that allows for *network-assigned*
MAC/MAC-prefix, and also support for MAC "default route" announcement.
As you advocate, an ARP registry could further enhance scaling for ARP
(if there are no show stopping limitations), but hierarchical MAC and
ARP registry should operate independently.  Support for virtual network
contexts would be required for both UNI and registry.  I am a big
proponent of VRFs and virtual LANs so a solution that takes these off
the table would be off my table too.

Best -- aldrin

Hi Aldrin,

>> How would you address gratuitous ARP when using the hierarchical MAC
addressing with a registry to store MAC-IP bindings?

We should ideally have a protocol that maps ARP messages into the
registry and vice versa. Why new protocol? Greater security and
reliability plus avoiding ARP broadcasts. Gratuitous ARP doesn't have an
acknowledge, so if you want to perform some actions based on this, you
need to get an "acknowledge" for it. Gratuitous ARP can get dropped.

The other issue is that a user can misuse it do MAC hijacking. But if
you have a hierarchical MAC you can't hijack because network knows your
location. Things like dot1x which solved these issues in the campus
space are not used in datacenter.

>> Also, when using MAC prefixes in complex L2VPN topologies how could
we address the risk of a hub forwarding into the wrong spoke context on
the leaf switch?  Each egress port on the leaf switch may be in a
different context (ex: EVI in EVPN).

We should avoid building any kind of VRF at the control plane, because
that's just not scalable. With hierarchical MAC, you have global L2
addressing so from a forwarding perspective it looks like static L3
routing and anyone can send packets anywhere. When a VM moves, this
doesn't change the global forwarding table (unless a virtual switch
moves - which is also possible). To segment customers we tag packets
with some tenant id, and it will be dropped at destination. The tags
will be present both at ingress and egress ports.

There are only two possible approaches - drop at source or drop at
destination. If we try to drop at source, we need a mapping to which
destinations are allowed (too many, and the scale is unpredictable). If
we drop at destination, we still need to know which destinations are
allowed, but that's just comprised of local host entries (no new
entries).

I'm not totally certain if I answered your question, so let me know.

Thanks, Ashish

Hi Ashish,

How would you address gratuitous ARP when using the hierarchical MAC
addressing with a registry to store MAC-IP bindings?

Also, when using MAC prefixes in complex L2VPN topologies how could we
address the risk of a hub forwarding into the wrong spoke context on the
leaf switch?  Each egress port on the leaf switch may be in a different
context (ex: EVI in EVPN).

Thanks -- aldrin

 

Folks,

The first draft captures 10 requirements / problems to be addressed in
the datacenter space. These were described over the email earlier.

The second draft discusses various approaches to addressing these
requirements. This draft analyzes the scaling properties of various
approaches and makes recommendations at the end for future work that can
be taken by this group.

Thanks, Ashish

---

From: Yakov Rekhter <[email-masked]>
Date: Tue, 3 Jan 2012 07:10:11 -0800

Ashish,

> need to get an "acknowledge" for it. Gratuitous ARP can get dropped.=20
> 
> space are not used in datacenter.=20
> 
> that's just not scalable. 

Would you care to provide detailed technical analysis to support your
claim about VRFs "just not scalable", or should this claim be treated 
as proof by emphatic assertion ?

Yakov.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 21:14:31 +0530

Yakov,

>> Would you care to provide detailed technical analysis to support your
claim about VRFs "just not scalable",

VM mobility requires insertion of host-routes. As VMs move and/or are
created/deleted, these host routes need to be propagated everywhere.
That's a control plane scaling problem, plus a problem about
convergence.

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Yakov Rekhter
Sent: Tuesday, January 03, 2012 8:40 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]; Aldrin Isaac
Subject: Re: [dc] new drafts

Ashish,

Yakov.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 16:53:54 +0100

Ashish,

VM mobility based on re-propagating host routes even with your own 
domain looks to me like broken .. VM mobility timing requirements are in 
milliseconds or 10s of ms .. routing distribution or even FIB update 
with host routes (assuming that we could propagate host routes ahead of 
migration then just trigger the activation) would be in 100s of ms or 
seconds depending on the scope. I agree with your assertion that this is 
not scalable.

However this has nothing to do with stating that VRFs are "just not 
scalable". VRF instantiation or VRF scaling is very opaque topic. 
Especially one could claim that VRFs in control plane are a very 
scalable tools we have at our disposal today.

Best,
R.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 22:09:17 +0530

Robert,

Here are some things to evaluate scalability against.

Assume a simple case that under a switch there are 250 VM, split amongst
10 customers. Each customer has a unique VRF. Normally, we would have
advertized a /24 route for that switch. In this case your routes to a
single switch are segmented and there are 10 VRFs, and you will very
likely have 250 route table entries total segmented by VRF-ids. That's a
routing table bloat from 1 entry to 250 entry. This happens everywhere.
I have assumed a public IP addressing, but the same thing will happen
for the private addressing as well.

Then, typically the number of VRFs you can support on a router is about
4K. These # of VRFs have to be supported at the access, so you have to
assume this is the limit from the access viewpoint. 4K is nothing - we
have 4K VLANs today to segment and that's nothing. Every segmentation
technique being talked about speaks of a million plus segments. Take
that to VRFs, you need a million VRFs on the control plane at the access
switch. Another problem with a VRF is that it will get and store a route
for a host, even when there is no host talking to it. With dynamic
learning or learning based on packet arrival you avoid these host routes
and limit them to active conversations only. That's a huge saving
because not every host talks to every host.

Then, at massive scale, the failure rates are also massive. At 5 nines
reliability, a hardware entity out of 100,000 will fail every 5.25
minutes. Access switches don't have high availability. Software fails
even faster - OS is generally 4 9's, which means one out of 10,000 fails
every 5.25 minutes. At millions of instances of such entities, there are
rapid failures happening. You have to only look at massive datacenters
today run by Web 2.0 companies, and they all echo this view. They
basically form clusters of the same application. Software moves the
workload from one cluster to another. The whole cluster can fail over.
That's not what you do in a consumer cloud, where you have to recover.
At massive failure rates, and rapid recovery rates, you are moving
things around and injecting host routes for reachability. It's a
convergence problem, especially with link-state algorithms.

If the VM can be moved, then all you need to do is install a temporary
redirect of packets to the new location. Each host will refresh the MAC
after 15-30 seconds. If the packets are redirected from old to new
location for these 30 seconds, the redirect can be aged automatically.
This happens all the time in mobile networks in what is called a "fast
handoff" where you redirect the packets until handoff is completed.

Thanks, Ashish

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 9:24 PM
To: Ashish Dalela (adalela)
Cc: Yakov Rekhter; Aldrin Isaac; [email-masked]
Subject: Re: [dc] new drafts

Ashish,

not scalable.

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 03 Jan 2012 17:55:58 +0100

Ashish,

> for the private addressing as well.

Normally in this case I would have 10 routes + 1 route for switch 
loopback not 250. However when those VMs will start moving between PEs 
you are right that worse case one could end up would be 250 non 
aggregetable routes per VRF.

First I don't think this is a problem scaling wise today as I would not 
assume that everyone will be moving.

Second we know today how to handle millions of routes in BGP.

Third I am not saying that this model should be used.

I am advocating that a hierarchical IP in IP model should be used. VRFs 
on those PEs could be used for isolation purposes. And any VM move needs 
to be only reflected in mapping plane and not in routing infrastructure 
of the network.

> assume this is the limit from the access viewpoint.

Nope ... Control plane VRFs have no bounds for limits. 4K comes from 
platform limitations.

Hint: Think about control plane and data plane separation. Pedro's draft 
already provides an example on how such separation can be accomplished.

> convergence problem, especially with link-state algorithms.

Not applicable to what I am advocating.

> redirect of packets to the new location.

What is this redirection ? Where do you install it ? In all switching 
elements of the network ? Redirection works when you encapsulate. 
Without any encapsulation how do you redirect by just touching a single 
network element ?

> Each host will refresh the MAC

Hmmmm interesting. We even have a draft which can be used for such 
redirection today ... draft-rekhter-l3vpn-virtual-hub

Cheers,
R.

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Tue, 3 Jan 2012 22:49:32 +0530

Robert,

I think you and I talking past each other.

I'm talking about datacenter switching and not BGP, PE, etc. The context
is scaling of access switches also called Top of Rack or ToR. 
So, your comment about millions of routes and BGP assumes a core router,
not an access switch. 

Reset context.

Thanks, Ashish

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 10:26 PM
To: Ashish Dalela (adalela)
Cc: Yakov Rekhter; [email-masked]; Aldrin Isaac
Subject: Re: [dc] new drafts

Ashish,

Cheers,
R.

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 3 Jan 2012 23:56:47 +0000

Ashish, 

I really like what you stated "articulating the problem itself therefore has become a challenge" in your "draft-dalela-dc-requirements-00". In my opinion, articulating which problem is IETF problem is probably even harder, given all the drafts flowing to IETF DC Interim meeting. 

Can you elaborate more on why "The Mobile IP approach does not handle data center mobility optimally"? Every smart phone has an IP address. Today's mobile IP works (even though not optimal). 

Your section 4.1 has very similar description as in "ARMD problem statement". When network is fragmented, you either have to enable all subnets on all network links or re-configure addresses on all switches/routers whenever there is any network fragmentation change. Then not only gateway routers have to deal with traffic on all subnets (which is ARMD problem), but all links have to carry broadcast traffic from all subnets (which is bandwidth utilization issue). 

Your section 4.3 "Data is relatively Immobile" assumes that VMs can move anywhere without considering where needed data storage is. It is not true with many VM management systems. Many VM/server management systems' algorithms do take in consideration on where the needed data storage is (and many other attributes) in determining where Application can be hosted. 

Section 5.1: broadcast is not the reason that L2 can't scale. It is "fragmented network can't scale". The main reason that L2 can't scale is that L2's addresses can't be aggregated. 

Linda Dunbar

> Ashish Dalela (adalela)
> Sent: Tuesday, January 03, 2012 11:20 AM
> To: [email-masked]
> > likely have 250 route table entries total segmented by VRF-ids.
> That's
> a

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Wed, 4 Jan 2012 06:36:03 +0530

Linda, 

Please see inline.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Wednesday, January 04, 2012 5:27 AM
To: Ashish Dalela (adalela)
Cc: [email-masked]
Subject: [dc] comments to your new draft

Ashish, 

I really like what you stated "articulating the problem itself therefore
has become a challenge" in your "draft-dalela-dc-requirements-00". In my
opinion, articulating which problem is IETF problem is probably even
harder, given all the drafts flowing to IETF DC Interim meeting. 

[AD] Thanks. Except for the Network SLA problem, there is already work
going on in IETF somewhere or the other. So, they are recognized to be
all IETF problems. The question now is - is there a "uber" problem
statement or there are many "small" problem statements? Correspondingly,
does not work of these workgroups needs to be pursued individually and
independently or coordinated?

Can you elaborate more on why "The Mobile IP approach does not handle
data center mobility optimally"? Every smart phone has an IP address.
Today's mobile IP works (even though not optimal). 

[AD] Datacenter needs shortest path and multiple paths. Mobile IP does
not give either. 

Your section 4.1 has very similar description as in "ARMD problem
statement". When network is fragmented, you either have to enable all
subnets on all network links or re-configure addresses on all
switches/routers whenever there is any network fragmentation change.
Then not only gateway routers have to deal with traffic on all subnets
(which is ARMD problem), but all links have to carry broadcast traffic
from all subnets (which is bandwidth utilization issue). 

[AD] Agree. I was trying to elaborate on use-cases that lead to
fragmentation.

Your section 4.3 "Data is relatively Immobile" assumes that VMs can move
anywhere without considering where needed data storage is. It is not
true with many VM management systems. Many VM/server management systems'
algorithms do take in consideration on where the needed data storage is
(and many other attributes) in determining where Application can be
hosted. 

[AD] In DC contexts there are discussions about elephant and mouse
flows. What I was trying to say (but not said it clearly perhaps) that
data immobility leads to elephant flows. These flows can't be toggled to
avoid packet reordering. This section needs to be reworded as I realize
now. Thanks.
 
Section 5.1: broadcast is not the reason that L2 can't scale. It is
"fragmented network can't scale". The main reason that L2 can't scale is
that L2's addresses can't be aggregated. 

[AD] Yes, it's both. At the control plane it is broadcasting and the
forwarding plane it is lack of aggregation. Will update this. Thanks.

Linda Dunbar