Subject: Re: [dc] new drafts

From: Lizhong Jin<[email-masked]>
Date: Wed, 4 Jan 2012 22:58:25 +0800

Hi Ashish,
If we implement VRF on access switch (or ToR), I agree there will be 
scalability problem. Also the cost will also be an issue, the access 
switch would be more expensive than before. How about implement VRF on 
aggregation router? If the aggregation router could solve scalability and 
high availability problem, then we should focus on how to setup connection 
between VM and aggregation router. Hope to see your comments.

Thanks
Lizhong

> 
> -----From "Ashish Dalela (adalela)" <[email-masked]> 
> Tue, 3 Jan 2012 22:09:17 +0530 -----
> 
> Receiver:
> 
> <[email-masked]>
> 
> cc:
> 
> Yakov Rekhter <[email-masked]>, [email-masked], Aldrin Isaac 
> <[email-masked]>
> 
> Subject:
> 
> Re: [dc] new drafts
> 
> Robert,
> 
> Here are some things to evaluate scalability against.
> 
> Assume a simple case that under a switch there are 250 VM, split amongst
> 10 customers. Each customer has a unique VRF. Normally, we would have
> advertized a /24 route for that switch. In this case your routes to a
> single switch are segmented and there are 10 VRFs, and you will very
> likely have 250 route table entries total segmented by VRF-ids. That's a
> routing table bloat from 1 entry to 250 entry. This happens everywhere.
> I have assumed a public IP addressing, but the same thing will happen
> for the private addressing as well.
> 
> Then, typically the number of VRFs you can support on a router is about
> 4K. These # of VRFs have to be supported at the access, so you have to
> assume this is the limit from the access viewpoint. 4K is nothing - we
> have 4K VLANs today to segment and that's nothing. Every segmentation
> technique being talked about speaks of a million plus segments. Take
> that to VRFs, you need a million VRFs on the control plane at the access
> switch. Another problem with a VRF is that it will get and store a route
> for a host, even when there is no host talking to it. With dynamic
> learning or learning based on packet arrival you avoid these host routes
> and limit them to active conversations only. That's a huge saving
> because not every host talks to every host.
> 
> Then, at massive scale, the failure rates are also massive. At 5 nines
> reliability, a hardware entity out of 100,000 will fail every 5.25
> minutes. Access switches don't have high availability. Software fails
> even faster - OS is generally 4 9's, which means one out of 10,000 fails
> every 5.25 minutes. At millions of instances of such entities, there are
> rapid failures happening. You have to only look at massive datacenters
> today run by Web 2.0 companies, and they all echo this view. They
> basically form clusters of the same application. Software moves the
> workload from one cluster to another. The whole cluster can fail over.
> That's not what you do in a consumer cloud, where you have to recover.
> At massive failure rates, and rapid recovery rates, you are moving
> things around and injecting host routes for reachability. It's a
> convergence problem, especially with link-state algorithms.
> 
> If the VM can be moved, then all you need to do is install a temporary
> redirect of packets to the new location. Each host will refresh the MAC
> after 15-30 seconds. If the packets are redirected from old to new
> location for these 30 seconds, the redirect can be aged automatically.
> This happens all the time in mobile networks in what is called a "fast
> handoff" where you redirect the packets until handoff is completed.
> 
> Thanks, Ashish
> 
> 

--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail is solely property of the sender's organization. This mail communication is confidential. Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the originator of the message. Any views expressed in this message are those of the individual sender.
This message has been scanned for viruses and Spam by ZTE Anti-Spam system.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 04 Jan 2012 16:17:17 +0100

Hi Lizhong,

How about neither ?

How about implementing VRF for control plane separation on the x86 
controller out of data plane and simply instructing either host hosting 
VMs or TOR or Access Switch to forward/encapsulate the packets correctly ?

Cheers,
R.

> aggregation router? If the aggregation router could solve scalability
> and high availability problem, then we should focus on how to setup
> connection between VM and aggregation router. Hope to see your comments.
>
> Thanks
> Lizhong

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Wed, 4 Jan 2012 09:53:07 -0600

This is an idea well worth investigating, in addition to compare the
options VRF on X in discussion.
Luyuan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Raszuk
> Sent: Wednesday, January 04, 2012 10:17 AM
> To: Lizhong Jin
> Cc: [email-masked]; [email-masked]; Ashish Dalela (adalela);
> [email-masked]
> Subject: Re: [dc] new drafts
> 
> Hi Lizhong,
> 
> How about neither ?
> 
> _______________________________________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dc

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Wed, 4 Jan 2012 21:46:34 +0530

Hi Lizhong,

When you do things at the aggregation you need two sets of encaps -
Agg-to-Agg and Agg-to-Access. 

If there are N hosts under an access, and M accesses under an Agg, and P
Aggs, then you need - 

N * M for access to agg

(P-1) * N * M for agg to agg

Total = N * M + (P-1) * N * M = N * M * P

This is the worst case.

For a better case, assume each VM talks to 25 VM outside its Agg.

The total = N * M * 25

Let's plug some numbers into this.

For a 48 port access, 50 VM per port, each with 2 VIF, N = 48 * 50 * 2 =
4800

For an agg with 100 ports (50 down and 50 up), M = 50

A million VM require 1,000,000 / (48 * 50) = 415 accesses.

Each access connects to 4 Aggs, so you need 415 * 4 / 50 = 33 Aggs = P.

Worst case total entries needed at each Agg = 4800 * 50 * 33 = 7,920,000

Better case total entries needed at each Agg = 4800 * 50 * 25 =
6,000,000

You may be wondering why for 1M VM, we need 6-8M host routes? It's
because of multi-pathing. A destination can be reached through many
paths, so you put routes for all paths.

When you have 6-8M routes, you can imagine the corresponding control
plane load.

The additional complexity is that to send packet from A to B, you need 3
encaps - access-to-agg, agg-to-agg and agg-to-access. When a VM moves,
you have to coordinate all these entries. That's another not so easy
problem.

Thanks, Ashish

From: Lizhong Jin [mailto:[email-masked]] 
Sent: Wednesday, January 04, 2012 8:28 PM
To: Ashish Dalela (adalela)
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]
Subject: Re: [dc] new drafts

Hi Ashish, 

Thanks 
Lizhong 

--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail
is solely property of the sender's organization. This mail communication
is confidential. Recipients named above are obligated to maintain
secrecy and are not permitted to disclose the contents of this
communication to others.
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the
originator of the message. Any views expressed in this message are those
of the individual sender.
This message has been scanned for viruses and Spam by ZTE Anti-Spam
system.

---

From: Lizhong Jin<[email-masked]>
Date: Thu, 5 Jan 2012 23:26:06 +0800

Hi Ashish,
Thank you for the analysis. The number of host routes in your example 
reaches 2M, becasue you assume each VM with two VIF. If each route perform 
4 path for ECMP, the total route will reach 8M as you described. This is 
the worst case, and at least, the scalability on aggregation router would 
be better than on access. And I agree, the connection between access and 
aggregation should be coordinated when VM moving, which is not easy to 
solve.

Regards
Lizhong

"Ashish Dalela (adalela)" <[email-masked]> wrote 2012/01/05 00:16:34:

> Hi Lizhong,
> 
> When you do things at the aggregation you need two sets of encaps – 
> If there are N hosts under an access, and M accesses under an Agg, 
> and P Aggs, then you need – 
> 
> N * M for access to agg
> Let’s plug some numbers into this.
> 
4800
6,000,000
> 
> You may be wondering why for 1M VM, we need 6-8M host routes? It’s 
> The additional complexity is that to send packet from A to B, you 
> need 3 encaps – access-to-agg, agg-to-agg and agg-to-access. When a 
> VM moves, you have to coordinate all these entries. That’s another 
> not so easy problem.
> 
> Thanks, Ashish
> 
> 
[email-masked]
> switch would be more expensive than before. How about implement VRF 
> on aggregation router? If the aggregation router could solve 
> scalability and high availability problem, then we should focus on 
> how to setup connection between VM and aggregation router. Hope to 
> see your comments. 
> 
> Thanks 
> Lizhong 
> 
> 
> >

---

From: Lizhong Jin<[email-masked]>
Date: Thu, 5 Jan 2012 23:34:49 +0800

Hi Robert,
I think this is also an option. When separating control plane and 
dataplane, do you mean to use openFlow to implement the protocol between 
control plane and dataplane? I have some concern with route convergence 
when implementing VRF on x86.

Regards
Lizhong

---

From: Michael Hammer <[email-masked]>
Date: Thu, 5 Jan 2012 11:17:12 -0500

It is good to consider worse cases when planning architectures.  When you
design only for best cases, it all collapses when the worse cases occur.
 So, it might help to identify those hidden biases that rely on certain
assumptions of what might or might not occur.  Often it seems like only one
cloud provider is assumed.

I would prefer to see a design that addresses:
- security as a first consideration, not bolted on afterwards,
- clear separation between customer and cloud provider mechanisms, such
that the actions of each are orthogonal.

That means that any mechanism used should clearly define how customer and
cloud provider use of such mechanism is isolated.
And if not isolated, where in the architecture does the customer or cloud
provider monitor/manage the cross-domain traffic.

It might also be good to identify which mechanisms depend on certain
assumptions, and recommend that they not be used when those assumptions no
longer hold.  The poster child for this would seem to be ARP, which seems
to assume a rather static LAN, which in DC is not the case.  That might
suggest that another means of learning mappings is needed.

Lesson here is that continuing to use every existing mechanism may need to
be questioned and perhaps discouraged, i.e. abandon the cruft.
So, the task at hand may not be solely identifying new elements to pile
onto the toolkit.

Mike

2012/1/5 Lizhong Jin <[email-masked]>

> > Better case total entries needed at each Agg = 4800 * 50 * 25 = 6,000,000
> >

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 05 Jan 2012 18:59:13 +0100

Openflow could be one option. The other option could be XMPP as example.

 > I have some concern with route convergence when implementing VRF on
 > x86.

Hmmm you mean that it is going to be faster then any network gear could 
support ;) ? Then just program directly hypervisors ....

Rgs,
R.