Subject: Re: [dc] Elevator Pitch

From: Yakov Rekhter <[email-masked]>
Date: Tue, 3 Jan 2012 06:32:43 -0800

Xuxiaohu,

> Hi all,
> 
> Since there are some differences in the problems and requirements
> between data center network (DCN) and data center interconnect
> (DCI), I try to list several problems and requirements for DCN and
> DCI separately as follows. Here the dat a centers mainly refer to
> those multi-tenant data centers which are operated by public cloud
> providers to deliver cloud service (i.e., IaaS) to their customers
> (i.e., tenants).
> 
> 1. DCN problems and requirements:
> 
> 1) VM mobility across multiple pods -> LAN/subnet extension across pods
> 
> 2) Some cluster applications use non-IP or link-local multicast (optional) ->
>     Layer2 networking
> 
> 3) Multi-tenancy isolation -> VPN/VLAN instance scalability
> 
> 4) Millions of VMs -> MAC/IP forwarding table scalability
> 
> 5) Increasing bandwidth demands for server-to-server connectivity 
>    (i.e., east-west traffic)-> ECMP and shortest path forwarding capabilities
> 
> 6) Network resiliency -> Fast convergence and multi-homing

Do you need fast routing convergence, or fast connectivity restoration ?

> 7) Thousands of network devices -> Simplified provisioning and operation
> 
> 
> 
> 2. DCI problems and requirements:
> 
> 1) VMs mobility across data centers -> LAN/subnet extension across 
>    data centers.
> 
> 2) Multi-tenancy isolation -> VLAN/VPN instance scalability
> 
> 3) Millions of VMs -> MAC/IP forwarding table scalability
> 
> 4) Optimal utilization of WAN bandwidth resource -> Unknown unicast 
>    and ARP broadcast suppression
> 
> 5) Network resiliency -> Fast convergence and multi-homing

Do you need fast routing convergence, or fast connectivity restoration ?

> 6) Load-balancing across data centers -> Active-active DC exits
> 
> 7) Suboptimal path caused by LAN/subnet extension across data center ->
>    Path optimization for both VPN access and Internet access

Yakov.

---

From: Yakov Rekhter <[email-masked]>
Date: Tue, 3 Jan 2012 06:50:20 -0800

Russ,

[clipped...]
 
> The issue of convergence presents another problem to think about... If
> convergence ends up being slower than the application timing out and
> searching for a new destination IP address, then why is switching IP
> addresses worse? The only way any of this makes sense is if it converges
> faster than a human would notice --and, increasingly, faster than a
> computer would notice.

I think we need to distinguish between routing convergence and
connectivity restoration time, as the two are not always the same.
With this in mind let me paraphrase what you said above as follows:

 The issue of connectivity restoration time presents another problem
 to think about... If this time ends up being longer than the
 application timing out and searching for a new destination IP
 address, then why is switching IP addresses worse? The only way
 any of this makes sense is if this time is faster than a human
 would notice --and, increasingly, faster than a computer would
 notice.

Yakov.

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 3 Jan 2012 07:53:25 -0800

On 31Dec2011, at 06.42, Derick Winkworth wrote:

> 100,000 VMs is how many physical servers?  At some point you have to
> eat the cost of putting the right gear in.  You can't drive down to
> Best Buy to buy your switches if you think you'll have 100k host route
> entries (or MAC addresses) in your access switches.
> 
> Russ is right, there are multiple platforms/vendors now supporting
> millions of routes/MAC addresses.  Maybe what you thought of as a
> distribution or core device is now and end-of-row "access" device?
> 
> Which kind of leads me to a point I wanted to make with regard to some
> of the lists of "requirements" I've seen.  Some of these things should
> be considered a network design problem, not an IETF protocol problem.
> Specifically, I'm thinking of hyperscale requirements...

I agree - there is a constellation of potential problem spaces, in that is a subset of real operational problems, and within that is a smaller subset of problems that the IETF should address.  The larger problem space is, among other things, network and system design problems, including scale, performance, economic, etc.  
> 
> 
> 
> On Sat, Dec 31, 2011 at 8:32 AM, Ashish Dalela (adalela)
> <[email-masked]> wrote:
>> 
>> 100K or whatever that number is, is a big deal at the access. And that number will increase with time. The thing to bear in mind is that these are ADDITIONAL entries, not in lieu of other entries. E.g. network routes, local host-port entries, ACLs etc exist today and will continue to exist. Host routes are in addition to that. Access switches generally had about 16K or 32K table sizes. And we are talking about adding 3-4 times more to that.
>> 
>> The other thing is that access to core ratio is about 1000:1. So, we need to multiply the incremental costs by 1000 to get the full impact to TCO.
>> 
>> In regard to scaling, there are 4 places to look at:
>> 
>> 1. Access
>> 2. Core
>> 3. Inter-datacenter
>> 4. Datacenter-internet
>> 
>> These have different scaling properties.
>> 
>> If you have different DC and DCI technologies, and you use Encap, you have high entries at Access, DCI and Datacenter-Internet.
>> 
>> If you have common DC and DCI technologies, you have high Access and Datacenter-Internet entries.
>> 
>> If you have hierarchical MAC, then you have only high in Datacenter-Internet boundary.
>> 
>> Net-net, out of the 4 things above, you can solve 3 (first 3). The fourth we have to distribute over multiple routers, which is not hard because it is always north-south traffic.
>> 
>> Thanks, Ashish
>> 
>> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Christopher LILJENSTOLPE
>> Sent: Thursday, December 29, 2011 10:58 PM
>> To: Russ White
>> Cc: [email-masked]
>> Subject: Re: [dc] Elevator Pitch
>> 
>> Greetings Russ et.al.,
>> 
>> On 29Dec2011, at 07.08, Russ White wrote:
>> 
>>> 
>>>> Actually, 100K host routes in leaf nodes may well be a Big Deal. It
>>>> depends on what type of devices need to understand host routes.
>>> 
>>> So let me try to abstract the problem a little... There are two things
>>> that cost money in building a network:
>>> 
>>> 1. State --the more you carry, the more hardware is going to cost.
>>> 2. Bandwidth --the less optimal your bandwidth utilization, the more
>>> you're spending on something that's not used.
>>> 
>>> The first rule of all network design is that the more state you put in a
>>> network, the more optimal your bandwidth usage. To reduce the cost of
>>> bandwidth, you have to spend money on state. To reduce the cost of
>>> state, you must increase your spending on bandwidth.
>> 
>> Agreed, mostly.  The question is where the state resides, control plane or data plane (do you differentiate between possible state and active state).  Control plane "possible state" should be cheaper (by orders of magnitude) than data plane "active state" - however, transitioning between the two is a complex problem (that has gotten somewhat easier as we have gotten smarter about distributed systems over time)
>>> 
>>> The second of network design is the balance point where the network is
>>> cheapest is different in every network.
>> 
>> agreed
>> 
>>> 
>>> So, IMHO, the point of protocol design should be to allow the design the
>>> maximum amount of flexibility and granularity in when and where to hide
>>> information, to make the right tradeoff for any particular network.
>>> 
>> agreed
>> 
>>> Now, to return to the discussion at hand: If you don't want to pay for
>>> devices that will handle 100 million routes, but you have 100 million
>>> devices, then you're going to have some degree of suboptimal bandwidth
>>> utilization. There's simply no way around this reality.
>> 
>> agreed
>>> 
>>> IMHO, however we resolve this problem, we need to resolve it in a way
>>> that allows an operator to support 100 million routes in every node, if
>>> they choose to optimize for bandwidth at the cost of hardware. Or to
>>> support 1 route in every node using really cheap hardware, if they want
>>> to optimize for state. Or for any point in the sliding scale in between
>>> (within reason).
>> 
>> agreed
>> 
>>> 
>>> So this control plane needs to:
>>> 
>>> 1. Be able to support 100 million destinations at the host level (that
>>> doesn't mean 100 million paths, but only 100 million destinations --two
>>> different problems).
>> 
>> yes
>> 
>>> 
>>> 2. Be able to aggregate to hide information at anyplace that's logical
>>> within the network.
>> 
>> yes
>> 
>>> 
>>> BTW, some folks would like to solve this problem by making the control
>>> plane react to the data plane --but this carries it's own baggage in
>>> complexity and in operational capabilities. Reactive control planes
>>> always converge more slowly, and waste bandwidth at a rate that's
>>> arguably higher than simple aggregation. So there's no "silver bullet"
>>> waiting in the wings in the form of caching at the control plane level.
>> 
>> however, I believe that aggregation time may be coming down, especially in a more homogenous environment (like a DC) vs a heterogeneous environment (like the DFZ).
>> 
>>> 
>>>>> 2. Why does this mobility need to be at layer 2 specifically? Are we
>>>>> assuming DDNS and other sorts of solutions in this space will simply
>>>>> never be fast enough/scale far enough/etc?
>>>> 
>>>> Like it or not, the key requirement for VM mobility is that the VM's
>>>> IP address does not change. That means the VM can't really move from
>>>> one IP subnet to another. That means either moving to bigger and
>>>> bigger L2s (all under one IP subnet) as the DC expands or the need to
>>>> inject /32 host routes.
>>> 
>>> Again, the same tradeoff as above --moving to a bigger l2 domain also
>>> means losing the ability to optimally direct traffic through the network
>>> (unless you put another control plane on top of the l2 and l3 control
>>> planes already in existence --which just adds the complexity you're
>>> trying to get away from in the host routes back into the network state!).
>> 
>> Or replacing one or two of those L2/L3 control planes with a control plane with a global view, which is easier to do in a constrained network like a traffic engineering core or a data center.  It's is a bounded problem.  As you stated, however, convergence MAY be slower, but I would argue that converging a network of Nx10K switching/routing nodes takes a non-trivial amount of time as well :)
>> 
>>> 
>>>> Neither of those approaches seems particularly scalable/desirable if
>>>> you look 10 years down the road and think of 1M+ physical machines in
>>>> a DC.
>>> 
>>> There is no "particularly desirable" solution, in reality, because
>>> building a network with no control plane state anyplace that uses
>>> bandwidth in a perfectly optimal way is simply impossible no matter how
>>> you slice it (unless you're going to go into quantum routing!).
>>> 
>>> :-)
>>> 
>>> Russ
>>> _______________________________________________
>>> dc mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/dc
>> 
>> --
>> 李柯睿
>> Check my PGP key here: https://www.asgaard.org/~cdl/cdl.asc
>> Current vCard here: https://www.asgaard.org/~cdl/cdl.vcf
>> 
>> _______________________________________________
>> dc mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/dc
>> _______________________________________________
>> dc mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/dc
> _______________________________________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dc

--  
李柯睿
Check my PGP key here: https://www.asgaard.org/~cdl/cdl.asc
Current vCard here: https://www.asgaard.org/~cdl/cdl.vcf

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 3 Jan 2012 08:01:21 -0800

> 100K or whatever that number is, is a big deal at the access. And that number will increase with time. The thing to bear in mind is that these are ADDITIONAL entries, not in lieu of other entries.…

That is assuming that we are adding that to "hardware" switches.  In some cases that will be the case, and sometimes not, or sometimes it may be a hybrid.

>> 1. State --the more you carry, the more hardware is going to cost.

If you are carrying it in hardware.  If not, the tradeoff will be in performance (available bandwidth) rather than $$.

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 3 Jan 2012 08:41:47 -0800

On 31Dec2011, at 05.27, Russ White wrote:

> Do you mean aggregation levels may be coming down?

I probably wasn't very clear hear, sorry.  What I was saying was that within the datacenter, the majority of the state being computed is the same for each element computing across the state.  That, combined with a flattening out of the network layers should decrease convergence time relative to the total amount of state.

> I would argue that
> aggregation is used to solve two problems:
> 
> 1. Rate of state change.
> 2. Physical table size.

agreed on both points

> 
> Note that you don't need both everywhere. For instance, it doesn't cost
> much, in the way of hardware, to hold millions of routes in the control
> plane, nor to provide the processor to deal with those routes. If you
> can afford 150k servers with moderately good processors and the memory
> to go with them, then you can afford 15k network devices with moderately
> good processors and the memory to go with them.

agreed

> 
> Where the cost is, is in the data plane, it seems --the forwarding table
> size. But we already know how to do aggregation (and even caching)
> between the control plane and the forwarding table.

yes

> 
> BTW, and aside on caching in the forwarding table. It appears to be a
> magic bullet for the problem of forwarding table size at first glance,
> but really it all depends on a lot of things... As long as the 80/20
> rule holds, caching works at this level. The question is --when you
> reach 50/50, does caching still work? What about 40/60? Or 20/80? And
> how does caching fail? It always fails catastrophically. Cliffs are not
> good hidden away in unmonitored/not well understood pieces of the network.

...but they are so much fun to discover the hard way :)

> 
> And of course when we try to cache at the control plane level in order
> to solve the forwarding table size problem, and when we try to mix
> cached control planes with data plane driven information, we are, IMHO,
> in "bad juju."

If that caching modality is not monitored, I agree.  However, I don't believe that this needs to be an unbounded asynchronous cache. However if we can adjust the cache system based on ratios, etc - and monitor it, it's no longer hidden, nor necessarily a cliff.  I'm not saying caching is the only way, but it may be one approach.

> 
> Note also that the speed at which topology changes impact the
> convergence of the protocol, and the scope of those changes, can be
> controlled effectively separately from the rate of change in leaf nodes
> on the tree --so long as the protocol is designed to handle this sort of
> separation effectively.

Protocol, or algorithm. 

> 
> So if we break the problem down into its component parts, and define
> what we need from each component, we might be able to reach a reasonable
> solution that provides convergence and mobility. There will always be a
> tradeoff, but it should be up to the operator to make that ultimate
> tradeoff along the points where it's simply impossible for a protocol to
> go beyond.

agreed.

> 
> And to deal with the information hiding verses optimal bandwidth usage
> problem, of course. :-)
> 
> I'm not certain I understand this... I think you mean like a DFZ, a
> control plane that knows every possible destination. But you have to
> separate knowing every possible destination from knowing every possible
> route to that destination. Even the DFZ in the 'net is really an
> aggregated suboptimal subset. I don't know of any network on this scale
> that has an optimal route to every destination, and I don't think it's
> really possible to build one unless you want to make processing power
> and control plane bandwidth usage unbounded.

For an unbounded network, I agree.  However, if it is within a bounded subset (i.e. a dc or collection of dc's) I believe it is possible.

> 
> It's quite possible to know every destination as a host, but not know
> the entire path to every one of those destinations in detail (a form of
> fisheye routing, for instance). Aggregation is, in reality, just a form
> of fisheye routing --you know the path to the aggregation point in
> detail, but you don't know the path beyond that.

However, if that set of optimal paths is computed for every source/dest pair (or at least for every unique best path) once, based on a global topological/demand view, a "global" set of best paths may be accomplished within that constrained universe.

> 
> The difference in aggregation at the reachability level is that you also
> don't know the actual state of the destinations hidden in the aggregate
> itself. To the degree that mobility isn't an issue, it's okay to tie
> topology to reachability in this way. When mobility becomes an issue,
> you need to unbundle the two in some way, treating detailed topology as
> one problem, and detailed reachability as another problem altogether.
> I'm guessing that unbundling these two is the most "logical" or "free
> and clear" path towards scaling for the requirements as they appear to
> exist.

Agreed, if there are destinations hidden by aggregation - however, a "global" view may not have that problem.

yes

> Russ

Chris

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 3 Jan 2012 09:04:33 -0800

On 30Dec2011, at 01.45, Xuxiaohu wrote:

>> -----邮件原件-----
>> 发件人: Xuxiaohu
>> 发送时间: 2011年12月30日 12:17
>> 收件人: 'Christopher LILJENSTOLPE'
>> 抄送: Thomas Narten; Russ White; [email-masked]
>> 主题: re: [dc] Elevator Pitch
>> 
>> 
>>> -----邮件原件-----
>>> 发件人: Christopher LILJENSTOLPE [mailto:[email-masked]]
>>> 发送时间: 2011年12月30日 1:20
>>> 收件人: Xuxiaohu
>>> 抄送: Thomas Narten; Russ White; [email-masked]
>>> 主题: Re: [dc] Elevator Pitch
>>> 
>>> Greetings Xuxiaohu,
>>> 
>>> On 29Dec2011, at 00.55, Xuxiaohu wrote:
>>> 
>>>> Hi Thomas,
>>>> 
>>>>> -----邮件原件-----
>>>>> 发件人: [email-masked] [mailto:[email-masked]] 代表
>> Thomas
>>>>> Narten
>>>>> 发送时间: 2011年12月29日 1:01
>>>>> 收件人: Russ White
>>>>> 抄送: [email-masked]
>>>>> 主题: Re: [dc] Elevator Pitch
>>>>> 
<snip - it was getting long>

> Hi Chris,
> 
> By the way, in the MAC over IP solution, the forwarding table of the leaf nodes contains the MAC routes to hosts and the gateways, similarly, in the IP over IP solution, the forwarding table of leaf nodes contains the routes to hosts and one or more default route to the gateways. If you believe the default route directed to the gateway in the IP over IP solution is not enough for forwarding the cross-data-center traffic, does that mean the MAC over IP solution is totally unworkable since there is only one MAC route directed to the gateway for forwarding the cross-data-center traffic?

Greetings Xiaohu,

I didn't actually state that IP over IP or MAC over IP were unworkable - I think that is a network design decision, and the answer will be different for different networks.  MAC's being non-aggregatable does make it a bit more "interesting" than IP over IP however.

What I am saying is that if all "east-west" traffic in the DC has to aggregate up to some level of core via a tree (read default route) topology, that would lead to some interesting scale issues (actually, it already does).  There is a reason that large networks tend not to run default for edge-to-edge connectivity (i.e. backbones) - almost always they are incomplete meshes.  The fact that we continue to use trees in the data center is an artifact of their enterprise origins, and the historical north-south traffic patterns.  I think the reason you are seeing so much activity now is due, in part, to the fact that the tree is starting to creak, and the north-south assumption is no longer valid.

	Chris

> 
> Best regards,
> Xiaohu
> 
>> Hi Chris,
>> 
>> Would you please give a concrete example where the communication between
>> different tenants is very common in the multi-tenant cloud data center?
>> 
>> Best regards,
>> Xiaohu
>> 
>>>> 
>>>> 
>>>>> Or will they want an alternative approach?
>>>>> 
>>>> In the DCI scenario where the PE routers are usually performed at the
>>> aggregation SWs or even core SWs, the PE routers would need a much large
>>> forwarding table. Provided the routing table containing millions of entries,
>>> which is available on most today's high-end routers, was still not large
>> enough,
>>> the on-demand FIB installation or on-demand route announcement
>>> mechanisms can be used further to scale the solution. Note that the trigger
>> for
>>> the FIB installation or route announcement is ARP request packets rather
>> than
>>> data packets. Hence it will not cause the so-called initial packet loss or
>> latency
>>> issue.
>>>> 
>>>> Maybe we should also take the development speed of routing/switching
>> chip
>>> and CPU technologies into account:)
>>> 
>>> It's more a question of cost/performance on off-chip memory/TCAMs.  That
>> is
>>> a slightly different curve :)
>>> 
>>> 	Chris
>>> 
>>>> 
>>>> Best regards,
>>>> Xiaohu
>>>> 
>>>>> Thomas
>>>>>

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 3 Jan 2012 09:07:08 -0800

On 29Dec2011, at 20.31, Aldrin Isaac wrote:

> 
> On Dec 29, 2011, at 4:27 PM, Christopher LILJENSTOLPE wrote:
> 
>> Greetings Aldrin,
>> On 29Dec2011, at 13.21, Aldrin Isaac wrote:
>> 
>>> Hi Christopher,
>>> 
>>>>> 
>>> Wrt your responses above, are you speaking specifically regarding data centers where end stations are 100% VM?  Could you lend your thoughts on how non-aggregating DCs ought to interwork with an aggregating global WAN/Internet? 
>> 
>> No, I was not assuming a 100% VM datacenter (which I think is unlikely in the near-to-mid term - there will be appliances, storage, etc for sometime that is not behind a HV).  However, I'm uncertain by what you mean a "non-aggregating" data center.  Any data center will, by nature aggregate, the question is where, and how many layers of aggregation?  Leaf (hypervisor and/or ToR for non HV elements), ToR, spine, core router?  I seriously doubt that any DC will expose it's enitire fleet of /32 (/128) hosts to the Internet (or a global WAN).   Some level of aggregation will be necessary.  As to how a DC would interact with another DC or a WAN backbone in a controller-based network, one could envision each DC as a network element that aggregates into the WAN with, potentially, cooperating controllers, or a controller hierarchy.  Other topologies are likely to exist, as well.
>> 	Chris
> 
> Hi Chris, 
> 
> I may have misunderstood your statement, "I believe that aggregation time is coming down", to mean you are a proponent of dramatically reducing or eliminating route aggregation in some way.  Maybe I'm wrong, but it seems there are at least a few folks that expect that hosts should be independently mobile across more than one site and that a solution should support that expectation at scale.

I agree that is a design goal - it was for me at my last job as well.

> 
> I think we're on the same page wrt needing some levels of aggregation.  As a matter of fact, the run-of-the-mill DC network design already has aggregation at the access gateway/router (in the form of subnet routes) and "host routes" (mac routes) within the access LAN.  You can make the access LAN (host route domain) smaller or bigger (hopefully not relying on STP) for varying degree of mobility.  

Run-of-the-mill I would agree.  However, the fact that we are all spending so much time and effort in this space would indicate that the current mode of operation seems to be either breaking down or not delivering the desired outcomes.

> 
> Could you highlight, from your perspective, the features current standards and design patterns cannot fundamentally support?  Since we're familiar with the limitations with STP, flooding and ARP for large LANs, I'm interested in knowing, in technical terms, what other flaws you believe exist and features you believe are missing.

Please go back and look at Tom Narten's presentation on the topic from Taipei.  

> 
> Thanks -- aldrin_______________________________________________
> dc mailing list
> [email-masked]

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 3 Jan 2012 09:13:02 -0800

Greetings,

On 29Dec2011, at 20.01, Xuxiaohu wrote:

>>>> 发件人: [email-masked] [mailto:[email-masked]] 代表 Thomas
>>>> Narten
>>>> 发送时间: 2011年12月29日 1:01
>>>> 收件人: Russ White
>>>> 抄送: [email-masked]
<snip>

> Hi Chris,
> 
> Would you please give a concrete example where the communication between different tenants is very common in the multi-tenant cloud data center?

Here's the question - if you are a tenant in a data center, and you are writing a mash-up application against some other content provider, how do you know if they are in the same data center, or not?  My guess is that there is quite a bit of traffic between tenants of EC2, btw.  I know that that was the intent at my last gig - we wanted SaaS-like providers to live in our DC's and develop an eco-system around our core services.  Other major cross-dc traffic - how about all of my services like spam filtering, backup, etc.  In a DC, I may call them "core services" but they are, in fact, another tenant.  How about large scale content providers that mash-up between their own offerings.  Many of those properties are viewed as "separate customers" by the infrastructure teams (can't name names here).  Any inter-offering mash-ups would definitely be cross-dc.
>> which is available on most today's high-end routers, was still not large enough,
>> mechanisms can be used further to scale the solution. Note that the trigger for
>> the FIB installation or route announcement is ARP request packets rather than
>> data packets. Hence it will not cause the so-called initial packet loss or latency
>> issue.
>>> 
>>> Maybe we should also take the development speed of routing/switching chip
>> It's more a question of cost/performance on off-chip memory/TCAMs.  That is

---

From: Thomas Narten <[email-masked]>
Date: Tue, 03 Jan 2012 15:55:41 -0500

+1 to Chris' comments.

Christopher LILJENSTOLPE <[email-masked]> writes:

> Secondly, the question we should be asking is "What does the IETF
>  NEED to do" not "What can the IETF do" We can do a great many
>  things, the bulk of which will not be helpful or a good use of
>  resources.

Agree completely. If those advocating that the IETF "do work" cannot
answer the above succinctly and in a way that fellow IETFers can
understand, the only conclusion that can be drawn is that the IETF
cannot (and should not) take any further action at this time.

> > On 28Dec2011, at 17.40, Bhumip Khasnabish wrote:
> > We need comments and suggestions from you and others to update this doc.
> > 
> > We also have another draft covering potential work items
> > (
> > http://tools.ietf.org/html/draft-khasnabish-cloud-industry-workitems-survey-01

I had a look at this document, and most if not all of the
"requirements" or "work items" are very very high level, and it is not
at all clear to me how they relate to IETF work.

> > We can discuss these further in the interim mtg.

I strongly disagree. See above. There needs to be much more prep work
done on describing a specific technical or operational problem that
the IETF would work on before there would be much use in having a f2f
discussion. IMO.

> I would like to propose a different approach, if I may.  If we took
>  a focused set of problem statements and ran them through the
>  following set of filters:

> 1) Is this a current/real or near-to-mid term probable issue and is
>  it substantial?
  
> 2) If yes, is it being adequately covered by another SDO and is it
>  in that SDO's domain?
  
> 3) if no, is it in the domain of IETF competency?

> 4) if yes, do we want to work on it?

Agree completely. Again, this list should be devoted exclusively to
teasing out problem areas for which the IETF would seem to be the
right place to so specific work.

Thomas

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 04 Jan 2012 03:11:58 +0000

> 发送时间: 2012年1月4日 1:13
> > Would you please give a concrete example where the communication
> between different tenants is very common in the multi-tenant cloud data
> center?
> 
> Here's the question - if you are a tenant in a data center, and you are writing a
> mash-up application against some other content provider, how do you know if
> they are in the same data center, or not?  My guess is that there is quite a bit
> of traffic between tenants of EC2, btw.  I know that that was the intent at my
> last gig - we wanted SaaS-like providers to live in our DC's and develop an
> eco-system around our core services.  Other major cross-dc traffic - how
> about all of my services like spam filtering, backup, etc.  In a DC, I may call
> them "core services" but they are, in fact, another tenant.  How about large
> scale content providers that mash-up between their own offerings.  Many of
> those properties are viewed as "separate customers" by the infrastructure
> teams (can't name names here).  Any inter-offering mash-ups would definitely
> be cross-dc.

Hi Chris,

I have no doubt about the possibility you mentioned above. However, once you are considering to optimize the forwarding path of inter-tenant traffic within the scope of L2VPN or L3VPN solutions, while taking the address space overlapping, firewall policy issues, etc. into account, you will find it is a much complex job. Whether or not it is worthwhile to do that optimization heavily depends on whether the volume of inter-tenant traffic is much significant. 

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 04 Jan 2012 03:20:16 +0000

> 发送时间: 2012年1月4日 1:05
> > By the way, in the MAC over IP solution, the forwarding table of the leaf
> nodes contains the MAC routes to hosts and the gateways, similarly, in the IP
> over IP solution, the forwarding table of leaf nodes contains the routes to hosts
> and one or more default route to the gateways. If you believe the default route
> directed to the gateway in the IP over IP solution is not enough for forwarding
> the cross-data-center traffic, does that mean the MAC over IP solution is
> totally unworkable since there is only one MAC route directed to the gateway
> for forwarding the cross-data-center traffic?
> 
> Greetings Xiaohu,
> 
> I didn't actually state that IP over IP or MAC over IP were unworkable - I think
> that is a network design decision, and the answer will be different for different
> networks.  MAC's being non-aggregatable does make it a bit more
> "interesting" than IP over IP however.
> 
> What I am saying is that if all "east-west" traffic in the DC has to aggregate up
> to some level of core via a tree (read default route) topology, that would lead to
> some interesting scale issues (actually, it already does).  There is a reason
> that large networks tend not to run default for edge-to-edge connectivity (i.e.
> backbones) - almost always they are incomplete meshes.  The fact that we
> continue to use trees in the data center is an artifact of their enterprise origins,
> and the historical north-south traffic patterns.  I think the reason you are
> seeing so much activity now is due, in part, to the fact that the tree is starting
> to creak, and the north-south assumption is no longer valid.

Hi Chris,

Agree to your above opinions. However, the question is whether the majority of the east-west traffic is intra-tenant or inter-tenant.

Best regards,
Xiaohu

> between

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 04 Jan 2012 03:50:38 +0000

Hi Yakov,

> -----邮件原件-----
> 发件人: Yakov Rekhter [mailto:[email-masked]]
> 发送时间: 2012年1月3日 22:33
> 收件人: Xuxiaohu
> 抄送: [email-masked]

Both. 

> Do you need fast routing convergence, or fast connectivity restoration ?

Both.

Best regards,
Xiaohu

---

From: Aldrin Isaac <[email-masked]>
Date: Tue, 3 Jan 2012 23:54:50 -0500

> Please go back and look at Tom Narten's presentation on the topic from Taipei.  

Thomas' presentation is limited to solving a set of problems for environments that will be primarily VM based (with low/no multicast) using overlay approach.  This mailing list, I believe, is inclusive of nvo3 but not limited to it, so my question was broader.

---

From: Patrick Frejborg <[email-masked]>
Date: Wed, 4 Jan 2012 09:52:18 +0200

> cannot (and should not) take any further action at this time.

I have two proposals - having them solved would be helpful in the
design and implementing phase of data center networks.

1. PVLAN
There are use cases where PVLAN is needed, e.g. implementing a
separate Ethernet network for backup traffic. In this use case the
host will have two NICs - one for "production" traffic (clients
reaching the server) and the backup NIC. Usually you assign the 0/0
route for the "production" NIC, the outcome is that the backup NIC
should have a very large subnet assigned so that the backup node can
be located in the very same subnet - and static routing is avoided at
the server (if you apply static routing at server there are two teams
taking care of the routing domain - moves, adds and changes becomes an
operational nightmare). Because the server resides in different
security zones it is a bad idea to have the backup NICs in the same L2
domain - it can be solved by implementing PVLAN and the backup NIC can
only reach the node which is providing the backup service.
The problem with PVLAN is interoperability between switches and switch
vendors,  from RFC5517 section 3:

   When a VLAN spans multiple switches, there is no standard mechanism
   to propagate port-level isolation information to other switches and,
   consequently, the isolation behavior fails in other switches

2. L2 hardening
I'm not aware of a BCP or informational RFC on how to mitigate L2
attacks, such as VLAN hopping, ARP spoofing, MAC spoofing etc I always
get into discussions with security officers how much can we trust the
virtualization of network devices. These issues can be mitigated on
some switches but with increased use of hypervisors these issues
should be taken into account at the virtual switches as well. Maybe it
would be helpful with an RFC listing all the L2 attacks and how they
can be mitigated so that the vendors pay better attention to these
threats.

Patrick

---

From: Yakov Rekhter <[email-masked]>
Date: Wed, 4 Jan 2012 05:01:12 -0800

Xuxiaohu,

> > > 2) Some cluster applications use non-IP or link-local multicast 
> > >     (optional) -> Layer2 networking
> > >

I understand the rationale for fast connectivity restoration. But
given that there is fast connectivity restoration, why would one
also need fast routing convergence ?

> > Do you need fast routing convergence, or fast connectivity restoration ?

The same question, as above.

Yakov.

---

From: Russ White <[email-masked]>
Date: Wed, 04 Jan 2012 10:14:22 -0500

> With this in mind let me paraphrase what you said above as follows:

Yes and no... How's that for equivocation? :-)

Fast reroute generally means two things --routing around the problem
quickly, and then adjusting the main control plane. IMHO, the first
stage needs to be in millisconds, and the second stage in seconds (5
seconds or less, if possible). My reasoning is that the time between the
first and second stage is a vulnerable moment in the network --if a
second failure occurs just at that moment, really bad things can happen.

So the first stage needs to be faster than people (or increasingly
computers) will notice, the second stage needs to happen faster than a
second failure is likely to happen (can't stop it all the time, but we
can make it rare to the point of being out of mind).

:-)

Russ

---

From: Russ White <[email-masked]>
Date: Wed, 04 Jan 2012 10:24:26 -0500

> If that caching modality is not monitored, I agree.  However, I
> don't believe that this needs to be an unbounded asynchronous cache. 
> However if we can adjust the cache system based on ratios, etc - and 
> monitor it, it's no longer hidden, nor necessarily a cliff.  I'm not 
> saying caching is the only way, but it may be one approach.

So long as the cache isn't "hidden from the hosts" in the network, but
rather is something the host knows about and can interact with
(understands the state of), caching can be useful. When application
state becomes unbundled from cache state, things tend to slow down
beyond "useful," I think, to provide the "working room" between the
cache and the application.

>> control plane that knows every possible destination. But you have 
>> to separate knowing every possible destination from knowing every 
>> possible route to that destination. Even the DFZ in the 'net is 
>> really an aggregated suboptimal subset. I don't know of any
>> network on this scale that has an optimal route to every
>> destination, and I don't think it's really possible to build one
>> unless you want to make processing power and control plane
>> bandwidth usage unbounded.
> 
> For an unbounded network, I agree.  However, if it is within a 
> bounded subset (i.e. a dc or collection of dc's) I believe it is 
> possible.

Except that we are already talking about relatively unbounded sizes, I
think --100 million routes (well, not really unbounded, but bounded at a
point far beyond what we have today).

> However, if that set of optimal paths is computed for every 
> source/dest pair (or at least for every unique best path) once,
> based on a global topological/demand view, a "global" set of best
> paths may be accomplished within that constrained universe.

Assuming you're doing the computation on a very high end set of hardware
--which implies "off line," I think. But this is a different model from
what we deal with today, and I'm not certain how willing the group would
be to go in that direction.

Fast reroute (see the other thread with Yakov) has made it possible to
precompute fast backup paths locally, potentially solving one of the
various problems with off line computation. So long as you have the
second stage react fast enough not to cause double failure failures...

> "global" view may not have that problem.

The key to a "global view," done off line (again, assuming this is what
you're driving at), is that you can actually optimize aggregation
--there are heuristics in existence that can do so, and I'm certain
research would yield more. The tradeoff is in the details of getting
forwarding state fast enough, building a separate control plane network,
and things like that.

:-)

Russ

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 04 Jan 2012 16:26:27 +0100

Hi Russ,

+1

And just to add to that fast connectivity restoration is possible only 
if you have sufficient relatively static information about backup paths 
ahead of failure already in the system.

As example .. if you are in very mobile situations where not DCs are 
moving but their clients are moving in a stochastic pattern you can't 
predict the backup path ahead of failure. In such cases where backup 
path can not be precomputed - fast connectivity restoration becomes 
quite a challenge. In those cases you are usually stuck with transient 
suboptimal data paths till re-convergence.

R.

---

From: DIEGO LOPEZ GARCIA <[email-masked]>
Date: Wed, 04 Jan 2012 17:16:20 +0100

Hi Robert,

> suboptimal data paths till re-convergence.

Definitely yes, and it is there where solutions a-la-Mobile-IP come into play. Suboptimal but functional and reasonably simple to deal with.

Be goode,

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D

e-mail: [email-masked]
Tel:      +34 913 129 041
Mobile: +34 682 051 091
-----------------------------------------

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at.
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 4 Jan 2012 17:44:17 +0000

Patrick, 

The PVLAN suggested by RFC 5517 is only within one switch. Those port properties can't be carried from one switch to another switch.  Are you suggesting that IETF should have a new work item to allow the VLAN port property to be carried from one switch to another? Then the Port property has to be carried in the data frame. It is almost like extending some bits for the VLAN. Is this work in the domain of IEEE802.1?

My understanding is that the 3 different port types (isolated, community, and promiscuous), are really for establishing e-tree services or different forwarding behavior for different hosts, but not for using less number of VIDs in providing client isolation.   

Other comments are inserted below: 

Linda 

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Patrick Frejborg
> Sent: Wednesday, January 04, 2012 1:52 AM
> To: Thomas Narten
> Cc: Ronald Bonica; Bhumip Khasnabish; [email-masked]; Christopher
> LILJENSTOLPE; So, Ning
> Subject: Re: [dc] Elevator Pitch (was: Scoping the Interim meeting)
> 
[Snip]
> reaching the server) and the backup NIC. 

[Linda] The "Backup traffic" is for the purpose of carrying traffic when the production NIC fails or is for the purpose of communicating with back end server/storage? 

>Usually you assign the 0/0
> be located in the very same subnet - 

[Linda] Is the "backup node" providing services to very large number of servers?   

> and static routing is avoided at

[Linda] Do you mean that "backup NIC" can only reach "backup (or backend) service"? 

> can be mitigated so that the vendors pay better attention to these

[Linda] That is a very good point. One of the ARMD objectives is to deliver an information RFC on ARP/ND related security recommendation. I know many companies have implemented those features. But it is difficult to motivate people to invest time in writing up those practices.

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 05 Jan 2012 01:33:35 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Yakov
> Rekhter
> 发送时间: 2012年1月4日 21:01
> 收件人: Xuxiaohu
> 抄送: [email-masked]

Hi Yakov,

I have a much similar question to you: given that there is fast connectivity restoration, why did we also need fast routing convergence for the Internet routing system?

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 05 Jan 2012 02:48:40 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Xuxiaohu
> 发送时间: 2012年1月5日 9:34
> 收件人: Yakov Rekhter
> 抄送: [email-masked]
> I have a much similar question to you: given that there is fast connectivity
> restoration, why did we also need fast routing convergence for the Internet
> routing system?

Hi Yakov,

By the way, the "fast convergence" here is mainly in contrast to the "slow convergence" of the spanning-tree protocol.

Best regards,
Xiaohu

---

From: Patrick Frejborg <[email-masked]>
Date: Thu, 5 Jan 2012 10:20:32 +0200

> The PVLAN suggested by RFC 5517 is only within one switch. Those port properties can't be carried from one switch to another switch.  Are you suggesting that IETF should have a new work item to all…

[Patrick] Hmm, can we avoid the data plane and instead use the control
plane to carry port properties by e.g. applying extensions to LLDP -
should be quite straightforward since LLDP make use of TLVs ?
It might be that this topic is better suited for IEEE802.1 than IETF -
anyway it is pain point for me, I have seen quite many data centers
using PVLAN.

> My understanding is that the 3 different port types (isolated, community, and promiscuous), are really for establishing e-tree services or different forwarding behavior for different hosts, but not…

[Patrick] The main benefit is to preserve global IP addresses by
avoiding a lot of subnets and static routes on the servers, i.e.
create a lot of isolated VLANs but span one large IP addresses over
those VLANs. You can't use RFC1918 addresses because when you move
customers' backend servers into your data center they have been
allocated RFC1918 addresses - and there is no sponsor with a budget
for a renumbering project to avoid overlapping addresses.

> [Linda] Is the "backup node" providing services to very large number of servers?

[Patrick] Yes, the servers are multi-tenant and are located in
different security zones

> [Linda] Do you mean that "backup NIC" can only reach "backup (or backend) service"?

[Patrick] Yes, that is the purpose. Then there might be other purposes
as well, e.g. management of the hypervisors, monitoring tools such as
Openview Tivoli, BMC Patrol, heart beat traffic for application
clusters etc. but these are usually assigned their own VLAN and these
have no need for PVLAN.

> [Linda] That is a very good point. One of the ARMD objectives is to deliver an information RFC on ARP/ND related security recommendation. I know many companies have implemented those features. But…

[Patrick] Have a look on the virtual switches, most of them have a
very poor set of security features. Seems that they aren't aware of
the security issues with L2 attacks.

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Fri, 6 Jan 2012 09:20:12 -0800

Greetings - in-line,

> I have no doubt about the possibility you mentioned above. However, once you are considering to optimize the forwarding path of inter-tenant traffic within the scope of L2VPN or L3VPN solutions, wh…

This is one that, if we miss, will come back and bite us.  If you have no way of getting traffic between "tenants" in a data centre, I have NO idea how you will deal with shared services (such as storage, backup, transit, etc) as well as the classical inter-tennant problem.  If the proposal is to just send it up the tree (as is done today), you will continue to have the same issues as we do now (that is driving all this work).  If you doubt this, ask yourself why there is a drive for greater cross-sectional bandwidth in the data center.  If you believe that the classical tree solution will work, then we are all chasing our tail here.  

	Chris

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Fri, 6 Jan 2012 09:24:10 -0800

Greetings,

> cache and the application.

Hense one of the drivers of SDN's.

> point far beyond what we have today).

I'm interested in the 100 M routes.  I'm not aware, at this time, of any data center with 100 M nodes in it.  If we do end up with that size, then yes, it does start getting interesting.

> second stage react fast enough not to cause double failure failures...

That would be one approach

> forwarding state fast enough, building a separate control plane network,

Yup - it's all fun.

---

From: Bhumip Khasnabish <[email-masked]>
Date: Fri, 6 Jan 2012 14:19:05 -0500

Hello Thomas,

Thanks.

(1) Pls note that we have follow up drafts
(VDCS, VPN4DC, VEPC, VPN-O-CS, VPN-O-DCS, VRM, security framework for VDCS,
etc.)
from different contributors
for many of the problems that are mentionedd in the work items draft.
I am sure that you are familiar with at least a few of these drafts.

(2) Do you have any Agenda in mind for the interim mtg.?
I was just suggesting that we can look at the DataCenter
design/development/operations problems based on high-priority
use cases, and can explore which ones can possibly be solved using
the solutions that IETF can develop promptly...

(3) We can do the "teasing out problem areas for which the IETF .."
off-line and
can use the F2F mtg to finalize the work items and can discuss possible
approaches.

Once again, many thanks for contributions.

Best.

Bhumip

-- 
Best.

Bhumip Khasnabish
[email-masked]
[email-masked]
+1-781-752-8003 (mobile)
http://tinyurl.com/bhumip

                   __o
             _ `\ <, _
.......... ( • ) / ( • ) ......................

---

From: Michael Hammer <[email-masked]>
Date: Fri, 6 Jan 2012 15:48:23 -0500

Christopher,

Do you assume that the only FW applications are at the top of the tree and
serve all customers?
Or, could there be customer-centric FW apps that demarcates the boundary
between two tenants
that would also be on VMs within the DC and globally addressable?
Would such traffic need to take other than an optimal route between VMs
inside the DC?

Trying to picture the anchor points for routing.

Mike

On Fri, Jan 6, 2012 at 12:20 PM, Christopher LILJENSTOLPE
<[email-masked]>wrote:

> > I have no doubt about the possibility you mentioned above. However, once
> you are considering to optimize the forwarding path of inter-tenant traffic
> within the scope of L2VPN or L3VPN solutions, while taking the address
> space overlapping, firewall policy issues, etc. into account, you will find
> it is a much complex job. Whether or not it is worthwhile to do that
> optimization heavily depends on whether the volume of inter-tenant traffic
> is much significant.
>
> This is one that, if we miss, will come back and bite us.  If you have no
> way of getting traffic between "tenants" in a data centre, I have NO idea
> how you will deal with shared services (such as storage, backup, transit,
> etc) as well as the classical inter-tennant problem.  If the proposal is to
> just send it up the tree (as is done today), you will continue to have the
> same issues as we do now (that is driving all this work).  If you doubt
> this, ask yourself why there is a drive for greater cross-sectional
> bandwidth in the data center.  If you believe that the classical tree
> solution will work, then we are all chasing our tail here.
>
>        Chris
>
> >
> > Best regards,
> > Xiaohu
> >
> >>> Best regards,
> >>> Xiaohu
> >>>
> >>>>>
> >>>>>

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 6 Jan 2012 20:35:34 -0500

> This is one that, if we miss, will come back and bite us.  If you have no way of getting traffic between "tenants" in a data centre, I have NO idea how you will deal with shared services (such as s…

100% agree.  Solution MUST have an efficient way to reach shared services, extranets, portals, inter-tenant transit, etc.  Solution is a non-starter without it.

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 07 Jan 2012 02:21:54 +0000

> 发送时间: 2012年1月7日 1:20
> within the scope of L2VPN or L3VPN solutions, while taking the address space
> overlapping, firewall policy issues, etc. into account, you will find it is a much
> complex job. Whether or not it is worthwhile to do that optimization heavily
> depends on whether the volume of inter-tenant traffic is much significant.
> 
> This is one that, if we miss, will come back and bite us.  If you have no way of
> getting traffic between "tenants" in a data centre, I have NO idea how you will

Hi Chris,

I'm not saying traffic between "tenants" should not be allowed in a data center. IMHO, the inter-tenant traffic originated from one tenant could be forwarded towards its gateway which will in turn forward the traffic to the destination within another tenant. If you want the ToRs (acting as ingress PE routers) to perform the inter-tenant routing so as to achieve path optimization, there would be a lot of hard problems to be considered:, 1) the forwarding table scalability issue will be worsen, for example, in the IP over IP solution, the VRF routing table size on the ToR will be increased since the routes from other tenants (e.g., VRFs) would have to be imported, in the MAC over IP solution, the ToR would have to support IRB (integrated routing and bridging) functions which required the ToR to have a MAC table for intra-tenant traffic forwarding and a routing table for inter-tenant traffic forwarding, the routing table size will be much larger due to the same reason as above ; 2) the ACLs for inter-tenant access control would have to be configured identically on all of the PE routers (i.e., ToRs) belonging to the same tenant; 3) if the address spaces of different tenants are overlapped, you would have to configure NAT service on all of the PE routers (i.e., ToRs) belonging to the same tenant, in addition, there will a lot of uncertain issues need to be considered... In a word, all of those will make the provisioning and management much more complicated.

Best regards,
Xiaohu  

> deal with shared services (such as storage, backup, transit, etc) as well as the
> classical inter-tennant problem.  If the proposal is to just send it up the tree
> (as is done today), you will continue to have the same issues as we do now
> (that is driving all this work).  If you doubt this, ask yourself why there is a
> drive for greater cross-sectional bandwidth in the data center.  If you believe
> that the classical tree solution will work, then we are all chasing our tail here.
> 
> 	Chris
> 
> >
> > Best regards,
> > Xiaohu
> >
> >>> Best regards,
> >>> Xiaohu
> >>>
> >>>>>
> >>>>>

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 9 Jan 2012 07:42:43 -0800

Xiaohu,

> restoration, why did we also need fast routing convergence for the Internet 

Because fast routing convergence is *one* possible way to accomplish
fast connectivity restoration. Having said this, I want to make it
clear that while it is one possible way, it is *not* the only way
possible. 

Moreover, when talking about routing convergence one need to
distinguish between the cases where (a) during routing convergence
connectivity is preserved, although perhaps along suboptimal routes,
and (b) during routing convergence connectivity is disrupted.

Yakov.