Subject: [dc] Scoping the Interim meeting

From: Ronald Bonica <[email-masked]>
Date: Tue, 20 Dec 2011 14:48:29 -0500

Folks,

During the last few days, this list has produced an impressive annotated bibliography of Internet Drafts. Thanks to all who have contributed.

In preparation for the upcoming interim meeting, this group should consider all of the "problem statement" drafts. Specifically, we should make a list of problems that are called out by a majority of these drafts. We will identify these problems as the core data center problem space.

We should also make another list of problems that are called out by only a small minority of drafts. Having made this second list, we will explain why the problem is not commonly viewed. Possible answers are:

- it is not a real problem
- the problem is not unique to the data center
- the problem is unique to a particular service being offered in the DC
- the problem is unique to a particular data center network architecture

It would be great if we could capture the output of this exercise in an Internet Draft. 

Comments?

--------------------------
Ron Bonica
vcard:       www.bonica.org/ron/ronbonica.vcf

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 20 Dec 2011 15:00:26 -0600

Ron,

We think this is a very good idea. 
We'd like to volunteer to edit this problem analysis draft together.
Welcome everyone's input.

Thanks,
Luyuan and Thomas

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Ronald Bonica
> Sent: Tuesday, December 20, 2011 2:48 PM
> To: [email-masked]
> Subject: [dc] Scoping the Interim meeting
> 
> Folks,
> 
> During the last few days, this list has produced an impressive
> annotated bibliography of Internet Drafts. Thanks to all who have
> contributed.
> 
> In preparation for the upcoming interim meeting, this group should
> consider all of the "problem statement" drafts. Specifically, we
should
> make a list of problems that are called out by a majority of these
> drafts. We will identify these problems as the core data center
problem
> space.
> 
> We should also make another list of problems that are called out by
> only a small minority of drafts. Having made this second list, we will
> explain why the problem is not commonly viewed. Possible answers are:
> 
> It would be great if we could capture the output of this exercise in
an
> Internet Draft.
> _______________________________________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dc

---

From: "Pat Thaler" <[email-masked]>
Date: Tue, 20 Dec 2011 15:12:09 -0800

There is a fifth possibility for why a problem might be only in a small minority of drafts
- it is a real problem that should be in the core data center problem space, but only a small  group of authors have identified it

Probably that explanation won't apply to most of the set, but it is possible in the early days of people looking at a problem space from different perspectives and we should apply our judgment rather than making the cut just based on the number of drafts that spotted the issue.

Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ronald Bonica
Sent: Tuesday, December 20, 2011 11:48 AM
To: [email-masked]
Subject: [dc] Scoping the Interim meeting

Folks,

Comments?

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 20 Dec 2011 20:20:02 -0600

Chris,

Good idea. Will reach out to Tom.
Luyuan

> From: Christopher LILJENSTOLPE [mailto:[email-masked]]
> Sent: Tuesday, December 20, 2011 6:51 PM
> To: Luyuan Fang (lufang)
> Cc: Ronald Bonica; [email-masked]; Thomas Narten
> Subject: Re: [dc] Scoping the Interim meeting
> 
> Greetings,
> 
> 	I would suggest that Tom Nadeau also be included, as he had quite
> a bit of input pulling the SDN stuff together.
> 
> 	Chris
> 
> On 20Dec2011, at 13.00, Luyuan Fang (lufang) wrote:
> 
> > Ron,
> >
> >> only a small minority of drafts. Having made this second list, we
> >> will explain why the problem is not commonly viewed. Possible
> answers are:
> >>
> --
> 李柯睿
> Check my PGP key here: https://www.asgaard.org/~cdl/cdl.asc
> Current vCard here: https://www.asgaard.org/~cdl/cdl.vcf

---

From: "Luyuan Fang (lufang)" <[email-masked]>
Date: Tue, 20 Dec 2011 20:48:06 -0600

Make sense. Thx.
Luyuan

Pat
> Thaler
> Sent: Tuesday, December 20, 2011 6:12 PM
> To: Ronald Bonica; [email-masked]
> There is a fifth possibility for why a problem might be only in a
small
> minority of drafts
> - it is a real problem that should be in the core data center problem
> space, but only a small  group of authors have identified it
> 
> Probably that explanation won't apply to most of the set, but it is
> possible in the early days of people looking at a problem space from
> different perspectives and we should apply our judgment rather than
> making the cut just based on the number of drafts that spotted the
> issue.
> 
> Pat
> 
should
problem
an

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 20 Dec 2011 15:50:33 -0800

Greetings,

	I would suggest that Tom Nadeau also be included, as he had quite a bit of input pulling the SDN stuff together.

	Chris

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Tue, 20 Dec 2011 15:50:33 -0800

Greetings,

	Chris

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 21 Dec 2011 14:54:00 +0000

I would like to help too. Please keep me included. 

Thanks, Linda Dunbar

> Luyuan Fang (lufang)
> Sent: Tuesday, December 20, 2011 3:00 PM
> Cc: Thomas Narten
> will

---

From: "So, Ning" <[email-masked]>
Date: Wed, 21 Dec 2011 10:23:30 -0500

Looks like we have quite a few people who are willing to help.  Has anyone started "putting words down on paper"?  Perhaps we can begin by formulating a rough structure of the draft, and have one/two volunteer(s) focusing on each section/problem, and all others can contribute text/ideas as they see fit.  

I would like to offer my service on this effort as well.  

Best regards,
 
Ning So
Verizon Corporate Technology
(office) 972-729-7905
(Cell) 972-955-0914

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ronald Bonica
Sent: Tuesday, December 20, 2011 1:48 PM
To: [email-masked]
Subject: [dc] Scoping the Interim meeting

Folks,

Comments?

---

From: Bhumip Khasnabish <[email-masked]>
Date: Wed, 21 Dec 2011 10:32:50 -0500

We have an existing draft that surveys industry work items related to
cloud computing, networking and services (DC items are included in it)

http://tools.ietf.org/id/draft-khasnabish-cloud-industry-workitems-survey-01.txt

We are planning to update that, and look forward to comments and
suggestions.

On Wed, Dec 21, 2011 at 10:23 AM, So, Ning <[email-masked]> wrote:

> Looks like we have quite a few people who are willing to help.  Has anyone
> started "putting words down on paper"?  Perhaps we can begin by formulating
> a rough structure of the draft, and have one/two volunteer(s) focusing on
> each section/problem, and all others can contribute text/ideas as they see
> fit.
>
> During the last few days, this list has produced an impressive annotated
> bibliography of Internet Drafts. Thanks to all who have contributed.
>
> consider all of the "problem statement" drafts. Specifically, we should
> make a list of problems that are called out by a majority of these drafts.
> We will identify these problems as the core data center problem space.
>
> We should also make another list of problems that are called out by only a
> small minority of drafts. Having made this second list, we will explain why
> the problem is not commonly viewed. Possible answers are:
>

---

From: Melinda Shore <[email-masked]>
Date: Wed, 21 Dec 2011 07:26:30 -0900

> http://tools.ietf.org/id/draft-khasnabish-cloud-industry-workitems-survey-01.txt

I would be very, very disappointed to see this receive time on an
IETF meeting agenda.  I think your time would have been far better
spent putting together an architectural framework document or another
framework document - something problem-focused and coherent.  And
I'm *still* not clear why you're not referencing NIST work - that's
a nontrivial oversight.

Melinda

---

From: Bhumip Khasnabish <[email-masked]>
Date: Wed, 21 Dec 2011 11:50:43 -0500

* Pls let us know which NIST doc (URL please)
you are referring to related to Cloud/DataCater work items.

* Also, we'll be updating the Ref. framework draft soon, appreciate your
constructive
comments and suggestions ..

http://tools.ietf.org/id/draft-khasnabish-cloud-reference-framework-01.txt

On Wed, Dec 21, 2011 at 11:26 AM, Melinda Shore <[email-masked]>wrote:

>> http://tools.ietf.org/id/**draft-khasnabish-cloud-**
>> industry-workitems-survey-01.**txt<http://tools.ietf.org/id/draft-khasnabish-cloud-industry-workitems-survey-01.txt>
>>
>
> ______________________________**_________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/dc<https://www.ietf.org/mailman/listinfo/dc>
>

---

From: Thomas Narten <[email-masked]>
Date: Wed, 21 Dec 2011 13:30:44 -0500

We sure have a lot of volunteers wanting to help. That is fine at one
level, but misses the bigger point.

The best way to help is to articulate a specific problem or area. 

Do so on this list in a single message. We don't need yet more
documents (at least not yet).

2-4 paragraphs (no more).  Think "elevator pitch"
(http://en.wikipedia.org/wiki/Elevator_pitch).

You only get a couple of minutes to make your case, and if you can't
do so, either you don't understand the problem yourself well enough,
or the value-proposition just isn't compelling.

So far, I haven't seen discussions of specific problems (or problem
areas). What I'm seeing is pointers to lots of drafts, many of which
(for those that I've looked at) don't crisply get at a real, tangible
problem that I understand and think the IETF can/should work on. Bonus
points for showing:

1) an operator (or some other clear customer/consumer technology) that
   says "yes, that is what I need, and if it was available today, I'd
   use it"

2) a vendor saying "here is what my customers are telling me will
   solve a problem of theirs", and we think an IETF standard is
   needed.

3) demonstration of a clear technical/protocol gap between what is
   available today, and what is needed to address the problem. 

Thomas

---

From: Melinda Shore <[email-masked]>
Date: Wed, 21 Dec 2011 09:39:35 -0900

> The best way to help is to articulate a specific problem or area.

That bears repeating.

> documents (at least not yet).

I'm working with a Yingjie Gu and Senthil Sivakumar on the state
migration problem, but it's unclear to me that it belongs in an
operations-focused group.  The problem occurs in data centers but
I don't think it's a data center problem per se.  Totally the
chairs' call.

Melinda

---

From: Thomas Narten <[email-masked]>
Date: Wed, 21 Dec 2011 13:46:13 -0500

Hi Melinda.

> I don't think it's a data center problem per se.  Totally the

Great!

That said, let me suggest we *not* get hung up on process and "what
area is the right place to discuss" before we even start.

Focus on the problem statement. If a problem has validity, we can
easily figure out later *where* to discuss it.

Thomas

---

From: Spencer Dawkins <[email-masked]>
Date: Wed, 21 Dec 2011 17:00:10 -0600

> or the value-proposition just isn't compelling.

Thomas may be too modest to suggest that people think about what's 
recommended in http://tools.ietf.org/html/rfc5434, but I can promise you 
that at least one IAB member is hoping to see people reading it :-)

For reference, the first item on the list of goals in RFC 5434 is 
showing that

   - there is a problem that needs solving, and the IETF is the right
	group to attempt solving it.

And, just in case "elevator pitch" is not a familiar idiom for you, the 
idea is that you get on an elevator, recognize someone on the elevator 
who will make a decision about your idea, and start talking - 
remembering that you don't know what floor your decision maker will get 
off on, so you need to say the most important thing first, then the 
second most, and so forth.

I hope this is helpful.

Spencer

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 22 Dec 2011 01:23:45 +0000

Fully agree. It's no late to summarize the real problems in one doc after various problems have been fully discussed and verified one-by-one in the mailing-list.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Thomas
> Narten
> 发送时间: 2011年12月22日 2:31
> 收件人: So, Ning
> 抄送: Ronald Bonica; [email-masked]
> 主题: Re: [dc] Scoping the Interim meeting
>

---

From: Melinda Shore <[email-masked]>
Date: Wed, 21 Dec 2011 16:29:04 -0900

> Fully agree. It's no late to summarize the real problems in one doc after various problems have been fully discussed and verified one-by-one in the mailing-list.

What would be that document's purpose?

Melinda

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 22 Dec 2011 01:38:49 +0000

As a comprehensive problem statement doc, it can be referenced later for solution design.

Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Melinda
> Shore
> 发送时间: 2011年12月22日 9:29
> 收件人: [email-masked]
> > Fully agree. It's no late to summarize the real problems in one doc after
> various problems have been fully discussed and verified one-by-one in the
> mailing-list.
>

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 22 Dec 2011 17:10:12 -0500

Folks,

Amplifying what Thomas said:

At this point, the most helpful thing that anyone can do is to summarize the problem(s) that we are trying to solve in a relatively terse email. Once we come to consensus on that brief summary, we can flesh it out into very short Internet Draft.

I look forward to hearing your elevator pitches.

                                                     Happy Holidays,
                                                        Ron

> Thomas Narten
> Sent: Wednesday, December 21, 2011 1:31 PM
> To: So, Ning
> Cc: Ronald Bonica; [email-masked]

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Fri, 23 Dec 2011 08:54:53 +0530

Here's my elevator pitch. I'll assume that elevator has to go to the
140th floor, with a few stops.

-------------

Large virtualized datacenters pose several networking challenges. 

First, native L2 or L3 forwarding technologies do not work. L2 networks
can't be large due to high number of broadcasts. L3 networks can't be
used because of VM mobility, since routing uses subnets and the IP
cannot be moved out of that subnet. Moving IP in a natively L3 network
requires installing a host route, which is an approach that can't be
scaled. Hence, a new approach is required.

Second, datacenters will span across locations, and in the hybrid cloud
case need to connect private-public domains. VM mobility across these
sites has the same issues as mobility within a datacenter, although the
technology to address it must run "over" the L3 Internet. 

Third, the need of a new approach is reinforced by multi-tenancy of
cloud networks. Native L2 networks are constrained by 4096 VLANs, which
may be too few segments to distinguish customers in a public cloud.
Further, when private and public clouds are connected, private VLAN
identities will get extended into the public domain, causing VLAN
overlap. Some method of uniquely identifying a customer in the public
cloud is needed.

Fourth, given that small, large, huge datacenters will need to be
interconnected and need to work seamlessly, the technology for the
datacenter must be agnostic of the architecture used in a specific case.
Each datacenter may architect the network differently depending on their
scale. This has been true of past of L2 (e.g. STP) and L3 (e.g. OSPF)
technologies that are topology agnostic.

Fifth, networking schemes spanning across these large domains must be
cognizant of "convergence" issues, especially arising out of VM mobility
and hardware / software failures, which will be common at a large scale.
Network convergence on mobility and failure will require control plane
signaling and forwarding plane churn. This churn needs to be minimized
in order to keep the network stable. 

Sixth, large cloud datacenters depend on economies of scale, and need to
reduce equipment cost and energy. This is possible only when the
technology scales easily both at a hardware and software level, keeping
the cost and energy consumed low. As an example, technologies that do
map-encap require per flow hardware entries, and per flow setup
signaling, which is very expensive at scale.

Seventh, move into a multi-tenant environment depends upon sufficient
"controls" for a user (preferably easy to use). Loss of control will
prevent users from using these facilities (cloud doesn't equal to
fuzzy). The "control" problem requires a standard interface across
private-public and public-public domains. This interface should
facilitate discovery and management of services, including real-time
debugging and statistics.

------------

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ronald Bonica
Sent: Friday, December 23, 2011 3:40 AM
To: Thomas Narten; So, Ning
Cc: [email-masked]
Subject: [dc] Elevator Pitch (was: Scoping the Interim meeting)

Folks,

At this point, the most helpful thing that anyone can do is to summarize
the problem(s) that we are trying to solve in a relatively terse email.
Once we come to consensus on that brief summary, we can flesh it out
into very short Internet Draft.

                                                     Happy Holidays,
                                                        Ron

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 23 Dec 2011 04:12:01 +0000

Hi,

Let me express my opinion on your first point. It's true that installing host route is not scalable in general. However, once the subnet has been extended across multiple data centers for VM mobility, you can 't use subnet for routing any more. Either MAC based forwarding or host route based routing (both of them are in the host granularity), you need to choose one. Using Mobile IP alike approach will stretch the forwarding path and hence will increase the forwarding latency. Once we choose the host route based approach, we can use the map&encap scheme like L3VPN to address the scaling issue on the P routers, and further we can even use the on-demand FIB-installation or on-demand routing announcement to address the FIB/RIB scaling issue on the PE routers. 

Best regards,
Xiaohu
.
> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Ashish
> Dalela (adalela)
> 发送时间: 2011年12月23日 11:25
> 收件人: Ronald Bonica; Thomas Narten; So, Ning
> 抄送: [email-masked]
> 主题: Re: [dc] Elevator Pitch (was: Scoping the Interim meeting)
> 
> 
>                                                      Happy
> Holidays,

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Fri, 23 Dec 2011 10:27:42 +0530

Xuxiaohu,

I'm trying to avoid getting into a specific solution at this point, as the request is to offer problem statements. 

Here are a couple of more requirements to the below:

-----------------------

Eighth, the packet forwarding scheme must be able to use all available paths to a destination (ECMP). This is for load-balancing available paths and maximizing utilization of available bandwidth. In a way that implies turning off STP because STP blocks many paths. The requirement holds true for intra-dc and inter-dc forwarding. The intra-dc is governed by server-to-server (east-west) flows, and inter-dc due to many users and bursts.

Ninth, there should be mechanisms to guarantee network SLAs in a multi-tenant environment. In particular, high traffic from one tenant should not adversely affect all other tenants on the network who are using the same shared network medium to communicate. Users entering a multi-tenant environment ought to be able to specify their network needs just as they specify compute and storage needs. VM mobility needs to take into account if sufficient network bandwidth will be available after move. It might also take into account latency.

Tenth, forwarding schemes must be optimized for unicast, broadcast and multicast traffics. In broadcast cases, this might imply broadcast control (if possible, both within and between a dc). In multicast, it implies choosing the most optimal points of packet replication to reduce bandwidth use (access, core, edge, etc). Note that multicast replication points may need to change with VM mobility.

-----------------------

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Friday, December 23, 2011 9:42 AM
To: Ashish Dalela (adalela); Ronald Bonica; Thomas Narten; So, Ning
Cc: [email-masked]
Subject: Re: [dc] Elevator Pitch (was: Scoping the Interim meeting)

Hi,

Best regards,
Xiaohu
.

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 23 Dec 2011 06:08:58 +0000

> -----邮件原件-----
> 发件人: Ashish Dalela (adalela) [mailto:[email-masked]]
> 发送时间: 2011年12月23日 12:58
> 收件人: Xuxiaohu; Ronald Bonica; Thomas Narten; So, Ning
> 抄送: [email-masked]
> 主题: RE: [dc] Elevator Pitch (was: Scoping the Interim meeting)
> 
> Xuxiaohu,
> 

Yes, I see. I'm also trying to avoid mentioning solutions, especially those solutions to be invented. 

Xiaohu

> Eighth, the packet forwarding scheme must be able to use all available paths to
> a destination (ECMP). This is for load-balancing available paths and maximizing
> utilization of available bandwidth. In a way that implies turning off STP because
> STP blocks many paths. The requirement holds true for intra-dc and inter-dc
> forwarding. The intra-dc is governed by server-to-server (east-west) flows, and
> inter-dc due to many users and bursts.
> 
> Ninth, there should be mechanisms to guarantee network SLAs in a
> multi-tenant environment. In particular, high traffic from one tenant should not
> adversely affect all other tenants on the network who are using the same
> shared network medium to communicate. Users entering a multi-tenant
> environment ought to be able to specify their network needs just as they
> specify compute and storage needs. VM mobility needs to take into account if
> sufficient network bandwidth will be available after move. It might also take
> into account latency.
> 
> Tenth, forwarding schemes must be optimized for unicast, broadcast and
> multicast traffics. In broadcast cases, this might imply broadcast control (if
> possible, both within and between a dc). In multicast, it implies choosing the
> most optimal points of packet replication to reduce bandwidth use (access,
> core, edge, etc). Note that multicast replication points may need to change
> with VM mobility.
> 
> -----------------------
> 
> Thanks, Ashish
> 
> 
> Xuxiaohu
> Let me express my opinion on your first point. It's true that installing host route
> is not scalable in general. However, once the subnet has been extended across
> multiple data centers for VM mobility, you can 't use subnet for routing any
> more. Either MAC based forwarding or host route based routing (both of them
> are in the host granularity), you need to choose one. Using Mobile IP alike
> approach will stretch the forwarding path and hence will increase the
> forwarding latency. Once we choose the host route based approach, we can
> use the map&encap scheme like L3VPN to address the scaling issue on the P
> routers, and further we can even use the on-demand FIB-installation or
> on-demand routing announcement to address the FIB/RIB scaling issue on the
> PE routers.
> 
> Best regards,
> Xiaohu
> .
> > -----邮件原件-----

---

From: "Mcdysan, David E" <[email-masked]>
Date: Fri, 23 Dec 2011 07:59:28 -0500

Pitch on an Elevator in a shorter building. ;)

Support an infrastructure for multiple tenants, each with computing
applications and associated resources ranging from a single VM to a very
large tenant with a complex, multi-site computing application. An
important case is supporting a very large single tenant computing
application.

Ensure that each tenant is securely separated from others and that only
tenants who mutually agree to communicate and/or share resources can do so.

Orchestrate all of the virtual and real appliances (e.g., firewalls. load
balancers, security, etc.) needed for a complex computing application
distributed across multiple sites.

Provide methods to optimize the placement and interconnection of
computing, storage, appliance and networking resources.

Work across a diverse set of networking technologies and architectures,
ranging from dedicated special-purpose networks (e.g., Ethernet/PBB L2/L3
VPNs) to access over the Inernet.

Be scalable on the high end to support hundreds of millions of Virtual
Machines -- think one VM pre wireless subscriber.

Be capable of meeting the quality objectives across a range of performance
profiles corresponding to classes of tenants.

Hope this helps,

Dave

On Thursday12/22/11 5:10 PM, "Ronald Bonica" <[email-masked]> wrote:

>Once we come to consensus on that brief summary, we can flesh it out into
>very short Internet Draft.
>

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 23 Dec 2011 09:24:24 -0600

Thanks, Dave.  This helps a lot.

Would you please help a bit more, by (a) suggesting whether each "piece" of this should be done in the IETF or elsewhere, and (b) commenting on whether existing solutions, to the extent they exist, don't meet your requirements?  I think you gave some useful hints, but I'd like to see if we can get more explicit.

For example, is orchestration simply something that is done in proprietary software and/or standardized in existing forums (DMTF, OCCI/OGF, OpenStack, etc.)?  Is the interconnection requirement satisfied by a combination of L2VPN and L3VPN technology, driven by orchestration platforms?

(Please note: I'm not suggesting that we already have the right tools. Rather, I'm trying to find a good description of the gaps.)

Cheers,
-Benson

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 23 Dec 2011 13:02:13 -0500

I'd be interested in knowing if large [cloud] data center operators would want to run a virtual LAN active-active across multiple hub sites except under some unusual circumstances.

Given need for managing scaling, inter-site bandwidth, etc, the following is how I would guess a scalable elastic environment might need to look:

1) Use L3-only overlay-based solution for VM-based customers/services that require spoke<->hub workload migration and that DO NOT require L2-based services.
2) Keep any single virtual LAN (or "l3-group" if l3-only) pinned to a single hub site.  A virtual LAN (or l3-group) would be assigned an IP subnet prefix and members would be assigned an address from that prefix.  A gateway at the site where the subnet is active would advertise the subnet into the core network.
3) Each hub site may further have sub-domains into which subnets may be pinned under normal circumstances.
4) Assign subnets to hub sites based on site capacity and other requirements (latency, etc).
5) Ensure that hub sites have sufficient capacity to handle the failure of peer hub site.
6) If all the members of a subnet cannot be hosted at a hub site for any reason, then move the entire subnet to another hub site.
7) If a site fails, move it's subnets to another hub site.

---

From: Benson Schliesser <[email-masked]>
Date: Fri, 23 Dec 2011 12:18:27 -0600

Hi, Aldrin.

Thanks. There are several good comments in your message that probably deserve more exploration.  But one of them catches my attention immediately:  Why do you assume that an overlay should be a "L3-Group" and not support L2?

To elaborate:  If the underlying mechanism (i.e. IP in the case of X-over-IP) provides the architectural benefits you need, and we can map inner packets (i.e. X in the case of X-over-IP) into the overlay based on either destination IP or MAC address, then does it really matter what kind of service it provides (L2 vs L3)?  I suppose that L2 services might have different bandwidth implications, given the presence of broadcast traffic etc. But I don't see how it materially affects the protocol architecture.

(That said:  I'm admittedly biased, as a collaborator, but I think the phrase from Thomas Narten's NVO3 presentation is quite nice: "L3 when you can, L2 when you must.")

Thoughts?

Cheers,
-Benson

---

From: Thomas Narten <[email-masked]>
Date: Fri, 23 Dec 2011 19:23:36 -0500

Hi Dave.

Thanks for this. Let me see if I can summarize and further tease this
out.

> important case is supporting a very large single tenant computing

I think you are stating that a key requirement is support for
multi-tenancy.  (And I would assume pretty much everyone agrees this
is a requirement...)

> tenants who mutually agree to communicate and/or share resources can

This sounds like sub-bullets under the definition of multi-tenancy.

> distributed across multiple sites.

This sounds like something very different and potential very big. By
"orchestration" do you mean the system by which you can place or move VMs and
all their associated dependent devices (FWs, LBs, etc.) around? How
you place them, how you move components around?

If so, this seems to me to be a combination of a (possibly)
centralized controller (of sorts), with a lot of individual components
distributed around the DC that provide necessary hooks/mechanisms for
the orchestrator to do its job. I.e., moving a VM (and a FW) requires
a number of steps. The orchestrator (at a high level) initiates those
steps, but there are other network components that also play a role
(the hypervisor, switches, etc.)

Where do you see the standardization gaps in the above? What would be
some example areas where IETF (or some other SDO) work is needed?

> computing, storage, appliance and networking resources.

What would the standardization angle of this be? (FWIW, I don't see
one right off, but I probably just don't understand what is
meant/needed here).

> VPNs) to access over the Inernet.

This is a sub-bullet of the orchestration requirements?

> Machines -- think one VM pre wireless subscriber.

Is this a sub-requirement of orchestration? or multi-tenancy? or both?

> profiles corresponding to classes of tenants.

Could you elaborate on this? Is this about meeting SLAs? What is the
standardization angle (i.e., where work is needed, where there is a
gap today).

Thomas

---

From: Aldrin Isaac <[email-masked]>
Date: Sat, 24 Dec 2011 13:27:09 -0500

Hi Benson,

Looking beyond the encapsulation, spoke<->hub site VM migration
service is, IMO, a WAN service for which LAN behavior (overlay or
otherwise) may not be a good fit.  That said, L3-only solutions appear
to be realistic only for VM-only environments.  Speaking for myself
today, a realistic vpn4dc solution would support a mix of managed and
colocated virtual and physical systems in the same virtual LAN.

Best -- aldrin

---

From: Bhumip Khasnabish <[email-masked]>
Date: Sat, 24 Dec 2011 13:28:25 -0500

·         Support of on-demand provisioning and management (availability)
of distributed virtualized computing, communications, and storage resources

·         Offering of seamless mobility of virtual machine (VM),
virtualized interface (VI), virtualized network elements (VNE) and
virtualized storage (VS) from  one domain to another for load balancing and
service continuity management

·         Management of security, privacy, data integrity, and log of Data
center services for audit and verification

·         Support of Distributed scaling and multi-tenancy support without
excessive complexity and resource consumption

·         Support of Naming and addressing of virtualized entities for
simplified mobility and service continuity management

·         Support of Realistic Service level agreement parameters for Data
center resources and services

* Support of Visualization of resources utilization and automation of
provisioning

Thanks for you kind attention.

Happy Holidays and all the best for 2012.

Bhumip

---

From: Russ White <[email-masked]>
Date: Sun, 25 Dec 2011 18:22:01 -0500

> scaled. Hence, a new approach is required.

Two question...

1. How many host routes do we reach before we say it "won't scale?" If
you're talking about leaf nodes in a routing protocol, 100k isn't really
that big of a deal. Where you do run into problems is between the
routing and forwarding tables --but that's an area for thought.

2. Why does this mobility need to be at layer 2 specifically? Are we
assuming DDNS and other sorts of solutions in this space will simply
never be fast enough/scale far enough/etc?

> in order to keep the network stable. 

Convergence speed and stability are generally opposed to one another,
but there are techniques that allow reasonable convergence speed for a
confined set of network topology changes while providing stability on
the back end by slowing down for subsequent changes.

> signaling, which is very expensive at scale.

I think the major challenge here is going to be forwarding table size
--but we have to balance against going the "traditional" path of
caching, or making the forwarding path dependent on the data being
forwarded. Anything where you're waiting on data to flow in order to
build a control plane entry is going to be very difficult to reconcile
with fast convergence and mobility --we faced this problem with reactive
protocols in the MANET space, and never really found a solid way to
overcome them (at least that I know of).

:-)

Russ

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 26 Dec 2011 01:43:49 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Russ White
> 发送时间: 2011年12月26日 7:22
> 收件人: [email-masked]
> 主题: Re: [dc] Elevator Pitch
> 
> 

Fully agree.

> never be fast enough/scale far enough/etc?

Good question.

> overcome them (at least that I know of).

Fully agree. The approach of making the forwarding path dependent on the data being
forwarded will probably open a can of worms.

Best regards,
Xiaohu

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Mon, 26 Dec 2011 10:08:30 +0530

Russ,

1. The scaling numbers for a single DC are expected to be 1-100 million.
It will reach the higher limits with 10G to the server. The numbers for
DC connectivity are not well-known. The typical access switch numbers
are much lower than 100K. E.g. if you are running OSPF with static
L3/VLAN configuration how much do you really need? A drastic increase in
table sizes at access is cost prohibitive.

2. I did not assume that mobility has to be L2, just that today's L2
will give mobility without any change. Today's L3 will not give that.
So, I'm trying to state a problem that this group needs to work upon.
Whether DDNS is a right approach, or other approaches, we can discuss
once we can agree that we have a problem to be solved and the problem is
clearly defined.

Your point about challenges with reactive protocols are spot-on I think.

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Russ
White
Sent: Monday, December 26, 2011 4:52 AM
To: [email-masked]
Subject: Re: [dc] Elevator Pitch

Two question...

:-)

Russ

---

From: Aldrin Isaac <[email-masked]>
Date: Sun, 25 Dec 2011 23:45:10 -0500

> the back end by slowing down for subsequent changes.

Hi Russ,

The network may have equipment with varying degrees of control-plane horsepower.  What if the "resting" update rate should become greater than the minimum update processing rate available across the network to maintain stability?  Could you elaborate on what kinds of scaling tools/techniques you have in mind?

Best -- aldrin

---

From: DIEGO LOPEZ GARCIA <[email-masked]>
Date: Mon, 26 Dec 2011 08:59:26 +0100

Hi,

> profiles corresponding to classes of tenants.

I'd add a couple of mentions:

1) To align these solutions with the interfaces in use and/or development by cloud providers and the user community (DMTF, OCCI/OGF, SCIM…)

2) To IPv6 technologies as a possible source of solutions for some of the current problems, and certainly as part of the networking scenario to be addressed.

All the best,

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D

e-mail: [email-masked]
Tel:      +34 913 129 041
Mobile: +34 682 051 091
-----------------------------------------

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at.
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

---

From: Tina TSOU <[email-masked]>
Date: Mon, 26 Dec 2011 08:12:14 +0000

Tina

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of DIEGO LOPEZ GARCIA
Sent: Sunday, December 25, 2011 11:59 PM
To: [email-masked]
Subject: Re: [dc] Elevator Pitch (was: Scoping the Interim meeting)

Hi,

1) To align these solutions with the interfaces in use and/or development by cloud providers and the user community (DMTF, OCCI/OGF, SCIM.)

[Tina:
IPv6 is easily added through a managed NAT service, no need for complicated Cloud networking, no need for tunnels.
IPv6 will be available for Cloud users...and Cloud Operators.
]

All the best,

Dr Diego R. Lopez
Telefonica I+D

e-mail: [email-masked]
Tel:      +34 913 129 041
Mobile: +34 682 051 091

---

From: Thomas Narten <[email-masked]>
Date: Wed, 28 Dec 2011 11:50:07 -0500

Hi Bhumip.

This post (and a few others) seem to me to be providing a laundry list
of buzz words and generic "requirements".

What this group needs to do is identify concrete problems for which
lack of a standard is preventing solutions.

For the items mentioned above, what is the actual concrete standards
work that needs to be done?

Bhumip Khasnabish <[email-masked]> writes:

> · Support of on-demand provisioning and management (availability) of
>           distributed virtualized computing, communications, and
>           storage resources

This is too broad and high-level to be helpful. That is, this comes
across as a motherhood-and-apple-pie wish list. We need to drill down
into something much more concrete. 

What standards are lacking today that prevent the above from being
possible?

How does the above translate into gaps that the IETF needs to fill?

> · Offering of seamless mobility of virtual machine (VM), virtualized
>           interface (VI), virtualized network elements (VNE) and
>           virtualized storage (VS) from one domain to another for load
>           balancing and service continuity management

Same comment again. Also, what is a "domain" above?

Also, there are already proprietary systems/products that provide VM
mobility. What is it that the IETF needs to do?

>           center services for audit and verification

Again, broad wish list. How does this translate into specific gaps
that the IETF needs to fill?

>           excessive complexity and resource consumption

Ditto.

>           simplified mobility and service continuity management

Ditto

>           center resources and services

Ditto.

> * Support of Visualization of resources utilization and automation of

Ditto.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 28 Dec 2011 12:00:46 -0500

> routing and forwarding tables --but that's an area for thought.

Actually, 100K host routes in leaf nodes may well be a Big Deal. It
depends on what type of devices need to understand host routes.

For the scenarios NVO3 is looking at, the edge device includes edge
switches and hypervisors. Yes, such devices can in theory support 100K
host routes (and the routing protocol needed to make that converge).

But at a cost.

And I suspect the cost is too high.

Do you think data center operators will want to run routing protocols
in hypervisors that have to handle 100K host routes? Especially if
such devices will need to be upgraded with better hardware to support
such a load?

Or will they want an alternative approach?

> never be fast enough/scale far enough/etc?

Like it or not, the key requirement for VM mobility is that the VM's
IP address does not change. That means the VM can't really move from
one IP subnet to another. That means either moving to bigger and
bigger L2s (all under one IP subnet) as the DC expands or the need to
inject /32 host routes.

Neither of those approaches seems particularly scalable/desirable if
you look 10 years down the road and think of 1M+ physical machines in
a DC.

Thomas

---

From: Bhumip Khasnabish <[email-masked]>
Date: Wed, 28 Dec 2011 20:40:52 -0500

Hello Thomas,

Yes, these are high-level very important one-liner data center related
topics/issues.

We have a draft that covers
SDO survey (http://tools.ietf.org/html/draft-khasnabish-cloud-sdo-survey-01).
This
is basically a summary of which SDO is trying to do what and what IETF can
do.
We need comments and suggestions from you and others to update this doc.

We also have another draft covering potential work items
(
http://tools.ietf.org/html/draft-khasnabish-cloud-industry-workitems-survey-01
).

We can discuss these further in the interim mtg.

In my email, the domain means admin. domain, apps/service domain,
resource domain, etc
There are many proprietary and app-/service-specific interim (neither
scalable nor distributed) solution for many of the problems that I mention,
but IETF
can prioritize and focus on a few and can help develop
scalable/distributed/interoperable solution, best practices, etc.

We can develop a template to prepare a list of topics and issues
for each of the items that IETF can contribute to.

Any other suggestions?

Thanks again.

Best.

Bhumip

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Wed, 28 Dec 2011 21:59:42 -0800

Greetings Bhumip,

This may sound harsh, and I don't want to diminish the amount of hard work that went into the documents listed below, but I think we may need to do a bit of a reset in this conversation.  Comments in line....

> is basically a summary of which SDO is trying to do what and what IETF can

I'm not sure of the value of this survey, to be frank.  It lists a number of SDO's that may be involved in "cloud" in some way.  Actually, looking at the list, I see very few that are doing anything that is actually driving the industry at this time.  More of the development in the industry of standards is coming from non-SDO or new semi-SDO organizations and vendors/providers of services.  For example, Amazon has published a set of API's that a number of other entities are busy implementing, ONF is busy standardizing the OpenFlow protocol, OpenFlow operational models, and a management framework.  OpenStack is busy defining a complete "cloud" operating environment and stack.  None of them show up in the SDO document, but, to be frank, are probably having a greater impact in this area than the majority of the SDO's listed in the document.  There are other examples if one wants to dig.

Secondly, the question we should be asking is "What does the IETF NEED to do" not "What can the IETF do"  We can do a great many things, the bulk of which will not be helpful or a good use of resources.

> http://tools.ietf.org/html/draft-khasnabish-cloud-industry-workitems-survey-01

Again, I'm not sure of the value of this document.  What is the source for these "work items"?  What real or near-to-mid term problems are they trying to address?  What ones are relevant to the IETF?  I basically see a catch-all list of a number of things that someone somewhere has thought might be a problem that may need to be solved in a "cloud" environment.  Some of the real issues I hear operators talking about (including myself if I channel myself from 8 months ago) (not theoretical, but "I have this problem now or will soon" are somehow covered in some of the work items - but the majority of work items listed don't really cover what I've heard of as current problems.  Even the work items that cover existing problems are very broad, and cover MUCH more than the real issues that are being discussed in the operational community.

> We can discuss these further in the interim mtg.

I'm not sure how fruitful such a conversation might be.  A boil-the-ocean interim meeting won't produce concrete actionable directions unless we are VERY focused in the conversation - not an IETF strong suit.

I would like to propose a different approach, if I may.  If we took a focused set of problem statements and ran them through the following set of filters:

1) Is this a current/real or near-to-mid term probable issue and is it substantial?
2) If yes, is it being adequately covered by another SDO and is it in that SDO's domain?
3) if no, is it in the domain of IETF competency?
4) if yes, do we want to work on it?

There are many other interesting problems outlined in the various documents that have been listed.  They could, in time be picked up by the IETF, liaised to some other SDO, or handed to the IRTF for in-depth study.   However, if we take the body of work that is being proposed, we will fail at boiling an ocean that, in many cases, we don't even live in.  We will take too long, cast too broad a net, and be bypassed before we even get WG drafts issued.

In Taipei, we heard, I think, two good sets of problem statements, those made by Tomas Narten, and Tom Nadeau.  More contributions in that vein, preferably with real operational input, would be very helpful in forming the conversation for the interim meeting.

> scalable/distributed/interoperable solution, best practices, etc.

In this we are closer in alignment, but I think we start from the operational problem side of the house, rather than the whole universe of possibilities.
> for each of the items that IETF can contribute to.

The IETF doesn't as a rule, contribute.  We either develop standards or we observe and occasionally liaise.  There is too broad a space here to be effective if we try and "contribute" to the whole problem space, in my opinion.
> Any other suggestions?

Thank you for starting the conversation,
	Chris

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 29 Dec 2011 08:55:52 +0000

Hi Thomas,

> 发送时间: 2011年12月29日 1:01
> 收件人: Russ White

If they need to be upgraded with better hardware, why not directly consider the physical edge switches (e.g., ToR switches) as leaf nodes? 

BTW, in the typical multi-tenant cloud data center environment where most of the tenants are believed to be SMBs, the route amount of any given tenant domain would not be too large. Hence, even there are more than 1M host routes in total, the route amount on any leaf node would not be too large since the tenant amount on any leaf node would not be too large. The maximum tenant amount on any leaf node is limited by at least the following two factors: 1) the port density of leave nodes; 2) the VM density of physical servers. In addition, in the case where the non-blocking DC fabric is not yet a reality due to its high cost, the VMs of a given tenant would be placed as close as possible, for example, being attached to a single leaf node if possible, so as to reduce the bandwidth consumption on the uplinks. In this way, the average tenant amount on a given leaf node would be even less.

> bigger L2s (all under one IP subnet) as the DC expands or the need to

In the DCI scenario where the PE routers are usually performed at the aggregation SWs or even core SWs, the PE routers would need a much large forwarding table. Provided the routing table containing millions of entries, which is available on most today's high-end routers, was still not large enough, the on-demand FIB installation or on-demand route announcement mechanisms can be used further to scale the solution. Note that the trigger for the FIB installation or route announcement is ARP request packets rather than data packets. Hence it will not cause the so-called initial packet loss or latency issue.

> you look 10 years down the road and think of 1M+ physical machines in

Maybe we should also take the development speed of routing/switching chip and CPU technologies into account:)

Best regards,
Xiaohu

---

From: Russ White <[email-masked]>
Date: Thu, 29 Dec 2011 10:08:28 -0500

> depends on what type of devices need to understand host routes.

So let me try to abstract the problem a little... There are two things
that cost money in building a network:

1. State --the more you carry, the more hardware is going to cost.
2. Bandwidth --the less optimal your bandwidth utilization, the more
you're spending on something that's not used.

The first rule of all network design is that the more state you put in a
network, the more optimal your bandwidth usage. To reduce the cost of
bandwidth, you have to spend money on state. To reduce the cost of
state, you must increase your spending on bandwidth.

The second of network design is the balance point where the network is
cheapest is different in every network.

So, IMHO, the point of protocol design should be to allow the design the
maximum amount of flexibility and granularity in when and where to hide
information, to make the right tradeoff for any particular network.

Now, to return to the discussion at hand: If you don't want to pay for
devices that will handle 100 million routes, but you have 100 million
devices, then you're going to have some degree of suboptimal bandwidth
utilization. There's simply no way around this reality.

IMHO, however we resolve this problem, we need to resolve it in a way
that allows an operator to support 100 million routes in every node, if
they choose to optimize for bandwidth at the cost of hardware. Or to
support 1 route in every node using really cheap hardware, if they want
to optimize for state. Or for any point in the sliding scale in between
(within reason).

So this control plane needs to:

1. Be able to support 100 million destinations at the host level (that
doesn't mean 100 million paths, but only 100 million destinations --two
different problems).

2. Be able to aggregate to hide information at anyplace that's logical
within the network.

BTW, some folks would like to solve this problem by making the control
plane react to the data plane --but this carries it's own baggage in
complexity and in operational capabilities. Reactive control planes
always converge more slowly, and waste bandwidth at a rate that's
arguably higher than simple aggregation. So there's no "silver bullet"
waiting in the wings in the form of caching at the control plane level.

> bigger L2s (all under one IP subnet) as the DC expands or the need to

Again, the same tradeoff as above --moving to a bigger l2 domain also
means losing the ability to optimally direct traffic through the network
(unless you put another control plane on top of the l2 and l3 control
planes already in existence --which just adds the complexity you're
trying to get away from in the host routes back into the network state!).

> you look 10 years down the road and think of 1M+ physical machines in

There is no "particularly desirable" solution, in reality, because
building a network with no control plane state anyplace that uses
bandwidth in a perfectly optimal way is simply impossible no matter how
you slice it (unless you're going to go into quantum routing!).

:-)

Russ

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Thu, 29 Dec 2011 09:20:03 -0800

Greetings Xuxiaohu,

> BTW, in the typical multi-tenant cloud data center environment where most of the tenants are believed to be SMBs, the route amount of any given tenant domain would not be too large. Hence, even the…

There are folks who are looking at the ToR as the leaf node, and, in my opinion, any solution will probably be a hybrid of ToR and hypervisor.  However, remember that the hypervisor switch is still the first aggregation point, and therefore, is the leaf in a network tree.  Some level of state aggregation (and accompanying loss) will occur at the hypervisor switch, making it the leaf.  Also, there are scaling issues with having 10's (or 100's in the case of some operators) vm's running on each server, and having 20-40 servers running into the ToR - all of a sudden, the host-table alone on the ToR becomes mildly interesting, even without other (intra-data center) routes on low CAM ToR switches.  Not all solutions will map to all (or most) customers.

Also, the comment to limit the route-tabel of tenants due to small numbers of hosts may not always (or usually) match reality.  What of the cross-data-center traffic for other services either provided by the hoster, or other tenants?  Continuing to route ALL of that traffic through routing/switching nodes external to the "fabric" will lead to it's own scaling issues.   Don't assume that the only routes that an SMB needs are it's own.

> Maybe we should also take the development speed of routing/switching chip and CPU technologies into account:)

It's more a question of cost/performance on off-chip memory/TCAMs.  That is a slightly different curve :)

	Chris

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Thu, 29 Dec 2011 09:27:55 -0800

Greetings Russ et.al.,

> state, you must increase your spending on bandwidth.

Agreed, mostly.  The question is where the state resides, control plane or data plane (do you differentiate between possible state and active state).  Control plane "possible state" should be cheaper (by orders of magnitude) than data plane "active state" - however, transitioning between the two is a complex problem (that has gotten somewhat easier as we have gotten smarter about distributed systems over time)
> cheapest is different in every network.

agreed

> information, to make the right tradeoff for any particular network.
agreed

> utilization. There's simply no way around this reality.

agreed
> to optimize for state. Or for any point in the sliding scale in between

agreed

> doesn't mean 100 million paths, but only 100 million destinations --two

yes

> 2. Be able to aggregate to hide information at anyplace that's logical

yes

> waiting in the wings in the form of caching at the control plane level.

however, I believe that aggregation time may be coming down, especially in a more homogenous environment (like a DC) vs a heterogeneous environment (like the DFZ).

> trying to get away from in the host routes back into the network state!).

Or replacing one or two of those L2/L3 control planes with a control plane with a global view, which is easier to do in a constrained network like a traffic engineering core or a data center.  It's is a bounded problem.  As you stated, however, convergence MAY be slower, but I would argue that converging a network of Nx10K switching/routing nodes takes a non-trivial amount of time as well :)

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Thu, 29 Dec 2011 09:27:55 -0800

Greetings Russ et.al.,

agreed

agreed

agreed

agreed

yes

yes

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Thu, 29 Dec 2011 09:27:55 -0800

Greetings Russ et.al.,

agreed

agreed

agreed

agreed

yes

yes

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 29 Dec 2011 16:21:57 -0500

Hi Christopher,

> Or replacing one or two of those L2/L3 control planes with a control plane with a global view, which is easier to do in a constrained network like a traffic engineering core or a data center.  It's…

Wrt your responses above, are you speaking specifically regarding data centers where end stations are 100% VM?  Could you lend your thoughts on how non-aggregating DCs ought to interwork with an aggregating global WAN/Internet? 

Thanks -- aldrin

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Thu, 29 Dec 2011 13:27:05 -0800

Greetings Aldrin,

> Wrt your responses above, are you speaking specifically regarding data centers where end stations are 100% VM?  Could you lend your thoughts on how non-aggregating DCs ought to interwork with an ag…

No, I was not assuming a 100% VM datacenter (which I think is unlikely in the near-to-mid term - there will be appliances, storage, etc for sometime that is not behind a HV).  However, I'm uncertain by what you mean a "non-aggregating" data center.  Any data center will, by nature aggregate, the question is where, and how many layers of aggregation?  Leaf (hypervisor and/or ToR for non HV elements), ToR, spine, core router?  I seriously doubt that any DC will expose it's enitire fleet of /32 (/128) hosts to the Internet (or a global WAN).   Some level of aggregation will be necessary.  As to how a DC would interact with another DC or a WAN backbone in a controller-based network, one could envision each DC as a network element that aggregates into the WAN with, potentially, cooperating controllers, or a controller hierarchy.  Other topologies are likely to exist, as well.
	Chris

---

From: Christopher LILJENSTOLPE <[email-masked]>
Date: Thu, 29 Dec 2011 13:30:35 -0800

Another clarification...

> Wrt your responses above, are you speaking specifically regarding data centers where end stations are 100% VM?  Could you lend your thoughts on how non-aggregating DCs ought to interwork with an ag…

Actually, I was making the case that, at large scale, convergence takes time, if it is distributed as it is today (STP, IGPs) or in a controller-based environment.  The convergence time question becomes more interesting, as it's a large computation problem, no mater if it is distributed or centralized.  Centralized may be faster, but there is latency between forwarding plane and control plane.  Less latency in the distributed model, but more computation elements that must converge.  

	Chris

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 30 Dec 2011 04:01:51 +0000

> -----邮件原件-----
> 发件人: Christopher LILJENSTOLPE [mailto:[email-masked]]
> 发送时间: 2011年12月30日 1:20
> 收件人: Xuxiaohu
> 抄送: Thomas Narten; Russ White; [email-masked]
> > If they need to be upgraded with better hardware, why not directly consider
> the physical edge switches (e.g., ToR switches) as leaf nodes?
> >
> > BTW, in the typical multi-tenant cloud data center environment where most
> of the tenants are believed to be SMBs, the route amount of any given tenant
> domain would not be too large. Hence, even there are more than 1M host
> routes in total, the route amount on any leaf node would not be too large since
> the tenant amount on any leaf node would not be too large. The maximum
> tenant amount on any leaf node is limited by at least the following two factors:
> 1) the port density of leave nodes; 2) the VM density of physical servers. In
> addition, in the case where the non-blocking DC fabric is not yet a reality due to
> its high cost, the VMs of a given tenant would be placed as close as possible,
> for example, being attached to a single leaf node if possible, so as to reduce the
> bandwidth consumption on the uplinks. In this way, the average tenant amount
> on a given leaf node would be even less.
> 
> There are folks who are looking at the ToR as the leaf node, and, in my opinion,
> any solution will probably be a hybrid of ToR and hypervisor.  However,
> remember that the hypervisor switch is still the first aggregation point, and
> therefore, is the leaf in a network tree.  Some level of state aggregation (and
> accompanying loss) will occur at the hypervisor switch, making it the leaf.
> Also, there are scaling issues with having 10's (or 100's in the case of some
> operators) vm's running on each server, and having 20-40 servers running into
> the ToR - all of a sudden, the host-table alone on the ToR becomes mildly
> interesting, even without other (intra-data center) routes on low CAM ToR
> switches.  Not all solutions will map to all (or most) customers.
> 
> Also, the comment to limit the route-tabel of tenants due to small numbers of
> hosts may not always (or usually) match reality.  What of the
> cross-data-center traffic for other services either provided by the hoster, or
> other tenants?  Continuing to route ALL of that traffic through
> routing/switching nodes external to the "fabric" will lead to it's own scaling
> issues.   Don't assume that the only routes that an SMB needs are it's own.

Hi Chris,

Would you please give a concrete example where the communication between different tenants is very common in the multi-tenant cloud data center?

Best regards,
Xiaohu

> > In the DCI scenario where the PE routers are usually performed at the
> aggregation SWs or even core SWs, the PE routers would need a much large
> forwarding table. Provided the routing table containing millions of entries,
> which is available on most today's high-end routers, was still not large enough,
> the on-demand FIB installation or on-demand route announcement
> mechanisms can be used further to scale the solution. Note that the trigger for
> the FIB installation or route announcement is ARP request packets rather than
> data packets. Hence it will not cause the so-called initial packet loss or latency
> issue.
> >

---

From: Aldrin Isaac <[email-masked]>
Date: Thu, 29 Dec 2011 23:31:41 -0500

Hi Chris, 

I may have misunderstood your statement, "I believe that aggregation time is coming down", to mean you are a proponent of dramatically reducing or eliminating route aggregation in some way.  Maybe I'm wrong, but it seems there are at least a few folks that expect that hosts should be independently mobile across more than one site and that a solution should support that expectation at scale.

I think we're on the same page wrt needing some levels of aggregation.  As a matter of fact, the run-of-the-mill DC network design already has aggregation at the access gateway/router (in the form of subnet routes) and "host routes" (mac routes) within the access LAN.  You can make the access LAN (host route domain) smaller or bigger (hopefully not relying on STP) for varying degree of mobility.  

Could you highlight, from your perspective, the features current standards and design patterns cannot fundamentally support?  Since we're familiar with the limitations with STP, flooding and ARP for large LANs, I'm interested in knowing, in technical terms, what other flaws you believe exist and features you believe are missing.

Thanks -- aldrin

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 30 Dec 2011 09:45:53 +0000

> -----邮件原件-----
> 发件人: Xuxiaohu
> 发送时间: 2011年12月30日 12:17
> 收件人: 'Christopher LILJENSTOLPE'
> 主题: re: [dc] Elevator Pitch
> 
> 
> > -----邮件原件-----

Hi Chris,

By the way, in the MAC over IP solution, the forwarding table of the leaf nodes contains the MAC routes to hosts and the gateways, similarly, in the IP over IP solution, the forwarding table of leaf nodes contains the routes to hosts and one or more default route to the gateways. If you believe the default route directed to the gateway in the IP over IP solution is not enough for forwarding the cross-data-center traffic, does that mean the MAC over IP solution is totally unworkable since there is only one MAC route directed to the gateway for forwarding the cross-data-center traffic?

Best regards,
Xiaohu

> Hi Chris,
> 
> Would you please give a concrete example where the communication between
> different tenants is very common in the multi-tenant cloud data center?
> 
> Best regards,
> Xiaohu
> 
> > >
> > >
> > > Maybe we should also take the development speed of routing/switching
> chip
> > and CPU technologies into account:)
> >
> > It's more a question of cost/performance on off-chip memory/TCAMs.  That
> is
> > a slightly different curve :)
> >
> > 	Chris
> >
> > >
> > > Best regards,
> > > Xiaohu
> > >
> > >> Thomas
> > >>

---

From: Bhumip Khasnabish <[email-masked]>
Date: Fri, 30 Dec 2011 12:56:24 -0500

Hello Chris,

Thank you very much for your comments and suggestions.

If you do not mind, please provide write ups, as appropriate, on Amazon
API, ONF, OpenFlow, Open Stack, etc.
that you mention below for inclusion in the next version of the SDO survey
draft.
For the work item survey draft, the intention is to provide a list. We have
seen many
followup drafts during the last two IETF mtgs articulating problems and
possible
solutions  (VDCS, VPN4DC, VEPC, VPN-O-CS, VPN-O-DCS, VRM, security framework
for VDCS, etc.) from both service providers and solution providers.
We'll be updating this draft soon, and can work on providing priority, if
that is helpful.

If we want to work on one near-term high-priority work item, may be
virtual machine (and virtual network element) mobility and interconnection
can be the focus..
(may be we need to use distributed control, distributed hash, etc.)

Any others?

Thanks and best wishes.

Bhumip

On Thu, Dec 29, 2011 at 12:59 AM, Christopher LILJENSTOLPE <
[email-masked]> wrote:

> Greetings Bhumip,
>
> This may sound harsh, and I don't want to diminish the amount of hard work
> that went into the documents listed below, but I think we may need to do a
> bit of a reset in this conversation.  Comments in line....
>
> > SDO survey (
> http://tools.ietf.org/html/draft-khasnabish-cloud-sdo-survey-01).
> > This
> I'm not sure of the value of this survey, to be frank.  It lists a number
> of SDO's that may be involved in "cloud" in some way.  Actually, looking at
> the list, I see very few that are doing anything that is actually driving
> the industry at this time.  More of the development in the industry of
> standards is coming from non-SDO or new semi-SDO organizations and
> vendors/providers of services.  For example, Amazon has published a set of
> API's that a number of other entities are busy implementing, ONF is busy
> standardizing the OpenFlow protocol, OpenFlow operational models, and a
> management framework.  OpenStack is busy defining a complete "cloud"
> operating environment and stack.  None of them show up in the SDO document,
> but, to be frank, are probably having a greater impact in this area than
> the majority of the SDO's listed in the document.  There are other examples
> if one wants to dig.
>
> Secondly, the question we should be asking is "What does the IETF NEED to
> do" not "What can the IETF do"  We can do a great many things, the bulk of
> which will not be helpful or a good use of resources.
>
> Again, I'm not sure of the value of this document.  What is the source for
> these "work items"?  What real or near-to-mid term problems are they trying
> to address?  What ones are relevant to the IETF?  I basically see a
> catch-all list of a number of things that someone somewhere has thought
> might be a problem that may need to be solved in a "cloud" environment.
>  Some of the real issues I hear operators talking about (including myself
> if I channel myself from 8 months ago) (not theoretical, but "I have this
> problem now or will soon" are somehow covered in some of the work items -
> but the majority of work items listed don't really cover what I've heard of
> as current problems.  Even the work items that cover existing problems are
> very broad, and cover MUCH more than the real issues that are being
> discussed in the operational community.
>
> > ).
> >
> I'm not sure how fruitful such a conversation might be.  A boil-the-ocean
> interim meeting won't produce concrete actionable directions unless we are
> VERY focused in the conversation - not an IETF strong suit.
>
> I would like to propose a different approach, if I may.  If we took a
> focused set of problem statements and ran them through the following set of
> filters:
>
> There are many other interesting problems outlined in the various
> documents that have been listed.  They could, in time be picked up by the
> IETF, liaised to some other SDO, or handed to the IRTF for in-depth study.
>   However, if we take the body of work that is being proposed, we will fail
> at boiling an ocean that, in many cases, we don't even live in.  We will
> take too long, cast too broad a net, and be bypassed before we even get WG
> drafts issued.
>
> In Taipei, we heard, I think, two good sets of problem statements, those
> made by Tomas Narten, and Tom Nadeau.  More contributions in that vein,
> preferably with real operational input, would be very helpful in forming
> the conversation for the interim meeting.
>
> >
> In this we are closer in alignment, but I think we start from the
> operational problem side of the house, rather than the whole universe of
> possibilities.
> >
> The IETF doesn't as a rule, contribute.  We either develop standards or we
> observe and occasionally liaise.  There is too broad a space here to be
> effective if we try and "contribute" to the whole problem space, in my
> opinion.
> >
> > Any other suggestions?
> >
>

-- 
Best.

Bhumip Khasnabish
[email-masked]
[email-masked]
+1-781-752-8003 (mobile)
http://tinyurl.com/bhumip

                   __o
             _ `\ <, _
.......... ( • ) / ( • ) ......................

---

From: Xuxiaohu <[email-masked]>
Date: Sat, 31 Dec 2011 02:33:42 +0000

Hi all,

Since there are some differences in the problems and requirements between data center network (DCN) and data center interconnect (DCI), I try to list several problems and requirements for DCN and DCI separately as follows. Here the data centers mainly refer to those multi-tenant data centers which are operated by public cloud providers to deliver cloud service (i.e., IaaS) to their customers (i.e., tenants).

1. DCN problems and requirements:

1) VM mobility across multiple pods -> LAN/subnet extension across pods

2) Some cluster applications use non-IP or link-local multicast (optional) -> Layer2 networking

3) Multi-tenancy isolation -> VPN/VLAN instance scalability

4) Millions of VMs -> MAC/IP forwarding table scalability

5) Increasing bandwidth demands for server-to-server connectivity (i.e., east-west traffic)-> ECMP and shortest path forwarding capabilities

6) Network resiliency -> Fast convergence and multi-homing

7) Thousands of network devices -> Simplified provisioning and operation

2. DCI problems and requirements:

1) VMs mobility across data centers -> LAN/subnet extension across data centers.

2) Multi-tenancy isolation -> VLAN/VPN instance scalability

3) Millions of VMs -> MAC/IP forwarding table scalability

4) Optimal utilization of WAN bandwidth resource -> Unknown unicast and ARP broadcast suppression

5) Network resiliency -> Fast convergence and multi-homing

6) Load-balancing across data centers -> Active-active DC exits

7) Suboptimal path caused by LAN/subnet extension across data center -> Path optimization for both VPN access and Internet access

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Xuxiaohu
> 发送时间: 2011年12月30日 12:02
> 收件人: Christopher LILJENSTOLPE
> > >> host routes (and the routing protocol needed to make that converge)
> > >>
> > >> But at a cost.
> > >>

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Sat, 31 Dec 2011 08:33:17 +0530

What is the difference? Multi-homing = multi-path = active-active. What you need in one place that you don't need in other?

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Saturday, December 31, 2011 8:04 AM
To: [email-masked]
Subject: Re: [dc] Elevator Pitch

Hi all,

Best regards,
Xiaohu

---

From: Aldrin Isaac <[email-masked]>
Date: Fri, 30 Dec 2011 23:38:27 -0500

Xuxiaohu

Good summary.

I would add to that:

* Support for both VM and physical end stations in same LAN/subnet.
* Support for flexible virtual topologies -- ex: a host can be a member of one or more LAN topologies (think private vlan on steroids)

The following subgoals would fall under the scalability goals, but highlighting them here anyway.
* Resource load on any network node should be proportional (or better) to the number of virtual networks joined by downstream nodes and the size of those virtual networks.
* Eliminate data plane learning at all network points including at the (physical and virtual) host-network interface (please).  A proper UNI is needed for this purpose.

Thank you.

---

From: Russ White <[email-masked]>
Date: Sat, 31 Dec 2011 08:27:38 -0500

> however, I believe that aggregation time may be coming down, especially in a more homogenous environment (like a DC) vs a heterogeneous environment (like the DFZ).

Do you mean aggregation levels may be coming down? I would argue that
aggregation is used to solve two problems:

1. Rate of state change.
2. Physical table size.

Note that you don't need both everywhere. For instance, it doesn't cost
much, in the way of hardware, to hold millions of routes in the control
plane, nor to provide the processor to deal with those routes. If you
can afford 150k servers with moderately good processors and the memory
to go with them, then you can afford 15k network devices with moderately
good processors and the memory to go with them.

Where the cost is, is in the data plane, it seems --the forwarding table
size. But we already know how to do aggregation (and even caching)
between the control plane and the forwarding table.

BTW, and aside on caching in the forwarding table. It appears to be a
magic bullet for the problem of forwarding table size at first glance,
but really it all depends on a lot of things... As long as the 80/20
rule holds, caching works at this level. The question is --when you
reach 50/50, does caching still work? What about 40/60? Or 20/80? And
how does caching fail? It always fails catastrophically. Cliffs are not
good hidden away in unmonitored/not well understood pieces of the network.

And of course when we try to cache at the control plane level in order
to solve the forwarding table size problem, and when we try to mix
cached control planes with data plane driven information, we are, IMHO,
in "bad juju."

Note also that the speed at which topology changes impact the
convergence of the protocol, and the scope of those changes, can be
controlled effectively separately from the rate of change in leaf nodes
on the tree --so long as the protocol is designed to handle this sort of
separation effectively.

So if we break the problem down into its component parts, and define
what we need from each component, we might be able to reach a reasonable
solution that provides convergence and mobility. There will always be a
tradeoff, but it should be up to the operator to make that ultimate
tradeoff along the points where it's simply impossible for a protocol to
go beyond.

And to deal with the information hiding verses optimal bandwidth usage
problem, of course. :-)

> Or replacing one or two of those L2/L3 control planes with a control plane with a global view, which is easier to do in a constrained network like a traffic engineering core or a data center.  It's…

I'm not certain I understand this... I think you mean like a DFZ, a
control plane that knows every possible destination. But you have to
separate knowing every possible destination from knowing every possible
route to that destination. Even the DFZ in the 'net is really an
aggregated suboptimal subset. I don't know of any network on this scale
that has an optimal route to every destination, and I don't think it's
really possible to build one unless you want to make processing power
and control plane bandwidth usage unbounded.

It's quite possible to know every destination as a host, but not know
the entire path to every one of those destinations in detail (a form of
fisheye routing, for instance). Aggregation is, in reality, just a form
of fisheye routing --you know the path to the aggregation point in
detail, but you don't know the path beyond that.

The difference in aggregation at the reachability level is that you also
don't know the actual state of the destinations hidden in the aggregate
itself. To the degree that mobility isn't an issue, it's okay to tie
topology to reachability in this way. When mobility becomes an issue,
you need to unbundle the two in some way, treating detailed topology as
one problem, and detailed reachability as another problem altogether.
I'm guessing that unbundling these two is the most "logical" or "free
and clear" path towards scaling for the requirements as they appear to
exist.

The issue of convergence presents another problem to think about... If
convergence ends up being slower than the application timing out and
searching for a new destination IP address, then why is switching IP
addresses worse? The only way any of this makes sense is if it converges
faster than a human would notice --and, increasingly, faster than a
computer would notice.

:-)

Russ

---

From: "Ashish Dalela (adalela)" <[email-masked]>
Date: Sat, 31 Dec 2011 20:02:39 +0530

100K or whatever that number is, is a big deal at the access. And that number will increase with time. The thing to bear in mind is that these are ADDITIONAL entries, not in lieu of other entries. E.g. network routes, local host-port entries, ACLs etc exist today and will continue to exist. Host routes are in addition to that. Access switches generally had about 16K or 32K table sizes. And we are talking about adding 3-4 times more to that.

The other thing is that access to core ratio is about 1000:1. So, we need to multiply the incremental costs by 1000 to get the full impact to TCO.

In regard to scaling, there are 4 places to look at:

1. Access
2. Core
3. Inter-datacenter
4. Datacenter-internet

These have different scaling properties. 

If you have different DC and DCI technologies, and you use Encap, you have high entries at Access, DCI and Datacenter-Internet. 

If you have common DC and DCI technologies, you have high Access and Datacenter-Internet entries.

If you have hierarchical MAC, then you have only high in Datacenter-Internet boundary. 

Net-net, out of the 4 things above, you can solve 3 (first 3). The fourth we have to distribute over multiple routers, which is not hard because it is always north-south traffic. 

Thanks, Ashish

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christopher LILJENSTOLPE
Sent: Thursday, December 29, 2011 10:58 PM
To: Russ White
Cc: [email-masked]
Subject: Re: [dc] Elevator Pitch

Greetings Russ et.al.,

agreed

agreed

agreed

agreed

yes

yes

---

From: Derick Winkworth <[email-masked]>
Date: Sat, 31 Dec 2011 08:42:12 -0600

100,000 VMs is how many physical servers?  At some point you have to
eat the cost of putting the right gear in.  You can't drive down to
Best Buy to buy your switches if you think you'll have 100k host route
entries (or MAC addresses) in your access switches.

Russ is right, there are multiple platforms/vendors now supporting
millions of routes/MAC addresses.  Maybe what you thought of as a
distribution or core device is now and end-of-row "access" device?

Which kind of leads me to a point I wanted to make with regard to some
of the lists of "requirements" I've seen.  Some of these things should
be considered a network design problem, not an IETF protocol problem.
Specifically, I'm thinking of hyperscale requirements...

---

From: Joel jaeggli <[email-masked]>
Date: Sat, 31 Dec 2011 16:38:32 -0800

> entries (or MAC addresses) in your access switches.

As a DC operator I have no interest in having 1m 100k or even 10k routes
in a tor...