From david.black@emc.com Fri Sep 16 06:12:25 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BBCEE21F8AF2 for ; Fri, 16 Sep 2011 06:12:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.548 X-Spam-Level: X-Spam-Status: No, score=-106.548 tagged_above=-999 required=5 tests=[AWL=0.051, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g4yUZYaxJcJt for ; Fri, 16 Sep 2011 06:12:24 -0700 (PDT) Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id A049D21F8B8E for ; Fri, 16 Sep 2011 06:12:24 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p8GDEbTC031694 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 16 Sep 2011 09:14:37 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.251]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor) for ; Fri, 16 Sep 2011 09:14:21 -0400 Received: from mxhub02.corp.emc.com (mxhub02.corp.emc.com [10.254.141.104]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p8GDEHcn000914 for ; Fri, 16 Sep 2011 09:14:20 -0400 Received: from mx14a.corp.emc.com ([169.254.1.78]) by mxhub02.corp.emc.com ([10.254.141.104]) with mapi; Fri, 16 Sep 2011 09:14:18 -0400 From: To: Date: Fri, 16 Sep 2011 09:14:15 -0400 Thread-Topic: [armd] soliciting typical network designs for ARMD Thread-Index: AcxoEpdWIL0jbdP5SwCcwvlPcQ1kPQMX2sLA Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> In-Reply-To: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Sep 2011 13:12:25 -0000 And two more drafts on this topic: http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt The edge switches could be the software switches in hypervisors.=20 Thanks, --David > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of W= arren Kumari > Sent: Wednesday, August 31, 2011 3:16 PM > To: Vishwas Manral > Cc: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD >=20 >=20 > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >=20 > > Hi Linda/ Anoop, > > > > Here is the example of the design I was talking about, as defined by go= ogle. >=20 > Just a clarification -- s/as defined by google/as described by someone wh= o happens to work for google/ >=20 > W >=20 > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani = wrote: > > > > >>>> > > (though I think if there was a standard way to map Multicast MAC to Mul= ticast IP, they could > probably use such a standard mechanisms). > > >>>> > > > > They can do that, but then this imposes requirements on the > > equipment to be able to do multicast forwarding, and even if does, > > because of pruning requirements the number of groups would be > > very large. The average data center switch probably won't handle > > that many groups. > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral = wrote: > > Hi Anoop, > > > > From what I know they do not use Multicast GRE (I hear the extra 4 byte= s in the GRE header is a > proprietery extension). > > > > I think a directory based mechanism is what is used (though I think if = there was a standard way to > map Multicast MAC to Multicast IP, they could probably use such a standar= d mechanisms). > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani = wrote: > > Hi Vishwas, > > > > How do they get multicast through the network in that case? > > Are they planning to use multicast GRE, or just use directory > > based lookups and not worry about multicast applications > > for now? > > > > Anoop > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral = wrote: > > Hi Linda, > > > > The data packets can be tunnelled at the ToR over say a GRE packet and = the core is a Layer-3 core > (except for the downstream ports). So we could have encapsulation/ decaps= ulation of L2 over GRE at the > ToR. > > > > The very same thing can be done at the hypervisor layer too, in which c= ase the entire DC network > would look like a Layer-3 flat network including the ToR to server link a= nd the hypervisor would do > the tunneling. > > > > I am not sure if you got the points above or not. I know cloud OS compa= nies that provide the service > and have big announced customers. > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wro= te: > > Vishwas, > > > > In my mind the bullet 1) in the list refers to ToR switches downstream = ports (facing servers) > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > Have you seen data center networks with ToR switches downstream ports (= i.e. facing servers) enabling > IP routing, even though the physical links are Ethernet? > > If yes, we should definitely include it in the ARMD draft. > > > > Thanks, > > Linda > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral wrote: > > Hi Linda, > > I am unsure what you mean by this, but: > > * layer 3 all the way to TOR (Top of Rack switches), > > We can also have a heirarchical network, with the core totally Layer-3 = (and having seperate > routing), from the hosts still in a large Layer-3 subnet. Another aspect = could be to have a totally > Layer-3 network. > > > > The difference between them is the link between the servers and the ToR= . > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wro= te: > > During the 81st IETF ARMD WG discussion, it was suggested that it is ne= cessary to document typical > data center network designs so that address resolution scaling issues can= be properly described. Many > data center operators have expressed that they can't openly reveal their = detailed network designs. > Therefore, we only want to document anonymous designs without too much de= tail. During the journey of > establishing ARMD, we have come across the following typical data center = network designs: > > * layer 3 all the way to TOR (Top of Rack switches), > > * large layer 2 with hundreds (or thousands) of ToRs being interconnec= ted by Layer 2. This > design will have thousands of hosts under the L2/L3 boundary router (s) > > * CLOS design with thousands of switches. This design will have thous= ands of hosts under the > L2/L3 boundary router(s) > > We have heard that each of the designs above has its own problems. ARMD= problem statements might > need to document DC problems under each typical design. > > Please send feedback to us (either to the armd email list or to the AR= MD chair Benson & Linda) to > indicate if we have missed any typical Data Center network designs. > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > Thank you very much. > > > > Linda & Benson > > > > >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From linda.dunbar@huawei.com Sat Sep 17 11:33:07 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1435B21F8880 for ; Sat, 17 Sep 2011 11:33:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.324 X-Spam-Level: X-Spam-Status: No, score=-6.324 tagged_above=-999 required=5 tests=[AWL=0.274, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gAg4vB0C9a9c for ; Sat, 17 Sep 2011 11:33:05 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id C5ACF21F87FC for ; Sat, 17 Sep 2011 11:33:05 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LRO00C2GJMY2Y@usaga04-in.huawei.com> for armd@ietf.org; Sat, 17 Sep 2011 13:35:22 -0500 (CDT) Received: from dfweml201-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LRO00E2VJMX8I@usaga04-in.huawei.com> for armd@ietf.org; Sat, 17 Sep 2011 13:35:21 -0500 (CDT) Received: from DFWEML403-HUB.china.huawei.com (10.193.5.151) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.270.1; Sat, 17 Sep 2011 11:35:12 -0700 Received: from DFWEML504-MBX.china.huawei.com ([169.254.4.45]) by dfweml403-hub.china.huawei.com ([10.193.5.151]) with mapi id 14.01.0270.001; Sat, 17 Sep 2011 11:35:17 -0700 Date: Sat, 17 Sep 2011 18:35:17 +0000 From: Linda Dunbar X-Originating-IP: [10.47.170.222] To: "armd@ietf.org" Message-id: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_Ky4PFcvLtpkwUIPbnFUTNw)" Content-language: en-US Accept-Language: en-US, zh-CN Thread-topic: Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-index: Acx1aIsSyTVtVcAKQ4yYdknwqYNy9g== X-MS-Has-Attach: X-MS-TNEF-Correlator: Subject: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2011 18:33:07 -0000 --Boundary_(ID_Ky4PFcvLtpkwUIPbnFUTNw) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT There are many drafts recently on various Overlay Network schemes for data centers with large number of tenants/hosts. When Overlay Network is deployed in a data center, local switches/routers within the data center network don't see the hosts addresses anymore except the nodes which perform the network address encapsulation/decapsulation. However, when a large data center has tens of thousands of tenants/hosts which communicate to peers outside the data center, all those tenants/hosts send data packets destined towards external peers to the Gateway routers of the data center. Each tenant has at least several hosts which need to communicate with external peers. For each host (which runs on a VM) which need to communicate with external world, it frequently sends ARP/ND requests to its default Gateway router. For an application with dual stacks on, it sends both ARP & ND requests to its default gateway router. Routers are built to forward large amount of traffic. Traffic forwarded from line card to line card don't go through CPU. But ARP/ND requests to the gateway routers have to be processed by the router's CPU. That is why we see Router's CPU utilization reaching 40% (Merit Network statistics) or higher in many data centers. I think that this kind of problems should be captured in the ARMD WG problem statement. What do you think? Linda Dunbar --Boundary_(ID_Ky4PFcvLtpkwUIPbnFUTNw) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

 

There are many drafts recently on various Overlay Network schemes for data centers with large number of tenants/hosts. When Overlay Network is deployed in a data center, local switches/routers within the data center network don’t see the hosts addresses anymore except the nodes which perform the network address encapsulation/decapsulation.

 

However, when a large data center has tens of thousands of tenants/hosts which communicate to peers outside the data center, all those tenants/hosts send data packets destined towards external peers to the Gateway routers of the data center. Each tenant has at least several hosts which need to communicate with external peers.

                                                                                                                          

For each host (which runs on a VM) which need to communicate with external world, it frequently sends ARP/ND requests to its default Gateway router. For an application with dual stacks on, it sends both ARP & ND requests to its default gateway router.

 

Routers are built to forward large amount of traffic. Traffic forwarded from line card to line card don’t go through CPU. But ARP/ND requests to the gateway routers have to be processed by the router’s CPU. That is why we see Router’s CPU utilization reaching 40% (Merit Network statistics) or higher  in many data centers.

 

I think that this kind of problems should be captured in the ARMD WG problem statement. What do you think?  

 

Linda Dunbar

--Boundary_(ID_Ky4PFcvLtpkwUIPbnFUTNw)-- From muraris@microsoft.com Sat Sep 17 13:00:03 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A067721F8669 for ; Sat, 17 Sep 2011 13:00:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.496 X-Spam-Level: X-Spam-Status: No, score=-110.496 tagged_above=-999 required=5 tests=[AWL=0.103, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dQz6zCDTZmBJ for ; Sat, 17 Sep 2011 13:00:02 -0700 (PDT) Received: from smtp.microsoft.com (mail2.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 74DA421F8663 for ; Sat, 17 Sep 2011 13:00:02 -0700 (PDT) Received: from TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Sat, 17 Sep 2011 13:02:20 -0700 Received: from TK5EX14MBXC300.redmond.corp.microsoft.com ([169.254.3.19]) by TK5EX14MLTC102.redmond.corp.microsoft.com ([157.54.79.180]) with mapi id 14.01.0339.002; Sat, 17 Sep 2011 13:02:20 -0700 From: Murari Sridharan To: "david.black@emc.com" , "armd@ietf.org" Thread-Topic: [armd] soliciting typical network designs for ARMD Thread-Index: AQHMVt0qWVZI0cjSpkywQ6H6jnPxdJUVhDmAgAOGoACAHuGmgIAYwDCAgAGOU4A= Date: Sat, 17 Sep 2011 20:02:19 +0000 Message-ID: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.34] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2011 20:00:03 -0000 FYI, here is a talk that I gave last week in relation to the nvgre draft be= low.=20 http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T Thanks Murari=20 -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of dav= id.black@emc.com Sent: Friday, September 16, 2011 6:14 AM To: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD And two more drafts on this topic: http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt The edge switches could be the software switches in hypervisors.=20 Thanks, --David > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=20 > Of Warren Kumari > Sent: Wednesday, August 31, 2011 3:16 PM > To: Vishwas Manral > Cc: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD >=20 >=20 > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >=20 > > Hi Linda/ Anoop, > > > > Here is the example of the design I was talking about, as defined by go= ogle. >=20 > Just a clarification -- s/as defined by google/as described by someone=20 > who happens to work for google/ >=20 > W >=20 > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani = wrote: > > > > >>>> > > (though I think if there was a standard way to map Multicast MAC to=20 > > Multicast IP, they could > probably use such a standard mechanisms). > > >>>> > > > > They can do that, but then this imposes requirements on the=20 > > equipment to be able to do multicast forwarding, and even if does,=20 > > because of pruning requirements the number of groups would be very=20 > > large. The average data center switch probably won't handle that=20 > > many groups. > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral = wrote: > > Hi Anoop, > > > > From what I know they do not use Multicast GRE (I hear the extra 4=20 > > bytes in the GRE header is a > proprietery extension). > > > > I think a directory based mechanism is what is used (though I think=20 > > if there was a standard way to > map Multicast MAC to Multicast IP, they could probably use such a standar= d mechanisms). > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani = wrote: > > Hi Vishwas, > > > > How do they get multicast through the network in that case? > > Are they planning to use multicast GRE, or just use directory based=20 > > lookups and not worry about multicast applications for now? > > > > Anoop > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral = wrote: > > Hi Linda, > > > > The data packets can be tunnelled at the ToR over say a GRE packet=20 > > and the core is a Layer-3 core > (except for the downstream ports). So we could have encapsulation/=20 > decapsulation of L2 over GRE at the ToR. > > > > The very same thing can be done at the hypervisor layer too, in=20 > > which case the entire DC network > would look like a Layer-3 flat network including the ToR to server=20 > link and the hypervisor would do the tunneling. > > > > I am not sure if you got the points above or not. I know cloud OS=20 > > companies that provide the service > and have big announced customers. > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wro= te: > > Vishwas, > > > > In my mind the bullet 1) in the list refers to ToR switches=20 > > downstream ports (facing servers) > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > Have you seen data center networks with ToR switches downstream=20 > > ports (i.e. facing servers) enabling > IP routing, even though the physical links are Ethernet? > > If yes, we should definitely include it in the ARMD draft. > > > > Thanks, > > Linda > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral wrote: > > Hi Linda, > > I am unsure what you mean by this, but: > > * layer 3 all the way to TOR (Top of Rack switches), We can also=20 > > have a heirarchical network, with the core totally Layer-3 (and=20 > > having seperate > routing), from the hosts still in a large Layer-3 subnet. Another=20 > aspect could be to have a totally > Layer-3 network. > > > > The difference between them is the link between the servers and the ToR= . > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wro= te: > > During the 81st IETF ARMD WG discussion, it was suggested that it is=20 > > necessary to document typical > data center network designs so that address resolution scaling issues=20 > can be properly described. Many data center operators have expressed that= they can't openly reveal their detailed network designs. > Therefore, we only want to document anonymous designs without too much=20 > detail. During the journey of establishing ARMD, we have come across the = following typical data center network designs: > > * layer 3 all the way to TOR (Top of Rack switches), > > * large layer 2 with hundreds (or thousands) of ToRs being=20 > > interconnected by Layer 2. This > design will have thousands of hosts under the L2/L3 boundary router=20 > (s) > > * CLOS design with thousands of switches. This design will have=20 > > thousands of hosts under the > L2/L3 boundary router(s) > > We have heard that each of the designs above has its own problems.=20 > > ARMD problem statements might > need to document DC problems under each typical design. > > Please send feedback to us (either to the armd email list or to the=20 > > ARMD chair Benson & Linda) to > indicate if we have missed any typical Data Center network designs. > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > Thank you very much. > > > > Linda & Benson > > > > >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From linda.dunbar@huawei.com Sun Sep 18 07:04:18 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7175F21F84CC for ; Sun, 18 Sep 2011 07:04:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.341 X-Spam-Level: X-Spam-Status: No, score=-6.341 tagged_above=-999 required=5 tests=[AWL=0.258, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cMCT0Bx1j00T for ; Sun, 18 Sep 2011 07:04:17 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id 4FC2F21F84BA for ; Sun, 18 Sep 2011 07:04:17 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LRQ00CT11UW2Y@usaga04-in.huawei.com> for armd@ietf.org; Sun, 18 Sep 2011 09:06:33 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LRQ003PC1UVNE@usaga04-in.huawei.com> for armd@ietf.org; Sun, 18 Sep 2011 09:06:32 -0500 (CDT) Received: from DFWEML403-HUB.china.huawei.com (10.193.5.151) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Sun, 18 Sep 2011 07:06:31 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.103]) by dfweml403-hub.china.huawei.com ([10.193.5.151]) with mapi id 14.01.0270.001; Sun, 18 Sep 2011 07:06:30 -0700 Date: Sun, 18 Sep 2011 14:06:30 +0000 From: Linda Dunbar In-reply-to: X-Originating-IP: [10.47.165.248] To: Murari Sridharan , "david.black@emc.com" , "armd@ietf.org" Message-id: <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-language: en-US Content-transfer-encoding: 7BIT Accept-Language: en-US, zh-CN Thread-topic: how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-index: AQHMdgwqIcvvzy1XgUWNiawstYen2w== X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Sep 2011 14:04:18 -0000 Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with external peers, the host either uses public IP address which is visible to external peers or uses private IP address which is translated to public address at the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center" advertise this host address to external peers? Or will all external peers go through enterprise's gateway to reach this host which is no longer residing in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > > > > > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From d3e3e3@gmail.com Sun Sep 18 09:00:07 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 86DAF21F8548 for ; Sun, 18 Sep 2011 09:00:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -104.097 X-Spam-Level: X-Spam-Status: No, score=-104.097 tagged_above=-999 required=5 tests=[AWL=-0.498, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8UMB2beDRhfq for ; Sun, 18 Sep 2011 09:00:06 -0700 (PDT) Received: from mail-fx0-f44.google.com (mail-fx0-f44.google.com [209.85.161.44]) by ietfa.amsl.com (Postfix) with ESMTP id 6B6BB21F855F for ; Sun, 18 Sep 2011 09:00:06 -0700 (PDT) Received: by fxd18 with SMTP id 18so3460851fxd.31 for ; Sun, 18 Sep 2011 09:02:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=BYRk6C+ClNvzL8Luv/Gnin3j4SFg+3VNABrj4/nFnrE=; b=F0FSveOroRo4dpcnP5OpPgrpLT3eYGmK8GLA7iQJj366HLIesOBu9Ged0LltLzLdPo sQ6gop9iZTSc6YLAKbC04okdd0K0NYNKsZGEEKcQyMaczOJ4ip4gCawY7x47TlToAIGo mPs8tPLQEgMG9PmHo/RfDqc3Wt6PdF1eIiC2w= Received: by 10.223.45.209 with SMTP id g17mr3413204faf.96.1316361746254; Sun, 18 Sep 2011 09:02:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.152.22.9 with HTTP; Sun, 18 Sep 2011 09:02:06 -0700 (PDT) In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> References: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> From: Donald Eastlake Date: Sun, 18 Sep 2011 12:02:06 -0400 Message-ID: To: Linda Dunbar Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Sep 2011 16:00:08 -0000 This sort of problem seems on target for the ARMD charter and should be verified and documented. Thanks, Donald On Sat, Sep 17, 2011 at 2:35 PM, Linda Dunbar wro= te: > > > There are many drafts recently on various Overlay Network schemes for dat= a > centers with large number of tenants/hosts. When Overlay Network is deplo= yed > in a data center, local switches/routers within the data center network > don=92t see the hosts addresses anymore except the nodes which perform th= e > network address encapsulation/decapsulation. > > > However, when a large data center has tens of thousands of tenants/hosts > which communicate to peers outside the data center, all those tenants/hos= ts > send data packets destined towards external peers to the Gateway routers = of > the data center. Each tenant has at least several hosts which need to > communicate with external peers. > > > For each host (which runs on a VM) which need to communicate with externa= l > world, it frequently sends ARP/ND requests to its default Gateway router. > For an application with dual stacks on, it sends both ARP & ND requests t= o > its default gateway router. > > > Routers are built to forward large amount of traffic. Traffic forwarded f= rom > line card to line card don=92t go through CPU. But ARP/ND requests to the > gateway routers have to be processed by the router=92s CPU. That is why w= e see > Router=92s CPU utilization reaching 40% (Merit Network statistics) or hig= her > =A0in many data centers. > > > I think that this kind of problems should be captured in the ARMD WG prob= lem > statement. What do you think? > > > Linda Dunbar From narave@microsoft.com Sun Sep 18 10:59:11 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6954421F84B1 for ; Sun, 18 Sep 2011 10:59:11 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zluM1Pky2MzU for ; Sun, 18 Sep 2011 10:59:10 -0700 (PDT) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 3A27F21F8496 for ; Sun, 18 Sep 2011 10:59:10 -0700 (PDT) Received: from TK5EX14HUBC106.redmond.corp.microsoft.com (157.54.80.61) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Sun, 18 Sep 2011 11:01:30 -0700 Received: from TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com (157.54.71.68) by TK5EX14HUBC106.redmond.corp.microsoft.com (157.54.80.61) with Microsoft SMTP Server (TLS) id 14.1.339.2; Sun, 18 Sep 2011 11:01:30 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.68]) with mapi id 14.01.0339.002; Sun, 18 Sep 2011 11:01:29 -0700 From: Narasimhan Venkataramaiah To: Linda Dunbar , Murari Sridharan , "david.black@emc.com" , "armd@ietf.org" Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHG Date: Sun, 18 Sep 2011 18:01:29 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> , <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.43] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Sep 2011 17:59:11 -0000 The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunb= ar [linda.dunbar@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > > > > > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd= From bschlies@cisco.com Sun Sep 18 11:22:09 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 74D1B21F84FC for ; Sun, 18 Sep 2011 11:22:09 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.157 X-Spam-Level: X-Spam-Status: No, score=-3.157 tagged_above=-999 required=5 tests=[AWL=-0.558, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zymMRNP51T6H for ; Sun, 18 Sep 2011 11:22:08 -0700 (PDT) Received: from rcdn-iport-4.cisco.com (rcdn-iport-4.cisco.com [173.37.86.75]) by ietfa.amsl.com (Postfix) with ESMTP id B957421F84E5 for ; Sun, 18 Sep 2011 11:22:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=bschlies@cisco.com; l=3618; q=dns/txt; s=iport; t=1316370270; x=1317579870; h=date:subject:from:to:message-id:in-reply-to:mime-version: content-transfer-encoding; bh=KkheFRhCJiRd+NgV+pI21/6TH2YQIGnKaVPYjXaSOPg=; b=HcDXo2Kcx2jhUj68FQ32/pgLUTyZOGa1m7gAvJ+Hn8RU18luwYg8RtAC DsNGBEQHVjU1XSJVbKULh/w0PLVgHKIkKj5iqmY9crYE3ou+T08xI/WlR pqK5yfHFpKMuv+Sb7omKJqRpEKClnQVimIuqn24rZznJ3s95xR89Y9VDT g=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgEFAEI2dk6tJXG8/2dsb2JhbABChFWiZXeBUwEBAQEDAQEBDwEKBhkBFxodAQgRAwECAwISAQEMBgIpJgEBBQMBAQQTFA6HWZVtgSYBjDqQJoEsgkABBAeBb4ERBJNJhSaMIw X-IronPort-AV: E=Sophos;i="4.68,401,1312156800"; d="scan'208";a="22289524" Received: from rcdn-core2-1.cisco.com ([173.37.113.188]) by rcdn-iport-4.cisco.com with ESMTP; 18 Sep 2011 18:24:29 +0000 Received: from xbh-rcd-101.cisco.com (xbh-rcd-101.cisco.com [72.163.62.138]) by rcdn-core2-1.cisco.com (8.14.3/8.14.3) with ESMTP id p8IIOTop006422 for ; Sun, 18 Sep 2011 18:24:29 GMT Received: from xmb-rcd-206.cisco.com ([72.163.62.213]) by xbh-rcd-101.cisco.com with Microsoft SMTPSVC(6.0.3790.4675); Sun, 18 Sep 2011 13:24:29 -0500 Received: from 10.82.243.93 ([10.82.243.93]) by XMB-RCD-206.cisco.com ([72.163.62.213]) via Exchange Front-End Server email.cisco.com ([72.163.62.136]) with Microsoft Exchange Server HTTP-DAV ; Sun, 18 Sep 2011 18:24:29 +0000 User-Agent: Microsoft-Entourage/12.30.0.110427 Date: Sun, 18 Sep 2011 14:24:28 -0400 From: Benson Schliesser To: "ARMD@ietf.org" Message-ID: Thread-Topic: Nomcom 2011-2012: Second Call for Nominations Thread-Index: Acx2MDNqOboLB50QBkaBDboSk8jkRw== In-Reply-To: <20110918020538.C7C5821F86EC@ietfa.amsl.com> Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable X-OriginalArrivalTime: 18 Sep 2011 18:24:29.0631 (UTC) FILETIME=[3462F4F0:01CC7630] Subject: [armd] FW: Nomcom 2011-2012: Second Call for Nominations X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Sep 2011 18:22:09 -0000 In case you haven't read this posting elsewhere, please see below. The NomCom really needs more nominations for various positions they're working to fill. If you are interested and/or know somebody that might be willing to serve, please consider helping. Cheers, -Benson=20 ------ Forwarded Message From: NomCom Chair Date: Sat, 17 Sep 2011 19:05:38 -0700 (PDT) To: IETF Announcement list Cc: Subject: Nomcom 2011-2012: Second Call for Nominations Hi All, We are halfway through the nomination period (it ends on October 2, 2011) and we need more nominees than we have received so far. We appreciate the folks that have taken the time to nominate people and those who have accepted so far. But the fact remains that the number of nominations have been below average and the acceptance rates of those nominated has also been low. We need **YOUR** input and participation! We cannot properly execute the task of selecting the best candidates for these positions with so few nominations and acceptances. So, please consider making nominations for the open positions, in particular those for which we have so few nominations =EF=BF=BD it takes just a few minutes of your time. Right now, we just need the names/email addresses. Why do we need more nominations? Well, even if you think a willing incumbent is doing a very good job and should be returned, his or her ability to serve again might be impacted by unforeseen circumstances between now and March. NomCom needs to consider multiple nominees to be prepared in the event one or more candidates is unable to serve come next March and to ensure we have chosen the best candidate. There are several ways you can help the Nomcom - You can nominate yourself. - You can nominate someone you know whom you think would do a good job. Don't worry about whether they might already be nominated. We would much prefer to receive the same nomination several times rather than miss a good person we should consider. How to submit Nominations: -------------------------- The list of positions we need to fill, and the provided Job Descriptions, and forms for nominations, can be found in the call for nominations at: https://datatracker.ietf.org/ann/nomcom/3049/ You can enter a nomination by going to the following URL: https://www.ietf.org/group/nomcom/2011/nominate You can also nominate someone by sending an email to nomcom11@ietf.org and giving us their name, email address and the open position you are nominating them for. We will take care of the rest. If you are asked for a user name and password, use an existing ietf login and password. If you need a login and password, request one from the following URL: https://datatracker.ietf.org/accounts/create/ Open List: ---------- As you already know, NomCom 2011-2012 will follow the policy for "Open Disclosure of Willing Nominees" described in RFC 5680. Feedback Collection: -------------------- The open list is currently available on the Nomcom page and the entire community is invited to provide feedback on all nominees. You can provide your comments on all willing nominees at the following URL: https://www.ietf.org/group/nomcom/2011/input/ Suresh Krishnan Chair, NomCom 2011-2012 nomcom-chair@ietf.org suresh.krishnan@ericsson.com _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www.ietf.org/mailman/listinfo/ietf-announce ------ End of Forwarded Message From prvs=2243993404=phe@ciena.com Mon Sep 19 05:59:55 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7E13C21F8B91 for ; Mon, 19 Sep 2011 05:59:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.725 X-Spam-Level: X-Spam-Status: No, score=0.725 tagged_above=-999 required=5 tests=[AWL=3.990, BAYES_00=-2.599, IP_NOT_FRIENDLY=0.334, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LO-PvwRdpUCf for ; Mon, 19 Sep 2011 05:59:54 -0700 (PDT) Received: from mx0b-00103a01.pphosted.com (mx0b-00103a01.pphosted.com [67.231.152.227]) by ietfa.amsl.com (Postfix) with ESMTP id 6138821F854D for ; Mon, 19 Sep 2011 05:59:54 -0700 (PDT) Received: from pps.filterd (m0001124 [127.0.0.1]) by mx0b-00103a01.pphosted.com (8.14.3/8.14.3) with SMTP id p8JCxtlU022877; Mon, 19 Sep 2011 09:02:09 -0400 Received: from mdwexght02.ciena.com (LIN1-118-36-29.ciena.com [63.118.36.29]) by mx0b-00103a01.pphosted.com with ESMTP id yx66wg0gg-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 19 Sep 2011 09:02:09 -0400 Received: from MDWEXGMB02.ciena.com ([::1]) by MDWEXGHT02.ciena.com ([::1]) with mapi; Mon, 19 Sep 2011 09:02:08 -0400 From: "He, Peng" To: Donald Eastlake , Linda Dunbar Content-Class: urn:content-classes:message Date: Mon, 19 Sep 2011 09:02:06 -0400 Thread-Topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-Index: Acx2HGVRi+PxXB+KSxKfHuqXT39J7AAr5ArA Message-ID: References: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US x-tm-as-product-ver: SMEX-10.0.0.1412-6.800.1017-18394.007 x-tm-as-result: No--52.071600-8.000000-31 x-tm-as-user-approved-sender: No x-tm-as-user-blocked-sender: No Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.4.6813, 1.0.211, 0.0.0000 definitions=2011-09-19_03:2011-09-19, 2011-09-19, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1109190102 Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 12:59:55 -0000 Agreed.. nice description for the problem statement. Regards, Peng > -----Original Message----- >From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >Donald Eastlake >Sent: Sunday, September 18, 2011 12:02 PM >To: Linda Dunbar >Cc: armd@ietf.org >Subject: Re: [armd] Bottleneck of Overlay network in a data center with >large number of tenants/hosts > >This sort of problem seems on target for the ARMD charter and should >be verified and documented. > >Thanks, >Donald > >On Sat, Sep 17, 2011 at 2:35 PM, Linda Dunbar >wrote: >> >> >> There are many drafts recently on various Overlay Network schemes for >data >> centers with large number of tenants/hosts. When Overlay Network is >deployed >> in a data center, local switches/routers within the data center >network >> don't see the hosts addresses anymore except the nodes which perform >the >> network address encapsulation/decapsulation. >> >> >> However, when a large data center has tens of thousands of >tenants/hosts >> which communicate to peers outside the data center, all those >tenants/hosts >> send data packets destined towards external peers to the Gateway >routers of >> the data center. Each tenant has at least several hosts which need to >> communicate with external peers. >> >> >> For each host (which runs on a VM) which need to communicate with >external >> world, it frequently sends ARP/ND requests to its default Gateway >router. >> For an application with dual stacks on, it sends both ARP & ND >requests to >> its default gateway router. >> >> >> Routers are built to forward large amount of traffic. Traffic >forwarded from >> line card to line card don't go through CPU. But ARP/ND requests to >the >> gateway routers have to be processed by the router's CPU. That is why >we see >> Router's CPU utilization reaching 40% (Merit Network statistics) or >higher >> in many data centers. >> >> >> I think that this kind of problems should be captured in the ARMD WG >problem >> statement. What do you think? >> >> >> Linda Dunbar >_______________________________________________ >armd mailing list >armd@ietf.org >https://www.ietf.org/mailman/listinfo/armd From ning.so@verizon.com Mon Sep 19 06:56:01 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3499D21F8BEF for ; Mon, 19 Sep 2011 06:56:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.049 X-Spam-Level: X-Spam-Status: No, score=-3.049 tagged_above=-999 required=5 tests=[AWL=0.550, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NCfMcsRvYKH7 for ; Mon, 19 Sep 2011 06:56:00 -0700 (PDT) Received: from fldsmtpe02.verizon.com (fldsmtpe02.verizon.com [140.108.26.141]) by ietfa.amsl.com (Postfix) with ESMTP id 7FBFA21F8BE8 for ; Mon, 19 Sep 2011 06:56:00 -0700 (PDT) X-IronPort-Anti-Spam-Filtered: false Received: from unknown (HELO fldsmtpi03.verizon.com) ([166.68.71.145]) by fldsmtpe02.verizon.com with ESMTP; 19 Sep 2011 13:58:22 +0000 From: "So, Ning" X-IronPort-AV: E=Sophos;i="4.68,405,1312156800"; d="scan'208";a="139893130" Received: from fhdp1lumxc7hb05.verizon.com (HELO FHDP1LUMXC7HB05.us.one.verizon.com) ([166.68.59.192]) by fldsmtpi03.verizon.com with ESMTP; 19 Sep 2011 13:58:21 +0000 Received: from FHDP1LUMXC7V41.us.one.verizon.com ([169.254.1.60]) by FHDP1LUMXC7HB05.us.one.verizon.com ([166.68.59.192]) with mapi; Mon, 19 Sep 2011 09:58:20 -0400 To: Donald Eastlake , Linda Dunbar Date: Mon, 19 Sep 2011 09:58:19 -0400 Thread-Topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-Index: Acx2HGOduOzJS+qdT12fMc3R9BB7rwAt8efw Message-ID: <6665BC1FEA04AB47B1F75FA641C43BC08DF665A0@FHDP1LUMXC7V41.us.one.verizon.com> References: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 13:56:01 -0000 +1. =A0 Best regards, =A0 Ning So Verizon Corporate Technology (office) 972-729-7905 (Cell) 972-955-0914 =A0 -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Don= ald Eastlake Sent: Sunday, September 18, 2011 11:02 AM To: Linda Dunbar Cc: armd@ietf.org Subject: Re: [armd] Bottleneck of Overlay network in a data center with lar= ge number of tenants/hosts This sort of problem seems on target for the ARMD charter and should be ver= ified and documented. Thanks, Donald On Sat, Sep 17, 2011 at 2:35 PM, Linda Dunbar wro= te: > > > There are many drafts recently on various Overlay Network schemes for=20 > data centers with large number of tenants/hosts. When Overlay Network=20 > is deployed in a data center, local switches/routers within the data=20 > center network don't see the hosts addresses anymore except the nodes=20 > which perform the network address encapsulation/decapsulation. > > > However, when a large data center has tens of thousands of=20 > tenants/hosts which communicate to peers outside the data center, all=20 > those tenants/hosts send data packets destined towards external peers=20 > to the Gateway routers of the data center. Each tenant has at least=20 > several hosts which need to communicate with external peers. > > > For each host (which runs on a VM) which need to communicate with=20 > external world, it frequently sends ARP/ND requests to its default Gatewa= y router. > For an application with dual stacks on, it sends both ARP & ND=20 > requests to its default gateway router. > > > Routers are built to forward large amount of traffic. Traffic=20 > forwarded from line card to line card don't go through CPU. But ARP/ND=20 > requests to the gateway routers have to be processed by the router's=20 > CPU. That is why we see Router's CPU utilization reaching 40% (Merit=20 > Network statistics) or higher > =A0in many data centers. > > > I think that this kind of problems should be captured in the ARMD WG=20 > problem statement. What do you think? > > > Linda Dunbar _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From florin.balus@alcatel-lucent.com Mon Sep 19 16:34:22 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9DEA921F8B2F for ; Mon, 19 Sep 2011 16:34:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K5yFjzD68mYs for ; Mon, 19 Sep 2011 16:34:21 -0700 (PDT) Received: from ihemail4.lucent.com (ihemail4.lucent.com [135.245.0.39]) by ietfa.amsl.com (Postfix) with ESMTP id 8425221F8B2B for ; Mon, 19 Sep 2011 16:34:21 -0700 (PDT) Received: from usnavsmail2.ndc.alcatel-lucent.com (usnavsmail2.ndc.alcatel-lucent.com [135.3.39.10]) by ihemail4.lucent.com (8.13.8/IER-o) with ESMTP id p8JNaene026924 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 19 Sep 2011 18:36:41 -0500 (CDT) Received: from USNAVSXCHHUB01.ndc.alcatel-lucent.com (usnavsxchhub01.ndc.alcatel-lucent.com [135.3.39.110]) by usnavsmail2.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p8JNadAr018614 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT); Mon, 19 Sep 2011 18:36:40 -0500 Received: from USNAVSXCHMBSC3.ndc.alcatel-lucent.com ([135.3.39.139]) by USNAVSXCHHUB01.ndc.alcatel-lucent.com ([135.3.39.110]) with mapi; Mon, 19 Sep 2011 18:36:39 -0500 From: "Balus, Florin Stelian (Florin)" To: Murari Sridharan , "david.black@emc.com" , "armd@ietf.org" Date: Mon, 19 Sep 2011 18:36:37 -0500 Thread-Topic: [armd] soliciting typical network designs for ARMD Thread-Index: AQHMVt0qWVZI0cjSpkywQ6H6jnPxdJUVhDmAgAOGoACAHuGmgIAYwDCAgAGOU4CAA0PKgA== Message-ID: <2073A6C5467C99478898544C6EBA3F4602BBFE512F@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.57 on 135.245.2.39 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.10 Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Sep 2011 23:34:22 -0000 These two drafts and draft-wkumari-dcops-l3-vmmobility-00.txt seem to be su= bmitted as either informational or experimental. Can the authors of these t= wo drafts clarify what do they plan to do with them? If relevant feedback i= s required I think they seem to fit the scope of L2VPN WG. At least two of = them talk about some new encapsulation used to provide L2 Multi-tenancy in = the Data Centers over some sort of overlay and UDP/IP tunnels. A new L2VPN = charter was released recently which includes requirements and solutions tar= geted at the Data Center space. Florin -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Mur= ari Sridharan Sent: Saturday, September 17, 2011 1:02 PM To: david.black@emc.com; armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD FYI, here is a talk that I gave last week in relation to the nvgre draft be= low.=20 http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T Thanks Murari=20 -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of dav= id.black@emc.com Sent: Friday, September 16, 2011 6:14 AM To: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD And two more drafts on this topic: http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt The edge switches could be the software switches in hypervisors.=20 Thanks, --David > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=20 > Of Warren Kumari > Sent: Wednesday, August 31, 2011 3:16 PM > To: Vishwas Manral > Cc: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD >=20 >=20 > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >=20 > > Hi Linda/ Anoop, > > > > Here is the example of the design I was talking about, as defined by go= ogle. >=20 > Just a clarification -- s/as defined by google/as described by someone=20 > who happens to work for google/ >=20 > W >=20 > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani = wrote: > > > > >>>> > > (though I think if there was a standard way to map Multicast MAC to=20 > > Multicast IP, they could > probably use such a standard mechanisms). > > >>>> > > > > They can do that, but then this imposes requirements on the=20 > > equipment to be able to do multicast forwarding, and even if does,=20 > > because of pruning requirements the number of groups would be very=20 > > large. The average data center switch probably won't handle that=20 > > many groups. > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral = wrote: > > Hi Anoop, > > > > From what I know they do not use Multicast GRE (I hear the extra 4=20 > > bytes in the GRE header is a > proprietery extension). > > > > I think a directory based mechanism is what is used (though I think=20 > > if there was a standard way to > map Multicast MAC to Multicast IP, they could probably use such a standar= d mechanisms). > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani = wrote: > > Hi Vishwas, > > > > How do they get multicast through the network in that case? > > Are they planning to use multicast GRE, or just use directory based=20 > > lookups and not worry about multicast applications for now? > > > > Anoop > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral = wrote: > > Hi Linda, > > > > The data packets can be tunnelled at the ToR over say a GRE packet=20 > > and the core is a Layer-3 core > (except for the downstream ports). So we could have encapsulation/=20 > decapsulation of L2 over GRE at the ToR. > > > > The very same thing can be done at the hypervisor layer too, in=20 > > which case the entire DC network > would look like a Layer-3 flat network including the ToR to server=20 > link and the hypervisor would do the tunneling. > > > > I am not sure if you got the points above or not. I know cloud OS=20 > > companies that provide the service > and have big announced customers. > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wro= te: > > Vishwas, > > > > In my mind the bullet 1) in the list refers to ToR switches=20 > > downstream ports (facing servers) > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > Have you seen data center networks with ToR switches downstream=20 > > ports (i.e. facing servers) enabling > IP routing, even though the physical links are Ethernet? > > If yes, we should definitely include it in the ARMD draft. > > > > Thanks, > > Linda > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral wrote: > > Hi Linda, > > I am unsure what you mean by this, but: > > * layer 3 all the way to TOR (Top of Rack switches), We can also=20 > > have a heirarchical network, with the core totally Layer-3 (and=20 > > having seperate > routing), from the hosts still in a large Layer-3 subnet. Another=20 > aspect could be to have a totally > Layer-3 network. > > > > The difference between them is the link between the servers and the ToR= . > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wro= te: > > During the 81st IETF ARMD WG discussion, it was suggested that it is=20 > > necessary to document typical > data center network designs so that address resolution scaling issues=20 > can be properly described. Many data center operators have expressed that= they can't openly reveal their detailed network designs. > Therefore, we only want to document anonymous designs without too much=20 > detail. During the journey of establishing ARMD, we have come across the = following typical data center network designs: > > * layer 3 all the way to TOR (Top of Rack switches), > > * large layer 2 with hundreds (or thousands) of ToRs being=20 > > interconnected by Layer 2. This > design will have thousands of hosts under the L2/L3 boundary router=20 > (s) > > * CLOS design with thousands of switches. This design will have=20 > > thousands of hosts under the > L2/L3 boundary router(s) > > We have heard that each of the designs above has its own problems.=20 > > ARMD problem statements might > need to document DC problems under each typical design. > > Please send feedback to us (either to the armd email list or to the=20 > > ARMD chair Benson & Linda) to > indicate if we have missed any typical Data Center network designs. > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > Thank you very much. > > > > Linda & Benson > > > > >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From linda.dunbar@huawei.com Tue Sep 20 07:56:02 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EAFE521F8CF5 for ; Tue, 20 Sep 2011 07:56:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.355 X-Spam-Level: X-Spam-Status: No, score=-6.355 tagged_above=-999 required=5 tests=[AWL=0.244, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YRXypgI6v4iw for ; Tue, 20 Sep 2011 07:56:02 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id EA6B321F8CF2 for ; Tue, 20 Sep 2011 07:56:01 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LRT008W1TLEOZ@usaga04-in.huawei.com> for armd@ietf.org; Tue, 20 Sep 2011 09:58:27 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LRT00AJ7TLDHT@usaga04-in.huawei.com> for armd@ietf.org; Tue, 20 Sep 2011 09:58:26 -0500 (CDT) Received: from DFWEML403-HUB.china.huawei.com (10.193.5.151) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 20 Sep 2011 07:58:26 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.103]) by dfweml403-hub.china.huawei.com ([10.193.5.151]) with mapi id 14.01.0270.001; Tue, 20 Sep 2011 07:58:22 -0700 Date: Tue, 20 Sep 2011 14:58:20 +0000 From: Linda Dunbar In-reply-to: <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> X-Originating-IP: [10.216.46.97] To: Narasimhan Venkataramaiah , Linda Dunbar , Murari Sridharan , "david.black@emc.com" , "armd@ietf.org" Message-id: <4A95BA014132FF49AE685FAB4B9F17F610CA8508@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=Windows-1252 Content-language: en-US Content-transfer-encoding: 7BIT Accept-Language: en-US, zh-CN Thread-topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-index: AQHMdi0CNEHNe+RVIE+79nf9cw77PpVWFS6g X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2011 14:56:03 -0000 Simha, What if entire subnet is moved to "cloud data center"? For applications which communicate with external peers, do they go through the enterprise data center? What if the enterprise data center is removed after moving all the applications to "cloud"? Linda > -----Original Message----- > From: Narasimhan Venkataramaiah [mailto:narave@microsoft.com] > Sent: Sunday, September 18, 2011 1:01 PM > To: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: RE: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal > from the enterprise point of view. Are you referring to a scenario > where the VMs subnet is split between the cloud and the enterprise? > Otherwise I don't see the implication on virtualization as its no > different than getting the traffic routed to the enterprise in the > first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible > to external peers or uses private IP address which is translated to > public address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all > external peers go through enterprise's gateway to reach this host which > is no longer residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On > Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC > to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be > very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra > 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I > think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory > based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE > packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can > also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and > the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling > issues > > > can be properly described. Many data center operators have > expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come > across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will > have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > > > > > > > > > > > > _______________________________________________ > > > armd mailing list > > > armd@ietf.org > > > https://www.ietf.org/mailman/listinfo/armd > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From narave@microsoft.com Tue Sep 20 08:13:13 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CCA5B21F8C2B for ; Tue, 20 Sep 2011 08:13:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ypwtjvQhW7kr for ; Tue, 20 Sep 2011 08:13:12 -0700 (PDT) Received: from smtp.microsoft.com (mailb.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 9BB0621F8C1E for ; Tue, 20 Sep 2011 08:13:12 -0700 (PDT) Received: from TK5EX14HUBC105.redmond.corp.microsoft.com (157.54.80.48) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Tue, 20 Sep 2011 08:15:38 -0700 Received: from TK5EX14MLTW651.wingroup.windeploy.ntdev.microsoft.com (157.54.71.39) by TK5EX14HUBC105.redmond.corp.microsoft.com (157.54.80.48) with Microsoft SMTP Server (TLS) id 14.1.339.2; Tue, 20 Sep 2011 08:15:38 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW651.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.39]) with mapi id 14.01.0339.002; Tue, 20 Sep 2011 08:15:38 -0700 From: Narasimhan Venkataramaiah To: Linda Dunbar , Murari Sridharan , "david.black@emc.com" , "armd@ietf.org" Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHGgANomAD//41iwA== Date: Tue, 20 Sep 2011 15:15:37 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD31BBA@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <4A95BA014132FF49AE685FAB4B9F17F610CA8508@dfweml503-mbx.china.huawei.com> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610CA8508@dfweml503-mbx.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.42] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2011 15:13:14 -0000 I am trying to understand the implication to network virtualization. Say an= enterprise has 2 data centers in different geographical regions and they d= ecided to move a subnet from one DC to another - whatever is done to route = the subnet to the new data center applies here too. What is different for a= virtualized network unless the subnet spans 2 data centers? Simha -----Original Message----- From: Linda Dunbar [mailto:linda.dunbar@huawei.com]=20 Sent: Tuesday, September 20, 2011 7:58 AM To: Narasimhan Venkataramaiah; Linda Dunbar; Murari Sridharan; david.black@= emc.com; armd@ietf.org Subject: RE: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Simha,=20 What if entire subnet is moved to "cloud data center"? For applications whi= ch communicate with external peers, do they go through the enterprise data = center? What if the enterprise data center is removed after moving all the = applications to "cloud"?=20 Linda > -----Original Message----- > From: Narasimhan Venkataramaiah [mailto:narave@microsoft.com] > Sent: Sunday, September 18, 2011 1:01 PM > To: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: RE: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > The easiest from the point of view of configuration would be to route=20 > everything back through the enterprise - not necessarily the optimal=20 > from the enterprise point of view. Are you referring to a scenario=20 > where the VMs subnet is split between the cloud and the enterprise? > Otherwise I don't see the implication on virtualization as its no=20 > different than getting the traffic routed to the enterprise in the=20 > first case. >=20 > Simha >=20 > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda=20 > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > Hi Murari, >=20 > Thank you very much for sharing the presentation. >=20 > One question: >=20 > For a host within an Enterprise site which needs to communicate with=20 > external peers, the host either uses public IP address which is=20 > visible to external peers or uses private IP address which is=20 > translated to public address at the Enterprise site's gateway. >=20 > When this host is moved to "Cloud data center", will the "Cloud Data=20 > center" advertise this host address to external peers? Or will all=20 > external peers go through enterprise's gateway to reach this host=20 > which is no longer residing in the enterprise site? >=20 > Thanks, Linda >=20 > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre=20 > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On > Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as=20 > > > > defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC > to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the=20 > > > > equipment to be able to do multicast forwarding, and even if=20 > > > > does, because of pruning requirements the number of groups would=20 > > > > be > very > > > > large. The average data center switch probably won't handle=20 > > > > that many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra > 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I > think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory > based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE > packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/=20 > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in=20 > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server=20 > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud=20 > > > > OS companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches=20 > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream=20 > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can > also > > > > have a heirarchical network, with the core totally Layer-3 (and=20 > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another=20 > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and > the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that=20 > > > > it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling > issues > > > can be properly described. Many data center operators have > expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come > across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being=20 > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary=20 > > > router > > > (s) > > > > * CLOS design with thousands of switches. This design will > have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > > > > > > > > > > > > _______________________________________________ > > > armd mailing list > > > armd@ietf.org > > > https://www.ietf.org/mailman/listinfo/armd > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From vishwas.ietf@gmail.com Tue Sep 20 15:10:55 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3BFBA21F8C48 for ; Tue, 20 Sep 2011 15:10:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.129 X-Spam-Level: X-Spam-Status: No, score=-3.129 tagged_above=-999 required=5 tests=[AWL=0.469, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SHS1nj6iYdOM for ; Tue, 20 Sep 2011 15:10:54 -0700 (PDT) Received: from mail-qy0-f179.google.com (mail-qy0-f179.google.com [209.85.216.179]) by ietfa.amsl.com (Postfix) with ESMTP id 56AD121F8C33 for ; Tue, 20 Sep 2011 15:10:04 -0700 (PDT) Received: by qyk33 with SMTP id 33so1110946qyk.10 for ; Tue, 20 Sep 2011 15:12:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cG+oo/GM81YeFgii1Kaw1gk6KxEAytXouUfT7V7sJqU=; b=OVRmyvyuNWYX2ED8jVc7oDNQe6T1X4dKppRfb7bVgsyU0zzkXY/wNf5/aE1NOYMasH Uh/HXyOcYwI7xX7Zw4SiPf4vYX+dNCJfcTOCKQLE0ctGd+hg1fDUi1oQmWR1nhUXuR3F 2o0FbRsYQP78MJfS3zWuabdJM9BGtotuomrbo= MIME-Version: 1.0 Received: by 10.229.40.17 with SMTP id i17mr1170311qce.205.1316556749852; Tue, 20 Sep 2011 15:12:29 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Tue, 20 Sep 2011 15:12:29 -0700 (PDT) In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> References: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> Date: Tue, 20 Sep 2011 15:12:29 -0700 Message-ID: From: Vishwas Manral To: Linda Dunbar Content-Type: multipart/alternative; boundary=00163683414c66d09204ad66c187 Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2011 22:10:55 -0000 --00163683414c66d09204ad66c187 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Linda, An overlay may not originate at the host, it could originate at the router or any other device. I am sure as the specs eveolve they will clearly seperate the functionality and packe= t formats (specified by IETF) from the location of the functionality (can be informational but not necessarily something we need to write something against). However if you have issues with the encapsulation (too much overhead/ hard to hash flows/ fragmentation or whatever else) that is something we could discuss. Also regarding ARP/ ND I think there are well knows ways to get over broadcast issues, which may require standards track work. Thanks, Vishwas On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar wro= te: > ** ** > > There are many drafts recently on various Overlay Network schemes for dat= a > centers with large number of tenants/hosts. When Overlay Network is deplo= yed > in a data center, local switches/routers within the data center network > don=92t see the hosts addresses anymore except the nodes which perform th= e > network address encapsulation/decapsulation. **** > > ** ** > > However, when a large data center has tens of thousands of tenants/hosts > which communicate to peers outside the data center, all those tenants/hos= ts > send data packets destined towards external peers to the Gateway routers = of > the data center. Each tenant has at least several hosts which need to > communicate with external peers.**** > > > **** > > For each host (which runs on a VM) which need to communicate with externa= l > world, it frequently sends ARP/ND requests to its default Gateway router. > For an application with dual stacks on, it sends both ARP & ND requests t= o > its default gateway router. **** > > ** ** > > Routers are built to forward large amount of traffic. Traffic forwarded > from line card to line card don=92t go through CPU. But ARP/ND requests t= o the > gateway routers have to be processed by the router=92s CPU. That is why w= e see > Router=92s CPU utilization reaching 40% (Merit Network statistics) or hig= her > in many data centers. **** > > ** ** > > I think that this kind of problems should be captured in the ARMD WG > problem statement. What do you think? **** > > ** ** > > Linda Dunbar**** > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --00163683414c66d09204ad66c187 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi Linda,
=A0
An overlay may not originate at the host, it could originate at the ro= uter or any other device. I am sure
as the specs eveolve they will clearly seperate the functionality and = packet formats (specified by IETF)=A0from the location of the functionality= (can be informational but not necessarily something we need to write somet= hing against).
=A0
However if you have issues with the encapsulation (too much overhead/ = hard to hash flows/ fragmentation=A0or whatever else)=A0that is something w= e could discuss.
=A0
Also regarding ARP/ ND I think there are well knows ways to get over b= roadcast issues, which may require standards track work.
=A0
Thanks,
Vishwas
On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <= span dir=3D"ltr"><linda.dunba= r@huawei.com> wrote:

=A0

There are many draft= s recently on various Overlay Network schemes for data centers with large n= umber of tenants/hosts. When Overlay Network is deployed in a data center, = local switches/routers within the data center network don=92t see the hosts= addresses anymore except the nodes which perform the network address encap= sulation/decapsulation.

=A0

However, when a larg= e data center has tens of thousands of tenants/hosts which communicate to p= eers outside the data center, all those tenants/hosts send data packets des= tined towards external peers to the Gateway routers of the data center. Eac= h tenant has at least several hosts which need to communicate with external= peers.

=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0

For each host (which= runs on a VM) which need to communicate with external world, it frequently= sends ARP/ND requests to its default Gateway router. For an application wi= th dual stacks on, it sends both ARP & ND requests to its default gatew= ay router.

=A0

Routers are built to= forward large amount of traffic. Traffic forwarded from line card to line = card don=92t go through CPU. But ARP/ND requests to the gateway routers hav= e to be processed by the router=92s CPU. That is why we see Router=92s CPU = utilization reaching 40% (Merit Network statistics) or higher =A0in many da= ta centers.

=A0

I think that this ki= nd of problems should be captured in the ARMD WG problem statement. What do= you think? =A0

=A0

Linda Dunbar<= u>


___________________________________= ____________
armd mailing list
armd@= ietf.org
ht= tps://www.ietf.org/mailman/listinfo/armd


--00163683414c66d09204ad66c187-- From vishwas.ietf@gmail.com Thu Sep 22 16:01:06 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D83A511E80C7 for ; Thu, 22 Sep 2011 16:01:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.144 X-Spam-Level: X-Spam-Status: No, score=-3.144 tagged_above=-999 required=5 tests=[AWL=0.454, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pYssUlWZ9mmw for ; Thu, 22 Sep 2011 16:01:05 -0700 (PDT) Received: from mail-qw0-f50.google.com (mail-qw0-f50.google.com [209.85.216.50]) by ietfa.amsl.com (Postfix) with ESMTP id 0FEC911E8098 for ; Thu, 22 Sep 2011 16:01:04 -0700 (PDT) Received: by qwj9 with SMTP id 9so6530132qwj.9 for ; Thu, 22 Sep 2011 16:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=i+S57aq0Hi5ZXB/bHY/2Ym4MOYNMPcJp9UHGsAKBxjo=; b=lDqMeftBTjDj+5321bH+EQZWitlAXwEJWTN+efGNSb6vUc33aSF6NcaQ2WOEGq6B8t P0uTIEdRy8ks9/L3sl4E2TISn9KVM/oT+w+Ecawshy5Gvgmxx7ShcNnuLmYJh9Fz1xSX ZiVcngqfaqnru6vEDSo03vfqwXecxW88Hz5r0= MIME-Version: 1.0 Received: by 10.229.65.229 with SMTP id k37mr2173913qci.281.1316732616225; Thu, 22 Sep 2011 16:03:36 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 16:03:36 -0700 (PDT) In-Reply-To: <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 16:03:36 -0700 Message-ID: From: Vishwas Manral To: Narasimhan Venkataramaiah Content-Type: multipart/alternative; boundary=0016e64ed514dab37d04ad8fb31a Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:01:07 -0000 --0016e64ed514dab37d04ad8fb31a Content-Type: text/plain; charset=ISO-8859-1 Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows from/to a tenant will be hashed to the same path (even with the changes in switches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for providing the randomization for hashing flows between same to/from on different paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < narave@microsoft.com> wrote: > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the VMs > subnet is split between the cloud and the enterprise? Otherwise I don't see > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible to > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all external > peers go through enterprise's gateway to reach this host which is no longer > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > > > --0016e64ed514dab37d04ad8fb31a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Simha,
=A0
The main (Standards Track)=A0change in your draft is the addition of T= NI.
=A0
A question I have is a TNI identifies a particular tenant and all flow= s from/to a tenant will be hashed to the same path (even with the changes i= n switches to do hashing to use TNI).
=A0
Why do you not use the last 8 bits which you have kept as reserved for= providing the randomization for hashing flows between same to/from on diff= erent paths?
=A0
Thanks,
Vishwas
On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Ven= kataramaiah <n= arave@microsoft.com> wrote:
The easiest from the point of vi= ew of configuration would be to route everything back through the enterpris= e - not necessarily the optimal from the enterprise point of view. Are you = referring to a scenario where the VMs subnet is split between the cloud and= the enterprise? Otherwise I don't see the implication on virtualizatio= n as its no different than getting the traffic routed to the enterprise in = the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Du= nbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sr= idharan-virtualization-nvgre-00" advertise its external facing hosts&#= 39; IP addresses to external world?

Hi Murari,

Thank you very much for sharing the= presentation.

One question:

For a host within an Enterprise = site which needs to communicate with external peers, the host either uses p= ublic IP address which is visible to external peers or uses private IP addr= ess which is translated to public address at the Enterprise site's gate= way.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.black@emc.com; <= a href=3D"mailto:armd@ietf.org">armd@ietf.org
> Subject: Re: [arm= d] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the n= vgre
> draft below.
> http://channel9.msdn.com/Even= ts/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf= .org [mailto:armd-bounces@ietf= .org] On Behalf Of
> david.black@emc.com
>= Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typica= l network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: [armd] soliciting = typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> >= ; > Hi Linda,
> > >
> > > The data packets can be tunnelled at the T= oR over say a GRE packet
> > > and the core is a Layer-3 core> > (except for the downstream ports). So we could have encapsulati= on/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> = > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refer= s to ToR switches
> > > downstream ports (facing servers)
&g= t; > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > = >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP = routing, even though the physical links are Ethernet?
> > > If = yes, we should definitely include it in the ARMD draft.
> > >
> > > Thanks,
> > > Linda
> &g= t; > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote: > > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wr= ote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> &g= t; data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >
=
--0016e64ed514dab37d04ad8fb31a-- From narave@microsoft.com Thu Sep 22 16:08:56 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3474921F8A91 for ; Thu, 22 Sep 2011 16:08:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.598 X-Spam-Level: X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9v2U6qKpubSU for ; Thu, 22 Sep 2011 16:08:54 -0700 (PDT) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.214]) by ietfa.amsl.com (Postfix) with ESMTP id 6FB1E21F8A80 for ; Thu, 22 Sep 2011 16:08:54 -0700 (PDT) Received: from TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 16:11:27 -0700 Received: from TK5EX14MLTW651.wingroup.windeploy.ntdev.microsoft.com (157.54.71.39) by TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) with Microsoft SMTP Server (TLS) id 14.1.339.2; Thu, 22 Sep 2011 16:11:26 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW651.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.39]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 16:11:26 -0700 From: Narasimhan Venkataramaiah To: Vishwas Manral Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHGgAcU1gD//4sFsA== Date: Thu, 22 Sep 2011 23:11:25 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.90] Content-Type: multipart/alternative; boundary="_000_65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1TK5EX14MBXW601w_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:08:56 -0000 --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1TK5EX14MBXW601w_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1TK5EX14MBXW601w_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get full entropy from the inner Ethernet fram= e. I take it that your comment would be to make it explicit in the draft. R= ight?

 

One

   such example could be to use the upper 8 bits= of the Key field to

   add flow based entropy and tag all the packet= s from a flow with an entropy label.

 

Simha

 <= /p>

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.o= rg
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft= is the addition of TNI.

 

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narave@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Otherwise I don't see the implicatio= n on virtualization as its no different than getting the traffic routed to = the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [<= a href=3D"mailto:armd-bounces@ietf.org">armd-bounces@ietf.org] on behal= f of Linda Dunbar [linda.dunbar@= huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@em= c.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.black@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@emc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: armd-bounces@ietf.= org [mailto:armd-bounces@ietf.= org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu= > wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu= > wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

--_000_65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1TK5EX14MBXW601w_-- From vishwas.ietf@gmail.com Thu Sep 22 16:16:58 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7518621F8B43 for ; Thu, 22 Sep 2011 16:16:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.158 X-Spam-Level: X-Spam-Status: No, score=-3.158 tagged_above=-999 required=5 tests=[AWL=0.440, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZRSNxjDGapjH for ; Thu, 22 Sep 2011 16:16:56 -0700 (PDT) Received: from mail-qw0-f50.google.com (mail-qw0-f50.google.com [209.85.216.50]) by ietfa.amsl.com (Postfix) with ESMTP id 69A5C21F8B38 for ; Thu, 22 Sep 2011 16:16:56 -0700 (PDT) Received: by qwj9 with SMTP id 9so6545418qwj.9 for ; Thu, 22 Sep 2011 16:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=GxSTfYj+fgZBPam9GlJ6l1E4l08QkRnf9BONqdJlSc4=; b=XqRShR57TCT/lOpIHWAYBpBJRlSBP2ZFp3dMP+QaQHUKgyMovGDPqlnHTnRmqv7LZW /+pf+iPkE4vTXp3PZNRm81HOYjJsenlzpCIpdAegjdbs5HrT3Ob4Ot3AM9UOe6KxJ5Ox 1af7tHec0Xxky34Mp/a8F8TpZgT1YUG5EZyvE= MIME-Version: 1.0 Received: by 10.229.231.149 with SMTP id jq21mr2198349qcb.243.1316733568768; Thu, 22 Sep 2011 16:19:28 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 16:19:28 -0700 (PDT) In-Reply-To: <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 16:19:28 -0700 Message-ID: From: Vishwas Manral To: Narasimhan Venkataramaiah Content-Type: multipart/alternative; boundary=0016e64ea93ea15cb404ad8fecf4 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:16:58 -0000 --0016e64ea93ea15cb404ad8fecf4 Content-Type: text/plain; charset=ISO-8859-1 Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (besides ofcourse that TNI needs to be parsed in the intermediate device for hashing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling you consolidate the addresses visible to the core and by providing a hash mechanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < narave@microsoft.com> wrote: > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to get > entropy. One step further would be for the switches to get full entropy from > the inner Ethernet frame. I take it that your comment would be to make it > explicit in the draft. Right?**** > > ** ** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > ** ** > > Simha**** > > ** ** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > ** ** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.*** > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes in > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the VMs > subnet is split between the cloud and the enterprise? Otherwise I don't see > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible to > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all external > peers go through enterprise's gateway to reach this host which is no longer > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > --0016e64ea93ea15cb404ad8fecf4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Simha,
=A0
I see this as the only difference between VXLAN and the NVGRE solution= (besides ofcourse that TNI needs to be parsed in the intermediate device f= or hashing and using lesser number of bytes).
=A0
I would think you should add it to your draft immediately. With tunnel= ing=A0you=A0consolidate=A0the addresses visible to the core and by providin= g a hash mechanism, you are providing some level of randomness.
=A0
The other thing you should look at is L3 (IPv4/ IPv6)=A0over NVGRE ins= tead of L2 alone.=A0I guess it would be the same comment for the VXLAN prop= osal too.
=A0
Thanks,
Vishwas
On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venk= ataramaiah <na= rave@microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get f= ull entropy from the inner Ethernet frame. I take it that your comment woul= d be to make it explicit in the draft. Right?

=A0

One

=A0=A0 such example could be to use the upper 8 bits of the= Key field to

=A0=A0 add flow based entropy and tag all the packets from = a flow with an entropy label.

=A0

Simh= a

<= /u>=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sri= dharan; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

The main (Standards Track)=A0change in your draft is= the addition of TNI.

=A0

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

=A0

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much for sharin= g the presentation.

One question:

For a host within an Enterp= rise site which needs to communicate with external peers, the host either u= ses public IP address which is visible to external peers or uses private IP= address which is translated to public address at the Enterprise site's= gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

--0016e64ea93ea15cb404ad8fecf4-- From muraris@microsoft.com Thu Sep 22 16:41:17 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6D6A21F0C40 for ; Thu, 22 Sep 2011 16:41:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.513 X-Spam-Level: X-Spam-Status: No, score=-110.513 tagged_above=-999 required=5 tests=[AWL=0.085, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AljglnmQJXGO for ; Thu, 22 Sep 2011 16:41:15 -0700 (PDT) Received: from smtp.microsoft.com (mailc.microsoft.com [131.107.115.214]) by ietfa.amsl.com (Postfix) with ESMTP id 883F221F8C59 for ; Thu, 22 Sep 2011 16:41:15 -0700 (PDT) Received: from TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 16:43:48 -0700 Received: from TK5EX14MBXC300.redmond.corp.microsoft.com ([169.254.3.19]) by TK5EX14MLTC102.redmond.corp.microsoft.com ([157.54.79.180]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 16:43:48 -0700 From: Murari Sridharan To: Vishwas Manral , Narasimhan Venkataramaiah Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgyEzmK1/alSgEuAPIK1uYKiVZVT4zqAgAadvQCAAAIvgIAAAj8A//+QzSA= Date: Thu, 22 Sep 2011 23:43:47 +0000 Message-ID: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.33] Content-Type: multipart/alternative; boundary="_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A337TK5EX14MBXC300r_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:41:17 -0000 --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A337TK5EX14MBXC300r_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A337TK5EX14MBXC300r_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Vishwas, Thanks for the f= eedback we will definitely consider adding that. I am not sure what you mea= n by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /p>

Thanks<= /p>

 

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.o= rg
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).<= /p>

 

I would think you should add it to your draft immedi= ately. With tunneling you consolidate the addresses visible = to the core and by providing a hash mechanism, you are providing some level= of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6= ) over NVGRE instead of L2 alone. I guess it would be the same co= mment for the VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

--_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A337TK5EX14MBXC300r_-- From vishwas.ietf@gmail.com Thu Sep 22 16:46:19 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 33DD321F8C64 for ; Thu, 22 Sep 2011 16:46:19 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.17 X-Spam-Level: X-Spam-Status: No, score=-3.17 tagged_above=-999 required=5 tests=[AWL=0.428, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f762fmztsUyp for ; Thu, 22 Sep 2011 16:46:17 -0700 (PDT) Received: from mail-qw0-f50.google.com (mail-qw0-f50.google.com [209.85.216.50]) by ietfa.amsl.com (Postfix) with ESMTP id 0E32521F8C60 for ; Thu, 22 Sep 2011 16:46:16 -0700 (PDT) Received: by qwj9 with SMTP id 9so6572157qwj.9 for ; Thu, 22 Sep 2011 16:48:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=3XiAYxa/h0Wr0MVfABqBJ3HchBYF8FECIIXZlBoMMms=; b=iPHBuc6ePOBcSyWix9xJxEX0NVqdsk5npZxPrM5GTEXqX5gnS3IdvaXuri/cW2FQB9 qgI+ZHMJqkHDANOOl+uBXfsFh3EKUrLC8GurjKxAvi9AadtJHmtebMJLxBoB953LYyeR K0C+ftPBq+QYAQY/nK1cRWKcb0geHqp/X64YU= MIME-Version: 1.0 Received: by 10.229.69.142 with SMTP id z14mr2248622qci.167.1316735328309; Thu, 22 Sep 2011 16:48:48 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 16:48:48 -0700 (PDT) In-Reply-To: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 16:48:48 -0700 Message-ID: From: Vishwas Manral To: Murari Sridharan Content-Type: multipart/alternative; boundary=0016e644c79881d25f04ad9055ac Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:46:19 -0000 --0016e644c79881d25f04ad9055ac Content-Type: text/plain; charset=ISO-8859-1 Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing it totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan wrote: > Vishwas, Thanks for the feedback we will definitely consider adding that. > I am not sure what you mean by doing L3 instead of L2. We allow any > arbitrary virtual topology including L3. **** > > ** ** > > Thanks**** > > ** ** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:19 PM > > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > ** ** > > Hi Simha,**** > > **** > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device for > hashing and using lesser number of bytes).**** > > **** > > I would think you should add it to your draft immediately. With > tunneling you consolidate the addresses visible to the core and by providing > a hash mechanism, you are providing some level of randomness.**** > > **** > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of > L2 alone. I guess it would be the same comment for the VXLAN proposal too. > **** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to get > entropy. One step further would be for the switches to get full entropy from > the inner Ethernet frame. I take it that your comment would be to make it > explicit in the draft. Right?**** > > **** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > **** > > Simha**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.*** > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes in > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the VMs > subnet is split between the cloud and the enterprise? Otherwise I don't see > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible to > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all external > peers go through enterprise's gateway to reach this host which is no longer > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > > ** ** > --0016e644c79881d25f04ad9055ac Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Murari,
=A0
What I am saying is the inner header should be allowed to be L3.
=A0
From the diagram you have that does not seem to be the case. Am I miss= ing it totally?
=A0
Thanks,
Vishwas
On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridhara= n <muraris@mi= crosoft.com> wrote:

Vish= was, Thanks for the feedback we will definitely consider adding that. I am = not sure what you mean by doing L3 instead of L2. We allow any arbitrary vi= rtual topology including L3.

<= /u>=A0

Than= ks

=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:19 PM=20


To: Narasimhan Venkataramaiah
Cc: Li= nda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).

=A0

I would think you should add it to your draft immedi= ately. With tunneling=A0you=A0consolidate=A0the addresses visible to the co= re and by providing a hash mechanism, you are providing some level of rando= mness.

=A0

The other thing you should look at is L3 (IPv4/ IPv6= )=A0over NVGRE instead of L2 alone.=A0I guess it would be the same comment = for the VXLAN proposal too.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave= @microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get f= ull entropy from the inner Ethernet frame. I take it that your comment woul= d be to make it explicit in the draft. Right?

=A0

One

=A0=A0 such example could be to use the upper 8 bits of the= Key field to

=A0=A0 add flow based entropy and tag all the packets from = a flow with an entropy label.

=A0

Simh= a

=A0<= /span>

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sri= dharan; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

The main (Standards Track)=A0change in your draft is= the addition of TNI.

=A0

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

=A0

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much for sharin= g the presentation.

One question:

For a host within an Enterp= rise site which needs to communicate with external peers, the host either u= ses public IP address which is visible to external peers or uses private IP= address which is translated to public address at the Enterprise site's= gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

=A0


--0016e644c79881d25f04ad9055ac-- From muraris@microsoft.com Thu Sep 22 16:48:03 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 415CD21F8B37 for ; Thu, 22 Sep 2011 16:48:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.525 X-Spam-Level: X-Spam-Status: No, score=-110.525 tagged_above=-999 required=5 tests=[AWL=0.073, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HPOhmAJRF+DS for ; Thu, 22 Sep 2011 16:48:01 -0700 (PDT) Received: from smtp.microsoft.com (mailb.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 3896A21F8B2F for ; Thu, 22 Sep 2011 16:48:01 -0700 (PDT) Received: from TK5EX14HUBC107.redmond.corp.microsoft.com (157.54.80.67) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 16:50:33 -0700 Received: from TK5EX14MBXC300.redmond.corp.microsoft.com ([169.254.3.19]) by TK5EX14HUBC107.redmond.corp.microsoft.com ([157.54.80.67]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 16:50:32 -0700 From: Murari Sridharan To: Vishwas Manral Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgyEzmK1/alSgEuAPIK1uYKiVZVT4zqAgAadvQCAAAIvgIAAAj8A//+QzSCAAHdlAP//iwsA Date: Thu, 22 Sep 2011 23:50:32 +0000 Message-ID: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.33] Content-Type: multipart/alternative; boundary="_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A3B4TK5EX14MBXC300r_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:48:03 -0000 --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A3B4TK5EX14MBXC300r_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf= .org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A3B4TK5EX14MBXC300r_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

You mean not an Ethernet = frame but some IP payload?

 

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; ar= md@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allow= ed to be L3.

 

From the diagram you have that does not seem to be t= he case. Am I missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@microsoft.com> wr= ote:

Vishwas, Thanks for= the feedback we will definitely consider adding that. I am not sure what y= ou mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /o:p>

Thanks<= /o:p>

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solu= tion (besides ofcourse that TNI needs to be parsed in the intermediate devi= ce for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tu= nneling you consolidate the addresses visible to the core an= d by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NV= GRE instead of L2 alone. I guess it would be the same comment for the = VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

--_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A3B4TK5EX14MBXC300r_-- From vishwas.ietf@gmail.com Thu Sep 22 16:51:01 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DBD8F21F8C94 for ; Thu, 22 Sep 2011 16:51:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.183 X-Spam-Level: X-Spam-Status: No, score=-3.183 tagged_above=-999 required=5 tests=[AWL=0.415, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2+ZIJs31avWQ for ; Thu, 22 Sep 2011 16:51:00 -0700 (PDT) Received: from mail-qw0-f50.google.com (mail-qw0-f50.google.com [209.85.216.50]) by ietfa.amsl.com (Postfix) with ESMTP id 9FBF021F8C91 for ; Thu, 22 Sep 2011 16:50:59 -0700 (PDT) Received: by qwj9 with SMTP id 9so6576243qwj.9 for ; Thu, 22 Sep 2011 16:53:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=rLIJ/DoTLRHoEE66YtNd0A6C+7m37KZgMJzIJt3mVEs=; b=W3143QfmpemXMlFOzgJcT0wNDgj1+WoIJwQizBbvC0TeUZYJLUZTJ9tr1tEpKvRgM9 rvHl7f/VqJ+x6C4OlJNcoTmjRSpQOl/ioPAoI5pFFWg+9JczC9K0xzXKK8/T2Y5pI9j9 jjNTpDMCOF6zm21R6lSeLB8JvHH2POW0ar7Rg= MIME-Version: 1.0 Received: by 10.229.69.142 with SMTP id z14mr2251005qci.167.1316735610803; Thu, 22 Sep 2011 16:53:30 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 16:53:30 -0700 (PDT) In-Reply-To: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 16:53:30 -0700 Message-ID: From: Vishwas Manral To: Murari Sridharan Content-Type: multipart/alternative; boundary=0016e644c7985857b104ad906615 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:51:02 -0000 --0016e644c7985857b104ad906615 Content-Type: text/plain; charset=ISO-8859-1 Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan wrote: > You mean not an Ethernet frame but some IP payload?**** > > ** ** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:49 PM > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > ** ** > > Murari,**** > > **** > > What I am saying is the inner header should be allowed to be L3. **** > > **** > > From the diagram you have that does not seem to be the case. Am I missing > it totally?**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote:**** > > Vishwas, Thanks for the feedback we will definitely consider adding that. I > am not sure what you mean by doing L3 instead of L2. We allow any arbitrary > virtual topology including L3. **** > > **** > > Thanks**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:19 PM **** > > > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device for > hashing and using lesser number of bytes).**** > > **** > > I would think you should add it to your draft immediately. With > tunneling you consolidate the addresses visible to the core and by providing > a hash mechanism, you are providing some level of randomness.**** > > **** > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of > L2 alone. I guess it would be the same comment for the VXLAN proposal too. > **** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to get > entropy. One step further would be for the switches to get full entropy from > the inner Ethernet frame. I take it that your comment would be to make it > explicit in the draft. Right?**** > > **** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > **** > > Simha**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.*** > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes in > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the VMs > subnet is split between the cloud and the enterprise? Otherwise I don't see > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible to > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all external > peers go through enterprise's gateway to reach this host which is no longer > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > > **** > > ** ** > --0016e644c7985857b104ad906615 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Murari,
=A0
Yes that is what I mean.
=A0
Thanks,
Vishwas
On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridhara= n <muraris@mi= crosoft.com> wrote:

You = mean not an Ethernet frame but some IP payload?

=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.blac= k@emc.com; armd@ietf= .org=20


Subject: Re: [armd] how does "draft-sridh= aran-virtualization-nvgre-00" advertise its external facing hosts'= IP addresses to external world?

=A0

Murari,

=A0

What I am saying is the inner header should be allow= ed to be L3.

=A0

From the diagram you have that does not seem to be t= he case. Am I missing it totally?

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Vish= was, Thanks for the feedback we will definitely consider adding that. I am = not sure what you mean by doing L3 instead of L2. We allow any arbitrary vi= rtual topology including L3.

=A0<= /span>

Than= ks

=A0=

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:19 PM


To: Naras= imhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; = armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).

=A0

I would think you should add it to your draft immedi= ately. With tunneling=A0you=A0consolidate=A0the addresses visible to the co= re and by providing a hash mechanism, you are providing some level of rando= mness.

=A0

The other thing you should look at is L3 (IPv4/ IPv6= )=A0over NVGRE instead of L2 alone.=A0I guess it would be the same comment = for the VXLAN proposal too.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave= @microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get f= ull entropy from the inner Ethernet frame. I take it that your comment woul= d be to make it explicit in the draft. Right?

=A0

One

=A0=A0 such example could be to use the upper 8 bits of the= Key field to

=A0=A0 add flow based entropy and tag all the packets from = a flow with an entropy label.

=A0

Simh= a

=A0<= /span>

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sri= dharan; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

The main (Standards Track)=A0change in your draft is= the addition of TNI.

=A0

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

=A0

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much for sharin= g the presentation.

One question:

For a host within an Enterp= rise site which needs to communicate with external peers, the host either u= ses public IP address which is visible to external peers or uses private IP= address which is translated to public address at the Enterprise site's= gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

=A0

=A0


--0016e644c7985857b104ad906615-- From muraris@microsoft.com Thu Sep 22 16:57:56 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A4D9C11E8098 for ; Thu, 22 Sep 2011 16:57:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.534 X-Spam-Level: X-Spam-Status: No, score=-110.534 tagged_above=-999 required=5 tests=[AWL=0.064, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l4LWyFsNbcqL for ; Thu, 22 Sep 2011 16:57:54 -0700 (PDT) Received: from smtp.microsoft.com (mailb.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 967E511E8073 for ; Thu, 22 Sep 2011 16:57:54 -0700 (PDT) Received: from TK5EX14HUBC106.redmond.corp.microsoft.com (157.54.80.61) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 17:00:27 -0700 Received: from TK5EX14MBXC300.redmond.corp.microsoft.com ([169.254.3.19]) by TK5EX14HUBC106.redmond.corp.microsoft.com ([157.54.80.61]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 17:00:26 -0700 From: Murari Sridharan To: Vishwas Manral Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgyEzmK1/alSgEuAPIK1uYKiVZVT4zqAgAadvQCAAAIvgIAAAj8A//+QzSCAAHdlAP//iwsAgAB2RgD//4yXuA== Date: Fri, 23 Sep 2011 00:00:26 +0000 Message-ID: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: multipart/alternative; boundary="_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A472TK5EX14MBXC300r_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 23:57:56 -0000 --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A472TK5EX14MBXC300r_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Do you have a scenario in mind? Sent from my Windows Phone ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf= .org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A472TK5EX14MBXC300r_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Do you have a= scenario in mind?

Sent from my Windows Phone
From: Vishwa= s Manral
Sent: 9/22/2= 011 4:55 PM
To: Murari= Sridharan
Cc: Narasi= mhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [a= rmd] how does "draft-sridharan-virtualization-nvgre-00" advertise= its external facing hosts' IP addresses to external world?

Hi Murari,
 
Yes that is what I mean.
 
Thanks,
Vishwas
On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridhara= n <muraris@microsoft.com><= /span> wrote:

You me= an not an Ethernet frame but some IP payload?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allow= ed to be L3.

 

From the diagram you have that does not seem to be t= he case. Am I missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Vishwa= s, Thanks for the feedback we will definitely consider adding that. I am no= t sure what you mean by doing L3 instead of L2. We allow any arbitrary virt= ual topology including L3.

 =

Thanks=

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immedi= ately. With tunneling you consolidate the addresses visible = to the core and by providing a hash mechanism, you are providing some level= of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6= ) over NVGRE instead of L2 alone. I guess it would be the same co= mment for the VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave= @microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get full entropy from the inner Ethernet fram= e. I take it that your comment would be to make it explicit in the draft. R= ight?

 

One

   such example could be to use the upper 8 bits of the Key= field to

   add flow based entropy and tag all the packets from a fl= ow with an entropy label.

 

Simha<= /span>

 =

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft= is the addition of TNI.

 

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Otherwise I don't see the implicatio= n on virtualization as its no different than getting the traffic routed to = the enterprise in the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 


--_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2A472TK5EX14MBXC300r_-- From vishwas.ietf@gmail.com Thu Sep 22 19:35:14 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C139E11E8094 for ; Thu, 22 Sep 2011 19:35:14 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.194 X-Spam-Level: X-Spam-Status: No, score=-3.194 tagged_above=-999 required=5 tests=[AWL=0.404, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2ng4+tCtb8Ji for ; Thu, 22 Sep 2011 19:35:12 -0700 (PDT) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 782DF11E808F for ; Thu, 22 Sep 2011 19:35:12 -0700 (PDT) Received: by qyk32 with SMTP id 32so6863282qyk.10 for ; Thu, 22 Sep 2011 19:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=nb2ypdT3oQWmisywdGALv70vF3Icd4dpi5+bX/nPvIg=; b=Cp99CSm6HAX+lSYgjR3g3mkZuEyfaYXoXXuhscntDPRsfBXUkPeJ6mxareHZDWiSd3 5je/ib7sb4LwkbHkpv60vrpVWlMlkpf7EbheASmnccanAnSBvh+QjDcrtovhc8d0amdh pYym8p9/F0wP5lodQfQ/QelZ/zK12wz72GWQ4= MIME-Version: 1.0 Received: by 10.229.65.229 with SMTP id k37mr2286027qci.281.1316745465157; Thu, 22 Sep 2011 19:37:45 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 19:37:45 -0700 (PDT) In-Reply-To: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 19:37:45 -0700 Message-ID: From: Vishwas Manral To: Murari Sridharan Content-Type: multipart/alternative; boundary=0016e64ed514b5d86604ad92b1ff Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 02:35:14 -0000 --0016e64ed514b5d86604ad92b1ff Content-Type: text/plain; charset=ISO-8859-1 Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan wrote: > Do you have a scenario in mind? > ------------------------------ > From: Vishwas Manral > Sent: 9/22/2011 4:55 PM > > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > Hi Murari, > > Yes that is what I mean. > > Thanks, > Vishwas > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan wrote: > >> You mean not an Ethernet frame but some IP payload?**** >> >> ** ** >> >> *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] >> *Sent:* Thursday, September 22, 2011 4:49 PM >> *To:* Murari Sridharan >> *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; >> armd@ietf.org >> >> *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" >> advertise its external facing hosts' IP addresses to external world?**** >> >> ** ** >> >> Murari,**** >> >> **** >> >> What I am saying is the inner header should be allowed to be L3. **** >> >> **** >> >> From the diagram you have that does not seem to be the case. Am I missing >> it totally?**** >> >> **** >> >> Thanks,**** >> >> Vishwas**** >> >> On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan >> wrote:**** >> >> Vishwas, Thanks for the feedback we will definitely consider adding that. >> I am not sure what you mean by doing L3 instead of L2. We allow any >> arbitrary virtual topology including L3. **** >> >> **** >> >> Thanks**** >> >> **** >> >> *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] >> *Sent:* Thursday, September 22, 2011 4:19 PM **** >> >> >> *To:* Narasimhan Venkataramaiah >> *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org >> *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" >> advertise its external facing hosts' IP addresses to external world?**** >> >> **** >> >> Hi Simha,**** >> >> **** >> >> I see this as the only difference between VXLAN and the NVGRE solution >> (besides ofcourse that TNI needs to be parsed in the intermediate device for >> hashing and using lesser number of bytes).**** >> >> **** >> >> I would think you should add it to your draft immediately. With >> tunneling you consolidate the addresses visible to the core and by providing >> a hash mechanism, you are providing some level of randomness.**** >> >> **** >> >> The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead >> of L2 alone. I guess it would be the same comment for the VXLAN proposal >> too.**** >> >> **** >> >> Thanks,**** >> >> Vishwas**** >> >> On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < >> narave@microsoft.com> wrote:**** >> >> The draft mentions exactly this as one use of the reserved 8 bits in >> Section 4. An NVGRE endpoint could use the 8 bits to further distribute >> flows belonging to a particular TNI and the switches use all 32 bits to get >> entropy. One step further would be for the switches to get full entropy from >> the inner Ethernet frame. I take it that your comment would be to make it >> explicit in the draft. Right?**** >> >> **** >> >> One**** >> >> such example could be to use the upper 8 bits of the Key field to**** >> >> add flow based entropy and tag all the packets from a flow with an >> entropy label.**** >> >> **** >> >> Simha**** >> >> **** >> >> *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] >> *Sent:* Thursday, September 22, 2011 4:04 PM >> *To:* Narasimhan Venkataramaiah >> *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org >> *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" >> advertise its external facing hosts' IP addresses to external world?**** >> >> **** >> >> Hi Simha,**** >> >> **** >> >> The main (Standards Track) change in your draft is the addition of TNI.** >> ** >> >> **** >> >> A question I have is a TNI identifies a particular tenant and all flows >> from/to a tenant will be hashed to the same path (even with the changes in >> switches to do hashing to use TNI).**** >> >> **** >> >> Why do you not use the last 8 bits which you have kept as reserved for >> providing the randomization for hashing flows between same to/from on >> different paths?**** >> >> **** >> >> Thanks,**** >> >> Vishwas**** >> >> On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < >> narave@microsoft.com> wrote:**** >> >> The easiest from the point of view of configuration would be to route >> everything back through the enterprise - not necessarily the optimal from >> the enterprise point of view. Are you referring to a scenario where the VMs >> subnet is split between the cloud and the enterprise? Otherwise I don't see >> the implication on virtualization as its no different than getting the >> traffic routed to the enterprise in the first case. >> >> Simha >> >> ________________________________________ >> From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda >> Dunbar [linda.dunbar@huawei.com] >> Sent: Sunday, September 18, 2011 7:06 AM >> To: Murari Sridharan; david.black@emc.com; armd@ietf.org >> Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" >> advertise its external facing hosts' IP addresses to external world?**** >> >> >> Hi Murari, >> >> Thank you very much for sharing the presentation. >> >> One question: >> >> For a host within an Enterprise site which needs to communicate with >> external peers, the host either uses public IP address which is visible to >> external peers or uses private IP address which is translated to public >> address at the Enterprise site's gateway. >> >> When this host is moved to "Cloud data center", will the "Cloud Data >> center" advertise this host address to external peers? Or will all external >> peers go through enterprise's gateway to reach this host which is no longer >> residing in the enterprise site? >> >> Thanks, Linda >> >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >> > Murari Sridharan >> > Sent: Saturday, September 17, 2011 3:02 PM >> > To: david.black@emc.com; armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > FYI, here is a talk that I gave last week in relation to the nvgre >> > draft below. >> > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T >> > >> > Thanks >> > Murari >> > >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >> > david.black@emc.com >> > Sent: Friday, September 16, 2011 6:14 AM >> > To: armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > And two more drafts on this topic: >> > >> > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt >> > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt >> > >> > The edge switches could be the software switches in hypervisors. >> > >> > Thanks, >> > --David >> > >> > >> > > -----Original Message----- >> > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf >> > > Of Warren Kumari >> > > Sent: Wednesday, August 31, 2011 3:16 PM >> > > To: Vishwas Manral >> > > Cc: armd@ietf.org >> > > Subject: Re: [armd] soliciting typical network designs for ARMD >> > > >> > > >> > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >> > > >> > > > Hi Linda/ Anoop, >> > > > >> > > > Here is the example of the design I was talking about, as defined >> > by google. >> > > >> > > Just a clarification -- s/as defined by google/as described by >> > someone >> > > who happens to work for google/ >> > > >> > > W >> > > >> > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt >> > > > >> > > > Thanks, >> > > > Vishwas >> > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani >> > wrote: >> > > > >> > > > >>>> >> > > > (though I think if there was a standard way to map Multicast MAC to >> > > > Multicast IP, they could >> > > probably use such a standard mechanisms). >> > > > >>>> >> > > > >> > > > They can do that, but then this imposes requirements on the >> > > > equipment to be able to do multicast forwarding, and even if does, >> > > > because of pruning requirements the number of groups would be very >> > > > large. The average data center switch probably won't handle that >> > > > many groups. >> > > > >> > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral >> > wrote: >> > > > Hi Anoop, >> > > > >> > > > From what I know they do not use Multicast GRE (I hear the extra 4 >> > > > bytes in the GRE header is a >> > > proprietery extension). >> > > > >> > > > I think a directory based mechanism is what is used (though I think >> > > > if there was a standard way to >> > > map Multicast MAC to Multicast IP, they could probably use such a >> > standard mechanisms). >> > > > >> > > > Thanks, >> > > > Vishwas >> > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani >> > wrote: >> > > > Hi Vishwas, >> > > > >> > > > How do they get multicast through the network in that case? >> > > > Are they planning to use multicast GRE, or just use directory based >> > > > lookups and not worry about multicast applications for now? >> > > > >> > > > Anoop >> > > > >> > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral >> > wrote: >> > > > Hi Linda, >> > > > >> > > > The data packets can be tunnelled at the ToR over say a GRE packet >> > > > and the core is a Layer-3 core >> > > (except for the downstream ports). So we could have encapsulation/ >> > > decapsulation of L2 over GRE at the ToR. >> > > > >> > > > The very same thing can be done at the hypervisor layer too, in >> > > > which case the entire DC network >> > > would look like a Layer-3 flat network including the ToR to server >> > > link and the hypervisor would do the tunneling. >> > > > >> > > > I am not sure if you got the points above or not. I know cloud OS >> > > > companies that provide the service >> > > and have big announced customers. >> > > > >> > > > Thanks, >> > > > Vishwas >> > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar >> > wrote: >> > > > Vishwas, >> > > > >> > > > In my mind the bullet 1) in the list refers to ToR switches >> > > > downstream ports (facing servers) >> > > running Layer 2 and ToR uplinks ports run IP Layer 3. >> > > > >> > > > Have you seen data center networks with ToR switches downstream >> > > > ports (i.e. facing servers) enabling >> > > IP routing, even though the physical links are Ethernet? >> > > > If yes, we should definitely include it in the ARMD draft. >> > > > >> > > > Thanks, >> > > > Linda >> > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral >> > wrote: >> > > > Hi Linda, >> > > > I am unsure what you mean by this, but: >> > > > * layer 3 all the way to TOR (Top of Rack switches), We can also >> > > > have a heirarchical network, with the core totally Layer-3 (and >> > > > having seperate >> > > routing), from the hosts still in a large Layer-3 subnet. Another >> > > aspect could be to have a totally >> > > Layer-3 network. >> > > > >> > > > The difference between them is the link between the servers and the >> > ToR. >> > > > >> > > > Thanks, >> > > > Vishwas >> > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar >> > wrote: >> > > > During the 81st IETF ARMD WG discussion, it was suggested that it >> > is >> > > > necessary to document typical >> > > data center network designs so that address resolution scaling issues >> > > can be properly described. Many data center operators have expressed >> > that they can't openly reveal their detailed network designs. >> > > Therefore, we only want to document anonymous designs without too >> > much >> > > detail. During the journey of establishing ARMD, we have come across >> > the following typical data center network designs: >> > > > * layer 3 all the way to TOR (Top of Rack switches), >> > > > * large layer 2 with hundreds (or thousands) of ToRs being >> > > > interconnected by Layer 2. This >> > > design will have thousands of hosts under the L2/L3 boundary router >> > > (s) >> > > > * CLOS design with thousands of switches. This design will have >> > > > thousands of hosts under the >> > > L2/L3 boundary router(s) >> > > > We have heard that each of the designs above has its own problems. >> > > > ARMD problem statements might >> > > need to document DC problems under each typical design. >> > > > Please send feedback to us (either to the armd email list or to >> > the >> > > > ARMD chair Benson & Linda) to >> > > indicate if we have missed any typical Data Center network designs. >> > > > >> > > > Your contribution can greatly accelerate the progress of ARMD WG. >> > > > >> > > > Thank you very much. >> > > > >> > > > Linda & Benson >> > > >**** >> >> **** >> >> ** ** >> >> > --0016e64ed514b5d86604ad92b1ff Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Murari think IP mobility. :)
=A0
On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridhara= n <muraris@mi= crosoft.com> wrote:
Do you have= a scenario in mind?
From: Vishwas Manral
Sent: 9/22/2011 4:55 PM=20

To: Murari Sridharan
Cc: Narasimhan= Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre= -00" advertise its external facing hosts' IP addresses to external= world?

Hi Murari,
=A0
Yes that is what I mean.
=A0
Thanks,
Vishwas
On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridhara= n <muraris@microsoft.com> wrote:

You = mean not an Ethernet frame but some IP payload?

=A0=

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.blac= k@emc.com; armd@ietf= .org


Subject: Re: [armd] how does "draft-sridharan-virtuali= zation-nvgre-00" advertise its external facing hosts' IP addresses= to external world?

=A0

Murari,

=A0

What I am saying is the inner header should be allow= ed to be L3.

=A0

From the diagram you have that does not seem to be t= he case. Am I missing it totally?

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Vish= was, Thanks for the feedback we will definitely consider adding that. I am = not sure what you mean by doing L3 instead of L2. We allow any arbitrary vi= rtual topology including L3.

=A0<= /span>

Than= ks

= =A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:19 PM


To: Naras= imhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; = armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).

=A0

I would think you should add it to your draft immedi= ately. With tunneling=A0you=A0consolidate=A0the addresses visible to the co= re and by providing a hash mechanism, you are providing some level of rando= mness.

=A0

The other thing you should look at is L3 (IPv4/ IPv6= )=A0over NVGRE instead of L2 alone.=A0I guess it would be the same comment = for the VXLAN proposal too.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave= @microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get f= ull entropy from the inner Ethernet frame. I take it that your comment woul= d be to make it explicit in the draft. Right?

=A0

One

=A0=A0 such example could be to use the upper 8 bits of the= Key field to

=A0=A0 add flow based entropy and tag all the packets from = a flow with an entropy label.

=A0

Simh= a

=A0<= /span>

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sri= dharan; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

The main (Standards Track)=A0change in your draft is= the addition of TNI.

=A0

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

=A0

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much for sharin= g the presentation.

One question:

For a host within an Enterp= rise site which needs to communicate with external peers, the host either u= ses public IP address which is visible to external peers or uses private IP= address which is translated to public address at the Enterprise site's= gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

=A0

=A0



--0016e64ed514b5d86604ad92b1ff-- From muraris@microsoft.com Thu Sep 22 19:38:46 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A8E5211E808E for ; Thu, 22 Sep 2011 19:38:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.541 X-Spam-Level: X-Spam-Status: No, score=-110.541 tagged_above=-999 required=5 tests=[AWL=0.057, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LwDDjQmNqcwL for ; Thu, 22 Sep 2011 19:38:41 -0700 (PDT) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id A5ADA21F8B5C for ; Thu, 22 Sep 2011 19:38:41 -0700 (PDT) Received: from TK5EX14HUBC107.redmond.corp.microsoft.com (157.54.80.67) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 19:41:14 -0700 Received: from TK5EX14MBXC300.redmond.corp.microsoft.com ([169.254.3.19]) by TK5EX14HUBC107.redmond.corp.microsoft.com ([157.54.80.67]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 19:41:14 -0700 From: Murari Sridharan To: Vishwas Manral Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgyEzmK1/alSgEuAPIK1uYKiVZVT4zqAgAadvQCAAAIvgIAAAj8A//+QzSCAAHdlAP//iwsAgAB2RgD//4yXuAAUKaaAAA6W6XA= Date: Fri, 23 Sep 2011 02:41:13 +0000 Message-ID: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.35] Content-Type: multipart/alternative; boundary="_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2AD9FTK5EX14MBXC300r_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 02:38:46 -0000 --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2AD9FTK5EX14MBXC300r_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable In the context of the proposal here where you can carry your IP around wher= ever you go doesn't that already equivalent to IP mobility? If I understand= you right you are simply saying don't just restrict the draft to MAC-in-GR= E but do IP-in-GRE right? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 7:38 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf= .org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2AD9FTK5EX14MBXC300r_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

In the context of the pro= posal here where you can carry your IP around wherever you go doesn’t= that already equivalent to IP mobility? If I understand you right you are simply saying don’t just restrict the draft to MAC-in-GRE bu= t do IP-in-GRE right?

 

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 7:38 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; ar= md@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari think IP mobility. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan &l= t;muraris@microsoft.com> wr= ote:

Do you have a scenario in mind?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.black@emc.co= m; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

Hi Murari,

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

You mean not an Eth= ernet frame but some IP payload?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allowed to be L3.

 

From the diagram you have that does not seem to be the case. Am I = missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Vishwas, Thanks for= the feedback we will definitely consider adding that. I am not sure what y= ou mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /o:p>

Thanks<= /o:p>

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solu= tion (besides ofcourse that TNI needs to be parsed in the intermediate devi= ce for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tu= nneling you consolidate the addresses visible to the core an= d by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NV= GRE instead of L2 alone. I guess it would be the same comment for the = VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

--_000_EF5EF2B13ED09B4F871D9A0DBCA463C216C2AD9FTK5EX14MBXC300r_-- From vishwas.ietf@gmail.com Thu Sep 22 20:05:21 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DAAE411E8091 for ; Thu, 22 Sep 2011 20:05:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.205 X-Spam-Level: X-Spam-Status: No, score=-3.205 tagged_above=-999 required=5 tests=[AWL=0.393, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bwAL9BmzLhjw for ; Thu, 22 Sep 2011 20:05:19 -0700 (PDT) Received: from mail-qy0-f179.google.com (mail-qy0-f179.google.com [209.85.216.179]) by ietfa.amsl.com (Postfix) with ESMTP id 905D511E808E for ; Thu, 22 Sep 2011 20:05:19 -0700 (PDT) Received: by qyk33 with SMTP id 33so3509674qyk.10 for ; Thu, 22 Sep 2011 20:07:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZbkjEZUrdAsjO0FaHgbUohWPpD/br5HRB8HiVcnSbWE=; b=phIIsh91v/Yq9AaE7QRtDdrMXI5oReboVz4Eu0GXY8e+/f4IzA3DZthEV+tXrcq1dG RP9WC1JPFprS9E080aydz146St5ebl+9JLKULxxg4yU/8HPqEcscEwbjN5DzP0RuUqxb fmkVCNAeohPwaxU8KKnaqqi+89j0wPMut0szE= MIME-Version: 1.0 Received: by 10.224.216.197 with SMTP id hj5mr2622006qab.24.1316747272341; Thu, 22 Sep 2011 20:07:52 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 20:07:52 -0700 (PDT) In-Reply-To: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 20:07:52 -0700 Message-ID: From: Vishwas Manral To: Murari Sridharan Content-Type: multipart/alternative; boundary=20cf300fb3c76d472d04ad931d15 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 03:05:22 -0000 --20cf300fb3c76d472d04ad931d15 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Murari, You could interpret it that way, but I see the scope increase is considerably more than just removing the MAC header, though that would be just one such advantage. Thanks, Vishwas On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan wr= ote: > In the context of the proposal here where you can carry your IP around > wherever you go doesn=92t that already equivalent to IP mobility? If I > understand you right you are simply saying don=92t just restrict the draf= t to > MAC-in-GRE but do IP-in-GRE right?**** > > ** ** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 7:38 PM > > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > ** ** > > Murari think IP mobility. :)**** > > **** > > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote:**** > > Do you have a scenario in mind?**** > ------------------------------ > > *From: *Vishwas Manral > *Sent: *9/22/2011 4:55 PM **** > > > *To: *Murari Sridharan > *Cc: *Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject: *Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > Hi Murari,**** > > **** > > Yes that is what I mean.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote:**** > > You mean not an Ethernet frame but some IP payload?**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:49 PM > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org **** > > > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Murari,**** > > **** > > What I am saying is the inner header should be allowed to be L3. **** > > **** > > From the diagram you have that does not seem to be the case. Am I missing > it totally?**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote:**** > > Vishwas, Thanks for the feedback we will definitely consider adding that.= I > am not sure what you mean by doing L3 instead of L2. We allow any arbitra= ry > virtual topology including L3. **** > > **** > > Thanks**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:19 PM **** > > > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device = for > hashing and using lesser number of bytes).**** > > **** > > I would think you should add it to your draft immediately. With > tunneling you consolidate the addresses visible to the core and by provid= ing > a hash mechanism, you are providing some level of randomness.**** > > **** > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead = of > L2 alone. I guess it would be the same comment for the VXLAN proposal too= . > **** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to g= et > entropy. One step further would be for the switches to get full entropy f= rom > the inner Ethernet frame. I take it that your comment would be to make it > explicit in the draft. Right?**** > > **** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > **** > > Simha**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.**= * > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes i= n > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the V= Ms > subnet is split between the cloud and the enterprise? Otherwise I don't s= ee > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible t= o > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all extern= al > peers go through enterprise's gateway to reach this host which is no long= er > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > > **** > > **** > > ** ** > > ** ** > --20cf300fb3c76d472d04ad931d15 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Murari,
=A0
You could interpret it that way, but I see the scope increase is consi= derably more than just removing the MAC header, though that would be just o= ne such advantage.
=A0
Thanks,
Vishwas

On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridhara= n <muraris@mi= crosoft.com> wrote:

In t= he context of the proposal here where you can carry your IP around wherever= you go doesn=92t that already equivalent to IP mobility? If I understand y= ou right you are simply saying don=92t just restrict the draft to MAC-in-GR= E but do IP-in-GRE right?

=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 7:38 PM=20


To: Murari Sridharan
Cc: Narasimhan = Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Murari think IP mobility. :)

=A0

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Do you have a scenar= io in mind?

From: = Vishwas Manral
Sent: 9/22/2011= 4:55 PM


To: Murari Srid= haran
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-= 00" advertise its external facing hosts' IP addresses to external = world?

Hi Murari,

=A0

Yes that is what I mean.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

You = mean not an Ethernet frame but some IP payload?

=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.blac= k@emc.com; armd@ietf= .org


Subject: Re: [armd] how does "draft-= sridharan-virtualization-nvgre-00" advertise its external facing hosts= ' IP addresses to external world?

=A0

Murari,

=A0

What I am saying is the inner header should be allow= ed to be L3.

=A0

From the diagram you have that does not seem to be t= he case. Am I missing it totally?

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Vish= was, Thanks for the feedback we will definitely consider adding that. I am = not sure what you mean by doing L3 instead of L2. We allow any arbitrary vi= rtual topology including L3.

=A0<= /span>

Than= ks

=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:19 PM


To: Naras= imhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; = armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).

=A0

I would think you should add it to your draft immedi= ately. With tunneling=A0you=A0consolidate=A0the addresses visible to the co= re and by providing a hash mechanism, you are providing some level of rando= mness.

=A0

The other thing you should look at is L3 (IPv4/ IPv6= )=A0over NVGRE instead of L2 alone.=A0I guess it would be the same comment = for the VXLAN proposal too.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave= @microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get f= ull entropy from the inner Ethernet frame. I take it that your comment woul= d be to make it explicit in the draft. Right?

=A0

One

=A0=A0 such example could be to use the upper 8 bits of the= Key field to

=A0=A0 add flow based entropy and tag all the packets from = a flow with an entropy label.

=A0

Simh= a

=A0<= /span>

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sri= dharan; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

The main (Standards Track)=A0change in your draft is= the addition of TNI.

=A0

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

=A0

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much for sharin= g the presentation.

One question:

For a host within an Enterp= rise site which needs to communicate with external peers, the host either u= ses public IP address which is visible to external peers or uses private IP= address which is translated to public address at the Enterprise site's= gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

=A0

=A0

=A0

=A0


--20cf300fb3c76d472d04ad931d15-- From narave@microsoft.com Thu Sep 22 20:09:50 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 906BE21F8508 for ; Thu, 22 Sep 2011 20:09:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.598 X-Spam-Level: X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3ML4P-H0OEEJ for ; Thu, 22 Sep 2011 20:09:41 -0700 (PDT) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.214]) by ietfa.amsl.com (Postfix) with ESMTP id C838F21F84DC for ; Thu, 22 Sep 2011 20:09:39 -0700 (PDT) Received: from TK5EX14MLTC103.redmond.corp.microsoft.com (157.54.79.174) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 20:12:13 -0700 Received: from TK5EX14MLTW651.wingroup.windeploy.ntdev.microsoft.com (157.54.71.39) by TK5EX14MLTC103.redmond.corp.microsoft.com (157.54.79.174) with Microsoft SMTP Server (TLS) id 14.1.339.2; Thu, 22 Sep 2011 20:12:12 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW651.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.39]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 20:12:12 -0700 From: Narasimhan Venkataramaiah To: Vishwas Manral , Murari Sridharan Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHGgAcU1gD//4sFsIAAeWoAgAAGy4CAAAFnAIAAAHwAgAAA1ACAAAHwAIAAK/SAgAAA+ICAAAdyAP//i4Ng Date: Fri, 23 Sep 2011 03:12:11 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD37351@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.90] Content-Type: multipart/alternative; boundary="_000_65755BEBE02F7C41BD4F137AED91DA5E2FD37351TK5EX14MBXW601w_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 03:09:50 -0000 --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD37351TK5EX14MBXW601w_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Do you mean in the context of network virtualization or in general its usef= ul? Mobility is already one aspect of network virtualization and its satisf= ied by the MAC in GRE option. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 8:08 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf= .org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, You could interpret it that way, but I see the scope increase is considerab= ly more than just removing the MAC header, though that would be just one su= ch advantage. Thanks, Vishwas On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan > wrote: In the context of the proposal here where you can carry your IP around wher= ever you go doesn't that already equivalent to IP mobility? If I understand= you right you are simply saying don't just restrict the draft to MAC-in-GR= E but do IP-in-GRE right? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 7:38 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD37351TK5EX14MBXW601w_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Do you mean in the contex= t of network virtualization or in general its useful? Mobility is already o= ne aspect of network virtualization and its satisfied by the MAC in GRE option.

 <= /p>

Simha

 <= /p>

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 8:08 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; ar= md@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

You could interpret it that way, but I see the scope= increase is considerably more than just removing the MAC header, though th= at would be just one such advantage.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan &l= t;muraris@microsoft.com> wr= ote:

In the context of t= he proposal here where you can carry your IP around wherever you go doesn&#= 8217;t that already equivalent to IP mobility? If I understand you right you are simply saying don’t just restrict = the draft to MAC-in-GRE but do IP-in-GRE right?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 7:38 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari think IP mobility. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Do you have a scenario in mind?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.co= m; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgr= e-00" advertise its external facing hosts' IP addresses to external wo= rld?

Hi Murari,

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

You mean not an Eth= ernet frame but some IP payload?

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allowed to be L3.

 

From the diagram you have that does not seem to be the case. Am I = missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Vishwas, Thanks for= the feedback we will definitely consider adding that. I am not sure what y= ou mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /o:p>

Thanks<= /o:p>

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solu= tion (besides ofcourse that TNI needs to be parsed in the intermediate devi= ce for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tu= nneling you consolidate the addresses visible to the core an= d by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NV= GRE instead of L2 alone. I guess it would be the same comment for the = VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

 

--_000_65755BEBE02F7C41BD4F137AED91DA5E2FD37351TK5EX14MBXW601w_-- From vishwas.ietf@gmail.com Thu Sep 22 20:15:30 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DC49B11E8090 for ; Thu, 22 Sep 2011 20:15:30 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.215 X-Spam-Level: X-Spam-Status: No, score=-3.215 tagged_above=-999 required=5 tests=[AWL=0.383, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Zi+XRJnGB1ie for ; Thu, 22 Sep 2011 20:15:28 -0700 (PDT) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 71B2A11E808E for ; Thu, 22 Sep 2011 20:15:28 -0700 (PDT) Received: by qyk32 with SMTP id 32so6891310qyk.10 for ; Thu, 22 Sep 2011 20:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=FuECHID71zZCOH880+iyGjpPtn0a4W98ecdyusG7ur4=; b=xAogQ4xwHViZdLCLS6Te+iHmMmV3XSn+gdLbzRO+Uy8ORMMed0wPBCoCVBIMZMzS6l BpsS1MJ5u470Njd6yqeGApOnuxI39NocxQpz3MdTem+aLYnyUuyaVks7ZOJqCcMc9fND Lfb+jGyBrQz+aadUjHqfruCgwFAiU2wBPgnlk= MIME-Version: 1.0 Received: by 10.229.69.142 with SMTP id z14mr2357263qci.167.1316747879501; Thu, 22 Sep 2011 20:17:59 -0700 (PDT) Received: by 10.229.28.70 with HTTP; Thu, 22 Sep 2011 20:17:59 -0700 (PDT) In-Reply-To: <65755BEBE02F7C41BD4F137AED91DA5E2FD37351@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD37351@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Date: Thu, 22 Sep 2011 20:17:59 -0700 Message-ID: From: Vishwas Manral To: Narasimhan Venkataramaiah Content-Type: multipart/alternative; boundary=0016e644c7989dd1ad04ad934155 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 03:15:31 -0000 --0016e644c7989dd1ad04ad934155 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Simha, I am talking about the difference between Layer-2 mobility and Layer-3 mobility. Thanks, Vishwas On Thu, Sep 22, 2011 at 8:12 PM, Narasimhan Venkataramaiah < narave@microsoft.com> wrote: > Do you mean in the context of network virtualization or in general its > useful? Mobility is already one aspect of network virtualization and its > satisfied by the MAC in GRE option. **** > > ** ** > > Simha**** > > ** ** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 8:08 PM > > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > ** ** > > Murari,**** > > **** > > You could interpret it that way, but I see the scope increase is > considerably more than just removing the MAC header, though that would be > just one such advantage.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan > wrote:**** > > In the context of the proposal here where you can carry your IP around > wherever you go doesn=92t that already equivalent to IP mobility? If I > understand you right you are simply saying don=92t just restrict the draf= t to > MAC-in-GRE but do IP-in-GRE right?**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 7:38 PM **** > > > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Murari think IP mobility. :)**** > > **** > > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote:**** > > Do you have a scenario in mind?**** > ------------------------------ > > *From: *Vishwas Manral > *Sent: *9/22/2011 4:55 PM **** > > > *To: *Murari Sridharan > *Cc: *Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject: *Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > Hi Murari,**** > > **** > > Yes that is what I mean.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote:**** > > You mean not an Ethernet frame but some IP payload?**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:49 PM > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org **** > > > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Murari,**** > > **** > > What I am saying is the inner header should be allowed to be L3. **** > > **** > > From the diagram you have that does not seem to be the case. Am I missing > it totally?**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote:**** > > Vishwas, Thanks for the feedback we will definitely consider adding that.= I > am not sure what you mean by doing L3 instead of L2. We allow any arbitra= ry > virtual topology including L3. **** > > **** > > Thanks**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:19 PM **** > > > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device = for > hashing and using lesser number of bytes).**** > > **** > > I would think you should add it to your draft immediately. With > tunneling you consolidate the addresses visible to the core and by provid= ing > a hash mechanism, you are providing some level of randomness.**** > > **** > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead = of > L2 alone. I guess it would be the same comment for the VXLAN proposal too= . > **** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to g= et > entropy. One step further would be for the switches to get full entropy f= rom > the inner Ethernet frame. I take it that your comment would be to make it > explicit in the draft. Right?**** > > **** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > **** > > Simha**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.**= * > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes i= n > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the V= Ms > subnet is split between the cloud and the enterprise? Otherwise I don't s= ee > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible t= o > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all extern= al > peers go through enterprise's gateway to reach this host which is no long= er > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > > **** > > **** > > **** > > **** > > ** ** > --0016e644c7989dd1ad04ad934155 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi Simha,
=A0
I am talking about the difference between Layer-2 mobility and Layer-3= mobility.
=A0
Thanks,
Vishwas
On Thu, Sep 22, 2011 at 8:12 PM, Narasimhan Venk= ataramaiah <na= rave@microsoft.com> wrote:

Do y= ou mean in the context of network virtualization or in general its useful? = Mobility is already one aspect of network virtualization and its satisfied = by the MAC in GRE option.

<= /u>=A0

Simh= a

<= /u>=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 8:08 PM=20


To: Murari Sridharan
Cc: Narasimhan = Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Murari,

=A0

You could interpret it that way, but I see the scope= increase is considerably more than just removing the MAC header, though th= at would be just one such advantage.

=A0

Thanks,

Vishwas<= /p>

On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

In t= he context of the proposal here where you can carry your IP around wherever= you go doesn=92t that already equivalent to IP mobility? If I understand y= ou right you are simply saying don=92t just restrict the draft to MAC-in-GR= E but do IP-in-GRE right?

=A0=

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 7:38 PM


To: Murar= i Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; = armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Murari think IP mobility. :)

=A0

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Do you have a scenar= io in mind?

From: = Vishwas Manral
Sent: 9/22/2011= 4:55 PM


To: Murari Srid= haran
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-= 00" advertise its external facing hosts' IP addresses to external = world?

Hi Murari,

=A0

Yes that is what I mean.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

You = mean not an Ethernet frame but some IP payload?

=A0<= /span>

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.blac= k@emc.com; armd@ietf= .org


Subject: Re: [armd] how does "draft-= sridharan-virtualization-nvgre-00" advertise its external facing hosts= ' IP addresses to external world?

=A0

Murari,

=A0

What I am saying is the inner header should be allow= ed to be L3.

=A0

From the diagram you have that does not seem to be t= he case. Am I missing it totally?

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Vish= was, Thanks for the feedback we will definitely consider adding that. I am = not sure what you mean by doing L3 instead of L2. We allow any arbitrary vi= rtual topology including L3.

=A0<= /span>

Than= ks

=A0

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:19 PM


To: Naras= imhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; = armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

I see this as the only difference between VXLAN and = the NVGRE solution (besides ofcourse that TNI needs to be parsed in the int= ermediate device for hashing and using lesser number of bytes).

=A0

I would think you should add it to your draft immedi= ately. With tunneling=A0you=A0consolidate=A0the addresses visible to the co= re and by providing a hash mechanism, you are providing some level of rando= mness.

=A0

The other thing you should look at is L3 (IPv4/ IPv6= )=A0over NVGRE instead of L2 alone.=A0I guess it would be the same comment = for the VXLAN proposal too.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkatar= amaiah <narave= @microsoft.com> wrote:

The draft mentions exactly this as one use of the re= served 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to furth= er distribute flows belonging to a particular TNI and the switches use all = 32 bits to get entropy. One step further would be for the switches to get f= ull entropy from the inner Ethernet frame. I take it that your comment woul= d be to make it explicit in the draft. Right?

=A0

One

=A0=A0 such example could be to use the upper 8 bits of the= Key field to

=A0=A0 add flow based entropy and tag all the packets from = a flow with an entropy label.

=A0

Simh= a

=A0<= /span>

From:<= span style=3D"FONT-SIZE: 10pt"> Vishwas Manral [mailto:vishwas.ietf@gmail.com]
= Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sri= dharan; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0

Hi Simha,

=A0

The main (Standards Track)=A0change in your draft is= the addition of TNI.

=A0

A question I have is a TNI identifies a particular t= enant and all flows from/to a tenant will be hashed to the same path (even = with the changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have ke= pt as reserved for providing the randomization for hashing flows between sa= me to/from on different paths?

=A0

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkata= ramaiah <narav= e@microsoft.com> wrote:

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much for sharin= g the presentation.

One question:

For a host within an Enterp= rise site which needs to communicate with external peers, the host either u= ses public IP address which is visible to external peers or uses private IP= address which is translated to public address at the Enterprise site's= gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

=A0

=A0

=A0

=A0

=A0


--0016e644c7989dd1ad04ad934155-- From narave@microsoft.com Thu Sep 22 20:32:15 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 96A3221F869E for ; Thu, 22 Sep 2011 20:32:15 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.598 X-Spam-Level: X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eXbWC5zd5+X4 for ; Thu, 22 Sep 2011 20:32:09 -0700 (PDT) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 9F56621F84AC for ; Thu, 22 Sep 2011 20:32:09 -0700 (PDT) Received: from TK5EX14MLTC104.redmond.corp.microsoft.com (157.54.79.159) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 22 Sep 2011 20:34:42 -0700 Received: from TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com (157.54.71.68) by TK5EX14MLTC104.redmond.corp.microsoft.com (157.54.79.159) with Microsoft SMTP Server (TLS) id 14.1.339.2; Thu, 22 Sep 2011 20:34:42 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.68]) with mapi id 14.01.0339.002; Thu, 22 Sep 2011 20:34:41 -0700 From: Narasimhan Venkataramaiah To: Vishwas Manral Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHGgAcU1gD//4sFsIAAeWoAgAAGy4CAAAFnAIAAAHwAgAAA1ACAAAHwAIAAK/SAgAAA+ICAAAdyAP//i4NggAB3UYD//4yTMA== Date: Fri, 23 Sep 2011 03:34:41 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD373E3@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD37351@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.90] Content-Type: multipart/alternative; boundary="_000_65755BEBE02F7C41BD4F137AED91DA5E2FD373E3TK5EX14MBXW601w_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 03:32:15 -0000 --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD373E3TK5EX14MBXW601w_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Say you have a virtual subnet that spans 2 physical subnets. A VM in a virt= ual subnet moving to another host within the same physical subnet is L2 mob= ility and a VM moving to another host on another in a different physical s= ubnet is L3 mobility. These 2 cases are the same for NV-GRE in terms of enc= apsulation as the resulting packets are the same. What does IP in GRE achie= ve that MAC in GRE does not? Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 8:18 PM To: Narasimhan Venkataramaiah Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I am talking about the difference between Layer-2 mobility and Layer-3 mobi= lity. Thanks, Vishwas On Thu, Sep 22, 2011 at 8:12 PM, Narasimhan Venkataramaiah > wrote: Do you mean in the context of network virtualization or in general its usef= ul? Mobility is already one aspect of network virtualization and its satisf= ied by the MAC in GRE option. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 8:08 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, You could interpret it that way, but I see the scope increase is considerab= ly more than just removing the MAC header, though that would be just one su= ch advantage. Thanks, Vishwas On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan > wrote: In the context of the proposal here where you can carry your IP around wher= ever you go doesn't that already equivalent to IP mobility? If I understand= you right you are simply saying don't just restrict the draft to MAC-in-GR= E but do IP-in-GRE right? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 7:38 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD373E3TK5EX14MBXW601w_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Say you have a virtual su= bnet that spans 2 physical subnets. A VM in a virtual subnet moving to anot= her host within the same physical subnet is L2 mobility and a VM moving to another host on another in  a different physical s= ubnet is L3 mobility. These 2 cases are the same for NV-GRE in terms of enc= apsulation as the resulting packets are the same. What does IP in GRE achie= ve that MAC in GRE does not?

 <= /p>

Simha

 <= /p>

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 8:18 PM
To: Narasimhan Venkataramaiah
Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; armd@ietf.o= rg
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I am talking about the difference between Layer-2 mo= bility and Layer-3 mobility.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 8:12 PM, Narasimhan Venkatar= amaiah <narave@microsoft.com= > wrote:

Do you mean in the = context of network virtualization or in general its useful? Mobility is alr= eady one aspect of network virtualization and its satisfied by the MAC in GRE option.

 <= /o:p>

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 8:08 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

You could interpret it that way, but I see the scope increase is c= onsiderably more than just removing the MAC header, though that would be ju= st one such advantage.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

In the context of t= he proposal here where you can carry your IP around wherever you go doesn&#= 8217;t that already equivalent to IP mobility? If I understand you right you are simply saying don’t just restrict = the draft to MAC-in-GRE but do IP-in-GRE right?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 7:38 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari think IP mobility. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Do you have a scenario in mind?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.co= m; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgr= e-00" advertise its external facing hosts' IP addresses to external wo= rld?

Hi Murari,

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

You mean not an Eth= ernet frame but some IP payload?

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allowed to be L3.

 

From the diagram you have that does not seem to be the case. Am I = missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Vishwas, Thanks for= the feedback we will definitely consider adding that. I am not sure what y= ou mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /o:p>

Thanks<= /o:p>

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solu= tion (besides ofcourse that TNI needs to be parsed in the intermediate devi= ce for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tu= nneling you consolidate the addresses visible to the core an= d by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NV= GRE instead of L2 alone. I guess it would be the same comment for the = VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

 

 

--_000_65755BEBE02F7C41BD4F137AED91DA5E2FD373E3TK5EX14MBXW601w_-- From gaberger@cisco.com Fri Sep 23 06:31:16 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C7E5621F8C5E for ; Fri, 23 Sep 2011 06:31:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.127 X-Spam-Level: X-Spam-Status: No, score=-1.127 tagged_above=-999 required=5 tests=[AWL=0.075, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eh4jwnK3ybC7 for ; Fri, 23 Sep 2011 06:31:11 -0700 (PDT) Received: from rcdn-iport-1.cisco.com (rcdn-iport-1.cisco.com [173.37.86.72]) by ietfa.amsl.com (Postfix) with ESMTP id 45A6321F8C5A for ; Fri, 23 Sep 2011 06:31:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=48056; q=dns/txt; s=iport; t=1316784826; x=1317994426; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=hEqPhqLzeHQjWuDrp6HfrxBGEJEdOgS/ZHDdC/GgITY=; b=csZcArts3xtefKbM1RPEMFkhFleTI5OrACZNAKY5InwJJyp0zmZYB03K 1v3PzmQ5+rPZEEjzo2R93Upo4rbvPaqnadUgFBqTekLmG1tZufmnY7ZNi 0MAPC7YSZOt4AKTMNg9hsolY3Wsol+PD/niK8+IgFWQLg5BHKS0r/KX6q 4=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Am0BAHiJfE6tJV2b/2dsb2JhbABCgk2WTIdbAYR8djxueIFTAQEBAwEBAQEPAQcBIjEEBwUHBwgRAwEBAQEgAQIEKAYfCQgGAQ0FCRIHh1YGl3QBnieHAQSHQowQhSCEdYc4 X-IronPort-AV: E=Sophos;i="4.68,430,1312156800"; d="scan'208,217";a="23563745" Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by rcdn-iport-1.cisco.com with ESMTP; 23 Sep 2011 13:33:44 +0000 Received: from [10.82.247.253] (rtp-vpn2-2042.cisco.com [10.82.247.253]) by rcdn-core-4.cisco.com (8.14.3/8.14.3) with ESMTP id p8NDXf5i032207; Fri, 23 Sep 2011 13:33:43 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Fri, 23 Sep 2011 09:33:40 -0400 From: Gary Berger To: Vishwas Manral , Narasimhan Venkataramaiah Message-ID: Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3399615223_12189497" Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 13:31:16 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3399615223_12189497 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable I think these are all interesting approaches.. Whether you demultiplex a customer on an 802.1ah I-SID, a VXLAN VID, NVGRE TNI or some other set of bits, we still seem to be carrying excessive state across the network whic= h will continue to make things more fragile. If we continue to drive the problem into the host-stack (because that=B9s where the customer has some choice) we will lose the benefit of building intelligent services in the network.=20 The problem space is still the same I.e. Because of the TCP pseudo-header binding the IP address to the point of attachment we have to encapsulate in order to hide this limiting factor). The arguments about "whatever over GRE= " are beyond the problem space IMHO but certainly a clear indication of the flawed model.=20 Some have suggested the answer is a distributed directory service (for broadcast containment) which allows the implementor to define their own bindings. This along with the programability of the network (e.g. OF) in order to instantiate those bindings into the forwarding plane. Certainly that solution can only be based on the scope of control but none-the-less, = I thought that this example was where this working group derived its charter. -g From: Vishwas Manral Date: Thu, 22 Sep 2011 20:17:59 -0700 To: Narasimhan Venkataramaiah Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? Hi Simha, =20 I am talking about the difference between Layer-2 mobility and Layer-3 mobility. =20 Thanks, Vishwas On Thu, Sep 22, 2011 at 8:12 PM, Narasimhan Venkataramaiah wrote: > Do you mean in the context of network virtualization or in general its us= eful? > Mobility is already one aspect of network virtualization and its satisfie= d by > the MAC in GRE option. > =20 > Simha > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 8:08 PM >=20 >=20 > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > =20 >=20 > Murari, >=20 > =20 >=20 > You could interpret it that way, but I see the scope increase is consider= ably > more than just removing the MAC header, though that would be just one suc= h > advantage. >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan > wrote: >=20 > In the context of the proposal here where you can carry your IP around > wherever you go doesn=B9t that already equivalent to IP mobility? If I > understand you right you are simply saying don=B9t just restrict the draft = to > MAC-in-GRE but do IP-in-GRE right? > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 7:38 PM >=20 >=20 > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Murari think IP mobility. :) >=20 > =20 >=20 > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: >=20 > Do you have a scenario in mind? >=20 >=20 > From: Vishwas Manral > Sent: 9/22/2011 4:55 PM >=20 >=20 > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > Hi Murari, >=20 > =20 >=20 > Yes that is what I mean. >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: >=20 > You mean not an Ethernet frame but some IP payload? > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:49 PM > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org=20 >=20 >=20 > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Murari, >=20 > =20 >=20 > What I am saying is the inner header should be allowed to be L3. >=20 > =20 >=20 > From the diagram you have that does not seem to be the case. Am I missing= it > totally? >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: >=20 > Vishwas, Thanks for the feedback we will definitely consider adding that.= I am > not sure what you mean by doing L3 instead of L2. We allow any arbitrary > virtual topology including L3. > =20 > Thanks > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:19 PM >=20 >=20 > To: Narasimhan Venkataramaiah > Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Hi Simha, >=20 > =20 >=20 > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device = for > hashing and using lesser number of bytes). >=20 > =20 >=20 > I would think you should add it to your draft immediately. With tunneling= you > consolidate the addresses visible to the core and by providing a hash > mechanism, you are providing some level of randomness. >=20 > =20 >=20 > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead = of L2 > alone. I guess it would be the same comment for the VXLAN proposal too. >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: >=20 > The draft mentions exactly this as one use of the reserved 8 bits in Sect= ion > 4. An NVGRE endpoint could use the 8 bits to further distribute flows > belonging to a particular TNI and the switches use all 32 bits to get ent= ropy. > One step further would be for the switches to get full entropy from the i= nner > Ethernet frame. I take it that your comment would be to make it explicit = in > the draft. Right? > =20 > One > such example could be to use the upper 8 bits of the Key field to > add flow based entropy and tag all the packets from a flow with an ent= ropy > label. > =20 > Simha > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:04 PM > To: Narasimhan Venkataramaiah > Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Hi Simha, >=20 > =20 >=20 > The main (Standards Track) change in your draft is the addition of TNI. >=20 > =20 >=20 > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes i= n > switches to do hashing to use TNI). >=20 > =20 >=20 > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths? >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from= the > enterprise point of view. Are you referring to a scenario where the VMs s= ubnet > is split between the cloud and the enterprise? Otherwise I don't see the > implication on virtualization as its no different than getting the traffi= c > routed to the enterprise in the first case. >=20 > Simha >=20 > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Du= nbar > [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advert= ise > its external facing hosts' IP addresses to external world? >=20 >=20 > Hi Murari, >=20 > Thank you very much for sharing the presentation. >=20 > One question: >=20 > For a host within an Enterprise site which needs to communicate with exte= rnal > peers, the host either uses public IP address which is visible to externa= l > peers or uses private IP address which is translated to public address at= the > Enterprise site's gateway. >=20 > When this host is moved to "Cloud data center", will the "Cloud Data cent= er" > advertise this host address to external peers? Or will all external peers= go > through enterprise's gateway to reach this host which is no longer residi= ng in > the enterprise site? >=20 > Thanks, Linda >=20 >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f >> > Murari Sridharan >> > Sent: Saturday, September 17, 2011 3:02 PM >> > To: david.black@emc.com; armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > FYI, here is a talk that I gave last week in relation to the nvgre >> > draft below. >> > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T >> > >> > Thanks >> > Murari >> > >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f >> > david.black@emc.com >> > Sent: Friday, September 16, 2011 6:14 AM >> > To: armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > And two more drafts on this topic: >> > >> > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt >> > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt >> > >> > The edge switches could be the software switches in hypervisors. >> > >> > Thanks, >> > --David >> > >> > >>> > > -----Original Message----- >>> > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behal= f >>> > > Of Warren Kumari >>> > > Sent: Wednesday, August 31, 2011 3:16 PM >>> > > To: Vishwas Manral >>> > > Cc: armd@ietf.org >>> > > Subject: Re: [armd] soliciting typical network designs for ARMD >>> > > >>> > > >>> > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >>> > > >>>> > > > Hi Linda/ Anoop, >>>> > > > >>>> > > > Here is the example of the design I was talking about, as define= d >> > by google. >>> > > >>> > > Just a clarification -- s/as defined by google/as described by >> > someone >>> > > who happens to work for google/ >>> > > >>> > > W >>> > > >>>> > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani >> > wrote: >>>> > > > >>>>>>>> > > > >>>> >>>> > > > (though I think if there was a standard way to map Multicast MAC= to >>>> > > > Multicast IP, they could >>> > > probably use such a standard mechanisms). >>>>>>>> > > > >>>> >>>> > > > >>>> > > > They can do that, but then this imposes requirements on the >>>> > > > equipment to be able to do multicast forwarding, and even if doe= s, >>>> > > > because of pruning requirements the number of groups would be ve= ry >>>> > > > large. The average data center switch probably won't handle tha= t >>>> > > > many groups. >>>> > > > >>>> > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Anoop, >>>> > > > >>>> > > > From what I know they do not use Multicast GRE (I hear the extra= 4 >>>> > > > bytes in the GRE header is a >>> > > proprietery extension). >>>> > > > >>>> > > > I think a directory based mechanism is what is used (though I th= ink >>>> > > > if there was a standard way to >>> > > map Multicast MAC to Multicast IP, they could probably use such a >> > standard mechanisms). >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani >> > wrote: >>>> > > > Hi Vishwas, >>>> > > > >>>> > > > How do they get multicast through the network in that case? >>>> > > > Are they planning to use multicast GRE, or just use directory ba= sed >>>> > > > lookups and not worry about multicast applications for now? >>>> > > > >>>> > > > Anoop >>>> > > > >>>> > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Linda, >>>> > > > >>>> > > > The data packets can be tunnelled at the ToR over say a GRE pack= et >>>> > > > and the core is a Layer-3 core >>> > > (except for the downstream ports). So we could have encapsulation/ >>> > > decapsulation of L2 over GRE at the ToR. >>>> > > > >>>> > > > The very same thing can be done at the hypervisor layer too, in >>>> > > > which case the entire DC network >>> > > would look like a Layer-3 flat network including the ToR to server >>> > > link and the hypervisor would do the tunneling. >>>> > > > >>>> > > > I am not sure if you got the points above or not. I know cloud O= S >>>> > > > companies that provide the service >>> > > and have big announced customers. >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar >> > wrote: >>>> > > > Vishwas, >>>> > > > >>>> > > > In my mind the bullet 1) in the list refers to ToR switches >>>> > > > downstream ports (facing servers) >>> > > running Layer 2 and ToR uplinks ports run IP Layer 3. >>>> > > > >>>> > > > Have you seen data center networks with ToR switches downstream >>>> > > > ports (i.e. facing servers) enabling >>> > > IP routing, even though the physical links are Ethernet? >>>> > > > If yes, we should definitely include it in the ARMD draft. >>>> > > > >>>> > > > Thanks, >>>> > > > Linda >>>> > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Linda, >>>> > > > I am unsure what you mean by this, but: >>>> > > > * layer 3 all the way to TOR (Top of Rack switches), We can al= so >>>> > > > have a heirarchical network, with the core totally Layer-3 (and >>>> > > > having seperate >>> > > routing), from the hosts still in a large Layer-3 subnet. Another >>> > > aspect could be to have a totally >>> > > Layer-3 network. >>>> > > > >>>> > > > The difference between them is the link between the servers and = the >> > ToR. >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar >> > wrote: >>>> > > > During the 81st IETF ARMD WG discussion, it was suggested that i= t >> > is >>>> > > > necessary to document typical >>> > > data center network designs so that address resolution scaling issu= es >>> > > can be properly described. Many data center operators have expresse= d >> > that they can't openly reveal their detailed network designs. >>> > > Therefore, we only want to document anonymous designs without too >> > much >>> > > detail. During the journey of establishing ARMD, we have come acros= s >> > the following typical data center network designs: >>>> > > > * layer 3 all the way to TOR (Top of Rack switches), >>>> > > > * large layer 2 with hundreds (or thousands) of ToRs being >>>> > > > interconnected by Layer 2. This >>> > > design will have thousands of hosts under the L2/L3 boundary router >>> > > (s) >>>> > > > * CLOS design with thousands of switches. This design will ha= ve >>>> > > > thousands of hosts under the >>> > > L2/L3 boundary router(s) >>>> > > > We have heard that each of the designs above has its own problem= s. >>>> > > > ARMD problem statements might >>> > > need to document DC problems under each typical design. >>>> > > > Please send feedback to us (either to the armd email list or to >> > the >>>> > > > ARMD chair Benson & Linda) to >>> > > indicate if we have missed any typical Data Center network designs. >>>> > > > >>>> > > > Your contribution can greatly accelerate the progress of ARMD WG= . >>>> > > > >>>> > > > Thank you very much. >>>> > > > >>>> > > > Linda & Benson >>>> > > > > =20 > =20 > =20 > =20 > =20 _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --B_3399615223_12189497 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable
I think these are = all interesting approaches.. Whether you demultiplex a customer on an 802.1a= h I-SID, a VXLAN VID,  NVGRE TNI or some other set of bits,  we st= ill seem to be carrying excessive state across the network which will contin= ue to make things more fragile. If we continue to drive the problem into the= host-stack (because that’s where the customer has some choice) we wil= l lose the benefit of building intelligent services in the network. 

The problem space is still the same I.e. Because of t= he TCP pseudo-header binding the IP address to the point of attachment we ha= ve to encapsulate in order to hide this limiting factor). The arguments abou= t "whatever over GRE" are beyond the problem space IMHO but certainly a clea= r indication of the flawed model. 

Some have suggested the answer is a distributed directory service (for broa= dcast containment) which allows the implementor to define their own bindings= . This along with the programability of the network (e.g. OF) in order to in= stantiate those bindings into the forwarding plane. Certainly that solution = can only be based on the scope of control but none-the-less, I thought that = this example was where this working group derived its charter.
-g


From: Vishwas Manral <v= ishwas.ietf@gmail.com>
Date: Thu, 22 Sep 2011 20:17:59 -0700
To: Narasimhan Venkataramaiah <nara= ve@microsoft.com>
Cc: "armd@ietf.org" <armd@ietf.org>
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its= external facing hosts' IP addresses to external world?

Hi Simha,
 
I am talking about the differe= nce between Layer-2 mobility and Layer-3 mobility.
 
Thanks,
Vishwas
On Thu, Sep 22= , 2011 at 8:12 PM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:

= Do you mean in the context of network virtualization or in general its usefu= l? Mobility is already one aspect of network virtualization and its satisfie= d by the MAC in GRE option.

 

<= p class=3D"MsoNormal">Simha

 

From: Vishwas M= anral [mailto:vishwa= s.ietf@gmail.com]
Sent: Thursday, September 22, 2011 8:08 PM =


To: Murari Sridharan<= br>Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] = how does "draft-sridharan-virtualization-nvgre-00" advertise its external fa= cing hosts' IP addresses to external world?

=

 

Murari,

 

You could in= terpret it that way, but I see the scope increase is considerably more than = just removing the MAC header, though that would be just one such advantage.<= u>

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 7:41 PM, Murari Sridharan <muraris@microsoft.com> wrote:

In the context of the proposal here where you= can carry your IP around wherever you go doesn’t that already equival= ent to IP mobility? If I understand you right you are simply saying don̵= 7;t just restrict the draft to MAC-in-GRE but do IP-in-GRE right?<= /u>

 

Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 7:38 PM


T= o: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunba= r; david.black@emc.com<= /a>; armd@ietf.org
= Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?

 <= /u>

Murari think IP mobility. :)<= /p>

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Do you have a scenario in mind?

From: Vishwas= Manral
Sent: 9/22/2011 4:55 PM

<= p class=3D"MsoNormal" style=3D"MARGIN-BOTTOM: 12pt">
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does = "draft-sridharan-virtualization-nvgre-00" advertise its external facing host= s' IP addresses to external world?

Hi Murari,

&n= bsp;

Yes that is what I mea= n.

 

=

Thanks,

Vishwas

On Thu= , Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com> wrote:

You mean not an Ethernet frame but some IP payload?

&nb= sp;

From: Vishwas Manral [mailto:= vishwas.ietf@gmail.c= om]
Sent: Thursday, September 22, 2011 4:49 PM
To: = Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] ho= w does "draft-sridharan-virtualization-nvgre-00" advertise its external faci= ng hosts' IP addresses to external world?

=

 

= Murari,

 

What I am saying is the inner header sh= ould be allowed to be L3.

=  

From the diagram you= have that does not seem to be the case. Am I missing it totally?<= /u>

 

Thanks,

= Vishwas

On Thu, Sep 22, 201= 1 at 4:43 PM, Murari Sridharan <muraris@microsoft.com> wrote:

Vishwas= , Thanks for the feedback we will definitely consider adding that. I am not = sure what you mean by doing L3 instead of L2. We allow any arbitrary virtual= topology including L3.

 

Thanks<= u>

=  

From: Vishwas Manral [= mailto:vishwas.ietf@= gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM =


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Mura= ri Sridharan; david.bla= ck@emc.com; armd@ietf.org=
Subject: Re: [armd] how does "draft-sridharan-virtualization-= nvgre-00" advertise its external facing hosts' IP addresses to external worl= d?

 =

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solutio= n (besides ofcourse that TNI needs to be parsed in the intermediate device f= or hashing and using lesser number of bytes).

 

I= would think you should add it to your draft immediately. With tunneling&nbs= p;you consolidate the addresses visible to the core and by providi= ng a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) = ;over NVGRE instead of L2 alone. I guess it would be the same comment f= or the VXLAN proposal too.

=  

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah= <narave@microsoft.= com> wrote:

The draft= mentions exactly this as one use of the reserved 8 bits in Section 4. An NV= GRE endpoint could use the 8 bits to further distribute flows belonging to a= particular TNI and the switches use all 32 bits to get entropy. One step fu= rther would be for the switches to get full entropy from the inner Ethernet = frame. I take it that your comment would be to make it explicit in the draft= . Right?

 

One

   such example could be to use the upper = 8 bits of the Key field to

   add flo= w based entropy and tag all the packets from a flow with an entropy label.

 

Simha=

 = ;

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com= ]
Sent: Thursday, September 22, 2011 4:04 PM
To: Na= rasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject:<= /b> Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise = its external facing hosts' IP addresses to external world?<= /u>

 

Hi Simha,

 = ;

The main (Standards Track= ) change in your draft is the addition of TNI.

<= div>

 

A question I have is a TNI identifies a particular tenant and all flows= from/to a tenant will be hashed to the same path (even with the changes in = switches to do hashing to use TNI).

 

Why do you = not use the last 8 bits which you have kept as reserved for providing the ra= ndomization for hashing flows between same to/from on different paths?

 

Thanks,

Vishwas

On Sun, Sep 18= , 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:

The easiest from the point of view of configuratio= n would be to route everything back through the enterprise - not necessarily= the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Othe= rwise I don't see the implication on virtualization as its no different than= getting the traffic routed to the enterprise in the first case.

Simh= a

________________________________________
From: armd-bounces@ietf.org [armd-bounces@ietf.org] on be= half of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd]= how does "draft-sridharan-virtualization-nvgre-00" advertise its external f= acing hosts' IP addresses to external world?


Hi Murari,

Thank you very much for sharing the p= resentation.

One question:

For a host within an Enterprise sit= e which needs to communicate with external peers, the host either uses publi= c IP address which is visible to external peers or uses private IP address w= hich is translated to public address at the Enterprise site's gateway.
When this host is moved to "Cloud data center", will the "Cloud Data cente= r" advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resid= ing in the enterprise site?

Thanks, Linda

> -----Original M= essage-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.black@e= mc.com; armd@ietf.org=
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgre=
> draft below.
> http://channel9.msdn.com/Events/BUILD/B= UILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd= -bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@emc.c= om
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re= : [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization= -nvgre-00.txt
>
> The edge switches could be the software sw= itches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: [armd]= soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &g= t;
> > > Here is the example of the design I was talking about, = as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to wo= rk for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt<= /a>
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu>= ; wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &g= t; Multicast IP, they could
> > probably use such a standard mechan= isms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equip= ment to be able to do multicast forwarding, and even if does,
> > &= gt; because of pruning requirements the number of groups would be very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > >= ; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrot= e:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > bytes= in the GRE header is a
> > proprietery extension).
> > &g= t;
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Mu= lticast MAC to Multicast IP, they could probably use such a
> standard= mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu>= ; wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they p= lanning to use multicast GRE, or just use directory based
> > > = lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:=
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and t= he core is a Layer-3 core
> > (except for the downstream ports). So= we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in<= br>> > > which case the entire DC network
> > would look l= ike a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know = cloud OS
> > > companies that provide the service
> > a= nd have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wro= te:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream po= rts (facing servers)
> > running Layer 2 and ToR uplinks ports run = IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enabling=
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> &= gt; > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > >   * layer 3 all the way to TOR (Top of Rack s= witches), We can also
> > > have a heirarchical network, with th= e core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a to= tally
> > Layer-3 network.
> > >
> > > The = difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com>= ;
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typi= cal
> > data center network designs so that address resolution scal= ing issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:<= br>> > >   * layer 3 all the way to TOR (Top of Rack switches)= ,
> > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design = will have thousands of hosts under the L2/L3 boundary router
> > (s= )
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> >= L2/L3 boundary router(s)
> > > We have heard that each of the d= esigns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedbac= k to us (either to the armd email list  or to
> the
> > = > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly acceler= ate the progress of ARMD WG.
> > >
> > > Thank you v= ery much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

 

<= br> _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/m= ailman/listinfo/armd --B_3399615223_12189497-- From warren@kumari.net Fri Sep 23 12:44:18 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9484921F8C1F for ; Fri, 23 Sep 2011 12:44:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -104.47 X-Spam-Level: X-Spam-Status: No, score=-104.47 tagged_above=-999 required=5 tests=[AWL=2.129, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vVPV9e9RyOZU for ; Fri, 23 Sep 2011 12:44:17 -0700 (PDT) Received: from vimes.kumari.net (vimes.kumari.net [198.186.192.250]) by ietfa.amsl.com (Postfix) with ESMTP id 6B0DF21F8C1D for ; Fri, 23 Sep 2011 12:44:17 -0700 (PDT) Received: from dhcp-172-19-119-117.cbf.corp.google.com (unknown [64.13.52.115]) by vimes.kumari.net (Postfix) with ESMTPSA id F0CFA1B40091; Fri, 23 Sep 2011 15:46:50 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warren Kumari In-Reply-To: <2073A6C5467C99478898544C6EBA3F4602BBFE512F@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Date: Fri, 23 Sep 2011 15:46:48 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <18111E6E-5578-4987-9B22-F9D27529182E@kumari.net> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <2073A6C5467C99478898544C6EBA3F4602BBFE512F@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> To: "Balus, Florin Stelian (Florin)" X-Mailer: Apple Mail (2.1084) Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 19:44:18 -0000 On Sep 19, 2011, at 7:36 PM, Balus, Florin Stelian (Florin) wrote: >=20 > These two drafts and draft-wkumari-dcops-l3-vmmobility-00.txt seem to = be submitted as either informational or experimental. Can the authors of = these two drafts clarify what do they plan to do with them? Honestly? My grand plan was to largely ignore the draft and then feel = guilty that I had not incorporated the comments that I have received. = Eventually the guilt would have caught up with me and I would have been = forced to make a decision as to if I was going to bother writing more, = or simply deleting it and pretending nothing had ever happened... I believe that my co-author ((Joel), who is more level-headed than me) = may be been planning on doing something more useful -- I've CCed him. The draft wasn't really intended to push a new encapsulation technology = (the desire was simply to demonstrate that big flat L2 networks are not = needed for VM mobility) -- the introduction of an encapsulation was only = done because I couldn't easily find an existing solution. If one is = created (that satisfied the needs of the protocol), that would be much = better... W=20 > If relevant feedback is required I think they seem to fit the scope of = L2VPN WG. At least two of them talk about some new encapsulation used to = provide L2 Multi-tenancy in the Data Centers over some sort of overlay = and UDP/IP tunnels. A new L2VPN charter was released recently which = includes requirements and solutions targeted at the Data Center space. >=20 > Florin >=20 > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf = Of Murari Sridharan > Sent: Saturday, September 17, 2011 1:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD >=20 > FYI, here is a talk that I gave last week in relation to the nvgre = draft below.=20 > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T >=20 > Thanks > Murari=20 >=20 > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf = Of david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD >=20 > And two more drafts on this topic: >=20 > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt >=20 > The edge switches could be the software switches in hypervisors.=20 >=20 > Thanks, > --David >=20 >=20 >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=20= >> Of Warren Kumari >> Sent: Wednesday, August 31, 2011 3:16 PM >> To: Vishwas Manral >> Cc: armd@ietf.org >> Subject: Re: [armd] soliciting typical network designs for ARMD >>=20 >>=20 >> On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >>=20 >>> Hi Linda/ Anoop, >>>=20 >>> Here is the example of the design I was talking about, as defined by = google. >>=20 >> Just a clarification -- s/as defined by google/as described by = someone=20 >> who happens to work for google/ >>=20 >> W >>=20 >>> http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt >>>=20 >>> Thanks, >>> Vishwas >>> On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani = wrote: >>>=20 >>>>>>>=20 >>> (though I think if there was a standard way to map Multicast MAC to=20= >>> Multicast IP, they could >> probably use such a standard mechanisms). >>>>>>>=20 >>>=20 >>> They can do that, but then this imposes requirements on the=20 >>> equipment to be able to do multicast forwarding, and even if does,=20= >>> because of pruning requirements the number of groups would be very=20= >>> large. The average data center switch probably won't handle that=20 >>> many groups. >>>=20 >>> On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral = wrote: >>> Hi Anoop, >>>=20 >>> =46rom what I know they do not use Multicast GRE (I hear the extra 4=20= >>> bytes in the GRE header is a >> proprietery extension). >>>=20 >>> I think a directory based mechanism is what is used (though I think=20= >>> if there was a standard way to >> map Multicast MAC to Multicast IP, they could probably use such a = standard mechanisms). >>>=20 >>> Thanks, >>> Vishwas >>> On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani = wrote: >>> Hi Vishwas, >>>=20 >>> How do they get multicast through the network in that case? >>> Are they planning to use multicast GRE, or just use directory based=20= >>> lookups and not worry about multicast applications for now? >>>=20 >>> Anoop >>>=20 >>> On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral = wrote: >>> Hi Linda, >>>=20 >>> The data packets can be tunnelled at the ToR over say a GRE packet=20= >>> and the core is a Layer-3 core >> (except for the downstream ports). So we could have encapsulation/=20 >> decapsulation of L2 over GRE at the ToR. >>>=20 >>> The very same thing can be done at the hypervisor layer too, in=20 >>> which case the entire DC network >> would look like a Layer-3 flat network including the ToR to server=20 >> link and the hypervisor would do the tunneling. >>>=20 >>> I am not sure if you got the points above or not. I know cloud OS=20 >>> companies that provide the service >> and have big announced customers. >>>=20 >>> Thanks, >>> Vishwas >>> On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar = wrote: >>> Vishwas, >>>=20 >>> In my mind the bullet 1) in the list refers to ToR switches=20 >>> downstream ports (facing servers) >> running Layer 2 and ToR uplinks ports run IP Layer 3. >>>=20 >>> Have you seen data center networks with ToR switches downstream=20 >>> ports (i.e. facing servers) enabling >> IP routing, even though the physical links are Ethernet? >>> If yes, we should definitely include it in the ARMD draft. >>>=20 >>> Thanks, >>> Linda >>> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = wrote: >>> Hi Linda, >>> I am unsure what you mean by this, but: >>> * layer 3 all the way to TOR (Top of Rack switches), We can also=20= >>> have a heirarchical network, with the core totally Layer-3 (and=20 >>> having seperate >> routing), from the hosts still in a large Layer-3 subnet. Another=20 >> aspect could be to have a totally >> Layer-3 network. >>>=20 >>> The difference between them is the link between the servers and the = ToR. >>>=20 >>> Thanks, >>> Vishwas >>> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar = wrote: >>> During the 81st IETF ARMD WG discussion, it was suggested that it is=20= >>> necessary to document typical >> data center network designs so that address resolution scaling issues=20= >> can be properly described. Many data center operators have expressed = that they can't openly reveal their detailed network designs. >> Therefore, we only want to document anonymous designs without too = much=20 >> detail. During the journey of establishing ARMD, we have come across = the following typical data center network designs: >>> * layer 3 all the way to TOR (Top of Rack switches), >>> * large layer 2 with hundreds (or thousands) of ToRs being=20 >>> interconnected by Layer 2. This >> design will have thousands of hosts under the L2/L3 boundary router=20= >> (s) >>> * CLOS design with thousands of switches. This design will have=20= >>> thousands of hosts under the >> L2/L3 boundary router(s) >>> We have heard that each of the designs above has its own problems.=20= >>> ARMD problem statements might >> need to document DC problems under each typical design. >>> Please send feedback to us (either to the armd email list or to the=20= >>> ARMD chair Benson & Linda) to >> indicate if we have missed any typical Data Center network designs. >>>=20 >>> Your contribution can greatly accelerate the progress of ARMD WG. >>>=20 >>> Thank you very much. >>>=20 >>> Linda & Benson >>>=20 >>>=20 >>=20 >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd >=20 From jmh@joelhalpern.com Fri Sep 23 12:51:55 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 638EF21F8B3E for ; Fri, 23 Sep 2011 12:51:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.556 X-Spam-Level: X-Spam-Status: No, score=-102.556 tagged_above=-999 required=5 tests=[AWL=0.043, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 10OhFipM6gfR for ; Fri, 23 Sep 2011 12:51:54 -0700 (PDT) Received: from hermes.out.tigertech.net (hermes-ipv6.tigertech.net [IPv6:2604:4f00::1:0:0:16]) by ietfa.amsl.com (Postfix) with ESMTP id 4EC4F21F8B33 for ; Fri, 23 Sep 2011 12:51:54 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by hermes.tigertech.net (Postfix) with ESMTP id 72AF44300F5; Fri, 23 Sep 2011 12:54:29 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at hermes.tigertech.net Received: from [10.10.10.104] (pool-71-161-52-133.clppva.btas.verizon.net [71.161.52.133]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by hermes.tigertech.net (Postfix) with ESMTPSA id B8997430064; Fri, 23 Sep 2011 12:54:27 -0700 (PDT) Message-ID: <4E7CE3F1.3000507@joelhalpern.com> Date: Fri, 23 Sep 2011 15:54:25 -0400 From: "Joel M. Halpern" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2 MIME-Version: 1.0 To: Warren Kumari References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <2073A6C5467C99478898544C6EBA3F4602BBFE512F@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> <18111E6E-5578-4987-9B22-F9D27529182E@kumari.net> In-Reply-To: <18111E6E-5578-4987-9B22-F9D27529182E@kumari.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2011 19:51:55 -0000 My own perspective was that it seemed useful to put on paper the approach to using a mapping system at the hypervisor. The exactly encapsulation is irrelevant to me. I was hoping it would prompt discussion of that piece of the idea. If it did, we then had to figure out whether there was a way to take it forward. Also, if there was interest, it meant that it was worth seeing if the ARMD data supported or disagreed with the view that this is part of the problem. At the moment, the data does not tell us. But if no one cares, then it is not going to be effective for me to push folks to find out more. Yours, Joel On 9/23/2011 3:46 PM, Warren Kumari wrote: > > On Sep 19, 2011, at 7:36 PM, Balus, Florin Stelian (Florin) wrote: > >> >> These two drafts and draft-wkumari-dcops-l3-vmmobility-00.txt seem to be submitted as either informational or experimental. Can the authors of these two drafts clarify what do they plan to do with them? > > Honestly? My grand plan was to largely ignore the draft and then feel guilty that I had not incorporated the comments that I have received. Eventually the guilt would have caught up with me and I would have been forced to make a decision as to if I was going to bother writing more, or simply deleting it and pretending nothing had ever happened... > > I believe that my co-author ((Joel), who is more level-headed than me) may be been planning on doing something more useful -- I've CCed him. > > The draft wasn't really intended to push a new encapsulation technology (the desire was simply to demonstrate that big flat L2 networks are not needed for VM mobility) -- the introduction of an encapsulation was only done because I couldn't easily find an existing solution. If one is created (that satisfied the needs of the protocol), that would be much better... > > > W >> If relevant feedback is required I think they seem to fit the scope of L2VPN WG. At least two of them talk about some new encapsulation used to provide L2 Multi-tenancy in the Data Centers over some sort of overlay and UDP/IP tunnels. A new L2VPN charter was released recently which includes requirements and solutions targeted at the Data Center space. >> >> Florin >> >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Murari Sridharan >> Sent: Saturday, September 17, 2011 1:02 PM >> To: david.black@emc.com; armd@ietf.org >> Subject: Re: [armd] soliciting typical network designs for ARMD >> >> FYI, here is a talk that I gave last week in relation to the nvgre draft below. >> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T >> >> Thanks >> Murari >> >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of david.black@emc.com >> Sent: Friday, September 16, 2011 6:14 AM >> To: armd@ietf.org >> Subject: Re: [armd] soliciting typical network designs for ARMD >> >> And two more drafts on this topic: >> >> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt >> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt >> >> The edge switches could be the software switches in hypervisors. >> >> Thanks, >> --David >> >> >>> -----Original Message----- >>> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf >>> Of Warren Kumari >>> Sent: Wednesday, August 31, 2011 3:16 PM >>> To: Vishwas Manral >>> Cc: armd@ietf.org >>> Subject: Re: [armd] soliciting typical network designs for ARMD >>> >>> >>> On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >>> >>>> Hi Linda/ Anoop, >>>> >>>> Here is the example of the design I was talking about, as defined by google. >>> >>> Just a clarification -- s/as defined by google/as described by someone >>> who happens to work for google/ >>> >>> W >>> >>>> http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt >>>> >>>> Thanks, >>>> Vishwas >>>> On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani wrote: >>>> >>>>>>>> >>>> (though I think if there was a standard way to map Multicast MAC to >>>> Multicast IP, they could >>> probably use such a standard mechanisms). >>>>>>>> >>>> >>>> They can do that, but then this imposes requirements on the >>>> equipment to be able to do multicast forwarding, and even if does, >>>> because of pruning requirements the number of groups would be very >>>> large. The average data center switch probably won't handle that >>>> many groups. >>>> >>>> On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral wrote: >>>> Hi Anoop, >>>> >>>> From what I know they do not use Multicast GRE (I hear the extra 4 >>>> bytes in the GRE header is a >>> proprietery extension). >>>> >>>> I think a directory based mechanism is what is used (though I think >>>> if there was a standard way to >>> map Multicast MAC to Multicast IP, they could probably use such a standard mechanisms). >>>> >>>> Thanks, >>>> Vishwas >>>> On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani wrote: >>>> Hi Vishwas, >>>> >>>> How do they get multicast through the network in that case? >>>> Are they planning to use multicast GRE, or just use directory based >>>> lookups and not worry about multicast applications for now? >>>> >>>> Anoop >>>> >>>> On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral wrote: >>>> Hi Linda, >>>> >>>> The data packets can be tunnelled at the ToR over say a GRE packet >>>> and the core is a Layer-3 core >>> (except for the downstream ports). So we could have encapsulation/ >>> decapsulation of L2 over GRE at the ToR. >>>> >>>> The very same thing can be done at the hypervisor layer too, in >>>> which case the entire DC network >>> would look like a Layer-3 flat network including the ToR to server >>> link and the hypervisor would do the tunneling. >>>> >>>> I am not sure if you got the points above or not. I know cloud OS >>>> companies that provide the service >>> and have big announced customers. >>>> >>>> Thanks, >>>> Vishwas >>>> On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wrote: >>>> Vishwas, >>>> >>>> In my mind the bullet 1) in the list refers to ToR switches >>>> downstream ports (facing servers) >>> running Layer 2 and ToR uplinks ports run IP Layer 3. >>>> >>>> Have you seen data center networks with ToR switches downstream >>>> ports (i.e. facing servers) enabling >>> IP routing, even though the physical links are Ethernet? >>>> If yes, we should definitely include it in the ARMD draft. >>>> >>>> Thanks, >>>> Linda >>>> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral wrote: >>>> Hi Linda, >>>> I am unsure what you mean by this, but: >>>> * layer 3 all the way to TOR (Top of Rack switches), We can also >>>> have a heirarchical network, with the core totally Layer-3 (and >>>> having seperate >>> routing), from the hosts still in a large Layer-3 subnet. Another >>> aspect could be to have a totally >>> Layer-3 network. >>>> >>>> The difference between them is the link between the servers and the ToR. >>>> >>>> Thanks, >>>> Vishwas >>>> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: >>>> During the 81st IETF ARMD WG discussion, it was suggested that it is >>>> necessary to document typical >>> data center network designs so that address resolution scaling issues >>> can be properly described. Many data center operators have expressed that they can't openly reveal their detailed network designs. >>> Therefore, we only want to document anonymous designs without too much >>> detail. During the journey of establishing ARMD, we have come across the following typical data center network designs: >>>> * layer 3 all the way to TOR (Top of Rack switches), >>>> * large layer 2 with hundreds (or thousands) of ToRs being >>>> interconnected by Layer 2. This >>> design will have thousands of hosts under the L2/L3 boundary router >>> (s) >>>> * CLOS design with thousands of switches. This design will have >>>> thousands of hosts under the >>> L2/L3 boundary router(s) >>>> We have heard that each of the designs above has its own problems. >>>> ARMD problem statements might >>> need to document DC problems under each typical design. >>>> Please send feedback to us (either to the armd email list or to the >>>> ARMD chair Benson& Linda) to >>> indicate if we have missed any typical Data Center network designs. >>>> >>>> Your contribution can greatly accelerate the progress of ARMD WG. >>>> >>>> Thank you very much. >>>> >>>> Linda& Benson >>>> >>>> >>> >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> > > From linda.dunbar@huawei.com Sat Sep 24 06:33:34 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2127221F85FF for ; Sat, 24 Sep 2011 06:33:34 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.368 X-Spam-Level: X-Spam-Status: No, score=-6.368 tagged_above=-999 required=5 tests=[AWL=0.230, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4FkSpZ4pfCtO for ; Sat, 24 Sep 2011 06:33:32 -0700 (PDT) Received: from usaga02-in.huawei.com (usaga02-in.huawei.com [206.16.17.70]) by ietfa.amsl.com (Postfix) with ESMTP id CD5D221F85F1 for ; Sat, 24 Sep 2011 06:33:31 -0700 (PDT) Received: from huawei.com (localhost [127.0.0.1]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS100A2G4G89P@usaga02-in.huawei.com> for armd@ietf.org; Sat, 24 Sep 2011 08:36:08 -0500 (CDT) Received: from dfweml201-edg.china.huawei.com ([172.18.4.104]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LS100C6Z4G7JQ@usaga02-in.huawei.com> for armd@ietf.org; Sat, 24 Sep 2011 08:36:08 -0500 (CDT) Received: from DFWEML402-HUB.china.huawei.com (10.193.5.102) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.270.1; Sat, 24 Sep 2011 06:35:59 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.103]) by DFWEML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Sat, 24 Sep 2011 06:35:57 -0700 Date: Sat, 24 Sep 2011 13:35:57 +0000 From: Linda Dunbar In-reply-to: X-Originating-IP: [10.47.136.108] To: Vishwas Manral , Murari Sridharan Message-id: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_T8xVqtKV2eEAkfPh1N7rdA)" Content-language: en-US Accept-Language: en-US Thread-topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-index: AQHMdi0CNEHNe+RVIE+79nf9cw77PpVagLYAgAACL4CAAAI/AIAABsyAgAABZgCAAAB8AIAAANUAgAAB8ACAACv0gIABSAxg X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Sep 2011 13:33:34 -0000 --Boundary_(ID_T8xVqtKV2eEAkfPh1N7rdA) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Protocols have been developed for IP mobility between Home Gateway and Remote Gateway. The question is if we want similar protocols among TORs or among vSwitches. Handset mobility is random, but VM migration is planned. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vishwas Manral Sent: Thursday, September 22, 2011 9:38 PM To: Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing it totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I am not sure what you mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (besides ofcourse that TNI needs to be parsed in the intermediate device for hashing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling you consolidate the addresses visible to the core and by providing a hash mechanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to further distribute flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be for the switches to get full entropy from the inner Ethernet frame. I take it that your comment would be to make it explicit in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entropy label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows from/to a tenant will be hashed to the same path (even with the changes in switches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for providing the randomization for hashing flows between same to/from on different paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route everything back through the enterprise - not necessarily the optimal from the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the cloud and the enterprise? Otherwise I don't see the implication on virtualization as its no different than getting the traffic routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with external peers, the host either uses public IP address which is visible to external peers or uses private IP address which is translated to public address at the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center" advertise this host address to external peers? Or will all external peers go through enterprise's gateway to reach this host which is no longer residing in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --Boundary_(ID_T8xVqtKV2eEAkfPh1N7rdA) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

Protocols have been developed for IP mobility between Home Gateway and Remote Gateway. The question is if we want similar protocols among TORs or among vSwitches. Handset mobility is random, but VM migration is planned.  

 

Linda

 

From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vishwas Manral
Sent: Thursday, September 22, 2011 9:38 PM
To: Murari Sridharan
Cc: armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world?

 

Murari think IP mobility. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com> wrote:

Do you have a scenario in mind?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world?

Hi Murari,

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com> wrote:

You mean not an Ethernet frame but some IP payload?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world?

 

Murari,

 

What I am saying is the inner header should be allowed to be L3.

 

From the diagram you have that does not seem to be the case. Am I missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com> wrote:

Vishwas, Thanks for the feedback we will definitely consider adding that. I am not sure what you mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 

Thanks

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solution (besides ofcourse that TNI needs to be parsed in the intermediate device for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tunneling you consolidate the addresses visible to the core and by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of L2 alone. I guess it would be the same comment for the VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:

The draft mentions exactly this as one use of the reserved 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to further distribute flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be for the switches to get full entropy from the inner Ethernet frame. I take it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key field to

   add flow based entropy and tag all the packets from a flow with an entropy label.

 

Simha

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the addition of TNI.

 

A question I have is a TNI identifies a particular tenant and all flows from/to a tenant will be hashed to the same path (even with the changes in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved for providing the randomization for hashing flows between same to/from on different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:

The easiest from the point of view of configuration would be to route everything back through the enterprise - not necessarily the optimal from the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the cloud and the enterprise? Otherwise I don't see the implication on virtualization as its no different than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world?


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with external peers, the host either uses public IP address which is visible to external peers or uses private IP address which is translated to public address at the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cloud Data center" advertise this host address to external peers? Or will all external peers go through enterprise's gateway to reach this host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.black@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@emc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as defined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by
> someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the
> > > equipment to be able to do multicast forwarding, and even if does,
> > > because of pruning requirements the number of groups would be very
> > > large.  The average data center switch probably won't handle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the extra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?
> > > Are they planning to use multicast GRE, or just use directory based
> > > lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too, in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know cloud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches
> > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstream
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.
> > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), We can also
> > > have a heirarchical network, with the core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested that it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling issues
> > can be properly described. Many data center operators have expressed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too
> much
> > detail. During the journey of establishing ARMD, we have come across
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),
> > >   * large layer 2 with hundreds (or thousands) of ToRs being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary router
> > (s)
> > >   * CLOS design  with thousands of switches. This design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own problems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list  or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network designs.
> > >
> > > Your contribution can greatly accelerate the progress of ARMD WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

--Boundary_(ID_T8xVqtKV2eEAkfPh1N7rdA)-- From narave@microsoft.com Sat Sep 24 10:34:22 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BD29221F8B79 for ; Sat, 24 Sep 2011 10:34:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.598 X-Spam-Level: X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5v8XCmF8EAJ6 for ; Sat, 24 Sep 2011 10:34:16 -0700 (PDT) Received: from smtp.microsoft.com (mail3.microsoft.com [131.107.115.214]) by ietfa.amsl.com (Postfix) with ESMTP id 1EB2521F8B78 for ; Sat, 24 Sep 2011 10:34:15 -0700 (PDT) Received: from TK5EX14MLTC104.redmond.corp.microsoft.com (157.54.79.159) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Sat, 24 Sep 2011 10:36:53 -0700 Received: from TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com (157.54.71.68) by TK5EX14MLTC104.redmond.corp.microsoft.com (157.54.79.159) with Microsoft SMTP Server (TLS) id 14.1.339.2; Sat, 24 Sep 2011 10:36:53 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.68]) with mapi id 14.01.0339.002; Sat, 24 Sep 2011 10:36:52 -0700 From: Narasimhan Venkataramaiah To: Linda Dunbar , Vishwas Manral , Murari Sridharan Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHGgAcU1gD//4sFsIAAeWoAgAAGy4CAAAFnAIAAAHwAgAAA1ACAAAHwAIAAK/SAgAJKO4D//81YUA== Date: Sat, 24 Sep 2011 17:36:51 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD3872E@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> <7C4DFCE962635144B8FAE8CA11D0BF1E058CCE4D4C@MX14A.corp.emc.com> <4A95BA014132FF49AE685FAB4B9F17F610CA2E67@dfweml503-mbx.china.huawei.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD2AF86@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <65755BEBE02F7C41BD4F137AED91DA5E2FD36DD1@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.90] Content-Type: multipart/alternative; boundary="_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3872ETK5EX14MBXW601w_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Sep 2011 17:34:22 -0000 --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3872ETK5EX14MBXW601w_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Posting back some offline emails From: Narasimhan Venkataramaiah Sent: Thursday, September 22, 2011 9:49 PM To: 'Vishwas Manral' Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser Subject: RE: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? In some cases the it may be more efficient to just send IP - but given MAC = in GRE is a superset, its simpler to just always do MAC in GRE from an engi= neering point of view for various hardware devices that parse the packets t= o provide added functionality in the path. They have to deal with one type = of packets. We would need MAC in GRE to stretch L2 to non NV-GRE subnets or= to carry non IP payload. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 9:43 PM To: Narasimhan Venkataramaiah Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? I agree PMIP may not directly work here. In such cases the Layer-2 header h= owever is still of no use. -Vishwas On Thu, Sep 22, 2011 at 9:07 PM, Narasimhan Venkataramaiah > wrote: Right - with NVGRE L2 mobility happens automatically - however if you were = to achieve it without NVGRE in a multitenant environment you would need som= e VLAN reconfiguration. NVGRE achieves that with Tenant IDs. PMIP is not multi-tenancy aware in the sense that the proxy won't be able t= o distinguish between multiple tenants using same IP address space in the v= irtual network. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 8:51 PM To: Narasimhan Venkataramaiah Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, With that aspect there is little difference in the functionality, though L3= -in-L3 can use PMIP and other mobility work already done. Also it begs the question why you even need L2 mobility in this case Moving off the list as Benson feels the replies are not substantial. Thanks, Vishwas On Thu, Sep 22, 2011 at 8:34 PM, Narasimhan Venkataramaiah > wrote: Say you have a virtual subnet that spans 2 physical subnets. A VM in a virt= ual subnet moving to another host within the same physical subnet is L2 mob= ility and a VM moving to another host on another in a different physical s= ubnet is L3 mobility. These 2 cases are the same for NV-GRE in terms of enc= apsulation as the resulting packets are the same. What does IP in GRE achie= ve that MAC in GRE does not? Simha From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Lin= da Dunbar Sent: Saturday, September 24, 2011 6:36 AM To: Vishwas Manral; Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Protocols have been developed for IP mobility between Home Gateway and Remo= te Gateway. The question is if we want similar protocols among TORs or amon= g vSwitches. Handset mobility is random, but VM migration is planned. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vis= hwas Manral Sent: Thursday, September 22, 2011 9:38 PM To: Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3872ETK5EX14MBXW601w_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Posting back some offline= emails

 <= /p>

From: Narasimh= an Venkataramaiah
Sent: Thursday, September 22, 2011 9:49 PM
To: 'Vishwas Manral'
Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schl= iesser
Subject: RE: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

In some cases the it may = be more efficient to just send IP – but given MAC in GRE is a superse= t, its simpler to just always do MAC in GRE from an engineering point of view for various hardware devices that parse the packets to provi= de added functionality in the path. They have to deal with one type of pack= ets. We would need MAC in GRE to stretch L2 to non NV-GRE subnets or to car= ry non IP payload.

 <= /p>

Simha

 <= /p>

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 9:43 PM
To: Narasimhan Venkataramaiah
Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schl= iesser
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

I agree PMIP may not directly work here. In such cas= es the Layer-2 header however is still of no use.

 

-Vishwas

On Thu, Sep 22, 2011 at 9:07 PM, Narasimhan Venkatar= amaiah <narave@microsoft.com= > wrote:

Right – with = NVGRE L2 mobility happens automatically – however if you were to achi= eve it without NVGRE in a multitenant environment you would need some VLAN reconfiguration. NVGRE achieves that with Tenant = IDs.

PMIP is not multi-t= enancy aware in the sense that the proxy won’t be able to distinguish= between multiple tenants using same IP address space in the virtual network. 

Simha

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 8:51 PM
To: Narasimhan Venkataramaiah
Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

With that aspect there is little difference in the functionality, = though L3-in-L3 can use PMIP and other mobility work already done.

Also it begs the question why you even need L2 mobility in this ca= se

Moving off the list as Benson feels the replies are not substantia= l.

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 8:34 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

Say you have a virt= ual subnet that spans 2 physical subnets. A VM in a virtual subnet moving t= o another host within the same physical subnet is L2 mobility and a VM moving to another host on another in  = a different physical subnet is L3 mobility. These 2 cases are the same for = NV-GRE in terms of encapsulation as the resulting packets are the same. Wha= t does IP in GRE achieve that MAC in GRE does not?

Simha

 <= /p>

 <= /p>

From: armd-bou= nces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Linda Dunbar
Sent: Saturday, September 24, 2011 6:36 AM
To: Vishwas Manral; Murari Sridharan
Cc: armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Protocols have been devel= oped for IP mobility between Home Gateway and Remote Gateway. The question = is if we want similar protocols among TORs or among vSwitches. Handset mobility is random, but VM migration is planned.  =

 <= /p>

Linda

 <= /p>

From: armd-bou= nces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vishwas Manral
Sent: Thursday, September 22, 2011 9:38 PM
To: Murari Sridharan
Cc: armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari think IP mobility. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan &l= t;muraris@microsoft.com> wr= ote:

Do you have a scenario in mind?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.black@emc.co= m; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

Hi Murari,

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

You mean not an Eth= ernet frame but some IP payload?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allowed to be L3.

 

From the diagram you have that does not seem to be the case. Am I = missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Vishwas, Thanks for= the feedback we will definitely consider adding that. I am not sure what y= ou mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /o:p>

Thanks<= /o:p>

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solu= tion (besides ofcourse that TNI needs to be parsed in the intermediate devi= ce for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tu= nneling you consolidate the addresses visible to the core an= d by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NV= GRE instead of L2 alone. I guess it would be the same comment for the = VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

--_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3872ETK5EX14MBXW601w_-- From bschlies@cisco.com Sat Sep 24 11:22:26 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9976721F8713 for ; Sat, 24 Sep 2011 11:22:26 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.296 X-Spam-Level: X-Spam-Status: No, score=-2.296 tagged_above=-999 required=5 tests=[AWL=-1.094, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2-I9Y9AvuuuF for ; Sat, 24 Sep 2011 11:22:21 -0700 (PDT) Received: from rcdn-iport-1.cisco.com (rcdn-iport-1.cisco.com [173.37.86.72]) by ietfa.amsl.com (Postfix) with ESMTP id A07BA21F86A1 for ; Sat, 24 Sep 2011 11:22:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=bschlies@cisco.com; l=54712; q=dns/txt; s=iport; t=1316888698; x=1318098298; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=a1qYHR7DVg0EzhytaggkHgLQ738aoq4qpLoKGWlusFk=; b=jT4gMSdwBqXfyHmALouP1+WafhbBVnhbn6a0rr0VdH+8NJiB1LFmnxrx 5AmzX0xauQo+XZ35WTVzwXoGVNsRpseGn/WQvJSzsBqx7WzEpUijAfKx8 bJDsUueK7WJeuHdH6VrgLWb8naPIUQQYAx2zFqxDnEcV44Uxkx6CLgrWn Y=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArwAAL8ffk6tJXG+/2dsb2JhbABCgk2WU4deAYR9djxueIFTAQEBAQIBAQEBDwEHASIxBAcFBwYBCA4DBAEBASABAgQoBh8JCAIEAQ0FCRIEA4dWBppOAZ0YhwsEh0Iwi2CFK4Rshzo X-IronPort-AV: E=Sophos;i="4.68,435,1312156800"; d="scan'208,217";a="23788451" Received: from rcdn-core2-3.cisco.com ([173.37.113.190]) by rcdn-iport-1.cisco.com with ESMTP; 24 Sep 2011 18:24:56 +0000 Received: from xbh-rcd-301.cisco.com (xbh-rcd-301.cisco.com [72.163.63.8]) by rcdn-core2-3.cisco.com (8.14.3/8.14.3) with ESMTP id p8OIOuUn000455; Sat, 24 Sep 2011 18:24:56 GMT Received: from xmb-rcd-206.cisco.com ([72.163.62.213]) by xbh-rcd-301.cisco.com with Microsoft SMTPSVC(6.0.3790.4675); Sat, 24 Sep 2011 13:24:56 -0500 Received: from 10.21.72.138 ([10.21.72.138]) by XMB-RCD-206.cisco.com ([72.163.62.213]) via Exchange Front-End Server email.cisco.com ([171.70.151.174]) with Microsoft Exchange Server HTTP-DAV ; Sat, 24 Sep 2011 18:24:55 +0000 User-Agent: Microsoft-Entourage/12.31.0.110725 Date: Sat, 24 Sep 2011 13:24:50 -0500 From: Benson Schliesser To: Narasimhan Venkataramaiah , Linda Dunbar , Vishwas Manral , Murari Sridharan Message-ID: Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMdgwxzKQpmaafWkqNIE83xQq6KJVTbCHGgAcU1gD//4sFsIAAeWoAgAAGy4CAAAFnAIAAAHwAgAAA1ACAAAHwAIAAK/SAgAJKO4D//81YUIAADgaw In-Reply-To: <65755BEBE02F7C41BD4F137AED91DA5E2FD3872E@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3399715491_61616018" X-OriginalArrivalTime: 24 Sep 2011 18:24:56.0132 (UTC) FILETIME=[42A93040:01CC7AE7] Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Sep 2011 18:22:26 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3399715491_61616018 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Thanks, Simha. Leaving aside your comments about support for non-IP payloads, etc, and focusing strictly on address resolution: Would carrying IP in the overlay inherently localize the address resolution functions (ARP, ND)? Conversely, does carrying L2 frames in the overlay suggest the extension or enlargement of ARP/ND domains? Or would you propose an ARP/ND proxy in the MAC overlay? It seems to me that scale of the address resolution function will be affected by the way it=B9s distributed (or not) throughout the overlay network. Likewise, this may have impact on the forwarding efficiency of traffic (i.e. when there are multiple inbound/outbound paths attached to th= e overlay). Cheers, -Benson On 9/24/11 12:36 PM, "Narasimhan Venkataramaiah" wrote: > Posting back some offline emails > =20 > From: Narasimhan Venkataramaiah > Sent: Thursday, September 22, 2011 9:49 PM > To: 'Vishwas Manral' > Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesse= r > Subject: RE: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > =20 > In some cases the it may be more efficient to just send IP =AD but given MA= C in > GRE is a superset, its simpler to just always do MAC in GRE from an > engineering point of view for various hardware devices that parse the pac= kets > to provide added functionality in the path. They have to deal with one ty= pe of > packets. We would need MAC in GRE to stretch L2 to non NV-GRE subnets or = to > carry non IP payload. > =20 > Simha > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 9:43 PM > To: Narasimhan Venkataramaiah > Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesse= r > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > =20 > I agree PMIP may not directly work here. In such cases the Layer-2 header > however is still of no use. > =20 > -Vishwas > On Thu, Sep 22, 2011 at 9:07 PM, Narasimhan Venkataramaiah > wrote: > Right =AD with NVGRE L2 mobility happens automatically =AD however if you wer= e to > achieve it without NVGRE in a multitenant environment you would need some= VLAN > reconfiguration. NVGRE achieves that with Tenant IDs. > PMIP is not multi-tenancy aware in the sense that the proxy won=B9t be able= to > distinguish between multiple tenants using same IP address space in the > virtual network.=20 > Simha > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 8:51 PM > To: Narasimhan Venkataramaiah > Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesse= r >=20 > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > =20 > Hi Simha, > With that aspect there is little difference in the functionality, though > L3-in-L3 can use PMIP and other mobility work already done. > Also it begs the question why you even need L2 mobility in this case > Moving off the list as Benson feels the replies are not substantial. > Thanks, > Vishwas > On Thu, Sep 22, 2011 at 8:34 PM, Narasimhan Venkataramaiah > wrote: > Say you have a virtual subnet that spans 2 physical subnets. A VM in a vi= rtual > subnet moving to another host within the same physical subnet is L2 mobil= ity > and a VM moving to another host on another in a different physical subne= t is > L3 mobility. These 2 cases are the same for NV-GRE in terms of encapsulat= ion > as the resulting packets are the same. What does IP in GRE achieve that M= AC in > GRE does not?=20 > Simha > =20 > =20 >=20 > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of L= inda > Dunbar > Sent: Saturday, September 24, 2011 6:36 AM > To: Vishwas Manral; Murari Sridharan > Cc: armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > =20 > Protocols have been developed for IP mobility between Home Gateway and Re= mote > Gateway. The question is if we want similar protocols among TORs or among > vSwitches. Handset mobility is random, but VM migration is planned. > =20 > Linda > =20 >=20 > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Vishwas Manral > Sent: Thursday, September 22, 2011 9:38 PM > To: Murari Sridharan > Cc: armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > =20 >=20 > Murari think IP mobility. :) >=20 > =20 >=20 > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: >=20 > Do you have a scenario in mind? >=20 >=20 > From: Vishwas Manral > Sent: 9/22/2011 4:55 PM >=20 >=20 > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > Hi Murari, >=20 > =20 >=20 > Yes that is what I mean. >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: >=20 > You mean not an Ethernet frame but some IP payload? > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:49 PM > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org=20 >=20 >=20 > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Murari, >=20 > =20 >=20 > What I am saying is the inner header should be allowed to be L3. >=20 > =20 >=20 > From the diagram you have that does not seem to be the case. Am I missing= it > totally? >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: >=20 > Vishwas, Thanks for the feedback we will definitely consider adding that.= I am > not sure what you mean by doing L3 instead of L2. We allow any arbitrary > virtual topology including L3. > =20 > Thanks > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:19 PM >=20 >=20 > To: Narasimhan Venkataramaiah > Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Hi Simha, >=20 > =20 >=20 > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device = for > hashing and using lesser number of bytes). >=20 > =20 >=20 > I would think you should add it to your draft immediately. With tunneling= you > consolidate the addresses visible to the core and by providing a hash > mechanism, you are providing some level of randomness. >=20 > =20 >=20 > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead = of L2 > alone. I guess it would be the same comment for the VXLAN proposal too. >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: >=20 > The draft mentions exactly this as one use of the reserved 8 bits in Sect= ion > 4. An NVGRE endpoint could use the 8 bits to further distribute flows > belonging to a particular TNI and the switches use all 32 bits to get ent= ropy. > One step further would be for the switches to get full entropy from the i= nner > Ethernet frame. I take it that your comment would be to make it explicit = in > the draft. Right? > =20 > One > such example could be to use the upper 8 bits of the Key field to > add flow based entropy and tag all the packets from a flow with an ent= ropy > label. > =20 > Simha > =20 > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:04 PM > To: Narasimhan Venkataramaiah > Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? >=20 > =20 >=20 > Hi Simha, >=20 > =20 >=20 > The main (Standards Track) change in your draft is the addition of TNI. >=20 > =20 >=20 > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes i= n > switches to do hashing to use TNI). >=20 > =20 >=20 > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths? >=20 > =20 >=20 > Thanks, >=20 > Vishwas >=20 > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from= the > enterprise point of view. Are you referring to a scenario where the VMs s= ubnet > is split between the cloud and the enterprise? Otherwise I don't see the > implication on virtualization as its no different than getting the traffi= c > routed to the enterprise in the first case. >=20 > Simha >=20 > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Du= nbar > [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advert= ise > its external facing hosts' IP addresses to external world? >=20 >=20 > Hi Murari, >=20 > Thank you very much for sharing the presentation. >=20 > One question: >=20 > For a host within an Enterprise site which needs to communicate with exte= rnal > peers, the host either uses public IP address which is visible to externa= l > peers or uses private IP address which is translated to public address at= the > Enterprise site's gateway. >=20 > When this host is moved to "Cloud data center", will the "Cloud Data cent= er" > advertise this host address to external peers? Or will all external peers= go > through enterprise's gateway to reach this host which is no longer residi= ng in > the enterprise site? >=20 > Thanks, Linda >=20 >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f >> > Murari Sridharan >> > Sent: Saturday, September 17, 2011 3:02 PM >> > To: david.black@emc.com; armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > FYI, here is a talk that I gave last week in relation to the nvgre >> > draft below. >> > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T >> > >> > Thanks >> > Murari >> > >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f >> > david.black@emc.com >> > Sent: Friday, September 16, 2011 6:14 AM >> > To: armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > And two more drafts on this topic: >> > >> > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt >> > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt >> > >> > The edge switches could be the software switches in hypervisors. >> > >> > Thanks, >> > --David >> > >> > >>> > > -----Original Message----- >>> > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behal= f >>> > > Of Warren Kumari >>> > > Sent: Wednesday, August 31, 2011 3:16 PM >>> > > To: Vishwas Manral >>> > > Cc: armd@ietf.org >>> > > Subject: Re: [armd] soliciting typical network designs for ARMD >>> > > >>> > > >>> > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >>> > > >>>> > > > Hi Linda/ Anoop, >>>> > > > >>>> > > > Here is the example of the design I was talking about, as define= d >> > by google. >>> > > >>> > > Just a clarification -- s/as defined by google/as described by >> > someone >>> > > who happens to work for google/ >>> > > >>> > > W >>> > > >>>> > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani >> > wrote: >>>> > > > >>>>>>>> > > > >>>> >>>> > > > (though I think if there was a standard way to map Multicast MAC= to >>>> > > > Multicast IP, they could >>> > > probably use such a standard mechanisms). >>>>>>>> > > > >>>> >>>> > > > >>>> > > > They can do that, but then this imposes requirements on the >>>> > > > equipment to be able to do multicast forwarding, and even if doe= s, >>>> > > > because of pruning requirements the number of groups would be ve= ry >>>> > > > large. The average data center switch probably won't handle tha= t >>>> > > > many groups. >>>> > > > >>>> > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Anoop, >>>> > > > >>>> > > > From what I know they do not use Multicast GRE (I hear the extra= 4 >>>> > > > bytes in the GRE header is a >>> > > proprietery extension). >>>> > > > >>>> > > > I think a directory based mechanism is what is used (though I th= ink >>>> > > > if there was a standard way to >>> > > map Multicast MAC to Multicast IP, they could probably use such a >> > standard mechanisms). >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani >> > wrote: >>>> > > > Hi Vishwas, >>>> > > > >>>> > > > How do they get multicast through the network in that case? >>>> > > > Are they planning to use multicast GRE, or just use directory ba= sed >>>> > > > lookups and not worry about multicast applications for now? >>>> > > > >>>> > > > Anoop >>>> > > > >>>> > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Linda, >>>> > > > >>>> > > > The data packets can be tunnelled at the ToR over say a GRE pack= et >>>> > > > and the core is a Layer-3 core >>> > > (except for the downstream ports). So we could have encapsulation/ >>> > > decapsulation of L2 over GRE at the ToR. >>>> > > > >>>> > > > The very same thing can be done at the hypervisor layer too, in >>>> > > > which case the entire DC network >>> > > would look like a Layer-3 flat network including the ToR to server >>> > > link and the hypervisor would do the tunneling. >>>> > > > >>>> > > > I am not sure if you got the points above or not. I know cloud O= S >>>> > > > companies that provide the service >>> > > and have big announced customers. >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar >> > wrote: >>>> > > > Vishwas, >>>> > > > >>>> > > > In my mind the bullet 1) in the list refers to ToR switches >>>> > > > downstream ports (facing servers) >>> > > running Layer 2 and ToR uplinks ports run IP Layer 3. >>>> > > > >>>> > > > Have you seen data center networks with ToR switches downstream >>>> > > > ports (i.e. facing servers) enabling >>> > > IP routing, even though the physical links are Ethernet? >>>> > > > If yes, we should definitely include it in the ARMD draft. >>>> > > > >>>> > > > Thanks, >>>> > > > Linda >>>> > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Linda, >>>> > > > I am unsure what you mean by this, but: >>>> > > > * layer 3 all the way to TOR (Top of Rack switches), We can al= so >>>> > > > have a heirarchical network, with the core totally Layer-3 (and >>>> > > > having seperate >>> > > routing), from the hosts still in a large Layer-3 subnet. Another >>> > > aspect could be to have a totally >>> > > Layer-3 network. >>>> > > > >>>> > > > The difference between them is the link between the servers and = the >> > ToR. >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar >> > wrote: >>>> > > > During the 81st IETF ARMD WG discussion, it was suggested that i= t >> > is >>>> > > > necessary to document typical >>> > > data center network designs so that address resolution scaling issu= es >>> > > can be properly described. Many data center operators have expresse= d >> > that they can't openly reveal their detailed network designs. >>> > > Therefore, we only want to document anonymous designs without too >> > much >>> > > detail. During the journey of establishing ARMD, we have come acros= s >> > the following typical data center network designs: >>>> > > > * layer 3 all the way to TOR (Top of Rack switches), >>>> > > > * large layer 2 with hundreds (or thousands) of ToRs being >>>> > > > interconnected by Layer 2. This >>> > > design will have thousands of hosts under the L2/L3 boundary router >>> > > (s) >>>> > > > * CLOS design with thousands of switches. This design will ha= ve >>>> > > > thousands of hosts under the >>> > > L2/L3 boundary router(s) >>>> > > > We have heard that each of the designs above has its own problem= s. >>>> > > > ARMD problem statements might >>> > > need to document DC problems under each typical design. >>>> > > > Please send feedback to us (either to the armd email list or to >> > the >>>> > > > ARMD chair Benson & Linda) to >>> > > indicate if we have missed any typical Data Center network designs. >>>> > > > >>>> > > > Your contribution can greatly accelerate the progress of ARMD WG= . >>>> > > > >>>> > > > Thank you very much. >>>> > > > >>>> > > > Linda & Benson >>>> > > > > =20 > =20 > =20 > =20 >=20 >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --B_3399715491_61616018 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Re: [armd] how does "draft-sridharan-virtualization-nvgre-00&qu= ot;advertise its external facing hosts' IP addresses to external world?</TIT= LE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:11pt= '>Thanks, Simha.<BR> <BR> Leaving aside your comments about support for non-IP payloads, etc, and foc= using strictly on address resolution: Would carrying IP in the overlay inher= ently localize the address resolution functions (ARP, ND)? Conversely, does = carrying L2 frames in the overlay suggest the extension or enlargement of AR= P/ND domains? Or would you propose an ARP/ND proxy in the MAC overlay?<BR> <BR> It seems to me that scale of the address resolution function will be affect= ed by the way it’s distributed (or not) throughout the overlay network= . Likewise, this may have impact on the forwarding efficiency of traffic (i.= e. when there are multiple inbound/outbound paths attached to the overlay).<= BR> <BR> Cheers,<BR> -Benson<BR> <BR> <BR> On 9/24/11 12:36 PM, "Narasimhan Venkataramaiah" <<a href=3D"nar= ave@microsoft.com">narave@microsoft.com</a>> wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><= SPAN STYLE=3D'font-size:11pt'>Posting back some offline emails<BR>  <BR> </SPAN><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'><B>From:</B> Narasimhan = Venkataramaiah <BR> <B>Sent:</B> Thursday, September 22, 2011 9:49 PM<BR> <B>To:</B> 'Vishwas Manral'<BR> <B>Cc:</B> Murari Sridharan; Linda Dunbar; <a href=3D"david.black@emc.com">da= vid.black@emc.com</a>; Benson Schliesser<BR> <B>Subject:</B> RE: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12= pt'> <BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'>In some cases the it may be more efficient to just send IP &= #8211; but given MAC in GRE is a superset, its simpler to just always do MAC= in GRE from an engineering point of view for various hardware devices that = parse the packets to provide added functionality in the path. They have to d= eal with one type of packets. We would need MAC in GRE to stretch L2 to non = NV-GRE subnets or to carry non IP payload. <BR>  <BR> Simha<BR>  <BR> </SPAN><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'><B>From:</B> Vishwas Man= ral [<a href=3D"mailto:vishwas.ietf@gmail.com">mailto:vishwas.ietf@gmail.com</= a>] <BR> <B>Sent:</B> Thursday, September 22, 2011 9:43 PM<BR> <B>To:</B> Narasimhan Venkataramaiah<BR> <B>Cc:</B> Murari Sridharan; Linda Dunbar; <a href=3D"david.black@emc.com">da= vid.black@emc.com</a>; Benson Schliesser<BR> <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12= pt'> <BR> I agree PMIP may not directly work here. In such cases the Layer-2 header h= owever is still of no use.<BR>  <BR> -Vishwas<BR> On Thu, Sep 22, 2011 at 9:07 PM, Narasimhan Venkataramaiah <<a href=3D"nar= ave@microsoft.com">narave@microsoft.com</a>> wrote:<BR> </SPAN><FONT COLOR=3D"#1F497D"><SPAN STYLE=3D'font-size:11pt'>Right – wit= h NVGRE L2 mobility happens automatically – however if you were to ach= ieve it without NVGRE in a multitenant environment you would need some VLAN = reconfiguration. NVGRE achieves that with Tenant IDs. <BR> PMIP is not multi-tenancy aware in the sense that the proxy won’t be = able to distinguish between multiple tenants using same IP address space in = the virtual network. <BR> Simha<BR> </SPAN></FONT><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'><B>From:</B> Vish= was Manral [<a href=3D"mailto:vishwas.ietf@gmail.com">mailto:vishwas.ietf@gmai= l.com</a>] <BR> <B>Sent:</B> Thursday, September 22, 2011 8:51 PM<BR> <B>To:</B> Narasimhan Venkataramaiah<BR> <B>Cc:</B> Murari Sridharan; Linda Dunbar; <a href=3D"david.black@emc.com">da= vid.black@emc.com</a>; Benson Schliesser <BR> <BR> <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT><SPAN STYLE=3D'font-size:12pt'> <BR> Hi Simha,<BR> With that aspect there is little difference in the functionality, though L3= -in-L3 can use PMIP and other mobility work already done.<BR> Also it begs the question why you even need L2 mobility in this case<BR> Moving off the list as Benson feels the replies are not substantial.<BR> Thanks,<BR> Vishwas<BR> On Thu, Sep 22, 2011 at 8:34 PM, Narasimhan Venkataramaiah <<a href=3D"nar= ave@microsoft.com">narave@microsoft.com</a>> wrote:<BR> </SPAN><FONT COLOR=3D"#1F497D"><SPAN STYLE=3D'font-size:11pt'>Say you have a vi= rtual subnet that spans 2 physical subnets. A VM in a virtual subnet moving = to another host within the same physical subnet is L2 mobility and a VM movi= ng to another host on another in  a different physical subnet is L3 mob= ility. These 2 cases are the same for NV-GRE in terms of encapsulation as th= e resulting packets are the same. What does IP in GRE achieve that MAC in GR= E does not? <BR> Simha<BR> </SPAN></FONT></FONT><SPAN STYLE=3D'font-size:11pt'><FONT FACE=3D"Calibri, Verd= ana, Helvetica, Arial"> <BR>  <BR> <BR> </FONT></SPAN><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><FONT SIZE=3D"2= "><SPAN STYLE=3D'font-size:10pt'><B>From:</B> <a href=3D"armd-bounces@ietf.org">= armd-bounces@ietf.org</a> [<a href=3D"mailto:armd-bounces@ietf.org">mailto:arm= d-bounces@ietf.org</a>] <B>On Behalf Of </B>Linda Dunbar<BR> <B>Sent:</B> Saturday, September 24, 2011 6:36 AM<BR> <B>To:</B> Vishwas Manral; Murari Sridharan<BR> <B>Cc:</B> <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12= pt'> <BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'>Protocols have been developed for IP mobility between Home G= ateway and Remote Gateway. The question is if we want similar protocols amon= g TORs or among vSwitches. Handset mobility is random, but VM migration is p= lanned.  <BR>  <BR> Linda<BR>  <BR> <BR> </SPAN><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'><B>From:</B> <a href=3D"ar= md-bounces@ietf.org">armd-bounces@ietf.org</a> [<a href=3D"mailto:armd-bounces= @ietf.org">mailto:armd-bounces@ietf.org</a>] <B>On Behalf Of </B>Vishwas Man= ral<BR> <B>Sent:</B> Thursday, September 22, 2011 9:38 PM<BR> <B>To:</B> Murari Sridharan<BR> <B>Cc:</B> <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12= pt'> <BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Mur= ari think IP mobility. :)<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>On = Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <<a href=3D"muraris@microsof= t.com">muraris@microsoft.com</a>> wrote:<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> Do you have a scenario in mind? </SPAN></FONT> <P ALIGN=3DCENTER> <FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'><HR ALIGN=3DCENTER = SIZE=3D"3" WIDTH=3D"100%"></SPAN></FONT> <P> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:11pt= '><BR> </SPAN><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'><B>From: </B>Vishwas Man= ral<BR> <B>Sent: </B>9/22/2011 4:55 PM</SPAN></FONT></FONT><FONT FACE=3D"Times New Ro= man"><SPAN STYLE=3D'font-size:12pt'> <BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'><BR= > </SPAN></FONT><FONT SIZE=3D"2"><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial= "><SPAN STYLE=3D'font-size:10pt'><B>To: </B>Murari Sridharan<BR> <B>Cc: </B>Narasimhan Venkataramaiah; Linda Dunbar; <a href=3D"david.black@em= c.com">david.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a><BR= > <B>Subject: </B>Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN = STYLE=3D'font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Hi = Murari,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Yes= that is what I mean.<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Tha= nks,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Vis= hwas<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>On = Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <<a href=3D"muraris@microsof= t.com">muraris@microsoft.com</a>> wrote:<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><SPAN STYLE=3D'font-size:11pt'><FONT COLOR=3D"#1F497D"><FONT FACE= =3D"Times New Roman">You mean not an Ethernet frame but some IP payload?<BR>  <BR> </FONT></FONT></SPAN><FONT FACE=3D"Times New Roman"><FONT SIZE=3D"2"><SPAN STYL= E=3D'font-size:10pt'><B>From:</B> Vishwas Manral [<a href=3D"mailto:vishwas.ietf= @gmail.com">mailto:vishwas.ietf@gmail.com</a>] <BR> <B>Sent:</B> Thursday, September 22, 2011 4:49 PM<BR> <B>To:</B> Murari Sridharan<BR> <B>Cc:</B> Narasimhan Venkataramaiah; Linda Dunbar; <a href=3D"david.black@em= c.com">david.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a> <B= R> </SPAN></FONT></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN = STYLE=3D'font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'><BR= > <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Mur= ari,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Wha= t I am saying is the inner header should be allowed to be L3. <BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Fro= m the diagram you have that does not seem to be the case. Am I missing it to= tally?<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Tha= nks,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Vis= hwas<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>On = Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <<a href=3D"muraris@microsof= t.com">muraris@microsoft.com</a>> wrote:<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><SPAN STYLE=3D'font-size:11pt'><FONT COLOR=3D"#1F497D"><FONT FACE= =3D"Times New Roman">Vishwas, Thanks for the feedback we will definitely consi= der adding that. I am not sure what you mean by doing L3 instead of L2. We a= llow any arbitrary virtual topology including L3. <BR>  <BR> Thanks<BR>  <BR> </FONT></FONT></SPAN><FONT FACE=3D"Times New Roman"><FONT SIZE=3D"2"><SPAN STYL= E=3D'font-size:10pt'><B>From:</B> Vishwas Manral [<a href=3D"mailto:vishwas.ietf= @gmail.com">mailto:vishwas.ietf@gmail.com</a>] <BR> <B>Sent:</B> Thursday, September 22, 2011 4:19 PM <BR> </SPAN></FONT></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN = STYLE=3D'font-size:11pt'><BR> </SPAN></FONT><FONT SIZE=3D"2"><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font= -size:10pt'><BR> <B>To:</B> Narasimhan Venkataramaiah<BR> <B>Cc:</B> Linda Dunbar; Murari Sridharan; <a href=3D"david.black@emc.com">da= vid.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN = STYLE=3D'font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Hi = Simha,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>I s= ee this as the only difference between VXLAN and the NVGRE solution (besides= ofcourse that TNI needs to be parsed in the intermediate device for hashing= and using lesser number of bytes).<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>I w= ould think you should add it to your draft immediately. With tunneling you c= onsolidate the addresses visible to the core and by providing a hash mechani= sm, you are providing some level of randomness.<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>The= other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of L2 = alone. I guess it would be the same comment for the VXLAN proposal too.<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Tha= nks,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Vis= hwas<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>On = Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <<a href=3D"narave@= microsoft.com">narave@microsoft.com</a>> wrote:<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>The= draft mentions exactly this as one use of the reserved 8 bits in Section 4.= An NVGRE endpoint could use the 8 bits to further distribute flows belongin= g to a particular TNI and the switches use all 32 bits to get entropy. One s= tep further would be for the switches to get full entropy from the inner Eth= ernet frame. I take it that your comment would be to make it explicit in the= draft. Right?<BR> </SPAN></FONT><FONT SIZE=3D"2"><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial= "><SPAN STYLE=3D'font-size:10pt'> <BR> One<BR>    such example could be to use the upper 8 bits of the Key = field to<BR>    add flow based entropy and tag all the packets from a flo= w with an entropy label.<BR>  <BR> </SPAN></FONT></FONT><FONT COLOR=3D"#1F497D"><FONT FACE=3D"Times New Roman"><SP= AN STYLE=3D'font-size:11pt'>Simha<BR>  <BR> </SPAN></FONT></FONT><FONT FACE=3D"Times New Roman"><FONT SIZE=3D"2"><SPAN STYL= E=3D'font-size:10pt'><B>From:</B> Vishwas Manral [<a href=3D"mailto:vishwas.ietf= @gmail.com">mailto:vishwas.ietf@gmail.com</a>] <BR> <B>Sent:</B> Thursday, September 22, 2011 4:04 PM<BR> <B>To:</B> Narasimhan Venkataramaiah<BR> <B>Cc:</B> Linda Dunbar; Murari Sridharan; <a href=3D"david.black@emc.com">da= vid.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> <B>Subject:</B> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external w= orld?<BR> </SPAN></FONT></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN = STYLE=3D'font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Hi = Simha,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>The= main (Standards Track) change in your draft is the addition of TNI.<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>A q= uestion I have is a TNI identifies a particular tenant and all flows from/to= a tenant will be hashed to the same path (even with the changes in switches= to do hashing to use TNI).<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Why= do you not use the last 8 bits which you have kept as reserved for providin= g the randomization for hashing flows between same to/from on different path= s?<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'> <B= R> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Tha= nks,<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>Vis= hwas<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'>On = Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <<a href=3D"narave= @microsoft.com">narave@microsoft.com</a>> wrote:<BR> The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the ent= erprise point of view. Are you referring to a scenario where the VMs subnet = is split between the cloud and the enterprise? Otherwise I don't see the imp= lication on virtualization as its no different than getting the traffic rout= ed to the enterprise in the first case.<BR> <BR> Simha<BR> <BR> ________________________________________<BR> From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a href=3D"a= rmd-bounces@ietf.org">armd-bounces@ietf.org</a>] on behalf of Linda Dunbar [= <a href=3D"linda.dunbar@huawei.com">linda.dunbar@huawei.com</a>]<BR> Sent: Sunday, September 18, 2011 7:06 AM<BR> To: Murari Sridharan; <a href=3D"david.black@emc.com">david.black@emc.com</a>= ; <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?<BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> </SPAN></FONT><FONT FACE=3D"Times New Roman"><SPAN STYLE=3D'font-size:12pt'><BR= > Hi Murari,<BR> <BR> Thank you very much for sharing the presentation.<BR> <BR> One question:<BR> <BR> For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to externa= l peers or uses private IP address which is translated to public address at = the Enterprise site's gateway.<BR> <BR> When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or will= all external peers go through enterprise's gateway to reach this host which= is no longer residing in the enterprise site?<BR> <BR> Thanks, Linda<BR> <BR> > -----Original Message-----<BR> > From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a hr= ef=3D"mailto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] On Behal= f Of<BR> > Murari Sridharan<BR> > Sent: Saturday, September 17, 2011 3:02 PM<BR> > To: <a href=3D"david.black@emc.com">david.black@emc.com</a>; <a href=3D"ar= md@ietf.org">armd@ietf.org</a><BR> > Subject: Re: [armd] soliciting typical network designs for ARMD<BR> ><BR> > FYI, here is a talk that I gave last week in relation to the nvgre<BR> > draft below.<BR> > <a href=3D"http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T">htt= p://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T</a><BR> ><BR> > Thanks<BR> > Murari<BR> ><BR> > -----Original Message-----<BR> > From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a hr= ef=3D"mailto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] On Behal= f Of<BR> > <a href=3D"david.black@emc.com">david.black@emc.com</a><BR> > Sent: Friday, September 16, 2011 6:14 AM<BR> > To: <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> > Subject: Re: [armd] soliciting typical network designs for ARMD<BR> ><BR> > And two more drafts on this topic:<BR> ><BR> > <a href=3D"http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.t= xt">http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt</a><BR> > <a href=3D"http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-0= 0.txt">http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt</a= ><BR> ><BR> > The edge switches could be the software switches in hypervisors.<BR> ><BR> > Thanks,<BR> > --David<BR> ><BR> ><BR> > > -----Original Message-----<BR> > > From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [= <a href=3D"mailto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] On = Behalf<BR> > > Of Warren Kumari<BR> > > Sent: Wednesday, August 31, 2011 3:16 PM<BR> > > To: Vishwas Manral<BR> > > Cc: <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> > > Subject: Re: [armd] soliciting typical network designs for ARMD<B= R> > ><BR> > ><BR> > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:<BR> > ><BR> > > > Hi Linda/ Anoop,<BR> > > ><BR> > > > Here is the example of the design I was talking about, as de= fined<BR> > by google.<BR> > ><BR> > > Just a clarification -- s/as defined by google/as described by<BR= > > someone<BR> > > who happens to work for google/<BR> > ><BR> > > W<BR> > ><BR> > > > <a href=3D"http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmob= ility-00.txt">http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.tx= t</a><BR> > > ><BR> > > > Thanks,<BR> > > > Vishwas<BR> > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani<BR> > <<a href=3D"anoop@alumni.duke.edu">anoop@alumni.duke.edu</a>> wrot= e:<BR> > > ><BR> > > > >>>><BR> > > > (though I think if there was a standard way to map Multicast= MAC to<BR> > > > Multicast IP, they could<BR> > > probably use such a standard mechanisms).<BR> > > > >>>><BR> > > ><BR> > > > They can do that, but then this imposes requirements on the<= BR> > > > equipment to be able to do multicast forwarding, and even if= does,<BR> > > > because of pruning requirements the number of groups would b= e very<BR> > > > large.  The average data center switch probably won't h= andle that<BR> > > > many groups.<BR> > > ><BR> > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral<BR> > <<a href=3D"vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>> wr= ote:<BR> > > > Hi Anoop,<BR> > > ><BR> > > > From what I know they do not use Multicast GRE (I hear the e= xtra 4<BR> > > > bytes in the GRE header is a<BR> > > proprietery extension).<BR> > > ><BR> > > > I think a directory based mechanism is what is used (though = I think<BR> > > > if there was a standard way to<BR> > > map Multicast MAC to Multicast IP, they could probably use such a= <BR> > standard mechanisms).<BR> > > ><BR> > > > Thanks,<BR> > > > Vishwas<BR> > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani<BR> > <<a href=3D"anoop@alumni.duke.edu">anoop@alumni.duke.edu</a>> wrot= e:<BR> > > > Hi Vishwas,<BR> > > ><BR> > > > How do they get multicast through the network in that case?<= BR> > > > Are they planning to use multicast GRE, or just use director= y based<BR> > > > lookups and not worry about multicast applications for now?<= BR> > > ><BR> > > > Anoop<BR> > > ><BR> > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral<BR> > <<a href=3D"vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>> wr= ote:<BR> > > > Hi Linda,<BR> > > ><BR> > > > The data packets can be tunnelled at the ToR over say a GRE = packet<BR> > > > and the core is a Layer-3 core<BR> > > (except for the downstream ports). So we could have encapsulation= /<BR> > > decapsulation of L2 over GRE at the ToR.<BR> > > ><BR> > > > The very same thing can be done at the hypervisor layer too,= in<BR> > > > which case the entire DC network<BR> > > would look like a Layer-3 flat network including the ToR to serve= r<BR> > > link and the hypervisor would do the tunneling.<BR> > > ><BR> > > > I am not sure if you got the points above or not. I know clo= ud OS<BR> > > > companies that provide the service<BR> > > and have big announced customers.<BR> > > ><BR> > > > Thanks,<BR> > > > Vishwas<BR> > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <<a href=3D"d= unbar.ll@gmail.com">dunbar.ll@gmail.com</a>><BR> > wrote:<BR> > > > Vishwas,<BR> > > ><BR> > > > In my mind the bullet 1) in the list refers to ToR switches<= BR> > > > downstream ports (facing servers)<BR> > > running Layer 2 and ToR uplinks ports run IP Layer 3.<BR> > > ><BR> > > > Have you seen data center networks with ToR switches downstr= eam<BR> > > > ports (i.e. facing servers) enabling<BR> > > IP routing, even though the physical links are Ethernet?<BR> > > > If yes, we should definitely include it in the ARMD draft.<B= R> > > ><BR> > > > Thanks,<BR> > > > Linda<BR> > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral<BR> > <<a href=3D"vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>> wr= ote:<BR> > > > Hi Linda,<BR> > > > I am unsure what you mean by this, but:<BR> > > >   * layer 3 all the way to TOR (Top of Rack switch= es), We can also<BR> > > > have a heirarchical network, with the core totally Layer-3 (= and<BR> > > > having seperate<BR> > > routing), from the hosts still in a large Layer-3 subnet. Another= <BR> > > aspect could be to have a totally<BR> > > Layer-3 network.<BR> > > ><BR> > > > The difference between them is the link between the servers = and the<BR> > ToR.<BR> > > ><BR> > > > Thanks,<BR> > > > Vishwas<BR> > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <<a href=3D"d= unbar.ll@gmail.com">dunbar.ll@gmail.com</a>><BR> > wrote:<BR> > > > During the 81st IETF ARMD WG discussion, it was suggested th= at it<BR> > is<BR> > > > necessary to document typical<BR> > > data center network designs so that address resolution scaling is= sues<BR> > > can be properly described. Many data center operators have expres= sed<BR> > that they can't openly reveal their detailed network designs.<BR> > > Therefore, we only want to document anonymous designs without too= <BR> > much<BR> > > detail. During the journey of establishing ARMD, we have come acr= oss<BR> > the following typical data center network designs:<BR> > > >   * layer 3 all the way to TOR (Top of Rack switch= es),<BR> > > >   * large layer 2 with hundreds (or thousands) of = ToRs being<BR> > > > interconnected by Layer 2. This<BR> > > design will have thousands of hosts under the L2/L3 boundary rout= er<BR> > > (s)<BR> > > >   * CLOS design  with thousands of switches. = This design will have<BR> > > > thousands of hosts under the<BR> > > L2/L3 boundary router(s)<BR> > > > We have heard that each of the designs above has its own pro= blems.<BR> > > > ARMD problem statements might<BR> > > need to document DC problems under each typical design.<BR> > > > Please send feedback to us (either to the armd email list &n= bsp;or to<BR> > the<BR> > > > ARMD chair Benson & Linda) to<BR> > > indicate if we have missed any typical Data Center network design= s.<BR> > > ><BR> > > > Your contribution can greatly accelerate the progress of ARM= D WG.<BR> > > ><BR> > > > Thank you very much.<BR> > > ><BR> > > > Linda & Benson<BR> > > ><BR>  <BR>  <BR>  <BR>  <BR> </SPAN></FONT><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'= font-size:11pt'><BR> <HR ALIGN=3DCENTER SIZE=3D"3" WIDTH=3D"95%"></SPAN></FONT><FONT SIZE=3D"2"><FONT FA= CE=3D"Consolas, Courier New, Courier"><SPAN STYLE=3D'font-size:10pt'>___________= ____________________________________<BR> armd mailing list<BR> <a href=3D"armd@ietf.org">armd@ietf.org</a><BR> <a href=3D"https://www.ietf.org/mailman/listinfo/armd">https://www.ietf.org/m= ailman/listinfo/armd</a><BR> </SPAN></FONT></FONT></BLOCKQUOTE> </BODY> </HTML> --B_3399715491_61616018-- From narave@microsoft.com Sun Sep 25 19:18:48 2011 Return-Path: <narave@microsoft.com> X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7D9A821F85B1 for <armd@ietfa.amsl.com>; Sun, 25 Sep 2011 19:18:48 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.598 X-Spam-Level: X-Spam-Status: No, score=-10.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Qhyca2zmEp5 for <armd@ietfa.amsl.com>; Sun, 25 Sep 2011 19:18:40 -0700 (PDT) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.215]) by ietfa.amsl.com (Postfix) with ESMTP id 6E71821F85B5 for <armd@ietf.org>; Sun, 25 Sep 2011 19:18:40 -0700 (PDT) Received: from TK5EX14HUBC104.redmond.corp.microsoft.com (157.54.80.25) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Sun, 25 Sep 2011 19:21:20 -0700 Received: from TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com (157.54.71.68) by TK5EX14HUBC104.redmond.corp.microsoft.com (157.54.80.25) with Microsoft SMTP Server (TLS) id 14.1.339.2; Sun, 25 Sep 2011 19:21:20 -0700 Received: from TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com ([169.254.1.117]) by TK5EX14MLTW652.wingroup.windeploy.ntdev.microsoft.com ([157.54.71.68]) with mapi id 14.01.0339.002; Sun, 25 Sep 2011 19:21:19 -0700 From: Narasimhan Venkataramaiah <narave@microsoft.com> To: Benson Schliesser <bschlies@cisco.com>, Linda Dunbar <linda.dunbar@huawei.com>, Vishwas Manral <vishwas.ietf@gmail.com>, "Murari Sridharan" <muraris@microsoft.com> Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHMeudIkbNdsFA7r0K+ePmu2p7juJVe49+A Date: Mon, 26 Sep 2011 02:21:18 +0000 Message-ID: <65755BEBE02F7C41BD4F137AED91DA5E2FD3B544@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> References: <65755BEBE02F7C41BD4F137AED91DA5E2FD3872E@TK5EX14MBXW601.wingroup.windeploy.ntdev.microsoft.com> <CAA38AA2.15970%bschlies@cisco.com> In-Reply-To: <CAA38AA2.15970%bschlies@cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.54.51.90] Content-Type: multipart/alternative; boundary="_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3B544TK5EX14MBXW601w_" MIME-Version: 1.0 Cc: "armd@ietf.org" <armd@ietf.org> Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> List-Unsubscribe: <https://www.ietf.org/mailman/options/armd>, <mailto:armd-request@ietf.org?subject=unsubscribe> List-Archive: <http://www.ietf.org/mail-archive/web/armd> List-Post: <mailto:armd@ietf.org> List-Help: <mailto:armd-request@ietf.org?subject=help> List-Subscribe: <https://www.ietf.org/mailman/listinfo/armd>, <mailto:armd-request@ietf.org?subject=subscribe> X-List-Received-Date: Mon, 26 Sep 2011 02:18:48 -0000 --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3B544TK5EX14MBXW601w_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Would carrying IP in the overlay inherently localize the address resolution= functions (ARP, ND)? [Simha] Not necessarily. ND can still work over an IP based overlay. ARP ca= n also be made to work by mapping the ARP to some IP packet that performs r= esolution in the overlay - somewhat like teredo that maps ND packets to UDP= bubble packets. But I can't see reason to do so. Conversely, does carrying L2 frames in the overlay suggest the extension or= enlargement of ARP/ND domains? Or would you propose an ARP/ND proxy in the= MAC overlay? [Simha] No. The virtual networks are separating the configuration done on t= he host vs the configuration done in the network to create a certain networ= k topology. The information that is required to create this separation can = be used to make ARP/ND more efficient. One option is to proxy ARP/ND. The o= ther option is to unicast the ARP/ND packet to a tunnel endpoint from where= the ARP/ND target is reached without flooding. This is not super useful fo= r ARP. However it is useful for ND, say, if SEND is used. It seems to me that scale of the address resolution function will be affect= ed by the way it's distributed (or not) throughout the overlay network. Lik= ewise, this may have impact on the forwarding efficiency of traffic (i.e. w= hen there are multiple inbound/outbound paths attached to the overlay). [Simha] For ARP/ND the forwarding efficiency can be made independent of the= number of paths in the overlay by using the information present at the tun= nel endpoints. In other words ARP/ND packets will be as efficient to forwar= d as any unicast packet. But i agree with your point for general broadcast = that realizing broadcast domains using IP multicast might just move the for= warding efficiency issue from L2 to L3. Simha From: Benson Schliesser [mailto:bschlies@cisco.com] Sent: Saturday, September 24, 2011 11:25 AM To: Narasimhan Venkataramaiah; Linda Dunbar; Vishwas Manral; Murari Sridhar= an Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"adver= tise its external facing hosts' IP addresses to external world? Thanks, Simha. Leaving aside your comments about support for non-IP payloads, etc, and foc= using strictly on address resolution: Would carrying IP in the overlay inhe= rently localize the address resolution functions (ARP, ND)? Conversely, doe= s carrying L2 frames in the overlay suggest the extension or enlargement of= ARP/ND domains? Or would you propose an ARP/ND proxy in the MAC overlay? It seems to me that scale of the address resolution function will be affect= ed by the way it's distributed (or not) throughout the overlay network. Lik= ewise, this may have impact on the forwarding efficiency of traffic (i.e. w= hen there are multiple inbound/outbound paths attached to the overlay). Cheers, -Benson On 9/24/11 12:36 PM, "Narasimhan Venkataramaiah" <narave@microsoft.com> wro= te: Posting back some offline emails From: Narasimhan Venkataramaiah Sent: Thursday, September 22, 2011 9:49 PM To: 'Vishwas Manral' Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser Subject: RE: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? In some cases the it may be more efficient to just send IP - but given MAC = in GRE is a superset, its simpler to just always do MAC in GRE from an engi= neering point of view for various hardware devices that parse the packets t= o provide added functionality in the path. They have to deal with one type = of packets. We would need MAC in GRE to stretch L2 to non NV-GRE subnets or= to carry non IP payload. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 9:43 PM To: Narasimhan Venkataramaiah Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? I agree PMIP may not directly work here. In such cases the Layer-2 header h= owever is still of no use. -Vishwas On Thu, Sep 22, 2011 at 9:07 PM, Narasimhan Venkataramaiah <narave@microsof= t.com> wrote: Right - with NVGRE L2 mobility happens automatically - however if you were = to achieve it without NVGRE in a multitenant environment you would need som= e VLAN reconfiguration. NVGRE achieves that with Tenant IDs. PMIP is not multi-tenancy aware in the sense that the proxy won't be able t= o distinguish between multiple tenants using same IP address space in the v= irtual network. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 8:51 PM To: Narasimhan Venkataramaiah Cc: Murari Sridharan; Linda Dunbar; david.black@emc.com; Benson Schliesser Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, With that aspect there is little difference in the functionality, though L3= -in-L3 can use PMIP and other mobility work already done. Also it begs the question why you even need L2 mobility in this case Moving off the list as Benson feels the replies are not substantial. Thanks, Vishwas On Thu, Sep 22, 2011 at 8:34 PM, Narasimhan Venkataramaiah <narave@microsof= t.com> wrote: Say you have a virtual subnet that spans 2 physical subnets. A VM in a virt= ual subnet moving to another host within the same physical subnet is L2 mob= ility and a VM moving to another host on another in a different physical s= ubnet is L3 mobility. These 2 cases are the same for NV-GRE in terms of enc= apsulation as the resulting packets are the same. What does IP in GRE achie= ve that MAC in GRE does not? Simha From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Lin= da Dunbar Sent: Saturday, September 24, 2011 6:36 AM To: Vishwas Manral; Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Protocols have been developed for IP mobility between Home Gateway and Remo= te Gateway. The question is if we want similar protocols among TORs or amon= g vSwitches. Handset mobility is random, but VM migration is planned. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vis= hwas Manral Sent: Thursday, September 22, 2011 9:38 PM To: Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com> w= rote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf= .org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com> w= rote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf= .org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com> w= rote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsof= t.com> wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microso= ft.com> wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunb= ar [linda.dunbar@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > <anoop@alumni.duke.edu> wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > <vishwas.ietf@gmail.com> wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > <anoop@alumni.duke.edu> wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > <vishwas.ietf@gmail.com> wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com> > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > <vishwas.ietf@gmail.com> wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > ________________________________ _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3B544TK5EX14MBXW601w_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)"> <!--[if !mso]><style>v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);} </style><![endif]--> <title>Re: [armd] how does "draft-sridharan-virtualization-nvgre-00&qu= ot;advertise its external facing hosts' IP addresses to external world?</ti= tle> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman","serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p {mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; font-size:12.0pt; font-family:"Times New Roman","serif";} p.MsoAcetate, li.MsoAcetate, div.MsoAcetate {mso-style-priority:99; mso-style-link:"Balloon Text Char"; margin:0in; margin-bottom:.0001pt; font-size:8.0pt; font-family:"Tahoma","sans-serif";} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} span.BalloonTextChar {mso-style-name:"Balloon Text Char"; mso-style-priority:99; mso-style-link:"Balloon Text"; font-family:"Tahoma","sans-serif";} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><i><span style=3D"font-size:11.0pt;font-family:"= ;Calibri","sans-serif";color:#1F497D">Would carrying IP in t= he overlay inherently localize the address resolution functions (ARP, ND)? <o:p></o:p></span></i></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D">[Simha] Not necessarily. = ND can still work over an IP based overlay. ARP can also be made to work by= mapping the ARP to some IP packet that performs resolution in the overlay - somewhat like teredo that maps ND packets to UDP bubble p= ackets. But I can’t see reason to do so.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D"><o:p> </o:p></span><= /p> <p class=3D"MsoNormal"><i><span style=3D"font-size:11.0pt;font-family:"= ;Calibri","sans-serif";color:#1F497D">Conversely, does carry= ing L2 frames in the overlay suggest the extension or enlargement of ARP/ND= domains? Or would you propose an ARP/ND proxy in the MAC overlay?<o:p></o:p></span></i></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D">[Simha] No. The virtual n= etworks are separating the configuration done on the host vs the configurat= ion done in the network to create a certain network topology. The information that is required to create this separation can be used to = make ARP/ND more efficient. One option is to proxy ARP/ND. The other option= is to unicast the ARP/ND packet to a tunnel endpoint from where the ARP/ND= target is reached without flooding. This is not super useful for ARP. However it is useful for ND, say, if SEN= D is used.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D"><o:p> </o:p></span><= /p> <p class=3D"MsoNormal"><i><span style=3D"font-size:11.0pt;font-family:"= ;Calibri","sans-serif";color:#1F497D">It seems to me that sc= ale of the address resolution function will be affected by the way it’= ;s distributed (or not) throughout the overlay network. Likewise, this may have impact on the forwarding efficiency of traffic (i.e. when th= ere are multiple inbound/outbound paths attached to the overlay).<o:p></o:p= ></span></i></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D">[Simha] For ARP/ND the fo= rwarding efficiency can be made independent of the number of paths in the o= verlay by using the information present at the tunnel endpoints. In other words ARP/ND packets will be as efficient to forward as any unica= st packet. But i agree with your point for general broadcast that realizing= broadcast domains using IP multicast might just move the forwarding effici= ency issue from L2 to L3.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D"><o:p> </o:p></span><= /p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D">Simha<o:p></o:p></span></= p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif";color:#1F497D"><o:p> </o:p></span><= /p> <div> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:"= ;Tahoma","sans-serif"">From:</span></b><span style=3D"font-s= ize:10.0pt;font-family:"Tahoma","sans-serif""> Benson S= chliesser [mailto:bschlies@cisco.com] <br> <b>Sent:</b> Saturday, September 24, 2011 11:25 AM<br> <b>To:</b> Narasimhan Venkataramaiah; Linda Dunbar; Vishwas Manral; Murari = Sridharan<br> <b>Cc:</b> armd@ietf.org<br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00"advertise its external facing hosts' IP addresses to external w= orld?<o:p></o:p></span></p> </div> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><span style=3D"font-s= ize:11.0pt;font-family:"Calibri","sans-serif"">Thanks, = Simha.<br> <br> Leaving aside your comments about support for non-IP payloads, etc, and foc= using strictly on address resolution: Would carrying IP in the overlay inhe= rently localize the address resolution functions (ARP, ND)? Conversely, doe= s carrying L2 frames in the overlay suggest the extension or enlargement of ARP/ND domains? Or would you propo= se an ARP/ND proxy in the MAC overlay?<br> <br> It seems to me that scale of the address resolution function will be affect= ed by the way it’s distributed (or not) throughout the overlay networ= k. Likewise, this may have impact on the forwarding efficiency of traffic (= i.e. when there are multiple inbound/outbound paths attached to the overlay).<br> <br> Cheers,<br> -Benson<br> <br> <br> On 9/24/11 12:36 PM, "Narasimhan Venkataramaiah" <<a href=3D"n= arave@microsoft.com">narave@microsoft.com</a>> wrote:</span><o:p></o:p><= /p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri","sans-serif"">Posting back some offline emails<br>  <br> </span><b><span style=3D"font-size:10.0pt;font-family:"Calibri",&= quot;sans-serif"">From:</span></b><span style=3D"font-size:10.0pt;font= -family:"Calibri","sans-serif""> Narasimhan Venkatarama= iah <br> <b>Sent:</b> Thursday, September 22, 2011 9:49 PM<br> <b>To:</b> 'Vishwas Manral'<br> <b>Cc:</b> Murari Sridharan; Linda Dunbar; <a href=3D"david.black@emc.com">= david.black@emc.com</a>; Benson Schliesser<br> <b>Subject:</b> RE: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif"">In some cases the it may be more efficient to just send IP = 211; but given MAC in GRE is a superset, its simpler to just always do MAC = in GRE from an engineering point of view for various hardware devices that parse the packets to provide added functionality in the path.= They have to deal with one type of packets. We would need MAC in GRE to st= retch L2 to non NV-GRE subnets or to carry non IP payload. <br>  <br> Simha<br>  <br> </span><b><span style=3D"font-size:10.0pt;font-family:"Calibri",&= quot;sans-serif"">From:</span></b><span style=3D"font-size:10.0pt;font= -family:"Calibri","sans-serif""> Vishwas Manral [<a hre= f=3D"mailto:vishwas.ietf@gmail.com">mailto:vishwas.ietf@gmail.com</a>] <br> <b>Sent:</b> Thursday, September 22, 2011 9:43 PM<br> <b>To:</b> Narasimhan Venkataramaiah<br> <b>Cc:</b> Murari Sridharan; Linda Dunbar; <a href=3D"david.black@emc.com">= david.black@emc.com</a>; Benson Schliesser<br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><br> I agree PMIP may not directly work here. In such cases the Layer-2 header h= owever is still of no use.<br>  <br> -Vishwas<br> On Thu, Sep 22, 2011 at 9:07 PM, Narasimhan Venkataramaiah <<a href=3D"n= arave@microsoft.com">narave@microsoft.com</a>> wrote:<br> <span style=3D"font-size:11.0pt;color:#1F497D">Right – with NVGRE L2 = mobility happens automatically – however if you were to achieve it wi= thout NVGRE in a multitenant environment you would need some VLAN reconfigu= ration. NVGRE achieves that with Tenant IDs. <br> PMIP is not multi-tenancy aware in the sense that the proxy won’t be = able to distinguish between multiple tenants using same IP address space in= the virtual network. <br> Simha<br> </span><b><span style=3D"font-size:10.0pt">From:</span></b><span style=3D"f= ont-size:10.0pt"> Vishwas Manral [<a href=3D"mailto:vishwas.ietf@gmail.com"= >mailto:vishwas.ietf@gmail.com</a>] <br> <b>Sent:</b> Thursday, September 22, 2011 8:51 PM<br> <b>To:</b> Narasimhan Venkataramaiah<br> <b>Cc:</b> Murari Sridharan; Linda Dunbar; <a href=3D"david.black@emc.com">= david.black@emc.com</a>; Benson Schliesser <br> <br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><br> Hi Simha,<br> With that aspect there is little difference in the functionality, though L3= -in-L3 can use PMIP and other mobility work already done.<br> Also it begs the question why you even need L2 mobility in this case<br> Moving off the list as Benson feels the replies are not substantial.<br> Thanks,<br> Vishwas<br> On Thu, Sep 22, 2011 at 8:34 PM, Narasimhan Venkataramaiah <<a href=3D"n= arave@microsoft.com">narave@microsoft.com</a>> wrote:<br> <span style=3D"font-size:11.0pt;color:#1F497D">Say you have a virtual subne= t that spans 2 physical subnets. A VM in a virtual subnet moving to another= host within the same physical subnet is L2 mobility and a VM moving to ano= ther host on another in  a different physical subnet is L3 mobility. These 2 cases are the same for NV-GRE in t= erms of encapsulation as the resulting packets are the same. What does IP i= n GRE achieve that MAC in GRE does not? <br> Simha<br> </span><span style=3D"font-size:11.0pt;font-family:"Calibri",&quo= t;sans-serif""><br>  <br> <br> </span><b><span style=3D"font-size:10.0pt;font-family:"Calibri",&= quot;sans-serif"">From:</span></b><span style=3D"font-size:10.0pt;font= -family:"Calibri","sans-serif""> <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a href=3D"mai= lto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] <b>On Behalf Of </b>Linda Dunbar<br> <b>Sent:</b> Saturday, September 24, 2011 6:36 AM<br> <b>To:</b> Vishwas Manral; Murari Sridharan<br> <b>Cc:</b> <a href=3D"armd@ietf.org">armd@ietf.org</a><br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif"">Protocols have been developed for IP mobility between Home Gat= eway and Remote Gateway. The question is if we want similar protocols among= TORs or among vSwitches. Handset mobility is random, but VM migration is planned.  <br>  <br> Linda<br>  <br> <br> </span><b><span style=3D"font-size:10.0pt;font-family:"Calibri",&= quot;sans-serif"">From:</span></b><span style=3D"font-size:10.0pt;font= -family:"Calibri","sans-serif""> <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a href=3D"mai= lto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] <b>On Behalf Of </b>Vishwas Manral<br> <b>Sent:</b> Thursday, September 22, 2011 9:38 PM<br> <b>To:</b> Murari Sridharan<br> <b>Cc:</b> <a href=3D"armd@ietf.org">armd@ietf.org</a><br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Murari think IP mobility. :)<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <<a href=3D"mur= aris@microsoft.com">muraris@microsoft.com</a>> wrote:<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> Do you have a scenario in mind? </span><o:p></o:p></p> <div class=3D"MsoNormal" align=3D"center" style=3D"text-align:center"> <hr size=3D"3" width=3D"100%" align=3D"center"> </div> <p style=3D"margin-bottom:12.0pt"><span style=3D"font-size:11.0pt;font-fami= ly:"Calibri","sans-serif""><br> </span><b><span style=3D"font-size:10.0pt;font-family:"Calibri",&= quot;sans-serif"">From: </span></b><span style=3D"font-size:10.0pt;font-family:"Calibri",= "sans-serif"">Vishwas Manral<br> <b>Sent: </b>9/22/2011 4:55 PM</span> <br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <b><span style=3D"font-size:10.0pt;font-family:"Calibri","sa= ns-serif"">To: </span></b><span style=3D"font-size:10.0pt;font-family:= "Calibri","sans-serif"">Murari Sridharan<br> <b>Cc: </b>Narasimhan Venkataramaiah; Linda Dunbar; <a href=3D"david.black@= emc.com"> david.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a><br> <b>Subject: </b>Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><span style=3D"font-size:11.0pt;font-family:"Calibri",&quo= t;sans-serif""><br> </span>Hi Murari,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Yes that is what I mean.<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Thanks,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Vishwas<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <<a href=3D"mur= aris@microsoft.com">muraris@microsoft.com</a>> wrote:<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><span style=3D"font-size:11.0pt;color:#1F497D">You mean not an Ether= net frame but some IP payload?<br>  <br> </span><b><span style=3D"font-size:10.0pt">From:</span></b><span style=3D"f= ont-size:10.0pt"> Vishwas Manral [<a href=3D"mailto:vishwas.ietf@gmail.com"= >mailto:vishwas.ietf@gmail.com</a>] <br> <b>Sent:</b> Thursday, September 22, 2011 4:49 PM<br> <b>To:</b> Murari Sridharan<br> <b>Cc:</b> Narasimhan Venkataramaiah; Linda Dunbar; <a href=3D"david.black@= emc.com"> david.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a> <br> </span><span style=3D"font-size:11.0pt;font-family:"Calibri",&quo= t;sans-serif""><br> </span><br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Murari,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>What I am saying is the inner header should be allowed to be L3. <br= > <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>From the diagram you have that does not seem to be the case. Am I mi= ssing it totally?<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Thanks,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Vishwas<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <<a href=3D"mur= aris@microsoft.com">muraris@microsoft.com</a>> wrote:<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><span style=3D"font-size:11.0pt;color:#1F497D">Vishwas, Thanks for t= he feedback we will definitely consider adding that. I am not sure what you= mean by doing L3 instead of L2. We allow any arbitrary virtual topology in= cluding L3. <br>  <br> Thanks<br>  <br> </span><b><span style=3D"font-size:10.0pt">From:</span></b><span style=3D"f= ont-size:10.0pt"> Vishwas Manral [<a href=3D"mailto:vishwas.ietf@gmail.com"= >mailto:vishwas.ietf@gmail.com</a>] <br> <b>Sent:</b> Thursday, September 22, 2011 4:19 PM <br> </span><span style=3D"font-size:11.0pt;font-family:"Calibri",&quo= t;sans-serif""><br> </span><span style=3D"font-size:10.0pt"><br> <b>To:</b> Narasimhan Venkataramaiah<br> <b>Cc:</b> Linda Dunbar; Murari Sridharan; <a href=3D"david.black@emc.com">= david.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a><br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><span style=3D"font-size:11.0pt;font-family:"Calibri",&quo= t;sans-serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Hi Simha,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>I see this as the only difference between VXLAN and the NVGRE soluti= on (besides ofcourse that TNI needs to be parsed in the intermediate device= for hashing and using lesser number of bytes).<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>I would think you should add it to your draft immediately. With tunn= eling you consolidate the addresses visible to the core and by providing a = hash mechanism, you are providing some level of randomness.<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE ins= tead of L2 alone. I guess it would be the same comment for the VXLAN propos= al too.<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Thanks,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Vishwas<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <<a hr= ef=3D"narave@microsoft.com">narave@microsoft.com</a>> wrote:<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>The draft mentions exactly this as one use of the reserved 8 bits in= Section 4. An NVGRE endpoint could use the 8 bits to further distribute fl= ows belonging to a particular TNI and the switches use all 32 bits to get e= ntropy. One step further would be for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?<br> <span style=3D"font-size:10.0pt;font-family:"Calibri","sans-= serif""><br> One<br>    such example could be to use the upper 8 bits of the Key = field to<br>    add flow based entropy and tag all the packets from a flo= w with an entropy label.<br>  <br> </span><span style=3D"font-size:11.0pt;color:#1F497D">Simha<br>  <br> </span><b><span style=3D"font-size:10.0pt">From:</span></b><span style=3D"f= ont-size:10.0pt"> Vishwas Manral [<a href=3D"mailto:vishwas.ietf@gmail.com"= >mailto:vishwas.ietf@gmail.com</a>] <br> <b>Sent:</b> Thursday, September 22, 2011 4:04 PM<br> <b>To:</b> Narasimhan Venkataramaiah<br> <b>Cc:</b> Linda Dunbar; Murari Sridharan; <a href=3D"david.black@emc.com">= david.black@emc.com</a>; <a href=3D"armd@ietf.org">armd@ietf.org</a><br> <b>Subject:</b> Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?<br> </span><span style=3D"font-size:11.0pt;font-family:"Calibri",&quo= t;sans-serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Hi Simha,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>The main (Standards Track) change in your draft is the addition of T= NI.<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>A question I have is a TNI identifies a particular tenant and all fl= ows from/to a tenant will be hashed to the same path (even with the changes= in switches to do hashing to use TNI).<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Why do you not use the last 8 bits which you have kept as reserved f= or providing the randomization for hashing flows between same to/from on di= fferent paths?<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Thanks,<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>Vishwas<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span>On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <<a h= ref=3D"narave@microsoft.com">narave@microsoft.com</a>> wrote:<br> The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the implication on virtualization as= its no different than getting the traffic routed to the enterprise in the = first case.<br> <br> Simha<br> <br> ________________________________________<br> From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a href= =3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a>] on behalf of Linda Du= nbar [<a href=3D"linda.dunbar@huawei.com">linda.dunbar@huawei.com</a>]<br> Sent: Sunday, September 18, 2011 7:06 AM<br> To: Murari Sridharan; <a href=3D"david.black@emc.com">david.black@emc.com</= a>; <a href=3D"armd@ietf.org"> armd@ietf.org</a><br> Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?<br> <span style=3D"font-size:11.0pt;font-family:"Calibri","sans-= serif""><br> </span><br> Hi Murari,<br> <br> Thank you very much for sharing the presentation.<br> <br> One question:<br> <br> For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.<br> <br> When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?<br> <br> Thanks, Linda<br> <br> > -----Original Message-----<br> > From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a = href=3D"mailto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] On = Behalf Of<br> > Murari Sridharan<br> > Sent: Saturday, September 17, 2011 3:02 PM<br> > To: <a href=3D"david.black@emc.com">david.black@emc.com</a>; <a href= =3D"armd@ietf.org"> armd@ietf.org</a><br> > Subject: Re: [armd] soliciting typical network designs for ARMD<br> ><br> > FYI, here is a talk that I gave last week in relation to the nvgre<br> > draft below.<br> > <a href=3D"http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T">h= ttp://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T</a><br> ><br> > Thanks<br> > Murari<br> ><br> > -----Original Message-----<br> > From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a> [<a = href=3D"mailto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>] On = Behalf Of<br> > <a href=3D"david.black@emc.com">david.black@emc.com</a><br> > Sent: Friday, September 16, 2011 6:14 AM<br> > To: <a href=3D"armd@ietf.org">armd@ietf.org</a><br> > Subject: Re: [armd] soliciting typical network designs for ARMD<br> ><br> > And two more drafts on this topic:<br> ><br> > <a href=3D"http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00= .txt">http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt</a><b= r> > <a href=3D"http://www.ietf.org/id/draft-sridharan-virtualization-nvgre= -00.txt">http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt= </a><br> ><br> > The edge switches could be the software switches in hypervisors.<br> ><br> > Thanks,<br> > --David<br> ><br> ><br> > > -----Original Message-----<br> > > From: <a href=3D"armd-bounces@ietf.org">armd-bounces@ietf.org</a>= [<a href=3D"mailto:armd-bounces@ietf.org">mailto:armd-bounces@ietf.org</a>= ] On Behalf<br> > > Of Warren Kumari<br> > > Sent: Wednesday, August 31, 2011 3:16 PM<br> > > To: Vishwas Manral<br> > > Cc: <a href=3D"armd@ietf.org">armd@ietf.org</a><br> > > Subject: Re: [armd] soliciting typical network designs for ARMD<b= r> > ><br> > ><br> > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:<br> > ><br> > > > Hi Linda/ Anoop,<br> > > ><br> > > > Here is the example of the design I was talking about, as de= fined<br> > by google.<br> > ><br> > > Just a clarification -- s/as defined by google/as described by<br= > > someone<br> > > who happens to work for google/<br> > ><br> > > W<br> > ><br> > > > <a href=3D"http://www.ietf.org/id/draft-wkumari-dcops-l3-vmm= obility-00.txt">http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00= .txt</a><br> > > ><br> > > > Thanks,<br> > > > Vishwas<br> > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani<br> > <<a href=3D"anoop@alumni.duke.edu">anoop@alumni.duke.edu</a>> wr= ote:<br> > > ><br> > > > >>>><br> > > > (though I think if there was a standard way to map Multicast= MAC to<br> > > > Multicast IP, they could<br> > > probably use such a standard mechanisms).<br> > > > >>>><br> > > ><br> > > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,<br> > > > because of pruning requirements the number of groups would b= e very<br> > > > large.  The average data center switch probably won't h= andle that<br> > > > many groups.<br> > > ><br> > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral<br> > <<a href=3D"vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>> = wrote:<br> > > > Hi Anoop,<br> > > ><br> > > > From what I know they do not use Multicast GRE (I hear the e= xtra 4<br> > > > bytes in the GRE header is a<br> > > proprietery extension).<br> > > ><br> > > > I think a directory based mechanism is what is used (though = I think<br> > > > if there was a standard way to<br> > > map Multicast MAC to Multicast IP, they could probably use such a= <br> > standard mechanisms).<br> > > ><br> > > > Thanks,<br> > > > Vishwas<br> > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani<br> > <<a href=3D"anoop@alumni.duke.edu">anoop@alumni.duke.edu</a>> wr= ote:<br> > > > Hi Vishwas,<br> > > ><br> > > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based<br> > > > lookups and not worry about multicast applications for now?<= br> > > ><br> > > > Anoop<br> > > ><br> > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral<br> > <<a href=3D"vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>> = wrote:<br> > > > Hi Linda,<br> > > ><br> > > > The data packets can be tunnelled at the ToR over say a GRE = packet<br> > > > and the core is a Layer-3 core<br> > > (except for the downstream ports). So we could have encapsulation= /<br> > > decapsulation of L2 over GRE at the ToR.<br> > > ><br> > > > The very same thing can be done at the hypervisor layer too,= in<br> > > > which case the entire DC network<br> > > would look like a Layer-3 flat network including the ToR to serve= r<br> > > link and the hypervisor would do the tunneling.<br> > > ><br> > > > I am not sure if you got the points above or not. I know clo= ud OS<br> > > > companies that provide the service<br> > > and have big announced customers.<br> > > ><br> > > > Thanks,<br> > > > Vishwas<br> > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <<a href=3D= "dunbar.ll@gmail.com">dunbar.ll@gmail.com</a>><br> > wrote:<br> > > > Vishwas,<br> > > ><br> > > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)<br> > > running Layer 2 and ToR uplinks ports run IP Layer 3.<br> > > ><br> > > > Have you seen data center networks with ToR switches downstr= eam<br> > > > ports (i.e. facing servers) enabling<br> > > IP routing, even though the physical links are Ethernet?<br> > > > If yes, we should definitely include it in the ARMD draft.<b= r> > > ><br> > > > Thanks,<br> > > > Linda<br> > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral<br> > <<a href=3D"vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>> = wrote:<br> > > > Hi Linda,<br> > > > I am unsure what you mean by this, but:<br> > > >   * layer 3 all the way to TOR (Top of Rack switch= es), We can also<br> > > > have a heirarchical network, with the core totally Layer-3 (= and<br> > > > having seperate<br> > > routing), from the hosts still in a large Layer-3 subnet. Another= <br> > > aspect could be to have a totally<br> > > Layer-3 network.<br> > > ><br> > > > The difference between them is the link between the servers = and the<br> > ToR.<br> > > ><br> > > > Thanks,<br> > > > Vishwas<br> > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <<a href=3D= "dunbar.ll@gmail.com">dunbar.ll@gmail.com</a>><br> > wrote:<br> > > > During the 81st IETF ARMD WG discussion, it was suggested th= at it<br> > is<br> > > > necessary to document typical<br> > > data center network designs so that address resolution scaling is= sues<br> > > can be properly described. Many data center operators have expres= sed<br> > that they can't openly reveal their detailed network designs.<br> > > Therefore, we only want to document anonymous designs without too= <br> > much<br> > > detail. During the journey of establishing ARMD, we have come acr= oss<br> > the following typical data center network designs:<br> > > >   * layer 3 all the way to TOR (Top of Rack switch= es),<br> > > >   * large layer 2 with hundreds (or thousands) of = ToRs being<br> > > > interconnected by Layer 2. This<br> > > design will have thousands of hosts under the L2/L3 boundary rout= er<br> > > (s)<br> > > >   * CLOS design  with thousands of switches. = This design will have<br> > > > thousands of hosts under the<br> > > L2/L3 boundary router(s)<br> > > > We have heard that each of the designs above has its own pro= blems.<br> > > > ARMD problem statements might<br> > > need to document DC problems under each typical design.<br> > > > Please send feedback to us (either to the armd email list &n= bsp;or to<br> > the<br> > > > ARMD chair Benson & Linda) to<br> > > indicate if we have missed any typical Data Center network design= s.<br> > > ><br> > > > Your contribution can greatly accelerate the progress of ARM= D WG.<br> > > ><br> > > > Thank you very much.<br> > > ><br> > > > Linda & Benson<br> > > ><br>  <br>  <br>  <br>  <span style=3D"font-size:11.0pt;font-family:"Calibri","= ;sans-serif""><o:p></o:p></span></p> <div class=3D"MsoNormal" align=3D"center" style=3D"text-align:center"><span= style=3D"font-size:11.0pt;font-family:"Calibri","sans-serif= ""> <hr size=3D"3" width=3D"95%" align=3D"center"> </span></div> <p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:Consolas= ">_______________________________________________<br> armd mailing list<br> <a href=3D"armd@ietf.org">armd@ietf.org</a><br> <a href=3D"https://www.ietf.org/mailman/listinfo/armd">https://www.ietf.org= /mailman/listinfo/armd</a></span><o:p></o:p></p> </div> </body> </html> --_000_65755BEBE02F7C41BD4F137AED91DA5E2FD3B544TK5EX14MBXW601w_-- From ghanwani@gmail.com Mon Sep 26 12:07:55 2011 Return-Path: <ghanwani@gmail.com> X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 30C2721F8CFE for <armd@ietfa.amsl.com>; Mon, 26 Sep 2011 12:07:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.422 X-Spam-Level: X-Spam-Status: No, score=-3.422 tagged_above=-999 required=5 tests=[AWL=-0.445, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MFe9-p0eJSQK for <armd@ietfa.amsl.com>; Mon, 26 Sep 2011 12:07:54 -0700 (PDT) Received: from mail-qy0-f179.google.com (mail-qy0-f179.google.com [209.85.216.179]) by ietfa.amsl.com (Postfix) with ESMTP id 8B9D921F8CF6 for <armd@ietf.org>; Mon, 26 Sep 2011 12:07:54 -0700 (PDT) Received: by qyk33 with SMTP id 33so6778174qyk.10 for <armd@ietf.org>; Mon, 26 Sep 2011 12:10:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=1PkdI2DLDayaS3lJn86E+WQo/QIWfJekNXuY/hW2/bQ=; b=j/tutqdSaCK22H5r4UHiVsVy/cTOAiAlwm8oM4dYgQfeb/TrvPBkM0PfQz2KO4Dkta prNuCOpcwdnuBO3USvDcMQHr7BtObCj+atTqVSQuBF3xjMGDDipI22ri0njaSBrIHWGa QoyjAhv4LJbNmGdpvN2JV74AyfujIyHBdbp/4= MIME-Version: 1.0 Received: by 10.229.2.167 with SMTP id 39mr4976070qcj.173.1317064237680; Mon, 26 Sep 2011 12:10:37 -0700 (PDT) Sender: ghanwani@gmail.com Received: by 10.229.219.197 with HTTP; Mon, 26 Sep 2011 12:10:37 -0700 (PDT) In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> References: <Acx1aIsSyTVtVcAKQ4yYdknwqYNy9g==> <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> Date: Mon, 26 Sep 2011 12:10:37 -0700 X-Google-Sender-Auth: V6MAGBx3jAyudUq816vvLdLXtn0 Message-ID: <CA+-tSzyecPTu4zhrdfFqWRdfr75gJMprJNJC_j+2XhBq4zhpZA@mail.gmail.com> From: Anoop Ghanwani <anoop@alumni.duke.edu> To: Linda Dunbar <linda.dunbar@huawei.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: "armd@ietf.org" <armd@ietf.org> Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> List-Unsubscribe: <https://www.ietf.org/mailman/options/armd>, <mailto:armd-request@ietf.org?subject=unsubscribe> List-Archive: <http://www.ietf.org/mail-archive/web/armd> List-Post: <mailto:armd@ietf.org> List-Help: <mailto:armd-request@ietf.org?subject=help> List-Subscribe: <https://www.ietf.org/mailman/listinfo/armd>, <mailto:armd-request@ietf.org?subject=subscribe> X-List-Received-Date: Mon, 26 Sep 2011 19:07:55 -0000 On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com> wr= ote: > > > There are many drafts recently on various Overlay Network schemes for dat= a > centers with large number of tenants/hosts. When Overlay Network is deplo= yed > in a data center, local switches/routers within the data center network > don=92t see the hosts addresses anymore except the nodes which perform th= e > network address encapsulation/decapsulation. > ... > For each host (which runs on a VM) which need to communicate with externa= l > world, it frequently sends ARP/ND requests to its default Gateway router. > For an application with dual stacks on, it sends both ARP & ND requests t= o > its default gateway router. I think the issue of whether or not an overlay network is in use is orthogo= nal to the issue of ARP/ND traffic that ARMD is trying to address. If we have an overlay network built over a data center core, we still run into the problem where ARP/ND traffic from a VM needs to be broadcast/multicast over the core (even though it uses the overlay). So unless some other mechanism is used to mitigate the ARP/ND traffic, we would be no better than a regular L2 fabric that spans the entire data center (at least from the standpoint of the volume of ARP/ND traffic traversing the core). From vishwas.ietf@gmail.com Mon Sep 26 13:28:39 2011 Return-Path: <vishwas.ietf@gmail.com> X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1EDD21F0CE9 for <armd@ietfa.amsl.com>; Mon, 26 Sep 2011 13:28:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.225 X-Spam-Level: X-Spam-Status: No, score=-3.225 tagged_above=-999 required=5 tests=[AWL=0.373, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FOF4etbGigWE for <armd@ietfa.amsl.com>; Mon, 26 Sep 2011 13:28:38 -0700 (PDT) Received: from mail-qy0-f179.google.com (mail-qy0-f179.google.com [209.85.216.179]) by ietfa.amsl.com (Postfix) with ESMTP id 4837A1F0CEA for <armd@ietf.org>; Mon, 26 Sep 2011 13:28:38 -0700 (PDT) Received: by qyk33 with SMTP id 33so6851090qyk.10 for <armd@ietf.org>; Mon, 26 Sep 2011 13:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Zf9BTNWZBvyhUIogjPXABbkl8Rvo/R7FcJff9oD85d0=; b=JE50wJh5504jyW8vNn/qcRxg6A8VSDjUCQm7AKeHN5ijspk16kGk5YCLOu0VRe/iy9 ezoA79XcdtLPhJ9n8/SNAKaSZjT5Q/VaxFJvFf/AckeqbTLMQssLHQlZcPwAWNjX0uDd t3vJ5bBM5M9l7JDF3zVpzEkOLHIaB9/H9bSe0= MIME-Version: 1.0 Received: by 10.229.196.233 with SMTP id eh41mr4476822qcb.187.1317069079465; Mon, 26 Sep 2011 13:31:19 -0700 (PDT) Received: by 10.229.91.131 with HTTP; Mon, 26 Sep 2011 13:31:19 -0700 (PDT) In-Reply-To: <CA+-tSzyecPTu4zhrdfFqWRdfr75gJMprJNJC_j+2XhBq4zhpZA@mail.gmail.com> References: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> <CA+-tSzyecPTu4zhrdfFqWRdfr75gJMprJNJC_j+2XhBq4zhpZA@mail.gmail.com> Date: Mon, 26 Sep 2011 13:31:19 -0700 Message-ID: <CAOyVPHQ8VvA_XUXJej9+KBv5tv9Vdd4hhr3VC8Bo4e_fy_M8jQ@mail.gmail.com> From: Vishwas Manral <vishwas.ietf@gmail.com> To: Anoop Ghanwani <anoop@alumni.duke.edu> Content-Type: multipart/alternative; boundary=0016369cffb4a04d3304adde0a6e Cc: "armd@ietf.org" <armd@ietf.org> Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> List-Unsubscribe: <https://www.ietf.org/mailman/options/armd>, <mailto:armd-request@ietf.org?subject=unsubscribe> List-Archive: <http://www.ietf.org/mail-archive/web/armd> List-Post: <mailto:armd@ietf.org> List-Help: <mailto:armd-request@ietf.org?subject=help> List-Subscribe: <https://www.ietf.org/mailman/listinfo/armd>, <mailto:armd-request@ietf.org?subject=subscribe> X-List-Received-Date: Mon, 26 Sep 2011 20:28:39 -0000 --0016369cffb4a04d3304adde0a6e Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Totally agree with Anoop here!!! -Vishwas On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu>wro= te: > On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com> > wrote: > > > > > > There are many drafts recently on various Overlay Network schemes for > data > > centers with large number of tenants/hosts. When Overlay Network is > deployed > > in a data center, local switches/routers within the data center network > > don=92t see the hosts addresses anymore except the nodes which perform = the > > network address encapsulation/decapsulation. > > > ... > > For each host (which runs on a VM) which need to communicate with > external > > world, it frequently sends ARP/ND requests to its default Gateway route= r. > > For an application with dual stacks on, it sends both ARP & ND requests > to > > its default gateway router. > > I think the issue of whether or not an overlay network is in use is > orthogonal > to the issue of ARP/ND traffic that ARMD is trying to address. > If we have an overlay network built over a data center core, > we still run into the problem where ARP/ND traffic from a VM needs to > be broadcast/multicast over the core (even though it uses the > overlay). So unless > some other mechanism is used to mitigate the ARP/ND traffic, we would > be no better than a regular L2 fabric that spans the entire data center (= at > least from the standpoint of the volume of ARP/ND traffic traversing > the core). > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > --0016369cffb4a04d3304adde0a6e Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable <div>Totally agree with Anoop here!!!</div> <div>=A0</div> <div>-Vishwas</div> <div>=A0</div> <div class=3D"gmail_quote">On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani= <span dir=3D"ltr"><<a href=3D"mailto:anoop@alumni.duke.edu">anoop@alumn= i.duke.edu</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"PADDING-LEFT: 1ex; MARGIN: 0px 0= px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"> <div class=3D"im">On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <<a hre= f=3D"mailto:linda.dunbar@huawei.com">linda.dunbar@huawei.com</a>> wrote:= <br>><br>><br></div> <div class=3D"im">> There are many drafts recently on various Overlay Ne= twork schemes for data<br>> centers with large number of tenants/hosts. = When Overlay Network is deployed<br>> in a data center, local switches/r= outers within the data center network<br> > don=92t see the hosts addresses anymore except the nodes which perform= the<br>> network address encapsulation/decapsulation.<br>><br></div>= ...<br> <div class=3D"im">> For each host (which runs on a VM) which need to com= municate with external<br>> world, it frequently sends ARP/ND requests t= o its default Gateway router.<br>> For an application with dual stacks o= n, it sends both ARP & ND requests to<br> > its default gateway router.<br><br></div>I think the issue of whether = or not an overlay network is in use is orthogonal<br>to the issue of ARP/ND= traffic that ARMD is trying to address.<br>If we have an overlay network b= uilt over a data center core,<br> we still run into the problem where ARP/ND traffic from a VM needs to<br>be= broadcast/multicast over the core (even though it uses the<br>overlay). = =A0So unless<br>some other mechanism is used to mitigate the ARP/ND traffic= , we would<br> be no better than a regular L2 fabric that spans the entire data center (at= <br>least from the standpoint of the volume of ARP/ND traffic traversing<br= >the core).<br> <div> <div></div> <div class=3D"h5">_______________________________________________<br>armd m= ailing list<br><a href=3D"mailto:armd@ietf.org">armd@ietf.org</a><br><a hre= f=3D"https://www.ietf.org/mailman/listinfo/armd" target=3D"_blank">https://= www.ietf.org/mailman/listinfo/armd</a><br> </div></div></blockquote></div><br> --0016369cffb4a04d3304adde0a6e-- From gaberger@cisco.com Tue Sep 27 07:33:26 2011 Return-Path: <gaberger@cisco.com> X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2614021F8BAD for <armd@ietfa.amsl.com>; Tue, 27 Sep 2011 07:33:26 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.142 X-Spam-Level: X-Spam-Status: No, score=-1.142 tagged_above=-999 required=5 tests=[AWL=0.060, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4dhFQwCQINuZ for <armd@ietfa.amsl.com>; Tue, 27 Sep 2011 07:33:24 -0700 (PDT) Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by ietfa.amsl.com (Postfix) with ESMTP id 3C72121F8B3F for <armd@ietf.org>; Tue, 27 Sep 2011 07:33:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=6372; q=dns/txt; s=iport; t=1317134170; x=1318343770; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=51Ze8d5mDo9dFeHqXeOO/AK+XDopHX/ZmKxJYEYOEO8=; b=WGxLFkDhJ1nonjMT0vfg0mjoLNU4soAn2myW9l6Em83KUlajQP40tqDs 4I28if1NhDPz7dssypDEjtA6QmicLfeEGgeGErQmnIytqlTREJQaTd5Bi TC3ydIpkeL6WGUQbfA9ZOgL5rpSPELccCpu+shXkvjqB3S4EkzCDBUTdz o=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgEFAHXegU6tJXG+/2dsb2JhbABBDoI/pDxzeIFTAQEBAwEBAQEPASoxCwUOCBEDAQIBTwYoCAYBDQUih1YGm00BnikDhwsEh0KMEIUihHaGZVM X-IronPort-AV: E=Sophos;i="4.68,449,1312156800"; d="scan'208,217";a="24334575" Received: from rcdn-core2-3.cisco.com ([173.37.113.190]) by rcdn-iport-7.cisco.com with ESMTP; 27 Sep 2011 14:36:09 +0000 Received: from [10.82.240.5] (rtp-vpn2-5.cisco.com [10.82.240.5]) by rcdn-core2-3.cisco.com (8.14.3/8.14.3) with ESMTP id p8REa7Yk030678; Tue, 27 Sep 2011 14:36:08 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Tue, 27 Sep 2011 10:36:05 -0400 From: Gary Berger <gaberger@cisco.com> To: Vishwas Manral <vishwas.ietf@gmail.com>, Anoop Ghanwani <anoop@alumni.duke.edu> Message-ID: <CAA75790.2B45B%gaberger@cisco.com> Thread-Topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts In-Reply-To: <CAOyVPHQ8VvA_XUXJej9+KBv5tv9Vdd4hhr3VC8Bo4e_fy_M8jQ@mail.gmail.com> Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3399964568_19844571" Cc: "armd@ietf.org" <armd@ietf.org> Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> List-Unsubscribe: <https://www.ietf.org/mailman/options/armd>, <mailto:armd-request@ietf.org?subject=unsubscribe> List-Archive: <http://www.ietf.org/mail-archive/web/armd> List-Post: <mailto:armd@ietf.org> List-Help: <mailto:armd-request@ietf.org?subject=help> List-Subscribe: <https://www.ietf.org/mailman/listinfo/armd>, <mailto:armd-request@ietf.org?subject=subscribe> X-List-Received-Date: Tue, 27 Sep 2011 14:33:26 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3399964568_19844571 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable +1 From: Vishwas Manral <vishwas.ietf@gmail.com> Date: Mon, 26 Sep 2011 13:31:19 -0700 To: Anoop Ghanwani <anoop@alumni.duke.edu> Cc: "armd@ietf.org" <armd@ietf.org> Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Totally agree with Anoop here!!! =20 -Vishwas =20 On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu> wrote: > On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com> > wrote: >> > >> > >> > There are many drafts recently on various Overlay Network schemes for = data >> > centers with large number of tenants/hosts. When Overlay Network is >> deployed >> > in a data center, local switches/routers within the data center networ= k >> > don=B9t see the hosts addresses anymore except the nodes which perform t= he >> > network address encapsulation/decapsulation. >> > > ... >> > For each host (which runs on a VM) which need to communicate with exte= rnal >> > world, it frequently sends ARP/ND requests to its default Gateway rout= er. >> > For an application with dual stacks on, it sends both ARP & ND request= s to >> > its default gateway router. >=20 > I think the issue of whether or not an overlay network is in use is ortho= gonal > to the issue of ARP/ND traffic that ARMD is trying to address. > If we have an overlay network built over a data center core, > we still run into the problem where ARP/ND traffic from a VM needs to > be broadcast/multicast over the core (even though it uses the > overlay). So unless > some other mechanism is used to mitigate the ARP/ND traffic, we would > be no better than a regular L2 fabric that spans the entire data center (= at > least from the standpoint of the volume of ARP/ND traffic traversing > the core). > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --B_3399964568_19844571 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable <html><head></head><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: s= pace; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size:= 14px; font-family: Calibri, sans-serif; "><div><div><div>+1</div></div></di= v><span id=3D"OLK_SRC_BODY_SECTION"><div style=3D"font-family:Calibri; font-size= :11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT= : medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; B= ORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"><= span style=3D"font-weight:bold">From: </span> Vishwas Manral <<a href=3D"mail= to:vishwas.ietf@gmail.com">vishwas.ietf@gmail.com</a>><br><span style=3D"fo= nt-weight:bold">Date: </span> Mon, 26 Sep 2011 13:31:19 -0700<br><span style= =3D"font-weight:bold">To: </span> Anoop Ghanwani <<a href=3D"mailto:anoop@alu= mni.duke.edu">anoop@alumni.duke.edu</a>><br><span style=3D"font-weight:bold= ">Cc: </span> "<a href=3D"mailto:armd@ietf.org">armd@ietf.org</a>" <<a href= =3D"mailto:armd@ietf.org">armd@ietf.org</a>><br><span style=3D"font-weight:bo= ld">Subject: </span> Re: [armd] Bottleneck of Overlay network in a data cent= er with large number of tenants/hosts<br></div><div><br></div><div>Totally a= gree with Anoop here!!!</div><div> </div><div>-Vishwas</div><div> = </div><div class=3D"gmail_quote">On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanw= ani <span dir=3D"ltr"><<a href=3D"mailto:anoop@alumni.duke.edu">anoop@alumni.= duke.edu</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"PAD= DING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><div= class=3D"im">On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <<a href=3D"mail= to:linda.dunbar@huawei.com">linda.dunbar@huawei.com</a>> wrote:<br>><b= r>><br></div><div class=3D"im">> There are many drafts recently on vario= us Overlay Network schemes for data<br>> centers with large number of ten= ants/hosts. When Overlay Network is deployed<br>> in a data center, local= switches/routers within the data center network<br> > don’t see the hosts addresses anymore except the nodes which per= form the<br>> network address encapsulation/decapsulation.<br>><br></d= iv>...<br><div class=3D"im">> For each host (which runs on a VM) which need= to communicate with external<br>> world, it frequently sends ARP/ND requ= ests to its default Gateway router.<br>> For an application with dual sta= cks on, it sends both ARP & ND requests to<br> > its default gateway router.<br><br></div>I think the issue of whether = or not an overlay network is in use is orthogonal<br>to the issue of ARP/ND = traffic that ARMD is trying to address.<br>If we have an overlay network bui= lt over a data center core,<br> we still run into the problem where ARP/ND traffic from a VM needs to<br>be= broadcast/multicast over the core (even though it uses the<br>overlay). &nb= sp;So unless<br>some other mechanism is used to mitigate the ARP/ND traffic,= we would<br> be no better than a regular L2 fabric that spans the entire data center (at= <br>least from the standpoint of the volume of ARP/ND traffic traversing<br>= the core).<br><div><div></div><div class=3D"h5">______________________________= _________________<br>armd mailing list<br><a href=3D"mailto:armd@ietf.org">arm= d@ietf.org</a><br><a href=3D"https://www.ietf.org/mailman/listinfo/armd" targe= t=3D"_blank">https://www.ietf.org/mailman/listinfo/armd</a><br></div></div></b= lockquote></div><br> _______________________________________________ armd mailing list <a href=3D"mailto:armd@ietf.org">armd@ietf.org</a> <a href=3D"https://www.ietf.org/mailman/listinfo/armd">https://www.ietf.org/m= ailman/listinfo/armd</a> </span></body></html> --B_3399964568_19844571-- From omgponies3145@gmail.com Tue Sep 27 09:46:54 2011 Return-Path: <omgponies3145@gmail.com> X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 928CD21F8B4A for <armd@ietfa.amsl.com>; Tue, 27 Sep 2011 09:46:54 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aXKplAf6mybG for <armd@ietfa.amsl.com>; Tue, 27 Sep 2011 09:46:53 -0700 (PDT) Received: from mail-vx0-f172.google.com (mail-vx0-f172.google.com [209.85.220.172]) by ietfa.amsl.com (Postfix) with ESMTP id A2CF021F8B2D for <armd@ietf.org>; Tue, 27 Sep 2011 09:46:53 -0700 (PDT) Received: by vcbfo11 with SMTP id fo11so5485476vcb.31 for <armd@ietf.org>; Tue, 27 Sep 2011 09:49:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=04Zttnm64MOUyFKnms/o5QrkLkvuR9434DnddyxaZPc=; b=WkUk3tJnOXvthvz4g9voZk/BefDiGm3FUWxmJQtZHV7J2qoUH28Tqy/sd9EkNZB7q4 d2g+4LFXCvzlrAFU/espSThuCqUhwDEkuWXBW8RKC60fpsuHUAXyoA5gLaKa8y5QEhoW p5wqrD3ZXaSq5Ih2HRTCcFX3bVNVgU3Mlk0A4= MIME-Version: 1.0 Received: by 10.52.72.50 with SMTP id a18mr7521201vdv.426.1317142179427; Tue, 27 Sep 2011 09:49:39 -0700 (PDT) Received: by 10.220.189.204 with HTTP; Tue, 27 Sep 2011 09:49:39 -0700 (PDT) In-Reply-To: <CAA75790.2B45B%gaberger@cisco.com> References: <CAOyVPHQ8VvA_XUXJej9+KBv5tv9Vdd4hhr3VC8Bo4e_fy_M8jQ@mail.gmail.com> <CAA75790.2B45B%gaberger@cisco.com> Date: Tue, 27 Sep 2011 09:49:39 -0700 Message-ID: <CAJW6G52_uu9_M-oXCazTotG+VTjuxOUv_=-eN0ntMQE0aSOOiQ@mail.gmail.com> From: John Menerick <omgponies3145@gmail.com> To: Gary Berger <gaberger@cisco.com> Content-Type: multipart/alternative; boundary=20cf3071ca30b9364504adef0fab Cc: "armd@ietf.org" <armd@ietf.org> Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> List-Unsubscribe: <https://www.ietf.org/mailman/options/armd>, <mailto:armd-request@ietf.org?subject=unsubscribe> List-Archive: <http://www.ietf.org/mail-archive/web/armd> List-Post: <mailto:armd@ietf.org> List-Help: <mailto:armd-request@ietf.org?subject=help> List-Subscribe: <https://www.ietf.org/mailman/listinfo/armd>, <mailto:armd-request@ietf.org?subject=subscribe> X-List-Received-Date: Tue, 27 Sep 2011 16:46:54 -0000 --20cf3071ca30b9364504adef0fab Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Agreed! John Menerick On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger <gaberger@cisco.com> wrote: > +1 > From: Vishwas Manral <vishwas.ietf@gmail.com> > Date: Mon, 26 Sep 2011 13:31:19 -0700 > To: Anoop Ghanwani <anoop@alumni.duke.edu> > Cc: "armd@ietf.org" <armd@ietf.org> > Subject: Re: [armd] Bottleneck of Overlay network in a data center with > large number of tenants/hosts > > Totally agree with Anoop here!!! > > -Vishwas > > On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu>w= rote: > >> On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com> >> wrote: >> > >> > >> > There are many drafts recently on various Overlay Network schemes for >> data >> > centers with large number of tenants/hosts. When Overlay Network is >> deployed >> > in a data center, local switches/routers within the data center networ= k >> > don=92t see the hosts addresses anymore except the nodes which perform= the >> > network address encapsulation/decapsulation. >> > >> ... >> > For each host (which runs on a VM) which need to communicate with >> external >> > world, it frequently sends ARP/ND requests to its default Gateway >> router. >> > For an application with dual stacks on, it sends both ARP & ND request= s >> to >> > its default gateway router. >> >> I think the issue of whether or not an overlay network is in use is >> orthogonal >> to the issue of ARP/ND traffic that ARMD is trying to address. >> If we have an overlay network built over a data center core, >> we still run into the problem where ARP/ND traffic from a VM needs to >> be broadcast/multicast over the core (even though it uses the >> overlay). So unless >> some other mechanism is used to mitigate the ARP/ND traffic, we would >> be no better than a regular L2 fabric that spans the entire data center >> (at >> least from the standpoint of the volume of ARP/ND traffic traversing >> the core). >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> > > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --20cf3071ca30b9364504adef0fab Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Agreed!<div><br></div><div><br></div><div>John Menerick<br><br><div class= =3D"gmail_quote">On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger <span dir=3D"= ltr"><<a href=3D"mailto:gaberger@cisco.com">gaberger@cisco.com</a>></= span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex;"><div style=3D"word-wrap:break-word;color:rg= b(0, 0, 0);font-size:14px;font-family:Calibri, sans-serif"><div><div><div>+= 1</div> </div></div><span><div style=3D"font-family:Calibri;font-size:11pt;text-ali= gn:left;color:black;border-bottom:medium none;border-left:medium none;paddi= ng-bottom:0in;padding-left:0in;padding-right:0in;border-top:#b5c4df 1pt sol= id;border-right:medium none;padding-top:3pt"> <span style=3D"font-weight:bold">From: </span> Vishwas Manral <<a href= =3D"mailto:vishwas.ietf@gmail.com" target=3D"_blank">vishwas.ietf@gmail.com= </a>><br><span style=3D"font-weight:bold">Date: </span> Mon, 26 Sep 2011= 13:31:19 -0700<br> <span style=3D"font-weight:bold">To: </span> Anoop Ghanwani <<a href=3D"= mailto:anoop@alumni.duke.edu" target=3D"_blank">anoop@alumni.duke.edu</a>&g= t;<br><span style=3D"font-weight:bold">Cc: </span> "<a href=3D"mailto:= armd@ietf.org" target=3D"_blank">armd@ietf.org</a>" <<a href=3D"mai= lto:armd@ietf.org" target=3D"_blank">armd@ietf.org</a>><br> <span style=3D"font-weight:bold">Subject: </span> Re: [armd] Bottleneck of = Overlay network in a data center with large number of tenants/hosts<br></di= v><div><div></div><div class=3D"h5"><div><br></div><div>Totally agree with = Anoop here!!!</div> <div>=A0</div><div>-Vishwas</div><div>=A0</div><div class=3D"gmail_quote">O= n Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <span dir=3D"ltr"><<a hr= ef=3D"mailto:anoop@alumni.duke.edu" target=3D"_blank">anoop@alumni.duke.edu= </a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"padding-left:1ex;margin:0px 0px = 0px 0.8ex;border-left:#ccc 1px solid"><div>On Sat, Sep 17, 2011 at 11:35 AM= , Linda Dunbar <<a href=3D"mailto:linda.dunbar@huawei.com" target=3D"_bl= ank">linda.dunbar@huawei.com</a>> wrote:<br> ><br>><br></div><div>> There are many drafts recently on various O= verlay Network schemes for data<br>> centers with large number of tenant= s/hosts. When Overlay Network is deployed<br>> in a data center, local s= witches/routers within the data center network<br> > don=92t see the hosts addresses anymore except the nodes which perform= the<br>> network address encapsulation/decapsulation.<br>><br></div>= ...<br><div>> For each host (which runs on a VM) which need to communica= te with external<br> > world, it frequently sends ARP/ND requests to its default Gateway rout= er.<br>> For an application with dual stacks on, it sends both ARP &= ND requests to<br> > its default gateway router.<br><br></div>I think the issue of whether = or not an overlay network is in use is orthogonal<br>to the issue of ARP/ND= traffic that ARMD is trying to address.<br>If we have an overlay network b= uilt over a data center core,<br> we still run into the problem where ARP/ND traffic from a VM needs to<br>be= broadcast/multicast over the core (even though it uses the<br>overlay). = =A0So unless<br>some other mechanism is used to mitigate the ARP/ND traffic= , we would<br> be no better than a regular L2 fabric that spans the entire data center (at= <br>least from the standpoint of the volume of ARP/ND traffic traversing<br= >the core).<br><div><div></div><div>_______________________________________= ________<br> armd mailing list<br><a href=3D"mailto:armd@ietf.org" target=3D"_blank">arm= d@ietf.org</a><br><a href=3D"https://www.ietf.org/mailman/listinfo/armd" ta= rget=3D"_blank">https://www.ietf.org/mailman/listinfo/armd</a><br></div></d= iv></blockquote> </div><br></div></div> _______________________________________________ armd mailing list <a href=3D"mailto:armd@ietf.org" target=3D"_blank">armd@ietf.org</a> <a href=3D"https://www.ietf.org/mailman/listinfo/armd" target=3D"_blank">ht= tps://www.ietf.org/mailman/listinfo/armd</a> </span></div> <br>_______________________________________________<br> armd mailing list<br> <a href=3D"mailto:armd@ietf.org">armd@ietf.org</a><br> <a href=3D"https://www.ietf.org/mailman/listinfo/armd" target=3D"_blank">ht= tps://www.ietf.org/mailman/listinfo/armd</a><br> <br></blockquote></div><br></div> --20cf3071ca30b9364504adef0fab-- From bschlies@cisco.com Tue Sep 27 10:33:07 2011 Return-Path: <bschlies@cisco.com> X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8FE9421F8DCA for <armd@ietfa.amsl.com>; Tue, 27 Sep 2011 10:33:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.241 X-Spam-Level: X-Spam-Status: No, score=-2.241 tagged_above=-999 required=5 tests=[AWL=-1.039, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TyDr9zHHe+fP for <armd@ietfa.amsl.com>; Tue, 27 Sep 2011 10:33:05 -0700 (PDT) Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by ietfa.amsl.com (Postfix) with ESMTP id CF2C521F8DF0 for <armd@ietf.org>; Tue, 27 Sep 2011 10:33:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=bschlies@cisco.com; l=9742; q=dns/txt; s=iport; t=1317144951; x=1318354551; h=date:subject:from:to:message-id:in-reply-to:mime-version; bh=+Te5zHxxxMJYXxOs2ULFOqSMoNTZK42PQWHFj7KSTM0=; b=cXmdugnsTIdhSRFNmgV1rKeoBWPWU/w0arWqgSS9dzM1jP239E3/+QLY sv3GUV8IWvgIkb1ZmsyZDPPPsNpFZxSjwoLSceHBdh77O09piC+9kQivY zOUoY08yDbhmgKTSLxbBZWn2HV8MLzW5oyx/Zk3vMFPFe/1gkiTL5cn3b I=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgEFAJAIgk6tJXG+/2dsb2JhbABBgk2kPHN4gVMBAQEBAgEBAQEPASoxBAwNAQgRAwECAU8GKAgBAQQTIodWBppHgSYBnjADhwsEh0Iwi2CFK4Rthzw X-IronPort-AV: E=Sophos;i="4.68,450,1312156800"; d="scan'208,217";a="24393500" Received: from rcdn-core2-3.cisco.com ([173.37.113.190]) by rcdn-iport-7.cisco.com with ESMTP; 27 Sep 2011 17:35:50 +0000 Received: from xbh-rcd-302.cisco.com (xbh-rcd-302.cisco.com [72.163.63.9]) by rcdn-core2-3.cisco.com (8.14.3/8.14.3) with ESMTP id p8RHZoGQ009403 for <armd@ietf.org>; Tue, 27 Sep 2011 17:35:50 GMT Received: from xmb-rcd-206.cisco.com ([72.163.62.213]) by xbh-rcd-302.cisco.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 27 Sep 2011 12:35:50 -0500 Received: from 10.155.40.90 ([10.155.40.90]) by XMB-RCD-206.cisco.com ([72.163.62.213]) via Exchange Front-End Server email.cisco.com ([171.70.151.174]) with Microsoft Exchange Server HTTP-DAV ; Tue, 27 Sep 2011 17:35:49 +0000 User-Agent: Microsoft-Entourage/12.31.0.110725 Date: Tue, 27 Sep 2011 10:35:47 -0700 From: Benson Schliesser <bschlies@cisco.com> To: "armd@ietf.org" <armd@ietf.org> Message-ID: <CAA75783.15B21%bschlies@cisco.com> Thread-Topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-Index: Acx9O+QUhEQ3udjBPUmw54z27G+HcA== In-Reply-To: <CAJW6G52_uu9_M-oXCazTotG+VTjuxOUv_=-eN0ntMQE0aSOOiQ@mail.gmail.com> Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3399964548_64776577" X-OriginalArrivalTime: 27 Sep 2011 17:35:50.0096 (UTC) FILETIME=[E5ED0500:01CC7D3B] Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." <armd.ietf.org> List-Unsubscribe: <https://www.ietf.org/mailman/options/armd>, <mailto:armd-request@ietf.org?subject=unsubscribe> List-Archive: <http://www.ietf.org/mail-archive/web/armd> List-Post: <mailto:armd@ietf.org> List-Help: <mailto:armd-request@ietf.org?subject=help> List-Subscribe: <https://www.ietf.org/mailman/listinfo/armd>, <mailto:armd-request@ietf.org?subject=subscribe> X-List-Received-Date: Tue, 27 Sep 2011 17:33:07 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3399964548_64776577 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable I also agree with Anoop=B9s comment. But architecturally speaking, what we achieve with an overlay mechanism is a form of compartmentalization (or =B3partitioning=B2, =B3isolation=B2, etc, if you prefer), e.g. of a given set of hosts across the datacenter. We can also achieve compartmentalization of the network by dividing it into regions or clusters, and we can use both regions and overlays together to get even more compartmentalization. Whatever mechanism we use, it might allow us to distribute the address resolution function. For instance, we can distribute L3 gateways across multiple physical nodes or distribute ARP/ND proxy functions. Thus, it strikes me that there may be an architectural issue for ARMD to explore. What is the relationship between compartmentalization and address resolution scale? Is it only =B3scale=B2 that is affected by compartmentalization, or are there other address resolution attributes also affected? Cheers, -Benson On 9/27/11 9:49 AM, "John Menerick" <omgponies3145@gmail.com> wrote: > Agreed! >=20 >=20 > John Menerick >=20 > On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger <gaberger@cisco.com> wrote: >> +1 >> From: Vishwas Manral <vishwas.ietf@gmail.com> >> Date: Mon, 26 Sep 2011 13:31:19 -0700 >> To: Anoop Ghanwani <anoop@alumni.duke.edu> >> Cc: "armd@ietf.org" <armd@ietf.org> >> Subject: Re: [armd] Bottleneck of Overlay network in a data center with >> large number of tenants/hosts >>=20 >> Totally agree with Anoop here!!! >> =A0 >> -Vishwas >> =A0 >> On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu> >> wrote: >>> On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com= > >>> wrote: >>>> > >>>> > >>>> > There are many drafts recently on various Overlay Network schemes fo= r >>>> data >>>> > centers with large number of tenants/hosts. When Overlay Network is >>>> deployed >>>> > in a data center, local switches/routers within the data center netw= ork >>>> > don=B9t see the hosts addresses anymore except the nodes which perform= the >>>> > network address encapsulation/decapsulation. >>>> > >>> ... >>>> > For each host (which runs on a VM) which need to communicate with >>>> external >>>> > world, it frequently sends ARP/ND requests to its default Gateway ro= uter. >>>> > For an application with dual stacks on, it sends both ARP & ND reque= sts >>>> to >>>> > its default gateway router. >>>=20 >>> I think the issue of whether or not an overlay network is in use is >>> orthogonal >>> to the issue of ARP/ND traffic that ARMD is trying to address. >>> If we have an overlay network built over a data center core, >>> we still run into the problem where ARP/ND traffic from a VM needs to >>> be broadcast/multicast over the core (even though it uses the >>> overlay). =A0So unless >>> some other mechanism is used to mitigate the ARP/ND traffic, we would >>> be no better than a regular L2 fabric that spans the entire data center= (at >>> least from the standpoint of the volume of ARP/ND traffic traversing >>> the core). >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >>=20 >> _______________________________________________ armd mailing list >> armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>=20 >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >>=20 >=20 >=20 >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --B_3399964548_64776577 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: [armd] Bottleneck of Overlay network in a data center with large= number of tenants/hosts I also agree with Anoop’s comment. But architecturally speaking, wha= t we achieve with an overlay mechanism is a form of compartmentalization (or= “partitioning”, “isolation”, etc, if you prefer), e= .g. of a given set of hosts across the datacenter.  We can also achieve= compartmentalization of the network by dividing it into regions or clusters= , and we can use both regions and overlays together to get even more compart= mentalization.  Whatever mechanism we use, it might allow us to distrib= ute the address resolution function. For instance, we can distribute L3 gate= ways across multiple physical nodes or distribute ARP/ND proxy functions.
Thus, it strikes me that there may be an architectural issue for ARMD to ex= plore.  What is the relationship between compartmentalization and addre= ss resolution scale? Is it only “scale” that is affected by comp= artmentalization, or are there other address resolution attributes also affe= cted?

Cheers,
-Benson


On 9/27/11 9:49 AM, "John Menerick" <omgponies3145@gmail.com> wrote:

<= SPAN STYLE=3D'font-size:11pt'>Agreed!


John Menerick

On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger <gaberger@cisco.com> wrote:
<= SPAN STYLE=3D'font-size:11pt'>+1
From: Vishwas Manral <vishwas.i= etf@gmail.com>
Date: Mon, 26 Sep 2011 13:31:19 -0700
To: Anoop Ghanwani <anoop@alumni= .duke.edu>
Cc: "armd@ietf.org" <armd@ietf.org>
Subject: Re: [armd] Bottleneck of Overlay network in a data center = with large number of tenants/hosts

Totally agree with Anoop here!!!
=A0
-Vishwas
=A0
On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu> wrote:
<= SPAN STYLE=3D'font-size:11pt'>On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar &= lt;linda.dunbar@huawei.com> wrote:<= BR> >
>
> There are many drafts recently on various Overlay Network schemes for = data
> centers with large number of tenants/hosts. When Overlay Network is de= ployed
> in a data center, local switches/routers within the data center networ= k
> don’t see the hosts addresses anymore except the nodes which per= form the
> network address encapsulation/decapsulation.
>
...
> For each host (which runs on a VM) which need to communicate with exte= rnal
> world, it frequently sends ARP/ND requests to its default Gateway rout= er.
> For an application with dual stacks on, it sends both ARP & ND req= uests to
> its default gateway router.

I think the issue of whether or not an overlay network is in use is orthogo= nal
to the issue of ARP/ND traffic that ARMD is trying to address.
If we have an overlay network built over a data center core,
we still run into the problem where ARP/ND traffic from a VM needs to
be broadcast/multicast over the core (even though it uses the
overlay). =A0So unless
some other mechanism is used to mitigate the ARP/ND traffic, we would
be no better than a regular L2 fabric that spans the entire data center (at=
least from the standpoint of the volume of ARP/ND traffic traversing
the core).
_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/m= ailman/listinfo/armd
=
_______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd

_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/m= ailman/listinfo/armd

=

___________= ____________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/m= ailman/listinfo/armd
--B_3399964548_64776577-- From gaberger@cisco.com Tue Sep 27 14:41:57 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3CA2421F8EE8 for ; Tue, 27 Sep 2011 14:41:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.152 X-Spam-Level: X-Spam-Status: No, score=-1.152 tagged_above=-999 required=5 tests=[AWL=0.050, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fyluAlTekzuD for ; Tue, 27 Sep 2011 14:41:54 -0700 (PDT) Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by ietfa.amsl.com (Postfix) with ESMTP id 49E4921F8EE7 for ; Tue, 27 Sep 2011 14:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=12079; q=dns/txt; s=iport; t=1317159881; x=1318369481; h=date:subject:from:to:message-id:in-reply-to:mime-version; bh=n56v9bN8m0SWdDnIFRHtgIuT1IttzAW6/7OuJk0CUqY=; b=IqmhQargxHNBbfHljIRgnf08JbCw022Uu17+3SN9DTSfxtMCR200RJRp 1uKJTWx7WmvI3AiCLAPV3KE+0ZmCZ9vCZHa7yKrme0gH7d6KyqD6A0boW YVXRGJ5zajI0T0D2K+/YH73Pt/dHGGHaZsrE+xhxaEupoe6vPYkeaMO28 A=; X-IronPort-AV: E=Sophos;i="4.68,451,1312156800"; d="scan'208,217";a="24454752" Received: from rcdn-core-6.cisco.com ([173.37.93.157]) by rcdn-iport-7.cisco.com with ESMTP; 27 Sep 2011 21:44:41 +0000 Received: from [10.82.225.136] (rtp-vpn1-392.cisco.com [10.82.225.136]) by rcdn-core-6.cisco.com (8.14.3/8.14.3) with ESMTP id p8RLiaXs030592; Tue, 27 Sep 2011 21:44:39 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Tue, 27 Sep 2011 17:44:34 -0400 From: Gary Berger To: Benson Schliesser , "armd@ietf.org" Message-ID: Thread-Topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3399990280_21347386" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 21:41:57 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3399990280_21347386 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Isn't that the purpose of the network-layer? Why do we need compartmentalization if we have a functioning network layer and proper address-bindings? The whole issue is in the scaling function of dynamic learning which is a side-effect of the mobility requirement in todays data centers. From: Benson Schliesser Date: Tue, 27 Sep 2011 10:35:47 -0700 To: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts I also agree with Anoop=B9s comment. But architecturally speaking, what we achieve with an overlay mechanism is a form of compartmentalization (or =B3partitioning=B2, =B3isolation=B2, etc, if you prefer), e.g. of a given set of hosts across the datacenter. We can also achieve compartmentalization of the network by dividing it into regions or clusters, and we can use both regions and overlays together to get even more compartmentalization. Whatever mechanism we use, it might allow us to distribute the address resolution function. For instance, we can distribute L3 gateways across multiple physical nodes or distribute ARP/ND proxy functions. Thus, it strikes me that there may be an architectural issue for ARMD to explore. What is the relationship between compartmentalization and address resolution scale? Is it only =B3scale=B2 that is affected by compartmentalization, or are there other address resolution attributes also affected? Cheers, -Benson On 9/27/11 9:49 AM, "John Menerick" wrote: > Agreed! >=20 >=20 > John Menerick >=20 > On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger wrote: >> +1 >> From: Vishwas Manral >> Date: Mon, 26 Sep 2011 13:31:19 -0700 >> To: Anoop Ghanwani >> Cc: "armd@ietf.org" >> Subject: Re: [armd] Bottleneck of Overlay network in a data center with >> large number of tenants/hosts >>=20 >> Totally agree with Anoop here!!! >> =20 >> -Vishwas >> =20 >> On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani >> wrote: >>> On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar >>> wrote: >>>> > >>>> > >>>> > There are many drafts recently on various Overlay Network schemes fo= r >>>> data >>>> > centers with large number of tenants/hosts. When Overlay Network is >>>> deployed >>>> > in a data center, local switches/routers within the data center netw= ork >>>> > don=B9t see the hosts addresses anymore except the nodes which perform= the >>>> > network address encapsulation/decapsulation. >>>> > >>> ... >>>> > For each host (which runs on a VM) which need to communicate with >>>> external >>>> > world, it frequently sends ARP/ND requests to its default Gateway ro= uter. >>>> > For an application with dual stacks on, it sends both ARP & ND reque= sts >>>> to >>>> > its default gateway router. >>>=20 >>> I think the issue of whether or not an overlay network is in use is >>> orthogonal >>> to the issue of ARP/ND traffic that ARMD is trying to address. >>> If we have an overlay network built over a data center core, >>> we still run into the problem where ARP/ND traffic from a VM needs to >>> be broadcast/multicast over the core (even though it uses the >>> overlay). So unless >>> some other mechanism is used to mitigate the ARP/ND traffic, we would >>> be no better than a regular L2 fabric that spans the entire data center= (at >>> least from the standpoint of the volume of ARP/ND traffic traversing >>> the core). >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >>=20 >> _______________________________________________ armd mailing list >> armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>=20 >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >>=20 >=20 >=20 >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --B_3399990280_21347386 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable
Isn't that the pur= pose of the network-layer?  Why do we need compartmentalization if we h= ave a functioning network layer and proper address-bindings? The whole issue= is in the scaling function of dynamic learning which is a side-effect of th= e mobility requirement in todays data centers.

= From: Benson Schliesser <bschlies@cisco.com>
Date: Tue, 27 Sep 2011 10:35:47 -0700
To: "armd@ietf.org" <armd@ietf.org>
Subject: Re: [armd] Bottleneck of Overlay network = in a data center with large number of tenants/hosts

=
Re: [armd] Bottleneck of Overlay network in a data center with l= arge number of tenants/hosts
I also agree with Anoop’s comme= nt. But architecturally speaking, what we achieve with an overlay mechanism = is a form of compartmentalization (or “partitioning”, “iso= lation”, etc, if you prefer), e.g. of a given set of hosts across the = datacenter.  We can also achieve compartmentalization of the network by= dividing it into regions or clusters, and we can use both regions and overl= ays together to get even more compartmentalization.  Whatever mechanism= we use, it might allow us to distribute the address resolution function. Fo= r instance, we can distribute L3 gateways across multiple physical nodes or = distribute ARP/ND proxy functions.

Thus, it strikes me that there may be an architectural issue for ARMD to ex= plore.  What is the relationship between compartmentalization and addre= ss resolution scale? Is it only “scale” that is affected by comp= artmentalization, or are there other address resolution attributes also affe= cted?

Cheers,
-Benson


On 9/27/11 9:49 AM, "John Menerick" <o= mgponies3145@gmail.com> wrote:

Agreed!=


John Menerick

On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger <gaberger@cisco.com> wrote:
+1
Fro= m: Vishwas Manral <vishwas.ietf@gma= il.com>
Date: Mon, 26 Sep 2011 13:31:19 -0700
To: Anoop Ghanwani <anoop@alumni.duke.edu<= /a>>
Cc: "armd@ietf.org" <armd@ietf.org>
Subject: Re: [armd] Bott= leneck of Overlay network in a data center with large number of tenants/host= s

Totally agree with Anoop here!!!
 
-Vishwas
 
On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu> wrote:
= On= Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com> wrote:
>
>
> There are many drafts recently on various Overlay Network schemes for = data
> centers with large number of tenants/hosts. When Overlay Network is de= ployed
> in a data center, local switches/routers within the data center networ= k
> don’t see the hosts addresses anymore except the nodes which per= form the
> network address encapsulation/decapsulation.
>
...
> For each host (which runs on a VM) which need to communicate with exte= rnal
> world, it frequently sends ARP/ND requests to its default Gateway rout= er.
> For an application with dual stacks on, it sends both ARP & ND req= uests to
> its default gateway router.

I think the issue of whether or not an overlay network is in use is orthogo= nal
to the issue of ARP/ND traffic that ARMD is trying to address.
If we have an overlay network built over a data center core,
we still run into the problem where ARP/ND traffic from a VM needs to
be broadcast/multicast over the core (even though it uses the
overlay).  So unless
some other mechanism is used to mitigate the ARP/ND traffic, we would
be no better than a regular L2 fabric that spans the entire data center (at=
least from the standpoint of the volume of ARP/ND traffic traversing
the core).
_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/lis= tinfo/armd

_______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd

_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/lis= tinfo/armd



_________________________________= ______________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/lis= tinfo/armd
_______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/m= ailman/listinfo/armd --B_3399990280_21347386-- From linda.dunbar@huawei.com Tue Sep 27 15:08:24 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B4F8D21F8FEB for ; Tue, 27 Sep 2011 15:08:24 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.379 X-Spam-Level: X-Spam-Status: No, score=-6.379 tagged_above=-999 required=5 tests=[AWL=0.219, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DXc7Jl8relDs for ; Tue, 27 Sep 2011 15:08:21 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id A04CB21F8FF2 for ; Tue, 27 Sep 2011 15:08:21 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS700LVGCAJ5T@usaga04-in.huawei.com> for armd@ietf.org; Tue, 27 Sep 2011 17:11:08 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS700H0XCAJRE@usaga04-in.huawei.com> for armd@ietf.org; Tue, 27 Sep 2011 17:11:07 -0500 (CDT) Received: from DFWEML401-HUB.china.huawei.com (10.193.5.101) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 27 Sep 2011 15:11:08 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.103]) by DFWEML401-HUB.china.huawei.com ([fe80::f07f:889f:78ef:8df3%13]) with mapi id 14.01.0270.001; Tue, 27 Sep 2011 15:11:00 -0700 Date: Tue, 27 Sep 2011 22:10:59 +0000 From: Linda Dunbar In-reply-to: X-Originating-IP: [10.192.11.155] To: Gary Berger , Benson Schliesser , "armd@ietf.org" Message-id: <4A95BA014132FF49AE685FAB4B9F17F610CAC53D@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_6drN+YyhlapLjk2COyN34g)" Content-language: en-US Accept-Language: en-US Thread-topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-index: Acx1aIsSyTVtVcAKQ4yYdknwqYNy9gHUhp6AAALRg4AAJeKSgAAEqi6AAAGcd4AACLBMAAAN05nw X-MS-Has-Attach: X-MS-TNEF-Correlator: References: Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 22:08:24 -0000 --Boundary_(ID_6drN+YyhlapLjk2COyN34g) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Overlay network enables operators to move VMs from one rack to another without re-configuring IP addresses. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Gary Berger Sent: Tuesday, September 27, 2011 4:45 PM To: Benson Schliesser; armd@ietf.org Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Isn't that the purpose of the network-layer? Why do we need compartmentalization if we have a functioning network layer and proper address-bindings? The whole issue is in the scaling function of dynamic learning which is a side-effect of the mobility requirement in todays data centers. From: Benson Schliesser > Date: Tue, 27 Sep 2011 10:35:47 -0700 To: "armd@ietf.org" > Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts I also agree with Anoop's comment. But architecturally speaking, what we achieve with an overlay mechanism is a form of compartmentalization (or "partitioning", "isolation", etc, if you prefer), e.g. of a given set of hosts across the datacenter. We can also achieve compartmentalization of the network by dividing it into regions or clusters, and we can use both regions and overlays together to get even more compartmentalization. Whatever mechanism we use, it might allow us to distribute the address resolution function. For instance, we can distribute L3 gateways across multiple physical nodes or distribute ARP/ND proxy functions. Thus, it strikes me that there may be an architectural issue for ARMD to explore. What is the relationship between compartmentalization and address resolution scale? Is it only "scale" that is affected by compartmentalization, or are there other address resolution attributes also affected? Cheers, -Benson On 9/27/11 9:49 AM, "John Menerick" wrote: Agreed! John Menerick On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger wrote: +1 From: Vishwas Manral Date: Mon, 26 Sep 2011 13:31:19 -0700 To: Anoop Ghanwani Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Totally agree with Anoop here!!! -Vishwas On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani wrote: On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar wrote: > > > There are many drafts recently on various Overlay Network schemes for data > centers with large number of tenants/hosts. When Overlay Network is deployed > in a data center, local switches/routers within the data center network > don't see the hosts addresses anymore except the nodes which perform the > network address encapsulation/decapsulation. > ... > For each host (which runs on a VM) which need to communicate with external > world, it frequently sends ARP/ND requests to its default Gateway router. > For an application with dual stacks on, it sends both ARP & ND requests to > its default gateway router. I think the issue of whether or not an overlay network is in use is orthogonal to the issue of ARP/ND traffic that ARMD is trying to address. If we have an overlay network built over a data center core, we still run into the problem where ARP/ND traffic from a VM needs to be broadcast/multicast over the core (even though it uses the overlay). So unless some other mechanism is used to mitigate the ARP/ND traffic, we would be no better than a regular L2 fabric that spans the entire data center (at least from the standpoint of the volume of ARP/ND traffic traversing the core). _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd ________________________________ _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --Boundary_(ID_6drN+YyhlapLjk2COyN34g) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

Overlay network enables operators to move VMs from one rack to another without re-configuring IP addresses.

 

Linda

 

From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Gary Berger
Sent: Tuesday, September 27, 2011 4:45 PM
To: Benson Schliesser; armd@ietf.org
Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

 

Isn't that the purpose of the network-layer?  Why do we need compartmentalization if we have a functioning network layer and proper address-bindings? The whole issue is in the scaling function of dynamic learning which is a side-effect of the mobility requirement in todays data centers.

 

From: Benson Schliesser <bschlies@cisco.com>
Date: Tue, 27 Sep 2011 10:35:47 -0700
To: "armd@ietf.org" <armd@ietf.org>
Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

 

I also agree with Anoop’s comment. But architecturally speaking, what we achieve with an overlay mechanism is a form of compartmentalization (or “partitioning”, “isolation”, etc, if you prefer), e.g. of a given set of hosts across the datacenter.  We can also achieve compartmentalization of the network by dividing it into regions or clusters, and we can use both regions and overlays together to get even more compartmentalization.  Whatever mechanism we use, it might allow us to distribute the address resolution function. For instance, we can distribute L3 gateways across multiple physical nodes or distribute ARP/ND proxy functions.

Thus, it strikes me that there may be an architectural issue for ARMD to explore.  What is the relationship between compartmentalization and address resolution scale? Is it only “scale” that is affected by compartmentalization, or are there other address resolution attributes also affected?

Cheers,
-Benson


On 9/27/11 9:49 AM, "John Menerick" <omgponies3145@gmail.com> wrote:

Agreed!


John Menerick

On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger <gaberger@cisco.com> wrote:

+1
From: Vishwas Manral <vishwas.ietf@gmail.com>
Date: Mon, 26 Sep 2011 13:31:19 -0700
To: Anoop Ghanwani <anoop@alumni.duke.edu>
Cc: "armd@ietf.org" <armd@ietf.org>
Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts

Totally agree with Anoop here!!!
 
-Vishwas
 
On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani <anoop@alumni.duke.edu> wrote:

On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar <linda.dunbar@huawei.com> wrote:
>
>
> There are many drafts recently on various Overlay Network schemes for data
> centers with large number of tenants/hosts. When Overlay Network is deployed
> in a data center, local switches/routers within the data center network
> don’t see the hosts addresses anymore except the nodes which perform the
> network address encapsulation/decapsulation.
>
...
> For each host (which runs on a VM) which need to communicate with external
> world, it frequently sends ARP/ND requests to its default Gateway router.
> For an application with dual stacks on, it sends both ARP & ND requests to
> its default gateway router.

I think the issue of whether or not an overlay network is in use is orthogonal
to the issue of ARP/ND traffic that ARMD is trying to address.
If we have an overlay network built over a data center core,
we still run into the problem where ARP/ND traffic from a VM needs to
be broadcast/multicast over the core (even though it uses the
overlay).  So unless
some other mechanism is used to mitigate the ARP/ND traffic, we would
be no better than a regular L2 fabric that spans the entire data center (at
least from the standpoint of the volume of ARP/ND traffic traversing
the core).
_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd


_______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd

_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd

 

_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd

_______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd

--Boundary_(ID_6drN+YyhlapLjk2COyN34g)-- From ghanwani@gmail.com Tue Sep 27 16:04:40 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3446121F8EC8 for ; Tue, 27 Sep 2011 16:04:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.072 X-Spam-Level: X-Spam-Status: No, score=-3.072 tagged_above=-999 required=5 tests=[AWL=-0.695, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, J_CHICKENPOX_32=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ySGKIU4XT3Jz for ; Tue, 27 Sep 2011 16:04:39 -0700 (PDT) Received: from mail-qy0-f179.google.com (mail-qy0-f179.google.com [209.85.216.179]) by ietfa.amsl.com (Postfix) with ESMTP id 7132221F8ECA for ; Tue, 27 Sep 2011 16:04:39 -0700 (PDT) Received: by qyk33 with SMTP id 33so8174593qyk.10 for ; Tue, 27 Sep 2011 16:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=b8dee5GKCUUfJAJlmuAUrOUqecKIT4K5NTGC6sAV9kk=; b=i3MD2XIMryCl6SAbbVdZcyHFO8okea5EXnHRHfpBpXh4u29vivX07/eE0PX7mAeUmz KXYYrAPSVlYRlm15wUQiMNL2t3CuD68KQzGYhQC1ufjl8b1mUfVbX+r+SUbrk5ojiv0k IB5DTEDbzuP2vTb2JOb5FFhjvBdaIb1Zt+hUk= MIME-Version: 1.0 Received: by 10.229.66.158 with SMTP id n30mr6149630qci.249.1317164845738; Tue, 27 Sep 2011 16:07:25 -0700 (PDT) Sender: ghanwani@gmail.com Received: by 10.229.219.197 with HTTP; Tue, 27 Sep 2011 16:07:25 -0700 (PDT) In-Reply-To: References: Date: Tue, 27 Sep 2011 16:07:25 -0700 X-Google-Sender-Auth: lS1pXhHCwiqML7FVScmpzQSFCnQ Message-ID: From: Anoop Ghanwani To: Benson Schliesser Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: armd@ietf.org Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2011 23:04:40 -0000 On Tue, Sep 27, 2011 at 10:35 AM, Benson Schliesser wr= ote: > I also agree with Anoop=92s comment. But architecturally speaking, what w= e > achieve with an overlay mechanism is a form of compartmentalization (or > =93partitioning=94, =93isolation=94, etc, if you prefer), e.g. of a given= set of > hosts across the datacenter. =A0We can also achieve compartmentalization = of > the network by dividing it into regions or clusters, and we can use both > regions and overlays together to get even more compartmentalization. Can you clarify what you mean by "cluster"? Regardless, in the absence of any other protocol mechanisms, having an overlay won't help ARP/ND in any way whatsoever. Are you using cluster to mean a physical section of the network that will help keep the overlays themselves contained in order to limit the propagation of ARP/ND messages? =A0If so, I agree. =A0However, at the limit, if we make each ToR its own cluster, then we don't even need the overlay. > =A0Whatever mechanism we use, it might allow us to distribute the address > resolution function. For instance, we can distribute L3 gateways across > multiple physical nodes or distribute ARP/ND proxy functions. It is easier for me to understand distribution of the ARP/ND proxy functions because that is purely a control-plane problem. I have trouble with the distributed L3 gateway because that is a data plane problem and we will hav to worry about optimality of the forwarding paths. =A0Can you shed more light on the distributed L3 gateway solution? > Thus, it strikes me that there may be an architectural issue for ARMD to > explore. =A0What is the relationship between compartmentalization and add= ress > resolution scale? Is it only =93scale=94 that is affected by > compartmentalization, or are there other address resolution attributes al= so > affected? What other attributes did you have in mind? =A0The scale issue is clear with respect to number of MAC addresses, the the MAC:IP bindings, and the volume of broadcast/multicast traffic that gets sent across core links. From christopher.morrow@gmail.com Tue Sep 27 22:09:44 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7748521F8B4E for ; Tue, 27 Sep 2011 22:09:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.549 X-Spam-Level: X-Spam-Status: No, score=-103.549 tagged_above=-999 required=5 tests=[AWL=0.050, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QJ9Z9pqvk3wY for ; Tue, 27 Sep 2011 22:09:38 -0700 (PDT) Received: from mail-pz0-f50.google.com (mail-pz0-f50.google.com [209.85.210.50]) by ietfa.amsl.com (Postfix) with ESMTP id 26CC421F8B4C for ; Tue, 27 Sep 2011 22:09:35 -0700 (PDT) Received: by pzk37 with SMTP id 37so20482543pzk.9 for ; Tue, 27 Sep 2011 22:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=cxiEHPtCkXvYWr1gvJ0ys6mn5lUcqJKWFPyALYmYjd8=; b=MrwIYovIPxXXIyRAu28JwyOG5gd4R+lcwD+VeDCiv7LVVlReFCMrjGHIWQQ+IaDwbu sSyTkx1MyRFgwyWvxw3zgFQ+gEmgjQja2Q/lx1P7Ov2Tpl7+ND4aYtXqtyDv1FyBhAjg FYtGXnQ2h6l8R/EkVFKYSHzAl3l7f8WW7wZgw= MIME-Version: 1.0 Received: by 10.68.59.10 with SMTP id v10mr42589153pbq.16.1317186738392; Tue, 27 Sep 2011 22:12:18 -0700 (PDT) Sender: christopher.morrow@gmail.com Received: by 10.68.47.6 with HTTP; Tue, 27 Sep 2011 22:12:18 -0700 (PDT) In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610CAC53D@dfweml503-mbx.china.huawei.com> References: <4A95BA014132FF49AE685FAB4B9F17F610CAC53D@dfweml503-mbx.china.huawei.com> Date: Wed, 28 Sep 2011 01:12:18 -0400 X-Google-Sender-Auth: g6YTgu3KMdjXS7-oMAC_-fY2-hA Message-ID: From: Christopher Morrow To: Linda Dunbar Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 05:09:45 -0000 On Tue, Sep 27, 2011 at 6:10 PM, Linda Dunbar wro= te: > Overlay network enables operators to move VMs from one rack to another > without re-configuring IP addresses. Or, to put it slightly differently.... o Build a large L3 enabled datacenter(s) network o overlay L2 across this (vpls, for one example) o feel free to move about the datacenter(s) with your vm-image There's still the problem at the edge nodes understanding what/where the nodes are at L2, there's addtionaly the operations problem of managing a changing VPLS (in this example) overlay (perhaps vlan 77 did not exist at rack 22, but you need it there now... quick, add it to the local network devices!!) It's intriguing, I'm not sure how well it fixes/compartmentalizes the problem, I'm also not clear on where the cost just got shifted to :) It also assumes that the network gear (potentially the bottom-of-the-barrel TOR device we were dealing with before) can perform VPLS functions. this could due with some study it seems, to me at least. -chris > Linda > > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of G= ary > Berger > Sent: Tuesday, September 27, 2011 4:45 PM > To: Benson Schliesser; armd@ietf.org > > Subject: Re: [armd] Bottleneck of Overlay network in a data center with > large number of tenants/hosts > > > > Isn't that the purpose of the network-layer? =A0Why do we need > compartmentalization if we have a functioning network layer and proper > address-bindings? The whole issue is in the scaling function of dynamic > learning which is a side-effect of the mobility requirement in todays dat= a > centers. > > > > From: Benson Schliesser > Date: Tue, 27 Sep 2011 10:35:47 -0700 > To: "armd@ietf.org" > Subject: Re: [armd] Bottleneck of Overlay network in a data center with > large number of tenants/hosts > > > > I also agree with Anoop=92s comment. But architecturally speaking, what w= e > achieve with an overlay mechanism is a form of compartmentalization (or > =93partitioning=94, =93isolation=94, etc, if you prefer), e.g. of a given= set of > hosts across the datacenter. =A0We can also achieve compartmentalization = of > the network by dividing it into regions or clusters, and we can use both > regions and overlays together to get even more compartmentalization. > =A0Whatever mechanism we use, it might allow us to distribute the address > resolution function. For instance, we can distribute L3 gateways across > multiple physical nodes or distribute ARP/ND proxy functions. > > Thus, it strikes me that there may be an architectural issue for ARMD to > explore. =A0What is the relationship between compartmentalization and add= ress > resolution scale? Is it only =93scale=94 that is affected by > compartmentalization, or are there other address resolution attributes al= so > affected? > > Cheers, > -Benson > > > On 9/27/11 9:49 AM, "John Menerick" wrote: > > Agreed! > > > John Menerick > > On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger wrote: > > +1 > From: Vishwas Manral > Date: Mon, 26 Sep 2011 13:31:19 -0700 > To: Anoop Ghanwani > Cc: "armd@ietf.org" > Subject: Re: [armd] Bottleneck of Overlay network in a data center with > large number of tenants/hosts > > Totally agree with Anoop here!!! > > -Vishwas > > On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani > wrote: > > On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar > wrote: >> >> >> There are many drafts recently on various Overlay Network schemes for da= ta >> centers with large number of tenants/hosts. When Overlay Network is >> deployed >> in a data center, local switches/routers within the data center network >> don=92t see the hosts addresses anymore except the nodes which perform t= he >> network address encapsulation/decapsulation. >> > ... >> For each host (which runs on a VM) which need to communicate with extern= al >> world, it frequently sends ARP/ND requests to its default Gateway router= . >> For an application with dual stacks on, it sends both ARP & ND requests = to >> its default gateway router. > > I think the issue of whether or not an overlay network is in use is > orthogonal > to the issue of ARP/ND traffic that ARMD is trying to address. > If we have an overlay network built over a data center core, > we still run into the problem where ARP/ND traffic from a VM needs to > be broadcast/multicast over the core (even though it uses the > overlay). =A0So unless > some other mechanism is used to mitigate the ARP/ND traffic, we would > be no better than a regular L2 fabric that spans the entire data center (= at > least from the standpoint of the volume of ARP/ND traffic traversing > the core). > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > > > ________________________________ > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > From xuxiaohu@huawei.com Wed Sep 28 00:04:40 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C829221F8B2F for ; Wed, 28 Sep 2011 00:04:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.279 X-Spam-Level: X-Spam-Status: No, score=-4.279 tagged_above=-999 required=5 tests=[AWL=2.320, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id npVqpgkO-tFv for ; Wed, 28 Sep 2011 00:04:39 -0700 (PDT) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [119.145.14.64]) by ietfa.amsl.com (Postfix) with ESMTP id 992F921F8CCA for ; Wed, 28 Sep 2011 00:04:38 -0700 (PDT) Received: from huawei.com (szxga05-in [172.24.2.49]) by szxga05-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS8003TV136JL@szxga05-in.huawei.com> for armd@ietf.org; Wed, 28 Sep 2011 15:06:42 +0800 (CST) Received: from szxrg01-dlp.huawei.com ([172.24.2.119]) by szxga05-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS800NAO1334J@szxga05-in.huawei.com> for armd@ietf.org; Wed, 28 Sep 2011 15:06:42 +0800 (CST) Received: from szxeml207-edg.china.huawei.com ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.1.9-GA) with ESMTP id AEE95734; Wed, 28 Sep 2011 15:06:22 +0800 Received: from SZXEML411-HUB.china.huawei.com (10.82.67.138) by szxeml207-edg.china.huawei.com (172.24.2.59) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 28 Sep 2011 15:06:19 +0800 Received: from SZXEML525-MBS.china.huawei.com ([169.254.8.181]) by szxeml411-hub.china.huawei.com ([10.82.67.138]) with mapi id 14.01.0270.001; Wed, 28 Sep 2011 15:06:15 +0800 Date: Wed, 28 Sep 2011 07:06:15 +0000 From: Xuxiaohu In-reply-to: X-Originating-IP: [10.108.4.115] To: Christopher Morrow , Linda Dunbar Message-id: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE736806@szxeml525-mbs.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-language: zh-CN Content-transfer-encoding: base64 Accept-Language: zh-CN, en-US Thread-topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-index: AQHMfIAQACzuTo2v30SdIKBzAk348ZVfl2KAgAEvFICAACVSgIAADOOAgABFgwCAAAdhgIAAdbcAgACjV+A= X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected References: <4A95BA014132FF49AE685FAB4B9F17F610CAC53D@dfweml503-mbx.china.huawei.com> Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 07:04:40 -0000 DQo+IC0tLS0t6YKu5Lu25Y6f5Lu2LS0tLS0NCj4g5Y+R5Lu25Lq6OiBhcm1kLWJvdW5jZXNAaWV0 Zi5vcmcgW21haWx0bzphcm1kLWJvdW5jZXNAaWV0Zi5vcmddIOS7o+ihqA0KPiBDaHJpc3RvcGhl ciBNb3Jyb3cNCj4g5Y+R6YCB5pe26Ze0OiAyMDEx5bm0OeaciDI45pelIDEzOjEyDQo+IOaUtuS7 tuS6ujogTGluZGEgRHVuYmFyDQo+IOaKhOmAgTogYXJtZEBpZXRmLm9yZw0KPiDkuLvpopg6IFJl OiBbYXJtZF0gQm90dGxlbmVjayBvZiBPdmVybGF5IG5ldHdvcmsgaW4gYSBkYXRhIGNlbnRlciB3 aXRoIGxhcmdlDQo+IG51bWJlciBvZiB0ZW5hbnRzL2hvc3RzDQo+IA0KPiBPbiBUdWUsIFNlcCAy NywgMjAxMSBhdCA2OjEwIFBNLCBMaW5kYSBEdW5iYXIgPGxpbmRhLmR1bmJhckBodWF3ZWkuY29t Pg0KPiB3cm90ZToNCj4gPiBPdmVybGF5IG5ldHdvcmsgZW5hYmxlcyBvcGVyYXRvcnMgdG8gbW92 ZSBWTXMgZnJvbSBvbmUgcmFjayB0byBhbm90aGVyDQo+ID4gd2l0aG91dCByZS1jb25maWd1cmlu ZyBJUCBhZGRyZXNzZXMuDQo+IA0KPiBPciwgdG8gcHV0IGl0IHNsaWdodGx5IGRpZmZlcmVudGx5 Li4uLg0KPiANCj4gICBvIEJ1aWxkIGEgbGFyZ2UgTDMgZW5hYmxlZCBkYXRhY2VudGVyKHMpIG5l dHdvcmsNCj4gICBvIG92ZXJsYXkgTDIgYWNyb3NzIHRoaXMgKHZwbHMsIGZvciBvbmUgZXhhbXBs ZSkNCj4gICBvIGZlZWwgZnJlZSB0byBtb3ZlIGFib3V0IHRoZSBkYXRhY2VudGVyKHMpIHdpdGgg eW91ciB2bS1pbWFnZQ0KPiANCj4gVGhlcmUncyBzdGlsbCB0aGUgcHJvYmxlbSBhdCB0aGUgZWRn ZSBub2RlcyB1bmRlcnN0YW5kaW5nIHdoYXQvd2hlcmUNCj4gdGhlIG5vZGVzIGFyZSBhdCBMMiwg dGhlcmUncyBhZGR0aW9uYWx5IHRoZSBvcGVyYXRpb25zIHByb2JsZW0gb2YNCj4gbWFuYWdpbmcg YSBjaGFuZ2luZyBWUExTIChpbiB0aGlzIGV4YW1wbGUpIG92ZXJsYXkgKHBlcmhhcHMgdmxhbiA3 Nw0KPiBkaWQgbm90IGV4aXN0IGF0IHJhY2sgMjIsIGJ1dCB5b3UgbmVlZCBpdCB0aGVyZSBub3cu Li4gcXVpY2ssIGFkZCBpdA0KPiB0byB0aGUgbG9jYWwgbmV0d29yayBkZXZpY2VzISEpDQoNClZQ TFMgY291bGQgYWN0dWFsbHkgYmUgY29uc2lkZXJlZCBhcyBhIG92ZXJsYXkgbmV0d29yayAoTDIg YWNyb3NzIEwzKSBzb2x1dGlvbiBmb3IgbXVsdGktdGVuYW50IGRhdGEgY2VudGVycyBkdWUgdG8g aXRzIG1hbnkgc2NhbGluZyBjYXBhYmlsaXRpZXMuIEluIGZhY3QsIHNvbWUgdmVuZG9ycyBhcmUg cHJvcG9zaW5nIFZQTFMgYXMgYSBwcmFjdGljYWwgZGF0YSBjZW50ZXIgbmV0d29yayBzb2x1dGlv biB0byBjbG91ZCBkYXRhIGNlbnRlciBwcm92aWRlcnMuIEFzIGZvciBob3cgdG8gcmVkdWNlIHRo ZSBjb25maWd1cmF0aW9uIGFuZCBtYWludGVuYW5jZSBidXJkZW5zIGFzc29jaWF0ZWQgd2l0aCB0 aGUgbGVnYWN5IFZQTFMgc29sdXRpb25zIChpLmUuLCBMRFAgVlBMUyBvciBCR1AgVlBMUyApLCBJ Uy1JUyBWUExTIChodHRwOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC14dS1sMnZwbi12cGxz LWlzaXMtMDEpIGlzIGFuIG9uZ29pbmcgYXR0ZW1wdCB0b3dhcmRzIHRoYXQgZGlyZWN0aW9uLg0K DQpCZXN0IHJlZ2FyZHMsDQpYaWFvaHUNCg0KPiBJdCdzIGludHJpZ3VpbmcsIEknbSBub3Qgc3Vy ZSBob3cgd2VsbCBpdCBmaXhlcy9jb21wYXJ0bWVudGFsaXplcyB0aGUNCj4gcHJvYmxlbSwgSSdt IGFsc28gbm90IGNsZWFyIG9uIHdoZXJlIHRoZSBjb3N0IGp1c3QgZ290IHNoaWZ0ZWQgdG8gOikN Cj4gSXQgYWxzbyBhc3N1bWVzIHRoYXQgdGhlIG5ldHdvcmsgZ2VhciAocG90ZW50aWFsbHkgdGhl DQo+IGJvdHRvbS1vZi10aGUtYmFycmVsIFRPUiBkZXZpY2Ugd2Ugd2VyZSBkZWFsaW5nIHdpdGgg YmVmb3JlKSBjYW4NCj4gcGVyZm9ybSBWUExTIGZ1bmN0aW9ucy4NCj4gDQo+IHRoaXMgY291bGQg ZHVlIHdpdGggc29tZSBzdHVkeSBpdCBzZWVtcywgdG8gbWUgYXQgbGVhc3QuDQo+IA0KPiAtY2hy aXMNCj4gDQo+ID4gTGluZGENCj4gPg0KPiA+DQo+ID4NCj4gPiBGcm9tOiBhcm1kLWJvdW5jZXNA aWV0Zi5vcmcgW21haWx0bzphcm1kLWJvdW5jZXNAaWV0Zi5vcmddIE9uIEJlaGFsZiBPZg0KPiBH YXJ5DQo+ID4gQmVyZ2VyDQo+ID4gU2VudDogVHVlc2RheSwgU2VwdGVtYmVyIDI3LCAyMDExIDQ6 NDUgUE0NCj4gPiBUbzogQmVuc29uIFNjaGxpZXNzZXI7IGFybWRAaWV0Zi5vcmcNCj4gPg0KPiA+ IFN1YmplY3Q6IFJlOiBbYXJtZF0gQm90dGxlbmVjayBvZiBPdmVybGF5IG5ldHdvcmsgaW4gYSBk YXRhIGNlbnRlciB3aXRoDQo+ID4gbGFyZ2UgbnVtYmVyIG9mIHRlbmFudHMvaG9zdHMNCj4gPg0K PiA+DQo+ID4NCj4gPiBJc24ndCB0aGF0IHRoZSBwdXJwb3NlIG9mIHRoZSBuZXR3b3JrLWxheWVy PyDCoFdoeSBkbyB3ZSBuZWVkDQo+ID4gY29tcGFydG1lbnRhbGl6YXRpb24gaWYgd2UgaGF2ZSBh IGZ1bmN0aW9uaW5nIG5ldHdvcmsgbGF5ZXIgYW5kIHByb3Blcg0KPiA+IGFkZHJlc3MtYmluZGlu Z3M/IFRoZSB3aG9sZSBpc3N1ZSBpcyBpbiB0aGUgc2NhbGluZyBmdW5jdGlvbiBvZiBkeW5hbWlj DQo+ID4gbGVhcm5pbmcgd2hpY2ggaXMgYSBzaWRlLWVmZmVjdCBvZiB0aGUgbW9iaWxpdHkgcmVx dWlyZW1lbnQgaW4gdG9kYXlzIGRhdGENCj4gPiBjZW50ZXJzLg0KPiA+DQo+ID4NCj4gPg0KPiA+ IEZyb206IEJlbnNvbiBTY2hsaWVzc2VyIDxic2NobGllc0BjaXNjby5jb20+DQo+ID4gRGF0ZTog VHVlLCAyNyBTZXAgMjAxMSAxMDozNTo0NyAtMDcwMA0KPiA+IFRvOiAiYXJtZEBpZXRmLm9yZyIg PGFybWRAaWV0Zi5vcmc+DQo+ID4gU3ViamVjdDogUmU6IFthcm1kXSBCb3R0bGVuZWNrIG9mIE92 ZXJsYXkgbmV0d29yayBpbiBhIGRhdGEgY2VudGVyIHdpdGgNCj4gPiBsYXJnZSBudW1iZXIgb2Yg dGVuYW50cy9ob3N0cw0KPiA+DQo+ID4NCj4gPg0KPiA+IEkgYWxzbyBhZ3JlZSB3aXRoIEFub29w 4oCZcyBjb21tZW50LiBCdXQgYXJjaGl0ZWN0dXJhbGx5IHNwZWFraW5nLCB3aGF0IHdlDQo+ID4g YWNoaWV2ZSB3aXRoIGFuIG92ZXJsYXkgbWVjaGFuaXNtIGlzIGEgZm9ybSBvZiBjb21wYXJ0bWVu dGFsaXphdGlvbiAob3INCj4gPiDigJxwYXJ0aXRpb25pbmfigJ0sIOKAnGlzb2xhdGlvbuKAnSwg ZXRjLCBpZiB5b3UgcHJlZmVyKSwgZS5nLiBvZiBhIGdpdmVuIHNldCBvZg0KPiA+IGhvc3RzIGFj cm9zcyB0aGUgZGF0YWNlbnRlci4gwqBXZSBjYW4gYWxzbyBhY2hpZXZlIGNvbXBhcnRtZW50YWxp emF0aW9uIG9mDQo+ID4gdGhlIG5ldHdvcmsgYnkgZGl2aWRpbmcgaXQgaW50byByZWdpb25zIG9y IGNsdXN0ZXJzLCBhbmQgd2UgY2FuIHVzZSBib3RoDQo+ID4gcmVnaW9ucyBhbmQgb3ZlcmxheXMg dG9nZXRoZXIgdG8gZ2V0IGV2ZW4gbW9yZSBjb21wYXJ0bWVudGFsaXphdGlvbi4NCj4gPiDCoFdo YXRldmVyIG1lY2hhbmlzbSB3ZSB1c2UsIGl0IG1pZ2h0IGFsbG93IHVzIHRvIGRpc3RyaWJ1dGUg dGhlIGFkZHJlc3MNCj4gPiByZXNvbHV0aW9uIGZ1bmN0aW9uLiBGb3IgaW5zdGFuY2UsIHdlIGNh biBkaXN0cmlidXRlIEwzIGdhdGV3YXlzIGFjcm9zcw0KPiA+IG11bHRpcGxlIHBoeXNpY2FsIG5v ZGVzIG9yIGRpc3RyaWJ1dGUgQVJQL05EIHByb3h5IGZ1bmN0aW9ucy4NCj4gPg0KPiA+IFRodXMs IGl0IHN0cmlrZXMgbWUgdGhhdCB0aGVyZSBtYXkgYmUgYW4gYXJjaGl0ZWN0dXJhbCBpc3N1ZSBm b3IgQVJNRCB0bw0KPiA+IGV4cGxvcmUuIMKgV2hhdCBpcyB0aGUgcmVsYXRpb25zaGlwIGJldHdl ZW4gY29tcGFydG1lbnRhbGl6YXRpb24gYW5kDQo+IGFkZHJlc3MNCj4gPiByZXNvbHV0aW9uIHNj YWxlPyBJcyBpdCBvbmx5IOKAnHNjYWxl4oCdIHRoYXQgaXMgYWZmZWN0ZWQgYnkNCj4gPiBjb21w YXJ0bWVudGFsaXphdGlvbiwgb3IgYXJlIHRoZXJlIG90aGVyIGFkZHJlc3MgcmVzb2x1dGlvbiBh dHRyaWJ1dGVzIGFsc28NCj4gPiBhZmZlY3RlZD8NCj4gPg0KPiA+IENoZWVycywNCj4gPiAtQmVu c29uDQo+ID4NCj4gPg0KPiA+IE9uIDkvMjcvMTEgOTo0OSBBTSwgIkpvaG4gTWVuZXJpY2siIDxv bWdwb25pZXMzMTQ1QGdtYWlsLmNvbT4gd3JvdGU6DQo+ID4NCj4gPiBBZ3JlZWQhDQo+ID4NCj4g Pg0KPiA+IEpvaG4gTWVuZXJpY2sNCj4gPg0KPiA+IE9uIFR1ZSwgU2VwIDI3LCAyMDExIGF0IDc6 MzYgQU0sIEdhcnkgQmVyZ2VyIDxnYWJlcmdlckBjaXNjby5jb20+IHdyb3RlOg0KPiA+DQo+ID4g KzENCj4gPiBGcm9tOiBWaXNod2FzIE1hbnJhbCA8dmlzaHdhcy5pZXRmQGdtYWlsLmNvbT4NCj4g PiBEYXRlOiBNb24sIDI2IFNlcCAyMDExIDEzOjMxOjE5IC0wNzAwDQo+ID4gVG86IEFub29wIEdo YW53YW5pIDxhbm9vcEBhbHVtbmkuZHVrZS5lZHU+DQo+ID4gQ2M6ICJhcm1kQGlldGYub3JnIiA8 YXJtZEBpZXRmLm9yZz4NCj4gPiBTdWJqZWN0OiBSZTogW2FybWRdIEJvdHRsZW5lY2sgb2YgT3Zl cmxheSBuZXR3b3JrIGluIGEgZGF0YSBjZW50ZXIgd2l0aA0KPiA+IGxhcmdlIG51bWJlciBvZiB0 ZW5hbnRzL2hvc3RzDQo+ID4NCj4gPiBUb3RhbGx5IGFncmVlIHdpdGggQW5vb3AgaGVyZSEhIQ0K PiA+DQo+ID4gLVZpc2h3YXMNCj4gPg0KPiA+IE9uIE1vbiwgU2VwIDI2LCAyMDExIGF0IDEyOjEw IFBNLCBBbm9vcCBHaGFud2FuaQ0KPiA8YW5vb3BAYWx1bW5pLmR1a2UuZWR1Pg0KPiA+IHdyb3Rl Og0KPiA+DQo+ID4gT24gU2F0LCBTZXAgMTcsIDIwMTEgYXQgMTE6MzUgQU0sIExpbmRhIER1bmJh cg0KPiA8bGluZGEuZHVuYmFyQGh1YXdlaS5jb20+DQo+ID4gd3JvdGU6DQo+ID4+DQo+ID4+DQo+ ID4+IFRoZXJlIGFyZSBtYW55IGRyYWZ0cyByZWNlbnRseSBvbiB2YXJpb3VzIE92ZXJsYXkgTmV0 d29yayBzY2hlbWVzIGZvcg0KPiBkYXRhDQo+ID4+IGNlbnRlcnMgd2l0aCBsYXJnZSBudW1iZXIg b2YgdGVuYW50cy9ob3N0cy4gV2hlbiBPdmVybGF5IE5ldHdvcmsgaXMNCj4gPj4gZGVwbG95ZWQN Cj4gPj4gaW4gYSBkYXRhIGNlbnRlciwgbG9jYWwgc3dpdGNoZXMvcm91dGVycyB3aXRoaW4gdGhl IGRhdGEgY2VudGVyIG5ldHdvcmsNCj4gPj4gZG9u4oCZdCBzZWUgdGhlIGhvc3RzIGFkZHJlc3Nl cyBhbnltb3JlIGV4Y2VwdCB0aGUgbm9kZXMgd2hpY2ggcGVyZm9ybQ0KPiB0aGUNCj4gPj4gbmV0 d29yayBhZGRyZXNzIGVuY2Fwc3VsYXRpb24vZGVjYXBzdWxhdGlvbi4NCj4gPj4NCj4gPiAuLi4N Cj4gPj4gRm9yIGVhY2ggaG9zdCAod2hpY2ggcnVucyBvbiBhIFZNKSB3aGljaCBuZWVkIHRvIGNv bW11bmljYXRlIHdpdGgNCj4gZXh0ZXJuYWwNCj4gPj4gd29ybGQsIGl0IGZyZXF1ZW50bHkgc2Vu ZHMgQVJQL05EIHJlcXVlc3RzIHRvIGl0cyBkZWZhdWx0IEdhdGV3YXkgcm91dGVyLg0KPiA+PiBG b3IgYW4gYXBwbGljYXRpb24gd2l0aCBkdWFsIHN0YWNrcyBvbiwgaXQgc2VuZHMgYm90aCBBUlAg JiBORCByZXF1ZXN0cyB0bw0KPiA+PiBpdHMgZGVmYXVsdCBnYXRld2F5IHJvdXRlci4NCj4gPg0K PiA+IEkgdGhpbmsgdGhlIGlzc3VlIG9mIHdoZXRoZXIgb3Igbm90IGFuIG92ZXJsYXkgbmV0d29y ayBpcyBpbiB1c2UgaXMNCj4gPiBvcnRob2dvbmFsDQo+ID4gdG8gdGhlIGlzc3VlIG9mIEFSUC9O RCB0cmFmZmljIHRoYXQgQVJNRCBpcyB0cnlpbmcgdG8gYWRkcmVzcy4NCj4gPiBJZiB3ZSBoYXZl IGFuIG92ZXJsYXkgbmV0d29yayBidWlsdCBvdmVyIGEgZGF0YSBjZW50ZXIgY29yZSwNCj4gPiB3 ZSBzdGlsbCBydW4gaW50byB0aGUgcHJvYmxlbSB3aGVyZSBBUlAvTkQgdHJhZmZpYyBmcm9tIGEg Vk0gbmVlZHMgdG8NCj4gPiBiZSBicm9hZGNhc3QvbXVsdGljYXN0IG92ZXIgdGhlIGNvcmUgKGV2 ZW4gdGhvdWdoIGl0IHVzZXMgdGhlDQo+ID4gb3ZlcmxheSkuIMKgU28gdW5sZXNzDQo+ID4gc29t ZSBvdGhlciBtZWNoYW5pc20gaXMgdXNlZCB0byBtaXRpZ2F0ZSB0aGUgQVJQL05EIHRyYWZmaWMs IHdlIHdvdWxkDQo+ID4gYmUgbm8gYmV0dGVyIHRoYW4gYSByZWd1bGFyIEwyIGZhYnJpYyB0aGF0 IHNwYW5zIHRoZSBlbnRpcmUgZGF0YSBjZW50ZXIgKGF0DQo+ID4gbGVhc3QgZnJvbSB0aGUgc3Rh bmRwb2ludCBvZiB0aGUgdm9sdW1lIG9mIEFSUC9ORCB0cmFmZmljIHRyYXZlcnNpbmcNCj4gPiB0 aGUgY29yZSkuDQo+ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18NCj4gPiBhcm1kIG1haWxpbmcgbGlzdA0KPiA+IGFybWRAaWV0Zi5vcmcNCj4gPiBodHRw czovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCj4gPg0KPiA+IF9fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fIGFybWQgbWFpbGluZyBsaXN0 DQo+ID4gYXJtZEBpZXRmLm9yZyBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZv L2FybWQNCj4gPg0KPiA+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fDQo+ID4gYXJtZCBtYWlsaW5nIGxpc3QNCj4gPiBhcm1kQGlldGYub3JnDQo+ID4gaHR0 cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9hcm1kDQo+ID4NCj4gPg0KPiA+DQo+ ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPg0KPiA+IF9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gYXJtZCBtYWlsaW5nIGxp c3QNCj4gPiBhcm1kQGlldGYub3JnDQo+ID4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9s aXN0aW5mby9hcm1kDQo+ID4NCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXyBhcm1kIG1haWxpbmcgbGlzdA0KPiA+IGFybWRAaWV0Zi5vcmcgaHR0cHM6 Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9hcm1kDQo+ID4NCj4gPiBfX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+IGFybWQgbWFpbGluZyBs aXN0DQo+ID4gYXJtZEBpZXRmLm9yZw0KPiA+IGh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4v bGlzdGluZm8vYXJtZA0KPiA+DQo+ID4NCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18NCj4gYXJtZCBtYWlsaW5nIGxpc3QNCj4gYXJtZEBpZXRmLm9yZw0K PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCg== From joelja@bogus.com Wed Sep 28 00:26:50 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 10A6921F85F1 for ; Wed, 28 Sep 2011 00:26:50 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.542 X-Spam-Level: X-Spam-Status: No, score=-102.542 tagged_above=-999 required=5 tests=[AWL=0.057, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oeoaz5q1XjIA for ; Wed, 28 Sep 2011 00:26:49 -0700 (PDT) Received: from nagasaki.bogus.com (nagasaki.bogus.com [IPv6:2001:418:1::81]) by ietfa.amsl.com (Postfix) with ESMTP id 1C2F021F84BA for ; Wed, 28 Sep 2011 00:26:49 -0700 (PDT) Received: from Zorch.local (c-98-234-216-143.hsd1.ca.comcast.net [98.234.216.143]) (authenticated bits=0) by nagasaki.bogus.com (8.14.4/8.14.4) with ESMTP id p8S7SNhq048724 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT); Wed, 28 Sep 2011 07:28:24 GMT (envelope-from joelja@bogus.com) Message-ID: <4E82CC96.1050608@bogus.com> Date: Wed, 28 Sep 2011 00:28:22 -0700 From: Joel jaeggli User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2 MIME-Version: 1.0 To: Xuxiaohu References: <4A95BA014132FF49AE685FAB4B9F17F610CAC53D@dfweml503-mbx.china.huawei.com> <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE736806@szxeml525-mbs.china.huawei.com> In-Reply-To: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE736806@szxeml525-mbs.china.huawei.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (nagasaki.bogus.com [147.28.0.81]); Wed, 28 Sep 2011 07:28:24 +0000 (UTC) Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 07:26:50 -0000 On 9/28/11 00:06 , Xuxiaohu wrote: > >> -----邮件原件----- 发件人: armd-bounces@ietf.org >> [mailto:armd-bounces@ietf.org] 代表 Christopher Morrow 发送时间: 2011年9月 >> 28日 13:12 收件人: Linda Dunbar 抄送: armd@ietf.org 主题: Re: [armd] >> Bottleneck of Overlay network in a data center with large number of >> tenants/hosts >> >> On Tue, Sep 27, 2011 at 6:10 PM, Linda Dunbar >> wrote: >>> Overlay network enables operators to move VMs from one rack to >>> another without re-configuring IP addresses. >> >> Or, to put it slightly differently.... >> >> o Build a large L3 enabled datacenter(s) network o overlay L2 >> across this (vpls, for one example) o feel free to move about the >> datacenter(s) with your vm-image >> >> There's still the problem at the edge nodes understanding >> what/where the nodes are at L2, there's addtionaly the operations >> problem of managing a changing VPLS (in this example) overlay >> (perhaps vlan 77 did not exist at rack 22, but you need it there >> now... quick, add it to the local network devices!!) > > VPLS could actually be considered as a overlay network (L2 across L3) > solution for multi-tenant data centers due to its many scaling > capabilities. In fact, some vendors are proposing VPLS as a practical > data center network solution to cloud data center providers. As for > how to reduce the configuration and maintenance burdens associated > with the legacy VPLS solutions (i.e., LDP VPLS or BGP VPLS ), IS-IS > VPLS (http://tools.ietf.org/html/draft-xu-l2vpn-vpls-isis-01) is an > ongoing attempt towards that direction. assume for a moment that I have 5000 (PE) devices supporting 100,000 (CE) physical machines with 500,000 VMs on them. takes a few paths to build my mesh. I presuming I end up with more that one vpls per DC as well. > Best regards, Xiaohu > >> It's intriguing, I'm not sure how well it fixes/compartmentalizes >> the problem, I'm also not clear on where the cost just got shifted >> to :) It also assumes that the network gear (potentially the >> bottom-of-the-barrel TOR device we were dealing with before) can >> perform VPLS functions. >> >> this could due with some study it seems, to me at least. >> >> -chris >> >>> Linda >>> >>> >>> >>> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On >>> Behalf Of >> Gary >>> Berger Sent: Tuesday, September 27, 2011 4:45 PM To: Benson >>> Schliesser; armd@ietf.org >>> >>> Subject: Re: [armd] Bottleneck of Overlay network in a data >>> center with large number of tenants/hosts >>> >>> >>> >>> Isn't that the purpose of the network-layer? Why do we need >>> compartmentalization if we have a functioning network layer and >>> proper address-bindings? The whole issue is in the scaling >>> function of dynamic learning which is a side-effect of the >>> mobility requirement in todays data centers. >>> >>> >>> >>> From: Benson Schliesser Date: Tue, 27 Sep >>> 2011 10:35:47 -0700 To: "armd@ietf.org" Subject: >>> Re: [armd] Bottleneck of Overlay network in a data center with >>> large number of tenants/hosts >>> >>> >>> >>> I also agree with Anoop’s comment. But architecturally speaking, >>> what we achieve with an overlay mechanism is a form of >>> compartmentalization (or “partitioning”, “isolation”, etc, if you >>> prefer), e.g. of a given set of hosts across the datacenter. We >>> can also achieve compartmentalization of the network by dividing >>> it into regions or clusters, and we can use both regions and >>> overlays together to get even more compartmentalization. Whatever >>> mechanism we use, it might allow us to distribute the address >>> resolution function. For instance, we can distribute L3 gateways >>> across multiple physical nodes or distribute ARP/ND proxy >>> functions. >>> >>> Thus, it strikes me that there may be an architectural issue for >>> ARMD to explore. What is the relationship between >>> compartmentalization and >> address >>> resolution scale? Is it only “scale” that is affected by >>> compartmentalization, or are there other address resolution >>> attributes also affected? >>> >>> Cheers, -Benson >>> >>> >>> On 9/27/11 9:49 AM, "John Menerick" >>> wrote: >>> >>> Agreed! >>> >>> >>> John Menerick >>> >>> On Tue, Sep 27, 2011 at 7:36 AM, Gary Berger >>> wrote: >>> >>> +1 From: Vishwas Manral Date: Mon, 26 >>> Sep 2011 13:31:19 -0700 To: Anoop Ghanwani >>> Cc: "armd@ietf.org" >>> Subject: Re: [armd] Bottleneck of Overlay network in a data >>> center with large number of tenants/hosts >>> >>> Totally agree with Anoop here!!! >>> >>> -Vishwas >>> >>> On Mon, Sep 26, 2011 at 12:10 PM, Anoop Ghanwani >> >>> wrote: >>> >>> On Sat, Sep 17, 2011 at 11:35 AM, Linda Dunbar >> >>> wrote: >>>> >>>> >>>> There are many drafts recently on various Overlay Network >>>> schemes for >> data >>>> centers with large number of tenants/hosts. When Overlay >>>> Network is deployed in a data center, local switches/routers >>>> within the data center network don’t see the hosts addresses >>>> anymore except the nodes which perform >> the >>>> network address encapsulation/decapsulation. >>>> >>> ... >>>> For each host (which runs on a VM) which need to communicate >>>> with >> external >>>> world, it frequently sends ARP/ND requests to its default >>>> Gateway router. For an application with dual stacks on, it >>>> sends both ARP & ND requests to its default gateway router. >>> >>> I think the issue of whether or not an overlay network is in use >>> is orthogonal to the issue of ARP/ND traffic that ARMD is trying >>> to address. If we have an overlay network built over a data >>> center core, we still run into the problem where ARP/ND traffic >>> from a VM needs to be broadcast/multicast over the core (even >>> though it uses the overlay). So unless some other mechanism is >>> used to mitigate the ARP/ND traffic, we would be no better than a >>> regular L2 fabric that spans the entire data center (at least >>> from the standpoint of the volume of ARP/ND traffic traversing >>> the core). _______________________________________________ armd >>> mailing list armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >>> >>> _______________________________________________ armd mailing >>> list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>> >>> _______________________________________________ armd mailing >>> list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>> >>> >>> >>> ________________________________ >>> >>> _______________________________________________ armd mailing >>> list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>> >>> _______________________________________________ armd mailing >>> list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>> >>> _______________________________________________ armd mailing >>> list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >>> >>> >> _______________________________________________ armd mailing list >> armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > From xuxiaohu@huawei.com Wed Sep 28 00:57:44 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E890521F8CC4 for ; Wed, 28 Sep 2011 00:57:44 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.137 X-Spam-Level: X-Spam-Status: No, score=-2.137 tagged_above=-999 required=5 tests=[AWL=-0.080, BAYES_00=-2.599, CN_BODY_35=0.339, MIME_BASE64_TEXT=1.753, MIME_CHARSET_FARAWAY=2.45, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7qNeDCOWmBER for ; Wed, 28 Sep 2011 00:57:44 -0700 (PDT) Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [119.145.14.67]) by ietfa.amsl.com (Postfix) with ESMTP id 3F81021F8CC1 for ; Wed, 28 Sep 2011 00:57:44 -0700 (PDT) Received: from huawei.com (szxga04-in [172.24.2.12]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS8007IA3KSWN@szxga04-in.huawei.com> for armd@ietf.org; Wed, 28 Sep 2011 16:00:29 +0800 (CST) Received: from szxrg01-dlp.huawei.com ([172.24.2.119]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS800JKP3KS0P@szxga04-in.huawei.com> for armd@ietf.org; Wed, 28 Sep 2011 16:00:28 +0800 (CST) Received: from szxeml207-edg.china.huawei.com ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.1.9-GA) with ESMTP id AEE98036; Wed, 28 Sep 2011 15:59:37 +0800 Received: from SZXEML410-HUB.china.huawei.com (10.82.67.137) by szxeml207-edg.china.huawei.com (172.24.2.59) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 28 Sep 2011 15:59:33 +0800 Received: from SZXEML525-MBS.china.huawei.com ([169.254.8.181]) by szxeml410-hub.china.huawei.com ([10.82.67.137]) with mapi id 14.01.0270.001; Wed, 28 Sep 2011 15:59:27 +0800 Date: Wed, 28 Sep 2011 07:59:25 +0000 From: Xuxiaohu In-reply-to: X-Originating-IP: [10.108.4.115] To: Anoop Ghanwani , Linda Dunbar Message-id: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE73684B@szxeml525-mbs.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=gb2312 Content-language: zh-CN Content-transfer-encoding: base64 Accept-Language: zh-CN, en-US Thread-topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-index: AQHMfIAQACzuTo2v30SdIKBzAk348ZVibQvA X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected References: <4A95BA014132FF49AE685FAB4B9F17F610214AC6@dfweml504-mbx.china.huawei.com> Cc: "armd@ietf.org" Subject: Re: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 07:57:45 -0000 DQo+IC0tLS0t08q8/tStvP4tLS0tLQ0KPiC3orz+yMs6IGFybWQtYm91bmNlc0BpZXRmLm9yZyBb bWFpbHRvOmFybWQtYm91bmNlc0BpZXRmLm9yZ10gtPqx7SBBbm9vcA0KPiBHaGFud2FuaQ0KPiC3 osvNyrG85DogMjAxMcTqOdTCMjfI1SAzOjExDQo+IMrVvP7IyzogTGluZGEgRHVuYmFyDQo+ILOt y806IGFybWRAaWV0Zi5vcmcNCj4g1vfM4jogUmU6IFthcm1kXSBCb3R0bGVuZWNrIG9mIE92ZXJs YXkgbmV0d29yayBpbiBhIGRhdGEgY2VudGVyIHdpdGggbGFyZ2UNCj4gbnVtYmVyIG9mIHRlbmFu dHMvaG9zdHMNCj4gDQo+IE9uIFNhdCwgU2VwIDE3LCAyMDExIGF0IDExOjM1IEFNLCBMaW5kYSBE dW5iYXIgPGxpbmRhLmR1bmJhckBodWF3ZWkuY29tPg0KPiB3cm90ZToNCj4gPg0KPiA+DQo+ID4g VGhlcmUgYXJlIG1hbnkgZHJhZnRzIHJlY2VudGx5IG9uIHZhcmlvdXMgT3ZlcmxheSBOZXR3b3Jr IHNjaGVtZXMgZm9yIGRhdGENCj4gPiBjZW50ZXJzIHdpdGggbGFyZ2UgbnVtYmVyIG9mIHRlbmFu dHMvaG9zdHMuIFdoZW4gT3ZlcmxheSBOZXR3b3JrIGlzDQo+IGRlcGxveWVkDQo+ID4gaW4gYSBk YXRhIGNlbnRlciwgbG9jYWwgc3dpdGNoZXMvcm91dGVycyB3aXRoaW4gdGhlIGRhdGEgY2VudGVy IG5ldHdvcmsNCj4gPiBkb26hr3Qgc2VlIHRoZSBob3N0cyBhZGRyZXNzZXMgYW55bW9yZSBleGNl cHQgdGhlIG5vZGVzIHdoaWNoIHBlcmZvcm0gdGhlDQo+ID4gbmV0d29yayBhZGRyZXNzIGVuY2Fw c3VsYXRpb24vZGVjYXBzdWxhdGlvbi4NCj4gPg0KPiAuLi4NCj4gPiBGb3IgZWFjaCBob3N0ICh3 aGljaCBydW5zIG9uIGEgVk0pIHdoaWNoIG5lZWQgdG8gY29tbXVuaWNhdGUgd2l0aCBleHRlcm5h bA0KPiA+IHdvcmxkLCBpdCBmcmVxdWVudGx5IHNlbmRzIEFSUC9ORCByZXF1ZXN0cyB0byBpdHMg ZGVmYXVsdCBHYXRld2F5IHJvdXRlci4NCj4gPiBGb3IgYW4gYXBwbGljYXRpb24gd2l0aCBkdWFs IHN0YWNrcyBvbiwgaXQgc2VuZHMgYm90aCBBUlAgJiBORCByZXF1ZXN0cyB0bw0KPiA+IGl0cyBk ZWZhdWx0IGdhdGV3YXkgcm91dGVyLg0KPiANCj4gSSB0aGluayB0aGUgaXNzdWUgb2Ygd2hldGhl ciBvciBub3QgYW4gb3ZlcmxheSBuZXR3b3JrIGlzIGluIHVzZSBpcyBvcnRob2dvbmFsDQo+IHRv IHRoZSBpc3N1ZSBvZiBBUlAvTkQgdHJhZmZpYyB0aGF0IEFSTUQgaXMgdHJ5aW5nIHRvIGFkZHJl c3MuDQo+IElmIHdlIGhhdmUgYW4gb3ZlcmxheSBuZXR3b3JrIGJ1aWx0IG92ZXIgYSBkYXRhIGNl bnRlciBjb3JlLA0KPiB3ZSBzdGlsbCBydW4gaW50byB0aGUgcHJvYmxlbSB3aGVyZSBBUlAvTkQg dHJhZmZpYyBmcm9tIGEgVk0gbmVlZHMgdG8NCj4gYmUgYnJvYWRjYXN0L211bHRpY2FzdCBvdmVy IHRoZSBjb3JlIChldmVuIHRob3VnaCBpdCB1c2VzIHRoZQ0KPiBvdmVybGF5KS4gIFNvIHVubGVz cw0KPiBzb21lIG90aGVyIG1lY2hhbmlzbSBpcyB1c2VkIHRvIG1pdGlnYXRlIHRoZSBBUlAvTkQg dHJhZmZpYywgd2Ugd291bGQNCj4gYmUgbm8gYmV0dGVyIHRoYW4gYSByZWd1bGFyIEwyIGZhYnJp YyB0aGF0IHNwYW5zIHRoZSBlbnRpcmUgZGF0YSBjZW50ZXIgKGF0DQo+IGxlYXN0IGZyb20gdGhl IHN0YW5kcG9pbnQgb2YgdGhlIHZvbHVtZSBvZiBBUlAvTkQgdHJhZmZpYyB0cmF2ZXJzaW5nDQo+ IHRoZSBjb3JlKS4NCg0KQWJzb2x1dGVseSBhZ3JlZS4gDQoNCkJlc2lkZXMsIGlmIHRoZSByZWFs IHJlYXNvbiBmb3IgdGhlIFJvdXRlcqGvcyBDUFUgdXRpbGl6YXRpb24gcmVhY2hpbmcgNDAlIChN ZXJpdCBOZXR3b3JrIHN0YXRpc3RpY3MpIG9yIGhpZ2hlciBpbiBtYW55IGRhdGEgY2VudGVycyBp cyBiZWNhdXNlIHRoZXJlIGFyZSB0b28gbWFueSBBUlAvTkQgcmVxdWVzdHMgdG8gdGhlIGdhdGV3 YXkgcm91dGVycywgdGhlIG1hdHVyZWQgQVJQIGNhY2hlIG1lY2hhbmlzbSBjb3VsZCBiZSBkZXBs b3llZCBvbiB0aGUgZWRnZSBuZXR3b3JrIGRldmljZXMgKGUuZy4sIFRvUiBzd2l0Y2hlcykgdG8g c29sdmUgdGhlIGFib3ZlIGlzc3VlLiBpZiB0aGUgYWJvdmUgcmVhc29uIGlzIG5vdCB0cnVlIG9y IG5vdCB0aGUgbWFqb3IgcmVhc29uIGZvciB0aGUgaGlnaCByb3V0ZXIgQ1BVIHV0aWxpemF0aW9u LCBmb3IgZXhhbXBsZSwgaXQgaXMgZHVlIHRvIHRoZSByZWFzb24gdGhhdCB0aGUgcm91dGVycyB0 aGF0IG5lZWQgdG8gcGVyaW9kaWNhbGx5IHNlbmQgQVJQL05EIHJlcXVlc3RzIHRvIGtlZXAgdGhl aXIgQVJQL05EIGNhY2hlIGVudHJpZXMgZnJvbSB0aW1pbmcgb3V0LCB3ZSBuZWVkIHRvIGNvbnNp ZGVyIG90aGVyIHNvbHV0aW9ucy4NCg0KQmVzdCByZWdhcmRzLA0KWGlhb2h1DQoNCj4gX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gYXJtZCBtYWlsaW5n IGxpc3QNCj4gYXJtZEBpZXRmLm9yZw0KPiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xp c3RpbmZvL2FybWQNCg== From xuxiaohu@huawei.com Wed Sep 28 01:01:32 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B851C21F8D03 for ; Wed, 28 Sep 2011 01:01:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.178 X-Spam-Level: X-Spam-Status: No, score=-4.178 tagged_above=-999 required=5 tests=[AWL=1.969, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4, SARE_SUB_ENC_UTF8=0.152] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hf5RaBhUkR4s for ; Wed, 28 Sep 2011 01:01:31 -0700 (PDT) Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [119.145.14.67]) by ietfa.amsl.com (Postfix) with ESMTP id DCA5421F8CDE for ; Wed, 28 Sep 2011 01:01:30 -0700 (PDT) Received: from huawei.com (szxga04-in [172.24.2.12]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS8007L23R5WN@szxga04-in.huawei.com> for armd@ietf.org; Wed, 28 Sep 2011 16:04:18 +0800 (CST) Received: from szxrg01-dlp.huawei.com ([172.24.2.119]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LS800JZ93R5WT@szxga04-in.huawei.com> for armd@ietf.org; Wed, 28 Sep 2011 16:04:17 +0800 (CST) Received: from szxeml208-edg.china.huawei.com ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.1.9-GA) with ESMTP id AEE98236; Wed, 28 Sep 2011 16:04:17 +0800 Received: from SZXEML407-HUB.china.huawei.com (10.82.67.94) by szxeml208-edg.china.huawei.com (172.24.2.60) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 28 Sep 2011 16:03:27 +0800 Received: from SZXEML525-MBS.china.huawei.com ([169.254.8.181]) by szxeml407-hub.china.huawei.com ([10.82.67.94]) with mapi id 14.01.0270.001; Wed, 28 Sep 2011 16:03:24 +0800 Date: Wed, 28 Sep 2011 08:03:23 +0000 From: Xuxiaohu In-reply-to: <4E82CC96.1050608@bogus.com> X-Originating-IP: [10.108.4.115] To: Joel jaeggli Message-id: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE73685B@szxeml525-mbs.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-language: zh-CN Content-transfer-encoding: base64 Accept-Language: zh-CN, en-US Thread-topic: [armd] Bottleneck of Overlay network in a data center with large number of tenants/hosts Thread-index: AQHMfIAQACzuTo2v30SdIKBzAk348ZVfl2KAgAEvFICAACVSgIAADOOAgABFgwCAAAdhgIAAdbcAgACjV+D//4KtAIAAk0Mg X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected References: <4A95BA014132FF49AE685FAB4B9F17F610CAC53D@dfweml503-mbx.china.huawei.com> <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE736806@szxeml525-mbs.china.huawei.com> <4E82CC96.1050608@bogus.com> Cc: "armd@ietf.org" Subject: [armd] =?utf-8?b?562U5aSNOiAgQm90dGxlbmVjayBvZiBPdmVybGF5IG5ldHdv?= =?utf-8?q?rk_in_a_data_center_with_large_number_of_tenants/hosts?= X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2011 08:01:32 -0000 DQo+IC0tLS0t6YKu5Lu25Y6f5Lu2LS0tLS0NCj4g5Y+R5Lu25Lq6OiBhcm1kLWJvdW5jZXNAaWV0 Zi5vcmcgW21haWx0bzphcm1kLWJvdW5jZXNAaWV0Zi5vcmddIOS7o+ihqCBKb2VsDQo+IGphZWdn bGkNCj4g5Y+R6YCB5pe26Ze0OiAyMDEx5bm0OeaciDI45pelIDE1OjI4DQo+IOaUtuS7tuS6ujog WHV4aWFvaHUNCj4g5oqE6YCBOiBhcm1kQGlldGYub3JnDQo+IOS4u+mimDogUmU6IFthcm1kXSBC b3R0bGVuZWNrIG9mIE92ZXJsYXkgbmV0d29yayBpbiBhIGRhdGEgY2VudGVyIHdpdGggbGFyZ2UN Cj4gbnVtYmVyIG9mIHRlbmFudHMvaG9zdHMNCj4gDQo+IE9uIDkvMjgvMTEgMDA6MDYgLCBYdXhp YW9odSB3cm90ZToNCj4gPg0KPiA+PiAtLS0tLemCruS7tuWOn+S7ti0tLS0tIOWPkeS7tuS6ujog YXJtZC1ib3VuY2VzQGlldGYub3JnDQo+ID4+IFttYWlsdG86YXJtZC1ib3VuY2VzQGlldGYub3Jn XSDku6PooaggQ2hyaXN0b3BoZXIgTW9ycm93IOWPkemAgeaXtumXtDoNCj4gMjAxMeW5tDnmnIgN Cj4gPj4gMjjml6UgMTM6MTIg5pS25Lu25Lq6OiBMaW5kYSBEdW5iYXIg5oqE6YCBOiBhcm1kQGll dGYub3JnIOS4u+mimDogUmU6IFthcm1kXQ0KPiA+PiBCb3R0bGVuZWNrIG9mIE92ZXJsYXkgbmV0 d29yayBpbiBhIGRhdGEgY2VudGVyIHdpdGggbGFyZ2UgbnVtYmVyIG9mDQo+ID4+IHRlbmFudHMv aG9zdHMNCj4gPj4NCj4gPj4gT24gVHVlLCBTZXAgMjcsIDIwMTEgYXQgNjoxMCBQTSwgTGluZGEg RHVuYmFyDQo+ID4+IDxsaW5kYS5kdW5iYXJAaHVhd2VpLmNvbT4gd3JvdGU6DQo+ID4+PiBPdmVy bGF5IG5ldHdvcmsgZW5hYmxlcyBvcGVyYXRvcnMgdG8gbW92ZSBWTXMgZnJvbSBvbmUgcmFjayB0 bw0KPiA+Pj4gYW5vdGhlciB3aXRob3V0IHJlLWNvbmZpZ3VyaW5nIElQIGFkZHJlc3Nlcy4NCj4g Pj4NCj4gPj4gT3IsIHRvIHB1dCBpdCBzbGlnaHRseSBkaWZmZXJlbnRseS4uLi4NCj4gPj4NCj4g Pj4gbyBCdWlsZCBhIGxhcmdlIEwzIGVuYWJsZWQgZGF0YWNlbnRlcihzKSBuZXR3b3JrIG8gb3Zl cmxheSBMMg0KPiA+PiBhY3Jvc3MgdGhpcyAodnBscywgZm9yIG9uZSBleGFtcGxlKSBvIGZlZWwg ZnJlZSB0byBtb3ZlIGFib3V0IHRoZQ0KPiA+PiBkYXRhY2VudGVyKHMpIHdpdGggeW91ciB2bS1p bWFnZQ0KPiA+Pg0KPiA+PiBUaGVyZSdzIHN0aWxsIHRoZSBwcm9ibGVtIGF0IHRoZSBlZGdlIG5v ZGVzIHVuZGVyc3RhbmRpbmcNCj4gPj4gd2hhdC93aGVyZSB0aGUgbm9kZXMgYXJlIGF0IEwyLCB0 aGVyZSdzIGFkZHRpb25hbHkgdGhlIG9wZXJhdGlvbnMNCj4gPj4gcHJvYmxlbSBvZiBtYW5hZ2lu ZyBhIGNoYW5naW5nIFZQTFMgKGluIHRoaXMgZXhhbXBsZSkgb3ZlcmxheQ0KPiA+PiAocGVyaGFw cyB2bGFuIDc3IGRpZCBub3QgZXhpc3QgYXQgcmFjayAyMiwgYnV0IHlvdSBuZWVkIGl0IHRoZXJl DQo+ID4+IG5vdy4uLiBxdWljaywgYWRkIGl0IHRvIHRoZSBsb2NhbCBuZXR3b3JrIGRldmljZXMh ISkNCj4gPg0KPiA+IFZQTFMgY291bGQgYWN0dWFsbHkgYmUgY29uc2lkZXJlZCBhcyBhIG92ZXJs YXkgbmV0d29yayAoTDIgYWNyb3NzIEwzKQ0KPiA+IHNvbHV0aW9uIGZvciBtdWx0aS10ZW5hbnQg ZGF0YSBjZW50ZXJzIGR1ZSB0byBpdHMgbWFueSBzY2FsaW5nDQo+ID4gY2FwYWJpbGl0aWVzLiBJ biBmYWN0LCBzb21lIHZlbmRvcnMgYXJlIHByb3Bvc2luZyBWUExTIGFzIGEgcHJhY3RpY2FsDQo+ ID4gZGF0YSBjZW50ZXIgbmV0d29yayBzb2x1dGlvbiB0byBjbG91ZCBkYXRhIGNlbnRlciBwcm92 aWRlcnMuIEFzIGZvcg0KPiA+IGhvdyB0byByZWR1Y2UgdGhlIGNvbmZpZ3VyYXRpb24gYW5kIG1h aW50ZW5hbmNlIGJ1cmRlbnMgYXNzb2NpYXRlZA0KPiA+IHdpdGggdGhlIGxlZ2FjeSBWUExTIHNv bHV0aW9ucyAoaS5lLiwgTERQIFZQTFMgb3IgQkdQIFZQTFMgKSwgSVMtSVMNCj4gPiBWUExTICho dHRwOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC14dS1sMnZwbi12cGxzLWlzaXMtMDEpIGlz IGFuDQo+ID4gb25nb2luZyBhdHRlbXB0IHRvd2FyZHMgdGhhdCBkaXJlY3Rpb24uDQo+IA0KPiBh c3N1bWUgZm9yIGEgbW9tZW50IHRoYXQgSSBoYXZlIDUwMDAgKFBFKSAgZGV2aWNlcyBzdXBwb3J0 aW5nIDEwMCwwMDANCj4gKENFKSBwaHlzaWNhbCBtYWNoaW5lcyB3aXRoIDUwMCwwMDAgVk1zIG9u IHRoZW0uIHRha2VzIGEgZmV3IHBhdGhzIHRvDQo+IGJ1aWxkIG15IG1lc2guIEkgcHJlc3VtaW5n IEkgZW5kIHVwIHdpdGggbW9yZSB0aGF0IG9uZSB2cGxzIHBlciBEQyBhcyB3ZWxsLg0KDQpDYW4g eW91IGV4cGxhaW4gbW9yZSBhYm91dCB5b3VyIHBvaW50Pw0KDQpYaWFvaHUNCg0KPiA+IEJlc3Qg cmVnYXJkcywgWGlhb2h1DQo+ID4NCj4gPj4gSXQncyBpbnRyaWd1aW5nLCBJJ20gbm90IHN1cmUg aG93IHdlbGwgaXQgZml4ZXMvY29tcGFydG1lbnRhbGl6ZXMNCj4gPj4gdGhlIHByb2JsZW0sIEkn bSBhbHNvIG5vdCBjbGVhciBvbiB3aGVyZSB0aGUgY29zdCBqdXN0IGdvdCBzaGlmdGVkDQo+ID4+ IHRvIDopIEl0IGFsc28gYXNzdW1lcyB0aGF0IHRoZSBuZXR3b3JrIGdlYXIgKHBvdGVudGlhbGx5 IHRoZQ0KPiA+PiBib3R0b20tb2YtdGhlLWJhcnJlbCBUT1IgZGV2aWNlIHdlIHdlcmUgZGVhbGlu ZyB3aXRoIGJlZm9yZSkgY2FuDQo+ID4+IHBlcmZvcm0gVlBMUyBmdW5jdGlvbnMuDQo+ID4+DQo+ ID4+IHRoaXMgY291bGQgZHVlIHdpdGggc29tZSBzdHVkeSBpdCBzZWVtcywgdG8gbWUgYXQgbGVh c3QuDQo+ID4+DQo+ID4+IC1jaHJpcw0KPiA+Pg0KPiA+Pj4gTGluZGENCj4gPj4+DQo+ID4+Pg0K PiA+Pj4NCj4gPj4+IEZyb206IGFybWQtYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOmFybWQtYm91 bmNlc0BpZXRmLm9yZ10gT24NCj4gPj4+IEJlaGFsZiBPZg0KPiA+PiBHYXJ5DQo+ID4+PiBCZXJn ZXIgU2VudDogVHVlc2RheSwgU2VwdGVtYmVyIDI3LCAyMDExIDQ6NDUgUE0gVG86IEJlbnNvbg0K PiA+Pj4gU2NobGllc3NlcjsgYXJtZEBpZXRmLm9yZw0KPiA+Pj4NCj4gPj4+IFN1YmplY3Q6IFJl OiBbYXJtZF0gQm90dGxlbmVjayBvZiBPdmVybGF5IG5ldHdvcmsgaW4gYSBkYXRhDQo+ID4+PiBj ZW50ZXIgd2l0aCBsYXJnZSBudW1iZXIgb2YgdGVuYW50cy9ob3N0cw0KPiA+Pj4NCj4gPj4+DQo+ ID4+Pg0KPiA+Pj4gSXNuJ3QgdGhhdCB0aGUgcHVycG9zZSBvZiB0aGUgbmV0d29yay1sYXllcj8g IFdoeSBkbyB3ZSBuZWVkDQo+ID4+PiBjb21wYXJ0bWVudGFsaXphdGlvbiBpZiB3ZSBoYXZlIGEg ZnVuY3Rpb25pbmcgbmV0d29yayBsYXllciBhbmQNCj4gPj4+IHByb3BlciBhZGRyZXNzLWJpbmRp bmdzPyBUaGUgd2hvbGUgaXNzdWUgaXMgaW4gdGhlIHNjYWxpbmcNCj4gPj4+IGZ1bmN0aW9uIG9m IGR5bmFtaWMgbGVhcm5pbmcgd2hpY2ggaXMgYSBzaWRlLWVmZmVjdCBvZiB0aGUNCj4gPj4+IG1v YmlsaXR5IHJlcXVpcmVtZW50IGluIHRvZGF5cyBkYXRhIGNlbnRlcnMuDQo+ID4+Pg0KPiA+Pj4N Cj4gPj4+DQo+ID4+PiBGcm9tOiBCZW5zb24gU2NobGllc3NlciA8YnNjaGxpZXNAY2lzY28uY29t PiBEYXRlOiBUdWUsIDI3IFNlcA0KPiA+Pj4gMjAxMSAxMDozNTo0NyAtMDcwMCBUbzogImFybWRA aWV0Zi5vcmciIDxhcm1kQGlldGYub3JnPiBTdWJqZWN0Og0KPiA+Pj4gUmU6IFthcm1kXSBCb3R0 bGVuZWNrIG9mIE92ZXJsYXkgbmV0d29yayBpbiBhIGRhdGEgY2VudGVyIHdpdGgNCj4gPj4+IGxh cmdlIG51bWJlciBvZiB0ZW5hbnRzL2hvc3RzDQo+ID4+Pg0KPiA+Pj4NCj4gPj4+DQo+ID4+PiBJ IGFsc28gYWdyZWUgd2l0aCBBbm9vcOKAmXMgY29tbWVudC4gQnV0IGFyY2hpdGVjdHVyYWxseSBz cGVha2luZywNCj4gPj4+IHdoYXQgd2UgYWNoaWV2ZSB3aXRoIGFuIG92ZXJsYXkgbWVjaGFuaXNt IGlzIGEgZm9ybSBvZg0KPiA+Pj4gY29tcGFydG1lbnRhbGl6YXRpb24gKG9yIOKAnHBhcnRpdGlv bmluZ+KAnSwg4oCcaXNvbGF0aW9u4oCdLCBldGMsIGlmIHlvdQ0KPiA+Pj4gcHJlZmVyKSwgZS5n LiBvZiBhIGdpdmVuIHNldCBvZiBob3N0cyBhY3Jvc3MgdGhlIGRhdGFjZW50ZXIuICBXZQ0KPiA+ Pj4gY2FuIGFsc28gYWNoaWV2ZSBjb21wYXJ0bWVudGFsaXphdGlvbiBvZiB0aGUgbmV0d29yayBi eSBkaXZpZGluZw0KPiA+Pj4gaXQgaW50byByZWdpb25zIG9yIGNsdXN0ZXJzLCBhbmQgd2UgY2Fu IHVzZSBib3RoIHJlZ2lvbnMgYW5kDQo+ID4+PiBvdmVybGF5cyB0b2dldGhlciB0byBnZXQgZXZl biBtb3JlIGNvbXBhcnRtZW50YWxpemF0aW9uLiBXaGF0ZXZlcg0KPiA+Pj4gbWVjaGFuaXNtIHdl IHVzZSwgaXQgbWlnaHQgYWxsb3cgdXMgdG8gZGlzdHJpYnV0ZSB0aGUgYWRkcmVzcw0KPiA+Pj4g cmVzb2x1dGlvbiBmdW5jdGlvbi4gRm9yIGluc3RhbmNlLCB3ZSBjYW4gZGlzdHJpYnV0ZSBMMyBn YXRld2F5cw0KPiA+Pj4gYWNyb3NzIG11bHRpcGxlIHBoeXNpY2FsIG5vZGVzIG9yIGRpc3RyaWJ1 dGUgQVJQL05EIHByb3h5DQo+ID4+PiBmdW5jdGlvbnMuDQo+ID4+Pg0KPiA+Pj4gVGh1cywgaXQg c3RyaWtlcyBtZSB0aGF0IHRoZXJlIG1heSBiZSBhbiBhcmNoaXRlY3R1cmFsIGlzc3VlIGZvcg0K PiA+Pj4gQVJNRCB0byBleHBsb3JlLiAgV2hhdCBpcyB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4N Cj4gPj4+IGNvbXBhcnRtZW50YWxpemF0aW9uIGFuZA0KPiA+PiBhZGRyZXNzDQo+ID4+PiByZXNv bHV0aW9uIHNjYWxlPyBJcyBpdCBvbmx5IOKAnHNjYWxl4oCdIHRoYXQgaXMgYWZmZWN0ZWQgYnkN Cj4gPj4+IGNvbXBhcnRtZW50YWxpemF0aW9uLCBvciBhcmUgdGhlcmUgb3RoZXIgYWRkcmVzcyBy ZXNvbHV0aW9uDQo+ID4+PiBhdHRyaWJ1dGVzIGFsc28gYWZmZWN0ZWQ/DQo+ID4+Pg0KPiA+Pj4g Q2hlZXJzLCAtQmVuc29uDQo+ID4+Pg0KPiA+Pj4NCj4gPj4+IE9uIDkvMjcvMTEgOTo0OSBBTSwg IkpvaG4gTWVuZXJpY2siIDxvbWdwb25pZXMzMTQ1QGdtYWlsLmNvbT4NCj4gPj4+IHdyb3RlOg0K PiA+Pj4NCj4gPj4+IEFncmVlZCENCj4gPj4+DQo+ID4+Pg0KPiA+Pj4gSm9obiBNZW5lcmljaw0K PiA+Pj4NCj4gPj4+IE9uIFR1ZSwgU2VwIDI3LCAyMDExIGF0IDc6MzYgQU0sIEdhcnkgQmVyZ2Vy IDxnYWJlcmdlckBjaXNjby5jb20+DQo+ID4+PiB3cm90ZToNCj4gPj4+DQo+ID4+PiArMSBGcm9t OiBWaXNod2FzIE1hbnJhbCA8dmlzaHdhcy5pZXRmQGdtYWlsLmNvbT4gRGF0ZTogTW9uLCAyNg0K PiA+Pj4gU2VwIDIwMTEgMTM6MzE6MTkgLTA3MDAgVG86IEFub29wIEdoYW53YW5pDQo+ID4+PiA8 YW5vb3BAYWx1bW5pLmR1a2UuZWR1PiBDYzogImFybWRAaWV0Zi5vcmciIDxhcm1kQGlldGYub3Jn Pg0KPiA+Pj4gU3ViamVjdDogUmU6IFthcm1kXSBCb3R0bGVuZWNrIG9mIE92ZXJsYXkgbmV0d29y ayBpbiBhIGRhdGENCj4gPj4+IGNlbnRlciB3aXRoIGxhcmdlIG51bWJlciBvZiB0ZW5hbnRzL2hv c3RzDQo+ID4+Pg0KPiA+Pj4gVG90YWxseSBhZ3JlZSB3aXRoIEFub29wIGhlcmUhISENCj4gPj4+ DQo+ID4+PiAtVmlzaHdhcw0KPiA+Pj4NCj4gPj4+IE9uIE1vbiwgU2VwIDI2LCAyMDExIGF0IDEy OjEwIFBNLCBBbm9vcCBHaGFud2FuaQ0KPiA+PiA8YW5vb3BAYWx1bW5pLmR1a2UuZWR1Pg0KPiA+ Pj4gd3JvdGU6DQo+ID4+Pg0KPiA+Pj4gT24gU2F0LCBTZXAgMTcsIDIwMTEgYXQgMTE6MzUgQU0s IExpbmRhIER1bmJhcg0KPiA+PiA8bGluZGEuZHVuYmFyQGh1YXdlaS5jb20+DQo+ID4+PiB3cm90 ZToNCj4gPj4+Pg0KPiA+Pj4+DQo+ID4+Pj4gVGhlcmUgYXJlIG1hbnkgZHJhZnRzIHJlY2VudGx5 IG9uIHZhcmlvdXMgT3ZlcmxheSBOZXR3b3JrDQo+ID4+Pj4gc2NoZW1lcyBmb3INCj4gPj4gZGF0 YQ0KPiA+Pj4+IGNlbnRlcnMgd2l0aCBsYXJnZSBudW1iZXIgb2YgdGVuYW50cy9ob3N0cy4gV2hl biBPdmVybGF5DQo+ID4+Pj4gTmV0d29yayBpcyBkZXBsb3llZCBpbiBhIGRhdGEgY2VudGVyLCBs b2NhbCBzd2l0Y2hlcy9yb3V0ZXJzDQo+ID4+Pj4gd2l0aGluIHRoZSBkYXRhIGNlbnRlciBuZXR3 b3JrIGRvbuKAmXQgc2VlIHRoZSBob3N0cyBhZGRyZXNzZXMNCj4gPj4+PiBhbnltb3JlIGV4Y2Vw dCB0aGUgbm9kZXMgd2hpY2ggcGVyZm9ybQ0KPiA+PiB0aGUNCj4gPj4+PiBuZXR3b3JrIGFkZHJl c3MgZW5jYXBzdWxhdGlvbi9kZWNhcHN1bGF0aW9uLg0KPiA+Pj4+DQo+ID4+PiAuLi4NCj4gPj4+ PiBGb3IgZWFjaCBob3N0ICh3aGljaCBydW5zIG9uIGEgVk0pIHdoaWNoIG5lZWQgdG8gY29tbXVu aWNhdGUNCj4gPj4+PiB3aXRoDQo+ID4+IGV4dGVybmFsDQo+ID4+Pj4gd29ybGQsIGl0IGZyZXF1 ZW50bHkgc2VuZHMgQVJQL05EIHJlcXVlc3RzIHRvIGl0cyBkZWZhdWx0DQo+ID4+Pj4gR2F0ZXdh eSByb3V0ZXIuIEZvciBhbiBhcHBsaWNhdGlvbiB3aXRoIGR1YWwgc3RhY2tzIG9uLCBpdA0KPiA+ Pj4+IHNlbmRzIGJvdGggQVJQICYgTkQgcmVxdWVzdHMgdG8gaXRzIGRlZmF1bHQgZ2F0ZXdheSBy b3V0ZXIuDQo+ID4+Pg0KPiA+Pj4gSSB0aGluayB0aGUgaXNzdWUgb2Ygd2hldGhlciBvciBub3Qg YW4gb3ZlcmxheSBuZXR3b3JrIGlzIGluIHVzZQ0KPiA+Pj4gaXMgb3J0aG9nb25hbCB0byB0aGUg aXNzdWUgb2YgQVJQL05EIHRyYWZmaWMgdGhhdCBBUk1EIGlzIHRyeWluZw0KPiA+Pj4gdG8gYWRk cmVzcy4gSWYgd2UgaGF2ZSBhbiBvdmVybGF5IG5ldHdvcmsgYnVpbHQgb3ZlciBhIGRhdGENCj4g Pj4+IGNlbnRlciBjb3JlLCB3ZSBzdGlsbCBydW4gaW50byB0aGUgcHJvYmxlbSB3aGVyZSBBUlAv TkQgdHJhZmZpYw0KPiA+Pj4gZnJvbSBhIFZNIG5lZWRzIHRvIGJlIGJyb2FkY2FzdC9tdWx0aWNh c3Qgb3ZlciB0aGUgY29yZSAoZXZlbg0KPiA+Pj4gdGhvdWdoIGl0IHVzZXMgdGhlIG92ZXJsYXkp LiAgU28gdW5sZXNzIHNvbWUgb3RoZXIgbWVjaGFuaXNtIGlzDQo+ID4+PiB1c2VkIHRvIG1pdGln YXRlIHRoZSBBUlAvTkQgdHJhZmZpYywgd2Ugd291bGQgYmUgbm8gYmV0dGVyIHRoYW4gYQ0KPiA+ Pj4gcmVndWxhciBMMiBmYWJyaWMgdGhhdCBzcGFucyB0aGUgZW50aXJlIGRhdGEgY2VudGVyIChh dCBsZWFzdA0KPiA+Pj4gZnJvbSB0aGUgc3RhbmRwb2ludCBvZiB0aGUgdm9sdW1lIG9mIEFSUC9O RCB0cmFmZmljIHRyYXZlcnNpbmcNCj4gPj4+IHRoZSBjb3JlKS4gX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18gYXJtZA0KPiA+Pj4gbWFpbGluZyBsaXN0IGFy bWRAaWV0Zi5vcmcNCj4gPj4+IGh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8v YXJtZA0KPiA+Pj4NCj4gPj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fIGFybWQgbWFpbGluZw0KPiA+Pj4gbGlzdCBhcm1kQGlldGYub3JnIGh0dHBzOi8v d3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vYXJtZA0KPiA+Pj4NCj4gPj4+IF9fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fIGFybWQgbWFpbGluZw0KPiA+ Pj4gbGlzdCBhcm1kQGlldGYub3JnIGh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGlu Zm8vYXJtZA0KPiA+Pj4NCj4gPj4+DQo+ID4+Pg0KPiA+Pj4gX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18NCj4gPj4+DQo+ID4+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXyBhcm1kIG1haWxpbmcNCj4gPj4+IGxpc3QgYXJtZEBpZXRmLm9yZyBo dHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCj4gPj4+DQo+ID4+PiBf X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXyBhcm1kIG1haWxp bmcNCj4gPj4+IGxpc3QgYXJtZEBpZXRmLm9yZyBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFu L2xpc3RpbmZvL2FybWQNCj4gPj4+DQo+ID4+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXyBhcm1kIG1haWxpbmcNCj4gPj4+IGxpc3QgYXJtZEBpZXRmLm9y ZyBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCj4gPj4+DQo+ID4+ Pg0KPiA+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXyBh cm1kIG1haWxpbmcgbGlzdA0KPiA+PiBhcm1kQGlldGYub3JnIGh0dHBzOi8vd3d3LmlldGYub3Jn L21haWxtYW4vbGlzdGluZm8vYXJtZA0KPiA+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fIGFybWQgbWFpbGluZyBsaXN0DQo+ID4gYXJtZEBpZXRmLm9yZyBo dHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCj4gPg0KPiANCj4gX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gYXJtZCBtYWls aW5nIGxpc3QNCj4gYXJtZEBpZXRmLm9yZw0KPiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFu L2xpc3RpbmZvL2FybWQNCg==