From kim.changhoon@gmail.com Sun Jul 31 18:03:12 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 11FB35E8001 for ; Sun, 31 Jul 2011 18:03:12 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.976 X-Spam-Level: X-Spam-Status: No, score=-2.976 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jtsvd-n9J0+m for ; Sun, 31 Jul 2011 18:03:11 -0700 (PDT) Received: from mail-yw0-f44.google.com (mail-yw0-f44.google.com [209.85.213.44]) by ietfa.amsl.com (Postfix) with ESMTP id CC7D421F8545 for ; Sun, 31 Jul 2011 18:03:10 -0700 (PDT) Received: by ywm21 with SMTP id 21so560557ywm.31 for ; Sun, 31 Jul 2011 18:03:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=TK6YbofwuKgidUZBVxQTjaWrrem+wh+Utfhy7CyvCoo=; b=cZsKB56akZk5i6Y09OssAUZ6splYCnqOaHcI7eaXsh2MYV3rM7IpF9yMdp4T04sfoX YRFT+YP5sbyoIdhHbIUunC6CdxotEpSUVay8oeNdlI3paIG0AbrJp8nU+nwzgLIXnlID JzkZpnX0vnnN6o7FSa3Qe44RY4mzOsL68vcec= MIME-Version: 1.0 Received: by 10.236.173.70 with SMTP id u46mr2599055yhl.65.1312160595550; Sun, 31 Jul 2011 18:03:15 -0700 (PDT) Sender: kim.changhoon@gmail.com Received: by 10.236.108.11 with HTTP; Sun, 31 Jul 2011 18:03:15 -0700 (PDT) In-Reply-To: <4E329CB2.8010908@cs.illinois.edu> References: <4E329CB2.8010908@cs.illinois.edu> Date: Sun, 31 Jul 2011 18:03:15 -0700 X-Google-Sender-Auth: JyA_5FXFTAd8X6D80AWR3C0_KQ8 Message-ID: From: Changhoon Kim To: armd@ietf.org Content-Type: multipart/alternative; boundary=20cf305e25392fa09404a9673281 Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2011 19:44:12 -0000 --20cf305e25392fa09404a9673281 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, I'd like to add a minor correction to the original email about SEATTLE. - "For example, for ARP queries, the key is an IP address and the value includes the IP address and the host=92s location". We meant "For example, = for ARP queries, the key is an IP address and the value includes the _MAC_ address and the host=92s location". Also, for those who are interested in the implementation details of SEATTLE= , we have another write-up that include a full section explaining our prototype SEATTLE switch. http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf Thanks. -- Chang On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar wrote: > > Hi, > > I've been watching this list with a lot of interest. We wanted to post fo= r your consideration a protocol called SEATTLE we've been developing, which targets large improvements in scalability of layer-2 in the context of data centers/cloud computing by eliminating the need for broadcasts. We think SEATTLE addresses some of the goals in ARMD's charter. More details below, but we'd be quite interested to discuss/answer questions if there's interest. > > SEATTLE achieves the same configuration-free properties as Ethernet bridging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need for network-wide broadcasting, achieving data-plane control overheads that grow logarithmically with network size. SEATTLE is backwards-compatible with existing Ethernets, simplifying incremental deployment, and supporting existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also supports more advanced (optional) features, such as the ability to search and look up services based on text strings, more flexible routing an= d more control over layer-2 routing policies, and the ability to anycast and load balance directly over named services. SEATTLE has two main features: > > Distributed in-network directory service: The switches collectively run a directory service that handles ARP and DHCP requests, as well as packets sent to unknown destination addresses. The directory service leverages consistent hashing and the link-state routing protocol to form a one-hop distributed hash table. The directory is a key-value store, where the hash of the key determines the switch(es) responsible for storing the information. For example, for ARP queries, the key is an IP address and the value includes the IP address and the host=92s location. When an ingress switch cannot handle an ARP/DHCP request or a data packet directly, the switch computes the hash and directs the packet through the responsible intermediate switch. The intermediate switch can response to ARP and DHCP queries, and direct data packets onward to their egress switch while returning the destination host=92s location to the ingress switch. > > Reactive caching and invalidation: To minimize the portion of traffic directed over longer paths, an ingress switch reactively caches the responses from the intermediate switch. For example, while an intermediate switch may handle an ARP query, the data packets typically flow directly along the shortest path. In addition, other hosts connected to the same ingress switch benefit from the cached information. However, the cached information becomes stale if a host moves to a new location. SEATTLE includes an efficient cache invalidation protocol that reactively updates the stale cache entries when a packet wrongly travels to the old egress switch. > > The first feature leads to good performance (by keeping traffic in the data plane, and avoiding broadcast) and self scaling (by having the directory service naturally scale with the size of the network), and the second ensures fast recovery after failures and migration. > > Our writeup on SEATTLE: > > http://www.cs.princeton.edu/~jrex/papers/seattle08.pdf > > presents more details on this protocol, our experiences designing and implementing a system prototype, and a performance evaluation of the protocol through simulations and deployment in a testbed. > > -- Matt > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --20cf305e25392fa09404a9673281 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi,
=A0
I'd like to add a minor correction to = the original email about SEATTLE.
=A0
- "For examp= le, for ARP queries, the key is an IP address and the value includes the IP= address and the host=92s location". We meant "For example, for A= RP queries, the key is an IP address and the value includes the=A0_MAC_ add= ress and the host=92s location".
=A0
Also, for those who are interested in the implementation= details of SEATTLE, we have another write-up that include a full section e= xplaining our prototype SEATTLE switch.
=A0
=A0
Thanks.
=A0
-- Chang
=A0
=A0=A0
On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar &= lt;caesar@cs.illinois.edu>= wrote:
>
> Hi,
>
> I've been wa= tching this list with a lot of interest. We wanted to post for your conside= ration a protocol called SEATTLE we've been developing, which targets l= arge improvements in scalability of layer-2 in the context of data centers/= cloud computing by eliminating the need for broadcasts. We think SEATTLE ad= dresses some of the goals in ARMD's charter. More details below, but we= 'd be quite interested to discuss/answer questions if there's inter= est.
>
> SEATTLE achieves the same configuration-free prope= rties as Ethernet bridging yet scales to large networks. Unlike other propo= sals (e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely elim= inates the need for network-wide broadcasting, achieving data-plane control= overheads that grow logarithmically with network size. SEATTLE is backward= s-compatible with existing Ethernets, simplifying incremental deployment, a= nd supporting existing Ethernet functions (e.g., VLANs and host bootstrappi= ng). SEATTLE also supports more advanced (optional) features, such as the a= bility to search and look up services based on text strings, more flexible = routing and more control over layer-2 routing policies, and the ability to = anycast and load balance directly over named services. SEATTLE has two main= features:
>
> Distributed in-network directory service: The swit= ches collectively run a directory service that handles ARP and DHCP request= s, as well as packets sent to unknown destination addresses. The directory = service leverages consistent hashing and the link-state routing protocol to= form a one-hop distributed hash table. The directory is a key-value store,= where the hash of the key determines the switch(es) responsible for storin= g the information. For example, for ARP queries, the key is an IP address a= nd the value includes the IP address and the host=92s location. When an ing= ress switch cannot handle an ARP/DHCP request or a data packet directly, th= e switch computes the hash and directs the packet through the responsible i= ntermediate switch. The intermediate switch can response to ARP and DHCP qu= eries, and direct data packets onward to their egress switch while returnin= g the destination host=92s location to the ingress switch.
>
> Reactive caching and invalidation: To minimize the= portion of traffic directed over longer paths, an ingress switch reactivel= y caches the responses from the intermediate switch. For example, while an = intermediate switch may handle an ARP query, the data packets typically flo= w directly along the shortest path. In addition, other hosts connected to t= he same ingress switch benefit from the cached information. However, the ca= ched information becomes stale if a host moves to a new location. SEATTLE i= ncludes an efficient cache invalidation protocol that reactively updates th= e stale cache entries when a packet wrongly travels to the old egress switc= h.
>
> The first feature leads to good performance (by ke= eping traffic in the data plane, and avoiding broadcast) and self scaling (= by having the directory service naturally scale with the size of the networ= k), and the second ensures fast recovery after failures and migration.
>
> Our writeup on SEATTLE:
>
>
> presents more details on this protocol, our experiences designing= and implementing a system prototype, and a performance evaluation of the p= rotocol through simulations and deployment in a testbed.
>
> -- Matt
>
>
> ____________= ___________________________________
> armd mailing list
<= div>> armd@ietf.org
=A0
=A0
--20cf305e25392fa09404a9673281-- From linda.dunbar@huawei.com Tue Aug 2 07:19:52 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 87B2421F8841 for ; Tue, 2 Aug 2011 07:19:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xGKVZecNKwvU for ; Tue, 2 Aug 2011 07:19:49 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id E3ED421F8801 for ; Tue, 2 Aug 2011 07:19:48 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LPB00KOI153S9@usaga04-in.huawei.com> for armd@ietf.org; Tue, 02 Aug 2011 09:19:51 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LPB009MA152XI@usaga04-in.huawei.com> for armd@ietf.org; Tue, 02 Aug 2011 09:19:51 -0500 (CDT) Received: from DFWEML401-HUB.china.huawei.com (10.193.5.101) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 02 Aug 2011 07:19:51 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.202]) by DFWEML401-HUB.china.huawei.com ([fe80::f07f:889f:78ef:8df3%13]) with mapi id 14.01.0270.001; Tue, 02 Aug 2011 07:19:50 -0700 Date: Tue, 02 Aug 2011 14:19:48 +0000 From: Linda Dunbar In-reply-to: X-Originating-IP: [10.47.135.70] To: Changhoon Kim , "armd@ietf.org" , Matthew Caesar Message-id: <4A95BA014132FF49AE685FAB4B9F17F60517CDCC@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_9vfezoaFQRz1CDavCSIPrg)" Content-language: en-US Accept-Language: en-US Thread-topic: [armd] For your consideration: SEATTLE Thread-index: AQHMTeTdALAlQ+jY2E+la4kw6nwQL5UHpySAgAH0X8A= X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <4E329CB2.8010908@cs.illinois.edu> Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2011 14:19:52 -0000 --Boundary_(ID_9vfezoaFQRz1CDavCSIPrg) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Kim and Mattew, Thank you very much for posting the SEATTLE detail to the ARMD list. SEATTLE does present a possible approach to scale address resolution scheme for Large Layer 2 network. ARMD WG is currently chartered to identify the Address Resolution issues in Data Center network, document best practices and workaround for Address Resolution scaling issues. At the last week's ARMD WG session, many Address Resolution scaling issues were discussed. Many people do feel that it is more productive for the WG to focus on a subset of problems which are solvable, so that the WG won't waste energy on too broad issues associated with Data Center. Is it be possible for you to write an IETF draft to describe the SEATTLE approach for next IETF? Thanks, Linda Dunbar From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Changhoon Kim Sent: Sunday, July 31, 2011 8:03 PM To: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE Hi, I'd like to add a minor correction to the original email about SEATTLE. - "For example, for ARP queries, the key is an IP address and the value includes the IP address and the host's location". We meant "For example, for ARP queries, the key is an IP address and the value includes the _MAC_ address and the host's location". Also, for those who are interested in the implementation details of SEATTLE, we have another write-up that include a full section explaining our prototype SEATTLE switch. http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf Thanks. -- Chang On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar > wrote: > > Hi, > > I've been watching this list with a lot of interest. We wanted to post for your consideration a protocol called SEATTLE we've been developing, which targets large improvements in scalability of layer-2 in the context of data centers/cloud computing by eliminating the need for broadcasts. We think SEATTLE addresses some of the goals in ARMD's charter. More details below, but we'd be quite interested to discuss/answer questions if there's interest. > > SEATTLE achieves the same configuration-free properties as Ethernet bridging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need for network-wide broadcasting, achieving data-plane control overheads that grow logarithmically with network size. SEATTLE is backwards-compatible with existing Ethernets, simplifying incremental deployment, and supporting existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also supports more advanced (optional) features, such as the ability to search and look up services based on text strings, more flexible routing and more control over layer-2 routing policies, and the ability to anycast and load balance directly over named services. SEATTLE has two main features: > > Distributed in-network directory service: The switches collectively run a directory service that handles ARP and DHCP requests, as well as packets sent to unknown destination addresses. The directory service leverages consistent hashing and the link-state routing protocol to form a one-hop distributed hash table. The directory is a key-value store, where the hash of the key determines the switch(es) responsible for storing the information. For example, for ARP queries, the key is an IP address and the value includes the IP address and the host's location. When an ingress switch cannot handle an ARP/DHCP request or a data packet directly, the switch computes the hash and directs the packet through the responsible intermediate switch. The intermediate switch can response to ARP and DHCP queries, and direct data packets onward to their egress switch while returning the destination host's location to the ingress switch. > > Reactive caching and invalidation: To minimize the portion of traffic directed over longer paths, an ingress switch reactively caches the responses from the intermediate switch. For example, while an intermediate switch may handle an ARP query, the data packets typically flow directly along the shortest path. In addition, other hosts connected to the same ingress switch benefit from the cached information. However, the cached information becomes stale if a host moves to a new location. SEATTLE includes an efficient cache invalidation protocol that reactively updates the stale cache entries when a packet wrongly travels to the old egress switch. > > The first feature leads to good performance (by keeping traffic in the data plane, and avoiding broadcast) and self scaling (by having the directory service naturally scale with the size of the network), and the second ensures fast recovery after failures and migration. > > Our writeup on SEATTLE: > > http://www.cs.princeton.edu/~jrex/papers/seattle08.pdf > > presents more details on this protocol, our experiences designing and implementing a system prototype, and a performance evaluation of the protocol through simulations and deployment in a testbed. > > -- Matt > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --Boundary_(ID_9vfezoaFQRz1CDavCSIPrg) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

Kim and Mattew,

 

Thank you very much for posting the SEATTLE detail to the ARMD list.

SEATTLE does present a possible approach to scale address resolution scheme for Large Layer 2 network.

ARMD WG is currently chartered to identify the Address Resolution issues in Data Center network, document best practices and workaround for Address Resolution scaling issues. At the last week’s ARMD WG session, many Address Resolution scaling issues were discussed. 

Many people do feel that it is more productive for the WG to focus on a subset of problems which are solvable, so that the WG won’t  waste energy on too broad issues associated with Data Center.

 

Is it be possible for you to write an IETF draft to describe the SEATTLE approach for next IETF?

 

Thanks,

Linda Dunbar

 

 

From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Changhoon Kim
Sent: Sunday, July 31, 2011 8:03 PM
To: armd@ietf.org
Subject: Re: [armd] For your consideration: SEATTLE

 

Hi,

 

I'd like to add a minor correction to the original email about SEATTLE.

 

- "For example, for ARP queries, the key is an IP address and the value includes the IP address and the host’s location". We meant "For example, for ARP queries, the key is an IP address and the value includes the _MAC_ address and the host’s location".

 

Also, for those who are interested in the implementation details of SEATTLE, we have another write-up that include a full section explaining our prototype SEATTLE switch.

 

 

Thanks.

 

-- Chang

 

  

On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar <caesar@cs.illinois.edu> wrote:

> 

> Hi,

> 

> I've been watching this list with a lot of interest. We wanted to post for your consideration a protocol called SEATTLE we've been developing, which targets large improvements in scalability of layer-2 in the context of data centers/cloud computing by eliminating the need for broadcasts. We think SEATTLE addresses some of the goals in ARMD's charter. More details below, but we'd be quite interested to discuss/answer questions if there's interest.

> 

> SEATTLE achieves the same configuration-free properties as Ethernet bridging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need for network-wide broadcasting, achieving data-plane control overheads that grow logarithmically with network size. SEATTLE is backwards-compatible with existing Ethernets, simplifying incremental deployment, and supporting existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also supports more advanced (optional) features, such as the ability to search and look up services based on text strings, more flexible routing and more control over layer-2 routing policies, and the ability to anycast and load balance directly over named services. SEATTLE has two main features:

> 

> Distributed in-network directory service: The switches collectively run a directory service that handles ARP and DHCP requests, as well as packets sent to unknown destination addresses. The directory service leverages consistent hashing and the link-state routing protocol to form a one-hop distributed hash table. The directory is a key-value store, where the hash of the key determines the switch(es) responsible for storing the information. For example, for ARP queries, the key is an IP address and the value includes the IP address and the host’s location. When an ingress switch cannot handle an ARP/DHCP request or a data packet directly, the switch computes the hash and directs the packet through the responsible intermediate switch. The intermediate switch can response to ARP and DHCP queries, and direct data packets onward to their egress switch while returning the destination host’s location to the ingress switch.

> 

> Reactive caching and invalidation: To minimize the portion of traffic directed over longer paths, an ingress switch reactively caches the responses from the intermediate switch. For example, while an intermediate switch may handle an ARP query, the data packets typically flow directly along the shortest path. In addition, other hosts connected to the same ingress switch benefit from the cached information. However, the cached information becomes stale if a host moves to a new location. SEATTLE includes an efficient cache invalidation protocol that reactively updates the stale cache entries when a packet wrongly travels to the old egress switch.

> 

> The first feature leads to good performance (by keeping traffic in the data plane, and avoiding broadcast) and self scaling (by having the directory service naturally scale with the size of the network), and the second ensures fast recovery after failures and migration.

> 

> Our writeup on SEATTLE:

> 

> 

> presents more details on this protocol, our experiences designing and implementing a system prototype, and a performance evaluation of the protocol through simulations and deployment in a testbed.

> 

> -- Matt

> 

> 

> _______________________________________________

> armd mailing list

 

 

--Boundary_(ID_9vfezoaFQRz1CDavCSIPrg)-- From prvs=1195030359=hshah@ciena.com Tue Aug 2 09:23:26 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1C0BB11E8083 for ; Tue, 2 Aug 2011 09:23:26 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.013 X-Spam-Level: X-Spam-Status: No, score=0.013 tagged_above=-999 required=5 tests=[AWL=-0.143, BAYES_00=-2.599, EXTRA_MPART_TYPE=1, HTML_MESSAGE=0.001, IP_NOT_FRIENDLY=0.334, SARE_GIF_ATTACH=1.42] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xwl9GdtusrqH for ; Tue, 2 Aug 2011 09:23:22 -0700 (PDT) Received: from mx0a-00103a01.pphosted.com (mx0a-00103a01.pphosted.com [67.231.144.234]) by ietfa.amsl.com (Postfix) with ESMTP id 3299E11E807B for ; Tue, 2 Aug 2011 09:23:22 -0700 (PDT) Received: from pps.filterd (m0000419 [127.0.0.1]) by mx0a-00103a01.pphosted.com (8.14.3/8.14.3) with SMTP id p72GHETg003448; Tue, 2 Aug 2011 12:23:13 -0400 Received: from mdwexght01.ciena.com (LIN1-118-36-28.ciena.com [63.118.36.28]) by mx0a-00103a01.pphosted.com with ESMTP id xxm6cg4a4-2 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 02 Aug 2011 12:23:13 -0400 Received: from MDWEXGMB02.ciena.com ([::1]) by MDWEXGHT01.ciena.com ([::1]) with mapi; Tue, 2 Aug 2011 12:23:11 -0400 From: "Shah, Himanshu" To: Linda Dunbar , Changhoon Kim , "armd@ietf.org" , Matthew Caesar Content-Class: urn:content-classes:message Date: Tue, 2 Aug 2011 12:23:09 -0400 Thread-Topic: [armd] For your consideration: SEATTLE Thread-Index: AQHMTeTdALAlQ+jY2E+la4kw6nwQL5UHpySAgAH0X8CAACgs0A== Message-ID: References: <4E329CB2.8010908@cs.illinois.edu> <4A95BA014132FF49AE685FAB4B9F17F60517CDCC@dfweml503-mbx.china.huawei.com> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F60517CDCC@dfweml503-mbx.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US x-tm-as-product-ver: SMEX-10.0.0.1412-6.800.1017-18300.000 x-tm-as-result: No--57.363600-8.000000-31 x-tm-as-user-approved-sender: No x-tm-as-user-blocked-sender: No Content-Type: multipart/related; boundary="_004_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_"; type="multipart/alternative" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.4.6813, 1.0.211, 0.0.0000 definitions=2011-08-02_04:2011-08-02, 2011-08-02, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=6 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1108020148 Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2011 16:23:26 -0000 --_004_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_ Content-Type: multipart/alternative; boundary="_000_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_" --_000_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Linda - I specifically asked the question at the mike about solutions related work,= in the last week's WG meeting, and if you recall, Ron Bonica, said EXPLICITLY that solutions CAN NOT BE di= scussed in this WG as this WG belongs to OPS area, and only requirement related discussions ar= e in charter. Has the scope of WG charter changed after the meetings were over?? Thanks, himanshu [cid:image8005c2.gif@bfb2481e.9e5b4b0f] From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Lin= da Dunbar Sent: Tuesday, August 02, 2011 10:20 AM To: Changhoon Kim; armd@ietf.org; Matthew Caesar Subject: Re: [armd] For your consideration: SEATTLE Kim and Mattew, Thank you very much for posting the SEATTLE detail to the ARMD list. SEATTLE does present a possible approach to scale address resolution scheme= for Large Layer 2 network. ARMD WG is currently chartered to identify the Address Resolution issues in= Data Center network, document best practices and workaround for Address Re= solution scaling issues. At the last week's ARMD WG session, many Address R= esolution scaling issues were discussed. Many people do feel that it is more productive for the WG to focus on a sub= set of problems which are solvable, so that the WG won't waste energy on t= oo broad issues associated with Data Center. Is it be possible for you to write an IETF draft to describe the SEATTLE ap= proach for next IETF? Thanks, Linda Dunbar From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Cha= nghoon Kim Sent: Sunday, July 31, 2011 8:03 PM To: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE Hi, I'd like to add a minor correction to the original email about SEATTLE. - "For example, for ARP queries, the key is an IP address and the value inc= ludes the IP address and the host's location". We meant "For example, for A= RP queries, the key is an IP address and the value includes the _MAC_ addre= ss and the host's location". Also, for those who are interested in the implementation details of SEATTLE= , we have another write-up that include a full section explaining our proto= type SEATTLE switch. http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf Thanks. -- Chang On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar > wrote: > > Hi, > > I've been watching this list with a lot of interest. We wanted to post fo= r your consideration a protocol called SEATTLE we've been developing, which= targets large improvements in scalability of layer-2 in the context of dat= a centers/cloud computing by eliminating the need for broadcasts. We think = SEATTLE addresses some of the goals in ARMD's charter. More details below, = but we'd be quite interested to discuss/answer questions if there's interes= t. > > SEATTLE achieves the same configuration-free properties as Ethernet bridg= ing yet scales to large networks. Unlike other proposals (e.g., TRILL, Rbri= dges, Viking, SmartBridges), SEATTLE completely eliminates the need for net= work-wide broadcasting, achieving data-plane control overheads that grow lo= garithmically with network size. SEATTLE is backwards-compatible with exist= ing Ethernets, simplifying incremental deployment, and supporting existing = Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also suppo= rts more advanced (optional) features, such as the ability to search and lo= ok up services based on text strings, more flexible routing and more contro= l over layer-2 routing policies, and the ability to anycast and load balanc= e directly over named services. SEATTLE has two main features: > > Distributed in-network directory service: The switches collectively run a= directory service that handles ARP and DHCP requests, as well as packets s= ent to unknown destination addresses. The directory service leverages consi= stent hashing and the link-state routing protocol to form a one-hop distrib= uted hash table. The directory is a key-value store, where the hash of the = key determines the switch(es) responsible for storing the information. For = example, for ARP queries, the key is an IP address and the value includes t= he IP address and the host's location. When an ingress switch cannot handle= an ARP/DHCP request or a data packet directly, the switch computes the has= h and directs the packet through the responsible intermediate switch. The i= ntermediate switch can response to ARP and DHCP queries, and direct data pa= ckets onward to their egress switch while returning the destination host's = location to the ingress switch. > > Reactive caching and invalidation: To minimize the portion of traffic dir= ected over longer paths, an ingress switch reactively caches the responses = from the intermediate switch. For example, while an intermediate switch may= handle an ARP query, the data packets typically flow directly along the sh= ortest path. In addition, other hosts connected to the same ingress switch = benefit from the cached information. However, the cached information become= s stale if a host moves to a new location. SEATTLE includes an efficient ca= che invalidation protocol that reactively updates the stale cache entries w= hen a packet wrongly travels to the old egress switch. > > The first feature leads to good performance (by keeping traffic in the da= ta plane, and avoiding broadcast) and self scaling (by having the directory= service naturally scale with the size of the network), and the second ensu= res fast recovery after failures and migration. > > Our writeup on SEATTLE: > > http://www.cs.princeton.edu/~jrex/papers/seattle08.pdf > > presents more details on this protocol, our experiences designing and imp= lementing a system prototype, and a performance evaluation of the protocol = through simulations and deployment in a testbed. > > -- Matt > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --_000_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <= meta content=3D"Microsoft Word 12 (filtered medium)" name=3DGenerator>
Linda –

 

I sp= ecifically asked the question at the mike about solutions related work, in = the last week’s WG meeting,

and if you recall, Ron Bonica, said EXPLICITLY that solutions CAN N= OT BE discussed in this WG

as this WG belongs to OPS area, and only requirement related discussions a= re in charter.

 =

Has the scope of WG charter ch= anged after the meetings were over??

 

Thanks,=

himanshu

 


3D"ciena

From: armd-bounc= es@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Linda Dunbar=
Sent: Tuesday, August 02, 2011 10:20 AM
To: Changhoon = Kim; armd@ietf.org; Matthew Caesar
Subject: Re: [armd] For your c= onsideration: SEATTLE

 

Kim and Mattew,

 

Thank you very much for posting the SEATTLE detail to= the ARMD list.

SEATTLE d= oes present a possible approach to scale address resolution scheme for Larg= e Layer 2 network.

ARMD= WG is currently chartered to identify the Address Resolution issues in Dat= a Center network, document best practices and workaround for Address Resolu= tion scaling issues. At the last week’s ARMD WG session, many Address= Resolution scaling issues were discussed. 

Many people do feel that it is more productive for = the WG to focus on a subset of problems which are solvable, so that the WG = won’t  waste energy on too broad issues associated with Data Cen= ter.

 

Is it be possible for you to write an I= ETF draft to describe the SEATTLE approach for next IETF?

 

Thanks,

Lin= da Dunbar

 

 

=

From: armd-bounces@ietf.org [mailto:armd-b= ounces@ietf.org] On Behalf Of Changhoon Kim
Sent: Sunday, = July 31, 2011 8:03 PM
To: armd@ietf.org
Subject: Re: [a= rmd] For your consideration: SEATTLE

 

Hi,

 

I'd like to add a minor correction to the original email ab= out SEATTLE.

 

- "For example, for ARP queries, = the key is an IP address and the value includes the IP address and the host= ’s location". We meant "For example, for ARP queries, the k= ey is an IP address and the value includes the _MAC_ address and the h= ost’s location".

&= nbsp;

Also, for those who are= interested in the implementation details of SEATTLE, we have another write= -up that include a full section explaining our prototype SEATTLE switch.

 

http://www.cs.princeton.edu/~chkim/papers/seattle_tocs1= 1.pdf

 

Thanks.

 

-- Chang<= o:p>

 

  

On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar <caesar@cs.illinois.edu> wrote:=

> 

> Hi,

&g= t; 

> I've been watch= ing this list with a lot of interest. We wanted to post for your considerat= ion a protocol called SEATTLE we've been developing, which targets large im= provements in scalability of layer-2 in the context of data centers/cloud c= omputing by eliminating the need for broadcasts. We think SEATTLE addresses= some of the goals in ARMD's charter. More details below, but we'd be quite= interested to discuss/answer questions if there's interest.

=

> 

> SEATTLE achieves the same configuration-free properties = as Ethernet bridging yet scales to large networks. Unlike other proposals (= e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely eliminates= the need for network-wide broadcasting, achieving data-plane control overh= eads that grow logarithmically with network size. SEATTLE is backwards-comp= atible with existing Ethernets, simplifying incremental deployment, and sup= porting existing Ethernet functions (e.g., VLANs and host bootstrapping). S= EATTLE also supports more advanced (optional) features, such as the ability= to search and look up services based on text strings, more flexible routin= g and more control over layer-2 routing policies, and the ability to anycas= t and load balance directly over named services. SEATTLE has two main featu= res:

> 

> Distributed in-network directory serv= ice: The switches collectively run a directory service that handles ARP and= DHCP requests, as well as packets sent to unknown destination addresses. T= he directory service leverages consistent hashing and the link-state routin= g protocol to form a one-hop distributed hash table. The directory is a key= -value store, where the hash of the key determines the switch(es) responsib= le for storing the information. For example, for ARP queries, the key is an= IP address and the value includes the IP address and the host’s loca= tion. When an ingress switch cannot handle an ARP/DHCP request or a data pa= cket directly, the switch computes the hash and directs the packet through = the responsible intermediate switch. The intermediate switch can response t= o ARP and DHCP queries, and direct data packets onward to their egress swit= ch while returning the destination host’s location to the ingress swi= tch.

> 

> Reactive caching and invalidation: To= minimize the portion of traffic directed over longer paths, an ingress swi= tch reactively caches the responses from the intermediate switch. For examp= le, while an intermediate switch may handle an ARP query, the data packets = typically flow directly along the shortest path. In addition, other hosts c= onnected to the same ingress switch benefit from the cached information. Ho= wever, the cached information becomes stale if a host moves to a new locati= on. SEATTLE includes an efficient cache invalidation protocol that reactive= ly updates the stale cache entries when a packet wrongly travels to the old= egress switch.

>&nbs= p;

> The first feature leads to= good performance (by keeping traffic in the data plane, and avoiding broad= cast) and self scaling (by having the directory service naturally scale wit= h the size of the network), and the second ensures fast recovery after fail= ures and migration.

>=  

> Our writeup on SEATTLE= :

> 

> 

> presents more details on this p= rotocol, our experiences designing and implementing a system prototype, and= a performance evaluation of the protocol through simulations and deploymen= t in a testbed.

>&nbs= p;

> -- Matt

> 

> 

> ___= ____________________________________________

> armd mailing list

 

 


= --_000_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_-- --_004_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_ Content-Type: image/gif; name="image8005c2.gif@bfb2481e.9e5b4b0f" Content-Description: image8005c2.gif@bfb2481e.9e5b4b0f Content-Disposition: inline; filename="image8005c2.gif@bfb2481e.9e5b4b0f"; size=3410; creation-date="Tue, 02 Aug 2011 12:23:12 GMT"; modification-date="Tue, 02 Aug 2011 12:23:12 GMT" Content-ID: Content-Transfer-Encoding: base64 R0lGODlh3AAeAPcAANINQfb29uiCnuqVq9AALPXb4vLBzCopFcsAGbOyq+dmc8nJxeJig8gAB/3/ //TS2/r2+Pr5+eydsuZ8mauqo++0w2JiVBgXAuVyke6qvGtqXdMSRru6s/z///np7dEIPtrZ1vG9 y4yLgfru8dXU0kxKOtEANfz5+kNCMdMQRNLRzuFVeu2htFpaTPfh6NAAMdw6ZfLE0fjl6umNps8A LeTk4tQUSVNSQzw7Kfr19u6luc4AKfTO2fC1xPGqspyblIWEeuHh3s0AJdoyWc7OytosWd5BbPLx 8ZaVjHZ1aXt6bvn39+FZfunq6dglU6alnuHh4MXGwds0YYKBdvz8/NIFPDUzIc8AMPGzus0AIOh+ mtAAA9IKQAMCAMLCvfTK1d5JcuFSdNUEF9orVeno5umFoPTM1pSUitgeUOqKpPry9u7u7L6+uPnx 8vXW36KhmXJxZSMiDn18cd7d2ubm5N9SedUNQt/f3Od5l/fe5Pfi5uNmiMTEvuNqidEAOO+jqvz+ /JKRiNUbR+JdgdMRRe3s65iXjtIUNaCfl9IEOueAnOVWZeRpivff5dcbTdckTdjX0/PI0+VvjdMO Q7e1sNYYTNIAOJGQhtrb2I+OhdQKQGZlWOzr6nh3a9YSQuqKo9ECOmBfUKWjneRpjP39/szMxjk5 J9ICOPT08/Dw7/Xz9eRwj4mJfz8+LcDAutMSRdMQRjIxH9MQQW5tX4iGe9DPy2loW0JAMP36/OBO dOqPp6ion6Sim/PM18jHwt09Z5mZj15dTtMUR9ICO9ACOVhXSNMEPEdFNh4dCcwAHy8uGzc2JcTD vvz19//9/4B/dNzb2ZSTidIBO/f39rCvqdYYQ1BPP++xwPv7/NQANueMo8/RzdfY1Ofo5Orn50lI OdEDO+yYr/C4xlBOP7i3sfz9/FdZSFpXR/Cuwf78/dAEMdIHNhAOAOLk4P7//uPj4NggUfDv7tkn V+qRqO6dpfv7+pOUi8jIxOZ3lN9IbqOjm+6uv8vKxudwhKmooP///yH5BAAAAAAALAAAAADcAB4A AAj/AP8JHEiw4MAObUgZXMiwocOHECNKnEixosWLGDO6YSRlRYyMIEOKHEmypEmLjdAkI5CsiriT MGPKnEkzpDtFLTcQQpDPQc2fQIMKJemuDoFXr2zsKAJhqNOnUKP+u4mgig1YCHL5lMq1q9eRMuQh GAvg49ezaNM+dIEnzJ5IauPKjeqg7taK7to84MHDxV2I7upCtOvA3dzDQiOEmMGI0Z49M67JmCpj RLMRMk4sjMHEBpdJmiaBMUDQRYwvZr4Y8PDPATp8DBjNK1DQHSAZPebh2zOKkRYWehALj+lAghQT O2hcoUFjBwJJ/x7Iq1QEnqMKBTvgeyHkhZ/vJrqH/xs4IZudFKcIXHO3gkByAkI0jReYLlw+eC+S X1nunhCepsMFGBI2gwixAzgATDJJCilMggAT//QCH3JCsECQA0xkYcIkGwhDyAYbyJKFHW4IhAEB JoDywgur1CGEHwpOAkByGQjkQiUImPBBggvKyAUNCPyihoBEWtQBEzl+COIkXHCRQjJ9RGeHHxt8 YEmNArmDRxangEgIOMpxUaUQihimhSWEpLmBDcN8kIIwOoEohDxDuiCPCQBwUUUibU4C5wbEIDBK kYRKJEEWoAjzygaTmHDFii80sIKUVFqJ5T8GmPDCoim8oMkKYFQxCSHJ1GEmmmoCoOkVfnxIiDCg mP8Qwj8u2JAMcx94AksiL4j5KgGy0FbosAudMIQQHm7wIzwT6KDDDGNA9wAAlV6ZJRNCpKATDWjM +g8DeJZ6apqEpGCCI3sw4sgLrnJxhS7/5FHEL5/sE8MDDxgArpgbVPGChcQGPFAMO8Dy4SQvMMHa QJf9w8OUVVqyj0AyVPMCiCac8pJA87C6QxnjEvKKH5WQ9k8kaJiQ5iQ0aPFPOY0wNAgNIAJAgACG CRzwADt4SMgVUkTAkBkQc2ECwNeYAA6jNICx1QnH0eCJsGeKDAANqxA0wQ5isqzIQA7EUAYDKzDw SS8CvMDhjDjrHPAeJix69QQNEU3lJASUIVAaBHD/IQwAV2Q9FSMvIKe3QFUrS8AABM3DNdNf/xMD DCvusIOB0mzA4AY2t+32sODK/QK8Q0/5yiRXHC7ADgCcvsOk/2BQuBAr3JU4FwTMkPM/aQjRNQ0C /POFMS8m+AIN8F2RbOe7w9T85yetcvHpNEBXuh+np87xFb6+4IQ4DBDwQhYwAIg4morrPlDvXef+ jxFZaEuIJVIoopsUxmjLPEYj/IFFOQypxSZqwRV+2OIeJinFEwJQEF50ggxe4MBEJICsD5nABl8o 3SnStAPrheAUidBJCmDxgUfRIBdtKMjtcrc79nGOBhIYgSbiRgg/yGNh/9DBDtZ2s+dJpB6L2MIW /+hBkHiIoBT/oEYXKCESciDBhxHxRxcQYZJNsIMEBQnGBXzxjVYohCBP8MdCeNA3nRDiBfDYh2am wpd/mEETrdqACYqQg38AoghXUFMKuGAMG2iDIF8AmQDQhzv17c13jLoCC3phDGmU6wV7IMge8si5 Hl4kD2LYQgMWcRdqsCMJ/2ADMngBgiYIZAnMIGARydAEftBBIJxgwxz+sYRxIKMUR4AGLv4BhRr8 IwCYEEgp+CC0I9BhDZj4ATIS8A9MQAMbA1lCFBaQDlV0owlReMdAVOCFIwxkAQtI4hSgyQ9XLOEf cDjAO/xhiKkwgx//IMMylgEJZxSkHDDIApweaf+CX/ShDwxQx6T0YAOVpckEdQjBPqSAPTPWEAYe 6IAHQqAFddDgBGkgJAvXh0jUseABxhjGK+ZXBBe0RgJ+WFolPVcRB/RjC2LwwUDucIw4KMMebDhA Kw5wA1TEwwKtwEE7ByKKW5TgANaYBhGsgYJvsEEfB4hDKEQQCwr8owQl+McZSgCCKcQBGcFYQwJu cYxxKOEWXkhAHH6QsxrYYhlW4AMFjvGNOJhiG/9AwjFuYQFUOGMWF0DGG5CggWkYogStCAUqgKCM d0QDCAFIZwmoAYSoctUgGcgCDWwQpw/QIAvJEEIDGPCPDsBgBx96BQAKlz/NyY9BhBCEEx6RiCv/ NOAQvDuF3Da6N64Jg2USyIEmXuAhczmBEXX4zgZ2GzyMUCESZiDINAJxAA2AwAsXSIISLiAKVrBD CSjogisGcokuaCAJyDDEJuLAijgcwBehiEUCIHGAYnDgqxz4hgUygYwpKFMOP+jCDdiAiFaYIxab WMNAdtEFFDxjGmfIrj6kSok4FGMWXWDFJdgxBWDUwhZxwIYontEJdlCDFVaYQwvGkQB2WEEENSDC LW5BgVQYxB0TQMAO3iSMHtvABpUQAmn/EY4ssKvHwvhAFT6Qnx184FV/MwENXnAFEzRgCA4oAyF3 YMh/fMLJL0zDP+qAgA/0+IzweZHmKqm3POgA/4cgaccy/8GHC2jTFEAIxQFYkQQckGMg9rhADZqA Alsc4xhIaMEN/kGLWwgECQfAgQVacAtTEGEWyxBIKMYRjVjMkhmxCCw0CNKNJBSjGFFAAg4K8Y9O WEEDB7AFEIoBBMQORAmtuAMQkiCCZeiDFqaAwixCEYBnWOMcEmzBJhriDgGoYyVKA0UiTmHlMBjG AYMI7aM0RQMhVCEXv8iCEFhVhSr4gTvrGMQD/iGJZOwAPg1orkAm0IBkXEG0GPiHHpygYxNUARQ7 yII88vlu0eLhH/gwAekmEgH//UUF7DgGCJR4D1RcgBW0YIcInpEJBgpEDl24QxDikIT1vqETT//4 hwa6YIgIkAEZXUhAILqgjH8Ygh1AQAIy5CACdnDjH0/gcAmUgUWBHOEHnehCC0SgDCVQIA4WuG8o gPEMMoigC0D4gQosYApDdCEJwGAHEpSADBLcwNG8wHAr/nEDZOiDEw2JxCAEkQ0/WMISeBrDpTow DynIQhMAEMQvJhADdzRDF373gwnMU4RVwEUg4ZCCEYwAgyLoYHc6kMcvjPCLIkjARgwQBLUtUQ0m NCIE86K85XNohI1J5AQKEOIfdneETXTBFuRQxj28YQUKTCMUprBCIAgSCGVAAwQoeMIdvoGDW0hw F4Hlwz82cQBO0LcT/4iABpBxgWAEwB/KwCv/NYjuiu8OpBTHwMEBKPEDK7QCGcpQgc1NkdhzWoAd 7EDEFG6ggmJ8QwPLsAuZcAvcAAebQALWEAuxQEU3Fwfyx2y44Sw6cA1uoAbPowYF8AV9YT4C0Qxu UAEskAEPMAIFcQJtAAFLkAMjsEsDcQIjkAMQAAEjsEatIQMGwAIVkAeG4Q4jgIIqqBmAMAJQ1BCY JEScRBCqsAAgQArtoBBBMA2/NE02NhBHEARTUQNQyAleoAJC4w4g4Auo8A/GJBA14HHlUAv8AIXT AAVU8A/TYIX/cAd38EXTQAJeMEuBEAscAA1w+A8kwAe+9A9UsABEMA2o0A1ieA930AT1oAqvL7QG hUAF3OAKsySIKsAPQgM9J+EAsRdTOkMLB0AEmjgsJ4AFITCERAIJlGBKThEQADs= --_004_B37E6A2CE5957F4E83C1D9845A0FFE386E51423EMDWEXGMB02ciena_-- From giles.heron@gmail.com Wed Aug 3 10:59:27 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4E87411E807E for ; Wed, 3 Aug 2011 10:59:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.404 X-Spam-Level: X-Spam-Status: No, score=-3.404 tagged_above=-999 required=5 tests=[AWL=0.195, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RfHDdT7enkni for ; Wed, 3 Aug 2011 10:59:26 -0700 (PDT) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by ietfa.amsl.com (Postfix) with ESMTP id 67CBC11E8078 for ; Wed, 3 Aug 2011 10:59:26 -0700 (PDT) Received: by wwe5 with SMTP id 5so754259wwe.13 for ; Wed, 03 Aug 2011 10:59:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=user-agent:date:subject:from:to:message-id:thread-topic :thread-index:mime-version:content-type:content-transfer-encoding; bh=gA26DOCC3mti00x5fQszkQiF0ljQK9QIYgmLuf6lk74=; b=aUGCh88OW20YWJfe6ztxlQR1QBBCsPN/dEx0t2hK7GDvYposUeE4aIQWxavVIM/qzF pLoL3w/b30nYa5oHmzgvZBEKZFcZcOB6vgSMFJpF7l1QqkjJYdk1TqscpFwA62ltVbfI fTpTWrAArnh2ujFfrTHkmz3Sa/yyAe+XITU2A= Received: by 10.216.136.18 with SMTP id v18mr3197577wei.23.1312394365286; Wed, 03 Aug 2011 10:59:25 -0700 (PDT) Received: from [10.55.88.129] (64-103-25-233.cisco.com [64.103.25.233]) by mx.google.com with ESMTPS id j54sm701686wed.23.2011.08.03.10.59.22 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 03 Aug 2011 10:59:23 -0700 (PDT) User-Agent: Microsoft-Entourage/12.30.0.110427 Date: Wed, 03 Aug 2011 19:01:24 +0100 From: Giles Heron To: Message-ID: Thread-Topic: MPLS 2011 - October 16-19, Washington DC Thread-Index: AcxSB1t7d2O98yJCKECCF+HdwUVDcA== Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Subject: [armd] MPLS 2011 - October 16-19, Washington DC X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2011 17:59:27 -0000 Hi ARMDers, registration for MPLS2011 (October 16-19, Omni Shoreham Hotel in Washington), the 14th annual International Conference on MPLS and related new technologies is now open at: http://www.mpls2011.com/registration/attendees.htm MPLS 2011 will include an extensive four day program consisting of tutorials, technical sessions, panels, and exhibits. The key topics to be discussed at this year's conference include: Scaling MPLS, Data Centers, MPLS Transport Profile, Cloud Computing, Future Networks, Mobile Backhaul, Resiliency, Network Management and Performance, Multi-layer and Optical Networks. In addition, there will be two panel discussions on current topics. The complete program is available at: http://www.mpls2011.com/program/technical_sessions.htm The event will be followed by a Public Interop demonstration. There will also be an exhibit floor, where leading network equipment vendors will showcase their new offerings, the list of current sponsors is available at: http://www.mpls2011.com/sponsors/sponsors.htm The conference hotel is the Omni Shoreham Hotel in Washington DC. Please note that there are only a limited number of rooms available this year at a reduced rate. Please make reservations at: http://www.mpls2011.com/hotel.htm Giles From dunbar.ll@gmail.com Tue Aug 9 10:21:34 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 829E611E809C for ; Tue, 9 Aug 2011 10:21:34 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wyoapgQIOakI for ; Tue, 9 Aug 2011 10:21:33 -0700 (PDT) Received: from mail-fx0-f44.google.com (mail-fx0-f44.google.com [209.85.161.44]) by ietfa.amsl.com (Postfix) with ESMTP id 620FE21F8CD2 for ; Tue, 9 Aug 2011 10:21:33 -0700 (PDT) Received: by fxe6 with SMTP id 6so265210fxe.31 for ; Tue, 09 Aug 2011 10:22:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=pSGDcLyaOiArYjAl3FWM1jJCmBAPRQ2n5hazhm/k+Bs=; b=rIGOb5o4uVyFRDW6fzhvIafONAv6Nr7FYSaJFLZ/0KNEd/K+yqmt+lYyB4h9XmDKpf pIUBSyyVd21G+xY29dosv8UMa9qxbgwnKvQVlgpUtobWSZIAIytSBN7b9yzAhqcoL2zI ptgejy/Oea70e9sJm2DX6S3oimds/NYgttBeo= MIME-Version: 1.0 Received: by 10.205.65.205 with SMTP id xn13mr2032000bkb.284.1312910521920; Tue, 09 Aug 2011 10:22:01 -0700 (PDT) Received: by 10.205.81.139 with HTTP; Tue, 9 Aug 2011 10:22:01 -0700 (PDT) Date: Tue, 9 Aug 2011 12:22:01 -0500 Message-ID: From: Linda Dunbar To: armd@ietf.org Content-Type: multipart/alternative; boundary=bcaec5430fb647ec3404aa15cdcf Subject: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 17:21:34 -0000 --bcaec5430fb647ec3404aa15cdcf Content-Type: text/plain; charset=ISO-8859-1 During the 81st IETF ARMD WG discussion, it was suggested that it is necessary to document typical data center network designs so that address resolution scaling issues can be properly described. Many data center operators have expressed that they can't openly reveal their detailed network designs. Therefore, we only want to document anonymous designs without too much detail. During the journey of establishing ARMD, we have come across the following typical data center network designs: 1. layer 3 all the way to TOR (Top of Rack switches), 2. large layer 2 with hundreds (or thousands) of ToRs being interconnected by Layer 2. This design will have thousands of hosts under the L2/L3 boundary router (s) 3. CLOS design with thousands of switches. This design will have thousands of hosts under the L2/L3 boundary router(s) We have heard that each of the designs above has its own problems. ARMD problem statements might need to document DC problems under each typical design. Please send feedback to us (either to the armd email list or to the ARMD chair Benson & Linda) to indicate if we have missed any typical Data Center network designs. Your contribution can greatly accelerate the progress of ARMD WG. Thank you very much. Linda & Benson --bcaec5430fb647ec3404aa15cdcf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=C2=A0so that addre= ss resolution scaling issues can be properly described. Many data center op= erators have expressed that they can't openly reveal their detailed net= work designs. Therefore, we only want to document anonymous designs without= too much detail. During the journey of establishing ARMD, we have come acr= oss the following typical data center network designs:
  1. = layer 3 all the way to TOR (Top of Rack switches),
  2. = large layer 2 with hundreds (or thousands) of ToRs being interconnected by = Layer 2. This design will have thousands of hosts under the L2/L3 boundary = router (s)
  3. = CLOS design =C2=A0with thousands of switches. This design will have thousan= ds of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. ARMD pr= oblem statements might need to document DC problems under each typical desi= gn.
Please send feedback to us (either to the=C2=A0armd email list =C2=A0or to= the ARMD chair Benson & Linda) to indicate if we have missed any typic= al Data Center network designs.
=C2=A0=
Your contribution ca= n greatly accelerate the progress of ARMD WG.
=C2=A0=
Thank you very much.=
=C2=A0=
Linda & Benson
--bcaec5430fb647ec3404aa15cdcf-- From vishwas.ietf@gmail.com Tue Aug 9 10:58:05 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5028F21F8A95 for ; Tue, 9 Aug 2011 10:58:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.937 X-Spam-Level: X-Spam-Status: No, score=-2.937 tagged_above=-999 required=5 tests=[AWL=0.661, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rfTvd5tUDNzd for ; Tue, 9 Aug 2011 10:58:04 -0700 (PDT) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id 7BCE521F8A23 for ; Tue, 9 Aug 2011 10:58:04 -0700 (PDT) Received: by qwc23 with SMTP id 23so164301qwc.31 for ; Tue, 09 Aug 2011 10:58:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PLUdRhEW63VA6EDqwByv3OJfF9MDBU2awpfVz8p7LWA=; b=p0e6gEpWEXw3qEfYZYRhYpObLXkQIQ0nh99O0jeC1gax242BmKD3WytEkVIKDuQDly EjoYH+4mT1j78w+fWgG/Tn8G9FbFe0UH/StypVnVkvqZ7iQcoe7jewH+QkDpF757kMDu eJc1kXDddPssQL2MxQ8EW+ekcwGrTTjdIYAcA= MIME-Version: 1.0 Received: by 10.229.107.5 with SMTP id z5mr5265755qco.229.1312912711067; Tue, 09 Aug 2011 10:58:31 -0700 (PDT) Received: by 10.229.65.91 with HTTP; Tue, 9 Aug 2011 10:58:30 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 10:58:30 -0700 Message-ID: From: Vishwas Manral To: Linda Dunbar Content-Type: multipart/alternative; boundary=0023544711bcc3a7a404aa164fca Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 17:58:05 -0000 --0023544711bcc3a7a404aa164fca Content-Type: text/plain; charset=ISO-8859-1 Hi Linda, I am unsure what you mean by this, but: 1. layer 3 all the way to TOR (Top of Rack switches), We can also have a heirarchical network, with the core totally Layer-3 (and having seperate routing), from the hosts still in a large Layer-3 subnet. Another aspect could be to have a totally Layer-3 network. The difference between them is the link between the servers and the ToR. Thanks, Vishwas On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: > During the 81st IETF ARMD WG discussion, it was suggested that it is > necessary to document typical data center network designs so that address > resolution scaling issues can be properly described. Many data center > operators have expressed that they can't openly reveal their detailed > network designs. Therefore, we only want to document anonymous designs > without too much detail. During the journey of establishing ARMD, we have > come across the following typical data center network designs: > > 1. layer 3 all the way to TOR (Top of Rack switches), > 2. large layer 2 with hundreds (or thousands) of ToRs being > interconnected by Layer 2. This design will have thousands of hosts under > the L2/L3 boundary router (s) > 3. CLOS design with thousands of switches. This design will have > thousands of hosts under the L2/L3 boundary router(s) > > We have heard that each of the designs above has its own problems. ARMD > problem statements might need to document DC problems under each typical > design. > Please send feedback to us (either to the armd email list or to the ARMD > chair Benson & Linda) to indicate if we have missed any typical Data Center > network designs. > > Your contribution can greatly accelerate the progress of ARMD WG. > > Thank you very much. > > Linda & Benson > --0023544711bcc3a7a404aa164fca Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Linda,
I am unsure what you mean by this, but:
  1. layer 3 all the way to TOR (Top of Rack switches),
    2. =
We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
=A0
The difference between them is the link between the servers and the To= R.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.= com> wrote:
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
  1. layer 3 all the way to TOR (Top of Rack switches),
  2. large layer 2 with hundreds (or thousands) of ToRs being interconnecte= d by Layer 2. This design will have thousands of hosts under the L2/L3 boun= dary router (s)
  3. CLOS design =A0with thousands of switches. This design will have thous= ands of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. AR= MD problem statements might need to document DC problems under each typical= design.
Please send feedback to us (either to the=A0armd email list =A0or to = the ARMD chair Benson & Linda) to indicate if we have missed any typica= l Data Center network designs.
= =A0
Your contributi= on can greatly accelerate the progress of ARMD WG.
= =A0
Thank you very = much.
= =A0
Linda & Ben= son
--0023544711bcc3a7a404aa164fca-- From dunbar.ll@gmail.com Tue Aug 9 11:51:22 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9545D21F8BB7 for ; Tue, 9 Aug 2011 11:51:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8hKEpZqcAB6E for ; Tue, 9 Aug 2011 11:51:21 -0700 (PDT) Received: from mail-ew0-f44.google.com (mail-ew0-f44.google.com [209.85.215.44]) by ietfa.amsl.com (Postfix) with ESMTP id 1B20B21F8B88 for ; Tue, 9 Aug 2011 11:51:20 -0700 (PDT) Received: by ewy19 with SMTP id 19so197483ewy.31 for ; Tue, 09 Aug 2011 11:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=842mEeSGxC0XyovtWw+7lHzn6HtP2aV7nnokWK33/OE=; b=X+Rm/Rg/oEKtXy7BVXvJnvCnoSFxzp4UTvHAJmLCq8PkvmSfSIIvcjWOpUiKwnubql L8FNu3HG/CVkbHq3PVJrQeXMU0/1mt0KXL7LhSl2K7yBadC+wqiueRwsW6fqLge3xTMw YXTvXexzlBVxAnwqJKDkF2PkCwMPcWo8QvYmg= MIME-Version: 1.0 Received: by 10.205.65.205 with SMTP id xn13mr2050755bkb.284.1312915909709; Tue, 09 Aug 2011 11:51:49 -0700 (PDT) Received: by 10.205.81.139 with HTTP; Tue, 9 Aug 2011 11:51:49 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 13:51:49 -0500 Message-ID: From: Linda Dunbar To: Vishwas Manral Content-Type: multipart/alternative; boundary=bcaec5430fb66b0e4404aa170ebf Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 18:51:22 -0000 --bcaec5430fb66b0e4404aa170ebf Content-Type: text/plain; charset=ISO-8859-1 Vishwas, In my mind the bullet 1) in the list refers to ToR switches downstream ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3. Have you seen data center networks with ToR switches downstream ports (i.e. facing servers) enabling IP routing, even though the physical links are Ethernet? If yes, we should definitely include it in the ARMD draft. Thanks, Linda On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral wrote: > Hi Linda, > I am unsure what you mean by this, but: > > 1. layer 3 all the way to TOR (Top of Rack switches), > > We can also have a heirarchical network, with the core totally Layer-3 (and > having seperate routing), from the hosts still in a large Layer-3 subnet. > Another aspect could be to have a totally Layer-3 network. > > The difference between them is the link between the servers and the ToR. > > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: > >> During the 81st IETF ARMD WG discussion, it was suggested that it is >> necessary to document typical data center network designs so that address >> resolution scaling issues can be properly described. Many data center >> operators have expressed that they can't openly reveal their detailed >> network designs. Therefore, we only want to document anonymous designs >> without too much detail. During the journey of establishing ARMD, we have >> come across the following typical data center network designs: >> >> 1. layer 3 all the way to TOR (Top of Rack switches), >> 2. large layer 2 with hundreds (or thousands) of ToRs being >> interconnected by Layer 2. This design will have thousands of hosts under >> the L2/L3 boundary router (s) >> 3. CLOS design with thousands of switches. This design will have >> thousands of hosts under the L2/L3 boundary router(s) >> >> We have heard that each of the designs above has its own problems. ARMD >> problem statements might need to document DC problems under each typical >> design. >> Please send feedback to us (either to the armd email list or to the ARMD >> chair Benson & Linda) to indicate if we have missed any typical Data Center >> network designs. >> >> Your contribution can greatly accelerate the progress of ARMD WG. >> >> Thank you very much. >> >> Linda & Benson >> > --bcaec5430fb66b0e4404aa170ebf Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Vishwas,
=A0
In my mind the bullet 1) in the list refers to ToR switches downstream= ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer = 3.
=A0
Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical links a= re Ethernet?=A0=A0
If yes, we should definitely include it in the ARMD draft.
=A0
Thanks,
Linda
On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = <vishwas.iet= f@gmail.com> wrote:
Hi Linda,
I am unsure what you mean by this, but:
  1. layer 3 all the way to TOR (Top of Rack switches),
    2. =
We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
=A0
The difference between them is the link between the servers and the To= R.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
  1. layer 3 all the way to TOR (Top of Rack switches),
  2. large layer 2 with hundreds (or thousands) of ToRs being interconnecte= d by Layer 2. This design will have thousands of hosts under the L2/L3 boun= dary router (s)
  3. CLOS design =A0with thousands of switches. This design will have thous= ands of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. AR= MD problem statements might need to document DC problems under each typical= design.
Please send feedback to us (either to the=A0armd email list =A0or to = the ARMD chair Benson & Linda) to indicate if we have missed any typica= l Data Center network designs.
= =A0
Your contributi= on can greatly accelerate the progress of ARMD WG.
= =A0
Thank you very = much.
= =A0
Linda & Ben= son
--bcaec5430fb66b0e4404aa170ebf-- From dunbar.ll@gmail.com Tue Aug 9 12:50:37 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CC70721F8B8E for ; Tue, 9 Aug 2011 12:50:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.765 X-Spam-Level: X-Spam-Status: No, score=-2.765 tagged_above=-999 required=5 tests=[AWL=-0.833, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_HTML_USL_OBFU=1.666] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mb9AIYNnDYZT for ; Tue, 9 Aug 2011 12:50:36 -0700 (PDT) Received: from mail-ew0-f44.google.com (mail-ew0-f44.google.com [209.85.215.44]) by ietfa.amsl.com (Postfix) with ESMTP id 04B4421F8B95 for ; Tue, 9 Aug 2011 12:50:34 -0700 (PDT) Received: by ewy19 with SMTP id 19so230909ewy.31 for ; Tue, 09 Aug 2011 12:51:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PKLFYSXZxGRlxonK7zcqLQRn7+DfTjfvENznOopvdzU=; b=NiJZynBDfoM1Zn/KpbfG9DC0XNs+wZO6vY821qWmLfFFRutEwPyKemuj6LIRIl3Ewf 4GvhTbxIboxkBEz/SM4ekEK4SeM9Fmh8xVEu693OGp3VzV06XM/uRWoXGouhtl/DS/tY TOlwIy71PbgkCrdUgaJcaMe2YlsN5VNZTYdPk= MIME-Version: 1.0 Received: by 10.205.65.205 with SMTP id xn13mr2063047bkb.284.1312919460294; Tue, 09 Aug 2011 12:51:00 -0700 (PDT) Received: by 10.205.81.139 with HTTP; Tue, 9 Aug 2011 12:51:00 -0700 (PDT) In-Reply-To: <4E329CB2.8010908@cs.illinois.edu> References: <4E329CB2.8010908@cs.illinois.edu> Date: Tue, 9 Aug 2011 14:51:00 -0500 Message-ID: From: Linda Dunbar To: Matthew Caesar Content-Type: multipart/alternative; boundary=bcaec5430fb60cae9804aa17e20a Cc: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 19:50:37 -0000 --bcaec5430fb60cae9804aa17e20a Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Matthew, Questions on the details: 1. Distributed in-network directory service: when Ingress switch receives a data packet with unknown, you stated that the ingress switch will send the packet to the Intermediate switch based on the hashing value. How does "intermediate switch" know if a received data packet is relayed from anothe= r switch or sent directly by hosts? If the "intermediate switch" still doesn'= t have forwarding entry for the destination address in the data packet, it will do a "hashing" and send to another "intermediate switch. This forwarding can form a loop. How does SEATTLE mitigate the loop issue? 2. Reactive caching and invalidation: Your brief description sounds a lot like the scheme described in "( https://datatracker.ietf.org/doc/draft-shah-armd-arp-reduction/". Can you elaborate the major differences? Thank you very much. Linda Dunbar On Fri, Jul 29, 2011 at 6:42 AM, Matthew Caesar wro= te: > Hi, > > I've been watching this list with a lot of interest. We wanted to post fo= r > your consideration a protocol called SEATTLE we've been developing, which > targets large improvements in scalability of layer-2 in the context of da= ta > centers/cloud computing by eliminating the need for broadcasts. We think > SEATTLE addresses some of the goals in ARMD's charter. More details below= , > but we'd be quite interested to discuss/answer questions if there's > interest. > > SEATTLE achieves the same configuration-free properties as Ethernet > bridging yet scales to large networks. Unlike other proposals (e.g., TRIL= L, > Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need f= or > network-wide broadcasting, achieving data-plane control overheads that gr= ow > logarithmically with network size. SEATTLE is backwards-compatible with > existing Ethernets, simplifying incremental deployment, and supporting > existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE > also supports more advanced (optional) features, such as the ability to > search and look up services based on text strings, more flexible routing = and > more control over layer-2 routing policies, and the ability to anycast an= d > load balance directly over named services. SEATTLE has two main features: > > Distributed in-network directory service: The switches collectively run a > directory service that handles ARP and DHCP requests, as well as packets > sent to unknown destination addresses. The directory service leverages > consistent hashing and the link-state routing protocol to form a one-hop > distributed hash table. The directory is a key-value store, where the has= h > of the key determines the switch(es) responsible for storing the > information. For example, for ARP queries, the key is an IP address and t= he > value includes the IP address and the host=92s location. When an ingress > switch cannot handle an ARP/DHCP request or a data packet directly, the > switch computes the hash and directs the packet through the responsible > intermediate switch. The intermediate switch can response to ARP and DHCP > queries, and direct data packets onward to their egress switch while > returning the destination host=92s location to the ingress switch. > > Reactive caching and invalidation: To minimize the portion of traffic > directed over longer paths, an ingress switch reactively caches the > responses from the intermediate switch. For example, while an intermediat= e > switch may handle an ARP query, the data packets typically flow directly > along the shortest path. In addition, other hosts connected to the same > ingress switch benefit from the cached information. However, the cached > information becomes stale if a host moves to a new location. SEATTLE > includes an efficient cache invalidation protocol that reactively updates > the stale cache entries when a packet wrongly travels to the old egress > switch. > > The first feature leads to good performance (by keeping traffic in the da= ta > plane, and avoiding broadcast) and self scaling (by having the directory > service naturally scale with the size of the network), and the second > ensures fast recovery after failures and migration. > > Our writeup on SEATTLE: > > http://www.cs.princeton.edu/~**jrex/papers/seattle08.pdf > > presents more details on this protocol, our experiences designing and > implementing a system prototype, and a performance evaluation of the > protocol through simulations and deployment in a testbed. > > -- Matt > > > ______________________________**_________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/**listinfo/armd > --bcaec5430fb60cae9804aa17e20a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Matthew,
=C2=A0
Questions on the details:
=C2=A0
1. Distributed in-network directory service: when Ingress switch recei= ves a data packet with unknown, you stated that the ingress switch will sen= d the packet to the Intermediate switch based on the hashing value. How doe= s "intermediate switch" know if a received data packet is relayed= from another switch or sent directly by hosts? If the "intermediate s= witch" still doesn't have forwarding entry for the destination add= ress in the data packet, it will do a "hashing" and send to anoth= er "intermediate switch. This forwarding can form a loop. How does SEA= TTLE mitigate the loop issue?
=C2=A0
=C2=A0
2. Reactive caching and invalidation: Your brief description sounds a = lot like the scheme described in "(https://da= tatracker.ietf.org/doc/draft-shah-armd-arp-reduction/". Can you elaborate the major differences?
=C2=A0
Thank you ver= y much.
=C2=A0=
Linda Dunbar<= /span>

On Fri, Jul 29, 2011 at 6:42 AM, Matthew Caesar = <caesar@cs.i= llinois.edu> wrote:
Hi,

I've been watchin= g this list with a lot of interest. We wanted to post for your consideratio= n a protocol called SEATTLE we've been developing, which targets large = improvements in scalability of layer-2 in the context of data centers/cloud= computing by eliminating the need for broadcasts. We think SEATTLE address= es some of the goals in ARMD's charter. More details below, but we'= d be quite interested to discuss/answer questions if there's interest.<= br>
SEATTLE achieves the same configuration-free properties as Ethernet bri= dging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rb= ridges, Viking, SmartBridges), SEATTLE completely eliminates the need for n= etwork-wide broadcasting, achieving data-plane control overheads that grow = logarithmically with network size. SEATTLE is backwards-compatible with exi= sting Ethernets, simplifying incremental deployment, and supporting existin= g Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also sup= ports more advanced (optional) features, such as the ability to search and = look up services based on text strings, more flexible routing and more cont= rol over layer-2 routing policies, and the ability to anycast and load bala= nce directly over named services. SEATTLE has two main features:

Distributed in-network directory service: The switches collectively run= a directory service that handles ARP and DHCP requests, as well as packets= sent to unknown destination addresses. The directory service leverages con= sistent hashing and the link-state routing protocol to form a one-hop distr= ibuted hash table. The directory is a key-value store, where the hash of th= e key determines the switch(es) responsible for storing the information. Fo= r example, for ARP queries, the key is an IP address and the value includes= the IP address and the host=E2=80=99s location. When an ingress switch can= not handle an ARP/DHCP request or a data packet directly, the switch comput= es the hash and directs the packet through the responsible intermediate swi= tch. The intermediate switch can response to ARP and DHCP queries, and dire= ct data packets onward to their egress switch while returning the destinati= on host=E2=80=99s location to the ingress switch.

Reactive caching and invalidation: To minimize the portion of traffic d= irected over longer paths, an ingress switch reactively caches the response= s from the intermediate switch. For example, while an intermediate switch m= ay handle an ARP query, the data packets typically flow directly along the = shortest path. In addition, other hosts connected to the same ingress switc= h benefit from the cached information. However, the cached information beco= mes stale if a host moves to a new location. SEATTLE includes an efficient = cache invalidation protocol that reactively updates the stale cache entries= when a packet wrongly travels to the old egress switch.

The first feature leads to good performance (by keeping traffic in the = data plane, and avoiding broadcast) and self scaling (by having the directo= ry service naturally scale with the size of the network), and the second en= sures fast recovery after failures and migration.

Our writeup on SEATTLE:

http://www.cs.princeton.edu/~<= u>jrex/papers/seattle08.pdf

presents more details on this pr= otocol, our experiences designing and implementing a system prototype, and = a performance evaluation of the protocol through simulations and deployment= in a testbed.

-- Matt


_____________________________________________= __
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd=

--bcaec5430fb60cae9804aa17e20a-- From vishwas.ietf@gmail.com Tue Aug 9 13:26:38 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4F14B5E8023 for ; Tue, 9 Aug 2011 13:26:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.966 X-Spam-Level: X-Spam-Status: No, score=-2.966 tagged_above=-999 required=5 tests=[AWL=0.632, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZkJFS6QlD8X5 for ; Tue, 9 Aug 2011 13:26:37 -0700 (PDT) Received: from mail-gx0-f172.google.com (mail-gx0-f172.google.com [209.85.161.172]) by ietfa.amsl.com (Postfix) with ESMTP id 657DA5E8021 for ; Tue, 9 Aug 2011 13:26:37 -0700 (PDT) Received: by gxk19 with SMTP id 19so297863gxk.31 for ; Tue, 09 Aug 2011 13:27:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9h8EXG/kQXrLm3rdQXSTZ4AK1nDo9Tvx1VLhQ1UT5fo=; b=Np9u6BOWht2dbhv5ZJQijP5/dbwqK2STzvHT3N8b/mz+XbqB3vjWPwhuWLQQ4XYJ1s +yo0KOeyR8Hd0FWLVLO/t8PRbkT4uu6DVogMObtwmyNY0+sf1VPm+m4JhkXrOhH5bMRW E1Zca6BHx0dHVwIgfSmHcXBFVauYYG3kt+f3o= MIME-Version: 1.0 Received: by 10.142.203.10 with SMTP id a10mr7004195wfg.51.1312921625232; Tue, 09 Aug 2011 13:27:05 -0700 (PDT) Received: by 10.142.126.9 with HTTP; Tue, 9 Aug 2011 13:27:05 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 13:27:05 -0700 Message-ID: From: Vishwas Manral To: Linda Dunbar Content-Type: multipart/alternative; boundary=000e0cd1446a17064904aa18630a Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 20:26:38 -0000 --000e0cd1446a17064904aa18630a Content-Type: text/plain; charset=ISO-8859-1 Hi Linda, The data packets can be tunnelled at the ToR over say a GRE packet and the core is a Layer-3 core (except for the downstream ports). So we could have encapsulation/ decapsulation of L2 over GRE at the ToR. The very same thing can be done at the hypervisor layer too, in which case the entire DC network would look like a Layer-3 flat network including the ToR to server link and the hypervisor would do the tunneling. I am not sure if you got the points above or not. I know cloud OS companies that provide the service and have big announced customers. Thanks, Vishwas On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wrote: > Vishwas, > > In my mind the bullet 1) in the list refers to ToR switches downstream > ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3. > > > Have you seen data center networks with ToR switches downstream ports (i.e. > facing servers) enabling IP routing, even though the physical links are > Ethernet? > If yes, we should definitely include it in the ARMD draft. > > Thanks, > Linda > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral wrote: > >> Hi Linda, >> I am unsure what you mean by this, but: >> >> 1. layer 3 all the way to TOR (Top of Rack switches), >> >> We can also have a heirarchical network, with the core totally Layer-3 >> (and having seperate routing), from the hosts still in a large Layer-3 >> subnet. Another aspect could be to have a totally Layer-3 network. >> >> The difference between them is the link between the servers and the ToR. >> >> Thanks, >> Vishwas >> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: >> >>> During the 81st IETF ARMD WG discussion, it was suggested that it is >>> necessary to document typical data center network designs so that address >>> resolution scaling issues can be properly described. Many data center >>> operators have expressed that they can't openly reveal their detailed >>> network designs. Therefore, we only want to document anonymous designs >>> without too much detail. During the journey of establishing ARMD, we have >>> come across the following typical data center network designs: >>> >>> 1. layer 3 all the way to TOR (Top of Rack switches), >>> 2. large layer 2 with hundreds (or thousands) of ToRs being >>> interconnected by Layer 2. This design will have thousands of hosts under >>> the L2/L3 boundary router (s) >>> 3. CLOS design with thousands of switches. This design will have >>> thousands of hosts under the L2/L3 boundary router(s) >>> >>> We have heard that each of the designs above has its own problems. ARMD >>> problem statements might need to document DC problems under each typical >>> design. >>> Please send feedback to us (either to the armd email list or to the ARMD >>> chair Benson & Linda) to indicate if we have missed any typical Data Center >>> network designs. >>> >>> Your contribution can greatly accelerate the progress of ARMD WG. >>> >>> Thank you very much. >>> >>> Linda & Benson >>> >> > --000e0cd1446a17064904aa18630a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Linda,
=A0
The data packets can be tunnelled at the ToR over say a GRE packet and= the core is a Layer-3 core (except for the downstream ports). So we could = have encapsulation/ decapsulation of L2 over GRE at the ToR.
=A0
The very same thing can be done at the hypervisor layer too, in which = case the entire DC network would look like a Layer-3 flat network including= the ToR to server link and the hypervisor would do the tunneling.
=A0
I am not sure if you got the points above or not. I know cloud OS comp= anies that provide the service and have big announced customers.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
Vishwas,
=A0
In my mind the bullet 1) in the list refers to ToR switches downstream= ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer = 3.
=A0
Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical links a= re Ethernet?=A0=A0
If yes, we should definitely include it in the ARMD draft.
=A0
Thanks,
Linda
On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = <vishwas.ietf@gmail.com> wrote:
Hi Linda,
I am unsure what you mean by this, but:
  1. layer 3 all the way to TOR (Top of Rack switches),
    2. =
We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
=A0
The difference between them is the link between the servers and the To= R.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
  1. layer 3 all the way to TOR (Top of Rack switches),
  2. large layer 2 with hundreds (or thousands) of ToRs being interconnecte= d by Layer 2. This design will have thousands of hosts under the L2/L3 boun= dary router (s)
  3. CLOS design =A0with thousands of switches. This design will have thous= ands of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. AR= MD problem statements might need to document DC problems under each typical= design.
Please send feedback to us (either to the=A0armd email list =A0or to = the ARMD chair Benson & Linda) to indicate if we have missed any typica= l Data Center network designs.
= =A0
Your contributi= on can greatly accelerate the progress of ARMD WG.
= =A0
Thank you very = much.
= =A0
Linda & Ben= son

--000e0cd1446a17064904aa18630a-- From vishwas.ietf@gmail.com Tue Aug 9 14:41:31 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 29CB211E80AD for ; Tue, 9 Aug 2011 14:41:31 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.992 X-Spam-Level: X-Spam-Status: No, score=-2.992 tagged_above=-999 required=5 tests=[AWL=0.606, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nD3MjWz1RXOf for ; Tue, 9 Aug 2011 14:41:30 -0700 (PDT) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 1655D11E808E for ; Tue, 9 Aug 2011 14:41:30 -0700 (PDT) Received: by qyk34 with SMTP id 34so2419744qyk.10 for ; Tue, 09 Aug 2011 14:41:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ElkSSpFaQzRgmvRElbVZPnM0SkBIL6jKlZm+d8FJGMs=; b=w7XnhVfm2Z3L8f8zynwj/cC/qqPpLWN/T5jQ51zC+BzKqxN5fKY+2Ojl1y5bsPaQJR HyC+g0340juH/SUV87NdvgfYUqLN5RAMKgjVWySQh65gHTLGw1qCxqPknuhC7uG2UeSD 3JX28zm+lfX6uLCvK/KLfN5Fexi0MaHH1Lwto= MIME-Version: 1.0 Received: by 10.229.136.144 with SMTP id r16mr5765111qct.153.1312926119387; Tue, 09 Aug 2011 14:41:59 -0700 (PDT) Received: by 10.229.65.91 with HTTP; Tue, 9 Aug 2011 14:41:59 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 14:41:59 -0700 Message-ID: From: Vishwas Manral To: Anoop Ghanwani Content-Type: multipart/alternative; boundary=00248c711805f6622a04aa196ee4 Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 21:41:31 -0000 --00248c711805f6622a04aa196ee4 Content-Type: text/plain; charset=ISO-8859-1 Hi Anoop, >From what I know they do not use Multicast GRE (I hear the extra 4 bytes in the GRE header is a proprietery extension). I think a directory based mechanism is what is used (though I think if there was a standard way to map Multicast MAC to Multicast IP, they could probably use such a standard mechanisms). Thanks, Vishwas On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani wrote: > Hi Vishwas, > > How do they get multicast through the network in that case? > Are they planning to use multicast GRE, or just use directory > based lookups and not worry about multicast applications > for now? > > Anoop > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral wrote: > >> Hi Linda, >> >> The data packets can be tunnelled at the ToR over say a GRE packet and the >> core is a Layer-3 core (except for the downstream ports). So we could have >> encapsulation/ decapsulation of L2 over GRE at the ToR. >> >> The very same thing can be done at the hypervisor layer too, in which case >> the entire DC network would look like a Layer-3 flat network including the >> ToR to server link and the hypervisor would do the tunneling. >> >> I am not sure if you got the points above or not. I know cloud OS >> companies that provide the service and have big announced customers. >> >> Thanks, >> Vishwas >> On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wrote: >> >>> Vishwas, >>> >>> In my mind the bullet 1) in the list refers to ToR switches downstream >>> ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3. >>> >>> >>> Have you seen data center networks with ToR switches downstream ports >>> (i.e. facing servers) enabling IP routing, even though the physical links >>> are Ethernet? >>> If yes, we should definitely include it in the ARMD draft. >>> >>> Thanks, >>> Linda >>> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral < >>> vishwas.ietf@gmail.com> wrote: >>> >>>> Hi Linda, >>>> I am unsure what you mean by this, but: >>>> >>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>> >>>> We can also have a heirarchical network, with the core totally Layer-3 >>>> (and having seperate routing), from the hosts still in a large Layer-3 >>>> subnet. Another aspect could be to have a totally Layer-3 network. >>>> >>>> The difference between them is the link between the servers and the ToR. >>>> >>>> Thanks, >>>> Vishwas >>>> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: >>>> >>>>> During the 81st IETF ARMD WG discussion, it was suggested that it is >>>>> necessary to document typical data center network designs so that address >>>>> resolution scaling issues can be properly described. Many data center >>>>> operators have expressed that they can't openly reveal their detailed >>>>> network designs. Therefore, we only want to document anonymous designs >>>>> without too much detail. During the journey of establishing ARMD, we have >>>>> come across the following typical data center network designs: >>>>> >>>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>>> 2. large layer 2 with hundreds (or thousands) of ToRs being >>>>> interconnected by Layer 2. This design will have thousands of hosts under >>>>> the L2/L3 boundary router (s) >>>>> 3. CLOS design with thousands of switches. This design will have >>>>> thousands of hosts under the L2/L3 boundary router(s) >>>>> >>>>> We have heard that each of the designs above has its own problems. ARMD >>>>> problem statements might need to document DC problems under each typical >>>>> design. >>>>> Please send feedback to us (either to the armd email list or to the >>>>> ARMD chair Benson & Linda) to indicate if we have missed any typical Data >>>>> Center network designs. >>>>> >>>>> Your contribution can greatly accelerate the progress of ARMD WG. >>>>> >>>>> Thank you very much. >>>>> >>>>> Linda & Benson >>>>> >>>> --00248c711805f6622a04aa196ee4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Anoop,
=A0
From what I know they do not use Multicast GRE (I hear the extra 4 byt= es in the GRE header is a proprietery extension).
=A0
I think a directory based mechanism is what is used (though I think if= there was a standard way to map=A0Multicast MAC=A0to Multicast IP,=A0they = could probably use such a standard mechanisms).
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani <= span dir=3D"ltr"><anoop@alumni.= duke.edu> wrote:
Hi Vishwas,=20

How do they get multicast through the network in that case?
Are they planning to use multicast GRE, or just use directory
based lookups and not worry about multicast applications
for now?

Anoop

On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <vi= shwas.ietf@gmail.com> wrote:
Hi Linda,
=A0
The data packets can be tunnelled at the ToR over say a GRE packet and= the core is a Layer-3 core (except for the downstream ports). So we could = have encapsulation/ decapsulation of L2 over GRE at the ToR.
=A0
The very same thing can be done at the hypervisor layer too, in which = case the entire DC network would look like a Layer-3 flat network including= the ToR to server link and the hypervisor would do the tunneling.
=A0
I am not sure if you got the points above or not. I know cloud OS comp= anies that provide the service and have big announced customers.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
Vishwas,
=A0
In my mind the bullet 1) in the list refers to ToR switches downstream= ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer = 3.
=A0
Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical links a= re Ethernet?=A0=A0
If yes, we should definitely include it in the ARMD draft.
=A0
Thanks,
Linda
On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = <vishwas.ietf@gmail.com> wrote:
Hi Linda,
I am unsure what you mean by this, but:
  1. layer 3 all the way to TOR (Top of Rack switches),
    2. =
We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
=A0
The difference between them is the link between the servers and the To= R.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
  1. layer 3 all the way to TOR (Top of Rack switches),
  2. large layer 2 with hundreds (or thousands) of ToRs being interconnecte= d by Layer 2. This design will have thousands of hosts under the L2/L3 boun= dary router (s)
  3. CLOS design =A0with thousands of switches. This design will have thous= ands of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. AR= MD problem statements might need to document DC problems under each typical= design.
Please send feedback to us (either to the=A0armd email list =A0or to = the ARMD chair Benson & Linda) to indicate if we have missed any typica= l Data Center network designs.
= =A0
Your contributi= on can greatly accelerate the progress of ARMD WG.
= =A0
Thank you very = much.
= =A0
Linda & Ben= son
--00248c711805f6622a04aa196ee4-- From ghanwani@gmail.com Tue Aug 9 14:03:30 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 565CD5E8007 for ; Tue, 9 Aug 2011 14:03:30 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.476 X-Spam-Level: X-Spam-Status: No, score=-4.476 tagged_above=-999 required=5 tests=[AWL=-1.500, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S4QBB73HBuTy for ; Tue, 9 Aug 2011 14:03:29 -0700 (PDT) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 4E6B621F8B77 for ; Tue, 9 Aug 2011 14:03:29 -0700 (PDT) Received: by qyk34 with SMTP id 34so2403667qyk.10 for ; Tue, 09 Aug 2011 14:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=7ECbwhvISLWCOZIVtGu+uf5r81WHoA7pdIE/Vl9LV3k=; b=bZm1UfRM0gTbRNCzQKjICjJ29fuBKboBSdYFcRPxKnslMhIn7SAKdn9t7bnKLbSNfw gn/rMKKXgEvUvuKQ170ObVJELSpi+kmxRPbAo9EieRwC7siB5FK0REFyw8AbC+wZDroC RHJ7oY+68X5KAqqfLhCLHh509bKaA5czdnlMc= MIME-Version: 1.0 Received: by 10.229.77.78 with SMTP id f14mr6360428qck.38.1312923836845; Tue, 09 Aug 2011 14:03:56 -0700 (PDT) Sender: ghanwani@gmail.com Received: by 10.229.216.138 with HTTP; Tue, 9 Aug 2011 14:03:56 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 14:03:56 -0700 X-Google-Sender-Auth: hYDMnHARRRMbUzvwMVpIg4db60A Message-ID: From: Anoop Ghanwani To: Vishwas Manral Content-Type: multipart/alternative; boundary=00235447143ce990c804aa18e6d0 Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 21:51:19 -0000 --00235447143ce990c804aa18e6d0 Content-Type: text/plain; charset=ISO-8859-1 Hi Vishwas, How do they get multicast through the network in that case? Are they planning to use multicast GRE, or just use directory based lookups and not worry about multicast applications for now? Anoop On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral wrote: > Hi Linda, > > The data packets can be tunnelled at the ToR over say a GRE packet and the > core is a Layer-3 core (except for the downstream ports). So we could have > encapsulation/ decapsulation of L2 over GRE at the ToR. > > The very same thing can be done at the hypervisor layer too, in which case > the entire DC network would look like a Layer-3 flat network including the > ToR to server link and the hypervisor would do the tunneling. > > I am not sure if you got the points above or not. I know cloud OS companies > that provide the service and have big announced customers. > > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wrote: > >> Vishwas, >> >> In my mind the bullet 1) in the list refers to ToR switches downstream >> ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3. >> >> >> Have you seen data center networks with ToR switches downstream ports >> (i.e. facing servers) enabling IP routing, even though the physical links >> are Ethernet? >> If yes, we should definitely include it in the ARMD draft. >> >> Thanks, >> Linda >> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: >> >>> Hi Linda, >>> I am unsure what you mean by this, but: >>> >>> 1. layer 3 all the way to TOR (Top of Rack switches), >>> >>> We can also have a heirarchical network, with the core totally Layer-3 >>> (and having seperate routing), from the hosts still in a large Layer-3 >>> subnet. Another aspect could be to have a totally Layer-3 network. >>> >>> The difference between them is the link between the servers and the ToR. >>> >>> Thanks, >>> Vishwas >>> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: >>> >>>> During the 81st IETF ARMD WG discussion, it was suggested that it is >>>> necessary to document typical data center network designs so that address >>>> resolution scaling issues can be properly described. Many data center >>>> operators have expressed that they can't openly reveal their detailed >>>> network designs. Therefore, we only want to document anonymous designs >>>> without too much detail. During the journey of establishing ARMD, we have >>>> come across the following typical data center network designs: >>>> >>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>> 2. large layer 2 with hundreds (or thousands) of ToRs being >>>> interconnected by Layer 2. This design will have thousands of hosts under >>>> the L2/L3 boundary router (s) >>>> 3. CLOS design with thousands of switches. This design will have >>>> thousands of hosts under the L2/L3 boundary router(s) >>>> >>>> We have heard that each of the designs above has its own problems. ARMD >>>> problem statements might need to document DC problems under each typical >>>> design. >>>> Please send feedback to us (either to the armd email list or to the >>>> ARMD chair Benson & Linda) to indicate if we have missed any typical Data >>>> Center network designs. >>>> >>>> Your contribution can greatly accelerate the progress of ARMD WG. >>>> >>>> Thank you very much. >>>> >>>> Linda & Benson >>>> >>> >> > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --00235447143ce990c804aa18e6d0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Vishwas,

How do they get multicast through the networ= k in that case?
Are they planning to use multicast GRE, or just u= se directory
based lookups and not worry about multicast applicat= ions
for now?

Anoop

On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <vishwas.ietf@gmail.com> wrote:
Hi Linda,
=A0
The data packets can be tunnelled at the ToR over say a GRE packet and= the core is a Layer-3 core (except for the downstream ports). So we could = have encapsulation/ decapsulation of L2 over GRE at the ToR.
=A0
The very same thing can be done at the hypervisor layer too, in which = case the entire DC network would look like a Layer-3 flat network including= the ToR to server link and the hypervisor would do the tunneling.
=A0
I am not sure if you got the points above or not. I know cloud OS comp= anies that provide the service and have big announced customers.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
Vishwas,
=A0
In my mind the bullet 1) in the list refers to ToR switches downstream= ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer = 3.
=A0
Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical links a= re Ethernet?=A0=A0
If yes, we should definitely include it in the ARMD draft.
=A0
Thanks,
Linda
On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = <vishwas.ietf@gmail.com> wrote:
Hi Linda,
I am unsure what you mean by this, but:
  1. layer 3 all the way to TOR (Top of Rack switches),
    2. <= /ol>
We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
=A0
The difference between them is the link between the servers and the To= R.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
  1. layer 3 all the way to TOR (Top of Rack switches),
  2. large layer 2 with hundreds (or thousands) of ToRs being interconnected= by Layer 2. This design will have thousands of hosts under the L2/L3 bound= ary router (s)
  3. CLOS design =A0with thousands of switches. This design will have thousa= nds of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. ARM= D problem statements might need to document DC problems under each typical = design.
Please send feedback to us (either to the=A0armd email list =A0or to t= he ARMD chair Benson & Linda) to indicate if we have missed any typical= Data Center network designs.
= =A0
Your contributio= n can greatly accelerate the progress of ARMD WG.
= =A0
Thank you very m= uch.
= =A0
Linda & Bens= on


_______________________________________________
armd mailing list
armd@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/armd


--00235447143ce990c804aa18e6d0-- From ghanwani@gmail.com Tue Aug 9 14:49:38 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D779A21F8C6E for ; Tue, 9 Aug 2011 14:49:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.726 X-Spam-Level: X-Spam-Status: No, score=-3.726 tagged_above=-999 required=5 tests=[AWL=-0.750, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 93SquH8BhWKo for ; Tue, 9 Aug 2011 14:49:38 -0700 (PDT) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id C1CAE21F8C69 for ; Tue, 9 Aug 2011 14:49:37 -0700 (PDT) Received: by qwc23 with SMTP id 23so280403qwc.31 for ; Tue, 09 Aug 2011 14:50:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=1mCKQovlP/nPpI/aa5B3trBOfaJmNJ1XsOwkE3CjZFM=; b=YNCprPzO2RLf+leDzvf4jLUZFcC62io6nbEZ+pismcasylCjm88NHoPiZN2Txub9ko bw5OJ5UhzyFvqa5a05g2qvjVNvOiZgaC+geHVV2Vedw6nwF6bPjWOA2BuU1T+5IL3oAB 9vVxDgFkCi9alFmDg4PWGou38Je22hqKQnQa0= MIME-Version: 1.0 Received: by 10.229.25.212 with SMTP id a20mr5781371qcc.148.1312926607131; Tue, 09 Aug 2011 14:50:07 -0700 (PDT) Sender: ghanwani@gmail.com Received: by 10.229.216.138 with HTTP; Tue, 9 Aug 2011 14:50:07 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Aug 2011 14:50:07 -0700 X-Google-Sender-Auth: AQRw7XAsrVQRZ1FE7UacdcJqXsc Message-ID: From: Anoop Ghanwani To: Vishwas Manral Content-Type: multipart/alternative; boundary=00163641708d08c4c404aa198cb3 Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 21:51:41 -0000 --00163641708d08c4c404aa198cb3 Content-Type: text/plain; charset=ISO-8859-1 >>>> (though I think if there was a standard way to map Multicast MAC to Multicast IP, they could probably use such a standard mechanisms). >>>> They can do that, but then this imposes requirements on the equipment to be able to do multicast forwarding, and even if does, because of pruning requirements the number of groups would be very large. The average data center switch probably won't handle that many groups. On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral wrote: > Hi Anoop, > > From what I know they do not use Multicast GRE (I hear the extra 4 bytes in > the GRE header is a proprietery extension). > > I think a directory based mechanism is what is used (though I think if > there was a standard way to map Multicast MAC to Multicast IP, they could > probably use such a standard mechanisms). > > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani wrote: > >> Hi Vishwas, >> >> How do they get multicast through the network in that case? >> Are they planning to use multicast GRE, or just use directory >> based lookups and not worry about multicast applications >> for now? >> >> Anoop >> >> On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral wrote: >> >>> Hi Linda, >>> >>> The data packets can be tunnelled at the ToR over say a GRE packet and >>> the core is a Layer-3 core (except for the downstream ports). So we could >>> have encapsulation/ decapsulation of L2 over GRE at the ToR. >>> >>> The very same thing can be done at the hypervisor layer too, in which >>> case the entire DC network would look like a Layer-3 flat network including >>> the ToR to server link and the hypervisor would do the tunneling. >>> >>> I am not sure if you got the points above or not. I know cloud OS >>> companies that provide the service and have big announced customers. >>> >>> Thanks, >>> Vishwas >>> On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wrote: >>> >>>> Vishwas, >>>> >>>> In my mind the bullet 1) in the list refers to ToR switches downstream >>>> ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3. >>>> >>>> >>>> Have you seen data center networks with ToR switches downstream ports >>>> (i.e. facing servers) enabling IP routing, even though the physical links >>>> are Ethernet? >>>> If yes, we should definitely include it in the ARMD draft. >>>> >>>> Thanks, >>>> Linda >>>> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral < >>>> vishwas.ietf@gmail.com> wrote: >>>> >>>>> Hi Linda, >>>>> I am unsure what you mean by this, but: >>>>> >>>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>>> >>>>> We can also have a heirarchical network, with the core totally Layer-3 >>>>> (and having seperate routing), from the hosts still in a large Layer-3 >>>>> subnet. Another aspect could be to have a totally Layer-3 network. >>>>> >>>>> The difference between them is the link between the servers and the >>>>> ToR. >>>>> >>>>> Thanks, >>>>> Vishwas >>>>> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: >>>>> >>>>>> During the 81st IETF ARMD WG discussion, it was suggested that it is >>>>>> necessary to document typical data center network designs so that address >>>>>> resolution scaling issues can be properly described. Many data center >>>>>> operators have expressed that they can't openly reveal their detailed >>>>>> network designs. Therefore, we only want to document anonymous designs >>>>>> without too much detail. During the journey of establishing ARMD, we have >>>>>> come across the following typical data center network designs: >>>>>> >>>>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>>>> 2. large layer 2 with hundreds (or thousands) of ToRs being >>>>>> interconnected by Layer 2. This design will have thousands of hosts under >>>>>> the L2/L3 boundary router (s) >>>>>> 3. CLOS design with thousands of switches. This design will have >>>>>> thousands of hosts under the L2/L3 boundary router(s) >>>>>> >>>>>> We have heard that each of the designs above has its own problems. >>>>>> ARMD problem statements might need to document DC problems under each >>>>>> typical design. >>>>>> Please send feedback to us (either to the armd email list or to the >>>>>> ARMD chair Benson & Linda) to indicate if we have missed any typical Data >>>>>> Center network designs. >>>>>> >>>>>> Your contribution can greatly accelerate the progress of ARMD WG. >>>>>> >>>>>> Thank you very much. >>>>>> >>>>>> Linda & Benson >>>>>> >>>>> --00163641708d08c4c404aa198cb3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
>>>>
(though I think if there was a standard way to map= =A0Multicast MAC=A0to Multicast IP,=A0they could probably use such a standa= rd mechanisms).
>>>>

They ca= n do that, but then this imposes requirements on the
equipment to be able to do multicast forwarding, and even if does,
because of pruning requirements the number of groups would be
<= div>very large. =A0The average data center switch probably won't handle=
that many groups.

On Tue, Aug= 9, 2011 at 2:41 PM, Vishwas Manral <vishwas.ietf@gmail.com> wrote:
Hi Anoop,
=A0
From what I know they do not use Multicast GRE (I hear the extra 4 byt= es in the GRE header is a proprietery extension).
=A0
I think a directory based mechanism is what is used (though I think if= there was a standard way to map=A0Multicast MAC=A0to Multicast IP,=A0they = could probably use such a standard mechanisms).
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani <= span dir=3D"ltr"><anoop@alumni.duke.edu> wrote:
Hi Vishwas,=20

How do they get multicast through the network in that case?
Are they planning to use multicast GRE, or just use directory
based lookups and not worry about multicast applications
for now?

Anoop

On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <<= a href=3D"mailto:vishwas.ietf@gmail.com" target=3D"_blank">vishwas.ietf@gma= il.com> wrote:
Hi Linda,
=A0
The data packets can be tunnelled at the ToR over say a GRE packet and= the core is a Layer-3 core (except for the downstream ports). So we could = have encapsulation/ decapsulation of L2 over GRE at the ToR.
=A0
The very same thing can be done at the hypervisor layer too, in which = case the entire DC network would look like a Layer-3 flat network including= the ToR to server link and the hypervisor would do the tunneling.
=A0
I am not sure if you got the points above or not. I know cloud OS comp= anies that provide the service and have big announced customers.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
Vishwas,
=A0
In my mind the bullet 1) in the list refers to ToR switches downstream= ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer = 3.
=A0
Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical links a= re Ethernet?=A0=A0
If yes, we should definitely include it in the ARMD draft.
=A0
Thanks,
Linda
On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = <vishwas.ietf@gmail.com> wrote:
Hi Linda,
I am unsure what you mean by this, but:
  1. layer 3 all the way to TOR (Top of Rack switches),
    2. <= /ol>
We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
=A0
The difference between them is the link between the servers and the To= R.
=A0
Thanks,
Vishwas
On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
  1. layer 3 all the way to TOR (Top of Rack switches),
  2. large layer 2 with hundreds (or thousands) of ToRs being interconnected= by Layer 2. This design will have thousands of hosts under the L2/L3 bound= ary router (s)
  3. CLOS design =A0with thousands of switches. This design will have thousa= nds of hosts under the L2/L3 boundary router(s)
We have heard that each of the designs above has its own problems. ARM= D problem statements might need to document DC problems under each typical = design.
Please send feedback to us (either to the=A0armd email list =A0or to t= he ARMD chair Benson & Linda) to indicate if we have missed any typical= Data Center network designs.
= =A0
Your contributio= n can greatly accelerate the progress of ARMD WG.
= =A0
Thank you very m= uch.
= =A0
Linda & Bens= on

--00163641708d08c4c404aa198cb3-- From kim.changhoon@gmail.com Tue Aug 9 14:55:01 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 770B921F8B5A for ; Tue, 9 Aug 2011 14:55:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.143 X-Spam-Level: X-Spam-Status: No, score=-2.143 tagged_above=-999 required=5 tests=[AWL=-0.833, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_HTML_USL_OBFU=1.666] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vHefxCh9goVv for ; Tue, 9 Aug 2011 14:55:00 -0700 (PDT) Received: from mail-yx0-f172.google.com (mail-yx0-f172.google.com [209.85.213.172]) by ietfa.amsl.com (Postfix) with ESMTP id BFF0221F8B26 for ; Tue, 9 Aug 2011 14:54:57 -0700 (PDT) Received: by yxp4 with SMTP id 4so357177yxp.31 for ; Tue, 09 Aug 2011 14:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Qeq99FoIVQkLFWnRjm49lX1aRZ0F4mBf/fdQbebAxhE=; b=Y2WYSxa14Lh1ne5UOLUX3VAT82xsLUgiOLiv9rV0yoc3RJucJ4seA34qB/9LoX5rOE avXMjxmD3bPM1XPKXdugBpaxvTpkpIEufvCHFCvdr2vTYB1eOcMH5MapaOyIR7sJkmCW v1hpe/Ceq92GCxdlCgkcHvhOUBPszJ3Cw2roI= MIME-Version: 1.0 Received: by 10.236.173.198 with SMTP id v46mr3957029yhl.237.1312926927179; Tue, 09 Aug 2011 14:55:27 -0700 (PDT) Sender: kim.changhoon@gmail.com Received: by 10.236.105.148 with HTTP; Tue, 9 Aug 2011 14:55:26 -0700 (PDT) In-Reply-To: References: <4E329CB2.8010908@cs.illinois.edu> Date: Tue, 9 Aug 2011 14:55:26 -0700 X-Google-Sender-Auth: oNe-3QnNDCCM7wRPeROU4yboNOs Message-ID: From: Changhoon Kim To: Linda Dunbar Content-Type: multipart/alternative; boundary=20cf305630371c4f1204aa199fa7 Cc: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 21:55:01 -0000 --20cf305630371c4f1204aa199fa7 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Linda, On your question 1, SEATTLE uses a particular encapsulation format (EtherIP in our current prototype) to deliver packets from an ingress to an intermediate (resolver) switch. Hence, an intermediate switch can recognize relayed packets w/o ambiguity because those packets will be destined to the intermediate switch _*and*_ delivered in the particular encap format. When an intermediate switch receives relayed packets from another ingress, and the intermediate switch doesn=92t have a forwarding entry for the actua= l destination MAC address, the intermediate switch discards the packet. We do not propose rehashing the packet and forwarding it to another intermediate exactly because of the concern you have: a forwarding loop. On your question 2, I took a look at the draft you mentioned (draft-shah-armd-arp-reduction-01.txt) and found out that the draft does no= t propose any cache update protocol that works among ToR switches. They propose the existing aging-based timeout along with gratuitous ARP (which essentially is a host-driven broadcast-based update that works only for hos= t arrival, but not for host departure or failure). In contrast to this, SEATTLE adopts an explicit, unicast-based host-information update mechanism triggered by switches. Suppose a host (h) moves from an old switch (X) to a new switch (Y) due to mobility. Another switch (Z) in the network may still have a forwarding entry for host h associated with host h=92s old location (i.e., X) and hence keep forwarding packets destined to h to X via encapsulation. To update the stale forwardin= g entry in switch Z, the old switch X directly sends an =93host-not-available= =94 message to the ingress switch (Z) whenever it receives packets destined to = h from Z. Since this is done via unicast and only when the stale entry is actually used for packet forwarding, the cache-update overhead in SEATTLE i= s very low. For details, please take a look at Section 4.3, =93Ensuring Seaml= ess Mobility=94 in the following write-up (page 16). http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf Thanks. -- Chang On Tue, Aug 9, 2011 at 12:51 PM, Linda Dunbar wrote: > Matthew, > > Questions on the details: > > 1. Distributed in-network directory service: when Ingress switch receives= a > data packet with unknown, you stated that the ingress switch will send th= e > packet to the Intermediate switch based on the hashing value. How does > "intermediate switch" know if a received data packet is relayed from anot= her > switch or sent directly by hosts? If the "intermediate switch" still does= n't > have forwarding entry for the destination address in the data packet, it > will do a "hashing" and send to another "intermediate switch. This > forwarding can form a loop. How does SEATTLE mitigate the loop issue? > > > 2. Reactive caching and invalidation: Your brief description sounds a lot > like the scheme described in "( > https://datatracker.ietf.org/doc/draft-shah-armd-arp-reduction/". Can you > elaborate the major differences? > > Thank you very much. > > Linda Dunbar > > On Fri, Jul 29, 2011 at 6:42 AM, Matthew Caesar w= rote: > >> Hi, >> >> I've been watching this list with a lot of interest. We wanted to post f= or >> your consideration a protocol called SEATTLE we've been developing, whic= h >> targets large improvements in scalability of layer-2 in the context of d= ata >> centers/cloud computing by eliminating the need for broadcasts. We think >> SEATTLE addresses some of the goals in ARMD's charter. More details belo= w, >> but we'd be quite interested to discuss/answer questions if there's >> interest. >> >> SEATTLE achieves the same configuration-free properties as Ethernet >> bridging yet scales to large networks. Unlike other proposals (e.g., TRI= LL, >> Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need = for >> network-wide broadcasting, achieving data-plane control overheads that g= row >> logarithmically with network size. SEATTLE is backwards-compatible with >> existing Ethernets, simplifying incremental deployment, and supporting >> existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATTL= E >> also supports more advanced (optional) features, such as the ability to >> search and look up services based on text strings, more flexible routing= and >> more control over layer-2 routing policies, and the ability to anycast a= nd >> load balance directly over named services. SEATTLE has two main features= : >> >> Distributed in-network directory service: The switches collectively run = a >> directory service that handles ARP and DHCP requests, as well as packets >> sent to unknown destination addresses. The directory service leverages >> consistent hashing and the link-state routing protocol to form a one-hop >> distributed hash table. The directory is a key-value store, where the ha= sh >> of the key determines the switch(es) responsible for storing the >> information. For example, for ARP queries, the key is an IP address and = the >> value includes the IP address and the host=92s location. When an ingress >> switch cannot handle an ARP/DHCP request or a data packet directly, the >> switch computes the hash and directs the packet through the responsible >> intermediate switch. The intermediate switch can response to ARP and DHC= P >> queries, and direct data packets onward to their egress switch while >> returning the destination host=92s location to the ingress switch. >> >> Reactive caching and invalidation: To minimize the portion of traffic >> directed over longer paths, an ingress switch reactively caches the >> responses from the intermediate switch. For example, while an intermedia= te >> switch may handle an ARP query, the data packets typically flow directly >> along the shortest path. In addition, other hosts connected to the same >> ingress switch benefit from the cached information. However, the cached >> information becomes stale if a host moves to a new location. SEATTLE >> includes an efficient cache invalidation protocol that reactively update= s >> the stale cache entries when a packet wrongly travels to the old egress >> switch. >> >> The first feature leads to good performance (by keeping traffic in the >> data plane, and avoiding broadcast) and self scaling (by having the >> directory service naturally scale with the size of the network), and the >> second ensures fast recovery after failures and migration. >> >> Our writeup on SEATTLE: >> >> http://www.cs.princeton.edu/~**jrex/papers/seattle08.pdf >> >> presents more details on this protocol, our experiences designing and >> implementing a system prototype, and a performance evaluation of the >> protocol through simulations and deployment in a testbed. >> >> -- Matt >> >> >> ______________________________**_________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/**listinfo/armd >> > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --20cf305630371c4f1204aa199fa7 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable

Linda,

=A0

On your question 1,=A0SEATTLE uses a particular e= ncapsulation format (EtherIP in our current prototype) to deliver packets from an ingress to an intermediate (resolver) switch. Hence, an intermediate switch can recognize relayed packets w/o ambiguity because those packets will be destined to the intermediate switch _and_ delivered in the particular encap format. =

=A0

When an intermediate switch receives relayed pack= ets from another ingress, and the intermediate switch doesn=92t have a forwarding en= try for the actual destination MAC address,=A0the intermediate switch=A0discards th= e packet. We do not propose rehashing the packet and forwarding it to another intermediate exactly beca= use of the concern you have: a forwarding loop.

=A0

On your question 2, I took a look at the draft yo= u mentioned (draft-shah-armd-arp-reduction-01.txt) and found out that the draft does not propose any cache update protocol tha= t works among ToR switches. They propose the existing aging-based timeout alo= ng with gratuitous ARP (which essentially is a host-driven broadcast-based upd= ate that works only for host arrival, but not for host departure or failure). <= /span>

=A0

In contrast to this, SEATTLE adopts an explicit, = unicast-based host-information update mechanism triggered by switches. Suppose a host (h) moves from an ol= d switch (X) to a new switch (Y) due to mobility. Another switch (Z) in the network may still have a forwarding entry for host h associated with host h= =92s old location (i.e., X) and hence keep forwarding packets destined to h to X= via encapsulation. To update the stale forwarding entry in switch Z, the old sw= itch X directly sends an =93host-not-available=94 message to the ingress switch = (Z) whenever it=A0receives packets destined to h from Z. Since this is done= via unicast and only when the stale entry is actually used for packet forwarding, the cache-upda= te overhead in SEATTLE is very low. For details, please take a look at Section 4.3, =93Ensuring Seamless Mobility=94 in the following write-up (page 16).<= /span>

=A0

http://www.cs.princeton.edu/~ch= kim/papers/seattle_tocs11.pdf

=A0

Thanks.

=A0

-- Chang

=A0

=A0


On Tue, Aug 9, 2011 at 12:51 PM, Lind= a Dunbar <dunba= r.ll@gmail.com> wrote:
Matthew,
=A0
Questions on the details:
=A0
1. Distributed in-network directory service: when Ingress switch recei= ves a data packet with unknown, you stated that the ingress switch will sen= d the packet to the Intermediate switch based on the hashing value. How doe= s "intermediate switch" know if a received data packet is relayed= from another switch or sent directly by hosts? If the "intermediate s= witch" still doesn't have forwarding entry for the destination add= ress in the data packet, it will do a "hashing" and send to anoth= er "intermediate switch. This forwarding can form a loop. How does SEA= TTLE mitigate the loop issue?
=A0
=A0
2. Reactive caching and invalidation: Your brief description sounds a = lot like the scheme described in "(https://datatracker.ietf.org/doc/d= raft-shah-armd-arp-reduction/". Can you el= aborate the major differences?
=A0
Thank you very much.
=A0
Linda Dunbar

On Fri, Jul 29, 2011 at= 6:42 AM, Matthew Caesar <caesar@cs.illinois.edu> wrote= :
Hi,
I've been watching this list with a lot of interest. We wanted to p= ost for your consideration a protocol called SEATTLE we've been develop= ing, which targets large improvements in scalability of layer-2 in the cont= ext of data centers/cloud computing by eliminating the need for broadcasts.= We think SEATTLE addresses some of the goals in ARMD's charter. More d= etails below, but we'd be quite interested to discuss/answer questions = if there's interest.

SEATTLE achieves the same configuration-free properties as Ethernet bri= dging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rb= ridges, Viking, SmartBridges), SEATTLE completely eliminates the need for n= etwork-wide broadcasting, achieving data-plane control overheads that grow = logarithmically with network size. SEATTLE is backwards-compatible with exi= sting Ethernets, simplifying incremental deployment, and supporting existin= g Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also sup= ports more advanced (optional) features, such as the ability to search and = look up services based on text strings, more flexible routing and more cont= rol over layer-2 routing policies, and the ability to anycast and load bala= nce directly over named services. SEATTLE has two main features:

Distributed in-network directory service: The switches collectively run= a directory service that handles ARP and DHCP requests, as well as packets= sent to unknown destination addresses. The directory service leverages con= sistent hashing and the link-state routing protocol to form a one-hop distr= ibuted hash table. The directory is a key-value store, where the hash of th= e key determines the switch(es) responsible for storing the information. Fo= r example, for ARP queries, the key is an IP address and the value includes= the IP address and the host=92s location. When an ingress switch cannot ha= ndle an ARP/DHCP request or a data packet directly, the switch computes the= hash and directs the packet through the responsible intermediate switch. T= he intermediate switch can response to ARP and DHCP queries, and direct dat= a packets onward to their egress switch while returning the destination hos= t=92s location to the ingress switch.

Reactive caching and invalidation: To minimize the portion of traffic d= irected over longer paths, an ingress switch reactively caches the response= s from the intermediate switch. For example, while an intermediate switch m= ay handle an ARP query, the data packets typically flow directly along the = shortest path. In addition, other hosts connected to the same ingress switc= h benefit from the cached information. However, the cached information beco= mes stale if a host moves to a new location. SEATTLE includes an efficient = cache invalidation protocol that reactively updates the stale cache entries= when a packet wrongly travels to the old egress switch.

The first feature leads to good performance (by keeping traffic in the = data plane, and avoiding broadcast) and self scaling (by having the directo= ry service naturally scale with the size of the network), and the second en= sures fast recovery after failures and migration.

Our writeup on SEATTLE:

http://www.cs.princeton.edu/~<= u>jrex/papers/seattle08.pdf

presents more details on this pr= otocol, our experiences designing and implementing a system prototype, and = a performance evaluation of the protocol through simulations and deployment= in a testbed.

-- Matt


_____________________________________________= __
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd=


_______________________________________________
armd mailing list
armd@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/armd


--20cf305630371c4f1204aa199fa7-- From linda.dunbar@huawei.com Wed Aug 10 14:23:02 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 36A9311E80B8 for ; Wed, 10 Aug 2011 14:23:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.088 X-Spam-Level: X-Spam-Status: No, score=-5.088 tagged_above=-999 required=5 tests=[AWL=-1.510, BAYES_00=-2.599, EXTRA_MPART_TYPE=1, HTML_MESSAGE=0.001, J_CHICKENPOX_33=0.6, RCVD_IN_DNSWL_MED=-4, SARE_GIF_ATTACH=1.42] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZieaTb2UuotI for ; Wed, 10 Aug 2011 14:22:58 -0700 (PDT) Received: from usaga02-in.huawei.com (usaga02-in.huawei.com [206.16.17.70]) by ietfa.amsl.com (Postfix) with ESMTP id 039E211E80B9 for ; Wed, 10 Aug 2011 14:22:58 -0700 (PDT) Received: from huawei.com (localhost [127.0.0.1]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LPQ00NSTE324D@usaga02-in.huawei.com> for armd@ietf.org; Wed, 10 Aug 2011 16:23:27 -0500 (CDT) Received: from dfweml201-edg.china.huawei.com ([172.18.4.104]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LPQ00C3XE31AT@usaga02-in.huawei.com> for armd@ietf.org; Wed, 10 Aug 2011 16:23:26 -0500 (CDT) Received: from DFWEML402-HUB.china.huawei.com (10.193.5.102) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 10 Aug 2011 14:23:21 -0700 Received: from DFWEML504-MBX.china.huawei.com ([169.254.4.122]) by DFWEML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Wed, 10 Aug 2011 14:23:25 -0700 Date: Wed, 10 Aug 2011 21:23:24 +0000 From: Linda Dunbar In-reply-to: X-Originating-IP: [10.192.11.66] To: "Shah, Himanshu" , Changhoon Kim , "armd@ietf.org" , Matthew Caesar Message-id: <4A95BA014132FF49AE685FAB4B9F17F6051894BA@dfweml504-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/related; boundary="Boundary_(ID_c9/k6vnfqQw81ualds/GIw)"; type="multipart/alternative" Content-language: en-US Accept-Language: en-US Thread-topic: [armd] For your consideration: SEATTLE Thread-index: AQHMTeTdALAlQ+jY2E+la4kw6nwQL5UHpySAgAH0X8CAACgs0IAM4WHA X-MS-Has-Attach: yes X-MS-TNEF-Correlator: References: <4E329CB2.8010908@cs.illinois.edu> <4A95BA014132FF49AE685FAB4B9F17F60517CDCC@dfweml503-mbx.china.huawei.com> Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2011 21:23:02 -0000 --Boundary_(ID_c9/k6vnfqQw81ualds/GIw) Content-type: multipart/alternative; boundary="Boundary_(ID_OFkpWp3cWXkhTzTcS2ODaw)" --Boundary_(ID_OFkpWp3cWXkhTzTcS2ODaw) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Himanshu, ARMD WG's current charter is to document scaling issues in data center networks and requirement. However, having some high level solution drafts can definitely help the WG to focus on the subset of problems which are solvable. At the ARMD WG discussion, some people stated that Address Resolution scaling is too big an issue if the desire is to build fat&big Layer 2. Therefore, it is beneficial to have a high level solution draft so that we know that the problem is solvable. You are welcome to initiate some discussion on the email list of the scheme proposed by your draft and present it at the next WG session if time allows. But we want to make sure that the Problem statement and requirement is #1 priority, and solution drafts are to scope the problems and details are to be addressed at the next stage. Linda From: Shah, Himanshu [mailto:hshah@ciena.com] Sent: Tuesday, August 02, 2011 11:23 AM To: Linda Dunbar; Changhoon Kim; armd@ietf.org; Matthew Caesar Subject: RE: [armd] For your consideration: SEATTLE Linda - I specifically asked the question at the mike about solutions related work, in the last week's WG meeting, and if you recall, Ron Bonica, said EXPLICITLY that solutions CAN NOT BE discussed in this WG as this WG belongs to OPS area, and only requirement related discussions are in charter. Has the scope of WG charter changed after the meetings were over?? Thanks, himanshu [ciena logo] From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Linda Dunbar Sent: Tuesday, August 02, 2011 10:20 AM To: Changhoon Kim; armd@ietf.org; Matthew Caesar Subject: Re: [armd] For your consideration: SEATTLE Kim and Mattew, Thank you very much for posting the SEATTLE detail to the ARMD list. SEATTLE does present a possible approach to scale address resolution scheme for Large Layer 2 network. ARMD WG is currently chartered to identify the Address Resolution issues in Data Center network, document best practices and workaround for Address Resolution scaling issues. At the last week's ARMD WG session, many Address Resolution scaling issues were discussed. Many people do feel that it is more productive for the WG to focus on a subset of problems which are solvable, so that the WG won't waste energy on too broad issues associated with Data Center. Is it be possible for you to write an IETF draft to describe the SEATTLE approach for next IETF? Thanks, Linda Dunbar From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Changhoon Kim Sent: Sunday, July 31, 2011 8:03 PM To: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE Hi, I'd like to add a minor correction to the original email about SEATTLE. - "For example, for ARP queries, the key is an IP address and the value includes the IP address and the host's location". We meant "For example, for ARP queries, the key is an IP address and the value includes the _MAC_ address and the host's location". Also, for those who are interested in the implementation details of SEATTLE, we have another write-up that include a full section explaining our prototype SEATTLE switch. http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf Thanks. -- Chang On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar > wrote: > > Hi, > > I've been watching this list with a lot of interest. We wanted to post for your consideration a protocol called SEATTLE we've been developing, which targets large improvements in scalability of layer-2 in the context of data centers/cloud computing by eliminating the need for broadcasts. We think SEATTLE addresses some of the goals in ARMD's charter. More details below, but we'd be quite interested to discuss/answer questions if there's interest. > > SEATTLE achieves the same configuration-free properties as Ethernet bridging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need for network-wide broadcasting, achieving data-plane control overheads that grow logarithmically with network size. SEATTLE is backwards-compatible with existing Ethernets, simplifying incremental deployment, and supporting existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also supports more advanced (optional) features, such as the ability to search and look up services based on text strings, more flexible routing and more control over layer-2 routing policies, and the ability to anycast and load balance directly over named services. SEATTLE has two main features: > > Distributed in-network directory service: The switches collectively run a directory service that handles ARP and DHCP requests, as well as packets sent to unknown destination addresses. The directory service leverages consistent hashing and the link-state routing protocol to form a one-hop distributed hash table. The directory is a key-value store, where the hash of the key determines the switch(es) responsible for storing the information. For example, for ARP queries, the key is an IP address and the value includes the IP address and the host's location. When an ingress switch cannot handle an ARP/DHCP request or a data packet directly, the switch computes the hash and directs the packet through the responsible intermediate switch. The intermediate switch can response to ARP and DHCP queries, and direct data packets onward to their egress switch while returning the destination host's location to the ingress switch. > > Reactive caching and invalidation: To minimize the portion of traffic directed over longer paths, an ingress switch reactively caches the responses from the intermediate switch. For example, while an intermediate switch may handle an ARP query, the data packets typically flow directly along the shortest path. In addition, other hosts connected to the same ingress switch benefit from the cached information. However, the cached information becomes stale if a host moves to a new location. SEATTLE includes an efficient cache invalidation protocol that reactively updates the stale cache entries when a packet wrongly travels to the old egress switch. > > The first feature leads to good performance (by keeping traffic in the data plane, and avoiding broadcast) and self scaling (by having the directory service naturally scale with the size of the network), and the second ensures fast recovery after failures and migration. > > Our writeup on SEATTLE: > > http://www.cs.princeton.edu/~jrex/papers/seattle08.pdf > > presents more details on this protocol, our experiences designing and implementing a system prototype, and a performance evaluation of the protocol through simulations and deployment in a testbed. > > -- Matt > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --Boundary_(ID_OFkpWp3cWXkhTzTcS2ODaw) Content-type: text/html; charset=us-ascii Content-transfer-encoding: quoted-printable

Himanshu,

 <= /p>

ARMD WG’s current c= harter is to document scaling issues in data center networks and requiremen= t.

However, having some high= level solution drafts can definitely help the WG to focus on the subset of= problems which are solvable. At the ARMD WG discussion, some people stated that Address Resolution scaling is too big an issue if = the desire is to build fat&big Layer 2. Therefore, it is beneficial to = have a high level solution draft so that we know that the problem is solvab= le. You are welcome to initiate some discussion on the email list of the scheme proposed by your draft and pres= ent it at the next WG session if time allows.

But we want to make sure = that the Problem statement and requirement is  #1 priority, and soluti= on drafts are to scope the problems and details are to be addressed at the next stage.  

 <= /p>

Linda

 <= /p>

 <= /p>

 <= /p>

From: Shah, Hi= manshu [mailto:hshah@ciena.com]
Sent: Tuesday, August 02, 2011 11:23 AM
To: Linda Dunbar; Changhoon Kim; armd@ietf.org; Matthew Caesar
Subject: RE: [armd] For your consideration: SEATTLE

 

Linda –<= /span>

 <= /p>

I specifically asked the = question at the mike about solutions related work, in the last week’s= WG meeting,

and if you recall, Ron Bo= nica, said EXPLICITLY that solutions CAN NOT BE discussed in this WG

as this WG belongs to OPS= area, and only requirement related discussions are in charter.<= /span>

 <= /p>

Has the scope of WG chart= er changed after the meetings were over??

 <= /p>

Thanks,=

himanshu

 <= /p>


3D"ciena

From: armd-bou= nces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Linda Dunbar
Sent: Tuesday, August 02, 2011 10:20 AM
To: Changhoon Kim; armd@ietf.org; Matthew Caesar
Subject: Re: [armd] For your consideration: SEATTLE

 

Kim and Mattew,

 <= /p>

Thank you very much for p= osting the SEATTLE detail to the ARMD list.

SEATTLE does present a po= ssible approach to scale address resolution scheme for Large Layer 2 networ= k.

ARMD WG is currently char= tered to identify the Address Resolution issues in Data Center network, doc= ument best practices and workaround for Address Resolution scaling issues. At the last week’s ARMD WG session, many Address Res= olution scaling issues were discussed. 

Many people do feel that = it is more productive for the WG to focus on a subset of problems which are= solvable, so that the WG won’t  waste energy on too broad issues associated with Data Center.

 <= /p>

Is it be possible for you= to write an IETF draft to describe the SEATTLE approach for next IETF?

 <= /p>

Thanks,

Linda Dunbar

 <= /p>

 <= /p>

From: armd-bou= nces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Changhoon Kim
Sent: Sunday, July 31, 2011 8:03 PM
To: armd@ietf.org
Subject: Re: [armd] For your consideration: SEATTLE

 

Hi,

 

I'd like to add a minor correction to the original e= mail about SEATTLE.

 

- "For example, for ARP queries, the key is an = IP address and the value includes the IP address and the host’s locat= ion". We meant "For example, for ARP queries, the key is an IP ad= dress and the value includes the _MAC_ address and the host’s location".

 

Also, for those who are interested in the implementa= tion details of SEATTLE, we have another write-up that include a full secti= on explaining our prototype SEATTLE switch.

 

 

Thanks.

 

-- Chang

 

  

On Fri, Jul 29, 2011 at 4:42 AM, Matthew Caesar <= caesar@cs.illinois.edu> wr= ote:

> 

> Hi,

> 

> I've been watching this list with a lot of inte= rest. We wanted to post for your consideration a protocol called SEATTLE we= 've been developing, which targets large improvements in scalability of lay= er-2 in the context of data centers/cloud computing by eliminating the need for broadcasts. We think SEATTLE address= es some of the goals in ARMD's charter. More details below, but we'd be qui= te interested to discuss/answer questions if there's interest.

> 

> SEATTLE achieves the same configuration-free pr= operties as Ethernet bridging yet scales to large networks. Unlike other pr= oposals (e.g., TRILL, Rbridges, Viking, SmartBridges), SEATTLE completely e= liminates the need for network-wide broadcasting, achieving data-plane control overheads that grow logarithmically with netw= ork size. SEATTLE is backwards-compatible with existing Ethernets, simplify= ing incremental deployment, and supporting existing Ethernet functions (e.g= ., VLANs and host bootstrapping). SEATTLE also supports more advanced (optional) features, such as the abili= ty to search and look up services based on text strings, more flexible rout= ing and more control over layer-2 routing policies, and the ability to anyc= ast and load balance directly over named services. SEATTLE has two main features:

> 

> Distributed in-network directory service: The s= witches collectively run a directory service that handles ARP and DHCP requ= ests, as well as packets sent to unknown destination addresses. The directo= ry service leverages consistent hashing and the link-state routing protocol to form a one-hop distributed hash tab= le. The directory is a key-value store, where the hash of the key determine= s the switch(es) responsible for storing the information. For example, for = ARP queries, the key is an IP address and the value includes the IP address and the host’s location. When = an ingress switch cannot handle an ARP/DHCP request or a data packet direct= ly, the switch computes the hash and directs the packet through the respons= ible intermediate switch. The intermediate switch can response to ARP and DHCP queries, and direct data packets onwar= d to their egress switch while returning the destination host’s locat= ion to the ingress switch.

> 

> Reactive caching and invalidation: To minimize = the portion of traffic directed over longer paths, an ingress switch reacti= vely caches the responses from the intermediate switch. For example, while = an intermediate switch may handle an ARP query, the data packets typically flow directly along the shortest pat= h. In addition, other hosts connected to the same ingress switch benefit fr= om the cached information. However, the cached information becomes stale if= a host moves to a new location. SEATTLE includes an efficient cache invalidation protocol that reactively = updates the stale cache entries when a packet wrongly travels to the old eg= ress switch.

> 

> The first feature leads to good performance (by= keeping traffic in the data plane, and avoiding broadcast) and self scalin= g (by having the directory service naturally scale with the size of the net= work), and the second ensures fast recovery after failures and migration.

> 

> Our writeup on SEATTLE:

> 

> 

> presents more details on this protocol, our exp= eriences designing and implementing a system prototype, and a performance e= valuation of the protocol through simulations and deployment in a testbed.<= o:p>

> 

> -- Matt

> 

> 

> _______________________________________________=

> armd mailing list

 

 

 

--Boundary_(ID_OFkpWp3cWXkhTzTcS2ODaw)-- --Boundary_(ID_c9/k6vnfqQw81ualds/GIw) Content-id: Content-type: image/gif; name=image001.gif Content-transfer-encoding: base64 Content-disposition: inline; filename=image001.gif; size=3410; creation-date="Wed, 10 Aug 2011 21:23:24 GMT"; modification-date="Wed, 10 Aug 2011 21:23:24 GMT" Content-description: image001.gif R0lGODlh3AAeAPcAANINQfb29uiCnuqVq9AALPXb4vLBzCopFcsAGbOyq+dmc8nJxeJig8gAB/3/ //TS2/r2+Pr5+eydsuZ8mauqo++0w2JiVBgXAuVyke6qvGtqXdMSRru6s/z///np7dEIPtrZ1vG9 y4yLgfru8dXU0kxKOtEANfz5+kNCMdMQRNLRzuFVeu2htFpaTPfh6NAAMdw6ZfLE0fjl6umNps8A LeTk4tQUSVNSQzw7Kfr19u6luc4AKfTO2fC1xPGqspyblIWEeuHh3s0AJdoyWc7OytosWd5BbPLx 8ZaVjHZ1aXt6bvn39+FZfunq6dglU6alnuHh4MXGwds0YYKBdvz8/NIFPDUzIc8AMPGzus0AIOh+ mtAAA9IKQAMCAMLCvfTK1d5JcuFSdNUEF9orVeno5umFoPTM1pSUitgeUOqKpPry9u7u7L6+uPnx 8vXW36KhmXJxZSMiDn18cd7d2ubm5N9SedUNQt/f3Od5l/fe5Pfi5uNmiMTEvuNqidEAOO+jqvz+ /JKRiNUbR+JdgdMRRe3s65iXjtIUNaCfl9IEOueAnOVWZeRpivff5dcbTdckTdjX0/PI0+VvjdMO Q7e1sNYYTNIAOJGQhtrb2I+OhdQKQGZlWOzr6nh3a9YSQuqKo9ECOmBfUKWjneRpjP39/szMxjk5 J9ICOPT08/Dw7/Xz9eRwj4mJfz8+LcDAutMSRdMQRjIxH9MQQW5tX4iGe9DPy2loW0JAMP36/OBO dOqPp6ion6Sim/PM18jHwt09Z5mZj15dTtMUR9ICO9ACOVhXSNMEPEdFNh4dCcwAHy8uGzc2JcTD vvz19//9/4B/dNzb2ZSTidIBO/f39rCvqdYYQ1BPP++xwPv7/NQANueMo8/RzdfY1Ofo5Orn50lI OdEDO+yYr/C4xlBOP7i3sfz9/FdZSFpXR/Cuwf78/dAEMdIHNhAOAOLk4P7//uPj4NggUfDv7tkn V+qRqO6dpfv7+pOUi8jIxOZ3lN9IbqOjm+6uv8vKxudwhKmooP///yH5BAAAAAAALAAAAADcAB4A AAj/AP8JHEiw4MAObUgZXMiwocOHECNKnEixosWLGDO6YSRlRYyMIEOKHEmypEmLjdAkI5CsiriT MGPKnEkzpDtFLTcQQpDPQc2fQIMKJemuDoFXr2zsKAJhqNOnUKP+u4mgig1YCHL5lMq1q9eRMuQh GAvg49ezaNM+dIEnzJ5IauPKjeqg7taK7to84MHDxV2I7upCtOvA3dzDQiOEmMGI0Z49M67JmCpj RLMRMk4sjMHEBpdJmiaBMUDQRYwvZr4Y8PDPATp8DBjNK1DQHSAZPebh2zOKkRYWehALj+lAghQT O2hcoUFjBwJJ/x7Iq1QEnqMKBTvgeyHkhZ/vJrqH/xs4IZudFKcIXHO3gkByAkI0jReYLlw+eC+S X1nunhCepsMFGBI2gwixAzgATDJJCilMggAT//QCH3JCsECQA0xkYcIkGwhDyAYbyJKFHW4IhAEB JoDywgur1CGEHwpOAkByGQjkQiUImPBBggvKyAUNCPyihoBEWtQBEzl+COIkXHCRQjJ9RGeHHxt8 YEmNArmDRxangEgIOMpxUaUQihimhSWEpLmBDcN8kIIwOoEohDxDuiCPCQBwUUUibU4C5wbEIDBK kYRKJEEWoAjzygaTmHDFii80sIKUVFqJ5T8GmPDCoim8oMkKYFQxCSHJ1GEmmmoCoOkVfnxIiDCg mP8Qwj8u2JAMcx94AksiL4j5KgGy0FbosAudMIQQHm7wIzwT6KDDDGNA9wAAlV6ZJRNCpKATDWjM +g8DeJZ6apqEpGCCI3sw4sgLrnJxhS7/5FHEL5/sE8MDDxgArpgbVPGChcQGPFAMO8Dy4SQvMMHa QJf9w8OUVVqyj0AyVPMCiCac8pJA87C6QxnjEvKKH5WQ9k8kaJiQ5iQ0aPFPOY0wNAgNIAJAgACG CRzwADt4SMgVUkTAkBkQc2ECwNeYAA6jNICx1QnH0eCJsGeKDAANqxA0wQ5isqzIQA7EUAYDKzDw SS8CvMDhjDjrHPAeJix69QQNEU3lJASUIVAaBHD/IQwAV2Q9FSMvIKe3QFUrS8AABM3DNdNf/xMD DCvusIOB0mzA4AY2t+32sODK/QK8Q0/5yiRXHC7ADgCcvsOk/2BQuBAr3JU4FwTMkPM/aQjRNQ0C /POFMS8m+AIN8F2RbOe7w9T85yetcvHpNEBXuh+np87xFb6+4IQ4DBDwQhYwAIg4morrPlDvXef+ jxFZaEuIJVIoopsUxmjLPEYj/IFFOQypxSZqwRV+2OIeJinFEwJQEF50ggxe4MBEJICsD5nABl8o 3SnStAPrheAUidBJCmDxgUfRIBdtKMjtcrc79nGOBhIYgSbiRgg/yGNh/9DBDtZ2s+dJpB6L2MIW /+hBkHiIoBT/oEYXKCESciDBhxHxRxcQYZJNsIMEBQnGBXzxjVYohCBP8MdCeNA3nRDiBfDYh2am wpd/mEETrdqACYqQg38AoghXUFMKuGAMG2iDIF8AmQDQhzv17c13jLoCC3phDGmU6wV7IMge8si5 Hl4kD2LYQgMWcRdqsCMJ/2ADMngBgiYIZAnMIGARydAEftBBIJxgwxz+sYRxIKMUR4AGLv4BhRr8 IwCYEEgp+CC0I9BhDZj4ATIS8A9MQAMbA1lCFBaQDlV0owlReMdAVOCFIwxkAQtI4hSgyQ9XLOEf cDjAO/xhiKkwgx//IMMylgEJZxSkHDDIApweaf+CX/ShDwxQx6T0YAOVpckEdQjBPqSAPTPWEAYe 6IAHQqAFddDgBGkgJAvXh0jUseABxhjGK+ZXBBe0RgJ+WFolPVcRB/RjC2LwwUDucIw4KMMebDhA Kw5wA1TEwwKtwEE7ByKKW5TgANaYBhGsgYJvsEEfB4hDKEQQCwr8owQl+McZSgCCKcQBGcFYQwJu cYxxKOEWXkhAHH6QsxrYYhlW4AMFjvGNOJhiG/9AwjFuYQFUOGMWF0DGG5CggWkYogStCAUqgKCM d0QDCAFIZwmoAYSoctUgGcgCDWwQpw/QIAvJEEIDGPCPDsBgBx96BQAKlz/NyY9BhBCEEx6RiCv/ NOAQvDuF3Da6N64Jg2USyIEmXuAhczmBEXX4zgZ2GzyMUCESZiDINAJxAA2AwAsXSIISLiAKVrBD CSjogisGcokuaCAJyDDEJuLAijgcwBehiEUCIHGAYnDgqxz4hgUygYwpKFMOP+jCDdiAiFaYIxab WMNAdtEFFDxjGmfIrj6kSok4FGMWXWDFJdgxBWDUwhZxwIYontEJdlCDFVaYQwvGkQB2WEEENSDC LW5BgVQYxB0TQMAO3iSMHtvABpUQAmn/EY4ssKvHwvhAFT6Qnx184FV/MwENXnAFEzRgCA4oAyF3 YMh/fMLJL0zDP+qAgA/0+IzweZHmKqm3POgA/4cgaccy/8GHC2jTFEAIxQFYkQQckGMg9rhADZqA Alsc4xhIaMEN/kGLWwgECQfAgQVacAtTEGEWyxBIKMYRjVjMkhmxCCw0CNKNJBSjGFFAAg4K8Y9O WEEDB7AFEIoBBMQORAmtuAMQkiCCZeiDFqaAwixCEYBnWOMcEmzBJhriDgGoYyVKA0UiTmHlMBjG AYMI7aM0RQMhVCEXv8iCEFhVhSr4gTvrGMQD/iGJZOwAPg1orkAm0IBkXEG0GPiHHpygYxNUARQ7 yII88vlu0eLhH/gwAekmEgH//UUF7DgGCJR4D1RcgBW0YIcInpEJBgpEDl24QxDikIT1vqETT//4 hwa6YIgIkAEZXUhAILqgjH8Ygh1AQAIy5CACdnDjH0/gcAmUgUWBHOEHnehCC0SgDCVQIA4WuG8o gPEMMoigC0D4gQosYApDdCEJwGAHEpSADBLcwNG8wHAr/nEDZOiDEw2JxCAEkQ0/WMISeBrDpTow DynIQhMAEMQvJhADdzRDF373gwnMU4RVwEUg4ZCCEYwAgyLoYHc6kMcvjPCLIkjARgwQBLUtUQ0m NCIE86K85XNohI1J5AQKEOIfdneETXTBFuRQxj28YQUKTCMUprBCIAgSCGVAAwQoeMIdvoGDW0hw F4Hlwz82cQBO0LcT/4iABpBxgWAEwB/KwCv/NYjuiu8OpBTHwMEBKPEDK7QCGcpQgc1NkdhzWoAd 7EDEFG6ggmJ8QwPLsAuZcAvcAAebQALWEAuxQEU3Fwfyx2y44Sw6cA1uoAbPowYF8AV9YT4C0Qxu UAEskAEPMAIFcQJtAAFLkAMjsEsDcQIjkAMQAAEjsEatIQMGwAIVkAeG4Q4jgIIqqBmAMAJQ1BCY JEScRBCqsAAgQArtoBBBMA2/NE02NhBHEARTUQNQyAleoAJC4w4g4Auo8A/GJBA14HHlUAv8AIXT AAVU8A/TYIX/cAd38EXTQAJeMEuBEAscAA1w+A8kwAe+9A9UsABEMA2o0A1ieA930AT1oAqvL7QG hUAF3OAKsySIKsAPQgM9J+EAsRdTOkMLB0AEmjgsJ4AFITCERAIJlGBKThEQADs= --Boundary_(ID_c9/k6vnfqQw81ualds/GIw)-- From ping@pingpan.org Wed Aug 10 17:03:00 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 900C721F8BA6 for ; Wed, 10 Aug 2011 17:03:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.088 X-Spam-Level: X-Spam-Status: No, score=-5.088 tagged_above=-999 required=5 tests=[AWL=-0.778, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4, SARE_HTML_USL_OBFU=1.666] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jMtNgyK6r0B7 for ; Wed, 10 Aug 2011 17:02:59 -0700 (PDT) Received: from exprod7og113.obsmtp.com (exprod7og113.obsmtp.com [64.18.2.179]) by ietfa.amsl.com (Postfix) with SMTP id 4F82321F8BA7 for ; Wed, 10 Aug 2011 17:02:57 -0700 (PDT) Received: from mail-ew0-f47.google.com ([209.85.215.47]) (using TLSv1) by exprod7ob113.postini.com ([64.18.6.12]) with SMTP ID DSNKTkMcPsYw1R6z7VL9YCwQg9BSnnIUAG4B@postini.com; Wed, 10 Aug 2011 17:03:31 PDT Received: by ewy5 with SMTP id 5so1057890ewy.34 for ; Wed, 10 Aug 2011 17:03:09 -0700 (PDT) Received: by 10.213.111.6 with SMTP id q6mr11507ebp.41.1313020989144; Wed, 10 Aug 2011 17:03:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.4.81 with HTTP; Wed, 10 Aug 2011 17:02:27 -0700 (PDT) In-Reply-To: References: <4E329CB2.8010908@cs.illinois.edu> From: Ping Pan Date: Wed, 10 Aug 2011 17:02:27 -0700 Message-ID: To: Changhoon Kim Content-Type: multipart/alternative; boundary=001636d34483a3fecd04aa2f85d4 Cc: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 00:03:00 -0000 --001636d34483a3fecd04aa2f85d4 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable It sounds to achieve the goal, we need two things (a) tunneling between two nodes, and (b) some mechanism to resolve MAC. Curious, doesn't TRILL ESADI suppose to be doing this too? Or am I missing something here. Regards, - Ping On Tue, Aug 9, 2011 at 2:55 PM, Changhoon Kim wrote= : > Linda, > > > > On your question 1, SEATTLE uses a particular encapsulation format (Ether= IP > in our current prototype) to deliver packets from an ingress to an > intermediate (resolver) switch. Hence, an intermediate switch can recogni= ze > relayed packets w/o ambiguity because those packets will be destined to t= he > intermediate switch _*and*_ delivered in the particular encap format. > > > > When an intermediate switch receives relayed packets from another ingress= , > and the intermediate switch doesn=92t have a forwarding entry for the act= ual > destination MAC address, the intermediate switch discards the packet. We = do > not propose rehashing the packet and forwarding it to another intermediat= e > exactly because of the concern you have: a forwarding loop. > > > > On your question 2, I took a look at the draft you mentioned > (draft-shah-armd-arp-reduction-01.txt) and found out that the draft does = not > propose any cache update protocol that works among ToR switches. They > propose the existing aging-based timeout along with gratuitous ARP (which > essentially is a host-driven broadcast-based update that works only for h= ost > arrival, but not for host departure or failure). > > > > In contrast to this, SEATTLE adopts an explicit, unicast-based > host-information update mechanism triggered by switches. Suppose a host (= h) > moves from an old switch (X) to a new switch (Y) due to mobility. Another > switch (Z) in the network may still have a forwarding entry for host h > associated with host h=92s old location (i.e., X) and hence keep forwardi= ng > packets destined to h to X via encapsulation. To update the stale forward= ing > entry in switch Z, the old switch X directly sends an =93host-not-availab= le=94 > message to the ingress switch (Z) whenever it receives packets destined t= o h > from Z. Since this is done via unicast and only when the stale entry is > actually used for packet forwarding, the cache-update overhead in SEATTLE= is > very low. For details, please take a look at Section 4.3, =93Ensuring Sea= mless > Mobility=94 in the following write-up (page 16). > > > > http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf > > > > Thanks. > > > > -- Chang > > > > > > On Tue, Aug 9, 2011 at 12:51 PM, Linda Dunbar wrote= : > >> Matthew, >> >> Questions on the details: >> >> 1. Distributed in-network directory service: when Ingress switch receive= s >> a data packet with unknown, you stated that the ingress switch will send= the >> packet to the Intermediate switch based on the hashing value. How does >> "intermediate switch" know if a received data packet is relayed from ano= ther >> switch or sent directly by hosts? If the "intermediate switch" still doe= sn't >> have forwarding entry for the destination address in the data packet, it >> will do a "hashing" and send to another "intermediate switch. This >> forwarding can form a loop. How does SEATTLE mitigate the loop issue? >> >> >> 2. Reactive caching and invalidation: Your brief description sounds a lo= t >> like the scheme described in "( >> https://datatracker.ietf.org/doc/draft-shah-armd-arp-reduction/". Can yo= u >> elaborate the major differences? >> >> Thank you very much. >> >> Linda Dunbar >> >> On Fri, Jul 29, 2011 at 6:42 AM, Matthew Caesar = wrote: >> >>> Hi, >>> >>> I've been watching this list with a lot of interest. We wanted to post >>> for your consideration a protocol called SEATTLE we've been developing, >>> which targets large improvements in scalability of layer-2 in the conte= xt of >>> data centers/cloud computing by eliminating the need for broadcasts. We >>> think SEATTLE addresses some of the goals in ARMD's charter. More detai= ls >>> below, but we'd be quite interested to discuss/answer questions if ther= e's >>> interest. >>> >>> SEATTLE achieves the same configuration-free properties as Ethernet >>> bridging yet scales to large networks. Unlike other proposals (e.g., TR= ILL, >>> Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the need= for >>> network-wide broadcasting, achieving data-plane control overheads that = grow >>> logarithmically with network size. SEATTLE is backwards-compatible with >>> existing Ethernets, simplifying incremental deployment, and supporting >>> existing Ethernet functions (e.g., VLANs and host bootstrapping). SEATT= LE >>> also supports more advanced (optional) features, such as the ability to >>> search and look up services based on text strings, more flexible routin= g and >>> more control over layer-2 routing policies, and the ability to anycast = and >>> load balance directly over named services. SEATTLE has two main feature= s: >>> >>> Distributed in-network directory service: The switches collectively run= a >>> directory service that handles ARP and DHCP requests, as well as packet= s >>> sent to unknown destination addresses. The directory service leverages >>> consistent hashing and the link-state routing protocol to form a one-ho= p >>> distributed hash table. The directory is a key-value store, where the h= ash >>> of the key determines the switch(es) responsible for storing the >>> information. For example, for ARP queries, the key is an IP address and= the >>> value includes the IP address and the host=92s location. When an ingres= s >>> switch cannot handle an ARP/DHCP request or a data packet directly, the >>> switch computes the hash and directs the packet through the responsible >>> intermediate switch. The intermediate switch can response to ARP and DH= CP >>> queries, and direct data packets onward to their egress switch while >>> returning the destination host=92s location to the ingress switch. >>> >>> Reactive caching and invalidation: To minimize the portion of traffic >>> directed over longer paths, an ingress switch reactively caches the >>> responses from the intermediate switch. For example, while an intermedi= ate >>> switch may handle an ARP query, the data packets typically flow directl= y >>> along the shortest path. In addition, other hosts connected to the same >>> ingress switch benefit from the cached information. However, the cached >>> information becomes stale if a host moves to a new location. SEATTLE >>> includes an efficient cache invalidation protocol that reactively updat= es >>> the stale cache entries when a packet wrongly travels to the old egress >>> switch. >>> >>> The first feature leads to good performance (by keeping traffic in the >>> data plane, and avoiding broadcast) and self scaling (by having the >>> directory service naturally scale with the size of the network), and th= e >>> second ensures fast recovery after failures and migration. >>> >>> Our writeup on SEATTLE: >>> >>> http://www.cs.princeton.edu/~**jrex/papers/seattle08.pdf >>> >>> presents more details on this protocol, our experiences designing and >>> implementing a system prototype, and a performance evaluation of the >>> protocol through simulations and deployment in a testbed. >>> >>> -- Matt >>> >>> >>> ______________________________**_________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/**listinfo/armd >>> >> >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> >> > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --001636d34483a3fecd04aa2f85d4 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
It sounds to achieve the goal, we need two things (a) tunneling betwee= n two nodes, and (b) some mechanism to resolve MAC.

Curious, doesn't TRILL=A0ESADI suppose to be doing this too? Or = am I missing something here.

Regards,

- Ping


On Tue, Aug 9, 2011 at 2:55 PM, Changhoo= n Kim <chkim= @cs.princeton.edu> wrote:

Linda,

=A0

On your question 1,=A0SEATTLE uses a = particular encapsulation format (EtherIP in our current prototype) to deliver packets from an ingress to an intermediate (resolver) switch. Hence, an intermediate switch can recognize relayed packets w/o ambiguity because those packets will be destined to the intermediate switch _and_ delivered in the particular encap format. =

=A0

When an intermediate switch receives = relayed packets from another ingress, and the intermediate switch doesn=92t have a forwarding en= try for the actual destination MAC address,=A0the intermediate switch=A0discards th= e packet. We do not propose rehashing the packet and forwarding it to another intermediate exactly beca= use of the concern you have: a forwarding loop.

=A0

On your question 2, I took a look at = the draft you mentioned (draft-shah-armd-arp-reduction-01.txt) and found out that the draft does not propose any cache update protocol tha= t works among ToR switches. They propose the existing aging-based timeout alo= ng with gratuitous ARP (which essentially is a host-driven broadcast-based upd= ate that works only for host arrival, but not for host departure or failure). <= /span>

=A0

In contrast to this, SEATTLE adopts a= n explicit, unicast-based host-information update mechanism triggered by switches. Suppose a host (h) moves from an ol= d switch (X) to a new switch (Y) due to mobility. Another switch (Z) in the network may still have a forwarding entry for host h associated with host h= =92s old location (i.e., X) and hence keep forwarding packets destined to h to X= via encapsulation. To update the stale forwarding entry in switch Z, the old sw= itch X directly sends an =93host-not-available=94 message to the ingress switch = (Z) whenever it=A0receives packets destined to h from Z. Since this is done= via unicast and only when the stale entry is actually used for packet forwarding, the cache-upda= te overhead in SEATTLE is very low. For details, please take a look at Section 4.3, =93Ensuring Seamless Mobility=94 in the following write-up (page 16).<= /span>

=A0

= http://www.cs.p= rinceton.edu/~chkim/papers/seattle_tocs11.pdf

=A0

Thanks.

=A0

-- Chang

=A0

=A0


On Tue, Aug 9, 2011 at 12:51 PM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
Matthew,
=A0
Questions on the details:
=A0
1. Distributed in-network directory service: when Ingress switch recei= ves a data packet with unknown, you stated that the ingress switch will sen= d the packet to the Intermediate switch based on the hashing value. How doe= s "intermediate switch" know if a received data packet is relayed= from another switch or sent directly by hosts? If the "intermediate s= witch" still doesn't have forwarding entry for the destination add= ress in the data packet, it will do a "hashing" and send to anoth= er "intermediate switch. This forwarding can form a loop. How does SEA= TTLE mitigate the loop issue?
=A0
=A0
=A0
Thank you very much.
=A0
Linda Dunbar

On Fri, Jul 29, 2011 at 6:42 AM, Mat= thew Caesar <caesar@cs.illinois.edu> wrote:
Hi,

I've been watching this list with a lot of interest. We wanted to p= ost for your consideration a protocol called SEATTLE we've been develop= ing, which targets large improvements in scalability of layer-2 in the cont= ext of data centers/cloud computing by eliminating the need for broadcasts.= We think SEATTLE addresses some of the goals in ARMD's charter. More d= etails below, but we'd be quite interested to discuss/answer questions = if there's interest.

SEATTLE achieves the same configuration-free properties as Ethernet bri= dging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rb= ridges, Viking, SmartBridges), SEATTLE completely eliminates the need for n= etwork-wide broadcasting, achieving data-plane control overheads that grow = logarithmically with network size. SEATTLE is backwards-compatible with exi= sting Ethernets, simplifying incremental deployment, and supporting existin= g Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also sup= ports more advanced (optional) features, such as the ability to search and = look up services based on text strings, more flexible routing and more cont= rol over layer-2 routing policies, and the ability to anycast and load bala= nce directly over named services. SEATTLE has two main features:

Distributed in-network directory service: The switches collectively run= a directory service that handles ARP and DHCP requests, as well as packets= sent to unknown destination addresses. The directory service leverages con= sistent hashing and the link-state routing protocol to form a one-hop distr= ibuted hash table. The directory is a key-value store, where the hash of th= e key determines the switch(es) responsible for storing the information. Fo= r example, for ARP queries, the key is an IP address and the value includes= the IP address and the host=92s location. When an ingress switch cannot ha= ndle an ARP/DHCP request or a data packet directly, the switch computes the= hash and directs the packet through the responsible intermediate switch. T= he intermediate switch can response to ARP and DHCP queries, and direct dat= a packets onward to their egress switch while returning the destination hos= t=92s location to the ingress switch.

Reactive caching and invalidation: To minimize the portion of traffic d= irected over longer paths, an ingress switch reactively caches the response= s from the intermediate switch. For example, while an intermediate switch m= ay handle an ARP query, the data packets typically flow directly along the = shortest path. In addition, other hosts connected to the same ingress switc= h benefit from the cached information. However, the cached information beco= mes stale if a host moves to a new location. SEATTLE includes an efficient = cache invalidation protocol that reactively updates the stale cache entries= when a packet wrongly travels to the old egress switch.

The first feature leads to good performance (by keeping traffic in the = data plane, and avoiding broadcast) and self scaling (by having the directo= ry service naturally scale with the size of the network), and the second en= sures fast recovery after failures and migration.

Our writeup on SEATTLE:

http://www.cs.princeton.edu/~<= u>jrex/papers/seattle08.pdf

presents more details on this pr= otocol, our experiences designing and implementing a system prototype, and = a performance evaluation of the protocol through simulations and deployment= in a testbed.

-- Matt


_____________________________________________= __
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd=


_______________________________________________
armd mailing list
armd@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/armd



_______________________________________________
armd mailing list
armd@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/armd


--001636d34483a3fecd04aa2f85d4-- From kim.changhoon@gmail.com Thu Aug 11 15:55:53 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8A9FA11E8092 for ; Thu, 11 Aug 2011 15:55:53 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.727 X-Spam-Level: X-Spam-Status: No, score=-1.727 tagged_above=-999 required=5 tests=[AWL=-0.416, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_HTML_USL_OBFU=1.666] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sk2ggVwuR+-N for ; Thu, 11 Aug 2011 15:55:52 -0700 (PDT) Received: from mail-vx0-f172.google.com (mail-vx0-f172.google.com [209.85.220.172]) by ietfa.amsl.com (Postfix) with ESMTP id 9C64311E809B for ; Thu, 11 Aug 2011 15:55:51 -0700 (PDT) Received: by vxi29 with SMTP id 29so2511757vxi.31 for ; Thu, 11 Aug 2011 15:56:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=b4Jf85LIGb/J+MlgpUIOzvX5J4q6RH+dLpEokyLTkss=; b=hDqMDk365+Vx6M47V+zGmsz36X+rRsGzkMr4eXzH2nvY+YCBiTToB9DFJlGXnHBKsL NZDpxbsef5lIcmEOeZ/OPPTElhLe6IuyQGsvLkI0n6r7MfVfbCO05SRKwA/a6CimeRlP tGwwW56nrzMVwSfST/kqj8z9YKZ4GToGvFtbw= MIME-Version: 1.0 Received: by 10.52.70.68 with SMTP id k4mr122227vdu.506.1313103386708; Thu, 11 Aug 2011 15:56:26 -0700 (PDT) Sender: kim.changhoon@gmail.com Received: by 10.52.158.100 with HTTP; Thu, 11 Aug 2011 15:56:26 -0700 (PDT) In-Reply-To: References: <4E329CB2.8010908@cs.illinois.edu> Date: Thu, 11 Aug 2011 15:56:26 -0700 X-Google-Sender-Auth: jwBm6SGzIEhN1-mgUyk01fdbS7I Message-ID: From: Changhoon Kim To: Ping Pan Content-Type: multipart/alternative; boundary=20cf307ac2cbeb0ba104aa42b4ed Cc: armd@ietf.org Subject: Re: [armd] For your consideration: SEATTLE X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 22:55:53 -0000 --20cf307ac2cbeb0ba104aa42b4ed Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Ping, Yes, as far as I understand, the TRILL ESADI protocol offers an optional mechanism for RBrides to exchange end-station information (address= , vlan id, and location) explicitly, in addition to learning from the data plane. The key differences between SEATTLE and ESADI are as follows. - In ESADI, every RBridge in a VLAN learns the whole end-station information announced by other RBridges in the same VLAN. In SEATTLE, ea= ch switch learns only a fraction of end-station information in a network (o= r a VLAN). Doing so reduces control-plane overhead needed to maintain a larg= e amount of end-station information (e.g., memory foot-print of the host-information table, such as ARP and L2 forwarding table) consistentl= y and timely. - ESADI uses multicast for end-station information dissemination. In contrast, SEATTLE switches register and resolve end-station information (between ingress and relay) via unicast, conducive to reliable delivery. Thanks to this reliability, SEATTLE switches never flood unica= st frames. - Because SEATTLE switches disseminate end-station information only when and where it is needed, SEATTLE switches can update stale end-station information (due to mobility or failure) more efficiently and rapidly. In summary, while ESADI is similar to the control-plane functions of SEATTLE, it doesn't particularly address the scalability problems that can arise in a large cloud DC where number and sizes (in terms of # of member end stations) of VLANs are huge, and, furthermore, lots of VLANs can potentially overlap even at leaf switches. -- Chang On Wed, Aug 10, 2011 at 5:02 PM, Ping Pan wrote: > It sounds to achieve the goal, we need two things (a) tunneling between t= wo > nodes, and (b) some mechanism to resolve MAC. > > Curious, doesn't TRILL ESADI suppose to be doing this too? Or am I missin= g > something here. > > Regards, > > - Ping > > > > On Tue, Aug 9, 2011 at 2:55 PM, Changhoon Kim wro= te: > >> Linda, >> >> >> >> On your question 1, SEATTLE uses a particular encapsulation format >> (EtherIP in our current prototype) to deliver packets from an ingress to= an >> intermediate (resolver) switch. Hence, an intermediate switch can recogn= ize >> relayed packets w/o ambiguity because those packets will be destined to = the >> intermediate switch _*and*_ delivered in the particular encap format. >> >> >> >> When an intermediate switch receives relayed packets from another ingres= s, >> and the intermediate switch doesn=92t have a forwarding entry for the ac= tual >> destination MAC address, the intermediate switch discards the packet. We= do >> not propose rehashing the packet and forwarding it to another intermedia= te >> exactly because of the concern you have: a forwarding loop. >> >> >> >> On your question 2, I took a look at the draft you mentioned >> (draft-shah-armd-arp-reduction-01.txt) and found out that the draft does= not >> propose any cache update protocol that works among ToR switches. They >> propose the existing aging-based timeout along with gratuitous ARP (whic= h >> essentially is a host-driven broadcast-based update that works only for = host >> arrival, but not for host departure or failure). >> >> >> >> In contrast to this, SEATTLE adopts an explicit, unicast-based >> host-information update mechanism triggered by switches. Suppose a host = (h) >> moves from an old switch (X) to a new switch (Y) due to mobility. Anothe= r >> switch (Z) in the network may still have a forwarding entry for host h >> associated with host h=92s old location (i.e., X) and hence keep forward= ing >> packets destined to h to X via encapsulation. To update the stale forwar= ding >> entry in switch Z, the old switch X directly sends an =93host-not-availa= ble=94 >> message to the ingress switch (Z) whenever it receives packets destined = to h >> from Z. Since this is done via unicast and only when the stale entry is >> actually used for packet forwarding, the cache-update overhead in SEATTL= E is >> very low. For details, please take a look at Section 4.3, =93Ensuring Se= amless >> Mobility=94 in the following write-up (page 16). >> >> >> >> http://www.cs.princeton.edu/~chkim/papers/seattle_tocs11.pdf >> >> >> >> Thanks. >> >> >> >> -- Chang >> >> >> >> >> >> On Tue, Aug 9, 2011 at 12:51 PM, Linda Dunbar wrote= : >> >>> Matthew, >>> >>> Questions on the details: >>> >>> 1. Distributed in-network directory service: when Ingress switch receiv= es >>> a data packet with unknown, you stated that the ingress switch will sen= d the >>> packet to the Intermediate switch based on the hashing value. How does >>> "intermediate switch" know if a received data packet is relayed from an= other >>> switch or sent directly by hosts? If the "intermediate switch" still do= esn't >>> have forwarding entry for the destination address in the data packet, i= t >>> will do a "hashing" and send to another "intermediate switch. This >>> forwarding can form a loop. How does SEATTLE mitigate the loop issue? >>> >>> >>> 2. Reactive caching and invalidation: Your brief description sounds a l= ot >>> like the scheme described in "( >>> https://datatracker.ietf.org/doc/draft-shah-armd-arp-reduction/". Can >>> you elaborate the major differences? >>> >>> Thank you very much. >>> >>> Linda Dunbar >>> >>> On Fri, Jul 29, 2011 at 6:42 AM, Matthew Caesar wrote: >>> >>>> Hi, >>>> >>>> I've been watching this list with a lot of interest. We wanted to post >>>> for your consideration a protocol called SEATTLE we've been developing= , >>>> which targets large improvements in scalability of layer-2 in the cont= ext of >>>> data centers/cloud computing by eliminating the need for broadcasts. W= e >>>> think SEATTLE addresses some of the goals in ARMD's charter. More deta= ils >>>> below, but we'd be quite interested to discuss/answer questions if the= re's >>>> interest. >>>> >>>> SEATTLE achieves the same configuration-free properties as Ethernet >>>> bridging yet scales to large networks. Unlike other proposals (e.g., T= RILL, >>>> Rbridges, Viking, SmartBridges), SEATTLE completely eliminates the nee= d for >>>> network-wide broadcasting, achieving data-plane control overheads that= grow >>>> logarithmically with network size. SEATTLE is backwards-compatible wit= h >>>> existing Ethernets, simplifying incremental deployment, and supporting >>>> existing Ethernet functions (e.g., VLANs and host bootstrapping). SEAT= TLE >>>> also supports more advanced (optional) features, such as the ability t= o >>>> search and look up services based on text strings, more flexible routi= ng and >>>> more control over layer-2 routing policies, and the ability to anycast= and >>>> load balance directly over named services. SEATTLE has two main featur= es: >>>> >>>> Distributed in-network directory service: The switches collectively ru= n >>>> a directory service that handles ARP and DHCP requests, as well as pac= kets >>>> sent to unknown destination addresses. The directory service leverages >>>> consistent hashing and the link-state routing protocol to form a one-h= op >>>> distributed hash table. The directory is a key-value store, where the = hash >>>> of the key determines the switch(es) responsible for storing the >>>> information. For example, for ARP queries, the key is an IP address an= d the >>>> value includes the IP address and the host=92s location. When an ingre= ss >>>> switch cannot handle an ARP/DHCP request or a data packet directly, th= e >>>> switch computes the hash and directs the packet through the responsibl= e >>>> intermediate switch. The intermediate switch can response to ARP and D= HCP >>>> queries, and direct data packets onward to their egress switch while >>>> returning the destination host=92s location to the ingress switch. >>>> >>>> Reactive caching and invalidation: To minimize the portion of traffic >>>> directed over longer paths, an ingress switch reactively caches the >>>> responses from the intermediate switch. For example, while an intermed= iate >>>> switch may handle an ARP query, the data packets typically flow direct= ly >>>> along the shortest path. In addition, other hosts connected to the sam= e >>>> ingress switch benefit from the cached information. However, the cache= d >>>> information becomes stale if a host moves to a new location. SEATTLE >>>> includes an efficient cache invalidation protocol that reactively upda= tes >>>> the stale cache entries when a packet wrongly travels to the old egres= s >>>> switch. >>>> >>>> The first feature leads to good performance (by keeping traffic in the >>>> data plane, and avoiding broadcast) and self scaling (by having the >>>> directory service naturally scale with the size of the network), and t= he >>>> second ensures fast recovery after failures and migration. >>>> >>>> Our writeup on SEATTLE: >>>> >>>> http://www.cs.princeton.edu/~**jrex/papers/seattle08.pdf >>>> >>>> presents more details on this protocol, our experiences designing and >>>> implementing a system prototype, and a performance evaluation of the >>>> protocol through simulations and deployment in a testbed. >>>> >>>> -- Matt >>>> >>>> >>>> ______________________________**_________________ >>>> armd mailing list >>>> armd@ietf.org >>>> https://www.ietf.org/mailman/**listinfo/armd >>>> >>> >>> >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >>> >>> >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> >> > --20cf307ac2cbeb0ba104aa42b4ed Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Ping,
=A0
Yes, as far as I understand, the TRILL E= SADI=A0protocol=A0offers an optional=A0mechanism for RBrides to exchange en= d-station information (address, vlan id, and location) explicitly, in addit= ion to=A0learning from the data plane. The key differences between SEATTLE = and ESADI are as follows.=A0
  • In=A0ESADI, every RBridge in a VLAN learns the=A0whole end-station = information announced by=A0other RBridges in the same VLAN. In SEATTLE, eac= h switch learns only a fraction of end-station information in=A0a network (= or a VLAN).=A0Doing so reduces=A0control-plane overhead=A0needed to maintai= n=A0a large amount of end-station information=A0(e.g., memory foot-print=A0= of the host-information table, such as ARP and L2 forwarding table) consist= ently and timely.
  • ESADI uses multicast for end-station information dissemination. In cont= rast, SEATTLE switches=A0register and resolve end-station information (betw= een ingress and=A0relay) via unicast,=A0conducive to=A0reliable delivery.= =A0Thanks to this reliability,=A0SEATTLE switches=A0never=A0flood unicast f= rames.
  • Because=A0SEATTLE switches disseminate end-station information only whe= n and where it is needed, SEATTLE switches can=A0update stale end-station i= nformation (due to mobility or failure) more efficiently and rapidly.
    • <= /ul>
    In summary, while ESADI is similar to the control-plane functions of S= EATTLE,=A0it doesn't particularly address the scalability=A0problems th= at can arise in a large cloud DC where number and sizes (in terms of # of m= ember end stations) of VLANs are huge, and, furthermore, lots of VLANs=A0ca= n potentially overlap even at leaf switches.
    =A0
    --=A0Chang=A0

    =A0
    On Wed, Aug 10, 2011 at 5:02 PM, Ping Pan <ping@pingpan.org>= wrote:
    It sounds to achieve the goal, we need tw= o things (a) tunneling between two nodes, and (b) some mechanism to resolve= MAC.

    Curious, doesn't TRILL=A0ESADI suppose to be doing this too? Or am I missi= ng something here.

    <= div>Regards,

    - Ping



    On Tue, Aug 9, 2011 at 2:55 PM, Changhoo= n Kim <chkim@cs.princeton.edu> wrote:

    Linda,

    =A0

    On your question 1,=A0SEATTLE u= ses a particular encapsulation format (EtherIP in our current prototype) to deliver packets from an ingress to an intermediate (resolver) switch. Hence, an intermediate switch can recognize relayed packets w/o ambiguity because those packets will be destined to the intermediate switch _and_ delivered in the particular encap format. =

    =A0

    When an intermediate switch rec= eives relayed packets from another ingress, and the intermediate switch doesn=92t have a forwarding en= try for the actual destination MAC address,=A0the intermediate switch=A0discards th= e packet. We do not propose rehashing the packet and forwarding it to another intermediate exactly beca= use of the concern you have: a forwarding loop.

    =A0

    On your question 2, I took a lo= ok at the draft you mentioned (draft-shah-armd-arp-reduction-01.txt) and found out that the draft does not propose any cache update protocol tha= t works among ToR switches. They propose the existing aging-based timeout alo= ng with gratuitous ARP (which essentially is a host-driven broadcast-based upd= ate that works only for host arrival, but not for host departure or failure). <= /span>

    =A0

    In contrast to this, SEATTLE ad= opts an explicit, unicast-based host-information update mechanism triggered by switches. Suppose a host (h) moves from an ol= d switch (X) to a new switch (Y) due to mobility. Another switch (Z) in the network may still have a forwarding entry for host h associated with host h= =92s old location (i.e., X) and hence keep forwarding packets destined to h to X= via encapsulation. To update the stale forwarding entry in switch Z, the old sw= itch X directly sends an =93host-not-available=94 message to the ingress switch = (Z) whenever it=A0receives packets destined to h from Z. Since this is done= via unicast and only when the stale entry is actually used for packet forwarding, the cache-upda= te overhead in SEATTLE is very low. For details, please take a look at Section 4.3, =93Ensuring Seamless Mobility=94 in the following write-up (page 16).<= /span>

    On Tue, Au= g 9, 2011 at 12:51 PM, Linda Dunbar <dunbar.ll@gmail.com> = wrote:
    Matthew,
    =A0
    Questions on the details:
    =A0
    1. Distributed in-network directory service: when Ingress switch recei= ves a data packet with unknown, you stated that the ingress switch will sen= d the packet to the Intermediate switch based on the hashing value. How doe= s "intermediate switch" know if a received data packet is relayed= from another switch or sent directly by hosts? If the "intermediate s= witch" still doesn't have forwarding entry for the destination add= ress in the data packet, it will do a "hashing" and send to anoth= er "intermediate switch. This forwarding can form a loop. How does SEA= TTLE mitigate the loop issue?
    =A0
    =A0
    2. Reactive caching and invalidation: Your brief description sounds a = lot like the scheme described in "(https://datatracker.ietf.org/doc/d= raft-shah-armd-arp-reduction/". Can you el= aborate the major differences?
    =A0
    Thank you very much.
    =A0
    Linda Dunbar

    On Fri, Jul 29, 2011 at 6:42 AM, Mat= thew Caesar <caesar@cs.illinois.edu> wrote:
    Hi,

    I've been watching this list with a lot of interest. We wanted to p= ost for your consideration a protocol called SEATTLE we've been develop= ing, which targets large improvements in scalability of layer-2 in the cont= ext of data centers/cloud computing by eliminating the need for broadcasts.= We think SEATTLE addresses some of the goals in ARMD's charter. More d= etails below, but we'd be quite interested to discuss/answer questions = if there's interest.

    SEATTLE achieves the same configuration-free properties as Ethernet bri= dging yet scales to large networks. Unlike other proposals (e.g., TRILL, Rb= ridges, Viking, SmartBridges), SEATTLE completely eliminates the need for n= etwork-wide broadcasting, achieving data-plane control overheads that grow = logarithmically with network size. SEATTLE is backwards-compatible with exi= sting Ethernets, simplifying incremental deployment, and supporting existin= g Ethernet functions (e.g., VLANs and host bootstrapping). SEATTLE also sup= ports more advanced (optional) features, such as the ability to search and = look up services based on text strings, more flexible routing and more cont= rol over layer-2 routing policies, and the ability to anycast and load bala= nce directly over named services. SEATTLE has two main features:

    Distributed in-network directory service: The switches collectively run= a directory service that handles ARP and DHCP requests, as well as packets= sent to unknown destination addresses. The directory service leverages con= sistent hashing and the link-state routing protocol to form a one-hop distr= ibuted hash table. The directory is a key-value store, where the hash of th= e key determines the switch(es) responsible for storing the information. Fo= r example, for ARP queries, the key is an IP address and the value includes= the IP address and the host=92s location. When an ingress switch cannot ha= ndle an ARP/DHCP request or a data packet directly, the switch computes the= hash and directs the packet through the responsible intermediate switch. T= he intermediate switch can response to ARP and DHCP queries, and direct dat= a packets onward to their egress switch while returning the destination hos= t=92s location to the ingress switch.

    Reactive caching and invalidation: To minimize the portion of traffic d= irected over longer paths, an ingress switch reactively caches the response= s from the intermediate switch. For example, while an intermediate switch m= ay handle an ARP query, the data packets typically flow directly along the = shortest path. In addition, other hosts connected to the same ingress switc= h benefit from the cached information. However, the cached information beco= mes stale if a host moves to a new location. SEATTLE includes an efficient = cache invalidation protocol that reactively updates the stale cache entries= when a packet wrongly travels to the old egress switch.

    The first feature leads to good performance (by keeping traffic in the = data plane, and avoiding broadcast) and self scaling (by having the directo= ry service naturally scale with the size of the network), and the second en= sures fast recovery after failures and migration.

    Our writeup on SEATTLE:

    http://www.cs.princeton.edu/~<= u>jrex/papers/seattle08.pdf

    presents more details on this pr= otocol, our experiences designing and implementing a system prototype, and = a performance evaluation of the protocol through simulations and deployment= in a testbed.

    -- Matt


    _____________________________________________= __
    armd mailing list
    armd@ietf.org
    https://www.ietf.org/mailman/listinfo/armd=


    _______________________________________________
    armd mailing list
    armd@ietf.org
    ht= tps://www.ietf.org/mailman/listinfo/armd



    _______________________________________________
    armd mailing list
    armd@ietf.org
    ht= tps://www.ietf.org/mailman/listinfo/armd



    --20cf307ac2cbeb0ba104aa42b4ed-- From warren@kumari.net Thu Aug 11 19:43:40 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 64D5F21F8686 for ; Thu, 11 Aug 2011 19:43:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.46 X-Spam-Level: X-Spam-Status: No, score=-102.46 tagged_above=-999 required=5 tests=[AWL=0.139, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K-s9MhmCUGOc for ; Thu, 11 Aug 2011 19:43:40 -0700 (PDT) Received: from vimes.kumari.net (vimes.kumari.net [198.186.192.250]) by ietfa.amsl.com (Postfix) with ESMTP id E7B0421F867A for ; Thu, 11 Aug 2011 19:43:39 -0700 (PDT) Received: from [192.168.1.4] (24-104-73-2-ip-static.hfc.comcastbusiness.net [24.104.73.2]) by vimes.kumari.net (Postfix) with ESMTPSA id 27AC01B40A8D for ; Thu, 11 Aug 2011 22:44:12 -0400 (EDT) From: Warren Kumari Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Date: Thu, 11 Aug 2011 19:44:09 -0700 References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> To: armd@ietf.org Message-Id: Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) Subject: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 02:43:40 -0000 Hi there all, At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a = bit of a hissy fit about how VM mobility can be implemented without the = need for L2 between the hosts. So, here is a very drafty draft, outlining the principle at a high = level=85 I have no real plans for this draft, it's more just an explanation of = the idea=85 W Begin forwarded message: > From: internet-drafts@ietf.org > Date: August 11, 2011 5:53:35 PM PDT > To: warren@kumari.net > Cc: joel.halpern@ericsson.com, warren@kumari.net > Subject: New Version Notification for = draft-wkumari-dcops-l3-vmmobility-00.txt >=20 > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has = been successfully submitted by Warren Kumari and posted to the IETF = repository. >=20 > Filename: draft-wkumari-dcops-l3-vmmobility > Revision: 00 > Title: Virtual Machine mobility in L3 Networks. > Creation date: 2011-08-11 > WG ID: Individual Submission > Number of pages: 8 >=20 > Abstract: > This document outlines how Virtual Machine mobility can be > accomplished in datacenter networks that are based on L3 > technologies. It is not really intended to solve (or fully define) > the problem, but rather to outline it at a very high level to > determine if standardization within the IETF makes sense. >=20 >=20 >=20 >=20 > The IETF Secretariat >=20 From vishwas.ietf@gmail.com Thu Aug 11 20:40:10 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5975021F8781 for ; Thu, 11 Aug 2011 20:40:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.017 X-Spam-Level: X-Spam-Status: No, score=-3.017 tagged_above=-999 required=5 tests=[AWL=0.581, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ImJl9DuqbMAL for ; Thu, 11 Aug 2011 20:40:09 -0700 (PDT) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id 1FADB21F877D for ; Thu, 11 Aug 2011 20:40:09 -0700 (PDT) Received: by qwc23 with SMTP id 23so1789477qwc.31 for ; Thu, 11 Aug 2011 20:40:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AwkuOA/0wljstps9Lq3p077eV/kOiifhu4caXpQk2tc=; b=o1SnnHd8mEl79x0RfFrU4ppKEUg7i7iB/EBk2i1yBaUnMQA61I/g1/wL6MjnnicmTU qaurEl6Ugzkq1ls/Atve5vmRUnHkW9b/0ZSXT9fKwJboZfk0Ys3Glibpff89xQK8uSqh mgH/c/V9Pjx4+T+Hf46xvE3lrjAsHKj3Hw3ug= MIME-Version: 1.0 Received: by 10.229.52.76 with SMTP id h12mr292220qcg.73.1313120444851; Thu, 11 Aug 2011 20:40:44 -0700 (PDT) Received: by 10.229.79.7 with HTTP; Thu, 11 Aug 2011 20:40:44 -0700 (PDT) In-Reply-To: References: Date: Thu, 11 Aug 2011 20:40:44 -0700 Message-ID: From: Vishwas Manral To: Anoop Ghanwani Content-Type: multipart/alternative; boundary=0016364eeb6ea9a4d604aa46add6 Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 03:40:10 -0000 --0016364eeb6ea9a4d604aa46add6 Content-Type: text/plain; charset=ISO-8859-1 Hi Linda/ Anoop, Here is the example of the design I was talking about, as defined by google. http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt Thanks, Vishwas On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani wrote: > > >>>> > (though I think if there was a standard way to map Multicast MAC to > Multicast IP, they could probably use such a standard mechanisms). > >>>> > > They can do that, but then this imposes requirements on the > equipment to be able to do multicast forwarding, and even if does, > because of pruning requirements the number of groups would be > very large. The average data center switch probably won't handle > that many groups. > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral wrote: > >> Hi Anoop, >> >> From what I know they do not use Multicast GRE (I hear the extra 4 bytes >> in the GRE header is a proprietery extension). >> >> I think a directory based mechanism is what is used (though I think if >> there was a standard way to map Multicast MAC to Multicast IP, they could >> probably use such a standard mechanisms). >> >> Thanks, >> Vishwas >> On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani wrote: >> >>> Hi Vishwas, >>> >>> How do they get multicast through the network in that case? >>> Are they planning to use multicast GRE, or just use directory >>> based lookups and not worry about multicast applications >>> for now? >>> >>> Anoop >>> >>> On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral >> > wrote: >>> >>>> Hi Linda, >>>> >>>> The data packets can be tunnelled at the ToR over say a GRE packet and >>>> the core is a Layer-3 core (except for the downstream ports). So we could >>>> have encapsulation/ decapsulation of L2 over GRE at the ToR. >>>> >>>> The very same thing can be done at the hypervisor layer too, in which >>>> case the entire DC network would look like a Layer-3 flat network including >>>> the ToR to server link and the hypervisor would do the tunneling. >>>> >>>> I am not sure if you got the points above or not. I know cloud OS >>>> companies that provide the service and have big announced customers. >>>> >>>> Thanks, >>>> Vishwas >>>> On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar wrote: >>>> >>>>> Vishwas, >>>>> >>>>> In my mind the bullet 1) in the list refers to ToR switches downstream >>>>> ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer 3. >>>>> >>>>> >>>>> Have you seen data center networks with ToR switches downstream ports >>>>> (i.e. facing servers) enabling IP routing, even though the physical links >>>>> are Ethernet? >>>>> If yes, we should definitely include it in the ARMD draft. >>>>> >>>>> Thanks, >>>>> Linda >>>>> On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral < >>>>> vishwas.ietf@gmail.com> wrote: >>>>> >>>>>> Hi Linda, >>>>>> I am unsure what you mean by this, but: >>>>>> >>>>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>>>> >>>>>> We can also have a heirarchical network, with the core totally Layer-3 >>>>>> (and having seperate routing), from the hosts still in a large Layer-3 >>>>>> subnet. Another aspect could be to have a totally Layer-3 network. >>>>>> >>>>>> The difference between them is the link between the servers and the >>>>>> ToR. >>>>>> >>>>>> Thanks, >>>>>> Vishwas >>>>>> On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar wrote: >>>>>> >>>>>>> During the 81st IETF ARMD WG discussion, it was suggested that it is >>>>>>> necessary to document typical data center network designs so that address >>>>>>> resolution scaling issues can be properly described. Many data center >>>>>>> operators have expressed that they can't openly reveal their detailed >>>>>>> network designs. Therefore, we only want to document anonymous designs >>>>>>> without too much detail. During the journey of establishing ARMD, we have >>>>>>> come across the following typical data center network designs: >>>>>>> >>>>>>> 1. layer 3 all the way to TOR (Top of Rack switches), >>>>>>> 2. large layer 2 with hundreds (or thousands) of ToRs being >>>>>>> interconnected by Layer 2. This design will have thousands of hosts under >>>>>>> the L2/L3 boundary router (s) >>>>>>> 3. CLOS design with thousands of switches. This design will have >>>>>>> thousands of hosts under the L2/L3 boundary router(s) >>>>>>> >>>>>>> We have heard that each of the designs above has its own problems. >>>>>>> ARMD problem statements might need to document DC problems under each >>>>>>> typical design. >>>>>>> Please send feedback to us (either to the armd email list or to the >>>>>>> ARMD chair Benson & Linda) to indicate if we have missed any typical Data >>>>>>> Center network designs. >>>>>>> >>>>>>> Your contribution can greatly accelerate the progress of ARMD WG. >>>>>>> >>>>>>> Thank you very much. >>>>>>> >>>>>>> Linda & Benson >>>>>>> >>>>>> > --0016364eeb6ea9a4d604aa46add6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
    Hi Linda/ Anoop,
    =A0
    Here is the example of the design I was talking about, as defined by g= oogle.
    http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
    =A0
    Thanks,
    Vishwas
    On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani <= span dir=3D"ltr"><anoop@alumni.= duke.edu> wrote:

    >>>>=20
    (though I think if there was a standard way to map=A0Multicast MAC=A0t= o Multicast IP,=A0they could probably use such a standard mechanisms).
    >>>>

    They can do that, but then this imposes requirements on the
    equipment to be able to do multicast forwarding, and even if does,
    because of pruning requirements the number of groups would be
    very large. =A0The average data center switch probably won't handl= e
    that many groups.

    On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral <= span dir=3D"ltr"><vishwas.ietf@gmail.com> wrote:
    Hi Anoop,
    =A0
    From what I know they do not use Multicast GRE (I hear the extra 4 byt= es in the GRE header is a proprietery extension).
    =A0
    I think a directory based mechanism is what is used (though I think if= there was a standard way to map=A0Multicast MAC=A0to Multicast IP,=A0they = could probably use such a standard mechanisms).
    =A0
    Thanks,
    Vishwas
    On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani <= span dir=3D"ltr"><anoop@alumni.duke.edu> wrote:
    Hi Vishwas,=20

    How do they get multicast through the network in that case?
    Are they planning to use multicast GRE, or just use directory
    based lookups and not worry about multicast applications
    for now?

    Anoop

    On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral <<= a href=3D"mailto:vishwas.ietf@gmail.com" target=3D"_blank">vishwas.ietf@gma= il.com> wrote:
    Hi Linda,
    =A0
    The data packets can be tunnelled at the ToR over say a GRE packet and= the core is a Layer-3 core (except for the downstream ports). So we could = have encapsulation/ decapsulation of L2 over GRE at the ToR.
    =A0
    The very same thing can be done at the hypervisor layer too, in which = case the entire DC network would look like a Layer-3 flat network including= the ToR to server link and the hypervisor would do the tunneling.
    =A0
    I am not sure if you got the points above or not. I know cloud OS comp= anies that provide the service and have big announced customers.
    =A0
    Thanks,
    Vishwas
    On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
    Vishwas,
    =A0
    In my mind the bullet 1) in the list refers to ToR switches downstream= ports (facing servers) running Layer 2 and ToR uplinks ports run IP Layer = 3.
    =A0
    Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical links a= re Ethernet?=A0=A0
    If yes, we should definitely include it in the ARMD draft.
    =A0
    Thanks,
    Linda
    On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = <vishwas.ietf@gmail.com> wrote:
    Hi Linda,
    I am unsure what you mean by this, but:
    1. layer 3 all the way to TOR (Top of Rack switches),
      2. =
    We can also have a heirarchical network, with the core totally Layer-3= (and having seperate routing), from the hosts still in a large Layer-3 sub= net. Another aspect could be to have a totally Layer-3 network.
    =A0
    The difference between them is the link between the servers and the To= R.
    =A0
    Thanks,
    Vishwas
    On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com> wrote:
    During the 81st IETF ARMD WG discussion, it was suggested that it is n= ecessary to document typical data center network designs=A0so that address = resolution scaling issues can be properly described. Many data center opera= tors have expressed that they can't openly reveal their detailed networ= k designs. Therefore, we only want to document anonymous designs without to= o much detail. During the journey of establishing ARMD, we have come across= the following typical data center network designs:
    1. layer 3 all the way to TOR (Top of Rack switches),
    2. large layer 2 with hundreds (or thousands) of ToRs being interconnecte= d by Layer 2. This design will have thousands of hosts under the L2/L3 boun= dary router (s)
    3. CLOS design =A0with thousands of switches. This design will have thous= ands of hosts under the L2/L3 boundary router(s)
    We have heard that each of the designs above has its own problems. AR= MD problem statements might need to document DC problems under each typical= design.
    Please send feedback to us (either to the=A0armd email list =A0or to = the ARMD chair Benson & Linda) to indicate if we have missed any typica= l Data Center network designs.
    = =A0
    Your contributi= on can greatly accelerate the progress of ARMD WG.
    = =A0
    Thank you very = much.
    = =A0
    Linda & Ben= son


    --0016364eeb6ea9a4d604aa46add6-- From vishwas.ietf@gmail.com Thu Aug 11 21:05:10 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 742EB21F8888 for ; Thu, 11 Aug 2011 21:05:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.039 X-Spam-Level: X-Spam-Status: No, score=-3.039 tagged_above=-999 required=5 tests=[AWL=0.559, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id udT99DLOvqbw for ; Thu, 11 Aug 2011 21:05:09 -0700 (PDT) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 960E621F888A for ; Thu, 11 Aug 2011 21:05:09 -0700 (PDT) Received: by qyk34 with SMTP id 34so128516qyk.10 for ; Thu, 11 Aug 2011 21:05:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=4C9lLTowCl+h/qAfpDwWvEFu7OpYYzQo7iHPgwoR/nA=; b=XI3j6A+0zRtaA4FddyYL5dOXTxR0+pVA+pd65wztZyaaXl6bXFJGKFaxKZcKvcPpwo MHrEJEot6HLgIWae3ydiAnes6xl7fYqgxfP9zNOvKiFMEHcMlNl563WxxqjDwPO949zM 5SiMtuIwk8H8tjpaEVMYcv+QCMIUWmGuPmtoc= MIME-Version: 1.0 Received: by 10.229.227.20 with SMTP id iy20mr276919qcb.225.1313121944125; Thu, 11 Aug 2011 21:05:44 -0700 (PDT) Received: by 10.229.79.7 with HTTP; Thu, 11 Aug 2011 21:05:44 -0700 (PDT) In-Reply-To: References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> Date: Thu, 11 Aug 2011 21:05:44 -0700 Message-ID: From: Vishwas Manral To: Warren Kumari Content-Type: multipart/alternative; boundary=00163630f24706c33004aa4707c7 Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 04:05:10 -0000 --00163630f24706c33004aa4707c7 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Warren, It is good to see the draft. I guess you can position this as an informational draft of how network operators are using techniques to overcome big layer-2 issues (the ARP issues themselves are resolved by the directory mechanism). I think this approach has been discussed in VNRG for some time now. May be you can use the terms defined there. Also if I understand right, we are still using a big layer-2 network, only thing by using a heirarchical overlay network we are doing away with the actual network devices switches/ routers using Layer-2? As is well known "Overlay networks" have issues with scalability, which become worse as the number of overlays increases (think of all IPsec VPN's)= . How do we deal with the same? Also what about issues with fragmentation/ management of hypervisor based shim etc? It would be good if you could give all such details. Thanks, Vishwas On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari wrote: > Hi there all, > > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a b= it > of a hissy fit about how VM mobility can be implemented without the need = for > L2 between the hosts. > > So, here is a very drafty draft, outlining the principle at a high level= =85 > > I have no real plans for this draft, it's more just an explanation of the > idea=85 > > W > > > Begin forwarded message: > > > From: internet-drafts@ietf.org > > Date: August 11, 2011 5:53:35 PM PDT > > To: warren@kumari.net > > Cc: joel.halpern@ericsson.com, warren@kumari.net > > Subject: New Version Notification for > draft-wkumari-dcops-l3-vmmobility-00.txt > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been > successfully submitted by Warren Kumari and posted to the IETF repository= . > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > Revision: 00 > > Title: Virtual Machine mobility in L3 Networks. > > Creation date: 2011-08-11 > > WG ID: Individual Submission > > Number of pages: 8 > > > > Abstract: > > This document outlines how Virtual Machine mobility can be > > accomplished in datacenter networks that are based on L3 > > technologies. It is not really intended to solve (or fully define) > > the problem, but rather to outline it at a very high level to > > determine if standardization within the IETF makes sense. > > > > > > > > > > The IETF Secretariat > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > --00163630f24706c33004aa4707c7 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
    Hi Warren,
    =A0
    It is good to see the draft. I guess you can position this as an infor= mational draft of how network operators are using techniques to overcome bi= g layer-2 issues (the ARP issues themselves are resolved by the directory m= echanism). I think this approach has been discussed in VNRG for some time n= ow. May be you can use the terms defined there.
    =A0
    Also if I understand right, we are still using a big layer-2 network, = only thing by using a heirarchical overlay network we are doing away with t= he actual network devices switches/ routers=A0using Layer-2?
    =A0
    As is well known "Overlay networks" have issues with scalabi= lity, which become worse as the number of overlays increases (think of all = IPsec VPN's). How do we deal with the same? Also what about issues with= fragmentation/ management of hypervisor based=A0shim etc?
    =A0
    It would be good if you could give all such details.
    =A0
    Thanks,
    Vishwas
    On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari <= span dir=3D"ltr"><warren@kumari.net= > wrote:
    Hi there all,

    At two prev= ious meetings (IETF in Quebec and NANOG in Denver) I threw a bit of a hissy= fit about how VM mobility can be implemented without the need for L2 betwe= en the hosts.

    So, here is a very drafty draft, outlining the principle at a high leve= l=85

    I have no real plans for this draft, it's more just an expl= anation of the idea=85

    W


    Begin forwarded message:

    > From: internet-drafts@ietf= .org
    > Date: August 11, 2011 5:53:35 PM PDT
    > To: warren@kumari.net
    > Cc: joel.halpern@ericsson.com, warren@kumari.net
    > Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobilit= y-00.txt
    >
    > A new version of I-D, draft-wkumari-dcops-l3-vmmob= ility-00.txt has been successfully submitted by Warren Kumari and posted to= the IETF repository.
    >
    > Filename: =A0 =A0 =A0draft-wkumari-dcops-l3-vmmobility
    >= Revision: =A0 =A0 =A000
    > Title: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Vir= tual Machine mobility in L3 Networks.
    > Creation date: =A0 =A0 =A0 = =A0 2011-08-11
    > WG ID: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Individual Su= bmission
    > Number of pages: 8
    >
    > Abstract:
    > =A0 This document= outlines how Virtual Machine mobility can be
    > =A0 accomplished in d= atacenter networks that are based on L3
    > =A0 technologies. =A0It is = not really intended to solve (or fully define)
    > =A0 the problem, but rather to outline it at a very high level to
    &= gt; =A0 determine if standardization within the IETF makes sense.
    >>
    >
    >
    > The IETF Secretariat
    >

    ________= _______________________________________
    armd mailing list
    armd@ietf.org
    = ht= tps://www.ietf.org/mailman/listinfo/armd

    --00163630f24706c33004aa4707c7-- From warren@kumari.net Thu Aug 11 21:32:48 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4779721F86D7 for ; Thu, 11 Aug 2011 21:32:48 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k0J-8c1otSZ9 for ; Thu, 11 Aug 2011 21:32:47 -0700 (PDT) Received: from vimes.kumari.net (vimes.kumari.net [198.186.192.250]) by ietfa.amsl.com (Postfix) with ESMTP id 4F48821F86C7 for ; Thu, 11 Aug 2011 21:32:47 -0700 (PDT) Received: from [172.29.164.5] (unknown [216.239.45.130]) by vimes.kumari.net (Postfix) with ESMTPSA id 5D9701B411A1; Fri, 12 Aug 2011 00:33:22 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=windows-1252 From: Warren Kumari In-Reply-To: Date: Thu, 11 Aug 2011 21:33:20 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> To: Vishwas Manral X-Mailer: Apple Mail (2.1084) Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 04:32:48 -0000 On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: > Hi Warren, > =20 > It is good to see the draft. And it's good to see that someone has already read it :-P > I guess you can position this as an informational draft of how network = operators are using techniques to overcome big layer-2 issues (the ARP = issues themselves are resolved by the directory mechanism). I think this = approach has been discussed in VNRG for some time now. May be you can = use the terms defined there. Yup. > =20 > Also if I understand right, we are still using a big layer-2 network, = only thing by using a heirarchical overlay network we are doing away = with the actual network devices switches/ routers using Layer-2? Kinda -- the very high level view is that datacenters can be built with = L3 designs (L2 designs, or whatever design happens to exists), and then = separate overlay networks, one per customer. > =20 > As is well known "Overlay networks" have issues with scalability, = which become worse as the number of overlays increases (think of all = IPsec VPN's). Yes, but as the overlay is only built between the VM hosts (and only = *those* that are actually communicating), the number of overlays that = needs to be tracked *per device* is fairly limited. Also, the overlays = are built on general purpose servers (and are invisible to the network) = and so you don't have all of the standard scaling issues that happen = with network devices... > How do we deal with the same? Also what about issues with = fragmentation/ management of hypervisor based shim etc? Yup, there is still lots of work to be done on specifying things like = that -- who does fragmentation / reassembly, a standard protocol for = populating the directory, how the DS informs the shims that a VM has = moved, etc are all (currently) coverd by hand-waving -- I do actually = have answers on most of that, but I don't have them written down=85. > =20 > It would be good if you could give all such details. > =20 > Thanks, > Vishwas > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari = wrote: > Hi there all, >=20 > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw = a bit of a hissy fit about how VM mobility can be implemented without = the need for L2 between the hosts. >=20 > So, here is a very drafty draft, outlining the principle at a high = level=85 >=20 > I have no real plans for this draft, it's more just an explanation of = the idea=85 >=20 > W >=20 >=20 > Begin forwarded message: >=20 > > From: internet-drafts@ietf.org > > Date: August 11, 2011 5:53:35 PM PDT > > To: warren@kumari.net > > Cc: joel.halpern@ericsson.com, warren@kumari.net > > Subject: New Version Notification for = draft-wkumari-dcops-l3-vmmobility-00.txt > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has = been successfully submitted by Warren Kumari and posted to the IETF = repository. > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > Revision: 00 > > Title: Virtual Machine mobility in L3 Networks. > > Creation date: 2011-08-11 > > WG ID: Individual Submission > > Number of pages: 8 > > > > Abstract: > > This document outlines how Virtual Machine mobility can be > > accomplished in datacenter networks that are based on L3 > > technologies. It is not really intended to solve (or fully = define) > > the problem, but rather to outline it at a very high level to > > determine if standardization within the IETF makes sense. > > > > > > > > > > The IETF Secretariat > > >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd >=20 From vishwas.ietf@gmail.com Thu Aug 11 22:38:50 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 89D0921F85B8 for ; Thu, 11 Aug 2011 22:38:50 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.06 X-Spam-Level: X-Spam-Status: No, score=-3.06 tagged_above=-999 required=5 tests=[AWL=0.538, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id myLYMsptKbEw for ; Thu, 11 Aug 2011 22:38:49 -0700 (PDT) Received: from mail-qy0-f179.google.com (mail-qy0-f179.google.com [209.85.216.179]) by ietfa.amsl.com (Postfix) with ESMTP id 6E8AF21F85B2 for ; Thu, 11 Aug 2011 22:38:49 -0700 (PDT) Received: by qyk35 with SMTP id 35so1587808qyk.10 for ; Thu, 11 Aug 2011 22:38:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MaIg8THKcL5pPTskn3Z50tulX/NX9aymnZnywVc91+I=; b=V9kvQ65GCyO3iTNEmYU10h7aCf4e5v/5GElTaLoQUBx/KNxh2eVewLq2LnovOJrJ6G AMuq0jL759lK9MH2u5v8kRTgoTUJzOXyjCEBO0O2qT/FUh65HfwLj2gt79ibiy5W/i+9 PUE9gC3EL5MvM4WUjIpMGHWhMhjI2cY4aDZns= MIME-Version: 1.0 Received: by 10.229.134.68 with SMTP id i4mr308169qct.263.1313127496281; Thu, 11 Aug 2011 22:38:16 -0700 (PDT) Received: by 10.229.79.7 with HTTP; Thu, 11 Aug 2011 22:38:16 -0700 (PDT) In-Reply-To: References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> Date: Thu, 11 Aug 2011 22:38:16 -0700 Message-ID: From: Vishwas Manral To: Warren Kumari Content-Type: multipart/alternative; boundary=e89a8f6465c5f5ee0d04aa485165 Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 05:38:50 -0000 --e89a8f6465c5f5ee0d04aa485165 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Warren, Great to read the stuff in draft, which we all know has existed in Google. You rightly mentioned that a lot of the parts still need to be standardized= . Am not sure whre that will be done as I hear ARMD has a very narrow focus. BTW I know people use L2TP tunnels instead of GRE in a lot of cases. Theinteroperability needs to be looked at too even in such cases. Also things like QoS need to be mentioned and looked at and mentioned. Thanks, Vishwas On Thu, Aug 11, 2011 at 9:33 PM, Warren Kumari wrote: > > On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: > > > Hi Warren, > > > > It is good to see the draft. > > And it's good to see that someone has already read it :-P > > > I guess you can position this as an informational draft of how network > operators are using techniques to overcome big layer-2 issues (the ARP > issues themselves are resolved by the directory mechanism). I think this > approach has been discussed in VNRG for some time now. May be you can use > the terms defined there. > > Yup. > > > > > Also if I understand right, we are still using a big layer-2 network, > only thing by using a heirarchical overlay network we are doing away with > the actual network devices switches/ routers using Layer-2? > > Kinda -- the very high level view is that datacenters can be built with L= 3 > designs (L2 designs, or whatever design happens to exists), and then > separate overlay networks, one per customer. > > > > > As is well known "Overlay networks" have issues with scalability, which > become worse as the number of overlays increases (think of all IPsec VPN'= s). > > Yes, but as the overlay is only built between the VM hosts (and only > *those* that are actually communicating), the number of overlays that nee= ds > to be tracked *per device* is fairly limited. Also, the overlays are buil= t > on general purpose servers (and are invisible to the network) and so you > don't have all of the standard scaling issues that happen with network > devices... > > > How do we deal with the same? Also what about issues with fragmentation= / > management of hypervisor based shim etc? > > Yup, there is still lots of work to be done on specifying things like tha= t > -- who does fragmentation / reassembly, a standard protocol for populatin= g > the directory, how the DS informs the shims that a VM has moved, etc are = all > (currently) coverd by hand-waving -- I do actually have answers on most o= f > that, but I don't have them written down=85. > > > > > It would be good if you could give all such details. > > > > Thanks, > > Vishwas > > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari > wrote: > > Hi there all, > > > > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a > bit of a hissy fit about how VM mobility can be implemented without the n= eed > for L2 between the hosts. > > > > So, here is a very drafty draft, outlining the principle at a high leve= l=85 > > > > I have no real plans for this draft, it's more just an explanation of t= he > idea=85 > > > > W > > > > > > Begin forwarded message: > > > > > From: internet-drafts@ietf.org > > > Date: August 11, 2011 5:53:35 PM PDT > > > To: warren@kumari.net > > > Cc: joel.halpern@ericsson.com, warren@kumari.net > > > Subject: New Version Notification for > draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has be= en > successfully submitted by Warren Kumari and posted to the IETF repository= . > > > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > > Revision: 00 > > > Title: Virtual Machine mobility in L3 Networks. > > > Creation date: 2011-08-11 > > > WG ID: Individual Submission > > > Number of pages: 8 > > > > > > Abstract: > > > This document outlines how Virtual Machine mobility can be > > > accomplished in datacenter networks that are based on L3 > > > technologies. It is not really intended to solve (or fully define) > > > the problem, but rather to outline it at a very high level to > > > determine if standardization within the IETF makes sense. > > > > > > > > > > > > > > > The IETF Secretariat > > > > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > > > --e89a8f6465c5f5ee0d04aa485165 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
    Hi Warren,
    =A0
    Great to read the stuff in draft, which we all know has existed in Goo= gle. You rightly mentioned that a lot of the parts still need to be standar= dized. Am not sure whre that will be done as I hear ARMD has a very narrow = focus.
    =A0
    BTW I know people use L2TP tunnels instead of GRE in a lot of cases. T= heinteroperability needs to be looked at too even in such cases. Also thing= s like QoS need to be mentioned and looked at and mentioned.
    =A0
    Thanks,
    Vishwas
    On Thu, Aug 11, 2011 at 9:33 PM, Warren Kumari <= span dir=3D"ltr"><warren@kumari.net= > wrote:

    On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote:
    > Hi Warren,
    >
    > It is good to see the draft.

    And it's good to see that someone has already read it :-P

    > I guess you can position this as an informationa= l draft of how network operators are using techniques to overcome big layer= -2 issues (the ARP issues themselves are resolved by the directory mechanis= m). I think this approach has been discussed in VNRG for some time now. May= be you can use the terms defined there.

    Yup.

    >
    > Also if I understand right, we are still= using a big layer-2 network, only thing by using a heirarchical overlay ne= twork we are doing away with the actual network devices switches/ routers u= sing Layer-2?

    Kinda -- the very high level view is that datacenters can be buil= t with L3 designs (L2 designs, or whatever design happens to exists), and t= hen separate overlay networks, one per customer.

    >
    > As is well known "Overlay networks&= quot; have issues with scalability, which become worse as the number of ove= rlays increases (think of all IPsec VPN's).

    Yes, but as th= e overlay is only built between the VM hosts (and only *those* that are act= ually communicating), the number of overlays that needs to be tracked *per = device* is fairly limited. Also, the overlays are built on general purpose = servers (and are invisible to the network) and so you don't have all of= the standard scaling issues that happen with network devices...

    > How do we deal with the same? Also what about is= sues with fragmentation/ management of hypervisor based shim etc?

    Yup, there is still lots of work to be done on specifying things like t= hat -- who does fragmentation / reassembly, a standard protocol for populat= ing the directory, how the DS informs the shims that a VM has moved, etc ar= e all (currently) coverd by hand-waving -- I do actually have answers on mo= st of that, but I don't have them written down=85.

    >
    > It would be good if you could give all s= uch details.
    >
    > Thanks,
    > Vishwas
    > On Thu, Aug 11= , 2011 at 7:44 PM, Warren Kumari <w= arren@kumari.net> wrote:
    > Hi there all,
    >
    > At two previous meetings (IETF in Quebec= and NANOG in Denver) I threw a bit of a hissy fit about how VM mobility ca= n be implemented without the need for L2 between the hosts.
    >
    > So, here is a very drafty draft, outlining the principle at a high lev= el=85
    >
    > I have no real plans for this draft, it's more ju= st an explanation of the idea=85
    >
    > W
    >
    >
    > = Begin forwarded message:
    >
    > > From: interne= t-drafts@ietf.org
    > > Date: August 11, 2011 5:53:35 PM PDT
    = > > To: warren@kumari.net > > Cc: joel.halpern@eri= csson.com, warren@kumari.net> > Subject: New Version Notification for draft-wkumari-dcops-l3-vm= mobility-00.txt
    > >
    > > A new version of I-D, draft-wkumari-dcops-l3-vmmobil= ity-00.txt has been successfully submitted by Warren Kumari and posted to t= he IETF repository.
    > >
    > > Filename: =A0 =A0 =A0draft-wk= umari-dcops-l3-vmmobility
    > > Revision: =A0 =A0 =A000
    > > Title: =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 Virtual Machine mobility in L3 Networks.
    > > Creation = date: =A0 =A0 =A0 =A0 2011-08-11
    > > WG ID: =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 Individual Submission
    > > Number of pages: 8
    > >
    > > Abstract:
    > > =A0 This document outlines ho= w Virtual Machine mobility can be
    > > =A0 accomplished in datacent= er networks that are based on L3
    > > =A0 technologies. =A0It is no= t really intended to solve (or fully define)
    > > =A0 the problem, but rather to outline it at a very high level to=
    > > =A0 determine if standardization within the IETF makes sense.=
    > >
    > >
    > >
    > >
    > > The IETF = Secretariat
    > >
    >
    > _______________________________________________> armd mailing list
    > armd@ietf= .org
    > https://www.ietf.org/mailman/listinfo/armd
    >


    --e89a8f6465c5f5ee0d04aa485165-- From david.black@emc.com Fri Aug 12 06:08:31 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9E26021F87D3 for ; Fri, 12 Aug 2011 06:08:31 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.797 X-Spam-Level: X-Spam-Status: No, score=-105.797 tagged_above=-999 required=5 tests=[AWL=0.802, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xXN4TfDa2SSU for ; Fri, 12 Aug 2011 06:08:30 -0700 (PDT) Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id BD22021F87C9 for ; Fri, 12 Aug 2011 06:08:30 -0700 (PDT) Received: from hop04-l1d11-si03.isus.emc.com (HOP04-L1D11-SI03.isus.emc.com [10.254.111.23]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p7CD95Ys002652 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 12 Aug 2011 09:09:05 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.221.145]) by hop04-l1d11-si03.isus.emc.com (RSA Interceptor); Fri, 12 Aug 2011 09:09:01 -0400 Received: from mxhub29.corp.emc.com (mxhub29.corp.emc.com [128.221.47.158]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p7CD91u1002065; Fri, 12 Aug 2011 09:09:01 -0400 Received: from mx14a.corp.emc.com ([169.254.1.245]) by mxhub29.corp.emc.com ([128.221.47.158]) with mapi; Fri, 12 Aug 2011 09:09:00 -0400 From: To: Date: Fri, 12 Aug 2011 09:08:59 -0400 Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt Thread-Index: AcxYqV2CczpWX7zbTo6Z3j7VUtea7wARpcNQ Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EMM-MHVC: 1 Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 13:08:31 -0000 Hi Warren, > > It is good to see the draft. >=20 > And it's good to see that someone has already read it :-P Make that at least two someones ;-). Thanks for getting the conversation s= tarted. A detailed example of related technology can be found in draft-hasmit-otv-0= 3 (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 functional= ity (MAC-in-IP) is targeted at carrier/provider networks rather than data cente= r networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy o= f the focus on carrier/provider networks, one of the areas of design emphasis is avoidance of flooding.=20 Please don't ask me about OTV details - the draft authors (e.g., Dino) woul= d be a better resource for those sorts of questions. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA=A0 01748 +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FAX: +1 (508) 293-778= 6 david.black@emc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of W= arren Kumari > Sent: Friday, August 12, 2011 12:33 AM > To: Vishwas Manral > Cc: armd@ietf.org > Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops= -l3-vmmobility-00.txt >=20 >=20 > On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: >=20 > > Hi Warren, > > > > It is good to see the draft. >=20 > And it's good to see that someone has already read it :-P >=20 > > I guess you can position this as an informational draft of how network = operators are using > techniques to overcome big layer-2 issues (the ARP issues themselves are = resolved by the directory > mechanism). I think this approach has been discussed in VNRG for some tim= e now. May be you can use the > terms defined there. >=20 > Yup. >=20 > > > > Also if I understand right, we are still using a big layer-2 network, o= nly thing by using a > heirarchical overlay network we are doing away with the actual network de= vices switches/ routers using > Layer-2? >=20 > Kinda -- the very high level view is that datacenters can be built with L= 3 designs (L2 designs, or > whatever design happens to exists), and then separate overlay networks, o= ne per customer. >=20 > > > > As is well known "Overlay networks" have issues with scalability, which= become worse as the number > of overlays increases (think of all IPsec VPN's). >=20 > Yes, but as the overlay is only built between the VM hosts (and only *tho= se* that are actually > communicating), the number of overlays that needs to be tracked *per devi= ce* is fairly limited. Also, > the overlays are built on general purpose servers (and are invisible to t= he network) and so you don't > have all of the standard scaling issues that happen with network devices.= .. >=20 > > How do we deal with the same? Also what about issues with fragmentation= / management of hypervisor > based shim etc? >=20 > Yup, there is still lots of work to be done on specifying things like tha= t -- who does fragmentation / > reassembly, a standard protocol for populating the directory, how the DS = informs the shims that a VM > has moved, etc are all (currently) coverd by hand-waving -- I do actually= have answers on most of > that, but I don't have them written down.. >=20 > > > > It would be good if you could give all such details. > > > > Thanks, > > Vishwas > > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari wrot= e: > > Hi there all, > > > > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a= bit of a hissy fit about how > VM mobility can be implemented without the need for L2 between the hosts. > > > > So, here is a very drafty draft, outlining the principle at a high leve= l. > > > > I have no real plans for this draft, it's more just an explanation of t= he idea. > > > > W > > > > > > Begin forwarded message: > > > > > From: internet-drafts@ietf.org > > > Date: August 11, 2011 5:53:35 PM PDT > > > To: warren@kumari.net > > > Cc: joel.halpern@ericsson.com, warren@kumari.net > > > Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobili= ty-00.txt > > > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has be= en successfully submitted by > Warren Kumari and posted to the IETF repository. > > > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > > Revision: 00 > > > Title: Virtual Machine mobility in L3 Networks. > > > Creation date: 2011-08-11 > > > WG ID: Individual Submission > > > Number of pages: 8 > > > > > > Abstract: > > > This document outlines how Virtual Machine mobility can be > > > accomplished in datacenter networks that are based on L3 > > > technologies. It is not really intended to solve (or fully define) > > > the problem, but rather to outline it at a very high level to > > > determine if standardization within the IETF makes sense. > > > > > > > > > > > > > > > The IETF Secretariat > > > > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From gaberger@cisco.com Fri Aug 12 07:01:13 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 13EDB21F881C for ; Fri, 12 Aug 2011 07:01:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.202 X-Spam-Level: X-Spam-Status: No, score=-1.202 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q86pwjhuKFpZ for ; Fri, 12 Aug 2011 07:01:11 -0700 (PDT) Received: from rcdn-iport-5.cisco.com (rcdn-iport-5.cisco.com [173.37.86.76]) by ietfa.amsl.com (Postfix) with ESMTP id 7A8B121F87ED for ; Fri, 12 Aug 2011 07:01:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=18297; q=dns/txt; s=iport; t=1313157709; x=1314367309; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=MueW2/aFrJCDKXb0Sev7Bj/7OeRE77q0FCp8RXjmjps=; b=OfAWadWhGxxBQBuYTqzREtS2JXBxr9Kx87pGF6aWO98A2qmLD2S3MM+l llkcr84K/qPVc3h4zONtLeVIL5Zr9AA1rB4DiO1SwSAv/1hm7Z2sHQy+H OSP4+ott+6oy94UV5uTlJvHVIvLiwPLYicZGUOXtey3fkNQeC4MSoG+AB w=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuoAAI8xRU6tJV2b/2dsb2JhbAA3BwOCTYNDkhiHQwGHI2h3gUABAQEBAQEBAQEBDwEqKgYBCQIFBwcIEQMBAQEBJy4fCQgGAQ0FCRmHTQSbBQGedIMmGoMHBJMQhQyMAQ X-IronPort-AV: E=Sophos;i="4.67,362,1309737600"; d="scan'208,217";a="12576091" Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by rcdn-iport-5.cisco.com with ESMTP; 12 Aug 2011 14:01:48 +0000 Received: from [10.82.225.231] (rtp-vpn1-487.cisco.com [10.82.225.231]) by rcdn-core-4.cisco.com (8.14.3/8.14.3) with ESMTP id p7CE1jxv000911; Fri, 12 Aug 2011 14:01:46 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Fri, 12 Aug 2011 10:01:45 -0400 From: Gary Berger To: , Message-ID: Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3395988107_71867817" Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 14:01:13 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3395988107_71867817 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit I would hope that we can find a more elegant approach to the problem of dealing with dynamic address learning and mobility instead of adding overlays. We have created the L2 scaling problem (which shows itself in mobility and control plane scalability) because of the fact that the data-link address, IP Address and what we use to identify the host (I.e. The DNS name) are all bound to the point of attachment. This created the desire for the "flat" network which as we know doesn't scale. It would be great if part of this working group can look from the point of view that the Internet is an experiment and we never evolved to a clean implementation. Decoupling the service (I.e. Where a socket call connects to) from the node address and the point of attachment will go a long way to fixing the scaling challenges. A better description can be found in Saltzer, et al RFC-1498 . Also, one of the chief OSI developers John Day has an interesting perspective on this here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf -g On 8/12/11 9:08 AM, "david.black@emc.com" wrote: > Hi Warren, > >>> > It is good to see the draft. >> >> And it's good to see that someone has already read it :-P > > Make that at least two someones ;-). Thanks for getting the conversation > started. > > A detailed example of related technology can be found in draft-hasmit-otv-03 > (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 functionality > (MAC-in-IP) is targeted at carrier/provider networks rather than data center > networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy of > the focus on carrier/provider networks, one of the areas of design emphasis > is avoidance of flooding. > > Please don't ask me about OTV details - the draft authors (e.g., Dino) would > be a better resource for those sorts of questions. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >> Warren Kumari >> Sent: Friday, August 12, 2011 12:33 AM >> To: Vishwas Manral >> Cc: armd@ietf.org >> Subject: Re: [armd] Fwd: New Version Notification for >> draft-wkumari-dcops-l3-vmmobility-00.txt >> >> >> On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: >> >>> > Hi Warren, >>> > >>> > It is good to see the draft. >> >> And it's good to see that someone has already read it :-P >> >>> > I guess you can position this as an informational draft of how network >>> operators are using >> techniques to overcome big layer-2 issues (the ARP issues themselves are >> resolved by the directory >> mechanism). I think this approach has been discussed in VNRG for some time >> now. May be you can use the >> terms defined there. >> >> Yup. >> >>> > >>> > Also if I understand right, we are still using a big layer-2 network, >>> only thing by using a >> heirarchical overlay network we are doing away with the actual network >> devices switches/ routers using >> Layer-2? >> >> Kinda -- the very high level view is that datacenters can be built with L3 >> designs (L2 designs, or >> whatever design happens to exists), and then separate overlay networks, one >> per customer. >> >>> > >>> > As is well known "Overlay networks" have issues with scalability, which >>> become worse as the number >> of overlays increases (think of all IPsec VPN's). >> >> Yes, but as the overlay is only built between the VM hosts (and only *those* >> that are actually >> communicating), the number of overlays that needs to be tracked *per device* >> is fairly limited. Also, >> the overlays are built on general purpose servers (and are invisible to the >> network) and so you don't >> have all of the standard scaling issues that happen with network devices... >> >>> > How do we deal with the same? Also what about issues with fragmentation/ >>> management of hypervisor >> based shim etc? >> >> Yup, there is still lots of work to be done on specifying things like that >> -- who does fragmentation / >> reassembly, a standard protocol for populating the directory, how the DS >> informs the shims that a VM >> has moved, etc are all (currently) coverd by hand-waving -- I do actually >> have answers on most of >> that, but I don't have them written down.. >> >>> > >>> > It would be good if you could give all such details. >>> > >>> > Thanks, >>> > Vishwas >>> > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari wrote: >>> > Hi there all, >>> > >>> > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a >>> bit of a hissy fit about how >> VM mobility can be implemented without the need for L2 between the hosts. >>> > >>> > So, here is a very drafty draft, outlining the principle at a high level. >>> > >>> > I have no real plans for this draft, it's more just an explanation of the >>> idea. >>> > >>> > W >>> > >>> > >>> > Begin forwarded message: >>> > >>>> > > From: internet-drafts@ietf.org >>>> > > Date: August 11, 2011 5:53:35 PM PDT >>>> > > To: warren@kumari.net >>>> > > Cc: joel.halpern@ericsson.com, warren@kumari.net >>>> > > Subject: New Version Notification for >>>> draft-wkumari-dcops-l3-vmmobility-00.txt >>>> > > >>>> > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has >>>> been successfully submitted by >> Warren Kumari and posted to the IETF repository. >>>> > > >>>> > > Filename: draft-wkumari-dcops-l3-vmmobility >>>> > > Revision: 00 >>>> > > Title: Virtual Machine mobility in L3 Networks. >>>> > > Creation date: 2011-08-11 >>>> > > WG ID: Individual Submission >>>> > > Number of pages: 8 >>>> > > >>>> > > Abstract: >>>> > > This document outlines how Virtual Machine mobility can be >>>> > > accomplished in datacenter networks that are based on L3 >>>> > > technologies. It is not really intended to solve (or fully define) >>>> > > the problem, but rather to outline it at a very high level to >>>> > > determine if standardization within the IETF makes sense. >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > The IETF Secretariat >>>> > > >>> > >>> > _______________________________________________ >>> > armd mailing list >>> > armd@ietf.org >>> > https://www.ietf.org/mailman/listinfo/armd >>> > >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > --B_3395988107_71867817 Content-type: text/html; charset="US-ASCII" Content-transfer-encoding: quoted-printable
    I would hope that we ca= n find a more elegant approach to the problem of dealing with dynamic addres= s learning and mobility instead of adding overlays. 

    We have created the L2 scaling problem (which shows itself in mobilit= y and control plane scalability) because of the fact that the data-link addr= ess, IP Address and what we use to identify the host (I.e. The DNS name) are= all bound to the point of attachment. This created the desire for the "flat= " network which as we know doesn't scale. It would be great if part of this = working group can look from the point of view that the Internet is an experi= ment and we never evolved to a clean implementation. Decoupling the service = (I.e. Where a socket call connects to) from the node address and the point o= f attachment will go a long way to fixing the scaling challenges. A bet= ter description  can be found in Saltzer, et al  RFC-1498. Also, one of the chief OSI develop= ers John Day has an interesting perspective on this here http://pouzin.pna= networks.com/images/KoreaNamingFund100218.pdf

    -= g

    On 8/12/11 9:08 AM, "david.black@emc.com" <david.black@emc.com> wrote:

    Hi Warren,

    > It is good to see the draft= .
    And it's good to see that someone has already read = it :-P

    Make that at least two someones= ;-).  Thanks for getting the conversation started.

    =
    A detailed example of related technology can be found in draft-ha= smit-otv-03
    (http://www.ietf.org/id/draft-hasmit-otv-03.txt).  This L2-= in-L3 functionality
    (MAC-in-IP) is targeted at carrier/provider ne= tworks rather than data center
    networks, and uses UDP encapsulatio= n (e.g., as opposed to GRE).  Courtesy of
    the focus on c= arrier/provider networks, one of the areas of design emphasis
    is a= voidance of flooding.

    Please don't ask me about OT= V details - the draft authors (e.g., Dino) would
    be a better resou= rce for those sorts of questions.

    Thanks,
    --David
    ----------------------------------------------------
    David L. Black, Distinguished Engineer
    EMC Corporation, 176 = South St., Hopkinton, MA  01748
    +1 (508) 293-7953  =            FAX: +1 (508) 2= 93-7786
    =
    ----------------------------------------------------

    -----Original Message= -----
    Sent: Friday, August 12, 201= 1 12:33 AM
    To: Vishwas Manral
    Subject: Re: [armd] Fwd: New Version= Notification for draft-wkumari-dcops-l3-vmmobility-00.txt
    =
    On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote:
    > Hi Warren,
    >
    > It is good t= o see the draft.
    And it's good to see that someone ha= s already read it :-P
    > I guess you can position t= his as an informational draft of how network operators are using
    = techniques to overcome big layer-2 issues (the ARP issues themselves are res= olved by the directory
    mechanism). I think this approach has been= discussed in VNRG for some time now. May be you can use the
    term= s defined there.
    Yup.
    >
    > Also if I understand right, we are still using a big layer-2 ne= twork, only thing by using a
    heirarchical overlay network we are = doing away with the actual network devices switches/ routers using
    Layer-2?
    Kinda -- the very high level view is that = datacenters can be built with L3 designs (L2 designs, or
    whatever= design happens to exists), and then separate overlay networks, one per cust= omer.
    >
    > As is well known "Overlay = networks" have issues with scalability, which become worse as the number
    of overlays increases (think of all IPsec VPN's).
    <= div> Yes, but as the overlay is only built between the VM hosts (and only *t= hose* that are actually
    communicating), the number of overlays th= at needs to be tracked *per device* is fairly limited. Also,
    the = overlays are built on general purpose servers (and are invisible to the netw= ork) and so you don't
    have all of the standard scaling issues tha= t happen with network devices...
    > How do we deal = with the same? Also what about issues with fragmentation/ management of hype= rvisor
    based shim etc?
    Yup, there is still= lots of work to be done on specifying things like that -- who does fragment= ation /
    reassembly, a standard protocol for populating the direct= ory, how the DS informs the shims that a VM
    has moved, etc are al= l (currently) coverd by hand-waving -- I do actually have answers on most of=
    that, but I don't have them written down..
    >
    > It would be good if you could give all such details.<= /div>
    >
    > Thanks,
    > Vishwas
    &= gt; On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari <warren@kumari.net> wrote:
    > Hi there all,=
    >
    > At two previous meetings (IETF in Quebec an= d NANOG in Denver) I threw a bit of a hissy fit about how
    VM mobi= lity can be implemented without the need for L2 between the hosts.
    >
    > So, here is a very drafty draft, outlining the princi= ple at a high level.
    >
    > I have no real plans fo= r this draft, it's more just an explanation of the idea.
    >
    > W
    >
    >
    > Begin forward= ed message:
    >
    > > Date:= August 11, 2011 5:53:35 PM PDT
    > > Subject: New = Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt
    = > >
    > > A new version of I-D, draft-wkumari-dcops-l3= -vmmobility-00.txt has been successfully submitted by
    Warren Kuma= ri and posted to the IETF repository.
    > >
    > &= gt; Filename:      draft-wkumari-dcops-l3-vmmo= bility
    > > Revision:      00<= /div>
    > > Title:        &= nbsp;        Virtual Machine mobility in = L3 Networks.
    > > Creation date:    &nbs= p;    2011-08-11
    > > WG ID:   = ;            &nb= sp; Individual Submission
    > > Number of pages: 8
    = > >
    > > Abstract:
    > >   T= his document outlines how Virtual Machine mobility can be
    > &g= t;   accomplished in datacenter networks that are based on L3
    > >   technologies.  It is not really inten= ded to solve (or fully define)
    > >   the problem,= but rather to outline it at a very high level to
    > > =   determine if standardization within the IETF makes sense.
    = > >
    > >
    > >
    > >
    > > The IETF Secretariat
    > >
    ><= /div>
    > _______________________________________________
    &= gt; armd mailing list
    >
    _______________________________________________
    arm= d mailing list

    ____= ___________________________________________
    armd mailing list

    --B_3395988107_71867817-- From warren@kumari.net Fri Aug 12 08:42:47 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 99A9E21F86D0 for ; Fri, 12 Aug 2011 08:42:47 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.483 X-Spam-Level: X-Spam-Status: No, score=-102.483 tagged_above=-999 required=5 tests=[AWL=0.116, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XDVmfyRRDdAd for ; Fri, 12 Aug 2011 08:42:46 -0700 (PDT) Received: from vimes.kumari.net (vimes.kumari.net [198.186.192.250]) by ietfa.amsl.com (Postfix) with ESMTP id C966E21F84F9 for ; Fri, 12 Aug 2011 08:42:46 -0700 (PDT) Received: from [192.168.1.4] (24-104-73-2-ip-static.hfc.comcastbusiness.net [24.104.73.2]) by vimes.kumari.net (Postfix) with ESMTPSA id 82DC91B404F8; Fri, 12 Aug 2011 11:43:23 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=windows-1252 From: Warren Kumari In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> Date: Fri, 12 Aug 2011 08:43:21 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <0D9FA771-02B1-47BE-9C22-96C8D10F55D7@kumari.net> References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> To: X-Mailer: Apple Mail (2.1084) Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 15:42:47 -0000 On Aug 12, 2011, at 6:08 AM, wrote: > Hi Warren, >=20 >>> It is good to see the draft. >>=20 >> And it's good to see that someone has already read it :-P >=20 > Make that at least two someones ;-). Thanks for getting the = conversation started. >=20 > A detailed example of related technology can be found in = draft-hasmit-otv-03 > (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 = functionality > (MAC-in-IP) is targeted at carrier/provider networks rather than data = center > networks, and uses UDP encapsulation (e.g., as opposed to GRE). = Courtesy of > the focus on carrier/provider networks, one of the areas of design = emphasis > is avoidance of flooding.=20 Oooh, thank you, sir=85 So far I have only skimmed it, but so far it looks really good as a = solution=85.. >=20 > Please don't ask me about OTV details - the draft authors (e.g., Dino) = would > be a better resource for those sorts of questions. Cool. Thanks again, W >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf = Of Warren Kumari >> Sent: Friday, August 12, 2011 12:33 AM >> To: Vishwas Manral >> Cc: armd@ietf.org >> Subject: Re: [armd] Fwd: New Version Notification for = draft-wkumari-dcops-l3-vmmobility-00.txt >>=20 >>=20 >> On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: >>=20 >>> Hi Warren, >>>=20 >>> It is good to see the draft. >>=20 >> And it's good to see that someone has already read it :-P >>=20 >>> I guess you can position this as an informational draft of how = network operators are using >> techniques to overcome big layer-2 issues (the ARP issues themselves = are resolved by the directory >> mechanism). I think this approach has been discussed in VNRG for some = time now. May be you can use the >> terms defined there. >>=20 >> Yup. >>=20 >>>=20 >>> Also if I understand right, we are still using a big layer-2 = network, only thing by using a >> heirarchical overlay network we are doing away with the actual = network devices switches/ routers using >> Layer-2? >>=20 >> Kinda -- the very high level view is that datacenters can be built = with L3 designs (L2 designs, or >> whatever design happens to exists), and then separate overlay = networks, one per customer. >>=20 >>>=20 >>> As is well known "Overlay networks" have issues with scalability, = which become worse as the number >> of overlays increases (think of all IPsec VPN's). >>=20 >> Yes, but as the overlay is only built between the VM hosts (and only = *those* that are actually >> communicating), the number of overlays that needs to be tracked *per = device* is fairly limited. Also, >> the overlays are built on general purpose servers (and are invisible = to the network) and so you don't >> have all of the standard scaling issues that happen with network = devices... >>=20 >>> How do we deal with the same? Also what about issues with = fragmentation/ management of hypervisor >> based shim etc? >>=20 >> Yup, there is still lots of work to be done on specifying things like = that -- who does fragmentation / >> reassembly, a standard protocol for populating the directory, how the = DS informs the shims that a VM >> has moved, etc are all (currently) coverd by hand-waving -- I do = actually have answers on most of >> that, but I don't have them written down.. >>=20 >>>=20 >>> It would be good if you could give all such details. >>>=20 >>> Thanks, >>> Vishwas >>> On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari = wrote: >>> Hi there all, >>>=20 >>> At two previous meetings (IETF in Quebec and NANOG in Denver) I = threw a bit of a hissy fit about how >> VM mobility can be implemented without the need for L2 between the = hosts. >>>=20 >>> So, here is a very drafty draft, outlining the principle at a high = level. >>>=20 >>> I have no real plans for this draft, it's more just an explanation = of the idea. >>>=20 >>> W >>>=20 >>>=20 >>> Begin forwarded message: >>>=20 >>>> From: internet-drafts@ietf.org >>>> Date: August 11, 2011 5:53:35 PM PDT >>>> To: warren@kumari.net >>>> Cc: joel.halpern@ericsson.com, warren@kumari.net >>>> Subject: New Version Notification for = draft-wkumari-dcops-l3-vmmobility-00.txt >>>>=20 >>>> A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has = been successfully submitted by >> Warren Kumari and posted to the IETF repository. >>>>=20 >>>> Filename: draft-wkumari-dcops-l3-vmmobility >>>> Revision: 00 >>>> Title: Virtual Machine mobility in L3 Networks. >>>> Creation date: 2011-08-11 >>>> WG ID: Individual Submission >>>> Number of pages: 8 >>>>=20 >>>> Abstract: >>>> This document outlines how Virtual Machine mobility can be >>>> accomplished in datacenter networks that are based on L3 >>>> technologies. It is not really intended to solve (or fully = define) >>>> the problem, but rather to outline it at a very high level to >>>> determine if standardization within the IETF makes sense. >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> The IETF Secretariat >>>>=20 >>>=20 >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >>>=20 >>=20 >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >=20 From vishwas.ietf@gmail.com Fri Aug 12 09:49:28 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C8E9821F84E1 for ; Fri, 12 Aug 2011 09:49:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.079 X-Spam-Level: X-Spam-Status: No, score=-3.079 tagged_above=-999 required=5 tests=[AWL=0.519, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IIBQ0WaYqUsS for ; Fri, 12 Aug 2011 09:49:27 -0700 (PDT) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id 31B8F21F8496 for ; Fri, 12 Aug 2011 09:49:27 -0700 (PDT) Received: by qwc23 with SMTP id 23so2162069qwc.31 for ; Fri, 12 Aug 2011 09:50:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=58ULGhgBpbfw7TgMbjsZAGYvF91ySep5MeHgkH7Blv8=; b=IbPDU7w0qNq9Bzc7u7KPbDJOwgU55bS86gZY8Pop8mvOA3OvjGHPiWAwBi954N40r7 ALDCNRWAJIlT+Y9S4yvCumyMcTzbolf6tsVqVOjwxNDIa2VCKCuFajkTgKSVTH4yc5pt d6WYor5IaWxn3YEivOEXa0OCvwZbQPC1eUENw= MIME-Version: 1.0 Received: by 10.229.13.210 with SMTP id d18mr782402qca.76.1313167804246; Fri, 12 Aug 2011 09:50:04 -0700 (PDT) Received: by 10.229.79.7 with HTTP; Fri, 12 Aug 2011 09:50:04 -0700 (PDT) In-Reply-To: References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> Date: Fri, 12 Aug 2011 09:50:04 -0700 Message-ID: From: Vishwas Manral To: Gary Berger Content-Type: multipart/alternative; boundary=00151758f44a80b0a704aa51b41e Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 16:49:28 -0000 --00151758f44a80b0a704aa51b41e Content-Type: text/plain; charset=ISO-8859-1 Hi Gary, Thanks for the links. I will read through them. Is OTV as pointed out by David Black also not an overlay? Thanks, Vishwas On Fri, Aug 12, 2011 at 7:01 AM, Gary Berger wrote: > I would hope that we can find a more elegant approach to the problem of > dealing with dynamic address learning and mobility instead of adding > overlays. > > We have created the L2 scaling problem (which shows itself in mobility and > control plane scalability) because of the fact that the data-link address, > IP Address and what we use to identify the host (I.e. The DNS name) are all > bound to the point of attachment. This created the desire for the "flat" > network which as we know doesn't scale. It would be great if part of this > working group can look from the point of view that the Internet is an > experiment and we never evolved to a clean implementation. Decoupling the > service (I.e. Where a socket call connects to) from the node address and the > point of attachment will go a long way to fixing the scaling challenges. A > better description can be found in Saltzer, et al RFC-1498. > Also, one of the chief OSI developers John Day has an interesting > perspective on this here > http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf > > -g > > On 8/12/11 9:08 AM, "david.black@emc.com" wrote: > > Hi Warren, > > > It is good to see the draft. > And it's good to see that someone has already read it :-P > > > Make that at least two someones ;-). Thanks for getting the conversation > started. > > A detailed example of related technology can be found in > draft-hasmit-otv-03 > (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 > functionality > (MAC-in-IP) is targeted at carrier/provider networks rather than data > center > networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy > of > the focus on carrier/provider networks, one of the areas of design emphasis > is avoidance of flooding. > > Please don't ask me about OTV details - the draft authors (e.g., Dino) > would > be a better resource for those sorts of questions. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Distinguished Engineer > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > david.black@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] > On Behalf Of Warren Kumari > Sent: Friday, August 12, 2011 12:33 AM > To: Vishwas Manral > Cc: armd@ietf.org > Subject: Re: [armd] Fwd: New Version Notification for > draft-wkumari-dcops-l3-vmmobility-00.txt > On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: > > Hi Warren, > > > > It is good to see the draft. > And it's good to see that someone has already read it :-P > > I guess you can position this as an informational draft of how network > operators are using > techniques to overcome big layer-2 issues (the ARP issues themselves are > resolved by the directory > mechanism). I think this approach has been discussed in VNRG for some time > now. May be you can use the > terms defined there. > Yup. > > > > Also if I understand right, we are still using a big layer-2 network, > only thing by using a > heirarchical overlay network we are doing away with the actual network > devices switches/ routers using > Layer-2? > Kinda -- the very high level view is that datacenters can be built with > L3 designs (L2 designs, or > whatever design happens to exists), and then separate overlay networks, one > per customer. > > > > As is well known "Overlay networks" have issues with scalability, which > become worse as the number > of overlays increases (think of all IPsec VPN's). > Yes, but as the overlay is only built between the VM hosts (and only > *those* that are actually > communicating), the number of overlays that needs to be tracked *per > device* is fairly limited. Also, > the overlays are built on general purpose servers (and are invisible to the > network) and so you don't > have all of the standard scaling issues that happen with network devices... > > How do we deal with the same? Also what about issues with > fragmentation/ management of hypervisor > based shim etc? > Yup, there is still lots of work to be done on specifying things like > that -- who does fragmentation / > reassembly, a standard protocol for populating the directory, how the DS > informs the shims that a VM > has moved, etc are all (currently) coverd by hand-waving -- I do actually > have answers on most of > that, but I don't have them written down.. > > > > It would be good if you could give all such details. > > > > Thanks, > > Vishwas > > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari > wrote: > > Hi there all, > > > > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a > bit of a hissy fit about how > VM mobility can be implemented without the need for L2 between the hosts. > > > > So, here is a very drafty draft, outlining the principle at a high level. > > > > I have no real plans for this draft, it's more just an explanation of the > idea. > > > > W > > > > > > Begin forwarded message: > > > > > From: internet-drafts@ietf.org > > > Date: August 11, 2011 5:53:35 PM PDT > > > To: warren@kumari.net > > > Cc: joel.halpern@ericsson.com, warren@kumari.net > > > Subject: New Version Notification for > draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been > successfully submitted by > Warren Kumari and posted to the IETF repository. > > > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > > Revision: 00 > > > Title: Virtual Machine mobility in L3 Networks. > > > Creation date: 2011-08-11 > > > WG ID: Individual Submission > > > Number of pages: 8 > > > > > > Abstract: > > > This document outlines how Virtual Machine mobility can be > > > accomplished in datacenter networks that are based on L3 > > > technologies. It is not really intended to solve (or fully define) > > > the problem, but rather to outline it at a very high level to > > > determine if standardization within the IETF makes sense. > > > > > > > > > > > > > > > The IETF Secretariat > > > > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --00151758f44a80b0a704aa51b41e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
    Hi Gary,
    =A0
    Thanks for the links. I will read through them.
    =A0
    Is OTV as pointed out by David Black also not an overlay?
    =A0
    Thanks,
    Vishwas
    On Fri, Aug 12, 2011 at 7:01 AM, Gary Berger <gaberger@cisco.com= > wrote:
    I would hope that we can find a more elegant approach to the problem o= f dealing with dynamic address learning and mobility instead of adding over= lays.=A0

    We have created the L2 scaling problem (which shows itself in mobility= and control plane scalability) because of the fact that the data-link addr= ess, IP Address and what we use to identify the host (I.e. The DNS name) ar= e all bound to the point of attachment. This created the desire for the &qu= ot;flat" network which as we know doesn't scale. It would be great= if part of this working group can look from the point of view that the Int= ernet is an experiment and we never evolved to a clean implementation. Deco= upling the service (I.e. Where a socket call connects to) from the node add= ress and the point of attachment will go a long way to fixing the scaling c= hallenges.=A0A better description =A0can be found in Saltzer, et al =A0RFC-1498. = Also, one of the chief OSI developers John Day has an interesting perspecti= ve on this here=A0http://pouzin.pnanetworks.com/images/Ko= reaNamingFund100218.pdf

    -g

    On 8/12/11 9:08 AM, "david.black@emc.com" <david.black@emc.com> wrote:

    Hi Warren,

    > It is good to see the draft.
    And it's good to see that someone has already read it :-P

    Make that at least two someones ;-).=A0=A0Thanks for getting the conve= rsation started.

    A detailed example of related technology can be found in draft-hasmit-= otv-03
    (http://www.ietf.org/id/draft-hasmit-otv-03.txt).=A0=A0This L2-i= n-L3 functionality
    (MAC-in-IP) is targeted at carrier/provider networks rather than data = center
    networks, and uses UDP encapsulation (e.g., as opposed to GRE).=A0=A0C= ourtesy of
    the focus on carrier/provider networks, one of the areas of design emp= hasis
    is avoidance of flooding.

    Please don't ask me about OTV details - the draft authors (e.g., D= ino) would
    be a better resource for those sorts of questions.

    Thanks,
    --David
    ----------------------------------------------------
    David L. Black, Distinguished Engineer
    EMC Corporation, 176 South St., Hopkinton, MA=A0 01748
    +1 (508) 293-7953=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FA= X: +1 (508) 293-7786
    david.black@e= mc.com=A0=A0=A0=A0=A0=A0=A0 Mobile: +1 (978) 394-7754
    ----------------------------------------------------

    -----Original Message-----
    From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Warren Kumari
    Sent: Friday, August 12, 2011 12:33 AM
    To: Vishwas Manral
    Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dc= ops-l3-vmmobility-00.txt
    On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote:
    > Hi Warren,
    >
    > It is good to see the draft.
    And it's good to see that someone has already read it :-P
    > I guess you can position this as an informational draft of how ne= twork operators are using
    techniques to overcome big layer-2 issues (the ARP issues themselves a= re resolved by the directory
    mechanism). I think this approach has been discussed in VNRG for some = time now. May be you can use the
    terms defined there.
    Yup.
    >
    > Also if I understand right, we are still using a big layer-2 netw= ork, only thing by using a
    heirarchical overlay network we are doing away with the actual network= devices switches/ routers using
    Layer-2?
    Kinda -- the very high level view is that datacenters can be built wit= h L3 designs (L2 designs, or
    whatever design happens to exists), and then separate overlay networks= , one per customer.
    >
    > As is well known "Overlay networks" have issues with sc= alability, which become worse as the number
    of overlays increases (think of all IPsec VPN's).
    Yes, but as the overlay is only built between the VM hosts (and only *= those* that are actually
    communicating), the number of overlays that needs to be tracked *per d= evice* is fairly limited. Also,
    the overlays are built on general purpose servers (and are invisible t= o the network) and so you don't
    have all of the standard scaling issues that happen with network devic= es...
    > How do we deal with the same? Also what about issues with fragmen= tation/ management of hypervisor
    based shim etc?
    Yup, there is still lots of work to be done on specifying things like = that -- who does fragmentation /
    reassembly, a standard protocol for populating the directory, how the = DS informs the shims that a VM
    has moved, etc are all (currently) coverd by hand-waving -- I do actua= lly have answers on most of
    that, but I don't have them written down..
    >
    > It would be good if you could give all such details.
    >
    > Thanks,
    > Vishwas
    > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari <warren@kumari.net> wrote:
    > Hi there all,
    >
    > At two previous meetings (IETF in Quebec and NANOG in Denver) I t= hrew a bit of a hissy fit about how
    VM mobility can be implemented without the need for L2 between the hos= ts.
    >
    > So, here is a very drafty draft, outlining the principle at a hig= h level.
    >
    > I have no real plans for this draft, it's more just an explan= ation of the idea.
    >
    > W
    >
    >
    > Begin forwarded message:
    >
    > > Date: August 11, 2011 5:53:35 PM PDT
    > > Subject: New Version Notification for draft-wkumari-dcops-l3= -vmmobility-00.txt
    > >
    > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.t= xt has been successfully submitted by
    Warren Kumari and posted to the IETF repository.
    > >
    > > Filename:=A0=A0=A0=A0=A0=A0draft-wkumari-dcops-l3-vmmobility=
    > > Revision:=A0=A0=A0=A0=A0=A000
    > > Title:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Virtu= al Machine mobility in L3 Networks.
    > > Creation date:=A0=A0=A0=A0=A0=A0=A0=A0 2011-08-11
    > > WG ID:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Indiv= idual Submission
    > > Number of pages: 8
    > >
    > > Abstract:
    > >=A0=A0 This document outlines how Virtual Machine mobility ca= n be
    > >=A0=A0 accomplished in datacenter networks that are based on = L3
    > >=A0=A0 technologies.=A0=A0It is not really intended to solve = (or fully define)
    > >=A0=A0 the problem, but rather to outline it at a very high l= evel to
    > >=A0=A0 determine if standardization within the IETF makes sen= se.
    > >
    > >
    > >
    > >
    > > The IETF Secretariat
    > >
    >
    > _______________________________________________
    > armd mailing list
    >
    _______________________________________________
    armd mailing list

    _______________________________________________
    armd mailing list

    _________________________= ______________________
    armd mailing list
    armd@ietf.org
    https://www.ietf.org/mailman/listinfo/armd


    --00151758f44a80b0a704aa51b41e-- From gaberger@cisco.com Fri Aug 12 09:51:43 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DCB1C21F854E for ; Fri, 12 Aug 2011 09:51:43 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.202 X-Spam-Level: X-Spam-Status: No, score=-1.202 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4vrzvTO7z3lC for ; Fri, 12 Aug 2011 09:51:42 -0700 (PDT) Received: from rcdn-iport-4.cisco.com (rcdn-iport-4.cisco.com [173.37.86.75]) by ietfa.amsl.com (Postfix) with ESMTP id 3DA4B21F852E for ; Fri, 12 Aug 2011 09:51:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=22563; q=dns/txt; s=iport; t=1313167940; x=1314377540; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=eUy8xZDK1XMqouO+DjPGbK5hivE4xK7xukdNKoQflwk=; b=jbVCQerUnVHp3P6LUdOBc1XChN5XWDFMHif/n5nZph3IVnT+a5qt207n unP5+xiR7gh2Uzv7tftR0Rnsvnn3K0ESr2UxdV3nAEnrQ2vOnL2MIvS+2 XaGPqbkkSshB+x8ME+a/R/S+8rUSZFO1KYpEk9T7zxW9HyapiHHx/o+Ri M=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtoAAGpZRU6tJV2d/2dsb2JhbAA4BwOCTYNDkhyHQ4ciaHeBQAEBAQEBAQEBAQEPASoqBgEJAgUHBwgRAwEBAQEnKAYfCQgGDgUJEgeHTQScJAGedYMmGoMHBJMQhQyEaYcY X-IronPort-AV: E=Sophos;i="4.67,363,1309737600"; d="scan'208,217";a="12647375" Received: from rcdn-core-6.cisco.com ([173.37.93.157]) by rcdn-iport-4.cisco.com with ESMTP; 12 Aug 2011 16:52:19 +0000 Received: from [10.82.225.231] (rtp-vpn1-487.cisco.com [10.82.225.231]) by rcdn-core-6.cisco.com (8.14.3/8.14.3) with ESMTP id p7CGqGf7015700; Fri, 12 Aug 2011 16:52:18 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Fri, 12 Aug 2011 12:52:16 -0400 From: Gary Berger To: Vishwas Manral Message-ID: Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3395998338_72486248" Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 16:51:44 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3395998338_72486248 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit From: Vishwas Manral Date: Fri, 12 Aug 2011 09:50:04 -0700 To: Gary Berger Cc: , , Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt Hi Gary, Thanks for the links. I will read through them. Is OTV as pointed out by David Black also not an overlay? @gaberger: I would say anything with encapsulation is utterly an overlay.. This is certainly a workable solution within scope. The question is if you want to address the core architectural flaws or paint over them with sexy implementations which incur their own side-effects.. Thanks, Vishwas On Fri, Aug 12, 2011 at 7:01 AM, Gary Berger wrote: > I would hope that we can find a more elegant approach to the problem of > dealing with dynamic address learning and mobility instead of adding overlays. > > We have created the L2 scaling problem (which shows itself in mobility and > control plane scalability) because of the fact that the data-link address, IP > Address and what we use to identify the host (I.e. The DNS name) are all bound > to the point of attachment. This created the desire for the "flat" network > which as we know doesn't scale. It would be great if part of this working > group can look from the point of view that the Internet is an experiment and > we never evolved to a clean implementation. Decoupling the service (I.e. Where > a socket call connects to) from the node address and the point of attachment > will go a long way to fixing the scaling challenges. A better description can > be found in Saltzer, et al RFC-1498 . > Also, one of the chief OSI developers John Day has an interesting perspective > on this here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf > > -g > > On 8/12/11 9:08 AM, "david.black@emc.com" wrote: > >> Hi Warren, >> >>>> > It is good to see the draft. >>> And it's good to see that someone has already read it :-P >> >> Make that at least two someones ;-). Thanks for getting the conversation >> started. >> >> A detailed example of related technology can be found in draft-hasmit-otv-03 >> (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 >> functionality >> (MAC-in-IP) is targeted at carrier/provider networks rather than data center >> networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy of >> the focus on carrier/provider networks, one of the areas of design emphasis >> is avoidance of flooding. >> >> Please don't ask me about OTV details - the draft authors (e.g., Dino) would >> be a better resource for those sorts of questions. >> >> Thanks, >> --David >> ---------------------------------------------------- >> David L. Black, Distinguished Engineer >> EMC Corporation, 176 South St., Hopkinton, MA 01748 >> +1 (508) 293-7953 FAX: +1 >> (508) 293-7786 >> david.black@emc.com Mobile: +1 (978) 394-7754 >> >> ---------------------------------------------------- >> >>> -----Original Message----- >>> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >>> Warren Kumari >>> Sent: Friday, August 12, 2011 12:33 AM >>> To: Vishwas Manral >>> Cc: armd@ietf.org >>> Subject: Re: [armd] Fwd: New Version Notification for >>> draft-wkumari-dcops-l3-vmmobility-00.txt >>> On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: >>>> > Hi Warren, >>>> > >>>> > It is good to see the draft. >>> And it's good to see that someone has already read it :-P >>>> > I guess you can position this as an informational draft of how network >>>> operators are using >>> techniques to overcome big layer-2 issues (the ARP issues themselves are >>> resolved by the directory >>> mechanism). I think this approach has been discussed in VNRG for some time >>> now. May be you can use the >>> terms defined there. >>> Yup. >>>> > >>>> > Also if I understand right, we are still using a big layer-2 network, >>>> only thing by using a >>> heirarchical overlay network we are doing away with the actual network >>> devices switches/ routers using >>> Layer-2? >>> Kinda -- the very high level view is that datacenters can be built with L3 >>> designs (L2 designs, or >>> whatever design happens to exists), and then separate overlay networks, one >>> per customer. >>>> > >>>> > As is well known "Overlay networks" have issues with scalability, which >>>> become worse as the number >>> of overlays increases (think of all IPsec VPN's). >>> Yes, but as the overlay is only built between the VM hosts (and only *those* >>> that are actually >>> communicating), the number of overlays that needs to be tracked *per device* >>> is fairly limited. Also, >>> the overlays are built on general purpose servers (and are invisible to the >>> network) and so you don't >>> have all of the standard scaling issues that happen with network devices... >>>> > How do we deal with the same? Also what about issues with fragmentation/ >>>> management of hypervisor >>> based shim etc? >>> Yup, there is still lots of work to be done on specifying things like that >>> -- who does fragmentation / >>> reassembly, a standard protocol for populating the directory, how the DS >>> informs the shims that a VM >>> has moved, etc are all (currently) coverd by hand-waving -- I do actually >>> have answers on most of >>> that, but I don't have them written down.. >>>> > >>>> > It would be good if you could give all such details. >>>> > >>>> > Thanks, >>>> > Vishwas >>>> > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari wrote: >>>> > Hi there all, >>>> > >>>> > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a >>>> bit of a hissy fit about how >>> VM mobility can be implemented without the need for L2 between the hosts. >>>> > >>>> > So, here is a very drafty draft, outlining the principle at a high level. >>>> > >>>> > I have no real plans for this draft, it's more just an explanation of the >>>> idea. >>>> > >>>> > W >>>> > >>>> > >>>> > Begin forwarded message: >>>> > >>>>> > > From: internet-drafts@ietf.org >>>>> > > Date: August 11, 2011 5:53:35 PM PDT >>>>> > > To: warren@kumari.net >>>>> > > Cc: joel.halpern@ericsson.com, warren@kumari.net >>>>> > > Subject: New Version Notification for >>>>> draft-wkumari-dcops-l3-vmmobility-00.txt >>>>> > > >>>>> > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has >>>>> been successfully submitted by >>> Warren Kumari and posted to the IETF repository. >>>>> > > >>>>> > > Filename: draft-wkumari-dcops-l3-vmmobility >>>>> > > Revision: 00 >>>>> > > Title: Virtual Machine mobility in L3 Networks. >>>>> > > Creation date: 2011-08-11 >>>>> > > WG ID: Individual Submission >>>>> > > Number of pages: 8 >>>>> > > >>>>> > > Abstract: >>>>> > > This document outlines how Virtual Machine mobility can be >>>>> > > accomplished in datacenter networks that are based on L3 >>>>> > > technologies. It is not really intended to solve (or fully define) >>>>> > > the problem, but rather to outline it at a very high level to >>>>> > > determine if standardization within the IETF makes sense. >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > The IETF Secretariat >>>>> > > >>>> > >>>> > _______________________________________________ >>>> > armd mailing list >>>> > armd@ietf.org >>>> > https://www.ietf.org/mailman/listinfo/armd >>>> > >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > --B_3395998338_72486248 Content-type: text/html; charset="US-ASCII" Content-transfer-encoding: quoted-printable


    From: Vishwas Manral = <vishwas.ietf@gmail.com>Date: Fri, 12 Aug 2011 09:50:04 -07= 00
    To: Gary Berger <gaberger@cisco.com>
    Cc: <david.black@e= mc.com>, <warren@kumari.net= >, <armd@ietf.org>
    Subject: Re: [armd] Fwd: New Version Notificati= on for draft-wkumari-dcops-l3-vmmobility-00.txt

    Hi Gary,
     
    Thanks for the links. I will read throu= gh them.
     
    Is OTV as pointed out by David Black als= o not an overlay?
     
    @gaberger: I would say a= nything with encapsulation is utterly an overlay.. This is certainly a worka= ble solution within scope. The question is if you want to address the core a= rchitectural flaws or paint over them with sexy implementations which incur = their own side-effects..


    Thanks,
    Vishwas
    On Fri, Aug 12, 2011 at 7:01 AM, Gary Berger <gaberger@cisco.com> wrote:
    I= would hope that we can find a more elegant approach to the problem of deali= ng with dynamic address learning and mobility instead of adding overlays.&nb= sp;

    We have created the L2 scaling problem (which s= hows itself in mobility and control plane scalability) because of the fact t= hat the data-link address, IP Address and what we use to identify the host (= I.e. The DNS name) are all bound to the point of attachment. This created th= e desire for the "flat" network which as we know doesn't scale. It would be = great if part of this working group can look from the point of view that the= Internet is an experiment and we never evolved to a clean implementation. D= ecoupling the service (I.e. Where a socket call connects to) from the node a= ddress and the point of attachment will go a long way to fixing the scaling = challenges. A better description  can be found in Saltzer, et al &= nbsp;RFC-1498. Also, one of the chief OSI developers John Day has an interesting perspe= ctive on this here http://pouzin.pnanetworks.com/images/Ko= reaNamingFund100218.pdf

    -= g

    On 8= /12/11 9:08 AM, "david.= black@emc.com" <= david.black@emc.com> wrote:

    Hi Warren,

    > It is good to see the draft.
    And it's good to see that someone has already read it :-P

    Make that at least two someones ;-).  Thank= s for getting the conversation started.

    A detailed = example of related technology can be found in draft-hasmit-otv-03
    = (ht= tp://www.ietf.org/id/draft-hasmit-otv-03.txt).  This L2-in-L3 = functionality
    (MAC-in-IP) is targeted at carrier/provider networks= rather than data center
    networks, and uses UDP encapsulation (e.g= ., as opposed to GRE).  Courtesy of
    the focus on carrier= /provider networks, one of the areas of design emphasis
    is avoidan= ce of flooding.

    Please don't ask me about OTV deta= ils - the draft authors (e.g., Dino) would
    be a better resource fo= r those sorts of questions.

    Thanks,
    --Dav= id
    ----------------------------------------------------
    = David L. Black, Distinguished Engineer
    EMC Corporation, 176 South = St., Hopkinton, MA  01748
    +1 (508) 293-7953  = ;           FAX: +1 (50= 8) 293-7786
    david.black@emc.com        Mobile:= +1 (978) 394-7754
    ----------------------------------------= ------------

    -----Original Message-----
    From: armd-bounces@i= etf.org [mailto:a= rmd-bounces@ietf.org] On Behalf Of Warren Kumari
    Sent: Friday,= August 12, 2011 12:33 AM
    To: Vishwas Manral
    Subject:= Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobil= ity-00.txt
    On Aug 11, 2011, at 9:05 PM, Vish= was Manral wrote:
    > Hi Warren,
    >
    <= div>> It is good to see the draft.
    And it's good to = see that someone has already read it :-P
    > I guess y= ou can position this as an informational draft of how network operators are = using
    techniques to overcome big layer-2 issues (the ARP issues th= emselves are resolved by the directory
    mechanism). I think this ap= proach has been discussed in VNRG for some time now. May be you can use the<= /div>
    terms defined there.
    Yup.
    >
    > Also if I understand right, we are still using a big l= ayer-2 network, only thing by using a
    heirarchical overlay network= we are doing away with the actual network devices switches/ routers using
    Layer-2?
    Kinda -- the very high level view is = that datacenters can be built with L3 designs (L2 designs, or
    what= ever design happens to exists), and then separate overlay networks, one per = customer.
    >
    > As is well known "Overlay= networks" have issues with scalability, which become worse as the number
    of overlays increases (think of all IPsec VPN's).
    Yes, but as the overlay is only built between the VM hosts (and only *tho= se* that are actually
    communicating), the number of overlays that = needs to be tracked *per device* is fairly limited. Also,
    the over= lays are built on general purpose servers (and are invisible to the network)= and so you don't
    have all of the standard scaling issues that hap= pen with network devices...
    > How do we deal with th= e same? Also what about issues with fragmentation/ management of hypervisor<= /div>
    based shim etc?
    Yup, there is still lots of w= ork to be done on specifying things like that -- who does fragmentation /
    reassembly, a standard protocol for populating the directory, how th= e DS informs the shims that a VM
    has moved, etc are all (currently= ) coverd by hand-waving -- I do actually have answers on most of
    t= hat, but I don't have them written down..
    >
    > It would be good if you could give all such details.
    >
    > Thanks,
    > Vishwas
    > On Thu, Aug 11, = 2011 at 7:44 PM, Warren Kumari <warren@kumari.net> wrote:
    > Hi there all,
    >
    > At two previous meetings (IETF in Quebec and NANOG= in Denver) I threw a bit of a hissy fit about how
    VM mobility can= be implemented without the need for L2 between the hosts.
    >
    > So, here is a very drafty draft, outlining the principle at a h= igh level.
    >
    > I have no real plans for this draft= , it's more just an explanation of the idea.
    >
    > W=
    >
    >
    > Begin forwarded message:
    =
    >
    > > Date: Aug= ust 11, 2011 5:53:35 PM PDT
    > > Subject: New Version Notification for draft-wkum= ari-dcops-l3-vmmobility-00.txt
    > >
    > > A new= version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been successfu= lly submitted by
    Warren Kumari and posted to the IETF repository.<= /div>
    > >
    > > Filename:    &nb= sp; draft-wkumari-dcops-l3-vmmobility
    > > Revision:&nbs= p;     00
    > > Title:  &nb= sp;            &= nbsp; Virtual Machine mobility in L3 Networks.
    > > Creation = date:         2011-08-11
    &= gt; > WG ID:          &= nbsp;      Individual Submission
    > >= ; Number of pages: 8
    > >
    > > Abstract:
    =
    > >   This document outlines how Virtual Machine mobili= ty can be
    > >   accomplished in datacenter network= s that are based on L3
    > >   technologies. &n= bsp;It is not really intended to solve (or fully define)
    > >=    the problem, but rather to outline it at a very high level to
    > >   determine if standardization within the IETF = makes sense.
    > >
    > >
    > >
    > >
    > > The IETF Secretariat
    > >= ;
    >
    > ____________________________________________= ___
    > armd mailing list
    >
    ____________________= ___________________________
    armd mailing list

    _________= ______________________________________
    armd mailing list
    armd@ietf.org


    _______________________________________________
    armd m= ailing list
    armd@ietf.org
    https://www.ietf= .org/mailman/listinfo/armd


    <= /html> --B_3395998338_72486248-- From bedard.phil@gmail.com Fri Aug 12 10:06:06 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EDA9B21F8698 for ; Fri, 12 Aug 2011 10:06:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.282 X-Spam-Level: X-Spam-Status: No, score=-2.282 tagged_above=-999 required=5 tests=[AWL=0.699, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, RCVD_IN_SORBS_WEB=0.619] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fdMtxLC0gR91 for ; Fri, 12 Aug 2011 10:06:06 -0700 (PDT) Received: from mail-gw0-f44.google.com (mail-gw0-f44.google.com [74.125.83.44]) by ietfa.amsl.com (Postfix) with ESMTP id CB56521F8661 for ; Fri, 12 Aug 2011 10:06:05 -0700 (PDT) Received: by gwb20 with SMTP id 20so2450472gwb.31 for ; Fri, 12 Aug 2011 10:06:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=user-agent:date:subject:from:to:cc:message-id:thread-topic :in-reply-to:mime-version:content-type:content-transfer-encoding; bh=0n04gwa5ILyDXA7eM8n3/YO12mB7wRLIdzcGMehKJUg=; b=XPvZvygGYKJJRHKeBUG+MaasDGjUPbs+0XvYtJA4eloyunA+cpoYmICC23hTpMiAWg MiG6fl5elr3cto7bxCZpNl/mvuDbI4d5IjR3n/6iNFCTStUpqyPlZq1wpSqYu9ZcmvTu MxAsb/33ELzv0TqYqisOHMeL8mCSnoixey648= Received: by 10.236.175.39 with SMTP id y27mr3619790yhl.54.1313168803212; Fri, 12 Aug 2011 10:06:43 -0700 (PDT) Received: from [10.63.85.125] (gw1.cox.com [24.248.74.254]) by mx.google.com with ESMTPS id r28sm915166yhm.52.2011.08.12.10.06.39 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 12 Aug 2011 10:06:41 -0700 (PDT) User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Fri, 12 Aug 2011 13:06:36 -0400 From: Phil Bedard To: , Message-ID: Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 17:06:07 -0000 I've read it as well and have heard of the approach or similar approaches in the past using both GRE and L2TP. My main concerns are around the burden it places on the hypervisor, especially in cases of very fragmented clusters of VMs where there may be a lot of encap/decap going on. Also the process of external traffic reaching the destination host. OTV extends L2 domains in the same way VPLS does and places the burden of configuring the edge network devices to make sure the network to Hypervisor switch connection is in the right L2 domain. There is no mapping service and instead uses a dynamic control plane to disseminate MAC and location information (IS-IS). Ethernet VPN, http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-02 is similar only using BGP to distribute MAC information and uses MPLS to encapsulate frames. Both could work for L3 mobility but require the network devices be configured as part of the mobility process. And of course means you need L3 devices supporting the aforementioned protocols. The Ethernet VPN concept using BGP (or any other control plane protocol) could be extended to tunneling mechanisms other than MPLS. I think the only way to move away from overlays is more of a paradigm shift where we are no longer using IP addresses or MAC addresses to send packets to end hosts. Phil On 8/12/11 9:08 AM, "david.black@emc.com" wrote: >Hi Warren, > >> > It is good to see the draft. >> >> And it's good to see that someone has already read it :-P > >Make that at least two someones ;-). Thanks for getting the conversation >started. > >A detailed example of related technology can be found in >draft-hasmit-otv-03 >(http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 >functionality >(MAC-in-IP) is targeted at carrier/provider networks rather than data >center >networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy >of >the focus on carrier/provider networks, one of the areas of design >emphasis >is avoidance of flooding. > >Please don't ask me about OTV details - the draft authors (e.g., Dino) >would >be a better resource for those sorts of questions. > >Thanks, >--David >---------------------------------------------------- >David L. Black, Distinguished Engineer >EMC Corporation, 176 South St., Hopkinton, MA 01748 >+1 (508) 293-7953 FAX: +1 (508) 293-7786 >david.black@emc.com Mobile: +1 (978) 394-7754 >---------------------------------------------------- > >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >>Warren Kumari >> Sent: Friday, August 12, 2011 12:33 AM >> To: Vishwas Manral >> Cc: armd@ietf.org >> Subject: Re: [armd] Fwd: New Version Notification for >>draft-wkumari-dcops-l3-vmmobility-00.txt >> >> >> On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: >> >> > Hi Warren, >> > >> > It is good to see the draft. >> >> And it's good to see that someone has already read it :-P >> >> > I guess you can position this as an informational draft of how >>network operators are using >> techniques to overcome big layer-2 issues (the ARP issues themselves >>are resolved by the directory >> mechanism). I think this approach has been discussed in VNRG for some >>time now. May be you can use the >> terms defined there. >> >> Yup. >> >> > >> > Also if I understand right, we are still using a big layer-2 network, >>only thing by using a >> heirarchical overlay network we are doing away with the actual network >>devices switches/ routers using >> Layer-2? >> >> Kinda -- the very high level view is that datacenters can be built with >>L3 designs (L2 designs, or >> whatever design happens to exists), and then separate overlay networks, >>one per customer. >> >> > >> > As is well known "Overlay networks" have issues with scalability, >>which become worse as the number >> of overlays increases (think of all IPsec VPN's). >> >> Yes, but as the overlay is only built between the VM hosts (and only >>*those* that are actually >> communicating), the number of overlays that needs to be tracked *per >>device* is fairly limited. Also, >> the overlays are built on general purpose servers (and are invisible to >>the network) and so you don't >> have all of the standard scaling issues that happen with network >>devices... >> >> > How do we deal with the same? Also what about issues with >>fragmentation/ management of hypervisor >> based shim etc? >> >> Yup, there is still lots of work to be done on specifying things like >>that -- who does fragmentation / >> reassembly, a standard protocol for populating the directory, how the >>DS informs the shims that a VM >> has moved, etc are all (currently) coverd by hand-waving -- I do >>actually have answers on most of >> that, but I don't have them written down.. >> >> > >> > It would be good if you could give all such details. >> > >> > Thanks, >> > Vishwas >> > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari >>wrote: >> > Hi there all, >> > >> > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw >>a bit of a hissy fit about how >> VM mobility can be implemented without the need for L2 between the >>hosts. >> > >> > So, here is a very drafty draft, outlining the principle at a high >>level. >> > >> > I have no real plans for this draft, it's more just an explanation of >>the idea. >> > >> > W >> > >> > >> > Begin forwarded message: >> > >> > > From: internet-drafts@ietf.org >> > > Date: August 11, 2011 5:53:35 PM PDT >> > > To: warren@kumari.net >> > > Cc: joel.halpern@ericsson.com, warren@kumari.net >> > > Subject: New Version Notification for >>draft-wkumari-dcops-l3-vmmobility-00.txt >> > > >> > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has >>been successfully submitted by >> Warren Kumari and posted to the IETF repository. >> > > >> > > Filename: draft-wkumari-dcops-l3-vmmobility >> > > Revision: 00 >> > > Title: Virtual Machine mobility in L3 Networks. >> > > Creation date: 2011-08-11 >> > > WG ID: Individual Submission >> > > Number of pages: 8 >> > > >> > > Abstract: >> > > This document outlines how Virtual Machine mobility can be >> > > accomplished in datacenter networks that are based on L3 >> > > technologies. It is not really intended to solve (or fully >>define) >> > > the problem, but rather to outline it at a very high level to >> > > determine if standardization within the IETF makes sense. >> > > >> > > >> > > >> > > >> > > The IETF Secretariat >> > > >> > >> > _______________________________________________ >> > armd mailing list >> > armd@ietf.org >> > https://www.ietf.org/mailman/listinfo/armd >> > >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd > >_______________________________________________ >armd mailing list >armd@ietf.org >https://www.ietf.org/mailman/listinfo/armd From david.black@emc.com Fri Aug 12 10:37:19 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 30E2911E8083 for ; Fri, 12 Aug 2011 10:37:19 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.829 X-Spam-Level: X-Spam-Status: No, score=-105.829 tagged_above=-999 required=5 tests=[AWL=0.769, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fxzOgNeSC41c for ; Fri, 12 Aug 2011 10:37:16 -0700 (PDT) Received: from mexforward.lss.emc.com (mexforward.lss.emc.com [128.222.32.20]) by ietfa.amsl.com (Postfix) with ESMTP id A8DFB21F8571 for ; Fri, 12 Aug 2011 10:37:11 -0700 (PDT) Received: from hop04-l1d11-si04.isus.emc.com (HOP04-L1D11-SI04.isus.emc.com [10.254.111.24]) by mexforward.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p7CHbjQ4016745 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 12 Aug 2011 13:37:46 -0400 Received: from mailhub.lss.emc.com (mailhub.lss.emc.com [10.254.222.129]) by hop04-l1d11-si04.isus.emc.com (RSA Interceptor); Fri, 12 Aug 2011 13:37:34 -0400 Received: from mxhub05.corp.emc.com (mxhub05.corp.emc.com [128.221.46.113]) by mailhub.lss.emc.com (Switch-3.4.3/Switch-3.4.3) with ESMTP id p7CHbX7S001021; Fri, 12 Aug 2011 13:37:34 -0400 Received: from mx14a.corp.emc.com ([169.254.1.245]) by mxhub05.corp.emc.com ([128.221.46.113]) with mapi; Fri, 12 Aug 2011 13:37:33 -0400 From: To: Date: Fri, 12 Aug 2011 13:37:28 -0400 Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt Thread-Index: AcxY+MT0jEOXkVdBRpK6mdGj6GPwTQAHU0kQ Message-ID: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2@MX14A.corp.emc.com> References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2MX14Acorpemcc_" MIME-Version: 1.0 Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 17:37:19 -0000 --_000_7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2MX14Acorpemcc_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Gary, Are you aware of the IETF work already in progress on that sort of decoupli= ng, e.g., LISP, HIP and ILNP? I don't understand why another effort in this space would be useful. Thanks, --David From: Gary Berger [mailto:gaberger@cisco.com] Sent: Friday, August 12, 2011 10:02 AM To: Black, David; warren@kumari.net Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l= 3-vmmobility-00.txt I would hope that we can find a more elegant approach to the problem of dea= ling with dynamic address learning and mobility instead of adding overlays. We have created the L2 scaling problem (which shows itself in mobility and = control plane scalability) because of the fact that the data-link address, = IP Address and what we use to identify the host (I.e. The DNS name) are all= bound to the point of attachment. This created the desire for the "flat" n= etwork which as we know doesn't scale. It would be great if part of this wo= rking group can look from the point of view that the Internet is an experim= ent and we never evolved to a clean implementation. Decoupling the service = (I.e. Where a socket call connects to) from the node address and the point = of attachment will go a long way to fixing the scaling challenges. A better= description can be found in Saltzer, et al RFC-1498. Also, one of the chief OSI developers John Day has an inte= resting perspective on this here http://pouzin.pnanetworks.com/images/Korea= NamingFund100218.pdf -g On 8/12/11 9:08 AM, "david.black@emc.com" > wrote: Hi Warren, > It is good to see the draft. And it's good to see that someone has already read it :-P Make that at least two someones ;-). Thanks for getting the conversation s= tarted. A detailed example of related technology can be found in draft-hasmit-otv-0= 3 (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 functional= ity (MAC-in-IP) is targeted at carrier/provider networks rather than data cente= r networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy o= f the focus on carrier/provider networks, one of the areas of design emphasis is avoidance of flooding. Please don't ask me about OTV details - the draft authors (e.g., Dino) woul= d be a better resource for those sorts of questions. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 david.black@emc.com Mobile: +1 (978) 394= -7754 ---------------------------------------------------- -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-boun= ces@ietf.org] On Behalf Of Warren Kumari Sent: Friday, August 12, 2011 12:33 AM To: Vishwas Manral Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l= 3-vmmobility-00.txt On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: > Hi Warren, > > It is good to see the draft. And it's good to see that someone has already read it :-P > I guess you can position this as an informational draft of how network op= erators are using techniques to overcome big layer-2 issues (the ARP issues themselves are re= solved by the directory mechanism). I think this approach has been discussed in VNRG for some time = now. May be you can use the terms defined there. Yup. > > Also if I understand right, we are still using a big layer-2 network, onl= y thing by using a heirarchical overlay network we are doing away with the actual network devi= ces switches/ routers using Layer-2? Kinda -- the very high level view is that datacenters can be built with L3 = designs (L2 designs, or whatever design happens to exists), and then separate overlay networks, one= per customer. > > As is well known "Overlay networks" have issues with scalability, which b= ecome worse as the number of overlays increases (think of all IPsec VPN's). Yes, but as the overlay is only built between the VM hosts (and only *those= * that are actually communicating), the number of overlays that needs to be tracked *per device= * is fairly limited. Also, the overlays are built on general purpose servers (and are invisible to the= network) and so you don't have all of the standard scaling issues that happen with network devices... > How do we deal with the same? Also what about issues with fragmentation/ = management of hypervisor based shim etc? Yup, there is still lots of work to be done on specifying things like that = -- who does fragmentation / reassembly, a standard protocol for populating the directory, how the DS in= forms the shims that a VM has moved, etc are all (currently) coverd by hand-waving -- I do actually h= ave answers on most of that, but I don't have them written down.. > > It would be good if you could give all such details. > > Thanks, > Vishwas > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari > wrote: > Hi there all, > > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a b= it of a hissy fit about how VM mobility can be implemented without the need for L2 between the hosts. > > So, here is a very drafty draft, outlining the principle at a high level. > > I have no real plans for this draft, it's more just an explanation of the= idea. > > W > > > Begin forwarded message: > > > From: internet-drafts@ietf.org > > Date: August 11, 2011 5:53:35 PM PDT > > To: warren@kumari.net > > Cc: joel.halpern@ericsson.com, warren= @kumari.net > > Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobility= -00.txt > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been= successfully submitted by Warren Kumari and posted to the IETF repository. > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > Revision: 00 > > Title: Virtual Machine mobility in L3 Networks. > > Creation date: 2011-08-11 > > WG ID: Individual Submission > > Number of pages: 8 > > > > Abstract: > > This document outlines how Virtual Machine mobility can be > > accomplished in datacenter networks that are based on L3 > > technologies. It is not really intended to solve (or fully define) > > the problem, but rather to outline it at a very high level to > > determine if standardization within the IETF makes sense. > > > > > > > > > > The IETF Secretariat > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --_000_7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2MX14Acorpemcc_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

    = Hi Gary,

     

    =

    Are you aware of the IETF work already in progress on that= sort of decoupling, e.g., LISP, HIP and ILNP?

     

    I don’t unders= tand why another effort in this space would be useful.

     

    Thanks,=
    --David

     

    From: Gary Berger [mai= lto:gaberger@cisco.com]
    Sent: Friday, August 12, 2011 10:02 AMTo: Black, David; warren@kumari.net
    Cc: armd@ietf.orgSubject: Re: [armd] Fwd: New Version Notification for draft-wkumari= -dcops-l3-vmmobility-00.txt

     

    I would hop= e that we can find a more elegant approach to the problem of dealing with d= ynamic address learning and mobility instead of adding overlays. =

     

    We have created the L2 scaling = problem (which shows itself in mobility and control plane scalability) beca= use of the fact that the data-link address, IP Address and what we use to i= dentify the host (I.e. The DNS name) are all bound to the point of attachme= nt. This created the desire for the "flat" network which as we kn= ow doesn't scale. It would be great if part of this working group can look = from the point of view that the Internet is an experiment and we never evol= ved to a clean implementation. Decoupling the service (I.e. Where a socket = call connects to) from the node address and the point of attachment will go= a long way to fixing the scaling challenges. A better description &nb= sp;can be found in Saltzer, et al  RFC-1498. Also, one of the chief OSI developers John Day ha= s an interesting perspective on this here http://pouzin.pnanetworks.co= m/images/KoreaNamingFund100218.pdf

     

    -g

    <= span style=3D'font-size:10.5pt;font-family:"Calibri","sans-serif";color:bla= ck'> 

    On 8/1= 2/11 9:08 AM, "david.black@emc.= com" <david.black@emc.co= m> wrote:

    =  

    Hi Warren,

    <= span style=3D'font-size:10.5pt;font-family:"Calibri","sans-serif";color:bla= ck'> 

    > It is good to see the draft.

    =

    And it's good to see that someone has alread= y read it :-P

     

    = Make that at least two someones ;-).  Thanks for getting the conv= ersation started.

     

    A detaile= d example of related technology can be found in draft-hasmit-otv-03

    (http://www.ietf.org/id/draft-hasmit-otv= -03.txt).  This L2-in-L3 functionality

    <= /div>

    (MAC-in-IP) is targeted at carrier/prov= ider networks rather than data center

    networks, and uses UDP encapsulation (e.g., as opposed= to GRE).  Courtesy of

    the focus on carrier/provider networks, one of the areas = of design emphasis

    is avoidance of flooding.

     

    Please don't ask me about OTV details - the draft authors (e.g., Dino) wo= uld

    be a better r= esource for those sorts of questions.

     

    Thanks,

    --David

    ------= ----------------------------------------------

    <= div>

    David L. Black, Distinguished Engineer

    EMC Corporation, 176 So= uth St., Hopkinton, MA  01748

    +1 (508) 293-7953      &nbs= p;      FAX: +1 (508) 293-7786

    david.black@emc.com        Mobile= : +1 (978) 394-7754

    ----------------------------------------------------

     

    -----Original Message-----=

    From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Warr= en Kumari

    Sent: = Friday, August 12, 2011 12:33 AM

    To: Vishwas Manral

    Cc: armd@ietf.org<= o:p>

    Subject: Re: [arm= d] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.t= xt

    On Aug 11, 201= 1, at 9:05 PM, Vishwas Manral wrote:

    > Hi Warren,

    > 

    > It is good to see the draft.

    <= div>

    And it's good to see that someone has already= read it :-P

    >= I guess you can position this as an informational draft of how network ope= rators are using

    = techniques to overcome big layer-2 issues (the ARP issues themselves are re= solved by the directory

    mechanism). I think this approach has been discussed in VNRG for som= e time now. May be you can use the

    terms defined there.

    Yup.

    > 

    <= span style=3D'font-size:10.5pt;font-family:"Calibri","sans-serif";color:bla= ck'>> Also if I understand right, we are still using a big layer-2 netwo= rk, only thing by using a

    heirarchical overlay network we are doing away with the actual net= work devices switches/ routers using

    Layer-2?

    Kinda -- the very high level view is that datacenters can be bu= ilt with L3 designs (L2 designs, or

    whatever design happens to exists), and then separate ov= erlay networks, one per customer.

    > 

    > As is well known "Overlay networks" have issues = with scalability, which become worse as the number

    of overlays increases (think of all IPsec= VPN's).

    Yes, b= ut as the overlay is only built between the VM hosts (and only *those* that= are actually

    com= municating), the number of overlays that needs to be tracked *per device* i= s fairly limited. Also,

    the overlays are built on general purpose servers (and are invisible= to the network) and so you don't

    have all of the standard scaling issues that happen with = network devices...

    > How do we deal with the same? Also what about issues with fragmentat= ion/ management of hypervisor

    based shim etc?

    Yup, there is still lots of work to be done on specifying thing= s like that -- who does fragmentation /

    reassembly, a standard protocol for populating the d= irectory, how the DS informs the shims that a VM

    has moved, etc are all (currently) coverd b= y hand-waving -- I do actually have answers on most of

    that, but I don't have them written d= own..

    >&n= bsp;

    > It would be = good if you could give all such details.

    > 

    > Thanks,

    > Vishwas

    > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari <warren@kumari.net> wrote:

    > Hi there all,=

    > 

    > At two previous meetings (IE= TF in Quebec and NANOG in Denver) I threw a bit of a hissy fit about how

    VM mobility can be = implemented without the need for L2 between the hosts.

    > 

    > So, here is a very drafty draft, outli= ning the principle at a high level.

    > 

    > I have no real plans for this draft, it's more just an ex= planation of the idea.

    > 

    = > W

    >&= nbsp;

    > <= /o:p>

    > Begin forwarded m= essage:

    >=  

    > > Date: Au= gust 11, 2011 5:53:35 PM PDT

    > > Subject: New V= ersion Notification for draft-wkumari-dcops-l3-vmmobility-00.txt=

    > >=

    > > A new version of I-D, dr= aft-wkumari-dcops-l3-vmmobility-00.txt has been successfully submitted by

    Warren Kumari and = posted to the IETF repository.

    > >

    > > Filename:      draft-wkumari= -dcops-l3-vmmobility

    <= span style=3D'font-size:10.5pt;font-family:"Calibri","sans-serif";color:bla= ck'>> > Revision:      00

    > > Title:  &n= bsp;            = ;  Virtual Machine mobility in L3 Networks.

    > > Creation date:   &= nbsp;     2011-08-11

    > > WG ID:      = ;           Individual Su= bmission

    > &= gt; Number of pages: 8

    > >

    &= gt; > Abstract:

    > >   This document outlines how Virtual Machine mobility= can be

    > >=    accomplished in datacenter networks that are based on L3<= /o:p>

    > >   = technologies.  It is not really intended to solve (or fully defin= e)

    > > = ;  the problem, but rather to outline it at a very high level to<= /o:p>

    > >   = determine if standardization within the IETF makes sense.=

    > >

    > >

    > >

    > >

    > > The IETF Secretariat

    > >

    > 

    > _______________________________________________

    > armd mailing list

    > https://www.ietf.org/mailman/listinfo/armd

    > 

    ______________________________________________= _

    armd mailing li= st

    <= /blockquote>

     

    __________________________________________= _____

    armd mailin= g list

    = --_000_7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2MX14Acorpemcc_-- From gaberger@cisco.com Fri Aug 12 10:50:40 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3763821F8634 for ; Fri, 12 Aug 2011 10:50:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.902 X-Spam-Level: X-Spam-Status: No, score=-0.902 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_65=0.6, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rZlofyTMECw1 for ; Fri, 12 Aug 2011 10:50:36 -0700 (PDT) Received: from rcdn-iport-8.cisco.com (rcdn-iport-8.cisco.com [173.37.86.79]) by ietfa.amsl.com (Postfix) with ESMTP id 1596721F861A for ; Fri, 12 Aug 2011 10:50:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=48680; q=dns/txt; s=iport; t=1313171474; x=1314381074; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=1A1dbepGfO1HFvjimM2DEJLPq6n0qpIk2Pm9uJ/UHJI=; b=k/KDpMyzwYiofWUtDrl6PKUECBnu5LSgGEK1/vamLzs0XmTkn9XIpgVI oTdb/4QmIITLGz0toqKXQtCGVKmgl2PGtN6PB4zBpX1cvENdX13ztgB/R 49yOd6v5MtsZGX2wI6ZY2eqVEf8ccRLt0g39wO65V2sk1FQo1uZ2c5iYP 8=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtoAAKNnRU6tJXG9/2dsb2JhbAA3BwOCTZVfh0OHImh3gUABAQEBAQEBAQEBDwEKEBAqBgEJAgUHBwgRAQIBAQEBIAEGLh8DBggGDgUJGYdNBJwDAZ5xgyYagwcEhzCLYIUMjAE X-IronPort-AV: E=Sophos;i="4.67,363,1309737600"; d="scan'208,217";a="12668790" Received: from rcdn-core2-2.cisco.com ([173.37.113.189]) by rcdn-iport-8.cisco.com with ESMTP; 12 Aug 2011 17:51:13 +0000 Received: from [10.82.225.231] (rtp-vpn1-487.cisco.com [10.82.225.231]) by rcdn-core2-2.cisco.com (8.14.3/8.14.3) with ESMTP id p7CHp9bb026700; Fri, 12 Aug 2011 17:51:10 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Fri, 12 Aug 2011 13:51:08 -0400 From: Gary Berger To: Message-ID: Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt In-Reply-To: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2@MX14A.corp.emc.com> Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3396001871_72671282" Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 17:50:40 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3396001871_72671282 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable I am not advocating any such thing.. I merely am trying to point to a bit o= f history and the problem at hand; as noted in the ARMD Call for Investigation: One such aspect,being investigated by the ARMD working group, is the scalin= g of address resolution between the network (L3) and link (L2) layers of modern datacenter networks. You cannot decouple the resolution and binding of addresses without understanding the limitations in the current architecture. LISP doesn't scale because of the path liveness problem, I am not familiar enough with EIP or ILNP but I am sure that this issue is long-lived and unsolved and at the root of our scaling challenges not just in DC but across the Internet.. -g From: Date: Fri, 12 Aug 2011 13:37:28 -0400 To: Gary Berger Cc: Subject: RE: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt Hi Gary, =20 Are you aware of the IETF work already in progress on that sort of decoupling, e.g., LISP, HIP and ILNP? =20 I don=B9t understand why another effort in this space would be useful. =20 Thanks, --David =20 From: Gary Berger [mailto:gaberger@cisco.com] Sent: Friday, August 12, 2011 10:02 AM To: Black, David; warren@kumari.net Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt =20 I would hope that we can find a more elegant approach to the problem of dealing with dynamic address learning and mobility instead of adding overlays.=20 =20 We have created the L2 scaling problem (which shows itself in mobility and control plane scalability) because of the fact that the data-link address, IP Address and what we use to identify the host (I.e. The DNS name) are all bound to the point of attachment. This created the desire for the "flat" network which as we know doesn't scale. It would be great if part of this working group can look from the point of view that the Internet is an experiment and we never evolved to a clean implementation. Decoupling the service (I.e. Where a socket call connects to) from the node address and th= e point of attachment will go a long way to fixing the scaling challenges. A better description can be found in Saltzer, et al RFC-1498 . Also, one of the chief OSI developer= s John Day has an interesting perspective on this here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf =20 -g =20 On 8/12/11 9:08 AM, "david.black@emc.com" wrote: =20 >=20 > Hi Warren, >=20 > =20 >>=20 >>> > It is good to see the draft. >>=20 >> And it's good to see that someone has already read it :-P >=20 > =20 >=20 > Make that at least two someones ;-). Thanks for getting the conversation > started. >=20 > =20 >=20 > A detailed example of related technology can be found in draft-hasmit-otv= -03 >=20 > (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 function= ality >=20 > (MAC-in-IP) is targeted at carrier/provider networks rather than data cen= ter >=20 > networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy= of >=20 > the focus on carrier/provider networks, one of the areas of design emphas= is >=20 > is avoidance of flooding. >=20 > =20 >=20 > Please don't ask me about OTV details - the draft authors (e.g., Dino) wo= uld >=20 > be a better resource for those sorts of questions. >=20 > =20 >=20 > Thanks, >=20 > --David >=20 > ---------------------------------------------------- >=20 > David L. Black, Distinguished Engineer >=20 > EMC Corporation, 176 South St., Hopkinton, MA 01748 >=20 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 >=20 > david.black@emc.com Mobile: +1 (978) 394-7754 >=20 > ---------------------------------------------------- >=20 > =20 >>=20 >> -----Original Message----- >>=20 >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >> Warren Kumari >>=20 >> Sent: Friday, August 12, 2011 12:33 AM >>=20 >> To: Vishwas Manral >>=20 >> Cc: armd@ietf.org >>=20 >> Subject: Re: [armd] Fwd: New Version Notification for >> draft-wkumari-dcops-l3-vmmobility-00.txt >>=20 >> On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: >>=20 >>> > Hi Warren, >>=20 >>> >=20 >>=20 >>> > It is good to see the draft. >>=20 >> And it's good to see that someone has already read it :-P >>=20 >>> > I guess you can position this as an informational draft of how networ= k >>> operators are using >>=20 >> techniques to overcome big layer-2 issues (the ARP issues themselves are >> resolved by the directory >>=20 >> mechanism). I think this approach has been discussed in VNRG for some ti= me >> now. May be you can use the >>=20 >> terms defined there. >>=20 >> Yup. >>=20 >>> >=20 >>=20 >>> > Also if I understand right, we are still using a big layer-2 network,= only >>> thing by using a >>=20 >> heirarchical overlay network we are doing away with the actual network >> devices switches/ routers using >>=20 >> Layer-2? >>=20 >> Kinda -- the very high level view is that datacenters can be built with = L3 >> designs (L2 designs, or >>=20 >> whatever design happens to exists), and then separate overlay networks, = one >> per customer. >>=20 >>> >=20 >>=20 >>> > As is well known "Overlay networks" have issues with scalability, whi= ch >>> become worse as the number >>=20 >> of overlays increases (think of all IPsec VPN's). >>=20 >> Yes, but as the overlay is only built between the VM hosts (and only *th= ose* >> that are actually >>=20 >> communicating), the number of overlays that needs to be tracked *per dev= ice* >> is fairly limited. Also, >>=20 >> the overlays are built on general purpose servers (and are invisible to = the >> network) and so you don't >>=20 >> have all of the standard scaling issues that happen with network devices= ... >>=20 >>> > How do we deal with the same? Also what about issues with fragmentati= on/ >>> management of hypervisor >>=20 >> based shim etc? >>=20 >> Yup, there is still lots of work to be done on specifying things like th= at -- >> who does fragmentation / >>=20 >> reassembly, a standard protocol for populating the directory, how the DS >> informs the shims that a VM >>=20 >> has moved, etc are all (currently) coverd by hand-waving -- I do actuall= y >> have answers on most of >>=20 >> that, but I don't have them written down.. >>=20 >>> >=20 >>=20 >>> > It would be good if you could give all such details. >>=20 >>> >=20 >>=20 >>> > Thanks, >>=20 >>> > Vishwas >>=20 >>> > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari wr= ote: >>=20 >>> > Hi there all, >>=20 >>> >=20 >>=20 >>> > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw= a >>> bit of a hissy fit about how >>=20 >> VM mobility can be implemented without the need for L2 between the hosts= . >>=20 >>> >=20 >>=20 >>> > So, here is a very drafty draft, outlining the principle at a high le= vel. >>=20 >>> >=20 >>=20 >>> > I have no real plans for this draft, it's more just an explanation of= the >>> idea. >>=20 >>> >=20 >>=20 >>> > W >>=20 >>> >=20 >>=20 >>> >=20 >>=20 >>> > Begin forwarded message: >>=20 >>> >=20 >>=20 >>>> > > From: internet-drafts@ietf.org >>=20 >>>> > > Date: August 11, 2011 5:53:35 PM PDT >>=20 >>>> > > To: warren@kumari.net >>=20 >>>> > > Cc: joel.halpern@ericsson.com, warren@kumari.net >>=20 >>>> > > Subject: New Version Notification for >>>> draft-wkumari-dcops-l3-vmmobility-00.txt >>=20 >>>> > > >>=20 >>>> > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has= been >>>> successfully submitted by >>=20 >> Warren Kumari and posted to the IETF repository. >>=20 >>>> > > >>=20 >>>> > > Filename: draft-wkumari-dcops-l3-vmmobility >>=20 >>>> > > Revision: 00 >>=20 >>>> > > Title: Virtual Machine mobility in L3 Networks. >>=20 >>>> > > Creation date: 2011-08-11 >>=20 >>>> > > WG ID: Individual Submission >>=20 >>>> > > Number of pages: 8 >>=20 >>>> > > >>=20 >>>> > > Abstract: >>=20 >>>> > > This document outlines how Virtual Machine mobility can be >>=20 >>>> > > accomplished in datacenter networks that are based on L3 >>=20 >>>> > > technologies. It is not really intended to solve (or fully defi= ne) >>=20 >>>> > > the problem, but rather to outline it at a very high level to >>=20 >>>> > > determine if standardization within the IETF makes sense. >>=20 >>>> > > >>=20 >>>> > > >>=20 >>>> > > >>=20 >>>> > > >>=20 >>>> > > The IETF Secretariat >>=20 >>>> > > >>=20 >>> >=20 >>=20 >>> > _______________________________________________ >>=20 >>> > armd mailing list >>=20 >>> > armd@ietf.org >>=20 >>> > https://www.ietf.org/mailman/listinfo/armd >>=20 >>> >=20 >>=20 >> _______________________________________________ >>=20 >> armd mailing list >>=20 >> armd@ietf.org >>=20 >> https://www.ietf.org/mailman/listinfo/armd >=20 > =20 >=20 > _______________________________________________ >=20 > armd mailing list >=20 > armd@ietf.org >=20 > https://www.ietf.org/mailman/listinfo/armd >=20 > =20 --B_3396001871_72671282 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable
    I am not advocating any such thing.. I merely am trying = to point to a bit of history and the problem at hand; as noted in the ARMD C= all for Investigation:

    One such aspect,being investigated by the ARMD w=
orking group, is the scaling of address resolution between the network (L3) =
and link (L2) layers of modern datacenter networks.
    You cannot decouple the resolution and bind=
ing of addresses without understanding the limitations in the current archit=
ecture. LISP doesn't scale because of the path liveness problem, I am not fa=
miliar enough with EIP or ILNP but I am sure that this issue is long-lived a=
nd unsolved and at the root of our scaling challenges not just in DC but acr=
oss the Internet.. 


    -g



    From: <david.black@emc.co= m>
    Date: Fri, 12 Aug 2011 1= 3:37:28 -0400
    To: Gary Berger <= gaberger@cisco.com>
    Cc: <armd@iet= f.org>
    Subject: RE: [armd] = Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt

    Hi Gary,

     

    Are you aware of the IETF work already in progress on that sort of= decoupling, e.g., LISP, HIP and ILNP?

    I don’t understand w= hy another effort in this space would be useful.

     

    Thanks,
    = --David

     =

    From: Gary Berger [mailto:gaberger@cisco.com]
    Sent: Friday, Au= gust 12, 2011 10:02 AM
    To: Black, David; warren@kumari.net
    Cc: armd@ietf.org
    Subject: Re: [armd] Fwd: New Version Notificati= on for draft-wkumari-dcops-l3-vmmobility-00.txt

    <= /div>

     

    I would hope that we can find a more elegant approach to the problem= of dealing with dynamic address learning and mobility instead of adding ove= rlays. 

     

    We have create= d the L2 scaling problem (which shows itself in mobility and control plane s= calability) because of the fact that the data-link address, IP Address and w= hat we use to identify the host (I.e. The DNS name) are all bound to the poi= nt of attachment. This created the desire for the "flat" network which as we= know doesn't scale. It would be great if part of this working group can loo= k from the point of view that the Internet is an experiment and we never evo= lved to a clean implementation. Decoupling the service (I.e. Where a socket = call connects to) from the node address and the point of attachment will go = a long way to fixing the scaling challenges. A better description  = ;can be found in Saltzer, et al  RFC-1498. Also, one of the chief OSI developers John Day has an i= nteresting perspective on this here http://pouzin.pnanetworks.com/images/K= oreaNamingFund100218.pdf

     

    &= nbsp;

    On 8/12/11 9:08 = AM, "david.black@emc.com" <david.black@emc.com> wrote:

     

    Hi Warren,<= /span>

     

    =

    > It is good to see t= he draft.

    And it'= s good to see that someone has already read it :-P

     

    <= /div>

    Make that at least two someones ;-).&nb= sp; Thanks for getting the conversation started.

    <= /div>

     

    =

    A detailed example of related technology can be f= ound in draft-hasmit-otv-03

    (http://= www.ietf.org/id/draft-hasmit-otv-03.txt).  This L2-in-L3 funct= ionality

    (MAC-in-= IP) is targeted at carrier/provider networks rather than data center

    networks, and uses UDP e= ncapsulation (e.g., as opposed to GRE).  Courtesy of

    the focus on carrier/provider = networks, one of the areas of design emphasis

    is avoidance of flooding.

     

    Please don't ask me about OTV details - the dra= ft authors (e.g., Dino) would

    be a better resource for those sorts of questions.

     

    <= /div>

    Thanks,

    --David

    ----------------------------------------------------=

    David L. Black, Disti= nguished Engineer

    EMC Corporation, 176 South St., Hopkinton, MA  01748=

    +1 (508) 293-7953   = ;          FAX: +1 (508) 293-77= 86

    david.black@emc.com     =    Mobile: +1 (978) 394-7754

    ----------------------------------------------------

     <= /span>

    -----Original = Message-----

    From= : armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f Warren Kumari

    S= ent: Friday, August 12, 2011 12:33 AM

    To: Vishwas Manral

    Cc: armd@ietf.org<= o:p>

    Subject: Re: [arm= d] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.tx= t

    On Aug 11, 2011= , at 9:05 PM, Vishwas Manral wrote:

    > Hi Warren,

    > 

    > It is good to see the draft.

    And it's good to see that someone has already r= ead it :-P

    > I= guess you can position this as an informational draft of how network operat= ors are using

    tec= hniques to overcome big layer-2 issues (the ARP issues themselves are resolv= ed by the directory

    terms defined there.

    Yup.

    <= span style=3D"font-size: 10.5pt; color: black; font-family: Calibri, sans-seri= f; ">> 

    &= gt; Also if I understand right, we are still using a big layer-2 network, on= ly thing by using a

    Layer-2?

    <= span style=3D"font-size: 10.5pt; color: black; font-family: Calibri, sans-seri= f; ">Kinda -- the very high level view is that datacenters can be built with= L3 designs (L2 designs, or

    whatever design happens to exists), and then separate overlay net= works, one per customer.

    > 

    of overlays increases (think of all IPsec VPN's).

    Yes, but as the overlay is onl= y built between the VM hosts (and only *those* that are actually<= /span>

    communicating), the number o= f overlays that needs to be tracked *per device* is fairly limited. Also,

    the overlays are bu= ilt on general purpose servers (and are invisible to the network) and so you= don't

    have all o= f the standard scaling issues that happen with network devices...=

    > How do we deal with th= e same? Also what about issues with fragmentation/ management of hypervisor<= o:p>

    based shim etc?

    Yup, there is stil= l lots of work to be done on specifying things like that -- who does fragmen= tation /

    reassemb= ly, a standard protocol for populating the directory, how the DS informs the= shims that a VM

    = has moved, etc are all (currently) coverd by hand-waving -- I do actually ha= ve answers on most of

    <= span style=3D"font-size: 10.5pt; color: black; font-family: Calibri, sans-seri= f; ">that, but I don't have them written down..

    <= div>

    > 

    > It would be good if you could give all such de= tails.

    >&= nbsp;

    > Thanks,

    > Vishwas

    > On Thu, Aug 11, 201= 1 at 7:44 PM, Warren Kumari <warren@ku= mari.net> wrote:

    > Hi there all,

    > 

    VM mobility can be implemented without the need for L2 betwe= en the hosts.

    >= ; 

    > So, = here is a very drafty draft, outlining the principle at a high level.

    > 

    > I have no real plans for= this draft, it's more just an explanation of the idea.

    > 

    > W

    > 

    > 

    > Begin forwarded message:

    > 

    > > Date: August 11, 2011 5:53:35 PM PDT=

    >= > Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobilit= y-00.txt

    > >= ;

    > > A new= version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been successfu= lly submitted by

    = Warren Kumari and posted to the IETF repository.

    =

    > >

    > > Filename:     &n= bsp;draft-wkumari-dcops-l3-vmmobility

    > > Revision:      = 00

    > > Titl= e:            &n= bsp;    Virtual Machine mobility in L3 Networks.

    > > Creation date: = ;        2011-08-11

    =

    > > WG ID:   &nbs= p;             I= ndividual Submission

    > > Number of pages: 8

    > >

    > > Abstract:

    > >   This document outlines how Virtual Machine= mobility can be

    = > >   accomplished in datacenter networks that are based on = L3

    > > = ;  technologies.  It is not really intended to solve (or full= y define)

    > &g= t;   the problem, but rather to outline it at a very high level to=

    > > &= nbsp; determine if standardization within the IETF makes sense.

    > >

    > >

    > >

    <= p class=3D"MsoNormal">> >

    > > The IETF Secretariat

    > >

    <= p class=3D"MsoNormal">> 

    > _______________________________________________

    > armd mailing li= st

    > https://www.ietf.org/mailman/listinfo/armd

    > 

    _______________________________________________=

    armd mailing lis= t

     

    =

    _______________________________________________

    armd mailing list<= o:p>

     

    <= /div>
    --B_3396001871_72671282-- From ning.so@verizon.com Fri Aug 12 11:12:42 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BC8B011E808F for ; Fri, 12 Aug 2011 11:12:42 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.452 X-Spam-Level: X-Spam-Status: No, score=-2.452 tagged_above=-999 required=5 tests=[AWL=-0.454, BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_65=0.6] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GCZOzLbXM4K9 for ; Fri, 12 Aug 2011 11:12:38 -0700 (PDT) Received: from fldsmtpe02.verizon.com (fldsmtpe02.verizon.com [140.108.26.141]) by ietfa.amsl.com (Postfix) with ESMTP id 4138611E8087 for ; Fri, 12 Aug 2011 11:12:38 -0700 (PDT) X-IronPort-Anti-Spam-Filtered: false Received: from unknown (HELO fldsmtpi03.verizon.com) ([166.68.71.145]) by fldsmtpe02.verizon.com with ESMTP; 12 Aug 2011 18:12:05 +0000 From: "So, Ning" X-IronPort-AV: E=Sophos;i="4.67,363,1309737600"; d="scan'208,217";a="113806958" Received: from fhdp1lumxc7hb03.verizon.com (HELO FHDP1LUMXC7HB03.us.one.verizon.com) ([166.68.59.190]) by fldsmtpi03.verizon.com with ESMTP; 12 Aug 2011 18:12:05 +0000 Received: from FHDP1LUMXC7V41.us.one.verizon.com ([169.254.1.38]) by FHDP1LUMXC7HB03.us.one.verizon.com ([166.68.59.190]) with mapi; Fri, 12 Aug 2011 14:12:05 -0400 To: Gary Berger , "david.black@emc.com" Date: Fri, 12 Aug 2011 14:12:03 -0400 Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt Thread-Index: AcxZGHPhuX/y3vATTTW5tWs7GF/xbQAAqgkA Message-ID: <6665BC1FEA04AB47B1F75FA641C43BC0813C4D9E@FHDP1LUMXC7V41.us.one.verizon.com> References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2@MX14A.corp.emc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_6665BC1FEA04AB47B1F75FA641C43BC0813C4D9EFHDP1LUMXC7V41u_" MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 18:12:42 -0000 --_000_6665BC1FEA04AB47B1F75FA641C43BC0813C4D9EFHDP1LUMXC7V41u_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Gary, Can you elaborate a bit on LISP's path liveness problem? Is it documented = anywhere? Best regards, Ning So Verizon Corporate Technology (office) 972-729-7905 (Cell) 972-955-0914 From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Gar= y Berger Sent: Friday, August 12, 2011 12:51 PM To: david.black@emc.com Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l= 3-vmmobility-00.txt I am not advocating any such thing.. I merely am trying to point to a bit o= f history and the problem at hand; as noted in the ARMD Call for Investigat= ion: One such aspect,being investigated by the ARMD working group, is the scalin= g of address resolution between the network (L3) and link (L2) layers of mo= dern datacenter networks. You cannot decouple the resolution and binding of addresses without underst= anding the limitations in the current architecture. LISP doesn't scale beca= use of the path liveness problem, I am not familiar enough with EIP or ILNP= but I am sure that this issue is long-lived and unsolved and at the root o= f our scaling challenges not just in DC but across the Internet.. -g From: > Date: Fri, 12 Aug 2011 13:37:28 -0400 To: Gary Berger > Cc: > Subject: RE: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l= 3-vmmobility-00.txt Hi Gary, Are you aware of the IETF work already in progress on that sort of decoupli= ng, e.g., LISP, HIP and ILNP? I don't understand why another effort in this space would be useful. Thanks, --David From: Gary Berger [mailto:gaberger@cisco.com] Sent: Friday, August 12, 2011 10:02 AM To: Black, David; warren@kumari.net Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l= 3-vmmobility-00.txt I would hope that we can find a more elegant approach to the problem of dea= ling with dynamic address learning and mobility instead of adding overlays. We have created the L2 scaling problem (which shows itself in mobility and = control plane scalability) because of the fact that the data-link address, = IP Address and what we use to identify the host (I.e. The DNS name) are all= bound to the point of attachment. This created the desire for the "flat" n= etwork which as we know doesn't scale. It would be great if part of this wo= rking group can look from the point of view that the Internet is an experim= ent and we never evolved to a clean implementation. Decoupling the service = (I.e. Where a socket call connects to) from the node address and the point = of attachment will go a long way to fixing the scaling challenges. A better= description can be found in Saltzer, et al RFC-1498. Also, one of the chief OSI developers John Day has an inte= resting perspective on this here http://pouzin.pnanetworks.com/images/Korea= NamingFund100218.pdf -g On 8/12/11 9:08 AM, "david.black@emc.com" > wrote: Hi Warren, > It is good to see the draft. And it's good to see that someone has already read it :-P Make that at least two someones ;-). Thanks for getting the conversation s= tarted. A detailed example of related technology can be found in draft-hasmit-otv-0= 3 (http://www.ietf.org/id/draft-hasmit-otv-03.txt). This L2-in-L3 functional= ity (MAC-in-IP) is targeted at carrier/provider networks rather than data cente= r networks, and uses UDP encapsulation (e.g., as opposed to GRE). Courtesy o= f the focus on carrier/provider networks, one of the areas of design emphasis is avoidance of flooding. Please don't ask me about OTV details - the draft authors (e.g., Dino) woul= d be a better resource for those sorts of questions. Thanks, --David ---------------------------------------------------- David L. Black, Distinguished Engineer EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 david.black@emc.com Mobile: +1 (978) 394= -7754 ---------------------------------------------------- -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-boun= ces@ietf.org] On Behalf Of Warren Kumari Sent: Friday, August 12, 2011 12:33 AM To: Vishwas Manral Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l= 3-vmmobility-00.txt On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote: > Hi Warren, > > It is good to see the draft. And it's good to see that someone has already read it :-P > I guess you can position this as an informational draft of how network op= erators are using techniques to overcome big layer-2 issues (the ARP issues themselves are re= solved by the directory mechanism). I think this approach has been discussed in VNRG for some time = now. May be you can use the terms defined there. Yup. > > Also if I understand right, we are still using a big layer-2 network, onl= y thing by using a heirarchical overlay network we are doing away with the actual network devi= ces switches/ routers using Layer-2? Kinda -- the very high level view is that datacenters can be built with L3 = designs (L2 designs, or whatever design happens to exists), and then separate overlay networks, one= per customer. > > As is well known "Overlay networks" have issues with scalability, which b= ecome worse as the number of overlays increases (think of all IPsec VPN's). Yes, but as the overlay is only built between the VM hosts (and only *those= * that are actually communicating), the number of overlays that needs to be tracked *per device= * is fairly limited. Also, the overlays are built on general purpose servers (and are invisible to the= network) and so you don't have all of the standard scaling issues that happen with network devices... > How do we deal with the same? Also what about issues with fragmentation/ = management of hypervisor based shim etc? Yup, there is still lots of work to be done on specifying things like that = -- who does fragmentation / reassembly, a standard protocol for populating the directory, how the DS in= forms the shims that a VM has moved, etc are all (currently) coverd by hand-waving -- I do actually h= ave answers on most of that, but I don't have them written down.. > > It would be good if you could give all such details. > > Thanks, > Vishwas > On Thu, Aug 11, 2011 at 7:44 PM, Warren Kumari > wrote: > Hi there all, > > At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a b= it of a hissy fit about how VM mobility can be implemented without the need for L2 between the hosts. > > So, here is a very drafty draft, outlining the principle at a high level. > > I have no real plans for this draft, it's more just an explanation of the= idea. > > W > > > Begin forwarded message: > > > From: internet-drafts@ietf.org > > Date: August 11, 2011 5:53:35 PM PDT > > To: warren@kumari.net > > Cc: joel.halpern@ericsson.com, warren= @kumari.net > > Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobility= -00.txt > > > > A new version of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been= successfully submitted by Warren Kumari and posted to the IETF repository. > > > > Filename: draft-wkumari-dcops-l3-vmmobility > > Revision: 00 > > Title: Virtual Machine mobility in L3 Networks. > > Creation date: 2011-08-11 > > WG ID: Individual Submission > > Number of pages: 8 > > > > Abstract: > > This document outlines how Virtual Machine mobility can be > > accomplished in datacenter networks that are based on L3 > > technologies. It is not really intended to solve (or fully define) > > the problem, but rather to outline it at a very high level to > > determine if standardization within the IETF makes sense. > > > > > > > > > > The IETF Secretariat > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --_000_6665BC1FEA04AB47B1F75FA641C43BC0813C4D9EFHDP1LUMXC7V41u_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

    Gary,

     

    <= p class=3DMsoNormal>Can you elaborate a bit on LISP’s path liv= eness problem?  Is it documented anywhere?

     

     

    Best regards,=

     =

    Ning So=

    Verizon Corporate Technology

    (office) 972-729-7905

    (Cell) 972-955-0914

     <= o:p>

     

    From: armd-bounces@ietf.org [mailt= o:armd-bounces@ietf.org] On Behalf Of Gary Berger
    Sent: Fr= iday, August 12, 2011 12:51 PM
    To: david.black@emc.com
    Cc:<= /b> armd@ietf.org
    Subject: Re: [armd] Fwd: New Version Notificati= on for draft-wkumari-dcops-l3-vmmobility-00.txt

    =

     

    = I am not advocating any such thing.. I merely am trying to point to a bit o= f history and the problem at hand; as noted in the ARMD Call for Investigat= ion:

     

    One such aspect,being investigated by the =
ARMD working group, is the scaling of address resolution between the networ=
k (L3) and link (L2) layers of modern datacenter networks.

    You cannot decouple the resolution and=
 binding of addresses without understanding the limitations in the current =
architecture. LISP doesn't scale because of the path liveness problem, I am=
 not familiar enough with EIP or ILNP but I am sure that this issue is long=
-lived and unsolved and at the root of our scaling challenges not just in D=
C but across the Internet.. =



    -g

    =

    From: <david.black@emc.com>
    Date: Fri, 12 Aug 2011 13:37:28 -0400
    To: Gary Be= rger <    gaberger@cisco.com>Cc: <armd@ietf.org>
    = Subject: RE: [armd] Fwd: New Version Notification for draft-wkumari-= dcops-l3-vmmobility-00.txt

    <= span style=3D'font-size:10.5pt;font-family:"Calibri","sans-serif";color:bla= ck'> 

    Hi Gary,

    &nb= sp;

     

    I don’t understand why another eff= ort in this space would be useful.<= /o:p>

     <= o:p>

    Thanks,
    --David

     

    =

    From: Gary Berger [mailto:gaberger@cisco.com]
    Sent: Fr= iday, August 12, 2011 10:02 AM
    To: Black, David; warren@kumari.net
    Cc: armd@ietf.org
    Subject: Re: [armd] Fwd: New Ver= sion Notification for draft-wkumari-dcops-l3-vmmobility-00.txt

     

    I would hope that we can find a more elegant approach to= the problem of dealing with dynamic address learning and mobility instead = of adding overlays. 

     

    We hav= e created the L2 scaling problem (which shows itself in mobility and contro= l plane scalability) because of the fact that the data-link address, IP Add= ress and what we use to identify the host (I.e. The DNS name) are all bound= to the point of attachment. This created the desire for the "flat&quo= t; network which as we know doesn't scale. It would be great if part of thi= s working group can look from the point of view that the Internet is an exp= eriment and we never evolved to a clean implementation. Decoupling the serv= ice (I.e. Where a socket call connects to) from the node address and the po= int of attachment will go a long way to fixing the scaling challenges. = ;A better description  can be found in Saltzer, et al  RFC-1498. Also, one of the chief O= SI developers John Day has an interesting perspective on this here htt= p://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf

     

    -g<= /o:p>

     

    On 8/12/11 9:08 AM, "d= avid.black@emc.com" <dav= id.black@emc.com> wrote:

     

    Hi Warren,=

     

    =

    > It is good to see the draft.

    And = it's good to see that someone has already read it :-P

     

    Make that at least two someones ;-).&nbs= p; Thanks for getting the conversation started.

    &nbs= p;

    A detailed example of related technology can be found = in draft-hasmit-otv-03=

    Fr= om: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On = Behalf Of Warren Kumari

    Sent: Friday, August 12, 2011 12:= 33 AM

    To: Vishwas Manral

    Subject: Re: [ar= md] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.= txt

    On Aug 11, 2011, at 9:05 PM, Vishwas Manral wrote:

    > Hi Warren,

    > 

    > It is good to see the draft.<= o:p>

    And it's good to = see that someone has already read it :-P=

    > I guess you= can position this as an informational draft of how network operators are u= sing

    techniques to overcome big layer-2 issues (the ARP i= ssues themselves are resolved by the directory

    mechanism)= . I think this approach has been discussed in VNRG for some time now. May b= e you can use the

    <= /div>

    terms defined there.

    Yup.

    > <= /o:p>

    > Also if I underst= and right, we are still using a big layer-2 network, only thing by using a<= /span>

    heirarchical overlay network we are doing away with the a= ctual network devices switches/ routers using

    Layer-2?

    Kinda -- the very high level view is that datacenters can= be built with L3 designs (L2 designs, or

    whatever design= happens to exists), and then separate overlay networks, one per customer.<= /span>

    > <= /span>

    > As is well known "= Overlay networks" have issues with scalability, which become worse as = the number

    of overlays increases (think of all IPsec VPN'= s).

    Yes, but as the overlay is only built between the VM = hosts (and only *those* that are actually

    communicating),= the number of overlays that needs to be tracked *per device* is fairly lim= ited. Also,

    <= div>

    the overlays are built on general purpose ser= vers (and are invisible to the network) and so you don't

    have all of the standard scaling issues that happen with network devices..= .

    > How do we deal with the same? Also what about issu= es with fragmentation/ management of hypervisor

    based shi= m etc?

    <= p class=3DMsoNormal>Yup, there is still lots of work to be done on spe= cifying things like that -- who does fragmentation /

    reas= sembly, a standard protocol for populating the directory, how the DS inform= s the shims that a VM<= /p>

    has moved, etc are all (currently) = coverd by hand-waving -- I do actually have answers on most of

    that, but I don't have them written down..

    >&n= bsp;

    > It would be good if you could give all such det= ails.

    > <= /o:p>

    > Thanks,

    > Vishwas<= /p>

    > On Thu, Aug 11, 2011 at 7:44 P= M, Warren Kumari <warren@kumari.net= > wrote:

    > Hi there all,

    >=  

    <= p class=3DMsoNormal>> At two previous meetings (IETF in Quebec and = NANOG in Denver) I threw a bit of a hissy fit about how

    VM mobility can be implemented without the need for L2 between the hosts.<= /span>

    > <= /span>

    > So, here is a very draf= ty draft, outlining the principle at a high level.

    >&n= bsp;

    > I have no real plans for this draft, it's more = just an explanation of the idea.

    > 

    = > W

    <= div>

    > =

    > 

    > Begin forwarded message:

    > 

    > > From: = internet-drafts@ietf.org<= /span>

    > > Date: August 11, 2= 011 5:53:35 PM PDT

    =

    > > To: warren@kumari.net<= /o:p>

    >= > Subject: New Version Notification for draft-wkumari-dcops-l3-vmmobili= ty-00.txt

    > >

    > > A new ver= sion of I-D, draft-wkumari-dcops-l3-vmmobility-00.txt has been successfully= submitted by

    Warren Kumari and posted to the IETF reposi= tory.

    > >

    > > Filename:&nbs= p;     draft-wkumari-dcops-l3-vmmobility

    > > Revision:      00

    > > Title:        &= nbsp;        Virtual Machine mobility in= L3 Networks.

    > > Creation date:   &= nbsp;     2011-08-11=

    > > WG ID:=             &nb= sp;    Individual Submission

    > > Num= ber of pages: 8

    > >

    > > Abs= tract:

    <= p class=3DMsoNormal>> >   This document outlines how V= irtual Machine mobility can be

    > >   acco= mplished in datacenter networks that are based on L3

    >= >   technologies.  It is not really intended to sol= ve (or fully define)

    > >   the problem, b= ut rather to outline it at a very high level to

    > >=    determine if standardization within the IETF makes sense.

    > >

    > >

    > = >

    > >

    > > The IETF Secre= tariat

    <= p class=3DMsoNormal>> ><= /o:p>

    > 

    > _______________________________________________

    > armd mailing list

    > <= /span>

    ____________________________= ___________________

    armd mailing list

     

    _______________________________________________

    armd m= ailing list

    <= div>

    armd@ietf.or= g

     =

    = --_000_6665BC1FEA04AB47B1F75FA641C43BC0813C4D9EFHDP1LUMXC7V41u_-- From sblake@extremenetworks.com Fri Aug 12 11:29:39 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 14DD221F85F7 for ; Fri, 12 Aug 2011 11:29:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6ep30vgAwpOw for ; Fri, 12 Aug 2011 11:29:38 -0700 (PDT) Received: from ussc-casht-p1.extremenetworks.com (ussc-casht-p2.extremenetworks.com [207.179.9.62]) by ietfa.amsl.com (Postfix) with ESMTP id 9511221F85AB for ; Fri, 12 Aug 2011 11:29:38 -0700 (PDT) Received: from [10.5.2.56] (10.5.2.56) by ussc-casht-p1.corp.extremenetworks.com (10.0.4.73) with Microsoft SMTP Server id 8.1.358.0; Fri, 12 Aug 2011 11:30:15 -0700 From: Steven Blake To: "So, Ning" Date: Fri, 12 Aug 2011 14:30:14 -0400 In-Reply-To: <6665BC1FEA04AB47B1F75FA641C43BC0813C4D9E@FHDP1LUMXC7V41.us.one.verizon.com> References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609FF2@MX14A.corp.emc.com> <6665BC1FEA04AB47B1F75FA641C43BC0813C4D9E@FHDP1LUMXC7V41.us.one.verizon.com> Organization: Extreme Networks Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.0.2 (3.0.2-3.fc15) Content-Transfer-Encoding: 8bit Message-ID: <1313173815.30418.2.camel@ecliptic.extremenetworks.com> MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 18:29:39 -0000 On Fri, 2011-08-12 at 11:12 -0700, So, Ning wrote: > Gary, > > > > Can you elaborate a bit on LISP’s path liveness problem? Is it > documented anywhere? http://tools.ietf.org/html/draft-meyer-loc-id-implications-01 Note that the problem is not specific to loc/id separation; it is a general problem for multi-addressed hosts. Regards, ///////////////////////////////////////////// Steven Blake sblake@extremenetworks.com Extreme Networks +1 919-884-3211 From pfrejborg@gmail.com Sat Aug 13 00:40:24 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1AA6A21F8779 for ; Sat, 13 Aug 2011 00:40:24 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.9 X-Spam-Level: X-Spam-Status: No, score=-2.9 tagged_above=-999 required=5 tests=[AWL=0.700, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id beag552fncIH for ; Sat, 13 Aug 2011 00:40:23 -0700 (PDT) Received: from mail-wy0-f172.google.com (mail-wy0-f172.google.com [74.125.82.172]) by ietfa.amsl.com (Postfix) with ESMTP id DF35421F8565 for ; Sat, 13 Aug 2011 00:40:22 -0700 (PDT) Received: by wyg8 with SMTP id 8so2958176wyg.31 for ; Sat, 13 Aug 2011 00:41:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=IHEXY3TRnsXh/qFwx0qd5F76UsilACV2Mrqbns9AyZU=; b=kIa6dt3DFY6/bDOrZY0HfJm3Lz8b1TntHiowDtCgET1rwQZ+EFHcJs4jAvosnmr1B9 Z/sNQgroIzSxqatmqJYlrTGaQFdIHxU1b1bys01BUdEgKZm5n6oYrURkabHIn44XTBCE 52K4iY+foJtjo3YbQTvcVIbtPfOviKz2IGhHg= MIME-Version: 1.0 Received: by 10.227.57.211 with SMTP id d19mr1621361wbh.93.1313221261165; Sat, 13 Aug 2011 00:41:01 -0700 (PDT) Received: by 10.227.29.18 with HTTP; Sat, 13 Aug 2011 00:41:01 -0700 (PDT) In-Reply-To: References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> Date: Sat, 13 Aug 2011 10:41:01 +0300 Message-ID: From: Patrick Frejborg To: Gary Berger Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Aug 2011 07:40:24 -0000 On Fri, Aug 12, 2011 at 5:01 PM, Gary Berger wrote: > I would hope that we can find a more elegant approach to the problem of > dealing with dynamic address learning and mobility instead of adding > overlays. > We have created the L2 scaling problem (which shows itself in mobility an= d > control plane scalability) because of the fact that the data-link address= , > IP Address and what we use to identify the host (I.e. The DNS name) are a= ll > bound to the point of attachment. This created the desire for the "flat" > network which as we know doesn't scale. It would be great if part of this > working group can look from the point of view that the Internet is an > experiment and we never evolved to a clean implementation. Decoupling the > service (I.e. Where a socket call connects to) from the node address and = the > point of attachment will go a long way to fixing the scaling challenges.= =A0A > better description =A0can be found in Saltzer, et al =A0RFC-1498. Also, o= ne of > the chief OSI developers John Day has an interesting perspective on this > here=A0http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf Have also a look on the SCAFFOLD paper ftp://ftp.cs.princeton.edu/reports/2010/885.pdf That would remove the L2 scaling problem for online services The approach requires a hierarchical addressing scheme, check out RFC6306, which is focusing on scalability for the routing architecture of the Internet. The identifier mechanism described in SCAFFOLD can be applied with the packet header described in RFC6306 though it is not discussed in detail in the document. This combination of the two separate works would introduce a session identifier (by MPTCP/SCTP) and service identifier (by SCAFFOLD) that are decoupled from the addressing scheme (locators). Patrick From gaberger@cisco.com Sat Aug 13 06:26:56 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6DC7821F874E for ; Sat, 13 Aug 2011 06:26:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.864 X-Spam-Level: X-Spam-Status: No, score=0.864 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_NUMERIC_HELO=2.067] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1zBv4QPInOHN for ; Sat, 13 Aug 2011 06:26:55 -0700 (PDT) Received: from rcdn-iport-5.cisco.com (rcdn-iport-5.cisco.com [173.37.86.76]) by ietfa.amsl.com (Postfix) with ESMTP id 48D3221F876A for ; Sat, 13 Aug 2011 06:26:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=2914; q=dns/txt; s=iport; t=1313242055; x=1314451655; h=subject:references:content-transfer-encoding:from: in-reply-to:message-id:date:to:cc:mime-version; bh=iLwg24bAhNvsJNJnHtjV7psD5gXLuW7DjdLgPhfxwQQ=; b=Aun2/SebI30hqH396WZeP8cjMHC6ix3llbfnLRu8qjXlyETGHMoSY5tj 4+hmZtt/9PPcTeLD7KPH3jDm7YF311r+YwL6/b2xi/WuURVtcxE9RPxN8 ViedCF7w7MY+VHs47ds1LMYn8GQUQG6Uwb3m9kObybRY6l6eHn+8Giidy M=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjwIAAp7Rk6tJXG+/2dsb2JhbAA3CqcQZgJ3gUABAQEBAQEBEgEnMwoCBQsCAQgOBAYuISgOAQEEEyKHTgSafgGeK4MmgkJfBJMShRWEYYMBhBo X-IronPort-AV: E=Sophos;i="4.67,366,1309737600"; d="scan'208";a="12863027" Received: from rcdn-core2-3.cisco.com ([173.37.113.190]) by rcdn-iport-5.cisco.com with ESMTP; 13 Aug 2011 13:27:34 +0000 Received: from xbh-rcd-101.cisco.com (xbh-rcd-101.cisco.com [72.163.62.138]) by rcdn-core2-3.cisco.com (8.14.3/8.14.3) with ESMTP id p7DDRXQp015505; Sat, 13 Aug 2011 13:27:33 GMT Received: from xmb-rcd-201.cisco.com ([72.163.62.208]) by xbh-rcd-101.cisco.com with Microsoft SMTPSVC(6.0.3790.4675); Sat, 13 Aug 2011 08:27:31 -0500 Received: from 144.254.231.93 ([144.254.231.93]) by XMB-RCD-201.cisco.com ([72.163.62.208]) with Microsoft Exchange Server HTTP-DAV ; Sat, 13 Aug 2011 13:27:31 +0000 References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> Content-Transfer-Encoding: quoted-printable From: "Gary Berger (gaberger)" thread-topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt thread-index: AcxZvMDcyk4ytuSbSsiD5+okAd+t1Q== Content-Type: text/plain; charset="us-ascii" In-Reply-To: Message-ID: <7CDCDBB6-0E31-4A35-BDD5-492FD6B731AC@cisco.com> Date: Sat, 13 Aug 2011 09:27:29 -0400 To: "Patrick Frejborg" MIME-Version: 1.0 (iPhone Mail 8C148) X-OriginalArrivalTime: 13 Aug 2011 13:27:31.0758 (UTC) FILETIME=[C13C4CE0:01CC59BC] Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Aug 2011 13:26:56 -0000 Hi Patrick, Yes I have read SCAFFOLD and I like the approach. Fundamentally you either s= olve lookup through exhaustive search or through hinting and a hierarchal ad= dress is critical to this. It doesn't solve the problem completely but it's a= good vision of where we can reformulate our view of binding layers. =46rom= an appdev point of view being able to target an abstract topology seems inc= redibly beneficial. The topology can be geographically bounded, segmented by= service or customer. There is a group lead by Martin Sustrik working on a s= calability protocol to abstract the network by communication patterns I.e pu= b/sub, req/rep, etc. but these proposals breakdown because we route on the p= oint of attachment not on the node. You can side-step with a proxy pattern (= SCAFFOLD) if you are willing to add latency and indirection. =20 Tks G Sent from my iPhone On Aug 13, 2011, at 3:41 AM, "Patrick Frejborg" wrote:= > On Fri, Aug 12, 2011 at 5:01 PM, Gary Berger wrote: >> I would hope that we can find a more elegant approach to the problem of >> dealing with dynamic address learning and mobility instead of adding >> overlays. >> We have created the L2 scaling problem (which shows itself in mobility an= d >> control plane scalability) because of the fact that the data-link address= , >> IP Address and what we use to identify the host (I.e. The DNS name) are a= ll >> bound to the point of attachment. This created the desire for the "flat" >> network which as we know doesn't scale. It would be great if part of this= >> working group can look from the point of view that the Internet is an >> experiment and we never evolved to a clean implementation. Decoupling the= >> service (I.e. Where a socket call connects to) from the node address and t= he >> point of attachment will go a long way to fixing the scaling challenges. A= >> better description can be found in Saltzer, et al RFC-1498. Also, one o= f >> the chief OSI developers John Day has an interesting perspective on this >> here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf >=20 > Have also a look on the SCAFFOLD paper > ftp://ftp.cs.princeton.edu/reports/2010/885.pdf > That would remove the L2 scaling problem for online services >=20 > The approach requires a hierarchical addressing scheme, check out > RFC6306, which is focusing on scalability for the routing architecture > of the Internet. >=20 > The identifier mechanism described in SCAFFOLD can be applied with the > packet header described in RFC6306 though it is not discussed in > detail in the document. This combination of the two separate works > would introduce a session identifier (by MPTCP/SCTP) and service > identifier (by SCAFFOLD) that are decoupled from the addressing scheme > (locators). >=20 > Patrick From rees@umich.edu Fri Aug 12 04:09:09 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 44EA321F86EA for ; Fri, 12 Aug 2011 04:09:09 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.965 X-Spam-Level: X-Spam-Status: No, score=-1.965 tagged_above=-999 required=5 tests=[AWL=0.634, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hLopXqJYMO30 for ; Fri, 12 Aug 2011 04:09:08 -0700 (PDT) Received: from merit-proxy01.merit.edu (merit-proxy01.merit.edu [207.75.116.193]) by ietfa.amsl.com (Postfix) with ESMTP id CDC8421F86DE for ; Fri, 12 Aug 2011 04:09:08 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by merit-proxy01.merit.edu (Postfix) with ESMTP id 1BE1E2039875; Fri, 12 Aug 2011 07:09:45 -0400 (EDT) X-Virus-Scanned: amavisd-new at merit-proxy01.merit.edu Received: from merit-proxy01.merit.edu ([127.0.0.1]) by localhost (merit-proxy01.merit.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wF2Vs+gZdpRS; Fri, 12 Aug 2011 07:09:44 -0400 (EDT) Received: from merit.edu (74-126-0-171.static.123.net [74.126.0.171]) by merit-proxy01.merit.edu (Postfix) with ESMTPSA id A7F0C203986E; Fri, 12 Aug 2011 07:09:44 -0400 (EDT) Date: Fri, 12 Aug 2011 07:09:43 -0400 From: Jim Rees To: Warren Kumari Message-ID: <20110812110943.GA2159@merit.edu> References: <20110812005335.21740.71063.idtracker@ietfa.amsl.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Mailman-Approved-At: Sun, 14 Aug 2011 16:01:40 -0700 Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Aug 2011 11:09:09 -0000 Warren Kumari wrote: Hi there all, At two previous meetings (IETF in Quebec and NANOG in Denver) I threw a bit of a hissy fit about how VM mobility can be implemented without the need for L2 between the hosts. This reminds me a bit of Mobile IP (rfc5944). From pfrejborg@gmail.com Sun Aug 14 23:25:33 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C5FBF11E80AE for ; Sun, 14 Aug 2011 23:25:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.249 X-Spam-Level: X-Spam-Status: No, score=-3.249 tagged_above=-999 required=5 tests=[AWL=0.350, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OPkfYZV42lsZ for ; Sun, 14 Aug 2011 23:25:32 -0700 (PDT) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by ietfa.amsl.com (Postfix) with ESMTP id 9145C11E809B for ; Sun, 14 Aug 2011 23:25:32 -0700 (PDT) Received: by wwf5 with SMTP id 5so3076909wwf.13 for ; Sun, 14 Aug 2011 23:26:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=mvHnhVDJ0aYBOHl9t47o8pOCA12wFkNxzseEdsqN9ic=; b=hziVsdgFFr18sjHkaFLnywLdJwvO1majgCf39Hs1EJUDSO60G2Db2ykHqi1U+iqOtZ 1Z6Dw4VQCViEBgOB1/npeQfAqXYUpaDRQ4yzFyEMA+/UA0iepl9fEAjf0+gnRbU3efh7 rypdvI58EaU70ZD6grCCcMr1OEHIth252Gu/s= MIME-Version: 1.0 Received: by 10.227.39.154 with SMTP id g26mr3179549wbe.37.1313389575902; Sun, 14 Aug 2011 23:26:15 -0700 (PDT) Received: by 10.227.38.206 with HTTP; Sun, 14 Aug 2011 23:26:15 -0700 (PDT) In-Reply-To: <7CDCDBB6-0E31-4A35-BDD5-492FD6B731AC@cisco.com> References: <7C4DFCE962635144B8FAE8CA11D0BF1E0589609E87@MX14A.corp.emc.com> <7CDCDBB6-0E31-4A35-BDD5-492FD6B731AC@cisco.com> Date: Mon, 15 Aug 2011 09:26:15 +0300 Message-ID: From: Patrick Frejborg To: "Gary Berger (gaberger)" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2011 06:25:33 -0000 Hi Gary, I'm interested to hear what you think is missing from the SCAFFOLD paper, think there are some minor issues but they can be improved. I also believe that routing always has to be applied against attachment points, i.e. locators are used for routing. Thus you to do a binding of locators and identifiers (nodes) somewhere, how the binding (of Loc/ID to produce the services) is designed is the real challenge - you could end up with scalability, privacy and latency issues - some additional latency is imposed but the later the binding is applied the less latency there is going to be. IMHO, if you do routing based on identifiers then there is really no decoupling of locators and identifiers - of course this depends on one's definition of Loc/ID split. Patrick On Sat, Aug 13, 2011 at 4:27 PM, Gary Berger (gaberger) wrote: > Hi Patrick, > > Yes I have read SCAFFOLD and I like the approach. Fundamentally you eithe= r solve lookup through exhaustive search or through hinting and a hierarcha= l address is critical to this. It doesn't solve the problem completely but = it's a good vision of where we can reformulate our view of binding layers. = =A0From an appdev point of view being able to target an abstract topology s= eems incredibly beneficial. The topology can be geographically bounded, seg= mented by service or customer. There is a group lead by Martin Sustrik =A0w= orking on a scalability protocol to abstract the network by communication p= atterns I.e pub/sub, req/rep, etc. but these proposals breakdown because we= route on the point of attachment not on the node. You can side-step with a= proxy =A0pattern (SCAFFOLD) if you are willing to add latency and indirect= ion. > > Tks > > G > > Sent from my iPhone > > On Aug 13, 2011, at 3:41 AM, "Patrick Frejborg" wro= te: > >> On Fri, Aug 12, 2011 at 5:01 PM, Gary Berger wrote: >>> I would hope that we can find a more elegant approach to the problem of >>> dealing with dynamic address learning and mobility instead of adding >>> overlays. >>> We have created the L2 scaling problem (which shows itself in mobility = and >>> control plane scalability) because of the fact that the data-link addre= ss, >>> IP Address and what we use to identify the host (I.e. The DNS name) are= all >>> bound to the point of attachment. This created the desire for the "flat= " >>> network which as we know doesn't scale. It would be great if part of th= is >>> working group can look from the point of view that the Internet is an >>> experiment and we never evolved to a clean implementation. Decoupling t= he >>> service (I.e. Where a socket call connects to) from the node address an= d the >>> point of attachment will go a long way to fixing the scaling challenges= . A >>> better description =A0can be found in Saltzer, et al =A0RFC-1498. Also,= one of >>> the chief OSI developers John Day has an interesting perspective on thi= s >>> here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf >> >> Have also a look on the SCAFFOLD paper >> ftp://ftp.cs.princeton.edu/reports/2010/885.pdf >> That would remove the L2 scaling problem for online services >> >> The approach requires a hierarchical addressing scheme, check out >> RFC6306, which is focusing on scalability for the routing architecture >> of the Internet. >> >> The identifier mechanism described in SCAFFOLD can be applied with the >> packet header described in RFC6306 though it is not discussed in >> detail in the document. This combination of the two separate works >> would introduce a session identifier (by MPTCP/SCTP) and service >> identifier (by SCAFFOLD) that are decoupled from the addressing scheme >> (locators). >> >> Patrick > From gaberger@cisco.com Tue Aug 16 06:25:54 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 930E721F8AF3 for ; Tue, 16 Aug 2011 06:25:54 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.102 X-Spam-Level: X-Spam-Status: No, score=-1.102 tagged_above=-999 required=5 tests=[AWL=0.100, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d8DcHhXoyfNb for ; Tue, 16 Aug 2011 06:25:53 -0700 (PDT) Received: from rcdn-iport-3.cisco.com (rcdn-iport-3.cisco.com [173.37.86.74]) by ietfa.amsl.com (Postfix) with ESMTP id 7AA4F21F8ADC for ; Tue, 16 Aug 2011 06:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=55000; q=dns/txt; s=iport; t=1313501201; x=1314710801; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=JXhpkcasd2ZCMIXIP1vbRsbwFEgKEpZE/Y1kYiN2Yhk=; b=kYspgU55WYEhAhwgKfkOsszN8t8iOsga3W5evFrjKmK4pHzCEM5HnCNV do13gdp0PNos91KU3himih7adoWnDoHVRHsXdPhgjDF2dOZ3VouBTIHZF cXZ431x5hhPDHVmuUpm4mqgZy346KxTUGDfHLUHfMeQ7qXW00im+HC8oj U=; X-Files: 7B78AD1C-E150-412D-A64A-DC1A9E885956.png : 25130 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ag0FANVvSk6tJV2b/2dsb2JhbAA3AQmCTYNDmg8BhyNud4FAAQEBAQEBAQUBDAEqMAoCBQ4IDgQGBQEBASgVAQkFIw4GDgQBBhyHTgSaKgGfNoMnAYMgBI4PhQOFDIRqAYca X-IronPort-AV: E=Sophos;i="4.67,380,1309737600"; d="png'150?scan'150,208,217,150";a="13550108" Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by rcdn-iport-3.cisco.com with ESMTP; 16 Aug 2011 13:26:37 +0000 Received: from [10.82.210.232] (rtp-vpn4-744.cisco.com [10.82.210.232]) by rcdn-core-4.cisco.com (8.14.3/8.14.3) with ESMTP id p7GDQXs4015998; Tue, 16 Aug 2011 13:26:35 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Tue, 16 Aug 2011 09:24:34 -0400 From: Gary Berger To: Patrick Frejborg Message-ID: Thread-Topic: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt In-Reply-To: Mime-version: 1.0 Content-type: multipart/mixed; boundary="B_3396331596_2372406" Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Aug 2011 13:25:54 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3396331596_2372406 Content-type: multipart/alternative; boundary="B_3396331596_2363215" --B_3396331596_2363215 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Patrick, Where to start.. Ugh.. Let me just say I think the approach Scaffold takes is innovative especially in the light of bringing together applications, services and networking.. Certainly we have learned that topology is a lot more complicated than just a physical graph and moving towards decoupling the physical topology will go along way with providing scalable services. As per SCAFFOLD, I have to say I have not thought through exhaustively the design but I can give you some things that just give me pause. * Some of the implementation issues are scary.. I.e. Changing the data plane, control plane, the network stack and programming model.. > * These are HUGE changes, not necessarily impossible but definitely a > challenge > * Kudos for the innovative thinking. I think there is growing practice in > looking at the network stack especially from a messaging point of view instead > of an opaque streaming interface[1] but it still is very scary and if we are > going to go to this extent maybe we should look at the entire protocol stack.. * Pushing the bindings into the server are creative and maybe a first attempt to look at these alternatives while they mature possibly migrating into the dataplane of the network. Operationally people are going to get a bit burned without the proper OAM tools. * I do like that you have added the elusive application service. Having a proper binding for this is important as well as the appropriate lookup services. Ip's "well-known" ports was a hack because we had no such directory. You do answer Saltzers first objective: 1. A given service may run at one or more nodes, and may need to move from one node to another without losing its identity as a service.[3] * But again we are missing the Node Address. Saltzer missed this, if you believe the conjecture it was because multi-homed node systems were not envisioned at this point. The IP Address is naming the subnetwork attachment point, this is ultimately the scaling problem..[2] What is the "identity as a node" he is talking about? An identity needs to be unique within the scope of the layer (we don't have this with MAC or IP today). 2. A given node may be connected to one or more network attachment points, and may need to move from one attachment point to another without losing its identity as a node.[3] * In the paper it doesn't seem you have the concept of an identity in fact under VM migration you talk about cycling the network interface to get a new Scaffold address.. In which case where is the identity of the node? * There is also overloading the hostAddr name with ASAddr and sockID in the example implementation.. This in practice may be challenging but I assume this is just an experimental viewpoint. "IMHO, if you do routing based on identifiers then there is really no decoupling of locators and identifiers - of course this depends on one's definition of Loc/ID split" I don't think this has to do with loc/id split, that in fact maybe a false path[2].. But you do routing on a "name/address" relative to the layer. Mux/Demux is just composition/decomposition in the network domain. If we take the perspective that devices are multi-homed as the canonical model; examples being a device moving from AP to AP, Wi-fi to 3G/4G or DC to DC. Scaffold goes to a lot of effort to make this seamless but the complexity should be an indicator something is wrong. I put together a short example here: http://www.slideshare.net/gaberger/scaffold-8866328 which describes Saltzer's model as well as Scaffold. I also took a stab at what it would actually look like if you incorporated a Node Identifier in the address architecture. I am not saying that this is an easy topic, its been 40 years in the making, the question is where do we concentrate our efforts in properly designing the address architecture and managing the layer bindings.. The latter as a charter in this group "the what", the former as a necessity in understanding "the why". 1. http://www.250bpm.com/concepts 2. http://pouzin.pnanetworks.com/images/LocIDSplit090309.pdf 3. http://tools.ietf.org/html/rfc1498 On 8/15/11 2:26 AM, "Patrick Frejborg" wrote: > Hi Gary, > > I'm interested to hear what you think is missing from the SCAFFOLD > paper, think there are some minor issues but they can be improved. > > I also believe that routing always has to be applied against > attachment points, i.e. locators are used for routing. Thus you to do > a binding of locators and identifiers (nodes) somewhere, how the > binding (of Loc/ID to produce the services) is designed is the real > challenge - you could end up with scalability, privacy and latency > issues - some additional latency is imposed but the later the binding > is applied the less latency there is going to be. > > IMHO, if you do routing based on identifiers then there is really no > decoupling of locators and identifiers - of course this depends on > one's definition of Loc/ID split. > > Patrick > > On Sat, Aug 13, 2011 at 4:27 PM, Gary Berger (gaberger) > wrote: >> Hi Patrick, >> >> Yes I have read SCAFFOLD and I like the approach. Fundamentally you either >> solve lookup through exhaustive search or through hinting and a hierarchal >> address is critical to this. It doesn't solve the problem completely but it's >> a good vision of where we can reformulate our view of binding layers. From >> an appdev point of view being able to target an abstract topology seems >> incredibly beneficial. The topology can be geographically bounded, segmented >> by service or customer. There is a group lead by Martin Sustrik working on a >> scalability protocol to abstract the network by communication patterns I.e >> pub/sub, req/rep, etc. but these proposals breakdown because we route on the >> point of attachment not on the node. You can side-step with a proxy pattern >> (SCAFFOLD) if you are willing to add latency and indirection. >> >> Tks >> >> G >> >> Sent from my iPhone >> >> On Aug 13, 2011, at 3:41 AM, "Patrick Frejborg" wrote: >> >>> On Fri, Aug 12, 2011 at 5:01 PM, Gary Berger wrote: >>>> I would hope that we can find a more elegant approach to the problem of >>>> dealing with dynamic address learning and mobility instead of adding >>>> overlays. >>>> We have created the L2 scaling problem (which shows itself in mobility and >>>> control plane scalability) because of the fact that the data-link address, >>>> IP Address and what we use to identify the host (I.e. The DNS name) are >>>> all >>>> bound to the point of attachment. This created the desire for the "flat" >>>> network which as we know doesn't scale. It would be great if part of this >>>> working group can look from the point of view that the Internet is an >>>> experiment and we never evolved to a clean implementation. Decoupling the >>>> service (I.e. Where a socket call connects to) from the node address and >>>> the >>>> point of attachment will go a long way to fixing the scaling challenges. A >>>> better description can be found in Saltzer, et al RFC-1498. Also, one of >>>> the chief OSI developers John Day has an interesting perspective on this >>>> here http://pouzin.pnanetworks.com/images/KoreaNamingFund100218.pdf >>> >>> Have also a look on the SCAFFOLD paper >>> ftp://ftp.cs.princeton.edu/reports/2010/885.pdf >>> That would remove the L2 scaling problem for online services >>> >>> The approach requires a hierarchical addressing scheme, check out >>> RFC6306, which is focusing on scalability for the routing architecture >>> of the Internet. >>> >>> The identifier mechanism described in SCAFFOLD can be applied with the >>> packet header described in RFC6306 though it is not discussed in >>> detail in the document. This combination of the two separate works >>> would introduce a session identifier (by MPTCP/SCTP) and service >>> identifier (by SCAFFOLD) that are decoupled from the addressing scheme >>> (locators). >>> >>> Patrick >> > --B_3396331596_2363215 Content-type: text/html; charset="US-ASCII" Content-transfer-encoding: quoted-printable
    Patrick,

    =
    Where to start.. Ugh.. Let me just say I think the approach Scaff= old takes is innovative especially in the light of bringing together applica= tions, services and networking.. Certainly we have learned that topology is = a lot more complicated than just a physical graph and moving towards decoupl= ing the physical topology will go along way with providing scalable services= .

    As per SCAFFOLD, I have to say I have not thought= through exhaustively the design but I can give you some things that just gi= ve me pause.
    • Some of the implementation issues are scary.. I.e.= Changing the data plane, control plane, the network stack and programming m= odel.. 
      • These are HUGE changes, not necessarily impossible = but definitely a challenge
      • Kudos for the innovative thinkin= g. I think there is growing practice in looking at the network stack especia= lly from a messaging point of view instead of an opaque streaming interface[= 1] but it still is very scary and if we are going to go to this extent maybe= we should look at the entire protocol stack..
    • Pushing the bind= ings into the server are creative and maybe a first attempt to look at these= alternatives while they mature possibly migrating into the dataplane of the= network. Operationally people are going to get a bit burned without the pro= per OAM tools.
    •  I do like that you have added the elusive appli= cation service. Having a proper binding for this is important as well as the= appropriate lookup services. Ip's "well-known" ports was a hack because we = had no such directory. You do answer Saltzers first objective:
    1. A given service may run at one or more nodes, and= may need to move from one node to another wit= hout losing its identity as a service.[3]
    • But again we a= re missing the Node Address. Saltzer missed this, if you believe the conject= ure it was because multi-homed node systems were not envisioned at this poin= t. The IP Address is naming the subnetwork attachment point, this is ultimat= ely the scaling problem..[2] What is the "identity as a node" he is talking = about? An identity needs to be unique within the scope of the layer (we don'= t have this with MAC or IP today).
    2. A= given node may be connected to one or more network attachment points, and m= ay need to move from one attachment point to another without losing its iden= tity as a node.[3]
    • In the paper it doesn't seem you have the =
concept of an identity in fact under VM migration you talk about cycling the=
 network interface to get a new Scaffold address.. In which case where is th=
e identity of the node?
    • Ther= e is also overloading the hostAddr name with ASAddr and sockID in the exampl= e implementation.. This in practice may be challenging but I assume this is = just an experimental viewpoint.

    "IMHO, if y= ou do routing based on identifiers then there is really no decoupling o= f locators and identifiers - of course this depends on one's definition= of Loc/ID split"

    I don't think this has to do with= loc/id split, that in fact maybe a false path[2].. But you do routing on a = "name/address" relative to the layer. Mux/Demux is just composition/decompos= ition in the network domain.

    If we take the perspec= tive that devices are multi-homed as the canonical model;  examples bei= ng  a device moving from AP to AP, Wi-fi to 3G/4G or DC to DC. Scaffold= goes to a lot of effort to make this seamless but the complexity should be = an indicator something is wrong.

    I put together a s= hort example here: http://www.slideshare.net/gaberger/scaffold-8866328 which= describes Saltzer's model as well as Scaffold. I also took a stab at what i= t would actually look like if you incorporated a Node Identifier in the addr= ess architecture. I am not saying that this is an easy topic, its been 40 ye= ars in the making, the question is where do we concentrate our efforts in pr= operly designing the address architecture and managing the layer bindings.. = The latter as a charter in this group "the what", the former as a necessity = in understanding "the why".


    1. http://www.250bpm.com/concepts
      2. <= li>http:= //pouzin.pnanetworks.com/images/LocIDSplit090309.pdf
    2. http://tools.ietf.org/html/rfc1498


    On 8/15/11 2:26 AM, "Patrick = Frejborg" <pfrejborg@gmail.com&g= t; wrote:

    Hi Gary,

    I'm interested to hear what you thin= k is missing from the SCAFFOLD
    paper, think there are some minor i= ssues but they can be improved.

    I also believe that= routing always has to be applied against
    attachment points, i.e. = locators are used for routing. Thus you to do
    a binding of locator= s and identifiers (nodes) somewhere, how the
    binding (of Loc/ID to= produce the services) is designed is the real
    challenge - you cou= ld end up with scalability, privacy and latency
    issues - some addi= tional latency is imposed but the later the binding
    is applied the= less latency there is going to be.

    IMHO, if you do= routing based on identifiers then there is really no
    decoupling o= f locators and identifiers - of course this depends on
    one's defin= ition of Loc/ID split.

    Patrick

    =
    On Sat, Aug 13, 2011 at 4:27 PM, Gary Berger (gaberger)
    <<= a href=3D"mailto:gaberger@cisco.com">gaberger@cisco.com> wrote:
    Hi Patrick,
    Yes I have read SCAFFOLD and I like the approach. Fundamentally= you either solve lookup through exhaustive search or through hinting and a = hierarchal address is critical to this. It doesn't solve the problem complet= ely but it's a good vision of where we can reformulate our view of binding l= ayers.  From an appdev point of view being able to target an abstract t= opology seems incredibly beneficial. The topology can be geographically boun= ded, segmented by service or customer. There is a group lead by Martin Sustr= ik  working on a scalability protocol to abstract the network by commun= ication patterns I.e pub/sub, req/rep, etc. but these proposals breakdown be= cause we route on the point of attachment not on the node. You can side-step= with a proxy  pattern (SCAFFOLD) if you are willing to add latency and= indirection.

    Tks

    G

    Sent from my iPhone

    On Aug 1= 3, 2011, at 3:41 AM, "Patrick Frejborg" <pfrejborg@gmail.com> wrote:

    On Fri, Aug 12, 2011 at 5:01 PM, Gary B= erger <gaberger@cisco.com> wro= te:
    I would hope t= hat we can find a more elegant approach to the problem of
    dealing= with dynamic address learning and mobility instead of adding
    ove= rlays.
    We have created the L2 scaling problem (which shows itself= in mobility and
    control plane scalability) because of the fact t= hat the data-link address,
    IP Address and what we use to identify= the host (I.e. The DNS name) are all
    bound to the point of attac= hment. This created the desire for the "flat"
    network which as we= know doesn't scale. It would be great if part of this
    working gr= oup can look from the point of view that the Internet is an
    exper= iment and we never evolved to a clean implementation. Decoupling the
    service (I.e. Where a socket call connects to) from the node address and= the
    point of attachment will go a long way to fixing the scaling= challenges. A
    better description  can be found in Saltzer, = et al  RFC-1498. Also, one of
    the chief OSI developers John = Day has an interesting perspective on this

    Have also a look on the SCAFFOLD paper
    That would remove the L2 scaling problem for on= line services

    The approach requires a hierarchical= addressing scheme, check out
    RFC6306, which is focusing on scala= bility for the routing architecture
    of the Internet.
    The identifier mechanism described in SCAFFOLD can be applied = with the
    packet header described in RFC6306 though it is not disc= ussed in
    detail in the document. This combination of the two sepa= rate works
    would introduce a session identifier (by MPTCP/SCTP) a= nd service
    identifier (by SCAFFOLD) that are decoupled from the a= ddressing scheme
    (locators).

    Patrick


    --B_3396331596_2363215-- --B_3396331596_2372406 Content-type: image/png; name="7B78AD1C-E150-412D-A64A-DC1A9E885956.png" Content-ID: <7B78AD1C-E150-412D-A64A-DC1A9E885956> Content-disposition: inline; filename="7B78AD1C-E150-412D-A64A-DC1A9E885956.png" Content-transfer-encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAW4AAABrCAYAAABXGGiIAAAXU2lDQ1BJQ0MgUHJvZmlsZQAA eAHVWXk8Vd3X3+fcEfcarnkm8zwPl8zzPI+pXPNM1xRlSFKokCGFFBIpGoWEDCnJlChFikKp NJjJe+h5nt/v/fze97/3n3d/Pmef71lr7bXXOWudvc9aBwDOeUpERAjMCEBoWBTV3kRfwNXN XQD3GuAAAeCBCKCneEdG6NnaWoL/tS2NAGib+VxmW9f/KvY/M5h8fCO9AYBsEbaXT6R3KILv AADre0dQowBArSD0gdioCASjHyOYhYoYiOA329j/D17Yxl47GIPekXG0NwAAwwEAnkChUP0B IAojdIEYb39ED9EQACwpzCcwDABmVwRrewdQfADgLERkpENDw7dxJ4LFvf5Nj/+/YQrF6x+d FIr/P/jPvSAjkYkNAyMjQihxOxf/l11oSDTyvHYaCekJYSHW275hQ45ZH4qhBXLmQY7fESE7 PkNkIC7fMCcHhLaNpcO8rG3+wtp+VGN7BCNjIduIKP1tjDwzyC8iytbxL3pifICBNYIJCD3P N9Lobz0Xgyjm2z6jR+i3qNH2TggWRnBbZIyDEYKRiIKm4wMcXf6S+eXja/gXHYb9Ao3N/sjA pMAos+25WBCf7woOt9i2AZkLVgUWIAT4gmhARfowIAMsgQEw/KuXAX6AgnBiEF4kCAYfERyK jAhHxoQjWOAvOYP/oBjvjPNHxv13jQLAG5GN/mfOP7MJIHP+rTMQ+CD4bzoFmWObt21d5P7A lH/N+bfEtr4da+Rr5OfkN/62CS2KVkSroPXRWmhtNBkIoNnQXEAGrYxWR+uhddCaCI8MjME0 otn/bxu39Yfe8ospDI/TcA5AuNv37vU3FzjvSAf+c/0fFoDAvvl7839bAECU70HkPQDAIDwi jhroHxAloIe8ub7SAmZh3rLSAoryCgrb7P83bXvN+mPsT/udtQhi6/8XjYKsSeqKANDq/4sW jrwjtflI6J/7F00UiXNOMgC37L2jqTF/9KG3TxhACxiQCOUEfEAIiCPPWRGoAk2gC4yAObAB jsAN7EPiJwCJQSqIBYfBEZAGMkE2yAfnQSkoB1XgOrgF7oFm8BA8Ak/BAHgBXoNJ8AF8Bgtg CaxDEISDiBAzxAnxQyKQFKQIqUPakBFkCdlDbpAn5A+FQdHQYegolAmdgc5Dl6Bq6CbUCD2E nkCD0CvoHTQH/YDWYBRMgFlgXlgUloPVYT3YAnaE98L+8AE4Hk6FT8OFcBl8Da6HH8JP4Rfw JPwZXkQBFB2KDSWIkkGpowxQNih3lB+KikpEZaAKUGWoWlQTqhv1HDWJmketorFoZrQAWgaJ U1O0E9obfQCdiD6JPo+uQtejO9HP0e/QC+jfGCKGByOF0cCYYVwx/phYTBqmAFOJuYvpwrzA fMAsYbFYNqwYVg1rinXDBmEPYU9iS7B12DbsIHYKu4jD4ThxUjgtnA2OgovCpeHO4a7hWnFD uA+4FTwdnh+viDfGu+PD8Cn4AvxVfAt+CD+DX6dhpBGh0aCxofGhiaPJoqmgaaLpp/lAs07L RCtGq0XrSBtEe4S2kLaWtov2De1POjq6XXRkOju6QLpkukK6G3SP6d7RrRJIBEmCAcGDEE04 TbhCaCO8IvwkEomiRF2iOzGKeJpYTewgThBX6JnpZenN6H3ok+iL6Ovph+i/MtAwiDDoMexj iGcoYLjN0M8wz0jDKMpowEhhTGQsYmxkHGVcZGJmUmCyYQplOsl0lekJ0ywJRxIlGZF8SKmk clIHaYoZxSzEbMDszXyUuYK5i/kDC5ZFjMWMJYglk+U6Sx/LAiuJVZnVmfUgaxHrA9ZJNhSb KJsZWwhbFtstthG2NXZedj12X/Z09lr2IfZlDm4OXQ5fjgyOOo4XHGucApxGnMGcOZz3OMe5 0FySXHZcsVwXuLq45rlZuDW5vbkzuG9xj/HAPJI89jyHeMp5enkWefl4TXgjeM/xdvDO87Hx 6fIF8eXxtfDN8TPza/MH8ufxt/J/EmAV0BMIESgU6BRYEOQRNBWMFrwk2Ce4vktsl9OulF11 u8aFaIXUhfyE8oTahRaE+YWthA8L1wiPidCIqIsEiJwV6RZZFhUTdRE9LnpPdFaMQ8xMLF6s RuyNOFFcR/yAeJn4sARWQl0iWKJEYkASllSRDJAskuyXgqVUpQKlSqQGpTHSZOkw6TLpURmC jJ5MjEyNzDtZNllL2RTZe7Jf5YTl3OVy5LrlfsuryIfIV8i/ViApmCukKDQp/FCUVPRWLFIc ViIqGSslKTUofVeWUvZVvqD8UoVZxUrluEq7yqaqmipVtVZ1Tk1YzVOtWG1UnUXdVv2k+mMy hqxPTiI3k1c1VDWiNG5pfNOU0QzWvKo5u1tst+/uit1TWru0KFqXtCa1BbQ9tS9qT+oI6lB0 ynTe6wrp+uhW6s7oSegF6V3T+6ovr0/Vv6u/bKBhkGDQZogyNDHMMOwzIhk5GZ03mjDeZexv XGO8YKJicsikzRRjamGaYzpqxmvmbVZttmCuZp5g3mlBsHCwOG/x3lLSkmrZZAVbmVvlWr2x FrEOs75nA2zMbHJtxm3FbA/Y3rfD2tnaFdl9tFewP2zf7cDssN/hqsOSo75jluNrJ3GnaKd2 ZwZnD+dq52UXQ5czLpOucq4Jrk/duNwC3Rrcce7O7pXui3uM9uTv+eCh4pHmMbJXbO/BvU/2 ce0L2fdgP8N+yv7bnhhPF8+rnhsUG0oZZdHLzKvYa8HbwPus92cfXZ88nzlfLd8zvjN+Wn5n /Gb9tfxz/ecCdAIKAuYDDQLPB34PMg0qDVoOtgm+ErwV4hJSF4oP9QxtDCOFBYd1hvOFHwwf jJCKSIuYPKBxIP/AAtWCWhkJRe6NbIhiQT4Oe6PFo49Fv4vRjimKWYl1jr19kOlg2MHeOMm4 9LiZeOP4y4fQh7wPtR8WPHzk8LsEvYRLiVCiV2J7klBSatKHZJPkqiO0R4KPPEuRTzmT8uuo y9GmVN7U5NSpYybHatLo06hpo8c1j5eeQJ8IPNGXrpR+Lv13hk9GT6Z8ZkHmxknvkz2nFE4V nto67Xe6L0s160I2NjsseyRHJ6fqDNOZ+DNTuVa59XkCeRl5v/L35z8pUC4oPUt7NvrsZKFl YcM54XPZ5zbOB5x/UaRfVFfMU5xevFziUzJ0QfdCbSlvaWbp2sXAiy8vmVyqLxMtKyjHlseU f6xwrui+rH65upKrMrNy80rYlckq+6rOarXq6qs8V7Nq4JromrlrHtcGrhteb6iVqb1Ux1aX eQPciL7x6abnzZFbFrfab6vfrr0jcqf4LvPdjHqoPq5+4V7AvckGt4bBRvPG9ibNprv3Ze9f aRZsLnrA+iCrhbYltWWrNb51sS2ibf6h/8Op9v3trztcO4Y77Tr7uiy6Hj8yftTRrdfd+ljr cfMTjSeNPeo9956qPq3vVem9+0zl2d0+1b76frX+hgHyQNPg7sGWIZ2hh88Nnz8aNht++sL6 xeCI08jLUY/RyZc+L2dfhbz6PhYztv46+Q3mTcY443jBBM9E2VuJt3WTqpMP3hm+633v8P71 lPfU5+nI6Y0PqR+JHwtm+GeqZxVnm+eM5wY+7fn04XPE5/X5tC9MX4q/in+98033W++C68KH 79TvWz9O/uT8eeWX8q/2RdvFiaXQpfXljBXOlapV9dXuNZe1mfXYDdxG4abEZtNvi99vtkK3 tiIoVMrOtwAK6WE/PwB+XEFyCDckdxhAvino/+QUOxJIugIhMgh2hmShz3An6ijaAaOLFcNx 4Tlo+Gm16KwJwcRs+kaGeSYZki9zOcsUmyR7HEcrFwO3C08F70/+3QKpgs+EmITtRU6JPhUH EkqSflJnpXtkluXE5e0UkhVrlF6owKoKanvVM8j1Gu92E7XUtT110nVv6r0xwBuqGnkbZ5s0 mE6YQxbCliZWQdZZNndsX9qtOLA5KjnZOIe6nHKtdXvq/m7Pgsfy3vX9wJOWwukl463nY++7 38/XnxLgELg7SCAYCp4MaQ29GHY0PCDC9oA6VSASH/ktaiS6JaYqNvdgYlxIvNshs8NaCWqJ qknkZL0jFikuR31To44dS8s7XnHidnpbRm/myMm3p2ZOf8n6kb2Ys3RmMXcxb60AfZa1UPqc yXnvoqTiwpLaC62lTy8OXxormyyfq/hVibrCWiVZrX/Voyb2Wt71W7WDdd9vMt1Suu1wJ/Ju dn31vaaGh40dTW337zfffVDXUt1a3lbyML89o+NwZ1CXwyPVbo7u1ceTT/p7Hj3t6H34rLmv rr9wIHLQYIg49Px50bDfC5URzMjoaNXLmFe6Y9ixbiS+VN7MjOdMaE5MvT01qTn5+V3pe/sp 1FTdtNP06oe8j9IfW2fsZ6Znj83JzU1/qvocNq80v/il7qv3N6ZvdxdsFz5+P/yD/cejn1m/ whYpS35IHE2vdW3Kbm3t+F8IugEHoRRRs+ibmGSsK04LL0MjRitGt4sgT9Sgt2PwZkxkKiW1 MM+xMrKps1M4TnDe4ZrgoeNV4tvDnyxwSbB112uhRRE6UX4xFXEzCU/JOKlc6ZsyvbKz8mgF QcXdSu7KUSqZqhVqjerPyO81fu3GanFrK+hY6YboZenfMBgw/GKMN+E1VTQzMney8LYMszpo nWhz1PaYXZr9CYcMx5NOGc6pLnGuAW6O7oZ7dDyM97rvi92f73mD0u7V493lc9e32O+Qv0uA fCAhcD5oILgppDq0KCwrPCWCesCDqhvJH7ke9SL6ekxarNdBozj5eOFDvIc5E1gTGZOwSUvJ 74/0pNw8mp8ae2xvmvlxwxOW6ZSMI5mXTz46NXH6a9Zi9nLO4pmfuQt5X/LnC76eXTnHeJ5c FFZcWdJ3Yap07uKHS2/LXpUPVjy+3FLZfKWn6stVwZq914qvv6pjuWF98wSyeq3ela33uVfU MNSEua/cvP/BsZbK1ua2lodX27M7Ejpju5IfZXWXPC5/cqHn9NPoXodnMn3ovrH+WwOZg0FD ds+Nho1e2I14jUa/TH11fCzhtd8bg3Gu8fmJxrfHJ13fybzHv/841TFd8uHAR90ZwszwbPlc 0qfAzz7zAV9Cv0Z8i1iI+E79EfMz7lfsYuCSyTLD8u0Vo5Wnq+6rX9YGNgibYzv+lwKdkAX0 EvZFYVFZaCl0PyYeK4edw13GB9DI0azS9tCVEmKJ9vSKDPQMS4yvmNpI1cy5LAms/mz27Foc EpysnBtcs9xDPC28tXzl/EUCBYJ5u7KE0oRjRCiiRmICYivivRKlkpFSptKCMrDMnOyo3GP5 JoWrioVKycqeKmRVrGq/Wr66K5mT/EqjRNNnt6IWVmtCu14nSzdAz1Bf1IDREBj+NJoxHjG5 b1pg5msuYj5pUWhpY4Wz6rA+amNmy2H7ya7FPtchwFHTieg04Xzd5bCruRur21v3qj3hyP6/ uvfBvuT9Bp54z0FKsVew924fgs+Y7xW/A/7q/hsBrYHJQbrBILgt5EioQRg6rCv8WIRexMqB GqobsmdXR9lE/YoujNkdMxGbfJD34IM4z3i2+LFDNYePJrgmiicuJXUk5x7xTzE8KpnKcYwu DaT9Oj514ll6XcbJTMpJ5VO4U2Onb2RlZAfnmJwhnXmUuyd3Pi8+X69A/+yJc/jzGUXTJZwX FEvJF8mXVMrkysUrBC9zVjJdoa2iqWZAIknrmuf147XX657f2Lglftv9zpm7g/dYGtwai5tG mzEPJFpMWr3akh5eaG/peNu59Uiw2+Cx/5OTPTefjvRu9kn07xk4OzjxXHH41Iuvow4vG8cE X+ePy72lfxc7nTkb98X6x9Kq3bb//9SWtvcErCoAuUie6XwKOeYAyLkHgOh9ANhpAbAlAuBI BvDxWgCb1AIo+Ng/+wcE0Ej1jQGpzPADCaCMZJqWwB0EgoMgHckor4EWMITUPDYgEiQB6SL5 YSR0CskHu6ApGIIFYX3YBz6OZHlD8BpKCGWFikdVoUbReLQGOhRdjn6FIWEskIysAwthdbHJ 2HYcBmeOy8a9xAviQ/CNNDgaF5oqmjVaK9pLtMt01nRVBDTBi9BBFCGmE7/SO9I3I5lODiNg PMA4zeTG1E8yJj1gVmeuZ9Fg6WC1Z51ii2bHshdwiHI0cFpzznKd4FbgnuIp5fXik+Jb4X8k kC/os0tZCCv0Wvi2SJZoiJiFuJQEUWJB8oXUfekLMomyHnJkeRb5BYVnileV0pUDVMxVZdVY 1bbUv5AnNIY0e3Z3aXVqd+v06Y7pzeovGQIjLLLO4U3xZjTmBAsWS0ErZWtrmzDbPLtm+w+O RCdlZzeXBNeLbp3uMx50e+X3Oe8/7FlB6fNa8RH2dfA75t8csBZkEHwuZDXMO3zogDG1OUo5 ui5W5uDN+N2HBhLCk3iSR1LyUi2PLR3PS5fO6Drpe5o1623Os9zx/K1CgfPkYssL+y/GlV2s GLsiU33xmnzt5M1Ld/bdo2usbd7bKtXO32X8uKyX0C8+uDScMyr+avDNhbdn3w999Jxb/UL6 du0H+CW/RF7eWs1Ya1gf3ri/Wf47YkttZ/2AdmoOJMANRJFagw6wAh5IbSER5IAK0Aj6kbrB JsQGyUHmkB9SEShDqgDvYTQsBlvCVPg83AF/Q/GgLFCHUXWoaTQX2h6die7CQBgtzCHMfcwG Vgd7FPsEx4hzw13G/cDr4XPxH2k0aXJp5mmNEZ9v0LnS3UEyYSphmEgmXqSnoz9IP8PgxtDH aMzYxqTN1EoyIPUwOzCPI5npGmsWmyTbU/YDHGwc9Zx2nB+54riJ3BU8ujzTvDl85vz0/OMC twVP7woU0hfmEP4s8kA0W8xPXF9CRJIkhZfGyOBl6eVI8kwKeIVVxVmlUeUelYeqD9V61F+T f2jS75bXstMO1InSpeoF6LsamBiSjZSN1U1MTPebJZpfsui2XLDmtjGyDUb2tDyHs475TnnO F11aXb+7q+xJ9ni2j29/lGe/l5C3n0++712/Pv/pgPUgtmClEMfQmLDz4W0Rn6jskcZRMdFX YsYOMsZZxWcdepkgmpiQNHXE/yhjak9a1Als+vFM9MkTp7mzOnJScl3zDc5qntMs0iwhl0pc Qpc9qoip5L7yoNqrhvXaeG3Xjf5bi3cV7h1ufNrM0GLYRm2v7Jzr1n9yq1ehr3hgfOjX8PeR mZdTY7Nvfr2F3tFOsXwQnjGdK5hX+5bxs3I5ZLVvPXWjY/PX79Ud/8PI28+EVJtkgDawQ2qd CaAA3AC94BNEg9SGrCAqVAi1QZ9gNtgQjoIr4TEUE8oMlYpqQ20ilZl4dBN6A6OHycCMYiWw R7DjOG1cGR6PD8cP05BpSmhhpBbygs6Q7j6BTHhItCV+pE9hEGRoY/RgXGLKJsmQnjGHsRBZ qlj1Wd+wxbHzsfdxnOb04tLnluRh4VnnHedr4D8jECpouUteiEMYK7wq8l30m9hP8U1Jeilh aV0ZT9lkuRL5BoXnij+VuVTMVFPUOsgEDQ/NG1o45Fu1RW+Xfq4hm1GtibsZk/mg5XnrcFsn e0WHMSd3515XU7fne/w8VvYd9YQoEV4vfNR8i/1pAo4E0QaXh1qFg4h71PAovuiO2Og4n0Nf EyuS446MpGykwsfwaYzHlU5Epg9nOp2cO30iWzbnVe6JfM2C74XV5/cV05ZcKVW7+KBMp7zt smFlT5Vt9XCN47WBWuO6xpvit87ewd9NqN9oSG8SvT/wIKVVtW2uvbjT5hG6+/6TyKdSvdN9 FwZch1ieD73IGjV/uTV27Y3N+Ozb6MnN9ynTqA8pM/Ds0U/oz0nzX78af4tbKPl+6kf0T8Of y7+uLlovvl4KWFpajlmeW/FY6V81WK1ZI65FrA2tq6wXrn/fMNso21jfdNy8/hv12/X3tS1o y2nr6rb/I/2UkHol0iCCPlJ+nNja+ikKAO4MAJs5W1vrZVtbm+VIsoH8A2kL+fO/YlsYi9Tc i99uo8fFN5K3z//e/gu44Xw0bQthrwAAIABJREFUeAHtXQdcFMcX/qiKGgQ7GMEIBhs2FKMi WP6AiYLGgBhFo6BRsIGxxQIWrFgADYkldlBjiRILVuwKNmwYFSwYRUFQEZV67z97Be6OOzhO wLbz+8Htzs68efPt7OzMm9nvaRAL4AOPAI8AjwCPwEeDgOZHoymvKI8AjwCPAI+AEAG+4+Yb Ao8AjwCPwEeGAN9xf2Q3jFeXR4BHgEdA+10h0NDQeFcRfH4eAR4BHoHPGoGSLjW+c8fNoV3S Qj/rO/SZV5570fPt5TNvBO9Q/U+x/agz+P0kTSX0PA4H1/iik6E2NDSqwLzvfOyIvoLL/yxA X/MqLE4b1Rx8EBQ8E0Md7dDDcz7+SXgt25yyE3F65VA009YQpXf0R0RcMtLi/oa/Y10Wx+K1 W8Fz5Uk8zH6XjTm5eH5zL+Y4mTCZxnANfyirR4nP3uBhTBgm29UR6qjdzB0BISEIWTQNHp1M oV2pH8KfPUPMHAdUq+uGNfFZSkp4g8SIsWjO1b+RP87nSSWjNNyMnAOnagxfoTzpi1Lp3vWQ UhCzchwGjZyF4OBZGDnwZ0z09MS881L3KjsJMet+hZvrMPgtCsIivzHwnvo7/vrdF6M2y2FJ 8VjXzxvhSTmymmU/REz4JNgJ28sXaOY+AyEhwUzWEHSqWwmVXFfjUrGYijCgjNs4sHwCBg70 xZzgIAR4/wi3yesR81SCs9z9Me+HwB3Rsm2Iw/dAENxl2uoseDt/h77jQnE08Y2s/sIz1o7i /kFg38bQ5tpmNQf4BrHyh36Lb3oMw5x/7iBbQa5PNkrmnor7gJiHhTHIOIM5diao22s14gUl RENRXnqOu3ef4116BJW0YKOfdwqskHfKX2aZc8+Sn4Uew8+IXMISxcUkUpiLEYvTIwu/s5RL efQyciTVY3XQ+nocHXmZJ6uOQhm5lBLmRnosDyz8KCZXNot6ZxkU42clp6t6kkS5JPJAei5h lCIRlXWJArsMobDke7TbsylpGThS4OWXkquFf1PCyEVPST0l2Oi5UViK6iCo3l6y6XFYfzI0 8aWjbwQi3V6dpgDbruQXkyE6z0ugbQObkJahK629nyXWX0BZt1eQk6Gx1H0XXRLcX0EOOjXI YcUtEksU52E/kvrItBcm6/pi6tKXw7AYTBkGgtQo8retTVqWU+jkK3FbyoujFY5GpGU2hDbd EeutTFaBNuxIvq2yqMzT5NeiCmkZO1OQwvumoG2+3Ede9XQIWi3J50h+S5Ap6WM6Ub39sFop vKeytRUk7SBPsypkYBdIlzMLtQrZxHJnhfNm0O0Vfam1sG+RS1zEaYnqJJbzSY64GRAqBk1U rlYdlVjqvNvHcfzWWxXzfWzJBEg/dwTnMpvC/dc+qF/BBM6rY/Hm6T6Mb6n/gVYmDReOnMPz V0+R9CJXpGOVtvAY3hHspclCFuJX+2LYxlsw/N4d35vqitJAA7oNf8KS+d1QURwj+nmLm9s2 40jOMxxZE4GbxY6unuHcgcvIbNIXv/Y1h0R6gUg5THVTETXbFwEn3qLNoL7oUEX8aGlawG1k DxgmbMCoURtLPqorKBCo0BYjJjpD/3EEfhm8BDFZKozrKhuiZiVmEc27hf3H41FGcyNpLT+q Y406fbD6ZiqeRo1HywolW6+TzZuHtFOLMXTSXkjNB8sMi8+84xYg48ljpDJ4tSzsYd+M68LV Ca+REOGHvn19MNv3JwxachJpwmeKxf8TgKHe/ggOHAWnb3ph3PZb4ulaNh5GToeDiSmaOYxF eLz0S0OJPHqGy0t6oRqbClfqMRW/+XZFXcM2GPnPw2KmZq/w78ETuJethTr2PdDmxS6MbG6I ChWs4S80OygpTx4KwQNE/todJnWbw2HUJsSL+1P5ZKVzrosq+qyLfr4VXvYD4L/5Mp6TDoz6 z8LktpXZfOcBju44gxcwROt2jSD7+qkA88ELMMO2eoEqWVcQkVAbDjW0kHdhB7ZeeFlwTdFR 1m0cPHgX2Rpfwv4Hazn5XAZpTHuiA51D2KZrrGOsgaYWdVDwYGniCzMLmGnl4cXhrdj1r/R9 VlRwUXE6qN2yFb7WYv3w9eM4kZBZVGLRtYxk/JfKjCRaZuhouBe9JeatlBTEhY+UMoUxU8vl ILH56zv4/eYDu7p10HxkOM7tFpvMzPvC1+s7tDEzgbnzApxKEzUAyriG8HGDMNR/NnzdPTFD YpahZJya54GBkwMROO4nOPZayExuKra14mtWCinE5kC9CqjQmJkD39xFxEgrZmqqBAuveVg4 0A5mxk3hPO8I7kSHYqidOQyNO8I7nHuG5fJmXMLGORtw+kUmnu5dhFFTdyCh2MGB+lUoaF/q y/gIc+bi1aWdWL5oAtx9j8CUdQZ/750GG72SvXFFFSdkxf4Oz5/+BPX+BdOnOuLttMGYsOcp 8PII5noswMkKNhgyYRJ+rHceS/uPx7oH2aCkbfil/2L82/MPnDk4D92rFtxlpfI0aqB5p5ao xQrOjn+Bmt//iK61BChq4JX7JAZ/BY7D6N9uizt3NiI1aYuOFl+I71sR+svc2RwkbZmC/vMf oOeaIzi4oAeqZpdlz22AjiMnwd2sItJvbMOs/h3RrPNIrIxJEdUj/V9EX3rONNSBfhU9Ns6W C7pGMPtS8iJmdbxyDqm2PhjhbMp6vWvYujMWirvQTDw5vw2B3r/gt0TFVuHCmAKCR7dx4zk3 njXEl3WqyCijWcsY9bghe14ibiaky1wr6UmBrFuIjn2mPPurS9ixfCEmuf+KSFM3+P+9E7+P 7oHWtcRzB41qaOLQERb5UwltGDb/RnQ9+zGe1eyOn7oaISurIup3aCdKl10NXWdHICZyPL6M nIrvJ+zDS3qIf3x+xOAzzTHOfypmuGnit0GjERL7Em+PzccA/xR08hqLCYtn4qf6OshU9qwo r0kZXqkEE0nduFIqmKJDx4ZsdpWLtzlN4LF+FSZZJuOf6b9g6dMeCI1cgcE65/H72EU48LKi bF49KwwY2Ibl1UXtHuOxfM4PMCvD3rUMRZch3u8sWhtftO6DUeMX45/4u7gQNh1OZmwUpzSk 49K6qfD29mZ/ozBx3QWpRY43uLn3b5x6wR5YI8kD+xCHj17HWz0LfPfzIPRsY4QK2W+Qkck6 55wkPEp+i9QzB3H4uQHa2TRjo7lKqFZTUn4R8qT0023eEV1th2HjrUtY7VyvcMclTqtt8BVa t7dCQzbSUhxUKw9Ixpl9p/FcrwVsrKoBkim4YqGlEMteMObu+PPodsx1bcEweovHJ0Ix3HEg Fseyzi8nA+mvVZ34v8T5/alo3bk17Hp3ZWPi17gVvg0n0gtelgUK68Lgq5Zo37Yhm9kUxEof FY+pdOqyOtaD/hf5vW7hQr5ojR9GTcSCf27g0YUwzHDiOiQVg24j2HS1h8fGy7i1ug/qSHCo VBt12AKuZgNr2DXQxbPDx3Dx1j78tuEGtOsZo5amJqrUMUb1F0exbH0MXma8wpuc/ZjUdyRC Dmah+xx3VIhU8qyoqFr5JNNGpbq1Yaipx2Z9OuyFq42aRjWgq1sZ+nrsOXr9Cq9yVDBTlaGy n2nHXVJE9dF68ByEhoayv+VYOJh7s0pCGuJi77Ep8iOc3PQbQsKfodP8RZjY1Riaul/DaXQ/ WFxcAu85f+PmC8kILhP3rt4AN14sHFKVyyucuPiYijVhbjMEU4Y3UtK5F6G/tPS8h7h6IVk6 pmyPKQnnjrFZgkl3/PpXNG7HbICPTW1A2Cmcx9tqTdGuOWcgeY2Hj9OgqAvOVzD9LLbcZKOp OrqoausEZ2YuwcN/8OeeRAUmJk1UrNEQNj//guENK+SLkDlQgKlm3a/R1JDJRSbSMyT3WZSL XrOXDDc50TJBYzNZo46MXO6E7Si5de0/qYGBbIoSyZLNWvpnKY/xKCERD+Q26bC3KlL+S4G2 43gEe7aAIGY1xnZvj86TwnH8spJnpfS1K1IiPb+Daw8V7c4pMtsHc1HZMOyDUfDDV6QKjL5k I1A8RfV2fTH656/zO0h6shPDWKd50GELri9phj1xIeLqaKHyF1+Ae8wLB+XyuLSqjjFl5VaE 2bfO0PpC0Xu6slL98exygRgNPXxhwEYfiQVRZXrE7On7p+/E24ML0IXZIGu3HYglm94irqE3 LqW/RramDXp7dMac6L2sU7+Op9QCRpKRIacY2yaYkFyVmUsq4uXxo/jPqBZ2L+Pwz0Pdzg2h tf0ODu86jSc/1pfNx+XlgqYZvu37BSQGJVGk9H85TKt2wAB3S6wP+g9xd5g5p4uhuB0QMu/+ i39ztGDg6IbejbilVeXLV4KEHZgVbobV8xpKFyY+ZguiV8/jkowsBcnKK6qmMeoye7dpoWah g5pf1oJOsi5sfzuKi67rMN9vLv5c8Tsi2nFdjnahZ6W8VBaVk4WEbcEIbzgb8yzLt+TSKk3R k1xasj8TOVVh5eTIFozYboXlqxGVloPsJ0exfNUp3D/6F8ITmCU1OxvZ6Y/xIFm0mETZmjCx tUMzlif61HWk00s8eZQhxstAibxzkKQoObAaqGBhCQuFq+YqlqdZH7aOTaD19gpOXUwDPX+K R2/K0sbNapmyHUs3xuWPPik7E1nalnAf0AFVwS1ALsRydwukH1qDFaeT80fP3GLZ+l9DEZ3J prOUiL3/6MFr5i8YM2YM+/PB1AluDPs8PN+5HH+cV7ZIWQUWLRqwUpQFeUxroMv0pZhmq4Uj oeE4L1l4YIuof/++A4/MBmH58oEwL+qJy76FzTP/ROpXX8rtiBHrIEjA36sP4LkqshSpLXn5 5r5GhspmJilBb1KR9loAwd0YHL+rCYsfe6G9xXcYOagpch8+RrJAvNhv0BWjf7JCzolATNjx HGaOY7Fi+QhY6LdC/xG9FDwr79K2pfRT6ZDY+tBfmLnkEb5qIDFPqpSxBInYetDeHTiu0BRX AjFFJS1ie6FKl5hsldKVZyJB2g068KcP2RhosSe3Mpm5zqPt52Lp0m4/cjDk4rTIwMaH1p56 QJLdv4X0y3pAp9ZKy5hPETeeUuqNCFro2ojYaJlgYEc+a09SYlYG3dk2gRzMDAj6FuTgu5Xu ZAlIkLSHfK1rkZZRO+rvv4X2h7qSsY4pOfgfpKQ8lidsOFkaGFNTe0/y6d+E6VWFLL020Y20 FwrlkSCFLi12JkNWtlZTT1qhUP/XlHjqT2JmBSaPpTNzo4XboiguLUdcRbY3OX4zeTatwq7X o56Lz1GaQJH+GfRg9xiy1GL1NHSmxZdSSJD1L4V5tiYDI0uyHzGS+puxffJarckr7CqTUQhB hREqtxe2B9e/+wCa6DOY3H1mU1DwPPL1GEmBkXdl71nWf3RyxXhytW1P9v2HkZeHO7n7/kZH HrwmDq/LKwaSWT1Xlu+GSMesRIre5EPWOlw/r0WGPefTifibcvd6Hm07GidVp+IwLaiq4NW/ 9E/gCHJxn0iBwUtptlc/6jtpHUU/yRQnkpflQtODgik4cCoNsTFh7aop228dT3GRS2mAWWWh jgY2I2hh0GzycXUmVxlZBeUS5VDajZ3k52AsvO8FbVP6xuRQ6sn55GRWi4wse5L3vNHUldun L7y/SZR6aSn15J4PtufbY8UJ1q7FeSX7+Y27k9dEVjeb1mTr+RtFp4ralCD1LK3wdidPv1nk M8CD/CNus3vE9pSHj6HuQ3zIL3Ax+Xm4kXfYvyxeUVuT1lG6ToWPVW4/8s+vyxTWhoIocPpg sjFm7Vb4fUAGxW8bQU0lbfxsNO3ybs3uAdcultDZy1vJ21LynJyi+IsSfFqT9+6rdEM6L3s+ 8tjzPsmW3UP91uQprGth/RXFqFwnqcwa3DHLqHb4FD9BVRsMtTPmIPnKGdzGl2hoXh+1Kys2 oqgtviwysq1eVw7fBho0hHmD2qgsbaYoojy+vRQBTnlfyrqLM+fSUauBKUzqGUqt28gp8iwc riYDsN3EDzE3ZqLte2yen2L7UadOvI1bro2+n1Md1GphJ9zm937KV6NUjVpoYc9tTOTDR4tA hQboYPfRav9ZK16Uxe2zBoavPI8AjwCHQDaeXL0HjbY2sKl9G3v+uoSnRW7h4VErDwR4U0l5 oMyXkY+AOtPC/Mz8wWePwKfYftSpEz/i/uwfBR4AHgEegY8NgVKxcXNvDD7wCKiKAN9eVEWK T6cIAb79cDvhSyG848aUUtBAHRGMVOfuI+ArUxh+4O8dev4A91AXDdjnxh97UGda+LHXmde/ 9BAou/bzGvcYS6BxY5Mi9u6XXj2kJanzIvp8TCX0HyInO8C8eQ94uHeDubEJmtosREwOY+mL +g3enTjHAw3gNHkRFvl7w6WNBSwcJ2J7vPRXbowR7OhyeNs1hoXDAHh5DYBD85bo2M0K5iMP 5H8oIn1TRMfsw4QrOxDo2Q66GoZo4xmA4DmTMNSpNcyaO2FkyBEk5jtjyMETRuBjZ94aTh4/ wsG8Luo27YFFMa+AhBB0YvnNubJHOKGZLuPzaOaEEZwe5uxLvU4hSChcOB9TCIEHCHdlL2zX MDwpdK24iBe4smORuL3UQSfvRdhx6hh2BHqiDXc/2nhiYfAcTB7aA83N2D0cuVyJ44PiyuGv lz4Cb/BfQpL4OWVfoR4YBRPO6YTwrz5GnMyEzkP28Zy3HRvMMdKtTt5YHpXIPi46ihDu2dU2 hZ3XfGy/8qL0VSupRKk93WodsvLUyle+mQT05qgvmVTux5wIcB8NiMn2jQbR9hcc4X0WxQfb s/3s9hQcL/4k581h8jHRIR2HFXRf+H1ANiXtHkFmOk3IfRP3IYE4sI9SNrk3IQOvyII4yTW5 36xIL9JHA/KKTBNf4T6ImEm2BlXIzHMHJXHlCMs1IdewByKy/6zrtMKpFQ3Y/pgofhl1d99G KVy6rEjy0gfpS8p9sZs8ui+jeLkyS+301VkKXXeRfebxbuHDaC8M95gdtDkmubBDBZWqp6C9 yN8PJiffsYLZCNqdlK2SZD5R0QiUvP2wex29krwdu5DT2EAKk3y0JrhHYf2G0tpHCu5LfDDZ QJ9sguNEymTF0VpXG3JdG1fsM1609oqvlrxORJ/JiDsHj69dR6KAfY4uZPViI6OGrvB108aj Z0o+265YC3XrVEDOuVjGM8GgfXsCC0b/icfdJ2BBf4uCjxV0LdB/wQR0TksF46wrYWAu1GzG InBMM9z/cypmHUwBHt/AhcQsZEooU3WbYKCvIzIesbe8YSsM9eyEGopMO1XbYeDQVoxUtAwC o+6M8BmKKdEpRZM5lUHRZSOS4d62D/q1rZnPK1MW5WhUs8OvgSPQ7P6fGDXrsBrtoyy0+pxk vkZ82M+wdjmMxvP/wu6g8ejf0UT47AriI7Dy8GXsC16OHVeecZ/RKg4cp/jcKdjSZDb++Klx wXOvOHW5xX4mHbcOjFtbwSJ7F+uspzJCfc78YYguS//EGLMCnr981JnDgthVgfjjYgZ0vmmJ RoxEJ/fSIfydaIhufexgLNdxahj3xbI5XRSQ7edLLOKgKtr2+wFWuIOdkdeQY9yCcWVn4B9f D/wSwfkJ1IBel3nYNaYxUK0jfujMGPIUhtro/ENHcHRXosBMQAcWwtOhEWp3+xGDmldjU8Iq MHMOwvkMbiMua9TbJ6PvwEkIDPCEXbsBWHL6KbIS9mOJby80q+MIj4HNoK3dEp6L1mBj1F1k XduN0NVRePjsGGa6ecAvyB/uzZvB+4BinkOJJur95jKPIvPgNnAagma7o3k1bxzgSPe4B2ne UEbO78fMFeZoPjQM8a/uIHKJD5ybNUIPDxeYC/2E6rIXXCVmwtrLnFqw9YxLC+HwRUf8epLj NGF8FQk7Mc5hklgm8225fCT6ek7FbM8OMHWcJ3ISoKisfJOWKrVifCZtf4C7VQU83HkQFwux 6Kkig0+jHgKMg/1yCNyHnED7pYEY2bKG1Ev6LW5FXUCGfgK2LRwHF6uO6BUcXZgLiJ7i1Myf MDplKDb5d1ZK86uefu+YS/HgXfVYVrzqid9nSsFTOunflQyYvtBvRe7LzlJqPkWCZOorfPFy L1/2p0X6lsMpTOgnMJserPiOcRhImzlKXpnCphKxDPE0GzbBzNQhMZ9wnCqGZOm+PJ8TQqZE BVNzmev0go76NBLWRc9hGd3KyqPMS3OpnY4h2QVfp9zbwWRXeyhFCE1FmXQ72JF0jIdRRGoG xQXaMb4KK/Ldd4J2L1lLp1KvUrCNvtgsk0OP1/YmA48IxjqRRy8iptD4fNOPrAaKzlRvL3dp bc8W5BGRzMQ8oQgvP4pk9qncywHU2pEzX4nNXxy3x9E0EsQFkjXH9eK7g6J3h9LKEwdoYTsD quGxmyHBzBb3V1LvodwxM5Pd2UDuHA+IvheTyfyOHhlHjToH023OapayjQYY1qaea+8qLUuh aU3p/UijSK8G7D5ImeEUAcPHqYSA6u2HtRkPhrtOK3IZ3pfsbXuS+/SNdDmfs4crjvG7xIbR RHtT0tL6hvyiuZbCgtBUokO1G5iSgY49BV6X+AoVXS7t/6rXqaDkz2TEzaBhn2jbzPgHVyID 4FTjNjaN/g5df9mPZ1zXlh8s4DLWDU21tBjvzkpcjf0D/c05BjFCTmaOmpSq+cKLPdDSq8j8 uXDmk+k4dGUPApxqIm7TWNh3/RUHlJl0lEqtymYUO8E6XOiYNUR9XU1UaNUPIxx1cObQMRz4 awuON7REk6pcE6iAhr37oPPTPdgQ+ZQRxrNZSGVrOHbrBGffwego44RBE5Wq14D2+okYEhKD PAdfjG+nnPxUqXrFXqiE6rVeYb3vGITEaMBhxki003mNS3/vRV6nNqjHFpT0zBrBQus+os7c AzGddRlpgI1jF1g7e2FYp87o62GDlxH/4MTLLCQeOIcazu0YqyAzkzEXXFNGtBFrkIaz2w9B v0cXEXNfjd5YdvYoQvvXVFqWWh8Oaumgoo7cVK1YDPgEaiOQexunjzyFcS9PTJ23AVtDvofu zpHoOnwrkvKfebYA2aI/Fmz/HT8bX8fWvTelnnFtVG7UAs0qH8UUj/k4KXbTprY+pZzx8+m4 hcAxV0WOU7H7/B742+riytIJmHNMeppvgk6jl2PT/P+BIqbh5+VXxCvQzNehRUPWLaThxq0k 4bC8VO9DahIevtXBl60bw1gomHUuzIHA1N0ncczfDppXgvHznONKXG2VRBMDGNWrgpwXyYh/ xFyrcXSzkkZcxxTmlV/jyTO2e6XIoImqPefg72nGODS2KyydV+GOWj1ZkYWwi7XRM/BPTKsb hbEdrOG86iazryfjxqUHeBXL3M6FhCBkVyZ6LpmPUR1qKBCmC5OeP6D7673YsOckDhytARe7 muJ0DF/u5SQMKcxxwTN2ri2eSrOH2aIJczWWUoKyxKIU/rxA0kNGyPulJZoaM5sbH8oHAcEb pD9nc+ROXdHSsALroAchYHxXZOzajgNP5Na19NvgW/vqSEp5KdVxM/OqYwD+3jwSX11cgH4j NiA+/2EpnyoUVcpn0nHfREjvXxH1VtRLaVTrDL+Q0WiBh7hwTX5DmD5ajlmAmQ7AwfFs4fAy t+TInPN+44ieNV7h7M7DCjx15+LF02dFbAcs6hawhdMDe3BEYIWf+jTGgxBP+EaJXybcLMEv EONb6CLxwg08LkqMStdykc14omu3aYsOzDUTblxFnAxncB1YWhRHHMW2Nl6/h6rjI9jsZSqs 4ueKfA+qVH5JEr3A9Zu1MP7QWUTObIV4fw/mx5N5hsnKRe6XXTBMyK3N8WsPQddqnOvWwkHD 2J5hqou9c8djo7EDbPUVNfcqqFEHuHg4Go8lLzHBfRw+dANvSlBW4dJFMfT4OHYeeYMWPzmj zce/DV9ZNT+8eJ16aNSyIlJSJZ0xc95gZAQ9ARusKHQ7VhktG9VjM17poI0a3ediT6gz3m77 hfnPjBI7AZdO836OFbXk96NJmZZaBxbGhzB/1VVx58pMHxkZeKvVDN1tTQqXrNsCo1YGwLnK Kfj/FICj3DSpanf4/z4YJqeCMSX0PPM4LsnGea32x8+zOc/u7PjABqyPuq9iJ86Rum+Ez9RD qDd8Gka3rYF6FhXx1/yNiJO83ZlvxVdvddGyewfUlxSp7m/WTZw+XRfDf+yE1m6D4KhzBvtO sJ0sbA6Rdf4Yjn7FnL52lvKMLldOXnoG3jxPwJntv2HN+Ww2exmPBV5WeM3iS3/dLQlRgRuZ V/D6cJwyC15WhPRXtWDNPG0/+m06pu6/x2YLz3AlfC7mHXku98BJFDfC/9zsUTkuF23YnnnO 90zhUBs2PTqiwr5ATFx1Ac+z7iFyegCOaFky5rySlFVYMrJvYr3PbOyr54H5o63L/cMOBRp9 PlEazIPRD62QGBmFa8JnKRtJD/6D7v+c0PXLFzi/ZQuiEsWv+/QL2H+mNYa7NhTPuqRhqoyG w5ZglXt1xMwfhhHrb6r4bEvLKIPjAnO3ekdMJfUylmuuTLbg5khGxhZkM2QqBQZOIBcrWzHZ eRaxj2rIS+h4oDbZeAXS9tjnTDvxvm2tytSg52QKE8Yx4vWIWdTfqh4ZNGhNNjYdyNqqK3ks PkQPONL5nLPk14gteun0prWPpXc859Gr2O200MOadGBAVh6zGXG+H43t35ksWzBC++DDovys VEHcYrI1qkdmNoNpeuB8muTSnqw9NwkdM0ggy0s8Siunu5AZI4DXMmNE/CuPUiK3sFYoxAkX FbWMrMnFZypNcHGiofkE76/pwX5/crTuTRMXziLfoX60jS3EZj04SHOdvmJ1aE8+m86KyfQz 6c4KZzLQ/5ps3QNp0+xuZGA9hGYHzSVvF0/6PfZloZKVRajeXjjdTcnaYwYFLRxNLm6hFJvJ MH51joKcvhY5stAyIVsf5rQi/S4dmduLjBm21j4bKDqROVGQBLYv3td+Hl3OlUSwxcnE47Rs AFu41etMk3Zdo7RMkYN85quDAAAgAElEQVQI5g2SLVi2INfFJ0QL14rKykqj2O1LycfemL26 xe3lZBRtX+hBVsw5g46VBy0IXkh+Y/uRraUV9fReJnLqICme/30nBFRvP6wY9o3FNh9HsnYa RwsDxlLfAX60O54tNOaco4BWhswRyjfkMWsmTRo5jVZGi/b05yUeoWVetmwTA+dsxYuWHX1A ea9iaYu3DbEXP1u0b0Quflsp9pXCB06tupWoTuISeHZAhlqpBrqLdR7rYRo6g/lKfN+LUcxE 1OkbTLf8Cymhjh/EHtSy+2S5VO8iL+wDReBTbD/q1OkzMZWUVytMx5X1W5ExdCQ6v/dOu7zq zJfDI8AjUN4I8MslpYq4PloM/pUten4I4Q0enj6CmMc5yNE7g/3nm+DbtvU+iFH3h4AOrwOP wMeMAG8q+Zjv3keouzrTwo+wmrzKZYTAp9h+1KkTbyopowbGi+UR4BHgESgrBPiOu6yQLXO5 HJ/4A6ltiWVeIF8AjwCPwAeCwGfYcXM7LQxR1bwb3L084dSMfQSt2wxOI36GOyNkqqrhgJAE js2ovANHqhSIQUN94dXJBHW/DUVcnpQOSvnEpdK802EunoT/CEPDHxEu/2XZO8l9l8zsY5/z wXBz9cLEgUMwL/oOruw9h4dl8qXmu+jJ5/34EBDg5T/DULORP/tW4OPT/p03YbMqq7V38f1l YvuDuw+n7SncPmsxAZCQbIjTiCOmGVDAyV2eSuaep4CWTYRc3YLU87TjwB0p7t/i+MRLR1FB agxt3hwjRb5VOnKlpZSoveTFUmC7r4Vc5By5VEu2b11EDCUtkT/+nBAoUfspEpg0RsLWkiz8 zlL+Fv8i05fdRXXq9BmOuKsxKtCB6FJD0YaaGug40A3NDbUYluUcHl3C8WuZwkI1qrVBHwdz qR0gavCJq6G+RrW26Nev7YdDX5l2A9FXc6FfpSK0Wk7A9iX2atSKz8IjoACBt5cQsVMDvR2a 4D087QoUKlnUZ9hxy/NWSwFGabhz9nf0qV4Hdr/uZ+7EBMi+vw5963TCpLB1mOvJXJ/V7gz3 QVbMpKIBbTNXBJ9nvCIq8zYr4sBOhuBhFFb/eQgJeem4tnslQjjOaxlzQDF84grLZ5/fRwbB 17k56vQYhIHmEhdNyjiqGQ7ZdxAxzg3jhPzazHQTE4phfYfBb7YH2pn2wJxTjMtaYVn53/9L gfmOhxlXsGP537j0VoxJSCg2nEyUEvoc50M84Ow5A0GTnWFhN0vEoc2lyL6F8FGe8PHpgbqG 3TCD05tFU1p58IhLqcgfvjcEBHd3Y+qP38D4i67w+qUv2pkZQtvcA2FiV4R5109gPzrDsTXH bFlEW3pvNSim4HedADDx7yriPeaXN5UwVTKP06Svq5CJz2F6w04Fd4Kpi2sYPT7qQ4yWiX0m /R0F32KfzWZGUwDje9axC6abFxVzRMtXLE8pBzabrAk5gIvg+y6CT1wZRzUJrlKgdRXSshxP +6J30pI/gmmitSKOaqap2AVbPuf4y4Pk06gnBd/OZBf/o+0DTEmn51pKVMKHLV9XZeclai8y mIg50yVmrcdrqaeOGC8hF3YtIYc2McbtaL9vSJ/jCxck0Fqn2gQjG3IPPEjX3oFHXFl9+Pjy RUD19iNuL5LnNZfRUVjoi00jGRTjZ021vPaTkBxBaVsqn7qpXqcCfT7DETeDqahQoS0GDWuD R5u24DDH43w0BjV7d4JRlwU4Hcym6jqmsKjPOLortIbHiM7AmV0IDPlbIUe0zKCZ8dddU8qB rQLvn1I+ceUc1QLm8FSX8XBXtvkfull/D9/hI+DtqYijmgHCuWCbMoR54uEC4c3ZXQjXt4O9 eQV2Xhd9lh3CxdAeeKKED1u2rkIhZfuvTg/4bVmFX7ow5sQTp3AjJxMv0pmpiRJxIvIuGjWu j0oapujZrzP0XlvCfUw31CsXHvGyrTYvvYQISJ5Xraqozlh9hdStedewZ+tL9Py2FSpx4pS1 pRIWVZ7JFRl6y7P8D7AsPTTu/xO+85uMVTv64LtdFdBrA8eSLb/0LKaJzDmJ/QfzWOfAcUSf EtaH44iuZCnPEZ2FJ9Ic2ByNiZgD+9ozxtesUpDwiXfEzO/7YSbHJ+60AS04juqKisrnXLRJ BzFHtc8UxlHtBnuOo3qlhKOakdcKnRFw6XOQdOsOkllnzhyXC4OGYUNYGt7DOqVlidKV23/m 7d7sy5sYaz0aiW0twTEsC0chzD1bDcZMey36GlJhCf3q1VChcQPU05bwiP8IJ45HfO90bN08 gbF+8+FzQ4BuncKBR9aYYCtu+8ra0gcMDN9xK7g5GsbfYtiPs/HDsul4ajEOe2twyxfyHTej hmWOCAS1m8Oq4Q1cFXJEdxNTh77G3StJQo7oKvnyK6AOx4G9QcSB3VjoeYa7yHFgF3Se+cll Djg+8Q2w3DxXSFwl4RPf1XIh4xNPRGOON9qc46guXL6MGHYi4qiegYGMo/qJ42IcVMhRrYWq NapB7+IJnHzsBTOhA4BsJB4+gAsZyssqqKt8qaV/Lohfjf6Oq2C25QiOdbmCkX/txzVhMfXR f+k87Os6F15+d1Hv3wyMXzQIjTXZ1sJrYh7x9kEY5c3xiDdF/J/OzCsOHz4fBN7g5r69uN59 GOzEz6DytvThosKbShTem1pwGD4QTa49hokz86quME06Yk/fgNnwXzGV7QApniO6EizV4MAW FV0Un3jHMuCo1kJ1Gwf8r8IBBExci9jn6YzXeC4mHjFC3/+pUleFgJVqZG7CVZx9UQnGdRgZ 68tUPMt/r75BYnQsKk7cjLBZ07Dkrw2YalOL8Szn4mnUynLgES/VavLCShsBeoCTh5Lw7fcF z7XytlTahZeivAJzt3pHTBX1Mr7vXHkP6OjKqeTCOY3lOHanr6KjicwbrSRkHyGfev0oLEWy y1O82KFlTFYuY8hvghs5DBXzZCvkbc73RCyRyH4Vc2AL0m5QZKCrYj5pYe6i+MRZAoXlZ9CD I/PIyViHdKx9aFN0YsG+8EIc1UxG1gM6tcydGqAG2U7aTjfSXtCdsOFkqS9yWtzUdQlzGsz2 vissS1FdhYoX+qdye2H6nFwswWQLXY8/S8tcmPNXPQfyP/6AMpN2kCd371CLrL1mMIevRoxf 2Z4Com/Qfi9LEV83Z6yHHhlZj2ROn5+z9V8HtXnEC1WEj3gvCKjafgRpMeL2wnGuX6b4S8vJ hT0LbO7M2oQdrbhf8KwLlLaltHKpo6p1klaGJ5liqCkKgjvL4bTQBOGrJFPpbCSE9IT5dHNE poTCUeKyUFFmPk4pAuoQ6igVpugC2woYNioUrz0H4usXqchiz2l28iH8cbUHdgV2VeIpR5Eg Pu5DRKDM2897qLQ6deJt3DI3itu7vBXrrxAqRh9H88GrePunDD4f+kkuksInY8hRE/wzvyU6 C73TM9NJxDk0at0AfGP/0O8fr5+qCPA2bhmkMhC3cwkm/DweW74ciQkdDcRX2ajt4Tnsj3nI Nlwwn4v7z+OhxCekTH7+5P0ioI06zj7wN9uNHtV1oMG2Q1Zt1h+hz50xrV99Bf4E36+2fOk8 AuoiwJtK1EWOz6cWAupMC9UqiM/0SSLwKbYfderEj7g/yebNV4pHgEfgU0aA77g/5bv7udaN nuPu3efc9gE+8AioiMBr3LuZyBazP47wYXfcgkRErZoGV/Mq0KjUCd5BQZgzyR2dzJvBcdxf iM+3Mz9AuKspDF3D8ERN3CntOOZ07wTniZMxtLsrJkfcgVJWbrZzYfs4dwycPA9zfFzg4LYU p9O4b/eUBfbxx5UdWORtB0NmdzXs5IWFiwMxV3heCWauU7HI34txg5vC3HEitouJcJRJUxzP HBWvGweXNnVh6H1AVvesWKwb9T3aGDeCt5BASrGEDzf2OaJ8v2O6pyhUkZ7sxWS7pmjuNJBx qjeGcd3GsFkUw77/lA9FyxGlLgJHFHVNviz+/H0jIHymHetDmxHCaWhbwHnRSaTlv80FSD8w CibcNeFffYw4mSnFyCmlfXYSEv57IxXxARxK7w1U55hVQZ1sJcgjTwQloKzLc8laqwY5rLhF oh3EOZQas4M2xySLz/PoVfQGWnf+lYrlPGY83I0KiKXuryCHGs604g5HsCQfMul2sCPpD9jO 6Iy48JgRMJlRy4DzxfP6CkmT9MkmOE4kVP785T7yqsf2mrYMoMuS7eOilCr+Z1zjNvqk7xVZ sGdbklOGsEkSWf6/arWXF7vJo4YOGQ7YRimFVOZ4lZtSZUYElixsDBl0e8UPZJR/f6QyFClH Kh0VgWOR16Rl8MdlgYDq7YcRow10Jt99dykz7RJt8mpHOlodKOAyI4jjguAehfUbSmsfZXNn hYMgmaJXjCZH2940NjCcTiUK6agKpyuFGNXrVFDYhz3iZjUqHDSg26Q1rCo/w7nYe+JRlTaq te2Dfm1rCncO0JNd8Ok/F9GphcdcheWxmKQDWLnxLf7Xtbnwk3UNk/awb3ACc1fHCDkwZPO8 wd1/7+Dtf4+QnM+sJMDb7KJG3LISlJ7pt8G39sbIu3YGMY9U1F2psE/lQh6e7f0Lh3SqIv2v 9dj5QH4e9ATXLjyEIDNb3BYqo+FAL7hlPMEzGQiKkyOTmD/52BF4HYsz1UZgxrdfoYJhKwyY NQa9dK8j8kSisGaC+AisPHwZ+4KXY8eVZ7JmNe5bgEHd4HLIAvN3bUPQ+B/RsZ6QjuqDQeUj 7LjZ1ry4S7j4uga+afmV+IMKFpewE+McJuEAm9acCd+KqIRnjNt6FVZH3cWzU/PgNnAagma7 o3k1b5ZGFv/sqzE4kVMfjc3Z59Nc0GCfmDOXZg+ionEnf2olugS2s7u9S298dWouhs46jDsn V2HZ9W4I8GxdeoTsul+gSiXpW8O58ApBP+dhmB00EU4Wjvkc0xz3tMRsE+Dti5CrEmIphkn8 XxjXdwgmB86Et3corgqrIM/TXQXajYZi4ayhzPTjB+9O5szRRJjYDMW5UyuMXbnyWlMCdm54 iVFBo9FGcAJrtt1E/vtSWB8jtO5ojux/JsPtl51I4Mxnet2wdNdImEluGfdbnBylOLK8Cq9l yPGdMxwb+yOmFN7f0mrzx2oiULkHFgd9C/ETDcY2hprs5W9al9vi+xa3oi4gQz8B2xYy86JV R/QKjoaI6i0dlwOHs28B2mDpsp/R0vAD3f1fMPhW74ihoF5GlXOJTSU6TanncC8aMaArmenr U1PPzRSfxc2NmenkzgZy5z5/lnA1y5gF7tLani3IIyKZpWWuybz8KLLga1cWJ/6UHfZSLsvk zTPyymbQnU1DyIxzpWWgzKQin4edy5tG5M4FKbvIs54eGbpuoscyX5FzdaglNoGIdOO4sR8L uadtyGLScRIadfIuUkCLyqJ0mafJr8U3NOmUyKCTFxtALSDmr5bj6V7oYUO1HFbQfYHYRRqa ks9R7nNfRdjl0ON34LUuaXvhOMy7dGe65V6nYDtDwteT6VSmDDgkSI0if1vGuw0t0rccRMui JSazgntQtByOw1sJjkVhLIfjkpWnytTtW0FtPt+jkrYfCVK5MX5kUW8kRb7Mk0Sx3xxKiw1j dAmmpKX1DflFs2dFaE7TIp3WP9BwVwey7TmYpq+7SGmyTU5KxrsfqlMn6WEdy/8BhwqNYNen N3q7DcZwlyZ4tG42Zmy7zRbhmOnEvC+mjGijRPlKqF7rFdb7jkFIjAYcZoxEOx0lSVWNptd4 cuc5Gri5wJr50ZjkuaDA+4qqMvLTvUXCtlmMJ7s32jYfhP2NpyPiDzcYielURcnqwdlvDbb9 YgMknsOxG+nIeZGONzkXsXnFI3Ts2BgcazY09fDFFxyTIftO6Mw2rEhoio4tRWMOzSpfgPP1 IQwyPN0O6PxlHoxs26AeW6TRM2sEC637iDpzj41sFWGniUrlxmv9Fv/uPgTd7zvDRKshevXv CJ3buxB2TNYIwrElzjh0FpEBTqgRtwGj7Z3wy4EkqelvMXKKwBFFXZPBkfGdD+v44bh9k9xr /pchkILDa0+h7dKJcJBhw2QbBVr0x4Ltv+Nn4+vYuvcmsq6fwZFnpug1ZBLmbQlDyA/a2On5 PYZvvi/Vnt4/qB9Px61ZC5adHeDIdg5MWLUaM22SsGlcCKLecLYM1nkzLmnFoTZ6Bv6JaXWj MLaDNZxXyU+1dVHPwrxgSiUUkovsLEY317wRTE/7ok7+ynNj+EalIm3PNPSP7IigjZsReXQh bK4FYujC02wCpk7QY7tK/BD65y5cePwSjw78Chvhp9rSslgDM6uO2zO6osnAMCRKpuOJ13Hh qQYq6MpP5xgF67UbeKqpC10dmTeAtFDxcTJucBzbsRyfdwhCdmWC4xMf1YHjRFSEnYTX2hiH OF5r51W4I2u7UFCGmlF5V7B19Q28PrEEI719MJ+xulXGHWzZeFzOfs3k634Fx6nbcP7YTNhq XsDSnwNx7K3YzlWcHKU4MrlFXVOzWny28kSAmfuOLsMaw6kI7mOi+OtZ4dpSdaGThdyMdObI zAydvm0BQ80aaDF4CsZ/m4ldm4+pvWOtLGr78XTc0rXXNIZFE0Mg+Q5uJRW3iPcC12/Wwnhu RDazFeL9PTBhz1NpadBtbg1bndu4EvdSFE9PcOv6S5h2bAXzLkvxhIizB7G/m1jaRRPn9x/B y9aWMNdkHWqrnxHwiyVunb4CFfzYyJSr8ongJlb1d0OQvh9iji3DoBZiy10lfRjopOHGLenR JSeVjYr1K0PndQJuJcoZ9AsVyr2kGMe2kE98DMaM4f6GoGu1bMYnrgi7J8i4Lua1jpwKq3iO 13ofxMgVkq5+BOHtiR3Y0/U37N30B0JDQ/H7tr3Y4GGK5zu3YM9j8X1P+A29fY+IX5pskdpm PELGWws73GvCNCrIUYoj076oa+pXjs9ZLghw6zzbMGdvY8zx71LMbKgyWjaqJ5xxttRKQ2qa pF8xgFG9KhBkSRa/y0XxYgv5ODvurJs4cyYZsGiP9vWVjbRzkJ6Rged3z2D7wo04n1cfjlNm wcuKkP5KrjMzssfgvto4dZpNlRhklHQRJ+JaYdRAKwXERJXRqGUT5MXdwoP8kaYWTNo0hTHY wt+BDVgfdV92H3Wxt6GYBLmJiD2biirGRjDAK6Q+E3mDR+22sO+ggdOr1yGK20ee/RKpL/NY IxOgescu6KB5BqtDj7G9q6wBp6WxzjUPWYV2vxgXweedhKhAeezelA+vNeNN/vv3C7B1/UZq NlQL/3PvBZO3R7Fu+23RImU9cxj/FYRVcZJF2RxkvMpkXuHtYMu1DVXkKMWRPaxFXSvmtvGX 3y8C9Gwffp12H99PdYU558opOx7bJyzAwZdPcX7LFkQlivdmp1/A/jOtMdy1ITQbdMMPNsmI PBgneoYpBQ8SdPA/l0748v1WR7b0dzWtM2nvKkJ5fo4ze9UMGtC0CkHHmjwWLKVAvzHU38aU DJr2o8WnnrKlSbY4mXiclg1oxLiaOe7da5SWF0crHI1J38yO3INW0WyW3tpjBgUtHE0ubqEU K7e4xSkgSD1GAY425DRhEnnYdibPsH8L74WWaMoc64Z5dSd79ym0cLYXOfddQCc5ruoc5pC0 EVsk1elNax+z8/zA9pXH7qAgH3syZAtoBjYjaMGihTTHy5YMxOeB22NJ6a5zQSLt9mzKuIRZ XuvhtGAi45VGbbINOEXP7mwiT0u2aCdcmLMjmwYNyGbIPIqIT5bh07a0b08NjG1pSMBmOrxv rixPt1KObW4/szx23GKu+rzWKrUXdt+PzHVli78W5DLvECVK1pO49jDPlXGGM3OjsRPNOPKA 8rgFQltTMmb3esj0BRQ4yZWsrEcw/m22X1dlOffprVIcM9jityKMp1HQ71Nlccy/3/xBWSGg UvvhCn91mgKEC9ZC0zRnM2N/OqJvNXLOUUAr9swYfEMes2bSpJHTaGX+gja32WEr+dh1IKeJ cynAx50GTN4p3ghRNrVSuU5SxfMkUwy1Ug10F+s81sM0dIbQzVipyv4EhKlDqPMJVJuvQikh 8Cm2H3Xq9HGaSkqpEZS+GPZJ9PqtyBg6Ep31ilsULP3SeYk8AjwCnwcC/Ij787jPH0wt1Rld fDDK84q8dwQ+xfajTp34Efd7b4q8AjwCPAI8AiVDgO+4S4YXn5pHgEeAR+C9I8B33O/9FvAK 8AjwCPAIlAwBvuMuGV58ah4BHoFPEgHekUIJb2s2Hkathp9rY0Z4XhOdvBcgeM4EuHdqDAsZ pwLv7ixBmuVNNQcIbPNn2jHMsHNBSILcRzsytXyD/xKSivjohnekIAOXCif0JBIzXNvB2LAw m6MK2UuQRBXnCoSsK+swykWRPkVdK4EafNJSR4B3pCC18Vv+kKEtH6XWeVakF+lL2Os4CVkx FGCtTzpC1jouQt5ZAot6dZZCGXOX9KcuXErFoaQOEBhz2OUVNIBjHZRhDpRIZ/pEryRvxy7k NDaQwk49UP7BjiSLHBtgIbZA3pGCBCn2K2ZtlDA+Sl1R7zCNokPD6Lx8Y1HZuUJR+hR1TT1t +VyKEVC9v+EdKZT6m1AlgbrmaGNVAznnYvGvkDZA1lkC6CEifIZiSnSKHD+zMukld4CgYdQH f/w2SOqTa4ns14gP+xnWLofReP5f2B00Hv07mih2eyTJosov70hBFZTUSJODJxFT0H/KKaTm 0xRwYnjnCmqA+XFk+cQdKcjTyn04N4XxCly4+Aw637REIyENK+cs4W9M9oqG454AtDy/HRuZ k4Qs490IXa2LH3prYM3IDchrXw931+yAfuBJhDoyIqr8IHaA4MA5QGiC1d3OiBwgzFXmAEEb BrVrKDB/sKnx5RC4DzmB9puPYmTLGooZx/LLVeNAoSOF5Rg6+xqadjVEzO9XYPXnRvjb1IIG R/I/eTZ26zaFRfpxrOMcKVhyZYoIdiZP2Q/dtvWRfniz0JGCJcenErkey0PXYHNeS9jf2onN 2v0wtz9w7Y0xvjgdjlMWM7EztD/jd8hjjhQC4bXiNdp/fR9rluoj8EkoHDKOYZZXUVirUWdl WfJuYOOwbphy4SauZXXH2shlGGBeSUgeJKqbKVL2HMGrXnPxh28nVEMyTs2agBV5zfD13XAs 1Z+PJ/56CN8YhYSsmtgduh66P/yILvU4HpMCJw07+y8WOmkYOr4Fo+iSBPaC3j4bU3YT2lqk 4/C68+yCufiiomtmyErYjyUr/8CazZmwtn+EDZsrY+qZI5hpXUUilP8tDwSEjhSkChI7Uqgg 50jhOHOksG3xH3BavAHhY9uhCvMrKnGksPki70hB8bxHKlZkKmEOEnp6kNeI/mRvZkBaTUfQ tnjOR5wCZwkyvv9UJfcvuQMEkV7SThaYMwaPBoyPpBW5DO9L9rY9yX36RrqcJj8Hl6qc5LAY UwnvSEECFPcrNj/ofUfBt1gbyDxOk77WJwu/s5SbxzlV4JxjMN5GFjgnCXY6X7Pzx8wF6Frq aeBJERmM+Z6ZQbzGH2CSFJsyinauIKDMaH9qYSFx3PCaYgPai511FHUtk+IC7QhaVszf4Qna vWQtneJ4bPhQKgiwrlgtObwjBamXWOkfVkVjO2f06u2Gn4b3gdWjTZg8Yydzo8Xolot0lqAi uX9pOEDIvY3TR57CuJcnps7bgK0h30N350h0Hb4VSRyVTYkD70ihSMh0TGFRvzJQoSaMazH3 oCkvkXttF9Ycr4UWHLUvC5oNv0X/zinYvCEKzypVQy3trfAdsgwxeV0xY3w7sXs7+VKKca7A Rl5nNm9FQscOaFmBoy/QRpUvJKPmoq6JueErW8OxWyc4+w5Gx0L86vK68OdliwDvSKFs8WWP WE1LWzg6OmPAhFBsmtkRiZsCsIQ5LyjaWYIq5P5s2q/EAUJ6lLyzhCLYpQVvkP5cCw06dWX+ 6CowDxqDEDC+KzJ2bceBJxIPByWBiXekUBK0uLS5Tx7hHjNiZXP+JYWhBkzNDfH2yTO8rPod Av+egLqHxqGDpRtW3eGIehWE4pwrMHb1axceQ7OCroKOv6hrCsrio94jArwjhXIGvwJMLRow r+vMi/etlGLKZtvtiiX3T1fqACGlkLOEqsrL06nH+LgrIiWVY7fmgjaqGxlBT8A6khxJR6I8 u1pXPktHCsqR0q5TF1/hLmLj0qQSaaCW5dcwyriNm1VH49CVPZhpFQ//76diz0v5+6KCcwVU hL5BRby+cQuJ8tmLvCalEn/4nhEQrfPwjhTK9Ta8ROyZy2wprSG6tjdRWnJeegbePE/Ame2/ Yc35bJg4jscCLyu8ZvESHxaizEU5QFAqvvAFDTN8+0MrJEZG4ZpwxJeNpAf/Qfd/Tuj6ZTbv SKEwYqUeo2HZB96OeTiw77zI807WNUQdNcKIwe1R6WkUAtdcQp5Jd0xZMAxWr1/hVY54K0le BjLePMPdu5exs1gnDcboaN8Kmqc3IDTqKVvqzURaajogfEEbFXGt1KvLC1QTAd6RQhHLAQzT Iq6qcimLEo/+SbMHtGCOAgzIymM2BQf60dj+NmRs0IJcF59gnrMVOEsQZNKdFc5koP812boH 0qbZ3ZiTgSE0O2guebt40u+xLwsXrswBQuGULIbt4447SKu92pEOGtGA4AiKSXwtSsnkbPNx JGuncbQwYCz1HeBHu7lFVN6RgkIkpSNVbS+CtBha5sItArdni3y3KenycnIx1iEd6wm07wFz bvBgD/lxji84sntfb5q8Tez4glsA5gjyZy+hhd5u5Pb7JcpkCuTd+Z0cDQzJzKY9dbG3Vc1J A9dePFuz7wtYv61vSfY2X5OxzRAKiLhNWQqvMY/gQcHk7/SVUG+fTWcpMasM3YNLA/uZHKva fnhHCsW87dShJCxG5Md9mXekUOT949tLkfDwF4tB4FNsP+rUqWDLajGA8ZdVQYB3pKAKSnwa HgEegXdDgHek8NzNJMMAAAp6SURBVG748blLiIA6o4sSFsEn/4QR+BTbjzp14kfcn3Aj56vG I8Aj8GkiwHfcn+Z95WvFI8Aj8AkjwHfcn/DN5atWgAC9SEZy/gc7BfH8EY+ACAFpPu5cvHj6 TAFP0YeDVbl13IKHUVjl5wpzbQ1U6uSNoOA5mORuB3OLbzFu+y0xSLl4Ev4jDA1/RLhaXyEW APvuPNqMRCh8BJpX1YaGRiXUdQxETIZoP3DRPL8FOhQc8XzcBViU7xE9OYAA52aoYOiOLQ9l d/Yj4wp2LPJCJ0N2jw3t4L1oNdaunQ/PNobQ0G0Hz4VLMWfyUDg1t0BzpzEIOXr/g36YyxfZ j7E0ad58AdIPjIKJhgZ7vrm/+hhx8hXSDgTAuVl1GLqE4+GHXMV33f7J6qa6iKxI8tIH6XtF MtofLryiywG2pKXTk1bcF8UIUmNo8+YYtndbLLZEnNtcntLh0c69vJj6TTki2kN+ewU5Gdam nmvvMvnF8PyK1Vb4UwzJFPF83Aphe9fIwkRh0hLjKNhGn2ATTPHC6DSK9GL7x6V5wAVP6aR/ VzLQakqeuxMZ5Rkf3hcCJepvhEoq4c0X3KOwfkNp7aNsuaqI739+e5C7XAanJa8TUbmNuBW/ vKqgSZumqJwTh9h/GR0pCxrV2qJfv7aoxvH6lJhzWygC786jLcBbPQfM8e/C9GCkQQ074X9N K6OiLmPBLYbnV6SBmv95Pm41gSvjbBq1YPPrbIxp9gB/jlqAg+kypN5lXDgvXn0ElPPmC+Ij sPLwZewLXo4dV56xL6w+rvCeO+4MxF24gdc6TdCyEWOA40L2HUSMc8O4A8l4ekbMuX2N49yO wkMB41qe+RMG+gVitnsrVPM+oGDqKuLRZmzLckGKR3tpYDE82pqoYtEMDXSFbw9k3z6G0zXG Y6bLl4CQ5/fbAucKYp5fUyHPr1yR6p4q5OMOQT/nYZgdNBFOFo6YcSpZ1Ng4Pu5x7hg4eR4C vH0RwvFxCwPH0/AXxvUdgsmBM+HtHSrk42ZvHsbHHQRf5+ao02MQBppXgXajoVg4ayiT4Qfv TuZoPjSMMTJyTZkR9JyaB7eB0xA02x3Nq4nciHFmqJluHvAL8od782bwPvBcXGZp/igum9M/ fvtk9B04CYEBnrBrNwBLTouwoLRzWD5sADz9psOzXVM4zjmONJknksk8OR3tatph5B97Efe8 BKRgFazQz90KeHgEkRdflWZFeVllgkBRz/tb3Iq6gAz9BGxjfNwuVh3RKzgaGfJ6UDJO/toJ NTuNwh97ruO5TFuST1zO5+868mfqqi5CbCrRadqThnsNowH2FqSv1ZI8t90SmU7YZ8Sb3Juw T98bkFdkGpMrmsbmm1YUci0rLr7w9FgNHu2se3R05UTq2aAK6VsOp7A7HDe4bFDM8yubJv+s GFMJz8edjxQ7uEtre3Kc28nsmN07Lz+KZNY0Ifd27aEU8SKP xYvc0ekYD6OI1CQ64tOBOgdf pzzKpZTtg8hQpzetfZxD+W3hxlUKGzdN7j6qYCoRqyWSo082wXHiGP6nvBFQvb9R5XlnZtXY MJpob0paWt+QX/QLVh2JqSSIbtwJp3HjwulOGdMWqF6nArTfy4i7QmM79OnVB24/DWFvu8dY N3k+tsWzkaKuBXMvNQRsXKM4qMy1rCC7OjzauvXR2eUnjJ46FM0frsGoGfvxTEa0Mp5fmUTF nPB83IoBqoTqtV5hvS9bFIzRgMOMkWin8wbX/tqC4w0t0aQq13QroGHvPuj8dA827NyN7eG6 6GFvzjzYaKFGn6U4e3EZ+htJnDzFY+vw8bjQcxz6m4tnd4oLLiZWC3oVJTKLScpffn8IqPS8 azNa5v5YsP13/Gx8HVv33hQzfjK1H23G8EGx6BnQj3mC4mbeH1Z4Lx23Zk1LdHbsDqcBk7Bq 01TYJG7AuCUn8YZho6Grq9x3o6pcy4owLoJHe0/YKNTJX11uDN+oAj5uDcMmcPCYjzUzGef2 P0dxMd/Zuwo8v4r0KBTH83EXgkQYURs9A//EtLpRGNvBGs6rbjLfoll48ugpM6cxCl3JtLWO Kcwrv8aTi9G4lqwDXclDplENFpZfSrWlZFy/ehpb1xzCE0lexQUric1FalIS3sIErZvWVpKG j/5gECjieS/Emy9cW6oudNIhompmtUi5iasxO7Bm38MP0v79Xjpu6ZuraWqBJnp5SL52G0nS FxQdK+RaVnGhqAgebbJdyh5m4mw+7O8mlnaR5+MWc4Mb1UQ1LU6xkvD8KqpICeM+Sz7uF7h+ sxbGHzqLyJmtEO/vgQl7XqBOXdZp3riKOJkFwjqw7MBs9npXcfjkf/kPmiDxJA7d4oYDXLDG hKXDUX3zFEzYmZifRnRNhf+UiAM7z0LQojf6tPlChQx8kveKQBHPu2Le/Mpsna1egdOMlmOx dHx1bB4zAzufyG0jfa8VExX+njtutoAQexZnXuvBoqs16isBRMS5/QB3bx5UwLWs4vCpSB5t oTdiqdJZx/zwEo7fTBM/4Nwi6n/oMK4frFjHrZTnl3GAJxzYgPVRpbzfNzcRsWdTUcXYCAZ4 hdRnmSJda7eFfQcNnF69DlFpbKEt+yVSX+ZBkCVA9Y5d0EHzDFaHHmMLdKw+aWmMuzoPWdny C3LGsLYzx6PfpmPq/ntsJPsMV8LnYt6R56wRJyEqcCPO59WH45RZ8LIipL96g6dRK4vhPpeC Uu1DRWVrw9JtEBx1zmDfCc65Bms/54/h6Fc/YrBrD/T4nxb2BczAqthnyErci+kTj0DLqKJY A23o201AqFoPI1sQXc/w2VcHw+cPR1uhKzO1K8ZnLA8EinzeX+D8li2IShS/1NMvYP+Z1hju 2lDK8XcN2E1bgPHV/8aYCX+rOUsrw4oWmLvVO2KqqZQxL/EorZo9gJpqgXSsPGhB8ELyG9uP bIxrU1PXJSKHqlkP6NQyd2qAGmQ7aTvdSGPOffM5t4Mp+vJShVzLsgqowaMtK4Cdibm+tUzI dsQMCgyYQpOWnRTtLX91mgJsa3NvC6k/HTLxOUxveD7uQkjKR6jaXkQL06Zk7TGDghaOJhe3 UIrN5HZQv6YH+/3J0bo3TVw4i3yH+tE24aIx51B6E3laGrL7okX6TfvR4lNPKS/tMm31sSU9 NCWPlafo9tkAaqejRQY2vhR28hBtCxpL9oZaBANb8gpcRWvWzCMPKwPGp21NHguWUqDfGOpv 25Ja9BxNwUfuib8/kK8Vf15eCKjefphGSnnzz1FAK9ZOON72WTNp0shptDI6me3P5xYrw8nH pgbBzIPFXaezAXaMj7822fhsolhVHIKrAUSJ6iSWz7MDMtRKNfB83EXCqQ4TWpEC+YufFQKf YvtRp07v2VTyqbU5no/7U7ujfH14BD5EBPgR94d4Vz5hndQZXXzCcPBVKyECn2L7UadO/Ii7 hA2HT84jwCPAI/C+ESiVLwm4NwYfeARURYBvL6oixadThADffoB37rjZIqcibPk4HgEeAR4B HoEyQoA3lZQRsLxYHgEeAR6BskKA77jLClleLo8AjwCPQBkhwHfcZQQsL5ZHgEeAR6CsEPg/ 6PGEU7sSLFoAAAAASUVORK5CYII= --B_3396331596_2372406-- From pfrejborg@gmail.com Tue Aug 16 09:51:28 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5F67E11E808E for ; Tue, 16 Aug 2011 09:51:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.365 X-Spam-Level: X-Spam-Status: No, score=-3.365 tagged_above=-999 required=5 tests=[AWL=0.233, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XzrqrUU3gS1T for ; Tue, 16 Aug 2011 09:51:26 -0700 (PDT) Received: from mail-wy0-f172.google.com (mail-wy0-f172.google.com [74.125.82.172]) by ietfa.amsl.com (Postfix) with ESMTP id 1F48B21F8B4A for ; Tue, 16 Aug 2011 09:51:25 -0700 (PDT) Received: by wyg8 with SMTP id 8so62286wyg.31 for ; Tue, 16 Aug 2011 09:52:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AZNLhavwwjcMKCsACv+zXSb29nQgc/wwpPXqaKOwifU=; b=hBX7P/y/jDbWmB79Bb6smjXrhW+6pEesTQ9Tqh3dzUFM/GMpxIz8p+7kA10Fq58dM7 bNTrhaB37CVK58BQinhmD+knJJ5vVS965lO3y/aji5md3wFrrMljSzYjE4ER+ijdm/Kf bZRxg/HkZxTMmOy9G23zqWFYGvjloohSwAujw= MIME-Version: 1.0 Received: by 10.227.60.201 with SMTP id q9mr3767570wbh.52.1313513534228; Tue, 16 Aug 2011 09:52:14 -0700 (PDT) Received: by 10.227.38.206 with HTTP; Tue, 16 Aug 2011 09:52:14 -0700 (PDT) In-Reply-To: References: Date: Tue, 16 Aug 2011 19:52:14 +0300 Message-ID: From: Patrick Frejborg To: Gary Berger Content-Type: multipart/related; boundary=485b3973eca19d87c104aaa23334 Cc: armd@ietf.org Subject: Re: [armd] Fwd: New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Aug 2011 16:51:28 -0000 --485b3973eca19d87c104aaa23334 Content-Type: multipart/alternative; boundary=485b3973eca19d87be04aaa23333 --485b3973eca19d87be04aaa23333 Content-Type: text/plain; charset=ISO-8859-1 Hi Gary, our discussion is way off the topic for ARMD, not sure if we can continue our discussion on the list - but I continue a while, and if somebody feel that we shouldn't have the discussion, please drop a note and I take our discussion off the list. First of all, I'm not the inventor behind SCAFFOLD. I have focused on RFC 6303 which is a controversial research work from RGG and focusing on routing scalability. I stumbled on SCAFFOLD somewhere in February-April, found it interesting and also "compatible" with RFC 6306 - then I found out what to do with the identifier. Comments inline On Tue, Aug 16, 2011 at 4:24 PM, Gary Berger wrote: > Patrick, > > Where to start.. Ugh.. Let me just say I think the approach Scaffold takes > is innovative especially in the light of bringing together applications, > services and networking.. Certainly we have learned that topology is a lot > more complicated than just a physical graph and moving towards decoupling > the physical topology will go along way with providing scalable services. > > As per SCAFFOLD, I have to say I have not thought through exhaustively the > design but I can give you some things that just give me pause. > > - Some of the implementation issues are scary.. I.e. Changing the data > plane, control plane, the network stack and programming model.. > - These are HUGE changes, not necessarily impossible but definitely > a challenge > > If you have a look on RFC 6306, you'll notice that the current data plane is compatible - no changes are needed but the middleboxes and some additional nodes needs to be changed/added. Only one change is needed to the control plane, i.e.to add support for ICMP extensions, but that shouldn't be a major showstopper for experiments. New extensions are needed to the networking stack, some applications will suffer (but what ever is done, some applications will suffer). My knowledge about the programming model is very limited, thus no comments on that. > > - Kudos for the innovative thinking. I think there is growing practice > in looking at the network stack especially from a messaging point of view > instead of an opaque streaming interface[1] but it still is very scary and > if we are going to go to this extent maybe we should look at the entire > protocol stack.. > - Pushing the bindings into the server are creative and maybe a first > attempt to look at these alternatives while they mature possibly migrating > into the dataplane of the network. Operationally people are going to get a > bit burned without the proper OAM tools. > - I do like that you have added the elusive application service. > Having a proper binding for this is important as well as the appropriate > lookup services. Ip's "well-known" ports was a hack because we had no such > directory. You do answer Saltzers first objective: > > 1. A given service may run at one or more nodes, and may need to move from > one node to another without losing its identity as a service.[3] > > - But again we are missing the Node Address. Saltzer missed this, if > you believe the conjecture it was because multi-homed node systems were not > envisioned at this point. The IP Address is naming the subnetwork attachment > point, this is ultimately the scaling problem..[2] What is the "identity as > a node" he is talking about? An identity needs to be unique within the scope > of the layer (we don't have this with MAC or IP today). > > Hmm. here I disagree :-) This is the thinking from the past - what we have been trying to do in the data centers during the past +10 years is to break up this host-to-host architecture. When I have designed data centers you get to a point when you have to decide what to do with the most important services that need an availibilty of five nines. There is desire to break the host-to-host concept and replace it with a host-to-many-hosts concept - thus you implement some kind of load balancing solution (SLB, DNS, etc) in front of your real servers so you can have a host-to-many-hosts solution. But what I realized, after reading the SCAFFOLD paper, is what we have been trying to accomplish that past +10 years is to create a host-to-service solution with help of these load balancing kludges - e.g. writing this e-mail I don't care on which node the mail gets composed, the most important thing is that the service is available! Did Saltzer think of online services, where there online services on the Internet at that time? I believe that mainframes (node) to produce a high available service was mainstream at that time - the online service concept (of the magnitude we see today) has arrived later. The demand for mission-critical services are growing, and here we sit with an outdated stack in our hands, trying to respond to the new requirements. Also, if you read RFC 6306 (e.g. Appendix C) you'll notice that the node has a two level "attachment point" - one level describing how the node is connected to the local network and how the local network is connected to the Internet. This model is like the highway where you have several exits to a destination and the nodes can leverage this information to reach another node. The current model that we use, is more or less like the railway - nodes have no choice to make a decision which path is used to reach the destination, instead the railway (network) tries to figure out which path works best for the nodes. If we could use a two-level addressing scheme we could replace multi-homing with multi-pathing and let the transport protocol figure out which path works best for them (like the driver on the highway). Oh, and the transport protocol is pretty good on liveness detection also. > 2. A given node may be connected to one or more network attachment points, > and may need to move from one attachment point to another without losing its > identity as a node.[3] > > - > > In the paper it doesn't seem you have the concept of an identity in fact under VM migration you talk about cycling the network interface to get a new Scaffold address.. In which case where is the identity of the node? > > > > - There is also overloading the hostAddr name with ASAddr and sockID in > the example implementation.. This in practice may be challenging but I > assume this is just an experimental viewpoint. > > Yes, drop this packet header structure and use the one described in RFC 6306 - it is better suited for the current Internet and there is placeholder for an identifier. MPTCP do have a token that can be used as a session identifier - then the application can be moved from one node to another - and I don't care if this e-mail gets composed in Finland or Belgium, as long as the service is available. For these two > > "IMHO, if you do routing based on identifiers then there is really > no decoupling of locators and identifiers - of course this depends on one's > definition of Loc/ID split" > > I don't think this has to do with loc/id split, that in fact maybe a false > path[2].. But you do routing on a "name/address" relative to the layer. > Mux/Demux is just composition/decomposition in the network domain. > > If we take the perspective that devices are multi-homed as the canonical > model; examples being a device moving from AP to AP, Wi-fi to 3G/4G or DC > to DC. Scaffold goes to a lot of effort to make this seamless but the > complexity should be an indicator something is wrong. > I don't think SCAFFOLD address this issue - this is better suited for MPTCP and they have work in progress to solve roaming issues. > > I put together a short example here: > http://www.slideshare.net/gaberger/scaffold-8866328 which describes > Saltzer's model as well as Scaffold. I also took a stab at what it would > actually look like if you incorporated a Node Identifier in the address > architecture. I am not saying that this is an easy topic, its been 40 years > in the making, the question is where do we concentrate our efforts in > properly designing the address architecture and managing the layer > bindings.. The latter as a charter in this group "the what", the former as a > necessity in understanding "the why". > I'll have a look on that - I do have a presentation about hIPv4 and uses cases but it need to be updated before releasing it. Patrick --485b3973eca19d87be04aaa23333 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Gary,

    our discussion is way off the topic for ARMD, not sure if w= e can continue our discussion on the list - but I continue a while, and if = somebody feel that we shouldn't have the discussion, please drop a note= and I take our discussion off the list.

    First of all, I'm not the inventor behind SCAFFOLD. I have focused = on RFC 6303 which is a controversial research work from RGG and focusing on= routing scalability. I stumbled on SCAFFOLD somewhere in February-April, f= ound it interesting and also "compatible" with RFC 6306 - then I = found out what to do with the identifier.

    Comments inline

    On Tue, Aug 16, 2011 = at 4:24 PM, Gary Berger <gaberger@cisco.com> wrote:
    Patrick,

    Wher= e to start.. Ugh.. Let me just say I think the approach Scaffold takes is i= nnovative especially in the light of bringing together applications, servic= es and networking.. Certainly we have learned that topology is a lot more c= omplicated than just a physical graph and moving towards decoupling the phy= sical topology will go along way with providing scalable services.

    As per SCAFFOLD, I have to say I have not thought throu= gh exhaustively the design but I can give you some things that just give me= pause.
    • Some of the implementation issues are scary.. I.e. Cha= nging the data plane, control plane, the network stack and programming mode= l..=A0
      • These are HUGE changes, not necessarily impossible but definitely a= challenge
    If you have a look o= n RFC 6306, you'll notice that the current data plane is compatible - n= o changes are needed but the middleboxes and some additional nodes needs to= be changed/added. Only one change is needed to the control plane, i.e.to add support for ICMP extensions, but that sho= uldn't be a major showstopper for experiments. New extensions are neede= d to the networking stack, some applications will suffer (but what ever is = done, some applications will suffer). My knowledge about the programming mo= del is very limited, thus no comments on that.
    =A0
      • Kudos for the innovative thinking. I think there is growin= g practice in looking at the network stack especially from a messaging poin= t of view instead of an opaque streaming interface[1] but it still is very = scary and if we are going to go to this extent maybe we should look at the = entire protocol stack..
    • Pushing the bindings into the server are creative and maybe a firs= t attempt to look at these alternatives while they mature possibly migratin= g into the dataplane of the network. Operationally people are going to get = a bit burned without the proper OAM tools.
    • =A0I do like that you have added the elusive application service. Havin= g a proper binding for this is important as well as the appropriate lookup = services. Ip's "well-known" ports was a hack because we had n= o such directory. You do answer Saltzers first objective:
    1. A given service may run at one or more nodes, and may need to m= ove from one node to another without losing its identity as a service= .[3]
    • But again we are missing the Node Address. Saltzer missed this, if = you believe the conjecture it was because multi-homed node systems were not= envisioned at this point. The IP Address is naming the subnetwork attachme= nt point, this is ultimately the scaling problem..[2] What is the "ide= ntity as a node" he is talking about? An identity needs to be unique w= ithin the scope of the layer (we don't have this with MAC or IP today).=
    Hmm. here I disagree :-)

    This is = the thinking from the past - what we have been trying to do in the data cen= ters during the past +10 years is to break up this host-to-host architectur= e. When I have designed data centers you get to a point when you have to de= cide what to do with the most important services that need an availibilty o= f five nines. There is desire to break the host-to-host concept and replace= it with a host-to-many-hosts concept - thus you implement some kind of loa= d balancing solution (SLB, DNS, etc) in front of your real servers so you c= an have a host-to-many-hosts solution. But what I realized, after reading t= he SCAFFOLD paper, is what we have been trying to accomplish that past +10 = years is to create a host-to-service solution with help of these load balan= cing kludges - e.g. writing this e-mail I don't care on which node the = mail gets composed, the most important thing is that the service is availab= le!
    Did Saltzer think of online services, where there online services on the In= ternet at that time? I believe that mainframes (node) to produce a high ava= ilable service was mainstream at that time - the online service concept (of= the magnitude we see today) has arrived later. The demand for mission-crit= ical services are growing, and here we sit with an outdated stack in our ha= nds, trying to respond to the new requirements.

    Also, if you read RFC 6306 (e.g. Appendix C) you'll notice that the= node has a two level "attachment point" - one level describing h= ow the node is connected to the local network and how the local network is = connected to the Internet. This model is like the highway where you have se= veral exits to a destination and the nodes can leverage this information to= reach another node. The current model that we use, is more or less like th= e railway - nodes have no choice to make a decision which path is used to r= each the destination, instead the railway (network) tries to figure out whi= ch path works best for the nodes. If we could use a two-level addressing sc= heme we could replace multi-homing with multi-pathing and let the transport= protocol figure out which path works best for them (like the driver on the= highway). Oh, and the transport protocol is pretty good on liveness detect= ion also.
    =A0
    2. A given node may be connected to one or more network attachment= points, and may need to move from one attachment point to another without = losing its identity as a node.[3]
    • In the paper it doesn=
9;t seem you have the concept of an identity in fact under VM migration you=
 talk about cycling the network interface to get a new Scaffold address.. I=
n which case where is the identity of the node?
    • There is also overloading the hostAddr name with AS= Addr and sockID in the example implementation.. This in practice may be cha= llenging but I assume this is just an experimental viewpoint.

    Yes, dro= p this packet header structure and use the one described in RFC 6306 - it i= s better suited for the current Internet and there is placeholder for an id= entifier.

    MPTCP do have a token that can be used as a session identifier - then t= he application can be moved from one node to another - and I don't care= if this e-mail gets composed in Finland or Belgium, as long as the service= is available.

    For these two

    "IMHO, if you do= routing based on identifiers then there is really no=A0decoupling of locat= ors and identifiers - of course this depends on=A0one's definition of L= oc/ID split"

    I don't think this has to do with loc/id spli= t, that in fact maybe a false path[2].. But you do routing on a "name/= address" relative to the layer. Mux/Demux is just composition/decompos= ition in the network domain.

    If we take the perspective that devices are multi-homed= as the canonical model; =A0examples being =A0a device moving from AP to AP= , Wi-fi to 3G/4G or DC to DC. Scaffold goes to a lot of effort to make this= seamless but the complexity should be an indicator something is wrong.

    I don't think SCAFFOLD address this i= ssue - this is better suited for MPTCP and they have work in progress to so= lve roaming issues.
    =A0

    I put together a= short example here:=A0http://www.slideshare.net/gaberger/scaffold-8= 866328=A0which describes Saltzer's model as well as Scaffold. I als= o took a stab at what it would actually look like if you incorporated a Nod= e Identifier in the address architecture. I am not saying that this is an e= asy topic, its been 40 years in the making, the question is where do we con= centrate our efforts in properly designing the address architecture and man= aging the layer bindings.. The latter as a charter in this group "the = what", the former as a necessity in understanding "the why".=

    I'll have a look on that - I do have = a presentation about hIPv4 and uses cases but it need to be updated before = releasing it.

    Patrick


    --485b3973eca19d87be04aaa23333-- --485b3973eca19d87c104aaa23334 Content-Type: image/png; name="7B78AD1C-E150-412D-A64A-DC1A9E885956.png" Content-Transfer-Encoding: base64 Content-ID: <7B78AD1C-E150-412D-A64A-DC1A9E885956> X-Attachment-Id: 29e2b99e3b8ae4e5_0.1 iVBORw0KGgoAAAANSUhEUgAAAW4AAABrCAYAAABXGGiIAAAXU2lDQ1BJQ0MgUHJvZmlsZQAAeAHV WXk8Vd3X3+fcEfcarnkm8zwPl8zzPI+pXPNM1xRlSFKokCGFFBIpGoWEDCnJlChFikKpNJjJe+h5 nt/v/fze97/3n3d/Pmef71lr7bXXOWudvc9aBwDOeUpERAjMCEBoWBTV3kRfwNXNXQD3GuAAAeCB CKCneEdG6NnaWoL/tS2NAGib+VxmW9f/KvY/M5h8fCO9AYBsEbaXT6R3KILvAADre0dQowBArSD0 gdioCASjHyOYhYoYiOA329j/D17Yxl47GIPekXG0NwAAwwEAnkChUP0BIAojdIEYb39ED9EQACwp zCcwDABmVwRrewdQfADgLERkpENDw7dxJ4LFvf5Nj/+/YQrF6x+dFIr/P/jPvSAjkYkNAyMjQihx Oxf/l11oSDTyvHYaCekJYSHW275hQ45ZH4qhBXLmQY7fESE7PkNkIC7fMCcHhLaNpcO8rG3+wtp+ VGN7BCNjIduIKP1tjDwzyC8iytbxL3pifICBNYIJCD3PN9Lobz0Xgyjm2z6jR+i3qNH2TggWRnBb ZIyDEYKRiIKm4wMcXf6S+eXja/gXHYb9Ao3N/sjApMAos+25WBCf7woOt9i2AZkLVgUWIAT4gmhA RfowIAMsgQEw/KuXAX6AgnBiEF4kCAYfERyKjAhHxoQjWOAvOYP/oBjvjPNHxv13jQLAG5GN/mfO P7MJIHP+rTMQ+CD4bzoFmWObt21d5P7AlH/N+bfEtr4da+Rr5OfkN/62CS2KVkSroPXRWmhtNBkI oNnQXEAGrYxWR+uhddCaCI8MjME0otn/bxu39Yfe8ospDI/TcA5AuNv37vU3FzjvSAf+c/0fFoDA vvl7839bAECU70HkPQDAIDwijhroHxAloIe8ub7SAmZh3rLSAoryCgrb7P83bXvN+mPsT/udtQhi 6/8XjYKsSeqKANDq/4sWjrwjtflI6J/7F00UiXNOMgC37L2jqTF/9KG3TxhACxiQCOUEfEAIiCPP WRGoAk2gC4yAObABjsAN7EPiJwCJQSqIBYfBEZAGMkE2yAfnQSkoB1XgOrgF7oFm8BA8Ak/BAHgB XoNJ8AF8BgtgCaxDEISDiBAzxAnxQyKQFKQIqUPakBFkCdlDbpAn5A+FQdHQYegolAmdgc5Dl6Bq 6CbUCD2EnkCD0CvoHTQH/YDWYBRMgFlgXlgUloPVYT3YAnaE98L+8AE4Hk6FT8OFcBl8Da6HH8JP 4RfwJPwZXkQBFB2KDSWIkkGpowxQNih3lB+KikpEZaAKUGWoWlQTqhv1HDWJmketorFoZrQAWgaJ U1O0E9obfQCdiD6JPo+uQtejO9HP0e/QC+jfGCKGByOF0cCYYVwx/phYTBqmAFOJuYvpwrzAfMAs YbFYNqwYVg1rinXDBmEPYU9iS7B12DbsIHYKu4jD4ThxUjgtnA2OgovCpeHO4a7hWnFDuA+4FTwd nh+viDfGu+PD8Cn4AvxVfAt+CD+DX6dhpBGh0aCxofGhiaPJoqmgaaLpp/lAs07LRCtGq0XrSBtE e4S2kLaWtov2De1POjq6XXRkOju6QLpkukK6G3SP6d7RrRJIBEmCAcGDEE04TbhCaCO8IvwkEomi RF2iOzGKeJpYTewgThBX6JnpZenN6H3ok+iL6Ovph+i/MtAwiDDoMexjiGcoYLjN0M8wz0jDKMpo wEhhTGQsYmxkHGVcZGJmUmCyYQplOsl0lekJ0ywJRxIlGZF8SKmkclIHaYoZxSzEbMDszXyUuYK5 i/kDC5ZFjMWMJYglk+U6Sx/LAiuJVZnVmfUgaxHrA9ZJNhSbKJsZWwhbFtstthG2NXZedj12X/Z0 9lr2IfZlDm4OXQ5fjgyOOo4XHGucApxGnMGcOZz3OMe50FySXHZcsVwXuLq45rlZuDW5vbkzuG9x j/HAPJI89jyHeMp5enkWefl4TXgjeM/xdvDO87Hx6fIF8eXxtfDN8TPza/MH8ufxt/J/EmAV0BMI ESgU6BRYEOQRNBWMFrwk2Ce4vktsl9OulF11u8aFaIXUhfyE8oTahRaE+YWthA8L1wiPidCIqIsE iJwV6RZZFhUTdRE9LnpPdFaMQ8xMLF6sRuyNOFFcR/yAeJn4sARWQl0iWKJEYkASllSRDJAskuyX gqVUpQKlSqQGpTHSZOkw6TLpURmCjJ5MjEyNzDtZNllL2RTZe7Jf5YTl3OVy5LrlfsuryIfIV8i/ ViApmCukKDQp/FCUVPRWLFIcViIqGSslKTUofVeWUvZVvqD8UoVZxUrluEq7yqaqmipVtVZ1Tk1Y zVOtWG1UnUXdVv2k+mMyhqxPTiI3k1c1VDWiNG5pfNOU0QzWvKo5u1tst+/uit1TWru0KFqXtCa1 BbQ9tS9qT+oI6lB0ynTe6wrp+uhW6s7oSegF6V3T+6ovr0/Vv6u/bKBhkGDQZogyNDHMMOwzIhk5 GZ03mjDeZexvXGO8YKJicsikzRRjamGaYzpqxmvmbVZttmCuZp5g3mlBsHCwOG/x3lLSkmrZZAVb mVvlWr2xFrEOs75nA2zMbHJtxm3FbA/Y3rfD2tnaFdl9tFewP2zf7cDssN/hqsOSo75jluNrJ3Gn aKd2ZwZnD+dq52UXQ5czLpOucq4Jrk/duNwC3Rrcce7O7pXui3uM9uTv+eCh4pHmMbJXbO/BvU/2 ce0L2fdgP8N+yv7bnhhPF8+rnhsUG0oZZdHLzKvYa8HbwPus92cfXZ88nzlfLd8zvjN+Wn5n/Gb9 tfxz/ecCdAIKAuYDDQLPB34PMg0qDVoOtgm+ErwV4hJSF4oP9QxtDCOFBYd1hvOFHwwfjJCKSIuY PKBxIP/AAtWCWhkJRe6NbIhiQT4Oe6PFo49Fv4vRjimKWYl1jr19kOlg2MHeOMm49LiZeOP4y4fQ h7wPtR8WPHzk8LsEvYRLiVCiV2J7klBSatKHZJPkqiO0R4KPPEuRTzmT8uuoy9GmVN7U5NSpYybH atLo06hpo8c1j5eeQJ8IPNGXrpR+Lv13hk9GT6Z8ZkHmxknvkz2nFE4Vnto67Xe6L0s160I2Njss eyRHJ6fqDNOZ+DNTuVa59XkCeRl5v/L35z8pUC4oPUt7NvrsZKFlYcM54XPZ5zbOB5x/UaRfVFfM U5xevFziUzJ0QfdCbSlvaWbp2sXAiy8vmVyqLxMtKyjHlseUf6xwrui+rH65upKrMrNy80rYlckq +6rOarXq6qs8V7Nq4JromrlrHtcGrhteb6iVqb1Ux1aXeQPciL7x6abnzZFbFrfab6vfrr0jcqf4 LvPdjHqoPq5+4V7AvckGt4bBRvPG9ibNprv3Ze9faRZsLnrA+iCrhbYltWWrNb51sS2ibf6h/8Op 9v3trztcO4Y77Tr7uiy6Hj8yftTRrdfd+ljrcfMTjSeNPeo9956qPq3vVem9+0zl2d0+1b76frX+ hgHyQNPg7sGWIZ2hh88Nnz8aNht++sL6xeCI08jLUY/RyZc+L2dfhbz6PhYztv46+Q3mTcY443jB BM9E2VuJt3WTqpMP3hm+633v8P71lPfU5+nI6Y0PqR+JHwtm+GeqZxVnm+eM5wY+7fn04XPE5/X5 tC9MX4q/in+98033W++C68KH79TvWz9O/uT8eeWX8q/2RdvFiaXQpfXljBXOlapV9dXuNZe1mfXY DdxG4abEZtNvi99vtkK3tiIoVMrOtwAK6WE/PwB+XEFyCDckdxhAvino/+QUOxJIugIhMgh2hmSh z3An6ijaAaOLFcNx4Tlo+Gm16KwJwcRs+kaGeSYZki9zOcsUmyR7HEcrFwO3C08F70/+3QKpgs+E mITtRU6JPhUHEkqSflJnpXtkluXE5e0UkhVrlF6owKoKanvVM8j1Gu92E7XUtT110nVv6r0xwBuq GnkbZ5s0mE6YQxbCliZWQdZZNndsX9qtOLA5KjnZOIe6nHKtdXvq/m7Pgsfy3vX9wJOWwukl463n Y++738/XnxLgELg7SCAYCp4MaQ29GHY0PCDC9oA6VSASH/ktaiS6JaYqNvdgYlxIvNshs8NaCWqJ qknkZL0jFikuR31To44dS8s7XnHidnpbRm/myMm3p2ZOf8n6kb2Ys3RmMXcxb60AfZa1UPqcyXnv oqTiwpLaC62lTy8OXxormyyfq/hVibrCWiVZrX/Voyb2Wt71W7WDdd9vMt1Suu1wJ/Judn31vaaG h40dTW337zfffVDXUt1a3lbyML89o+NwZ1CXwyPVbo7u1ceTT/p7Hj3t6H34rLmvrr9wIHLQYIg4 9Px50bDfC5URzMjoaNXLmFe6Y9ixbiS+VN7MjOdMaE5MvT01qTn5+V3pe/sp1FTdtNP06oe8j9If W2fsZ6Znj83JzU1/qvocNq80v/il7qv3N6ZvdxdsFz5+P/yD/cejn1m/whYpS35IHE2vdW3Kbm3t +F8IugEHoRRRs+ibmGSsK04LL0MjRitGt4sgT9Sgt2PwZkxkKiW1MM+xMrKps1M4TnDe4ZrgoeNV 4tvDnyxwSbB112uhRRE6UX4xFXEzCU/JOKlc6ZsyvbKz8mgFQcXdSu7KUSqZqhVqjerPyO81fu3G anFrK+hY6YboZenfMBgw/GKMN+E1VTQzMney8LYMszponWhz1PaYXZr9CYcMx5NOGc6pLnGuAW6O 7oZ7dDyM97rvi92f73mD0u7V493lc9e32O+Qv0uAfCAhcD5oILgppDq0KCwrPCWCesCDqhvJH7ke 9SL6ekxarNdBozj5eOFDvIc5E1gTGZOwSUvJ74/0pNw8mp8ae2xvmvlxwxOW6ZSMI5mXTz46NXH6 a9Zi9nLO4pmfuQt5X/LnC76eXTnHeJ5cFFZcWdJ3Yap07uKHS2/LXpUPVjy+3FLZfKWn6stVwZq9 14qvv6pjuWF98wSyeq3ela33uVfUMNSEua/cvP/BsZbK1ua2lodX27M7Ejpju5IfZXWXPC5/cqHn 9NPoXodnMn3ovrH+WwOZg0FDds+Nho1e2I14jUa/TH11fCzhtd8bg3Gu8fmJxrfHJ13fybzHv/84 1TFd8uHAR90ZwszwbPlc0qfAzz7zAV9Cv0Z8i1iI+E79EfMz7lfsYuCSyTLD8u0Vo5Wnq+6rX9YG NgibYzv+lwKdkAX0EvZFYVFZaCl0PyYeK4edw13GB9DI0azS9tCVEmKJ9vSKDPQMS4yvmNpI1cy5 LAms/mz27FocEpysnBtcs9xDPC28tXzl/EUCBYJ5u7KE0oRjRCiiRmICYivivRKlkpFSptKCMrDM nOyo3GP5JoWrioVKycqeKmRVrGq/Wr66K5mT/EqjRNNnt6IWVmtCu14nSzdAz1Bf1IDREBj+NJox HjG5b1pg5msuYj5pUWhpY4Wz6rA+amNmy2H7ya7FPtchwFHTieg04Xzd5bCruRur21v3qj3hyP6/ uvfBvuT9Bp54z0FKsVew924fgs+Y7xW/A/7q/hsBrYHJQbrBILgt5EioQRg6rCv8WIRexMqBGqob smdXR9lE/YoujNkdMxGbfJD34IM4z3i2+LFDNYePJrgmiicuJXUk5x7xTzE8KpnKcYwuDaT9Oj51 4ll6XcbJTMpJ5VO4U2Onb2RlZAfnmJwhnXmUuyd3Pi8+X69A/+yJc/jzGUXTJZwXFEvJF8mXVMrk ysUrBC9zVjJdoa2iqWZAIknrmuf147XX657f2Lglftv9zpm7g/dYGtwai5tGmzEPJFpMWr3akh5e aG/peNu59Uiw2+Cx/5OTPTefjvRu9kn07xk4OzjxXHH41Iuvow4vG8cEX+ePy72lfxc7nTkb98X6 x9Kq3bb//9SWtvcErCoAuUie6XwKOeYAyLkHgOh9ANhpAbAlAuBIBvDxWgCb1AIo+Ng/+wcE0Ej1 jQGpzPADCaCMZJqWwB0EgoMgHckor4EWMITUPDYgEiQB6SL5YSR0CskHu6ApGIIFYX3YBz6OZHlD 8BpKCGWFikdVoUbReLQGOhRdjn6FIWEskIysAwthdbHJ2HYcBmeOy8a9xAviQ/CNNDgaF5oqmjVa K9pLtMt01nRVBDTBi9BBFCGmE7/SO9I3I5lODiNgPMA4zeTG1E8yJj1gVmeuZ9Fg6WC1Z51ii2bH shdwiHI0cFpzznKd4FbgnuIp5fXik+Jb4X8kkC/os0tZCCv0Wvi2SJZoiJiFuJQEUWJB8oXUfekL MomyHnJkeRb5BYVnileV0pUDVMxVZdVY1bbUv5AnNIY0e3Z3aXVqd+v06Y7pzeovGQIjLLLO4U3x ZjTmBAsWS0ErZWtrmzDbPLtm+w+ORCdlZzeXBNeLbp3uMx50e+X3Oe8/7FlB6fNa8RH2dfA75t8c sBZkEHwuZDXMO3zogDG1OUo5ui5W5uDN+N2HBhLCk3iSR1LyUi2PLR3PS5fO6Drpe5o1623Os9zx /K1CgfPkYssL+y/GlV2sGLsiU33xmnzt5M1Ld/bdo2usbd7bKtXO32X8uKyX0C8+uDScMyr+avDN hbdn3w999Jxb/UL6du0H+CW/RF7eWs1Ya1gf3ri/Wf47YkttZ/2AdmoOJMANRJFagw6wAh5IbSER 5IAK0Aj6kbrBJsQGyUHmkB9SEShDqgDvYTQsBlvCVPg83AF/Q/GgLFCHUXWoaTQX2h6die7CQBgt zCHMfcwGVgd7FPsEx4hzw13G/cDr4XPxH2k0aXJp5mmNEZ9v0LnS3UEyYSphmEgmXqSnoz9IP8Pg xtDHaMzYxqTN1EoyIPUwOzCPI5npGmsWmyTbU/YDHGwc9Zx2nB+54riJ3BU8ujzTvDl85vz0/OMC twVP7woU0hfmEP4s8kA0W8xPXF9CRJIkhZfGyOBl6eVI8kwKeIVVxVmlUeUelYeqD9V61F+Tf2jS 75bXstMO1InSpeoF6LsamBiSjZSN1U1MTPebJZpfsui2XLDmtjGyDUb2tDyHs475TnnOF11aXb+7 q+xJ9ni2j29/lGe/l5C3n0++712/Pv/pgPUgtmClEMfQmLDz4W0Rn6jskcZRMdFXYsYOMsZZxWcd epkgmpiQNHXE/yhjak9a1Als+vFM9MkTp7mzOnJScl3zDc5qntMs0iwhl0pcQpc9qoip5L7yoNqr hvXaeG3Xjf5bi3cV7h1ufNrM0GLYRm2v7Jzr1n9yq1ehr3hgfOjX8PeRmZdTY7Nvfr2F3tFOsXwQ njGdK5hX+5bxs3I5ZLVvPXWjY/PX79Ud/8PI28+EVJtkgDawQ2qdCaAA3AC94BNEg9SGrCAqVAi1 QZ9gNtgQjoIr4TEUE8oMlYpqQ20ilZl4dBN6A6OHycCMYiWwR7DjOG1cGR6PD8cP05BpSmhhpBby gs6Q7j6BTHhItCV+pE9hEGRoY/RgXGLKJsmQnjGHsRBZqlj1Wd+wxbHzsfdxnOb04tLnluRh4Vnn Hedr4D8jECpouUteiEMYK7wq8l30m9hP8U1JeilhaV0ZT9lkuRL5BoXnij+VuVTMVFPUOsgEDQ/N G1o45Fu1RW+Xfq4hm1GtibsZk/mg5XnrcFsne0WHMSd3515XU7fne/w8VvYd9YQoEV4vfNR8i/1p Ao4E0QaXh1qFg4h71PAovuiO2Og4n0NfEyuS446MpGykwsfwaYzHlU5Epg9nOp2cO30iWzbnVe6J fM2C74XV5/cV05ZcKVW7+KBMp7ztsmFlT5Vt9XCN47WBWuO6xpvit87ewd9NqN9oSG8SvT/wIKVV tW2uvbjT5hG6+/6TyKdSvdN9FwZch1ieD73IGjV/uTV27Y3N+Ozb6MnN9ynTqA8pM/Ds0U/oz0nz X78af4tbKPl+6kf0T8Ofy7+uLlovvl4KWFpajlmeW/FY6V81WK1ZI65FrA2tq6wXrn/fMNso21jf dNy8/hv12/X3tS1oy2nr6rb/I/2UkHol0iCCPlJ+nNja+ikKAO4MAJs5W1vrZVtbm+VIsoH8A2kL +fO/YlsYi9Tci99uo8fFN5K3z//e/gu44Xw0bQthrwAAIABJREFUeAHtXQdcFMcX/qiKGgQ7GMEI Bhs2FKMiWP6AiYLGgBhFo6BRsIGxxQIWrFgADYkldlBjiRILVuwKNmwYFSwYRUFQEZV67z97Be6O OzhOwLbz+8Htzs68efPt7OzMm9nvaRAL4AOPAI8AjwCPwEeDgOZHoymvKI8AjwCPAI+AEAG+4+Yb Ao8AjwCPwEeGAN9xf2Q3jFeXR4BHgEdA+10h0NDQeFcRfH4eAR4BHoHPGoGSLjW+c8fNoV3SQj/r O/SZV5570fPt5TNvBO9Q/U+x/agz+P0kTSX0PA4H1/iik6E2NDSqwLzvfOyIvoLL/yxAX/MqLE4b 1Rx8EBQ8E0Md7dDDcz7+SXgt25yyE3F65VA009YQpXf0R0RcMtLi/oa/Y10Wx+K1W8Fz5Uk8zH6X jTm5eH5zL+Y4mTCZxnANfyirR4nP3uBhTBgm29UR6qjdzB0BISEIWTQNHp1MoV2pH8KfPUPMHAdU q+uGNfFZSkp4g8SIsWjO1b+RP87nSSWjNNyMnAOnagxfoTzpi1Lp3vWQUhCzchwGjZyF4OBZGDnw Z0z09MS881L3KjsJMet+hZvrMPgtCsIivzHwnvo7/vrdF6M2y2FJ8VjXzxvhSTmymmU/REz4JNgJ 28sXaOY+AyEhwUzWEHSqWwmVXFfjUrGYijCgjNs4sHwCBg70xZzgIAR4/wi3yesR81SCs9z9Me+H wB3Rsm2Iw/dAENxl2uoseDt/h77jQnE08Y2s/sIz1o7i/kFg38bQ5tpmNQf4BrHyh36Lb3oMw5x/ 7iBbQa5PNkrmnor7gJiHhTHIOIM5diao22s14gUlRENRXnqOu3ef4116BJW0YKOfdwqskHfKX2aZ c8+Sn4Uew8+IXMISxcUkUpiLEYvTIwu/s5RLefQyciTVY3XQ+nocHXmZJ6uOQhm5lBLmRnosDyz8 KCZXNot6ZxkU42clp6t6kkS5JPJAei5hlCIRlXWJArsMobDke7TbsylpGThS4OWXkquFf1PCyEVP ST0l2Oi5UViK6iCo3l6y6XFYfzI08aWjbwQi3V6dpgDbruQXkyE6z0ugbQObkJahK629nyXWX0BZ t1eQk6Gx1H0XXRLcX0EOOjXIYcUtEksU52E/kvrItBcm6/pi6tKXw7AYTBkGgtQo8retTVqWU+jk K3FbyoujFY5GpGU2hDbdEeutTFaBNuxIvq2yqMzT5NeiCmkZO1OQwvumoG2+3Ede9XQIWi3J50h+ S5Ap6WM6Ub39sFopvKeytRUk7SBPsypkYBdIlzMLtQrZxHJnhfNm0O0Vfam1sG+RS1zEaYnqJJbz SY64GRAqBk1UrlYdlVjqvNvHcfzWWxXzfWzJBEg/dwTnMpvC/dc+qF/BBM6rY/Hm6T6Mb6n/gVYm DReOnMPzV0+R9CJXpGOVtvAY3hHspclCFuJX+2LYxlsw/N4d35vqitJAA7oNf8KS+d1QURwj+nmL m9s240jOMxxZE4GbxY6unuHcgcvIbNIXv/Y1h0R6gUg5THVTETXbFwEn3qLNoL7oUEX8aGlawG1k DxgmbMCoURtLPqorKBCo0BYjJjpD/3EEfhm8BDFZKozrKhuiZiVmEc27hf3H41FGcyNpLT+qY406 fbD6ZiqeRo1HywolW6+TzZuHtFOLMXTSXkjNB8sMi8+84xYg48ljpDJ4tSzsYd+M68LVCa+REOGH vn19MNv3JwxachJpwmeKxf8TgKHe/ggOHAWnb3ph3PZb4ulaNh5GToeDiSmaOYxFeLz0S0OJPHqG y0t6oRqbClfqMRW/+XZFXcM2GPnPw2KmZq/w78ETuJethTr2PdDmxS6MbG6IChWs4S80OygpTx4K wQNE/todJnWbw2HUJsSL+1P5ZKVzrosq+qyLfr4VXvYD4L/5Mp6TDoz6z8LktpXZfOcBju44gxcw ROt2jSD7+qkA88ELMMO2eoEqWVcQkVAbDjW0kHdhB7ZeeFlwTdFR1m0cPHgX2Rpfwv4Hazn5XAZp THuiA51D2KZrrGOsgaYWdVDwYGniCzMLmGnl4cXhrdj1r/R9VlRwUXE6qN2yFb7WYv3w9eM4kZBZ VGLRtYxk/JfKjCRaZuhouBe9JeatlBTEhY+UMoUxU8vlILH56zv4/eYDu7p10HxkOM7tFpvMzPvC 1+s7tDEzgbnzApxKEzUAyriG8HGDMNR/NnzdPTFDYpahZJya54GBkwMROO4nOPZayExuKra14mtW CinE5kC9CqjQmJkD39xFxEgrZmqqBAuveVg40A5mxk3hPO8I7kSHYqidOQyNO8I7nHuG5fJmXMLG ORtw+kUmnu5dhFFTdyCh2MGB+lUoaF/qy/gIc+bi1aWdWL5oAtx9j8CUdQZ/750GG72SvXFFFSdk xf4Oz5/+BPX+BdOnOuLttMGYsOcp8PII5noswMkKNhgyYRJ+rHceS/uPx7oH2aCkbfil/2L82/MP nDk4D92rFtxlpfI0aqB5p5aoxQrOjn+Bmt//iK61BChq4JX7JAZ/BY7D6N9uizt3NiI1aYuOFl+I 71sR+svc2RwkbZmC/vMfoOeaIzi4oAeqZpdlz22AjiMnwd2sItJvbMOs/h3RrPNIrIxJEdUj/V9E X3rONNSBfhU9Ns6WC7pGMPtS8iJmdbxyDqm2PhjhbMp6vWvYujMWirvQTDw5vw2B3r/gt0TFVuHC mAKCR7dx4zk3njXEl3WqyCijWcsY9bghe14ibiaky1wr6UmBrFuIjn2mPPurS9ixfCEmuf+KSFM3 +P+9E7+P7oHWtcRzB41qaOLQERb5UwltGDb/RnQ9+zGe1eyOn7oaISurIup3aCdKl10NXWdHICZy PL6MnIrvJ+zDS3qIf3x+xOAzzTHOfypmuGnit0GjERL7Em+PzccA/xR08hqLCYtn4qf6OshU9qwo r0kZXqkEE0nduFIqmKJDx4ZsdpWLtzlN4LF+FSZZJuOf6b9g6dMeCI1cgcE65/H72EU48LKibF49 KwwY2Ibl1UXtHuOxfM4PMCvD3rUMRZch3u8sWhtftO6DUeMX45/4u7gQNh1OZmwUpzSk49K6qfD2 9mZ/ozBx3QWpRY43uLn3b5x6wR5YI8kD+xCHj17HWz0LfPfzIPRsY4QK2W+Qkck655wkPEp+i9Qz B3H4uQHa2TRjo7lKqFZTUn4R8qT0023eEV1th2HjrUtY7VyvcMclTqtt8BVat7dCQzbSUhxUKw9I xpl9p/FcrwVsrKoBkim4YqGlEMteMObu+PPodsx1bcEweovHJ0Ix3HEgFseyzi8nA+mvVZ34v8T5 /alo3bk17Hp3ZWPi17gVvg0n0gtelgUK68Lgq5Zo37Yhm9kUxEofFY+pdOqyOtaD/hf5vW7hQr5o jR9GTcSCf27g0YUwzHDiOiQVg24j2HS1h8fGy7i1ug/qSHCoVBt12AKuZgNr2DXQxbPDx3Dx1j78 tuEGtOsZo5amJqrUMUb1F0exbH0MXma8wpuc/ZjUdyRCDmah+xx3VIhU8qyoqFr5JNNGpbq1Yaip x2Z9OuyFq42aRjWgq1sZ+nrsOXr9Cq9yVDBTlaGyn2nHXVJE9dF68ByEhoayv+VYOJh7s0pCGuJi 77Ep8iOc3PQbQsKfodP8RZjY1Riaul/DaXQ/WFxcAu85f+PmC8kILhP3rt4AN14sHFKVyyucuPiY ijVhbjMEU4Y3UtK5F6G/tPS8h7h6IVk6pmyPKQnnjrFZgkl3/PpXNG7HbICPTW1A2Cmcx9tqTdGu OWcgeY2Hj9OgqAvOVzD9LLbcZKOpOrqoausEZ2YuwcN/8OeeRAUmJk1UrNEQNj//guENK+SLkDlQ gKlm3a/R1JDJRSbSMyT3WZSLXrOXDDc50TJBYzNZo46MXO6E7Si5de0/qYGBbIoSyZLNWvpnKY/x KCERD+Q26bC3KlL+S4G243gEe7aAIGY1xnZvj86TwnH8spJnpfS1K1IiPb+Daw8V7c4pMtsHc1HZ MOyDUfDDV6QKjL5kI1A8RfV2fTH656/zO0h6shPDWKd50GELri9phj1xIeLqaKHyF1+Ae8wLB+Xy uLSqjjFl5VaE2bfO0PpC0Xu6slL98exygRgNPXxhwEYfiQVRZXrE7On7p+/E24ML0IXZIGu3HYgl m94irqE3LqW/RramDXp7dMac6L2sU7+Op9QCRpKRIacY2yaYkFyVmUsq4uXxo/jPqBZ2L+Pwz0Pd zg2htf0ODu86jSc/1pfNx+XlgqYZvu37BSQGJVGk9H85TKt2wAB3S6wP+g9xd5g5p4uhuB0QMu/+ i39ztGDg6IbejbilVeXLV4KEHZgVbobV8xpKFyY+ZguiV8/jkowsBcnKK6qmMeoye7dpoWahg5pf 1oJOsi5sfzuKi67rMN9vLv5c8Tsi2nFdjnahZ6W8VBaVk4WEbcEIbzgb8yzLt+TSKk3Rk1xasj8T OVVh5eTIFozYboXlqxGVloPsJ0exfNUp3D/6F8ITmCU1OxvZ6Y/xIFm0mETZmjCxtUMzlif61HWk 00s8eZQhxstAibxzkKQoObAaqGBhCQuFq+YqlqdZH7aOTaD19gpOXUwDPX+KR2/K0sbNapmyHUs3 xuWPPik7E1nalnAf0AFVwS1ALsRydwukH1qDFaeT80fP3GLZ+l9DEZ3JprOUiL3/6MFr5i8YM2YM +/PB1AluDPs8PN+5HH+cV7ZIWQUWLRqwUpQFeUxroMv0pZhmq4UjoeE4L1l4YIuof/++A4/MBmH5 8oEwL+qJy76FzTP/ROpXX8rtiBHrIEjA36sP4LkqshSpLXn55r5GhspmJilBb1KR9loAwd0YHL+r CYsfe6G9xXcYOagpch8+RrJAvNhv0BWjf7JCzolATNjxHGaOY7Fi+QhY6LdC/xG9FDwr79K2pfRT 6ZDY+tBfmLnkEb5qIDFPqpSxBInYetDeHTiu0BRXAjFFJS1ie6FKl5hsldKVZyJB2g068KcP2Rho sSe3Mpm5zqPt52Lp0m4/cjDk4rTIwMaH1p56QJLdv4X0y3pAp9ZKy5hPETeeUuqNCFro2ojYaJlg YEc+a09SYlYG3dk2gRzMDAj6FuTgu5XuZAlIkLSHfK1rkZZRO+rvv4X2h7qSsY4pOfgfpKQ8lids OFkaGFNTe0/y6d+E6VWFLL020Y20FwrlkSCFLi12JkNWtlZTT1qhUP/XlHjqT2JmBSaPpTNzo4Xb oiguLUdcRbY3OX4zeTatwq7Xo56Lz1GaQJH+GfRg9xiy1GL1NHSmxZdSSJD1L4V5tiYDI0uyHzGS +puxffJarckr7CqTUQhBhREqtxe2B9e/+wCa6DOY3H1mU1DwPPL1GEmBkXdl71nWf3RyxXhytW1P 9v2HkZeHO7n7/kZHHrwmDq/LKwaSWT1Xlu+GSMesRIre5EPWOlw/r0WGPefTifibcvd6Hm07GidV p+IwLaiq4NW/9E/gCHJxn0iBwUtptlc/6jtpHUU/yRQnkpflQtODgik4cCoNsTFh7aop228dT3GR S2mAWWWhjgY2I2hh0GzycXUmVxlZBeUS5VDajZ3k52AsvO8FbVP6xuRQ6sn55GRWi4wse5L3vNHU ldunL7y/SZR6aSn15J4PtufbY8UJ1q7FeSX7+Y27k9dEVjeb1mTr+RtFp4ralCD1LK3wdidPv1nk M8CD/CNus3vE9pSHj6HuQ3zIL3Ax+Xm4kXfYvyxeUVuT1lG6ToWPVW4/8s+vyxTWhoIocPpgsjFm 7Vb4fUAGxW8bQU0lbfxsNO3ybs3uAdcultDZy1vJ21LynJyi+IsSfFqT9+6rdEM6L3s+8tjzPsmW 3UP91uQprGth/RXFqFwnqcwa3DHLqHb4FD9BVRsMtTPmIPnKGdzGl2hoXh+1Kys2oqgtviwysq1e Vw7fBho0hHmD2qgsbaYoojy+vRQBTnlfyrqLM+fSUauBKUzqGUqt28gp8iwcriYDsN3EDzE3ZqLt e2yen2L7UadOvI1bro2+n1Md1GphJ9zm937KV6NUjVpoYc9tTOTDR4tAhQboYPfRav9ZK16Uxe2z BoavPI8AjwCHQDaeXL0HjbY2sKl9G3v+uoSnRW7h4VErDwR4U0l5oMyXkY+AOtPC/Mz8wWePwKfY ftSpEz/i/uwfBR4AHgEegY8NgVKxcXNvDD7wCKiKAN9eVEWKT6cIAb79cDvhSyG848aUUtBAHRGM VOfuI+ArUxh+4O8dev4A91AXDdjnxh97UGda+LHXmde/9BAou/bzGvcYS6BxY5Mi9u6XXj2kJanz Ivp8TCX0HyInO8C8eQ94uHeDubEJmtosREwOY+mL+g3enTjHAw3gNHkRFvl7w6WNBSwcJ2J7vPRX bowR7OhyeNs1hoXDAHh5DYBD85bo2M0K5iMP5H8oIn1TRMfsw4QrOxDo2Q66GoZo4xmA4DmTMNSp NcyaO2FkyBEk5jtjyMETRuBjZ94aTh4/wsG8Luo27YFFMa+AhBB0YvnNubJHOKGZLuPzaOaEEZwe 5uxLvU4hSChcOB9TCIEHCHdlL2zXMDwpdK24iBe4smORuL3UQSfvRdhx6hh2BHqiDXc/2nhiYfAc TB7aA83N2D0cuVyJ44PiyuGvlz4Cb/BfQpL4OWVfoR4YBRPO6YTwrz5GnMyEzkP28Zy3HRvMMdKt Tt5YHpXIPi46ihDu2dU2hZ3XfGy/8qL0VSupRKk93WodsvLUyle+mQT05qgvmVTux5wIcB8NiMn2 jQbR9hcc4X0WxQfbs/3s9hQcL/4k581h8jHRIR2HFXRf+H1ANiXtHkFmOk3IfRP3IYE4sI9SNrk3 IQOvyII4yTW536xIL9JHA/KKTBNf4T6ImEm2BlXIzHMHJXHlCMs1IdewByKy/6zrtMKpFQ3Y/pgo fhl1d99GKVy6rEjy0gfpS8p9sZs8ui+jeLkyS+301VkKXXeRfebxbuHDaC8M95gdtDkmubBDBZWq p6C9yN8PJiffsYLZCNqdlK2SZD5R0QiUvP2wex29krwdu5DT2EAKk3y0JrhHYf2G0tpHCu5LfDDZ QJ9sguNEymTF0VpXG3JdG1fsM1609oqvlrxORJ/JiDsHj69dR6KAfY4uZPViI6OGrvB108ajZ0o+ 265YC3XrVEDOuVjGM8GgfXsCC0b/icfdJ2BBf4uCjxV0LdB/wQR0TksF46wrYWAu1GzGInBMM9z/ cypmHUwBHt/AhcQsZEooU3WbYKCvIzIesbe8YSsM9eyEGopMO1XbYeDQVoxUtAwCo+6M8BmKKdEp RZM5lUHRZSOS4d62D/q1rZnPK1MW5WhUs8OvgSPQ7P6fGDXrsBrtoyy0+pxkvkZ82M+wdjmMxvP/ wu6g8ejf0UT47AriI7Dy8GXsC16OHVeecZ/RKg4cp/jcKdjSZDb++KlxwXOvOHW5xX4mHbcOjFtb wSJ7F+uspzJCfc78YYguS//EGLMCnr981JnDgthVgfjjYgZ0vmmJRoxEJ/fSIfydaIhufexgLNdx ahj3xbI5XRSQ7edLLOKgKtr2+wFWuIOdkdeQY9yCcWVn4B9fD/wSwfkJ1IBel3nYNaYxUK0jfujM GPIUhtro/ENHcHRXosBMQAcWwtOhEWp3+xGDmldjU8IqMHMOwvkMbiMua9TbJ6PvwEkIDPCEXbsB WHL6KbIS9mOJby80q+MIj4HNoK3dEp6L1mBj1F1kXduN0NVRePjsGGa6ecAvyB/uzZvB+4BinkOJ Jur95jKPIvPgNnAagma7o3k1bxzgSPe4B2neUEbO78fMFeZoPjQM8a/uIHKJD5ybNUIPDxeYC/2E 6rIXXCVmwtrLnFqw9YxLC+HwRUf8epLjNGF8FQk7Mc5hklgm8225fCT6ek7FbM8OMHWcJ3ISoKis fJOWKrVifCZtf4C7VQU83HkQFwux6Kkig0+jHgKMg/1yCNyHnED7pYEY2bKG1Ev6LW5FXUCGfgK2 LRwHF6uO6BUcXZgLiJ7i1MyfMDplKDb5d1ZK86uefu+YS/HgXfVYVrzqid9nSsFTOunflQyYvtBv Re7LzlJqPkWCZOorfPFyL1/2p0X6lsMpTOgnMJserPiOcRhImzlKXpnCphKxDPE0GzbBzNQhMZ9w nCqGZOm+PJ8TQqZEBVNzmev0go76NBLWRc9hGd3KyqPMS3OpnY4h2QVfp9zbwWRXeyhFCE1FmXQ7 2JF0jIdRRGoGxQXaMb4KK/Ldd4J2L1lLp1KvUrCNvtgsk0OP1/YmA48IxjqRRy8iptD4fNOPrAaK zlRvL3dpbc8W5BGRzMQ8oQgvP4pk9qncywHU2pEzX4nNXxy3x9E0EsQFkjXH9eK7g6J3h9LKEwdo YTsDquGxmyHBzBb3V1LvodwxM5Pd2UDuHA+IvheTyfyOHhlHjToH023OapayjQYY1qaea+8qLUuh aU3p/UijSK8G7D5ImeEUAcPHqYSA6u2HtRkPhrtOK3IZ3pfsbXuS+/SNdDmfs4crjvG7xIbRRHtT 0tL6hvyiuZbCgtBUokO1G5iSgY49BV6X+AoVXS7t/6rXqaDkz2TEzaBhn2jbzPgHVyID4FTjNjaN /g5df9mPZ1zXlh8s4DLWDU21tBjvzkpcjf0D/c05BjFCTmaOmpSq+cKLPdDSq8j8uXDmk+k4dGUP ApxqIm7TWNh3/RUHlJl0lEqtymYUO8E6XOiYNUR9XU1UaNUPIxx1cObQMRz4awuON7REk6pcE6iA hr37oPPTPdgQ+ZQRxrNZSGVrOHbrBGffwego44RBE5Wq14D2+okYEhKDPAdfjG+nnPxUqXrFXqiE 6rVeYb3vGITEaMBhxki003mNS3/vRV6nNqjHFpT0zBrBQus+os7cAzGddRlpgI1jF1g7e2FYp87o 62GDlxH/4MTLLCQeOIcazu0YqyAzkzEXXFNGtBFrkIaz2w9Bv0cXEXNfjd5YdvYoQvvXVFqWWh8O aumgoo7cVK1YDPgEaiOQexunjzyFcS9PTJ23AVtDvofuzpHoOnwrkvKfebYA2aI/Fmz/HT8bX8fW vTelnnFtVG7UAs0qH8UUj/k4KXbTprY+pZzx8+m4hcAxV0WOU7H7/B742+riytIJmHNMeppvgk6j l2PT/P+BIqbh5+VXxCvQzNehRUPWLaThxq0k4bC8VO9DahIevtXBl60bw1gomHUuzIHA1N0ncczf DppXgvHznONKXG2VRBMDGNWrgpwXyYh/xFyrcXSzkkZcxxTmlV/jyTO2e6XIoImqPefg72nGODS2 KyydV+GOWj1ZkYWwi7XRM/BPTKsbhbEdrOG86iazryfjxqUHeBXL3M6FhCBkVyZ6LpmPUR1qKBCm C5OeP6D7673YsOckDhytARe7muJ0DF/u5SQMKcxxwTN2ri2eSrOH2aIJczWWUoKyxKIU/rxA0kNG yPulJZoaM5sbH8oHAcEbpD9nc+ROXdHSsALroAchYHxXZOzajgNP5Na19NvgW/vqSEp5KdVxM/Oq YwD+3jwSX11cgH4jNiA+/2EpnyoUVcpn0nHfREjvXxH1VtRLaVTrDL+Q0WiBh7hwTX5DmD5ajlmA mQ7AwfFs4fAyt+TInPN+44ieNV7h7M7DCjx15+LF02dFbAcs6hawhdMDe3BEYIWf+jTGgxBP+EaJ XybcLMEvEONb6CLxwg08LkqMStdykc14omu3aYsOzDUTblxFnAxncB1YWhRHHMW2Nl6/h6rjI9js ZSqs4ueKfA+qVH5JEr3A9Zu1MP7QWUTObIV4fw/mx5N5hsnKRe6XXTBMyK3N8WsPQddqnOvWwkHD 2J5hqou9c8djo7EDbPUVNfcqqFEHuHg4Go8lLzHBfRw+dANvSlBW4dJFMfT4OHYeeYMWPzmjzce/ DV9ZNT+8eJ16aNSyIlJSJZ0xc95gZAQ9ARusKHQ7VhktG9VjM17poI0a3ediT6gz3m77hfnPjBI7 AZdO836OFbXk96NJmZZaBxbGhzB/1VVx58pMHxkZeKvVDN1tTQqXrNsCo1YGwLnKKfj/FICj3DSp anf4/z4YJqeCMSX0PPM4LsnGea32x8+zOc/u7PjABqyPuq9iJ86Rum+Ez9RDqDd8Gka3rYF6FhXx 1/yNiJO83ZlvxVdvddGyewfUlxSp7m/WTZw+XRfDf+yE1m6D4KhzBvtOsJ0sbA6Rdf4Yjn7FnL52 lvKMLldOXnoG3jxPwJntv2HN+Ww2exmPBV5WeM3iS3/dLQlRgRuZV/D6cJwyC15WhPRXtWDNPG0/ +m06pu6/x2YLz3AlfC7mHXku98BJFDfC/9zsUTkuF23YnnnO90zhUBs2PTqiwr5ATFx1Ac+z7iFy egCOaFky5rySlFVYMrJvYr3PbOyr54H5o63L/cMOBRp9PlEazIPRD62QGBmFa8JnKRtJD/6D7v+c 0PXLFzi/ZQuiEsWv+/QL2H+mNYa7NhTPuqRhqoyGw5ZglXt1xMwfhhHrb6r4bEvLKIPjAnO3ekdM JfUylmuuTLbg5khGxhZkM2QqBQZOIBcrWzHZeRaxj2rIS+h4oDbZeAXS9tjnTDvxvm2tytSg52QK E8Yx4vWIWdTfqh4ZNGhNNjYdyNqqK3ksPkQPONL5nLPk14gteun0prWPpXc859Gr2O200MOadGBA Vh6zGXG+H43t35ksWzBC++DDovysVEHcYrI1qkdmNoNpeuB8muTSnqw9NwkdM0ggy0s8Siunu5AZ I4DXMmNE/CuPUiK3sFYoxAkXFbWMrMnFZypNcHGiofkE76/pwX5/crTuTRMXziLfoX60jS3EZj04 SHOdvmJ1aE8+m86KyfQz6c4KZzLQ/5ps3QNp0+xuZGA9hGYHzSVvF0/6PfZloZKVRajeXjjdTcna YwYFLRxNLm6hFJvJMH51joKcvhY5stAyIVsf5rQi/S4dmduLjBm21j4bKDqROVGQBLYv3td+Hl3O lUSwxcnE47RsAFu41etMk3Zdo7RMkYN85quDAAAgAElEQVQI5g2SLVi2INfFJ0QL14rKykqj2O1L ycfemL26xe3lZBRtX+hBVsw5g46VBy0IXkh+Y/uRraUV9fReJnLqICme/30nBFRvP6wY9o3FNh9H snYaRwsDxlLfAX60O54tNOaco4BWhswRyjfkMWsmTRo5jVZGi/b05yUeoWVetmwTA+dsxYuWHX1A ea9iaYu3DbEXP1u0b0Quflsp9pXCB06tupWoTuISeHZAhlqpBrqLdR7rYRo6g/lKfN+LUcxE1Okb TLf8Cymhjh/EHtSy+2S5VO8iL+wDReBTbD/q1OkzMZWUVytMx5X1W5ExdCQ6v/dOu7zqzJfDI8Aj UN4I8MslpYq4PloM/pUten4I4Q0enj6CmMc5yNE7g/3nm+DbtvU+iFH3h4AOrwOPwMeMAG8q+Zjv 3keouzrTwo+wmrzKZYTAp9h+1KkTbyopowbGi+UR4BHgESgrBPiOu6yQLXO5HJ/4A6ltiWVeIF8A jwCPwAeCwGfYcXM7LQxR1bwb3L084dSMfQSt2wxOI36GOyNkqqrhgJAEjs2ovANHqhSIQUN94dXJ BHW/DUVcnpQOSvnEpdK802EunoT/CEPDHxEu/2XZO8l9l8zsY5/zwXBz9cLEgUMwL/oOruw9h4dl 8qXmu+jJ5/34EBDg5T/DULORP/tW4OPT/p03YbMqq7V38f1lYvuDuw+n7SncPmsxAZCQbIjTiCOm GVDAyV2eSuaep4CWTYRc3YLU87TjwB0p7t/i+MRLR1FBagxt3hwjRb5VOnKlpZSoveTFUmC7r4Vc 5By5VEu2b11EDCUtkT/+nBAoUfspEpg0RsLWkiz8zlL+Fv8i05fdRXXq9BmOuKsxKtCB6FJD0Yaa Gug40A3NDbUYluUcHl3C8WuZwkI1qrVBHwdzqR0gavCJq6G+RrW26Nev7YdDX5l2A9FXc6FfpSK0 Wk7A9iX2atSKz8IjoACBt5cQsVMDvR2a4D087QoUKlnUZ9hxy/NWSwFGabhz9nf0qV4Hdr/uZ+7E BMi+vw5963TCpLB1mOvJXJ/V7gz3QVbMpKIBbTNXBJ9nvCIq8zYr4sBOhuBhFFb/eQgJeem4tnsl QjjOaxlzQDF84grLZ5/fRwbB17k56vQYhIHmEhdNyjiqGQ7ZdxAxzg3jhPzazHQTE4phfYfBb7YH 2pn2wJxTjMtaYVn53/9LgfmOhxlXsGP537j0VoxJSCg2nEyUEvoc50M84Ow5A0GTnWFhN0vEoc2l yL6F8FGe8PHpgbqG3TCD05tFU1p58IhLqcgfvjcEBHd3Y+qP38D4i67w+qUv2pkZQtvcA2FiV4R5 109gPzrDsTXHbFlEW3pvNSim4HedADDx7yriPeaXN5UwVTKP06Svq5CJz2F6w04Fd4Kpi2sYPT7q Q4yWiX0m/R0F32KfzWZGUwDje9axC6abFxVzRMtXLE8pBzabrAk5gIvg+y6CT1wZRzUJrlKgdRXS shxP+6J30pI/gmmitSKOaqap2AVbPuf4y4Pk06gnBd/OZBf/o+0DTEmn51pKVMKHLV9XZeclai8y mIg50yVmrcdrqaeOGC8hF3YtIYc2McbtaL9vSJ/jCxck0Fqn2gQjG3IPPEjX3oFHXFl9+PjyRUD1 9iNuL5LnNZfRUVjoi00jGRTjZ021vPaTkBxBaVsqn7qpXqcCfT7DETeDqahQoS0GDWuDR5u24DDH 43w0BjV7d4JRlwU4Hcym6jqmsKjPOLortIbHiM7AmV0IDPlbIUe0zKCZ8dddU8qBrQLvn1I+ceUc 1QLm8FSX8XBXtvkfull/D9/hI+DtqYijmgHCuWCbMoR54uEC4c3ZXQjXt4O9eQV2Xhd9lh3CxdAe eKKED1u2rkIhZfuvTg/4bVmFX7ow5sQTp3AjJxMv0pmpiRJxIvIuGjWuj0oapujZrzP0XlvCfUw3 1CsXHvGyrTYvvYQISJ5Xraqozlh9hdStedewZ+tL9Py2FSpx4pS1pRIWVZ7JFRl6y7P8D7AsPTTu /xO+85uMVTv64LtdFdBrA8eSLb/0LKaJzDmJ/QfzWOfAcUSfEtaH44iuZCnPEZ2FJ9Ic2ByNiZgD +9ozxtesUpDwiXfEzO/7YSbHJ+60AS04juqKisrnXLRJBzFHtc8UxlHtBnuOo3qlhKOakdcKnRFw 6XOQdOsOkllnzhyXC4OGYUNYGt7DOqVlidKV23/m7d7sy5sYaz0aiW0twTEsC0chzD1bDcZMey36 GlJhCf3q1VChcQPU05bwiP8IJ45HfO90bN08gbF+8+FzQ4BuncKBR9aYYCtu+8ra0gcMDN9xK7g5 GsbfYtiPs/HDsul4ajEOe2twyxfyHTejhmWOCAS1m8Oq4Q1cFXJEdxNTh77G3StJQo7oKvnyK6AO x4G9QcSB3VjoeYa7yHFgF3Se+cllDjg+8Q2w3DxXSFwl4RPf1XIh4xNPRGOON9qc46guXL6MGHYi 4qiegYGMo/qJ42IcVMhRrYWqNapB7+IJnHzsBTOhA4BsJB4+gAsZyssqqKt8qaV/Lohfjf6Oq2C2 5QiOdbmCkX/txzVhMfXRf+k87Os6F15+d1Hv3wyMXzQIjTXZ1sJrYh7x9kEY5c3xiDdF/J/OzCsO Hz4fBN7g5r69uN59GOzEz6DytvThosKbShTem1pwGD4QTa49hokz86quME06Yk/fgNnwXzGV7QAp niO6EizV4MAWFV0Un3jHMuCo1kJ1Gwf8r8IBBExci9jn6YzXeC4mHjFC3/+pUleFgJVqZG7CVZx9 UQnGdRgZ68tUPMt/r75BYnQsKk7cjLBZ07Dkrw2YalOL8Szn4mnUynLgES/VavLCShsBeoCTh5Lw 7fcFz7XytlTahZeivAJzt3pHTBX1Mr7vXHkP6OjKqeTCOY3lOHanr6KjicwbrSRkHyGfev0oLEWy y1O82KFlTFYuY8hvghs5DBXzZCvkbc73RCyRyH4Vc2AL0m5QZKCrYj5pYe6i+MRZAoXlZ9CDI/PI yViHdKx9aFN0YsG+8EIc1UxG1gM6tcydGqAG2U7aTjfSXtCdsOFkqS9yWtzUdQlzGsz2vissS1Fd hYoX+qdye2H6nFwswWQLXY8/S8tcmPNXPQfyP/6AMpN2kCd371CLrL1mMIevRoxf2Z4Com/Qfi9L EV83Z6yHHhlZj2ROn5+z9V8HtXnEC1WEj3gvCKjafgRpMeL2wnGuX6b4S8vJhT0LbO7M2oQdrbhf 8KwLlLaltHKpo6p1klaGJ5liqCkKgjvL4bTQBOGrJFPpbCSE9IT5dHNEpoTCUeKyUFFmPk4pAuoQ 6igVpugC2woYNioUrz0H4usXqchiz2l28iH8cbUHdgV2VeIpR5EgPu5DRKDM2897qLQ6deJt3DI3 itu7vBXrrxAqRh9H88GrePunDD4f+kkuksInY8hRE/wzvyU6C73TM9NJxDk0at0AfGP/0O8fr5+q CPA2bhmkMhC3cwkm/DweW74ciQkdDcRX2ajt4Tnsj3nINlwwn4v7z+OhxCekTH7+5P0ioI06zj7w N9uNHtV1oMG2Q1Zt1h+hz50xrV99Bf4E36+2fOk8AuoiwJtK1EWOz6cWAupMC9UqiM/0SSLwKbYf derEj7g/yebNV4pHgEfgU0aA77g/5bv7udaNnuPu3efc9gE+8AioiMBr3LuZyBazP47wYXfcgkRE rZoGV/Mq0KjUCd5BQZgzyR2dzJvBcdxfiM+3Mz9AuKspDF3D8ERN3CntOOZ07wTniZMxtLsrJkfc gVJWbrZzYfs4dwycPA9zfFzg4LYUp9O4b/eUBfbxx5UdWORtB0NmdzXs5IWFiwMxV3heCWauU7HI 34txg5vC3HEitouJcJRJUxzPHBWvGweXNnVh6H1AVvesWKwb9T3aGDeCt5BASrGEDzf2OaJ8v2O6 pyhUkZ7sxWS7pmjuNJBxqjeGcd3GsFkUw77/lA9FyxGlLgJHFHVNviz+/H0jIHymHetDmxHCaWhb wHnRSaTlv80FSD8wCibcNeFffYw4mSnFyCmlfXYSEv57IxXxARxK7w1U55hVQZ1sJcgjTwQloKzL c8laqwY5rLhFoh3EOZQas4M2xySLz/PoVfQGWnf+lYrlPGY83I0KiKXuryCHGs604g5HsCQfMul2 sCPpD9jO6Iy48JgRMJlRy4DzxfP6CkmT9MkmOE4kVP785T7yqsf2mrYMoMuS7eOilCr+Z1zjNvqk 7xVZsGdbklOGsEkSWf6/arWXF7vJo4YOGQ7YRimFVOZ4lZtSZUYElixsDBl0e8UPZJR/f6QyFClH Kh0VgWOR16Rl8MdlgYDq7YcRow10Jt99dykz7RJt8mpHOlodKOAyI4jjguAehfUbSmsfZXNnhYMg maJXjCZH2940NjCcTiUK6agKpyuFGNXrVFDYhz3iZjUqHDSg26Q1rCo/w7nYe+JRlTaqte2Dfm1r CncO0JNd8Ok/F9GphcdcheWxmKQDWLnxLf7Xtbnwk3UNk/awb3ACc1fHCDkwZPO8wd1/7+Dtf4+Q nM+sJMDb7KJG3LISlJ7pt8G39sbIu3YGMY9U1F2psE/lQh6e7f0Lh3SqIv2v9dj5QH4e9ATXLjyE IDNb3BYqo+FAL7hlPMEzGQiKkyOTmD/52BF4HYsz1UZgxrdfoYJhKwyYNQa9dK8j8kSisGaC+Ais PHwZ+4KXY8eVZ7JmNe5bgEHd4HLIAvN3bUPQ+B/RsZ6QjuqDQeUj7LjZ1ry4S7j4uga+afmV+IMK FpewE+McJuEAm9acCd+KqIRnjNt6FVZH3cWzU/PgNnAagma7o3k1b5ZGFv/sqzE4kVMfjc3Z59Nc 0GCfmDOXZg+ionEnf2olugS2s7u9S298dWouhs46jDsnV2HZ9W4I8GxdeoTsul+gSiXpW8O58ApB P+dhmB00EU4Wjvkc0xz3tMRsE+Dti5CrEmIphkn8XxjXdwgmB86Et3corgqrIM/TXQXajYZi4ayh zPTjB+9O5szRRJjYDMW5UyuMXbnyWlMCdm54iVFBo9FGcAJrtt1E/vtSWB8jtO5ojux/JsPtl51I 4Mxnet2wdNdImEluGfdbnBylOLK8Cq9lyPGdMxwb+yOmFN7f0mrzx2oiULkHFgd9C/ETDcY2hprs 5W9al9vi+xa3oi4gQz8B2xYy86JVR/QKjoaI6i0dlwOHs28B2mDpsp/R0vAD3f1fMPhW74ihoF5G lXOJTSU6TanncC8aMaArmenrU1PPzRSfxc2NmenkzgZy5z5/lnA1y5gF7tLani3IIyKZpWWuybz8 KLLga1cWJ/6UHfZSLsvkzTPyymbQnU1DyIxzpWWgzKQin4edy5tG5M4FKbvIs54eGbpuoscyX5Fz daglNoGIdOO4sR8LuadtyGLScRIadfIuUkCLyqJ0mafJr8U3NOmUyKCTFxtALSDmr5bj6V7oYUO1 HFbQfYHYRRqaks9R7nNfRdjl0ON34LUuaXvhOMy7dGe65V6nYDtDwteT6VSmDDgkSI0if1vGuw0t 0rccRMuiJSazgntQtByOw1sJjkVhLIfjkpWnytTtW0FtPt+jkrYfCVK5MX5kUW8kRb7Mk0Sx3xxK iw1jdAmmpKX1DflFs2dFaE7TIp3WP9BwVwey7TmYpq+7SGmyTU5KxrsfqlMn6WEdy/8BhwqNYNen N3q7DcZwlyZ4tG42Zmy7zRbhmOnEvC+mjGijRPlKqF7rFdb7jkFIjAYcZoxEOx0lSVWNptd4cuc5 Gri5wJr50ZjkuaDA+4qqMvLTvUXCtlmMJ7s32jYfhP2NpyPiDzcYielURcnqwdlvDbb9YgMknsOx G+nIeZGONzkXsXnFI3Ts2BgcazY09fDFFxyTIftO6Mw2rEhoio4tRWMOzSpfgPP1IQwyPN0O6Pxl Hoxs26AeW6TRM2sEC637iDpzj41sFWGniUrlxmv9Fv/uPgTd7zvDRKshevXvCJ3buxB2TNYIwrEl zjh0FpEBTqgRtwGj7Z3wy4EkqelvMXKKwBFFXZPBkfGdD+v44bh9k9xr/pchkILDa0+h7dKJcJBh w2QbBVr0x4Ltv+Nn4+vYuvcmsq6fwZFnpug1ZBLmbQlDyA/a2On5PYZvvi/Vnt4/qB9Px61ZC5ad HeDIdg5MWLUaM22SsGlcCKLecLYM1nkzLmnFoTZ6Bv6JaXWjMLaDNZxXyU+1dVHPwrxgSiUUkovs LEY317wRTE/7ok7+ynNj+EalIm3PNPSP7IigjZsReXQhbK4FYujC02wCpk7QY7tK/BD65y5cePwS jw78Chvhp9rSslgDM6uO2zO6osnAMCRKpuOJ13HhqQYq6MpP5xgF67UbeKqpC10dmTeAtFDxcTJu cBzbsRyfdwhCdmWC4xMf1YHjRFSEnYTX2hiHOF5r51W4I2u7UFCGmlF5V7B19Q28PrEEI719MJ+x ulXGHWzZeFzOfs3k634Fx6nbcP7YTNhqXsDSnwNx7K3YzlWcHKU4MrlFXVOzWny28kSAmfuOLsMa w6kI7mOi+OtZ4dpSdaGThdyMdObIzAydvm0BQ80aaDF4CsZ/m4ldm4+pvWOtLGr78XTc0rXXNIZF E0Mg+Q5uJRW3iPcC12/WwnhuRDazFeL9PTBhz1NpadBtbg1bndu4EvdSFE9PcOv6S5h2bAXzLkvx hIizB7G/m1jaRRPn9x/By9aWMNdkHWqrnxHwiyVunb4CFfzYyJSr8ongJlb1d0OQvh9iji3DoBZi y10lfRjopOHGLenRJSeVjYr1K0PndQJuJcoZ9AsVyr2kGMe2kE98DMaM4f6GoGu1bMYnrgi7J8i4 Lua1jpwKq3iO13ofxMgVkq5+BOHtiR3Y0/U37N30B0JDQ/H7tr3Y4GGK5zu3YM9j8X1P+A29fY+I X5pskdpmPELGWws73GvCNCrIUYoj076oa+pXjs9ZLghw6zzbMGdvY8zx71LMbKgyWjaqJ5xxttRK Q2qapF8xgFG9KhBkSRa/y0XxYgv5ODvurJs4cyYZsGiP9vWVjbRzkJ6Rged3z2D7wo04n1cfjlNm wcuKkP5KrjMzssfgvto4dZpNlRhklHQRJ+JaYdRAKwXERJXRqGUT5MXdwoP8kaYWTNo0hTHYwt+B DVgfdV92H3Wxt6GYBLmJiD2biirGRjDAK6Q+E3mDR+22sO+ggdOr1yGK20ee/RKpL/NYIxOgescu 6KB5BqtDj7G9q6wBp6WxzjUPWYV2vxgXweedhKhAeezelA+vNeNN/vv3C7B1/UZqNlQL/3PvBZO3 R7Fu+23RImU9cxj/FYRVcZJF2RxkvMpkXuHtYMu1DVXkKMWRPaxFXSvmtvGX3y8C9Gwffp12H99P dYU558opOx7bJyzAwZdPcX7LFkQlivdmp1/A/jOtMdy1ITQbdMMPNsmIPBgneoYpBQ8SdPA/l074 8v1WR7b0dzWtM2nvKkJ5fo4ze9UMGtC0CkHHmjwWLKVAvzHU38aUDJr2o8WnnrKlSbY4mXiclg1o xLiaOe7da5SWF0crHI1J38yO3INW0WyW3tpjBgUtHE0ubqEUK7e4xSkgSD1GAY425DRhEnnYdibP sH8L74WWaMoc64Z5dSd79ym0cLYXOfddQCc5ruoc5pC0EVsk1elNax+z8/zA9pXH7qAgH3syZAto BjYjaMGihTTHy5YMxOeB22NJ6a5zQSLt9mzKuIRZXuvhtGAi45VGbbINOEXP7mwiT0u2aCdcmLMj mwYNyGbIPIqIT5bh07a0b08NjG1pSMBmOrxvrixPt1KObW4/szx23GKu+rzWKrUXdt+PzHVli78W 5DLvECVK1pO49jDPlXGGM3OjsRPNOPKA8rgFQltTMmb3esj0BRQ4yZWsrEcw/m22X1dlOffprVIc M9jityKMp1HQ71Nlccy/3/xBWSGgUvvhCn91mgKEC9ZC0zRnM2N/OqJvNXLOUUAr9swYfEMes2bS pJHTaGX+gja32WEr+dh1IKeJcynAx50GTN4p3ghRNrVSuU5SxfMkUwy1Ug10F+s81sM0dIbQzVip yv4EhKlDqPMJVJuvQikh8Cm2H3Xq9HGaSkqpEZS+GPZJ9PqtyBg6Ep31ilsULP3SeYk8AjwCnwcC /Ij787jPH0wt1RldfDDK84q8dwQ+xfajTp34Efd7b4q8AjwCPAI8AiVDgO+4S4YXn5pHgEeAR+C9 I8B33O/9FvAK8AjwCPAIlAwBvuMuGV58ah4BHoFPEgHekUIJb2s2Hkathp9rY0Z4XhOdvBcgeM4E uHdqDAsZpwLv7ixBmuVNNQcIbPNn2jHMsHNBSILcRzsytXyD/xKSivjohnekIAOXCif0JBIzXNvB 2LAwm6MK2UuQRBXnCoSsK+swykWRPkVdK4EafNJSR4B3pCC18Vv+kKEtH6XWeVakF+lL2Os4CVkx FGCtTzpC1jouQt5ZAot6dZZCGXOX9KcuXErFoaQOEBhz2OUVNIBjHZRhDpRIZ/pEryRvxy7kNDaQ wk49UP7BjiSLHBtgIbZA3pGCBCn2K2ZtlDA+Sl1R7zCNokPD6Lx8Y1HZuUJR+hR1TT1t+VyKEVC9 v+EdKZT6m1AlgbrmaGNVAznnYvGvkDZA1lkC6CEifIZiSnSKHD+zMukld4CgYdQHf/w2SOqTa4ns 14gP+xnWLofReP5f2B00Hv07mih2eyTJosov70hBFZTUSJODJxFT0H/KKaTm0xRwYnjnCmqA+XFk +cQdKcjTyn04N4XxCly4+Aw637REIyENK+cs4W9M9oqG454AtDy/HRuZk4Qs490IXa2LH3prYM3I DchrXw931+yAfuBJhDoyIqr8IHaA4MA5QGiC1d3OiBwgzFXmAEEbBrVrKDB/sKnx5RC4DzmB9puP YmTLGooZx/LLVeNAoSOF5Rg6+xqadjVEzO9XYPXnRvjb1IIGR/I/eTZ26zaFRfpxrOMcKVhyZYoI diZP2Q/dtvWRfniz0JGCJcenErkey0PXYHNeS9jf2onN2v0wtz9w7Y0xvjgdjlMWM7EztD/jd8hj jhQC4bXiNdp/fR9rluoj8EkoHDKOYZZXUVirUWdlWfJuYOOwbphy4SauZXXH2shlGGBeSUgeJKqb KVL2HMGrXnPxh28nVEMyTs2agBV5zfD13XAs1Z+PJ/56CN8YhYSsmtgduh66P/yILvU4HpMCJw07 +y8WOmkYOr4Fo+iSBPaC3j4bU3YT2lqk4/C68+yCufiiomtmyErYjyUr/8CazZmwtn+EDZsrY+qZ I5hpXUUilP8tDwSEjhSkChI7Uqgg50jhOHOksG3xH3BavAHhY9uhCvMrKnGksPki70hB8bxHKlZk KmEOEnp6kNeI/mRvZkBaTUfQtnjOR5wCZwkyvv9UJfcvuQMEkV7SThaYMwaPBoyPpBW5DO9L9rY9 yX36RrqcJj8Hl6qc5LAYUwnvSEECFPcrNj/ofUfBt1gbyDxOk77WJwu/s5SbxzlV4JxjMN5GFjgn CXY6X7Pzx8wF6FrqaeBJERmM+Z6ZQbzGH2CSFJsyinauIKDMaH9qYSFx3PCaYgPai511FHUtk+IC 7QhaVszf4QnavWQtneJ4bPhQKgiwrlgtObwjBamXWOkfVkVjO2f06u2Gn4b3gdWjTZg8Yydzo8Xo lot0lqAiuX9pOEDIvY3TR57CuJcnps7bgK0h30N350h0Hb4VSRyVTYkD70ihSMh0TGFRvzJQoSaM azH3oCkvkXttF9Ycr4UWHLUvC5oNv0X/zinYvCEKzypVQy3trfAdsgwxeV0xY3w7sXs7+VKKca7A Rl5nNm9FQscOaFmBoy/QRpUvJKPmoq6JueErW8OxWyc4+w5Gx0L86vK68OdliwDvSKFs8WWPWE1L Wzg6OmPAhFBsmtkRiZsCsIQ5LyjaWYIq5P5s2q/EAUJ6lLyzhCLYpQVvkP5cCw06dWX+6CowDxqD EDC+KzJ2bceBJxIPByWBiXekUBK0uLS5Tx7hHjNiZXP+JYWhBkzNDfH2yTO8rPodAv+egLqHxqGD pRtW3eGIehWE4pwrMHb1axceQ7OCroKOv6hrCsrio94jArwjhXIGvwJMLRowr+vMi/etlGLKZtvt iiX3T1fqACGlkLOEqsrL06nH+LgrIiWVY7fmgjaqGxlBT8A6khxJR6I8u1pXPktHCsqR0q5TF1/h LmLj0qQSaaCW5dcwyriNm1VH49CVPZhpFQ//76diz0v5+6KCcwVUhL5BRby+cQuJ8tmLvCalEn/4 nhEQrfPwjhTK9Ta8ROyZy2wprSG6tjdRWnJeegbePE/Ame2/Yc35bJg4jscCLyu8ZvESHxaizEU5 QFAqvvAFDTN8+0MrJEZG4ZpwxJeNpAf/Qfd/Tuj6ZTbvSKEwYqUeo2HZB96OeTiw77zI807WNUQd NcKIwe1R6WkUAtdcQp5Jd0xZMAxWr1/hVY54K0leBjLePMPdu5exs1gnDcboaN8Kmqc3IDTqKVvq zURaajogfEEbFXGt1KvLC1QTAd6RQhHLAQzTIq6qcimLEo/+SbMHtGCOAgzIymM2BQf60dj+NmRs 0IJcF59gnrMVOEsQZNKdFc5koP812boH0qbZ3ZiTgSE0O2guebt40u+xLwsXrswBQuGULIbt4447 SKu92pEOGtGA4AiKSXwtSsnkbPNxJGuncbQwYCz1HeBHu7lFVN6RgkIkpSNVbS+CtBha5sItArdn i3y3KenycnIx1iEd6wm07wFzbvBgD/lxji84sntfb5q8Tez4glsA5gjyZy+hhd5u5Pb7JcpkCuTd +Z0cDQzJzKY9dbG3Vc1JA9dePFuz7wtYv61vSfY2X5OxzRAKiLhNWQqvMY/gQcHk7/SVUG+fTWcp MasM3YNLA/uZHKvafnhHCsW87dShJCxG5Md9mXekUOT949tLkfDwF4tB4FNsP+rUqWDLajGA8ZdV QYB3pKAKSnwaHgEegXdDgHek8NzNJMMAAAp6SURBVG748blLiIA6o4sSFsEn/4QR+BTbjzp14kfc n3Aj56vGI8Aj8GkiwHfcn+Z95WvFI8Aj8AkjwHfcn/DN5atWgAC9SEZy/gc7BfH8EY+ACAFpPu5c vHj6TAFP0YeDVbl13IKHUVjl5wpzbQ1U6uSNoOA5mORuB3OLbzFu+y0xSLl4Ev4jDA1/RLhaXyEW APvuPNqMRCh8BJpX1YaGRiXUdQxETIZoP3DRPL8FOhQc8XzcBViU7xE9OYAA52aoYOiOLQ9ld/Yj 4wp2LPJCJ0N2jw3t4L1oNdaunQ/PNobQ0G0Hz4VLMWfyUDg1t0BzpzEIOXr/g36YyxfZj7E0ad58 AdIPjIKJhgZ7vrm/+hhx8hXSDgTAuVl1GLqE4+GHXMV33f7J6qa6iKxI8tIH6XtFMtofLryiywG2 pKXTk1bcF8UIUmNo8+YYtndbLLZEnNtcntLh0c69vJj6TTki2kN+ewU5GdamnmvvMvnF8PyK1Vb4 UwzJFPF83Aphe9fIwkRh0hLjKNhGn2ATTPHC6DSK9GL7x6V5wAVP6aR/VzLQakqeuxMZ5Rkf3hcC JepvhEoq4c0X3KOwfkNp7aNsuaqI739+e5C7XAanJa8TUbmNuBW/vKqgSZumqJwTh9h/GR0pCxrV 2qJfv7aoxvH6lJhzWygC786jLcBbPQfM8e/C9GCkQQ074X9NK6OiLmPBLYbnV6SBmv95Pm41gSvj bBq1YPPrbIxp9gB/jlqAg+kypN5lXDgvXn0ElPPmC+IjsPLwZewLXo4dV56xL6w+rvCeO+4MxF24 gdc6TdCyEWOA40L2HUSMc8O4A8l4ekbMuX2N49yOwkMB41qe+RMG+gVitnsrVPM+oGDqKuLRZmzL ckGKR3tpYDE82pqoYtEMDXSFbw9k3z6G0zXGY6bLl4CQ5/fbAucKYp5fUyHPr1yR6p4q5OMOQT/n YZgdNBFOFo6YcSpZ1Ng4Pu5x7hg4eR4CvH0RwvFxCwPH0/AXxvUdgsmBM+HtHSrk42ZvHsbHHQRf 5+ao02MQBppXgXajoVg4ayiT4QfvTuZoPjSMMTJyTZkR9JyaB7eB0xA02x3Nq4nciHFmqJluHvAL 8od782bwPvBcXGZp/igum9M/fvtk9B04CYEBnrBrNwBLTouwoLRzWD5sADz9psOzXVM4zjmONJkn ksk8OR3tatph5B97Efe8BKRgFazQz90KeHgEkRdflWZFeVllgkBRz/tb3Iq6gAz9BGxjfNwuVh3R KzgaGfJ6UDJO/toJNTuNwh97ruO5TFuST1zO5+868mfqqi5CbCrRadqThnsNowH2FqSv1ZI8t90S mU7YZ8Sb3JuwT98bkFdkGpMrmsbmm1YUci0rLr7w9FgNHu2se3R05UTq2aAK6VsOp7A7HDe4bFDM 8yubJv+sGFMJz8edjxQ7uEtre3Kc28nsmN07Lz+KZNY0Ifd27aEU8SKPxYvc0ekYD6OI1CQ64tOB OgdfpzzKpZTtg8hQpzetfZxD+W3hxlUKGzdN7j6qYCoRqyWSo082wXHiGP6nvBFQvb9R5XlnZtXY MJpob0paWt+QX/QLVh2JqSSIbtwJp3HjwulOGdMWqF6nArTfy4i7QmM79OnVB24/DWFvu8dYN3k+ tsWzkaKuBXMvNQRsXKM4qMy1rCC7OjzauvXR2eUnjJ46FM0frsGoGfvxTEa0Mp5fmUTFnPB83IoB qoTqtV5hvS9bFIzRgMOMkWin8wbX/tqC4w0t0aQq13QroGHvPuj8dA827NyN7eG66GFvzjzYaKFG n6U4e3EZ+htJnDzFY+vw8bjQcxz6m4tnd4oLLiZWC3oVJTKLScpffn8IqPS8azNa5v5YsP13/Gx8 HVv33hQzfjK1H23G8EGx6BnQj3mC4mbeH1Z4Lx23Zk1LdHbsDqcBk7Bq01TYJG7AuCUn8YZho6Gr q9x3o6pcy4owLoJHe0/YKNTJX11uDN+oAj5uDcMmcPCYjzUzGef2P0dxMd/Zuwo8v4r0KBTH83EX gkQYURs9A//EtLpRGNvBGs6rbjLfoll48ugpM6cxCl3JtLWOKcwrv8aTi9G4lqwDXclDplENFpZf SrWlZFy/ehpb1xzCE0lexQUric1FalIS3sIErZvWVpKGj/5gECjieS/Emy9cW6oudNIhompmtUi5 iasxO7Bm38MP0v79Xjpu6ZuraWqBJnp5SL52G0nSFxQdK+RaVnGhqAgebbJdyh5m4mw+7O8mlnaR 5+MWc4Mb1UQ1LU6xkvD8KqpICeM+Sz7uF7h+sxbGHzqLyJmtEO/vgQl7XqBOXdZp3riKOJkFwjqw 7MBs9npXcfjkf/kPmiDxJA7d4oYDXLDGhKXDUX3zFEzYmZifRnRNhf+UiAM7z0LQojf6tPlChQx8 kveKQBHPu2Le/Mpsna1egdOMlmOxdHx1bB4zAzufyG0jfa8VExX+njtutoAQexZnXuvBoqs16isB RMS5/QB3bx5UwLWs4vCpSB5toTdiqdJZx/zwEo7fTBM/4Nwi6n/oMK4frFjHrZTnl3GAJxzYgPVR pbzfNzcRsWdTUcXYCAZ4hdRnmSJda7eFfQcNnF69DlFpbKEt+yVSX+ZBkCVA9Y5d0EHzDFaHHmML dKw+aWmMuzoPWdnyC3LGsLYzx6PfpmPq/ntsJPsMV8LnYt6R56wRJyEqcCPO59WH45RZ8LIipL96 g6dRK4vhPpeCUu1DRWVrw9JtEBx1zmDfCc65Bms/54/h6Fc/YrBrD/T4nxb2BczAqthnyErci+kT j0DLqKJYA23o201AqFoPI1sQXc/w2VcHw+cPR1uhKzO1K8ZnLA8EinzeX+D8li2IShS/1NMvYP+Z 1hju2lDK8XcN2E1bgPHV/8aYCX+rOUsrw4oWmLvVO2KqqZQxL/EorZo9gJpqgXSsPGhB8ELyG9uP bIxrU1PXJSKHqlkP6NQyd2qAGmQ7aTvdSGPOffM5t4Mp+vJShVzLsgqowaMtK4Cdibm+tUzIdsQM CgyYQpOWnRTtLX91mgJsa3NvC6k/HTLxOUxveD7uQkjKR6jaXkQL06Zk7TGDghaOJhe3UIrN5HZQ v6YH+/3J0bo3TVw4i3yH+tE24aIx51B6E3laGrL7okX6TfvR4lNPKS/tMm31sSU9NCWPlafo9tkA aqejRQY2vhR28hBtCxpL9oZaBANb8gpcRWvWzCMPKwPGp21NHguWUqDfGOpv25Ja9BxNwUfuib8/ kK8Vf15eCKjefphGSnnzz1FAK9ZOON72WTNp0shptDI6me3P5xYrw8nHpgbBzIPFXaezAXaMj782 2fhsolhVHIKrAUSJ6iSWz7MDMtRKNfB83EXCqQ4TWpEC+YufFQKfYvtRp07v2VTyqbU5no/7U7uj fH14BD5EBPgR94d4Vz5hndQZXXzCcPBVKyECn2L7UadO/Ii7hA2HT84jwCPAI/C+ESiVLwm4NwYf eARURYBvL6oixadThADffoB37rjZIqcibPk4HgEeAR4BHoEyQoA3lZQRsLxYHgEeAR6BskKA77jL ClleLo8AjwCPQBkhwHfcZQQsL5ZHgEeAR6CsEPg/6PGEU7sSLFoAAAAASUVORK5CYII= --485b3973eca19d87c104aaa23334-- From enordstr@cs.princeton.edu Tue Aug 16 14:07:51 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0ACED21F8AEA for ; Tue, 16 Aug 2011 14:07:51 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.698 X-Spam-Level: X-Spam-Status: No, score=-5.698 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_43=0.6, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rwz3qZqAyZgv for ; Tue, 16 Aug 2011 14:07:48 -0700 (PDT) Received: from redflag.CS.Princeton.EDU (redflag.CS.Princeton.EDU [128.112.136.72]) by ietfa.amsl.com (Postfix) with ESMTP id 7C92321F8A95 for ; Tue, 16 Aug 2011 14:07:48 -0700 (PDT) Received: from hadron.cs.princeton.edu (hadron.CS.Princeton.EDU [128.112.95.77]) (authenticated bits=0) by redflag.CS.Princeton.EDU (8.13.8/8.13.8) with ESMTP id p7GL8XKK032121 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 16 Aug 2011 17:08:33 -0400 Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_26F7412A-A92D-464F-8631-C4AE9475D1D9" From: =?iso-8859-1?Q?Erik_Nordstr=F6m?= In-Reply-To: Date: Tue, 16 Aug 2011 17:08:33 -0400 Message-Id: References: To: Patrick Frejborg X-Mailer: Apple Mail (2.1244.3) X-Proofpoint-Virus-Version: vendor=nai engine=5400 definitions=6440 signatures=658826 X-Proofpoint-Spam-Reason: safe Cc: Mike Freedman , armd@ietf.org Subject: Re: [armd] New Version Notification for draft-wkumari-dcops-l3-vmmobility-00.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Aug 2011 21:07:51 -0000 --Apple-Mail=_26F7412A-A92D-464F-8631-C4AE9475D1D9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Hi All, I am one of the authors of the SCAFFOLD paper and I learned of this = discussion just a few days ago. I thought I'd chip in with a few = comments, updates and clarifications regarding our work. I hope I can = cover most of the SCAFFOLD-related issues that have popped up. My = apologies if some of this is out of the list's scope, but I hoped to = also add more context for other list subscribers. First of all, SCAFFOLD has been superseded by our new, less radical, = approach to service-centric networking, called Serval (for Service = Access Layer). You'll find a link to a manuscript of a recent Serval = paper below. Serval realizes most features of SCAFFOLD (e.g., service naming with a = group abstraction, service-layer anycast, and so forth) while sitting on = top of IP instead of redefining the entire stack. This makes Serval = easier to deploy and integrate with existing infrastructure and data = centers. In fact, we have a running prototype implementation for the = Linux kernel (as a module) that runs also on Android. We can hence = deploy and run Serval today on existing hosts by simply inserting our = kernel module. The existing TCP/IP stack can, of course, continue to run = in parallel. We also have an in-network translator that allows legacy = hosts to connect to Serval networks, and have implemented legacy NAT = traversal through UDP encapsulation (as somewhat standard). I think this = addresses many of the complexity and deployment issues that have been = raised in relation to our earlier work on SCAFFOLD. While SCAFFOLD focused on replicated services, Serval takes a broader = view of the interest in handling 'multiplicity'. As such, it also adds = support for multi-homed / multi-interfaced hosts, supporting interface = and (virtual host) migration (allowing IP addresses to change without = breaking TCP connections) and multi-path TCP (although the latter has = not yet been implemented). Because the IP addresses of ongoing flows = can change, we've demo'd migrating VMs across layer-3 boundaries. Serval realizes the above functionality by defining a Service Access = Layer (SAL) in-between the network and transport layer. The SAL = implements primitives for building scalable service resolution schemes = that allow mapping a service name to an IP address (i.e., service = instance) based on a service name in the first packet of a connection. = This resolution allows efficient load balancing of connections on this = first packet only. Serval provides flexible service resolution = primitives instead of a one-size-fits-all solution. In fact, we = explicitly designed Serval so that it is easy to build and use multiple = resolution schemes concurrently (e.g., broadcast in the local segment, = directory services or static configuration in the data center, or = leveraging DNS or service routing schemes similar to BGP or LISP+ALT in = the wide area).=20 As pointed out earlier on the list, SCAFFOLD and Serval bear resemblance = to HIP and other layer 3.5s. The main difference, in terms of naming, is = that Serval names services (i.e., a group of processes) and not hosts. = Serval also untangles the overloaded use of ports and addresses by = introducing serviceIDs and flowIDs (previously called sockIDs) that = explicitly name services and flows, respectively. (I guess our flowID = would work as the sessionID that was mentioned earlier in this thread.) = We actually decided against adding an explicit host identifier to our = architecture. The main reason was that we found no compelling need for = such an identifier that would warrant an additional name/ID/address = mapping scheme. In most cases, end hosts can track each other across = mobility/migration events via in-band signaling without explicit host = identities. In some corner cases (e.g., simultaneous migration) one can = use rendezvous points, or similar solutions. For those hosts that for = some reason need their own globally unique name/ID, it is perfectly fine = to assign them their own serviceID (i.e., forming a singleton group). Below you will find the link to the manuscript of our most recent Serval = paper. We encourage those interested to take a look at this paper that = accurately reflects our current work. We are happy to continue the = discussion regarding the use of Serval in ARMD settings and we also = welcome suggestions and feedback. http://sns.cs.princeton.edu/projects/serval/ Best regards, --Erik On Aug 16, 2011, at 12:52 PM, Patrick Frejborg wrote: > Hi Gary, >=20 > our discussion is way off the topic for ARMD, not sure if we can = continue our discussion on the list - but I continue a while, and if = somebody feel that we shouldn't have the discussion, please drop a note = and I take our discussion off the list. >=20 > First of all, I'm not the inventor behind SCAFFOLD. I have focused on = RFC 6303 which is a controversial research work from RGG and focusing on = routing scalability. I stumbled on SCAFFOLD somewhere in February-April, = found it interesting and also "compatible" with RFC 6306 - then I found = out what to do with the identifier. >=20 > Comments inline >=20 > On Tue, Aug 16, 2011 at 4:24 PM, Gary Berger = wrote: > Patrick, >=20 > Where to start.. Ugh.. Let me just say I think the approach Scaffold = takes is innovative especially in the light of bringing together = applications, services and networking.. Certainly we have learned that = topology is a lot more complicated than just a physical graph and moving = towards decoupling the physical topology will go along way with = providing scalable services. >=20 > As per SCAFFOLD, I have to say I have not thought through exhaustively = the design but I can give you some things that just give me pause. > Some of the implementation issues are scary.. I.e. Changing the data = plane, control plane, the network stack and programming model..=20 > These are HUGE changes, not necessarily impossible but definitely a = challenge > If you have a look on RFC 6306, you'll notice that the current data = plane is compatible - no changes are needed but the middleboxes and some = additional nodes needs to be changed/added. Only one change is needed to = the control plane, i.e.to add support for ICMP extensions, but that = shouldn't be a major showstopper for experiments. New extensions are = needed to the networking stack, some applications will suffer (but what = ever is done, some applications will suffer). My knowledge about the = programming model is very limited, thus no comments on that. > =20 > Kudos for the innovative thinking. I think there is growing practice = in looking at the network stack especially from a messaging point of = view instead of an opaque streaming interface[1] but it still is very = scary and if we are going to go to this extent maybe we should look at = the entire protocol stack.. > Pushing the bindings into the server are creative and maybe a first = attempt to look at these alternatives while they mature possibly = migrating into the dataplane of the network. Operationally people are = going to get a bit burned without the proper OAM tools. > I do like that you have added the elusive application service. Having = a proper binding for this is important as well as the appropriate lookup = services. Ip's "well-known" ports was a hack because we had no such = directory. You do answer Saltzers first objective: > 1. A given service may run at one or more nodes, and may need to move = from one node to another without losing its identity as a service.[3] > But again we are missing the Node Address. Saltzer missed this, if you = believe the conjecture it was because multi-homed node systems were not = envisioned at this point. The IP Address is naming the subnetwork = attachment point, this is ultimately the scaling problem..[2] What is = the "identity as a node" he is talking about? An identity needs to be = unique within the scope of the layer (we don't have this with MAC or IP = today). > Hmm. here I disagree :-) >=20 > This is the thinking from the past - what we have been trying to do in = the data centers during the past +10 years is to break up this = host-to-host architecture. When I have designed data centers you get to = a point when you have to decide what to do with the most important = services that need an availibilty of five nines. There is desire to = break the host-to-host concept and replace it with a host-to-many-hosts = concept - thus you implement some kind of load balancing solution (SLB, = DNS, etc) in front of your real servers so you can have a = host-to-many-hosts solution. But what I realized, after reading the = SCAFFOLD paper, is what we have been trying to accomplish that past +10 = years is to create a host-to-service solution with help of these load = balancing kludges - e.g. writing this e-mail I don't care on which node = the mail gets composed, the most important thing is that the service is = available! > Did Saltzer think of online services, where there online services on = the Internet at that time? I believe that mainframes (node) to produce a = high available service was mainstream at that time - the online service = concept (of the magnitude we see today) has arrived later. The demand = for mission-critical services are growing, and here we sit with an = outdated stack in our hands, trying to respond to the new requirements. >=20 > Also, if you read RFC 6306 (e.g. Appendix C) you'll notice that the = node has a two level "attachment point" - one level describing how the = node is connected to the local network and how the local network is = connected to the Internet. This model is like the highway where you have = several exits to a destination and the nodes can leverage this = information to reach another node. The current model that we use, is = more or less like the railway - nodes have no choice to make a decision = which path is used to reach the destination, instead the railway = (network) tries to figure out which path works best for the nodes. If we = could use a two-level addressing scheme we could replace multi-homing = with multi-pathing and let the transport protocol figure out which path = works best for them (like the driver on the highway). Oh, and the = transport protocol is pretty good on liveness detection also. > =20 > 2. A given node may be connected to one or more network attachment = points, and may need to move from one attachment point to another = without losing its identity as a node.[3] > In the paper it doesn't seem you have the concept of an identity in = fact under VM migration you talk about cycling the network interface to = get a new Scaffold address.. In which case where is the identity of the = node? > There is also overloading the hostAddr name with ASAddr and sockID in = the example implementation.. This in practice may be challenging but I = assume this is just an experimental viewpoint. > <7B78AD1C-E150-412D-A64A-DC1A9E885956.png> >=20 > Yes, drop this packet header structure and use the one described in = RFC 6306 - it is better suited for the current Internet and there is = placeholder for an identifier. >=20 > MPTCP do have a token that can be used as a session identifier - then = the application can be moved from one node to another - and I don't care = if this e-mail gets composed in Finland or Belgium, as long as the = service is available.=20 >=20 > For these two=20 >=20 > "IMHO, if you do routing based on identifiers then there is really no = decoupling of locators and identifiers - of course this depends on one's = definition of Loc/ID split" >=20 > I don't think this has to do with loc/id split, that in fact maybe a = false path[2].. But you do routing on a "name/address" relative to the = layer. Mux/Demux is just composition/decomposition in the network = domain. >=20 > If we take the perspective that devices are multi-homed as the = canonical model; examples being a device moving from AP to AP, Wi-fi = to 3G/4G or DC to DC. Scaffold goes to a lot of effort to make this = seamless but the complexity should be an indicator something is wrong. >=20 > I don't think SCAFFOLD address this issue - this is better suited for = MPTCP and they have work in progress to solve roaming issues. > =20 >=20 > I put together a short example here: = http://www.slideshare.net/gaberger/scaffold-8866328 which describes = Saltzer's model as well as Scaffold. I also took a stab at what it would = actually look like if you incorporated a Node Identifier in the address = architecture. I am not saying that this is an easy topic, its been 40 = years in the making, the question is where do we concentrate our efforts = in properly designing the address architecture and managing the layer = bindings.. The latter as a charter in this group "the what", the former = as a necessity in understanding "the why". >=20 > I'll have a look on that - I do have a presentation about hIPv4 and = uses cases but it need to be updated before releasing it. >=20 > Patrick=20 >=20 >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd --Apple-Mail=_26F7412A-A92D-464F-8631-C4AE9475D1D9 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
    Hi = All,

    I = am one of the authors of the SCAFFOLD paper and I learned of this = discussion just a few days ago. I thought I'd chip in with a few = comments, updates and clarifications regarding our work. I hope I can = cover most of the SCAFFOLD-related issues that have popped up. My = apologies if some of this is out of the list's scope, but I hoped to = also add more context for other list subscribers.

    First of all, SCAFFOLD has been = superseded by our new, less radical, approach to service-centric = networking, called Serval (for Service Access Layer). You'll find a link = to a manuscript of a recent Serval paper below.

    Serval realizes most features of = SCAFFOLD (e.g., service naming with a group abstraction, service-layer = anycast, and so forth) while sitting on top of IP instead of redefining = the entire stack. This makes Serval easier to deploy and integrate with = existing infrastructure and data centers. In fact, we have a running = prototype implementation for the Linux kernel (as a module) that runs = also on Android. We can hence deploy and run Serval today on existing = hosts by simply inserting our kernel module. The existing TCP/IP stack = can, of course, continue to run in parallel. We also have an in-network = translator that allows legacy hosts to connect to Serval networks, and = have implemented legacy NAT traversal through UDP encapsulation (as = somewhat standard). I think this addresses many of the complexity and = deployment issues that have been raised in relation to our earlier work = on SCAFFOLD.



    As = pointed out earlier on the list, SCAFFOLD and Serval bear resemblance to = HIP and other layer 3.5s. The main difference, in terms of naming, is = that Serval names services (i.e., a group of processes) and not hosts. = Serval also untangles the overloaded use of ports and addresses by = introducing serviceIDs and flowIDs (previously called sockIDs) that = explicitly name services and flows, respectively. (I guess our flowID = would work as the sessionID that was mentioned earlier in this thread.) = We actually decided against adding an explicit host identifier to our = architecture. The main reason was that we found no compelling need for = such an identifier that would warrant an additional name/ID/address = mapping scheme. In most cases, end hosts can track each other across = mobility/migration events via in-band signaling without explicit host = identities. In some corner cases (e.g., simultaneous migration) one can = use rendezvous points, or similar solutions. For those hosts that for = some reason need their own globally unique name/ID, it is perfectly fine = to assign them their own serviceID (i.e., forming a singleton = group).





    On Aug 16, 2011, at 12:52 = PM, Patrick Frejborg wrote:

    Hi = Gary,

    our discussion is way off the topic for ARMD, not sure if = we can continue our discussion on the list - but I continue a while, and = if somebody feel that we shouldn't have the discussion, please drop a = note and I take our discussion off the list.

    First of all, I'm not the inventor behind SCAFFOLD. I have focused = on RFC 6303 which is a controversial research work from RGG and focusing = on routing scalability. I stumbled on SCAFFOLD somewhere in = February-April, found it interesting and also "compatible" with RFC 6306 = - then I found out what to do with the identifier.

    Comments inline

    On Tue, Aug 16, = 2011 at 4:24 PM, Gary Berger <gaberger@cisco.com> = wrote:
    Patrick,

    Where to start.. = Ugh.. Let me just say I think the approach Scaffold takes is innovative = especially in the light of bringing together applications, services and = networking.. Certainly we have learned that topology is a lot more = complicated than just a physical graph and moving towards decoupling the = physical topology will go along way with providing scalable = services.

    As per SCAFFOLD, I have to say I have not thought = through exhaustively the design but I can give you some things that just = give me pause.
    • Some of the implementation issues are = scary.. I.e. Changing the data plane, control plane, the network stack = and programming model.. 
      • These are HUGE changes, not necessarily impossible but = definitely a challenge
    If = you have a look on RFC 6306, you'll notice that the current data plane = is compatible - no changes are needed but the middleboxes and some = additional nodes needs to be changed/added. Only one change is needed to = the control plane, i.e.to add support for = ICMP extensions, but that shouldn't be a major showstopper for = experiments. New extensions are needed to the networking stack, some = applications will suffer (but what ever is done, some applications will = suffer). My knowledge about the programming model is very limited, thus = no comments on that.
     
      • Kudos for the innovative thinking. I think there is = growing practice in looking at the network stack especially from a = messaging point of view instead of an opaque streaming interface[1] but = it still is very scary and if we are going to go to this extent maybe we = should look at the entire protocol stack..
    • Pushing the bindings into the server are creative and maybe a = first attempt to look at these alternatives while they mature possibly = migrating into the dataplane of the network. Operationally people are = going to get a bit burned without the proper OAM tools.
    •  I do like that you have added the elusive application service. = Having a proper binding for this is important as well as the appropriate = lookup services. Ip's "well-known" ports was a hack because we had no = such directory. You do answer Saltzers first objective:
    1. A = given service may run at one or more nodes, and may need to move = from = one node to another without losing its identity as a = service.[3]
    • But again we are missing the Node Address. Saltzer missed this, = if you believe the conjecture it was because multi-homed node systems = were not envisioned at this point. The IP Address is naming the = subnetwork attachment point, this is ultimately the scaling problem..[2] = What is the "identity as a node" he is talking about? An identity needs = to be unique within the scope of the layer (we don't have this with MAC = or IP today).
    Hmm. here I disagree :-)

    This = is the thinking from the past - what we have been trying to do in the = data centers during the past +10 years is to break up this host-to-host = architecture. When I have designed data centers you get to a point when = you have to decide what to do with the most important services that need = an availibilty of five nines. There is desire to break the host-to-host = concept and replace it with a host-to-many-hosts concept - thus you = implement some kind of load balancing solution (SLB, DNS, etc) in front = of your real servers so you can have a host-to-many-hosts solution. But = what I realized, after reading the SCAFFOLD paper, is what we have been = trying to accomplish that past +10 years is to create a host-to-service = solution with help of these load balancing kludges - e.g. writing this = e-mail I don't care on which node the mail gets composed, the most = important thing is that the service is available!
    Did Saltzer think of online services, where there online services on the = Internet at that time? I believe that mainframes (node) to produce a = high available service was mainstream at that time - the online service = concept (of the magnitude we see today) has arrived later. The demand = for mission-critical services are growing, and here we sit with an = outdated stack in our hands, trying to respond to the new = requirements.

    Also, if you read RFC 6306 (e.g. Appendix C) you'll notice that the = node has a two level "attachment point" - one level describing how the = node is connected to the local network and how the local network is = connected to the Internet. This model is like the highway where you have = several exits to a destination and the nodes can leverage this = information to reach another node. The current model that we use, is = more or less like the railway - nodes have no choice to make a decision = which path is used to reach the destination, instead the railway = (network) tries to figure out which path works best for the nodes. If we = could use a two-level addressing scheme we could replace multi-homing = with multi-pathing and let the transport protocol figure out which path = works best for them (like the driver on the highway). Oh, and the = transport protocol is pretty good on liveness detection also.
     
    2. A = given node may be connected to one or more network attachment points, = and may need to move from one attachment point to another without losing = its identity as a node.[3]
    • In the paper it doesn't seem you have the concept of an =
identity in fact under VM migration you talk about cycling the network =
interface to get a new Scaffold address.. In which case where is the =
identity of the node?
    • There is also overloading the hostAddr name with = ASAddr and sockID in the example implementation.. This in practice may = be challenging but I assume this is just an experimental = viewpoint.
    <7B78AD1C-E150-412D-A64A-DC1A9E885956.png>

    Yes, drop this packet = header structure and use the one described in RFC 6306 - it is better = suited for the current Internet and there is placeholder for an = identifier.

    MPTCP do have a token that can be used as a session identifier - = then the application can be moved from one node to another - and I don't = care if this e-mail gets composed in Finland or Belgium, as long as the = service is available.

    For these two

    "IMHO, if you do = routing based on identifiers then there is really no decoupling of = locators and identifiers - of course this depends on one's = definition of Loc/ID split"

    I don't think this has to do with loc/id = split, that in fact maybe a false path[2].. But you do routing on a = "name/address" relative to the layer. Mux/Demux is just = composition/decomposition in the network domain.

    If we take the perspective that devices are = multi-homed as the canonical model;  examples being  a device = moving from AP to AP, Wi-fi to 3G/4G or DC to DC. Scaffold goes to a lot = of effort to make this seamless but the complexity should be an = indicator something is wrong.

    I don't think SCAFFOLD address this = issue - this is better suited for MPTCP and they have work in progress = to solve roaming issues.
     

    I put = together a short example here: http://www.slideshare.net/gaberger/scaffold-8866328&= nbsp;which describes Saltzer's model as well as Scaffold. I also took a = stab at what it would actually look like if you incorporated a Node = Identifier in the address architecture. I am not saying that this is an = easy topic, its been 40 years in the making, the question is where do we = concentrate our efforts in properly designing the address architecture = and managing the layer bindings.. The latter as a charter in this group = "the what", the former as a necessity in understanding "the why".

    I'll have a look on that - I do have a = presentation about hIPv4 and uses cases but it need to be updated before = releasing it.

    Patrick


    _______________________________________________
    armd mailing = list
    armd@ietf.org
    https://www.ietf.org/ma= ilman/listinfo/armd

    = --Apple-Mail=_26F7412A-A92D-464F-8631-C4AE9475D1D9-- From linda.dunbar@huawei.com Wed Aug 17 13:23:02 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 98D4021F8BE8 for ; Wed, 17 Aug 2011 13:23:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.178 X-Spam-Level: X-Spam-Status: No, score=-6.178 tagged_above=-999 required=5 tests=[AWL=0.420, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XjtTrdHcDyQg for ; Wed, 17 Aug 2011 13:23:00 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id 8178621F8781 for ; Wed, 17 Aug 2011 13:22:04 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQ300KWQ9Y7NF@usaga04-in.huawei.com> for armd@ietf.org; Wed, 17 Aug 2011 15:22:55 -0500 (CDT) Received: from dfweml201-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQ300JES9Y13M@usaga04-in.huawei.com> for armd@ietf.org; Wed, 17 Aug 2011 15:22:54 -0500 (CDT) Received: from DFWEML401-HUB.china.huawei.com (10.193.5.101) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 17 Aug 2011 13:22:48 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.37]) by DFWEML401-HUB.china.huawei.com ([fe80::f07f:889f:78ef:8df3%13]) with mapi id 14.01.0270.001; Wed, 17 Aug 2011 13:22:53 -0700 Date: Wed, 17 Aug 2011 20:22:52 +0000 From: Linda Dunbar X-Originating-IP: [10.192.11.66] To: "warren@kumari.net" Message-id: <4A95BA014132FF49AE685FAB4B9F17F605193340@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_GHz8rIZqzWRzDJqYbgK9aw)" Content-language: en-US Accept-Language: en-US Thread-topic: Why need GRE tunnel for overlay network to scale ARP/ND in DC? Thread-index: AcxdG25QLJtpy/DzSLOaHMPoOQYWGA== X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: 83w= A4sC BAKv CZzG Ct5D DyIJ EA6L H0t4 Kb7F LG/h Mp5F PkVs QE7S QKpp SVkV VJdJ; 2; YQByAG0AZABAAGkAZQB0AGYALgBvAHIAZwA7AHcAYQByAHIAZQBuAEAAawB1AG0AYQByAGkALgBuAGUAdAA=; Sosha1_v1; 7; {12B992B3-60A3-42BC-A1E9-7503F291ED0B}; bABpAG4AZABhAC4AZAB1AG4AYgBhAHIAQABoAHUAYQB3AGUAaQAuAGMAbwBtAA==; Wed, 17 Aug 2011 20:22:48 GMT; VwBoAHkAIABuAGUAZQBkACAARwBSAEUAIAB0AHUAbgBuAGUAbAAgAGYAbwByACAAbwB2AGUAcgBsAGEAeQAgAG4AZQB0AHcAbwByAGsAIAB0AG8AIABzAGMAYQBsAGUAIABBAFIAUAAvAE4ARAAgAGkAbgAgAEQAQwA/AA== x-cr-puzzleid: {12B992B3-60A3-42BC-A1E9-7503F291ED0B} Cc: "armd@ietf.org" Subject: [armd] Why need GRE tunnel for overlay network to scale ARP/ND in DC? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 20:23:02 -0000 --Boundary_(ID_GHz8rIZqzWRzDJqYbgK9aw) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Warren, Thank you very much for posting "draft-wkumari-dcops-l3-vmmobility-00". It shows a possible approach to scale Address Resolution for large number of hosts in data center. Your draft proposes an overlay network boarded by Hypervisor (or virtual switch within a server in IEEE802.1 Data Center terminology). The boarder node, i.e. Hypervisor in your draft, can intercept ARP/ND requests, pass it to a "mapping server" (or directory server), and cache the response. The response from the "mapping server" not only has the target host's MAC but also the physical server's IP where target host resides). When the Source host sends a data frame to the target host, the "Hypervisor" encapsulate the data packet with the IP header (with SA = IP of the physical server where Source host resides, and DA= IP of the physical server where target host resides). My question is why need GRE tunnel for the overlay network? Why can't it be simple IP network? The ARP/ND statistics work done by Merit Network showed that the pain points of large Layer 2 network is at the L2/L3 Gateway (or boundary) node. Your draft describes the way for "overlay Edge node" (i.e. Hypervisor in your draft) to intercept ARP/ND requests and reroute them to "mapping server". This method can definitely reduce the amount of ARP/ND requests to be processed by L2/L3 gateway nodes. However, L2/L3 Gateway nodes still have to maintain a huge mapping table (target host address & Customer ID <-> Server Address). Are you suggesting L2/L3 Gateway also using external "mapping server"? I didn't see any description in your draft on how servers/guest OS get notification when some VMs have moved. Can you give some explanation on how to avoid traffic being forwarded to the wrong servers when target VMs have moved? Thank you very much. Linda Dunbar --Boundary_(ID_GHz8rIZqzWRzDJqYbgK9aw) Content-type: text/html; charset=us-ascii Content-transfer-encoding: quoted-printable

    Warren,

     

    Thank you very much for posting “draft-wkumari= -dcops-l3-vmmobility-00”. It shows a possible approach to scale Addre= ss Resolution for large number of hosts in data center.

     

    Your draft proposes an overlay network boarded by Hy= pervisor (or virtual switch within a server in IEEE802.1 Data Center termin= ology).  The boarder node, i.e. Hypervisor in your draft, can intercep= t ARP/ND requests, pass it to a "mapping server" (or directory server), and cache the response. The response f= rom the "mapping server" not only has the target host's MAC but a= lso the physical server's IP where target host resides). When the Source ho= st sends a data frame to the target host, the "Hypervisor" encapsulate the data packet with the IP header (with SA =3D IP of the phys= ical server where Source host resides, and DA=3D IP of the physical server = where target host resides).

     

    My question is why need GRE tunnel for the overlay n= etwork? Why can’t it be simple IP network?

     

    The ARP/ND statistics work done by Merit Network sho= wed that the pain points of large Layer 2 network is at the L2/L3 Gateway (= or boundary) node.  Your draft describes the way for  “over= lay Edge node” (i.e. Hypervisor in your draft) to intercept ARP/ND requests and reroute them to “mapping server”= . This method can definitely reduce the amount of ARP/ND requests to be pro= cessed by L2/L3 gateway nodes. However, L2/L3 Gateway nodes still have to m= aintain a huge mapping table (target host address & Customer ID <-> Server Address). Are you suggesting L2/L3 Gate= way also using external “mapping server”?

     

    I didn’t see any description in your draft on = how servers/guest OS get notification when some VMs have moved. Can you giv= e some explanation on how to avoid traffic being forwarded to the wrong ser= vers when target VMs have moved?

     

    Thank you very much.

     

    Linda Dunbar

    --Boundary_(ID_GHz8rIZqzWRzDJqYbgK9aw)-- From jmh@joelhalpern.com Wed Aug 17 14:15:46 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 71F4F21F8C11 for ; Wed, 17 Aug 2011 14:15:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.532 X-Spam-Level: X-Spam-Status: No, score=-102.532 tagged_above=-999 required=5 tests=[AWL=0.067, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TA+FTQv0TyFE for ; Wed, 17 Aug 2011 14:15:45 -0700 (PDT) Received: from hgblob.out.tigertech.net (hgblob-ipv6.tigertech.net [IPv6:2604:4f00::1:0:0:22]) by ietfa.amsl.com (Postfix) with ESMTP id C369E21F8C08 for ; Wed, 17 Aug 2011 14:15:45 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by hgblob.tigertech.net (Postfix) with ESMTP id E6D753245DAB; Wed, 17 Aug 2011 14:16:37 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at hgblob.tigertech.net Received: from [10.10.10.105] (pool-71-161-51-177.clppva.btas.verizon.net [71.161.51.177]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by hgblob.tigertech.net (Postfix) with ESMTPSA id 2AD9E324405E; Wed, 17 Aug 2011 14:16:37 -0700 (PDT) Message-ID: <4E4C2FB2.4000707@joelhalpern.com> Date: Wed, 17 Aug 2011 17:16:34 -0400 From: "Joel M. Halpern" User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20110812 Thunderbird/6.0 MIME-Version: 1.0 To: Linda Dunbar References: <4A95BA014132FF49AE685FAB4B9F17F605193340@dfweml503-mbx.china.huawei.com> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F605193340@dfweml503-mbx.china.huawei.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: "armd@ietf.org" Subject: Re: [armd] Why need GRE tunnel for overlay network to scale ARP/ND in DC? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 21:15:46 -0000 Linda, I must have missed a set of statistics. The earlier descriptions I had heard, and the descriptions from folks at Yahoo (I will let Warren speak for Google) have stated that the problem is at the end device and its virtual switch, and the process of mapping to the actual MAC addresses to use. If instead the key point is where external traffic (either from users, or other families of servers) enter the subnet where the virtual machine is located, then I would appreciate the pointers to those statistics. (Sorting out this kind of confusion is exactly what ARMD was chartered for, so the fact of confusion comes as no surprise.) Yours, Joel On 8/17/2011 4:22 PM, Linda Dunbar wrote: > Warren, > > Thank you very much for posting “draft-wkumari-dcops-l3-vmmobility-00”. > It shows a possible approach to scale Address Resolution for large > number of hosts in data center. > > Your draft proposes an overlay network boarded by Hypervisor (or virtual > switch within a server in IEEE802.1 Data Center terminology). The > boarder node, i.e. Hypervisor in your draft, can intercept ARP/ND > requests, pass it to a "mapping server" (or directory server), and cache > the response. The response from the "mapping server" not only has the > target host's MAC but also the physical server's IP where target host > resides). When the Source host sends a data frame to the target host, > the "Hypervisor" encapsulate the data packet with the IP header (with SA > = IP of the physical server where Source host resides, and DA= IP of the > physical server where target host resides). > > My question is why need GRE tunnel for the overlay network? Why can’t it > be simple IP network? > > The ARP/ND statistics work done by Merit Network showed that the pain > points of large Layer 2 network is at the L2/L3 Gateway (or boundary) > node. Your draft describes the way for “overlay Edge node” (i.e. > Hypervisor in your draft) to intercept ARP/ND requests and reroute them > to “mapping server”. This method can definitely reduce the amount of > ARP/ND requests to be processed by L2/L3 gateway nodes. However, L2/L3 > Gateway nodes still have to maintain a huge mapping table (target host > address & Customer ID <-> Server Address). Are you suggesting L2/L3 > Gateway also using external “mapping server”? > > I didn’t see any description in your draft on how servers/guest OS get > notification when some VMs have moved. Can you give some explanation on > how to avoid traffic being forwarded to the wrong servers when target > VMs have moved? > > Thank you very much. > > Linda Dunbar > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From linda.dunbar@huawei.com Wed Aug 17 14:43:25 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6E96A21F8AE9 for ; Wed, 17 Aug 2011 14:43:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.221 X-Spam-Level: X-Spam-Status: No, score=-6.221 tagged_above=-999 required=5 tests=[AWL=0.378, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ye5KPapN1clU for ; Wed, 17 Aug 2011 14:43:24 -0700 (PDT) Received: from usaga02-in.huawei.com (usaga02-in.huawei.com [206.16.17.70]) by ietfa.amsl.com (Postfix) with ESMTP id CE38F21F8754 for ; Wed, 17 Aug 2011 14:43:24 -0700 (PDT) Received: from huawei.com (localhost [127.0.0.1]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQ300KDQDPRN5@usaga02-in.huawei.com> for armd@ietf.org; Wed, 17 Aug 2011 16:44:16 -0500 (CDT) Received: from dfweml201-edg.china.huawei.com ([172.18.4.104]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LQ300ADHDP77L@usaga02-in.huawei.com> for armd@ietf.org; Wed, 17 Aug 2011 16:44:15 -0500 (CDT) Received: from DFWEML402-HUB.china.huawei.com (10.193.5.102) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 17 Aug 2011 14:43:50 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.37]) by DFWEML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Wed, 17 Aug 2011 14:43:54 -0700 Date: Wed, 17 Aug 2011 21:43:54 +0000 From: Linda Dunbar In-reply-to: <4E4C2FB2.4000707@joelhalpern.com> X-Originating-IP: [10.192.11.66] To: "Joel M. Halpern" Message-id: <4A95BA014132FF49AE685FAB4B9F17F605193422@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-language: en-US Content-transfer-encoding: 7BIT Accept-Language: en-US Thread-topic: [armd] Why need GRE tunnel for overlay network to scale ARP/ND in DC? Thread-index: AQHMXSL6Ovr5hDg6jUu3i73CeKJPBpUhkRdQ X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <4A95BA014132FF49AE685FAB4B9F17F605193340@dfweml503-mbx.china.huawei.com> <4E4C2FB2.4000707@joelhalpern.com> Cc: "armd@ietf.org" Subject: Re: [armd] Why need GRE tunnel for overlay network to scale ARP/ND in DC? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 21:43:25 -0000 Joel, Merit Networks' Jim Rees and Manish Karir (CC'ed) presented their ARP/ND statistics at 81st IETF ARMD WG session. Their slides can be found at https://datatracker.ietf.org/meeting/81/materials.html under ARMD Address Resolution Statistics. Their draft is https://datatracker.ietf.org/doc/draft-karir-armd-statistics/ Linda > -----Original Message----- > From: Joel M. Halpern [mailto:jmh@joelhalpern.com] > Sent: Wednesday, August 17, 2011 4:17 PM > To: Linda Dunbar > Cc: warren@kumari.net; armd@ietf.org > Subject: Re: [armd] Why need GRE tunnel for overlay network to scale > ARP/ND in DC? > > Linda, I must have missed a set of statistics. > The earlier descriptions I had heard, and the descriptions from folks > at > Yahoo (I will let Warren speak for Google) have stated that the problem > is at the end device and its virtual switch, and the process of mapping > to the actual MAC addresses to use. > > If instead the key point is where external traffic (either from users, > or other families of servers) enter the subnet where the virtual > machine > is located, then I would appreciate the pointers to those statistics. > (Sorting out this kind of confusion is exactly what ARMD was chartered > for, so the fact of confusion comes as no surprise.) > > Yours, > Joel > > On 8/17/2011 4:22 PM, Linda Dunbar wrote: > > Warren, > > > > Thank you very much for posting "draft-wkumari-dcops-l3-vmmobility- > 00". > > It shows a possible approach to scale Address Resolution for large > > number of hosts in data center. > > > > Your draft proposes an overlay network boarded by Hypervisor (or > virtual > > switch within a server in IEEE802.1 Data Center terminology). The > > boarder node, i.e. Hypervisor in your draft, can intercept ARP/ND > > requests, pass it to a "mapping server" (or directory server), and > cache > > the response. The response from the "mapping server" not only has the > > target host's MAC but also the physical server's IP where target host > > resides). When the Source host sends a data frame to the target host, > > the "Hypervisor" encapsulate the data packet with the IP header (with > SA > > = IP of the physical server where Source host resides, and DA= IP of > the > > physical server where target host resides). > > > > My question is why need GRE tunnel for the overlay network? Why can't > it > > be simple IP network? > > > > The ARP/ND statistics work done by Merit Network showed that the pain > > points of large Layer 2 network is at the L2/L3 Gateway (or boundary) > > node. Your draft describes the way for "overlay Edge node" (i.e. > > Hypervisor in your draft) to intercept ARP/ND requests and reroute > them > > to "mapping server". This method can definitely reduce the amount of > > ARP/ND requests to be processed by L2/L3 gateway nodes. However, > L2/L3 > > Gateway nodes still have to maintain a huge mapping table (target > host > > address & Customer ID <-> Server Address). Are you suggesting L2/L3 > > Gateway also using external "mapping server"? > > > > I didn't see any description in your draft on how servers/guest OS > get > > notification when some VMs have moved. Can you give some explanation > on > > how to avoid traffic being forwarded to the wrong servers when target > > VMs have moved? > > > > Thank you very much. > > > > Linda Dunbar > > > > > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd From florin.balus@alcatel-lucent.com Wed Aug 17 16:26:26 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8EF9021F8BB7 for ; Wed, 17 Aug 2011 16:26:26 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[AWL=-0.001, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1dTS3G7ma-Hr for ; Wed, 17 Aug 2011 16:26:24 -0700 (PDT) Received: from ihemail2.lucent.com (ihemail2.lucent.com [135.245.0.35]) by ietfa.amsl.com (Postfix) with ESMTP id 7CF2A21F8BBA for ; Wed, 17 Aug 2011 16:26:24 -0700 (PDT) Received: from usnavsmail4.ndc.alcatel-lucent.com (usnavsmail4.ndc.alcatel-lucent.com [135.3.39.12]) by ihemail2.lucent.com (8.13.8/IER-o) with ESMTP id p7HNRFlF023374 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Wed, 17 Aug 2011 18:27:15 -0500 (CDT) Received: from USNAVSXCHHUB02.ndc.alcatel-lucent.com (usnavsxchhub02.ndc.alcatel-lucent.com [135.3.39.111]) by usnavsmail4.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p7HNREu8027567 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT) for ; Wed, 17 Aug 2011 18:27:14 -0500 Received: from USNAVSXCHMBSC3.ndc.alcatel-lucent.com ([135.3.39.145]) by USNAVSXCHHUB02.ndc.alcatel-lucent.com ([135.3.39.111]) with mapi; Wed, 17 Aug 2011 18:27:14 -0500 From: "Balus, Florin Stelian (Florin)" To: "armd@ietf.org" Date: Wed, 17 Aug 2011 18:27:12 -0500 Thread-Topic: ARMD problem space & existing standards Thread-Index: AcxdNTFE+My2+OQJRqaZzuH96c0GBg== Message-ID: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_2073A6C5467C99478898544C6EBA3F4602BB419A2CUSNAVSXCHMBSC_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.57 on 135.245.2.35 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.12 Subject: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Aug 2011 23:26:26 -0000 --_000_2073A6C5467C99478898544C6EBA3F4602BB419A2CUSNAVSXCHMBSC_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The discussion on ARMD problem space seem to gravitate around the issues ba= sic VLAN deployments had in the past with flood containment. Before startin= g new protocol work maybe we should pause and start from what is missing in= the latest IEEE/IETF technologies when it comes to addressing these issues= . Here is a list of existing standards I am familiar with that are deployed i= n large Service Provider Ethernet Networks where we had to deal with simila= r issues: - IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS or IP-onl= y, VPN "shim" is the ISID and/or PW label. - IEEE 802.1ah PBB - "core tunneling" is based on native Ethernet swit= ching; VPN "shim" is the ISID 24 bit tag; may be transported over the above= VPLS tunnels (see IETF PBB-VPLS work). Also there are additional initiatives in IEEE and IETF to add DC oriented f= eatures (e.g. Multi-pathing and control plane alternative to MAC learning) = - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). This is not a comprehensive list, just an example of possible ways to addre= ss the issues without reinventing the wheel. A basic function of these solutions is that any flooded packet is delivered= through the core efficiently only to the edge switches and server blades t= hat have at least an interested VPN endpoint (VM). The service auto-discove= ry component provides on demand service connectivity only between the inter= ested endpoints (e.g. Server Blades/VMs) inside or across DCs. So these sol= utions have the potential to address in my opinion the DC wide flooding. If further flood optimization inside an individual tenant domain is require= d, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may be used a= lso with these technologies: e.g. the mapping server approach can map VM MA= Cs to the above shims + Tunnels as an alternative to regular IEEE MAC Learn= ing. The discussion in draft-wkumari on separation of VM addressing between tena= nts and from the DC internal addressing highlights also a very important re= quirement that needs to be addressed for true isolation between tenants. Bu= t this could be achieved by re-using the existing VPN standards - this is a= fundamental principle that was addressed already in the existing encapsula= tions. In addition the core tunneling is based on Service Provider addressi= ng. Sorry for the long email. I am hoping people new to VPN work will benefit a= s much as we benefited from the good information on DC requirements. Florin --_000_2073A6C5467C99478898544C6EBA3F4602BB419A2CUSNAVSXCHMBSC_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

    -      IETF-L2VPN WG VPLS/PBB-VP= LS – “core tunneling” can be MPLS or IP-only, VPN “shim”= is the ISID and/or PW label.

    -      IEEE 802.1ah PBB – = “core tunneling” is based on native Ethernet switching; VPN “shimR= 21; is the ISID 24 bit tag; may be transported over the above VPLS tunnels (see= IETF PBB-VPLS work).

    The discussion on ARMD problem space seem = to gravitate around the issues basic VLAN deployments had in the past = with flood containment. Before starting new protocol work maybe we = should pause and start from what is missing in the latest IEEE/IETF = technologies when it comes to addressing these = issues.

     
    - IETF-L2VPN WG = VPLS/PBB-VPLS =96 =93core tunneling=94 can be MPLS or IP-only, VPN = =93shim=94 is the ISID and/or PW label.
    - IEEE 802.1ah PBB =96 = =93core tunneling=94 is based on native Ethernet switching; VPN =93shim=94= is the ISID 24 bit tag; may be transported over the above VPLS tunnels = (see IETF PBB-VPLS work).
    Also there are additional initiatives in = IEEE and IETF to add DC oriented features (e.g. Multi-pathing and = control plane alternative to MAC learning) - see IETF L2VPN EVPN, = PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). =  
     
     
    If further flood optimization inside an = individual tenant domain is required, the procedures in = draft-wkumari-dcops-l3-vmmobility-00.txt may be used also with these = technologies: e.g. the mapping server approach can map VM MACs to the = above shims + Tunnels as an alternative to regular IEEE MAC = Learning.
     
    Sorry for the long email. I am hoping = people new to VPN work will benefit as much as we benefited from the = good information on DC requirements.
    Florin
    Message-id: <4A95BA014132FF49AE685FAB4B9F17F605193C18@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_B/UlpbB/dTR5Fl0kjoLx6A)" Content-language: en-US Accept-Language: en-US Thread-topic: [armd] ARMD problem space & existing standards Thread-index: AcxdNTFE+My2+OQJRqaZzuH96c0GBgAWtWsAAAlLM8A= X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Cc: "armd@ietf.org" Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 15:06:26 -0000 --Boundary_(ID_B/UlpbB/dTR5Fl0kjoLx6A) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Thanks Florin for pointing out L2VPN & IEEE802.1ah Overlay Network. There is also TRILL encapsulation. All of these encapsulation schemes, including the IP encapsulation introduced by "draft-wkumari-dcops-l3-vmmobility-00", achieve the goal of hiding individual hosts' (or VMs) addresses from the backbone network. All of these encapsulation overlay mechanisms face the common challenge in Data Center: edge node of the overlay network has to maintain very large mapping for all remote hosts <-> their corresponding Edge nodes. The more number of hosts attached to Overlay Network Edge nodes, the more mapping has to be maintained by the Edge nodes. The goal of ARMD is to identify those problems. ARMD is not to introduce another encapsulation scheme. "draft-wkumari-dcops-l3-vmmobility-00" introduces "Mapping Server" to solve this problem. http://datatracker.ietf.org/doc/draft-dunbar-trill-directory-assisted-edge/ introduces the Directory Server to solve the problem for TRILL. Then there is a need for directory assistance for IEEE802.1ah, or other encapsulation schemes in Data Center because the massive number of hosts. It may be better for the industry to have a common approach on how Edge node can get assistance from directory servers. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Aldrin Isaac Sent: Wednesday, August 17, 2011 10:17 PM To: Balus, Florin Stelian (Florin) Cc: armd@ietf.org Subject: Re: [armd] ARMD problem space & existing standards IMHO, the additional initiatives that Florin describes in his email if implemented at (or near) the access switch and coupled with evolving standards such as VEPA and SRIOV technology and existing control plane scaling options (ex: scalable route reflector design, ORF, etc) could provide a VM-to-VM/end-to-end solution that scale well and enable extremely flexible virtual topologies without the need for an additional overlay. -- aldrin On Aug 17, 2011, at 7:27 PM, Balus, Florin Stelian (Florin) wrote: The discussion on ARMD problem space seem to gravitate around the issues basic VLAN deployments had in the past with flood containment. Before starting new protocol work maybe we should pause and start from what is missing in the latest IEEE/IETF technologies when it comes to addressing these issues. Here is a list of existing standards I am familiar with that are deployed in large Service Provider Ethernet Networks where we had to deal with similar issues: - IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS or IP-only, VPN "shim" is the ISID and/or PW label. - IEEE 802.1ah PBB - "core tunneling" is based on native Ethernet switching; VPN "shim" is the ISID 24 bit tag; may be transported over the above VPLS tunnels (see IETF PBB-VPLS work). Also there are additional initiatives in IEEE and IETF to add DC oriented features (e.g. Multi-pathing and control plane alternative to MAC learning) - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). This is not a comprehensive list, just an example of possible ways to address the issues without reinventing the wheel. A basic function of these solutions is that any flooded packet is delivered through the core efficiently only to the edge switches and server blades that have at least an interested VPN endpoint (VM). The service auto-discovery component provides on demand service connectivity only between the interested endpoints (e.g. Server Blades/VMs) inside or across DCs. So these solutions have the potential to address in my opinion the DC wide flooding. If further flood optimization inside an individual tenant domain is required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may be used also with these technologies: e.g. the mapping server approach can map VM MACs to the above shims + Tunnels as an alternative to regular IEEE MAC Learning. The discussion in draft-wkumari on separation of VM addressing between tenants and from the DC internal addressing highlights also a very important requirement that needs to be addressed for true isolation between tenants. But this could be achieved by re-using the existing VPN standards - this is a fundamental principle that was addressed already in the existing encapsulations. In addition the core tunneling is based on Service Provider addressing. Sorry for the long email. I am hoping people new to VPN work will benefit as much as we benefited from the good information on DC requirements. Florin _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --Boundary_(ID_B/UlpbB/dTR5Fl0kjoLx6A) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

    Thanks Florin for pointing out L2VPN & IEEE802.1ah Overlay Network. There is also TRILL encapsulation. All of these encapsulation schemes, including the IP encapsulation introduced by “draft-wkumari-dcops-l3-vmmobility-00”, achieve the goal of hiding individual hosts’ (or VMs) addresses from the backbone network.

     

    All of these encapsulation overlay mechanisms face the common challenge in Data Center: edge node of the overlay network has to maintain very large mapping for all remote hosts <-> their corresponding Edge nodes.

    The more number of hosts attached to Overlay Network Edge nodes, the more mapping has to be maintained by the Edge nodes. The goal of ARMD is to identify those problems. ARMD is not to introduce another encapsulation scheme.

     

    “draft-wkumari-dcops-l3-vmmobility-00” introduces “Mapping Server” to solve this problem. http://datatracker.ietf.org/doc/draft-dunbar-trill-directory-assisted-edge/ introduces the Directory Server to solve the problem for TRILL.

    Then there is a need for directory assistance for IEEE802.1ah, or other encapsulation schemes in Data Center because the massive number of hosts. It may be better for the industry to have a common approach on how Edge node can get assistance from directory servers.  

     

    Linda

     

    From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Aldrin Isaac
    Sent: Wednesday, August 17, 2011 10:17 PM
    To: Balus, Florin Stelian (Florin)
    Cc: armd@ietf.org
    Subject: Re: [armd] ARMD problem space & existing standards

     

    IMHO, the additional initiatives that Florin describes in his email if implemented at (or near) the access switch and coupled with evolving standards such as VEPA and SRIOV technology and existing control plane scaling options (ex: scalable route reflector design, ORF, etc) could provide a VM-to-VM/end-to-end solution that scale well and enable extremely flexible virtual topologies without the need for an additional overlay.  -- aldrin

     

     

    On Aug 17, 2011, at 7:27 PM, Balus, Florin Stelian (Florin) wrote:



    The discussion on ARMD problem space seem to gravitate around the issues basic VLAN deployments had in the past with flood containment. Before starting new protocol work maybe we should pause and start from what is missing in the latest IEEE/IETF technologies when it comes to addressing these issues.

     

    Here is a list of existing standards I am familiar with that are deployed in large Service Provider Ethernet Networks where we had to deal with similar issues:

    -      IETF-L2VPN WG VPLS/PBB-VPLS – “core tunneling” can be MPLS or IP-only, VPN “shim” is the ISID and/or PW label.

    -      IEEE 802.1ah PBB – “core tunneling” is based on native Ethernet switching; VPN “shim” is the ISID 24 bit tag; may be transported over the above VPLS tunnels (see IETF PBB-VPLS work).

     

    Also there are additional initiatives in IEEE and IETF to add DC oriented features (e.g. Multi-pathing and control plane alternative to MAC learning) - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB).  

     

    This is not a comprehensive list, just an example of possible ways to address the issues without reinventing the wheel.

     

    A basic function of these solutions is that any flooded packet is delivered through the core efficiently only to the edge switches and server blades that have at least an interested VPN endpoint (VM). The service auto-discovery component provides on demand service connectivity only between the interested endpoints (e.g. Server Blades/VMs) inside or across DCs. So these solutions have the potential to address in my opinion the DC wide flooding.

     

    If further flood optimization inside an individual tenant domain is required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may be used also with these technologies: e.g. the mapping server approach can map VM MACs to the above shims + Tunnels as an alternative to regular IEEE MAC Learning.

     

    The discussion in draft-wkumari on separation of VM addressing between tenants and from the DC internal addressing highlights also a very important requirement that needs to be addressed for true isolation between tenants. But this could be achieved by re-using the existing VPN standards - this is a fundamental principle that was addressed already in the existing encapsulations. In addition the core tunneling is based on Service Provider addressing.

     

    Sorry for the long email. I am hoping people new to VPN work will benefit as much as we benefited from the good information on DC requirements.

     

    Florin

     

    _______________________________________________
    armd mailing list
    armd@ietf.org
    https://www.ietf.org/mailman/listinfo/armd

     

    --Boundary_(ID_B/UlpbB/dTR5Fl0kjoLx6A)-- From florin.balus@alcatel-lucent.com Thu Aug 18 09:30:07 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 661B021F8B74 for ; Thu, 18 Aug 2011 09:30:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id amoiEBDAuP7L for ; Thu, 18 Aug 2011 09:30:02 -0700 (PDT) Received: from ihemail2.lucent.com (ihemail2.lucent.com [135.245.0.35]) by ietfa.amsl.com (Postfix) with ESMTP id 2C00F21F8B77 for ; Thu, 18 Aug 2011 09:30:02 -0700 (PDT) Received: from usnavsmail1.ndc.alcatel-lucent.com (usnavsmail1.ndc.alcatel-lucent.com [135.3.39.9]) by ihemail2.lucent.com (8.13.8/IER-o) with ESMTP id p7IGUle6029673 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 18 Aug 2011 11:30:48 -0500 (CDT) Received: from USNAVSXCHHUB02.ndc.alcatel-lucent.com (usnavsxchhub02.ndc.alcatel-lucent.com [135.3.39.111]) by usnavsmail1.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p7IGUj6o010863 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT); Thu, 18 Aug 2011 11:30:47 -0500 Received: from USNAVSXCHMBSC3.ndc.alcatel-lucent.com ([135.3.39.145]) by USNAVSXCHHUB02.ndc.alcatel-lucent.com ([135.3.39.111]) with mapi; Thu, 18 Aug 2011 11:30:46 -0500 From: "Balus, Florin Stelian (Florin)" To: Linda Dunbar , Aldrin Isaac Date: Thu, 18 Aug 2011 11:30:45 -0500 Thread-Topic: [armd] ARMD problem space & existing standards Thread-Index: AcxdNTFE+My2+OQJRqaZzuH96c0GBgAWtWsAAAlLM8AAATNwwA== Message-ID: <2073A6C5467C99478898544C6EBA3F4602BB419AE3@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> <4A95BA014132FF49AE685FAB4B9F17F605193C18@dfweml503-mbx.china.huawei.com> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F605193C18@dfweml503-mbx.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_2073A6C5467C99478898544C6EBA3F4602BB419AE3USNAVSXCHMBSC_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.57 on 135.245.2.35 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.9 Cc: "armd@ietf.org" Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 16:30:07 -0000 --_000_2073A6C5467C99478898544C6EBA3F4602BB419AE3USNAVSXCHMBSC_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable What I listed in my email is a list of standards that were developed to wor= k together in L2VPN and in sync with IEEE specification (encapsulation/tunn= eling/VPN shim). This is valuable development especially for inter-DC/VPN-t= enant connect since it interoperates with what is already deployed. Any new encapsulations for DC space should to go through the same kind of L= 2VPN interop discussion/scrutiny since the requirements at least for DC int= erconnect are similar with what we saw in L2VPN in the past. We need to ens= ure there is a good reason to introduce new VPN shims. I would say the same= thing applies to the native Ethernet space - IEEE 802.1 Task Force should = get a liaison every time we try to invent new Ethernet VPN shim - see discu= ssion we had in TRILL on two VLAN tags as VPN shim. IEEE already provided ~= 4 years ago an evolution of the 12 bit VLAN tag to 24 bit PBB ISID tag and= the appropriate encapsulation hierarchy to hide the (VM) MACs from the bac= kbone nodes. New control planes/server driven networking should re-use as much as possib= le the existing encaps to simplify at least the forwarding options. See also in-line... From: Linda Dunbar [mailto:linda.dunbar@huawei.com] Sent: Thursday, August 18, 2011 8:07 AM To: Aldrin Isaac; Balus, Florin Stelian (Florin) Cc: armd@ietf.org Subject: RE: [armd] ARMD problem space & existing standards Thanks Florin for pointing out L2VPN & IEEE802.1ah Overlay Network. There i= s also TRILL encapsulation. All of these encapsulation schemes, including t= he IP encapsulation introduced by "draft-wkumari-dcops-l3-vmmobility-00", a= chieve the goal of hiding individual hosts' (or VMs) addresses from the bac= kbone network. All of these encapsulation overlay mechanisms face the common challenge in = Data Center: edge node of the overlay network has to maintain very large ma= pping for all remote hosts <-> their corresponding Edge nodes. The more number of hosts attached to Overlay Network Edge nodes, the more m= apping has to be maintained by the Edge nodes. The goal of ARMD is to ident= ify those problems. ARMD is not to introduce another encapsulation scheme. "draft-wkumari-dcops-l3-vmmobility-00" introduces "Mapping Server" to solve= this problem. http://datatracker.ietf.org/doc/draft-dunbar-trill-directory= -assisted-edge/ introduces the Directory Server to solve the problem for TR= ILL. Then there is a need for directory assistance for IEEE802.1ah, or other enc= apsulation schemes in Data Center because the massive number of hosts. It m= ay be better for the industry to have a common approach on how Edge node ca= n get assistance from directory servers. FB> I assume that by Edge nodes you mean the place where the overlay tunnel= starts... When I wrote the initial email I had in mind what the overlay so= lutions are generally trying to address: allow a L2 tenant domain (ELAN) wi= th hosts distributed across a large backbone domain with scalable tunneling= , usually using routing protocols in the control plane. In DC talk that mea= ns related hosts can be distributed anywhere inside or across DCs while con= taining the amount of flooding/solving DC wide flooding. Implicitly this me= ans server blades are not bombarded with ARP/ND requests from any unrelated= server blade (from outside their tenant domain). If the issue is the number of hosts per VPN - i.e. a server blade has 32 VM= s every one with its own L2 tenant domain and every domain with X hosts - t= here are different issues one might try to solve: - Further optimize flooding per tenant domain and processing of ARP/ND= requests at the receiving end by cutting the initiation of flooded traffic= at the source: i.e. eliminate ARP transmission/MAC learning through Mappin= g server/Directory etc... - X*32 >> host entries the server blade can handle - not sure what can= be done about this in the network? One either needs to add memory in the s= erver blade or just don't do it as the DC guys said in the ARMD session. Th= e solution is proper planning of the average number of hosts per L2 domain = (i.e. one may allow a large number for some L2 domains but not for all)/dro= p a VRF (routing) instance when needed. Same discussion applies at a differ= ent scale if your edge node is the ToR. Most of the VPLS/PBB implementation= s I am aware of have per VPN/tenant and per interface enforcement of the nu= mber of hosts (MAC addresses, flooded/mcast traffic) to ensure the proper e= ngineering. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Ald= rin Isaac Sent: Wednesday, August 17, 2011 10:17 PM To: Balus, Florin Stelian (Florin) Cc: armd@ietf.org Subject: Re: [armd] ARMD problem space & existing standards IMHO, the additional initiatives that Florin describes in his email if impl= emented at (or near) the access switch and coupled with evolving standards = such as VEPA and SRIOV technology and existing control plane scaling option= s (ex: scalable route reflector design, ORF, etc) could provide a VM-to-VM/= end-to-end solution that scale well and enable extremely flexible virtual t= opologies without the need for an additional overlay. -- aldrin On Aug 17, 2011, at 7:27 PM, Balus, Florin Stelian (Florin) wrote: The discussion on ARMD problem space seem to gravitate around the issues ba= sic VLAN deployments had in the past with flood containment. Before startin= g new protocol work maybe we should pause and start from what is missing in= the latest IEEE/IETF technologies when it comes to addressing these issues= . Here is a list of existing standards I am familiar with that are deployed i= n large Service Provider Ethernet Networks where we had to deal with simila= r issues: - IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS or IP-onl= y, VPN "shim" is the ISID and/or PW label. - IEEE 802.1ah PBB - "core tunneling" is based on native Ethernet swit= ching; VPN "shim" is the ISID 24 bit tag; may be transported over the above= VPLS tunnels (see IETF PBB-VPLS work). Also there are additional initiatives in IEEE and IETF to add DC oriented f= eatures (e.g. Multi-pathing and control plane alternative to MAC learning) = - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). This is not a comprehensive list, just an example of possible ways to addre= ss the issues without reinventing the wheel. A basic function of these solutions is that any flooded packet is delivered= through the core efficiently only to the edge switches and server blades t= hat have at least an interested VPN endpoint (VM). The service auto-discove= ry component provides on demand service connectivity only between the inter= ested endpoints (e.g. Server Blades/VMs) inside or across DCs. So these sol= utions have the potential to address in my opinion the DC wide flooding. If further flood optimization inside an individual tenant domain is require= d, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may be used a= lso with these technologies: e.g. the mapping server approach can map VM MA= Cs to the above shims + Tunnels as an alternative to regular IEEE MAC Learn= ing. The discussion in draft-wkumari on separation of VM addressing between tena= nts and from the DC internal addressing highlights also a very important re= quirement that needs to be addressed for true isolation between tenants. Bu= t this could be achieved by re-using the existing VPN standards - this is a= fundamental principle that was addressed already in the existing encapsula= tions. In addition the core tunneling is based on Service Provider addressi= ng. Sorry for the long email. I am hoping people new to VPN work will benefit a= s much as we benefited from the good information on DC requirements. Florin _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --_000_2073A6C5467C99478898544C6EBA3F4602BB419AE3USNAVSXCHMBSC_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

    What I listed in my email is a list of standards that were d= eveloped to work together in L2VPN and in sync with IEEE specification (encapsulatio= n/tunneling/VPN shim). This is valuable development especially for inter-DC/VPN-tenant conn= ect since it interoperates with what is already deployed.

     

    Any new encapsulations for DC space should to go through the same kind of L2VPN interop discussion/scrutiny since the requirements at le= ast for DC interconnect are similar with what we saw in L2VPN in the past. We n= eed to ensure there is a good reason to introduce new VPN shims. I would say th= e same thing applies to the native Ethernet space - IEEE 802.1 Task Force sho= uld get a liaison every time we try to invent new Ethernet VPN shim – see= discussion we had in TRILL on two VLAN tags as VPN shim. IEEE already provided ~ 4 yea= rs ago an evolution of the 12 bit VLAN tag to 24 bit PBB ISID tag and the appropriate encapsulation hierarchy to hide the (VM) MACs from the backbone nodes.

     

    New control planes/server driven networking should re-use as much as possible the existing encaps to simplify at least the forwarding options.

     

    See also in-line…

     

    From: Linda Dunbar [mailto:linda.dunbar@huawei.com]
    Sent: Thursday, August 18, 2011 8:07 AM
    To: Aldrin Isaac; Balus, Florin Stelian (Florin)
    Cc: armd@ietf.org
    Subject: RE: [armd] ARMD problem space & existing standards=

     

    Thanks Florin for pointing out L2VPN & IEEE802.1ah Overl= ay Network. There is also TRILL encapsulation. All of these encapsulation sche= mes, including the IP encapsulation introduced by “draft-wkumari-dcops-l3-vmmobility-00”, achieve the goal of hid= ing individual hosts’ (or VMs) addresses from the backbone network. =

     

    All of these encapsulation overlay mechanisms face the commo= n challenge in Data Center: edge node of the overlay network has to maintain = very large mapping for all remote hosts <-> their corresponding Edge nodes= .

     

    The more number of hosts attached to Overlay Network Edge no= des, the more mapping has to be maintained by the Edge nodes. The goal of ARMD i= s to identify those problems. ARMD is not to introduce another encapsulation sch= eme.

     

    “draft-wkumari-dcops-l3-vmmobility-00” introduce= s “Mapping Server” to solve this problem. http://datatracker.ietf.org/doc/draft-dunbar-trill-directory-assi= sted-edge/ introduces the Directory Server to solve the problem for TRILL. =

    Then there is a need for directory assistance for IEEE802.1a= h, or other encapsulation schemes in Data Center because the massive number of hosts. It may be better for the industry to have a common approach on how Edge node can get assistance from director= y servers.  

     

    FB> I assume that by Edge nodes you mean the place where th= e overlay tunnel starts… When I wrote the initial email I had in mind w= hat the overlay solutions are generally trying to address: allow a L2 tenant domain (ELAN) with hosts distributed across a large backbone domain with scalable tunneling, usually using routing protocols in the control plane. In DC talk that means related hosts can be distributed anywhere inside or across DCs w= hile containing the amount of flooding/solving DC wide flooding. Implicitly this means server blades are not bombarded with ARP/ND requests from any unrelat= ed server blade (from outside their tenant domain).

     

    If the issue is the number of hosts per VPN – i.e. a ser= ver blade has 32 VMs every one with its own L2 tenant domain and every domain w= ith X hosts – there are different issues one might try to solve:

    -&nb= sp;     Further optimize flooding per tenant domain and processing of ARP/ND requests at the receiving end by cutting the initiation of flooded traffic at the source: i.e. eliminate ARP transmission/MAC learning through= Mapping server/Directory etc…

     

    -&nb= sp;     X*32 >> host entries the server blade can handle –= not sure what can be done about this in the network? One either needs to add me= mory in the server blade or just don’t do it as the DC guys said in the AR= MD session. The solution is proper planning of the average number of hosts per= L2 domain (i.e. one may allow a large number for some L2 domains but not for all)/dro= p a VRF (routing) instance when needed. Same discussion applies at a different scal= e if your edge node is the ToR. Most of the VPLS/PBB implementations I am aware = of have per VPN/tenant and per interface enforcement of the number of hosts (M= AC addresses, flooded/mcast traffic) to ensure the proper engineering.

     

     

    Linda

     

    From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Al= drin Isaac
    Sent: Wednesday, August 17, 2011 10:17 PM
    To: Balus, Florin Stelian (Florin)
    Cc: armd@ietf.org
    Subject: Re: [armd] ARMD problem space & existing standards=

     

    IMHO, the additional initiatives that Florin describes= in his email if implemented at (or near) the access switch and coupled with evolving standards such as VEPA and SRIOV technology and existing cont= rol plane scaling options (ex: scalable route reflector design, ORF, etc) could provide a VM-to-VM/end-to-end solution that scale well and enable extremely= flexible virtual topologies without the need for an additional overlay.  -- ald= rin

     

     

    On Aug 17, 2011, at 7:27 PM, Balus, Florin Stelian (Fl= orin) wrote:

     

    The discussion on ARMD problem space seem to gravitate around the issues basic = VLAN deployments had in the past with flood containment. Before starting new protocol work maybe we should pause and start from what is missing in the latest IEEE/IETF technologies when it comes to addressing these issues.

     

    Here is a list of existing standards I am familiar with that are deployed in lar= ge Service Provider Ethernet Networks where we had to deal with similar issues= :

    -      IETF-L2VPN WG VPLS/PBB-VPLS –= ; “core tunneling” can be MPLS or IP-only, VPN “shim”= is the ISID and/or PW label.

    -      IEEE 802.1ah PBB – “co= re tunneling” is based on native Ethernet switching; VPN “shimR= 21; is the ISID 24 bit tag; may be transported over the above VPLS tunnels (see IETF PBB-VPLS work).

     

    Also there are additional initiatives in IEEE and IETF to add DC oriented featur= es (e.g. Multi-pathing and control plane alternative to MAC learning) - see IE= TF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB).  

     

    This is not a comprehensive list, just an example of possible ways to address th= e issues without reinventing the wheel.

     

    A basic function of these solutions is that any flooded packet is delivered through the core efficiently only to the edge switches and server blades th= at have at least an interested VPN endpoint (VM). The service auto-discovery component provides on demand service connectivity only between the interest= ed endpoints (e.g. Server Blades/VMs) inside or across DCs. So these solutions have the potential to address in my opinion the DC wide flooding.

     

    If further flood optimization inside an individual tenant domain is required, = the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may be used also wit= h these technologies: e.g. the mapping server approach can map VM MACs to the above shims + Tunnels as an alternative to regular IEEE MAC Learning.

     

    The discussion in draft-wkumari on separation of VM addressing between tenants = and from the DC internal addressing highlights also a very important requiremen= t that needs to be addressed for true isolation between tenants. But this cou= ld be achieved by re-using the existing VPN standards - this is a fundamental principle that was addressed already in the existing encapsulations. In addition the core tunneling is based on Service Provider addressing.=

     

    Sorry for the long email. I am hoping people new to VPN work will benefit as much= as we benefited from the good information on DC requirements.

     

    Florin

     

    __= _____________________________________________
    armd mailing list
    armd@ietf.org
    https://www.ietf.org= /mailman/listinfo/armd

     

    --_000_2073A6C5467C99478898544C6EBA3F4602BB419AE3USNAVSXCHMBSC_-- From bedard.phil@gmail.com Thu Aug 18 10:44:35 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D674D21F8B8F for ; Thu, 18 Aug 2011 10:44:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MzxqoQzWS9+Z for ; Thu, 18 Aug 2011 10:44:35 -0700 (PDT) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by ietfa.amsl.com (Postfix) with ESMTP id B507121F8B8C for ; Thu, 18 Aug 2011 10:44:34 -0700 (PDT) Received: by wwf5 with SMTP id 5so1560722wwf.13 for ; Thu, 18 Aug 2011 10:45:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=NqXsRZ1SfSzjKA5ah7d2FIGNMMq2r9kkcJhYk4UFyvU=; b=JQdV54sS/5uhdPriBmJW9XPT6GfjFf8HYVMTXsrIUQZPnhrR3VN/I6G5ouwYHsjj83 ft+txBQxNg2C305b+J6U/zBOyVXAWgNIZ/LY0nO2Nw6XGKLaNpDyuw2l/+3rDhLMUFq4 dM2N543/qyz68QTeJzyn03dpGPLiM90n8JxuY= MIME-Version: 1.0 Received: by 10.216.169.211 with SMTP id n61mr5586070wel.83.1313689528553; Thu, 18 Aug 2011 10:45:28 -0700 (PDT) Received: by 10.216.87.196 with HTTP; Thu, 18 Aug 2011 10:45:28 -0700 (PDT) In-Reply-To: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Date: Thu, 18 Aug 2011 13:45:28 -0400 Message-ID: From: Phil Bedard To: "Balus, Florin Stelian (Florin)" , "armd@ietf.org" Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 17:44:35 -0000 I think the issue thus far is implementing this in access switches close to the hosts... These are all technologies used to enhance the scale of L2 topologies. The Google case in one where I believe there is a L3 IP hop between the two VM hosts which necessitates tunneling. In order for VMs to retain their connectivity the L2 has to be dynamically tunneled over L3 from server to server and gateway to server. In a L2 network you need to reconfigure the upstream switch port when a VM moves or preconfigure ports. I see extensions to LLDP like VEPA/multichannel which may allow more dynamic reconfiguration without a management tool having to reconfigure the upstream network device. With regards to ARP scale, whether emulated via VPLS or native, there are probably multiple solutions. ARP snooping is probably not a difficult operation since most do it today for IGMP and/or PIM and many devices perform proxy-arp. I mentioned Ethernet VPN earlier as a solution which can be implemented with a scalable protocol like BGP to push MAC information or one could alternatively use a directory lookup instead. EVPN I believe has a provision to perform ARP snooping. Phil On 8/17/11, Balus, Florin Stelian (Florin) wrote: > The discussion on ARMD problem space seem to gravitate around the issues > basic VLAN deployments had in the past with flood containment. Before > starting new protocol work maybe we should pause and start from what is > missing in the latest IEEE/IETF technologies when it comes to addressing > these issues. > > Here is a list of existing standards I am familiar with that are deployed in > large Service Provider Ethernet Networks where we had to deal with similar > issues: > > - IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS or > IP-only, VPN "shim" is the ISID and/or PW label. > > - IEEE 802.1ah PBB - "core tunneling" is based on native Ethernet > switching; VPN "shim" is the ISID 24 bit tag; may be transported over the > above VPLS tunnels (see IETF PBB-VPLS work). > > Also there are additional initiatives in IEEE and IETF to add DC oriented > features (e.g. Multi-pathing and control plane alternative to MAC learning) > - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). > > This is not a comprehensive list, just an example of possible ways to > address the issues without reinventing the wheel. > > A basic function of these solutions is that any flooded packet is delivered > through the core efficiently only to the edge switches and server blades > that have at least an interested VPN endpoint (VM). The service > auto-discovery component provides on demand service connectivity only > between the interested endpoints (e.g. Server Blades/VMs) inside or across > DCs. So these solutions have the potential to address in my opinion the DC > wide flooding. > > If further flood optimization inside an individual tenant domain is > required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may be > used also with these technologies: e.g. the mapping server approach can map > VM MACs to the above shims + Tunnels as an alternative to regular IEEE MAC > Learning. > > The discussion in draft-wkumari on separation of VM addressing between > tenants and from the DC internal addressing highlights also a very important > requirement that needs to be addressed for true isolation between tenants. > But this could be achieved by re-using the existing VPN standards - this is > a fundamental principle that was addressed already in the existing > encapsulations. In addition the core tunneling is based on Service Provider > addressing. > > Sorry for the long email. I am hoping people new to VPN work will benefit as > much as we benefited from the good information on DC requirements. > > Florin > > -- Sent from my mobile device From florin.balus@alcatel-lucent.com Thu Aug 18 13:42:13 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1958F21F8561 for ; Thu, 18 Aug 2011 13:42:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t4wT+Smb-7Ep for ; Thu, 18 Aug 2011 13:42:12 -0700 (PDT) Received: from ihemail2.lucent.com (ihemail2.lucent.com [135.245.0.35]) by ietfa.amsl.com (Postfix) with ESMTP id EB3EA21F8559 for ; Thu, 18 Aug 2011 13:42:11 -0700 (PDT) Received: from usnavsmail2.ndc.alcatel-lucent.com (usnavsmail2.ndc.alcatel-lucent.com [135.3.39.10]) by ihemail2.lucent.com (8.13.8/IER-o) with ESMTP id p7IKh5k5029866 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 18 Aug 2011 15:43:05 -0500 (CDT) Received: from USNAVSXCHHUB03.ndc.alcatel-lucent.com (usnavsxchhub03.ndc.alcatel-lucent.com [135.3.39.112]) by usnavsmail2.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p7IKh5UB012861 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT); Thu, 18 Aug 2011 15:43:05 -0500 Received: from USNAVSXCHMBSC3.ndc.alcatel-lucent.com ([135.3.39.145]) by USNAVSXCHHUB03.ndc.alcatel-lucent.com ([135.3.39.112]) with mapi; Thu, 18 Aug 2011 15:43:05 -0500 From: "Balus, Florin Stelian (Florin)" To: Phil Bedard , "armd@ietf.org" Date: Thu, 18 Aug 2011 15:43:03 -0500 Thread-Topic: [armd] ARMD problem space & existing standards Thread-Index: AcxdzqAwajO5rTH0ReKy3TLPnx8KYgAAgTuw Message-ID: <2073A6C5467C99478898544C6EBA3F4602BB419BF2@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.57 on 135.245.2.35 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.10 Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 20:42:13 -0000 Thanks Phil for the comments. See in-line... -----Original Message----- From: Phil Bedard [mailto:bedard.phil@gmail.com]=20 Sent: Thursday, August 18, 2011 10:45 AM To: Balus, Florin Stelian (Florin); armd@ietf.org Subject: Re: [armd] ARMD problem space & existing standards I think the issue thus far is implementing this in access switches close to the hosts...=20 FB> When it comes to internal DC network I understand VLANs/Ethernet switch= ing are usually used in access switches (ToRs). Backbone Ethernet tunneling= was developed by IEEE to maintain the same basic procedures and as far as = I know the chipsets built for these kind of devices support both TRILL and = PBB. These are all technologies used to enhance the scale of L2 topologies. FB> maybe a better way to put it is these technologies allow related L2 swi= tching instances to be located far away from each other, across one or a co= mbination of L2 or L3 (IP/MPLS) backbones. There are metro, national and ev= en intercontinental deployments of these technologies. The Google case in one where I believe there is a L3 IP hop between the two VM hosts which necessitates tunneling. In order for VMs to retain their connectivity the L2 has to be dynamically tunneled over L3 from server to server and gateway to server. =20 FB> I understand there is value in re-using the VLAN + IP routing environme= nt. That was the whole point of PBB-VPLS model and interop work in L2VPN: t= o allow a similar (Backbone) VLAN + (IP or MPLS) tunnel combination.=20 In a L2 network you need to reconfigure the upstream switch port when a VM= moves or preconfigure ports. I see extensions to LLDP like VEPA/multichannel which may allow more dynamic reconfiguration without a management tool having to reconfigure the upstream network device. With regards to ARP scale, whether emulated via VPLS or native, there are probably multiple solutions. ARP snooping is probably not a difficult operation since most do it today for IGMP and/or PIM and many devices perform proxy-arp. I mentioned Ethernet VPN earlier as a solution which can be implemented with a scalable protocol like BGP to push MAC information or one could alternatively use a directory lookup instead. EVPN I believe has a provision to perform ARP snooping. FB> Yes there is some discussion on ARP containment in EVPN draft - see sec= tion 13 in http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-01 = ... Phil On 8/17/11, Balus, Florin Stelian (Florin) wrote: > The discussion on ARMD problem space seem to gravitate around the issues > basic VLAN deployments had in the past with flood containment. Before > starting new protocol work maybe we should pause and start from what is > missing in the latest IEEE/IETF technologies when it comes to addressing > these issues. > > Here is a list of existing standards I am familiar with that are deployed= in > large Service Provider Ethernet Networks where we had to deal with simila= r > issues: > > - IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS or > IP-only, VPN "shim" is the ISID and/or PW label. > > - IEEE 802.1ah PBB - "core tunneling" is based on native Ethernet > switching; VPN "shim" is the ISID 24 bit tag; may be transported over the > above VPLS tunnels (see IETF PBB-VPLS work). > > Also there are additional initiatives in IEEE and IETF to add DC oriented > features (e.g. Multi-pathing and control plane alternative to MAC learnin= g) > - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). > > This is not a comprehensive list, just an example of possible ways to > address the issues without reinventing the wheel. > > A basic function of these solutions is that any flooded packet is deliver= ed > through the core efficiently only to the edge switches and server blades > that have at least an interested VPN endpoint (VM). The service > auto-discovery component provides on demand service connectivity only > between the interested endpoints (e.g. Server Blades/VMs) inside or acros= s > DCs. So these solutions have the potential to address in my opinion the D= C > wide flooding. > > If further flood optimization inside an individual tenant domain is > required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may = be > used also with these technologies: e.g. the mapping server approach can m= ap > VM MACs to the above shims + Tunnels as an alternative to regular IEEE MA= C > Learning. > > The discussion in draft-wkumari on separation of VM addressing between > tenants and from the DC internal addressing highlights also a very import= ant > requirement that needs to be addressed for true isolation between tenants= . > But this could be achieved by re-using the existing VPN standards - this = is > a fundamental principle that was addressed already in the existing > encapsulations. In addition the core tunneling is based on Service Provid= er > addressing. > > Sorry for the long email. I am hoping people new to VPN work will benefit= as > much as we benefited from the good information on DC requirements. > > Florin > > --=20 Sent from my mobile device From wim.henderickx@alcatel-lucent.com Thu Aug 18 22:11:41 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E7DD421F86AA for ; Thu, 18 Aug 2011 22:11:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.249 X-Spam-Level: X-Spam-Status: No, score=-4.249 tagged_above=-999 required=5 tests=[AWL=2.000, BAYES_00=-2.599, HELO_EQ_FR=0.35, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ALr5lrFSpSzQ for ; Thu, 18 Aug 2011 22:11:40 -0700 (PDT) Received: from smail3.alcatel.fr (smail3.alcatel.fr [64.208.49.56]) by ietfa.amsl.com (Postfix) with ESMTP id A2A9421F8514 for ; Thu, 18 Aug 2011 22:11:39 -0700 (PDT) Received: from FRMRSSXCHHUB01.dc-m.alcatel-lucent.com (FRMRSSXCHHUB01.dc-m.alcatel-lucent.com [135.120.45.61]) by smail3.alcatel.fr (8.14.3/8.14.3/ICT) with ESMTP id p7J5CNOk011183 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT); Fri, 19 Aug 2011 07:12:23 +0200 Received: from FRMRSSXCHMBSB1.dc-m.alcatel-lucent.com ([135.120.45.41]) by FRMRSSXCHHUB01.dc-m.alcatel-lucent.com ([135.120.45.61]) with mapi; Fri, 19 Aug 2011 07:12:23 +0200 From: "Henderickx, Wim (Wim)" To: "Balus, Florin Stelian (Florin)" , Phil Bedard , "armd@ietf.org" Date: Fri, 19 Aug 2011 07:12:23 +0200 Thread-Topic: [armd] ARMD problem space & existing standards Thread-Index: AcxdzqAwajO5rTH0ReKy3TLPnx8KYgAAgTuwABdiQJA= Message-ID: <14C7F4F06DB5814AB0DE29716C4F6D671A6AAF6E@FRMRSSXCHMBSB1.dc-m.alcatel-lucent.com> References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> <2073A6C5467C99478898544C6EBA3F4602BB419BF2@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> In-Reply-To: <2073A6C5467C99478898544C6EBA3F4602BB419BF2@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Accept-Language: nl-NL, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: nl-NL, en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.69 on 155.132.188.83 Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Aug 2011 05:11:42 -0000 Sorry, to join the discussion a bit later.=20 Below is also a draft (written by Rahul and a little bit of me) which can a= lso be referenced in this discussion. It is identifying some solution for p= otential issues in the DC. http://tools.ietf.org/html/draft-raggarwa-data-center-mobility-00 -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Bal= us, Florin Stelian (Florin) Sent: donderdag 18 augustus 2011 22:43 To: Phil Bedard; armd@ietf.org Subject: Re: [armd] ARMD problem space & existing standards Thanks Phil for the comments. See in-line... -----Original Message----- From: Phil Bedard [mailto:bedard.phil@gmail.com]=20 Sent: Thursday, August 18, 2011 10:45 AM To: Balus, Florin Stelian (Florin); armd@ietf.org Subject: Re: [armd] ARMD problem space & existing standards I think the issue thus far is implementing this in access switches close to the hosts...=20 FB> When it comes to internal DC network I understand VLANs/Ethernet switch= ing are usually used in access switches (ToRs). Backbone Ethernet tunneling= was developed by IEEE to maintain the same basic procedures and as far as = I know the chipsets built for these kind of devices support both TRILL and = PBB. These are all technologies used to enhance the scale of L2 topologies. FB> maybe a better way to put it is these technologies allow related L2 swi= tching instances to be located far away from each other, across one or a co= mbination of L2 or L3 (IP/MPLS) backbones. There are metro, national and ev= en intercontinental deployments of these technologies. The Google case in one where I believe there is a L3 IP hop between the two VM hosts which necessitates tunneling. In order for VMs to retain their connectivity the L2 has to be dynamically tunneled over L3 from server to server and gateway to server. =20 FB> I understand there is value in re-using the VLAN + IP routing environme= nt. That was the whole point of PBB-VPLS model and interop work in L2VPN: t= o allow a similar (Backbone) VLAN + (IP or MPLS) tunnel combination.=20 In a L2 network you need to reconfigure the upstream switch port when a VM= moves or preconfigure ports. I see extensions to LLDP like VEPA/multichannel which may allow more dynamic reconfiguration without a management tool having to reconfigure the upstream network device. With regards to ARP scale, whether emulated via VPLS or native, there are probably multiple solutions. ARP snooping is probably not a difficult operation since most do it today for IGMP and/or PIM and many devices perform proxy-arp. I mentioned Ethernet VPN earlier as a solution which can be implemented with a scalable protocol like BGP to push MAC information or one could alternatively use a directory lookup instead. EVPN I believe has a provision to perform ARP snooping. FB> Yes there is some discussion on ARP containment in EVPN draft - see sec= tion 13 in http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-01 = ... Phil On 8/17/11, Balus, Florin Stelian (Florin) wrote: > The discussion on ARMD problem space seem to gravitate around the issues > basic VLAN deployments had in the past with flood containment. Before > starting new protocol work maybe we should pause and start from what is > missing in the latest IEEE/IETF technologies when it comes to addressing > these issues. > > Here is a list of existing standards I am familiar with that are deployed= in > large Service Provider Ethernet Networks where we had to deal with simila= r > issues: > > - IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS or > IP-only, VPN "shim" is the ISID and/or PW label. > > - IEEE 802.1ah PBB - "core tunneling" is based on native Ethernet > switching; VPN "shim" is the ISID 24 bit tag; may be transported over the > above VPLS tunnels (see IETF PBB-VPLS work). > > Also there are additional initiatives in IEEE and IETF to add DC oriented > features (e.g. Multi-pathing and control plane alternative to MAC learnin= g) > - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). > > This is not a comprehensive list, just an example of possible ways to > address the issues without reinventing the wheel. > > A basic function of these solutions is that any flooded packet is deliver= ed > through the core efficiently only to the edge switches and server blades > that have at least an interested VPN endpoint (VM). The service > auto-discovery component provides on demand service connectivity only > between the interested endpoints (e.g. Server Blades/VMs) inside or acros= s > DCs. So these solutions have the potential to address in my opinion the D= C > wide flooding. > > If further flood optimization inside an individual tenant domain is > required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may = be > used also with these technologies: e.g. the mapping server approach can m= ap > VM MACs to the above shims + Tunnels as an alternative to regular IEEE MA= C > Learning. > > The discussion in draft-wkumari on separation of VM addressing between > tenants and from the DC internal addressing highlights also a very import= ant > requirement that needs to be addressed for true isolation between tenants= . > But this could be achieved by re-using the existing VPN standards - this = is > a fundamental principle that was addressed already in the existing > encapsulations. In addition the core tunneling is based on Service Provid= er > addressing. > > Sorry for the long email. I am hoping people new to VPN work will benefit= as > much as we benefited from the good information on DC requirements. > > Florin > > --=20 Sent from my mobile device _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From christopher.morrow@gmail.com Fri Aug 19 08:42:34 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8724E21F8A30 for ; Fri, 19 Aug 2011 08:42:34 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.599 X-Spam-Level: X-Spam-Status: No, score=-103.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZFbxsPK4DWbW for ; Fri, 19 Aug 2011 08:42:33 -0700 (PDT) Received: from mail-yi0-f44.google.com (mail-yi0-f44.google.com [209.85.218.44]) by ietfa.amsl.com (Postfix) with ESMTP id 76C3321F876A for ; Fri, 19 Aug 2011 08:42:33 -0700 (PDT) Received: by yie12 with SMTP id 12so2647729yie.31 for ; Fri, 19 Aug 2011 08:43:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=wiUZIZdjl3ohr3kNVo2ZeciO8ZLvxBe8I3ew8pheTtY=; b=N+nv9P0rhli08+sA2vsdJ7CV+Lw8kyN4tWeO+HxK5JaasXmMGDTMY2S0gMLn512Kok OczfmK2iG6aJQ5M2NYCPK4YvSD1KoA+e4FATHp6YkdQeh6WhBD3aj5ExcYZvf1JKIQig ArmbG3yT3rrC0ZE3Al115UVQDCoKvPbtD/9PA= MIME-Version: 1.0 Received: by 10.236.179.72 with SMTP id g48mr8790400yhm.50.1313768607243; Fri, 19 Aug 2011 08:43:27 -0700 (PDT) Sender: christopher.morrow@gmail.com Received: by 10.231.116.164 with HTTP; Fri, 19 Aug 2011 08:43:27 -0700 (PDT) In-Reply-To: <2073A6C5467C99478898544C6EBA3F4602BB419BF2@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> <2073A6C5467C99478898544C6EBA3F4602BB419BF2@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Date: Fri, 19 Aug 2011 11:43:27 -0400 X-Google-Sender-Auth: PMWpb0MNw7B-5X9qzxMoOlvALvw Message-ID: From: Christopher Morrow To: "Balus, Florin Stelian (Florin)" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "armd@ietf.org" Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Aug 2011 15:42:34 -0000 On Thu, Aug 18, 2011 at 4:43 PM, Balus, Florin Stelian (Florin) wrote: > Thanks Phil for the comments. See in-line... > > -----Original Message----- > From: Phil Bedard [mailto:bedard.phil@gmail.com] > Sent: Thursday, August 18, 2011 10:45 AM > To: Balus, Florin Stelian (Florin); armd@ietf.org > Subject: Re: [armd] ARMD problem space & existing standards > > I think the issue thus far is implementing this in access switches > close to the hosts... This seems correct to me, though there's likely some pain on the first L3 hop in the DC as well. It seems that once you step out of the immediate L2 domain, normal backbone routing things are still fine in v4/v6. You MAY have issues wrt routing scale, but of course you have those anyway. The particular problem(s) that ARMD was/is looking at are the scaling properties of the edge nodes (access-switch, TOR, first L3 hop, server-side), and how those interact in a much larger potential L2 domain. Today, for instance, in a normal DC, a single rack is potentially it's own L2 domain (there are lots of architectures, let us keep it simple though). That domain, in v4, is contained and relatively small (/24 or smaller). In v6 though, that domain is potentially much larger (/64 default deployments for lots of reasons). Hosts on the /24-equivalent can only make so many ARP requests, the network gear has only to cache a limited number of entries, and things 'just work'. Hosts on a /64 though, or a larger flat L2 deployment, see much more ARP traffic, the associated network gear has to maintain larger queues/caches and deal with a larger flow of ARP/etc requests. The pathway from line-port to routing-engine/route-processor/smarts in a network device are severely constrained (as compared with the line-port -> line-port speeds/pathways). As gear gets toward the less-expensive range, it also gets less capable in this respect. Scaling up to multiple /24's of space across a distributed L2 is .... much harder to deal with, traffic-to-cpu wise. Additionally, less expensive devices are generally not fortified with faster/more-capable cpus. The drive to lower costs in the DC (cheaper network deployments, more vm-guests's per vm-host, etc is exacerbating this problem, or so it would seem, in a large flat L2 network. > > FB> When it comes to internal DC network I understand VLANs/Ethernet swit= ching are usually used in access switches (ToRs). Backbone Ethernet tunneli= ng was developed by IEEE to maintain the same basic procedures and as far a= s I know the chipsets built for these kind of devices support both TRILL an= d PBB. > > These are all technologies used to enhance the scale of L2 topologies. > the breadth, they aren't really necessary for 'scale'... you can just connect lots of L2 switches into a large SCALE LAN, but making it work across the wide-area is where the above comes into play. (work WELL at least) > FB> maybe a better way to put it is these technologies allow related L2 s= witching instances to be located far away from each other, across one or a = combination of L2 or L3 (IP/MPLS) backbones. There are metro, national and = even intercontinental deployments of these technologies. > right, breadth, not 'scale' (distance not numbers) > =A0The Google case in one where I believe there is a L3 IP hop between > the two VM hosts which necessitates tunneling. =A0In order for VMs to > retain their connectivity the L2 has to be dynamically tunneled over > L3 from server to server and gateway to server. > > FB> I understand there is value in re-using the VLAN + IP routing environ= ment. That was the whole point of PBB-VPLS model and interop work in L2VPN:= to allow a similar (Backbone) VLAN + (IP or MPLS) tunnel combination. > > In a L2 network you need =A0to reconfigure the upstream switch port when = a VM moves or > preconfigure ports. =A0 I see extensions to LLDP like VEPA/multichannel > which may allow more dynamic reconfiguration without a management tool > having to reconfigure the upstream network device. > > With regards to ARP scale, whether emulated via VPLS or native, there > are probably multiple solutions. =A0ARP snooping is probably not a > difficult operation since most do it today for IGMP and/or PIM and > many devices perform proxy-arp. arp snooping and the like work today on smaller deployments (numbers-wise) they probably have severe challenges when you attempt to scale up the number of arping nodes downstream from a port on a switch though. You get to some of that (florin) in the discussion of numbers three messages previous. -chris > I mentioned Ethernet VPN earlier as a solution which can be > implemented with a scalable protocol like BGP to push MAC information > or one could alternatively use a directory lookup instead. =A0 EVPN I > believe has a provision to perform ARP snooping. > > FB> Yes there is some discussion on ARP containment in EVPN draft - see s= ection 13 in http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-0= 1 ... > > Phil > > On 8/17/11, Balus, Florin Stelian (Florin) > wrote: >> The discussion on ARMD problem space seem to gravitate around the issues >> basic VLAN deployments had in the past with flood containment. Before >> starting new protocol work maybe we should pause and start from what is >> missing in the latest IEEE/IETF technologies when it comes to addressing >> these issues. >> >> Here is a list of existing standards I am familiar with that are deploye= d in >> large Service Provider Ethernet Networks where we had to deal with simil= ar >> issues: >> >> - =A0 =A0 =A0IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPLS = or >> IP-only, VPN "shim" is the ISID and/or PW label. >> >> - =A0 =A0 =A0IEEE 802.1ah PBB - "core tunneling" is based on native Ethe= rnet >> switching; VPN "shim" is the ISID 24 bit tag; may be transported over th= e >> above VPLS tunnels (see IETF PBB-VPLS work). >> >> Also there are additional initiatives in IEEE and IETF to add DC oriente= d >> features (e.g. Multi-pathing and control plane alternative to MAC learni= ng) >> - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). >> >> This is not a comprehensive list, just an example of possible ways to >> address the issues without reinventing the wheel. >> >> A basic function of these solutions is that any flooded packet is delive= red >> through the core efficiently only to the edge switches and server blades >> that have at least an interested VPN endpoint (VM). The service >> auto-discovery component provides on demand service connectivity only >> between the interested endpoints (e.g. Server Blades/VMs) inside or acro= ss >> DCs. So these solutions have the potential to address in my opinion the = DC >> wide flooding. >> >> If further flood optimization inside an individual tenant domain is >> required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may= be >> used also with these technologies: e.g. the mapping server approach can = map >> VM MACs to the above shims + Tunnels as an alternative to regular IEEE M= AC >> Learning. >> >> The discussion in draft-wkumari on separation of VM addressing between >> tenants and from the DC internal addressing highlights also a very impor= tant >> requirement that needs to be addressed for true isolation between tenant= s. >> But this could be achieved by re-using the existing VPN standards - this= is >> a fundamental principle that was addressed already in the existing >> encapsulations. In addition the core tunneling is based on Service Provi= der >> addressing. >> >> Sorry for the long email. I am hoping people new to VPN work will benefi= t as >> much as we benefited from the good information on DC requirements. >> >> Florin >> >> > > -- > Sent from my mobile device > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > From dunbar.ll@gmail.com Fri Aug 19 08:57:50 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7473121F8AAF for ; Fri, 19 Aug 2011 08:57:50 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.995 X-Spam-Level: X-Spam-Status: No, score=-2.995 tagged_above=-999 required=5 tests=[AWL=0.603, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z4UxE6CUrxiJ for ; Fri, 19 Aug 2011 08:57:48 -0700 (PDT) Received: from mail-bw0-f44.google.com (mail-bw0-f44.google.com [209.85.214.44]) by ietfa.amsl.com (Postfix) with ESMTP id 4353C21F8A96 for ; Fri, 19 Aug 2011 08:57:48 -0700 (PDT) Received: by bkar4 with SMTP id r4so2780948bka.31 for ; Fri, 19 Aug 2011 08:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=LOGi2w3BOFbX0li4EJczo4VC6WtfWcrnnw9UsVEMxsc=; b=fUONSmNRYCutiZ2phkBhJ8umOpnVJEBcgtLvhO/1vNcFDOpVMl/mMqBZ8OE1BnzZWj LUUZSKAOcuQOepYWcM+EsIBmO/ktcnLprAEkUc0BnN2u82b+rqJCSWAiWOU+6Zic10G7 jrtrDXEpq7f8xqEwrGBP7sEZAYCVV9bVHKHWA= MIME-Version: 1.0 Received: by 10.204.141.12 with SMTP id k12mr609939bku.22.1313769522953; Fri, 19 Aug 2011 08:58:42 -0700 (PDT) Received: by 10.205.82.76 with HTTP; Fri, 19 Aug 2011 08:58:42 -0700 (PDT) In-Reply-To: <2073A6C5467C99478898544C6EBA3F4602BB419AE3@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> <4A95BA014132FF49AE685FAB4B9F17F605193C18@dfweml503-mbx.china.huawei.com> <2073A6C5467C99478898544C6EBA3F4602BB419AE3@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Date: Fri, 19 Aug 2011 10:58:42 -0500 Message-ID: From: Linda Dunbar To: "Balus, Florin Stelian (Florin)" Content-Type: multipart/alternative; boundary=0015175cd5b8bb7dd204aaddcdff Cc: "armd@ietf.org" Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Aug 2011 15:57:50 -0000 --0015175cd5b8bb7dd204aaddcdff Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Florin, See my comments inserted below: On Thu, Aug 18, 2011 at 11:30 AM, Balus, Florin Stelian (Florin) < florin.balus@alcatel-lucent.com> wrote: > What I listed in my email is a list of standards that were developed to > work together in L2VPN and in sync with IEEE specification > (encapsulation/tunneling/VPN shim). This is valuable development especial= ly > for inter-DC/VPN-tenant connect since it interoperates with what is alrea= dy > deployed. > [Linda] Agree with you 100%. > **** > > ** ** > > Any new encapsulations for DC space should to go through the same kind of > L2VPN interop discussion/scrutiny since the requirements at least for DC > interconnect are similar with what we saw in L2VPN in the past. We need t= o > ensure there is a good reason to introduce new VPN shims. > [Linda] Thank you for pointing this out. I agree with you. There are so man= y enapsulation schemes already (802.1h's MACinMAC, VPLS, TRILL. etc). > I would say the same thing applies to the native Ethernet space - IEEE > 802.1 Task Force should get a liaison every time we try to invent new > Ethernet VPN shim =96 see discussion we had in TRILL on two VLAN tags as = VPN > shim. IEEE already provided ~ 4 years ago an evolution of the 12 bit VLAN > tag to 24 bit PBB ISID tag and the appropriate encapsulation hierarchy to > hide the (VM) MACs from the backbone nodes.**** > > ** ** > > New control planes/server driven networking should re-use as much as > possible the existing encaps to simplify at least the forwarding options.= * > *** > > ** ** > > See also in-line=85**** > > ** ** > > *From:* Linda Dunbar [mailto:linda.dunbar@huawei.com] > *Sent:* Thursday, August 18, 2011 8:07 AM > *To:* Aldrin Isaac; Balus, Florin Stelian (Florin) > *Cc:* armd@ietf.org > *Subject:* RE: [armd] ARMD problem space & existing standards**** > > ** ** > > Thanks Florin for pointing out L2VPN & IEEE802.1ah Overlay Network. There > is also TRILL encapsulation. All of these encapsulation schemes, includin= g > the IP encapsulation introduced by =93draft-wkumari-dcops-l3-vmmobility-0= 0=94, > achieve the goal of hiding individual hosts=92 (or VMs) addresses from th= e > backbone network. **** > > ** ** > > All of these encapsulation overlay mechanisms face the common challenge i= n > Data Center: edge node of the overlay network has to maintain very large > mapping for all remote hosts <-> their corresponding Edge nodes. **** > > ** ** > > The more number of hosts attached to Overlay Network Edge nodes, the more > mapping has to be maintained by the Edge nodes. The goal of ARMD is to > identify those problems. ARMD is not to introduce another encapsulation > scheme. **** > > ** ** > > =93draft-wkumari-dcops-l3-vmmobility-00=94 introduces =93Mapping Server= =94 to solve > this problem. > http://datatracker.ietf.org/doc/draft-dunbar-trill-directory-assisted-edg= e/introduces the Directory Server to solve the problem for TRILL. > **** > > Then there is a need for directory assistance for IEEE802.1ah, or other > encapsulation schemes in Data Center because the massive number of hosts.= It > may be better for the industry to have a common approach on how Edge node > can get assistance from directory servers. **** > > ** ** > > FB> I assume that by Edge nodes you mean the place where the overlay tunn= el > starts=85 > [Linda] Correct! > When I wrote the initial email I had in mind what the overlay solutions > are generally trying to address: allow a L2 tenant domain (ELAN) with hos= ts > distributed across a large backbone domain with scalable tunneling, usual= ly > using routing protocols in the control plane. In DC talk that means relat= ed > hosts can be distributed anywhere inside or across DCs while containing t= he > amount of flooding/solving DC wide flooding. Implicitly this means server > blades are not bombarded with ARP/ND requests from any unrelated server > blade (from outside their tenant domain). > [Linda] That was the initial intent of ARMD. But the statistics from Merit Networks shows that the biggest pain is L2/L3 boundary router. IMO, both ARP/ND caching and "Mapping Server" can alleviate this problem. > **** > > ** ** > > If the issue is the number of hosts per VPN =96 i.e. a server blade has 3= 2 > VMs every one with its own L2 tenant domain and every domain with X hosts= =96 > there are different issues one might try to solve: > [Linda] If there are 40 server blades per chassis, then there are 40*32 =3D 1280 VMs connected to one Blade switch. If the ToR has 40 downstream ports connected to Blade Switch, then there are 40*1280 =3D 51200 VMs attached to the ToR. > **** > > **- **Further optimize flooding per tenant domain and processing of > ARP/ND requests at the receiving end by cutting the initiation of flooded > traffic at the source: i.e. eliminate ARP transmission/MAC learning throu= gh > Mapping server/Directory etc=85**** > > > [Linda] Very good suggestion. draft-shah-armd-arp-reduction-01 and draft-wkumari-dcops-l3-vmmobility-00 can achieve some of this. SEATTLE proposes a Distributed Hashing scheme, which may scale better. > **** > > **- **X*32 >> host entries the server blade can handle =96 not sure > what can be done about this in the network? One either needs to add memor= y > in the server blade or just don=92t do it as the DC guys said in the ARMD > session. > [Linda] I think the "mapping server" proposed by Warren's draft and SEATTLE approach can solve this problem. > The solution is proper planning of the average number of hosts per L2 > domain (i.e. one may allow a large number for some L2 domains but not for > all)/drop a VRF (routing) instance when needed. Same discussion applies a= t a > different scale if your edge node is the ToR. Most of the VPLS/PBB > implementations I am aware of have per VPN/tenant and per interface > enforcement of the number of hosts (MAC addresses, flooded/mcast traffic)= to > ensure the proper engineering.**** > > ** ** > > ** ** > > Linda**** > > ** ** > > *From:* armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] *On Behalf O= f > *Aldrin Isaac > *Sent:* Wednesday, August 17, 2011 10:17 PM > *To:* Balus, Florin Stelian (Florin) > *Cc:* armd@ietf.org > *Subject:* Re: [armd] ARMD problem space & existing standards**** > > ** ** > > IMHO, the additional initiatives that Florin describes in his email if > implemented at (or near) the access switch and coupled with evolving > standards such as VEPA and SRIOV technology and existing control plane > scaling options (ex: scalable route reflector design, ORF, etc) could > provide a VM-to-VM/end-to-end solution that scale well and enable extreme= ly > flexible virtual topologies without the need for an additional overlay. = -- > aldrin**** > > ** ** > > ** ** > > On Aug 17, 2011, at 7:27 PM, Balus, Florin Stelian (Florin) wrote:**** > > ** ** > > The discussion on ARMD problem space seem to gravitate around the issues > basic VLAN deployments had in the past with flood containment. Before > starting new protocol work maybe we should pause and start from what is > missing in the latest IEEE/IETF technologies when it comes to addressing > these issues.**** > > **** > > Here is a list of existing standards I am familiar with that are deployed > in large Service Provider Ethernet Networks where we had to deal with > similar issues:**** > > - IETF-L2VPN WG VPLS/PBB-VPLS =96 =93core tunneling=94 can be MPLS o= r > IP-only, VPN =93shim=94 is the ISID and/or PW label.**** > > - IEEE 802.1ah PBB =96 =93core tunneling=94 is based on native Ether= net > switching; VPN =93shim=94 is the ISID 24 bit tag; may be transported over= the > above VPLS tunnels (see IETF PBB-VPLS work).**** > > **** > > Also there are additional initiatives in IEEE and IETF to add DC oriented > features (e.g. Multi-pathing and control plane alternative to MAC learnin= g) > - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). **** > > **** > > This is not a comprehensive list, just an example of possible ways to > address the issues without reinventing the wheel.**** > > **** > > A basic function of these solutions is that any flooded packet is deliver= ed > through the core efficiently only to the edge switches and server blades > that have at least an interested VPN endpoint (VM). The service > auto-discovery component provides on demand service connectivity only > between the interested endpoints (e.g. Server Blades/VMs) inside or acros= s > DCs. So these solutions have the potential to address in my opinion the D= C > wide flooding.**** > > **** > > If further flood optimization inside an individual tenant domain is > required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt may = be > used also with these technologies: e.g. the mapping server approach can m= ap > VM MACs to the above shims + Tunnels as an alternative to regular IEEE MA= C > Learning.**** > > **** > > The discussion in draft-wkumari on separation of VM addressing between > tenants and from the DC internal addressing highlights also a very import= ant > requirement that needs to be addressed for true isolation between tenants= . > But this could be achieved by re-using the existing VPN standards - this = is > a fundamental principle that was addressed already in the existing > encapsulations. In addition the core tunneling is based on Service Provid= er > addressing.**** > > **** > > Sorry for the long email. I am hoping people new to VPN work will benefit > as much as we benefited from the good information on DC requirements.**** > > **** > > Florin**** > > **** > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd**** > > ** ** > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --0015175cd5b8bb7dd204aaddcdff Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
    Florin,
    =A0
    See my comments inserted below:
    =A0


    =A0
    On Thu, Aug 18, 2011 at 11:30 AM, Balus, Florin = Stelian (Florin) <florin.balus@alcatel-lucent.com> wrote:

    What= I listed in my email is a list of standards that were developed to work to= gether in L2VPN and in sync with IEEE specification (encapsulation/tunnelin= g/VPN shim). This is valuable development especially for inter-DC/VPN-tenan= t connect since it interoperates with what is already deployed.

    [Linda] Agree with you 100%.

    <= /u>

    <= /u>=A0

    Any = new encapsulations for DC space should to go through the same kind of L2VPN= interop discussion/scrutiny since the requirements at least for DC interco= nnect are similar with what we saw in L2VPN in the past. We need to ensure = there is a good reason to introduce new VPN shims.

    =A0
    [Linda] Thank you for pointing this out. I agree with you. There are s= o many enapsulation schemes already (802.1h's MACinMAC, VPLS, TRILL. et= c).
    =A0

    I wo= uld say the same thing applies to the native Ethernet space - IEEE 802.1 Ta= sk Force should get a liaison every time we try to invent new Ethernet VPN = shim =96 see discussion we had in TRILL on two VLAN tags as VPN shim. IEEE = already provided ~ 4 years ago an evolution of the 12 bit VLAN tag to 24 bi= t PBB ISID tag and the appropriate encapsulation hierarchy to hide the (VM)= MACs from the backbone nodes.

    <= /u>=A0

    New = control planes/server driven networking should re-use as much as possible t= he existing encaps to simplify at least the forwarding options.

    <= /u>=A0

    See = also in-line=85

    <= /u>=A0

    From:<= span style=3D"FONT-SIZE: 10pt"> Linda Dunbar [mailto:linda.dunbar@huawei.com]
    = Sent: Thursday, August 18, 2011 8:07 AM
    To: Aldrin Isaac; Balus, Florin Stelian (Florin)
    Cc: armd@ietf.org
    Subje= ct: RE: [armd] ARMD problem space & existing standards

    =A0

    Than= ks Florin for pointing out L2VPN & IEEE802.1ah Overlay Network. There i= s also TRILL encapsulation. All of these encapsulation schemes, including t= he IP encapsulation introduced by =93draft-wkumari-dcops-l3-vmmobility-00= =94, achieve the goal of hiding individual hosts=92 (or VMs) addresses from= the backbone network.

    <= /u>=A0

    All = of these encapsulation overlay mechanisms face the common challenge in Data= Center: edge node of the overlay network has to maintain very large mappin= g for all remote hosts <-> their corresponding Edge nodes. =

    <= /u>=A0

    The = more number of hosts attached to Overlay Network Edge nodes, the more mappi= ng has to be maintained by the Edge nodes. The goal of ARMD is to identify = those problems. ARMD is not to introduce another encapsulation scheme. <= /u>

    <= /u>=A0

    =93d= raft-wkumari-dcops-l3-vmmobility-00=94 introduces =93Mapping Server=94 to s= olve this problem. http://datatracker.ietf.o= rg/doc/draft-dunbar-trill-directory-assisted-edge/ introduces the Direc= tory Server to solve the problem for TRILL.

    Then= there is a need for directory assistance for IEEE802.1ah, or other encapsu= lation schemes in Data Center because the massive number of hosts. It may b= e better for the industry to have a comm= on approach on how Edge node can get assistance from directory servers. =A0=

    <= /u>=A0

    FB>= I assume that by Edge nodes you mean the place where the overlay tunnel st= arts=85

    =A0
    [Linda] Correct!
    =A0

    When I= wrote the initial email I had in mind what the overlay solutions are gener= ally trying to address: allow a L2 tenant domain (ELAN) with hosts distribu= ted across a large backbone domain with scalable tunneling, usually using r= outing protocols in the control plane. In DC talk that means related hosts = can be distributed anywhere inside or across DCs while containing the amoun= t of flooding/solving DC wide flooding. Implicitly this means server blades= are not bombarded with ARP/ND requests from any unrelated server blade (fr= om outside their tenant domain).

    =A0
    [Linda] That was the initial intent of ARMD. But the statistics from M= erit Networks shows that the biggest pain is L2/L3 boundary router. IMO, bo= th ARP/ND caching and "Mapping Server" can alleviate this problem= .
    =A0
    =A0
    =A0
    =A0

    =A0

    If the= issue is the number of hosts per VPN =96 i.e. a server blade has 32 VMs ev= ery one with its own L2 tenant domain and every domain with X hosts =96 the= re are different issues one might try to solve:

    =A0
    [Linda] If there are 40 server blades per chassis, then there are 40*3= 2 =3D 1280 VMs connected to one Blade switch. If the ToR has 40 downstream = ports connected to Blade Switch, then there are 40*1280 =3D 51200 VMs attac= hed to the ToR.
    =A0

    -=A0=A0=A0=A0=A0 Further optimize fl= ooding per tenant domain and processing of ARP/ND requests at the receiving= end by cutting the initiation of flooded traffic at the source: i.e. elimi= nate ARP transmission/MAC learning through Mapping server/Directory etc=85<= u>

    =A0

    =
    [Linda] Very good suggestion. draft-shah-armd-arp-reduction-01 and dra= ft-wkumari-dcops-l3-vmmobility-00 can achieve some of this. SEATTLE propose= s a Distributed Hashing scheme, which may scale better.
    =A0

    -=A0=A0=A0=A0=A0 X*32 >> host = entries the server blade can handle =96 not sure what can be done about thi= s in the network? One either needs to add memory in the server blade or jus= t don=92t do it as the DC guys said in the ARMD session.

    [Linda] I think the "mapping server" proposed by Warren'= s draft and SEATTLE approach can solve this problem.
    =A0

    The solution is proper pla= nning of the average number of hosts per L2 domain (i.e. one may allow a la= rge number for some L2 domains but not for all)/drop a VRF (routing) instan= ce when needed. Same discussion applies at a different scale if your edge n= ode is the ToR. Most of the VPLS/PBB implementations I am aware of have per= VPN/tenant and per interface enforcement of the number of hosts (MAC addre= sses, flooded/mcast traffic) to ensure the proper engineering.

    <= /u>=A0

    <= /u>=A0

    Lind= a

    <= /u>=A0

    From:<= span style=3D"FONT-SIZE: 10pt"> armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf O= f Aldrin Isaac
    Sent: Wednesday, August 17, 2011 10:17 PM
    To: Balus, Flori= n Stelian (Florin)
    Cc: armd@ietf.org
    Subject: Re: [armd] ARMD problem space= & existing standards

    =A0

    IMHO, the additional initiatives that Florin describ= es in his email if implemented at (or near) the access switch and coupled w= ith evolving standards such as VEPA and SRIOV technology=A0and existing con= trol plane scaling options (ex: scalable route reflector design, ORF, etc) = could provide a VM-to-VM/end-to-end solution that scale well and enable ext= remely flexible virtual topologies without the need for an additional overl= ay. =A0-- aldrin

    =A0

    =A0

    On Aug 17, 2011, at 7:27 PM, Balus, Florin Stelian (= Florin) wrote:

    =A0

    The discussion on AR= MD problem space seem to gravitate around the issues basic VLAN deployments= had in the past with flood containment. Before starting new protocol work = maybe we should pause and start from what is missing in the latest IEEE/IET= F technologies when it comes to addressing these issues.

    =A0

    Here is a list of ex= isting standards I am familiar with that are deployed in large Service Prov= ider Ethernet Networks where we had to deal with similar issues:

    -=A0=A0=A0=A0=A0=A0IETF-L2VPN WG VPLS/PBB-VPLS =96 =93core tunneling=94 can be = MPLS or IP-only, VPN =93shim=94 is the ISID and/or PW label.

    -=A0=A0=A0=A0=A0=A0IEEE 802.1ah PBB =96 =93core tunneling=94 is based on native= Ethernet switching; VPN =93shim=94 is the ISID 24 bit tag; may be transpor= ted over the above VPLS tunnels (see IETF PBB-VPLS work).

    =A0

    Also there are addit= ional initiatives in IEEE and IETF to add DC oriented features (e.g. Multi-= pathing and control plane alternative to MAC learning) - see IETF L2VPN EVP= N, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). =A0

    =A0

    This is not a compre= hensive list, just an example of possible ways to address the issues withou= t reinventing the wheel.

    =A0

    A basic function of = these solutions is that any flooded packet is delivered through the core ef= ficiently only to the edge switches and server blades that have at least an= interested VPN endpoint (VM). The service auto-discovery component provide= s on demand service connectivity only between the interested endpoints (e.g= . Server Blades/VMs) inside or across DCs. So these solutions have the pote= ntial to address in my opinion the DC wide flooding.

    =A0

    If further flood opt= imization inside an individual tenant domain is required, the procedures in= draft-wkumari-dcops-l3-vmmobility-00.txt may be used also with these techn= ologies: e.g. the mapping server approach can map VM MACs to the above shim= s + Tunnels as an alternative to regular IEEE MAC Learning.

    =A0

    The discussion in dr= aft-wkumari on separation of VM addressing between tenants and from the DC = internal addressing highlights also a very important requirement that needs= to be addressed for true isolation between tenants. But this could be achi= eved by re-using the existing VPN standards - this is a fundamental princip= le that was addressed already in the existing encapsulations. In addition t= he core tunneling is based on Service Provider addressing.

    =A0

    Sorry for the long e= mail. I am hoping people new to VPN work will benefit as much as we benefit= ed from the good information on DC requirements.

    =A0

    Florin

    =A0

    _______________________________________________
    armd mailing list
    = armd@ietf.org
    https= ://www.ietf.org/mailman/listinfo/armd

    =A0


    ___= ____________________________________________
    armd mailing list
    armd@ietf.org
    https://www.ietf.org/mailman/= listinfo/armd


    --0015175cd5b8bb7dd204aaddcdff-- From linda.dunbar@huawei.com Fri Aug 19 09:15:56 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4845221F8B1D for ; Fri, 19 Aug 2011 09:15:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.284 X-Spam-Level: X-Spam-Status: No, score=-6.284 tagged_above=-999 required=5 tests=[AWL=0.315, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9q44BGfOkyFi for ; Fri, 19 Aug 2011 09:15:55 -0700 (PDT) Received: from usaga04-in.huawei.com (usaga04-in.huawei.com [206.16.17.180]) by ietfa.amsl.com (Postfix) with ESMTP id 1C0F821F85FF for ; Fri, 19 Aug 2011 09:15:55 -0700 (PDT) Received: from huawei.com (usaga04-in [172.18.4.101]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQ6005WRNW3BS@usaga04-in.huawei.com> for armd@ietf.org; Fri, 19 Aug 2011 11:16:52 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQ6002J7NW20K@usaga04-in.huawei.com> for armd@ietf.org; Fri, 19 Aug 2011 11:16:51 -0500 (CDT) Received: from DFWEML401-HUB.china.huawei.com (10.193.5.101) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Fri, 19 Aug 2011 09:16:52 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.37]) by DFWEML401-HUB.china.huawei.com ([fe80::f07f:889f:78ef:8df3%13]) with mapi id 14.01.0270.001; Fri, 19 Aug 2011 09:16:42 -0700 Date: Fri, 19 Aug 2011 16:16:41 +0000 From: Linda Dunbar In-reply-to: X-Originating-IP: [10.192.11.66] To: Christopher Morrow , "Balus, Florin Stelian (Florin)" , Thomas Narten Message-id: <4A95BA014132FF49AE685FAB4B9F17F605194C34@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-language: en-US Content-transfer-encoding: quoted-printable Accept-Language: en-US Thread-topic: [armd] ARMD problem space & existing standards Thread-index: AcxdNTFE+My2+OQJRqaZzuH96c0GBgA1Bj0AAAYzuIAAJ9P5gAANku/A X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <2073A6C5467C99478898544C6EBA3F4602BB419A2C@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> <2073A6C5467C99478898544C6EBA3F4602BB419BF2@USNAVSXCHMBSC3.ndc.alcatel-lucent.com> Cc: "armd@ietf.org" Subject: Re: [armd] ARMD problem space & existing standards X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Aug 2011 16:15:56 -0000 Chris,=20 Thank you very much for the good description on pain points in DC networks.= ARMD's first goal is to document all those pain points.=20 Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Christopher Morrow > Sent: Friday, August 19, 2011 10:43 AM > To: Balus, Florin Stelian (Florin) > Cc: armd@ietf.org > Subject: Re: [armd] ARMD problem space & existing standards >=20 > On Thu, Aug 18, 2011 at 4:43 PM, Balus, Florin Stelian (Florin) > wrote: > > Thanks Phil for the comments. See in-line... > > > > -----Original Message----- > > From: Phil Bedard [mailto:bedard.phil@gmail.com] > > Sent: Thursday, August 18, 2011 10:45 AM > > To: Balus, Florin Stelian (Florin); armd@ietf.org > > Subject: Re: [armd] ARMD problem space & existing standards > > > > I think the issue thus far is implementing this in access switches > > close to the hosts... >=20 > This seems correct to me, though there's likely some pain on the first > L3 hop in the DC as well. It seems that once you step out of the > immediate L2 domain, normal backbone routing things are still fine in > v4/v6. You MAY have issues wrt routing scale, but of course you have > those anyway. >=20 > The particular problem(s) that ARMD was/is looking at are the scaling > properties of the edge nodes (access-switch, TOR, first L3 hop, > server-side), and how those interact in a much larger potential L2 > domain. Today, for instance, in a normal DC, a single rack is > potentially it's own L2 domain (there are lots of architectures, let > us keep it simple though). That domain, in v4, is contained and > relatively small (/24 or smaller). In v6 though, that domain is > potentially much larger (/64 default deployments for lots of reasons). >=20 > Hosts on the /24-equivalent can only make so many ARP requests, the > network gear has only to cache a limited number of entries, and things > 'just work'. >=20 > Hosts on a /64 though, or a larger flat L2 deployment, see much more > ARP traffic, the associated network gear has to maintain larger > queues/caches and deal with a larger flow of ARP/etc requests. The > pathway from line-port to routing-engine/route-processor/smarts in a > network device are severely constrained (as compared with the > line-port -> line-port speeds/pathways). As gear gets toward the > less-expensive range, it also gets less capable in this respect. > Scaling up to multiple /24's of space across a distributed L2 is .... > much harder to deal with, traffic-to-cpu wise. Additionally, less > expensive devices are generally not fortified with faster/more-capable > cpus. >=20 > The drive to lower costs in the DC (cheaper network deployments, more > vm-guests's per vm-host, etc is exacerbating this problem, or so it > would seem, in a large flat L2 network. >=20 > > > > FB> When it comes to internal DC network I understand VLANs/Ethernet > switching are usually used in access switches (ToRs). Backbone Ethernet > tunneling was developed by IEEE to maintain the same basic procedures > and as far as I know the chipsets built for these kind of devices > support both TRILL and PBB. > > > > These are all technologies used to enhance the scale of L2 topologies. > > >=20 > the breadth, they aren't really necessary for 'scale'... you can just > connect lots of L2 switches into a large SCALE LAN, but making it work > across the wide-area is where the above comes into play. (work WELL at > least) >=20 > > FB> maybe a better way to put it is these technologies allow related > L2 switching instances to be located far away from each other, across > one or a combination of L2 or L3 (IP/MPLS) backbones. There are metro, > national and even intercontinental deployments of these technologies. > > >=20 > right, breadth, not 'scale' (distance not numbers) >=20 > > =A0The Google case in one where I believe there is a L3 IP hop between > > the two VM hosts which necessitates tunneling. =A0In order for VMs to > > retain their connectivity the L2 has to be dynamically tunneled over > > L3 from server to server and gateway to server. > > > > FB> I understand there is value in re-using the VLAN + IP routing > environment. That was the whole point of PBB-VPLS model and interop > work in L2VPN: to allow a similar (Backbone) VLAN + (IP or MPLS) tunnel > combination. > > > > In a L2 network you need =A0to reconfigure the upstream switch port > when a VM moves or > > preconfigure ports. =A0 I see extensions to LLDP like VEPA/multichannel > > which may allow more dynamic reconfiguration without a management > tool > > having to reconfigure the upstream network device. > > > > With regards to ARP scale, whether emulated via VPLS or native, there > > are probably multiple solutions. =A0ARP snooping is probably not a > > difficult operation since most do it today for IGMP and/or PIM and > > many devices perform proxy-arp. >=20 > arp snooping and the like work today on smaller deployments > (numbers-wise) they probably have severe challenges when you attempt > to scale up the number of arping nodes downstream from a port on a > switch though. You get to some of that (florin) in the discussion of > numbers three messages previous. >=20 > -chris >=20 > > I mentioned Ethernet VPN earlier as a solution which can be > > implemented with a scalable protocol like BGP to push MAC information > > or one could alternatively use a directory lookup instead. =A0 EVPN I > > believe has a provision to perform ARP snooping. > > > > FB> Yes there is some discussion on ARP containment in EVPN draft - > see section 13 in http://tools.ietf.org/html/draft-raggarwa-sajassi- > l2vpn-evpn-01 ... > > > > Phil > > > > On 8/17/11, Balus, Florin Stelian (Florin) > > wrote: > >> The discussion on ARMD problem space seem to gravitate around the > issues > >> basic VLAN deployments had in the past with flood containment. > Before > >> starting new protocol work maybe we should pause and start from what > is > >> missing in the latest IEEE/IETF technologies when it comes to > addressing > >> these issues. > >> > >> Here is a list of existing standards I am familiar with that are > deployed in > >> large Service Provider Ethernet Networks where we had to deal with > similar > >> issues: > >> > >> - =A0 =A0 =A0IETF-L2VPN WG VPLS/PBB-VPLS - "core tunneling" can be MPL= S or > >> IP-only, VPN "shim" is the ISID and/or PW label. > >> > >> - =A0 =A0 =A0IEEE 802.1ah PBB - "core tunneling" is based on native > Ethernet > >> switching; VPN "shim" is the ISID 24 bit tag; may be transported > over the > >> above VPLS tunnels (see IETF PBB-VPLS work). > >> > >> Also there are additional initiatives in IEEE and IETF to add DC > oriented > >> features (e.g. Multi-pathing and control plane alternative to MAC > learning) > >> - see IETF L2VPN EVPN, PBB-EVPN drafts and IEEE 802.1aq SPBM (PBB). > >> > >> This is not a comprehensive list, just an example of possible ways > to > >> address the issues without reinventing the wheel. > >> > >> A basic function of these solutions is that any flooded packet is > delivered > >> through the core efficiently only to the edge switches and server > blades > >> that have at least an interested VPN endpoint (VM). The service > >> auto-discovery component provides on demand service connectivity > only > >> between the interested endpoints (e.g. Server Blades/VMs) inside or > across > >> DCs. So these solutions have the potential to address in my opinion > the DC > >> wide flooding. > >> > >> If further flood optimization inside an individual tenant domain is > >> required, the procedures in draft-wkumari-dcops-l3-vmmobility-00.txt > may be > >> used also with these technologies: e.g. the mapping server approach > can map > >> VM MACs to the above shims + Tunnels as an alternative to regular > IEEE MAC > >> Learning. > >> > >> The discussion in draft-wkumari on separation of VM addressing > between > >> tenants and from the DC internal addressing highlights also a very > important > >> requirement that needs to be addressed for true isolation between > tenants. > >> But this could be achieved by re-using the existing VPN standards - > this is > >> a fundamental principle that was addressed already in the existing > >> encapsulations. In addition the core tunneling is based on Service > Provider > >> addressing. > >> > >> Sorry for the long email. I am hoping people new to VPN work will > benefit as > >> much as we benefited from the good information on DC requirements. > >> > >> Florin > >> > >> > > > > -- > > Sent from my mobile device > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From linda.dunbar@huawei.com Tue Aug 23 06:49:27 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5ADFF21F8B7C for ; Tue, 23 Aug 2011 06:49:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.308 X-Spam-Level: X-Spam-Status: No, score=-6.308 tagged_above=-999 required=5 tests=[AWL=0.291, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ESfCp4P8iAw3 for ; Tue, 23 Aug 2011 06:49:26 -0700 (PDT) Received: from usaga02-in.huawei.com (usaga02-in.huawei.com [206.16.17.70]) by ietfa.amsl.com (Postfix) with ESMTP id B2AA221F8B2F for ; Tue, 23 Aug 2011 06:49:26 -0700 (PDT) Received: from huawei.com (localhost [127.0.0.1]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQD00AXVVPIOG@usaga02-in.huawei.com> for armd@ietf.org; Tue, 23 Aug 2011 08:48:54 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LQD00AKUVPIP4@usaga02-in.huawei.com> for armd@ietf.org; Tue, 23 Aug 2011 08:48:54 -0500 (CDT) Received: from DFWEML402-HUB.china.huawei.com (10.193.5.102) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 23 Aug 2011 06:48:54 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.37]) by DFWEML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Tue, 23 Aug 2011 06:48:53 -0700 Date: Tue, 23 Aug 2011 13:48:51 +0000 From: Linda Dunbar X-Originating-IP: [10.192.11.66] To: "armd@ietf.org" Message-id: <4A95BA014132FF49AE685FAB4B9F17F605195763@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-language: en-US Content-transfer-encoding: 7BIT Accept-Language: en-US Thread-topic: Nomcom 2011-2012: Call for Nominations Thread-index: AQHMYZtk/ySvbArLKUWWv2LKpmSnHQ== X-MS-Has-Attach: X-MS-TNEF-Correlator: Subject: [armd] FW: Nomcom 2011-2012: Call for Nominations X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Aug 2011 13:49:27 -0000 Please participate to nominate the best candidates for the positions that need to be filled. Thanks and Regards, Linda & Benson -----Original Message----- From: ietf-announce-bounces@ietf.org [mailto:ietf-announce-bounces@ietf.org] On Behalf Of NomCom Chair Sent: Tuesday, August 23, 2011 1:30 AM To: IETF Announcement list Cc: ietf@ietf.org Subject: Nomcom 2011-2012: Call for Nominations Hi All, The 2011-2012 Nominating committee is seeking nominations from now until October 2, 2011. The list of open positions can be found at: https://www.ietf.org/group/nomcom/2011/ Nominations may be made directly on the NomCom 2011-2012 pages by selecting the Nominate link at the top of the page. The URL for NomCom 2011-2012 pages is: https://www.ietf.org/group/nomcom/2011/ Nominations may also be made by email to nomcom11@ietf.org. If you do so, please include the word "Nominate" in the Subject and indicate in the email who is being nominated, their email address (to confirm acceptance of the nomination), and the position for which you are making the nomination. If you wish to nominate someone via email for more than one position, please use separate emails to do so. Self nomination is welcome. NomCom 2011-2012 will follow the policy for "Open Disclosure of Willing Nominees" described in RFC 5680. As stated in RFC 5680: "The list of nominees willing to be considered for positions under review in the current NomCom cycle is not confidential". Willing Nominees for each position will be publicly listed. The public nominee list will be updated at least once a week and possibly more often as nominations are received. With the exception of publicly listing willing nominees, the confidentiality requirements of RFC 3777 remain in effect. All feedback and NomCom deliberations will remain confidential and not disclosed. Because the list of nominees this year is public, we will accept feedback on the nominees starting August 23, 2011. Per RFC 5680, we will accept feedback from the entire IETF community on all the nominees. If you wish to provide anonymous feedback, the chair or any of the members will be happy to handle this for you. The Nominating Committee chair can be reached at nomcom-chair@ietf.org and the entire nominating committee can be reached at nomcom11@ietf.org. The email addresses of individual NomCom members is also on the NomCom 2011-2012 pages. In addition to nominations, the Nominating Committee is actively seeking community input on the jobs that need to be filled. We have received the job descriptions from the IAB, IESG, and IAOC and they can be found at: https://www.ietf.org/group/nomcom/2011/iab-requirements https://www.ietf.org/group/nomcom/2011/iesg-requirements https://www.ietf.org/group/nomcom/2011/iaoc-requirements However, we also need the community's views and input on the jobs within each organization. If you have ideas on job responsibilities (more, less, different), please let us know. Please send suggestions and feedback to nomcom11@ietf.org. Thank you, Suresh Krishnan Chair, NomCom 2011-2012 nomcom-chair@ietf.org suresh.krishnan@ericsson.com _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www.ietf.org/mailman/listinfo/ietf-announce _______________________________________________ opsarea-chairs mailing list opsarea-chairs@ietf.org https://www.ietf.org/mailman/listinfo/opsarea-chairs From xuxiaohu@huawei.com Wed Aug 24 19:46:38 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BF74921F8B2C; Wed, 24 Aug 2011 19:46:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.603 X-Spam-Level: X-Spam-Status: No, score=-3.603 tagged_above=-999 required=5 tests=[AWL=2.680, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4, SARE_MILLIONSOF=0.315] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 27qze9hUPXZ3; Wed, 24 Aug 2011 19:46:37 -0700 (PDT) Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [119.145.14.67]) by ietfa.amsl.com (Postfix) with ESMTP id 00B6721F884C; Wed, 24 Aug 2011 19:46:37 -0700 (PDT) Received: from huawei.com (szxga04-in [172.24.2.12]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQG00MW4QFO36@szxga04-in.huawei.com>; Thu, 25 Aug 2011 10:47:48 +0800 (CST) Received: from szxrg01-dlp.huawei.com ([172.24.2.119]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQG009UFQFOFQ@szxga04-in.huawei.com>; Thu, 25 Aug 2011 10:47:48 +0800 (CST) Received: from 172.24.2.119 (EHLO szxeml203-edg.china.huawei.com) ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.1.9-GA FastPath queued) with ESMTP id ADK31533; Thu, 25 Aug 2011 10:47:48 +0800 (CST) Received: from SZXEML405-HUB.china.huawei.com (10.82.67.60) by szxeml203-edg.china.huawei.com (172.24.2.55) with Microsoft SMTP Server (TLS) id 14.1.270.1; Thu, 25 Aug 2011 10:47:42 +0800 Received: from SZXEML525-MBS.china.huawei.com ([169.254.8.40]) by szxeml405-hub.china.huawei.com ([169.254.211.222]) with mapi id 14.01.0270.001; Thu, 25 Aug 2011 10:47:46 +0800 Date: Thu, 25 Aug 2011 02:47:47 +0000 From: Xuxiaohu X-Originating-IP: [10.110.98.53] To: "l2vpn@ietf.org" Message-id: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE725622@szxeml525-mbs.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_qd/1kR5tiQoJdowZb58ERw)" Content-language: zh-CN Accept-Language: zh-CN, en-US Thread-topic: Could millions of ARP entries on DC gateways be reduced to one Thread-index: Acxi0w1AcibzuVsiSkK+GwRxBeP6pg== X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected X-Mailman-Approved-At: Thu, 25 Aug 2011 07:49:27 -0700 Cc: "armd@ietf.org" Subject: [armd] Could millions of ARP entries on DC gateways be reduced to one X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Aug 2011 02:46:38 -0000 --Boundary_(ID_qd/1kR5tiQoJdowZb58ERw) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: quoted-printable Hi all, Another obvious advantage of host route based IP-only L2VPN (e.g., Virtual = Subnet) over VPLS, as a DCI solution, is to reduce the ARP table size on DC= gateways by several order of magnitude. Assume there are millions of CE ho= sts (i.e.,VMs) within a single VLAN/subnet, if VPLS is used as a DCI soluti= on, DC gateways (i.e., DC exit routers) would have to know millions of ARP = entries corresponding to these VMs. In contrast, in the Virtual Subnet solu= tion, DC gateways are directly connected to the PE routers of the IP-only L= 2VPN which act as ARP proxies, MAC addresses of those ARP entries for VMs o= n DC gateways are identical (i.e., ARP Proxy's MAC). Thus these millions of= ARP entries can be aggregated into one entry (1.0.0.0/8->ARP proxy's MAC).= That's to say, the exact-matching algorithm for ARP cache lookup is change= d to the longest-matching algorithm. Of course, there is no free lunch. The= side-effect of this change is that DC gateways could send out packets dest= ined for non-existing VMs to the PE routers of that IP-only L2VPN. Fortunat= ely, once those packets arrive at the PE routers of that IP-only L2VPN, whi= ch in turn will drop them directly since there is no matching routes for th= em. Since this topic is also interesting to ARMD WG, I copy this email to ARMD. Best regards, Xiaohu --Boundary_(ID_qd/1kR5tiQoJdowZb58ERw) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

    Hi all,

     

    Another obvious advantage of host route based IP-only L2VPN (e.g., Virtual Subnet) over VPLS, as a DCI solution, is to reduce the ARP table size on DC gateways by several order of magnitude. Assume there are millions of CE hosts (i.e.,VMs) within a single VLAN/subnet, if VPLS is used as a DCI solution, DC gateways (i.e., DC exit routers) would have to know millions of ARP entries corresponding to these VMs. In contrast, in the Virtual Subnet solution, DC gateways are directly connected to the PE routers of the IP-only L2VPN which act as ARP proxies, MAC addresses of those ARP entries for VMs on DC gateways are identical (i.e., ARP Proxy’s MAC). Thus these millions of ARP entries can be aggregated into one entry (1.0.0.0/8->ARP proxy’s MAC). That’s to say, the exact-matching algorithm for ARP cache lookup is changed to the longest-matching algorithm. Of course, there is no free lunch. The side-effect of this change is that DC gateways could send out packets destined for non-existing VMs to the PE routers of that IP-only L2VPN. Fortunately, once those packets arrive at the PE routers of that IP-only L2VPN, which in turn will drop them directly since there is no matching routes for them.

     

    Since this topic is also interesting to ARMD WG, I copy this email to ARMD.

     

    Best regards,

    Xiaohu

    --Boundary_(ID_qd/1kR5tiQoJdowZb58ERw)-- From dunbar.ll@gmail.com Thu Aug 25 10:02:46 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2E15A21F8AD8; Thu, 25 Aug 2011 10:02:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.937 X-Spam-Level: X-Spam-Status: No, score=-2.937 tagged_above=-999 required=5 tests=[AWL=0.345, BAYES_00=-2.599, HTML_MESSAGE=0.001, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_MILLIONSOF=0.315] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NTtAZsZxg3lY; Thu, 25 Aug 2011 10:02:45 -0700 (PDT) Received: from mail-bw0-f44.google.com (mail-bw0-f44.google.com [209.85.214.44]) by ietfa.amsl.com (Postfix) with ESMTP id 9D5DF21F8AD6; Thu, 25 Aug 2011 10:02:44 -0700 (PDT) Received: by bkar4 with SMTP id r4so2216461bka.31 for ; Thu, 25 Aug 2011 10:03:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VEDo/snjBtHPNeHccRqgRn0hSjZl8fSVIaJyPkqgXH4=; b=qMyv4bP4n1yTLAV0bHjLV/9vxhCsAu4vKhixKnPza7xobr01qE5tZPSqOveMwge5ai N6BzzdmL4oAM2tOc7o7Z5yG/oZ85Vrgr+omdPKU4/6Bl3d71o9SwGhIvSXXh5jeSvnnx s6qbzvhkgQS2N9rJb+th+AOtHJtPCt6XasUmU= MIME-Version: 1.0 Received: by 10.204.141.12 with SMTP id k12mr12192bku.22.1314291835055; Thu, 25 Aug 2011 10:03:55 -0700 (PDT) Received: by 10.204.41.73 with HTTP; Thu, 25 Aug 2011 10:03:55 -0700 (PDT) In-Reply-To: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE725622@szxeml525-mbs.china.huawei.com> References: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE725622@szxeml525-mbs.china.huawei.com> Date: Thu, 25 Aug 2011 12:03:55 -0500 Message-ID: From: Linda Dunbar To: Xuxiaohu Content-Type: multipart/alternative; boundary=0015175cd5b8f5acd604ab5769c2 Cc: "l2vpn@ietf.org" , "armd@ietf.org" Subject: Re: [armd] Could millions of ARP entries on DC gateways be reduced to one X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Aug 2011 17:02:46 -0000 --0015175cd5b8f5acd604ab5769c2 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Xiao Hu, That is a very interesting proposal. Do you mean that there is a single ARP proxy which represents millions of VMs? All the packets destined to any of those "millions of VMs" will be rerouted to "the single ARP proxy"? Will this create too much pressure on this single point? How is your proposed approach comparing with SEATTLE's distributed approach= ? Linda Dunbar On Wed, Aug 24, 2011 at 9:47 PM, Xuxiaohu wrote: > Hi all,**** > > ** ** > > Another obvious advantage of host route based IP-only L2VPN (e.g., Virtua= l > Subnet) over VPLS, as a DCI solution, is to reduce the ARP table size on = DC > gateways by several order of magnitude. Assume there are millions of CE > hosts (i.e.,VMs) within a single VLAN/subnet, if VPLS is used as a DCI > solution, DC gateways (i.e., DC exit routers) would have to know millions= of > ARP entries corresponding to these VMs. In contrast, in the Virtual Subne= t > solution, DC gateways are directly connected to the PE routers of the > IP-only L2VPN which act as ARP proxies, MAC addresses of those ARP entrie= s > for VMs on DC gateways are identical (i.e., ARP Proxy=92s MAC). Thus thes= e > millions of ARP entries can be aggregated into one entry (1.0.0.0/8->ARP > proxy=92s MAC). That=92s to say, the exact-matching algorithm for ARP cac= he > lookup is changed to the longest-matching algorithm. Of course, there is = no > free lunch. The side-effect of this change is that DC gateways could send > out packets destined for non-existing VMs to the PE routers of that IP-on= ly > L2VPN. Fortunately, once those packets arrive at the PE routers of that > IP-only L2VPN, which in turn will drop them directly since there is no > matching routes for them. **** > > ** ** > > Since this topic is also interesting to ARMD WG, I copy this email to ARM= D. > **** > > ** ** > > Best regards,**** > > Xiaohu**** > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > --0015175cd5b8f5acd604ab5769c2 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
    Xiao Hu,
    =A0
    That is a very interesting proposal.
    Do you mean that there is a single ARP proxy which represents millions= of VMs? All the packets destined to any of those "millions of VMs&quo= t; will be rerouted to "the single ARP proxy"? Will this create t= oo much pressure on this single point?
    How is your proposed approach comparing with SEATTLE's distributed= approach?
    =A0
    Linda Dunbar
    On Wed, Aug 24, 2011 at 9:47 PM, Xuxiaohu <xuxiaohu@huawei.com<= /a>> wrote:

    Hi all,

    =A0

    Another obvious advantage of host route based IP-only L2VPN (e.g= ., Virtual Subnet) over VPLS, as a DCI solution, is to reduce the ARP table= size on DC gateways by several order of magnitude. Assume there are millio= ns of CE hosts (i.e.,VMs) within a single VLAN/subnet, if VPLS is used as a= DCI solution, DC gateways (i.e., DC exit routers) would have to know milli= ons of ARP entries corresponding to these VMs. In contrast, in the Virtual = Subnet solution, DC gateways are directly connected to the PE routers of th= e IP-only L2VPN which act as ARP proxies, MAC addresses of those ARP entrie= s for VMs on DC gateways are identical (i.e., ARP Proxy=92s MAC). Thus thes= e millions of ARP entries can be aggregated into one entry (1.0.0.0/8->ARP proxy=92s MAC). That= =92s to say, the exact-matching algorithm for ARP cache lookup is changed t= o the longest-matching algorithm. Of course, there is no free lunch. The si= de-effect of this change is that DC gateways could send out packets destine= d for non-existing VMs to the PE routers of that IP-only L2VPN. Fortunately= , once those packets arrive at the PE routers of that IP-only L2VPN, which = in turn will drop them directly since there is no matching routes for them.=

    =A0

    Since this topic is also interesting to ARMD WG, I copy this ema= il to ARMD.

    =A0

    Best regards,

    Xiaohu


    _________________= ______________________________
    armd mailing list
    armd@ietf.org
    ht= tps://www.ietf.org/mailman/listinfo/armd


    --0015175cd5b8f5acd604ab5769c2-- From xuxiaohu@huawei.com Thu Aug 25 18:59:14 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6A2F621F86BE; Thu, 25 Aug 2011 18:59:14 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.65 X-Spam-Level: X-Spam-Status: No, score=0.65 tagged_above=-999 required=5 tests=[AWL=-2.716, BAYES_00=-2.599, CHARSET_FARAWAY_HEADER=3.2, HTML_MESSAGE=0.001, J_CHICKENPOX_35=0.6, MIME_8BIT_HEADER=0.3, MIME_BASE64_TEXT=1.753, MIME_CHARSET_FARAWAY=2.45, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_MED=-4, SARE_MILLIONSOF=0.315, SARE_SUB_ENC_GB2312=1.345] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 60sRhodt7fVP; Thu, 25 Aug 2011 18:59:13 -0700 (PDT) Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [119.145.14.67]) by ietfa.amsl.com (Postfix) with ESMTP id 36C1621F86C0; Thu, 25 Aug 2011 18:59:13 -0700 (PDT) Received: from huawei.com (szxga04-in [172.24.2.12]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQI00FGBIWNB2@szxga04-in.huawei.com>; Fri, 26 Aug 2011 10:00:24 +0800 (CST) Received: from szxrg01-dlp.huawei.com ([172.24.2.119]) by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQI00HJLIWLZ1@szxga04-in.huawei.com>; Fri, 26 Aug 2011 10:00:23 +0800 (CST) Received: from 172.24.2.119 (EHLO szxeml203-edg.china.huawei.com) ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.1.9-GA FastPath queued) with ESMTP id ADL19985; Fri, 26 Aug 2011 10:00:23 +0800 (CST) Received: from SZXEML408-HUB.china.huawei.com (10.82.67.95) by szxeml203-edg.china.huawei.com (172.24.2.55) with Microsoft SMTP Server (TLS) id 14.1.270.1; Fri, 26 Aug 2011 10:00:16 +0800 Received: from SZXEML525-MBS.china.huawei.com ([169.254.8.40]) by szxeml408-hub.china.huawei.com ([169.254.27.188]) with mapi id 14.01.0270.001; Fri, 26 Aug 2011 10:00:23 +0800 Date: Fri, 26 Aug 2011 02:00:22 +0000 From: Xuxiaohu In-reply-to: X-Originating-IP: [10.110.98.53] To: Linda Dunbar Message-id: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE7258BE@szxeml525-mbs.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_56762OkhtVRPuc8Wc3znzg)" Content-language: zh-CN Accept-Language: zh-CN, en-US Thread-topic: [armd] Could millions of ARP entries on DC gateways be reduced to one Thread-index: Acxi0w1AcibzuVsiSkK+GwRxBeP6pgAMt1qAACKD51A= X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected References: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE725622@szxeml525-mbs.china.huawei.com> Cc: "l2vpn@ietf.org" , "armd@ietf.org" Subject: [armd] =?gb2312?b?tPC4tDogIENvdWxkIG1pbGxpb25zIG9mIEFSUCBlbnRy?= =?gb2312?b?aWVzIG9uIERDIGdhdGV3YXlzIGJlIHJlZHVjZWQgdG8gb25l?= X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2011 01:59:14 -0000 --Boundary_(ID_56762OkhtVRPuc8Wc3znzg) Content-type: text/plain; charset=gb2312 Content-transfer-encoding: base64 DQoNCreivP7IyzogTGluZGEgRHVuYmFyIFttYWlsdG86ZHVuYmFyLmxsQGdtYWlsLmNvbV0NCrei y83KsbzkOiAyMDExxOo41MIyNsjVIDE6MDQNCsrVvP7IyzogWHV4aWFvaHUNCrOty806IGwydnBu QGlldGYub3JnOyBhcm1kQGlldGYub3JnDQrW98ziOiBSZTogW2FybWRdIENvdWxkIG1pbGxpb25z IG9mIEFSUCBlbnRyaWVzIG9uIERDIGdhdGV3YXlzIGJlIHJlZHVjZWQgdG8gb25lDQoNClhpYW8g SHUsDQoNClRoYXQgaXMgYSB2ZXJ5IGludGVyZXN0aW5nIHByb3Bvc2FsLg0KRG8geW91IG1lYW4g dGhhdCB0aGVyZSBpcyBhIHNpbmdsZSBBUlAgcHJveHkgd2hpY2ggcmVwcmVzZW50cyBtaWxsaW9u cyBvZiBWTXM/IEFsbCB0aGUgcGFja2V0cyBkZXN0aW5lZCB0byBhbnkgb2YgdGhvc2UgIm1pbGxp b25zIG9mIFZNcyIgd2lsbCBiZSByZXJvdXRlZCB0byAidGhlIHNpbmdsZSBBUlAgcHJveHkiPyBX aWxsIHRoaXMgY3JlYXRlIHRvbyBtdWNoIHByZXNzdXJlIG9uIHRoaXMgc2luZ2xlIHBvaW50Pw0K DQpIaSBMaW5kYSwNCg0KSXShr3MganVzdCBhbiBleGFtcGxlLiBJbiBwcmFjdGljZSwgaWYgdGhl IHByZXNzdXJlIG9uIHRoZSBMMlZQTiBQRSByb3V0ZXIgKGlmIFZpcnR1YWwgU3VibmV0IGlzIHVz ZWQgZm9yIERDSSkgd2l0aGluIHRoZSBEQyBpcyB0b28gbGFyZ2UsIHlvdSBjb3VsZCBjb25uZWN0 IHRoZSBEQyBleGl0IHJvdXRlciB0byBtdWx0aXBsZSBMMlZQTiBQRSByb3V0ZXJzIHRvIGFjaGll dmUgbG9hZC1iYWxhbmNpbmcuIEZvciBleGFtcGxlLCB0aGUgdHJhZmZpYyB3aXRoaW4gVkxBTiAx LTEwMDAgaXMgZm9yd2FyZGVkIHRocm91Z2ggUEUtMSBvZiB0aGF0IEwyVlBOLCAgYW5kIHRoZSB0 cmFmZmljIHdpdGhpbiBWTEFOIDEwMDEtMjAwMCBpcyBmb3J3YXJkZWQgdGhyb3VnaCBQRS0yIG9m IHRoYXQgTDJWUE6hrSAgU3VjaCBsb2FkLWJhbGFuY2luZyB1c2FnZSBpcyB2ZXJ5IGNvbW1vbiBu byBtYXR0ZXIgd2hhdCBzcGVjaWFsIEwyIHRlY2hub2xvZ3kgKGUuZy4sIFNUUCwgU1BCLFRSSUxM LFZQTFMgoa0pIGlzIHVzZWQuDQoNCkhvdyBpcyB5b3VyIHByb3Bvc2VkIGFwcHJvYWNoIGNvbXBh cmluZyB3aXRoIFNFQVRUTEUncyBkaXN0cmlidXRlZCBhcHByb2FjaD8NCg0KU0VBVFRMRSBpcyBv bmUgb2YgdGhlIG1hbnkgTUFDLXJvdXRpbmcgYmFzZWQgTDJWUE4gc29sdXRpb25zLiBXaXRoIGFu eSBNQUMtcm91dGluZyBiYXNlZCBMMlZQTiwgdGhlIGdhdGV3YXkgcm91dGVyIHNob3VsZCBhbmQg d291bGQgZ2V0IHRoZSByZWFsIE1BQyBhZGRyZXNzZXMgb2YgdGhlIHJlcXVlc3RlZCBob3N0cyBp biBBUlAgcmVzb2x1dGlvbi4gU2luY2UgdGhlc2UgTUFDIGFkZHJlc3NlcyBhcmUgbm90IGlkZW50 aWNhbCwgdGhlIEFSUCBjYWNoZSBlbnRyaWVzIG9uIHRoZSBnYXRld2F5IHJvdXRlciBjb3VsZCBu b3QgYmUgYWdncmVnYXRlZCBhdCBhbGwuDQoNClhpYW9odQ0KDQpMaW5kYSBEdW5iYXINCk9uIFdl ZCwgQXVnIDI0LCAyMDExIGF0IDk6NDcgUE0sIFh1eGlhb2h1IDx4dXhpYW9odUBodWF3ZWkuY29t PG1haWx0bzp4dXhpYW9odUBodWF3ZWkuY29tPj4gd3JvdGU6DQpIaSBhbGwsDQoNCkFub3RoZXIg b2J2aW91cyBhZHZhbnRhZ2Ugb2YgaG9zdCByb3V0ZSBiYXNlZCBJUC1vbmx5IEwyVlBOIChlLmcu LCBWaXJ0dWFsIFN1Ym5ldCkgb3ZlciBWUExTLCBhcyBhIERDSSBzb2x1dGlvbiwgaXMgdG8gcmVk dWNlIHRoZSBBUlAgdGFibGUgc2l6ZSBvbiBEQyBnYXRld2F5cyBieSBzZXZlcmFsIG9yZGVyIG9m IG1hZ25pdHVkZS4gQXNzdW1lIHRoZXJlIGFyZSBtaWxsaW9ucyBvZiBDRSBob3N0cyAoaS5lLixW TXMpIHdpdGhpbiBhIHNpbmdsZSBWTEFOL3N1Ym5ldCwgaWYgVlBMUyBpcyB1c2VkIGFzIGEgRENJ IHNvbHV0aW9uLCBEQyBnYXRld2F5cyAoaS5lLiwgREMgZXhpdCByb3V0ZXJzKSB3b3VsZCBoYXZl IHRvIGtub3cgbWlsbGlvbnMgb2YgQVJQIGVudHJpZXMgY29ycmVzcG9uZGluZyB0byB0aGVzZSBW TXMuIEluIGNvbnRyYXN0LCBpbiB0aGUgVmlydHVhbCBTdWJuZXQgc29sdXRpb24sIERDIGdhdGV3 YXlzIGFyZSBkaXJlY3RseSBjb25uZWN0ZWQgdG8gdGhlIFBFIHJvdXRlcnMgb2YgdGhlIElQLW9u bHkgTDJWUE4gd2hpY2ggYWN0IGFzIEFSUCBwcm94aWVzLCBNQUMgYWRkcmVzc2VzIG9mIHRob3Nl IEFSUCBlbnRyaWVzIGZvciBWTXMgb24gREMgZ2F0ZXdheXMgYXJlIGlkZW50aWNhbCAoaS5lLiwg QVJQIFByb3h5oa9zIE1BQykuIFRodXMgdGhlc2UgbWlsbGlvbnMgb2YgQVJQIGVudHJpZXMgY2Fu IGJlIGFnZ3JlZ2F0ZWQgaW50byBvbmUgZW50cnkgKDEuMC4wLjAvOC08aHR0cDovLzEuMC4wLjAv OC0+PkFSUCBwcm94eaGvcyBNQUMpLiBUaGF0oa9zIHRvIHNheSwgdGhlIGV4YWN0LW1hdGNoaW5n IGFsZ29yaXRobSBmb3IgQVJQIGNhY2hlIGxvb2t1cCBpcyBjaGFuZ2VkIHRvIHRoZSBsb25nZXN0 LW1hdGNoaW5nIGFsZ29yaXRobS4gT2YgY291cnNlLCB0aGVyZSBpcyBubyBmcmVlIGx1bmNoLiBU aGUgc2lkZS1lZmZlY3Qgb2YgdGhpcyBjaGFuZ2UgaXMgdGhhdCBEQyBnYXRld2F5cyBjb3VsZCBz ZW5kIG91dCBwYWNrZXRzIGRlc3RpbmVkIGZvciBub24tZXhpc3RpbmcgVk1zIHRvIHRoZSBQRSBy b3V0ZXJzIG9mIHRoYXQgSVAtb25seSBMMlZQTi4gRm9ydHVuYXRlbHksIG9uY2UgdGhvc2UgcGFj a2V0cyBhcnJpdmUgYXQgdGhlIFBFIHJvdXRlcnMgb2YgdGhhdCBJUC1vbmx5IEwyVlBOLCB3aGlj aCBpbiB0dXJuIHdpbGwgZHJvcCB0aGVtIGRpcmVjdGx5IHNpbmNlIHRoZXJlIGlzIG5vIG1hdGNo aW5nIHJvdXRlcyBmb3IgdGhlbS4NCg0KU2luY2UgdGhpcyB0b3BpYyBpcyBhbHNvIGludGVyZXN0 aW5nIHRvIEFSTUQgV0csIEkgY29weSB0aGlzIGVtYWlsIHRvIEFSTUQuDQoNCkJlc3QgcmVnYXJk cywNClhpYW9odQ0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXw0KYXJtZCBtYWlsaW5nIGxpc3QNCmFybWRAaWV0Zi5vcmc8bWFpbHRvOmFybWRAaWV0Zi5v cmc+DQpodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCg0K --Boundary_(ID_56762OkhtVRPuc8Wc3znzg) Content-type: text/html; charset=gb2312 Content-transfer-encoding: quoted-printable

     = ;

     = ;

    =B7=A2=BC=FE=C8=CB: Linda D= unbar [mailto:dunbar.ll@gmail.com]
    =B7=A2= =CB=CD=CA=B1=BC=E4: 2011=C4=EA8=D4=C226=C8=D5 1:04
    =CA=D5=BC=FE=C8=CB: Xuxiaohu
    =B3=AD=CB=CD: l2vpn@ietf.org; armd@ietf.org
    =D6=F7=CC=E2: Re: [armd] Could millions of ARP entries on DC gateways be reduced to one=

     

    Xiao Hu,

     

    That is a very interesting prop= osal.

    Do you mean that there is a sin= gle ARP proxy which represents millions of VMs? All the packets destined to= any of those "millions of VMs" will be rerouted to "the sin= gle ARP proxy"? Will this create too much pressure on this single point?

     = ;

    Hi Linda,<= o:p>

     = ;

    It=A1=AFs = just an example. In practice, if the pressure on the L2VPN PE router (if Vi= rtual Subnet is used for DCI) within the DC is too large, you could connect the DC exit router to multiple L2VPN PE routers to achieve load-ba= lancing. For example, the traffic within VLAN 1-1000 is forwarded through P= E-1 of that L2VPN,  and the traffic within VLAN 1001-2000 is forwarded= through PE-2 of that L2VPN=A1=AD  Such load-balancing usage is very common no matter what special L2 technology (e.g., STP, SPB,= TRILL,VPLS =A1=AD) is used.

     = ;

    How is your proposed approach c= omparing with SEATTLE's distributed approach?

     = ;

    SEATTLE is= one of the many MAC-routing based L2VPN solutions. With any MAC-routing ba= sed L2VPN, the gateway router should and would get the real MAC addresses of the requested hosts in ARP resolution. Since these MAC ad= dresses are not identical, the ARP cache entries on the gateway router coul= d not be aggregated at all.

     = ;

    Xiaohu

     

    Linda Dunbar<= /p>

    On Wed, Aug 24, 2011 at 9:47 PM= , Xuxiaohu <xuxiaohu@huawei.com> wrote:

    Hi all,

     

    Another obvious advantage of host route based= IP-only L2VPN (e.g., Virtual Subnet) over VPLS, as a DCI solution, is to r= educe the ARP table size on DC gateways by several order of magnitude. Assume there are millions of CE hosts (i.e.= ,VMs) within a single VLAN/subnet, if VPLS is used as a DCI solution, DC ga= teways (i.e., DC exit routers) would have to know millions of ARP entries c= orresponding to these VMs. In contrast, in the Virtual Subnet solution, DC gateways are directly connected to the = PE routers of the IP-only L2VPN which act as ARP proxies, MAC addresses of = those ARP entries for VMs on DC gateways are identical (i.e., ARP Proxy=A1= =AFs MAC). Thus these millions of ARP entries can be aggregated into one entry (1.0.0.0/8->ARP proxy=A1=AFs MAC). That=A1=AFs to say= , the exact-matching algorithm for ARP cache lookup is changed to the longe= st-matching algorithm. Of course, there is no free lunch. The side-effect of this change is that DC gateways could send out p= ackets destined for non-existing VMs to the PE routers of that IP-only L2VP= N. Fortunately, once those packets arrive at the PE routers of that IP-only= L2VPN, which in turn will drop them directly since there is no matching routes for them.

     

    Since this topic is also interesting to ARMD = WG, I copy this email to ARMD.

     

    Best regards,

    Xiaohu

    =
    _______________________________________________
    armd mailing list
    armd@ietf.org
    ht= tps://www.ietf.org/mailman/listinfo/armd

     

    --Boundary_(ID_56762OkhtVRPuc8Wc3znzg)-- From linda.dunbar@huawei.com Thu Aug 25 21:07:38 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0657521F8785 for ; Thu, 25 Aug 2011 21:07:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.728 X-Spam-Level: X-Spam-Status: No, score=-5.728 tagged_above=-999 required=5 tests=[AWL=-0.330, BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_46=0.6, J_CHICKENPOX_66=0.6, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T4h8OTN20Hae for ; Thu, 25 Aug 2011 21:07:36 -0700 (PDT) Received: from usaga02-in.huawei.com (usaga02-in.huawei.com [206.16.17.70]) by ietfa.amsl.com (Postfix) with ESMTP id E868A21F8586 for ; Thu, 25 Aug 2011 21:07:35 -0700 (PDT) Received: from huawei.com (localhost [127.0.0.1]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQI00JENOUQ4Q@usaga02-in.huawei.com> for armd@ietf.org; Thu, 25 Aug 2011 23:08:50 -0500 (CDT) Received: from dfweml202-edg.china.huawei.com ([172.18.4.104]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LQI00GCROUQ39@usaga02-in.huawei.com> for armd@ietf.org; Thu, 25 Aug 2011 23:08:50 -0500 (CDT) Received: from DFWEML401-HUB.china.huawei.com (10.193.5.101) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.270.1; Thu, 25 Aug 2011 21:08:51 -0700 Received: from DFWEML504-MBX.china.huawei.com ([169.254.4.82]) by DFWEML401-HUB.china.huawei.com ([fe80::f07f:889f:78ef:8df3%13]) with mapi id 14.01.0270.001; Thu, 25 Aug 2011 21:08:49 -0700 Date: Fri, 26 Aug 2011 04:08:48 +0000 From: Linda Dunbar X-Originating-IP: [10.47.136.107] To: "armd@ietf.org" Message-id: <4A95BA014132FF49AE685FAB4B9F17F60C0EB3FA@dfweml504-mbx.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_i3k3q9Dp6yEgj6SuzdlJIA)" Content-language: en-US Accept-Language: en-US Thread-topic: 81st IETF ARMD WG session minutes Thread-index: Acxjpd05ppP4J8v5TWKj0BD3bvD6Yg== X-MS-Has-Attach: X-MS-TNEF-Correlator: Subject: [armd] 81st IETF ARMD WG session minutes X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Aug 2011 04:07:38 -0000 --Boundary_(ID_i3k3q9Dp6yEgj6SuzdlJIA) Content-type: text/plain; charset=utf-8 Content-transfer-encoding: base64 VGhhbmtzIHRvIFBoaWxpcCBFYXJkbGV5IGFuZCBEb25hbGQgRWFzdGxha2UgZm9yIHRha2luZyB0 aGUgbm90ZXMgZHVyaW5nIDgxc3QgSUVURiBBUk1EIFdHIHNlc3Npb24uIEhlcmUgYXJlIHRoZSBt aW51dGVzIGNvbWJpbmVkIGZyb20gYm90aCBvZiB0aGVpciBub3Rlcy4NCklmIHdlIG1pc3MgYW55 dGhpbmcsIHBsZWFzZSBsZXQgdXMga25vdy4gV2UgbmVlZCB0byB1cGxvYWQgdGhlIG1pbnV0ZXMg YnkgRnJpZGF5IGV2ZW5pbmcuDQoNCkxpbmRhICYgQmVuc29uDQoNCk1lZXRpbmcgTWludXRlcyBB Uk1EIElFVEYgODFzdCBRdWViZWMgQ2l0eQ0KDQpBZGRyZXNzIFJlc29sdXRpb24gZm9yIE1hc3Np dmUgbnVtYmVycyBvZiBob3N0cyBpbiB0aGUgRGF0YSBjZW50ZXIgKEFSTUQpDQpMb2NhdGlvbjoy MDUgQUJDIG1lZXRpbmcgcm9vbSwgUXVlYmVjIENpdHkgQ29udmVudGlvbiBDZW50cmUsIFF1ZWJl YywgQ2FuYWRhDQpUaW1lOjI5LUp1bHktMjAxMSwgMDkwMC0xMTMwIC0gRnJpZGF5LCBNb3JuaW5n IFNlc3Npb24gSQ0KQ2hhaXJzOkJlbnNvbiBTY2hsaWVzc2VyIChic2NobGllc0BjaXNjby5jb208 bWFpbHRvOmJzY2hsaWVzQGNpc2NvLmNvbT4pICYgTGluZGEgRHVuYmFyIChsaW5kYS5kdW5iYXJA aHVhd2VpLmNvbTxtYWlsdG86bGluZGEuZHVuYmFyQGh1YXdlaS5jb20+KQ0KSmFiYmVyOmFybWRA amFiYmVyLmlldGYub3JnDQpVUkw6aHR0cDovL3Rvb2xzLmlldGYub3JnL3dnL2FybWQvDQoNCi0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCjEuICAgICBNZWV0aW5nICBBZG1pbmlzdHJpdmlhIChj aGFpcnMgLSAwNSBtaW4pDQril6ZXZWxjb21lDQril6ZNYWlsaW5nIGxpc3QgYW5kIFVSTA0K4pem TWludXRlcyBTY3JpYmUNCuKXpkphYmJlciBTY3JpYmUNCuKXpkJsdWUgU2hlZXRzDQpTbGlkZXMg bm93IGJlaW5nIHVwbG9hZCB0byBtZWV0aW5nIG1hdGVyaWFscywgYWxyZWFkeSBvbiB3ZWJleA0K DQoNCjIuICAgICBBUk1EIENhbGwgZm9yIEludmVzdGlnYXRpb24gKGNoYWlycyAtIDEwIG1pbikN CmRyYWZ0LWlldGYtYXJtZC1jYWxsLWZvci1pbnZlc3RpZ2F0aW9uLTAwPGh0dHA6Ly90b29scy5p ZXRmLm9yZy9odG1sL2RyYWZ0LWlldGYtYXJtZC1jYWxsLWZvci1pbnZlc3RpZ2F0aW9uLTAwPg0K YXNrcyBhIGZldyBxdWVzdGlvbnMgdG8gcHJvbXB0IHRoZSB0aG91Z2h0LiBEb2N1bWVudCBpcyBv bmx5IHRlbXBvcmFyeS4gV2lsbCBiZSBhbGxvd2VkIHRvIGV4cGlyZS4NCk1hbnkgcGVvcGxlIHJh aXNlZCBoYW5kIHNob3dpbmcgdGhhdCB0aGV5IHJlYWQgdGhlIGRyYWZ0LiBUaGVyZSBpcyBubyBx dWVzdGlvbnMNClRoZXJlIHdhcyBhIE5BTk9HIHNlc3Npb24gd2hlcmUgb3BlcmF0b3JzIHRhbGtl ZCBhYm91dCB0aGVpciBwcm9ibGVtcw0KRGF2aWQgQmxhY2s6IGFyZSB5b3UgYWxzbyBpbnRlcmVz dGVkIGluIERDIGRlc2lnbmVy4oCZcyBmZWVkYmFjayBhYm91dCBob3cgYWRkcmVzcyByZXNvbHV0 aW9uIHdvcmtzPw0KDQoNCjMuICAgICBQcm9ibGVtIFN0YXRlbWVudCBmb3IgQVJNRChUaG9tYXMg TmFydGVuIC0gMTUgbWluKQ0KZHJhZnQtbmFydGVuLWFybWQtcHJvYmxlbS1zdGF0ZW1lbnQtMDA8 aHR0cDovL3Rvb2xzLmlldGYub3JnL2h0bWwvZHJhZnQtbmFydGVuLWFybWQtcHJvYmxlbS1zdGF0 ZW1lbnQtMDA+DQpkZWxpYmVyYXRlbHkgbmFycm93bHkgc2NvcGVkDQpmaW5kIHRvIGlkZW50aWZ5 IHJlcHJlc2VudGF0aXZlIHByb2JsZW1zIHNlZW4gaW4gdGhlIGZpZWxkDQpIaW1hbnNodSA6ICAt IGhvdyBtYW55IERDIG9wZXJhdHBycyBhcmUgaGVyZS4gQWJvdXQgNT8NCkNvbW1lbnQgcmVnYXJk IHRvIFRob21hcyByZW1hcmtzOiBoaXN0b3J5IG9mIFdHIHZlcnkgc3RyYW5nZS4gU2NvcGUga2Vl cHMgZ2V0dGluZyByZWR1Y2VkLiBBcmUgd2UgZ29pbmcgdG8gZG8gYW55IHdvcmsgaGVyZT8NCkJl bnNvbjogcmVxdWlyZW1lbnRzIGlzIHdvcmssIG5vIHByb3RvY29scw0KVGhlbiB3aGVyZT8NCkNo YWlycyDigJMgd2XigJlyZSBub3QgZG9pbmcgcHJvdG9jb2wgYXQgdGhpcyBzdGFnZSBvZiBBUk1E LiBNYXliZSBpbiB0aGUgZnV0dXJlIGFmdGVyIHJlLWNoYXJ0ZXJpbmcuDQpSb24gQm9uaWNhLCBP cHMgQUQg4oCTIG5vIHByb3RvY29sIGRldmVsb3BtZW50IGluIE9wcyBhcmVhLg0KTmFydGVuIOKA kyBsZXTigJlzIG5vdCBnZXQgaHVuZyB1cCBpbiBwcm9jZXNzLiBMZXTigJlzIGlkZW50aWZ5IGFu ZCBhZ3JlZSBwcm9ibGVtcyAvcmVxdWlyZW1lbnRzIGFuZCB0aGVuIHdvcmsgb3V0IHdoZXJlIHRv IGRvIHdvcmsuIElmIGl0IHRha2VzIGEgeWVhciB0byBkbyByZXF1aXJlbWVudHMsIHRoZW4gSSB0 aGluayB3ZSBoYXZlIGJsb3duIGl0Lg0KSmFiYmVyIOKAkyB1cGxvYWQgdGhlIHNsaWRlcyEhDQoN CjQuICAgICBOQU5PRyA1MiBPcGVyYXRvcnMgUGVyc3BlY3RpdmUoU3VzYW4gSGFyZXMgLSAxNSBt aW4pDQpkcmFmdC1oYXJlcy1hcm1kLW5hbm9nNTItMDA8aHR0cDovL3Rvb2xzLmlldGYub3JnL2h0 bWwvZHJhZnQtaGFyZXMtYXJtZC1uYW5vZzUyLTAwPg0KVGhlIGdvYWwgZm9yIE5BTk9HIEFSTUQg c2Vzc2lvbiBpcyBmb3Igb3BlcmF0b3JzIHRvIHBsZWFzZSB0ZWxsIHVzIHdoYXQgdGhlIHBhaW4g cG9pbnRzIGFyZS4NClNwZWFrZXJzIGF0IE5BTk9HIEFSTUQgdHJhY2s6IEFkaG9zdCwgR29vZ2xl LCBZYWhvbywgTWVyaXQsIGFuZCBBVCZUIFJlc2VhcmNoDQpQbGVhc2UgZ28gdG8gd3d3Lm5hbm9n Lm9yZzxodHRwOi8vd3d3Lm5hbm9nLm9yZz4gIHRvIHNlZSBzbGlkZXMgdGhhdCB3ZXJlIHByZXNl bnRlZCB0aGVyZSDigJMgbWlkLXJhbmdlIERDIG9wZXJhdG9yLCBHb29nbGUgYnVzaW5lc3MgY2Fz ZSBldGMNCldoeSBMYXllciAyIHdpdGhpbiBEYXRhIENlbnRlcnM/DQpGb3IgbWFueSBtaWQtcmFu Z2UgREMgb3BlcmF0b3JzLCBlc3BlY2lhbGx5IHRoZSBtdWx0aS10ZW5hbnQgZGF0YSBjZW50ZXJz IOKAkyBtdWx0aXBsZSBjdXN0b21lcnMgZG9u4oCZdCB3YW50IHRvIGNoYW5nZSB0aGVpciBhcHBz LCBuZWVkcyBMYXllciAyIGluIERDIGZvciB0aGVpciBidXNpbmVzcyBtb2RlbCB0byB3b3JrLiBT ZWUgQVJQIHByb2JsZW1zLg0KR28gR3JpZOKAmXMgQXBwbGljYXRpb24gTDINCldoYXQgV2UgTGVh cm5lZDoNCiAgICAgTG90cyBvZiBNaWRkbGUgdGllciBEQ3MgaGF2ZSBtdWx0aS10ZW5hbnQNCiAg ICAgTmVlZCBMYXllciAyIGZvciBidXNpbmVzcyBtb2RlbA0KICAgICBNaWRkbGUgVGllciBEQ3Mg c2VlIEFSUCBwcm9ibGVtcw0KICAgICBXYW50IHRvIGNvbGxhYm9yYXRlIHdpdGggb3RoZXIgTWlk ZGxlLVRpZXJzDQoNClRob21hcyBOYXJ0ZW4g4oCTIHNsaWRlIDUsIEkgcnVsZWQgTDIgc3Bhbm5p bmcgbXVsdGlwbGUgc2l0ZXMgYXMgb3V0IG9mIHNjb3BlLg0KU3VzYW4g4oCTIHdoZW4gSSBhc2tl ZCBBZEhvc3Qg4oCTIFRoZXkgaGF2ZSAzIHNpdGVzLCBhbGwgaW4gc2luZ2xlIGNpdHkNClJvYiBT aGFraXIg4oCTIG1haW4gaXNzdWUgaXMgc2NhbGluZy4gSSBidWlsZCBWUExTIHdoaWNoIGNhbiBz cGFuIGdsb2JlLg0KQW5zd2VyOiBWUExTIHNwYW5zIGdsb2JlLiBIb3dldmVyLCB0aGUgbnVtYmVy IG9mIG5vZGVzIGludGVyY29ubmVjdGVkIGJ5IFZQTFMgaXMgbm90IGFzIG1hbnkgYXMgdGhlIG51 bWJlciBob3N0cyBpbiBEYXRhIENlbnRlci4gIFRoZXJlZm9yZSB0aGUgYWRkcmVzcyBzY2FsaW5n IGluIERhdGEgQ2VudGVycyBoYXMgZGlmZmVyZW50IGRpbWVuc2lvbiB0aGFuIFZQTFMuDQpYIChV Syk6IEkgZG8gVlBMU3MgdGhhdCBhcmUgdmVyeSBsYXJnZS4gTGF0ZW5jeSBjYW4gYmUgYSBwcm9i bGVtIGJ1dCB0aGlzIGdyb3VwIGlzIGFib3V0IGFkZHJlc3MgcmVzb2x1dGlvbi4gSXQgaXMgc2Nh bGluZywgcmlnaHQ/IFdlIHNob3VsZCBkb2N1bWVudCByZWFsIHRvcG9sb2dpZXMgYW5kIHNheSB3 aGF0IHRoZSBwcm9ibGVtcyBhcmUgd2l0aCB0aGF0IHRvcG9sb2d5LiBJIGFtIHdpbGxpbmcgdG8g aGVscCB3cml0ZSB0aGlzLiBFYXNpZXIgdG8gc2hhcmUgcHJpdmF0ZWx5LiBNeSBwcm9ibGVtIHdp dGggZ2V0dGluZyBwZW9wbGUgdG8gaW50ZXJhY3Qgd2l0aCB0aGUgSUVURiBpcyB0aGF0IHBlb3Bs ZSAgdGhpbmsgaXQgaXMgaXJyZWxldmFudC4NCkJlbnNvbjogTGF0ZW5jeSBtYXkgYmUgYSBwcm9i bGVtIG5vdCBpbiBzY29wZSwgYnV0IHdoYXQgaWYgbGF0ZW5jeSBhZmZlY3RzIGFkZHJlc3MgcmVz b2x1dGlvbi4NCk5pbmcgU286IFBlb3BsZSB0cnkgdG8gZG8gb25lIHNpemUgZml0cyBhbGwgbW9k ZWxzLiBXaGVuIHdlIHRhbGsgdG8gY3VzdG9tZXJzLCB0aGV5IGhhdmUgYWxyZWFkeSBiZWVuIHJ1 bm5pbmcgdGhlaXIgb3duIGRhdGEgY2VudGVycyBhcm91bmQgdGhlIHdvcmxkLiBUaGV5IHdhbnQg dG8gb2ZmLWxvYWQgc29tZSBzdHVmZiBvbiB1cy4gRm9yIGFuIGluZGVmaW5pdGUgdGltZSB0aGV5 IHdpbGwgd2FudCB0byBjb250aW51ZSB0byBvcGVyYXRlIHRoZWlyIG93biBkYXRhIGNlbnRlcnMg YWxvbmcgd2l0aCB1c2luZyBvdXJzLg0KDQoNCldhcnJlbiDigJMgY2FuIHlvdSBzdXBwbHkgcGlj IHdpdGggeW91ciBwcm9ibGVtPw0KUm9iIChVSyBvcGVyYXRvcikg4oCTIFdlIHNob3VsZCBoYXZl IHRvcG9sb2d5IHBpY3MgdG8gaWRlbnRpZnkgYWN0dWFsIGlzc3Vlcy4gTmVlZCB0byBnZW5lcmFs aXplIHNvIG5vdCBjb21tZXJjaWFsIGluZm8uIEkgd2lsbCBoZWxwLg0KUm9iIOKAkyBJIHRhbGsg d2l0aCBwZW9wbGUgd2hvIHRoaW5rIGlldGYgYXJlIGlycmVsZXZhbnQuDQpOaW5nICBTbyDigJMg d2Ugd2FudCB0byBydW4gbXVsdGktdGVuYW50IERDLiBDdXN0b21lcnMgd2FudCB0byBvZmZsb2Fk IHRoZWlyIGludGVybmFsIERDcyB0byB1cywgYnV0IHdpbGwga2VlcCBydW5uaW5nIHRoZWlyIG93 biBEQyDigJMgc28gd2lsbCBydW4gb3ZlciBib3RoLiBTbyB0aGUgYXJjaGl0ZWN0dXJlIGRlc2Ny aWJlZCBpbiBzbGlkZSA1IG1ha2VzIHNlbnNlIHRvIHVzLg0KQW5kcmV3IE1jR3JlZ29yIOKAkyBu ZWVkIHN3aXRjaCB2ZW5kb3JzIGFzIHdlbGwgKGhhbmRzIOKAkyB0aGV5IGFyZSBoZXJlKS4gVGhl IHNjZW5hcmlvIGlzIGZhaXJseSBmcmVxdWVudC4NCkNhdGh5IEVybnNvbiDigJMgbmVlZCBwcm92 aWRlcnMgdG8gY29tZSB0byBOQU5PRyB0byBnaXZlIGlucHV0Lg0KRXJpYyBLbGluZTog4oCTIHdo ZXJlIHNsaWRlcyBzYXkgQVJQIGRvIHRoZXkgbWVhbiBhZGRyZXNzIHJlc29sdXRpb24/IEFuc3dl cjogIFllcw0KUmFuZHkgQnVzaDogTWFueSBvcGVyYXRvcnMgdGhpbmsgdGhleSBnYXZlIGNvbnNp ZGVyYWJsZSBpbnB1dCBhdCBOQU5PRy4gSSB0aGluayB3ZSBoYXZlIGFuIGVhciByZXNvbHV0aW9u IHByb2JsZW0uDQpCZW5zb246IFllcywgd2UgaGFkIGEgZ29vZCB0dXJub3V0LiAgT3BlcmF0b3Jz IGNhbiB0YWxrIHRvIHVzIHByaXZhdGVseS4gV2UgYWxzbyB3YW50IGRpc2N1c3Npb24gb24gdGhl IGxpc3QgYWJvdXQgdGhpcy4gTmVlZCB0byBlbmQgdXAgd2l0aCB0ZXh0Lg0KDQpOaW5nIFNvOiBS ZXNwb25kaW5nIHRvIHRoZSBkb27piKXmqpsgZG8gdGhhdCBjb21tZW50OiBWZW5kb3Igc2F5cyBk b27piKXmqpsgZG8gdGhhdC4gV2Ugc2F5IGRvbumIpeaqmyBkbyB0aGF0LiBDdXN0b21lciBzYXlz IHRoYXTpiKXmqpogd2hhdCBJIHdhbnQgYW5kIHdoYXQgeW91ciBjb21wZXRpdG9ycyBhcmUgcHJv dmlkaW5nLiBTbyB3ZSBjb21lIGJhY2sgdG8gcHVzaCB2ZW5kb3IgdG8gZml4IGl0LiBTdWNoIGFz IEJHUCByb3V0ZSBwcm9ibGVtLg0KQmVuc29uOiBCb3RoIHZpZXdzIHNlZW0gcmVhc29uYWJsZS4g RGlmZmVyZW50IHNjZW5hcmlvcy4NCldhcnJlbjog4oCTIGRvY3RvciBzYXlzIGRvbuKAmXQgZG8g aXQgaWYgaXQgaHVydHMuIEJpZyBMYXllciAyIGhhcyBwcm9ibGVtcy4gVGhlcmUgaXMgYSByZWFz b24gdGhlIEludGVybmV0IGlzIG5vdCBhIGJpZyBMYXllciAyIG5ldHdvcmsuIE1heWJlIGEgQkNQ IHNheWluZyBqdXN0IGRvbuKAmXQgZG8gdGhhdCB3aWxsIGJlIHRoZSByZXN1bHQuDQpWTSBtb2Jp bGl0eSB0aHJvdWdoIG92ZXJsYXlzLCBzdWJuZXQgZXRjIGFzIHNvbHV0aW9ucw0KTmluZyBTTzog 4oCTIGN1c3RvbWVycyBjYW4gaW5zaXN0IHdlIGRvIHRoaXMuDQpSYW5keSDigJMgTGV04oCZcyBn ZXQgZG93biB0byB0ZWNobmljYWwgZGlzY3Vzc2lvbiBvciBlbHNlIGdvIGZvciBjb2ZmZWUuDQoN Cg0KNS4gICAgIEFkZHJlc3MgUmVzb2x1dGlvbiBTdGF0aXN0aWNzOiBKaW0gUmVlcywgTWFuaXNo IEthcmlyIOKAkyBNZXJpdCBOZXR3b3JrDQoNCmRyYWZ0LWthcmlyLWFybWQtc3RhdGlzdGljcy0w MTxodHRwOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC1rYXJpci1hcm1kLXN0YXRpc3RpY3Mt MDE+DQo1MDAgaG9zdHMsIGNvbmZpZ3VyYWJsZSBpbnRlcmNvbm5lY3QgcmF0ZS4NCkRpbm8g4oCT IGRpZCB5b3Ugc3R1ZHkgdW5rbm93biB1bmljYXN0IChicmlkZ2UgZmxvb2RzKT8gV2UgaGF2ZSBz cGlrZXMgZHVlIHRvIHRoYXQgYnV0IGRpZG7igJl0IGFuYWx5emUgbXVjaC4NCkFub29wIEdoYXdh bmkg4oCTIGlzIGl0IHNwYW5uaW5nIHRyZWU/IFllcy4NCkJlbnNvbiDigJMgVk0gZG9lcyB0cmlj ayBzbw0KRGlubyDigJMgdXBzdHJlYW0gbGlua3MgYXQgdG9wIGFyZSBMMz8gWWVzDQpDUFUgbG9h ZCBzcGlrZXMgdG8gMzAlDQpFcmljIEtsaW5lIOKAkyB3aHkgaXMgY3B1IGxvYWQgaGlnaGVyIGZv ciBhcnAgdGhhbiBORD8NCk1hbmlzaCDigJMgYXJwIGJyb2FkY2FzdHMgZ28gZXZlcnl3aGVyZSwg TkQgdXNlcyBtdWx0aWNhc3QuDQpFcmljIFZ5bmtlIOKAkyBXaGF0IGlzIHRoZSBzaXplPyBhbnN3 ZXIgLzIyLg0KQW5vb3A6ICBsb3dlciBsb2FkIGR1ZSB0byBub3QgZ2V0dGluZyBORCBtZXNzYWdl cyDigJMgc3BlY3VsYXRpb24gb24gdGhlIGFycCB2cyBuZCBpc3N1ZS4NCkRpbm86ICB3YXMgcGFj a2V0cyB0byBzd2l0Y2ggdGhlIHNhbWUgZm9yIHY0ICYgdjY/DQo5MDAtMTAwMCB2NiwgMTAwMC0x NDAwIHY0DQpBUlAvTkQgdHJhZmZpYyBzY2FsZWQgbGluZWFybHkgd2l0aCBudW1iZXIgb2YgaG9z dHMuDQpEaW5vIOKAkyB3aGF04oCZcyB0aGUgYmxhZGU/IERlbGwsIDIgY29yZXMgb24gZWFjaCBi bGFkZQ0KVGhvbWFzIE5hcnRlbiDigJMgd2hhdCBwYXBlcnMgYXJlIGF2YWlsYWJsZSwgdGhlIG5h bm9nIGNoYXJ0cyBhcmUgbm90IHJlYWRhYmxlLiBUaGVzZSBzbGlkZXMgYXJlIGEgYml0IGJldHRl ciwgd291bGQgbGlrZSB0byBtYWtlIGRhdGEgYXZhaWxhYmxlLg0KVGhvbWFzOiBJbnRlcmVzdGlu ZyB3b3JrLiBXaGF0IHBhcGVycyBvciBkb2NzIGFyZSBhdmFpbGFibGU/IENoYXJ0IHJlc29sdXRp b24gbm90IGdvb2Qgb24gTkFOT0cgZG9jdW1lbnRzLg0KSmltOiBzbGlkZXMgYmV0dGVyDQpIaW1h bnNodSDigJMgd2h5IGRvZXMgVk0gbWlncmF0aW9uIGhhdmUgbm8gaW1wYWN0PyBWTXdhcmUgcGxh eXMgdHJpY2tzIHRvIG1pbmltaXNlIHNlcnZpY2UgZGlzcnVwdGlvbiBkdXJpbmcgbWlncmF0aW9u Lg0KRGF2aWQgQmxhY2sg4oCTIHdoaWNoIFZNd2FyZSBzd2l0Y2g/DQpNYW5pc2gg4oCTIFdlIGJ5 cGFzc2VkIHN3aXRjaA0KV2lsbGlhbSBHbG9idXMg4oCTIGRpZCB5b3UgbG9vayBhdCBzd2l0Y2gg b3IgYmxhZGVzPyBNb25pdG9yaW5nIHRyYWZmaWMgb3V0c2lkZSBjYWJpbmV0Lg0KV2VzIEdlb3Jn ZSDigJMgc3BlbmRpbmcgYSBsb25nIHRpbWUgY29uamVjdHVyaW5nIG9uIFZNd2FyZSwgY2FuIHdl IGFzayB0aGVtIGRpcmVjdGx5Lg0KV2FycmVuIOKAkyBleHRyYXBvbGF0aW5nIHRvIDIwayBob3N0 cyBzdWdnZXN0cyBvdmVyIDEwMCUgbG9hZC4gQWN0dWFsbHkgYXJwIGxvYWQgaXMgaW4gdGhlIG5v aXNlLg0KTWFuaXNoIOKAkyBub3QgbnVtYmVyIG9mIGhvc3RzIGJ1dCB0cmFmZmljIHBhdHRlcm4u DQpSYW5keSBCdXNoIOKAkyBvbmUgdHJhaWxlciBpcyBvbiBvcmRlciBvZiAyMGsgaG9zdHMuIFlv deKAmWQgbGlrZSB0cmFpbGVyIHRvIGJlIEwyLiBUaGlzIGlzIGlkZWEgb2YgdGhlIHNjYWxlLg0K VGhvbWFzIE5hcnRlbiDigJMgd291bGQgYmUgdXNlZnVsIHRvIGRvY3VtZW50IGRpZmZlcmVudCBp bXBsZW1lbnRhdGlvbnMgb2YgQVJQLCB0aGV5IGNhbiBoYXZlIHZlcnkgZGlmZmVyZW50IGFsZ29y aXRobXMuDQpSb2IgU2hha2lyIOKAkyBjYXJlIHdoZXJlIHByb2JsZW0gaXMgaW4gaW1wbGVtZW50 YXRpb24gb2YgcHJvdG9jb2wgb3IgcHJvdG9jb2wgc3BhY2UuDQpGbG9yaW4gYmFsdXMg4oCTIG1h eSBiZSBpbnRlcmVzdGluZyB0byBnZXQgZGF0YSBmcm9tIG90aGVyIGVudmlyb25lbWVudCwgZWcg VlBMUy4gU2V2ZXJhbCAxMDAwIGhvc3RzIHN1cHBvcnRlZCBvbiBvbmUgYm94LCBhcnAgbG9hZCBp cyAzLTYlIG9mIGNwdS4NCldhcnJlbiDigJMgbXkgcXVpY2sgcm91Z2ggZXN0aW1hdGUsIDIwayBo b3N0cyBpbiBtZXNoIGlzIDY2MCByZXNvbHV0aW9ucyBwZXIgc2VjIOKAkyBzaG91bGQgYmUgdmVy eSBlYXN5IHRvIGRvLg0KDQoNCjYuICAgICBBZGRyZXNzIFJlc29sdXRpb24gaXNzdWVzIGluZHVj ZWQgYnkgVlBOIG9yaWVudGVkIGRhdGEgY2VudHJlIHNlcnZpY2VzIChOaW5nIFNvOiBWZXJpem9u KQ0KDQpkcmFmdC1zby1hcm1kLXZkY3MtYXItMDA8aHR0cDovL3Rvb2xzLmlldGYub3JnL2h0bWwv ZHJhZnQtc28tYXJtZC12ZGNzLWFyLTAwPg0KTmV0d29yayBjb250cm9sbGluZyBEQyAmIG1vYmls aXR5IG9mIGFwcHMgZXRjLiBBbiBpZGVhIGF0IHRoaXMgc3RhZ2UuDQpEaW5vIOKAkyBpbiB0ZXJt cyBvZiBzY2FsYWJpbGl0eSBzaG91bGRu4oCZdCBtYXR0ZXIsIGl04oCZcyBtYW5hZ2VtZW50LiA/ DQpOaW5nIOKAkyBlZGdlIHJvdXRlcnMgZ2V0dGluZyBoaXQgYnkgY29tcGxleGl0eQ0KVGhvbWFz IE4g4oCTIHNsaWRlIDgg4oCTIGlzIGFkZHJlc3MgY29uZmxpY3QgdGhlIHBhaW4gcG9pbnQ/IERp ZmZlcmVudCBpc3N1ZSB0byBhZGRyZXNzIHJlc29sdXRpb24uDQpBbm9vcCBbQnJvY2FkZV0g4oCT IHNvbHZlIGNsYXNoZXMgd2l0aCBpc29sYXRpb24gdGVjaG5vbG9neQ0KQW5kcmV3IE1jR3JlZ29y IOKAkyBtaXRpZ2lhdGlvbiB0ZWNobmlxdWVzIGNhbiBtYWtlIGFkZHJlc3MgcmVzb2x1dGlvbiB3 b3JzZS8uDQpNYW5pc2gg4oCTIGkgdGhpbmsgeW91ciBwcm9ibGVtIGJlbG9uZ3MgdG8gdGhlIEFS TUQgd2cNCk1hbmNoZXcgQ2llbmEgLSArMQ0KUm9iIFNoYWtpciBVSyDigJMgbmVlZCB0byBsaW1p dCBmYWlsdXJlcyBldGMsIHNvIHdvbuKAmXQgY29ubmVjdCB0byBzYW1lIGJveC4NCk5pbmcgU28g 4oCTIGFscmVhZHkgdGhlcmUgYXJlIHNldmVyYWwgYm94ZXMuDQpNYW5pc2gg4oCTIHdhbnQgdG8g bWF4aW1pc2Ugc2hhcmluZy4NCkZsb3JlbnQg4oCTIGtleSBpcyB0byBsaW1pdCBudW1iZXIgb2Yg aG9zdHMgcGVyIGJyb2FkY2FzdCBkb21haW47IHRoaXMgbWlnaHQgYmUgNSBob3N0cyBmb3IgdnBs cy4NCkRhdmlkIER5c29uIOKAkyB3aGF04oCZcyB0aGUgbGltaXQ/DQpXYXJyZW4g4oCTIGFuZCB0 aGVzZSA1IHRoaW5ncyBjYW4gYmUgcm91dGVycy4NCldhcnJlbiDigJMgY2FuIGRvIHN0dWZmIG9u IEh5cGVydmlzb3IuIEhvdyBtdWNoIG9mIHRoZSBicm9hZGNhc3RzIGlzIEFSUD8g4oCTIGJyb2Fk Y2FzdCBtaXRpZ2F0aW9uIG1heWJlIHRoZSBrZXkgYXBwcm9hY2guDQpXZXMg4oCTIGZvY3VzIG9u IG1pZCBzaXplIERDcy4gV3JpdGUgdXAgaG93IGRpZCBleHBlcmltZW50cyAob3IgY29kZSkNCg0K Ny4gICAgIFZ1bG5lcmFiaWxpdHkgSXNzdWVzIGluIE1pZ3JhdGlvbuKAkyBZaXpob3UgTGkNCmRy YWZ0LWxpeXotYXJtZC12bS1taWdyYXRpb24tcHMtMDE8aHR0cDovL3Rvb2xzLmlldGYub3JnL2h0 bWwvZHJhZnQtbGl5ei1hcm1kLXZtLW1pZ3JhdGlvbi1wcy0wMT4NCk5hcnRlbiDigJMgd2hlbiBW TTEgbW92ZXMgaGFzIHNhbWUgTUFDIGFkZHJlc3M/IFllcy4NCldhcnJlbiDigJMgYXJwIG1hcHMg ZnJvbSBpcCB0byBtYWMgYWRkcmVzcy4gSUYxICYgSUYyIGFyZSBzYW1lIHN1Ym5ldD8gWWVzLiAg TDIgbmVlZHMgdG8ga25vdyBNQUMgYWRkcmVzcyBoYXMgbW92ZWQsIHRoaXMgaXMgbm90IGFycCBi dXQgYnJpZGdlIGxlYXJuaW5nDQpUaG9tYXMg4oCTIFRoZXJlIGlzIGEgc3dpdGNoIGxlYXJuaW5n IHByb2JsZW0gd2hlbiB5b3UgbW92ZQ0KRXJpayBOb3JkbWFyayDigJMganVzdCB3YW50IHRvIHJl bGlhYmx5IHVwZGF0ZSBsZWFybmluZyB0YWJsZXMsIHByb2JhYmx5IGVhc2llc3QgdG8gZG8gd2l0 aCBhIGJyb2FkY2FzdCBwYWNrZXQsIHNvIEFSUCBwYWNrZXQgaXMganVzdCBjb252ZW5pZW50Lg0K QW5kcmV3IOKAkyB5ZXMsIGJ1dCBncmF0dWl0b3VzIGFycCBpcyB0aGUgcmlnaHQgdGhpbmcgdG8g ZG8uDQpEYXZpZCBCbGFjayDigJMgdGhpcyBjb25jZXB0IGhhcyBiZWVuIHVzZWQgc3VjY2Vzc2Z1 bGx5IGluIHRoZSBwYXN0Lg0KVGhvbWFzIOKAkyBkdXBsaWNhdGUgYWRkcmVzcyBkZXRlY3Rpb24g d2FzIGxhdGUgYWRkaXRpb24NCldhcnJlbiDigJMgREhDUCBzZXJ2ZXIgZG9lcyBsZWFzZS4NCkJl bnNvbiDigJMgZG9lcyBjb25nZXN0aW9uIGJyZWFrIGFycCBhbmQgbmQgZGlmZmVyZW50bHk/DQpE aW5vIOKAkyBwcm9ibGVtIGlzIGxvdHMgb2YgZGF0YSBmcm9tIGRhdGEgdG8gY29udHJvbCBwbGFu ZSwgYXJwIGNhbiBnZXQgbG9zdCBoZXJlLiAgUmVhbCBpc3N1ZSBmb3Igcm91dGVyICYgc3dpdGNo IHZlbmRvcnMuDQo9DQpNZWV0aW5nIGNsb3NlZC4NCg0KDQoNCg0KDQoNCg0K --Boundary_(ID_i3k3q9Dp6yEgj6SuzdlJIA) Content-type: text/html; charset=utf-8 Content-transfer-encoding: base64 PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVu dD0iTWljcm9zb2Z0IFdvcmQgMTIgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8q IEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk65a6L5L2TOw0K CXBhbm9zZS0xOjIgMSA2IDAgMyAxIDEgMSAxIDE7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWls eToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2IDMgMiA0O30NCkBmb250 LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5vc2UtMToyIDE1IDUgMiAyIDIgNCAz IDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OiJcQOWui+S9kyI7DQoJcGFub3NlLTE6 MiAxIDYgMCAzIDEgMSAxIDEgMTt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3Jt YWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGluOw0KCW1hcmdpbi1i b3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJp Iiwic2Fucy1zZXJpZiI7fQ0KYTpsaW5rLCBzcGFuLk1zb0h5cGVybGluaw0KCXttc28tc3R5bGUt cHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30N CmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZA0KCXttc28tc3R5bGUtcHJpb3Jp dHk6OTk7DQoJY29sb3I6cHVycGxlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0KcC5N c29MaXN0UGFyYWdyYXBoLCBsaS5Nc29MaXN0UGFyYWdyYXBoLCBkaXYuTXNvTGlzdFBhcmFncmFw aA0KCXttc28tc3R5bGUtcHJpb3JpdHk6MzQ7DQoJbWFyZ2luLXRvcDowaW47DQoJbWFyZ2luLXJp Z2h0OjBpbjsNCgltYXJnaW4tYm90dG9tOjBpbjsNCgltYXJnaW4tbGVmdDouNWluOw0KCW1hcmdp bi1ib3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1l cyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0KcC5Nc29MaXN0UGFyYWdyYXBoQ3hTcEZpcnN0LCBsaS5N c29MaXN0UGFyYWdyYXBoQ3hTcEZpcnN0LCBkaXYuTXNvTGlzdFBhcmFncmFwaEN4U3BGaXJzdA0K CXttc28tc3R5bGUtcHJpb3JpdHk6MzQ7DQoJbXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJ bWFyZ2luLXRvcDowaW47DQoJbWFyZ2luLXJpZ2h0OjBpbjsNCgltYXJnaW4tYm90dG9tOjBpbjsN CgltYXJnaW4tbGVmdDouNWluOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNpemU6 MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0KcC5Nc29M aXN0UGFyYWdyYXBoQ3hTcE1pZGRsZSwgbGkuTXNvTGlzdFBhcmFncmFwaEN4U3BNaWRkbGUsIGRp di5Nc29MaXN0UGFyYWdyYXBoQ3hTcE1pZGRsZQ0KCXttc28tc3R5bGUtcHJpb3JpdHk6MzQ7DQoJ bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJbWFyZ2luLXRvcDowaW47DQoJbWFyZ2luLXJp Z2h0OjBpbjsNCgltYXJnaW4tYm90dG9tOjBpbjsNCgltYXJnaW4tbGVmdDouNWluOw0KCW1hcmdp bi1ib3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1l cyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0KcC5Nc29MaXN0UGFyYWdyYXBoQ3hTcExhc3QsIGxpLk1z b0xpc3RQYXJhZ3JhcGhDeFNwTGFzdCwgZGl2Lk1zb0xpc3RQYXJhZ3JhcGhDeFNwTGFzdA0KCXtt c28tc3R5bGUtcHJpb3JpdHk6MzQ7DQoJbXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJbWFy Z2luLXRvcDowaW47DQoJbWFyZ2luLXJpZ2h0OjBpbjsNCgltYXJnaW4tYm90dG9tOjBpbjsNCglt YXJnaW4tbGVmdDouNWluOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTIu MHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0Kc3Bhbi5FbWFp bFN0eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtY29tcG9zZTsNCglmb250LWZhbWls eToiQ2FsaWJyaSIsInNhbnMtc2VyaWYiOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERl ZmF1bHQNCgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7fQ0KQHBhZ2UgV29yZFNlY3Rpb24x DQoJe3NpemU6OC41aW4gMTEuMGluOw0KCW1hcmdpbjoxLjBpbiAxLjI1aW4gMS4waW4gMS4yNWlu O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLyogTGlzdCBEZWZp bml0aW9ucyAqLw0KQGxpc3QgbDANCgl7bXNvLWxpc3QtaWQ6Mjg2MzUyMTIyOw0KCW1zby1saXN0 LXR5cGU6aHlicmlkOw0KCW1zby1saXN0LXRlbXBsYXRlLWlkczotNjUyNTg3NDk0IDY3Njk4NzAz IDY3Njk4NzEzIDY3Njk4NzE1IDY3Njk4NzAzIDY3Njk4NzEzIDY3Njk4NzE1IDY3Njk4NzAzIDY3 Njk4NzEzIDY3Njk4NzE1O30NCkBsaXN0IGwwOmxldmVsMQ0KCXttc28tbGV2ZWwtdGFiLXN0b3A6 bm9uZTsNCgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7DQoJbWFyZ2luLWxlZnQ6LjI1 aW47DQoJdGV4dC1pbmRlbnQ6LS4yNWluO30NCkBsaXN0IGwwOmxldmVsMg0KCXttc28tbGV2ZWwt dGFiLXN0b3A6MS4waW47DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0Ow0KCXRleHQt aW5kZW50Oi0uMjVpbjt9DQpAbGlzdCBsMDpsZXZlbDMNCgl7bXNvLWxldmVsLXRhYi1zdG9wOjEu NWluOw0KCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsNCgl0ZXh0LWluZGVudDotLjI1 aW47fQ0KQGxpc3QgbDA6bGV2ZWw0DQoJe21zby1sZXZlbC10YWItc3RvcDoyLjBpbjsNCgltc28t bGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7DQoJdGV4dC1pbmRlbnQ6LS4yNWluO30NCkBsaXN0 IGwwOmxldmVsNQ0KCXttc28tbGV2ZWwtdGFiLXN0b3A6Mi41aW47DQoJbXNvLWxldmVsLW51bWJl ci1wb3NpdGlvbjpsZWZ0Ow0KCXRleHQtaW5kZW50Oi0uMjVpbjt9DQpAbGlzdCBsMDpsZXZlbDYN Cgl7bXNvLWxldmVsLXRhYi1zdG9wOjMuMGluOw0KCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246 bGVmdDsNCgl0ZXh0LWluZGVudDotLjI1aW47fQ0KQGxpc3QgbDA6bGV2ZWw3DQoJe21zby1sZXZl bC10YWItc3RvcDozLjVpbjsNCgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7DQoJdGV4 dC1pbmRlbnQ6LS4yNWluO30NCkBsaXN0IGwwOmxldmVsOA0KCXttc28tbGV2ZWwtdGFiLXN0b3A6 NC4waW47DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0Ow0KCXRleHQtaW5kZW50Oi0u MjVpbjt9DQpAbGlzdCBsMDpsZXZlbDkNCgl7bXNvLWxldmVsLXRhYi1zdG9wOjQuNWluOw0KCW1z by1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsNCgl0ZXh0LWluZGVudDotLjI1aW47fQ0Kb2wN Cgl7bWFyZ2luLWJvdHRvbTowaW47fQ0KdWwNCgl7bWFyZ2luLWJvdHRvbTowaW47fQ0KLS0+PC9z dHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVk aXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28g OV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6ZXh0PSJl ZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0tPg0KPC9o ZWFkPg0KPGJvZHkgbGFuZz0iRU4tVVMiIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUiPg0KPGRp diBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPlRoYW5rcyB0byBQ aGlsaXAgRWFyZGxleSBhbmQgRG9uYWxkIEVhc3RsYWtlIGZvciB0YWtpbmcgdGhlIG5vdGVzIGR1 cmluZyA4MTxzdXA+c3Q8L3N1cD4gSUVURiBBUk1EIFdHIHNlc3Npb24uIEhlcmUgYXJlIHRoZSBt aW51dGVzIGNvbWJpbmVkIGZyb20gYm90aCBvZiB0aGVpciBub3Rlcy4NCjxvOnA+PC9vOnA+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+SWYgd2UgbWlzcyBhbnl0aGluZywgcGxlYXNlIGxldCB1 cyBrbm93LiBXZSBuZWVkIHRvIHVwbG9hZCB0aGUgbWludXRlcyBieSBGcmlkYXkgZXZlbmluZy4N CjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5MaW5kYSAmYW1wOyBCZW5zb248bzpwPjwvbzpwPjwv cD4NCjxkaXYgc3R5bGU9Im1zby1lbGVtZW50OnBhcmEtYm9yZGVyLWRpdjtib3JkZXI6bm9uZTti b3JkZXItYm90dG9tOnNvbGlkIHdpbmRvd3RleHQgMS4wcHQ7cGFkZGluZzowaW4gMGluIDEuMHB0 IDBpbiI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0iYm9yZGVyOm5vbmU7cGFkZGluZzow aW4iPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBh bGlnbj0iY2VudGVyIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4wcHQ7dGV4dC1hbGlnbjpjZW50 ZXIiPg0KTWVldGluZyBNaW51dGVzIEFSTUQgSUVURiA4MTxzdXA+c3Q8L3N1cD4gUXVlYmVjIENp dHk8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEwLjBwdCI+PHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4wcHQiPjxzcGFuIHN0eWxlPSJjb2xvcjpi bGFjayI+QWRkcmVzcyBSZXNvbHV0aW9uIGZvciBNYXNzaXZlIG51bWJlcnMgb2YgaG9zdHMgaW4g dGhlIERhdGEgY2VudGVyIChBUk1EKTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEwLjBwdCI+PHNwYW4gc3R5bGU9ImNvbG9y OmJsYWNrIj5Mb2NhdGlvbjoyMDUgQUJDIG1lZXRpbmcgcm9vbSwgUXVlYmVjIENpdHkgQ29udmVu dGlvbiBDZW50cmUsIFF1ZWJlYywgQ2FuYWRhPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTAuMHB0Ij48c3BhbiBzdHlsZT0i Y29sb3I6YmxhY2siPlRpbWU6MjktSnVseS0yMDExLCAwOTAwLTExMzAgLSBGcmlkYXksIE1vcm5p bmcgU2Vzc2lvbiBJPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg c3R5bGU9Im1hcmdpbi1ib3R0b206MTAuMHB0Ij48c3BhbiBsYW5nPSJERSIgc3R5bGU9ImNvbG9y OmJsYWNrIj5DaGFpcnM6QmVuc29uIFNjaGxpZXNzZXIgKDwvc3Bhbj48YSBocmVmPSJtYWlsdG86 YnNjaGxpZXNAY2lzY28uY29tIj48c3BhbiBsYW5nPSJERSI+YnNjaGxpZXNAY2lzY28uY29tPC9z cGFuPjwvYT48c3BhbiBsYW5nPSJERSIgc3R5bGU9ImNvbG9yOmJsYWNrIj4pICZhbXA7IExpbmRh IER1bmJhciAoPC9zcGFuPjxhIGhyZWY9Im1haWx0bzpsaW5kYS5kdW5iYXJAaHVhd2VpLmNvbSI+ PHNwYW4gbGFuZz0iREUiPmxpbmRhLmR1bmJhckBodWF3ZWkuY29tPC9zcGFuPjwvYT48c3BhbiBs YW5nPSJERSIgc3R5bGU9ImNvbG9yOmJsYWNrIj4pPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTAuMHB0Ij48c3BhbiBsYW5n PSJERSIgc3R5bGU9ImNvbG9yOmJsYWNrIj5KYWJiZXI6YXJtZEBqYWJiZXIuaWV0Zi5vcmc8bzpw PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJv dHRvbToxMC4wcHQiPjxhIGhyZWY9IlVSTDpodHRwOi8vdG9vbHMuaWV0Zi5vcmcvd2cvYXJtZC8i PjxzcGFuIGxhbmc9IkRFIj5VUkw6aHR0cDovL3Rvb2xzLmlldGYub3JnL3dnL2FybWQvPC9zcGFu PjwvYT48c3BhbiBsYW5nPSJERSIgc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4wcHQi PjxzcGFuIGxhbmc9IkRFIiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFu JnF1b3Q7LCZxdW90O3NlcmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj48bzpwPiZuYnNwOzwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4w cHQiPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS08bzpw PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTGlzdFBhcmFncmFwaCIgc3R5bGU9Im1z by1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBw dDttYXJnaW4tbGVmdDouMjVpbjt0ZXh0LWluZGVudDotLjI1aW47bXNvLWxpc3Q6bDAgbGV2ZWwx IGxmbzEiPg0KPCFbaWYgIXN1cHBvcnRMaXN0c10+PGI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5 OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjpibGFjayI+ PHNwYW4gc3R5bGU9Im1zby1saXN0Oklnbm9yZSI+MS48c3BhbiBzdHlsZT0iZm9udDo3LjBwdCAm cXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDsiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOw0KPC9z cGFuPjwvc3Bhbj48L3NwYW4+PC9iPjwhW2VuZGlmXT48Yj48c3BhbiBzdHlsZT0iZm9udC1mYW1p bHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOmJsYWNr Ij5NZWV0aW5nJm5ic3A7IEFkbWluaXN0cml2aWEgKGNoYWlycyAtIDA1IG1pbikNCjxvOnA+PC9v OnA+PC9zcGFuPjwvYj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdp bi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdp bi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtBcmlhbCZxdW90 OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj7il6Y8L3NwYW4+PHNwYW4gc3R5 bGU9ImNvbG9yOmJsYWNrIj5XZWxjb21lPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBp bjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0i Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xv cjpibGFjayI+4pemPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+TWFpbGluZyBsaXN0 IGFuZCBVUkw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVv dDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj7il6Y8L3Nw YW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5NaW51dGVzIFNjcmliZTxvOnA+PC9vOnA+PC9z cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6 MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1 aW4iPg0KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3Nh bnMtc2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPuKXpjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6 YmxhY2siPkphYmJlciBTY3JpYmU8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21h cmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJmb250 LWZhbWlseTomcXVvdDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOmJs YWNrIj7il6Y8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5CbHVlIFNoZWV0czxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2lu LWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5TbGlkZXMgbm93IGJlaW5n IHVwbG9hZCB0byBtZWV0aW5nIG1hdGVyaWFscywgYWxyZWFkeSBvbiB3ZWJleDwvc3Bhbj48c3Bh biBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LCZxdW90O3Nl cmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4wcHQiPjxzcGFuIHN0eWxlPSJjb2xv cjpibGFjayI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb0xpc3RQ YXJhZ3JhcGgiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47 bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW47dGV4dC1pbmRlbnQ6LS4yNWlu O21zby1saXN0OmwwIGxldmVsMSBsZm8xIj4NCjwhW2lmICFzdXBwb3J0TGlzdHNdPjxiPjxzcGFu IHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYm cXVvdDs7Y29sb3I6YmxhY2siPjxzcGFuIHN0eWxlPSJtc28tbGlzdDpJZ25vcmUiPjIuPHNwYW4g c3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7Ij4mbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsNCjwvc3Bhbj48L3NwYW4+PC9zcGFuPjwvYj48IVtlbmRpZl0+PGI+PHNw YW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJp ZiZxdW90Oztjb2xvcjpibGFjayI+QVJNRCBDYWxsIGZvciBJbnZlc3RpZ2F0aW9uIChjaGFpcnMg LSAxMCBtaW4pPG86cD48L286cD48L3NwYW4+PC9iPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJv dHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPGEgaHJlZj0iaHR0cDovL3Rvb2xzLmll dGYub3JnL2h0bWwvZHJhZnQtaWV0Zi1hcm1kLWNhbGwtZm9yLWludmVzdGlnYXRpb24tMDAiPmRy YWZ0LWlldGYtYXJtZC1jYWxsLWZvci1pbnZlc3RpZ2F0aW9uLTAwPC9hPg0KPHNwYW4gc3R5bGU9 ImZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OywmcXVvdDtzZXJpZiZxdW90 OyI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1z by1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBw dDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPmFza3MgYSBm ZXcgcXVlc3Rpb25zIHRvIHByb21wdCB0aGUgdGhvdWdodC4gRG9jdW1lbnQgaXMgb25seSB0ZW1w b3JhcnkuDQo8L3NwYW4+V2lsbCBiZSBhbGxvd2VkIHRvIGV4cGlyZTxzcGFuIHN0eWxlPSJjb2xv cjpibGFjayI+LiA8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0 OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4y NWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+TWFueSBwZW9wbGUgcmFpc2VkIGhhbmQg c2hvd2luZyB0aGF0IHRoZXkgcmVhZCB0aGUgZHJhZnQuIFRoZXJlIGlzIG5vIHF1ZXN0aW9uczwv c3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1y aWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4g c3R5bGU9ImNvbG9yOmJsYWNrIj5UaGVyZSB3YXMgYSBOQU5PRyBzZXNzaW9uIHdoZXJlIG9wZXJh dG9ycyB0YWxrZWQgYWJvdXQgdGhlaXIgcHJvYmxlbXM8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9y OmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+RGF2 aWQgQmxhY2s6IGFyZSB5b3UgYWxzbyBpbnRlcmVzdGVkIGluIERDIGRlc2lnbmVy4oCZcyBmZWVk YmFjayBhYm91dCBob3cgYWRkcmVzcyByZXNvbHV0aW9uIHdvcmtzPw0KPC9zcGFuPjxzcGFuIHN0 eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTAuMHB0Ij48c3BhbiBzdHlsZT0iY29sb3I6Ymxh Y2siPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29MaXN0UGFyYWdy YXBoIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdp bi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluO3RleHQtaW5kZW50Oi0uMjVpbjttc28t bGlzdDpsMCBsZXZlbDEgbGZvMSI+DQo8IVtpZiAhc3VwcG9ydExpc3RzXT48Yj48c3BhbiBzdHls ZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7 O2NvbG9yOmJsYWNrIj48c3BhbiBzdHlsZT0ibXNvLWxpc3Q6SWdub3JlIj4zLjxzcGFuIHN0eWxl PSJmb250OjcuMHB0ICZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyI+Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7DQo8L3NwYW4+PC9zcGFuPjwvc3Bhbj48L2I+PCFbZW5kaWZdPjxiPjxzcGFuIHN0 eWxlPSJmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVv dDs7Y29sb3I6YmxhY2siPlByb2JsZW0gU3RhdGVtZW50IGZvciBBUk1EKFRob21hcyBOYXJ0ZW4g LSAxNSBtaW4pPG86cD48L286cD48L3NwYW4+PC9iPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJv dHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPGEgaHJlZj0iaHR0cDovL3Rvb2xzLmll dGYub3JnL2h0bWwvZHJhZnQtbmFydGVuLWFybWQtcHJvYmxlbS1zdGF0ZW1lbnQtMDAiPmRyYWZ0 LW5hcnRlbi1hcm1kLXByb2JsZW0tc3RhdGVtZW50LTAwPC9hPg0KPHNwYW4gc3R5bGU9ImZvbnQt ZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OywmcXVvdDtzZXJpZiZxdW90OyI+PG86 cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJn aW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJn aW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPmRlbGliZXJhdGVseSBu YXJyb3dseSBzY29wZWQ8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3At YWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0 Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+ZmluZCB0byBpZGVudGlmeSByZXBy ZXNlbnRhdGl2ZSBwcm9ibGVtcyBzZWVuIGluIHRoZSBmaWVsZDwvc3Bhbj48c3BhbiBzdHlsZT0i Y29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJv dHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij5IaW1hbnNodSA6Jm5ic3A7IC0gaG93IG1hbnkgREMgb3BlcmF0cHJzIGFyZSBoZXJlLiBBYm91 dCA1Pzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21h cmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjVpbiI+DQo8 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkNvbW1lbnQgcmVnYXJkIHRvIFRob21hcyByZW1hcmtz OiBoaXN0b3J5IG9mIFdHIHZlcnkgc3RyYW5nZS4gU2NvcGUga2VlcHMgZ2V0dGluZyByZWR1Y2Vk LiBBcmUgd2UgZ29pbmcgdG8gZG8gYW55IHdvcmsgaGVyZT88bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJn aW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi41aW4iPg0KPHNw YW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5CZW5zb246IHJlcXVpcmVtZW50cyBpcyB3b3JrLCBubyBw cm90b2NvbHM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi41aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5UaGVu IHdoZXJlPzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbTox MC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5DaGFp cnMg4oCTIHdl4oCZcmUgbm90IGRvaW5nIHByb3RvY29sIGF0IHRoaXMgc3RhZ2Ugb2YgQVJNRC4g TWF5YmUgaW4gdGhlIGZ1dHVyZSBhZnRlciByZS1jaGFydGVyaW5nLg0KPC9zcGFuPjxzcGFuIHN0 eWxlPSJmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssJnF1b3Q7c2VyaWYm cXVvdDs7Y29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFy Z2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9y OmJsYWNrIj5Sb24gQm9uaWNhLCBPcHMgQUQg4oCTIG5vIHByb3RvY29sIGRldmVsb3BtZW50IGlu IE9wcyBhcmVhLg0KPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48 L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFs dDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDou MjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPk5hcnRlbiDigJMgbGV04oCZcyBub3Qg Z2V0IGh1bmcgdXAgaW4gcHJvY2Vzcy4gTGV04oCZcyBpZGVudGlmeSBhbmQgYWdyZWUgcHJvYmxl bXMgL3JlcXVpcmVtZW50cyBhbmQgdGhlbiB3b3JrIG91dCB3aGVyZSB0byBkbyB3b3JrLiBJZiBp dCB0YWtlcyBhIHllYXIgdG8gZG8gcmVxdWlyZW1lbnRzLCB0aGVuIEkgdGhpbmsgd2UgaGF2ZSBi bG93biBpdC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+SmFi YmVyIOKAkyB1cGxvYWQgdGhlIHNsaWRlcyEhPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LWZhbWls eTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssJnF1b3Q7c2VyaWYmcXVvdDs7Y29sb3I6Ymxh Y2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29MaXN0UGFyYWdyYXBoIiBz dHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0 b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluO3RleHQtaW5kZW50Oi0uMjVpbjttc28tbGlzdDps MCBsZXZlbDEgbGZvMSI+DQo8IVtpZiAhc3VwcG9ydExpc3RzXT48Yj48c3BhbiBzdHlsZT0iZm9u dC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9y OmJsYWNrIj48c3BhbiBzdHlsZT0ibXNvLWxpc3Q6SWdub3JlIj40LjxzcGFuIHN0eWxlPSJmb250 OjcuMHB0ICZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7DQo8L3NwYW4+PC9zcGFuPjwvc3Bhbj48L2I+PCFbZW5kaWZdPjxiPjxzcGFuIHN0eWxlPSJm b250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29s b3I6YmxhY2siPk5BTk9HIDUyIE9wZXJhdG9ycyBQZXJzcGVjdGl2ZShTdXNhbiBIYXJlcyAtIDE1 IG1pbik8bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5 bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9t OjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8YSBocmVmPSJodHRwOi8vdG9vbHMuaWV0Zi5v cmcvaHRtbC9kcmFmdC1oYXJlcy1hcm1kLW5hbm9nNTItMDAiPmRyYWZ0LWhhcmVzLWFybWQtbmFu b2c1Mi0wMDwvYT4NCjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9t YW4mcXVvdDssJnF1b3Q7c2VyaWYmcXVvdDsiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdo dDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5 bGU9ImNvbG9yOmJsYWNrIj5UaGUgZ29hbCBmb3IgTkFOT0cgQVJNRCBzZXNzaW9uIGlzIGZvciBv cGVyYXRvcnMgdG8gcGxlYXNlIHRlbGwgdXMgd2hhdCB0aGUgcGFpbiBwb2ludHMgYXJlLjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2lu LWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5TcGVha2VycyBhdCBOQU5P RyBBUk1EIHRyYWNrOiBBZGhvc3QsIEdvb2dsZSwgWWFob28sIE1lcml0LCBhbmQgQVQmYW1wO1Qg UmVzZWFyY2g8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NClBsZWFzZSBnbyB0byA8YSBocmVmPSJodHRwOi8v d3d3Lm5hbm9nLm9yZyI+d3d3Lm5hbm9nLm9yZzwvYT4gPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij4NCiZuYnNwO3RvIHNlZSBzbGlkZXMgdGhhdCB3ZXJlIHByZXNlbnRlZCB0aGVyZSDigJMgbWlk LXJhbmdlIERDIG9wZXJhdG9yLCBHb29nbGUgYnVzaW5lc3MgY2FzZSBldGM8L3NwYW4+PHNwYW4g c3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OywmcXVvdDtzZXJp ZiZxdW90Oztjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z b05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjtt YXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29s b3I6YmxhY2siPldoeSBMYXllciAyIHdpdGhpbiBEYXRhIENlbnRlcnM/PG86cD48L286cD48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDow aW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVp biI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkZvciBtYW55IG1pZC1yYW5nZSBEQyBvcGVy YXRvcnMsIGVzcGVjaWFsbHkgdGhlIG11bHRpLXRlbmFudCBkYXRhIGNlbnRlcnMg4oCTIG11bHRp cGxlIGN1c3RvbWVycyBkb27igJl0IHdhbnQgdG8gY2hhbmdlIHRoZWlyIGFwcHMsIG5lZWRzIExh eWVyIDIgaW4gREMgZm9yIHRoZWlyIGJ1c2luZXNzIG1vZGVsIHRvIHdvcmsuIFNlZSBBUlAgcHJv YmxlbXMuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEw LjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkdvIEdy aWTigJlzIEFwcGxpY2F0aW9uIEwyPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z b05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjtt YXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29s b3I6YmxhY2siPldoYXQgV2UgTGVhcm5lZDogPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9InRleHQtYXV0b3NwYWNlOm5vbmUiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OuWui+S9kyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7TG90cyBvZiBNaWRkbGUgdGllciBEQ3MgaGF2ZSBtdWx0aS10ZW5hbnQ8bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0idGV4dC1hdXRvc3Bh Y2U6bm9uZSI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk65a6L5L2T Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgTmVlZCBMYXllciAyIGZvciBidXNpbmVzcyBtb2Rl bDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJ0ZXh0 LWF1dG9zcGFjZTpub25lIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWls eTrlrovkvZMiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBNaWRkbGUgVGllciBEQ3Mgc2VlIEFS UCBwcm9ibGVtczxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0 eWxlPSJ0ZXh0LWF1dG9zcGFjZTpub25lIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtm b250LWZhbWlseTrlrovkvZMiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBXYW50IHRvIGNvbGxh Ym9yYXRlIHdpdGggb3RoZXIgTWlkZGxlLVRpZXJzPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJp Z2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBz dHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4m cXVvdDssJnF1b3Q7c2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPjxvOnA+Jm5ic3A7PC9vOnA+PC9z cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6 MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1 aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5UaG9tYXMgTmFydGVuIOKAkyBzbGlkZSA1 LCBJIHJ1bGVkIEwyIHNwYW5uaW5nIG11bHRpcGxlIHNpdGVzIGFzIG91dCBvZiBzY29wZS4NCjwv c3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1y aWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4g c3R5bGU9ImNvbG9yOmJsYWNrIj5TdXNhbiDigJMgd2hlbiBJIGFza2VkIEFkSG9zdCDigJMgVGhl eSBoYXZlIDMgc2l0ZXMsIGFsbCBpbiBzaW5nbGUgY2l0eTwvc3Bhbj48c3BhbiBzdHlsZT0iY29s b3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0 eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRv bToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5S b2IgU2hha2lyIOKAkyBtYWluIGlzc3VlIGlzIHNjYWxpbmcuIEkgYnVpbGQgVlBMUyB3aGljaCBj YW4gc3BhbiBnbG9iZS48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs IiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1i b3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFj ayI+QW5zd2VyOiBWUExTIHNwYW5zIGdsb2JlLiBIb3dldmVyLCB0aGUgbnVtYmVyIG9mIG5vZGVz IGludGVyY29ubmVjdGVkIGJ5IFZQTFMgaXMgbm90IGFzIG1hbnkgYXMgdGhlIG51bWJlciBob3N0 cyBpbiBEYXRhIENlbnRlci4mbmJzcDsgVGhlcmVmb3JlIHRoZSBhZGRyZXNzIHNjYWxpbmcgaW4g RGF0YSBDZW50ZXJzIGhhcyBkaWZmZXJlbnQgZGltZW5zaW9uIHRoYW4gVlBMUy4NCjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRv cC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxl ZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5YIChVSyk6IEkgZG8gVlBMU3Mg dGhhdCBhcmUgdmVyeSBsYXJnZS4gTGF0ZW5jeSBjYW4gYmUgYSBwcm9ibGVtIGJ1dCB0aGlzIGdy b3VwIGlzIGFib3V0IGFkZHJlc3MgcmVzb2x1dGlvbi4gSXQgaXMgc2NhbGluZywgcmlnaHQ/IFdl IHNob3VsZCBkb2N1bWVudCByZWFsIHRvcG9sb2dpZXMgYW5kIHNheSB3aGF0IHRoZSBwcm9ibGVt cyBhcmUgd2l0aCB0aGF0IHRvcG9sb2d5LiBJIGFtIHdpbGxpbmcgdG8NCiBoZWxwIHdyaXRlIHRo aXMuIEVhc2llciB0byBzaGFyZSBwcml2YXRlbHkuIE15IHByb2JsZW0gd2l0aCBnZXR0aW5nIHBl b3BsZSB0byBpbnRlcmFjdCB3aXRoIHRoZSBJRVRGIGlzIHRoYXQgcGVvcGxlJm5ic3A7IHRoaW5r IGl0IGlzIGlycmVsZXZhbnQuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJn aW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6 YmxhY2siPkJlbnNvbjogTGF0ZW5jeSBtYXkgYmUgYSBwcm9ibGVtIG5vdCBpbiBzY29wZSwgYnV0 IHdoYXQgaWYgbGF0ZW5jeSBhZmZlY3RzIGFkZHJlc3MgcmVzb2x1dGlvbi48bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0 OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4y NWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+TmluZyBTbzogUGVvcGxlIHRyeSB0byBk byBvbmUgc2l6ZSBmaXRzIGFsbCBtb2RlbHMuIFdoZW4gd2UgdGFsayB0byBjdXN0b21lcnMsIHRo ZXkgaGF2ZSBhbHJlYWR5IGJlZW4gcnVubmluZyB0aGVpciBvd24gZGF0YSBjZW50ZXJzIGFyb3Vu ZCB0aGUgd29ybGQuIFRoZXkgd2FudCB0byBvZmYtbG9hZCBzb21lIHN0dWZmIG9uIHVzLiBGb3Ig YW4gaW5kZWZpbml0ZSB0aW1lIHRoZXkgd2lsbCB3YW50IHRvDQogY29udGludWUgdG8gb3BlcmF0 ZSB0aGVpciBvd24gZGF0YSBjZW50ZXJzIGFsb25nIHdpdGggdXNpbmcgb3Vycy48bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0idGV4dC1hdXRvc3BhY2U6 bm9uZSI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk65a6L5L2TIj48 bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0i bXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAu MHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2Zv bnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OywmcXVvdDtzZXJpZiZxdW90Oztj b2xvcjpibGFjayI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJn aW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6 YmxhY2siPldhcnJlbiDigJMgY2FuIHlvdSBzdXBwbHkgcGljIHdpdGggeW91ciBwcm9ibGVtPyA8 L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4t cmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFu IHN0eWxlPSJjb2xvcjpibGFjayI+Um9iIChVSyBvcGVyYXRvcikg4oCTIFdlIHNob3VsZCBoYXZl IHRvcG9sb2d5IHBpY3MgdG8gaWRlbnRpZnkgYWN0dWFsIGlzc3Vlcy4gTmVlZCB0byBnZW5lcmFs aXplIHNvIG5vdCBjb21tZXJjaWFsIGluZm8uIEkgd2lsbCBoZWxwLg0KPC9zcGFuPjxzcGFuIHN0 eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJn aW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6 YmxhY2siPlJvYiDigJMgSSB0YWxrIHdpdGggcGVvcGxlIHdobyB0aGluayBpZXRmIGFyZSBpcnJl bGV2YW50Lg0KPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDow aW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVp biI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPk5pbmcmbmJzcDsgU28g4oCTIHdlIHdhbnQg dG8gcnVuIG11bHRpLXRlbmFudCBEQy4gQ3VzdG9tZXJzIHdhbnQgdG8gb2ZmbG9hZCB0aGVpciBp bnRlcm5hbCBEQ3MgdG8gdXMsIGJ1dCB3aWxsIGtlZXAgcnVubmluZyB0aGVpciBvd24gREMg4oCT IHNvIHdpbGwgcnVuIG92ZXIgYm90aC4gU28gdGhlIGFyY2hpdGVjdHVyZSBkZXNjcmliZWQgaW4g c2xpZGUgNSBtYWtlcyBzZW5zZSB0byB1cy4NCjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6Ymxh Y2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJt c28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4w cHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5BbmRyZXcg TWNHcmVnb3Ig4oCTIG5lZWQgc3dpdGNoIHZlbmRvcnMgYXMgd2VsbCAoaGFuZHMg4oCTIHRoZXkg YXJlIGhlcmUpLiBUaGUgc2NlbmFyaW8gaXMgZmFpcmx5IGZyZXF1ZW50Lg0KPC9zcGFuPjxzcGFu IHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z b05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjtt YXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29s b3I6YmxhY2siPkNhdGh5IEVybnNvbiDigJMgbmVlZCBwcm92aWRlcnMgdG8gY29tZSB0byBOQU5P RyB0byBnaXZlIGlucHV0LiZuYnNwOw0KPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+ PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1t YXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDtt YXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkVyaWMgS2xpbmU6 IOKAkyB3aGVyZSBzbGlkZXMgc2F5IEFSUCBkbyB0aGV5IG1lYW4gYWRkcmVzcyByZXNvbHV0aW9u PyBBbnN3ZXI6Jm5ic3A7IFllczwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2lu LWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5SYW5keSBCdXNoOiBNYW55 IG9wZXJhdG9ycyB0aGluayB0aGV5IGdhdmUgY29uc2lkZXJhYmxlIGlucHV0IGF0IE5BTk9HLiBJ IHRoaW5rIHdlIGhhdmUgYW4gZWFyIHJlc29sdXRpb24gcHJvYmxlbS48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBp bjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWlu Ij4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+QmVuc29uOiBZZXMsIHdlIGhhZCBhIGdvb2Qg dHVybm91dC4mbmJzcDsgT3BlcmF0b3JzIGNhbiB0YWxrIHRvIHVzIHByaXZhdGVseS4gV2UgYWxz byB3YW50IGRpc2N1c3Npb24gb24gdGhlIGxpc3QgYWJvdXQgdGhpcy4gTmVlZCB0byBlbmQgdXAg d2l0aCB0ZXh0LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0 eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRv bToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48 bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0i bXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAu MHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+TmluZyBT bzogUmVzcG9uZGluZyB0byB0aGUgZG9uPC9zcGFuPjxzcGFuIGxhbmc9IlpILUNOIiBzdHlsZT0i Zm9udC1mYW1pbHk65a6L5L2TO2NvbG9yOmJsYWNrIj7piKXmqps8L3NwYW4+PHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj4gZG8gdGhhdCBjb21tZW50OiBWZW5kb3Igc2F5cyBkb248L3NwYW4+PHNw YW4gbGFuZz0iWkgtQ04iIHN0eWxlPSJmb250LWZhbWlseTrlrovkvZM7Y29sb3I6YmxhY2siPumI peaqmzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPg0KIGRvIHRoYXQuIFdlIHNheSBk b248L3NwYW4+PHNwYW4gbGFuZz0iWkgtQ04iIHN0eWxlPSJmb250LWZhbWlseTrlrovkvZM7Y29s b3I6YmxhY2siPumIpeaqmzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiBkbyB0aGF0 LiBDdXN0b21lciBzYXlzIHRoYXQ8L3NwYW4+PHNwYW4gbGFuZz0iWkgtQ04iIHN0eWxlPSJmb250 LWZhbWlseTrlrovkvZM7Y29sb3I6YmxhY2siPumIpeaqmjwvc3Bhbj48c3BhbiBzdHlsZT0iY29s b3I6YmxhY2siPiB3aGF0IEkgd2FudCBhbmQNCiB3aGF0IHlvdXIgY29tcGV0aXRvcnMgYXJlIHBy b3ZpZGluZy4gU28gd2UgY29tZSBiYWNrIHRvIHB1c2ggdmVuZG9yIHRvIGZpeCBpdC4gU3VjaCBh cyBCR1Agcm91dGUgcHJvYmxlbS48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21h cmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xv cjpibGFjayI+QmVuc29uOiBCb3RoIHZpZXdzIHNlZW0gcmVhc29uYWJsZS4gRGlmZmVyZW50IHNj ZW5hcmlvcy48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+V2Fy cmVuOiDigJMgZG9jdG9yIHNheXMgZG9u4oCZdCBkbyBpdCBpZiBpdCBodXJ0cy4gQmlnIExheWVy IDIgaGFzIHByb2JsZW1zLiBUaGVyZSBpcyBhIHJlYXNvbiB0aGUgSW50ZXJuZXQgaXMgbm90IGEg YmlnIExheWVyIDIgbmV0d29yay4gTWF5YmUgYSBCQ1Agc2F5aW5nIGp1c3QgZG9u4oCZdCBkbyB0 aGF0IHdpbGwgYmUgdGhlIHJlc3VsdC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGlu O21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJj b2xvcjpibGFjayI+Vk0gbW9iaWxpdHkgdGhyb3VnaCBvdmVybGF5cywgc3VibmV0IGV0YyBhcyBz b2x1dGlvbnM8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBS b21hbiZxdW90OywmcXVvdDtzZXJpZiZxdW90Oztjb2xvcjpibGFjayI+PG86cD48L286cD48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDow aW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVp biI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPk5pbmcgU086IOKAkyBjdXN0b21lcnMgY2Fu IGluc2lzdCB3ZSBkbyB0aGlzLiA8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpw PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdp bi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdp bi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+UmFuZHkg4oCTIExldOKA mXMgZ2V0IGRvd24gdG8gdGVjaG5pY2FsIGRpc2N1c3Npb24gb3IgZWxzZSBnbyBmb3IgY29mZmVl Ljwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEwLjBwdCI+PHNwYW4g c3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFz cz0iTXNvTGlzdFBhcmFncmFwaEN4U3BGaXJzdCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDow aW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVp bjt0ZXh0LWluZGVudDotLjI1aW47bXNvLWxpc3Q6bDAgbGV2ZWwxIGxmbzEiPg0KPCFbaWYgIXN1 cHBvcnRMaXN0c10+PGI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVv dDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjpibGFjayI+PHNwYW4gc3R5bGU9Im1zby1s aXN0Oklnbm9yZSI+NS48c3BhbiBzdHlsZT0iZm9udDo3LjBwdCAmcXVvdDtUaW1lcyBOZXcgUm9t YW4mcXVvdDsiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOw0KPC9zcGFuPjwvc3Bhbj48L3NwYW4+ PC9iPjwhW2VuZGlmXT48Yj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZx dW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj5BZGRyZXNzIFJlc29sdXRp b24gU3RhdGlzdGljczogSmltIFJlZXMsIE1hbmlzaCBLYXJpciDigJMgTWVyaXQgTmV0d29yazxv OnA+PC9vOnA+PC9zcGFuPjwvYj48L3A+DQo8cCBjbGFzcz0iTXNvTGlzdFBhcmFncmFwaEN4U3BM YXN0IiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdp bi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxhIGhyZWY9Imh0dHA6Ly90b29s cy5pZXRmLm9yZy9odG1sL2RyYWZ0LWthcmlyLWFybWQtc3RhdGlzdGljcy0wMSI+ZHJhZnQta2Fy aXItYXJtZC1zdGF0aXN0aWNzLTAxPC9hPjxiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVv dDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPjxvOnA+ PC9vOnA+PC9zcGFuPjwvYj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1h cmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21h cmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+NTAwIGhvc3RzLCBj b25maWd1cmFibGUgaW50ZXJjb25uZWN0IHJhdGUuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJp Z2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBz dHlsZT0iY29sb3I6YmxhY2siPkRpbm8g4oCTIGRpZCB5b3Ugc3R1ZHkgdW5rbm93biB1bmljYXN0 IChicmlkZ2UgZmxvb2RzKT8gV2UgaGF2ZSBzcGlrZXMgZHVlIHRvIHRoYXQgYnV0IGRpZG7igJl0 IGFuYWx5emUgbXVjaC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs IiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1i b3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFj ayI+QW5vb3AgR2hhd2FuaSDigJMgaXMgaXQgc3Bhbm5pbmcgdHJlZT8gWWVzLiA8L3NwYW4+PHNw YW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OywmcXVvdDtz ZXJpZiZxdW90Oztjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBp bjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0i Y29sb3I6YmxhY2siPkJlbnNvbiDigJMgVk0gZG9lcyB0cmljayBzbyA8L3NwYW4+PHNwYW4gc3R5 bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9y bWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdp bi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpi bGFjayI+RGlubyDigJMgdXBzdHJlYW0gbGlua3MgYXQgdG9wIGFyZSBMMz8gWWVzIDwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDow aW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj5DUFUgbG9hZCBzcGlrZXMgdG8gMzAlPC9zcGFuPjxzcGFuIHN0eWxlPSJj b2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg c3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90 dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2si PkVyaWMgS2xpbmUg4oCTIHdoeSBpcyBjcHUgbG9hZCBoaWdoZXIgZm9yIGFycCB0aGFuIE5EPyA8 L3NwYW4+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdp bi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNw YW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5NYW5pc2gg4oCTIGFycCBicm9hZGNhc3RzIGdvIGV2ZXJ5 d2hlcmUsIE5EIHVzZXMgbXVsdGljYXN0Lg0KPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFj ayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1z by1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBw dDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkVyaWMgVnlu a2Ug4oCTIFdoYXQgaXMgdGhlIHNpemU/IGFuc3dlciAvMjIuPG86cD48L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFy Z2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkFub29wOiZuYnNwOyBsb3dlciBsb2FkIGR1ZSB0byBu b3QgZ2V0dGluZyBORCBtZXNzYWdlcyDigJMgc3BlY3VsYXRpb24gb24gdGhlIGFycCB2cyBuZCBp c3N1ZS4NCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJv bWFuJnF1b3Q7LCZxdW90O3NlcmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBp bjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWlu Ij4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+RGlubzombmJzcDsgd2FzIHBhY2tldHMgdG8g c3dpdGNoIHRoZSBzYW1lIGZvciB2NCAmYW1wOyB2Nj88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4t cmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFu IHN0eWxlPSJjb2xvcjpibGFjayI+OTAwLTEwMDAgdjYsIDEwMDAtMTQwMCB2NDxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1h bHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6 LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5BUlAvTkQgdHJhZmZpYyBzY2FsZWQg bGluZWFybHkgd2l0aCBudW1iZXIgb2YgaG9zdHMuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJp Z2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBz dHlsZT0iY29sb3I6YmxhY2siPkRpbm8g4oCTIHdoYXTigJlzIHRoZSBibGFkZT8gRGVsbCwgMiBj b3JlcyBvbiBlYWNoIGJsYWRlPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtU aW1lcyBOZXcgUm9tYW4mcXVvdDssJnF1b3Q7c2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2lu LWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5UaG9tYXMgTmFydGVuIOKA kyB3aGF0IHBhcGVycyBhcmUgYXZhaWxhYmxlLCB0aGUgbmFub2cgY2hhcnRzIGFyZSBub3QgcmVh ZGFibGUuIFRoZXNlIHNsaWRlcyBhcmUgYSBiaXQgYmV0dGVyLCB3b3VsZCBsaWtlIHRvIG1ha2Ug ZGF0YSBhdmFpbGFibGUuDQo8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10 b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1s ZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+VGhvbWFzOiBJbnRlcmVzdGlu ZyB3b3JrLiBXaGF0IHBhcGVycyBvciBkb2NzIGFyZSBhdmFpbGFibGU/IENoYXJ0IHJlc29sdXRp b24gbm90IGdvb2Qgb24gTkFOT0cgZG9jdW1lbnRzLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1y aWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4g c3R5bGU9ImNvbG9yOmJsYWNrIj5KaW06IHNsaWRlcyBiZXR0ZXI8bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjtt YXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4N CjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+SGltYW5zaHUg4oCTIHdoeSBkb2VzIFZNIG1pZ3Jh dGlvbiBoYXZlIG5vIGltcGFjdD8gVk13YXJlIHBsYXlzIHRyaWNrcyB0byBtaW5pbWlzZSBzZXJ2 aWNlIGRpc3J1cHRpb24gZHVyaW5nIG1pZ3JhdGlvbi4NCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdp bi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNw YW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5EYXZpZCBCbGFjayDigJMgd2hpY2ggVk13YXJlIHN3aXRj aD8gPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4m cXVvdDssJnF1b3Q7c2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21h cmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0K PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5NYW5pc2gg4oCTIFdlIGJ5cGFzc2VkIHN3aXRjaDwv c3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1y aWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4g c3R5bGU9ImNvbG9yOmJsYWNrIj5XaWxsaWFtIEdsb2J1cyDigJMgZGlkIHlvdSBsb29rIGF0IHN3 aXRjaCBvciBibGFkZXM/IE1vbml0b3JpbmcgdHJhZmZpYyBvdXRzaWRlIGNhYmluZXQuPC9zcGFu PjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0 OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHls ZT0iY29sb3I6YmxhY2siPldlcyBHZW9yZ2Ug4oCTIHNwZW5kaW5nIGEgbG9uZyB0aW1lIGNvbmpl Y3R1cmluZyBvbiBWTXdhcmUsIGNhbiB3ZSBhc2sgdGhlbSBkaXJlY3RseS48L3NwYW4+PHNwYW4g c3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21h cmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xv cjpibGFjayI+V2FycmVuIOKAkyBleHRyYXBvbGF0aW5nIHRvIDIwayBob3N0cyBzdWdnZXN0cyBv dmVyIDEwMCUgbG9hZC4gQWN0dWFsbHkgYXJwIGxvYWQgaXMgaW4gdGhlIG5vaXNlLjwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDow aW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj5NYW5pc2gg4oCTIG5vdCBudW1iZXIgb2YgaG9zdHMgYnV0IHRyYWZmaWMg cGF0dGVybi48L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBp bjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWlu Ij4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+UmFuZHkgQnVzaCDigJMgb25lIHRyYWlsZXIg aXMgb24gb3JkZXIgb2YgMjBrIGhvc3RzLiBZb3XigJlkIGxpa2UgdHJhaWxlciB0byBiZSBMMi4g VGhpcyBpcyBpZGVhIG9mIHRoZSBzY2FsZS48L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNv LW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0 O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+VGhvbWFzIE5h cnRlbiDigJMgd291bGQgYmUgdXNlZnVsIHRvIGRvY3VtZW50IGRpZmZlcmVudCBpbXBsZW1lbnRh dGlvbnMgb2YgQVJQLCB0aGV5IGNhbiBoYXZlIHZlcnkgZGlmZmVyZW50IGFsZ29yaXRobXMuDQo8 L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4t cmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFu IHN0eWxlPSJjb2xvcjpibGFjayI+Um9iIFNoYWtpciDigJMgY2FyZSB3aGVyZSBwcm9ibGVtIGlz IGluIGltcGxlbWVudGF0aW9uIG9mIHByb3RvY29sIG9yIHByb3RvY29sIHNwYWNlLjwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDow aW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj5GbG9yaW4gYmFsdXMg4oCTIG1heSBiZSBpbnRlcmVzdGluZyB0byBnZXQg ZGF0YSBmcm9tIG90aGVyIGVudmlyb25lbWVudCwgZWcgVlBMUy4gU2V2ZXJhbCAxMDAwIGhvc3Rz IHN1cHBvcnRlZCBvbiBvbmUgYm94LCBhcnAgbG9hZCBpcyAzLTYlIG9mIGNwdS4NCjwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDow aW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj5XYXJyZW4g4oCTIG15IHF1aWNrIHJvdWdoIGVzdGltYXRlLCAyMGsgaG9z dHMgaW4gbWVzaCBpcyA2NjAgcmVzb2x1dGlvbnMgcGVyIHNlYyDigJMgc2hvdWxkIGJlIHZlcnkg ZWFzeSB0byBkby48L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4w cHQiPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb0xpc3RQYXJhZ3JhcGhDeFNwRmlyc3QiIHN0eWxlPSJtc28tbWFyZ2lu LXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2lu LWxlZnQ6LjI1aW47dGV4dC1pbmRlbnQ6LS4yNWluO21zby1saXN0OmwwIGxldmVsMSBsZm8xIj4N CjwhW2lmICFzdXBwb3J0TGlzdHNdPjxiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtD YWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPjxzcGFuIHN0 eWxlPSJtc28tbGlzdDpJZ25vcmUiPjYuPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGlt ZXMgTmV3IFJvbWFuJnF1b3Q7Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsNCjwvc3Bhbj48L3Nw YW4+PC9zcGFuPjwvYj48IVtlbmRpZl0+PGI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90 O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjpibGFjayI+QWRkcmVz cyBSZXNvbHV0aW9uIGlzc3VlcyBpbmR1Y2VkIGJ5IFZQTiBvcmllbnRlZCBkYXRhIGNlbnRyZSBz ZXJ2aWNlcyAoTmluZyBTbzogVmVyaXpvbik8bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KPHAg Y2xhc3M9Ik1zb0xpc3RQYXJhZ3JhcGhDeFNwTGFzdCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFs dDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDou MjVpbiI+DQo8YSBocmVmPSJodHRwOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC1zby1hcm1k LXZkY3MtYXItMDAiPmRyYWZ0LXNvLWFybWQtdmRjcy1hci0wMDwvYT48Yj48c3BhbiBzdHlsZT0i Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2Nv bG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4t Ym90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6Ymxh Y2siPk5ldHdvcmsgY29udHJvbGxpbmcgREMgJmFtcDsgbW9iaWxpdHkgb2YgYXBwcyBldGMuIEFu IGlkZWEgYXQgdGhpcyBzdGFnZS4NCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6JnF1 b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LCZxdW90O3NlcmlmJnF1b3Q7O2NvbG9yOmJsYWNrIj48 bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1h cmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21h cmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+RGlubyDigJMgaW4g dGVybXMgb2Ygc2NhbGFiaWxpdHkgc2hvdWxkbuKAmXQgbWF0dGVyLCBpdOKAmXMgbWFuYWdlbWVu dC4gPzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21h cmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0K PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5OaW5nIOKAkyBlZGdlIHJvdXRlcnMgZ2V0dGluZyBo aXQgYnkgY29tcGxleGl0eTwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRv cC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxl ZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5UaG9tYXMgTiDigJMgc2xpZGUg OCDigJMgaXMgYWRkcmVzcyBjb25mbGljdCB0aGUgcGFpbiBwb2ludD8gRGlmZmVyZW50IGlzc3Vl IHRvIGFkZHJlc3MgcmVzb2x1dGlvbi4NCjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2si PjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28t bWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7 bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5Bbm9vcCBbQnJv Y2FkZV0g4oCTIHNvbHZlIGNsYXNoZXMgd2l0aCBpc29sYXRpb24gdGVjaG5vbG9neTwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDow aW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj5BbmRyZXcgTWNHcmVnb3Ig4oCTIG1pdGlnaWF0aW9uIHRlY2huaXF1ZXMg Y2FuIG1ha2UgYWRkcmVzcyByZXNvbHV0aW9uIHdvcnNlLy4NCjwvc3Bhbj48c3BhbiBzdHlsZT0i Y29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJv dHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij5NYW5pc2gg4oCTIGkgdGhpbmsgeW91ciBwcm9ibGVtIGJlbG9uZ3MgdG8gdGhlIEFSTUQgd2c8 L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4t cmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFu IHN0eWxlPSJjb2xvcjpibGFjayI+TWFuY2hldyBDaWVuYSAtICYjNDM7MTwvc3Bhbj48c3BhbiBz dHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFy Z2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9y OmJsYWNrIj5Sb2IgU2hha2lyIFVLIOKAkyBuZWVkIHRvIGxpbWl0IGZhaWx1cmVzIGV0Yywgc28g d29u4oCZdCBjb25uZWN0IHRvIHNhbWUgYm94Ljwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6Ymxh Y2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJt c28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4w cHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5OaW5nIFNv IOKAkyBhbHJlYWR5IHRoZXJlIGFyZSBzZXZlcmFsIGJveGVzLjwvc3Bhbj48c3BhbiBzdHlsZT0i Y29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJv dHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij5NYW5pc2gg4oCTIHdhbnQgdG8gbWF4aW1pc2Ugc2hhcmluZy4gPC9zcGFuPjxzcGFuIHN0eWxl PSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4t Ym90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6Ymxh Y2siPkZsb3JlbnQg4oCTIGtleSBpcyB0byBsaW1pdCBudW1iZXIgb2YgaG9zdHMgcGVyIGJyb2Fk Y2FzdCBkb21haW47IHRoaXMgbWlnaHQgYmUgNSBob3N0cyBmb3IgdnBscy4NCjwvc3Bhbj48c3Bh biBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47 bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNv bG9yOmJsYWNrIj5EYXZpZCBEeXNvbiDigJMgd2hhdOKAmXMgdGhlIGxpbWl0Pzwvc3Bhbj48c3Bh biBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47 bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNv bG9yOmJsYWNrIj5XYXJyZW4g4oCTIGFuZCB0aGVzZSA1IHRoaW5ncyBjYW4gYmUgcm91dGVycy4g PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0K PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2lu LXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3Bh biBzdHlsZT0iY29sb3I6YmxhY2siPldhcnJlbiDigJMgY2FuIGRvIHN0dWZmIG9uIEh5cGVydmlz b3IuIEhvdyBtdWNoIG9mIHRoZSBicm9hZGNhc3RzIGlzIEFSUD8g4oCTIGJyb2FkY2FzdCBtaXRp Z2F0aW9uIG1heWJlIHRoZSBrZXkgYXBwcm9hY2guDQo8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9y OmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206 MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+V2Vz IOKAkyBmb2N1cyBvbiBtaWQgc2l6ZSBEQ3MuIFdyaXRlIHVwIGhvdyBkaWQgZXhwZXJpbWVudHMg KG9yIGNvZGUpPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb0xpc3RQYXJhZ3JhcGgiIHN0eWxlPSJtc28tbWFyZ2luLXRv cC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxl ZnQ6LjI1aW47dGV4dC1pbmRlbnQ6LS4yNWluO21zby1saXN0OmwwIGxldmVsMSBsZm8xIj4NCjwh W2lmICFzdXBwb3J0TGlzdHNdPjxiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTomcXVvdDtDYWxp YnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6YmxhY2siPjxzcGFuIHN0eWxl PSJtc28tbGlzdDpJZ25vcmUiPjcuPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMg TmV3IFJvbWFuJnF1b3Q7Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsNCjwvc3Bhbj48L3NwYW4+ PC9zcGFuPjwvYj48IVtlbmRpZl0+PGI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O0Nh bGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjpibGFjayI+VnVsbmVyYWJp bGl0eSBJc3N1ZXMgaW4gTWlncmF0aW9u4oCTIFlpemhvdSBMaTxvOnA+PC9vOnA+PC9zcGFuPjwv Yj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1 dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG87bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPGEgaHJl Zj0iaHR0cDovL3Rvb2xzLmlldGYub3JnL2h0bWwvZHJhZnQtbGl5ei1hcm1kLXZtLW1pZ3JhdGlv bi1wcy0wMSI+ZHJhZnQtbGl5ei1hcm1kLXZtLW1pZ3JhdGlvbi1wcy0wMTwvYT4NCjxzcGFuIHN0 eWxlPSJmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssJnF1b3Q7c2VyaWYm cXVvdDsiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxl PSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbTox MC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5OYXJ0 ZW4g4oCTIHdoZW4gVk0xIG1vdmVzIGhhcyBzYW1lIE1BQyBhZGRyZXNzPyBZZXMuIDwvc3Bhbj4N CjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0 OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHls ZT0iY29sb3I6YmxhY2siPldhcnJlbiDigJMgYXJwIG1hcHMgZnJvbSBpcCB0byBtYWMgYWRkcmVz cy4gSUYxICZhbXA7IElGMiBhcmUgc2FtZSBzdWJuZXQ/IFllcy4mbmJzcDsgTDIgbmVlZHMgdG8g a25vdyBNQUMgYWRkcmVzcyBoYXMgbW92ZWQsIHRoaXMgaXMgbm90IGFycCBidXQgYnJpZGdlIGxl YXJuaW5nPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47 bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+ DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPlRob21hcyDigJMgVGhlcmUgaXMgYSBzd2l0Y2gg bGVhcm5pbmcgcHJvYmxlbSB3aGVuIHlvdSBtb3ZlPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpi bGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjEw LjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPkVyaWsg Tm9yZG1hcmsg4oCTIGp1c3Qgd2FudCB0byByZWxpYWJseSB1cGRhdGUgbGVhcm5pbmcgdGFibGVz LCBwcm9iYWJseSBlYXNpZXN0IHRvIGRvIHdpdGggYSBicm9hZGNhc3QgcGFja2V0LCBzbyBBUlAg cGFja2V0IGlzIGp1c3QgY29udmVuaWVudC48L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNv LW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0 O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+QW5kcmV3IOKA kyB5ZXMsIGJ1dCBncmF0dWl0b3VzIGFycCBpcyB0aGUgcmlnaHQgdGhpbmcgdG8gZG8uPC9zcGFu PjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowaW47bWFyZ2luLXJpZ2h0 OjBpbjttYXJnaW4tYm90dG9tOjEwLjBwdDttYXJnaW4tbGVmdDouMjVpbiI+DQo8c3BhbiBzdHls ZT0iY29sb3I6YmxhY2siPkRhdmlkIEJsYWNrIOKAkyB0aGlzIGNvbmNlcHQgaGFzIGJlZW4gdXNl ZCBzdWNjZXNzZnVsbHkgaW4gdGhlIHBhc3QuDQo8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJs YWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0i bXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAu MHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+VGhvbWFz IOKAkyBkdXBsaWNhdGUgYWRkcmVzcyBkZXRlY3Rpb24gd2FzIGxhdGUgYWRkaXRpb248L3NwYW4+ PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4tcmlnaHQ6 MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFuIHN0eWxl PSJjb2xvcjpibGFjayI+V2FycmVuIOKAkyBESENQIHNlcnZlciBkb2VzIGxlYXNlLjwvc3Bhbj48 c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6MGluO21hcmdpbi1yaWdodDow aW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6LjI1aW4iPg0KPHNwYW4gc3R5bGU9 ImNvbG9yOmJsYWNrIj5CZW5zb24g4oCTIGRvZXMgY29uZ2VzdGlvbiBicmVhayBhcnAgYW5kIG5k IGRpZmZlcmVudGx5Pzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1h bHQ6MGluO21hcmdpbi1yaWdodDowaW47bWFyZ2luLWJvdHRvbToxMC4wcHQ7bWFyZ2luLWxlZnQ6 LjI1aW4iPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5EaW5vIOKAkyBwcm9ibGVtIGlzIGxv dHMgb2YgZGF0YSBmcm9tIGRhdGEgdG8gY29udHJvbCBwbGFuZSwgYXJwIGNhbiBnZXQgbG9zdCBo ZXJlLiZuYnNwOyBSZWFsIGlzc3VlIGZvciByb3V0ZXIgJmFtcDsgc3dpdGNoIHZlbmRvcnMuDQo8 L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBpbjttYXJnaW4t cmlnaHQ6MGluO21hcmdpbi1ib3R0b206MTAuMHB0O21hcmdpbi1sZWZ0Oi4yNWluIj4NCjxzcGFu IHN0eWxlPSJjb2xvcjpibGFjayI+PTwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPjxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4t Ym90dG9tOjEwLjBwdCI+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj5NZWV0aW5nIGNsb3NlZC48 L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMC4wcHQiPjxzcGFuIHN0 eWxlPSJjb2xvcjpibGFjayI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTAuMHB0Ij48c3BhbiBzdHlsZT0iY29s b3I6YmxhY2siPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEwLjBwdCI+PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNr Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibWFyZ2luLWJvdHRvbToxMC4wcHQiPjxzcGFuIGxhbmc9IkRFIiBzdHlsZT0iY29sb3I6Ymxh Y2siPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0 eWxlPSJtYXJnaW4tYm90dG9tOjEwLjBwdCI+PHNwYW4gbGFuZz0iREUiIHN0eWxlPSJmb250LWZh bWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssJnF1b3Q7c2VyaWYmcXVvdDs7Y29sb3I6 YmxhY2siPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkRFIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1s Pg0K --Boundary_(ID_i3k3q9Dp6yEgj6SuzdlJIA)-- From xuxiaohu@huawei.com Sat Aug 27 03:01:27 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8CB4221F86C1; Sat, 27 Aug 2011 03:01:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.844 X-Spam-Level: X-Spam-Status: No, score=0.844 tagged_above=-999 required=5 tests=[AWL=-2.522, BAYES_00=-2.599, CHARSET_FARAWAY_HEADER=3.2, HTML_MESSAGE=0.001, J_CHICKENPOX_35=0.6, MIME_8BIT_HEADER=0.3, MIME_BASE64_TEXT=1.753, MIME_CHARSET_FARAWAY=2.45, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_MED=-4, SARE_MILLIONSOF=0.315, SARE_SUB_ENC_GB2312=1.345] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9qoddTMu1wW1; Sat, 27 Aug 2011 03:01:26 -0700 (PDT) Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [119.145.14.66]) by ietfa.amsl.com (Postfix) with ESMTP id 4CEA821F86B6; Sat, 27 Aug 2011 03:01:26 -0700 (PDT) Received: from huawei.com (szxga03-in [172.24.2.9]) by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQK00H94ZVF6U@szxga03-in.huawei.com>; Sat, 27 Aug 2011 18:02:04 +0800 (CST) Received: from szxrg02-dlp.huawei.com ([172.24.2.119]) by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQK00B43ZUYFU@szxga03-in.huawei.com>; Sat, 27 Aug 2011 18:02:03 +0800 (CST) Received: from 172.24.2.119 (EHLO szxeml201-edg.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.1.9-GA FastPath queued) with ESMTP id ADL24341; Sat, 27 Aug 2011 18:02:03 +0800 (CST) Received: from SZXEML409-HUB.china.huawei.com (10.82.67.136) by szxeml201-edg.china.huawei.com (172.24.2.39) with Microsoft SMTP Server (TLS) id 14.1.270.1; Sat, 27 Aug 2011 18:01:53 +0800 Received: from SZXEML525-MBS.china.huawei.com ([169.254.8.40]) by szxeml409-hub.china.huawei.com ([169.254.89.141]) with mapi id 14.01.0270.001; Sat, 27 Aug 2011 18:02:01 +0800 Date: Sat, 27 Aug 2011 10:02:00 +0000 From: Xuxiaohu X-Originating-IP: [10.110.98.53] To: Xuxiaohu , Linda Dunbar Message-id: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE725AF8@szxeml525-mbs.china.huawei.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_ZhAyDDwmGEGwkrXnmhzZcQ)" Content-language: zh-CN Accept-Language: zh-CN, en-US Thread-topic: [armd] Could millions of ARP entries on DC gateways be reduced to one Thread-index: Acxi0w1AcibzuVsiSkK+GwRxBeP6pgAMt1qAACKD51AARFKasA== X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected References: <1FEE3F8F5CCDE64C9A8E8F4AD27C19EE725622@szxeml525-mbs.china.huawei.com> Cc: "l2vpn@ietf.org" , "armd@ietf.org" Subject: [armd] =?gb2312?b?tPC4tDogIENvdWxkIG1pbGxpb25zIG9mIEFSUCBlbnRy?= =?gb2312?b?aWVzIG9uIERDIGdhdGV3YXlzIGJlIHJlZHVjZWQgdG8gb25l?= X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Aug 2011 10:01:27 -0000 --Boundary_(ID_ZhAyDDwmGEGwkrXnmhzZcQ) Content-type: text/plain; charset=gb2312 Content-transfer-encoding: base64 SGkgYWxsLA0KDQpBbiB1cGRhdGVkIHZlcnNpb24gb2YgVmlydHVhbCBTdWJuZXQgY2FuIGJlIGZv dW5kIGF0IGh0dHA6Ly90b29scy5pZXRmLm9yZy9odG1sL2RyYWZ0LXh1LXZpcnR1YWwtc3VibmV0 LTA2LiBWaXJ0dWFsIFN1Ym5ldCBpcyBhIGhvc3Qgcm91dGUgYmFzZWQgSVAtb25seSBMMlZQTiB3 aGljaCBjYW4gYmUgdXNlZCBhcyBhIHNjYWxhYmxlIGFwcHJvYWNoIGZvciBkYXRhIGNlbnRlciBp bnRlcmNvbm5lY3QuIENvbXBhcmVkIHRvIE1BQyBmb3J3YXJkaW5nIGJhc2VkIEwyVlBOIGFwcHJv YWNoZXMsIFZTIGNvdWxkIG9mZmVyIHlvdSBtYW55IGRlc2lyYWJsZSBiZW5lZml0cyB3aGljaCBh cmUgbWVudGlvbmVkIGluIGEgc2VjdGlvbiBvZiBDb21wYXJpc29uIHdpdGggVlBMUy4NCg0KQW55 IGNvbW1lbnRzIGFyZSB3ZWxjb21lLg0KDQpCZXN0IHJlZ2FyZHMsDQpYaWFvaHUNCg0Kt6K8/sjL OiBYdXhpYW9odQ0Kt6LLzcqxvOQ6IDIwMTHE6jjUwjI2yNUgMTA6MTINCsrVvP7IyzogJ0xpbmRh IER1bmJhcicNCrOty806IGwydnBuQGlldGYub3JnOyBhcm1kQGlldGYub3JnDQrW98ziOiC08Li0 OiBbYXJtZF0gQ291bGQgbWlsbGlvbnMgb2YgQVJQIGVudHJpZXMgb24gREMgZ2F0ZXdheXMgYmUg cmVkdWNlZCB0byBvbmUNCg0KDQoNCreivP7IyzogTGluZGEgRHVuYmFyIFttYWlsdG86ZHVuYmFy LmxsQGdtYWlsLmNvbV0NCreiy83KsbzkOiAyMDExxOo41MIyNsjVIDE6MDQNCsrVvP7IyzogWHV4 aWFvaHUNCrOty806IGwydnBuQGlldGYub3JnOyBhcm1kQGlldGYub3JnDQrW98ziOiBSZTogW2Fy bWRdIENvdWxkIG1pbGxpb25zIG9mIEFSUCBlbnRyaWVzIG9uIERDIGdhdGV3YXlzIGJlIHJlZHVj ZWQgdG8gb25lDQoNClhpYW8gSHUsDQoNClRoYXQgaXMgYSB2ZXJ5IGludGVyZXN0aW5nIHByb3Bv c2FsLg0KRG8geW91IG1lYW4gdGhhdCB0aGVyZSBpcyBhIHNpbmdsZSBBUlAgcHJveHkgd2hpY2gg cmVwcmVzZW50cyBtaWxsaW9ucyBvZiBWTXM/IEFsbCB0aGUgcGFja2V0cyBkZXN0aW5lZCB0byBh bnkgb2YgdGhvc2UgIm1pbGxpb25zIG9mIFZNcyIgd2lsbCBiZSByZXJvdXRlZCB0byAidGhlIHNp bmdsZSBBUlAgcHJveHkiPyBXaWxsIHRoaXMgY3JlYXRlIHRvbyBtdWNoIHByZXNzdXJlIG9uIHRo aXMgc2luZ2xlIHBvaW50Pw0KDQpIaSBMaW5kYSwNCg0KSXShr3MganVzdCBhbiBleGFtcGxlLiBJ biBwcmFjdGljZSwgaWYgdGhlIHByZXNzdXJlIG9uIHRoZSBMMlZQTiBQRSByb3V0ZXIgKGlmIFZp cnR1YWwgU3VibmV0IGlzIHVzZWQgZm9yIERDSSkgd2l0aGluIHRoZSBEQyBpcyB0b28gbGFyZ2Us IHlvdSBjb3VsZCBjb25uZWN0IHRoZSBEQyBleGl0IHJvdXRlciB0byBtdWx0aXBsZSBMMlZQTiBQ RSByb3V0ZXJzIHRvIGFjaGlldmUgbG9hZC1iYWxhbmNpbmcuIEZvciBleGFtcGxlLCB0aGUgdHJh ZmZpYyB3aXRoaW4gVkxBTiAxLTEwMDAgaXMgZm9yd2FyZGVkIHRocm91Z2ggUEUtMSBvZiB0aGF0 IEwyVlBOLCAgYW5kIHRoZSB0cmFmZmljIHdpdGhpbiBWTEFOIDEwMDEtMjAwMCBpcyBmb3J3YXJk ZWQgdGhyb3VnaCBQRS0yIG9mIHRoYXQgTDJWUE6hrSAgU3VjaCBsb2FkLWJhbGFuY2luZyB1c2Fn ZSBpcyB2ZXJ5IGNvbW1vbiBubyBtYXR0ZXIgd2hhdCBzcGVjaWFsIEwyIHRlY2hub2xvZ3kgKGUu Zy4sIFNUUCwgU1BCLFRSSUxMLFZQTFMgoa0pIGlzIHVzZWQuDQoNCkhvdyBpcyB5b3VyIHByb3Bv c2VkIGFwcHJvYWNoIGNvbXBhcmluZyB3aXRoIFNFQVRUTEUncyBkaXN0cmlidXRlZCBhcHByb2Fj aD8NCg0KU0VBVFRMRSBpcyBvbmUgb2YgdGhlIG1hbnkgTUFDLXJvdXRpbmcgYmFzZWQgTDJWUE4g c29sdXRpb25zLiBXaXRoIGFueSBNQUMtcm91dGluZyBiYXNlZCBMMlZQTiwgdGhlIGdhdGV3YXkg cm91dGVyIHNob3VsZCBhbmQgd291bGQgZ2V0IHRoZSByZWFsIE1BQyBhZGRyZXNzZXMgb2YgdGhl IHJlcXVlc3RlZCBob3N0cyBpbiBBUlAgcmVzb2x1dGlvbi4gU2luY2UgdGhlc2UgTUFDIGFkZHJl c3NlcyBhcmUgbm90IGlkZW50aWNhbCwgdGhlIEFSUCBjYWNoZSBlbnRyaWVzIG9uIHRoZSBnYXRl d2F5IHJvdXRlciBjb3VsZCBub3QgYmUgYWdncmVnYXRlZCBhdCBhbGwuDQoNClhpYW9odQ0KDQpM aW5kYSBEdW5iYXINCk9uIFdlZCwgQXVnIDI0LCAyMDExIGF0IDk6NDcgUE0sIFh1eGlhb2h1IDx4 dXhpYW9odUBodWF3ZWkuY29tPG1haWx0bzp4dXhpYW9odUBodWF3ZWkuY29tPj4gd3JvdGU6DQpI aSBhbGwsDQoNCkFub3RoZXIgb2J2aW91cyBhZHZhbnRhZ2Ugb2YgaG9zdCByb3V0ZSBiYXNlZCBJ UC1vbmx5IEwyVlBOIChlLmcuLCBWaXJ0dWFsIFN1Ym5ldCkgb3ZlciBWUExTLCBhcyBhIERDSSBz b2x1dGlvbiwgaXMgdG8gcmVkdWNlIHRoZSBBUlAgdGFibGUgc2l6ZSBvbiBEQyBnYXRld2F5cyBi eSBzZXZlcmFsIG9yZGVyIG9mIG1hZ25pdHVkZS4gQXNzdW1lIHRoZXJlIGFyZSBtaWxsaW9ucyBv ZiBDRSBob3N0cyAoaS5lLixWTXMpIHdpdGhpbiBhIHNpbmdsZSBWTEFOL3N1Ym5ldCwgaWYgVlBM UyBpcyB1c2VkIGFzIGEgRENJIHNvbHV0aW9uLCBEQyBnYXRld2F5cyAoaS5lLiwgREMgZXhpdCBy b3V0ZXJzKSB3b3VsZCBoYXZlIHRvIGtub3cgbWlsbGlvbnMgb2YgQVJQIGVudHJpZXMgY29ycmVz cG9uZGluZyB0byB0aGVzZSBWTXMuIEluIGNvbnRyYXN0LCBpbiB0aGUgVmlydHVhbCBTdWJuZXQg c29sdXRpb24sIERDIGdhdGV3YXlzIGFyZSBkaXJlY3RseSBjb25uZWN0ZWQgdG8gdGhlIFBFIHJv dXRlcnMgb2YgdGhlIElQLW9ubHkgTDJWUE4gd2hpY2ggYWN0IGFzIEFSUCBwcm94aWVzLCBNQUMg YWRkcmVzc2VzIG9mIHRob3NlIEFSUCBlbnRyaWVzIGZvciBWTXMgb24gREMgZ2F0ZXdheXMgYXJl IGlkZW50aWNhbCAoaS5lLiwgQVJQIFByb3h5oa9zIE1BQykuIFRodXMgdGhlc2UgbWlsbGlvbnMg b2YgQVJQIGVudHJpZXMgY2FuIGJlIGFnZ3JlZ2F0ZWQgaW50byBvbmUgZW50cnkgKDEuMC4wLjAv OC08aHR0cDovLzEuMC4wLjAvOC0+PkFSUCBwcm94eaGvcyBNQUMpLiBUaGF0oa9zIHRvIHNheSwg dGhlIGV4YWN0LW1hdGNoaW5nIGFsZ29yaXRobSBmb3IgQVJQIGNhY2hlIGxvb2t1cCBpcyBjaGFu Z2VkIHRvIHRoZSBsb25nZXN0LW1hdGNoaW5nIGFsZ29yaXRobS4gT2YgY291cnNlLCB0aGVyZSBp cyBubyBmcmVlIGx1bmNoLiBUaGUgc2lkZS1lZmZlY3Qgb2YgdGhpcyBjaGFuZ2UgaXMgdGhhdCBE QyBnYXRld2F5cyBjb3VsZCBzZW5kIG91dCBwYWNrZXRzIGRlc3RpbmVkIGZvciBub24tZXhpc3Rp bmcgVk1zIHRvIHRoZSBQRSByb3V0ZXJzIG9mIHRoYXQgSVAtb25seSBMMlZQTi4gRm9ydHVuYXRl bHksIG9uY2UgdGhvc2UgcGFja2V0cyBhcnJpdmUgYXQgdGhlIFBFIHJvdXRlcnMgb2YgdGhhdCBJ UC1vbmx5IEwyVlBOLCB3aGljaCBpbiB0dXJuIHdpbGwgZHJvcCB0aGVtIGRpcmVjdGx5IHNpbmNl IHRoZXJlIGlzIG5vIG1hdGNoaW5nIHJvdXRlcyBmb3IgdGhlbS4NCg0KU2luY2UgdGhpcyB0b3Bp YyBpcyBhbHNvIGludGVyZXN0aW5nIHRvIEFSTUQgV0csIEkgY29weSB0aGlzIGVtYWlsIHRvIEFS TUQuDQoNCkJlc3QgcmVnYXJkcywNClhpYW9odQ0KDQpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXw0KYXJtZCBtYWlsaW5nIGxpc3QNCmFybWRAaWV0Zi5vcmc8 bWFpbHRvOmFybWRAaWV0Zi5vcmc+DQpodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2FybWQNCg0K --Boundary_(ID_ZhAyDDwmGEGwkrXnmhzZcQ) Content-type: text/html; charset=gb2312 Content-transfer-encoding: quoted-printable

    Hi all,

     = ;

    An updated= version of Virtual Subnet can be found at http://to= ols.ietf.org/html/draft-xu-virtual-subnet-06. Virtual Subnet is a host = route based IP-only L2VPN which can be used as a scalable approach for data= center interconnect. Compared to MAC forwarding based L2VPN approaches, VS could offer you many desirable b= enefits which are mentioned in a section of Comparison with VPLS.

     = ;

    Any commen= ts are welcome.

     = ;

    Best regar= ds,

    Xiaohu

     = ;

    =B7=A2=BC=FE=C8=CB: Xuxiaoh= u
    =B7=A2= =CB=CD=CA=B1=BC=E4: 2011=C4=EA8=D4=C226=C8=D5 10:12
    =CA=D5=BC=FE=C8=CB: 'Linda Dunbar'
    =B3=AD=CB=CD: l2vpn@ietf.org; armd@ietf.org
    =D6=F7=CC=E2: =B4=F0=B8=B4: [armd] Could millions of ARP en= tries on DC gateways be reduced to one

     

     = ;

     = ;

    =B7=A2=BC=FE=C8=CB: Linda D= unbar [mailto:dunbar.ll@gmail.com]
    =B7=A2= =CB=CD=CA=B1=BC=E4: 2011=C4=EA8=D4=C226=C8=D5 1:04
    =CA=D5=BC=FE=C8=CB: Xuxiaohu
    =B3=AD=CB=CD: l2vpn@ietf.org; armd@ietf.org
    =D6=F7=CC=E2: Re: [armd] Could millions of ARP entries on DC gateways be reduced to one=

     

    Xiao Hu,

     

    That is a very interesting prop= osal.

    Do you mean that there is a sin= gle ARP proxy which represents millions of VMs? All the packets destined to= any of those "millions of VMs" will be rerouted to "the sin= gle ARP proxy"? Will this create too much pressure on this single point?

     = ;

    Hi Linda,<= o:p>

     = ;

    It=A1=AFs = just an example. In practice, if the pressure on the L2VPN PE router (if Vi= rtual Subnet is used for DCI) within the DC is too large, you could connect the DC exit router to multiple L2VPN PE routers to achieve load-ba= lancing. For example, the traffic within VLAN 1-1000 is forwarded through P= E-1 of that L2VPN,  and the traffic within VLAN 1001-2000 is forwarded= through PE-2 of that L2VPN=A1=AD  Such load-balancing usage is very common no matter what special L2 technology (e.g., STP, SPB,= TRILL,VPLS =A1=AD) is used.

     = ;

    How is your proposed approach c= omparing with SEATTLE's distributed approach?

     = ;

    SEATTLE is= one of the many MAC-routing based L2VPN solutions. With any MAC-routing ba= sed L2VPN, the gateway router should and would get the real MAC addresses of the requested hosts in ARP resolution. Since these MAC ad= dresses are not identical, the ARP cache entries on the gateway router coul= d not be aggregated at all.

     = ;

    Xiaohu

     

    Linda Dunbar<= /p>

    On Wed, Aug 24, 2011 at 9:47 PM= , Xuxiaohu <xuxiaohu@huawei.com> wrote:

    Hi all,

     

    Another obvious advantage of host route based= IP-only L2VPN (e.g., Virtual Subnet) over VPLS, as a DCI solution, is to r= educe the ARP table size on DC gateways by several order of magnitude. Assume there are millions of CE hosts (i.e.= ,VMs) within a single VLAN/subnet, if VPLS is used as a DCI solution, DC ga= teways (i.e., DC exit routers) would have to know millions of ARP entries c= orresponding to these VMs. In contrast, in the Virtual Subnet solution, DC gateways are directly connected to the = PE routers of the IP-only L2VPN which act as ARP proxies, MAC addresses of = those ARP entries for VMs on DC gateways are identical (i.e., ARP Proxy=A1= =AFs MAC). Thus these millions of ARP entries can be aggregated into one entry (1.0.0.0/8->ARP proxy=A1=AFs MAC). That=A1=AFs to say= , the exact-matching algorithm for ARP cache lookup is changed to the longe= st-matching algorithm. Of course, there is no free lunch. The side-effect of this change is that DC gateways could send out p= ackets destined for non-existing VMs to the PE routers of that IP-only L2VP= N. Fortunately, once those packets arrive at the PE routers of that IP-only= L2VPN, which in turn will drop them directly since there is no matching routes for them.

     

    Since this topic is also interesting to ARMD = WG, I copy this email to ARMD.

     

    Best regards,

    Xiaohu

    =
    _______________________________________________
    armd mailing list
    armd@ietf.org
    ht= tps://www.ietf.org/mailman/listinfo/armd

     

    --Boundary_(ID_ZhAyDDwmGEGwkrXnmhzZcQ)-- From warren@kumari.net Wed Aug 31 12:14:41 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2EADE21F8F49 for ; Wed, 31 Aug 2011 12:14:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.553 X-Spam-Level: X-Spam-Status: No, score=-102.553 tagged_above=-999 required=5 tests=[AWL=0.046, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gnInwiIQoei7 for ; Wed, 31 Aug 2011 12:14:40 -0700 (PDT) Received: from vimes.kumari.net (vimes.kumari.net [198.186.192.250]) by ietfa.amsl.com (Postfix) with ESMTP id EE15121F8F3F for ; Wed, 31 Aug 2011 12:14:39 -0700 (PDT) Received: from dot.her.corp.google.com (unknown [74.202.225.33]) by vimes.kumari.net (Postfix) with ESMTPSA id A10531B404C9; Wed, 31 Aug 2011 15:16:10 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=windows-1252 From: Warren Kumari In-Reply-To: Date: Wed, 31 Aug 2011 15:16:09 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> References: To: Vishwas Manral X-Mailer: Apple Mail (2.1084) Cc: armd@ietf.org Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Aug 2011 19:14:41 -0000 On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > Hi Linda/ Anoop, > =20 > Here is the example of the design I was talking about, as defined by = google. Just a clarification -- s/as defined by google/as described by someone = who happens to work for google/ W > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > =20 > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani = wrote: >=20 > >>>> > (though I think if there was a standard way to map Multicast MAC to = Multicast IP, they could probably use such a standard mechanisms). > >>>> >=20 > They can do that, but then this imposes requirements on the > equipment to be able to do multicast forwarding, and even if does, > because of pruning requirements the number of groups would be > very large. The average data center switch probably won't handle > that many groups. >=20 > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral = wrote: > Hi Anoop, > =20 > =46rom what I know they do not use Multicast GRE (I hear the extra 4 = bytes in the GRE header is a proprietery extension). > =20 > I think a directory based mechanism is what is used (though I think if = there was a standard way to map Multicast MAC to Multicast IP, they = could probably use such a standard mechanisms). > =20 > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani = wrote: > Hi Vishwas, >=20 > How do they get multicast through the network in that case? > Are they planning to use multicast GRE, or just use directory > based lookups and not worry about multicast applications > for now? >=20 > Anoop >=20 > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral = wrote: > Hi Linda, > =20 > The data packets can be tunnelled at the ToR over say a GRE packet and = the core is a Layer-3 core (except for the downstream ports). So we = could have encapsulation/ decapsulation of L2 over GRE at the ToR. > =20 > The very same thing can be done at the hypervisor layer too, in which = case the entire DC network would look like a Layer-3 flat network = including the ToR to server link and the hypervisor would do the = tunneling. > =20 > I am not sure if you got the points above or not. I know cloud OS = companies that provide the service and have big announced customers. > =20 > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar = wrote: > Vishwas, > =20 > In my mind the bullet 1) in the list refers to ToR switches downstream = ports (facing servers) running Layer 2 and ToR uplinks ports run IP = Layer 3. > =20 > Have you seen data center networks with ToR switches downstream ports = (i.e. facing servers) enabling IP routing, even though the physical = links are Ethernet? =20 > If yes, we should definitely include it in the ARMD draft. > =20 > Thanks, > Linda > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral = wrote: > Hi Linda, > I am unsure what you mean by this, but: > =95 layer 3 all the way to TOR (Top of Rack switches), > We can also have a heirarchical network, with the core totally Layer-3 = (and having seperate routing), from the hosts still in a large Layer-3 = subnet. Another aspect could be to have a totally Layer-3 network. > =20 > The difference between them is the link between the servers and the = ToR. > =20 > Thanks, > Vishwas > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar = wrote: > During the 81st IETF ARMD WG discussion, it was suggested that it is = necessary to document typical data center network designs so that = address resolution scaling issues can be properly described. Many data = center operators have expressed that they can't openly reveal their = detailed network designs. Therefore, we only want to document anonymous = designs without too much detail. During the journey of establishing = ARMD, we have come across the following typical data center network = designs: > =95 layer 3 all the way to TOR (Top of Rack switches), > =95 large layer 2 with hundreds (or thousands) of ToRs being = interconnected by Layer 2. This design will have thousands of hosts = under the L2/L3 boundary router (s) > =95 CLOS design with thousands of switches. This design will = have thousands of hosts under the L2/L3 boundary router(s) > We have heard that each of the designs above has its own problems. = ARMD problem statements might need to document DC problems under each = typical design. > Please send feedback to us (either to the armd email list or to the = ARMD chair Benson & Linda) to indicate if we have missed any typical = Data Center network designs. > =20 > Your contribution can greatly accelerate the progress of ARMD WG. > =20 > Thank you very much. > =20 > Linda & Benson >=20 >=20 From linda.dunbar@huawei.com Wed Aug 31 12:35:36 2011 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 32BDD21F8F55 for ; Wed, 31 Aug 2011 12:35:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.307 X-Spam-Level: X-Spam-Status: No, score=-6.307 tagged_above=-999 required=5 tests=[AWL=0.292, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5YJXSE89iKez for ; Wed, 31 Aug 2011 12:35:35 -0700 (PDT) Received: from usaga02-in.huawei.com (usaga02-in.huawei.com [206.16.17.70]) by ietfa.amsl.com (Postfix) with ESMTP id 5794521F8F54 for ; Wed, 31 Aug 2011 12:35:35 -0700 (PDT) Received: from huawei.com (localhost [127.0.0.1]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LQT0052D55SNP@usaga02-in.huawei.com> for armd@ietf.org; Wed, 31 Aug 2011 14:37:04 -0500 (CDT) Received: from dfweml201-edg.china.huawei.com ([172.18.4.104]) by usaga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPS id <0LQT00AM455NNZ@usaga02-in.huawei.com> for armd@ietf.org; Wed, 31 Aug 2011 14:37:04 -0500 (CDT) Received: from DFWEML402-HUB.china.huawei.com (10.193.5.102) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.270.1; Wed, 31 Aug 2011 12:36:58 -0700 Received: from DFWEML503-MBX.china.huawei.com ([169.254.3.37]) by DFWEML402-HUB.china.huawei.com ([::1]) with mapi id 14.01.0270.001; Wed, 31 Aug 2011 12:36:44 -0700 Date: Wed, 31 Aug 2011 19:36:43 +0000 From: Linda Dunbar In-reply-to: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> X-Originating-IP: [10.192.11.66] To: Warren Kumari , Vishwas Manral Message-id: <4A95BA014132FF49AE685FAB4B9F17F6102091D3@dfweml503-mbx.china.huawei.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-language: en-US Content-transfer-encoding: 7BIT Accept-Language: en-US Thread-topic: [armd] soliciting typical network designs for ARMD Thread-index: AQHMVt0qRHXKx8xJ1kiMH1hmlbd0hJUVhDmAgAOGoACAHuGmgP//jx7w X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <35BAFA1F-25E8-442E-8FE6-2D5691DCBEAC@kumari.net> Cc: "armd@ietf.org" Subject: Re: [armd] soliciting typical network designs for ARMD X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Aug 2011 19:35:36 -0000 Warren, Does the referenced design map the Multicast MAC to Multicast IP? Or map one multicast in MAC to multiple uni-cast messages? Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Warren Kumari > Sent: Wednesday, August 31, 2011 2:16 PM > To: Vishwas Manral > Cc: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > Hi Linda/ Anoop, > > > > Here is the example of the design I was talking about, as defined by > google. > > Just a clarification -- s/as defined by google/as described by someone > who happens to work for google/ > > W > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > wrote: > > > > >>>> > > (though I think if there was a standard way to map Multicast MAC to > Multicast IP, they could probably use such a standard mechanisms). > > >>>> > > > > They can do that, but then this imposes requirements on the > > equipment to be able to do multicast forwarding, and even if does, > > because of pruning requirements the number of groups would be > > very large. The average data center switch probably won't handle > > that many groups. > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > wrote: > > Hi Anoop, > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > bytes in the GRE header is a proprietery extension). > > > > I think a directory based mechanism is what is used (though I think > if there was a standard way to map Multicast MAC to Multicast IP, they > could probably use such a standard mechanisms). > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > wrote: > > Hi Vishwas, > > > > How do they get multicast through the network in that case? > > Are they planning to use multicast GRE, or just use directory > > based lookups and not worry about multicast applications > > for now? > > > > Anoop > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > wrote: > > Hi Linda, > > > > The data packets can be tunnelled at the ToR over say a GRE packet > and the core is a Layer-3 core (except for the downstream ports). So we > could have encapsulation/ decapsulation of L2 over GRE at the ToR. > > > > The very same thing can be done at the hypervisor layer too, in which > case the entire DC network would look like a Layer-3 flat network > including the ToR to server link and the hypervisor would do the > tunneling. > > > > I am not sure if you got the points above or not. I know cloud OS > companies that provide the service and have big announced customers. > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > wrote: > > Vishwas, > > > > In my mind the bullet 1) in the list refers to ToR switches > downstream ports (facing servers) running Layer 2 and ToR uplinks ports > run IP Layer 3. > > > > Have you seen data center networks with ToR switches downstream ports > (i.e. facing servers) enabling IP routing, even though the physical > links are Ethernet? > > If yes, we should definitely include it in the ARMD draft. > > > > Thanks, > > Linda > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > wrote: > > Hi Linda, > > I am unsure what you mean by this, but: > > * layer 3 all the way to TOR (Top of Rack switches), > > We can also have a heirarchical network, with the core totally Layer- > 3 (and having seperate routing), from the hosts still in a large Layer- > 3 subnet. Another aspect could be to have a totally Layer-3 network. > > > > The difference between them is the link between the servers and the > ToR. > > > > Thanks, > > Vishwas > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > wrote: > > During the 81st IETF ARMD WG discussion, it was suggested that it is > necessary to document typical data center network designs so that > address resolution scaling issues can be properly described. Many data > center operators have expressed that they can't openly reveal their > detailed network designs. Therefore, we only want to document anonymous > designs without too much detail. During the journey of establishing > ARMD, we have come across the following typical data center network > designs: > > * layer 3 all the way to TOR (Top of Rack switches), > > * large layer 2 with hundreds (or thousands) of ToRs being > interconnected by Layer 2. This design will have thousands of hosts > under the L2/L3 boundary router (s) > > * CLOS design with thousands of switches. This design will have > thousands of hosts under the L2/L3 boundary router(s) > > We have heard that each of the designs above has its own problems. > ARMD problem statements might need to document DC problems under each > typical design. > > Please send feedback to us (either to the armd email list or to the > ARMD chair Benson & Linda) to indicate if we have missed any typical > Data Center network designs. > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > Thank you very much. > > > > Linda & Benson > > > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd