Subject: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Jan 2012 00:22:52 +0000

Thomas, et al,

Here are my comments to draft-narten-nv03-overlay-problem-statment-01 and suggested wording change:

3.1. Limitations of Existing Virtual Network Models.

Since IEEE802.1 defined VLAN separation is mentioned, it would be appropriate to mention PBB's ISID separation.

The limitation for PBB and VLAN should include that MAC addresses can't be aggregated, therefore forwarding table can be very large for large data centers.

Second paragraph: Why "VLANs are a pure bridging construct while VRF is pure routing construct" is a problem?

4. Network Overlays

Should add a subsection to describe Virtual Network Instance ID.

4.x. Virtual Network Instance ID

Virtual Network Instance is for segregating traffic belonging to different tenants or different zones of one tenant. When a data center uses Overlay Network to hide hosts addresses, it is important that Virtual Network Instance identifier can properly represent the zones or the tenants in entire data center, especially when the overlay edge nodes are access switches, i.e. not embedded in hypervisors.

When overlay edge nodes are access switches, the data frames before entering the Overlay Network or after exiting the Overlay Network might carry traditional VLAN-ID for proper traffic segregation. The Virtual Network Instance ID value carried by the Overlay Header  of the data frames might be 24 bits (as described in 3.2). Those VLAN-ID for data frames under each overlay edge node are only locally significant. Proper mapping has to be maintained.

6.2 (TRILL) &6.3 (L2VPN)

It is necessary to point out that both TRILL and L2VPN carry the VLAN-ID embedded in the original Ethernet frames across the Overlay Network and the VLAN-ID maintain the same meaning in two separate L2 islands.

When VLAN-ID tagged Ethernet frames traverse across the Overlay Network for Data Center, the VLAN-ID carried by Ethernet frames lose its significance. In another words, a different VLAN-ID might be re-assigned to the Ethernet frames by the Egress overlay edge.

5.3. Associating a VNID with an endpoint.

You stated that "typically, it is a virtual NIC coming up that triggers this association". It is not quite right. Virtual NIC, same as physical NIC for a non-virtualized server, usually is not aware of which network segment it is attached to. For Ethernet data frames coming out from a traditional server (i.e. server without virtualization), the data frame is not tagged, i.e. not having VID associated with it. It is the first switch which assign the VID for the data frame.

It is more accurate to say  "virtual switch" or "Hypervisor" within the server which can trigger the association.

Since overlay header can be also added by first access switches (i.e. switches not embedded in physical servers), it is very possible that  data frames arriving at the overlay edge (i.e. the first access switch) is already VLAN tagged. Then association (or mapping) from the VLAN-Tag to VNID has to be operator administrated.

Linda Dunbar

---

From: Paul Unbehagen <[email-masked]>
Date: Thu, 26 Jan 2012 17:50:44 -0700

Inline below

--
Paul Unbehagen

On Jan 26, 2012, at 5:22 PM, Linda Dunbar <[email-masked]> wrote:

> Since IEEE802.1 defined VLAN separation is mentioned, it would be appropriate to mention PBB’s ISID separation.

Agreed, clear service separation and the 16 million services should be clearly explained, additionally the ease of edge provisioning over STP's per hop configuration. 

>  
> The limitation for PBB and VLAN should include that MAC addresses can’t be aggregated, therefore forwarding table can be very large for large data centers.

Disagree, this isn't what we see in live deployments. Since the Mac learning of the host stations only happens at the edge switches and the services are distributed across many edge switches the ISID service Mac table sizes don't end up being that large on any given node. Additionally the core never sees any macs but the nodes participating in the backbone from edge switch to edge switch. 

>  
> Second paragraph: Why “VLANs are a pure bridging construct while VRF is pure routing construct” is a problem?
>  
> 4. Network Overlays
>  
> You stated that “typically, it is a virtual NIC coming up that triggers this association”. It is not quite right. Virtual NIC, same as physical NIC for a non-virtualized server, usually is not aware of which network segment it is attached to. For Ethernet data frames coming out from a traditional server (i.e. server without virtualization), the data frame is not tagged, i.e. not having VID associated with it. It is the first switch which assign the VID for the data frame.
>  
> It is more accurate to say  “virtual switch” or “Hypervisor” within the server which can trigger the association.
>  
> _______________________________________________
> dc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dc

---

From: Rakesh Saha <[email-masked]>
Date: Thu, 26 Jan 2012 23:43:03 -0800

Hi Linda,

>You stated that “typically, it is a virtual NIC coming up that triggers 
this association”. It is not quite right. Virtual NIC, same as physical 
NIC for a non-virtualized server, usually is not aware of which >network 
segment it is attached to. For Ethernet data frames coming out from a 
traditional server (i.e. server without virtualization), the data frame is 
not tagged, i.e. not having VID associated with it. It is >the first 
switch which assign the VID for the data frame. 

Regarding "For Ethernet data frames coming out from a traditional server 
(i.e. server without virtualization), the data frame is not tagged",..." 
It is >the first switch which assign the VID for the data frame" ...

non-virtualized servers are capable of sending VLAN tagged packets. It is 
a standard feature of today'as traditional Operating Systems (NIC drivers) 
 to allow VLANs to be associated with an ethernet interface.
So a traditional non-virtualized server may send tagged and untagged 
frames.

Thanks,
Rakesh.
-=-

From:   Linda Dunbar <[email-masked]>
To:     [email-masked], "[email-masked]" 
<[email-masked]>, Murari Sridharan <[email-masked]>, Dinesh 
Dutt <[email-masked]>, "[email-masked]" <[email-masked]>
Cc:     "[email-masked]" <[email-masked]>
Date:   01/26/2012 04:25 PM
Subject:        [dc] comments and suggestions to 
draft-narten-nv03-overlay-problem-statment-01
Sent by:        [email-masked]

Thomas, et al, 
 
Here are my comments to draft-narten-nv03-overlay-problem-statment-01 and 
suggested wording change:

Since IEEE802.1 defined VLAN separation is mentioned, it would be 
appropriate to mention PBB’s ISID separation. 
 
The limitation for PBB and VLAN should include that MAC addresses can’t be 
aggregated, therefore forwarding table can be very large for large data 
centers. 
 
Second paragraph: Why “VLANs are a pure bridging construct while VRF is 
pure routing construct” is a problem? 
 
4. Network Overlays

 
Virtual Network Instance is for segregating traffic belonging to different 
tenants or different zones of one tenant. When a data center uses Overlay 
Network to hide hosts addresses, it is important that Virtual Network 
Instance identifier can properly represent the zones or the tenants in 
entire data center, especially when the overlay edge nodes are access 
switches, i.e. not embedded in hypervisors. 
 
When overlay edge nodes are access switches, the data frames before 
entering the Overlay Network or after exiting the Overlay Network might 
carry traditional VLAN-ID for proper traffic segregation. The Virtual 
Network Instance ID value carried by the Overlay Header  of the data 
frames might be 24 bits (as described in 3.2). Those VLAN-ID for data 
frames under each overlay edge node are only locally significant. Proper 
mapping has to be maintained. 
 
6.2 (TRILL) &6.3 (L2VPN)
 
It is necessary to point out that both TRILL and L2VPN carry the VLAN-ID 
embedded in the original Ethernet frames across the Overlay Network and 
the VLAN-ID maintain the same meaning in two separate L2 islands. 
 
When VLAN-ID tagged Ethernet frames traverse across the Overlay Network 
for Data Center, the VLAN-ID carried by Ethernet frames lose its 
significance. In another words, a different VLAN-ID might be re-assigned 
to the Ethernet frames by the Egress overlay edge. 

You stated that “typically, it is a virtual NIC coming up that triggers 
this association”. It is not quite right. Virtual NIC, same as physical 
NIC for a non-virtualized server, usually is not aware of which network 
segment it is attached to. For Ethernet data frames coming out from a 
traditional server (i.e. server without virtualization), the data frame is 
not tagged, i.e. not having VID associated with it. It is the first switch 
which assign the VID for the data frame. 
 
It is more accurate to say  “virtual switch” or “Hypervisor” within the 
server which can trigger the association. 
 
Since overlay header can be also added by first access switches (i.e. 
switches not embedded in physical servers), it is very possible that  data 
frames arriving at the overlay edge (i.e. the first access switch) is 
already VLAN tagged. Then association (or mapping) from the VLAN-Tag to 
VNID has to be operator administrated. 

Linda Dunbar

---

From: Thomas Narten <[email-masked]>
Date: Fri, 27 Jan 2012 08:42:43 -0500

> > appropriate to mention PBBâ€™s ISID separation.

> Agreed, clear service separation and the 16 million services should
> be clearly explained, additionally the ease of edge provisioning
> over STP's per hop configuration.

Will do.

> > The limitation for PBB and VLAN should include that MAC addresses
> > can't be aggregated, therefore forwarding table can be very large
> > for large data centers.

> Disagree, this isn't what we see in live deployments. Since the Mac
> learning of the host stations only happens at the edge switches and
> the services are distributed across many edge switches the ISID
> service Mac table sizes don't end up being that large on any given
> node. Additionally the core never sees any macs but the nodes
> participating in the backbone from edge switch to edge switch.

I think we need to distinguish between PBB-V and PBB-M.

With PBB-M, you get mac-in-mac encapsulation, so the PBB Bridge nodes
(as part of forwarding) only look at (and build tables for) the MACs
in the outer header. Consequently, you don't get MAC table explosion
here as your core network gets bigger. Is this what you mean above?

For PBB-V, the PBB bridges are routing on client MAC addrs (C-MACs),
so you presumably will run into issues with table size as your network
core increases in size.

Agreed?

Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Jan 2012 16:47:09 +0000

Paul,

See my comments below:

[Linda] If PBB’s MAC-in-MAC Overlay encapsulation is done at the access switches in data center, then the core switches forwarding table won’t explode. However, if the Mac-in-Mac overlay encapsulation is performed by servers’ hypervisor, the switches’ forwarding table could explode if there are hundreds of thousands of servers in data center, which is getting more and more common in large data centers.

Linda

---

From: David Allan I <[email-masked]>
Date: Fri, 27 Jan 2012 11:50:57 -0500

So what you are saying is if you architect your network wrong, you have a problem.

That issue is not confined to PBB

Dave

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Friday, January 27, 2012 8:47 AM
To: Paul Unbehagen
Cc: Thomas Narten; [email-masked]; [email-masked]; Dinesh Dutt; Murari Sridharan; [email-masked]
Subject: Re: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

Paul,

See my comments below:

[Linda] If PBB's MAC-in-MAC Overlay encapsulation is done at the access switches in data center, then the core switches forwarding table won't explode. However, if the Mac-in-Mac overlay encapsulation is performed by servers' hypervisor, the switches' forwarding table could explode if there are hundreds of thousands of servers in data center, which is getting more and more common in large data centers.

Linda

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 27 Jan 2012 16:50:50 +0000

Rakesh,

I agree with you that some traditional servers can attach VLAN-ID for data frames coming out.
But many servers deployed today don’t attach VID.

Linda

From: Rakesh Saha [mailto:[email-masked]]
Sent: Friday, January 27, 2012 1:43 AM
To: Linda Dunbar
Cc: [email-masked]; [email-masked]; [email-masked]; Dinesh Dutt; [email-masked]; Murari Sridharan; [email-masked]
Subject: Re: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

Hi Linda,

>You stated that “typically, it is a virtual NIC coming up that triggers this association”. It is not quite right. Virtual NIC, same as physical NIC for a non-virtualized server, usually is not aware of which >network segment it is attached to. For Ethernet data frames coming out from a traditional server (i.e. server without virtualization), the data frame is not tagged, i.e. not having VID associated with it. It is >the first switch which assign the VID for the data frame.

Regarding "For Ethernet data frames coming out from a traditional server (i.e. server without virtualization), the data frame is not tagged",..." It is >the first switch which assign the VID for the data frame" ...

non-virtualized servers are capable of sending VLAN tagged packets. It is a standard feature of today'as traditional Operating Systems (NIC drivers)  to allow VLANs to be associated with an ethernet interface.
So a traditional non-virtualized server may send tagged and untagged frames.

Thanks,
Rakesh.
-=-

From:        Linda Dunbar <[email-masked]>
To:        [email-masked], "[email-masked]" <[email-masked]>, Murari Sridharan <[email-masked]>, Dinesh Dutt <[email-masked]>, "[email-masked]" <[email-masked]>
Cc:        "[email-masked]" <[email-masked]>
Date:        01/26/2012 04:25 PM
Subject:        [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01
Sent by:        [email-masked]
________________________________

Thomas, et al,

4. Network Overlays

6.2 (TRILL) &6.3 (L2VPN)

Linda Dunbar

---

From: AshwoodsmithPeter <[email-masked]>
Date: Fri, 27 Jan 2012 18:23:51 +0000

> I think we need to distinguish between PBB-V and PBB-M.

Thomas, No such beasts as PBB-V and PBB-M ;) I think you are referring to SPBV and SPBM which have the properties you describe.

Peter

---

From: Thomas Narten <[email-masked]>
Date: Fri, 27 Jan 2012 13:29:08 -0500

> Thomas, No such beasts as PBB-V and PBB-M ;) I think you are
>  referring to SPBV and SPBM which have the properties you describe.

Indeed! Too many acronyms... Sigh. :-(

Thomas

---

From: Paul Unbehagen <[email-masked]>
Date: Fri, 27 Jan 2012 12:18:28 -0700

Also, all implementations and deployments of SPB are SPBm, the scaling and ability to deal with Mac explosions help in the DC. 

--
Paul Unbehagen

Sent from my iPad

---

From: Thomas Narten <[email-masked]>
Date: Fri, 27 Jan 2012 14:23:59 -0500

> Also, all implementations and deployments of SPB are SPBm, the
>  scaling and ability to deal with Mac explosions help in the DC.

Can others confirm that SPB-M is what deployments are using? And that
SPB-V is essentially irrelevant?

The benefits of SPB-M over SPB-V seem pretty compelling to me. It
would be nice if we could focus all SPB discussions on SPB-M rather
than worry about SPB-V.

Thomas

---

From: AshwoodsmithPeter <[email-masked]>
Date: Fri, 27 Jan 2012 19:22:02 +0000

Agreed, just for clarity Shortest Path Bridging Mac-in-Mac mode (SPBM) is the one that matters in this context (DC) and SPBM 'basically' uses the PBB (mac-in-mac) datapath.

On the ENCAP location, I suppose you 'could' do the full PBB encap in a hypervisor, if that hypervisor were controlling 1000's of VMs, but otherwise the only part of the PBB encap that would make sense in the hypervisor would be perhaps to add the ISID (I-TAG) instead of (C/S-TAGs) and save the TOR the trouble of the mapping.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Friday, January 27, 2012 1:29 PM
To: AshwoodsmithPeter
Cc: [email-masked]; [email-masked]; Dinesh Dutt; Linda Dunbar; Paul Unbehagen; Murari Sridharan; [email-masked]
Subject: Re: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

Thomas

---

From: "Pat Thaler" <[email-masked]>
Date: Fri, 27 Jan 2012 21:35:13 +0000

Linda,

Sometimes both can be true of the same server. A server that is can attach a VLAN-ID to data frames coming out may not be configured to do so. A large part of the problem for VLANs and overlay networks with virtualization is a control plane problem on how to get the network configuration information where it needs to be as virtual servers are moved.

5.3 doesn’t say that the virtual NIC adds the VLAN IDs to frames or that the virtual NIC itself triggers the association.

“Typically, it is a virtual NIC (the one connected to the VM)

 coming up that triggers this association. The access switch can then
   determine the VNID to be associated with this virtual NIC.”
To me this says that bringing up the virtual NIC causes something to trigger an association of a VNID with a VM. That something may not be the vNIC. The association might be made by the hypervisor as part of the installation of the virtual NIC.

For example, in the draft of IEEE 802.1Qbg, a hypervisor can use VDP (VSI Discovery and configuration Protocol) to tell the attached switch that a VSI (Virtual Station Interface, e.g. a vNIC) is coming up and ask it to create the association. VDP can either tell the attached switch what VLAN to use for the VSI or it can leave the VID field null to let the attached switch fill in the VLAN based on information such as VSI profile identifier and a 32-bit GroupID.

Regards,
Pat

From: [email-masked] [mailto:[email-masked]] On Behalf Of Linda Dunbar
Sent: Friday, January 27, 2012 8:51 AM
To: Rakesh Saha
Cc: [email-masked]; [email-masked]; Dinesh Dutt; [email-masked]; Murari Sridharan; [email-masked]; [email-masked]
Subject: Re: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

Rakesh,

Linda

From: Rakesh Saha [mailto:[email-masked]]<mailto:[mailto:[email-masked]]>
Sent: Friday, January 27, 2012 1:43 AM
To: Linda Dunbar
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Dinesh Dutt; [email-masked]<mailto:[email-masked]>; Murari Sridharan; [email-masked]<mailto:[email-masked]>
Subject: Re: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

Hi Linda,

Thanks,
Rakesh.
-=-

From:        Linda Dunbar <[email-masked]<mailto:[email-masked]>>
To:        [email-masked]<mailto:[email-masked]>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Murari Sridharan <[email-masked]<mailto:[email-masked]>>, Dinesh Dutt <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc:        "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date:        01/26/2012 04:25 PM
Subject:        [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01
Sent by:        [email-masked]<mailto:[email-masked]>
________________________________

Thomas, et al,

4. Network Overlays

6.2 (TRILL) &6.3 (L2VPN)

Linda Dunbar
 _______________________________________________
dc mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/dc

---

From: AshwoodsmithPeter <[email-masked]>
Date: Fri, 27 Jan 2012 21:36:26 +0000

Thomas, I'm not aware of any SPB-V implementations ... yet ... but there are a number of interoperable SPB-M implementations (see http://ieee802.org/1/files/public/docs2011/aq-ashwood-smith-spbm-3rd-interop-0718-v01.pdf ).

My understanding is that there are a bunch of live deployments of SPB-M already including DC's but I am not privy to all the details of course.

So I think concentrating on SPB-M would make sense, but if somebody feels otherwise by all means speak up.

Peter Ashwood-Smith

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]] 
Sent: Friday, January 27, 2012 2:24 PM
To: Paul Unbehagen
Cc: AshwoodsmithPeter; [email-masked]; [email-masked]; Dinesh Dutt; Linda Dunbar; Murari Sridharan; [email-masked]
Subject: Re: [dc] comments and suggestions to draft-narten-nv03-overlay-problem-statment-01

Thomas