Subject: [spring] Different MSDs for different traffic types on the same headend.

From: Nat Kao <[email-masked]>
Date: Fri, 13 Dec 2019 19:41:48 +0800

Hello, SPRING WG.

How do we deal with an SR-TE policy headend with different MSDs for
different types of traffic?
For example, a headend H can impose:
6 transport labels for plain IPv4 packets;
5 transport labels + 1 IPv6 ExpNull label for plain IPv6 packets;
3 transport labels + 3 VPN  labels for VPN packets.

a) For a plain IPv4 route R4 and a VPN route Rv both steered into the SR-TE
policy P1 with SID list <S1, S2, S3, S4, S5>, what will H perform in this
situation?
b) What is the MSD of H? 6, 5 or 3?

Thanks,
Nat.

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 13 Dec 2019 10:42:33 -0800

Hi Nat,

Please read https://tools.ietf.org/html/rfc8491#section-5
Currently defined MSD types are:
1: BMI
2: ERLD

Specifically to BMI:
Base MPLS Imposition MSD (BMI-MSD) signals the total number of MPLS labels that can be imposed, including all service/transport/special labels.
The answer to your question is 6

Cheers,
Jeff
On Dec 13, 2019, 3:42 AM -0800, Nat Kao <[email-masked]>, wrote:
> Hello, SPRING WG.
>
> How do we deal with an SR-TE policy headend with different MSDs for different types of traffic?
> a) For a plain IPv4 route R4 and a VPN route Rv both steered into the SR-TE policy P1 with SID list <S1, S2, S3, S4, S5>, what will H perform in this situation?
> _______________________________________________
> spring mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spring

---

From: Nat Kao <[email-masked]>
Date: Sat, 14 Dec 2019 15:42:48 +0800

Hi, Jeff.

Thanks for the BMI-MSD reference. If I understand correctly:

BMI-MSD = Transport Label Depth + Service Label Depth
Only former can be utilized by SR-TE policies.

Currently do we have any method to determine the composition of BMI?
We need to know the transport label depth when doing service route
per-destination steering.

This problem arises when trying to steer a plain IPv4 route and a VPN
service route into the same SR-TE policy that exceeds the transport label
depth of the service route. I'm trying to figure out the standard behavior
in this case since the headend we use currently produces some interesting
results.

Regards,
Nat.

On Sat, Dec 14, 2019 at 2:42 AM Jeff Tantsura <[email-masked]>
wrote:

> Base MPLS Imposition MSD (BMI-MSD) signals the total number of MPLS labels
> that can be imposed, including all service/transport/special labels.
> a) For a plain IPv4 route R4 and a VPN route Rv both steered into the
> SR-TE policy P1 with SID list <S1, S2, S3, S4, S5>, what will H perform in
> this situation?

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 15 Dec 2019 02:59:07 -0500

Jeff

With SR-MPLS with SR-TR let’s say if you use cSPF snd don’t have an ERO
strict explicit path defined or is a loose path, then the for the cSPF
would the transport labels be 1.  For loose would also be 1 also.  If the
path were explicit defined to egress PE and was 7 hops from ingress to
egress then transport would be 6.  And if L3 vpn service sid was signaled
that would be 1 vpn label.

Let me know if I have that right.

In Nats scenario for IPv6 he has 3 vpnv6 labels.

Why is that?

With both SR-MPLS and SRv6 the L3 vpn AFI/SAFI MBGP services overlay single
label sits on top off SR as if does today with MPLS so why 3 vpn labels.

So with this draft with BGP-LS and BMI-MSD you can flood into the IGP the
SID depth so all the nodes along the SR-TE path don’t go over the maximum
which would result in an error.

If you set your MTU high enough in the core like 9216, does that overcome
the SID depth issues with SR-TE?

Warm regards,

Gyan

On Sat, Dec 14, 2019 at 2:43 AM Nat Kao <[email-masked]> wrote:

> On Sat, Dec 14, 2019 at 2:42 AM Jeff Tantsura <[email-masked]
> <[email-masked]>> wrote:
>
>> Hi Nat,
>>
-- 

Gyan S. Mishra

IT Network Engineering & Technology

Verizon Communications Inc. (VZ)

13101 Columbia Pike FDC1 3rd Floor

Silver Spring, MD 20904

United States

Phone: 301 502-1347

Email: [email-masked]

www.linkedin.com/in/networking-technologies-consultant

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 15 Dec 2019 03:04:40 -0500

Spring

Does anyone know if there are plans to add RSVP PCALC path and reserve
model to both SR-TE for SR-MPLS and TI-LFA for SRv6.  That would be good so
that we can still use the benefits of MOLS TE for bandwidth management
using auto bandwidthor PCALC bandwidth management in pcep  distributed or
centralized controller model.

Warm regards,

Gyan

On Sun, Dec 15, 2019 at 2:59 AM Gyan Mishra <[email-masked]> wrote:

> With both SR-MPLS and SRv6 the L3 vpn AFI/SAFI MBGP services overlay
> single label sits on top off SR as if does today with MPLS so why 3 vpn
> labels.
>

Gyan S. Mishra

Silver Spring, MD 20904

United States

Phone: 301 502-1347

---

From: Jeff Tantsura <[email-masked]>
Date: Mon, 16 Dec 2019 15:19:05 -0800

Gyan,

MSD is only relevant for a device that either imposes the label stack (head-end) or manipulates it (BSID anchor). There are some other constrains when it comes to entropy labels and ERLD, please read the respective drafts.
In general, SID stack would grow when TE is in use (any time you need to use additional SID to deviate from SPT), another case is when additional SID’s are used for services on the nodes, other than the tail-end.
That’s why we've designed MSD to be very flexible to accommodate all the different use cases, it is upto computational logic to decide how to deal with different constrains (MSD types)

Cheers,
Jeff

P.S. you might want to see the NANOG MSD presentation I did some time ago.
https://pc.nanog.org/static/published/meetings/NANOG71/1424/20171004_Tantsura_The_Critical_Role_v1.pdf
On Dec 14, 2019, 11:59 PM -0800, Gyan Mishra <[email-masked]>, wrote:
> Jeff
>
> With SR-MPLS with SR-TR let’s say if you use cSPF snd don’t have an ERO strict explicit path defined or is a loose path, then the for the cSPF would the transport labels be 1.  For loose would also be 1 also.  If the path were explicit defined to egress PE and was 7 hops from ingress to egress then transport would be 6.  And if L3 vpn service sid was signaled that would be 1 vpn label.
>
> With both SR-MPLS and SRv6 the L3 vpn AFI/SAFI MBGP services overlay single label sits on top off SR as if does today with MPLS so why 3 vpn labels.
>
> So with this draft with BGP-LS and BMI-MSD you can flood into the IGP the SID depth so all the nodes along the SR-TE path don’t go over the maximum which would result in an error.
>
> If you set your MTU high enough in the core like 9216, does that overcome the SID depth issues with SR-TE?
>
> Warm regards,
>
> Gyan
>
> > > We need to know the transport label depth when doing service route per-destination steering.
> > >
> > > This problem arises when trying to steer a plain IPv4 route and a VPN service route into the same SR-TE policy that exceeds the transport label depth of the service route. I'm trying to figure out the standard behavior in this case since the headend we use currently produces some interesting results.
> > >
> > > Regards,
> > > Nat.
> > >
> > > > On Sat, Dec 14, 2019 at 2:42 AM Jeff Tantsura <[email-masked]> wrote:
> > > > > Hi Nat,
> > > > >

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 16 Dec 2019 20:25:29 -0500

Thanks Jeff!!

Both SR-MPLS & SRv6 in general I am guessing most deployments have been
centralized controller based model to take advantage of PCEP  and SR-TE
policy as necessary automatically instead of statically defined explicit
paths.

For small deployments I guess you could get away with non controller based
deployments.

The MSD limit does seem low even with proprietary asic 4-10 seems low.

Is the limiting factor with MSD the ASIC merchant or proprietary.

What is the MTU of the packet when you hit 10 hops or 10 adjacencies SIDS?

Also if you are doing tunnel stitching using BSID the MSD limit gets worse.

Sounds like SR-TE has a way to go on the asic side to support very long
explicit paths.

Warm regards,

Gyan

On Mon, Dec 16, 2019 at 6:19 PM Jeff Tantsura <[email-masked]>
wrote:

> Gyan,
>
> MSD is only relevant for a device that either imposes the label stack
> (head-end) or manipulates it (BSID anchor). There are some other constrains
> when it comes to entropy labels and ERLD, please read the respective drafts.
> In general, SID stack would grow when TE is in use (any time you need to
> use additional SID to deviate from SPT), another case is when additional
> SID’s are used for services on the nodes, other than the tail-end.
> That’s why we've designed MSD to be very flexible to accommodate all the
> different use cases, it is upto computational logic to decide how to deal
> with different constrains (MSD types)
>
> Cheers,
> Jeff
>
> 13101 Columbia Pike
> <https://www.google.com/maps/search/13101+Columbia+Pike?entry=gmail&source=g>
> FDC1 3rd Floor
>
> Silver Spring, MD 20904
>
> United States
>
> Phone: 301 502-1347
>

Gyan S. Mishra

Silver Spring, MD 20904

United States

Phone: 301 502-1347

---

From: Nat Kao <[email-masked]>
Date: Tue, 17 Dec 2019 13:05:13 +0800

Hi, Jeff.

Consider a headend that can perform 1 of the following 2 modes(but not
both):
1) Plain IPv4: 6 transport labels + 0 service label => traffic can be
steered into a 6-label SR-TE policy.
2) Any type of VPN: 3 transport labels + 1~3 service labels => traffic
cannot be steered into a 6-label SR-TE policy.

a) As defined in RFC8491, the BMI-MSD is 6 for this headend. Do we have a
standardized way to signal the transport label depth in mode 2?
   Maybe in a different MSD type?
b) Since plain IPv4 and VPN routes can be steered into the same SR-TE
policy, do we have a standardized headend behavior in this situation?
   (Should I open a new thread to discuss this? It seems not quite
MSD-relative.)

Thanks.
Nat

On Tue, Dec 17, 2019 at 11:14 AM Nat Kao <[email-masked]> wrote:

> Hi, Jeff.
>
> Consider a headend that can perform 1 of the following 2 modes(but not
> both):
> 1) Plain IPv4: 6 transport labels + 0 service label => traffic can be
> steered into a 6-label SR-TE policy.
> 2) Any type of VPN: 3 transport labels + 1~3 service labels => traffic
> cannot be steered into a 6-label SR-TE policy.
>
> a) As defined in RFC8491, the BMI-MSD is 6 for this headend. Do we have a
> standardized way to signal the transport label depth in mode 2?
>    Maybe in a different MSD type?
> b) Since plain IPv4 and VPN routes can be steered into the same SR-TE
> policy, do we have a standardized headend behavior in this situation?
>    (Should I open a new thread to discuss this? It seems not quite
> MSD-relative.)
>
> Thanks.
> Nat.
>
>
> On Tue, Dec 17, 2019 at 7:19 AM Jeff Tantsura <[email-masked]>
> wrote:
>
>> Gyan,
>>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 17 Dec 2019 10:58:50 +0100

Hi Nat,

I am having a bit of difficulty understanding reasoning and the way you are
separating transport from service labels.

The processing label limit usually comes from data plane capabilities of
the platform namely LFIB or hardware below.

Such layer is function agnostic and it does not matter what role the label
serves.

Label lookup results in pointer to a adj. rewrite on the outbound side. It
really does not matter if the adjacent peer is your P router, segment
endpoint or CE.

Of course there are number of processing exceptions - for example concept
of aggregate VPN label, additional header modifications etc ...

Likely some platforms put such artificial boundary between transport and
service hence your request. But if so IMHO this is more of an issue with
specific platform or its marketing message then IETF :).

Many thx,
R.

On Tue, Dec 17, 2019 at 6:06 AM Nat Kao <[email-masked]> wrote:

> Thanks..
>>> So with this draft with BGP-LS and BMI-MSD you can flood into the IGP
>>> the SID depth so all the nodes along the SR-TE path don’t go over the
>>> maximum which would result in an error.
>>>
>>> If you set your MTU high enough in the core like 9216, does that
>>> overcome the SID depth issues with SR-TE?
>>>
>>> Warm regards,
>>>
>>> Gyan
>>>

---

From: Nat Kao <[email-masked]>
Date: Tue, 17 Dec 2019 19:45:44 +0800

Hello, Robert.

Surely the current BMI-MSD definition is sufficient for platforms without
artificial boundaries.
In this ideal case, maximum labels available for SR-TE policy can be
inferred from BMI-MSD and VPN routes.

However we have 3 different artificial boundaries across 3 different
platforms now.
Hardcoding these boundaries might not scale well. (Especially different OS
versions may behave differently.)
A standard mechanism to discover this boundary would greatly help
constructing SR-TE policy candidate paths.

Regards,
Nat.

On Tue, Dec 17, 2019 at 5:58 PM Robert Raszuk <[email-masked]> wrote:

> Hi Nat,
>
> I am having a bit of difficulty understanding reasoning and the way you
> are separating transport from service labels.
>
>>>> In general, SID stack would grow when TE is in use (any time you need
>>>> to use additional SID to deviate from SPT), another case is when additional
>>>> That’s why we've designed MSD to be very flexible to accommodate all
>>>> the different use cases, it is upto computational logic to decide how to
>>>> deal with different constrains (MSD types)
>>>>
>>>> Cheers,
>>>> Jeff
>>>>

---

From: "Ketan Talaulikar (ketant)" <[email-masked]>
Date: Tue, 17 Dec 2019 12:21:23 +0000

Hi Nat,

The MSD framework enables us to define more/new MSD types. If there is a real use-case and requirement (as you express) and the necessary MSD type(s) can be formally defined then perhaps the WG can evaluate it.

Thanks,
Ketan

From: spring <[email-masked]> On Behalf Of Nat Kao
Sent: 17 December 2019 17:16
To: Robert Raszuk <[email-masked]>
Cc: SPRING WG <[email-masked]>; Nat Kao <[email-masked]>
Subject: Re: [spring] Different MSDs for different traffic types on the same headend.

Hello, Robert.
Surely the current BMI-MSD definition is sufficient for platforms without artificial boundaries.
In this ideal case, maximum labels available for SR-TE policy can be inferred from BMI-MSD and VPN routes.

However we have 3 different artificial boundaries across 3 different platforms now.
Hardcoding these boundaries might not scale well. (Especially different OS versions may behave differently.)
A standard mechanism to discover this boundary would greatly help constructing SR-TE policy candidate paths.

Regards,
Nat.

On Tue, Dec 17, 2019 at 5:58 PM Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:
Hi Nat,

I am having a bit of difficulty understanding reasoning and the way you are separating transport from service labels.

The processing label limit usually comes from data plane capabilities of the platform namely LFIB or hardware below.

Such layer is function agnostic and it does not matter what role the label serves.

Label lookup results in pointer to a adj. rewrite on the outbound side. It really does not matter if the adjacent peer is your P router, segment endpoint or CE.

Of course there are number of processing exceptions - for example concept of aggregate VPN label, additional header modifications etc ...

Likely some platforms put such artificial boundary between transport and service hence your request. But if so IMHO this is more of an issue with specific platform or its marketing message then IETF :).

Many thx,
R.

On Tue, Dec 17, 2019 at 6:06 AM Nat Kao <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Jeff.
Consider a headend that can perform 1 of the following 2 modes(but not both):
1) Plain IPv4: 6 transport labels + 0 service label => traffic can be steered into a 6-label SR-TE policy.
2) Any type of VPN: 3 transport labels + 1~3 service labels => traffic cannot be steered into a 6-label SR-TE policy.
a) As defined in RFC8491, the BMI-MSD is 6 for this headend. Do we have a standardized way to signal the transport label depth in mode 2?
   Maybe in a different MSD type?
b) Since plain IPv4 and VPN routes can be steered into the same SR-TE policy, do we have a standardized headend behavior in this situation?
   (Should I open a new thread to discuss this? It seems not quite MSD-relative.)

Thanks...
Nat

On Tue, Dec 17, 2019 at 7:19 AM Jeff Tantsura <[email-masked]<mailto:[email-masked]>> wrote:
Gyan,

Cheers,
Jeff

P.S. you might want to see the NANOG MSD presentation I did some time ago.
https://pc.nanog.org/static/published/meetings/NANOG71/1424/20171004_Tantsura_The_Critical_Role_v1.pdf
On Dec 14, 2019, 11:59 PM -0800, Gyan Mishra <[email-masked]<mailto:[email-masked]>>, wrote:

Jeff

Why is that?

Warm regards,

Gyan

On Sat, Dec 14, 2019 at 2:43 AM Nat Kao <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Jeff.

Regards,
Nat.

On Sat, Dec 14, 2019 at 2:42 AM Jeff Tantsura <[email-masked]<mailto:[email-masked]>> wrote:
Hi Nat,

Specifically to BMI:

Cheers,
Jeff
On Dec 13, 2019, 3:42 AM -0800, Nat Kao <[email-masked]<mailto:[email-masked]>>, wrote:

Hello, SPRING WG.

Thanks,
Nat.
_______________________________________________
spring mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/spring
_______________________________________________
spring mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/spring
--
Gyan S. Mishra
IT Network Engineering & Technology
Verizon Communications Inc. (VZ)
13101 Columbia Pike FDC1 3rd Floor
Silver Spring, MD 20904
United States
Phone: 301 502-1347
Email: [email-masked]<mailto:[email-masked]>
www.linkedin.com/in/networking-technologies-consultant<http://www.linkedin.com/in/networking-technologies-consultant>

_______________________________________________
spring mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/spring

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 17 Dec 2019 18:44:34 -0500

Hi Ketan

Is this the artificial boundary below for SR-TE from the document provided
by Jeff.

 MSD supported by different HW/SW differs widely :
– Linux (kernel 4.10): 2 SID’s, some improvements recently (as of 4.11) –
Low end off the shelf (merchant) silicon, e.g. BCM Trident2: 3-5 SID’s –
High end off the shelf (merchant) silicon, e.g BCM Jericho1 : 4-7 SID’s –
Vendor’ silicon, e.g. Juniper’ Trio, Nokia’ FP3: 4-10+ SID’s

So if you are doing MPLS-TE w/o SR-TE policies reusing they MPLS data plane
the label stack is identical to MPLS just with different label range for
SRGB 16000-23999 for absolute or indexed label.
Label stack for SR-MPLS is identical to MPLS with single topmost label for
label switching through the core and bottom of stack services label for L3
vpn services.

For clarity on nomenclature below please comment:

Transport label - This is the SID label stack insertion at the ingress PE
head end to the SR-MPLS domain performing SR-TE.  Each hop along the way
hop the top SID label is popped until SL==0 at which time PHP default or
UHP explicit null “pipe mode” is performed. (Similar to topmost in MPLS
label switch world but now its a stack of labels  instead of a single label)

Once you get past the transport label now you are on common ground with
M-BGP overlay services

Services label- L3 vpn service SID signaled.
So with the service label that is signaled from ingress to egress PE for
service.  Is this a single label like traditional L3 vpn aggregate label or
more then one or many labels.

The services label my guess should be fixed and not have any vendor
specific boundary constraints.

Gyan

On Tue, Dec 17, 2019 at 7:21 AM Ketan Talaulikar (ketant) <[email-masked]>
wrote:

> Hi Nat,
>
>
>
> The MSD framework enables us to define more/new MSD types. If there is a
> real use-case and requirement (as you express) and the necessary MSD
> type(s) can be formally defined then perhaps the WG can evaluate it.
>
>
>
> Thanks,
>
> Ketan
>
>
>
> *From:* spring <[email-masked]> *On Behalf Of *Nat Kao
> *Sent:* 17 December 2019 17:16
> *To:* Robert Raszuk <[email-masked]>
> *Cc:* SPRING WG <[email-masked]>; Nat Kao <[email-masked]>
> *Subject:* Re: [spring] Different MSDs for different traffic types on the
> same headend.
>
>
>
> Hello, Robert.
>
-- 

Gyan S. Mishra

Silver Spring, MD 20904

United States

Phone: 301 502-1347

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 17 Dec 2019 16:42:28 -0800

Robert,

While you are correct wrt single ASIC platforms, on a line-card based platforms (at least ones I’m familiar with and that includes merchant silicon)  in some case service labels were imposed on egress LC while transport on ingress, in any case, they are imposed at different stages in the pipeline. When you program HW there are clear differences and trade-offs associated.
On many NPU’s, there’s a way to recirculate packets inside ASIC/NPU to provide additional manipulations, usually comes with performance penalty, perfect use case for an additional MSD type (Cisco is doing exactly that on Jericho+ to increase MSD).
In general - as long as a device can provide any additional information that could be used by computation logic to compute the most optimized path, it is a good thing.

Cheers,
Jeff
On Dec 17, 2019, 1:59 AM -0800, Robert Raszuk <[email-masked]>, wrote:
> > > > > 1) Plain IPv4: 6 transport labels + 0 service label => traffic can be steered into a 6-label SR-TE policy..
> > > > > > On Tue, Dec 17, 2019 at 7:19 AM Jeff Tantsura <[email-masked]> wrote:
> > > > > > > Gyan,
> > > > > > >

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 17 Dec 2019 16:44:50 -0800

Hi Nat,

Please bring your use case to the wg, MSD has specifically been designed to address new types with a very little effort.

Cheers,
Jeff

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 17 Dec 2019 17:41:03 -0800

Gyan,

I wrote this presentation ~3 years ago, so the numbers are not necessarily current.

Please see inline, hope this helps you to better understand the technology behind the terminology.
Let me know if anything is unclear.

Cheers,
Jeff
On Dec 17, 2019, 3:45 PM -0800, Gyan Mishra <[email-masked]>, wrote:
>
> Hi Ketan
>
> Is this the artificial boundary below for SR-TE from the document provided by Jeff.
>
> – Linux (kernel 4.10): 2 SID’s, some improvements recently (as of 4.11) – Low end off the shelf (merchant) silicon, e.g. BCM Trident2: 3-5 SID’s – High end off the shelf (merchant) silicon, e.g BCM Jericho1 : 4-7 SID’s – Vendor’ silicon, e.g. Juniper’ Trio, Nokia’ FP3: 4-10+ SID’s
>
> So if you are doing MPLS-TE w/o SR-TE policies reusing they MPLS data plane the label stack is identical to MPLS just with different label range for SRGB 16000-23999 for absolute or indexed label.
[jeff]SRGB is either configurable or a static label range, it doesn’t have to be identical with SR domain, indexing provides the abstraction (and there’s life outside of Cisco ;-))
> Label stack for SR-MPLS is identical to MPLS with single topmost label for label switching through the core and bottom of stack services label for L3 vpn services.
[jeff] NH in the VPN route is recursively resolved to a LSP (or policy in modern lingo) that could have 1 or more SID’s associated with it (). In MPLS case, without TE, would be the label/N-SID of the egress PE, the SID stack would grow any time you need to use Adj-SID’s to deviate from SPT. Specifically to SR, egress PE can also associate a particular path (color) with the route and signal to the ingress PE what path to take in order to reach it (NH).
> Transport label - This is the SID label stack insertion at the ingress PE head end to the SR-MPLS domain performing SR-TE.  Each hop along the way hop the top SID label is popped until SL==0 at which time PHP default or UHP explicit null “pipe mode” is performed. (Similar to topmost in MPLS label switch world but now its a stack of labels  instead of a single label)
[jeff] There’s no SL field in MPLS header, there’s however BoS bit that should be set on the most bottom label (in this case - VPN label), PHP is explicitly signaled by the egress PE (P/NP-flag in Prefix-SID)
>
> Once you get past the transport label now you are on common ground with M-BGP overlay services
>
> So with the service label that is signaled from ingress to egress PE for service.  Is this a single label like traditional L3 vpn aggregate label or more then one or many labels.
[jeff] there’s no differences with any other VPNoverFoo, service labels are signaled in exactly same way independently of the transport and are locally significant (that’s why LxVPN’s are such a great technology)
>
> The services label my guess should be fixed and not have any vendor specific boundary constraints.
[jeff] they are locally significant, in any implementation - label blocks are associated with particular functions and programmed in HW, so the service label would come out of the service range, which is significant to the device advertising it.
In modern OS’s the ranges are configurable, in older ones, it is static and there’s usually a header file that looks like:
#define MPLS_L3VPN_MIN_LABEL  0x40000
#define MPLS_L3VPN_MAX_LABEL 0x85FFF

> Gyan
>
> > On Tue, Dec 17, 2019 at 7:21 AM Ketan Talaulikar (ketant) <[email-masked]> wrote:
> > > Hi Nat,
> > >

---

From: Nat Kao <[email-masked]>
Date: Wed, 18 Dec 2019 14:20:35 +0800

Hi, Jeff & Ketan.

For better discussion, let's assume these first:

"TP-MSD" denotes the maximum transport label depth for traffic without
service labels.
"SRV-TP-MSD" denotes the maximum transport label depth for traffic with 1+
service label(s).

Here's one of the use cases:

A customized best-effort TE process:

1. Decide which route R (incl. VPN routes) to diverge from its
shortest-path.

2. Compute SID-list S for R that satisfies the (TE or whatever) constraints.

3a. If R is a plain IPv4/IPv6 route, we consult TP-MSD.
    If length(S) <= headend's TP-MSD:
        Inject an SR-TE policy candidate path P with color X by BGP.
        Inject a new route R' with color X and larger localpref value.
    else:
        This path is not feasible, consider another route.

3b. If R is a service route, we consult SRV-TP-MSD.
    If length(S) <= headend's SRV-TP-MSD:
        Inject an SR-TE policy candidate path P with color X by BGP.
        Inject a new route R' with color X and larger localpref value.
    else:
        This path is not feasible, consider another route.

4. Repeat 1 to 3 until the main objectives are met.(best-effort approach,
some might fail)

Notes:

a) Clearly, BMI-MSD >= TP-MSD >= SRV-TP-MSD
b) TP-MSD & SRV-TP-MSD can be signaled by the same methods as BMI-MSD.
c) If a change on headend that modifies transport label depths(ex:
recirculation), TP-MSD & SRV-TP-MSD shall reflect that change accordingly.
d) If a change on headend that modifies transport label depths of certain
LCs, TP-MSD & SRV-TP-MSD of those links shall reflect that change, too.

Hope this helps.

Regards,
Nat

On Wed, Dec 18, 2019 at 8:44 AM Jeff Tantsura <[email-masked]>
wrote:

> Hi Nat,
>
> Please bring your use case to the wg, MSD has specifically been designed
> to address new types with a very little effort.
>
> Cheers,
> Jeff
> *From:* spring <[email-masked]> *On Behalf Of* Nat Kao

---

From: Gyan Mishra <[email-masked]>
Date: Wed, 18 Dec 2019 01:27:38 -0500

Thanks Jeff!!

Kind regards

I have a personal Cisco VIRL server server which I just finished building
out the SP topology with two SP SR domains.  Cisco XRv supports SRv6 in
6.6.1 so I have loaded the latest image 7.0.1 so hopefully have all
features.  Plan to get it all up and running tomorrow.

Question-  Since SR-MPLS uses an IPv4 data plane and since SRv6 uses IPv6
data plane is it possible to run SRv6 as an overlay both SR-MPLS and SRv6
together as two ships in the night.  Would save on having to go back and
forth between simulations if they can both live together  so I can test
both simultaneously.

In-line questions follow up

On Tue, Dec 17, 2019 at 8:41 PM Jeff Tantsura <[email-masked]>
wrote:

> So if you are doing MPLS-TE w/o SR-TE policies reusing they MPLS data
> plane the label stack is identical to MPLS just with different label range
> for SRGB 16000-23999 for absolute or indexed label.
>
> [jeff]SRGB is either configurable or a static label range, it doesn’t have
> to be identical with SR domain, indexing provides the abstraction (and
> there’s life outside of Cisco ;-))
>
> [jeff] NH in the VPN route is recursively resolved to a LSP (or policy in
> modern lingo) that could have 1 or more SID’s associated with it (). In
> MPLS case, without TE, would be the label/N-SID of the egress PE, the SID
> stack would grow any time you need to use Adj-SID’s to deviate from SPT.
> Specifically to SR, egress PE can also associate a particular path (color)
> with the route and signal to the ingress PE what path to take in order to
> reach it (NH).
>

    [gyan] So if you are running just SR-MPLS using SPF path and don’t
deviate from that then with prefix/node SID which I believe is like the
egress PE ibgp NH loopback FEC destination, that you have a SID label
binding for your LSP to build topmost reusing the MPLS data plane.  In the
case described there would be only 1 transport label.  Correct?

You would only have an Adj-SID if you are doing SR-TE.  Correct?  In that
case if you had one Adj-SID in the middle of the path like a loose explicit
route object ERO, then would still be only 1 transport label in the stack
as if you were not doing SR-TE.  Or would it be 2 labels, not sure.

So let’s say If your SR-TE path deviated from the spf along 10 hops, so
then you would have 10 Adj-SIDs and so 10 labels in the stack.  So this is
where MSD and BGP-LS IGP extensions come into play to signal back to the
head end that the maximum SID depth has exceeded resulting in an error.  At
that point the
SR-TE LSP would be broken and you would have to manually build policy with
less strict hops.

I guess If you were using PCEP centralized controller that would signal MSD
exceeded but I think you would still be in a broken LSP state.  The pcep
controller would have to be sitting next to the head end for the SR-TE
policy for the LSP to fix automatically.  Is that correct?

So for SR-TE due to the MSD issues and SID depth asic limitations even with
the latest asic, you really have to deploy a pcep controller based
architecture near the head end with SR-MPLS if using SR-TE.  If you are not
using SR-TE then I believe there is really no benefit of going to SR-MPLS.

You agree.

I guess you save on having to maintain ldp state on each node.  Thats  not
a big deal or business justification to switch to SR-MPLS as for operators
“MPLS” is their bread & butter they are use to so why change.

I think the big benefit for both SR-MPLS and SRv6 is individual coloring of
vpn flows.  Cisco calls it per-VRF TE, but it’s ugly to configure and
manage.

So SR-MPLS and SRv6 is the future solution for operators that need the per
vpn coloring.

Given the issues with SR-TE with SID depth asic limitations is it possible
to use traditional RSVP TE.

I am going to try in my virl setup as that could be a good workaround for
SR-TE hardware limitations until the hardware is up to snuff to handle very
long explicit paths.

I believe their is a draft with teas in the works for IP TE to use rsvp
bandwidth signaling path and reserve messages with both SR-MPLS and SRv6.

> [jeff] There’s no SL field in MPLS header, there’s however BoS bit that
> should be set on the most bottom label (in this case - VPN label), PHP is
> explicitly signaled by the egress PE (P/NP-flag in Prefix-SID)
>
>
> [jeff] there’s no differences with any other VPNoverFoo, service labels
> are signaled in exactly same way independently of the transport and are
> locally significant (that’s why LxVPN’s are such a great technology)
>
>
> [jeff] they are locally significant, in any implementation - label blocks
> are associated with particular functions and programmed in HW, so the
> service label would come out of the service range, which is significant to
> the device advertising it.
> In modern OS’s the ranges are configurable, in older ones, it is static
> and there’s usually a header file that looks like:
>> *Subject:* Re: [spring] Different MSDs for different traffic types on
>> the same headend.
>>
>>
>>
>> Hello, Robert.
>>
>> Label lookup results in pointer to a adj. rewrite on the outbound side.
>> It really does not matter if the adjacent peer is your P router, segment
>> endpoint or CE.
>>
>>
>>
>> Base MPLS Imposition MSD (BMI-MSD) signals the total number of
>> MPLS labels that can be imposed, including all
>> service/transport/special labels.
>>
>> <https://www..google.com/maps/search/13101+Columbia+Pike?entry=gmail&source=g>
>> FDC1 3rd Floor
>>
>> Silver Spring, MD 20904
>>
>> United States
>>
>> Phone: 301 502-1347
>>

Gyan S. Mishra

Silver Spring, MD 20904

United States

Phone: 301 502-1347

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 18 Dec 2019 16:13:17 -0800

Pleasure, see inline
/*somewhat philosophical ;-)

Cheers,
Jeff
On Dec 17, 2019, 10:28 PM -0800, Gyan Mishra <[email-masked]>, wrote:
>
> Thanks Jeff!!
>
> Kind regards
>
> I have a personal Cisco VIRL server server which I just finished building out the SP topology with two SP SR domains.  Cisco XRv supports SRv6 in 6.6.1 so I have loaded the latest image 7.0.1 so hopefully have all features.  Plan to get it all up and running tomorrow.
>
> Question-  Since SR-MPLS uses an IPv4 data plane and since SRv6 uses IPv6 data plane is it possible to run SRv6 as an overlay both SR-MPLS and SRv6 together as two ships in the night.  Would save on having to go back and forth between simulations if they can both live together  so I can test both simultaneously.
[jeff] I’m not sure what you mean by overlay, just run ipv4+mpls for SR-MPLS and ipv6 for SRv6, perhaps different interfaces rather than dual stack
> > On Tue, Dec 17, 2019 at 8:41 PM Jeff Tantsura <[email-masked]> wrote:
> > > Gyan,
> > >
> >     [gyan] So if you are running just SR-MPLS using SPF path and don’t deviate from that then with prefix/node SID which I believe is like the egress PE ibgp NH loopback FEC destination, that you have a SID label binding for your LSP to build topmost reusing the MPLS data plane.  In the case described there would be only 1 transport label.  Correct?
[jeff] yes, the only difference with LDP/RSVP would be the “globally significant label” (N-SID of egress PE), so there’s no swap (there is but to the same value)
> >
> > You would only have an Adj-SID if you are doing SR-TE.  Correct?  In that case if you had one Adj-SID in the middle of the path like a loose explicit route object ERO, then would still be only 1 transport label in the stack as if you were not doing SR-TE.  Or would it be 2 labels, not sure.
[jeff] if Adj-SID is locally significant (L flag not set) (default in most implementation) is must be evaluated only by the node that it is local to, which should result in 2 label construct (N-SID,Adj-SID), there are ways to optimize but this is the idea.
> >
> > So let’s say If your SR-TE path deviated from the spf along 10 hops, so then you would have 10 Adj-SIDs and so 10 labels in the stack.  So this is where MSD and BGP-LS IGP extensions come into play to signal back to the head end that the maximum SID depth has exceeded resulting in an error.  At that point the
> > SR-TE LSP would be broken and you would have to manually build policy with less strict hops.
[jeff] depends on implementation, it could as well be 20. No, this is other way around - (simplified version) BGP-LS distributes SR related data to the PCE and then PCE computes a path that meets the constrains, MSD being one of them, in other words - if PCE can’t compute a path to a destination that meets the constrains, it will not be instantiated
> >
> > I guess If you were using PCEP centralized controller that would signal MSD exceeded but I think you would still be in a broken LSP state.  The pcep controller would have to be sitting next to the head end for the SR-TE policy for the LSP to fix automatically.  Is that correct?
[jeff]see previous answer, if MSD (or any other) constrains can’t be met, the path is not going to be instantiated
PCEP is the protocol between PCE and PCC and has little to do with computation logic, PCE could compute a path but use for example RESTCONF to communicate with the head-end
> >
> > So for SR-TE due to the MSD issues and SID depth asic limitations even with the latest asic, you really have to deploy a pcep controller based architecture near the head end with SR-MPLS if using SR-TE.  If you are not using SR-TE then I believe there is really no benefit of going to SR-MPLS.
[jeff] you mean PCE controller. I disagree with you, reducing number of protocols in the network is a great goal (reduces complexity and number of interactions between them). Look at RSVP - there are many people using it but a only little part of them is actually doing TE, most use it for FRR and tactical TE.
To cite a great book ;-) - complexity doesn’t disappear, it moves to different places. In SR we have significantly reduced amount of  state and complexity associated with maintaining it in the network, but it is still there, it has just moved to a controller. In RSVP for example, you could do in-band  BW reservations, in SR it is “soft” controller has to do it and then network gets configured. Multicast works in IP because there’s enough state to compute loop-free MDT and so forth
> >
> > You agree.
> >
> > I guess you save on having to maintain ldp state on each node.  Thats  not a big deal or business justification to switch to SR-MPLS as for operators “MPLS” is their bread & butter they are use to so why change.
[jeff] No, I don’t agree -  the value is not in removing LDP state but reducing overall complexity, interactions between IGPs and LDP are complex and error prone, over the time resulted in 100’s of high severity bugs and broken networks (if you are old enough you should remember :))
> >
> > I think the big benefit for both SR-MPLS and SRv6 is individual coloring of vpn flows.  Cisco calls it per-VRF TE, but it’s ugly to configure and manage.
[jeff] well, complain to your account, building networking features is quite different from intuitive interfaces ;-) color is a piece of metadata associated with a route, on itself not directly related to SR, nothing prevents you of signaling a RSVP LSP or similar, there’s some work on generalization of colored path advertisement which I find very valuable (draft-chan-idr-bgp-lu2/  draft-szarecki-idr-bgp-lcu-traffic-steering)
> >
> > So SR-MPLS and SRv6 is the future solution for operators that need the per vpn coloring.
[jeff] different people have different requirements, our job here is to build technology that is flexible enough to address them at scale while manageable.
> >
> > Given the issues with SR-TE with SID depth asic limitations is it possible to use traditional RSVP TE.
[jeff] I don’t think you are right, while limitations are there, there are solvable, by a better silicon (or trading off capacity for functionality - looping/recycling packets), by a more clever computation, by brining some of the state back (take a look at draft-chunduri-lsr-isis-preferred-path-routing)
> >
> > I am going to try in my virl setup as that could be a good workaround for SR-TE hardware limitations until the hardware is up to snuff to handle very long explicit paths.
> >
> > I believe their is a draft with teas in the works for IP TE to use rsvp bandwidth signaling path and reserve messages with both SR-MPLS and SRv6.
[jeff] good engineering is always a right balance of trade-offs, extremities on either side (no state/all the state in a packet) (OpenFlow is a great example) are rigid and won’t survive when deployed at scale.

---

From: Gyan Mishra <[email-masked]>
Date: Wed, 18 Dec 2019 21:12:17 -0500

Thank you Jeff for taking the time for the detailed response!!

Cheers & Happy Holidays!!

Gyan

On Wed, Dec 18, 2019 at 7:13 PM Jeff Tantsura <[email-masked]>
wrote:

> [jeff] I’m not sure what you mean by overlay, just run ipv4+mpls for
> SR-MPLS and ipv6 for SRv6, perhaps different interfaces rather than dual
> stack
>
>
>> [jeff]SRGB is either configurable or a static label range, it doesn’t
>> have to be identical with SR domain, indexing provides the abstraction (and
>> Label stack for SR-MPLS is identical to MPLS with single topmost label
>> for label switching through the core and bottom of stack services label for
>> L3 vpn services.
>>
> [jeff] yes, the only difference with LDP/RSVP would be the “globally
> significant label” (N-SID of egress PE), so there’s no swap (there is but
> to the same value)
>
>
> [jeff] if Adj-SID is locally significant (L flag not set) (default in most
> implementation) is must be evaluated only by the node that it is local to,
> which should result in 2 label construct (N-SID,Adj-SID), there are ways to
> optimize but this is the idea.
>
>
> [jeff] depends on implementation, it could as well be 20. No, this is
> other way around - (simplified version) BGP-LS distributes SR related data
> to the PCE and then PCE computes a path that meets the constrains, MSD
> being one of them, in other words - if PCE can’t compute a path to a
> destination that meets the constrains, it will not be instantiated
>
>
> I guess If you were using PCEP centralized controller that would signal
> MSD exceeded but I think you would still be in a broken LSP state.  The
> pcep controller would have to be sitting next to the head end for the SR-TE
> PCEP is the protocol between PCE and PCC and has little to do with
> computation logic, PCE could compute a path but use for example RESTCONF to
> communicate with the head-end
>
>
> So for SR-TE due to the MSD issues and SID depth asic limitations even
> with the latest asic, you really have to deploy a pcep controller based
> [jeff] you mean PCE controller. I disagree with you, reducing number of
> protocols in the network is a great goal (reduces complexity and number of
> interactions between them). Look at RSVP - there are many people using it
> but a only little part of them is actually doing TE, most use it for FRR
> and tactical TE.
> To cite a great book ;-) - complexity doesn’t disappear, it moves to
> different places. In SR we have significantly reduced amount of  state and
> complexity associated with maintaining it in the network, but it is still
> there, it has just moved to a controller. In RSVP for example, you could do
> in-band  BW reservations, in SR it is “soft” controller has to do it and
> then network gets configured. Multicast works in IP because there’s enough
> state to compute loop-free MDT and so forth
>
>
> You agree.
>
> [jeff] No, I don’t agree -  the value is not in removing LDP state but
> reducing overall complexity, interactions between IGPs and LDP are complex
> and error prone, over the time resulted in 100’s of high severity bugs and
> broken networks (if you are old enough you should remember :))
>
>
> I think the big benefit for both SR-MPLS and SRv6 is individual coloring
> of vpn flows.  Cisco calls it per-VRF TE, but it’s ugly to configure and
> manage.
>
> [jeff] well, complain to your account, building networking features is
> quite different from intuitive interfaces ;-) color is a piece of metadata
> associated with a route, on itself not directly related to SR, nothing
> prevents you of signaling a RSVP LSP or similar, there’s some work on
> generalization of colored path advertisement which I find very valuable
> (draft-chan-idr-bgp-lu2/  draft-szarecki-idr-bgp-lcu-traffic-steering)
>
>
> [jeff] different people have different requirements, our job here is to
> build technology that is flexible enough to address them at scale while
> manageable.
>
>
> [jeff] I don’t think you are right, while limitations are there, there are
> solvable, by a better silicon (or trading off capacity for functionality -
> looping/recycling packets), by a more clever computation, by brining some
> of the state back (take a look
> at draft-chunduri-lsr-isis-preferred-path-routing)
>
>
> [jeff] good engineering is always a right balance of trade-offs,
> extremities on either side (no state/all the state in a packet) (OpenFlow
> is a great example) are rigid and won’t survive when deployed at scale.
>
>
>
>>> Of course there are number of processing exceptions - for example
>>> concept of aggregate VPN label, additional header modifications etc ...
>>>
>>>
>>>
>>> a) As defined in RFC8491, the BMI-MSD is 6 for this headend. Do we have
>>> a standardized way to signal the transport label depth in mode 2?
>>>

Gyan S. Mishra

Silver Spring, MD 20904

United States

Phone: 301 502-1347