Subject: [spring] to drop or to forward unlabelled (Re: Question on RFC8660)

From: Martin Horneffer <[email-masked]>
Date: Thu, 27 Aug 2020 12:34:33 +0200

Hello everyone,

may I come back the the question below? Or rather let me update it a little:

In case an SR-MPLS path is broken, should a node rather drop the packet, 
or forward it?
This can happen whenever the IGP points to a certain next hop, but that 
neither supplies a valid SID, nor allows LDP-stitching for whatever 
reason. For PUSH as well as for CONTINUE.

We have been using MPLS transport and a BGP free core since about two 
decades now, using LDP. In the analog case, LDP creates "unlabelled" 
entries in the LFIB, does the equivalent of a POP operation and forwards 
the packet to the next-hop as chosen by the IGP.

This behavior obviously breaks any traffic that relies on a service 
label, but it can protect some traffic.
In our case a huge percentage of all traffic still is public IPv4. This 
needs MPLS only for a transport label, be it LDP or SR-MPLS. If this 
traffic gets forwarded unlabelled, it follows an IGP default route to a 
central device, where it is 1) redirected to the correct destination and 
2) counted in a way that operators can quickly see whether and where 
this kind of failure occurs at some point in the network.

After more operational experience and several internal discussions we 
agreed that we want packets to be forwarded unlabelled rather than 
dropped. Anyone to share, or oppose this position?

Best regards, Martin

Am 31.01.20 um 16:50 schrieb Martin Horneffer:
> Hello everyone,
>
> again it seems the interesting questions only show up when applying 
> something to the live network...
>
> We ran into something that poses a question related to RFC8660: What 
> is the exact meaning of section 2.10.1, "Forwarding for PUSH and 
> CONTINUE of Global SIDs", when the chosen neighbor doesn't provide a 
> valid MPLS path?
>
> The relevant sections reads:
>
>       -  Else, if there are other usable next hops, use them to forward
>          the incoming packet.  The method by which the router "R0"
>          decides on the possibility of using other next hops is beyond
>          the scope of this document.  For example, the MCC on "R0" may
>          chose the send an IPv4 packet without pushing any label to
>          another next hop.
>
> Does the part "send an IPv4 packet without pushing any label" apply to 
> PUSH and CONTINUE, or just to PUSH?
> Does R0 have to validate that neighbor N can correctly process to 
> packet? Or can it forward the packet regardless?
>
> The reason for asking is that we are now seeing issues similar to ones 
> we had when starting with LDP based MPLS about two decades ago: 
> traffic being black holed even though a path to the destination 
> exists, because the MPLS path is interrupted somewhere in the middle.
>
> With LDP we know the case of LFIBentries called "unlabelled". While 
> this does break connectivity for many kinds of service, e.g. those 
> relying on an additional service labels, it still works for plain 
> IP(v4) traffic. In our cases, this works perfectly fine for all 
> internal routing and control traffic. And even for IPv4 traffic that 
> gets collected by a central router that injects a default route.
>
> However, depending on the exact interpretation of the above paragraph, 
> an implementor might feel obliged to chose the next paragraph:
>
>       -  Otherwise, drop the packet.
>
> Which is, at least in our case, very unfortunate...
>
> Any advice or opinion appreciated!
>
>
> Best regards, Martin
>
>
>
>
> _______________________________________________
> spring mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spring

---

From: "Ketan Talaulikar (ketant)" <[email-masked]>
Date: Thu, 27 Aug 2020 11:10:31 +0000

Hi Martin,

I share your position.

Thanks,
Ketan

-----Original Message-----
From: spring <[email-masked]> On Behalf Of Martin Horneffer
Sent: 27 August 2020 16:05
To: [email-masked]
Subject: [spring] to drop or to forward unlabelled (Re: Question on RFC8660)

Hello everyone,

In case an SR-MPLS path is broken, should a node rather drop the packet, or forward it?
This can happen whenever the IGP points to a certain next hop, but that neither supplies a valid SID, nor allows LDP-stitching for whatever reason. For PUSH as well as for CONTINUE.

We have been using MPLS transport and a BGP free core since about two decades now, using LDP. In the analog case, LDP creates "unlabelled" 
entries in the LFIB, does the equivalent of a POP operation and forwards the packet to the next-hop as chosen by the IGP.

This behavior obviously breaks any traffic that relies on a service label, but it can protect some traffic.
In our case a huge percentage of all traffic still is public IPv4. This needs MPLS only for a transport label, be it LDP or SR-MPLS. If this traffic gets forwarded unlabelled, it follows an IGP default route to a central device, where it is 1) redirected to the correct destination and
2) counted in a way that operators can quickly see whether and where this kind of failure occurs at some point in the network.

After more operational experience and several internal discussions we agreed that we want packets to be forwarded unlabelled rather than dropped. Anyone to share, or oppose this position?

Best regards, Martin

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 27 Aug 2020 14:45:39 +0200

Martin,

> it follows an IGP default route to a central device,

So putting aside that such default must point domian wide to the same
address (could be anycast if you are careful) once this "central device"
receives a packet and does a BGP full table lookup it will again try to
encapsulate it in MPLS towards exit.

What if the path is again via a device which breaks LDP LSP or SR path and
the packet will again go back to central device creating a very nice loop ?

Sure central device could give up on MPLS all together and IP encap to
egress via your BGP free core - but you are not mentioning that.

Bottom line if you would not have BGP free core (or Internet route free
core) I would say sure good idea to continue. But since you do I think this
is a lot of hidden traps number of networks may fall into by doing it. So
at least it should not be a default behaviour.

Thx,
R.

---

From: <[email-masked]>
Date: Fri, 28 Aug 2020 11:31:38 +0800 (CST)

Hi Martin,

For the matched LFIB entry with unlabeled outgoing forwarding information, I agree your opinion that public traffic need to be forwarded. 

I think the above LFIB entry can easiely drop VPN traffic according to no bottom flag of incomming LDP/SR label.

Regards,

PSF

原始邮件

发件人：MartinHorneffer <[email-masked]>
收件人：[email-masked] <[email-masked]>;
日 期 ：2020年08月27日 18:35
主 题 ：[spring] to drop or to forward unlabelled (Re: Question on RFC8660)

Hello everyone,

Best regards, Martin

---

From: Martin Horneffer <[email-masked]>
Date: Fri, 28 Aug 2020 12:10:28 +0200

Hi Robert,

the setup I sketched does not cover double failures nor 100 % of all 
topological cases for forwarding.
And in fact, forwarding the traffic is not the main purpose. The main 
purpose it to DETECT the failure in a useful way.

For the same reason we would sure not want to add yet another 
encapsulation technology.

But, of course, if the behavior (drop or forward unlabelled) would be 
configurable, that would be perfectly ok from my point of view.

Best regards, Martin

Am 27.08.20 um 14:45 schrieb Robert Raszuk:
> address (could be anycast if you are careful) once this "central 
> device" receives a packet and does a BGP full table lookup it will 
> again try to encapsulate it in MPLS towards exit.
>
> What if the path is again via a device which breaks LDP LSP or SR 
> path and the packet will again go back to central device creating a 
> very nice loop ?
>
> Bottom line if you would not have BGP free core (or Internet route 
> free core) I would say sure good idea to continue. But since you do I 
> think this is a lot of hidden traps number of networks may fall into 
> by doing it. So at least it should not be a default behaviour.
>
> Thx,
> R.
>
>
>
>
>
> On Thu, Aug 27, 2020 at 12:35 PM Martin Horneffer <[email-masked] 
> <mailto:maho@lab..dtag.de>> wrote:
>
>     Hello everyone,
>
>     > With LDP we know the case of LFIBentries called "unlabelled"..
>     While
>     > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Martin Horneffer <[email-masked]>
Date: Fri, 28 Aug 2020 12:13:23 +0200

Hi PSF,

limiting the "forward unlabelled" behavior to packets with the 
bottom-of-stack bit set is an interesting idea. I haven't though about 
that yet. In my case that would definitely help and be perfectly ok.

Best regards, Martin

Am 28.08.20 um 05:31 schrieb [email-masked]:
>
>
> Hi Martin,
>
>
> For the matched LFIB entry with unlabeled outgoing forwarding 
> information, I agree your opinion that public traffic need to be 
> forwarded.
>
> I think the above LFIB entry can easiely drop VPN traffic according to 
> no bottom flag of incomming LDP/SR label.
>
>
> Regards,
>
> PSF
>
>
>
> 原始邮件
> *发件人：*MartinHorneffer <[email-masked]>
> *收件人：[email-masked] <[email-masked]>;
> *日 期 ：*2020年08月27日 18:35
> *主 题 ：**[spring] to drop or to forward unlabelled (Re: Question on 
> RFC8660)*
> Hello everyone,
>

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 28 Aug 2020 18:31:08 +0000

It all hinges on "useable next hops". If the connected next hop
(a bgp next hop may not be connected) has a correct route for the
packet, then it is useable.

One more consideration: With the BGP free core, external devices
cannot send packets to your internal routers. Once you allow to
forward packets without a labelled route, you need to make sure that
this protection remains.

Regards,
Jakob.

-----Original Message-----
From: spring <[email-masked]> On Behalf Of Martin Horneffer
Sent: Thursday, August 27, 2020 3:35 AM
To: [email-masked]
Subject: [spring] to drop or to forward unlabelled (Re: Question on RFC8660)

Hello everyone,

Best regards, Martin

---

From: Tarek Saad <[email-masked]>
Date: Sun, 30 Aug 2020 00:36:52 +0000

Hi Martin,

See inline for some comments.

On 8/27/20, 6:35 AM, "spring on behalf of Martin Horneffer" <[email-masked] on behalf of [email-masked]> wrote:

    Hello everyone,

[TS]: The SR Path may be composed of multiple SIDs (i.e. label stack) -- where the top SID destination is not the SR Path endpoint.. I sense from "gets forwarded unlabelled " as you will discard the full label stack (?) and forward the packet unlabeled to the central device? Or are you encapsulating the remaining MPLS packet over IP and IP routing packet to the central device to be inspected? In either case, I expect somehow the packet to be ultimately delivered to the original SR Path endpoint/egress?

    After more operational experience and several internal discussions we 
    agreed that we want packets to be forwarded unlabelled rather than 
    dropped. Anyone to share, or oppose this position?
[TS]: again, I'm not clear on when you say "forward unlabelled" - do you mean packets will follow the top SID's the IP IGP route? Or are you peaking into the destination IP (public IPv4) of packet encapsulated in MPLS to determine how to forward it?

Regards,
Tarek

    Best regards, Martin

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 30 Aug 2020 10:21:13 -0400

Hi Martin

That is a very common scenario that occurs where one path where LDP
neighbor has lost its label binding for FEC destinations.

Most providers don’t label all prefixes and only label interesting traffic
meaning FEC destination which is the egress endpoint loopback iBGP peers
for an LSP.

So their is an ACL that is used for label allocation, label advertisement
and label accept policy to created the label binding LIB and and Label
forwarding table LFIB.

So the LSP is built from ingress PE to egress PE and the label mapping
along the path occurs downstream unsolicited in the opposite direction of
the LSP via label mapping message from the egress PE that the LSP is being
built to - to the ingress PE from the downstream node to the upstream node.

Most operators have a BGP free and PIM free core and usually don’t dual
stack the core and the concept of softwire mesh framework RFC 5565 so 6to4
softwire would be v6 edge over a v4 core or v4 edge over a v6 core.

So with the BGP free core all PE are BGP route reflector clients and have a
peer to RRs that sit in the P core.

The P core is BGP free so only runs an IGP ISIS PE OSPF.  This point is
critical and that is that since the core is BGP free it can not perform L3
forwarding so all packets must be L2.5 label switched via topmost label
either LDP or RSVP or SR-TE.

So now that we have established the core construct let’s dig into the
problem with the label mapping being broken on a node along a path.

So with LDPv4 or LDPv6 or SR-MPLS we have a concept called “ldp-IGP”
synchronization.  SR-MPLS uses IGP sync for SR sid labels as well since SR
is reusing the MPLS data plane.

So on all MPLS core enabled interfaces on both PE and P LDP sync is
enabled.

So the goal of ldp-IGP sync is to prevent black hole of packets on MPLS
interfaces cause by MPLS LDP failure and so how it works is the IGP prefix
that needs to build a label binding for FEC destination for label swapping
to occurs in the core so the IGP must not come up and forward traffic along
the path until MPLS is up or traffic will be “black holed” and dropped.  So
how this is accomplished is the IGP sets the prefix metric to maximum 65535
for ospf and ISIS wide metric does the same so the path now cannot be
utilized until ldp comes up and the label binding message is received via
downstream unsolicited to create outgoing label for the LFIB entry.  Once
LDP is up and the LFIB entry outing label is populated the IGP prefix is
“unset” so now is set to its normal IGP metric and traffic can now flow
over the path.

For LDP troubleshooting first you have to see if discovery link LDP is up
on the interface and then have to check if both Ldp adjacencent nodes have
a route in rib for each other’s LDP router id for the ldp neighbor to come
UP and label mapping message to be sent.  As you mentioned if their is
summarization for FEC which results in a loop the binding does not get
built for the FEC resulting in unlabeled FECs in the LFIB.

Hope that explanation helps.

Kind Regards

Gyan

On Sat, Aug 29, 2020 at 8:37 PM Tarek Saad <[email-masked]> wrote:

> On 8/27/20, 6:35 AM, "spring on behalf of Martin Horneffer" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>
>
>     Hello everyone,
>
>
>
> [TS]: The SR Path may be composed of multiple SIDs (i.e. label stack) --
> where the top SID destination is not the SR Path endpoint.. I sense from
> "gets forwarded unlabelled " as you will discard the full label stack (?)
> and forward the packet unlabeled to the central device? Or are you
> encapsulating the remaining MPLS packet over IP and IP routing packet to
> the central device to be inspected? In either case, I expect somehow the
> packet to be ultimately delivered to the original SR Path endpoint/egress?
>
>
>
> [TS]: again, I'm not clear on when you say "forward unlabelled" - do you
> mean packets will follow the top SID's the IP IGP route? Or are you peaking
> into the destination IP (public IPv4) of packet encapsulated in MPLS to
> determine how to forward it?
>
>
>
> Regards,
>
> Tarek
>
>
>
>     Best regards, Martin
>
>
>
>
>
> --

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*M 301 502-134713101 Columbia Pike *Silver Spring, MD