Subject: re: Multicast vs Unicast (was BFD on LAG interfaces)

From: Pablo Frank <[email-masked]>
Date: Wed, 4 Jan 2012 11:12:13 -0500

I know there's been some arguments made in favour of option 1 that
suggest that existing ASICs may be better able to handle
IP-encapsulated BFD.  Frankly, most vendors are using flexible-enough
parsing hardware so I'm not sure this is a very strong argument
anymore.  So I decided to focus on just looking at it from a pure
software IP-stack perspective.

It seems that either variant of option 1 requires an implementer to do
some pretty ugly hacking to get around existing stack implementations.
 I tried getting 1a to work using standard Linux socket APIs and ran
into all sorts of problems.  The fundamental issue is that the member
links of a LAG are not normally IP-enabled so trying to bind an IP
multicast group membership didn't work.  The other issue was trying to
force the UDP frames out a particular underlying member link was not
possible using the existing socket options for UDP sockets.  The only
way I got it to work was using AF_PACKET raw sockets and setting up
some Berkeley Packet Filters to filter out the frames with the correct
L2 multicast MAC address, ethertype, and L3 multicast IP address
(since L2 multicast address is not specific enough).  But in this
case, I'm essentially bypassing the IP stack so I don't get to
benefit-from/reuse any of the normal IP stack capabilities (i.e. no IP
header validation, checksum checking, etc.).  This was really ugly.  I
didn't even try 1b.

Option 2, on the other hand, was trivially easy using AF_PACKET DGRAM
sockets.  It was very straightforward to bind the TBA ethertype, the
TBA multicast MAC, and the interface to use.

The fact that option 1 was so convoluted isn't surprising given the
obvious layering violations that are taking place here.  So unless
there are gigantic barriers in place to prevent us from getting a MAC
address and ethertype assigned for this kind of low-level BFD, I don't
think we should really be seriously considering either variant of
option 1.

regards,
Pablo

> The two options before the WG are:
>
> 1. Use IP encapsulation
> 2. Use direct L2 encapsulation.
>
> With (1) we have an option of (1a) Using Multicast and (1b) Using Unicast.
>
> Can we arrive on a consensus between choosing 1a and 1b?
>
> Given that the new draft has this nice Appendix that succinctly
> discusses this, can the proponents of (1b) justify why they think (1a)
> is not good enough.
>
> Lets try closing the different threads so that we can converge towards
> a solution.
>
> Toms

---

From: Robert Rennison <[email-masked]>
Date: Wed, 4 Jan 2012 15:39:43 -0500

Pablo I agree, I am sensing a lot of over-engineering going on in some
of the discussions re encapsulations, my gut feel going when I saw the
thread start was, it will quickly settle on using a new Ethertype.

I lost track of or was never really convinced of what concrete reason
folks would want to be using anything other than an L2 encaps, for
example the individual links do not have an IP address on them so then
there were debates about using IP mcast addresses and demuxing on port
, yeuch.

So on the K.I.S.S principle my vote goes too for  trying  to getting a
new Ethertype and do not try to get a spec allowing a multiple of
encaps as this has the potential to balloon out of all proportion.

Cheers

Rob Rennison

P.s  I am also a fan of sticking to having BFD be an adjunct to or  an
accelerator for LACP so have  the micro flow or per interface BFD
sessions provide an early indication to LACP that there is a problem
with the link. One still   potentially needs an overarching slower per
LAG group BFD session to keep the upper layers happy if they are
expecting BFD on the IP interface. I say potentially because if the
interface is now protected by an augmented LACP perhaps the upper
layers do not need an explicit BFD, that's a more nuanced discussion.

---

From: Jeffrey Haas <[email-masked]>
Date: Sun, 8 Jan 2012 21:28:15 -0500

[Speaking currently as an individual contributor.]

I believe that Appendix A does a very nice job in summarizing many of the
issues accompanying each of the varying proposed solutions.  However, in
catching up on the full set of threads for this topic, I have to wonder if
we're conflating two highly related set of requirements that should be
analyzed separately.

Our fundamental problem is that ethernet LAGs, e.g. when using LACP, may
blackhole traffic for large periods of time due to convergence issues in
that protocol.  Similarly, such blackholing may exist when a LAG is
statically configured.

(Clearly such issues would exist for other forms of LAGs, but our primary
discussion has centered around Ethernet.  I will largely restrict this
response to that problem space.)

It is fair to observe that in the case of LAGs managed by LACP that the real
problem is it converges too slow.  This is something that could be addressed
by IEEE.  BFD, running as an application on top of Ethernet, could be used
to inform an LACP implementation to converge faster.  IMO, in such a case,
Ethernet encapsulation of BFD makes a lot of sense.

BFD provides its functionality by exercising the forwarding path.  In the
case of a LAG used for IP, the functionality we want to see is that when a
link is up, it is forwarding IP traffic.  To me, this frames the
requirements somewhat differently than the prior discussion although many of
the same observations still hold.

Where I believe we're conflating the issues is using BFD in one
encapsulation, IP or Ethernet, to test the health of the other service.
I.e. using IP BFD to inform LACP or Ethernet BFD to inform IP forwarding
health.

Thus, if we're wanting to verify that IP unicast traffic is correctly being
forwarded across a LAG element, BFD should be sent as IP unicast.  Among
other things, we'll be using the same MAC.  If we were instead to be using
IP multicast for BFD, the MAC will be different and the fate of the BFD
session is less strongly coupled to the ability to properly forward the
traffic.

This isn't to say that I don't agree nor sympathize with the issues
documented in Appendix A of the draft.  Multicast does make things
*significantly* easier.

To take a step back, the problem space I believe we're trying to address in
the IETF (rather than in IEEE) is the fact that IP forwarding may fail
because elements of a LAG are not being taken down fast enough.

I would suggest that a per link BFD solution needs the following properties:
- Uses an encapsulation meant to exercise the forwarding path of the traffic
  it is meant to protect.  This is consistent with the original design of
  BFD.  
- SHOULD NOT alter the core BFD state machine if at all possible.  E.g., MPLS-TP
  had good reasons to vary it in some circumstances, but was pushed to be
  compliant when circumstances did not require the deviation.
- If the session bootstrapping mechanism deviates from RFC 5880/5881, it
  MUST NOT cause interoperability issues with BFD sessions on non-LAG links.
  (I.e. a different UDP destination port.)
- SHOULD be port-centric.  This means a session should not use a return path
  that does not exercise the path to be verified for that session.  While
  this is fine for traditional IP BFD, per link BFD with IP encapsulation
  taking the wrong return path would leave it in question whether it was the
  forward path or the reverse path that were damaged.
- MAY be used as inputs to an aggregated BFD session state between the two
  systems.

As I mentioned above, I'm very sympathetic with the issues that such
requirements would place upon an implementor.  There are at least two
classes of issues here:
- We now need a per-link bootstrapping mechanism different from the existing
  RFC 5880/5881.  
- BFD done on a traditional host (routing processor/routing engine/etc.)
  instead of at the line card level becomes messy due to layering issues.
  (I.e. "here's IP destined for this host, please ignore the routing
  table".)  However, I don't think this problem is terribly much worse than
  IS-IS or VRRP on such hosts.  (I agree that it is still gross.)
  
I'd like us to see a bit more discussion around requirements before
completely diving into solutions.  However, I think the discussion around
the solutions has helped to clarify the issues and the draft has done a good
job on driving that discussion.

-- Jeff

---

From: Alexander Vainshtein <[email-masked]>
Date: Mon, 9 Jan 2012 08:05:50 +0200

Rob, Pablo and all,

I concur with Rob: BFD on LAG members using a dedicated Ethertype and used as an accelerator for LACP detecting the failures of LAG members looks like the right thing to do.

Please note also that if micro-BFD sessions are used in conjunction with LACP, there is also no need for a new well-known multicast MAC address, you can use MAC address of the peer port - learned via LACP - as the destination address (and, of course, the local port MAC address as the source address). 

As Jeff has indicated in one of his messages on this thread (http://www.ietf.org/mail-archive/web/rtg-bfd/current/msg01204.html), the operators tend to use LACP with LAG in any case. As for the exceptions Jeff has mentioned, IMHO "oldish  gear" is not very relevant (it would not support BFD as well). This leaves "extremely buggy LACP implementations"... but I do not believe that providing workaround for someone's bugs is an important objective.

My 2c,
     Sasha

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Robert Rennison [[email-masked]]
Sent: Wednesday, January 04, 2012 10:39 PM
To: Pablo Frank
Cc: [email-masked]
Subject: Re: Multicast vs Unicast (was BFD on LAG interfaces)

Cheers

Rob Rennison

>> Lets try closing the different threads so that we can converge towards

This e-mail message is intended for the recipient only and contains information which is CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this transmission in error, please inform us by e-mail, phone or fax, and then delete the original and all copies thereof.

---

From: Jeff Tantsura <[email-masked]>
Date: Mon, 9 Jan 2012 01:24:42 -0500

I completely agree with Jeff's position here, my proposal is again to use
unicast IP, it addresses both single and multi session BFD over LAG while
requres minimum changes to the existing implementations, keep in mind -
any new implementation should still be interoperable with "legacy" BFD
over LAG, being RP or LC based, so no changes such as another/new UDP port.

BFD is an IP protocol and should not be used on L2 LAG implementations, I
have already described in my previous mail how IMHO messaging with
internal structures should look like (RIB/etc)
-- 
Regards,
Jeff

---

From: Tom Sanders <[email-masked]>
Date: Mon, 9 Jan 2012 14:11:05 +0530

> issues accompanying each of the varying proposed solutions.  However, in

+1

> Ethernet encapsulation of BFD makes a lot of sense.

I agree.

And btw, isnt this the problem that this draft is attempting to address?

> link is up, it is forwarding IP traffic.  To me, this frames the

Not really.

I would not want to restrict ourselves by merely saying "forwarding IP
traffic". I dont see why a solution to preclude the possibility of
supporting VPLS or VPWS in which case one could have a LAG thats used
for forwarding L2 frames.

The idea is to get rid of LACP. The idea is to use BFD for fast
detection since folks know and understand BFD. We could have achieved
the same using ethernet OAM but folks did not want to do that.

> Thus, if we're wanting to verify that IP unicast traffic is correctly being

AS i mentioned earlier, I think we'll be very myopic if we restrict
ourselves to IP unicast traffic. What prevents us from using that link
for multicast traffic or L2 traffic?

> session is less strongly coupled to the ability to properly forward the

I think micro BFD sessions are required for required for link
liveliness and not IP liveliness. If on operator wants to ensure IP
layer liveliness then they must run BFD in the Big Pipe mode
(described in Sec 4, bullet "b"). The micro BFD sessions work along
with LACP to inform the LAG manager of the current state of the links
in the LAG. They are not meant for IP connectivity. Manav and Marc - I
hope i am not too much off the mark!

BBP (Sec 2.1) uses IP encapsulation which is fine as its this thats
meant for IP connectivity verification.

> documented in Appendix A of the draft.  Multicast does make things

Which is why i was earlier rooting for multicast. However, recent
discussions on the list have got me thinking and i think i may
drifting towards BFD over pure L2 encap mode now.

> because elements of a LAG are not being taken down fast enough.

No. I think this is the disconnect. Its meant to detect fast link failures.

>  (I.e. a different UDP destination port.)

This means that you assume that BFD only works in mode described in
Sec 4, bullet "a".

I am not sure if this was discussed.

>  IS-IS or VRRP on such hosts.  (I agree that it is still gross.)

IS-IS uses a well known L2 address so its really not a violation.

> job on driving that discussion.

Yes, i agree. We all appreciate the work done by the authors of
draft-mmm-bfd-on-lags and its definitely helped clear the fog
surrounding the problem and the solution space.

I would urge the folks to first discuss whether its to be L2 or L3,
before getting into the specifics of whether it should be Unicast or
Multicast.

--
Toms

---

From: Alexander Vainshtein <[email-masked]>
Date: Mon, 9 Jan 2012 11:18:27 +0200

Tom, and all,

I strongly disagree with the following statement in your message:  "The idea is to get rid of LACP ".

Regards,
     Sasha

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Tom Sanders
> Sent: Monday, January 09, 2012 10:41 AM
> To: Jeffrey Haas
> Cc: [email-masked]
> Subject: Re: BFD on LAG requirements (was Re: Multicast vs Unicast (was
> BFD on LAG interfaces))
> 
> >

---

From: Mach Chen <[email-masked]>
Date: Mon, 09 Jan 2012 09:40:08 +0000

Hi Tom,

Good point!

One clarification about the state:
> The idea is to get rid of LACP...

The draft does not intend to replace LACP, it's an accelerator for failures detection, micro-BFD sessions could be running either with or without LACP.

I think the requirement(fast failures detection for member links of a LAG) is obvious, and our draft has listed most of the candidate solutions, existing discussions mainly focus on which solution is the right one and we hope finally the WG can converge on only one solution. 

Best regards,
Mach
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Tom Sanders
> Sent: Monday, January 09, 2012 4:41 PM
> To: Jeffrey Haas
> Cc: [email-masked]
> Subject: Re: BFD on LAG requirements (was Re: Multicast vs Unicast (was BFD
> on LAG interfaces))
> 
> >

---

From: Mach Chen <[email-masked]>
Date: Mon, 09 Jan 2012 09:55:24 +0000

Hi Jeff,

For L3 solution, based on our experience, the implementation difference is trivial between unicast and multicast, but with multicast, as summarized in the Appendix, it has many merits (e.g., no extra configurations/bootstrapping mechanisms required) but unicast has not.

I think the WG should firstly focus on discussing whether L2 or L3 solution is the right direction. 

Best regards,
Mach

> Jeff Tantsura
> Sent: Monday, January 09, 2012 2:25 PM
> To: Jeffrey Haas; [email-masked]

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 9 Jan 2012 11:44:55 +0100

Hello Jeff,

> keep in mind -

Are you talking about the existing, proprietary BFD-on-member-link implementations when you refer to "legacy BFD over LAG"?  While it is desirable for the particular companies to minimize the change we cannot make backward compatibility a requirement. 

And personally, _if_ we end up with unicast IP (we talk about the IP address configured on the LAG, I assume?) and want to use the aggregator MAC then I would very much like to see a different UDP port. Reason is section 4, case (b) in the draft-mmm.

If you talk about the single BFD over LAG (what is named BFD over Big Pipe in draft-mmm) then I agree, this BFD session should be RFC5880/RFC5881 with no modification.

Regards, Marc

--
Marc Binderberger           <[email-masked]>

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 9 Jan 2012 12:02:02 +0100

Hello Alexander,

> Please note also that if micro-BFD sessions are used in conjunction with LACP, there is also no need for a new well-known multicast MAC address, you can use MAC address of the peer port - learned v…

true but it binds the solution to LACP as a must-have. While I agree that LACP is usually used by the customers it also means we have to invent another solution for the few customers that don't use LACP (for whatever reason).

Requesting a well-known MAC is just clean to me for the layer-2 solution.

Regarding layer-2 vs. layer-3: I agree the first thought may be "BFD in Ethernet" when looking at the problem. And nothing wrong with it. The layer-3 topic comes from additional "requirements" from customers that want to see BFD is related to layer-3 processing. So we try to square a circle (or circle a square?).

Regards, Marc

---

From: Alexander Vainshtein <[email-masked]>
Date: Mon, 9 Jan 2012 12:29:54 +0200

Mach and all,
One rather obvious problem with the IP-based encapsulation (multicast or unicast) is IMHO that it is a clear layer violation.
Pablo has already commented on implementation problems with this approach in his message ( http://www.ietf.org/mail-archive/web/rtg-bfd/current/msg01207.html) and their origin in the said layer violation.

I also have certain doubts interpreting the statements like " BFD provides its functionality by exercising the forwarding path" in Jeffrey's message (http://www.ietf.org/mail-archive/web/rtg-bfd/current/msg01212.html ). 
The forwarding path for IP traffic sent treats LAG as the interface (or, in the 802.1AX terms, IP is a client of the LAG MAC - see Figure 5.3 in the IEEE 802.1AX-2008). This means that:
- Source MAC address of IP packets transmitted via LAGF is MAC address of the LAG (does not depend on the specific LAG member over which it is actually transmitted)
- Destination MAC address of the unicast IP packet is MAC address of the peer LAG (discovered via ARP for IPv4 or ND for IPv6)
- IP layer does not exercise any kind of explicit control over selection of the LAG member over which the packet. Instead selection is done by the LAG distributor.  In fact, with most distributors, all BFD packets would be transmitted via a single LAG member link because they would be identified as belonging to the same flow. And BFD that exercises this forwarding path is what draft-mmm calls the "Big Pipe BFD session".  (BTW, Big Pipe BFD that distribute BFD packets over all LAG members actually exercise a different forwarding path).

 Did I miss something?

Regards,
     Sasha

> Behalf Of Mach Chen
> Sent: Monday, January 09, 2012 11:55 AM
> To: Jeff Tantsura; Jeffrey Haas; [email-masked]
> Subject: RE: BFD on LAG requirements (was Re: Multicast vs Unicast (was
> BFD on LAG interfaces))
> 
> Hi Jeff,
> 
> For L3 solution, based on our experience, the implementation difference is
> trivial between unicast and multicast, but with multicast, as summarized in
> the Appendix, it has many merits (e.g., no extra
> configurations/bootstrapping mechanisms required) but unicast has not.
> 
> Behalf Of
> BFD
> > >requirements somewhat differently than the prior discussion although
> many
> > >of

---

From: Alexander Vainshtein <[email-masked]>
Date: Mon, 9 Jan 2012 14:14:25 +0200

Marc hi,
Lots of thanks for a prompt response.

Please see a couple of comments inline below.

Regards,
     Sasha

> From: Marc Binderberger [mailto:[email-masked]]
> Sent: Monday, January 09, 2012 1:02 PM
> To: Robert Rennison; Alexander Vainshtein; Pablo Frank
> Cc: [email-masked]
> > Please note also that if micro-BFD sessions are used in conjunction with
> > LACP, there is also no need for a new well-known multicast MAC address,
> > you can use MAC address of the peer port - learned via LACP - as the
> > destination address (and, of course, the local port MAC address as the
> > source address).
> 
> true but it binds the solution to LACP as a must-have. While I agree that
> LACP is usually used by the customers it also means we have to invent
> another solution for the few customers that don't use LACP (for whatever
> reason).
> 
[[[Sasha]]]  I am OK with using a multicast MAC address from the IETF block if the peer port address is not known.
If it is known, using it seems a reasonable choice as well. This is not a big issue anyway.
> 
> Regarding layer-2 vs. layer-3: I agree the first thought may be "BFD in
> Ethernet" when looking at the problem. And nothing wrong with it. The
> layer-3 topic comes from additional "requirements" from customers that
> want to see BFD is related to layer-3 processing. So we try to square a circle
> (or circle a square?).
>
[[[Sasha]]] I fully agree with you on this point. I have already explained (in another email) that only the Big Pipe BFD session would exercise the same forwarding path as true IP traffic. 

IMHO and FWIW forcing Layer 3 into micro-BFD sessions does not provide any tangible benefits but definitely adds implementation and, potentially, operational problems - because, at the end of the day, they are NOT Layer 3. 

Once could argue whether IETF is the right place to define something that is not Layer 3 - but this is a different story.

> > I concur with Rob: BFD on LAG members using a dedicated Ethertype and
> used as an accelerator for LACP detecting the failures of LAG members looks
> like the right thing to do.
> >
> > Please note also that if micro-BFD sessions are used in conjunction with
> LACP, there is also no need for a new well-known multicast MAC address,
> you can use MAC address of the peer port - learned via LACP - as the
> destination address (and, of course, the local port MAC address as the
> source address).
> >
> > As Jeff has indicated in one of his messages on this thread
> (http://www.ietf.org/mail-archive/web/rtg-bfd/current/msg01204.html),
> the operators tend to use LACP with LAG in any case. As for the exceptions
> Jeff has mentioned, IMHO "oldish  gear" is not very relevant (it would not
> support BFD as well). This leaves "extremely buggy LACP
> implementations"... but I do not believe that providing workaround for
> someone's bugs is an important objective.
> >
> > My 2c,
> >     Sasha
> >
> > This e-mail message is intended for the recipient only and contains
> information which is CONFIDENTIAL and which may be proprietary to ECI
> Telecom. If you have received this transmission in error, please inform us by
> e-mail, phone or fax, and then delete the original and all copies thereof.
> >
> 
> --

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 9 Jan 2012 08:47:02 -0500

> Pablo has already commented on implementation problems with this approach in his message ( http://www.ietf.org/mail-archive/web/rtg-bfd/current/msg01207.html) and their origin in the said layer vio…

As I mentioned in my own post, I agree that it's a gross thing to do if
you're generating packets on a central routing engine rather than on a line
card.  But it certainly won't be the first time routers bang the bits on the
wire directly rather than running things through the stack.

After working through the requirements, if we end up with a solution that
doesn't require such a thing, that's a win.

> - IP layer does not exercise any kind of explicit control over selection of the LAG member over which the packet. Instead selection is done by the LAG distributor.  In fact, with most distributors,…

This is a fair observation.  It is entirely possible that an implementation
may only process IP traffic to the LAG MAC rather than to a LAG member MAC.

Thinking about a possible implementation rather than discussing
requirements, my expectation would be that the BFD IP packets are generated on
the LAG member and destined to the LAG MAC.  If the LAG BFD implementation
is not distributed to the individual interfaces, the only way to determine
that a given packet was coming in a given interface would be a priori
knowledge via the discriminators.  While it's possible a given routing
engine may have knowledge of what individual port of a LAG the packet came
in, it's just as likely that you'd get the LAG aggregate port interface
index back.

I'll ponder some text for this, but for the moment I'd prefer to focus on
requirements for now.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 9 Jan 2012 09:03:31 -0500

Tom,

> the same using ethernet OAM but folks did not want to do that.

I believe I strongly disagree with the idea we're trying to get rid of LACP.
However, moving this back to requirements it can be observed we could
accomplish the goal making sure IP traffic is delivered across the link
either by:
1. Testing the IP forwarding ability of a given link.
2. Testing the frame forwarding ability of a given link.

2 seems to be closer to what you're looking for.  BFD over Ethernet in such
a case would work here.  It could then inform protocols such as LACP so it
could converge faster.

One of the reasons I think 1 is a better requirement is that when we're
talking about aggregate links (regardless of link layer type), the operation
we're really concerned about is whether we can forward our higher layer
traffic over it.  This not only involves ensuring that the logical aggregate
link is presented to the implementation but whether the implementation can
use it for that purpose.

Put a slightly different way, just because you have a LAG doesn't mean that
your forwarding is working over a component.  I suspect any number of
operators could pick their favorite bug where this has happened.

Please note that I'm not suggesting we try to verify load balancing for LAG
members.  Doing so is wildly out of scope for BFD.

> for multicast traffic or L2 traffic?

The goal is to exercise the forwarding path for the service protected by
BFD.  If your traffic is IPv4 unicast, your BFD session should be as well.
If you traffic is MPLS, ditto.

While I'm unclear how many implementations actually do this, it is arguable
that if you're protecting a given CoS for IPv4 unicast, you should have a
distinct IPv4 unicast BFD session using that CoS.  This is because queueing
and forwarding hardware may forward differently and you want to know if you
get bi-directional forwarding for that CoS within your session's tolerance.
However, I suspect most vendors simply choose to go with fate sharing of the
address family; it minimizes the sessions you need.

Similarly, when trying to provide feedback for LACP, one design choice could
be to use the fate of IPv4 forwarding on a component to decide a LAG member
is up or down based on fate sharing.  A somewhat similar choice is faced
when protecting IS-IS sessions on a link: IS-IS is not IP traffic.

-- Jeff

---

From: Alexander Vainshtein <[email-masked]>
Date: Mon, 9 Jan 2012 17:06:03 +0200

Jeff,

I may be completely wrong here, but it seems that you are more worried about buggy LAG implementations (like in "you have a LAG doesn't mean that your forwarding is working over a component") than about "true" link failures (fiber cut etc.).

I am not sure BFD over LAG members is the right tool to help with buggy LAG implementations. In fact, I strongly suspect that it would rather add its own bugs on top of the already existing ones...

My understanding of the original problem that BFD over LAG members tries to solve is like following

1. We are dealing with a LAG between a pair of, say adjacent routers that is comprised of links that do not support detection of link failure at the physical layer (e.g., loss of optical power)  -   if they were, we would not need BFD to do the same. 
2. We want to detect failure of individual LAG members much faster than could be done by using standard LACP.
3. Fast detection of failure of the LAG member would result in:
	- Immediate bi-directional removal of this link from distribution in order to avoid back-holing of traffic. 
	  This may be followed by inclusion of some "spare links" into distribution (if these links exist and are alive)
	- Bringing down the entire LAG if the "Insufficient active links" condition has been detected. 
	  Bringing LAG down affects all its MAC clients in the usual way.
4. The situation when LAG goes down due to insufficient number of active links must be reversible, i.e., the LAG must be able to go UP once this condition disappears.
5. We are looking for a solution that is:
	- Based on the base BFD spec (RFC 5880) - among other reasons because of its operational simplicity
	- Operates normally in the presence of LACP and, optionally, may operate normally if LACP is not used.

This is very close to what I've written 3 weeks ago in my message to the list - see http://www.ietf.org/mail-archive/web/rtg-bfd/current/msg01151.html.
There is an implicit "implementation sanity" assumption here, e.g., that if the link has been found to be bi-directionally collecting and distributing, any packet LAG receives from its MAC client and decides to transmit via such a link would be received by the peer MAC client at the other end.

Could we take this problem statement as the starting point? If not, what, in your opinion, is the problem we try to address?

Regards,
     Sasha

> Behalf Of Jeffrey Haas
> Sent: Monday, January 09, 2012 4:04 PM
> To: Tom Sanders
> Cc: [email-masked]

---

From: Pablo Frank <[email-masked]>
Date: Mon, 9 Jan 2012 11:30:34 -0500

Jeff,

I think you ask a very good fundamental question at the beginning of this
thread.  Essentially, are we trying to monitor L2 continuity or L3
continuity across the LAG?

If the goal is to monitor L2 connectivity, then clearly the direct L2
encapsulation of BFD achieves that and is fairly trivial to implement.  Of
course, the IEEE could do this just as easily with Ethernet OAM and I've
heard rumours that they intend to address this issue soon.  Is there
a liaison with the IEEE who can confirm this?  However, as has been pointed
out, L2 connectivity does not necessarily mean that you can forward L3
successfully.

If the goal is to monitor the L3 forwarding path, then I don't think the
proposed solutions in the draft do that either.  Injecting IP unicast or
multicast frames directly into the member links of a LAG does not
necessarily test the L3 datapath because normal packets would be subject to
the LAG distribution algorithm.  And that's kinda the rub.  If I want to
ensure that my L3 frames will cross a LAG correctly, I need to have
visibility into the LAG distributor and source frames that will predictably
test the individual legs of the LAG.  For IP, this means varying the IP
src/dst and UDP src/dst (how would you do this with the normal IP BFD
encap?).  For MPLS sections, I suppose one could vary the entropy label.
 Of course, all this assumes that the behaviour of the distributor is known
and predictable.  I'm just not sure how you can write a standard around
this.

But there's an additional issue with running BFD at the L3 level instead of
L2.  If that's what you choose to do then I don't think that a L3 BFD
session has any business trying to influence the state of LACP that's
running down at the link layer.  I think if you choose to implement a
"LAG-aware" L3 BFD session, then all you should be doing with the state
information is influencing your L3 operational state (and taking corrective
action at L3 such as protection switching, re-routing, etc.).

It seems to me that we can't have our cake and eat it too.  There isn't a
single solution that will solve both the L2 and L3 problem domains in one
shot, nor should we try to achieve that.  It seems to me that the L2
solution should aim to solve the LACP slow convergence problem and not
attempt to verify L3 connectivity.  The IETF could try solving this with a
L2 BFD/LAG link approach but an IEEE solution is equally valid and they
likely *should* own this problem.  There might be a room for a L3
"LAG-aware" solution but the goal here is verifying L3 operational state
and it should certainly NOT be trying to influence the state of the L2 LAG
members.  In my mind, these two approaches could easily coexist and
integrates well into a layered protection model.

regards,
Pablo

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 9 Jan 2012 17:48:14 +0100

Hello Alexander,

> I may be completely wrong here, but it seems that you are more worried about buggy LAG implementations (like in "you have a LAG doesn't mean that your forwarding is working over a component") than…

not sure this is about buggy LAG implementations. More about layer-3 issues. 
The following scenario: you have a LAG, 2 links across two I/O cards. The BBP session may use - by accident - only link1 on I/O card 1 while the layer-3 hw on I/O card 2 is broken. Assuming you have many IP flows across the LAG you probably blackhole 50% of your traffic.

So BBP alone may not detect the situation and layer-3 based micro sessions would be required.

At least this is how I was reading Jeff's email.

Regards, Marc

---

From: Alexander Vainshtein <[email-masked]>
Date: Mon, 9 Jan 2012 19:59:46 +0200

Marc hi,

Lots of thanks for a prompt response and an excellent example of what could go wrong ("layer-3 hw on I/O card 2 is broken"). 

I can take it a bit further: Please consider a 1-hop IPv4/IPv6 BFD implementation (RFC 5881) where received BFD packets are identified early in the forwarding path and sent to the appropriate HW or SW entity, while "regular" IPv4 packets (with destination addresses not owned by the receiving router) are sent for the longest prefix match look-up in a corrupted table ("broken HW" or "SW bug") and, as a consequence, are black-holed. There is even no need in LAG... 

Please note that since IP and MPLS lookups are in different tables, IP traffic with the same destination addresses but encapsulated in MPLS could be perfectly healthy...

In other words, it is important to agree in advance which kind of failures you want to detect. 

To my taste,  1-hop BFD reliably covers the media between the pair of adjacent routers (be it a physical link, a LAN or something else). Once your BFD packets have been successfully received by the router, the actual coverage is implementation-specific.

Did I miss something?

Regards,
Sasha

________________________________________
From: Marc Binderberger [[email-masked]]
Sent: Monday, January 09, 2012 6:48 PM
To: Alexander Vainshtein
Cc: Jeffrey Haas; [email-masked]
Subject: Re: BFD on LAG requirements (was Re: Multicast vs Unicast (was BFD on LAG interfaces))

Hello Alexander,

Regards, Marc

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 9 Jan 2012 19:31:04 -0500

Sasha,

> Lots of thanks for a prompt response and an excellent example of what could go wrong ("layer-3 hw on I/O card 2 is broken").

And for what it's worth, Marc did a fine job summarizing one of the concerns
I'm trying to cover in the requirements.

> Please note that since IP and MPLS lookups are in different tables, IP traffic with the same destination addresses but encapsulated in MPLS could be perfectly healthy...

If the expected traffic is MPLS encapsulated IP, BFD should in such a case
be using MPLS as a transport.

> To my taste,  1-hop BFD reliably covers the media between the pair of adjacent routers (be it a physical link, a LAN or something else). Once your BFD packets have been successfully received by the…

We largely agree.  The point of my suggested requirements is that if any
IPv4, for example, can get through the component link, you're doing as well
as you can.  

Doing BFD at layer 2 tells us that the link is up but not necessarily that
IP forwarding is working at all.  But it's one step closer to the traffic
that would be passing through the link.

To pollute this response slightly, please also consider viability of
deployment from a standards basis.  IEEE was, per prior discussion with Dave
Ward, not very receptive to doing BFD at layer 2.  If our only real goal is
end to end link validation or as an adjunct to LACP, I suspect IEEE's
response would be "just use Ethernet OAM".  Part of the reason we're having
this discussion is because IP service providers are not too keen for any
number of reasons to deploy Ethernet OAM.  If they are, their solution space
is already covered.

A secondary consideration is also re-use of any standard for other types of
link aggregates.  Ethernet is the current case with the most interest, so it
makes sense to start there.  However, we'll probably consider others as
well.

-- Jeff

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Tue, 10 Jan 2012 06:04:06 +0530

Hi Pablo,

[clipped]

> If the goal is to monitor the L3 forwarding path, then I don't think the proposed 
> solutions in the draft do that either.  Injecting IP unicast or multicast frames 
> directly into the member links of a LAG does not necessarily test the L3 datapath 
> because normal packets would be subject to the LAG distribution algorithm.  And 
> that's kinda the rub.  If I want to ensure that my L3 frames will cross a LAG correctly, 
> I need to have visibility into the LAG distributor and source frames that will predictably 
> test the individual legs of the LAG.  For IP, this means varying the IP src/dst and UDP 
> src/dst (how would you do this with the normal IP BFD encap?).  For MPLS sections, 

This is incorrect. You don't need to do anything of this sort. 

Its implementation specific and I am sure all vendors can send IP/UDP packets with the same source IP/dest IP pair over different ports of the LAG.

Cheers, Manav

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 9 Jan 2012 19:45:03 -0500

Pablo,

> and predictable.  I'm just not sure how you can write a standard around

See other responses for my thoughts on layer 2 BFD.

I completely agree with you with regard to layer 3.  We fundamentally cannot
exercise the load balancer in a useful way with BFD.  I don't think we
should try.

What we do get out of layer 3 BFD on a per link basis is:
- The link is up.
- It is able to pass our traffic of interest.

> action at L3 such as protection switching, re-routing, etc.).

That is a fair observation.  By previous analogy to protecting IS-IS
sessions, vendors will use IPv4 BFD state to influence IS-IS despite the
fact that they're operating over different encapsulations.  It could also be
fairly pointed out that layer 2 BFD for our per-link case may also be good
enough.  

It is also potentially arguable that if you have a per-link BFD session for
IPv4 and one for IPv6, if IPv6 stops working you probably don't want to tear
the link down.  However, if the load balancer can take as inputs for its
hash the per-address family health of the link, the load balancer can choose
to do Something Appropriate.

So, to address your point, if we use BFD over layer 3, I believe that we can
get "good enough" link health in spite of not using the same transport as
LACP *and* that we get a useful input to LAG traffic balancers as well.
And to reiterate, I agree with you that we're not fully exercising the load
balancer using BFD.

> likely *should* own this problem.

We will certainly be working with IEEE to help refine the problem and
solution spaces.

> There might be a room for a L3
> members.

This point is certainly one of the issues I was hoping we'd drive to
conclusion by discussing requirements.

-- Jeff

---

From: Pablo Frank <[email-masked]>
Date: Wed, 11 Jan 2012 12:38:25 -0500

Hi Jeff

See PF>> below

> - It is able to pass our traffic of interest.

PF>> I guess it's the 2nd point that I'm skeptical about.  If you are not
going through the normal IP forwarding path, what exactly does it prove? If
both sender and receiver are bypassing normal IP stack operation to send
and receive these frames, it doesn't matter that they're IP encapsulated.
 They could be encapsulated in protocol foo and you could achieve the same
thing.  I guess, fundamentally, unless the L3 BFD is subject to the same
forwarding rules as other L3 traffic, it doesn't prove anything more than
just running BFD directly over L2.

Even if you were to assign, say, link-local IP addresses to the individual
links and exchange frames this way then all you're proving is that you can
exchange packets in the link-local subnet.  The IP routes that point out
the LAG interface (as opposed to the individual legs) might still be busted
for any number of reasons.

regards,
Pablo

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 11 Jan 2012 15:11:51 -0500

Dear Pablo, et al.,
I think that since RFC 5880 BFD is not using Up/Down MEP model fate sharing within a node is implementation specific.

    Regards,
        Greg

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Pablo Frank
Sent: Wednesday, January 11, 2012 9:38 AM
To: Jeffrey Haas
Cc: [email-masked]
Subject: Re: BFD on LAG requirements (was Re: Multicast vs Unicast (was BFD on LAG interfaces))

Hi Jeff

See PF>> below

On Mon, Jan 9, 2012 at 7:45 PM, Jeffrey Haas <[email-masked]<mailto:[email-masked]>> wrote:
Pablo,

PF>> I guess it's the 2nd point that I'm skeptical about.  If you are not going through the normal IP forwarding path, what exactly does it prove? If both sender and receiver are bypassing normal IP stack operation to send and receive these frames, it doesn't matter that they're IP encapsulated.  They could be encapsulated in protocol foo and you could achieve the same thing.  I guess, fundamentally, unless the L3 BFD is subject to the same forwarding rules as other L3 traffic, it doesn't prove anything more than just running BFD directly over L2.

Even if you were to assign, say, link-local IP addresses to the individual links and exchange frames this way then all you're proving is that you can exchange packets in the link-local subnet.  The IP routes that point out the LAG interface (as opposed to the individual legs) might still be busted for any number of reasons.

regards,
Pablo

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 12 Jan 2012 05:02:38 +0530

Hi Pablo,

I disagree with your conclusion that micro BFD sessions using L3 encapsulation does not get us anything.

Lets look at the two most widely deployed routing protocols, IS-IS and OSPF.

IS-IS packets don't follow the exact data path as the IP packets (since IS-IS uses L2 encap) and yet once the IS-IS adjacency goes up we trust that we can use that link for IP forwarding. Even OSPF for that matter does not follow the exact same data path as IP unicast packets since OSPF packets are multicast (except for DDs which are unicast). In both cases the protocols don't follow the same data path as IP unicast and yet we conclude that the IP routes are reachable based on the information provided by these protocols.

L3 encap micro BFD sessions will tell you that:

(1) The link is up
(2) The receiver is able to process L3 packets arriving on that link

Its imperative that we ensure that each link is able to process L3 packets since each link [1] will be used for L3 forwarding. And by running BFD on each member link (via micro BFD sessions) we ensure exactly that. And this is also precisely the reason why i believe that using multicast encap for micro BFD sessions is nearly as good as using unicast encap.

Cheers, Manav

[1] The details of how each link gets used is implementation specific and its usually decided based on a hash of certain well known fields from the data packet.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Pablo Frank
Sent: Wednesday, January 11, 2012 11:08 PM
To: Jeffrey Haas
Cc: [email-masked]
Subject: Re: BFD on LAG requirements (was Re: Multicast vs Unicast (was BFD on LAG interfaces))

Hi Jeff

See PF>> below

regards,
Pablo

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 12 Jan 2012 09:48:48 -0500

> IS-IS packets don't follow the exact data path as the IP packets (since IS-IS uses L2 encap) and yet once the IS-IS adjacency goes up we trust that we can use that link for IP forwarding. Even OSPF…

I'm going to pick on the OSPF case for a second.

There have been about three incidences I can remember over the years where
OSPF was failing to work right for me that were eventually diagnosed to be
multicast vs. unicast reachability issues on the exact same link.  In these
cases, the issue was outright buggy implementation.  (And in 2/3 of the
cases, resetting the interface cleared the problem.)

While I can agree with Pablo that we're not testing traffic *through* the
box but only *to* the box (a very shallow test for a box we probably want to
use as a transit router), I still think testing to the layer 3 is one step
better than not.  Bugs such as the ones discussed above make me leery of
doing otherwise.

(And we're not going to even talk about IS-IS bugs where IP wasn't up...)

-- Jeff

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 12 Jan 2012 19:15:41 -0500

Hi Manav,
I wouldn't interpret Pablo's view as "micro BFD sessions using L3 encapsulation does not get us anything" but rather as use of L3 addressing vs. L2 addressing doesn't add incremental value to micro-BFD. And the main reason for this conclusion, IMHO, is that IP addresses that might be used by micro-BFD would not be used by customer's traffic. Hence, we cannot claim that FIBs are properly set based on state of micro-BFD with L3 addressing.

    Regards,
        Greg

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bhatia, Manav (Manav)
Sent: Wednesday, January 11, 2012 3:33 PM
To: Pablo Frank; Jeffrey Haas
Cc: [email-masked]
Subject: RE: BFD on LAG requirements (was Re: Multicast vs Unicast (was BFD on LAG interfaces))

Hi Pablo,

(1) The link is up

Cheers, Manav

Hi Jeff

See PF>> below

regards,
Pablo