Subject: RE: BFD on LAG interfaces

From: Jeff Tantsura <[email-masked]>
Date: Tue, 3 Jan 2012 03:42:40 -0500

Hi,

Let's not make things too complicated :)

IMHO:
>From RIB point of view -  it doesn't care how many links are in the LAG, as long as its neighbor is reachable (BFD) and min-link has not been met - LAG is UP from RIB (and its clients) point of view and hence forwarding should use it.
If min-link has been met indeed - RIB doesn't care about reachability anymore, from its point of view - LAG is DOWN which is communicated to FP && every client registered.

Having implemented both single and multi session BFD over LAG I can tell you - customers (at least ours :)) don't care about running LACP in addition to BFD.
 Logical explanation why it is needed works just fine.I'd be really interested to hear differently.

So let me try to go thru the scenario below.
LAG UP
Before physical layer first and then 802.1ax agree that LAG is UP, RIB doesn't get bothered at all, so no forwarding  lookup could possibly yield that LAG as an outgoing interface and hence no blackholing.
Nobo - normally static is as client of RIB as any other protocol interested in BFD state, it shouldn't care about 802.1ax, as long as RIB doesn't tell it.

Additional link UP
Don't care (SS)/span additional BFD session (MS)

LINK DOWN
single session over LAG (could be LoS or BFD if the link in question is by chance the same link BFD got hashed over)

if 
 min-link not met
then ignore
else
 bring LAG DOWN && notify CP (RIB) && notify FP (initiate FRR/etc)

multi session BFD over LAG (will be LoS or BFD associated with this particular LAG constituent)
same logic as before

My 0.2

Regards,
Jeff

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Nobo Akiya (nobo)
Sent: Thursday, December 29, 2011 4:33 PM
To: Marc Binderberger; Alexander Vainshtein; Tom Sanders
Cc: [email-masked]
Subject: RE: BFD on LAG interfaces

Hello Marc, et al,

> The problem is to bring up the link with protocol P1 but bring it down 
> with protocol P2. The underlying assumption is that P1 is interrupted 
> as well when P2 goes down. True for a clean link cut, questionable if 
> something goes wrong with the layer-3 alone. Then P1 (LACP or L1) may 
> be up and thus the LAG port is active while P2 (BFD) stays down.
> 
> Maybe we can live with it (?).

"Can we live with it" is indeed the primary question. If consensus is yes, then secondary question becomes what action should LMM take (if any at all)? LMM will know P1 is up but P2 has not come up for "long" time:
hours or even days. Should LMM continue to use such link, or should it time out (and stop using the link) eventually? If we want to keep it simple, then the answer here would be "no timeout mechanism" here as well.

> Maybe someone has implemented the "P1 brings up, P2 brings down" and
can
> talk about the experience?

I'm curious on this. Side effect here is traffic blackhole. To elaborate the side effect, here are couple of scenarios which can happen.
Assumption here is there is one link which P1 is up and P2 has not come up due to layer-3 specific failure. Let's label this link L1.

(1) Routing protocols "handshake" happens to take place over non-L1 link and traffic shifts over 802.1ax, but traffic going over L1 will be blackholed.
(2) Static route, once 802.1ax state is up, will install route over it, but traffic going over L1 will be blackholed.

Certainly would like to hear input from vendors if someone has implemented this model, but would also like to hear from operators if this potential issue is something "we can live this".

Regards,
Nobo

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Marc Binderberger
> Sent: Friday, December 30, 2011 2:24 AM
> To: Alexander Vainshtein; Tom Sanders
> Cc: [email-masked]
> Subject: Re: BFD on LAG interfaces
> 
> Hello Tom, Alexander et al.,
> 
> in general I agree with your statements. BFD is _not_ LACP. BFD should 
> be planned as a booster for the detection. And keep it simple. On the 
> other hand we need answers for the various cases.
> 
> The problem is to bring up the link with protocol P1 but bring it down 
> with protocol P2. The underlying assumption is that P1 is interrupted 
> as well when P2 goes down. True for a clean link cut, questionable if 
> something goes wrong with the layer-3 alone. Then P1 (LACP or L1) may 
> be up and thus the LAG port is active while P2 (BFD) stays down.
> 
> Maybe we can live with it (?).
> 
> 
> As far as I know about existing implementations they seem to avoid the 
> above situation by either:
> 
> (i) use BFD to control - alone or in combination with LACP - the port 
> going active and going down
> 
> (ii) use BFD to update routing protocols alone but do not affect the
LAG.
> 
> 
> Maybe someone has implemented the "P1 brings up, P2 brings down" and
can
> talk about the experience?
> 
> 
> Regards, Marc
> 
> 
> 
> 
> On 2011-12-29, at 16:38 , Tom Sanders wrote:
> 
> > Sasha,
> >
> > I agree. Instead of us speculating, can the operators who are on
this
> > list speak on whether they are looking at BFD managing their LAGs 
> > without LACP? If Yes, then why do they want this? It would be a
shame
> > to over-engineer a solution when a simple one is available.
> >
> > Toms
> >
> > On 28 December 2011 00:29, Alexander Vainshtein 
> > <[email-masked]> wrote:
> >> Manav,
> >> The scheme you've presented looks OK to me.
> >>
> >> Regarding your question - why somebody would NOT want LACP - I can
> only add that micro-BFD sessions, in my opinion, are used to replace 
> lacking failure indications from the physical layer. In these
scenarios
> working without LACP would be quite problematic in any case...
> >>
> >> (Not sure if this is the right answer, but it is the best one I can
> give:-)
> >>
> >> Regards,
> >>     Sasha
> >>
> >> ________________________________________
> >> From: [email-masked] [[email-masked]] On Behalf
> Of Bhatia, Manav (Manav) [[email-masked]]
> >> Sent: Tuesday, December 27, 2011 5:43 PM
> >> To: Nitin Bahadur; [email-masked]
> >> Subject: RE: BFD on LAG interfaces
> >>
> >> Yes, this sounds reasonable.
> >>
> >> What we're looking at right now is this:
> >>
> >> 1. We use LACP to bring up the links.
> >> 2. We establish micro BFD sessions per link (using ingress port +
source
> MAC as a unique identifier)
> >> 3. If BFD session goes down, we inform the 802.1ax so that it can
bring
> down the state so that the link is NOT used for forwarding.
> >> 4. Wait for LACP to bring up the member port of the LAG. Once its
UP,
> BFD sessions again establish and the process repeats.
> >>
> >> So, we use BFD only to bring down the link.
> >>
> >> Once the link is up, we use IP encapsulated BFD (no L2 encap
required)
> with either unicast or multicast addressing scheme.
> >>
> >> This solution however requires LACP to work in tandem with the
micro
> BFD sessions for the LAG. This will work for all scenarios except the 
> one where a provider does NOT want to run LACP on the LAG (and I am
not
> sure why somebody would NOT want LACP btw).
> >>
> >> Cheers, Manav
> >>
> >>> -----Original Message-----
> >>> From: Nitin Bahadur [mailto:[email-masked]]
> >>> Sent: Friday, December 23, 2011 1:29 AM
> >>> To: Nitin Bahadur; Bhatia, Manav (Manav); Jeff Tantsura; 
> >>> [email-masked]
> >>> Subject: Re: BFD on LAG interfaces
> >>>
> >>>
> >>> And just to clarify a little bit more....
> >>>
> >>> I don't want BFD to replace LACP. I want it to supplement LACP. 
> >>> Just like today if we run BFD for OSPF adjacencies, we don't get 
> >>> rid of OSPF hellos...
> >>> But if the BFD session goes down, we tell OSPF about it and OSPF 
> >>> reacts to that. Similarly (pardon my lack of knowledge about 
> >>> LACP), LACP can react to BFD member sessions down. The link can be 
> >>> brought up by LACP at a slower interval. The goal should be fast 
> >>> failure detection and not necessarily fast Link-up following a 
> >>> failure correction.
> >>>
> >>> Thanks
> >>> Nitin
> >>>
> >>
> >> This e-mail message is intended for the recipient only and contains
> information which is CONFIDENTIAL and which may be proprietary to ECI 
> Telecom. If you have received this transmission in error, please
inform
> us by e-mail, phone or fax, and then delete the original and all
copies
> thereof.
> >>
> >
> >
> >
> > --
> > Toms.
> >
> 
> --
> Marc Binderberger           <[email-masked]>

---

From: Alexander Vainshtein <[email-masked]>
Date: Tue, 3 Jan 2012 11:25:31 +0200

Jeff,
You've said " customers (at least ours :)) don't care about running LACP in addition to BFD".

Does "don't care"  here mean that do not mind running both, or object to running both? A clarification on that point seems important.

Regards,
     Sasha

> From: Jeff Tantsura [mailto:[email-masked]]
> Sent: Tuesday, January 03, 2012 10:43 AM
> To: Nobo Akiya (nobo); Marc Binderberger; Alexander Vainshtein; Tom
> Sanders
> Cc: [email-masked]
> From RIB point of view -  it doesn't care how many links are in the LAG, as
> long as its neighbor is reachable (BFD) and min-link has not been met - LAG
> is UP from RIB (and its clients) point of view and hence forwarding should
> use it.
> If min-link has been met indeed - RIB doesn't care about reachability
> anymore, from its point of view - LAG is DOWN which is communicated to FP
> && every client registered.
> 
> Having implemented both single and multi session BFD over LAG I can tell
> you - customers (at least ours :)) don't care about running LACP in addition
> to BFD.
> Before physical layer first and then 802.1ax agree that LAG is UP, RIB doesn't
> get bothered at all, so no forwarding  lookup could possibly yield that LAG as
> an outgoing interface and hence no blackholing.
> Nobo - normally static is as client of RIB as any other protocol interested in
> BFD state, it shouldn't care about 802.1ax, as long as RIB doesn't tell it.
> 
> Additional link UP
> single session over LAG (could be LoS or BFD if the link in question is by
> chance the same link BFD got hashed over)
> 
> if
>  min-link not met
> then ignore
> else
> Behalf Of Nobo Akiya (nobo)
> "Can we live with it" is indeed the primary question. If consensus is yes,
> then secondary question becomes what action should LMM take (if any at
> all)? LMM will know P1 is up but P2 has not come up for "long" time:
> hours or even days. Should LMM continue to use such link, or should it time
> out (and stop using the link) eventually? If we want to keep it simple, then
> the answer here would be "no timeout mechanism" here as well.
> 
> I'm curious on this. Side effect here is traffic blackhole. To elaborate the
> side effect, here are couple of scenarios which can happen.
> Assumption here is there is one link which P1 is up and P2 has not come up
> due to layer-3 specific failure. Let's label this link L1.
> 
> (1) Routing protocols "handshake" happens to take place over non-L1 link
> and traffic shifts over 802.1ax, but traffic going over L1 will be blackholed.
> Certainly would like to hear input from vendors if someone has
> implemented this model, but would also like to hear from operators if this
> potential issue is something "we can live this".
> 
> Regards,
> Nobo
> 

This e-mail message is intended for the recipient only and contains information which is CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this transmission in error, please inform us by e-mail, phone or fax, and then delete the original and all copies thereof.

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 3 Jan 2012 04:41:34 -0500

Hi Sasha,

The first, running both. They'd run LACP anyway even before BFD over LAG had become available, there's no benefit is disabling it after.
In my past, for over 10 years I used to do network architecture and design in multivendor environment, to my memory the only case one wouldn't use LACP was either lack of support (oldish Cisco gear) or extremely buggy implementations (no names here :))

Again - I'm talking for myself and my customers, some might not like it for whatever reasons...

P.S. Dear BFDers - Happy New Year!

Regards,
Jeff

P.S. Dear BFDers - Happy New Year!

-----Original Message-----
From: Alexander Vainshtein [mailto:[email-masked]] 
Sent: Tuesday, January 03, 2012 1:26 AM
To: Jeff Tantsura
Cc: [email-masked]; Nobo Akiya (nobo); Marc Binderberger; Tom Sanders
Subject: RE: BFD on LAG interfaces

Jeff,

Regards,
     Sasha

> From RIB point of view -  it doesn't care how many links are in the 
> LAG, as long as its neighbor is reachable (BFD) and min-link has not 
> been met - LAG is UP from RIB (and its clients) point of view and 
> hence forwarding should use it.
> anymore, from its point of view - LAG is DOWN which is communicated to 
> FP && every client registered.
> 
> Having implemented both single and multi session BFD over LAG I can 
> tell you - customers (at least ours :)) don't care about running LACP 
> in addition to BFD.
> Before physical layer first and then 802.1ax agree that LAG is UP, RIB 
> doesn't get bothered at all, so no forwarding  lookup could possibly 
> yield that LAG as an outgoing interface and hence no blackholing.
> Nobo - normally static is as client of RIB as any other protocol 
> interested in BFD state, it shouldn't care about 802.1ax, as long as RIB doesn't tell it.
> 
> Additional link UP
> single session over LAG (could be LoS or BFD if the link in question 
> is by chance the same link BFD got hashed over)
> 
> if
>  min-link not met
> then ignore
> else
> multi session BFD over LAG (will be LoS or BFD associated with this 
> particular LAG constituent) same logic as before
> 
> My 0.2
> 
> Regards,
> Jeff
> 
> > The problem is to bring up the link with protocol P1 but bring it 
> > down with protocol P2. The underlying assumption is that P1 is 
> > interrupted as well when P2 goes down. True for a clean link cut, 
> > questionable if something goes wrong with the layer-3 alone. Then P1 
> > (LACP or L1) may be up and thus the LAG port is active while P2 (BFD) stays down.
> >
> "Can we live with it" is indeed the primary question. If consensus is 
> yes, then secondary question becomes what action should LMM take (if 
> any at all)? LMM will know P1 is up but P2 has not come up for "long" time:
> hours or even days. Should LMM continue to use such link, or should it 
> time out (and stop using the link) eventually? If we want to keep it 
> simple, then the answer here would be "no timeout mechanism" here as well.
> 
> I'm curious on this. Side effect here is traffic blackhole. To 
> elaborate the side effect, here are couple of scenarios which can happen.
> Assumption here is there is one link which P1 is up and P2 has not 
> come up due to layer-3 specific failure. Let's label this link L1.
> 
> (1) Routing protocols "handshake" happens to take place over non-L1 
> link and traffic shifts over 802.1ax, but traffic going over L1 will be blackholed.
> (2) Static route, once 802.1ax state is up, will install route over 
> it, but traffic going over L1 will be blackholed.
> 
> implemented this model, but would also like to hear from operators if 
> this potential issue is something "we can live this".
> 
> Regards,
> Nobo
> 
> > in general I agree with your statements. BFD is _not_ LACP. BFD 
> > should be planned as a booster for the detection. And keep it 
> > simple. On the other hand we need answers for the various cases.
> >
> > The problem is to bring up the link with protocol P1 but bring it 
> > down with protocol P2. The underlying assumption is that P1 is 
> > interrupted as well when P2 goes down. True for a clean link cut, 
> > questionable if something goes wrong with the layer-3 alone. Then P1 
> > (LACP or L1) may be up and thus the LAG port is active while P2 (BFD) stays down.
> >
> > As far as I know about existing implementations they seem to avoid 
> > the above situation by either:
> >
> > (i) use BFD to control - alone or in combination with LACP - the 
> > port going active and going down
> >
> > BFD sessions for the LAG. This will work for all scenarios except 
> > the one where a provider does NOT want to run LACP on the LAG (and I 
> > am
> not
> > >>> LACP), LACP can react to BFD member sessions down. The link can 
> > >>> be brought up by LACP at a slower interval. The goal should be 
> > >>> fast failure detection and not necessarily fast Link-up 
> > >>> following a failure correction.
> > >>>
> > >>> Thanks
> > >>> Nitin
> > >>>
> > >>
> > information which is CONFIDENTIAL and which may be proprietary to 
> > ECI Telecom. If you have received this transmission in error, please
> inform

---

From: Alexander Vainshtein <[email-masked]>
Date: Tue, 3 Jan 2012 12:41:09 +0200

Jeff,
Lots of thanks for a prompt and unambiguous response!

Happy New year to all!

Regards,
     Sasha

> Sent: Tuesday, January 03, 2012 11:42 AM
> To: Alexander Vainshtein
> The first, running both. They'd run LACP anyway even before BFD over LAG
> had become available, there's no benefit is disabling it after.
> In my past, for over 10 years I used to do network architecture and design in
> multivendor environment, to my memory the only case one wouldn't use
> LACP was either lack of support (oldish Cisco gear) or extremely buggy
> implementations (no names here :))
> 
> Does "don't care"  here mean that do not mind running both, or object to
> running both? A clarification on that point seems important.
> 
> Regards,
>      Sasha
> 
> Telecom. If you have received this transmission in error, please inform us by
> e-mail, phone or fax, and then delete the original and all copies thereof.

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Tue, 3 Jan 2012 17:19:25 -0800

Happy new year to everyone on the list ;)

Jeff, et al,

I agree with your statements on "RIB point of view", a link will get
used if it's "blessed". However, my concern is in the asymmetric
"blessing" logic described in this email. Below were stated.

>> So, we use BFD only to bring down the link.

This can create severe traffic impacting scenario if a system ever falls
into a scenario where BFD fails but LACP continues to work (ex: L3
specific failure). IMHO, if BFD is used to bring down the link, then it
should be used to bring up the link as well. With this logic, dependency
to LACP is no longer required either. I'd be curious to see what other
folks thinks on this.

Regards,
Nobo

> Sent: Tuesday, January 03, 2012 7:41 PM
> To: Jeff Tantsura
> > Again - I'm talking for myself and my customers, some might not like
> it for
> > whatever reasons...
> >
> > You've said " customers (at least ours :)) don't care about running
> LACP in
> > addition to BFD".
> >
> > Does "don't care"  here mean that do not mind running both, or
object
> to
> > > anymore, from its point of view - LAG is DOWN which is
communicated
> to
> > >  Logical explanation why it is needed works just fine.I'd be
really
> > > interested to hear differently.
> > >
> > > interested in BFD state, it shouldn't care about 802.1ax, as long
> as RIB
> > doesn't tell it.
> > >
> > > Additional link UP
> > > > questionable if something goes wrong with the layer-3 alone.
Then
> P1
> > > > (LACP or L1) may be up and thus the LAG port is active while P2
> (BFD)
> > stays down.
> > > >
> > > any at all)? LMM will know P1 is up but P2 has not come up for
"long"
> time:
> > > hours or even days. Should LMM continue to use such link, or
should
> it
> > > simple, then the answer here would be "no timeout mechanism" here
> as
> > well.
> > >
> > > link and traffic shifts over 802.1ax, but traffic going over L1
will
> be
> > blackholed.
> > > > questionable if something goes wrong with the layer-3 alone.
Then
> P1
> > > > (LACP or L1) may be up and thus the LAG port is active while P2
> (BFD)
> > stays down.
> > > >
> > > > >> Regarding your question - why somebody would NOT want LACP -
> I
> > > > >> can
> > > > >> (Not sure if this is the right answer, but it is the best one
> I
> > > > >> can
> > > > give:-)
> > > > >>
> > > > >> Regards,
> > > > >>     Sasha
> > > > >>
> > > > >> 2. We establish micro BFD sessions per link (using ingress
port
> +
> > > > the one where a provider does NOT want to run LACP on the LAG
(and
> I
ECI
inform
> us by
inform
copies