Subject: Updated BFD over LAGs draft

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Wed, 1 Feb 2012 15:29:25 +0530

Folks,

After several rounds of tele conferences and multiple internal iterations of this draft we (British Telecom, Alcatel-Lucent, Cisco, Juniper, Huawei and Ericsson) have converged at a solution that we all believe is the best way forward.

Would request the WG to look at the latest version of the BFD over LAGs draft:
http://tools.ietf.org/html/draft-mmm-bfd-on-lags-03

Toms,

Let me answer your questions:

> So where are we with the discussion of BFD over LAGs?

We've been having offline discussions since quite some time and the -03 version of the draft is a result of those extensive discussions.

> 
> Did we reach any conclusion on the layer that the "micro BFD sessions"
> should operate in? Is there any update on whether the "micro 
> BFD sessions" need to use unicast or multicast addressing? 

Yes, to both.

We believe its best to run it as Layer 3 Unicast.

> Are the authors of draft-mmm-* coming up with a new version 
> that captures some progress?

Yup, we have new co-authors and several contributors on the new version!

> 
> We have also not heard what the WG chairs feel about the 
> whole discussion. It would be instructive to hear their take 
> on this as well.

We have one of the chairs as a co-author now. So, we know at least what one of them feels about this! 

Cheers, Manav

---

From: Alexander Vainshtein <[email-masked]>
Date: Wed, 1 Feb 2012 17:38:40 +0200

Manav, and all,
I've read the latest version of the draft and I have several issues with the new text.

1. In Section 1 you say that " Currently Link aggregation control protocol (LACP) is used to detect failures on a per physical member link."
    This statement looks inaccurate to me because any (if not all) LAG implementations react to physical link failures (where available) 
    regardless of usage or non-usage of LACP. IMHO added value of running BFD over LAG members only exists when, for some reason,
    the member links do not report physical failures (at all or fast enough).

2. In the same Section you say that one of the benefits of running BFD over LAG members is ability "to verify L3 connectivity per member link".
    The correctness of this statement is IMHO implementation-specific, same as in the case of, say, 1-hop IPv4/IPv6 BFD as per RFC 5881:
     A faulty implementation may be able to correctly identify BFD packets in IP/UDP encapsulation and pass them to the appropriate
    entity (SW or HW) that  handles these packets while not being able to forward received IP packets with " remote" destination addresses.

3.  In Section 2.2. I've found a couple of statements that imply availability of the information about the physical ingress port of an IP packet to
     the appropriate "handling" entity.  This looks to me like a strong requirement because (and this has been discussed on the list) 
     some existing  implementations do not provide this information (as it does not belong to IP layer which treats the entire LAG as a single virtual port). 
    IMHO you should explicitly present this requirement.

4. In the same section, there are certain issues regarding MAC addresses in the Ethernet encapsulation of the micro-BFD packets:
 -    The Source MAC address is required to match the MAC address of the physical port on which the micro-BFD session runs. This 
       implies bypass of the "regular" IP stack (because it would use the same Source MAC address for all the packets sent via LAG) and,
       incidentally, compromises the claim that UDP/IP-encapsulation of the micro-BFD packets helps to verify IP connectivity over the link.
 -    The Destination MAC address of the micro-BFD packets is supposed to change in the life cycle of the session: it starts with using a
       dedicated MAC address to be allocated by IANA but goes on using "learned" MAC address (which presumably is the source MAC address
       in the received micro-BFD packet. IMHO such behavior is neither trivial to implement nor adds any benefits: if the peer is supposed to receive
       packets with the dedicated Destination JAC address, why should this address be ever changed? The claim that this trick 
      "reduced requirements for Ethernet HW" does not seem to be justified to me
 -    Last but not least, the draft states that the dedicated MAC address is allocated by IANA as per RFC 5342, but it does not explain whether
      a unicast or a multicast address should be allocated.

5. There seem to be a contradiction regarding activation of a micro-BFD session as described in Section 3 and Section 3.1:
   - Section 3 states that this session must be initiated when a link is attached to the Aggregator  and deleted when it is detached from it 
     (The member link state machine is shown in Figure 5-14 of 802.1ax)
   - Section 3.1 states that in the presence of LACP this session must be initiated when the link advances to its Distributing state.
   - To me the text in Section 3.1 makes plenty of sense because in the presence of LACP the link only advances to the Distributing
      state only after it sees that its peer is Collecting. But this definition differs from that in Section 3

6. The text in the same section states that " BFD, as a layer 3 protocol, is viewed as running  across the LAG, with  load balancing constraints 
    ensuring  particular micro BFD sessions are effectively bound to particular member links". Does this mean that micro-BFD sessions send and receive
    packets using the Agg: MA-UNITDATA service primitives? If yes, how can transmission of customer traffic via Distributing member links be prevented
    while micro-BFD packets go thru?

7.  802.1ax does not specify any "load balancing constraints" in the LAG Distributor beyond flow preservation. And there is no indication
      In the draft regarding how these constraints would look like.

8.  IMHO and FWIW the draft would become much more readable if it would explicitly indicate the relationship of micro-BFD with the LAG
     architectural blocks as shown 0n Figure 5-3 of 802.1ax.
    
Hopefully, these notes will be useful.

Regards,
     Sasha

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Bhatia, Manav (Manav)
> Sent: Wednesday, February 01, 2012 11:59 AM
> To: Tom Sanders; [email-masked]
> Subject: Updated BFD over LAGs draft
> 
> Folks,
> +
> After several rounds of tele conferences and multiple internal iterations of
> this draft we (British Telecom, Alcatel-Lucent, Cisco, Juniper, Huawei and
> Ericsson) have converged at a solution that we ll believe is the best way
> forward.
> 
> We've been having offline discussions since quite some time and the -03
> version of the draft is a result of those extensive discussions.
> 
> >

This e-mail message is intended for the recipient only and contains information which is CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this transmission in error, please inform us by e-mail, phone or fax, and then delete the original and all copies thereof.

---

From: Tom Sanders <[email-masked]>
Date: Thu, 2 Feb 2012 01:30:37 +0530

Hi Manav,

Thanks very much for the update.

Wow .. a lot has changed between -02 and -03!

> We believe its best to run it as Layer 3 Unicast.

I was, and still continue to be, a big fan of multicast as that saves
you a lot of configuration overhead. Any particular reason why you
didnt go with that approach?

BTW, if i am not mistaken, that was your preferred approach too, wasnt it?

> Yup, we have new co-authors and several contributors on the new version!

Whoa .. and thats some list you have! :-)

> We have one of the chairs as a co-author now. So, we know at least what one of them feels about this!

Aha.

-- 
Toms.

---

From: Tom Sanders <[email-masked]>
Date: Thu, 2 Feb 2012 01:40:50 +0530

I also think -02 did a good job with describing how BFD runs over a
LAG considering it as one virtual entity (BBP?). I find that entire
section missing in -03. I think its important to have that too, as one
could run micro BFD sessions to ensure link liveliness, and BFD over
the virtual LAG for LAG liveliness. You can argue that the aggregated
state of the micro sessions will give you the state of the LAG, but
there is a subtle difference. The micro BFD sessions only interact
with the LAG manager while the other BFD session will directly
interact with the L3 applications like OSPF, etc.

I think its useful to describe how BFD runs over a LAG considering it
as one virtual port.

Toms.

-- 
Toms.

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 1 Feb 2012 22:21:03 +0100

Hello Tom,

we split off the "BBP" into a separate draft, to focus on the per-member-link mode that was mainly described (and now solely) in the draft-mmm-bfd-on-lags.

Interestingly we have some fifty-fifty opinion if this split-off draft, describing BBP, has enough content to publish it. Maybe we should publish and leave the decision to the IETF list :-)

Regards, Marc

--
Marc Binderberger           <[email-masked]>

---

From: Marc Binderberger <[email-masked]>
Date: Thu, 2 Feb 2012 00:28:37 +0100

Hello Alexander,

lots of feedback - thanks!

>    This statement looks inaccurate [...]

fair enough. Correct, we talk about failures that are not indicates by loss-of-signal or other layer-1 "down" signals. Unfortunately that seems to happen often enough, otherwise no one would be interested in BFD (and other active protocols) ;-)
I think we can mention this in the text.

>    The correctness of this statement is IMHO implementation-specific [...]

indeed. If it helps we could talk about "allows to verify aspects of L3 connectivity". If and how much of the L3 connectivity and code path is tested with the BFD implementation is indeed and deliberately left to the implementor.

>     some existing  implementations do not provide this information 

First of all it's up to the implementor how s/he is matching the incoming packet to a port and more important to the BFD session running on that port. Remember when you proposed that LACP requests the BFD session and provides the remote port MAC address with the request? Providing the packet including the source MAC would then qualify as "interface information", enough to match the incoming packet to the session which in turn is for the particular port.

Alternatively: we once discussed the drop-off/inject mechanism, using the MAC service interface between the port MAC and the Link aggregation sublayer. One could drop/add IP/UDP packets at this point. That may not have much in common with the normal L3 path but that is actually not the problem of the draft. What it "guarantees" is that such an API is possible, based on a well defined MAC service Interface.

We had a lot of this API discussion and I think what it was missing was: the API is possible - the drop/add example is enough for this statement - and anything else is up to the implementor.

>       incidentally, compromises the claim that UDP/IP-encapsulation of the micro-BFD packets helps to verify IP connectivity over the link.

It is a balance between staying close to the layer-3 path and doing what is written in the draft. And it is again a matter of the implementors ideas. You could well run the full "normal" path and as a last step overwrite all the MAC, port, etc values. That would catch all the errors that may happen in the normal path.

Regarding the change in the MAC address I leave this to George. I don't know how many devices are affected and need this "support".
My understanding is that Ethernet hardware exists where anything but the embedded MAC is kind of an exception and the handling is "expensive". The mechanism itself seems relatively simple to me:

                   [..] This will be used for the initial BFD down
   packets of a session, while sending BFD Init and Up packets with the
   MAC address learned from the received BFD packets.

>      a unicast or a multicast address should be allocated.

Good point. We should discuss this :-) Both should work and I haven't seen any advantage for unicast or for multicast.

> 5. There seem to be a contradiction regarding activation of a micro-BFD session as described in Section 3 and Section 3.1: [...]

1:0 for you. Slipped through when we merged text :-/ Authors will discuss this but I _think_ the ...

   The micro BFD session for a particular port MUST be requested when
   the port is attached to an aggregator.  The session MUST be deleted
   when the port is detached from the aggregator.

... should be part of section 3.2.

>    packets using the Agg: MA-UNITDATA service primitives?

No, we are not trying to make any such statement about the service interface to use. What is wants to say is that "ensuring particular micro BFD sessions are
effectively bound to particular member links" is required. I.e. that the ssion packets go out from the right port.
If you think this text part is confusing then we may have to reword this a bit.

>      In the draft regarding how these constraints would look like.

We probably have to reword it if this "constraints" misleads implementors what to do or what they could do.

> Hopefully, these notes will be useful.

Yes, thanks a lot!

Regards, Marc

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 1 Feb 2012 20:47:20 -0500

[Speaking as a co-chair]
On Sat, Jan 21, 2012 at 01:20:44PM +0200, Alexander Vainshtein wrote:
> It has been my understanding that the discussion around this draft have resulted in the WG chairs requesting a slot for a F2F session at the Paris IETF - for the first time in several years.
> 
> I would take it as an indication (of a kind) of how they feel about the whole discussion.

Meeting time is best used for presentation of new ideas or working on
consensus.  The discussion in the WG list has been energetic and a lot of
good ideas have been thrown around.  The recently published -03 is the
result of many contributors trying to work out the differences in the
proposal.  

My hope is that by the time we've hit our time slot in Paris that we'll have
great consensus and will rapidly close any remaining issues.

It feels like we're making very good progress.  Let's keep it going.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 1 Feb 2012 21:08:45 -0500

[Speaking as an individual contributor]

> BTW, if i am not mistaken, that was your preferred approach too, wasnt it?

My sense is that it's the rough consensus of the group.  The general
argument is that we're closer to exercising the forwarding path of the
involved traffic - at least for IP unicast, which is our first target.

Arguments have been made, with some validity, that some of the things
necessary to bootstrap the session require a level of control beyond what
you'd get out of simple socket APIs.  I would offer a counter to this
argument that while that's true on the sending side under some of those
circumstances, the receiving side may not have similar considerations.

IMO, if we're going to have to bang the bits on the wire ourselves, let's do
so in such a way that implementations that need few changes need as few
changes from stock IP BFD as possible.  This lets us leverage existing code
and chipsets where possible.

> > We have one of the chairs as a co-author now. So, we know at least what one of them feels about this!

This editor (note role) believes in driving consensus.  The discussion has
been healthy and I think the resulting -03 is a great step.

While I know the RFC editor isn't a giant fan of very large author/editor
lists, at this stage in the game it's a good way to show strong, early
support from multiple vendors for the direction of the proposal.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 1 Feb 2012 21:16:28 -0500

Marc,

>    MAC address learned from the received BFD packets.

I won't speak for George's exact scenario, but I have at least one
supporting one.

Some Ethernet chips have a very small MAC cache that can be used for local
addresses.  Presume such a chip that supports 8 entries.  If you have the
standard unicast MAC address, you now have 7 MACs that can be used for other
purposes such as multicast or other unicast purposes such as VRRP.  Once
you've exceeded the maximum number of entries, you may need to always punt
traffic up to your primary CPU (pick your vendor specific lingo) for
processing.  In such circumstances, that traffic may at least be rate
limited.

In such circumstances, using the well known MAC address for bootstrapping
purposes at the sedate BFD setup rate may be acceptable.

My memory may be failing, but discussion around a similar issue may possibly
be found in the VRRP mailing list archives.  Multicast coders may also have
similar experiences to share.

-- Jeff

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 2 Feb 2012 07:59:23 +0200

Marc hi,
Lots of thanks for a prompt response.

Please see some additional comments inline below. As always, I hope they will be useful.

Regards,
     Sasha

________________________________________
From: Marc Binderberger [[email-masked]]
Sent: Thursday, February 02, 2012 1:28 AM
To: Alexander Vainshtein
Cc: Bhatia, Manav (Manav); [email-masked]
Subject: Re: Updated BFD over LAGs draft

Hello Alexander,

[[Sasha]] This would be nice.

[[Sasha]] "some implementation-specific asppects of L3 connectivity" sounds like an accurate description of what you guys try to achieve. So why not to say that? Please note also that there are two sides to ths "partial L3 connectivity verification":
- you may declare a link OK while actually received IP packets are not forwarded properly
- the link may be OK but mico-BFD session fails. This could be the case when, say, your IP stack only accepts unicast IP packets with port (or LAG) destination MAC address (AFAIK, quite a few implementations do that). If you use your "dedicated MAC address" to bring up the session while this is not resolved, the session simly will not come up no matter what really happens...
Maybe you should discuss these two situations (say, in a "Guidelines for the Implementor" Annex to the draft)?

[[Sasha]] Yes, source MAC address could be used as the link identifier.

[[Sasha]] IMHO and FWIW there are two alternative models for the micro-BFD option:
- What you call the  "inject/drop-off" option (i.e., micro-BFD sits somewhere near LACP in the LAG sublayers architecture) - quite straightforward with minimal similarity to the "normal" L3 path
- Aggregator MAC client that relies on some Distributor properties to send micro-BFD packets via the specific links - impoved similarity to the regular L3 path but requires these special Distributor properties.

There may be some mixed models as well. IMHO these things should be spelled out explicitly.

We had a lot of this API discussion and I think what it was missing was: the API is possible - the drop/add example is enough for this statement - and anything else is up to the implementor.
[[Sasha]] Yes of course - and this is why I would suggest a "Guidelines to teh Implementor" Annex. You do not really want a standard with too many options for non-interoperable implementations IMHO.

[[Sasha]] Why not to send all these packets with the LAG MAC address as the destination?
Anyway your L3 stack would need this address for sending unicast IP traffic (be it via ARP, PPPoE, DHCP snooping, configuration or anything else).

IMHO and FWIW you need a dedicated MC MAC in the "inject/drop-off" architecture. In the "Aggregator MAC client"
architecture you should send packets to the peer LAG MAC address but provide specific Source MAC addresses to distribute them to the appropriate member links.

[[Sasha]] How does your Ethernet port oeperate as part of a LAG without micro-BFD? The peer sends its unicast P packets with the LAG MAC address as destination - same for all the member links, so it is apt to differ from the physical ("embedded") port MAC address at least for some of the member ports (assuming that there is more than one link in your LAG). I.e., if your HW does not accept unicast Ethernet packets with Destination MAC different from its "embedded" address, it cannot be used in LAG regardless of micro-BFD. Did I miss something here?

[[Sasha]] I understand the need for a multicast MAC address in the "drop-off/inject" architecture. I do not see a need for a dedicated unicast MAC addess.

[[Sasha]] OK

Yes, thanks a lot!

Regards, Marc

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 2 Feb 2012 09:26:09 +0200

Jeff,
Lots of thanks for a prompt and clarifying response.

I fully agree with your explanation regarding limited size of the MAC filtering cache in various Ethernet chips.
This problem is, indeed, well recognized and must be addressed.

IMHO and FWIW such a limitation would be best served if micro-BFD packets would use the peer LAG MAC address as the destination address.
This is definitely possible if LACP is present and initiates micro-BFD sessions as described in Section 3.1 of the draft.

LAG without BFD presents lots of problems. E.g., one could imagine the case (micro-BFD notwithstanding) when:
- Layer 1 faults of the member links cannot be detected/used so that failed links remain Distributing
- ARP request/response packets are distributed to failed links (because this is how the Distributor works) and hence blackholed
- As a consequence, unicast IP would not be able use the seemingly alive LAG at all.

Do we really want to remedy this with micro-BFD sessions? If yes, then 
(a) IMHO it is worth explicitly noting such a scenario
(b) A well-known MAC address for micro-BFD sessions would really help.

My 2c,
     Sasha

________________________________________
From: Jeffrey Haas [[email-masked]]
Sent: Thursday, February 02, 2012 4:16 AM
To: Marc Binderberger
Cc: Alexander Vainshtein; [email-masked]
Subject: Re: Updated BFD over LAGs draft

Marc,

-- Jeff

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 2 Feb 2012 21:03:03 +0530

Hi Sasha,

Thanks for the comments. These are indeed very helpful.

[clipped]

> indeed. If it helps we could talk about "allows to verify 
> aspects of L3 connectivity". If and how much of the L3 
> connectivity and code path is tested with the BFD 
> implementation is indeed and deliberately left to the implementor.
> 
> [[Sasha]] "some implementation-specific asppects of L3 
> connectivity" sounds like an accurate description of what you 
> guys try to achieve. So why not to say that? Please note also 
> that there are two sides to ths "partial L3 connectivity 
> verification":
> - the link may be OK but mico-BFD session fails. This could 
> be the case when, say, your IP stack only accepts unicast IP 
> packets with port (or LAG) destination MAC address (AFAIK, 
> quite a few implementations do that). If you use your 
> "dedicated MAC address" to bring up the session while this is 
> not resolved, the session simly will not come up no matter 
> what really happens...

IS-IS has similar properties as the micro BFD sessions that we're describing. Theoretically its possible in some buggy implementations for IS-IS adjacency to come up, while the IP traffic learnt over the ISIS session to get dropped. This however has not detered people from using IS-IS.

> Maybe you should discuss these two situations (say, in a 
> "Guidelines for the Implementor" Annex to the draft)?
> 
> > 3.  In Section 2.2. I've found a couple of statements that 
> imply availability of the information about the physical 
> ingress port of an IP packet to
> >     the appropriate "handling" entity.  This looks to me 
> like a strong requirement because (and this has been 
> discussed on the list)

I think this information is trivially available in all implementations. If not, then how do such implementations handle IS-IS?

> > 4. In the same section, there are certain issues regarding 
> MAC addresses in the Ethernet encapsulation of the micro-BFD packets:
> > -    The Source MAC address is required to match the MAC 
> address of the physical port on which the micro-BFD session runs. This
> >       implies bypass of the "regular" IP stack (because it 
> would use the same Source MAC address for all the packets 
> sent via LAG) and,
> >       incidentally, compromises the claim that 
> UDP/IP-encapsulation of the micro-BFD packets helps to verify 
> IP connectivity over the link.
> 
> It is a balance between staying close to the layer-3 path and 
> doing what is written in the draft. And it is again a matter 
> of the implementors ideas. You could well run the full 
> "normal" path and as a last step overwrite all the MAC, port, 
> etc values. That would catch all the errors that may happen 
> in the normal path.
> 
> Anyway your L3 stack would need this address for sending 
> unicast IP traffic (be it via ARP, PPPoE, DHCP snooping, 
> configuration or anything else).

We cannot use LAG MAC address as the dest MAC since this requires resolving ARP before the micro sessions are set up. This means that ARP resolution must work on links that are currently down. This to me is an egregious violation of the ARP standards.

[clipped]
 
> > 6. The text in the same section states that " BFD, as a 
> layer 3 protocol, is viewed as running  across the LAG, with  
> load balancing constraints
> >    ensuring  particular micro BFD sessions are effectively 
> bound to particular member links". Does this mean that 
> micro-BFD sessions send and receive
> No, we are not trying to make any such statement about the 
> service interface to use. What is wants to say is that 
> "ensuring particular micro BFD sessions are effectively bound 
> to particular member links" is required. I.e. that the ssion 
> packets go out from the right port.

I agree that the text in -03 isnt very clear. We will fix this in -04. 

Cheers, Manav

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 2 Feb 2012 21:25:52 +0530

Hi Toms,

> I was, and still continue to be, a big fan of multicast as 
> that saves you a lot of configuration overhead. Any 
> particular reason why you didnt go with that approach?
> 

Yes, I was initially rooting for multicast encapsulation.

My biggest gripe with Unicast was the extensive configuration overload that it entailed. This can be fixed by trivially defining an auto-discovery mechanism that will discover the remote destination IP addresses.

My other gripe with Unicast was with resolving ARP before the micro sessions can be boot strapped. This opened a hole for DoS attacks and if nothing else is a gross violation of the basic IP tenets. This has been fixed in -03 by using a well known IANA assigned MAC for the initial BFD packets.

The primary advantage of using Unicast is that its closer to how other traffic will be dealt with and it makes it possible for us to reuse pretty much the same code and ASIC capabilities that exist today for supporting regular BFD (BBP flavor).

Cheers, Manav

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 2 Feb 2012 17:58:27 +0200

Manav,

Please see some answers/comments inline below (bold blue to make them readable).

I have removed the fragments of the original emails that are not relevant to these comments.

Regards,

     Sasha

> From: Bhatia, Manav (Manav) [mailto:[email-masked]]

> Sent: Thursday, February 02, 2012 5:33 PM

> To: Alexander Vainshtein; Marc Binderberger

> Cc: [email-masked]

> Subject: RE: Updated BFD over LAGs draft

... snipped by Sasha ...

> IS-IS has similar properties as the micro BFD sessions that we're describing.

> Theoretically its possible in some buggy implementations for IS-IS

> adjacency to come up, while the IP traffic learnt over the ISIS session to get

> dropped. This however has not detered people from using IS-IS.

[[[Sasha]]] I fully agree regarding similarity between IS-IS and micro-BFD sessions and popularity of IS-IS. However, to me it cuts both ways:

If IS-IS not using IP encapsulation did not deter people from using it for creating intra-AS IP connectivity, I do not see why micro-BFD without IP encapsulation should be treated differently... but this is actually a side issue.

... snipped by Sasha ...

> > 3.  In Section 2.2. I've found a couple of statements that imply availability of the information about the physical

> > ingress port of an IP packet to   the appropriate "handling" entity.  This looks to me like a strong requirement because (and this has been

> > discussed on the list) some existing  implementations do not provide this information

> I think this information is trivially available in all implementations. If not,

> then how do such implementations handle IS-IS?

[[[Sasha]]] If you run IS-IS over LAG, you only need to know which LAG its packets have been received from.

If you run it over a single physical link, you need the information regarding the physical ingress port.

But I do not see why IS-IS would be interested in the physical ingress port in a LAG.

> > [[Sasha]] Why not to send all these packets with the LAG MAC

> > address as the destination?

> We cannot use LAG MAC address as the dest MAC since this requires

> resolving ARP before the micro sessions are set up. This means that ARP

> resolution must work on links that are currently down. This to me is an

> egregious violation of the ARP standards.

[[[Sasha]]] First of all, ARP in LAG does not run over a specific link - it runs over LAG and resolves the peer LAG IP address to the peer LAG MAC address.

This means that running IP over LAG without LACP over a bunch of links that do not report Layer 1 faults (and hence remain permanently UP) is problematic:

If your Distributor selects a de-facto faulty link to send your ARP, it will never be resolved, and unicast IP will never come up.

However, with LACP this will not happen, because faulty links will be eventually excluded from distribution. At the same time, if LACP is used with LAG, the peer LAG MAC address is also known (LACP carries it across). To me this means that the only possible benefit of using some special MAC address in micro-BFD is LAG without LACP and without Layer 1 failure detection in the member links. Do we really want to remedy this? How often would such a scenario be deployed taking into account that without micro-BFD there is running chance of never bringing up IP forwarding across LAG with some healthy links?

... snipped by Sasha ...

[[[Sasha]]]

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 2 Feb 2012 21:43:07 +0530

Hi Sasha,

[Manav]  This is as you say is a side issue. By using IP encapsulation we leverage the existing code and the chip capabilities to a large extent. The extent of leverage one gets is implementation and platform dependent.

The reason i had brought up IS-IS was to counter your point about micro BFD sessions not relaying the true health of the member links for IP traffic.

... snipped by Sasha ...

[Manav] If an implementation can pass the information about the physical ingress port in one scenario then why cant it pass the same information in case of a LAG? Isnt this an implementation specific issue that we're dealing with here. I would like to know which commercial implementation that wants to run micro BFD sessions will not be capable of passing such information that enables the SW to learn about the ingress port the packet arrived on?

[Manav] I believe a solution gets automatically rejected if we come across even one scenario where it doesnt work. Regular ARP as you yourself noted can fail in the absence of LACP. I think we want a solution that works both in the presence and the absence of LACP.

Cheers, Manav

---

From: Alexander Vainshtein <[email-masked]>
Date: Thu, 2 Feb 2012 19:23:46 +0200

Hi Manav,

Some responses inline (bold purple).

Regards,
     Sasha

From: Bhatia, Manav (Manav) [mailto:[email-masked]]
Sent: Thursday, February 02, 2012 6:13 PM
To: Alexander Vainshtein
Cc: [email-masked]
Subject: RE: Updated BFD over LAGs draft

Hi Sasha,

[[[Sasha]]] We all seem to agree that this is a side issue, so let's stop here.

... snipped by Sasha ...

[[[Sasha]]] One suspicious use case could be the regular Linux IP stack with its socket interface. And, by your logic below, if there is one case when it does not work, this is enough to bring the entire construction down, right?

To make it absolutely clear, I only wanted to explicitly state the requirement on the LAG behavior if it is not part of an appropriate standard.

[Manav] I believe a solution gets automatically rejected if we come across even one scenario where it doesnt work.

[[[Sasha]]] But we've already agreed that behavior of micro-BFD sessions in the presence of LACP and without LACP differs.  Or didn't we?

Regular ARP as you yourself noted can fail in the absence of LACP. I think we want a solution that works both in the presence and the absence of LACP.

[[[Sasha]]] Please see above. The solution that works with LACP is already different from one that works without LACP. So one more point of difference does not weight so much (in my eyes). And IMHO and FWIW, changing the destination MAC address depending on the micro-BFD session state hardly helps with the objective to reuse existing code and HW.

BTW, what really prevents you from using broadcast Destination MAC in the encapsulation? Do you think that LAG members can pass thru a L2 switch on the way? Would not that screw the entire LAG operation (e.g., that a broadcast service frame is always sent via one link and received from one link)?

Cheers, Manav

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Sun, 5 Feb 2012 22:51:24 -0800

> 
> IS-IS has similar properties as the micro BFD sessions that we're
> describing. Theoretically its possible in some buggy implementations
> for IS-IS adjacency to come up, while the IP traffic learnt over the
> ISIS session to get dropped. This however has not detered people from
> using IS-IS.
> 

Not to divert the main discussion, but simply to clarify this point -
describing the issues associated with a lack of fate sharing between a
BFD session and the peer session between the BFD clients as a "buggy
[client] implementation" isn't accurate - or fair to the implementation
owners. Using IS-IS as an example, the base protocol specifies an
adjacency state machine which does not depend on BFD session state. In
order to address the fate sharing issue, the protocol adjacency state
machinery needs to be revised to incorporate BFD session state
appropriately. This has interoperability implications and therefore
requires a change in the specification. In the case of IS-IS RFC 6213
defines the necessary protocol extensions.

   Les

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Tue, 7 Feb 2012 00:29:29 +0530

Hi Les,

When I cited ISIS I did not mean ISIS and BFD interaction - I had meant vanilla RFC 1195 ISIS and IP traffic.

Cheers, Manav 

> From: Les Ginsberg (ginsberg) [mailto:[email-masked]] 
> Sent: Monday, February 06, 2012 12:21 PM
> To: Bhatia, Manav (Manav); Alexander Vainshtein; Marc Binderberger
> Cc: [email-masked]
> Not to divert the main discussion, but simply to clarify this 
> point - describing the issues associated with a lack of fate 
> sharing between a BFD session and the peer session between 
> the BFD clients as a "buggy [client] implementation" isn't 
> accurate - or fair to the implementation owners. Using IS-IS 
> as an example, the base protocol specifies an adjacency state 
> machine which does not depend on BFD session state. In order 
> to address the fate sharing issue, the protocol adjacency 
> state machinery needs to be revised to incorporate BFD 
> session state appropriately. This has interoperability 
> implications and therefore requires a change in the 
> specification. In the case of IS-IS RFC 6213 defines the 
> necessary protocol extensions.
> 
>    Les
> 
> 
> 
>