Subject: Re: [spring] [Idr] New draft for data center gateways

From: Robert Raszuk <[email-masked]>
Date: Mon, 23 May 2016 20:14:39 +0200

Hi Adrian,

Many thx for sharing the document. Just starting to read it one assertion
repeated at least twice got my attention:

"Segment routing (SR) [I-D.ietf-spring-segment-routing] is a popular
protocol mechanism for operating within a DC"

By "popular protocol mechanism" when building non blocking L3 CLOS IP
fabric are you referring to the below proposal or something else ?

https://tools.ietf.org/html/draft-filsfils-spring-large-scale-interconnect-02

Also by "popular" do you mean something which is perhaps deployed in
production to any level or author's definition of "popular" refers to
something real deployment are yet to see ?

Honestly having been involved in building large scale data centers for the
last few years I have not see SR being anywhere close to be used there.
Especially MPLS flavor of it :)

If anything within DCs draft-lapukhov-bgp-sdn-00 is easy to use and
provides more then sufficient flexibility within properly constructed
intra-dc fabric in terms of traffic engineering.

Many thx,
Robert.

On Mon, May 23, 2016 at 7:42 PM, Adrian Farrel <[email-masked]> wrote:

> Hi,
>
> Just posted
> https://datatracker.ietf.org/doc/draft-drake-bess-datacenter-gateway/
>
> We think this covers an important hole. The document defines a mechanism
> using
> the BGP Tunnel Encapsulation attribute to allow each gateway router to
> advertise
> the routes to the prefixes in the data center site to which it provides
> access,
> and also to advertise on behalf of each other gateway to the same data
> center
> site.
>
> We posted this as  BESS draft, so discussion there, please until such time
> that
> the various chairs tell us to move the discussion somewhere else.
>
> Thanks,
> Adrian
>
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 23 May 2016 22:14:16 +0200

Dear Authors,

Question 1:

Assume that prefix X has been advertised with tunnel attribute as described
in the draft with both GW1 and GW2 entry points to Egress DC site.

So remote GW in Ingress DC site receives at least one such advertisement
and as each contains both GW1 and GW2 entries it can engineer flows to
those.

So far so good ... but what happens when link between PE (on left side) and
GW1 goes down ?

BGP will after some time remove that path via all ASes, but GW2 will keep
advertsing prefix X as still reachable via both GW1 and GW2 within tunnel
encapsulation attribute as from his perspective nothing will be wrong.

How remote GW in Ingress DC site is now supposed to know that GW1 link to
PE in AS2 went down and stop pushing traffic towards it ?

- - -

Question 2:

What happens if all L3 DC CLOS IP Fabrics use eBGP not IGP ?

- - -

Question 3:

Is per prefix tunnel attribute where say /32 routes may be exchanges flat
(ref calico approach - https://www.projectcalico.org/) really a scalable
solution ?

Best,
Robert.

PS. Just purely as information your inter DC traffic steering problem
description is easily solved already today with one level of indirection -
Example: LISP. Not sure if we need additional flat protocol extensions
here.

On Mon, May 23, 2016 at 8:14 PM, Robert Raszuk <[email-masked]> wrote:

>> We posted this as  BESS draft, so discussion there, please until such
>> time that

---

From: John E Drake <[email-masked]>
Date: Tue, 24 May 2016 21:08:57 +0000

Robert,

Comments inline

Yours Irrespectively,

John

From: spring [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Monday, May 23, 2016 4:14 PM
To: Adrian Farrel
Cc: idr wg; [email-masked]
Subject: Re: [spring] [Idr] New draft for data center gateways

Dear Authors,

Question 1:

Assume that prefix X has been advertised with tunnel attribute as described in the draft with both GW1 and GW2 entry points to Egress DC site.

So remote GW in Ingress DC site receives at least one such advertisement and as each contains both GW1 and GW2 entries it can engineer flows to those.

So far so good ... but what happens when link between PE (on left side) and GW1 goes down ?

BGP will after some time remove that path via all ASes, but GW2 will keep advertsing prefix X as still reachable via both GW1 and GW2 within tunnel encapsulation attribute as from his perspective nothing will be wrong.

How remote GW in Ingress DC site is now supposed to know that GW1 link to PE in AS2 went down and stop pushing traffic towards it ?

[JD]  We were assuming that a) GW/backbone links would be advertised in BGP LS (optionally w/ EPE) and b) a GW that is disconnected from the backbone does not advertise an auto-discovery route.  This will be made explicit in the next revision.

- - -

Question 2:

[JD]   We make no assumptions about what control plane is used within a DC.  After all, the draft is about DC “interconnect”.  (Btw, it should be spelled ‘Clos’ since it’s a person’s last name.)

- - -

Question 3:

Is per prefix tunnel attribute where say /32 routes may be exchanges flat (ref calico approach - https://www.projectcalico.org/) really a scalable solution ?

[JD]   Are you simply repeating the opinion you expressed when the tunnel-encaps draft first came out, or do you have a scalability concern that hasn't already been discussed?

Best,
Robert.

PS. Just purely as information your inter DC traffic steering problem description is easily solved already today with one level of indirection - Example: LISP. Not sure if we need additional flat protocol extensions here.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 24 May 2016 23:27:44 +0200

Hi John,

*[JD]  We were assuming that a) GW/backbone links would be advertised in
> BGP LS (optionally w/ EPE) and b) a GW that is disconnected from the
> backbone does not advertise an auto-discovery route.  This will be made
> explicit in the next revision.  *
>

​Ad a) - EPE works fine with next hop self on the ASBRs of the bacbone. So
if your proposal requires external links to be passive in the backbone IGP
then the draft needs to state it well. Besides it may not be sufficient if
more then one backbone instance is used to interconnect DCs.

Ad b) - The current set of procedures described in section 2 for
constructing auto discovery routes is not sufficient. GW may be
interconnected to more then one backbone each serving different set of
prefixes within each DC (you call it X). So much more would need to go into
the auto-discovery to make this work in practice. And even if you would go
that way it will be slow to re-advertise all prefixes with different tunnel
attribute content.

*[JD]   Are you simply repeating the opinion you expressed when the
> tunnel-encaps draft first came out, or do you have a scalability concern
> that hasn't already been discussed?*
>
>
​I am expressing the observation that tunnel-encaps may work well to pass
static information or information which may change by configiration.

Here you are using it to pass remote reachability info embedded into it. I
find it far from optimal which as combined with possible DC prefix scale is
likely to cause more issues then benefits.

That's why I suggest use of indirection layer to solve this overall
problem. As mentioned one could be LISP or any other SD-WAN solutions.

Thx,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 24 May 2016 23:59:04 +0200

John, Adrian & Eric,

Btw .. indirection I am suggesting may not only be realized by choice of a
new overlay system.

Alternative design would be to only encode a given's DC "COLOR" or "ID"
into the tunnel encapsulation attribute which would not change for each X
then to establish full mesh EBGP multihop sessions between interesting GWs
with each again carrying the tunnel attribute with same set of "IDs" or
"COLORS" as would do atomic prefixes.

- If such full eBGP mesh would be getting large use of stock BGP Route
Server may be used at any DC or in the backbone.

- Over such sessions single prefixes would be exchanged - something what
you refer to auto discovery routes in current document but those would be
only send over EBGP not IBGP.

- This would also remove any need to run intra-DC auto discovery.

- The connectivity restoration time would be significantly reduced as
multihop BFD may be used (directly or to RS).

The above described proposal changes your current architecture however I
believe it significantly simplifies things, removes the concern of scale
and simplifies or even eliminates any new development needed to deploy it
today provided that given's vendor policy already allows to match routes
with specific attribute payload.

Cheers,
R.

PS. If this game is under same administration (which I think it is based on
the requirements) then to mark the routes use of standard or extended
community can be used instead of tunnel encapsulation attribute.