Subject: Re: [hackathon] [Rift] [RIFT] Routing in Fat Trees "chaos testing" hackathon at IETF 104 in Prague

From: Tony Przygienda <[email-masked]>
Date: Sun, 3 Mar 2019 11:05:20 -0800

Preparing stuff here along instructions, ran it on a t2.micro (spot request
is cheap and works well)

b) no command line completion on interactive ;-{
c) the meta config file restricts to 3 levels only (5 stage folded), good
enough for now but testing more than 3 levels is very interesting
proposition for the future
d) running the example I see tons of those

2019-03-03 18:35:08,355:ERROR:node.if.tx:[spine-2-2:veth-105g-4e] Error
[Errno 111] Connection refused sending IPv6 TIE ProtocolPacket(

and then

2019-03-03 18:35:08,274:ERROR:node.if.tx:[super-4:veth-4e-105g] Error
[Errno 111] Connection refused sending IPv4 TIDE ProtocolPacket

but I see no UDP losses

Udp:
    16103 packets received
    88 packets to unknown port received
    0 packet receive errors
    16224 packets sent
    0 receive buffer errors
    0 send buffer errors

Consequently I see on start  often 2-3 pairs of leafs that can't ping each
other, I did not analyze further, every time different pairs, sometimes all
come up.

monkey testing works fine ... Next iteration will be of course doing that
once positive disaggregation is implemented in python RIFT ;-)

I think @ hackathon first thing we have to write a simple script that
collects the kernel routing tables & compares them for # expected entries
since leaf-2-leaf has such a wide path diversity it's hard to debug,
observe that due to ECMP hashing you can get a leaf being able to ping 2
other leafs in another pod but not the third since it takes different
spine/superspine combination. Generally, to test connectivity on fabric
well, you need a fan-out of UDP source/sink pairs that lead to exhaustive
ECMP coverage (as well as tests quality of load balancing once we get to
the bandwidth balancing in python RIFT)

Yes, connectivity to spine loopbacks and so on has been discussed on the
core team list and is in the draft (-04 is in publication queue) ...
Generally, most robust IP fabric architecture is leaf-2-leaf since relying
on every leaf being able to reach every spine leads unavoidably into
exposing more and more topology towards the leafs with according
scalabiltiy problems on the leafs. So, a fabric where every leaf can reach
every leaf is in a sense "working fine" even if links are lost or some
spines crash. That modulo bandwidth balancing since we want to account for
load being balanced (unsurprisingly, RIFT does all that ;-) if we talk a
single prefix (loopback) on a spine/superspine then PGP and even simpler
stuff can work well but if we assume that specific superspines terminate
lots prefixes and rely on leafs being able to reach them then the fabric
"inverts" and scalbility is lost. Variants of the problem exist by e.g.
having superspines exposing anycast addresses. Once deployment experience
starts to come in I can see a "operational deployment" draft coming up (I
think someone already suggested to write one) ...

thanks

--- tony

On Mon, Feb 25, 2019 at 5:15 PM Bruno Rijsman <[email-masked]>
wrote:

> Hi everyone,
>
> At the Routing In Fat Trees (RIFT) IETF 104 hackathon in Prague we will be
> working on "chaos monkey" testing of two RIFT implementations: the
> RIFT-Python open source implementation and a vendor RIFT implementation.
>
> This hackathon is inspired by the concept of chaos engineering and in
> particular by the chaos monkey tool, which intentionally and randomly shuts
> down servers in a production environment. This is intended to build
> confidence in the system's capability to withstand turbulent and unexpected
> conditions.
>
> In this hackathon we will do something similar for RIFT networks. We will
> build a large fat tree network of RIFT routers. Then we will randomly
> insert failures into the network and make sure that the network behaves
> correctly. We will make sure the RIFT routers don't crash or otherwise
> behave badly. And we will make sure that the routers forwarding tables and
> link-state databases converge correctly.
>
> We will use (and possibly enhance) the chaos testing framework as well as
> the visualization and verification tools that are already part of the
> RIFT-Python open source project.
>
> Even if you are not much interested in the RIFT routing protocol, but you
> are interested in testing large network topologies using container-like
> technology, or in visualization tools, or in chaos testing, this might be
> the right hackathon for you.
>
> *A detailed description of the planned activities at the hackathon is
> available at: http://bit.ly/rift-hackathon-ietf-104
> <http://bit.ly/rift-hackathon-ietf-104>*
>
> The project is listed on the IETF 104 hackathon wiki page:
> https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon (info copied
> below)
>
> Feel free to join us.
>
> Bruno Rijsman
>
>
> Routing In Fat Trees (RIFT) Chaos Monkey Testing
>
> * Champions
>   * Tony Przygienda <prz at juniper.net> (will join in person)
>   * Bruno Rijsman <brunorijsman at gmail.com> (will join remotely)
> * Project(s)
>   * ​RIFT is a STD track WG working on a modern IP fabric/DC routing
> protocol
>   * ​Detailed description of the planned activities at the hackathon at
>     http://bit.ly/rift-hackathon-ietf-104
>   * We have two implementations now:
>     * RIFT-Python open source
>     * Vendor implementation
>     * The open source one has a chaos monkey framework in place we can use
>       to stress protocol implementations and interoperability between
> them.
>       Unusual hackathon but that will be very, very useful for further
>       development and quality of implementations while breaking completely
>       new grounds (chaos testing on model based protocol specifications).
> * Preconditions:
>   * Python knowledge
>   * Familiarity with the open source RIFT code on ​
> https://github.com/brunorijsman/rift-python
>   * RIFT Specification in some detail ​RIFT draft
>   * Familiarity with creating virtual machines in AWS
>
> _______________________________________________
> RIFT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rift
>