Subject: [tcpm] Anycast TCP

From: William Herrin <[email-masked]>
Date: Wed, 29 Mar 2017 22:46:49 -0400

http://bill.herrin.us/network/anycasttcp.html

Enjoy!

-Bill Herrin

-- 
William Herrin ................ [email-masked]  [email-masked]
Dirtside Systems ......... Web: <http://www.dirtside.com/>

---

From: William Herrin <[email-masked]>
Date: Thu, 30 Mar 2017 17:03:28 -0400

> http://bill.herrin.us/network/anycasttcp.html

Greetings,

If there's a more appropriate WG to introduce this to, I sure would
appreciate a pointer.

Regards,
Bill Herrin

---

From: David Lang <[email-masked]>
Date: Thu, 30 Mar 2017 14:28:20 -0700 (PDT)

you should list an optimization flood D, which is has a relationship to C like B 
does to A

no manipulation of sequence numbers, when you get something unknown, send it to 
all peers, but a peer can respond and say "I own that connection", eliminating 
future floods for that connection.

As a further optimization (of B and D), let the peers that you flood to send a 
nack, not just an ack, if you get a nack from all peers, you reset the 
connection.

It may be worth the slowdown to have the "do you own this connection" query be 
explicit, not implicit by forwarding the packets. This increases delays when 
something needs to be forwarded (at least one more round trip between the 
peers), but should greatly simplify the logic needed to communicate the status.

One question, if you use the "flood when not a known connection" procedure, do 
you still need all the kernel hooks? It seems to me that most of them could go 
away if you accepted the potential for a collision if multiple machines pick the 
same starting sequence number.

On Linux, IPTables already has a hook that can send packets to userspace. If you 
have a rule that accepts packets with known connections, and then follow that 
with a rule that forwards non-syn packets to the userspace program, do you still 
need a hook? The conntrack hooks allow you to manipulate the connection tables 
(so that you can take over a TCP connection that is on a cluster peer, very 
similar to this use case). So it seems to me that you may be able to implement a 
PoC of this on existing systems. Then measure where the bottlenecks are and what 
the 'miss rate' really is in the real world

David Lang

On Wed, 29 Mar 2017, William Herrin wrote:

> Date: Wed, 29 Mar 2017 22:46:49 -0400
> From: William Herrin <[email-masked]>
> To: [email-masked]
> Subject: [tcpm] Anycast TCP
>

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 30 Mar 2017 21:30:30 +0000

On 2017-3-30, at 16:03, William Herrin <[email-masked]> wrote:
> If there's a more appropriate WG to introduce this to, I sure would appreciate a pointer.

Until you submit an ID, which with it carries an obligation to disclose IPR, I expect you will not be able to engage in a discussion with many.

Lars

---

From: William Herrin <[email-masked]>
Date: Thu, 30 Mar 2017 23:00:30 -0400

On Thu, Mar 30, 2017 at 5:28 PM, David Lang <[email-masked]> wrote:

> you should list an optimization flood D, which is has a relationship to C
> like B does to A
>
> no manipulation of sequence numbers, when you get something unknown, send
> it to all peers, but a peer can respond and say "I own that connection",
> eliminating future floods for that connection.
>

Hi David,

Agreed. I didn't flesh out the flood versions because they all suffer from
a trivial attack vector: forged payload packets with randomized source
addresses and ports get flooded everywhere. That escalates a DOS against
one node into a DOS against all of them. If you see a way to mitigate the
problem then I think it'll be worth diving deeper in to the flood approach.

> As a further optimization (of B and D), let the peers that you flood to
> send a nack, not just an ack, if you get a nack from all peers, you reset
> the connection.
>

For completeness. I agree that would restore an ability to send a rst in
the flood case at the cost of briefly holding some discardable state.

It may be worth the slowdown to have the "do you own this connection" query
> be explicit, not implicit by forwarding the packets. This increases delays
> when something needs to be forwarded (at least one more round trip between
> the peers), but should greatly simplify the logic needed to communicate the
> status.
>

I don't follow how that would simplify the logic. Would you clarify?

One question, if you use the "flood when not a known connection" procedure,
> do you still need all the kernel hooks? It seems to me that most of them
> could go away if you accepted the potential for a collision if multiple
> machines pick the same starting sequence number.
>

Some of the flood versions need only the RST hook, as noted in Flood C.

> On Linux, IPTables already has a hook that can send packets to userspace.
> If you have a rule that accepts packets with known connections, and then
> follow that with a rule that forwards non-syn packets to the userspace
> program, do you still need a hook?

Outside a test harness, yes. Conntrack doubles the state information kept
for the TCP connections. However, that could be a reasonable shortcut for
prototyping and proving out the variants that don't adjust the sequence
number.

> The conntrack hooks allow you to manipulate the connection tables (so that
> you can take over a TCP connection that is on a cluster peer, very similar
> to this use case). So it seems to me that you may be able to implement a
> PoC of this on existing systems.

Good idea. Something like this:

sysctl net.netfilter.nf_conntrack_tcp_loose=0
sysctl net.netfilter.nf_conntrack_helper=0
iptables -A INPUT -p tcp --dport 80 -m conntrack \
  --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j NFQUEUE --queue-num 0 --queue-bypass

And then use libnetfilter_queue to catch the packets in userspace.

However, that only simulates the RST hook so could only be used to evaluate
the flood versions.

Then measure where the bottlenecks are and what the 'miss rate' really is
> in the real world

We already have pretty good miss-rate numbers from research that's been
done for anycast DNS. I don't have a link handy but the short version is
that better than 99% of clients consistently hit the same anycast node.

Regards,
Bill Herrin

---

From: William Herrin <[email-masked]>
Date: Thu, 30 Mar 2017 23:01:40 -0400

On Thu, Mar 30, 2017 at 5:30 PM, Eggert, Lars <[email-masked]> wrote:

> Until you submit an ID, which with it carries an obligation to disclose
> IPR, I expect you will not be able to engage in a discussion with many.
>

Hi Lars,

That would be a discouraging shift from the open flow of ideas the last
time I participated in a WG.

For the record, while I've integrated constructive criticism from a couple
very generous individuals, the design is mine alone. I assert no IP rights
though I'd certainly appreciate being invited to join as an author of any
IDs that eventually result. To the best of my knowledge, there is no
comparable work which would allow another to assert IP rights. Certainly I
did not find any when I searched.

Regards,
Bill Herrin

---

From: David Lang <[email-masked]>
Date: Thu, 30 Mar 2017 21:30:27 -0700 (PDT)

> problem then I think it'll be worth diving deeper in to the flood approach.

I think that the nack capability is a significant limit in the damage a flood 
can do (since the nacks will quickly shutdown the multiplication)

> I don't follow how that would simplify the logic. Would you clarify?

In your example, you were talking about sending the traffic out through tunnels 
to the nodes, and they would have to watch the traffic in the tunnel to detect 
that something new was showing up and send an ack.

If instead the node receiving the unknown traffic has an 'out of band' request 
to the other nodes "do you have connection X" with an explicit ack/nack, the 
other nodes don't have to watch the gre tunnel and decide if they need to send 
an ack based on that.

this involves a round-trip to the other nodes, but you are only sending the 
connection info (which can be packed many to a packet) and you don't send the 
actual payload packets out unless you get an ack back from someone.

This would also cut down on the dos multiplication factor as each malicious 
datastream isn't replicated in full, only it only generates a small chunk of 
traffic to the other nodes.

It wouldn't completely eliminate the vulnerability, but it would greatly reduce 
it.

> Some of the flood versions need only the RST hook, as noted in Flood C.

if you can create a raw IP packet, can't you create one that is a rst for a 
particular connection?

> However, that only simulates the RST hook so could only be used to evaluate

true, but that may be enough to show the value of the concept and evaluate if 
the flood versions can be tamed enough to be useful.

along similar lines, I'll point out multicast-mac (in linux, it's known as 
CLUSTERIP) that lets you put a number of boxes in one place with the same IP 
address and have each of them handle a subset of the traffic (the nic still sees 
the full load on each machine, but userspace only sees a subset.

so you have

big_pipe ->
   switch (multiple ->)
     server nic ->
       clusterip (traffic reduced to 1/servers at location) ->
         conntrack ->
           new/established (normal userspace handling)
           netfilter ->
             userspace, check nak list ->
               if on nak list, reset
               else send query to peers ->
                 if ack, send data to peer through gre
                 else add to nak list reset connection

It may be possible to do some trickery with conntrack to have the kernel send 
the naks (thinking out loud, if you set the conntrack state for the connection 
to fin, will it send the nak for you before the iptables new/established check?)

I don't like ideas that require any flags get set on packets that end up going 
to clients. Good firewalls block unknown flags (as we saw when ecn was 
introduced), so keeping all the tweaks local to the networks where the anycast 
servers are is a far better option.

David Lang