Subject: Re: [Dots] DOTS & NAT (was RE: DOTS Requirements review (-06))

From: "Jon Shallow" <[email-masked]>
Date: Wed, 1 Nov 2017 14:01:18 -0000

Hi Andrew,

I have been thinking this through as to how to practically implement this, come up with some recommendations and then with some suggested text as a possible substitute.

Recommendations
===============

1) In peace time, only the DOTS Client sends heartbeats (so no need to worry about NAT) at a lower rate
2) In attack time, both the DOTS client and DOTS server send heartbeats (and continue to send them, even if missing heartbeat count is exceeded)
3) signal 'config' includes both peace time and attack time heartbeat rates configuration support
4) Heartbeat rates switch over on both DOTS client and DOTS server when mitigating, and switch back to peace time when there are no mitigation requests

My thinking behind the recommendations
===================================

Two scenario situations
* Peace time
* Under attack of consequence (flooded inbound pipe)

My experience is that the transition time from 'Peace time' to 'Under attack of consequence' is very small - and once it has occurred any traffic over the DOTS signal channel that is of CoAP type "Confirmable" will fail.  So, any signal of type 'config' will fail, but signal of type 'mitigate' will probably get through as it is of type 'non-confirmable'.

We have determined from 'flooded inbound pipe' tests that the heartbeat requests from the DOTS client get through to the DOTS server, but nothing gets through to the DOTS client.  So the DOTS server can determine the session is still up and keep the session "warm" should the DOTS client need to blindly send off a mitigation request.  The caveat here is that the DOTS client needs to continue to send heartbeat requests even though the DOTS client has exceeded the missing heartbeat response counts.

During time of attack, to me, it is important that the heartbeats run at a sufficient frequency to keep any NAT / stateful devices allowing through 2-way traffic - we have generally agreed elsewhere that this should be the order of 30 seconds, but could be shorter or longer.

In peacetime, I do not believe that we need to keep state through any NAT / stateful devices for traffic initiated by DOTS server.  If the DOTS server sends no heartbeat requests (or any other initiated traffic), then NAT is not an issue.  Any heartbeats sent by a DOTS client should get the heartbeat responses back through a NAT / stateful device as the RTT should not be large (certainly should be less than 30 seconds!).  The DOTS client could reduce the heartbeat frequency to, say, once an hour (empirical guess) with no issues and the DOTS server can keep the session open ready for attack time as it is continuing to see the heartbeat requests.

As it may not be possible to reconfigure heartbeat rates once an attack has started, it makes some sense to me to have a peacetime heartbeat rate (only DOTS client sends) and attack time heartbeat rate (both DOTS client and DOTS server send).  Once a DOTS server has accepted a mitigation request, and for the continuation of the mitigation request the attack time heartbeat rate is used.  Once the mitigation request is dropped (and there are no other active mitigation requests from the DOTS client) then the peacetime heartbeat rate is used.

Suggested Updates
================

My suggested update to SIG-003 of the requirements spec is as follows:-

   SIG-003  Channel Health Monitoring: DOTS agents MUST support exchange
      of heartbeat messages over the signal channel to monitor channel
      health.  Peer DOTS agents SHOULD regularly send heartbeats to each
      other while a mitigation request is active.  The heartbeat
      interval during active mitigation is not specified, but SHOULD be
      frequent enough to maintain any on-path NAT bindings during
      mitigation.  When there is no active mitigation, there is no need for the 
      DOTS server to send heartbeats, and the DOTS client can use a lower 
      Heartbeat rate as then there is no requirement to maintain any on-path
      NAT bindings.

      To support scenarios in which loss of heartbeat is used to trigger
      mitigation, and to keep the channel active, DOTS clients MAY
      solicit heartbeat exchanges after successful mutual
      authentication.  When DOTS agents are exchanging heartbeats and no
      mitigation request is active, either agent MAY request changes to
      the heartbeat rate.  For example, a DOTS server might want to
      delay or cease heartbeat exchanges when an active DOTS client has
      not requested mitigation, in order to control load.

      Following mutual authentication, a signal channel MUST be
      considered active until a DOTS agent explicitly ends the session,
      or either DOTS agent fails to receive heartbeats from the other
      after a mutually agreed upon timeout period has elapsed.  Because
      heartbeat loss is much more likely during volumetric attack, DOTS
      agents SHOULD avoid signal channel termination when mitigation is
      active and heartbeats are not received by either DOTS agent for an
      extended period.  In such circumstances, DOTS clients MAY attempt
      to reestablish the signal channel.  DOTS servers SHOULD monitor
      the attack, using feedback from the mitigator and other available
      sources, and MAY use the absence of attack traffic and lack of
      client heartbeat requests as an indication the signal channel is defunct.

==========

Regards

Jon

---

From: "Konda, Tirumaleswar Reddy" <[email-masked]>
Date: Thu, 2 Nov 2017 10:48:38 +0000

>-----Original Message-----
> From: Jon Shallow [mailto:[email-masked]]
> Sent: Wednesday, November 1, 2017 7:31 PM
> To: Konda, Tirumaleswar Reddy <[email-masked]>;
> Mortensen, Andrew <[email-masked]>; [email-masked]
> Subject: RE: [Dots] DOTS & NAT (was RE: DOTS Requirements review (-06))
> 
> Hi Andrew,
> 
> I have been thinking this through as to how to practically implement this,
> come up with some recommendations and then with some suggested text as
> a possible substitute.
> 
> Recommendations
> ===============
> 
> 2) In attack time, both the DOTS client and DOTS server send heartbeats (and
> continue to send them, even if missing heartbeat count is exceeded)
> 4) Heartbeat rates switch over on both DOTS client and DOTS server when
> mitigating, and switch back to peace time when there are no mitigation
> requests
> 
> My experience is that the transition time from 'Peace time' to 'Under attack
> of consequence' is very small - and once it has occurred any traffic over the
> DOTS signal channel that is of CoAP type "Confirmable" will fail.  So, any
> signal of type 'config' will fail, but signal of type 'mitigate' will probably get
> through as it is of type 'non-confirmable'.

It depends on the type of DDoS attack, In case of slowloris DDoS attack, the incoming link to the DOTS client will most likely not be saturated.
In addition, in some deployments, where UDP is blocked, DOTS-over-TCP is required.

> 
> We have determined from 'flooded inbound pipe' tests that the heartbeat
> requests from the DOTS client get through to the DOTS server, but nothing
> gets through to the DOTS client.  So the DOTS server can determine the
> session is still up and keep the session "warm" should the DOTS client need
> to blindly send off a mitigation request.  The caveat here is that the DOTS
> client needs to continue to send heartbeat requests even though the DOTS
> client has exceeded the missing heartbeat response counts.

Yes, this mechanism is discussed in the next re-version. 

> 
> During time of attack, to me, it is important that the heartbeats run at a
> sufficient frequency to keep any NAT / stateful devices allowing through 2-
> way traffic - we have generally agreed elsewhere that this should be the
> order of 30 seconds, but could be shorter or longer.
> 
> In peacetime, I do not believe that we need to keep state through any NAT /
> stateful devices for traffic initiated by DOTS server.  If the DOTS server sends
> no heartbeat requests (or any other initiated traffic), then NAT is not an issue.
> Any heartbeats sent by a DOTS client should get the heartbeat responses
> back through a NAT / stateful device as the RTT should not be large (certainly
> should be less than 30 seconds!).  The DOTS client could reduce the
> heartbeat frequency to, say, once an hour (empirical guess) with no issues
> and the DOTS server can keep the session open ready for attack time as it is
> continuing to see the heartbeat requests.

The DOTS server also needs to perform heartbeat checks to detect and delete defunct DOTS session, otherwise the DOTS server could end-up consuming cryptographic state 
for DOTS session abruptly disconnected by the DOTS client. Please see the UDP usage best practices in https://tools.ietf.org/html/rfc8085 

<snip>
   A UDP application with a long-lived association between the sender
   and receiver, ought to be designed so that the sender periodically
   checks that the receiver still wants ("consents") to receive traffic
   and need to be designed to stop if there is no explicit confirmation
   of this [RFC7675].  Applications that require communications in two
   directions to implement protocol functions (such as reliability or
   congestion control) will need to independently check both directions
   of communication, and may have to exchange keep-alive messages to
   traverse middleboxes (see Section 3.5).

> 
> As it may not be possible to reconfigure heartbeat rates once an attack has
> started, it makes some sense to me to have a peacetime heartbeat rate (only
> DOTS client sends) and attack time heartbeat rate (both DOTS client and
> DOTS server send).  Once a DOTS server has accepted a mitigation request,
> and for the continuation of the mitigation request the attack time heartbeat
> rate is used.  Once the mitigation request is dropped (and there are no other
> active mitigation requests from the DOTS client) then the peacetime
> heartbeat rate is used.

I don't understand the need for different heartbeat rates. 
What problems do you see with the current mechanism ?

-Tiru

---

From: "Jon Shallow" <[email-masked]>
Date: Thu, 2 Nov 2017 12:02:38 -0000

Hi Tiru,

See inline.

Regards

Jon

-----Original Message-----
From: Dots [mailto: [email-masked]] On Behalf Of Konda, Tirumaleswar
Reddy
Sent: 02 November 2017 10:49
To: Jon Shallow; Mortensen, Andrew; [email-masked]
Subject: Re: [Dots] DOTS & NAT (was RE: DOTS Requirements review (-06))

> I have been thinking this through as to how to practically implement 
> this, come up with some recommendations and then with some suggested 
> text as a possible substitute.
> 
> Recommendations
> ===============
> 
> 2) In attack time, both the DOTS client and DOTS server send 
> heartbeats (and continue to send them, even if missing heartbeat count 
> is exceeded)
> 4) Heartbeat rates switch over on both DOTS client and DOTS server 
> when mitigating, and switch back to peace time when there are no 
> mitigation requests
> 
> My experience is that the transition time from 'Peace time' to 'Under 
> attack of consequence' is very small - and once it has occurred any 
> traffic over the DOTS signal channel that is of CoAP type 
> "Confirmable" will fail.  So, any signal of type 'config' will fail, 
> but signal of type 'mitigate' will probably get through as it is of type
'non-confirmable'.

It depends on the type of DDoS attack, In case of slowloris DDoS attack, the
incoming link to the DOTS client will most likely not be saturated.
In addition, in some deployments, where UDP is blocked, DOTS-over-TCP is
required.

Jon> If the link is not saturated then DOTS will continue to work as
expected.  I am thinking of the boundary conditions - an example being pipe
full where DOTS still needs to be functional.  
Jon> Agreed - CoAP over TLS is not ideal and is likely to fail under
boundary conditions.

> 
> We have determined from 'flooded inbound pipe' tests that the 
> heartbeat requests from the DOTS client get through to the DOTS 
> server, but nothing gets through to the DOTS client.  So the DOTS 
> server can determine the session is still up and keep the session 
> "warm" should the DOTS client need to blindly send off a mitigation 
> request.  The caveat here is that the DOTS client needs to continue to 
> send heartbeat requests even though the DOTS client has exceeded the
missing heartbeat response counts.

> 
> During time of attack, to me, it is important that the heartbeats run 
> at a sufficient frequency to keep any NAT / stateful devices allowing 
> through 2- way traffic - we have generally agreed elsewhere that this 
> should be the order of 30 seconds, but could be shorter or longer.
> 
> In peacetime, I do not believe that we need to keep state through any 
> NAT / stateful devices for traffic initiated by DOTS server.  If the 
> DOTS server sends no heartbeat requests (or any other initiated traffic),
then NAT is not an issue.
> Any heartbeats sent by a DOTS client should get the heartbeat 
> responses back through a NAT / stateful device as the RTT should not 
> be large (certainly should be less than 30 seconds!).  The DOTS client 
> could reduce the heartbeat frequency to, say, once an hour (empirical 
> guess) with no issues and the DOTS server can keep the session open 
> ready for attack time as it is continuing to see the heartbeat requests.

The DOTS server also needs to perform heartbeat checks to detect and delete
defunct DOTS session, otherwise the DOTS server could end-up consuming
cryptographic state for DOTS session abruptly disconnected by the DOTS
client. Please see the UDP usage best practices in
https://tools.ietf.org/html/rfc8085 

Jon> If the DOTS server fails to see a heartbeat CoAP ping request from the
DOTS client within the agreed heartbeat repeat rate * some factor
(suggestion 2 * + tolerance to handle a missing heartbeat), then the session
can be declared as dead.  I would also expect the DOTS client to be sending
a session close at the DTLS level if the client is going away (obviously not
if the client crashes!).

<snip>

> 
> As it may not be possible to reconfigure heartbeat rates once an 
> attack has started, it makes some sense to me to have a peacetime 
> heartbeat rate (only DOTS client sends) and attack time heartbeat rate 
> (both DOTS client and DOTS server send).  Once a DOTS server has 
> accepted a mitigation request, and for the continuation of the 
> mitigation request the attack time heartbeat rate is used.  Once the 
> mitigation request is dropped (and there are no other active 
> mitigation requests from the DOTS client) then the peacetime heartbeat
rate is used.

Jon> When mitigation is active, heartbeats need to be running at a rate that
keeps traffic flowing through middleboxes that may be NAT devices.  In peace
time, this is creating a lot of chit-chat that is not really needed and
could be slowed down (some people appear to be saying that they want no or
very little heartbeats).  If the rate becomes less frequent than the
recognized 30 seconds (even though RFC recommendations are for higher
values) the DOTS server will start to suffer heartbeat failure if traffic is
passing through a NAT device.

Jon> By there being two different sets of heartbeat values (which could be
the same values) and which set is used is based on whether mitigating or not
removes the need for having to do a CON Configuration change whilst under
attack to set the required heartbeat parameters for the attack conditions.

-Tiru

> Suggested Updates

_______________________________________________
Dots mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dots

---

From: "Konda, Tirumaleswar Reddy" <[email-masked]>
Date: Thu, 2 Nov 2017 16:08:24 +0000

> Sent: Thursday, November 2, 2017 5:33 PM
> From: Dots [mailto: [email-masked]] On Behalf Of Konda,
> Tirumaleswar Reddy
> The DOTS server also needs to perform heartbeat checks to detect and
> delete defunct DOTS session, otherwise the DOTS server could end-up
> consuming cryptographic state for DOTS session abruptly disconnected by
> the DOTS client. Please see the UDP usage best practices in
> Jon> If the DOTS server fails to see a heartbeat CoAP ping request from
> Jon> the
> can be declared as dead.  I would also expect the DOTS client to be sending a
> session close at the DTLS level if the client is going away (obviously not if the
> client crashes!).

Your proposal violates the UDP usage guidelines discussed in RFC8085 (please see the last paragraph in Section 6). 

> Jon> When mitigation is active, heartbeats need to be running at a rate
> Jon> that
> keeps traffic flowing through middleboxes that may be NAT devices.  In
> peace time, this is creating a lot of chit-chat that is not really needed and
> could be slowed down (some people appear to be saying that they want no
> or very little heartbeats). If the rate becomes less frequent than the

I don't think the proposed keep-alive interval is the problem, WebRTC uses consent checks (30 second expiry on consent https://tools.ietf.org/html/rfc7675) and 
QUIC sends keepalives every 15 to 30 seconds.

-Tiru

> 
> Jon> By there being two different sets of heartbeat values (which could
> Jon> be
> >       mitigation.  When there is no active mitigation, there is no
> > need
> for the