Subject: [AAA-WG]: [issue] Only DWR/DWA to be used to supervise transport connection

From: "Bo Landarv" <[email-masked]>
Date: Thu, 11 Apr 2002 10:50:16 +0200

Submitter name: Bo Landarv
Submitter email address: [email-masked]
Date first submitted: April 11, 2002
Reference: 
Document: transport-06
Comment type: T
Priority: 1
Section: Appendix A
Rationale/Explanation of issue:

Appendix A gives the impression that any request can be used to
supervise the connection. If we have an outstanding session-related
request it can take the role of the DWR.

The DWR/DWA sending and receiving is a simple mechanism between
two neighbours. An arbitrary request/answer pair, on the other hand,
might involve proxies, complicated processing etc and should not
IMHO be able to in any way affect the status of the connection.

Requested change:

State that "Pending" is set to true if and only if there is an 
outstanding, unanswered DWR. 
Procedure OnSendRequest should not call action SetWatchdog().

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 11 Apr 2002 06:54:20 -0700 (PDT)

Assigned issue #334.

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 11 Apr 2002 08:04:57 -0700 (PDT)

Let me see if I can summarize your issue and determine the source of the
problem.

That goal of the heartbeat is to allow peers to determine the
"liveness" of their neighbors without adding unnecessary traffic. To quote
from RFC 2865, section 2.6:

"  Some implementers have adopted the practice of sending test RADIUS
   requests to see if a server is alive.  This practice is strongly
   discouraged, since it adds to load and harms scalability without
   providing any additional useful information."

As a result, a heartbeat is only sent if there has been no other traffic
to verify liveness of the peer. In a proxy or relay system, it is possible
that the response will be delayed due to many factors, such as an
overloaded or failed downstream proxy or server. It is important that
issues encountered downstream not induce failover on a perfectly
functional connection, since the immediate neighbor may have little to 
do with the cause of the delay. 

In the transport state machine described in Appendix A, normal traffic 
sent to a peer is included as part of the watchdog activity in order to
avoid sending additional traffic unless absolutely necessary. The watchdog
timer is reset whenever a response is received, including receipt of a
watchdog response. On sending a message, if the send queue is empty, then
the watchdog timer is reset. If the watchdog timer expires and the send
queue is empty, then a watchdog packet is sent. 

If the watchdog timer expires and a response is pending, then failover is 
initiated. 

Let's look at a hypothetical case:

A ------------- B ----------------- C

                B' ---------------  C'

Diameter client A is using relay B as the primary to talk to server C. 
However it also has relay B' as its secondary, which it can use to talk to
server C or C'. 

Let's assume that B and B' are both healthy and that the connection
between them and A are fine, but that there is a problem with C. Ideally,
we'd want B to failover from C to C', and have this not affect the
connection between A and B, since there is nothing wrong with this. 

If there was nothing in the send queue when A sent its request to B, then
the watchdog timer would have been reset. Since there was a problem with
C, B has to initiate its own failover to C', and as a result, a response
may not be sent before A's watchdog timer expires. Since there is
something in the send queue (A's request to B), failover will be
initiated. This is wrong because had a watchdog been sent to B, then it
would have received a reply and no failover would have occurred. 

Therefore, I think that the requested change is that failover only be
initiated on failure to receive a response to a watchdog packet. While a
successful response to any Diameter request can reset the watchdog timer,
only a failure to receive a watchdog response can cause failover. 

One way to fix this might be to distinguish between what is in the send 
queue once the watchdog timer expires. Today, failover is initiated if the
timer expires with something in the send queue, no matter what it
is. However, when the watchdog timer is reset, the type of
request ("Watchdog", or "Non-Watchdog") that is pending can be put into a
state variable. This would enable failover only to be initiated when the
timer expires with a "Watchdog" pending. If a "Non-Watchdog" is pending,
then the watchdog timer can be reset and a watchdog packet can be sent.

The issue with this is that it could require two watchdog intervals to
detect a failure instead of one, in case there really is a failure in the
dowstream relay or proxy.

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 11 Apr 2002 08:49:23 -0700 (PDT)

The requested change is:

State that "Pending" is set to true if and only if there is an 
outstanding, unanswered DWR.  Procedure OnSendRequest should not call
action SetWatchdog().

I think that this is correct. Let me describe what the overall algorithm
now looks like:

1. The watchdog timer is reset whenever any response is received from the 
peer, including receipt of a watchdog response. 

2. If the watchdog timer expires and there is no pending watchdog
response, then the watchdog packet is sent and the watchdog timer is
reset. 

3. If the watchdog timer expires and there is a pending watchdog response,
then failover is initiated. 

This simplifies things, since there is no need to deal with requests other
than watchdogs. It appears to deal with the example:

1. If no packet had been sent in a while, then a watchdog packet would
have been sent and the timer reset. As long as B kept answering the
watchdog packets, the connection would stay up, regardless of whether B
was failing over to C'. 

2. As long as requests sent through B were being answered, the watchdog
timer would be continuously reset and no watchdog packets would need to be
sent. However, when C dies, the responses would stop and once the
timer expired, then a watchdog would be sent from A to B. Since B would
respond, the connection from A to B would stay up.

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 11 Apr 2002 15:57:33 -0700 (PDT)

I've proposed new text in a "transport-07" strawman document to
address issue #334. The new version is available for inspection at:

http://www.drizzle.com/~aboba/AAA/draft-ietf-aaa-transport-07.txt

Please let me know ASAP if the proposed text is satisfactory. 
Substantial changes were required in section 3.4 as well as in
Appendix A to provide the fix, so that a careful read of the new text
would be appreciated.

---

From: "Bo Landarv" <[email-masked]>
Date: Fri, 12 Apr 2002 11:31:32 +0200

Hi!

I am happy with the changes.
I just have some comments/questions.

* The base draft some versions ago made the distinction between 
  Ti/Tr/Tw timer values. Maybe to separate Ti and Tr doesn't 
  give that much, but how about the distinction between the 
  time when to send the next DWR and when to expect a DWA? 
  I think it might be useful to set different times for these.

* In chapter 3.4.1 and Appendix A it says "Tw responses are pending",
  "watchdog responses are pending", "unanswered watchdog requests".
  Always in plural and not singular. Is this because you include
  retransmissions of lower protocol layers? I would have felt less
  confused with singular.

* Chapter 3.4.1 item 2 "On sending a watchdog request, if no watchdog
  responses are pending, then Tw is reset."
  I get some bad vibrations here. Couldn't we say something like
  "On sending a watchdog request, Tw is reset" and
  "We only send a watchdog request when there is no watchdog response
  pending."

* Appendix A. You switched the order of the functional approach and
  the state machine. Perfectly fine with me.

* Appendix A. In the functional approach you have two assignments
  for variable Pending with "==". Should only be one = token.

* Appendix A. Definition of Sendwatchdog(pcb) instead of 
  SendWatchdog(pcb).

* Appendix A. A new event Senddog was added. I don't think you
  need this nor the transitions for Senddog. 
  SendWatchdog() does the job, doesn't it?

That's all.
/Bo Landarv

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Bernard Aboba
> Sent: den 12 april 2002 00:58
> To: Bo Landarv
> Cc: [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [AAA-WG]: [issue 334] Only DWR/DWA to be used to supervise
> transport connection
> 
>

---

From: Bernard Aboba <[email-masked]>
Date: Sat, 13 Apr 2002 19:03:41 -0700 (PDT)

Thanks for the comments on the proposed transport-07 text. A new
version reflecting the feedback is available for inspection at:

Comments welcome.

---

From: "Bo Landarv" <[email-masked]>
Date: Mon, 15 Apr 2002 10:29:54 +0200

I had one comment in my last mail which I would like to hear some
reasoning about.

The base draft originally made the distinction between Ti/Tr/Tw
timer values. The transport draft only mentions Tw to do the job
also of Ti/Tr.

I think it would be a good idea to be able to set different timer
values for interval when to probe the connection (Ti/Tr) and
for when to require reception of a DWA answer (Tw).

I could post this as an issue if you would like.

/Bo

> Sent: den 14 april 2002 04:04
> To: Bo Landarv
> Subject: Re: [AAA-WG]: [issue 334] updated transport-07 text
> 
>

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 15 Apr 2002 07:04:26 -0700 (PDT)

As long as a connection remains open, a reliable transport will retransmit
an outstanding segment. Since the transport handles retransmissions, it
doesn't make sense for the watchdog to be retransmitted at the application
layer. 

Thus, if there were separate timers for the "waiting interval" and the
"time between watchdogs" the reality would still be that the watchdog was
being continually retransmitted until the connection was closed. 

Moreover, research such as [Paxson] shows that in practice, substantial
"waiting intervals" are required to assure against spurious
failover. For example, route oscillations may persist as long as 30
seconds. That means that in practice the "waiting interval" and the time
between watchdogs are of the same order. 

Since the "waiting interval" is of the same order as the
"watchdog" interval, in order to reduce the complexity of the algorithm,
multiple timers were eliminated.