Subject: RE: Detection of DHCP server failures

From: "HIBBS, BARR (SBCSI)" <[email-masked]>
Date: Thu, 6 Apr 2000 09:03:09 -0700

> From: 	Bernard Aboba[SMTP:[email-masked]]
> Sent: 	Wednesday, April 05, 2000 11:12 AM
> 
> In the development of failover and load balancing protocols,
> we have learned that there are many ways that a given service
> can fail. In the most extreme case, this can include catastrophic
> hardware failures that take down the entire machine. However,
> services can also fail in more subtle ways. For example, a
> process or thread may fail, leaving other processes or
> threads intact. Or the process or thread may remain intact,
> but respond incorrectly to queries. Or a hardware subsystem
> may fail, leaving others intact. 
> 
...I certainly agree with your observation, Bernard, and am pleased you've
introduced the topic for discussion.

> In the example of a DHCP server running the DHCP failover
> protocol, the detection of DHCP failure is tied to the
> inability to respond to DHCP failover protocol messages.
> This means that the failover mechanism is not in itself
> tied to the operation of the DHCP server.
> 
...yes, but I believe that the mindset was that processing of packets from
dhcp clients was not a separate process thread from processing of
interserver messages.  Kim, can you expand on this?

> This introduces a number of potential problems, including:
> 
> a. The possibility that the thread or process handling
>    DHCP failover will fail, but not the actual DHCP
>    server process or thread. I believe that the DHCP
>    failover protocol is immune to problems resulting from
>    this. 
> 
...why?  if there are two independent process threads, they presumably must
communicate.  If the packet process completely fails, won't its lack of
response to the failover process be noticed?  But at the same time, if the
packet process only partially fails, continuing to respond to the failover
process but not to client packets, how would the failover process detect
this?

> b. The possibility that the DHCP server process or thread
>    will die but not that the DHCP failover thread or
>    process. In this case, failure will not be detected.
> 
...I don't think you've stated this correctly.  See my comments above.

> c. The possibility that the DHCP server will have a failure
>    that may cause it to send out invalid DHCP packets,
>    while the DHCP failover process or thread remains
>    intact. 
> 
...again, how can this be detected?  If the packet process is sending and
receiving data, the obvious signs of process health such as advancing
message counters won't reveal a problem, and if the packet process and
failover process continue to communicate, the failover process could easily
be fooled.

> d. The possibility that a partial hardware failure may 
>    cripple the DHCP server, while still allowing it to
>    respond to DHCP failover messages. For example, a
>    multi-homed DHCP server could lose a NIC, disabling
>    DHCP service for one or more subnets, without being
>    detected by DHCP failover. 
> 
...yes, but this is no different from some types of hardware failures that
might go undetected in a non-failover server.  For example, I rely on the
networking software to report failure or completion of send requests.
Suppose the NIC suddenly failed such that data was shifted out of the
transmit registers, but not actually written on the wire.  That would likely
not be detected by the network layer, so the server would not detect a send
failure.

> The cases described above, plus others that may be imagined,
> lead me to believe that the DHCP failover mechanism alone
> cannot provide a reliable mechanism for the true determination
> of DHCP server health. 
> 
> To provide such a determination it is necessary to use a variety
> of health monitoring mechanisms, including examination of data
> from the DHCP server MIB, external monitoring involving submission
> of "test" DHCPDISCOVER packets, etc. 
> 
> Through these mechanisms, it may be determined that a DHCP server
> is malfunctioning even though it (and the DHCP Failover protocol)
> believe it is doing just fine. 
> 
> In this case, it would appear to me that a mechanism may be needed
> in the DHCP failover protocol for a secondary to indicate to the
> primary "you're malfunctioning" so that the secondary can take over,
> even though the primary may think it's operating just fine. 
> 
Agreed!  I'm not sure that this is something that is appropriate for an RFC
though, as the methods for detection of various failures are very much
system-specific.

--Barr

---

From: "Bernard Aboba" <[email-masked]>
Date: Fri, 7 Apr 2000 02:27:35 -0700

Thanks everyone for the thoughtful responses.

>The load balancing draft (most recent version) has
>the concept of allowing the "secondary" server to start responding to
>requests if it "feels" that the "primary" is not responding in a timely
>fashion. This handles some, not all, of the issues you raise.

I agree that several (most? all?) of these issues boil
down to allowing the secondary to take over due to a "feeling".

Some questions are what the "feeling" can result
from, whether there are some corner conditions in which this
won't work and what the implications are. I agree that
timeliness is important. It seems to me that the "feeling" can
also come from other sources, such as a secondary looking at
retransmissions,
from an external monitor sending DHCPDISCOVER packets and analyzing
the semantics of the response; or from a management station polling
the DHCP server and analyzing data from the DHCP MIB.

The "action" may be: the secondary should start responding. Should
it inform the primary that it has taken this action? In the case
of an external monitor, the observed
behavior may indicate a possible malfunction of the primary
sufficiently grave as to be worth attempting to shut it down or
restart it. This has no protocol implications, however.

>I believe that the mindset was that processing of packets from

This is really an implementation issue, but I think that it cannot
be necessarily assumed.

>But at the same time, if the

It could be detected by other means referred to above, namely external
monitors, analysis of SNMP data, etc. When detected, the secondary may
want to step in even though the primary is still indicating that it
believes it is healthy.

>not be detected by the network layer, so the server would not detect a send

Yes, and I have seen this kind of failure in practice. The point is that the
DHCP server is not necessarily the best judge of its own health. However,
an external monitor may detect the failure and so the secondary may learn
that the primary is not as healthy as it says it is.

>If the packet process is sending and

Yes, and I have also seen this kind of failure occur in real systems. The
solution is to have an external monitor do semantic analysis. In the load
balancing business this is called "active content verification" and is
considered essential for verifying (mostly web) server health. If performed,
this can be another input to the secondary giving it a "feeling" that the
primary is sick.

>though, as the methods for detection of various failures are very much

I'm not saying that it is very relevant to inclusion in the RFC thought I
think
a paragraph on the subject might be useful. I does appear to have
implications for
how the protocol is designed though and I would like to understand if all
these
cases are in fact covered.

>Some of the failure cases (such as internal to the DHCP server) are
>likely NEVER to be fully eliminated. For example, if a DHCP Server
>is really broken and starts giving out bad leases, there's not much
>one can do except to warn an operator.

Actually, there is something that can be done, namely having an
external monitor inform the DHCP server that it is malfunctioning
so that it can be shut down or be restarted. Of course, that's not
relevant to the failover protocol design.