Subject: minutes from failover draft conference call

From: Kim Kinnear <[email-masked]>
Date: Fri, 27 Oct 2000 16:48:15 -0400

We held a conference call today (Friday, October 27, 2000)
concerning the failover draft:

http://www.ietf.org/internet-drafts/draft-ietf-dhc-failover-07.txt

The following people attended (and a big thanks to all of them):

Ralph Droms       DHC WG chair        EDT   [email-masked]
Steve Gonczi      Network Engines     EDT   [email-masked]
Richard Jones     Checkpoint Software PDT   [email-masked]
Kim Kinnear       Cisco Systems       EDT   [email-masked]
Thomas Narten     IETF area dir       EDT   [email-masked]
Mark Stapp        Cisco Systems       EDT   [email-masked]

The following people wanted to attend, but this particular
date/time didn't work for them:

Bernie Volz       IPWorks             EDT   [email-masked]
Greg Rabil        Lucent Technologies EDT   [email-masked]
Ted Lemon         Nominum             EDT   [email-masked]

The agenda was:

   a) Issues surrounding lease ownership and pool allocation for
   secondary servers.

   b) Handling of Dynamic bootp leases and "reservations.

   c) Massive resychronization required when bringing in new
   secondary server.

   d) What to do to move this draft towards last call?

   e) If and when should we do another failover conference call?

Here are the minutes (which I've written in third person for,
believe it or not, clarity):

----------------------------------------------------------
Issues surrounding lease ownership and pool allocation for
secondary servers.

We discussed the issues of lease ownership, loadbalancing, and
trying to balance the sizes of the pools of available leases on
the primary and secondary servers.

Steve Gonczi raised the issue that he felt that an IP address
should be owned by one server or another throughout its life
cycle, as opposed to the current model, where it is owned by one
server or another only when it is available.  He felt that this
would in general yield a more effective balance between the
available pools on the primary and secondary servers.

Kim Kinnear pointed out that in his view having IP addresses only
owned by one server or another when available wasn't the solution
to any problem with achieving a good balance between the failover
partner's available pool, since lease ownership wasn't really
that involved with the balancing issue.

Kim stated that, based on his thinking concerning Steve's issues,
that he felt that a more dynamic balancing approach should be
found that the current one, but that lease ownership didn't seem
like a solution (nor a problem).

After considerable discussion (approximately 45 minutes), we
discovered that we were imagining different models of use for the
protocol.  Steve was thinking of a 50/50 loadbalancing approach
with a 50/50 available IP address split -- and it seemed like
having the server's manage their "own" pool might cut the traffic
for having the primary pass addresses to the secondary.  Kim,
Mark Stapp, and Richard Jones all said that they were thinking
that people would sometimes use failover without loadbalancing at
all, and sometimes use loadbalancing very asymmetrically.  Kim
pointed out that he felt that a secondary server could not (in
general) wait for leases to expire as a way to replenish its pool
of available addresses, but that after using some addresses it
would have to have the main replenish its pool.

Steve pointed out that some of his concerns came from a real
failover implementation, but that it was operating in a testing
regime, where leases were very short and turned over a great
deal.  He generally agreed that the current approach to ownership
would work for him, since the environment in which it would be
used would probably need to have the secondary's available pool
replenished prior to any given lease expiring (that might be
owned by the secondary, and thus help to replenish the
secondary's pool of available addresses).

Kim said that he would work on a more dynamic balancing
mechanism, that was biased toward the primary making the
decisions -- to avoid having misconfigurations between the
partners turn into addresses flowing in a loop between them.
Richard said that he liked the idea of having the primary make
the IP address allocation decisions for the available leases for
the failover pair, since it yielded one place where the
parameters that would control this decision could be controlled.

---------------------------------------------------
Handling of Dynamic bootp leases and "reservations".

Steve brought up the need to tell the partner that a lease was
for a dynamic BOOTP client.  Kim mentioned that dynamic BOOTP was
in the draft, and that the problem wasn't with the information
that would flow about leases when the partners were
communicating.  He said that the problem was with the leases made
by one server when not communicating with the other server -- any
lease made from a pool which supports dynamic bootp will never
expire, and so the mclt has no meaning for any IP address in that
pool -- whether or not actually leased to a BOOTP device.  Thus,
any address pool which supports dynamic bootp will never be able
to be used by the partner, even in partner down.

We talked about reservations for a while.  Kim said that he
integrated the reservation work in the last round or two of the
draft, and Steve pointed out that it was not very well
integrated.  Kim said that he would take care of that.

Steve also said that, in his view, the reserved status of an IP
address was different from the state, and that having a state of
BACKUP-RESERVED was overloading the state in an unclean way.  Kim
responded that, as the reservation attribute was interesting only
when receiving the BNDUPD and only for available IP addresses
assigned to you, that this was not really a particularly
inelegant way to approach this problem (since both servers should
be configured to know that this IP address is reserved).  Richard
said that the BACKUP-RESERVED was a safety thing, to avoid a
problem with the servers were reconfigured at slightly different
times.  Steve was not entirely convinced, and still feels that
this approach is less clean than he would like, but agreed to
live with it., Kim said that, if another attribute appeared that
required transmission between failover partners, that he would
pull the BACKUP-RESERVED state and make it an attribute on the
BNDUPD along with the new bind update attribute.  At present
there are no other attributes that are not well transmitted in
the existing options and packet format.

There was general agreement that this was acceptable.

------------------------------------------------------
Massive resychronization required when bringing in new 
secondary server.

Kim pointed out that, when servers are being resychronized, where
the secondary server has not ever seen the primary but the
primary *has* seen a secondary, the secondary asked for the
primary to send it "everything".  The secondary knows in this
case that it has no information in its stable storage, so it
wouldn't technically have to get *everything*, but just the
"relevant" address information from the primary.  Kim proposed
adding a new message to get just the relevant (i.e., changed at
some point) information, to make this reintegration approach more
efficient than the one current in use.

Steve asked the excellent question about the value of sending any
lease state information in a BNDUPD that is not "relevant", i.e.,
what would be the value of sending information on a blank
database record (as he put it).

After some thought, the group realized that Steve was right, and
Kim agreed to recast the update request all to be only update
request all that is "relevant", i.e., what has changed since the
ground state of the server.

Steve asked if two failover servers could ever get out of sync,
and said that he felt that they couldn't.  Kim related a single
real situation where it appeared to happen due to a restoration
from a backup.  Steve said that given this, we needed some way to
trigger the failover pair to send all of the relevant stuff from
each to the other, and that if this were to be part of a user
capability, it could certainly be used to get two failover
partners back in sync.

Kim agreed to add this capability into the draft.

------------------------------------------------
What to do to move this draft towards last call?

Kim said that he had a new port number from Bernie, and that
other than the changes that he had from today, and from some
editorial comments he had received, he didn't know what else was
required.  Steve said that he'd send editorial comments, but the
group agreed that they didn't know of any other problems.  Kim
said he would propose that the draft go to last call after San
Diego IETF, and Ralph Droms said that he felt that was the best
way to approach this.  Ralph said that we should just put it to
last call let the process reveal any problems that people have
with the draft.

----------------------------------------------------------
If and when should we do another failover conference call?

We heard from Steve that Ted Lemon has implemented a considerable
portion of the recent failover drafts, and we decided to ensure
that the next (and probably the last this cycle) failover
conference call would be held at a date and time such that he
could attend, so that we could find out from him just what he's
learned when trying to implement the draft.  The tentative dates
are November 7 or November 8, 12-3 either day.  Kim said he'd
send out mail proposing this.

==========================================================

That's it for now.

Another *big* thanks to all who attended.

Cheers -- Kim

---

From: Bernie Volz <[email-masked]>
Date: Fri, 27 Oct 2000 17:31:07 -0400

Kim:

Excellent summary and thanks for getting it out so soon after the
discussion. Sorry I wasn't able to make the call.

Regarding the "Massive resychronization required when bringing in new
secondary server" issue, couldn't there be an assumption that when a
UPDREQALL was requested, that the requesting server assumes some initial
state for ALL addresses before making the request. Thus, only those
addresses that are in some way different from the initial state would
require a BNDUPD (I would assume the initial state is for all addresses to
be in FREE state with no client information). BTW: Is that assumption ever
explicitly stated? Is it needed? AND, isn't this actually a better way to go
if the servers have slightly different configurations (since in that case,
even if ALL was sent, all wouldn't really have been sent).

If that was done, what need is there for a new message? (And from the sounds
of it, that's what you might have concluded.)

You also indicate that there was a real life situation where a restored
backup of a server caused some interesting problems. I would guess that the
server was not asking for UPDREQALL but just UPDREQ? Or? Did the initial
state handling not detect this problem? Might we need some additional
protection during startup of the servers? Or, was the UPDREQALL not sending
'all'?

Regarding open issues, I think the security section still needed some work
(and that was on mine and Ted's plate at one point if I recall correctly).
Neither of us has done anything on it.

I would also like to get feedback from Ted and any operational experience he
might be able to relay.

- Bernie Volz
  IPWorks

-----Original Message-----
From: Kim Kinnear [mailto:[email-masked]]
Sent: Friday, October 27, 2000 4:48 PM
To: DHCPv4 discussion list
Cc: [email-masked]
Subject: minutes from failover draft conference call

The agenda was:

That's it for now.

Cheers -- Kim