Subject: RE: New version of draft-ietf-ldup-replica-req

From: [email-masked] (Richard V Huber)
Date: Wed, 15 Nov 2000 17:45:10 -0500

Albert -

I've restated/summarized the issues you raised and then added my
replies.  Please let me know if I misunderstood or missed some of your
comments.

1. Model 3 should not be allowed.  Only model 2 should be supported.

   These are requirements for the LDUP protocol, not for specific LDUP
   implementations.  There may be situations where model 3 (or model 1,
   if the overhead issues can be worked out) is useful (as witness the
   reference to an existing implementation of model 3), so we feel the
   protocol should support them (requirements G1, G2).

   Now from a practical point of view you may well be correct in
   saying that model 2 is a lot more useful that model 3.  In fact, the
   applications that I am personally involved with require model 2, and
   I would not buy a directory that only offered model 3.  But that is
   no reason to keep people who CAN use model 3 from having it.

2. The requirements should not become an RFC until there is a URP draft
   that supports eventual consistency.

   As I understand it, you want model 2.  The current requirements say
   that model 2 is a MUST, so I'm not sure what would be gained by
   delaying the requirements draft.  Also, as a general principle, the
   requirements should come before the design, so I don't feel we
   should wait for the design before we publish the requirements as an
   RFC.

3. Atomicity issues.

   We've discussed some of the problems with atomicity in multi-master
   environments in Appendix B.5.  What specific issues do you have with
   this?  Do you see a way to make atomicity "work" in these cases?
   Also, I would note that two of the authors of the revised draft
   (Ryan Moats and I) have quite a lot of experience in using
   directories for service provider applications, so user experience is
   not being ignored.

4. The subentry draft may add new replication needs.

   True.  But any new LDAP feature may add replication needs.  We can't
   wait for every possible feature to make sure that we have covered
   them all.  The right thing to do here is to ask the author of the
   subentry draft to include a section on replication issues rather
   than holding up the requirements draft.

Rick Huber

: From: "Albert Langer" <[email-masked]>
: To: "'Richard V Huber'" <[email-masked]>, <[email-masked]>
: Cc: <[email-masked]>, <[email-masked]>, <[email-masked]>,
:         <[email-masked]>, <[email-masked]>,
:         <[email-masked]>, <[email-masked]>, <[email-masked]>
: Subject: RE: New version of draft-ietf-ldup-replica-req
: 
: This is just to confirm that my opposition to both LDUP WG and IETF final
: call still stands, as explained below (previously posted as a PS under an
: inappropriate topic thread).
: 
: I am CCing to the Area Directors as advance notice that the question of
: whether LDUP is making a fundamental technical mistake is being raised.
: 
: Contrary to the "assumption" I made in item 1 below. There has been no
: initial draft, nor any discussion or other indication that work is actually
: proceeding, to revise architectural and protocol drafts to achieve "eventual
: convergence". I still believe there is likely to be a consensus in the WG
: that "eventual convergence" is a requirement. That would exclude "Model 3:
: Limited Effort Eventual (or Probabilistic) Consistency" and therefore
: contradict the following stated requirement:
: 
: "4.1 General
: 
: G1.  LDAP Replication MUST support models 2 (Eventual Consistency) and
: 3 (Limited Effort Eventual Consistency) above."
: 
: I am not aware of any justification or other discussion that has been
: presented in support of this requirement to support "Limited Effort Eventual
: Consistency", which amounts to permitting "Eventual Inconsistency". Steve,
: an author of the URP draft, mentioned that he took it for granted that
: eventual convergence was in fact required, indicated that he would be
: presenting a revised URP draft to achieve that and suggested that related
: changes would be needed in the architectural draft. No other view was
: expressed by anyone.
: 
: I formally request that the WG chairs put this as a separate issue to the WG
: prior to WG "final call" on the requirements draft and establish what
: consensus, if any, does exist on this matter. If it is an oversight it can
: easily be fixed. If in fact LDUP is seriously proposing "eventual
: inconsistency", that should be made clear now.
: 
: On the other matters below (and others not listed), I assume the absence of
: expressed opposition from others, and the report of discussions at the last
: LDUP meeting at IETF, does reflect a "rough consensus" within the WG against
: "atomicity" etc. My understanding is that my continuing belief that this is
: a fundamental technical mistake should therefore be expressed as opposition
: to LDUP's final position rather than delaying LDUP from publishing that as
: it's final position. The WG has had an adequate opportunity to change it's
: position.
: 
: BTW Recent discussions since the draft seemed to indicate some degree of
: agreement on a requirement to replicate administration and distribution
: knowledge. From my reading of the email archives, this necessity was pointed
: out, "vigorously", a long time ago. I believe it is always a mistake to
: design anything without establishing requirements first and that mistake is
: again being made on that issue, which affects subentry controls etc, too.
: 
: My previous comments follow:
: 
: ***
: PS This also a "hi" to all in view of long absence. Sorry I haven't been
: able to respond to latest requirements doc due to other commitments (still
: stuck in those for a while yet). But then nobody else has either. I do
: believe it was a significant improvement and genuine effort to address my
: objections.
: 
: For the record:
: 
: 1. I'm assuming some means of guaranteeing eventual convergence *will* be
: proposed with at least an initial draft prior to final call on the
: requirements draft. I believe there is probably a consensus for eventual
: convergence and the apparant ambiguity in the requirements draft about this
: may be unintentional and therefore easily fixed. A fix would remove the
: contradictory references to non-convergent models in the final para of
: section 3. The whole discussion of models in section 3 is just confusing if
: in fact the intention is to require guaranteed eventual convergence. Model 2
: should be clearly stated as *the* (only) relevant model.
: 
: Actual requirements 4.1 G1 and G2 are also confusing. But the real test of
: whether there is in fact an intention to guarantee convergence is however
: whether anyone is actually working on a means to achieve it. The current URP
: and architecture simply does not.
: 
: I still haven't seen a redraft of URP and architecture to meet the apparant
: consensus to support reliable eventual convergence under all circumstances.
: 
: 2. My objection to non-atomicity still stands for final call but I'm
: assuming there is little point arguing about it further within LDUP unless
: others want to take it up. The draft now at least makes it possible to have
: serious discussion about the issue in IETF review and does include several
: requirements which cannot in fact be met without atomicity, such as accurate
: replication of access controls and attributes such as "modifiers name".
: although I disagree with much of the analysis in appendix B.5 and also the
: continued misunderstanding of applications for multi-master replication in
: appendix A on usage scenarios. There are still some inadequacies in
: presentation of the issue but the fundamental question is now simply whether
: or not atomicity should be a requirement rather than whether a document that
: shows no comprehension of the issue at all should be presented to the IETF
: at all. I still believe a fundamental technical mistake is being made on
: that question. I also still believe the requirements draft should be
: actively seeking input from directory users on the impact of non-atomicity.
: 
: 3. I agree with the requirement 4.1 G6 for no unbounded conflict resolution
: records. This is a better description than "oscillation". It is not met by
: my MDCR draft and would need further work if in fact the current URP and
: architecture drafts do not come up with a means of guaranteed eventual
: convergence based only on timestamps. I'm not planning to do any work on it
: unless some interest is shown by others as a result of failure to deliver a
: viable URP and architecture with eventual convergence.
: 
: Sorry I simply don't have time to follow up on these issues at the moment.
: 
: Am just reminding that they are still there in case others want to take them
: up before the document goes from LDUP to IETF final call.
: ***

---

From: "Albert Langer" <[email-masked]>
Date: Thu, 16 Nov 2000 12:44:26 +1100

[Richard]

[Albert]
Thanks for the response.

A vague feeling that there is "no reason to keep people who CAN use model 3
from having it" is hardly grounds for imposing a mandatory requirement that
LDUP develop protocols to meet their needs. The most that could follow from
that is "SHOULD", not "MUST". But even "SHOULD" would just be the same sort
of waffle as the pious wishes for a general solution to meta-directory
issues in previous drafts. We simply aren't working on a problem that can be
solved without guaranteed eventual convergence.

Model 3 ("limited effort consistency") is needed for multi-master
replication among such a large number of independent nodes that it becomes
impossible to maintain globally consistent state information as to which
nodes are actually participating at any given time. There may also be other
situations in which it is needed (eg nodes that are not "servers" attempting
to provide a reliable service to "clients" but just trying to get a better
picture of the world around them, without storage capacity for global
information). Examples that come to mind with "servers" include the
replication of newsgroups and (distributed) email lists. Despite the best
efforts of "backbones" it is impractical to ensure that all nodes do receive
all updates so the feeds diverge due to batches not being collected within
the "limited best effort" criteria for purges. Model 2 is simply not capable
of scaling to the size of systems such as Usenet with currently available
technology on a multi-master basis. It is inherently confined to situations
where a relatively small number of "servers" (hundreds, perhaps thousands,
but not millions) are maintaining a common "backbone" for a much larger
number of "slave" servers and end user clients - such as for example LDAP
directories ;-)

For example the current LDUP architecture requires each node to maintain and
exchange information about the state of all other nodes, which adds a very
significant overhead to each replication session and severely limits
scalability. Perhaps that could be improved on within the scope of model 2,
but it is a non-trivial problem for which nobody in LDUP has come up with a
suggestion.

I believe there will be a growing demand for improved protocols meeting
requirements for "model 3" as the return to understanding the internet as a
"peer to peer" network gathers pace. Examples include "Freenet" and
"Groove", which could easily point to future peer to peer replication
systems reaching millions of replicating nodes but could not hope to
maintain more than "best effort" guarantees. Some may well involve the use
of LDAP clients and even exchange information analagous to directory
entries, though I cannot think of any example of work in that area. But if
and when they do, that is *all* they will have in common with LDUP work. The
design issues for "model 3" replication protocols has virtually nothing in
common with those for "model 2" that actually relates to replication. LDUP
has not been chartered to do that work, has not done any such work, and
knows nothing about the issues involved.

We have to provide a viable solution for replication of LDAP directories.
They clearly require eventual convergence and would be useless with "limited
effort" convergence since the inevitable divergence would negate the
benefits desired from actual needs for LDAP directories.

[Richard]

[Albert]
I agree that the current wording does require model 2 and that it is quite
proper to publish requirements before design.

My problem is that the requirements are being published long after the
design and the current design is not capable of meeting those requirements.
I have repeatedly explained why it is  simply not possible to guarantee
convergence using the "timestamps" approach in the current architecture in
the face of real world situations where DSAs have clocks that are not only
unsynchronized but can even jump backwards due to time zone mistakes etc and
where they crash and get later restored from backups. This has been
dismissed as a "religious" issue despite references to the extensive
research done on the subject for the Coda file system project and the
commercial deployment of replication protocols that do guarantee eventual
convergence in Active Directory.

Since no work has actually been done to fix the problem, my concern is that
if model 3 is listed as just as much a "MUST" requirement as model 2, LDUP
could resort to "incremental" delivery and proudly unleash on the world a
broken protocol that guarantees "eventual divergence", while deferring for
"subsequent work" anything actually useful.

The "eventual divergence" in the current architecture does not provide any
of the scalability requirements that do motivate model 3. It is simply
broken and must be fixed. Adding another MUST in parallel may not be
intended to weaken the force of the first MUST, but it can nevertheless have
that effect.

[Richard]
3. Atomicity issues.

[Albert]
I have published a draft demonstrating that atomicity can be made to work.
Atomicity *requires* that only one conflicting change be eventually
successful. Essentially the only way that can work in a multi-master
situation is by revoking some changes. B.5.1 and 5.2 accept that reality for
create and rename conflicts but inexplicably does not consider that approach
for other kinds of conflicts. I have shown it can be done without
administrative intervention, which may be tolerable for create and rename
conflicts but certainly not for more frequent conflicts.

On the second sentence, I do not believe the experience of the authors is
ever a useful substitute for actively making a "Request For Comment" from
users likely to be affected by a problematic issue. The latest draft is a
vast improvement on the previous in at least mentioning potential problems.
However it does not actively seek user input because it seeks to minimize
the problems and/or present them as unavoidable. Instead it should give
clear examples of how existing and essential applications would actually be
affected.

Instead of saying:

"The common characteristics of the examples are:

. Several directory users/applications are changing the same data
. They are changing the data at the same time
. They are using different directory servers to make these changes
. They are changing data that are parts of a distinguished name or they
  are using ModifyRequest to both read and write a given attribute
  value in a single atomic request

If any one of these conditions is reversed, the types of problems
described above will not occur.  There are many useful applications of
multi-master directories where at least one of the above conditions
does not occur.  For cases where all four do occur, application
designers should be aware of the possible consequences."

The draft should instead say:

***
If two users update Access Control Information from clients accessing
different servers the eventual result may differ unpredictably from that
intended by either of them. Even if the result does converge to the result
intended by one of them, in the simplest possible case, with changes to just
2 attributes, up to 6 separate intermediate states may be used to determine
access controls actually applied to the entry at various servers during the
course of replication while convergence is achieved. Changes to larger
numbers of attributes result in a combinatorial explosion of possible
intermediate states. Access granted or denied to entries is essentially
unpredictable during the course of a replication update, which may extend
over hours or days.

There may be useful applications for a directory service that does not
support access control information. Users expecting a directory with
predictable access control should be aware of the consequences of using LDUP
standards for multi-master replication.

The following excerpt provides some details from Alison's "Contribution to
Profiles Document (Consistency Discussion)":

http://www.imc.org/ietf-ldup/mail-archive/msg00548.html

The attached Word version, with more easily read tables, is:

http://www.imc.org/ietf-ldup/mail-archive/doc00000.doc

***
3.1  Latest Known Wins Consistency

The "Latest Known Wins" approach has the outcome that all attributes will
"eventually" reach the latest value entered into the system.  During the
"eventuality" being reached, intermediate states reflecting a combination of
states which have existed over time may be held by an individual DSA.  These
intermediate states are best described using an example.  Two operations, T1
at time 1 and T2 at time 2, occur on an entry updating attributes A and B.
This may be viewed as follows :
T0     T1     T2
A0     A1     A2
B0     B1     B2

"Latest Known Wins" consistency has the semantics that all directories will
eventually have the state A2, B2 (providing no further operations in the
distributed system affect attributes A and B).  However, at times between
either update occurring and this "eventuality", the directory state may be
any combination of the individual states of A and B depending on the
primitives currently received and processed by each DSA.

That is, any of the following states may exist :
* A0, B0 (Neither Operation has yet occurred)
* A0, B1 (Partial completion of Operation 1)
* A0, B2 (Partial completion of Operation 2)
* A1, B0 (Partial completion of Operation 1)
* A1, B1 (Operation 1 has occurred)
* A1, B2 (Operation 1 completed, Partial completion of Operation 2)
* A2, B0 (Partial completion of Operation 2)
* A2, B1 (Operation 1 completed, Partial completion of Operation 2)
* A2, B2 (Operation 2 has occurred)

***

I'll try to prepare a detailed response on specific issues in B.5 for a
separate message.

In case I don't get time, perhaps the above will at least provide food for
thought.

[Richard]

[Albert]
We should not be adding new features at all. Replication of administrative
information is a necessity for a viable directory service. That requirement
has been overlooked and should be stated. Subentries are not a "feature" but
a mechanism. In the course of considering how that mechanism is used to
achieve requirements for replication of information about replication it has
suddenly been realised that requirements for replication of other kinds of
administrative information are a good deal more complex than previously
supposed.

What is happening is not that someone has come up with a proposed new
feature, but that the penny has started to drop about certain aspects of a
global distributed directory that were not clear before, but have to be
dealt with when you go beyond an "Access" Protocol and start getting into
"Service" protocols such as replication.

---

From: [email-masked] (Christopher Apple)
Date: Thu, 16 Nov 2000 16:52:59 -0500

My responses below.....

Chris Apple

[email-masked]

-----Original Message-----
From: Albert Langer [mailto:[email-masked]]
Sent: Wednesday, November 15, 2000 8:44 PM
To: 'Richard V Huber'
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]; [email-masked]; [email-masked];
[email-masked]; [email-masked]
Subject: RE: New version of draft-ietf-ldup-replica-req

[Richard]

[Albert]
Thanks for the response.

CHRIS> Rick's language did not suggest to me that there was any sense of
vagueness about whether or not there are users of LDAP implementations that
would view Model 3 support as a MUST. It would not be listed as a MUST at
this stage of revising the requirements document if the authors (some of the
top LDAP experts in the world) and many other members of the working group
didn't know of a good reason for it being stated as that strong of a
requirement.
And you point out one of those good reasons in the next paragraph that you
wrote....

CHRIS> We'll let the WG express concensus or demonstrate the lack thereof
on whether or not "limited effort" convergence is useless or useful. I am
aware of your opinion on this matter. Your opinion will be considered along
with alternative viewpoints after the WG Last Call is over. John Strassner
and I will make a decision about whether concensus exists on the draft as
it is currently written. One opinion will not derail a recommendation
from John and I about requesting an IESG Last Call or requesting that
another revision of the document be undertaken.

[Richard]

[Albert]

CHRIS> The WG has achieved concensus on the use of "timestamps" in a way
that
you do not agree is best. That still means that the WG has achieved
concensus.
I am skeptical that John and I would allow an incremental delivery of the
nature
that you seem to be concerned about. You seem to be expressing doubt about
that,
but we are the Co-Chairs, you are not, and one or both Co-Chairs can be
replaced
by the Applications Area Co-ADs if allow something really screwy to happen.
Also, it *is* possible to revise or update an RFC once published, so that
even
if we were to publish any WG document as an RFC and WG consensus were to
change
to be consistent with your point of view, we could do it. A need to fix an
alleged
problem with the design should not and will not result in John and I
deciding to
hold up the requirements document from going through a WG Last Call.

[Richard]
3. Atomicity issues.

[Albert]

CHRIS> I want some time to think about the first paragraph and related text
below.

CHRIS> What I am prepared to say about the 2nd paragraph, is that I believe
you
are mincing words for sport. I have no tolerance for that sort of behavior
as
a WG Co-Chair. It simply wastes time and creates a hostile and unproductive
WG
environment. Rick stated that the concerns of users are not being ignored
due
to his experience. I firmly believe that user concerns are being adequately
addressed *because* Rick is one of the co-authors of the requirements
document
as Rick's experience is both historical and *current*. While working with
Rick
in the past, I have observed a level of concern over how end users will be
affected
by technical decisions that is quite admirable - and Rick also follows
through by
actually making technical decisions with user needs in mind. I also think
that
you are stretching the practical or real-world use of an RFC. It is true
that
the acronym expands to "Request for Comment." However, the real-world way of
using an RFC is as a reference, a common source of stable requirements for
building
implementations, and as a snap-shot in time of a set of concepts that will
hopefully
*become* standardized at some point in the future. RFCs can and have been
recycled
at the same level to address significant issues that should prevent their
progression
to the next level towards becoming a full standard. We'll do that if we have
to.
User input has been collected through the WG Members and Document Editors
and taken
into consideration since before the WG was even chartered. You may have a
different
perspective on user experience concerns than Rick does, but that difference
does not
equate to user experience being ignored or even glossed over.

CHRIS>Since what you write below is relevant to the paragraph I need to
think about a
bit more, I'll defer my response to it for another day or so.

CHRIS>However, we will be proceeding with a WG Last Call of the requirements
document.
It may be that we have to discuss this issue during the WG Last Call period.

CHRIS>Final comment on this e-mail is below under item 4.

Instead of saying:

***

***

***

[Richard]

[Albert]

CHRIS>I'm going to make the decision on this here and now.

CHRIS>Authors of ALL other WG documents MUST include a section
on replication issues that documents concerns, requirements, etc.
which we may not understand well enough or even be aware of at
this point in time. This section MUST be present with an indication
of non-applicability if no specific replication issues are known at the
time a particular document is written. We can revise requirements document
after it becomes an RFC to include any requirements from these document
sections if necessary later on and before those documents become published
those sections would be replaced with a statement referencing
the revised/updated requirements document.

CHRIS> Albert: if you have an issue with this decision, please take it
up directly with John Strassner, myself, and the Applications Area ADs.
It is not necessary or productive to involve the entire WG in a discussion
related to this as it is largely a matter of process. I won't discuss this
with you further on the mailing list for the WG and respectfully ask that no
other WG Member does either unless the Co-ADs decide that it is appropriate
and ask me directly to open up this topic for discussion again.

CHRIS> This issue is now resolved and Document Editors have guidelines for
how we will document replication requirements that are unknown or unclear
at this time.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 16 Nov 2000 14:48:58 -0800

At 16.52 -0500 00-11-16, Christopher Apple wrote:
>This issue is now resolved and Document Editors have guidelines for

I don't know if you expected any response from me as AD, but I just 
want to mention that

(a) I have read all comments, and don't see any reason why I should 
interfere with the process internal to the wg which I think the wg 
chairs are handling the correct way

(b) wg chairs are the ones which declare when the wg has consensus on 
a document, and they request an IETF wide last call

(c) comments that come in during IETF wide last call create the 
foundation for the decision which IESG will make regarding status of 
the requested RFC

(d) every decision made by wg chairs, AD's, IESG etc can be appealed 
by contacting, as Chris explains, basically the level above the one 
which you felt made the wrong decision

(e) rough consensus doesn't mean 100% of the people on the wg mailing 
list have to agree (then read bullet (b) again)

    Patrik Faltstrom
    Co-Area Director, Applications Area
    IETF