Subject: Re: I-D ACTION:draft-ietf-ldup-lcup-04.txt

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 10 Mar 2003 14:24:55 -0800

At 12:16 PM 3/10/2003, John McMeeking wrote:
>It seem like the log-based vs state-based distinctions may not be necessary
>if we borrow some ideas from LDUP.
>
>LDUP has the notion that deleted entries be maintained by the server for
>some period of time.  A client search never returns these entries, but the
>server keeps knowledge that the entry with UUID 'x-uuid' was deleted at
>time 'x-deleted-time'.  This could be implemented using a log (search log
>for deleted entries) or state (meta-data).  LDUP doesn't care (though at
>some level it starts to be awfully hard to be state/log based agnostic).
>
>If a cookie has some sort of time information, then a server can:
>- find entries that have been added (createTimestamp attribute or
>equivalent)
>- find entries that have been modified (modifyTimestamp or equivalent)
>- find entries that have been deleted (delete timestamp described above)

Without additional historical information, the server however
would not in general be able to determine whether or not an
updated entry presently not currently within the content was
in the content at the time indicated by the cookie.  The
server must assume it was in the content at the time indicated
by the cookie and generate an appropriate message to the client.

>Modify DN presents problems.  Perhaps we could add a superior timestamp
>(new superior) and RDN timestamp (rename).  Not knowing where the entry
>came from, all modify DN operations (with appropriate time) would be
>treated as "entered result set" if they are now in the result set, and as
>"left result set" if they are not now in the result set.

>I think modify would not follow the letter of the draft.  Any change to an
>entry would have to be treated as a change to the attributes the client was
>interested in.

Yes.  Say the updated entry is within scope of the search but
currently doesn't match the search criteria.  Without additional
historical information, the server would not be able to determine
if the entry was in the content at the time indicated by the cookie.
The server must assume it was in the content at the time indicated
by the cookie and generate an appropriate message to the client.

>But I don't think that should cause problems, unless there
>are a lot of these changes relative to changes the client is really
>interested in (whatever "a lot" is).

The problem here is that the server is unable to determine
whether an updated entry in the CONTEXT was previously in the
content without additional historical information and hence
must assume that it was in the CONTENT at the time indicated
by the cookie.

That is, the problem severity is relative to the changes to
the CONTEXT not just to CONTENT.

>A server probably wouldn't want to keep deleted entry state information
>around forever.  A server implementation should be able to prune this --
>could be as arbitrary as discarding deleted entry information after a fixed
>period.  Any LCUP cookie that corresponded to a time prior to this period
>would require a resync.  Deletes should be relatively infrequent, so it
>should be possible for this time period to be fairly long.
>
>Assuming everyone views this as appropriate state information, a log-based
>implementation is not required.  I doubt LCUP needs to say any of this, but
>maybe others think such "hints" are appropriate.

Well, I thin that if we believe that certain implementation
approaches would lead to implementations which would do harm,
that we should state these beliefs and, where appropriate,
detail implementation-approach-neutral imperatives of what
behaviors are considered harmful.  Lack of "eventual convergence
data consistency" and/or "overly chattiness" would be harmful.

Kurt

Kurt

---

From: John Merrells <[email-masked]>
Date: Mon, 10 Mar 2003 15:14:17 -0800

Kurt D. Zeilenga wrote:

>behaviors are considered harmful.  
>
In other words you're suggesting that the specification document that 
implementors
should implement the specification correctly.

>Lack of "eventual convergence data consistency" 
>
What does LCUP currently state?

You're looking for a guarentee from LCUP that the client state be
exactly the same as a server state at some point?

>and/or "overly chattiness" would be harmful.
>
That's implementation defined.

John

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 10 Mar 2003 17:30:16 -0800

At 03:14 PM 3/10/2003, John Merrells wrote:
>In other words you're suggesting that the specification document that implementors should implement the specification correctly.

I'm suggesting the specification discuss areas where the
implementor should take care not to do harm.

>What does LCUP currently state?

Presently, the I-D is a bit lacking in its data consistency
statements.  It needs to be more explicit in stating that
servers are to either generate messages which will lead to
convergence or return an error.  This, I think, can be resolved
by rewording portions of 3.3.7.

>exactly the same as a server state at some point?

I am looking for the LCUP to be designed to support content
synchronization with eventual convergent data consistency.

I'm not yet convinced that LCUP provides this, hence this
dialog.

>That's implementation defined.

Chattiness should be defined in terms of the protocol
and the operations they perform.  Not only should
protocols be designed to avoid overly chattiness,
where the protocol implementation allows for
chatty protocol exchanges, the specification should
discuss chattiness issues and state appropriate
imperatives.

The LCUP I-D makes some statements in this area, I believe
more is necessary and will over specific suggestions
at a later date.

Kurt

---

From: [email-masked] (Rich Megginson)
Date: Mon, 10 Mar 2003 19:40:25 -0700

>by rewording portions of 3.3.7.
Reword to say what?  Change the SHOULD in the first paragraph of 3.3.7 
to a MUST?

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 10 Mar 2003 20:11:27 -0800

At 06:40 PM 3/10/2003, Rich Megginson wrote:
>Reword to say what?
>Change the SHOULD in the first paragraph of 3.3.7 to a MUST?

The first paragraph says what the server is to do if it detects
an inconsistency.  It does not require the server to implement
any inconsistency detection.

It's my opinion that the server should be REQUIRED to either
produce a set of messages which cause convergence at conformant
client or return an appropriate error.

Kurt

---

From: John Merrells <[email-masked]>
Date: Mon, 10 Mar 2003 21:05:21 -0800

>more is necessary and will over specific suggestions
Sure, add implementation advise, but you can't mandate against
bad implementations.

John

---

From: [email-masked] (Rich Megginson)
Date: Tue, 11 Mar 2003 07:18:14 -0700

>client or return an appropriate error.
To summarize the discussion of the past two days - there are a couple of 
problems with the current LCUP 04 draft:

1) The draft has an implicit assumption that the server will store some 
historical change information in order to reduce the amount of data sent 
to the client during resync.  This precludes server implementations that 
do not store any historical information.  Such a server could simply use 
the createTimestamp or modifyTimestamp in conjunction with a cookie that 
simply has a timestamp as its value.  This server could just return all 
entries whose createTimestamp and modifyTimestamp are greater than the 
timestamp in the cookie, and return only a DN or UUID for other entries 
within the search scope.  The client would presume those entries have 
not changed.  At the end of the sync session, the client would assume 
that entries that have not been received in full (changed entries) or 
entries for which a DN or UUID has not been received (unchanged entries) 
have been deleted or moved out of the search scope.

I think the LCUP draft could be changed to allow this, but this would be 
an optional feature of the protocol.

However, I would like to know of any existing or planned server 
implementation that does not store any historical information, either in 
a state or log based system.  Could I see a "show of hands"?

2) The draft should state an absolute convergence requirement.  That is, 
the server is REQUIRED to do one of the following:
Produce a set of operations that will bring the client into complete 
synchronization with the server
Tell the client that incremental synchronization is not possible and 
require the client to do a full resync

I believe this could be easily done with some slight rewording of 
section 3.3.7 of the LCUP 04 draft.

Are there any other problems?  Come on, don't be shy.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 11 Mar 2003 07:11:20 -0800

At 06:18 AM 3/11/2003, Rich Megginson wrote:
>To summarize the discussion of the past two days - there are a couple of problems with the current LCUP 04 draft:
>
>1) The draft has an implicit assumption that the server will store some historical change information in order to reduce the amount of data sent to the client during resync.  This precludes server implementations that do not store any historical information.  Such a server could simply use the createTimestamp or modifyTimestamp

(or entryCSN)

>in conjunction with a cookie that simply has a timestamp as its value. This server could just return all entries whose createTimestamp and modifyTimestamp are greater than the timestamp in the cookie, and return only a DN or UUID for other entries within the search scope.  The client would presume those entries have not changed.  At the end of the sync session, the client would assume that entries that have not been received in full (changed entries) or entries for which a DN or UUID has not been received (unchanged entries) have been deleted or moved out of the search scope.

>I think the LCUP draft could be changed to allow this, but this would be an optional feature of the protocol.

I assume you mean optional to the server...

>However, I would like to know of any existing or planned server implementation that does not store any historical information, either in a state or log based system.  Could I see a "show of hands"?

OpenLDAP has "early" no-history implementations of LCUP (circa
-03) and LDAP-Sync presently available for experimentation.

>2) The draft should state an absolute convergence requirement.  That is, the server is REQUIRED to do one of the following:
>Produce a set of operations that will bring the client into complete synchronization with the server
>Tell the client that incremental synchronization is not possible and require the client to do a full resync
>
>I believe this could be easily done with some slight rewording of section 3.3.7 of the LCUP 04 draft.
>

Many (mostly minor).  When I first looked at -04, I found many of the
same problems I found in -03 (and raised in my "laundry list"
and/or the WG Last Call).  I have not yet attempted to put an
updated "laundry list" together for -04, I prefer to remain focused
on a couple of critical issues.

Kurt