Subject: Re: LCUP and PSearch on Sync-And-Persist

From: [email-masked] (Rich Megginson)
Date: Wed, 03 Apr 2002 19:34:36 -0700

>                       Mircea Pana
>                       <[email-masked]        To:       IETF LDUP <[email-masked]>
>                       om>                      cc:
>                       Sent by:                 Subject:  LCUP and PSearch on Sync-And-Persist
>                       owner-ietf-ldup@m
>                       ail.imc.org
>
>
>                       03/26/2002 10:25
>                       AM
>                       Please respond to
>                       mpana
>
>
>
> Both LCUP and PSearch are unclear on the order of results sent back to a
> client in case of a sync-and-persist search request. An order seems to
> be suggested but not formally recommended.
>
> If a client issues a search requests with the appropriate control and
> specifies updateType=synchronizeAndPersist (LCUP) or changesOnly=FALSE
> (PSearch), in response the server will start by sending all the data
> needed to synchronize the client. If a change occurs to an entry which
> satisfy the search criteria while the server is still synchronizing the
> client then:
> 1. the server will pause the synchronization, sent that changed entry to
> the client and then resume the synchronization
> -or-
> 2 (suggested). the server will queue the changed entry and sent it to
> the client after the synchronization is completed
> -or-
> 3. the server will ignore the change
>
> Which one of the above is recommended by LCUP / PSearch?

I'm speaking only for LCUP, not PSearch.
I think the above assumes that entries sent during the synchronize phase are somehow different from the entries sent during the
persistent phase.  Why should the client care if either 1 or 2 is used?  For 1), why would the server have to pause?  It could
just insert the changed entry either into it's outgoing sync queue, or it could put it on the end of the queue.  I don't think it
matters as long as the implementor has designed the server such that the server knows exactly where it left off if the connection
is cut.  The cookie sent back to the client should contain all the information required to resync that client, hopefully with no
overlaps.

> How is a client informed that the synchronization is complete and the
> only results that it may eventually receive are changes?

I'm not sure why the client would need to know that in the LCUP case.  I would appreciate more information about what type of
application you have that would require that information.

> Regards,
> Mircea Pana.

---

From: Mircea Pana <[email-masked]>
Date: Thu, 04 Apr 2002 14:06:32 -0500

> would appreciate more information about what type of

The application (LDAP Client) processes data that is heavily interdependent
(references and subordination) thus a single entry may not have a meaning by
itself. Also, in the synchronization phase, a large number of entries is
expected by the client in a relatively short period of time.

So, the results are buffered but not processed by the application while
synchronizing since it is assumed that the information is incomplete. Also, a
session lost while synchronizing is not "recoverable" from the application point
of view since the information received may violate the semantics.

Regards,
Mircea.

---

From: [email-masked] (Rich Megginson)
Date: Thu, 04 Apr 2002 12:20:14 -0700

> of view since the information received may violate the semantics.

So, if you lose a connection, or the server crashes, or any number of reasons to lose a session, you have to resync from scratch
every time?

---

From: Mircea Pana <[email-masked]>
Date: Mon, 08 Apr 2002 12:02:24 -0400

> So, if you lose a connection, or the server crashes, or any number of
> reasons to lose a session, you have to resync from scratch
> every time?

I have two kinds of client apps.:

1. A lightweight client without persistent storage resync "from scratch" every
time. Due to the relatively heavy data processing required, it will wait for
"sync" state to be reached before compiling the initial data. Subsequently it
will act on change. Such client has a limited scope (expects to receive a
moderate number of entries).

2. A full featured client with persistent storage keeps a copy of the processed
data (not 1:1 with the raw LDAP entries). Once past the "(re)sync." phase, it
compiles the data received and takes the necessary actions. It can "recover" a
broken session based on the cookie of the last successfully processed data
element.

If the Directory server does not provide an "in sync." indication, the client
apps. mentioned above need to treat all responses incrementally which would
affect their performance. So, IMO, even though not a must, supporting such
feature is a strong wish.

However, there is a case that I don't see how to resolve without an "in sync."
indication from the server:
Let's say that an application with persistent storage has received, processed
and stored a fair amount of data from the Directory server when the session
closes and the cookie is either lost or corrupted or the client receives a
"reload required". The only option the client has is to start the next session
"from scratch". The "old" data can not be cleared away immediately because that
may lead to temporary disruption of critical services - unacceptable! So, the
application would attempt to "sync." from the Directory server, process the new
data, compare it with the old one and apply the delta before going into the
"persistence" mode.

Any suggestions would be greatly appreciated.

Thanks,
Mircea.

---

From: [email-masked] (Rich Megginson)
Date: Mon, 08 Apr 2002 10:28:21 -0600

> feature is a strong wish.

Why can't the clients first do a synchronizeOnly session?  If the server is not under a heavy update load, this will get the
clients sufficiently close.  If the server is under a heavy update load, the clients may have to wait indefinitely to be "in
sync".

> data, compare it with the old one and apply the delta before going into the

Why can't the client first do a synchronizeOnly session?  If that session completes successfully, the client knows that it is
"reasonably" close to being "in sync" with the server (depending on the server update load).

If the client loses the cookie, perhaps the server could "guess" about the state of the client if the client still knows some
information like the last update time of the last entry received.

I'm not sure why the client would receive a reload required assuming the client does not lose the cookie, the server is not
reloaded, and the client can "keep up" with the updates on the server.

Why would the application have to compare the old one with the new one?  Why couldn't it just use the new one directly, unless
this is an implementation detail?

Please know that I am trying to understand your problem, to see if LCUP can solve it right now, that LCUP may be able to solve it
with some minor modifications, or that LCUP is simply not suitable for your problem.

---

From: Mircea Pana <[email-masked]>
Date: Mon, 08 Apr 2002 14:39:12 -0400

Rich,

> [...]Why can't the clients first do a synchronizeOnly session?

This sounds like a good idea. When it starts, the client will have to perform
two operations in sequence (SyncOnly followed by SyncAndPersist) instead of just
one, but it doesn't sound like it would be a significant overload. Thanks for
the suggestion.

> [...]I'm not sure why the client would receive a reload required assuming the

The cookie may become invalid if the server and/or the application are re-loaded
from backups for instance.

> Why would the application have to compare the old one with the new one?

The application actually compiles the data and sends it over to various service
engines. In a full resync session the application compares the new data set with
the old one to determine the old data units to be removed. These correspond to a
subset of the entries deleted after the last LCUP session but before the current
one. (I suppose that in case of a "full sync." session, the Directory server
will not send the previously deleted entries. Right?)

I still have a question: In case of a SyncOnly session, the server "knows" when
there is no more data to send to the client and closes the session with a
SearchResultsDone (with lcupSuccess). Since the Directory server also has this
state information in case of a SyncAndPersist session, would it be difficult or
wrong to include a notification in the appropriate EntryUpdateControlValue?

I appreciate your help.
Thanks a lot,
Mircea.

>
> Please know that I am trying to understand your problem, to see if LCUP can
> solve it right now, that LCUP may be able to solve it

---

From: [email-masked] (Rich Megginson)
Date: Mon, 08 Apr 2002 13:19:08 -0600

> from backups for instance.

Right.

> will not send the previously deleted entries. Right?)

Right.  So a client acting as an intermediary would want to prevent doing a full resync of its clients.

> wrong to include a notification in the appropriate EntryUpdateControlValue?

That necessarily means server implementors will have to sort the entries into two groups.  I could forsee that many clients may
not care to know when they are "in sync" in a syncAndPersist session, and the server would just be able to insert updated entries
anywhere it chooses to in the outgoing data stream.  Of course this is highly implementation dependent - I don't think this would
work given the model I spoke of using CSNs for ordering.  Or maybe we can make this behavior switchable by the client.  I'll speak
to the other authors about this.

---

From: Mircea Pana <[email-masked]>
Date: Mon, 08 Apr 2002 17:14:11 -0400

> That necessarily means server implementors will have to sort the entries
> into two groups.  I could forsee that many clients may
> not care to know when they are "in sync" in a syncAndPersist session, and
> the server would just be able to insert updated entries
> anywhere it chooses to in the outgoing data stream.

I would argue that the lack of sequence is annoying for consumers with complex
data models and as LDAP gains market share, such models become more and more
frequent. These LCUP clients would have to sort and hold data until meaningful.
Of course, I speak from the point of view if the client (trying to make my life
easier ;-) but I'm sure that the server side has its own challenges. So,
whatever the resolution will be, thanks for taking these issues into
consideration.

Mircea.