Subject: Client update protocol

From: Randy Turner <[email-masked]>
Date: Mon, 10 Feb 2003 10:39:42 -0800

Hello all,

I recently read over the LCUP document and this text looks very good 
for DIT synchronization.  What I am looking for is some type of "event" 
mechanism to trigger a client to connect/bind and initiate LCUP.  This 
would require the directory server to have knowledge of "who" is 
interested in being notified of DIT updates.  I may have 500,000 
clients for a directory, and I would prefer not to have to have the 
clients poll the server.  If a particular LCUP context is modified, I 
would like some type of async notification sent to interested parties.  
  Most of the work of actually synchronizing the DIT is covered by the 
LCUP document, I just need the clients to be notified "when" to perform 
LCUP. Ideally, the trigger mechanism would be some type of lightweight, 
connectionless notification, similar to an SNMP trap but more specific.

There is activity going on with global, scalable event notification 
architectures that is not specific to any IETF work, and we are looking 
into this.  However, I wanted to make sure I understand the taxonomy of 
any and all work related to solving this type of problem.

If there is work going on in this area in this, or another working 
group, I would appreciate any and all pointers, or other comments.

Thanks in advance,
Randy

---

From: [email-masked] (Rich Megginson)
Date: Mon, 10 Feb 2003 14:26:04 -0700

Randy Turner wrote:

> for DIT synchronization.  What I am looking for is some type of 
> "event" mechanism to trigger a client to connect/bind and initiate 
> LCUP.  This would require the directory server to have knowledge of 
> "who" is interested in being notified of DIT updates.  I may have 
> 500,000 clients for a directory, and I would prefer not to have to 
> have the clients poll the server.  If a particular LCUP context is 
> modified, I would like some type of async notification sent to 
> interested parties.   Most of the work of actually synchronizing the 
> DIT is covered by the LCUP document, I just need the clients to be 
> notified "when" to perform LCUP. Ideally, the trigger mechanism would 
> be some type of lightweight, connectionless notification, similar to 
> an SNMP trap but more specific. 

How many "interested parties" will there be?  Will the "interested 
parties" actually initiate an LCUP request?

A standard LDAP server is not designed to initiate connections to other 
servers or clients (except in the case of replication, but that is not a 
standard yet, nor would it likely serve your purpose).  Some LDAP 
servers do support callbacks or triggers that usually involve having to 
write your own plugin or some other mechanism.

This sounds like the sort of thing that message bus/queue architectures 
are trying to solve.  You could have an LDAP client that issues an LCUP 
persistent search against an LDAP server, and the client would put a 
message on the bus/queue every time it received an LCUP response.  Other 
clients register with the bus/queue and receive

BTW, for those interested, we are still working on the LCUP 04 draft and 
hope to have something very soon.

> architectures that is not specific to any IETF work, and we are 
> looking into this.  However, I wanted to make sure I understand the 
> taxonomy of any and all work related to solving this type of problem. 

Can you provide more information about this work?

---

From: Randy Turner <[email-masked]>
Date: Tue, 11 Feb 2003 09:54:43 -0800

Hi,

Thanks for the reply.  Regarding your earlier comments...

I might have 30,000 LDAP clients that want to know if a particular LCUP 
context has changed.  They might "subscribe" to this particular event 
so they get notified.  Notification methods such as that suggested by 
persistent search (and variants) or polling (connect/bind/search) would 
be sub-optimal with this many clients potentially involved. The LCUP 
sync mechanism will work great...I just need a client to be "tapped on 
the shoulder" when it should connect/bind and  issue the LCUP search 
method. The same entity that is notified, will be the same entity that 
issues the special LCUP context search.

At an IETF meeting a couple of years ago I had a discussion about such 
a technique (before I actually had a need for it) during early 
discussions about CLDAP and how this type of scenario could be included 
in the problems addressed by CLDAP.

Your suggestion about a "middleware" entity that maintains a persistent 
LCUP search sounds like it would be a good start,  looking "northbound" 
to the source DIT.  Looking back towards the clients, there would still 
be a need for clients to register their interest, as well as the actual 
lightweight delivery mechanism.

If there had been an extensible "trap/inform" group in the MADMAN MIB, 
then that might be a potential solutions as well.  Not perfect, but a 
step in the right direction. Ultimately, the solution will probably 
involve work going on outside the LDAP WG, again I'm just trying to 
find anyone with a problem domain similar to this.

I don't remember the exact names of the projects regarding event 
notification systems...but if you do a google (or other) search for 
"scalable event notification", this will show you a set of starting 
points that I used for my own research. Most of the recent work has 
been in academia, but I believe there are implementations of these 
systems available, as well as some organizations looking to go 
commercial with similar technology.

Thanks again for the reply,
Randy

On Monday, February 10, 2003, at 01:26 PM, Rich Megginson wrote:

> A standard LDAP server is not designed to initiate connections to 
> other servers or clients (except in the case of replication, but that 
> is not a standard yet, nor would it likely serve your purpose).  Some 
> LDAP servers do support callbacks or triggers that usually involve 
> having to write your own plugin or some other mechanism.
>
> This sounds like the sort of thing that message bus/queue 
> architectures are trying to solve.  You could have an LDAP client that 
> issues an LCUP persistent search against an LDAP server, and the 
> client would put a message on the bus/queue every time it received an 
> LCUP response.  Other clients register with the bus/queue and receive
>
> BTW, for those interested, we are still working on the LCUP 04 draft 
> and hope to have something very soon.
>
> <smime.p7s>

---

From: [email-masked] (Rich Megginson)
Date: Tue, 11 Feb 2003 11:06:51 -0700

Randy Turner wrote:

> I might have 30,000 LDAP clients that want to know if a particular 
> LCUP context has changed.  They might "subscribe" to this particular 
> event so they get notified.  Notification methods such as that 
> suggested by persistent search (and variants) or polling 
> (connect/bind/search) would be sub-optimal with this many clients 
> potentially involved. The LCUP sync mechanism will work great...I just 
> need a client to be "tapped on the shoulder" when it should 
> connect/bind and  issue the LCUP search method. The same entity that 
> is notified, will be the same entity that issues the special LCUP 
> context search. 

Then, will you have 30,000 clients making LCUP search requests at the 
same time, if they are all interested in the same event?

> discussions about CLDAP and how this type of scenario could be 
> included in the problems addressed by CLDAP. 

This seems like a perfect application for multicast.  Making 30000 
simultaneous or sequential TCP/IP connections is sub optimal.

> Your suggestion about a "middleware" entity that maintains a 
> persistent LCUP search sounds like it would be a good start,  looking 
> "northbound" to the source DIT.  Looking back towards the clients, 
> there would still be a need for clients to register their interest, as 
> well as the actual lightweight delivery mechanism.
>
>>> notified "when" to perform LCUP. Ideally, the trigger mechanism 
>>> would be some type of lightweight, connectionless notification, 
>>> similar to an SNMP trap but more specific.
>>
>>
>> architectures are trying to solve.  You could have an LDAP client 
>> that issues an LCUP persistent search against an LDAP server, and the

---

From: Randy Turner <[email-masked]>
Date: Tue, 11 Feb 2003 11:13:55 -0800

If all 30,000 clients have registered interest in the same context, 
then yes it's possible to get 30,000 LCUP request, although it would be 
easy to arrange for these requests to be load-balanced or intelligently 
skewed so they are not simultaneous to one instance of a server. In my 
particular application, only a small fraction would register interest 
in the exact same DIT context, but it would be prudent to design the 
system to handle the worst case loading.

Regarding multicast, you are right, some type of multicast might be 
appropriate. However, my experience with inter-AS (really between 
different service provider networks) multicast has been fraught with 
business issues rather than technical issues, so I'm not sure I can 
rely on multicast being a "generally available" solution to the 
problem.  This may have changed in the last year or so.

Randy

On Tuesday, February 11, 2003, at 10:06 AM, Rich Megginson wrote:

>> potentially involved. The LCUP sync mechanism will work great...I 
>> just need a client to be "tapped on the shoulder" when it should 
>> At an IETF meeting a couple of years ago I had a discussion about 
>> such a technique (before I actually had a need for it) during early 
>> there would still be a need for clients to register their interest, 
>> as well as the actual lightweight delivery mechanism.
>>
>> If there had been an extensible "trap/inform" group in the MADMAN 
>> MIB, then that might be a potential solutions as well.  Not perfect, 
>> but a step in the right direction. Ultimately, the solution will 
>> probably involve work going on outside the LDAP WG, again I'm just 
>> trying to find anyone with a problem domain similar to this.
>>
>>>> I recently read over the LCUP document and this text looks very 
>>>> good for DIT synchronization.  What I am looking for is some type 
>>>> of "event" mechanism to trigger a client to connect/bind and 
>>>> initiate LCUP.  This would require the directory server to have 
>>>> knowledge of "who" is interested in being notified of DIT updates.  
>>>> I may have 500,000 clients for a directory, and I would prefer not 
>>>> to have to have the clients poll the server.  If a particular LCUP 
>>>> context is modified, I would like some type of async notification 
>>>> sent to interested parties.   Most of the work of actually 
>>>> synchronizing the DIT is covered by the LCUP document, I just need 
>>>> the clients to be notified "when" to perform LCUP. Ideally, the 
>>>> trigger mechanism would be some type of lightweight, connectionless 
>>>> notification, similar to an SNMP trap but more specific.
>>>
>>>
>>> other servers or clients (except in the case of replication, but 
>>> that is not a standard yet, nor would it likely serve your purpose). 
>>>  Some LDAP servers do support callbacks or triggers that usually 
>>> involve having to write your own plugin or some other mechanism.
>>>
>>> that issues an LCUP persistent search against an LDAP server, and 
>>> the client would put a message on the bus/queue every time it 
>>> received an LCUP response.  Other clients register with the 
>>> bus/queue and receive
>>>