From hyc@highlandsun.com Thu Apr 2 07:43:15 2009 Return-Path: X-Original-To: ldapext@core3.amsl.com Delivered-To: ldapext@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8957E3A68FE for ; Thu, 2 Apr 2009 07:43:15 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.74 X-Spam-Level: X-Spam-Status: No, score=-0.74 tagged_above=-999 required=5 tests=[BAYES_20=-0.74] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qXWFGFj8QEl2 for ; Thu, 2 Apr 2009 07:43:13 -0700 (PDT) Received: from mail.highlandsun.com (mail.highlandsun.com [70.87.222.79]) by core3.amsl.com (Postfix) with ESMTP id 56A5028C0FD for ; Thu, 2 Apr 2009 07:43:07 -0700 (PDT) Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.highlandsun.com (Postfix) with ESMTP id 9221710F87 for ; Thu, 2 Apr 2009 10:44:08 -0400 (EDT) Message-ID: <49D4CF35.9040500@highlandsun.com> Date: Thu, 02 Apr 2009 07:44:05 -0700 From: Howard Chu User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1b3pre) Gecko/20090317 SeaMonkey/2.0a1pre Firefox/3.0.3 MIME-Version: 1.0 To: Ldapext Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: [ldapext] Request Progress Reporting X-BeenThere: ldapext@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: LDAP Extension Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2009 14:43:15 -0000 This discussion began on the OpenLDAP-Devel mailing list, talking about returning a progress indicator to a client for a long running search operation. http://www.openldap.org/lists/openldap-devel/200903/msg00082.html It was suggested that progress reporting may be useful in general, not just for search requests. Moving the discussion here to get more input. The idea currently is to attach a "ProgressReport" control to any request. The control will specify a time interval to indicate how often progress updates should be returned. ProgressReport updates may be attached in Response controls accompanying Search Response messages, Result messages, and also in Intermediate Response messages. The ProgressReport update will contain two integers of arbitrary units: one representing the total amount of work, and one representing the current amount already executed. E.g., for a Search request, the total value may represent the number of candidate entries to be evaluated in the search, and the current value may represent the number of entries already evaluated. Typically a write request should execute quickly, but if a write has other side-effects, it may take longer. E.g., if a server is configured with synchronous replication, where the client write does not complete until some number of replicas have received the update, then the progress report may be useful to indicate how many replicas have been updated. There are some security concerns here regarding the control giving out information that clients wouldn't otherwise have access to. E.g., access controls may limit a client to seeing only a subset of entries in a directory; a ProgressReport may allow them to discover the true number of entries. It's not clear to me that this is critical or useful to an attacker, but it would make sense for implementations to also control who is authorized to use the ProgressReport control, and to fudge the units used in the update data so that they don't correspond 1-to-1 with any actual counts of anything meaningful. Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From hyc@highlandsun.com Thu Apr 2 08:01:06 2009 Return-Path: X-Original-To: ldapext@core3.amsl.com Delivered-To: ldapext@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 127E328C224 for ; Thu, 2 Apr 2009 08:01:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.67 X-Spam-Level: X-Spam-Status: No, score=-1.67 tagged_above=-999 required=5 tests=[AWL=0.930, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K5f+yeFzycNZ for ; Thu, 2 Apr 2009 08:01:05 -0700 (PDT) Received: from mail.highlandsun.com (mail.highlandsun.com [70.87.222.79]) by core3.amsl.com (Postfix) with ESMTP id 05CDD3A6900 for ; Thu, 2 Apr 2009 08:01:05 -0700 (PDT) Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.highlandsun.com (Postfix) with ESMTP id 65CE310F87 for ; Thu, 2 Apr 2009 11:02:06 -0400 (EDT) Message-ID: <49D4D36B.4000609@highlandsun.com> Date: Thu, 02 Apr 2009 08:02:03 -0700 From: Howard Chu User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1b3pre) Gecko/20090317 SeaMonkey/2.0a1pre Firefox/3.0.3 MIME-Version: 1.0 To: Ldapext References: <49D4CF35.9040500@highlandsun.com> In-Reply-To: <49D4CF35.9040500@highlandsun.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [ldapext] Request Progress Reporting X-BeenThere: ldapext@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: LDAP Extension Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2009 15:01:06 -0000 Howard Chu wrote: > This discussion began on the OpenLDAP-Devel mailing list, talking about > returning a progress indicator to a client for a long running search operation. > > http://www.openldap.org/lists/openldap-devel/200903/msg00082.html > > It was suggested that progress reporting may be useful in general, not just > for search requests. Moving the discussion here to get more input. > > The idea currently is to attach a "ProgressReport" control to any request. The > control will specify a time interval to indicate how often progress updates > should be returned. I forgot to mention - time intervals should probably be in units smaller than seconds, but probably not smaller than milliseconds. Human factors tells us that most users don't perceive lag until intervals exceed 100ms. > ProgressReport updates may be attached in Response controls accompanying > Search Response messages, Result messages, and also in Intermediate Response > messages. > > The ProgressReport update will contain two integers of arbitrary units: one > representing the total amount of work, and one representing the current amount > already executed. > > E.g., for a Search request, the total value may represent the number of > candidate entries to be evaluated in the search, and the current value may > represent the number of entries already evaluated. > > Typically a write request should execute quickly, but if a write has other > side-effects, it may take longer. E.g., if a server is configured with > synchronous replication, where the client write does not complete until some > number of replicas have received the update, then the progress report may be > useful to indicate how many replicas have been updated. > > There are some security concerns here regarding the control giving out > information that clients wouldn't otherwise have access to. E.g., access > controls may limit a client to seeing only a subset of entries in a directory; > a ProgressReport may allow them to discover the true number of entries. It's > not clear to me that this is critical or useful to an attacker, but it would > make sense for implementations to also control who is authorized to use the > ProgressReport control, and to fudge the units used in the update data so that > they don't correspond 1-to-1 with any actual counts of anything meaningful. > > Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From hbf@bombur.uio.no Thu Apr 2 08:54:04 2009 Return-Path: X-Original-To: ldapext@core3.amsl.com Delivered-To: ldapext@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 40A503A6DF8 for ; Thu, 2 Apr 2009 08:54:04 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.58 X-Spam-Level: X-Spam-Status: No, score=-2.58 tagged_above=-999 required=5 tests=[AWL=0.019, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IfE3oyv1pzVB for ; Thu, 2 Apr 2009 08:54:03 -0700 (PDT) Received: from mail-out2.uio.no (mail-out2.uio.no [129.240.10.58]) by core3.amsl.com (Postfix) with ESMTP id 05D613A6E17 for ; Thu, 2 Apr 2009 08:53:06 -0700 (PDT) Received: from mail-mx1.uio.no ([129.240.10.29]) by mail-out2.uio.no with esmtp (Exim 4.69) (envelope-from ) id 1LpPF0-0003W5-Oq; Thu, 02 Apr 2009 17:54:06 +0200 Received: from bombur.uio.no ([129.240.6.233]) by mail-mx1.uio.no with esmtp (Exim 4.69) (envelope-from ) id 1LpPF0-0005z0-Fg; Thu, 02 Apr 2009 17:54:06 +0200 Received: from hbf by bombur.uio.no with local (Exim 4.44) id 1LpPF0-0007Xv-E0; Thu, 02 Apr 2009 17:54:06 +0200 From: Hallvard B Furuseth MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: Date: Thu, 2 Apr 2009 17:54:06 +0200 To: Howard Chu In-Reply-To: <49D4D36B.4000609@highlandsun.com> References: <49D4CF35.9040500@highlandsun.com> <49D4D36B.4000609@highlandsun.com> X-Mailer: VM 7.18 under Emacs 22.2.1 Sender: Hallvard Breien Furuseth X-UiO-Spam-info: not spam, SpamAssassin (score=-5.0, required=5.0, autolearn=disabled, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO) X-UiO-Scanned: EB38034506391EF97AB4FFA94097312F52DBA9E6 X-UiO-SPAM-Test: remote_host: 129.240.6.233 spam_score: -49 maxlevel 200 minaction 1 bait 0 mail/h: 1 total 1454 max/h 8 blacklist 0 greylist 0 ratelimit 0 Cc: Ldapext Subject: Re: [ldapext] Request Progress Reporting X-BeenThere: ldapext@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: LDAP Extension Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2009 15:54:04 -0000 Howard Chu writes: > I forgot to mention - time intervals should probably be in units > smaller than seconds, but probably not smaller than milliseconds. > Human factors tells us that most users don't perceive lag until > intervals exceed 100ms. "Surely only interactive programs will find a use for this feature." Hmm. BTW, we'll also need to define a dummy Intermediate Response, so servers can send this control when they have nothing else to say. -- Hallvard From andrew.findlay@skills-1st.co.uk Thu Apr 2 09:24:01 2009 Return-Path: X-Original-To: ldapext@core3.amsl.com Delivered-To: ldapext@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C9C203A6B2A for ; Thu, 2 Apr 2009 09:24:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IycvlCk4EmSt for ; Thu, 2 Apr 2009 09:24:00 -0700 (PDT) Received: from gnu.ourshack.com (gnu.ourshack.com [84.45.68.121]) by core3.amsl.com (Postfix) with ESMTP id 417313A69A8 for ; Thu, 2 Apr 2009 09:24:00 -0700 (PDT) Received: from 88-97-25-132.dsl.zen.co.uk ([88.97.25.132] helo=tile.skills-1st.co.uk) by gnu.ourshack.com with esmtpsa (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.68) (envelope-from ) id 1LpPit-0002bJ-Te; Thu, 02 Apr 2009 17:25:00 +0100 Received: from andrew by tile.skills-1st.co.uk with local (Exim 4.69) (envelope-from ) id 1LpPit-00058d-3D; Thu, 02 Apr 2009 17:24:59 +0100 Date: Thu, 2 Apr 2009 17:24:59 +0100 From: Andrew Findlay To: Howard Chu Message-ID: <20090402162458.GF9515@tile.skills-1st.co.uk> References: <49D4CF35.9040500@highlandsun.com> <49D4D36B.4000609@highlandsun.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <49D4D36B.4000609@highlandsun.com> User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Andrew Findlay Cc: Ldapext Subject: Re: [ldapext] Request Progress Reporting X-BeenThere: ldapext@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: LDAP Extension Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2009 16:24:01 -0000 On Thu, Apr 02, 2009 at 08:02:03AM -0700, Howard Chu wrote: >> The idea currently is to attach a "ProgressReport" control to any request. The >> control will specify a time interval to indicate how often progress updates >> should be returned. > > I forgot to mention - time intervals should probably be in units smaller > than seconds, but probably not smaller than milliseconds. Human factors > tells us that most users don't perceive lag until intervals exceed 100ms. I would suggest that report messages should include some representation of wallclock time at the server, so that clients can allow for network latency and jitter when interpreting them. This is a common strategy when making SNMP queries: a client will usually request the values of some counters and also the value of the server agent's "time since boot" counter. >> ProgressReport updates may be attached in Response controls accompanying >> Search Response messages, Result messages, and also in Intermediate Response >> messages. Clients must understand that an indication of progress is not an indication that the operation will ever finish, nor that it will be successful if it does. Andrew -- ----------------------------------------------------------------------- | From Andrew Findlay, Skills 1st Ltd | | Consultant in large-scale systems, networks, and directory services | | http://www.skills-1st.co.uk/ +44 1628 782565 | ----------------------------------------------------------------------- From hyc@highlandsun.com Thu Apr 2 19:58:14 2009 Return-Path: X-Original-To: ldapext@core3.amsl.com Delivered-To: ldapext@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5115728C27C for ; Thu, 2 Apr 2009 19:58:14 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.134 X-Spam-Level: X-Spam-Status: No, score=-2.134 tagged_above=-999 required=5 tests=[AWL=0.465, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dEVHBo9RSuk0 for ; Thu, 2 Apr 2009 19:58:13 -0700 (PDT) Received: from mail.highlandsun.com (mail.highlandsun.com [70.87.222.79]) by core3.amsl.com (Postfix) with ESMTP id 28C2728C28A for ; Thu, 2 Apr 2009 19:58:06 -0700 (PDT) Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.highlandsun.com (Postfix) with ESMTP id 1BC5210F87; Thu, 2 Apr 2009 22:59:07 -0400 (EDT) Message-ID: <49D57B7B.6010908@highlandsun.com> Date: Thu, 02 Apr 2009 19:59:07 -0700 From: Howard Chu User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1b3pre) Gecko/20090317 SeaMonkey/2.0a1pre Firefox/3.0.3 MIME-Version: 1.0 To: Hallvard B Furuseth References: <49D4CF35.9040500@highlandsun.com> <49D4D36B.4000609@highlandsun.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: Ldapext Subject: Re: [ldapext] Request Progress Reporting X-BeenThere: ldapext@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: LDAP Extension Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2009 02:58:14 -0000 Hallvard B Furuseth wrote: > Howard Chu writes: >> I forgot to mention - time intervals should probably be in units >> smaller than seconds, but probably not smaller than milliseconds. >> Human factors tells us that most users don't perceive lag until >> intervals exceed 100ms. > > "Surely only interactive programs will find a use for this feature." > Hmm. That's hardly relevant. Much smaller intervals will also place an unreasonable burden on the server. Also, you have platforms like Windows where the system clock period is only 55ms anyway; there'd be no point in asking for anything smaller. > BTW, we'll also need to define a dummy Intermediate Response, so > servers can send this control when they have nothing else to say. I was suggesting we would define a ProgressReport Intermediate Response, which would be sent in those situations. A dummy Intermediate Response could be more useful in general, but I'm not sure whether it's within the scope of this discussion. Certainly that would make processing a lot more uniform. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ From Kurt.Zeilenga@Isode.com Wed Apr 8 18:24:16 2009 Return-Path: X-Original-To: ldapext@core3.amsl.com Delivered-To: ldapext@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C09003A6824 for ; Wed, 8 Apr 2009 18:24:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.874 X-Spam-Level: X-Spam-Status: No, score=-2.874 tagged_above=-999 required=5 tests=[AWL=-0.275, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UrG7Vs6HUaAX for ; Wed, 8 Apr 2009 18:24:16 -0700 (PDT) Received: from rufus.isode.com (rufus.isode.com [62.3.217.251]) by core3.amsl.com (Postfix) with ESMTP id CBAC23A684C for ; Wed, 8 Apr 2009 18:24:15 -0700 (PDT) Received: from [192.168.1.102] ((unknown) [75.141.233.128]) by rufus.isode.com (submission channel) via TCP with ESMTPSA id ; Thu, 9 Apr 2009 02:25:22 +0100 X-SMTP-Protocol-Errors: NORDNS Message-Id: From: Kurt Zeilenga To: Hallvard B Furuseth In-Reply-To: Date: Wed, 8 Apr 2009 18:25:19 -0700 References: <49D4CF35.9040500@highlandsun.com> <49D4D36B.4000609@highlandsun.com> X-Mailer: Apple Mail (2.930.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Cc: Ldapext Subject: Re: [ldapext] Request Progress Reporting X-BeenThere: ldapext@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: LDAP Extension Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2009 01:24:16 -0000 On Apr 2, 2009, at 8:54 AM, Hallvard B Furuseth wrote: > BTW, we'll also need to define a dummy Intermediate Response, so > servers can send this control when they have nothing else to say. After consideration of whether it would be better to define an Intermediate Response specifically for this extension, or define a general "dummy" Intermediate Response to which one can attach this and other controls, I've concluded the latter is better (though it can be defined in the same document). -- Kurt