Subject: RE: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

From: "Babich, Alan" <[email-masked]>
Date: Thu, 6 Jul 2000 17:44:10 -0700

To use an analogy, I use the UNIX "find" command a lot to find
files. If it's used at the root of the file system, it includes
all the remotely mounted file systems too. Even just searching
the local files system takes quite a while. But when that's what 
I want to do, that's what I need to get done. If I decide it's 
taking too long, I simply kill it.

So, is there a way to kill a request that's taking too long?
If so, that helps quite a bit. A general mechanism for canceling 
operations addresses the problem for all long running operations.
You want a general solution, not a special solution for each
long running operation.

Alan Babich

-----Original Message-----
From: Jim Davis [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 1:09 PM
To: Greg Stein; [email-masked]
Subject: Re: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

At 07:15 AM 7/6/00 -0700, Greg Stein wrote:
>What is the general consensus on PROPFIND with Depth: infinity? I quoted a
>couple messages below that tend to favor disallowing them. I got that
>impression from some other comments on this list, but couldn't find
specific
>references.
>
>For clarity: can prople give opinions on simply disabling PROPFIND
infinity?

I oppose disabling infinity.  it is useful (as other emails have shown).

I agree to adding a principled way to refuse a request that's too expensive.

---

From: Brian Morin <[email-masked]>
Date: Thu, 6 Jul 2000 13:43:32 -0700

Wouldn't the scenario Hartmut gave be better addressed with a DASL request
to pull up only the list of modified files?

Doing lots of depth 1 propfinds is not all that bad.  If you pipeline the
requests you can elminate a lot of the latency, assuming a tree with
non-trivial breadth.  You get hiarchical structure built in (propfind XML
entries in a depth infinity response do not appear to be ordered, leaving
the client to rebuild the hiarchy from URLs.)  Finally, you are sending a
lot of small requests to a  web server, which is usually designed to
efficently handle a lot of small requests.

-----Original Message-----
From: Hartmut Warncke [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 8:23 AM
To: WebDAV WG
Subject: Re: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

I am of the opinion that disallowing  depth infinity PROPFINDS
would cut off one of the main WebDAV features.  In order to
explain that opinion I  would  like to give you a short example
how we use WebDAV in Adobe GoLive 5.0 (the following sample
is not a theoretical one it's out of the everday life of a GoLive user):

Imagine that you need the modification dates of  1000 resources
on the server in order to synchronize the resources on the server
with the resources on your local machine. Even if the resources
are located in a directory with a very deep hierarchy it's not a
problem with a depth infinity PROPFIND. You ask the server for
the dates and get the response for all the 1000 resources within one
Network request/ response.

On the other hand if you stick to PROPFIND Depth: 1 you end up
in a tremendous drawback (a ftp-like solution) because the client
software has to go down the hierarchy step by step and ask for the
content of each directory  level. The User has  to be very patient
in that scenario ...

So to my mind the infinity PROPFIND is a main WebDAV feature
for practical purposes.

In addition to that I would like to draw your attention to the fact
that it is very difficult to base a software implementation on a
WebDAV RFC which changes frequently because we are not able
to ship a new Release of our software every two weeks.

Hartmut

"Clemm, Geoff" wrote:

> I think we should disallow Depth:infinity on a PROPFIND, and require that
> the client pass in some maximum depth to let the server know when it
> should stop.
>
> Cheers,
> Geoff

---

From: "Clemm, Geoff" <[email-masked]>
Date: Thu, 6 Jul 2000 17:20:12 -0400

I agree with the "client proposes, server disposes" guideline,
but currently we are (in my view, inappropriately) limiting what
the client can propose.

In particular, we are not allowing the client to propose an
upper limit such as "20" or "100", even when the client knows
that to be the appropriate upper limit for its PROPFIND request.

Cheers,
Geoff

-----Original Message-----
From: Jim Davis [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 5:14 PM
To: WebDAV WG
Subject: RE: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

At 05:58 PM 7/6/00 +0100, Gary Barnett wrote:
>I think that creating a specification that builds in non-deterministic
>behaviour would be a real pain.
>
>I think that the idea of passing a depth value (with perhaps a default
value
>which all servers support) makes sense from a client perspective.

What we gain from the indeterminacy is flexibility.  Otherwise, we either
set the minimum standard high (and rule out cheap implementations) or set
it low (thus requiring all clients to use inefficient methods, and making
powerful implementations either useless or non-standard.)

Yaron put it like this

"The client proposes, the server disposes".

Clients should ask for what they want, and be prepared to get less than
that.

---

From: "Babich, Alan" <[email-masked]>
Date: Thu, 6 Jul 2000 20:21:01 -0700

I, too agree with the "the client proposes, the server disposes" theory.
One advantage is that's a general way for the server to defend against
naive and malicious clients from accidental or deliberate denial 
of service attacks.

To extend my UNIX find analogy, if all the programmers at my company
did UNIX find commands on the root directory, that would load the
servers pretty good. You can generally trust your fellow programmers
not to continually do stupid things, so that would be a temporary
problem. But in the world of the internet, you have much less 
sophisticated people that could unwittingly do expensive things 
repeatedly, plus you have a few malicious hackers that
might repeatedly do expensive things deliberately. This supports
the "server disposes" theory.

But, as to the idea of letting the client propose 20 or 100
as the limit (in addition to 0, 1, and infinity), the increased 
complexity might not be worth the decreased simplicity. I like
the KISS principle. The UNIX find command doesn't have a way
to specify a limit on the depth. It only has depth infinity.
The UNIX find command has withstood the test of time, and I know
of no UNIX implementations that have extended it to specify a limit
of N. Since the UNIX find command is doing something somewhat 
similar, that tends to support my gut feeling that we need not 
bother to allow specification of depth N.

So, what I propose is the following: Why don't we wait
for feedback from real users to see if we really need depth N?
Cut features until it's not useful for the first release,
then add only features demanded by the customers in later
releases.

Alan Babich

-----Original Message-----
From: Clemm, Geoff [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 2:20 PM
To: 'Jim Davis'; WebDAV WG
Subject: RE: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

Cheers,
Geoff

-----Original Message-----
From: Jim Davis [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 5:14 PM
To: WebDAV WG
Subject: RE: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

value

Yaron put it like this

---

From: "Wright, Tom" <[email-masked]>
Date: Fri, 7 Jul 2000 08:45:28 +0100

While what you say is true, it puts a huge load in the network, one of the
purposes of infinite requests was to perform an operation all in one go on
the server with out excessive comms from the client to the server ala
pipelining.  

Handling one large request is far more efficient that handling lots of small
requests, particularly when the resources are not on a file system , e.g.
accessing a Database or other custom repository.

I strongly oppose the removal of PROPFIND infinite depth requests.

As an aside, as for n depth requests, how deep is n ?? It seems a bit
arbitrary.

Regards

Tom
<All opinions are my own, not necessarily that of my employer>

-----Original Message-----
From: Hartmut Warncke [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 8:23 AM
To: WebDAV WG
Subject: Re: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

Hartmut

---

From: Jim Whitehead <[email-masked]>
Date: Fri, 7 Jul 2000 08:43:13 -0700

Accidentally caught by the spam filter. I have added Kevin's email to the
accept2 list for w3c-dist-auth, so his emails should not be bounced in the
future.

- Jim

-----Original Message-----
From: Kevin Dyer [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 8:35 AM
To: Greg Stein; [email-masked]
Subject: [Moderator Action] RE: [[email-masked]: Re: [dav-dev] Depth
Infinity Requests]

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Greg Stein
> Sent: Thursday, July 06, 2000 10:15 AM
> To: [email-masked]
> Subject: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]
>
>
> impression from some other comments on this list, but couldn't
> find specific
> references.
>
> For clarity: can prople give opinions on simply disabling
> PROPFIND infinity?
>
> JimW: we should probably note (explicitly) in the spec that a server may
> return a 403 (Forbidden) if a client requests a PROPFIND with a Depth of
> infinity.
>

I'm in agreement with Jim.  We should not allow infinite depth requests at
all.  Depending at what level the request is started from and the complexity
of the PROP data, the request could place a significant resource strain on
the server.

A side benefit, of not allowing infinite depth requests, servers as a
security speedbump.  It forces an end application to make more requests to
retrieve the entire tree.  Which in turn should raise a flag with someone or
something watching the logs for unusual behavior.

				Just my 0.02 Galactic Credits,

					Kevin

____________________________________________

Kevin J. Dyer
Sr. Technologist, Product Management
[email-masked]

TEL:     978-322-2011
FAX:     978-441-0071
MOBILE:  978-314-9855

MatrixOne, Inc.
Two Executive Drive
Chelmsford, MA  01824  USA
www.matrixone.com

Leading Provider of Internet Business Collaboration Solutions
____________________________________________

---

From: Jim Whitehead <[email-masked]>
Date: Fri, 7 Jul 2000 08:43:53 -0700

Another message caught yesterday by the spam filter. :-(

- Jim

-----Original Message-----
From: Kevin Dyer [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 10:30 AM
To: Kevin Wiggen; WebDAV WG
Subject: [Moderator Action] RE: [[email-masked]: Re: [dav-dev] Depth
Infinity Requests]

> [mailto:[email-masked]]On Behalf Of Kevin Wiggen
> Sent: Thursday, July 06, 2000 12:28 PM
> To: [email-masked]; WebDAV WG
> I believe we should follow the DASL spec on this one.  Dasl allows a
server
> to send a 507 if the query produced more results than the server is
willing
> to transmit.  Partial Results have been transmitted.

| From Tim Ellison
|
| The server should be free to refuse depth requests (>1<g>) that it decides
| are too expensive.  Of course, that does not preclude the spec from
| allowing the header value to be 0 .. n or infinity, but it does further
| complicate the client that has to deal with refusals and 'do the work'.

Then the server can respond to an infinite request with either a 403, if the
admin decides to be frugal with server resources, or a 507 and send up to
the
high water mark of depth.  It leaves more wiggle room but it looks like the
best solution.

>
> This is the implementation on the Xythos Storage Server for both SEARCH
and
> PROPFIND.

	Kevin

---

From: Jim Whitehead <[email-masked]>
Date: Fri, 7 Jul 2000 08:44:39 -0700

This is the last message caught in the spam filter yesterday.

- Jim

-----Original Message-----
From: Kevin Dyer [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 10:40 AM
To: [email-masked]
Subject: [Moderator Action] Re: [dav-dev] Depth Infinity Requests]

> [mailto:[email-masked]]On Behalf Of Gary Barnett
> Sent: Thursday, July 06, 2000 12:58 PM
> To: WebDAV WG
value
> which all servers support) makes sense from a client perspective.
But you still need to be able to tell the client that you've asked for too
much
and I'm only going to give you back this amount.  Or you've asked for
something
that I can't generate now.  Maybe the 403 is not the answer, but a new
return
code that actually tells the client what the high water mark is.

I apologize if this is in the latest version, shouldn't the client be able
to
request from the server all of the high and low watermark settings we've
been
talking about as part of a handshake?

Kevin

---

From: "Clemm, Geoff" <[email-masked]>
Date: Fri, 7 Jul 2000 16:21:22 -0400

The proposal to allow a depth value other than 0,1,infinity
has been motivated by some very clear and common use cases
(i.e. either deliberately or mistakenly invoking PROPFIND on
the root of a very large resource tree).

Arguments of the form "it would be simpler to only allow 0,1,infinity"
do nothing to address the use case that is of concern.

One of the postings used the fact that a "find" request does not have
a depth argument to support the position that a PROPFIND request
should not need one either.  I believe this argument does not hold
for a variety of reasons, such as the fact that "find" is almost
always used with a predicate (commonly "name"), and that the OS
commonly provides job control and a data pipe mechanism to control
the behavior of find, neither of which are available with a
PROPFIND request.

Cheers,
Geoff

-----Original Message-----
From: Babich, Alan [mailto:[email-masked]]
Sent: Friday, July 07, 2000 3:36 PM
To: 'Jim Davis'; WebDAV WG
Subject: RE: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

I wasn't trying to promote the UNIX find command as an
example of good design. I was only trying to make a point 
about the KISS principle. In fact, "find" does not have the
simplest design for the normal case.

The only way I ever use the UNIX find command is:

    find <directory> -name <file name specification> -print

This just displays the full path name of the file(s) I'm 
looking for. I completely ignore all the other powerful 
functionality. So, from the point of view of the way I use it, 
it is very simple. It would be even simpler if I didn't have 
to figure out that I had to add "-name" and "-print" to get 
what I want. The designers and implementors wasted (as far 
as my case is concerned) all their valuable time and effort 
implementing features that I don't use, and made it more 
complicated for me to figure out (as a side effect of the 
additional features). I assume that somebody out there thinks 
that some of the other features are critical, so that
the effort and increased complexity isn't entirely in vain.

My point is that my experience with this particular command 
supports the approach of "cut features until you are about 
to cut too far, then let customers tell you what features to 
add on subsequent releases". That approach tends to maximize
the simplicity of the normal case and ease of implementation,
and things tend to get added in priority order.

So, I'm in favor of (a) in your e-mail below -- no change.

Alan Babich

-----Original Message-----
From: Jim Davis [mailto:[email-masked]]
Sent: Thursday, July 06, 2000 11:22 PM
To: WebDAV WG
Subject: RE: [[email-masked]: Re: [dav-dev] Depth Infinity Requests]

At 08:21 PM 7/6/00 -0700, Babich, Alan wrote:

>... the increased 
>to specify a limit on the depth. 

Poor example, in support of a good argument.  The find command has dozens
of options, including general boolean combinations of predicates.  If Unix
find is your idea of something simple, I would hate to see something you
thought complex.

That said, it seems to me there are now three related proposals on the
floor.  They are
 1. change the values accepted in the depth header
 2. change the responses the server may return
 3. provide means for client to abort a request that is taking too long.

Of these, there are four alternatives for proposal 1:

a) depth may be only 0 or 1, or infinity (what we have now)

b) depth may be only 0 or 1

c) depth may be an integer, but not infinity.

d) depth may be an integer, or infinity

Proposal 2 is to add a response code the server may use to reject a
request, either because the specified depth is too great, or because the
results would be larger than the server is willing to return.  This
proposal only makes sense if for proposal 1 one prefers a, c, or d.

Proposal 3 is to add a mechanism where the client can abort a request that
has taken too long.

[I believe, but am not sure, that a client can simply close the connection]

Proposal 1 is the linch-pin.

the arguments for alternative a are that it's just what we have now.

the arguments f(I have seen) for b are
 * it is even simpler than what we have now

 * if one rejects it,  then if any client that can tolerate a refusal
(proposal 2) would have to be able to operate in depth-one mode anyway.  if
one takes the trouble to code the client to be able to work with
single-depth requests anyway, why would one even bother using the depths
greater than one?  it's just more code to write.

 * given pipelining, there would be little performance penalty.

---

From: Jim Whitehead <[email-masked]>
Date: Tue, 18 Jul 2000 10:51:24 -0700

Accidentally caught by the spam filter -- I've added Yaron's email address
to the accept2 list.

- Jim

-----Original Message-----
From: Yaron Goland [mailto:[email-masked]]
Sent: Monday, July 17, 2000 2:41 PM
To: [email-masked]'
Subject: [Moderator Action] RE: [[email-masked]: Re: [dav-dev] Depth
Infinity Requests]

Jim Doubek says: "Note that for realistic sized repositories, say 50K to
100K files, any depth=infinity request near the repository root is going to
be too expensive. For instance, an allprop request in such a case will be
several to tens of megabytes, and may take minutes to produce."

My company is looking to WebDAV to provide us with versioning functionality.
One of the most basic functions we perform is synching to a tree, which is
huge and which starts at the root.

Ignoring the issues of additional client complexity introduced by removing
propfind = infinity, without propfind = infinity network performance gets
significantly worse. What was previously a single request/response now
becomes Log(N) request/responses where N is the size of the longest chain in
the tree. Even if you have a fairly compact tree with one weirdo chain, your
performance is held hostage to that chain. Note: I am treating multiple
pipelined request/responses as being one request/response for the purposes
of counting round trips.

The proposal that we allow clients to submit individual integers is sort of
the worst of all worlds. It makes clients more difficult to program, it
makes servers more difficult to program (it is easier to support infinity
than a specific integer which requires keep track of a counter all the way
down the tree), it makes network performance worse and it doesn't map to any
common scenarios. In the majority of cases I have seen in my years of client
programming the client wants everything below a specific starting point. The
main exceptions are 0 and 1 but both of those are already covered.

As such I believe that we should keep depth = infinity and that we should
not allow for specific integers to be submitted.

I have a lot of sympathy for server makers who see depth = infinity as a
denial of service attack. It definitely screws server performance. The humor
of this thread is - We've already been down this path before. Once upon a
time WebDAV was only going to support depth 0 and 1. If you want to read a
summary of the arguments and how they were settled see the section entitled
<Out of our Depth> in
http://lists.w3.org/Archives/Public/w3c-dist-auth/1998OctDec/0303.html.

I completely agree with people who want to use the DASLesque solution of
returning a "Too much server processing time required" error message. I
think this is a great idea. One of the features my servers provide are
Service Level Agreements. I will check your SLA when I process your request
and if your SLA isn't high enough I will boot your depth = infinity request.
Obviously anonymous log-ins would get very little processing assigned to
them.

Let's not throw the baby out with the bath water. Depth = infinity is a
powerful feature that covers one of the most common editing scenarios "Synch
this tree" in a manner that has great network performance. This is a big
benefit of WebDAV. However let us also empower server authors to reject
requests that abuse their server. I think that is the best compromise.

		Yaron

P.S. Long experience teaches me that a number of the server authors reading
this letter are gleefully rubbing their hands together going "Great! I will
reject every propfind = infinity request with a "too much server processing
time" error. Problem Solved!!!" I suggest you read
http://lists.w3.org/Archives/Public/w3c-dist-auth/1999JulSep/0359.html and
keep in mind that clients will treat the "too much server processing time"
error as catastrophic. This type of behavior probably won't help your server
sales.