Subject: RE: [RFC2518 Issue] PROPFIND 'allprop' usage

From: "Wright, Tom" <[email-masked]>
Date: Fri, 24 Nov 2000 08:03:38 -0000

Various replies below

Regards

Tom Wright
Xerox

All comments are my own opinions and not really from my company.

-----Original Message-----
From: Hartmut Warncke [mailto:[email-masked]]
Sent: 23 November 2000 12:06
To: Greg Stein
Cc: Hartmut Warncke; WebDAV WG
Subject: Re: [RFC2518 Issue] PROPFIND 'allprop' usage

Greg,

> Given that the default configuration of mod_dav prevents a Depth:infinity
> PROPFIND, I'd guess that a rejection of Depth:infinity is a definite
> possibility. What does GoLive do when it gets the rejection? Do you simply
> skip that feature, send a bunch of Depth:1 requests, or something else?

>We do  *not*  send a bunch of Depth:1 requests, some WebDAV functionality
is simply
>*not*  working in GoLive 5 when the server is not willing to respond to
depth infinity
>requests. We have simply tried to implement the WebDAV protocol as it is
described in
>RFC2518 and RFC2518 does  *not*  allow a server to deny a depth infinity
access. At
>the point in time when the "DavDepthInfinity" flag was introduced into
mod_dav it was
>much too late for us to implement such a "bunch of Depth:1
requests"-solution because
>we were right before GM.

>And I think this is a general problem. There are a lot of very huge
products of huge
>companies out there which support WebDAV and I am very lucky about that
especially
>because I think that the idea of WebDAV is very powerful and advanced. But
we all have
>to keep in mind that a huge product has a well defined life cycle and
workflow which
>cannot be changed frequently. And of course a product cannot be changed
when it is
>already in the box. So when some changes take place in the protocol or its
important
>implementations like mod_dav and this will effect negatively or even break
products
>which are already in the box or right before GM, a customer who have paid
for the
>product has to wait at least for the next product release (which will take
months or
e>ven years).

Though Issuing patches is a well known tried an trusted appropach to this
kind of problem.

>So I think it's very important to get some degree of stability and
continuity in the
>WebDAV protocol and its implementations which does not mean to let the
protocal
>unchanged during the next years but we should keep its key features.
Otherwise a lot
>of companies will think twice before they implement WebDAV support and  -
without any
>doubt -  every protocol needs a widespread acceptance from commercial
companies and
>products  to be succesful.

All of your responses are missing the point Hartmut.  If a server generates
a 2 or 4 Gigabyte XML response in response to your thousands of files on the
server, 1) its going to hog the server, and even cause it to fail if your
building XML in memory, 2) clog the network with traffic and 3) how on earth
is the client going to cope with parsing a 2 or 4 gig ( or larger ) XML
response, are you simply hoping that you never get a response so large that
you cant handle it ? Sending a few hundred byte request ( which may be
reasonable to the user ) can result in potentially crippling the server,
network or client with unexpectedly large amounts of data, after all most
users have no idea of the size of the response before hand.  I agree we need
to do our best to maintain backwards compatability, but it sounds like your
check all in one go approach is a naive implementation. There is simply no
need to retrieve more than a directory ( ala depth 1 ) of files/folders
properties at a time.  Heck you dont even need webdav to do this, using
if-modified-since is good enough for files, unless you need to check
properties as well.

Your arguement about it being efficient is hard to understand.  Whats
efficent, your use of server and client resources, network band width etc ?

I am actually implementing propfind as we speak, and have full sympathy with
your position and agree we must not break backwards compatibility, however
the current spec is 1) a draft and 2) not considerate of resources, after
all they are not infinite.  I am suprised greg has not argued the case of
infinite depth and got the spec changed!!

Maybe the best compromise is as shaun suggested is to have a new status
code, saying i have done a depth 1 instead of a depth infinity propfind
response, the rest is up to you kind of code, some new 200 code perhaps. The
current situation of allowing infinite depth allprop with an unknown number
of properties for an unknown number of resources is 'silly'.  The protocol
is not just for clients, servers have to implement it also, and they need to
be able to throatle malicious/unintentional requests which will potentially
flood the network and cripple the server/client.

As a whole, i hope Jim is reading these emails, as there is clearly a number
of people unhappy with the current specification.  Maybe it should be
changed before the next release?

Best regards

Tom

---

From: Kevin Wiggen <[email-masked]>
Date: Sun, 26 Nov 2000 15:02:33 -0800

2)  Allprop.

I have not seen any answers for WHY people want to use Allprop, except for
there are clients that use it today.  This is why I would like to Deprecate
the use of Allprop.  Yes servers will continue to support it for backward
compatibility, but I believe that NEW clients should NEVER use ALLPROP.  If
a client NEEDS all the properties (and not just the properties it knows how
to handle), it can do a PROPNAME followed by a PROPFIND asking for all the
properties.  Thus the corner case of needing all the properties is still
available, but is not the generic case.

I also would like to remove ALLPROP as the default for PROPFIND.

I have seen the number of things people are using Properties for.  Its both
genius and scary :)  But I can see no reason why a client would care about
most of them unless they specifically knew what they meant.  In which case
they know to ask for them.

I think the spec should change to disallow ALLPROP from clients.

Kevin

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Wright, Tom
Sent: Friday, November 24, 2000 12:04 AM
To: 'WebDAV WG'
Subject: RE: [RFC2518 Issue] PROPFIND 'allprop' usage

Various replies below

Regards

Tom Wright
Xerox

-----Original Message-----
From: Hartmut Warncke [mailto:[email-masked]]
Sent: 23 November 2000 12:06
To: Greg Stein
Cc: Hartmut Warncke; WebDAV WG
Subject: Re: [RFC2518 Issue] PROPFIND 'allprop' usage

Greg,

is simply
depth infinity
described in
access. At
mod_dav it was

products of huge
especially
we all have
workflow which
when it is
important
products
for the
months or
e>ven years).

continuity in the
protocal
Otherwise a lot
without any
companies and

Best regards

Tom

---

From: Kevin Wiggen <[email-masked]>
Date: Sun, 26 Nov 2000 14:59:17 -0800

I think this should be split into 2 threads (as someone suggested earlier)
and so I will attempt to do so here :)

1)  Depth infinity searches.

I can see some very good uses for depth infinity searches and can also see
the large DOS attack.   To alleviate the situation I will suggest (again)
that we add a 507 Insufficient Storage status code response to a Propfind.
This will allow a server to respond with partial results when the result set
is to large.  This was how DASL suggested that servers handle the Result Set
Truncation possibility of searches that were to large, and since Propfind is
nothing but a limited search, I believe the 507 works well in this case
also.

In this way the server is free to answer as much as they can, and still give
a proper error condition.  In cases where the depth infinity is not rather
large, the 207 is returned as normal.  In the case of LARGE depth infinity
propfinds, the 507 is returned with partial propfind data.

Kevin

-----Original Message-----

Various replies below

Regards

Tom Wright
Xerox

-----Original Message-----
From: Hartmut Warncke [mailto:[email-masked]]
Sent: 23 November 2000 12:06
To: Greg Stein
Cc: Hartmut Warncke; WebDAV WG
Subject: Re: [RFC2518 Issue] PROPFIND 'allprop' usage

Greg,

is simply
depth infinity
described in
access. At
mod_dav it was

products of huge
especially
we all have
workflow which
when it is
important
products
for the
months or
e>ven years).

continuity in the
protocal
Otherwise a lot
without any
companies and

Best regards

Tom

---

From: "Geoffrey M. Clemm" <[email-masked]>
Date: Sun, 26 Nov 2000 20:34:45 -0500 (EST)

I heartily support the removal of allprop from the protocol.

If client writers start now to replace their use of allprop-PROPFIND
with a propname-PROPFIND/PROPFIND pair, they have plenty of time
to be in compliance with a revision of 2518 that does not support
allprop.

Cheers,
Geoff

---

From: "Geoffrey M. Clemm" <[email-masked]>
Date: Sun, 26 Nov 2000 20:47:00 -0500 (EST)

I support the use of 507 to indicate a server wishes to truncate
the PROPFIND.

I assume that if a server will respond with partial results, that the
507 will be returned under the appropriate response elements (whose
properties were omitted), while the overall status of the request
would be a 207.

Cheers,
Geoff

   Date: Sun, 26 Nov 2000 14:59:17 -0800
   From: Kevin Wiggen <[email-masked]>

   Kevin

---

From: Greg Stein <[email-masked]>
Date: Sun, 26 Nov 2000 18:09:57 -0800

> to be in compliance with a revision of 2518 that does not support

I vehemently agree. :-)

I also support limitations on Depth:infinity requests (as if you couldn't
guess :-).

[ mod_dav (by default) just tosses them, responding with 403 (Forbidden);
  since that is a legal return for a PROPFIND, it seemed to make sense in my
  situation (and clients would have to deal with it anyways). ]

Returning 507 would be a bit more difficult implementation-wise. However, I
think we really shouldn't allow that mechanism. What is a client to do when
it gets a 507? How does it know *what* was left out, and *how* to get those
results? Did the server do a depth-first or a breadth-first response of
properties? Which collections did it recurse into and which did it not? Did
it stop *partway* through a collection? How can a client tell?

With all of those problems, a client has to be *extremely* smart to recover.
It would have to sort through the returned hrefs, analyze the structure, and
try to determine where the server left off.

Pathological case: let's say that I implemented the server with a database
mapping URLs to property sets. I have a spiffy query that returns all rows
that start with a particular URL base. For performance purposes, I *do* not
sort the response, and the database does not guarantee one. Oh oh... what
now? I've got responses in a scatter plot all over the URL namespace.

One alternative would be for the server to "prune" responses and mark these
prunings in the response. The client could then know exactly what is missing
("<this> resource" or "<that> collection"). This mechanism would also be
nice for a "propname" or a "prop" style of PROPFIND with Depth:infinity. I
could easily see a server wanting to prune those, too.

Personally, I say put the 507 and specifically allow 403 responses.

[ I wouldn't mind if Depth:infinity was tossed altogether, but that is too
  extreme, and that depth is *very* handy for debugging :-) ]

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---

From: Greg Stein <[email-masked]>
Date: Sun, 26 Nov 2000 18:11:47 -0800

On Sun, Nov 26, 2000 at 03:02:33PM -0800, Kevin Wiggen wrote:
>...

We either have to toss the default, or leave it. We can't really change the
default to something else (the only valid one would be 'propname').

I'm fine with returning 400 (Bad Request) for a PROPFIND with no body.

Cheers,
-g

---

From: "Lisa Dusseault" <[email-masked]>
Date: Sun, 26 Nov 2000 19:53:30 -0800

I agree with the desire for the client to find out *what's* missing, and the
207 multistatus allows a server to fill in a "507" response for each
property that it chooses to omit due to processing or space limitations.
E.g. a depth-0 'allprop' response could still legitimately omit the value of
the ACL property but mark its presence with the 507 status code.

However, I'm not sure it's so easy for a server to indicate that it has
omitted a whole hierarchy in a 207 multistatus response.  What if the client
asks for depth-infinity 'allprop' on a directory with sub-directories?  I
guess the server could:
 - for every item in the directory that is not a sub-directory, respond
normally with property values or property status codes
 - for sub-directories that it wishes not to recurse into, include this kind
of XML

	<DAV:response>
		<DAV:href>/subdir</DAV:href>
		<DAV:status>HTTP/1.1 507 Insufficient Storage</DAV:status>
	</DAV:response>

This appears legal under RFC 2518's definition of the elements that can/must
go in the response element.

   <!ELEMENT response (href, ((href*, status)|(propstat+)),
   responsedescription?) >

I don't see any way for the server to return the properties for 'subdir'
while simultaneously indicating to the client that it did not recurse into
'subdir'.  Thus in order to limit its answer, the server would omit the
property set for 'subdir' as well.  Not the best of solutions; anybody have
brighter ideas?

Lisa

> [mailto:[email-masked]]On Behalf Of Greg Stein
> Sent: Sunday, November 26, 2000 6:10 PM
> To: [email-masked]

---

From: "Geoffrey M. Clemm" <[email-masked]>
Date: Sun, 26 Nov 2000 23:58:55 -0500 (EST)

Note that the 507 proposal is not for dealing with allprop,
but rather with dealing with depth infinity, so I've modified
the subject line (Kevin valiantly tried to separate the issues,
but they seem to keep getting mashed back together :-).

   From: Greg Stein <[email-masked]>

   Returning 507 would be a bit more difficult
   implementation-wise. However, I think we really shouldn't allow
   that mechanism. What is a client to do when it gets a 507?

If it gets a 507 as the top level status, it knows it has to retry
the request with a smaller depth.  If it gets a 207 as the top level
status, and a 507 as the status of some member, it has to retry the 
request on that member directly.

   How does
   it know *what* was left out,

It knows by what results are 507's and what are 200's.  Note: the
proposal is not that you use a 507 to omit some of the properties in a
response element, but rather to omit *all* of the requested properties
in the response element.

   and *how* to get those results?

See above.

   Did the server do a depth-first or a breadth-first response of
   properties?

I assume you mean, a depth-first or breadth-first search of
the members?  If so, you would be able to tell by what response
elements are 200's and what are 507's.

   Which collections did it recurse into and which did it
   not? Did it stop *partway* through a collection? How can a client
   tell?

We would need to define server behavior here.  For example: "you have
to return the properties of a collection if you are going to return
properties of any of its members, and you return 507 if you are not
going to return properties of the collection or any of its members,
due to space limitations".

If we define it properly, it only has to retry the 507's.

Just prune out all the responses whose URL match certain prefixes,
and return a 507 for the collections that you pruned out.

   One alternative would be for the server to "prune" responses and mark these
   prunings in the response. The client could then know exactly what is missing
   ("<this> resource" or "<that> collection").

Yes.  (What don't you like about this alternative?)

   This mechanism would also be
   nice for a "propname" or a "prop" style of PROPFIND with Depth:infinity. I
   could easily see a server wanting to prune those, too.

I'm not quite sure what you are referring to here.  The 507's are for
pruning responses when the depth is too great, and is independent of
whether it is a propname or prop style PROPFIND.

   Personally, I say punt the 507 and specifically allow 403 responses.

The 403 would definitely leave the client in the dark ... it wouldn't
know whether or not to retry the PROPFIND with a different depth.
Whereas the 507 tells the client very clearly that it is the depth
value that is too great, while still giving it some of what it asked
for.

Cheers,
Geoff

---

From: "Geoffrey M. Clemm" <[email-masked]>
Date: Mon, 27 Nov 2000 00:03:37 -0500 (EST)

From: "Lisa Dusseault" <[email-masked]>

If the server wants to return properties for subdir, but not for its
children, it would just return a 507 for each of its children.

Cheers,
Geoff

---

From: "Lisa Dusseault" <[email-masked]>
Date: Sun, 26 Nov 2000 22:48:17 -0800

I agree with everything except...

> From: Geoffrey M. Clemm
> Subject: Depth infinity PROPFIND (was: [RFC2518 Issue] PROPFIND
> 'allprop' usage)
>
> Note: the

I disagree about the scope of the proposal, at least the proposal that _I'd_
like to make.  I know this isn't exactly the same issue, but servers want to
(anthropomorphically-speaking) be able to omit property values which are too
expensive to compute when the PROPFIND covers many resources.  Thus, with a
PROPFIND request covering a number of resources simultaneously, the server
could choose to include the 507 response rather than the property value.
This is even easier than omitting sub-directories, because 2518 already has
examples of returning an error for a specific property (p 25-26 shows two
properties omitted due to permissions).

It's because these two issues, expensive properties and depth-infinity
PROPFIND, are so similar that we've been unable to separate them in mail
threads.  Luckily I think they can be dealt with in a very similar manner.

I'd like it to be explained, in RFC2518 ideally or at least understood
through the mailing list, that a client must be prepared to get incomplete
PROPFIND results containing 507 codes in status elements to indicate that it
was too expensive to calculate certain properties or to recurse into certain
trees.  With this much information, the client is perfectly able to send
more specific requests to fill in the holes, whether they be missing
property-values or missing resource-details.

This ought not to be too uncomfortable a change for existing clients: since
this is legal within 2518, it should not absolutely break them.  And, there
is nothing forcing servers to return incomplete results, so any server
implementor can choose to remain compatible with existing clients relying on
depth-infinity PROPFIND returning complete results.

Lisa