Subject: RE: [RFC2518 Issue] PROPFIND 'allprop' usage

From: "Hall, Shaun" <[email-masked]>
Date: Wed, 22 Nov 2000 15:27:47 -0000

> -----Original Message-----
> From: Hartmut Warncke [mailto:[email-masked]]
> Sent: 22 November 2000 13:12
> To: Lisa Dusseault
> Cc: WebDAV WG
> Subject: Re: [RFC2518 Issue] PROPFIND 'allprop' usage
> 
> 
> 
> Lisa,
> 
> 
> > One more question, because maybe, as Richard Humphreys 
> suggests, "depth
> > infinity" is the problem.
> >
> > Hartmut, does GoLive5 use depth: infinity PROPFIND requests 
> at all?  If yes,
> > does it use them for custom property discovery?
> 
> Yes, GoLive uses those infinity requests because it's a very 
> effiicient way to
> synchronize the WebDAV server content with the content of a 
> site on the local
> machine with  *o n e*  network request. We are requesting build-in DAV
> properties (like "getlastmodified" or "lockdiscovery") as 
> well as custom
> properties (which were PROPPATCHED by GoLive before) with 
> depth infinity.
> 
> To my mind depth infinity requests are -from the client 
> perspective - a very
> efficient and interesting WebDAV feature.
> Dropping  infinite requests would break GoLive 5!

The implications of PROPFIND with an infinite depth affect both the server
and/or the client. Supposing you have a large WebDAV enabled repository. The
PROPFIND request (a few hundred bytes) could potentially mean that (tens of)
GBs of data is to be returned. Even if the server could generate such a
large response (which is a feat in its own right), it will load the network,
and what will happen when the client receives it? Pity the poor user sitting
there for a long time (hours?), only to be told "timed out" or "out of
memory" or something similar.

> 
> By the way if a server would be allowed to refuse depth 
> infinity requests (like
> mod_dav with its "DavDepthInfinity" flag in its configuration 
> file), the client
> needs a mechanism to figure out if the server is willing to 
> respond to depth
> infinity requests or not. This mechanism has to be precisely 
> defined by RFC2518.
> 

Changing the specification could break existing products, which is a bad
thing. Backwards compatibilty should be maintained.

We think the way forward for the WebDAV specification is to allow servers
the ability to refuse such requests and inform the client. A mechanism
should be defined for the client to understand this. If the client received
a response which basically stated that the server was refusing to service an
infinite depth request, it could issue multiple requests with a Depth:1.

Regards

Shaun Hall
Xerox Europe

> 
> 
> > If not, then we could compromise our way through this by 
> stating that a
> > WebDAV server SHOULD (MUST?) respond to a PROPFIND depth-0 
> 'allprop' request
> > with all custom properties, but that it MAY respond to a 
> PROPFIND depth>0
> > 'allprop' request with a more limited set of properties 
> (suggested to be the
> > non-locking-related properties defined in RFC2518 presently).
> >
> > Would that work?
> 
> I think "allprop" should always return all properties (custom 
> and DAV build-in).
> Perhaps it makes sense to drop PROPFIND-ALLPROP requests with 
> depth infinity but
> please keep the PROPFIND-PROP requests with depth infinity 
> because they are a
> very powerful and efficient instrument. Dropping those 
> requests would lead us
> -in some situations- into a time consuming disaster with a 
> huge amount of
> network traffic.
> 
> Best, Hartmut
> 
> 
> > > -----Original Message-----
> > > From: Hartmut Warncke [mailto:[email-masked]]
> > > Sent: Monday, November 13, 2000 8:07 AM
> > > To: Lisa Dusseault
> > > Cc: WebDAV WG
> > > Subject: Re: [RFC2518 Issue] PROPFIND 'allprop' usage
> > >
> > >
> > >
> > >
> > > > To clarify, does the GoLive 5 WebDAV client rely on using
> > > PROPFIND 'allprop'
> > > > requests to get all custom properties on **any** webDAV server?
> > >  or against
> > > > a specific webDAV server?
> > >
> > > On  **any** WebDAV server.
> > >
> > > > Have you tested this against various servers?
> > > > Does it work with all of them?
> > >
> > > Yes, it's working against IIS and mod_dav for example.
> > >
> > > Best, Hartmut
> > >
> > >
> > > > > -----Original Message-----
> > > > > From: [email-masked]
> > > > > [mailto:[email-masked]]On Behalf Of 
> Hartmut Warncke
> > > > > Sent: Monday, November 13, 2000 4:07 AM
> > > > > To: Lisa Dusseault; WebDAV WG
> > > > > Subject: Re: [RFC2518 Issue] PROPFIND 'allprop' usage
> > > > >
> > > > >
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I have major concerns regarding the change of  the response
> > > on a PROPFIND
> > > > > 'allprop' request in the way you described it.
> > > > >
> > > > > Such a change would be  *v e r y  harmful*   for the GoLive 5
> > > > > WebDAV client.
> > > > > When we send a PROPFIND 'allprop' request we expect all
> > > > > properties which are
> > > > > defined on the resource, especially  the 
> Lockproperties and all custom
> > > > > properties defined by GoLive 5 (which we have 
> PROPPATCHED before).
> > > > >
> > > > > We are probably able to replace PROPFIND 'allprop' requests by
> > > > > PROPFIND 'prop'
> > > > > requests in future GoLive releases (which would be indeed much
> > > > > more efficient)
> > > > > but the suggested change in the protocol would be a  
> *disaster*
> > > > > for GoLive 5
> > > > > which is already in the box.
> > > > >
> > > > > Best, Hartmut
> > > > >
> > > > >
> > > > > Lisa Dusseault wrote:
> > > > >
> > > > > > Past discussions have shown that servers frequently 
> have trouble
> > > > > > implementing PROPFIND 'allprop'.  Jim asked me to 
> summarize the past
> > > > > > discussion & list the open issues so that we can get this
> > > > > fixed, if it can
> > > > > > be fixed, in revisions to 2518.
> > > > > >
> > > > > > There are already cases where not all properties 
> will be returned:
> > > > > > RFC2518: "In the case of allprop and propname, if a 
> principal
> > > > > does not have
> > > > > > the
> > > > > >    right to know whether a particular property exists then
> > > the property
> > > > > >    should be silently excluded from the response."
> > > > > >
> > > > > > John Stracke's proposal for reducing/specifying the scope of
> > > > > 'allprop', and
> > > > > > discussion of the motivation:
> > > > > >  -
> > > > > 
> http://lists.w3.org/Archives/Public/w3c-dist-auth/1999JulSep/0092.html
> > > > > >  -
> > > > > 
> http://lists.w3.org/Archives/Public/w3c-dist-auth/1999JulSep/0310.html
> > > > > >
> > > > > > It has been a point of discussion for Advanced Collections:
> > > > > >  -
> > > > > 
> http://lists.w3.org/Archives/Public/w3c-dist-auth/1999JanMar/0008.html
> > > > > > "Clients need to know whether the property is computed on
> > > the fly before
> > > > > > requesting it.  There is no way to find out.  The impact of
> > > > > computing on the
> > > > > > fly is especially significant when a client 
> requests allprop.
> > > > > There may be
> > > > > > other properties that are computed on the fly as well.
> > > DAV:getetag is
> > > > > > computed, and some versioning history properties may also
> > > be computed."
> > > > > >
> > > > > > Also in Versioning:
> > > > > >  -
> > > > 
> http://lists.w3.org/Archives/Public/w3c-dist-auth/1999JanMar/0075.html
> > > > > "There has also been a massive growth in the number 
> of available DAV
> > > > > properties.  PROPFIND allprop operations may lead to 
> very large
> > > > > responses even with Depth: 1, which would slow down 
> performance
> > > > > for users due to network speeds.  It might be 
> worthwhile to add this
> > > > > facet to the open issue ALLPROP_AND_COMPUTED."
> > > > >
> > > > > Also in ACLs, Babich argues that clients who request 
> 'allprop' don't
> > > > really
> > > > > want to see the ACL property, thus they ought to specifically
> > > ask for it.
> > > > >  -
> > 
> http://lists.w3.org/Archives/Public/w3c-dist-auth/1998JulSep/0101.html
> > > >
> > > > Several server implementors have voiced the opinion 
> that 'allprop'
> > should
> > > be
> > > > "put out of its misery" (GMC) or at least weakened.  
> Often this is
> > because
> > > > of standard or custom properties that must be 
> calculated by the server
> > > (e.g.
> > > > 'lockdiscovery'), and the calculation can become extraordinarily
> > expensive
> > > > with an 'allprop' of depth: infinity.
> > > >
> > > > The only server-side argument for keeping 'allprop' is that
> > > server-to-server
> > > > COPY requires it; but if anybody has implemented this 
> yet and can't use
> > > > 'propname' instead, please speak up.
> > > >
> > > > Summary:
> > > > There thus seems to be a consensus among server 
> implementors and those
> > > > designing new features for DAV.  What's missing in 
> order to resolve this
> > > > issue for fixing RFC2518 is input from clients.
> > > >
> > > > 1.  Is anybody aware of clients that rely on particular 
> properties being
> > > > returned in 'allprop'?  If the properties relied upon 
> include any more
> > > than
> > > > the set DAV:{creationdate, displayname, getcontentlanguage,
> > > > getcontentlength, getcontenttype, getetag, getlastmodified} (the
> > > properties
> > > > required for DAV level 1 support) I would be very 
> surprised.  Thus,
> > > servers
> > > > may be able to restrict the required property set to this set.
> > > >
> > > > 2.  Is anybody aware of clients that rely on 'allprop', 
> rather than
> > > > 'propname', for property discovery?  This would be a 
> more serious issue
> > if
> > > > major client implementations actually rely on doing 
> property discovery
> > > using
> > > > 'allprop', and attempt this against various 
> implementations of WebDAV
> > > > servers.
> > > >
> > > > These seem to be our options for modifying RFC2518 
> (remember, it has to
> > be
> > > a
> > > > simple mod):
> > > >  - deprecate 'allprop' and tell clients not to use it, 
> but to use
> > > 'propname'
> > > > instead
> > > >  - define 'allprop' to be the set of properties 
> required for DAV level 1
> > > > support (although servers could freely return more properties if
> > desired)
> > > >  - explicitly allow servers to return an error code (507?) for
> > properties
> > > > that were too expensive to calculate for a 'allprop' 
> request, but still
> > > > allowing the client to do property discovery through 'allprop'
> > > >
> > > > Please voice your preferences among these options, 
> objections, or other
> > > > ideas.
> > > >
> > > > thanks,
> > > > Lisa
> > > > Xythos
>

---

From: Hartmut Warncke <[email-masked]>
Date: Wed, 22 Nov 2000 17:58:26 +0100

> memory" or something similar.

Yes, of course, I completely understand your point of view. But as a matter of
fact we like this "depth infinity" feature very much because we can use it in a
very efficient way in some situations and therefore we are struggleing so hard
for keeping that feature.

> thing. Backwards compatibilty should be maintained.

Good to hear that. But I am not quite sure how you will do that because if a
server has the possibility to refuse a depth infinity request GoLive 5 will not
be able to handle that situation (synchronization of client and server content
will not work anymore).

> infinite depth request, it could issue multiple requests with a Depth:1.

If you have to synchronize a very large site with thousands of files, replacing
depth infinity requests by depth 1 requests would be a huge performance disaster
for us.

Best, Hartmut

---

From: Greg Stein <[email-masked]>
Date: Wed, 22 Nov 2000 13:37:21 -0800

On Wed, Nov 22, 2000 at 05:58:26PM +0100, Hartmut Warncke wrote:
>...

Given that the default configuration of mod_dav prevents a Depth:infinity
PROPFIND, I'd guess that a rejection of Depth:infinity is a definite
possibility. What does GoLive do when it gets the rejection? Do you simply
skip that feature, send a bunch of Depth:1 requests, or something else?

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---

From: Hartmut Warncke <[email-masked]>
Date: Thu, 23 Nov 2000 13:06:16 +0100

Greg,

> skip that feature, send a bunch of Depth:1 requests, or something else?

We do  *not*  send a bunch of Depth:1 requests, some WebDAV functionality is simply
*not*  working in GoLive 5 when the server is not willing to respond to depth infinity
requests. We have simply tried to implement the WebDAV protocol as it is described in
RFC2518 and RFC2518 does  *not*  allow a server to deny a depth infinity access. At
the point in time when the "DavDepthInfinity" flag was introduced into mod_dav it was
much too late for us to implement such a "bunch of Depth:1 requests"-solution because
we were right before GM.

And I think this is a general problem. There are a lot of very huge products of huge
companies out there which support WebDAV and I am very lucky about that especially
because I think that the idea of WebDAV is very powerful and advanced. But we all have
to keep in mind that a huge product has a well defined life cycle and workflow which
cannot be changed frequently. And of course a product cannot be changed when it is
already in the box. So when some changes take place in the protocol or its important
implementations like mod_dav and this will effect negatively or even break products
which are already in the box or right before GM, a customer who have paid for the
product has to wait at least for the next product release (which will take months or
even years).

So I think it's very important to get some degree of stability and continuity in the
WebDAV protocol and its implementations which does not mean to let the protocal
unchanged during the next years but we should keep its key features. Otherwise a lot
of companies will think twice before they implement WebDAV support and  - without any
doubt -  every protocol needs a widespread acceptance from commercial companies and
products  to be succesful.

Best, Hartmut

---

From: Greg Stein <[email-masked]>
Date: Thu, 23 Nov 2000 04:28:00 -0800

On Thu, Nov 23, 2000 at 01:06:16PM +0100, Hartmut Warncke wrote:
> requests.

Eek. You mentioned something about a "sync" feature which uses the
Depth:infinity request. Is that it, or are there other times when you use
that request type? Is the sync feature used often?

Basically, I'm curious about the user impact here. It would be helpful for
me if I understood the end-user impact a bit better.

[ btw, I just sent mail to Stephanie to see if I can get a copy of GoLive 5
  so that I can do some of this interop testing ]

> We have simply tried to implement the WebDAV protocol as it is described in

Yes... you beat me up on that one already :-(

> product has to wait at least for the next product release (which will take months or

I've worked on products with release cycles on the order of two years.
Believe me... I understand the issue, and sympathize greatly.

On the other hand, I wasn't interested in releasing mod_dav with a known,
simple, DoS attack (in the default configuration). Admins can always turn
the feature back on, but I'd prefer to see them do it with the understanding
of what it means.

> products  to be succesful.

Agreed.

Cheers,
-g

---

From: Hartmut Warncke <[email-masked]>
Date: Thu, 23 Nov 2000 15:13:27 +0100

Greg,

> that request type? Is the sync feature used often?

Yes, it is mainly the sync feature which uses this request but it's a very important and
often used feature. With synchronizing a file "a.html" I mean to perform a PROPFIND request
on "a.html" in order to figure out if it has been changed after the last time you have
downloaded that file to your local machine. Synchronizing a huge folder means to check this
out for the folder and all its content. So if the user wants to synchronize a folder with a
lot of resources in it it is very efficient to do that with one PROPFIND-infinity request.

> Believe me... I understand the issue, and sympathize greatly.

Yes, I know and believe that!

> the feature back on, but I'd prefer to see them do it with the understanding

Yes, I know and understand your point of view but, as I already pointed out,  we like those
requests because they are pretty efficient for us and we are very sensitive to changes of
the protocol or its most important implementations.

Best, Hartmut