Subject: WebDAV Bindings - Issue Yaron.AtomicDelete

From: Yaron Goland <[email-masked]>
Date: Thu, 27 Jan 2000 12:23:10 -0800

The following is a summary of my conversations with Daniel Lovinger,
technical lead for FAT in Windows NT and Brian Andrew, technical lead for
NTFS. Any technical mistakes below are due to my own failure to properly
understand what Dan and Brian told me.

I explained to Brian and Dan that I needed to implement WebDAV over existing
NTFS and FAT file systems and that it was critical that our WebDAV
implementation allow for multiple simultaneous protocol access through
NFS/Win32/SMB/FTP/WebDAV/etc. I asked them, given this requirement, if it
would be possible to simulate an atomic delete by just moving the tree we
wanted to delete and then begin the file-by-file delete.

Both Brian and Dan agreed that the move operation is atomic and that this
would allow one to make the entire sub-tree disappear from a particular
point in the namespace in a single atomic act. However both Brian and Dan
pointed out that the problems really begin once you have moved the tree. The
both pointed out that if I insist on allowing multiple access to the files
in question through WebDAV as well as FTP/SMB/NFS/Win32/etc. then it would
be impossible to perform an atomic delete.

The crux of the problem is that NFS, amongst others, does not necessarily
refer to files using their path location. They often refer to files using
File IDs which are completely independent of location. This means that the
attempt to move the files from one part of the namespace to another in order
to hide them won't work with NFS. NFS and similar systems will continue to
be able to operate on the files, regardless of where they are moved to.

This means that even if the move on the sub-directory succeeds it is
possible for a NFS client to come in and open (without the filesharedelete
flag) or change the ACL on a file and hence prevent us from deleting the
file. Even if we attempt to change the ACL on a file to prevent others from
accessing the file another user with a File ID and a higher access can
always override us. 

Note the assumption that we are running the WebDAV thread using the
authentication of the requesting user. This is standard practice in NT as we
have found that running threads at the administrative level is a security
and performance threat. However, even if we were willing to run the WebDAV
thread at administrative level in a shared file system it is possible for
other administrators to override the local administrator thus causing the
previous problems.

The result is that the atomic move provided by the file system doesn't buy
us much because it doesn't "hide" the files from everyone. This means that
there is no way for us to be sure, once we begin actually deleting files,
that we will successfully be able to delete all of this files. This then
means that we can end up with a half deleted tree. A tree we can not
subsequently move back after the failures because to do so would violate the
atomic guarantee of an all-or-nothing delete.

So the bottom line is that we can't implement an atomic delete and allow for
multiple simultaneous file system access through multiple protocols. Since
our users absolutely demand multiple simultaneous file system access through
multiple protocols we can not implement atomic delete.

			Yaron

---

From: Jim Whitehead <[email-masked]>
Date: Thu, 27 Jan 2000 16:02:29 -0800

Yaron,

An excellent, meaty, post.

It leaves me wondering, however, whether the inability to perform atomic
deletes is incompatible with the new bind delete language suggested by Judy
in her post (which doesn't use the word "atomic" at all):

http://lists.w3.org/Archives/Public/w3c-dist-auth/2000JanMar/0182.html

Or, put another way, how would Microsoft's likely implementation of bindings
interact with DELETE, and does that run into the atomic deletion semantics
you mention?

- Jim

Yaron Goland writes:
> The following is a summary of my conversations with
> Daniel Lovinger, technical lead for FAT in Windows NT
> and Brian Andrew, technical lead for NTFS. Any technical
> mistakes below are due to my own failure to properly
> ... snip ...

---

From: Greg Stein <[email-masked]>
Date: Thu, 27 Jan 2000 17:57:55 -0800 (PST)

On Thu, 27 Jan 2000, Yaron Goland wrote:
>...

Although I do not agree with atomic delete, I do need to point out
something here (to be fair). You have introduced an extra requirement --
simultaneous access through different protocols. From a WebDAV standpoint,
NTFS and FAT can implement an atomic delete. If you partially delete a
tree, there is nothing requiring you to move it back: from the client's
standpoint, you *have* deleted the resources.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---

From: "Geoffrey M. Clemm" <[email-masked]>
Date: Fri, 28 Jan 2000 10:09:59 -0500

Like Jim, I'd like to thank Yaron allocating out some non-existent
free time for pursuing this issue!  A few comments:

   From: Yaron Goland <[email-masked]>

   I explained to Brian and Dan that I needed to implement WebDAV over existing
   NTFS and FAT file systems and that it was critical that our WebDAV
   implementation allow for multiple simultaneous protocol access through
   NFS/Win32/SMB/FTP/WebDAV/etc.

I agree.  Note that this of course does not mean that a WebDAV
DELETE is anything like the operations called "delete" in any of
these other protocols.

Furthermore, I believe it is reasonable for a particular
protocol to provide access to only a subset of the repository.
In particular, it is reasonable to say that you can't point
WebDAV at the root of a file system, but only one level down.
This leaves a place to put "hidden" administrative or implementation
data (such as collections that have been DELETE'd but not yet
garbage collected).

   I asked them, given this requirement, if it
   would be possible to simulate an atomic delete by just moving the tree we
   wanted to delete and then begin the file-by-file delete.

   Both Brian and Dan agreed that the move operation is atomic and that this
   would allow one to make the entire sub-tree disappear from a particular
   point in the namespace in a single atomic act.

And as long as the place it was moved to was invisible to WebDAV access,
the "atomic" definition of DELETE is satisfied.

   However both Brian and Dan

As long as these files are not visible to WebDAV clients, it doesn't matter
if other protocol operations continue to work on them.

   file.

That's OK, since the file is in the "invisible" location on the file
volume, it doesn't matter to WebDAV that some other protocol still has
access to those bits.  The file may also still exist in some backup
archive form on that host, and we don't care about that either.

   Even if we attempt to change the ACL on a file to prevent others from
   accessing the file another user with a File ID and a higher access can
   always override us. 

That's OK, as long as to a WebDAV client, it looked like the atomic delete
happened.

   The result is that the atomic move provided by the file system doesn't buy
   us much because it doesn't "hide" the files from everyone.

As indicated above, we can't guarantee that a DELETE will hide the files
from everyone.  In the case of multiple bindings, we are explicitly saying
that a DELETE doesn't even hide the file from WebDAV clients (i.e. they
can still see that file from any of the remaining bindings).

   This means that
   there is no way for us to be sure, once we begin actually deleting files,
   that we will successfully be able to delete all of this files.

If DELETE means that you no longer see the specified resource at the
specified URL, then we can successfully delete all the resources.  If
DELETE means that the bits are permanently erased from all possible
forms of access, then it is impossible to implement (in the presence
of backups, etc.).

   This then

By the "not visible at this URL" definition, a MOVE successfully performs
the delete.  Garbage collecting the space after the DELETE is not something
the protocol should (or could) define.

I think the key questions here is:

Does delete mean "this resource is no longer visible at this URL" (as
proposed in the bindings protocol), or "the bits in this resource are
permanently removed from all possible access".

If we pick the former definition (which is the only one that is
compatible with the needs of multiple bindings and versioning), I
believe it is clear that atomic delete is feasible.

Cheers,
Geoff