Subject: RE: Interest in standardizing Batch methods?

From: "Jim Whitehead" <[email-masked]>
Date: Thu, 17 Jul 2003 10:22:09 -0700

Accidentally caught by the spam filter.

- Jim
-----Original Message-----
From: Winston Spencer [mailto:[email-masked]]
Sent: Wednesday, July 16, 2003 1:59 PM
To: [email-masked]
Subject: [Moderator Action] RE: Interest in standardizing Batch methods?

I can't see how anyone could  realize the important of a standardize batch
method.  At the end of the day, it saves money.

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 17 Jul 2003 19:42:51 +0200

Well,

but at the start of the day, somebody needs to write down a spec that we can
discuss. So far, nobody has done that.

Julian
--
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

  -----Original Message-----
  From: [email-masked] [mailto:[email-masked]]On
Behalf Of Jim Whitehead
  Sent: Thursday, July 17, 2003 7:22 PM
  To: WebDAV
  Subject: RE: Interest in standardizing Batch methods?

  - Jim

---

From: Chris Knight <[email-masked]>
Date: Thu, 17 Jul 2003 11:27:27 -0700

Julian Reschke wrote:

>Well,
> 
>but at the start of the day, somebody needs to write down a spec that we
>can discuss. So far, nobody has done that.
>  
>
And boy, the semantics of batch requests are pretty hairy. At the end of 
the day is there much benefit? Is each request in a batch atomic and/or 
is the entire batch? (Do you have multiple levels of transactions?) How 
do you cache a batch? (Boy, there's a long topic...)

The HTTP overhead for multiple requests (made over the same TCP 
connection, mind you) is small (arguably, couldn't be much smaller) and 
if you want atomic batches, locks do an ok job of it.

I've certainly considered something like a PUT/PROPPATCH combination 
(for example if you had "required dead properties") but it's probably 
just easier to do a null lock and the two requests separately.

---

From: Geoffrey M Clemm <[email-masked]>
Date: Thu, 17 Jul 2003 15:17:09 -0400

I agree with Chris.  The easy things aren't worth doing (you get virtually
all of the benefit by streaming your requests and not blocking waiting for
the response to each request), and the hard things are very complex and
any given approach is unlikely to be a good basis for interoperation.

Cheers,
Geoff

---

From: [email-masked]
Date: Thu, 17 Jul 2003 21:30:27 +0100

I am sure that you guys, who don't agree with standardising Batch 
method, have good reasons for that. But shall we be more clear about 
those reasons which might not quite clear for all. Of course I know 
some of those reasons by chasing the thread with the same subject in 
the WebDAV mailing list. But I think the WebDAV protocol is almost 
enough mature to interest many organisations and Web users. So that 
they would like to apply this protocol rather HTTP protocol, and use 
the WebDAV advantages such as metadata, access control,... to author 
Web resources. 

Now the question is: if the WebDAV is going to be a substitute for HTTP 
protocol, can it handle authoring all kind of Web resources as it 
claims, or still we need other conventions based on the HTTP protocol 
in order to support authoring Web resources except file systems. 

So if you believe the WebDAV as a protocol for authoring all Web 
resources (and not only the file systems), then how we can handle Web 
resources such as emails and databases. In order to implement some 
operations on those resources, you might use a sequence of WebDAV 
methods which need to be considered as one transaction. In those cases 
lock method cannot help. So what is your suggestion to handle those 
operations? 

I think a mechanism for supporting transactions is a strong point that 
WebDAV needs.

---

From: Geoffrey M Clemm <[email-masked]>
Date: Thu, 17 Jul 2003 17:02:15 -0400

If you are interested in defining a transactioning model, then
it would be best to make that explicit, because the term "batch"
is used to encompass many things.  To date, the WebDAV approach has
been to define specific transacted operations (e.g. PROPPATCH),
because that is something that can be implemented on a wide range
of repositories, while generic transactioning support is likely to
only be feasible on repositories that are actually databases.

But enough repositories really are databases that it probably is
worthwhile trying to define generic transactioning support for
WebDAV.

Cheers,
Geoff

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 17 Jul 2003 23:20:46 +0200

True, but...

...speaking from personal experience working on a document management
system, the temptation "give up" and just state "transactions are going to
solve this problem in general" is big. However, transactions can be very
expensive to implement, in particular in a HTTP server scenario. How many
concurrent transactions are you going to support? What are you going to do
if a client starts a transaction but doesn't finish it within a few minutes?

IMHO it makes a lot of sense to try to *avoid* them by

a) defining methods with the "right" level of transactionality (for
instance, see PROPPATCH or ORDERPATCH) or

b) try to do things using the existing locking and/or versioning machinery.

So at a minimum, I'd expect a working group activity on transactions to
start with a list of use cases. Then we can take a look at the use cases and
decide whether they require a generic transaction mechanism, or whether
something simpler can do that as well (and please don't say: I want to do
JDBC over HTTP).

Julian

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Geoffrey M Clemm
> Sent: Thursday, July 17, 2003 11:02 PM
> To: WebDAV
> Subject: Re: Interest in standardizing Batch methods?
>
>
>

---

From: Geoffrey M Clemm <[email-masked]>
Date: Thu, 17 Jul 2003 18:03:28 -0400

I agree.  Note that my statement "it probably is
worthwhile trying to define generic transactioning support for
WebDAV" explicitly did not suggest that such an attempt would
succeed (:-).  In particular, I believe it is likely that such
an attempt would fail for the reasons Julian describes.

Cheers,
Geoff

"Julian Reschke" <[email-masked]> wrote on 07/17/2003 05:20:46 PM:

> > > > the response to each request), and the hard things are very 
complex
> > > > and
> > > > > And boy, the semantics of batch requests are pretty hairy. At 
the
> > > > end of
> > > > > is the entire batch? (Do you have multiple levels of 
transactions?)
> > > > How
> > > > > connection, mind you) is small (arguably, couldn't be much 
smaller)
> > > > and

---

From: "Eric Sedlar" <[email-masked]>
Date: Thu, 17 Jul 2003 15:03:24 -0700

I think something along the lines of what NFSv4 supports with the COMPOUND
method would be in order.  The big benefit of doing a COMPOUND/BATCH method
is the atomicity.  Note that the transaction you want to do may operate on
multiple resources, and while LOCK will help, rolling back incomplete
changes is a big pain in the ass for the client to do.  Do you expect the
client to implement a log file (like a database) which will be updated on
each successful response, and then require that the client know how to apply
undo from the log file in case of an error?

Let's look at a use case:  I want to PROPPATCH resources /a1, /a2 ... /a10
with the same dead property called foo to some new value either as a unit or
not at all.  With today's WebDAV, I LOCK all of these resources and start
pipelining the PROPPATCH requests.  As the replies come back in, I log them
to a file on the client that is fsync'ed to the disk (so that I'm sure my
logging makes it to disk in case the client crashes at some point, the way
databases do).  It turns out that there was an error in the PROPPATCH on /a8
(ACL violation) and I want to rollback all of the preceding proppatches in
the client log.  Now the client starts to replay the log (since the last
transaction start), re-PROPPATCHing /a1 through /a7 hoping there is no other
error, and then cleans up the log entries (or marks them as rolled back).
Just in case this work sounds easy for client implementors to do, it's not.
Plus, this strategy would only work if I happened to know the old values of
the foo property so that I could undo them.  PROPPATCH doesn't tell me what
the old values were (you need the equivalent of the SQL statement "UPDATE
foo RETURNING foo INTO :1") to do this.  In general, undoing WebDAV
operations is not easy and not always possible (e.g. some versioning
operations).

Given that NFSv4 could handle COMPOUND, I don't think that this is too hard
or out of scope for WebDAV.  Also, I believe that none of the publically
announced NFSv4 implementations are built on databases, so I don't think
this is too hard for server implementors in general, but we could make BATCH
methods an optional feature to address this concern.

As to Julian's particular concern about transactions started but not
completed, the same solution should be applied as for any method in which
part of the request has arrived but the rest has not after a few
minutes--you timeout the entire request.

--Eric

----- Original Message -----
From: "Julian Reschke" <[email-masked]>
To: "Geoffrey M Clemm" <[email-masked]>; "WebDAV"
<[email-masked]>
Sent: Thursday, July 17, 2003 2:20 PM
Subject: RE: Interest in standardizing Batch methods?

transactions?)
smaller)