Subject: Ftp-WG: thoughts on ftp extensions and caching

From: Dave Cridland <[email-masked]>
Date: Tue, 6 Jun 2000 06:35:28 -0500

I'm unconvinced that it's needed, though, in any case.

Yes, if a HTTP->FTP proxy system really wants to use Content-Length headers,
that's fine, but given that we're mainly interested in HTTP/1.1, I can't see
the issue here - Chunked Encoding should work fine for giving the size of
the data transfer, simply by buffering a handful of k. The startup time
would then be as it is now, and the whole thing fits into both standards as
of now. Obviously in the case of HTTP/1.0, there's no need to give any
indication of the download length, so there's no need to attempt to support
it.

Putting in a MLST line in the response for a file being downloaded makes
sense to me, simply because it can be done with ease - and gives cues on
whether the proxy should consider this file for caching. This is a
performance advantage for the proxy, and the way MLST is designed, a minumum
hit on the server.

If Squid, Apache and the others are actually issuing SIZE commands and
Content-Length headers, then it's my thinking that they're doing things the
wrong way in any case.

-----Original Message-----
From: Stephen Tihor [mailto:[email-masked]]
Sent: Tuesday, June 06, 2000 4:39 AM
To: FTPEXT Working Group
Subject: Ftp-WG: thoughts on ftp extensions and caching

Nope it would not. Seems to be letting the HTTP tail wag the host file
system dog though.

---

From: Stephen Head <[email-masked]>
Date: Tue, 6 Jun 2000 11:57:56 -0500

Dave Cridland wrote:

> Yes, if a HTTP->FTP proxy system really wants to use Content-Length
> headers, that's fine, but given that we're mainly interested in
> HTTP/1.1, I can't see the issue here - Chunked Encoding should work
> fine for giving the size of the data transfer, simply by buffering a
> handful of k. The startup time would then be as it is now, and the
> whole thing fits into both standards as of now. Obviously in the case
> of HTTP/1.0, there's no need to give any indication of the download
> length, so there's no need to attempt to support it.

Although again the use of chunked-encoding can result in added
processing burden
in proxies.  Also any benefit from the potential use of Content-Length
along the way
to the destination client is completely foiled, and avoiding it seems to
imply it
has relatively little value to begin with, etc....

> Putting in a MLST line in the response for a file being downloaded
> makes sense to me, simply because it can be done with ease - and gives
> cues on whether the proxy should consider this file for caching. This
> is a performance advantage for the proxy, and the way MLST is
> designed, a minumum hit on the server.
>
> Content-Length headers, then it's my thinking that they're doing
> things the wrong way in any case.

Perhaps but it might conceivably also be interpreted as acts of
desperation on the part of the authors (?)
in lieu of a reliable standard to begin with.

> Nope it would not. Seems to be letting the HTTP tail wag the host file

Maybe though caching seems to be attracting considerable interest these
days.

RFC 959 contains the following which also may be relevant to
discussions:

         A comment on transfer modes.  The stream transfer mode is
         inherently unreliable, since one can not determine if the
         connection closed prematurely or not.  The other transfer modes

         (Block, Compressed) do not close the connection to indicate the

         end of file.  They have enough FTP encoding that the data
         connection can be parsed to determine the end of the file.

It seems as if it would be nice to be able to clear this up one way or
another.

-s

---

From: Stephen Tihor <[email-masked]>
Date: Tue, 6 Jun 2000 12:24:56 -0500

For predictive cacheing approximate sizes or (min,max) pairs are pretty good.
With a defined indefinite value/return type

---

From: Stephen Head <[email-masked]>
Date: Tue, 6 Jun 2000 13:20:55 -0500

> With a defined indefinite value/return type

This gets into the thoughts I excised from an earlier message.  As I understand
it the current draft-10 does not give mininum or maximum indications.
If it did, it would be better than a simple "approximation" which could go
either way.

I didn't really want to complicate the notion too much at least without cause
since as I've addressed before I am of the opinion exact sizes
is worth the effort and can be pre-computed for given files.

I would personally view something along the lines of Approx-min-size
and Approx-max-size as a last resort given my current awareness of
the issue(s).

-s

---

From: Stephen Tihor <[email-masked]>
Date: Tue, 6 Jun 2000 13:45:47 -0500

I belive we cna cheaply provide a guarenteed minimum or maximum size for almost
ervery file and aboth for only slightly fewer files, very efficiency. its the
exact number in between that is hard.  Of course a facotr of 2x or 4x may be
involved.

---

From: Stephen Head <[email-masked]>
Date: Tue, 6 Jun 2000 14:28:57 -0500

> exact number in between that is hard.  Of course a facotr of 2x or 4x may be

Has a survey been done or attempted as to what percentage of ftp servers
now in service use a OS in which giving the exact size would require undue
computation?
Of those, what percentage of files actually exhibit the problem?
Will a FTP put followed by a get result in exhibiting the problem?
Or is it due to some other factor(s), such as deleted records embedded in the
file?
(Apologies if this has already been discussed in detail in the wg already;
if not, some estimates might be of help in getting consensus.)

I have difficulty agreeing with the opinion that approximate sizes or min, max
pairs
are "pretty good" for caching servers.  To me it seems it is not yet demonstrated
to the point where there are compelling circumstances in which no alternatives
are available.  The algorithm implied for caching decision would seem to be to
replace
a simple conditional with something along the lines of:

  if Approx-max-size fact returned
    then
       if Approx-max-size-value > (Configurable-fudge-factor *
Configurable-highwatermark-size)
          then
             bail out
          else
             /* attempt to cache */
             begin caching

and later, elsewhere:

  if Actual-running-accumulation-size > (Configurable-fudge-factor *
Configurable-highwatermark-size)
    then
        terminate caching
        delete cache-entry

while the destination client is subjected to a potential quality of service
fluctuation
in overall response time, the complexity of the caching server is increased
substantially
(affecting reliability and time to market) and testing cases must be expanded
considerably
to incorporate the new and more complex code paths and new boundary conditions.
This is especially true if the Configurable-fudge-factor is underestimated
relative
to the actual load on a live network.  (What will the recommended default(s) be?)
Add a corresponding amount of complexity for anyone wishing to use Approx-min-size

in their caching policies.  Can we envision that there would be demand such that
benchmarks would ever be expanded to incorporate such test cases, at least in
the reasonably forseeable future?

Also approximations do not appear to help the purported inherent unreliability
of stream transfer mode mentioned in RFC959, if that is viewed as an issue...

-s

---

From: Stephen Tihor <[email-masked]>
Date: Tue, 6 Jun 2000 14:35:43 -0500

> now in service use a OS in which giving the exact size would require undue

This is not the way to create good internet standards. This is how you get
Microsft incompatibleware and vendor locking.

> Of those, what percentage of files actually exhibit the problem?
Of the ones I am aware, most.
> Will a FTP put followed by a get result in exhibiting the problem?
Unless the FTP server specifically maaintained a cache of size information. 
I am not aware of any doing this though one can imagine a variety of such
strategies - several of which IO object to sinc they are incompatible with good
security practices on ystems otehr than dedicated FTP server appliances. 

> if not, some estimates might be of help in getting consensus.)

Again, this sounds like a There are less than x% of these so we can make them
all work very very hard to pretend they are brand X which happens to work this
way.

---

From: Stephen Head <[email-masked]>
Date: Tue, 6 Jun 2000 17:26:33 -0500

> Microsft incompatibleware and vendor locking.

Sorry, didn't quite get this, or at least intend to imply that some formal
questionnaire be sent out.  I was trying to address current actual practice only
and (for example) rough estimates from knowledgeable administrator-types
as some kind of starting point to get an estimate of the measure of difficulty
of implementing whatever alternatives one is already considering
versus new alternatives.  It is an attempt to be fair under the presumtion
that the more readily available advance knowledge assembled for consideration,
the better...

> Of the ones I am aware, most.

What I mean is, is there a sense of which OSs and what percentage,
for Image/ASCII/whatever types, and for each OS which native file types
have the problem and under what circumstances?  Conceding that ASCII
is already a pain in the neck even for Un*x (so don't need to consider
that particular case), but recalling that Exact-image-size appears to be
a fully useable substitute for caching purposes as I mentioned earlier
 Any ventures as to breakdown guesstimates?  And is there a cutoff
that the wg is willing to accept, for example if only XXXOS YYY files have
a problem and if only 0.1% of ftp servers use XXXOS and only 0.5% of
such servers have YYY files with the problem, is this deemed sufficient
to withhold adoption of some feature that would require pre-computation
on XXXOS wrt YYY files?  (what are the specific names and values
to substitute here)

> security practices on ystems otehr than dedicated FTP server appliances.

[So then staging a transfer through a FTP server could potentially result
in loss of useful exact-size variety header info under both current and your
preferred future standards.  In general, is this a good thing?]

[Not sure what the reference to security concerns was about-- explain if you like.

I was positing the hypothetical example of a relatively simple A->B, B->C
transfer sequence.  And trying to determine how image files are typically
generated
(wrt choice of appropriate native OS file type) on the server to begin with for
OSs with these issues.]

> all work very very hard to pretend they are brand X which happens to work this

No, it's not an attempt by me to discriminate against anyone's favorite OS,
and apologies if that appeared to be the case.  I thought there was a prima facie
case for the general network-related usefulness of the information independent
of the OS and proceeded on that presumption.

I was just looking for specific cases.  It's just a bit hard to attempt to
understand
and/or try to offer solutions concerning a difficult case without a definitive
description
of the difficulty to begin with.  Sorry if I am baying at the wrong moon here;
again if the point has already been discussed thoroughly and conceded for
image files then maybe it's not worth revisiting at this time.

-s