Subject: Ftp-WG: thoughts on ftp extensions and caching

From: Dave Cridland <[email-masked]>
Date: Mon, 5 Jun 2000 07:17:38 -0500

Sounds better, however, I'd prefer it to be Size that's mandatory, and
Binary-Size optional based on whether the client has turned it on or not.
After all, I have my doubts that any proxy-caching system really needs to
know what the exact size is in order to make a policy decision on whether to
cache the entire file, continue the download after the proxy's client has
quit, etc.

-----Original Message-----
From: Stephen Head [mailto:[email-masked]]
Sent: Saturday, June 03, 2000 12:37 AM
To: FTPEXT Working Group
Subject: Ftp-WG: thoughts on ftp extensions and caching

How about:

The control channel responses for RETR are a 125 response
(non STREAM mode) and a 150 response (STREAM mode).

An extended 125 response consists of:

        "125-Ext" CRLF
         1*( SP entry CRLF )
         "125 End" CRLF

An extended 150 response consists of:

        "150-Ext" CRLF
         1*( SP entry CRLF )
         "150 End" CRLF

where:

        entry            = [ facts ] SP pathname
        facts            = 1*( fact ";" )
        fact             = factname "=" value
        factname         = "Binary-Size" / "Size" / "Modify" / "Create"
/
                           "Type" / "Unique" / "Perm" /
                           "Lang" / "Media-Type" / "CharSet" /
                           os-depend-fact / local-fact
        os-depend-fact   = <IANA assigned OS name> "." token
        local-fact       = "X." token
        value            = *RCHAR

and where pathname is the fully qualified pathname of the object*
and facts MUST include the following factname-value pairs**
if FEAT returns "X150" or "X125":

        Binary-Size
        Modify

Note that a caching proxy server has the option of fetching
the binary version of the file, even if it is ASCII that is being
requested by the client.  This makes reporting the Size
(in contrast to Binary-Size, which would be the binary size
of the file only) appear to be somewhat less critical.

-s

* permits managing file object cache using fully qualified pathname,
presumed to be unique (even in presence of symbolic links on
the server).

** presumeably the states of the other stuff is explicitly managed
by the client and so not needed.  I haven't exhaustively checked
that this is the case (eg for 3-party transfers, etc.).

-----

Dave Cridland wrote:

>
>
> Before I commence wittering, a caveat: It's before noon here, and I'm
> hungover, and can't remember everything in the MLST spec, and
> unfortunately I'm at work and somewhat busy.
>
> However, I suggest the following:
>
> In responses to RETR commands, the Server-PI SHOULD respond with the
> following as the text of the response:
>
> response = identifier SP mlst-line
>
> identifier = "<EXT>" (or some such - more than a character should
> avoid confusion)
>
> And also include the extra facts, not normally put in:
>
> "download-mode", "download-type", "download-structure" - defining the
> method of downloading. Assuming I've remembered these right.
>
> The "size" fact, if present, should indicate to a reasonable degree
> the size of the file being downloaded - perhaps not wholly accurately
> for ASCII mode downloads, but enough for policy based caching systems
> to take a view on whether to cache or not.
>
> Perhaps include an optional "download-size" fact for the exact size.

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 08:07:37 -0500

I am not quite sure how you are defining Binary size. But I suspect 
from context the author is uner the delusion that binary-size ==
size when transfered in image mode == number of bytes required oin disk to store. 
This is -- of course -- false in the general case.  There are a number of 
operating systems where this is not true though its usually the case 
for UNIX and MS-DOS based systems with vanillia storage subsystems. 

I believe an approximate size is the best thing to send and if someone really
cares thenthey will want the bounds (minimum, maximum) of the bytestream
which are quick to compute, albeit inaccurate.

---

From: Stephen Head <[email-masked]>
Date: Mon, 5 Jun 2000 10:35:39 -0500

I specified binary size with three primary motivations in mind:

1, it is (reputedly) easier for the server to compute
  than the other leading brand (size in ASCII type);

2, the proxy cache server can always store the image
  version of the file, and serve either the image or ASCII
  version depending on demand; and

3, the proxy cache server should not or at least might not
  necessarily use a UN*X file system to store file objects,
  so filesystem fragmentation concerns (etc.) are or at least
  appear as if they should be of less importance in this context.

-s

-----

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 11:19:26 -0500

It still seems that an approximate size should serve just fine. 
No?

---

From: Stephen Head <[email-masked]>
Date: Mon, 5 Jun 2000 11:33:24 -0500

Binary-size (Image-size would probably be an improvement on name)
also appears to be useful to squid  (see squid-2.2.STABLE4/src/ftp.c,
etc.) to generate Content-Length and (for restart) Content-Range
HTTP headers for ftp-related requests.  Apache also uses it
for similar purposes (see apache_1.3.6/src/modules//proxy_ftp.c)
--though just from skimming source code it appears not to care
whether ASCII or Image mode and seems very if not overly trusting of
the ftp server's response to SIZE.
Harvest (docs/old-manual/node130.html#SECTION00089700000000000000)
uses object size as a parameter in its cache decision flowchart,
but in any case I didn't want to preclude it if it could be obtained
cheaply.
(This could be added to the motivations list I previously sent out, as
#4.)

A very early form of my "thoughts" used plain old Size but I was
told it could potentially cause an undue amount of overhead, eg
in the case of ASCII type.  So I modified it before I sent out the group

version.  Either way works for me, but I am trying to make it as easy
as possible for a server to be conforming in order to attempt
to guarantee the maximum possible compliance (eg become the default
and if failing that become a near-universally-supported option).

-s

-----

(Dave Cridland wrote:

> Binary-Size optional based on whether the client has turned it on or
> not. After all, I have my doubts that any proxy-caching system really
> needs to know what the exact size is in order to make a policy
> decision on whether to cache the entire file, continue the download
> after the proxy's client has quit, etc.
>

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 11:53:36 -0500

If you want it to be easy then it shoudl be speced out as an rough-size sort of
option.

---

From: Stephen Head <[email-masked]>
Date: Mon, 5 Jun 2000 12:13:06 -0500

> If you want it to be easy then it shoudl be speced out as an rough-size sort of

Before doing that, I would think it should be considered the relative ease
of supplying the various alternative responses.  In particular, what is
difficult about supplying the image type size?  Also, the tendency
(see RFCs 1945, 2616 re Content-Length) is to specify exact # of
OCTETs.  Also, what should or can Squid and Apache do with
approximations in contrast to exact size when fetching ftp files?

-s

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 12:46:33 -0500

If your operating systmes does not track end bybtes in an easilly accessible way an image size may be rounded to the enxt nearest container unit.
Example VMS files will yield a container object size in round disk blocks. as the natural size. 
The bytestream is considerably smaller. The image mode stream is somewhat smaller since it can see more internal structure  plus somewhat larger since it adds overhead blocks.   the later I believ shoudl be estiamted in at a maximum value by the size code but as long as the result need only be an approximate and not the exact number
of bytes sent over the wire its ffine.

THe protocols where i see content lengths the content is already being redndered and processed, here we are talking aboutlooking at a raw disk object or virtual object which may have no exact size.

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 13:18:43 -0500

I have reviewed the size fact and its definiton is, as I recalled, acceptable
in this context.   what specific NEED exists for a more accurate number 
lets see if its worth a potentially large delay (equal to trasmission time over a very fast medium)?

---

From: Stephen Head <[email-masked]>
Date: Mon, 5 Jun 2000 13:53:46 -0500

Apache and Squid convert ftp SIZE response information into
HTTP Content-Length headers.  The following is from RFC 1945
(HTTP/1.0):

      Note: Some older servers supply an invalid Content-Length when
      sending a document that contains server-side includes dynamically
      inserted into the data stream. It must be emphasized that this
      will not be tolerated by future versions of HTTP. Unless the
      client knows that it is receiving a response from a compliant
      server, it should not depend on the Content-Length value being
      correct.

The following is from RFC 2616 (HTTP/1.1):

   3.If a Content-Length header field (section 14.13) is present, its
     decimal value in OCTETs represents both the entity-length and the
     transfer-length. The Content-Length header field MUST NOT be sent
     if these two lengths are different (i.e., if a Transfer-Encoding
     header field is present). If a message is received with both a
     Transfer-Encoding header field and a Content-Length header field,
     the latter MUST be ignored.

and:

   When a Content-Length is given in a message where a message-body is
   allowed, its field value MUST exactly match the number of OCTETs in
   the message-body. HTTP/1.1 user agents MUST notify the user when an
   invalid length is received and detected.

So it would appear that there are some situations in which http server
and proxy servers could return incorrect results as defined by RFC 2616
if they use some ftp SIZE variant or 150 response variant that returns
an approximation and not an exact size.  When faced with approximations,
I also tend to ask what the worst case is, and 100% does not sound
very appealing to me, even if it is only worst case (DOS attacks?).

I'm looking into the concern about VMS filesystem not returning a true size
for an image file size but my reflex response
is that ftp servers are better and I want to believe more often running
UN*X and variants.  A question here: for that subset of servers that
for some reason cannot respond with the exact size, how about
indicating that via a fact such as Approx-size and/or Approx-image-size
in the response so the downstream components are appropriately alerted?
(More thoughts here -- excised for the moment.)

It appears to me that I have somehow wandered into a sensitive area.
FWIW, it was not my intent to do so-- maybe I should have sent out
a couple of MMFs first instead :-).

-s

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 15:10:25 -0500

The current definiton for the Size fact is approximate.  If want ones to define
an EXACT -size then and only then can servers that require an exact number rely
on it.

---

From: Stephen Head <[email-masked]>
Date: Mon, 5 Jun 2000 15:40:41 -0500

I was trying to answer the previous question ("what specific NEED exists...").
The illustration I provided was that there exist current servers that rely
on the non-standard SIZE command results to form Content-Length
and Content-Range HTTP headers for which the trend, standards, and general
expectation is that they report the exact number of OCTETS.  It seems to me
that this is potentially a valid area that the ftp-wg can contribute an extended
standard to help satisfy that need.  Conceivably, it's the way HTTP permits
FTP urls to be fetched and/or the server writers that created this particular
problem.

Obviously, a server whose goal is interoperability can't use a networking feature
before it has been standardized.  That is always the case with every
new emerging standard/extension.  Identify need, then identify options,
then discuss and choose best option.

My opinion is that exact size is more generally useful than approximate size
and in most cases is just as easy to obtain from a reasonable OS.

If that view is not generally shared, then no biggie, then how about
Exact-image-size and an Exact-size facts as companions to existing
Size fact instead of redefining Size and adding Approximate-size variant(s).
I'm not hung up on naming (if that is the main issue) (?)...

-s

-----

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 15:50:10 -0500

Aside from the prejudicial definiton of reasonable OS. (Ease is in fact an attribute of the primitiiviness of the data model and the file system.) its a 
fine idea. but such facts shoudl come with cautionary ntoes that they may put a very serious load on the operating systemn and thus should not be asked 
for causally or when avoidable.

---

From: Stephen Head <[email-masked]>
Date: Mon, 5 Jun 2000 16:40:21 -0500

But it does not put a serious load on the OS if the information is pre-generated
(eg via a sufficiently smart server and/or attendant utility which "preps"
a given ftp server NVS via a parallel directory tree that contains files which
contain the exact file size in the ftp server readable format).

Doing it part of the time instead of most if not all of the time generally won't
result in avoiding the load altogether, but just offloads
the responsibility towards the leaf cache nodes and destination client
in addition to loading the net with more request-responses.
The point of commonality being the origin ftp server, that is the logical
optimal place where the computation should take place IMHO (and also have
it done once and for all for a given version of the file object).

BTW http servers tend to require very fast initial (i.e. HTTP/1.x header,
ideally incorporating a valid Content-Length/Content-Range header line)
response, the more reason to have this kind of info readily available
the first time and every time (I had nothing to do with the design of HTTP,
just looking at and reacting to the situation as it now stands).
I would not want to advance anything that doesn't ultimately make good
sense so feel free to check and be skeptical.

For dividing and conquering's sake, I can amend the notion to
use Exact-Image-Size and Exact-Size for names of exact image size
and exact size facts, respectively.  As a separate train of thought
I would regard as friendly the alternative of renaming Size to
Approx-Size (and variations).

-s

-----

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 5 Jun 2000 22:38:39 -0500

Nope it would not. Seems to be letting the HTTP tail wag the host file
system dog though.