Subject: Ftp-WG: thoughts on ftp extensions and caching

From: Dave Cridland <[email-masked]>
Date: Wed, 31 May 2000 05:02:55 -0500

Before I commence wittering, a caveat: It's before noon here, and I'm
hungover, and can't remember everything in the MLST spec, and unfortunately
I'm at work and somewhat busy.

However, I suggest the following:

In responses to RETR commands, the Server-PI SHOULD respond with the
following as the text of the response:

response = identifier SP mlst-line

identifier = "<EXT>" (or some such - more than a character should avoid
confusion)

And also include the extra facts, not normally put in:

"download-mode", "download-type", "download-structure" - defining the method
of downloading. Assuming I've remembered these right.

The "size" fact, if present, should indicate to a reasonable degree the size
of the file being downloaded - perhaps not wholly accurately for ASCII mode
downloads, but enough for policy based caching systems to take a view on
whether to cache or not.

Perhaps include an optional "download-size" fact for the exact size.

-----Original Message-----
From: Stephen Head [mailto:[email-masked]]
Sent: Tuesday, May 30, 2000 8:24 PM
To: [email-masked]
Subject: Ftp-WG: thoughts on ftp extensions and caching

I read/skimmed the current (Feb. 2000) and previous ftp extensions
document and would like to inquire about a couple of things.
First, I would like to compliment the authors of the current document
about the improved feature functionality.

Next, I would be interested to learn if there is any dialogue ongoing
between Marshall Rose, in his incarnation as author of the  Blocks
set of proposals, and the FTP Extensions working group-- if not,
I would like to input that I think it might bevery mutually beneficial
to both parties.

Finally, one of the things I am working on is a ftp caching scheme.
It seems to me that although augmenting ftp caching is not
mentioned as a goal by the charter of the ftp extensions working
group, nevertheless there seems to be considerable parallels
between needs for caching request responses, and needs
for restarting/continuing responses.

One thing that would appear to economize the number of messages
across the network is the inclusion of the kind of information
that would permit a ftp proxy cache server (PCS)to determine how
big a file is and if the file should be cached as a policy issue
in the beginning of a transfer.  This kind of information appears
in a MLST response as described in draft 10 of the extensions:
a unique fully-qualified TVFS pathname, the size of the file,
and a last-modified date and time.  The pathname can be used
as a key for later cache associative matching and file content
retrieval.  The last modified date and time can be used for
cache copy expiration purposes.  The size of the file could be used
to avoid the overhead of storing objects that are deemed to be
too large (or too small, etc.) for the cache.  It appears that
a ftp PCS would be required to issue a MLST and wait for
the response before starting the transfer to obtain this kind
of information.  However, note that many versions of ftp
server applications (eg BSD, solaris) already provide,
in a variation of response 150 that does not appear in RFC959
(that I could find in brief examination)
a unique fully qualified pathname and size already, for example:

  150 ASCII data connection for /etc/motd (208.48.116.174,51740) (53
bytes).

I tried this on the ietf ftp site and got a somewhat shortened
variation (relative instead of fully qualified pathname) but still
an apparently extended response over RFC 959:

  150 Opening ASCII mode data connection for
draft-ietf-ftpext-mlst-10.txt (126451 bytes).

It would seem at least potentially beneficial to re-standardize
response 150 (and its non-stream analog, response 125,
for the perhaps few ftp server applications that support
other than streaming mode, if only for completeness' sake)
to pass the fully qualified pathname, last modified date/time
and size MLST values back as part of the response.  A
convention for doing this might be, for example, mandating
this as part of support for the MLST command in the FEAT
command, and/or some convention that can be recognized
implicitly by a ftp PCS such as a convention for extended
responses "150 X ..." and "125 X ...".  It may be that the
value of this notion to the overall network in reducing
traffic from ftp PCSs and possibly ftp spiders could be
small to moderate, in addition to reducing a network
latency component in response time from any users
downstream of a ftp PCS.

I have been made aware through offline discussions
that computing an accurate file size based on request
type is potentially expensive, due to the need to perform
character set mappings and so on.  Possible solutions
that occur to me at this time include restricting
the guarantee of accuracy for the size value to apply
to binary files only, and (possibly somewhat less
palatable to most) having the ftp server application
pre-calculate the values for non-binary cases
for later convenient reporting.

In any event it seems that many ftp servers already
issue extra-standard information in 150 responses,
that an extended and standardized version of such
responses would be useful for caching proxy servers
towards minimizing response latency and overall
message traffic, and that it could be that a convenient
and appropriate time to extend the standard
to incorporate such improvements, if there is
common agreement on details, coincident with
the release of an extended standard for permitting
reliable restarting/continuation of responses.

In any case, perhaps the idea is at least worth kicking
around for a while even if it does not meet with the
ultimate favor of the group.  If there are any questions
I would be happy to try to answer them-- I am
not an ftp expert.  Thanks,

Steve Head
Entera
Fremont, Calif.
[email-masked]

PS-- thanks to Robert Elz for kindly taking the time
to offer useful comments on an earlier version of this.