Subject: Ftp-WG: thoughts on MLSx/PWD/TVFS for caching/restart

From: Stephen Head <[email-masked]>
Date: Sat, 10 Jun 2000 16:54:35 -0500

For caching, it appears it would be useful to get a relatively unique
handle for the object being retrieved (in as passive a manner
as possible).  One can't reliably use the 150 response because
it is not standard and there are variants and parsing problems.
One can't reliably use MLSx because the pathname returned
is not guaranteed fully qualified, by definition and (so I am told)
intent, due to need to operate on various filesystems
that don't have fully qualified pathnames.  One can't use Unique
because it is not guaranteed to be unique across connections by
definition
and intent (and I would not try to force it to be for implementation
reasons).  One can't use a concatenation of PWD and MSLx
outputs because PWD is not guaranteed to work, not guaranteed
unique, has failure modes (eg a concurrent rmdir ..) and even might
(421 response allowed in RFC 959) cause the connection to drop.
So it appears one can't just issue a PWD and check the results-
such a proxy cache server trick would get it disconnected from
certain origin servers and unduly interrupt the sequence of requests
being made from the destination client.

The sum of all that is as far as I can tell it appears to be that one
can only reliably cache if there is at a maximum one CWD issued
to the origin server per control channel connection with a NVFS or
TVFS.

Given my understanding of the intent of the TVFS to model Un*x-like
tree structured filesystems, this seems to be an unfortunate side-effect

of the definitions of the features in d-10.

To the concern that it only hurts the potential for caching, it could
be said that it also appears to miss an opportunity for permitting
restarts
across a prematurely closed control connection.

I could think of two potential ways to address this problem.

One is to add a FEAT command which permits the server to
advertise in advance support for the PWD command.  This would
avoid the 421 response danger described above.  I've appended
something for this at the end of this message.

Alternatively a relatively small additional tweak of the MLSx command
could be made in the case of advertised TVFS support could be made,
changing the reporting of the fully qualified pathname from SHOULD
to MUST.  This would still permit NVFS-not-TVFS server systems
to squeak by without the need for reporting fully qualified pathnames
when they choose (or are unable to comply), because SHOULD
would still be operable in those cases.  If my understanding
of TVFS is correct, servers which have TVFSs will always have fully
qualified pathnames available to report for MLSx.
Of these alternatives, I like the second better because it seems (?)
more in the spirit of the extensions and it also doesn't involve the
extra network load of sending PWDs and getting the responses
in association with sending MLSxs.

If the second alternative were chosen, it is conceivable that one
could envision a later, separate extension to the current d-10
which could fix this, but in the meantime there would be created
a specification hole in which some (a small few?) systems implementing
d-10 TVFS and MLSx would take SHOULD as advisory only and not report
fully qualified pathnames in MLSx.  That would seem to create
at least the possibility of a compatibility problem for extending MLSx.
If so, as far as I can tell, it would involve new variants of MLSx,
which could be viewed as very distasteful and possibly
messy enough that it would be unlikely to be approved, at least along
those lines.  Adding a FEAT for PWD would still seem to be
viable at a later date, but still seems less elegant as well as
somewhat more overhead everywhere.

To the concern that I am being selfish by bringing up ftp caching
by myself, it is not my intent to be selfish here.  At least as
of a couple of weeks ago (the last time I checked), Inktomi
advertised ftp caching on their feature list on their web site
http://www.inktomi.com/).  More caching companies seem to be appearing
on the radar seemingly almost every month, and Inktomi is
the current leader in at least the non-streaming caching arena.
I think a little credit might be extended to the area of caching
given that it is a relatively new and growing area.

(I expect that most caching would, in practice, occur for TVFS-style
ftp server filesystems.)

I also think that a test of a really robust standard is
one which can withstand the test of time (and exposure to
unexpected new uses) best by adhering to good fundamental
principles.  So I think it is at least conceivable that allowing
this issue might benefit more than just one application.
There is also a downside risk (perhaps illustrated in part by
Squid, etc.) to attempt to use tempting but not necessarily
correct shortcuts to address the issue in lieu of a standard.

I am finding it very time consuming to look though old wg discussions
(and I haven't at least yet had a chance to write a script to
get it all in batch and then read it) so if this duplicates
any old discussion that was made and dispensed with in some manner
(or if I have missed something, always possible :-) my apologies.

-s

PS, FWIW A quick and dirty FEAT response for PWD:

X.X. FEAT response for PWD

  When replying to the FEAT command [6], an FTP server process that
  supports the PWD command MUST include a line containing the single
  word "PWD".  This MAY be sent in upper or lower case, or a mixture
  of both (it is case insensitive) but SHOULD be transmitted in upper
  case only.  That is, the response SHOULD be

    C> Feat
    S> 211- <any descriptive text>
    S> ...
    S> PWD
    S> ...
    S> 211 End

  The ellipses indicate place holders where other features may be
  included, and are not required.  The one space indentation of the
  feature lines is mandatory [X].

---

From: Stephen Tihor <[email-masked]>
Date: Sun, 11 Jun 2000 10:26:25 -0500

If the concern is to get a stable, unique, fully qualified pathname supported 
it seems more logical to just define an additional fact that would be that and
identify the conditions udnerwhich it might exist and suggest strategies
for an implemetnation to follow when it does not.

---

From: Stephen Head <[email-masked]>
Date: Sun, 11 Jun 2000 17:27:11 -0500

I believe the PWD fact notion is for practical purposes
as close to what you describe as one could reasonably
hope for.

I do believe however that adopting either
alternative iff TVFS fact supported
seemed at least to me very much in keeping with
the spirit of the original TVFS intent
(speaking as a reader only and not as an author
of the TVFS concept).  I don't know what is so
illogical about either alternative.  Actually, they
don't preclude each other (and implementing both
would give clients more flexibility for getting info).

-s

---

From: Alun Jones <[email-masked]>
Date: Mon, 12 Jun 2000 14:09:07 -0500

>(421 response allowed in RFC 959) cause the connection to drop.

In this case, the 421 indicates that the connection is being dropped - it 
does not mean that the PWD _caused_ the connection to drop.  In exactly the 
circumstances that would generally elicit a 421 error from PWD, a 421 error 
would generally arrive from FEAT, or even NOOP.  _All_ commands should be 
allowed to produce the response "421", because it is an indication of an 
event occurring _before_ the command has been received.

>being made from the destination client.

Nope.  Wrong.  PWD / MLST should work for most caching; the only issue 
comes when two differently named directories are actually the same - by 
links, mount points, etc.  In such a situation, any method you care to 
choose has its own weaknesses, and cannot be relied upon; hence, a proxy 
must make some "best effort" to find those directories that are the same, 
and to suck up the doubly cached files otherwise.

>to the origin server per control channel connection with a NVFS or

In TVFS, you have the concept of a root.  Otherwise, there is not 
necessarily any way to get back to the same directory you logged in to, 
once you have ventured outside of it.  Practically speaking, there almost 
always is some way to do this, but it is not part of the FTP protocol for 
good reason - covering every possible way to change directories would make 
FTP into a protocol that has to morph once more every time a new file 
system was introduced.  [We can not assume that a hierarchical tree is the 
be-all and end-all of directory structures]

>across a prematurely closed control connection.

No - the CWD sequence and/or RETR that you used to originally fetch the 
file will still be valid to _reach_ that file, and an MLST / SIZE / DATE 
combination can give you a good enough idea that the file _is_ the one you 
thought it was.

>something for this at the end of this message.

That's somewhat ludicrous.  As I mentioned earlier, the same effect 
(usually a timeout, or system administrator-forced disconnect) that would 
"cause PWD" to respond 421 would "cause FEAT" to respond 421 also.  The 
reason being that the 421, and the disconnection, are generally actioned 
_before_ the server receives (and usually, before the client sends) the 
command in question.  You don't avoid the (non-existent) problem that you 
claim to be providing a fix for.

If you have a reason other than this to provide a FEAT response for PWD, 
then please suggest it, but avoidance of a possible 421 is not appropriate 
reason.  The 421 is _not_ provided as a valid response in the case of "we 
don't support PWD".

>qualified pathnames available to report for MLSx.

And the client _will_ be able to tell the difference between a fully 
qualified pathname (which begins with '/') and one that is not.

Just my two ha'p'orth.

Alun.
~~~~

--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us
1602 Harvest Moon Place | at web site http://www.wftpd.com or email
Cedar Park TX 78613     | us at [email-masked].  VISA / MC accepted.
Fax +1 (512) 378 3246   | NT-based ISPs, be sure to read details of
Phone +1 (512) 378 3246 | WFTPD Pro, NT service version - $100.

---

From: Robert Elz <[email-masked]>
Date: Mon, 12 Jun 2000 14:30:56 -0500

I agree with Alun here.

I'm also not sure what problem exactly is being solved.  Maybe it
is just because I have no clue at all what is involved with caching
of FTP files, but ...

I don't think that perfect caching of FTP is possible.  That is,
fetching data to the cache once (and assuming the cache doesn't discard
he data, and it remains stable on the server) never fetching it again.
The FTP model just doesn't guarantee the kind of information to make
that possible.

On the other hand, adequate caching is I think possible (or ought to be).
That means, being able to detect that a file being requested is in the
cache, and returning the copy (when appropriate).  But without
necessarily preventing the same file being in the cache (under different
names) more than once (ie: sometimes it may be fetched when had the
cache but known, it could have found a local copy already).

I don't think I'd support any changes to FTP that are aimed towards
making perfect caching (as I have defined it) work, as I don't think
it is possible anyway without discarding too many other aims of FTP.

On the other hand, if there's something lacking in FTP that would
prevent adequate caching from working, then that perhaps should be
fixed.

It is also probably worth noting that while lots of things aren't
guaranteed from FTP servers, most of them will be found in practice.
That means that a cache (or any client) can't rely on evernything
being nice and clean, it can optimise itself for that case, and
do better when "nice" servers are encountered (which is likely to
be most of the time).   So, for example, while a client (cache perhaps)
might not be able to guarantee it can find a full path name for a file,
most of the time it will be able to do that.   That ought mean that
an adequate cache ought to be able to approach being a perfect cache
if it is willing to do the work - for most servers on the net, yet
still function for the other oddball cases.

kre

---

From: Alun Jones <[email-masked]>
Date: Mon, 12 Jun 2000 15:18:40 -0500

>The FTP model just doesn't guarantee the kind of information to make

I presume also that, if such a specific need does require specific 
additions that have not yet made themselves clear to those of us that have 
been supporting servers "in the wild" for several years (and I'm one of the 
junior lot in this case), a new RFC draft might be the better place to 
include that, rather than in this (already about a decade or so overdue) draft.

As I see it, the main purpose of the MLST draft has so far been to document 
the de-facto standard commands of SIZE and MDTM, the de-facto operation of 
REST for stream-mode communication, and finally the introduction of MLST 
and MLSD to avoid what is by _far_ the most common complaint of FTP authors 
- both server and client - that is of no reliable, machine-parsable listing 
format.  In that sense, we are to some extent taking FTP back from an 
effective stance of "everyone does it like one or other variety of Unix" to 
a stance of "everyone does it like this".  It just so happens that the TVFS 
mimics Unix (although that's mostly immaterial), but MLST does not.

Some of the things that are asked for in this cache topic - specifically 
enough information to determine that "these are not the files you are 
looking for" - are not by any means absolute.  Others - the file storage 
size, and the file transfer size - would require a byte-by-byte read 
through the file prior to transfer, and the locking of that file both prior 
to, and during, the transfer.  Unfortunately, the cache is going to have to 
_guess_ some of this information - that makes the cache's job somewhat 
harder.  However, the alternative would be to either make the FTP server's 
job a whole lot harder, or potentially reduce the number of operating 
systems that could support an FTP server.

Alun.
~~~~

---

From: Stephen Head <[email-masked]>
Date: Mon, 12 Jun 2000 15:36:40 -0500

> I agree with Alun here.

I'll concede most if not all of Alun's points and thank him
for his patient response.

> The FTP model just doesn't guarantee the kind of information to make

I also agree that perfect caching of FTP is not likely to be possible.
Still, there appears to be some market demand for it, regardless
(I did not drive it).  The expectation is apparently to use a number
of admin-configurable, heuristic methods including size, mod date,
and timeouts (ugh).

> cache but known, it could have found a local copy already).

Yes, by getting pwd-type info I am attempting to make at least
a reasonable effort to avoid that.  I don't expect to succeed every time,
just make a reasonable try.  The consequences of multiple copies
for the same thing could be a crowding out of other valid info in the
cache(s),
especially if the copies are relatively large.

> prevent adequate caching from working, then that perhaps should be

The principle I started out trying to follow was to maximize
transparency at both ends:  if the destination client did command C1,
a maximal-transparency proxy server would try its best to
pass the request along as command C1 (maybe varying the
data connection address).  That didn't work too well with the wg.
While RETR can be replaced with a PASV-MLST-PWD-RETR
sequence, the more additional commands that are added by
the proxy server, the more overhead and alternative code paths
are exercised on the origin server, and the more downside risk
and performance hit.  Minimizing the size of the substitute list
of commands minimizes that risk and hit.  Proxy server supporters
would presumeably want to strive to avoid the situation in which
the original destination client command issued to the origin server
works, but the substitute proxy server generated command stream
for some possibly bogus reason beyond the proxy server's direct
control does not work, even though it should by the standard.

At this point my situation has been whittled down, as far as I can
tell, to either accepting PWD in addition to MLSx, or the following.

One could return to the idea supplied by Stephen Tihor (thanks,
and sorry for misunderstanding it, I think, the first time around)
of a pwdir type fact for MLSx.  The advantage of such a fact
would be that it would appear to avoid the need for MLSx-PWD cliches
being issued ad nauseum by proxy servers.  It may be in keeping
with the philosophy of MLSx in the first place of rolling a bunch
of info into one response (?).  It would be nice if TVFSs were
required to send such a type if requested in MLSx responses,
so the proxy servers would not be forced to try to issue PWDs
anyway in case the MLSx responses didn't contain a fully qualified
path.  From a proxy cache server perspective, part of the "state"
of a file is where it is located/locatable on a [TVFS] server.

OTOH, a pwdir type fact *is* yet another feature and depending on how
the WG is run it *may* delay final elevation of the draft to a standard
(dunno, depending on how you all run things).

In hindsight, I regret that I did not arrive at this right away
but my mind often gets numb going over all the hoary ftp stuff.

I think those are the tradeoffs and I can respect the consensus
of the group either way.  I can try to come up with something for a pwdir
just in case there is considerable vocal wg approval (?) expressed towards
the notion of going forward on that.

At any rate, plowing through this stuff rigorously from the perspective
I took apparently helped reveal a couple of minor omissions that are
nevertheless nice to fix for a full standard, I guess.  So not a complete
waste... :-)

-s

---

From: Stephen Head <[email-masked]>
Date: Mon, 12 Jun 2000 16:06:22 -0500

Alun Jones wrote:

...

> mimics Unix (although that's mostly immaterial), but MLST does not.

Yes, I recognize the frustration.  However, I did think that draft 10
is a great improvement over draft 7 (the previous draft I was looking
at before I was asked to help put out some fires somewhere else)
for universal purposes.  I would have suggested something along
the lines of MLSx myself if I had been continuously engaged in the issue
(though if I were writing it, with some kind of notion of pwd included
for TVFS case).

> harder.

Just to say that I gave up on any refinements to size, for the good reasons
expressed by you and others in previous responses.  The fully qualified pathname
admittedly should, as a workaround for systems that support it, usually be
available through a separate PWD command as a workaround for a lack of some
pwdir-style fact  as I noted in a previous response.  Presumeably,
for those systems that have it, it is mostly a matter of reporting it in contrast
to some huge computational load.  (If for any significant server situations
not the case, then perhaps not such a good idea despite what caching servers
would prefer.)

-s

---

From: Stephen Head <[email-masked]>
Date: Tue, 13 Jun 2000 13:07:51 -0500

I had some offline discussions about the comparative merits of changing
some wording in some cases in 7.3.1 to make reporting the fully qualified
pathname always recommended (ie SHOULD), versus required (MUST),
versus other possible options for the TVFS case.  SHOULD already
seems to apply for the TVFS MLST CRLF case (ie no argument in request).

I arrived at the tentative conclusion that the notion (credit to Alun)
of having MLSx *should* (instead of *may*)
return a fully qualified pathname for the TVFS case
seems as if it should be "good enough" for my plight
with regard to ftp proxy caching.

Here is (I believe) why:

  -> when it is "should", one can reasonably (?)
        expect "most" implementations to follow
        the recommendation  for mutual benefit
        of clients and servers (ASSUMPTION)

  -> an ftp proxy cache server can, if the fully qualified
         pathname is not returned with MLST response,
         either issue a followup PWD request,
         or choose to give up on trying to cache the file.
         The PWD request will incur a network latency
         quantum.  Giving up risks missing a later
         cache hit opportunity.  It would just be a caching policy
        decision tradeoff.

   -> it is symmetric with the TVFS-> "SHOULD"
         return fully qualified pathname spec for MLST CRLF
         (ie no argument specified in request)
         already in d-10, 7.3.1 (if my reading is correct).
         [Virtually all the arguments against
          having TVFS-> "SHOULD" return fully qualified
          pathname for the MLST case appear to apply
          just as much to this case too.  If so,
          the same level of recommendation (whether
          should, may, or must) might logically be applicable
          to the other, analogous cases, or at least so it would appear.]

  -> implementors that go against SHOULD in this case
        incur the potential risks of either supporting extra
        RETR requests that could otherwise have been cached,
        or getting extra PWD requests.

Assuming (?) that 80% of the ftpd folks follow the SHOULD
recommendation were it to be specified, then
80% of the time, the ftp proxy server <-> ftp server
interaction would be "optimal" 80% of the time.

This might look something like the following
(additions/changes are boxed)

7.3.1. Notes about the Filename

   The filename returned in the MLST response should be

----------------------------------
 |      , in the case in which TVFS is not supported,      |
----------------------------------

   the same name as
   was specified in the MLST command, or, where TVFS is supported, a
   fully qualified TVFS path naming the same file.  Where no argument
   was given to the MLST command, the server-PI may either include an
   empty filename in the response, or it may supply a name that refers
   to the current directory, if such a name is available.  Where TVFS is
   supported, a fully qualified path name of the current directory
   SHOULD be returned.

  ...

   If the server-FTP process is able, and the "type" fact is being
   returned, it MAY return in the MLSD response, an entry whose type is
   "cdir", which names the directory from which the contents of the
   listing were obtained.  Where TVFS is supported, the name MAY

   XXX XXX XXX

----------------------------------------
|   the cdir fact SHOULD be returned and the name SHOULD    |
----------------------------------------

   be the
   fully qualified path name of the directory, or MAY be

   XXX XXX XXX

---------------
 |   and otherwise is     |
---------------

  any other path
   name which is valid to refer to that directory from the current
   working directory of the server-FTP.

I imagine other alternative wordings are possible.

If it is OK to take a quick (unofficial) canvass, does anyone have
big problems with this?  I don't have any formal ties to this wg
so the notion can stand or fall based on whatever you all think.
If it is wildly unpopular, or some procedural issue gets in the way,
at least it could be said that the idea was aired but not acted
on.

BTW there are some possible problems with "pipelining" requests
on the control channel:
  * it might cause less carefully written ftp servers to barf; and
  * if the ftp server does two sends to the TCP level
     and the Nagle algorithm is not disabled, TCP might
     insert a delay.
thanks (again) to Alun for noting these potential problems
with this.

-s

---

From: Stephen Head <[email-masked]>
Date: Wed, 28 Jun 2000 18:02:59 -0500

I have received no objections to the thoughts I had on
this topic in response to my request for same.  Perhaps
this is a good omen.

I will be moving on from my current Silicon Valley
anthill to the next, so if anyone has any questions
for me feel free to reach me at

  eclectix  _AT_  pacbell _DOT_ net

Thanks to all for their patience and good luck,

Steve Head
Entera

-----

...