Subject: Ftp-WG: thoughts on ftp extensions and caching

From: Dave Cridland <[email-masked]>
Date: Wed, 7 Jun 2000 06:14:02 -0500

Following on from my earlier attempt at breaking down the problem into its
parts:

a) Seems to be fixed in practise, but not documented. Presumably the
solution works for uploading as well, which as I said before is going to be
the more common instance of this in my opinion.

b) Seems still to hold.

c) Nobody seems to like the idea of using approximate sizes, and in some
cases these are not available at all.

I think that, where available, and "reasonable", the reasonability of a
particular estimate being defined by configuration, the approximate size is
still useful.

Stephen Tihor suggests a factor of 4 between "size" and "transfer-size",
however, I think he'd probably agree that this is only in theory rather than
in practise. Certainly while UNIX text files can have a factor of 2 between
the "size" and the "transfer-size", when transferred as ASCII, this is never
going to be the case unless the file consists only of the character '\n',
which my gut tells me is somewhat unlikely in practise, and my brain agrees
totally, for once.

I'd managed to completely forget about dynamic archive creation, of course,
but this simply means that no "size" fact is available for that file. Which
in itself may cause a problem with the MLST draft - if a fact cannot be
found for only some files in a listing, then does the MLST draft allow us to
drop the fact for just those files? Reading the draft, it appears not to.
I'm happy to be proved wrong, of course.

-----Original Message-----
From: Gregory A Lundberg [mailto:[email-masked]]
Sent: Wednesday, June 07, 2000 2:39 AM
To: FTPEXT Working Group
Subject: Ftp-WG: thoughts on ftp extensions and caching

> (or at least, so it would seem).  If so, the problem referred to in RFC
959
> does not exist in practice (and if so becomes a candidate for wordsmithing
> out).
>
> [In what form will the eventual FTP extension reach the RFC stage?--
> if rolling RFC959 in its entirety with a new RFC #, then this might be
> opportune to address at that time.]

In practice, WU-FTPD 2.6.0 made the switch.  What we've found is a few
(actually, very few) clients which will lock up waiting for the reply before
closing the data channel.  The list is fairly short, most vendors made
updates when it was pointed out to them (if they'd not already run across
the problem .. seems some far-less-often used daemons were doing the same
thing).

We support backward compatability with a compile-time option.  From the
questions on the support mailing lists, while the subject is a FAQ (and
covered there), it seems the sites most in need of the backward
compatability are those in support of specific products where the client is
a legacy and having users upgrade is infeasable or impractical in the
short/middle term.  Sites where most users come in using graphical clients
such as IE and NS have no such problems and generate very few pleas for
help.

> Storing and later retrieving on demand such information concurrently with
file
> retrieval
> is additional work but not necessarily involving a significant delay if
done
> intelligently.

What I'm referring to is massive pre-processing such as tar/gzip like we do
in wu-ftpd's ftpconversions.  I can (and have) ftp'd multi-Gbyte directories
as a tar/gzip piped to the data channel.  If I needed to wait for the
tar/gzip to complete it could be several minutes and the server host would
be required to store the entire tar/gzip in temp storage (which I hope is
disk otherwise a sufficiently large archive will cause massive swapping on
the host).  Technically, it's possible.  As a practical matter, though, I
would never ask our users to live with such.

--

On the subject of rolling RFC 959 into a new RFC; I'd like to see that.
Heck, I'd like to see simply clarification and modernization of the language
in RFC 959.  I'm not sure, however, I'd like to see that begun until we've
sealed and shipped the MLST and HOST RFCs.  Let's not derail the work on
those with a massive new project, yet.

---

From: Robert Elz <[email-masked]>
Date: Wed, 7 Jun 2000 07:50:05 -0500

Date:        Wed, 7 Jun 2000 06:14:02 -0500
    From:        Dave Cridland <[email-masked]>
    Message-ID:  <[email-masked]>

  | Which
  | in itself may cause a problem with the MLST draft - if a fact cannot be
  | found for only some files in a listing, then does the MLST draft allow us to
  | drop the fact for just those files? Reading the draft, it appears not to.

That certainly wasn't the intent.   Any fact is supposed to be able to
be dropped when it cannot be supplied, or makes no sense.

In practice, dynamic archive type things tend not to appear in directory
listings anyway, as they don't really exist - the "RETR dir.tar.gz"
hack is truly that - using the ".tar.gz" part of the filename as a
set of options to the command to indicate to the server what it is
supposed to do with the entity requested (which is just "dir" in this
example).   It's totally legal FTP, but also horribly ugly.

kre

ps: is there no-one else but me here who believes that attempting to
standardise (in any form) the responses from RETR is just a little too
far from rational after all this time to even be discussing, however
benefitial it might be in some contexts?

---

From: Alun Jones <[email-masked]>
Date: Wed, 7 Jun 2000 10:22:00 -0500

Sorry I haven't been able to chime in on this sooner.

>benefitial it might be in some contexts?

I'm with you on that one, Robert, on two counts, that may actually appear 
to be contradictory:

1. RFC 959 (among many other protocol documents) makes it clear that the 
goal of the number is to provide information to the client, and the goal of 
the text is to provide information to the user.  Very few commands supply 
information in the text of a response that is expected to be read by the 
client - PASV, CWD, PWD, STOU, are the only ones that I can think of from 
RFC 959.  Later added commands provide more use of the response text for 
server to client communications, such as SIZE, MDTM, FEAT, etc.  However, 
only once (to my recollection), in RFC 1123, has an accepted command been 
provided with a new standard for its text response.  I'd like to see a lot 
more reason than "it saves issuing one command" before requiring a format 
of text, and one that is not directly unrelated to the command at 
hand.  STOU really _needs_ to tell the client what it stored the file 
as.  CWD and PWD _need_ to tell the client what your new directory 
is.  PASV _needs_ to tell the client where it's listening.  RETR does _not_ 
need to tell the client, and may not even know, how many bytes to expect.

2. It's already common for the RETR command to return the number of bytes 
expected in the 150 response, where reasonable.  Many FTP clients use this 
value to generate a progress indicator.  It's not as reliable as SIZE, 
perhaps, but it is common, and it is used.  Requiring a new format in the 
150 response would have the rather unhelpful effect of removing 
functionality from those clients.

Alun.
~~~~

--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us
1602 Harvest Moon Place | at web site http://www.wftpd.com or email
Cedar Park TX 78613     | us at [email-masked].  VISA / MC accepted.
Fax +1 (512) 378 3246   | NT-based ISPs, be sure to read details of
Phone +1 (512) 378 3246 | WFTPD Pro, NT service version - $100.