Subject: Ftp-WG: Transfer Size info, was: thoughts on ftp extensions a

From: Dave Cridland <[email-masked]>
Date: Wed, 7 Jun 2000 11:27:57 -0500

As far as the SHOULDs and MUST NOTs go, we have got a perfectly good FEAT
which will allow a client to identify whether the server can send the
extended 150/226 responses, and a perfectly good OPT which allows the client
to ask for them. In the absence of a client request, the server probably
MUST use the defacto standard if it supports our new FEAT, but with a client
request it MUST use our new MLST style message, perhaps at the beginning and
the end.

That, and defining the size mentioned in the defacto 150 response as being
only a transfer size (or no mention if it can't discover that quickly), and
defining a "transfer-size" fact, and we're away. Aren't we? Or does that
break something further I haven't yet thought of?

(Incidentally, while WU-FTPD is quite right in specifying a 550 response for
the "SIZE xxx.Z", surely it really ought to recognise the fact that this is
a valid entity, and respond with some other textual part?)

-----Original Message-----
From: Gregory A Lundberg [mailto:[email-masked]]
Sent: Wednesday, June 07, 2000 4:49 PM
To: FTPEXT Working Group
Subject: Ftp-WG: Transfer Size info, was: thoughts on ftp extensions and
caching

> Uh, would it be possible to have a second intermediate response code
> sent after the transfer, reporting the number of octets transmitted?
>
> This response would come before "transfer complete".
>
> This way the server doesn't have to pre-calculate the size, yet can tell
> the client exactly how many bytes the file was.

Adding an intermediate response could break a vast number of clients.

If the desire is for a proxy to have the information required to make a
policy decision prior to the data transfer, the information should be
available prior to sending the data transfer command.  For RETR-type, the
MLST facts, and possibly the SIZE command, should be sufficient for this
purpose.  For STOR-type, there would need to be some means to communicate
the expected size to the remote server; the ALLO command comes to mind here.

If the desire is for a client to be able to determine whether a specific
upload/download is complete; proper use of TCP and sequencing of the
data-channel activity with the command-channel replies is sufficient.

So, the question becomes simply: should there be a standardized format for
the intermediate and final reply?

Let's examine what's done now (using a command-line client and WU-FTPD).
Here's a sample session:

    $ ftp ftp.vr.net
    Connected to www.vr.net.
    220 ftp.vr.net FTP server ready.
    Name (ftp.vr.net:lundberg): anonymous
    331 Guest login ok, send your complete e-mail address as password.
    Password:
    230 Guest login ok, access restrictions apply.
    Remote system type is UNIX.
    Using binary mode to transfer files.
    ftp> ls /pub/pgp-keys/[email-masked]
    200 PORT command successful.
    150 Opening ASCII mode data connection for /bin/ls.
    -r--r--r--   1 lundberg vrnet         712 Jul  9  1999
/pub/pgp-keys/[email-masked]
    226 Transfer complete.
    ftp> bin
    200 Type set to I.
    ftp> size /pub/pgp-keys/[email-masked]
    213 712
    ftp> asc
    200 Type set to A.
    ftp> size /pub/pgp-keys/[email-masked]
    213 727

It appears, at least for WU-FTPD, the size command already provides the
necessary information to make a policy decision for this file.

    ftp> size /pub/pgp-keys/[email-masked]
    550 /pub/pgp-keys/[email-masked]: No such file or directory.

Although WU-FTPD does not go to the work of running any complex conversions
on the file simply to support the SIZE command.  Ah well.  For a
cache/no-cache decision, at least, there appears to be enough information
already available.

    ftp> bin
    200 Type set to I.
    ftp> get /pub/pgp-keys/[email-masked] aaaa
    local: aaaa remote: /pub/pgp-keys/[email-masked]
    200 PORT command successful.
    150 Opening BINARY mode data connection for
/pub/pgp-keys/[email-masked] (712 bytes).

On this reply we see, among other things, the exact size.  Should this be
formalized?  Why not.  While the original intention of the FTP was that the
numeric codes be sufficient for the client, by formalizing the information
on the 150 reply we enable (or ease) automata.

The minimum requirement should remain the numeric replies; the server MAY
send additional information in some predetermined format; the client MUST be
able to handle a 150 reply with non-formatted textual information and even
MUST be able to handle a "150 " reply with no textual information at all.

    226 Transfer complete.
    712 bytes received in 0.00109 secs (6.4e+02 Kbytes/sec)

Here my client has helpfully displayed the number of bytes transferred.  As
a human, I can easily compare this to the size given on the 150 reply to see
the transfer was complete.  Using a formatted reply enables additional
cross-checks, which is probably a Good Thing (even if not strictly
required).  The server MAY send the size information again; the client MUST
NOT depend upon the server actually doing so.

    ftp> get /pub/pgp-keys/[email-masked] bbbb
    local: bbbb remote: /pub/pgp-keys/[email-masked]
    200 PORT command successful.
    150 Opening ASCII mode data connection for /pub/pgp-keys/[email-masked]
(712 bytes).

Now we see that the WU-FTPD is giving an incorrect fact (which I guess I
should note as a bug :).  We know from the SIZE command earlier, the daemon
_could_ determine the correct value.  But that might mean doing what we did
for the SIZE command; processing the entire file to determine the size and
discarding the resulting data stream.  Since we're about to transmit the
file, pre-processing just to send an accurate size seems a waste of time and
resources.  If the server does not already have the size information
available (ie., by stat'ing the file) or the information can be expected to
be incorrect, I would say it MUST NOT send that information in the initial,
intermediate reply.

    226 Transfer complete.
    727 bytes received in 0.00515 secs (1.4e+02 Kbytes/sec)

Again, my client knows how many bytes (octets) were received.  But as a
human I am confused; the numbers do not match.  At this point, the server
SHOULD send the actual number of bytes (octets) transmitted.

Now here's an issue: how should it be worded?  I'd like to see the server
only sending the size information once.  But sometimes that information is
available prior to the transfer, sometimes not until after.  If sent before
the transfer the client (proxy?) can use the information to make policy
decisions (cache/no-cache).  But it's

    ftp> quot allo 727
    202 ALLO command ignored.

Sigh.  Well, this isn't really what ALLO was intended for.  I suppose
WU-FTPD _could_ do a filesystem space availability check.  But that would be
very hard to do inside a chroot'd environment.  The best way to check would
be to actually try creating a file of the desired size, but we don't know
where to create the file and even if we did I'd think long and hard about
the DoS possibilities.

    ftp> bin
    200 Type set to I.
    ftp> put aaaa /incoming/aaaa
    local: aaaa remote: /incoming/aaaa
    200 PORT command successful.
    150 Opening BINARY mode data connection for /incoming/aaaa.
    226 Transfer complete.
    712 bytes sent in 0.000797 secs (8.7e+02 Kbytes/sec)
    ftp> asc
    200 Type set to A.
    ftp> put bbbb /incoming/bbbb
    local: bbbb remote: /incoming/bbbb
    200 PORT command successful.
    150 Opening ASCII mode data connection for /incoming/bbbb.
    226 Transfer complete.
    727 bytes sent in 0.000873 secs (8.1e+02 Kbytes/sec)

Obviously the server has no size information.  But it probably SHOULD give a
read-back of the bytes received.

    ftp> quit
    221 Goodbye.

As to Alun's comments about changing the accepted textual reply ...

If the existing definition is "unspecified", and if the new specification is
no stronger than SHOULD on the server, and the client MUST be able to handle
the case of "unspecified" or even missing textual information, I don't see a
problem.

The RFC gives example relies for those which are "unspecified".  I see those
as no more than recommendations.  Also, a client which depended upon the
recommended replies would have problems handling a server which was using
non-English text in replies.

The best method, if we don't want to change the final reply, is to use
extended replies, such as:

226-727 bytes received
226 Transfer complete.