Subject: Ftp-WG: thoughts on ftp extensions and caching

From: Alun Jones <[email-masked]>
Date: Wed, 7 Jun 2000 12:03:20 -0500

At 10:43 AM 6/7/2000, you wrote:
>When I read through the MLST draft trying to see whether I was allowed to 
>drop a fact when the client had requested it, I couldn't find anything 
>about it, either way, which suggested to me in my pedantically crippled 
>way that it wasn't allowed - or at least, that it probably shouldn't be 
>done from a server PoV, but should be allowed for in a client context. It 
>may well be just me, though, I don't know if anyone else reads it that 
>way. Of course, I might have missed the critical phrase while skimming 
>through it, at which point I apologise profusely for my stupidity. :-)

End of 7.2:

   "Facts
    should be provided in each output line only if they both provide
    relevant information about the file named on the same line, and they
    are in the set requested by the user-PI.  There is no requirement
    that the same set of facts be provided for each file, or that the
    facts presented occur in the same order for each file."

Sounds like facts should _not_ be provided if they don't provide relevant 
information about the named file.  Perhaps a clarification is in order, but 
I read it as meaning that a nonsensical fact should not be displayed (e.g. 
a size on a directory).  Note also that section 7.7.3 shows an MSLD example 
that does _not_ include the Size fact for its directory entries.  7.7.9 
includes a similar example that shows dropping of the size facts on 
directories, but since that looks like my server, I'd say I'm biased :-)

>Standardising the responses from RETR is surely fine if it only becomes 
>active with a OPT setting? Or not?
>
>Having said that, Alun Jones makes very good sense indeed in his point 
>(2). Perhaps all that is needed is a mention of this de-facto standard in 
>more detail. If so, all the better.
>
>I do still like the concept of the transfer-size fact being defined, though.

The way I've written my server is that the expression "(N bytes)" is 
output, and is essentially the only item in parentheses on that line 
(unless you want to try and break your FTP client by creating a file called 
"(-1 bytes)" and retrieve that :-))  That appears to be standard - here's 
an example:

150 "/uplap.bat" file ready to send (437 bytes) in ASCII mode

Any transfer where I do not have a ready count of the bytes to send, gets 
no byte count - nothing in the parentheses.

If you're going to document that, I'd suggest also documenting that the 
byte count given may, for all sorts of reasons, not be the same as the 
number of bytes sent.  Let's say, for instance, you're transferring a text 
file that logs every FTP response message sent - that will _never_ be the 
same size as reported!  [I also have customers who use my server to 
'stream' video files of 13GB or so - when they first appear on the server 
to be downloaded, they'll look like they only have a few KB in them, so the 
value given at the start of the transfer is _greatly_ different from the 
number of bytes transferred - by several orders of magnitude!]

Alun.
~~~~
P.S.  On the subject of changing file sizes, it's clear that many people 
are unaware of file stores that use paged or record structures on files, 
where many pages/records are "empty" and hence not stored.  A file could 
occupy a few bytes on disk, but when transferred in File / Image mode, be 
several megabytes (mostly zero, or whatever the fill character is defined 
to be).
--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us
1602 Harvest Moon Place | at web site http://www.wftpd.com or email
Cedar Park TX 78613     | us at [email-masked].  VISA / MC accepted.
Fax +1 (512) 378 3246   | NT-based ISPs, be sure to read details of
Phone +1 (512) 378 3246 | WFTPD Pro, NT service version - $100.

---

From: Stephen Head <[email-masked]>
Date: Wed, 7 Jun 2000 13:30:47 -0500

> no byte count - nothing in the parentheses.

Whoa there pilgrim, does this adequately express how a client should
interpret that format under all circumstances?  In particular, considering
the permissible character set from which file names are comprised,
as in the following sequence:

...

230 User smh logged in.
ftp> put /tmp/rot "/tmp/xxx (437 bytes)"
200 PORT command successful.
150 ASCII data connection for /tmp/xxx (437 bytes) (a.b.c.d,49720).
226 Transfer complete.
local: /tmp/rot remote: /tmp/xxx (437 bytes)
108042 bytes sent in 0.017 seconds (6176.30 Kbytes/s)
ftp> put /tmp/rot /tmp/rot55
200 PORT command successful.
150 ASCII data connection for /tmp/rot55 (a.b.c.d,49722).
226 Transfer complete.
local: /tmp/rot remote: /tmp/rot55
108042 bytes sent in 0.018 seconds (5991.47 Kbytes/s)
ftp> get /tmp/rot "/tmp/xxx (437 bytes)"
200 PORT command successful.
150 ASCII data connection for /tmp/rot (a.b.c.d,49724) (9139 bytes).
226 ASCII Transfer complete.
local: /tmp/xxx (437 bytes) remote: /tmp/rot
9340 bytes received in 0.058 seconds (157.37 Kbytes/s)
ftp>

...

[a.b.c.d] uname -a
SunOS tomcat 5.7 Generic_106541-07 sun4u sparc SUNW,UltraSPARC-IIi-Engine

In particular, how should a client reliably parse the so-called de-facto
150 response given that filenames can contain all those funky characters?

If forwards, then they could run into files with the same name (granted,
unlikely, but still apparently "legal", at least in the common-practice sense).

If this is correct, but not widely implemented, perhaps at least this is a valid
issue
for standardization in some manner, at the earliest convenience.

(Or what have I missed...)

> number of bytes transferred - by several orders of magnitude!]

Noted, though I still would tend to expect the majority of files transferred
around the net to be of the static exact-size variety (and so
good potential candidates for caching).

-s

---

From: "Gregory A Lundberg" <[email-masked]>
Date: Wed, 7 Jun 2000 13:45:59 -0500

> ftp> put /tmp/rot "/tmp/xxx (437 bytes)"

Not to mention what RFC 2640 could do to you.

> good potential candidates for caching).

I dunno.  I get the feeling that if we put something akin to a CGI interface
in WU-FTPD you'd see a lot more on-the-fly stuff.  Just from the questions
I've fielded on the support lists and via phone conversations.  I know it
smacks of HTTP-izing FTP, but people want it, I know some who are doing it
now, and I think many more would use it.

---

From: Stephen Head <[email-masked]>
Date: Wed, 7 Jun 2000 14:01:21 -0500

> 150 ASCII data connection for /tmp/xxx (437 bytes) (a.b.c.d,49720).

Sorry, that example did not quite illustrate what I expected,
haste made waste, for the vicariously interested my local
machine uname -a is

SunOS viking.entera.com 5.7 Generic_106542-08 i86pc i386 i86pc

but the following comes close (and is actually what I had
in an earlier version of my message before I tried to fine tune
it):

ftp> put /tmp/rot "/tmp/(437 bytes)"
200 PORT command successful.
150 ASCII data connection for /tmp/(437 bytes) (208.48.116.174,49681).
226 Transfer complete.
local: /tmp/rot remote: /tmp/(437 bytes)
108042 bytes sent in 0.022 seconds (4749.27 Kbytes/s)
ftp> get "/tmp/(437 bytes)" /tmp/fooi
200 PORT command successful.
150 ASCII data connection for /tmp/(437 bytes) (208.48.116.174,49685) (105088
bytes).
226 ASCII Transfer complete.
local: /tmp/fooi remote: /tmp/(437 bytes)
108042 bytes received in 0.035 seconds (3035.99 Kbytes/s)
ftp> put /tmp/rot "/tmp (437 bytes)"

...

(Note in particular the two occurences of the pattern "(nnn bytes)"
in the second and last 150 response.)

(Looks as if one perhaps can't quite presume the doublequotes do
consistent duty on solaris 5.7 solaris ftp...)

-s

---

From: Alun Jones <[email-masked]>
Date: Wed, 7 Jun 2000 14:45:22 -0500

...
>ftp> put /tmp/rot "/tmp/xxx (437 bytes)"

Uh, yeah, that's what my "creating a file called '(-1 bytes)'" note was 
about.  It's not perfect, and it's not documented, but it seems to be what 
other FTP servers do, and it seems to be, for the most part, quite usable.

>150 ASCII data connection for /tmp/xxx (437 bytes) (a.b.c.d,49720).

To be honest, I've never seen the local port assignment returned inside of 
an FTP server response.  That's definitely a new one on me.

>150 response given that filenames can contain all those funky characters?

"so-called de-facto"?  It's about as 'de-facto' as the 'ls -l' format is 
for file listings, and it's perhaps more 'de-facto' than the SIZE and MDTM 
commands were before the MLST draft came along.  Whether it's reliable or 
not, it _is_ here, it _is_ widespread, and it _is_ used by many of the FTP 
clients I run into on a daily basis.  While that's not a statistically 
valid survey, I think you will find that a removal or disabling of that 
functionality, by requiring a different 150 response, would produce a lot 
of complaint from client users.

>unlikely, but still apparently "legal", at least in the common-practice 

Were it my FTP client, I'd parse backwards from the end of the last line, 
but then I'm perfect :-)

>for standardization in some manner, at the earliest convenience.

Perhaps we need to collect examples of 150 text messages from various 
servers, to know what is usual.  Anyone want to send me examples of their 
150 responses, so I can collate?  [RETR only, please, and no mucky 
filenames like that given in the examples above!]

>good potential candidates for caching).

Maybe so, but it is worth informing FTP client authors that this does not 
_always_ happen.  In most file systems / operating systems, you can append 
to a file while it is open for reading by an FTP server, and that extra 
data would then appear in the transfer.  To provide a means by which a 
client can 'tell' the size of a file before it is transferred implies to 
some programmers that this is where the file shall stop.  'tain't 
necessarily so, and anything that documents such a measure as being a 
foretelling of the size of the file would do well to remind programmers 
that the eventual size may be different.

Alun.
~~~~

---

From: Stephen Head <[email-masked]>
Date: Wed, 7 Jun 2000 17:05:45 -0500

> of complaint from client users.

The point I was trying to make was that there appears (or at least,
appears to me) to be some variation between the existing relevant
standard (959) and each of the various forms
of the de-facto-whatever :-) standard.  (Also a FEAT variable
permit more latitude in the response and avoid compatibility problems
it would seem to me.)

> Were it my FTP client, I'd parse backwards from the end of the last line,

Thanks; perhaps this can be noted that this is the correct way to parse it
if this format is chosen (I was concerned about how frequently parsing backwards
is done in practice-- this is one of the concerns I sought to avoid
by using FEAT and a spiffier format loosely based on d-10).

> filenames like that given in the examples above!]

I already sent out two variants a few messages back
for illustrative purposes.  They differed
in that one used a fully-qualified name and the other did not.
Also, ASCII was inaccurate even in the static case.
I also suggested the notion of using FEAT in some manner
to help clean it up.  For caching purposes, fully qualified name and accurate size

are better than 'base' filename and approx-size.  Modified date would
also, I believe, be useful to avoid sending separate MDTM for caching
(if parsing backwards is the ticket for the extended 150 response,
then there should be some room to play with between the filename
and the byte count to put a datestamp).

> that the eventual size may be different.

Yes...

These discussions have helped clarify
to me some of the origin of the extended response but
its reliability for use is still somewhat of a concern
from my viewpoint.  I thought it would be an overall win
to bring these issues to the attention of the wg, in the form
of some constructive potential solutions.  Well, back to the
pub XXX work,

-s