From ftp-wg-owner@hethmon.com Wed Oct 2 16:36:10 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA06876 for ; Wed, 2 Oct 2002 16:36:09 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021002154611-52002-10 ; Wed, 02 Oct 2002 15:46:11 -0500 Received: from web21109.mail.yahoo.com (web21109.mail.yahoo.com [216.136.227.111]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021002154606-33652-9 ; Wed, 02 Oct 2002 15:46:07 -0500 Message-ID: <20021002203617.83706.qmail@web21109.mail.yahoo.com> Received: from [147.178.2.110] by web21109.mail.yahoo.com via HTTP; Wed, 02 Oct 2002 13:36:17 PDT MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 2 Oct 2002 15:46:09 -0500 X-OldDate: Wed, 2 Oct 2002 13:36:17 -0700 (PDT) Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Pat LaVarre To: ftp-wg@hethmon.com Subject: Ftp-WG: to mlst-16 from mlst-15 > From: Robert Elz [mailto:kre@munnari.OZ.AU] > Sent: Mon 9/23/2002 9:52 AM > Subject: Ftp-WG: MLST draft 16 finally submitted = ftp://munnari.oz.au/internet-drafts/ + draft-ietf-ftpext-mlst-16.txt.gz - draft-ietf-ftpext-mlst-15.txt.gz Apart from seeming artifacts of pagination, the differences I see are: + draft-ietf-ftpext-mlst-16.txt September 2002 + Elz & Hethmon [Expires March 2003] = - draft-ietf-ftpext-mlst-15.txt April 2002 - Elz & Hethmon [Expires October 2002] = 1. Introduction = + This document updates the File Transfer Protocol - This document amends the File Transfer Protocol = 2.2. Pathnames ... + Note: for pathnames transferred over a data + connection, there is no way to represent a pathname + containing the characters CR and LF in sequence, and + distinguish that from the end of line indication. + Hence, pathnames containing the CRLF pair of + characters cannot be transmitted over a data + connection. Data connections only contain file + names transmitted from server-FTP to user-FTP as the + result of one of the directory listing commands. + Files with names containing the CRLF sequence must + either have that sequence converted to some other + form, such that the other form can be recognised and + be correctly converted back to CRLF, or be omitted + from the listing. ... + NVT also distinguishes between CR, LF, and the end + line CRLF, and so would permit pathnames containing + the pair of characters CR and LF to be correctly + transmitted. However, because such a sequence + cannot be transmitted over a data connection (as + part of the result of a LIST, NLST, or MLSD command) + such pathnames are best avoided. = 3. File Modification Time (MDTM) ... + Because the User and server FTPs' clocks are not + necessarily synchronised, User FTPs intending to use + this method should usually obtain the modification + time of the file from the server before the initial + RETRieval, and compare that with the modification + time before a RESTart. If they differ, the files + may have changed, and RESTart would be inadvisable. + Where this is not possible, the User FTP should make + sure to allow for possible clock skew when comparing + times. = 6.1. TVFS File Names ... + With the sole exception of the "/" character, any + valid IS10646 character [11] may be used in a TVFS + file name. When transmitted, file name characters + are encoded using the UTF-8 encoding [2]. Note that + the two character sequence CR LF occurring in a file + name will make that name impossible to transmit over + a data connection. Consequently, it should be + avoided, or if that is impossible to achieve, it + MUST be encoded in some reversible way. = - With the sole exception of the "/" character, any - valid IS10646 character [11] may be used in a TVFS - file name. When transmitted, file name characters - are encoded using the UTF-8 encoding [2]. = 7.7.8. A stress test of case (in)dependence ... = Next, notice the labels of the facts. These are = also case independent strings, + Server-FTP is permitted to return them in any + case desired. - Server-FTP is permitted to return them in any - case they desire. = 11. Security Considerations ... + Server FTP should take care not to reveal sensitive + information about files to unauthorised parties. In + particular, some underlying filesystems provide a + file identifier which, if known, can allow many of + the filesystem protection mechanisms to be + by-passed. That identifier would not be a suitable + choice to use as the basis of the value of the + unique fact. +12. Normative References -12. References = Editors' Addresses + Robert Elz + Prince of Songkla University + Department of Computer Enginering + Hat Yai, 90112 + Thailand + + Email: kre@munnari.OZ.AU = - Robert Elz - University of Melbourne - Department of Computer Science - Parkville, Vic 3052 - Australia - - Email: kre@munnari.OZ.AU P.S. Courtesy Google of this list I remembered: ftp://munnari.oz.au/internet-drafts/ and so I found: ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-16.txt.gz [39075 bytes] ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-15.txt.gz [38357 bytes] ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-14.txt.Z [50360 bytes] Sun's `jar -xvf` and Microsoft's Win XP choked over these .Z and .gz compressions, but Mac OS X just plain worked. __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com From ftp-wg-owner@hethmon.com Wed Oct 2 16:55:35 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA07434 for ; Wed, 2 Oct 2002 16:55:34 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021002160051-60141-11 ; Wed, 02 Oct 2002 16:00:51 -0500 Received: from web21101.mail.yahoo.com (web21101.mail.yahoo.com [216.136.227.103]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021002160047-7850-10 ; Wed, 02 Oct 2002 16:00:48 -0500 Message-ID: <20021002205058.68727.qmail@web21101.mail.yahoo.com> Received: from [147.178.2.110] by web21101.mail.yahoo.com via HTTP; Wed, 02 Oct 2002 13:50:58 PDT MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 2 Oct 2002 16:00:50 -0500 X-OldDate: Wed, 2 Oct 2002 13:50:58 -0700 (PDT) Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Pat LaVarre To: ftp-wg@hethmon.com Subject: Ftp-WG: in mlst-16, chmod bypass, we deprecate which? > Subject: Ftp-WG: to mlst-16 from mlst-15 ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-16.txt.gz > = 11. Security Considerations > ... > + Server FTP should take care not to reveal > + sensitive information about files to unauthorised > + parties. In particular, some underlying > + filesystems provide a file identifier which, if > + known, can allow many of the filesystem protection > + mechanisms to be by-passed. That identifier would > + not be a suitable choice to use as the basis of > + the value of the unique fact. Would a classic Unix inode number be an example of an identifier that lends itself to abuse, or did we have something else in mind? Curiously yours, thanks in advance, Pat LaVarre __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com From ftp-wg-owner@hethmon.com Wed Oct 2 16:57:51 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA07562 for ; Wed, 2 Oct 2002 16:57:51 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021002160317-61357-17 ; Wed, 02 Oct 2002 16:03:17 -0500 Received: from web21102.mail.yahoo.com (web21102.mail.yahoo.com [216.136.227.104]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021002160314-36256-16 ; Wed, 02 Oct 2002 16:03:15 -0500 Message-ID: <20021002205323.62739.qmail@web21102.mail.yahoo.com> Received: from [147.178.2.110] by web21102.mail.yahoo.com via HTTP; Wed, 02 Oct 2002 13:53:23 PDT MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 2 Oct 2002 16:03:16 -0500 X-OldDate: Wed, 2 Oct 2002 13:53:23 -0700 (PDT) Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Pat LaVarre To: ftp-wg@hethmon.com Subject: Ftp-WG: in mlst-16, only Dos Eol excluded from pathnames? > Subject: Ftp-WG: to mlst-16 from mlst-15 ftp://munnari.oz.au/internet-drafts/draft-ietf-ftpext-mlst-16.txt.gz > = 2.2. Pathnames > ... > + Note: for pathnames transferred over a data > + connection, there is no way to represent a > + pathname containing the characters CR and LF in > + sequence, and distinguish that from the end of > + line indication. This is a clear, limited statement, thank you. Am I correct to think this says, by omission, that there is a correct protocol for the transmission, by control and data connection, of filenames that contain LF, or CR, or LF CR, just not CR LF? We exclude only CR LF, aka the MS Dos Eol, aka the Telnet Eol. We do not exclude the isolated LF aka Unix Eol. We do not exclude the isolated CR aka Mac Eol. We do not exclude mixtures thereof, except for CR followed directly by LF. > > there is a correct protocol And someone on Earth knows what that protocol is? > > there is a correct protocol And we don't think exercising that protocol would in fact confuse most actual Ftp clients, Ftp servers, and Ftp firewalls? Curiously yours, thanks in advance, Pat LaVarre __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com From ftp-wg-owner@hethmon.com Thu Oct 3 02:40:27 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA27729 for ; Thu, 3 Oct 2002 02:40:26 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003014549-24622-11 ; Thu, 03 Oct 2002 01:45:49 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003014535-63460-9 ; Thu, 03 Oct 2002 01:45:47 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g936YjQ29545 for ; Thu, 3 Oct 2002 13:34:46 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g936Ya706240 for ; Thu, 3 Oct 2002 13:34:39 +0700 (ICT) In-Reply-To: <20021002205058.68727.qmail@web21101.mail.yahoo.com> References: <20021002205058.68727.qmail@web21101.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <6238.1033626876@munnari.OZ.AU> Date: Thu, 3 Oct 2002 01:45:48 -0500 X-OldDate: Thu, 03 Oct 2002 13:34:36 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: in mlst-16, chmod bypass, we deprecate which? Date: Wed, 2 Oct 2002 16:00:50 -0500 From: Pat LaVarre Message-ID: <20021002205058.68727.qmail@web21101.mail.yahoo.com> | Would a classic Unix inode number be an example of an | identifier that lends itself to abuse, or did we have | something else in mind? No, and yes - NFS file handles. kre From ftp-wg-owner@hethmon.com Thu Oct 3 02:47:53 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA27819 for ; Thu, 3 Oct 2002 02:47:52 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003015318-25017-14 ; Thu, 03 Oct 2002 01:53:18 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003015219-10126-14 ; Thu, 03 Oct 2002 01:53:15 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g936fLQ01226 for ; Thu, 3 Oct 2002 13:41:23 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g936fF706255 for ; Thu, 3 Oct 2002 13:41:15 +0700 (ICT) In-Reply-To: <20021002205323.62739.qmail@web21102.mail.yahoo.com> References: <20021002205323.62739.qmail@web21102.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <6253.1033627275@munnari.OZ.AU> Date: Thu, 3 Oct 2002 01:53:16 -0500 X-OldDate: Thu, 03 Oct 2002 13:41:15 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: in mlst-16, only Dos Eol excluded from pathnames? Date: Wed, 2 Oct 2002 16:03:16 -0500 From: Pat LaVarre Message-ID: <20021002205323.62739.qmail@web21102.mail.yahoo.com> | Am I correct to think this says, by omission, that | there is a correct protocol for the transmission, by | control and data connection, of filenames that contain | LF, or CR, or LF CR, just not CR LF? Yes, because CR LF is the ascii mode ftp file transfer EOL signal. | We exclude only | CR LF, aka the MS Dos Eol, aka the Telnet Eol. We do | not exclude the isolated LF aka Unix Eol. We do not | exclude the isolated CR aka Mac Eol. We do not | exclude mixtures thereof, except for CR followed | directly by LF. Correct. But nor do we require anyone to actually support those oddities. They're just possible to transmit according to the protocol, should some filesystem be odd enough to use them. | And someone on Earth knows what that protocol is? Yes, you send the bytes. | And we don't think exercising that protocol would in | fact confuse most actual Ftp clients, Ftp servers, and | Ftp firewalls? Yes, I do expect that it would confuse some. They're all broken and always have been. There's nothing now in FTP that prohibits any of these odd filenames, other than what the servers happen to support. As long as servers go on supporting the same basic kinds of filenames that they currently support, and clients keep on expecting what they currently expect, the abstract possibility of weird characters appearing in filenames will continue to exist with about the same possibility of actually being encountered in the wild as now does. kre From ftp-wg-owner@hethmon.com Thu Oct 3 02:57:46 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id CAA27945 for ; Thu, 3 Oct 2002 02:57:45 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003020311-22484-10 ; Thu, 03 Oct 2002 02:03:11 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003020303-61354-9 ; Thu, 03 Oct 2002 02:03:09 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g936qdQ04260 for ; Thu, 3 Oct 2002 13:52:40 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g936qW706273 for ; Thu, 3 Oct 2002 13:52:32 +0700 (ICT) In-Reply-To: <20021002203617.83706.qmail@web21109.mail.yahoo.com> References: <20021002203617.83706.qmail@web21109.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <6271.1033627952@munnari.OZ.AU> Date: Thu, 3 Oct 2002 02:03:10 -0500 X-OldDate: Thu, 03 Oct 2002 13:52:32 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: to mlst-16 from mlst-15 Date: Wed, 2 Oct 2002 15:46:09 -0500 From: Pat LaVarre Message-ID: <20021002203617.83706.qmail@web21109.mail.yahoo.com> | Apart from seeming artifacts of pagination, the | differences I see are: Thanks for that... And yes, I believe those are the changes (and I think they'd all been mentioned on the list before, at least in principle). | + This document updates the File Transfer Protocol | - This document amends the File Transfer Protocol That one was actually an IESG(AD) directive... | = 2.2. Pathnames | + Note: for pathnames transferred over a data | + connection, there is no way to represent a pathname | + containing the characters CR and LF in sequence, [...] This was discussed at some length on the list. | = 3. File Modification Time (MDTM) | + Because the User and server FTPs' clocks are not | + necessarily synchronised, [...] This was another IESG (AD?) request. | = 6.1. TVFS File Names | + [...] Note that | + the two character sequence CR LF occurring in a file | + name will make that name impossible to transmit over | + a data connection. Consequently, it should be | + avoided, or if that is impossible to achieve, it | + MUST be encoded in some reversible way. As above. | = 7.7.8. A stress test of case (in)dependence | + Server-FTP is permitted to return them in any | + case desired. | - Server-FTP is permitted to return them in any | - case they desire. Grammar... (hardly a change of substance...) | = 11. Security Considerations | + Server FTP should take care not to reveal sensitive | + information about files to unauthorised parties. [...] Another IESG (AD) request. | +12. Normative References | -12. References A pre-emptive strike (attempt to avoid a later holdup...) That one actually wasn't ever mentioned on the list, now I think of it. If someone wants to determine that any of the refs aren't normative, be my guest... | = Editors' Addresses Definitely not a substantive change... | P.S. Courtesy Google of this list I remembered: | ftp://munnari.oz.au/internet-drafts/ Yes. I tend to keep ancient copies of anything I find personally interesting, sometimes old copies of lots... | Sun's `jar -xvf` and Microsoft's Win XP choked over | these .Z and .gz compressions, but Mac OS X just plain worked. The .Z is ancient unix "compress" (LZ) compression. .gz is gzip. Now munnari has more horsepower, and less work to do than it used to have, I will probably (eventually) enable decompress on the fly for these files, so you can just fetch without the ".Z" or ".gz" and you'd get the TXT version returned. (I guess I could also make it do that if someone attempted an ACSII mode file transfer, but that might just violate the principle of least surprise, even if it would be more useful than just transferring a useless hunk of mangled data...) kre ps: the "IESG (AD)" above is because it isn't always clear which messages are coming from the IESG as a whole, which from a part of it, or which just from the AD (not that there's any reason it has to be either). From ftp-wg-owner@hethmon.com Thu Oct 3 09:52:07 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id JAA09100 for ; Thu, 3 Oct 2002 09:52:06 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003085715-23125-10 ; Thu, 03 Oct 2002 08:57:15 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003085713-8587-9 ; Thu, 03 Oct 2002 08:57:13 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 018327 for ; Thu, 3 Oct 2002 09:47:17 -0400 (EDT) Message-Id: <4.3.2.7.2.20021003084552.023048f8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <6271.1033627952@munnari.OZ.AU> References: <20021002203617.83706.qmail@web21109.mail.yahoo.com> <20021002203617.83706.qmail@web21109.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Thu, 3 Oct 2002 08:57:14 -0500 X-OldDate: Thu, 03 Oct 2002 08:50:15 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions At 02:03 AM 10/3/2002, you wrote: >The .Z is ancient unix "compress" (LZ) compression. .gz is gzip. Winzip and several other tools handle .Z quite comfortably, ancient though it is. Though I can remember back in the days of DOS that there wasn't enough memory to do 16-bit compression :-) >Now munnari has more horsepower, and less work to do than it used to >have, I will probably (eventually) enable decompress on the fly for >these files, so you can just fetch without the ".Z" or ".gz" and you'd >get the TXT version returned. (I guess I could also make it do that >if someone attempted an ACSII mode file transfer, but that might just >violate the principle of least surprise, even if it would be more useful >than just transferring a useless hunk of mangled data...) It might be more appropriate to simply refuse to transfer such a file in ASCII mode. To be honest, I've half-toyed with the idea of ignoring RFC 959's requirement that the default transfer method be ASCII - but then again, on a Windows system, where the EOL translation is a null one, it's not terribly important. It's about time that the MLST draft made it into RFC-hood, though. I haven't seen any major changes in quite some time, and what changes I can remember have been well within the range of what normal RFC readers would have figured out for themselves. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Thu Oct 3 10:14:34 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA11318 for ; Thu, 3 Oct 2002 10:14:33 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003092000-1292-15 ; Thu, 03 Oct 2002 09:20:00 -0500 Received: from watsun.cc.columbia.edu (watsun.cc.columbia.edu [128.59.39.2]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003091958-51391-14 ; Thu, 03 Oct 2002 09:19:58 -0500 Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA03671 for FTPEXT Working Group ; Thu, 3 Oct 2002 10:10:11 -0400 (EDT) In-Reply-To: Your message of Thu, 3 Oct 2002 08:57:14 -0500 Message-ID: Date: Thu, 3 Oct 2002 09:19:59 -0500 X-OldDate: Thu, 3 Oct 2002 10:10:10 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Jeffrey Altman To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions before you come to conclusions about mlst please read http://www.kermit-project.org/newftp.html in particular the section on NLST vs MLSD --- sorry I have been so quite recently but I spent a few weeks in and out of surgery. From ftp-wg-owner@hethmon.com Thu Oct 3 10:29:17 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA12581 for ; Thu, 3 Oct 2002 10:29:17 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003093443-1305-20 ; Thu, 03 Oct 2002 09:34:44 -0500 Received: from watsun.cc.columbia.edu (watsun.cc.columbia.edu [128.59.39.2]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003093441-40717-15 ; Thu, 03 Oct 2002 09:34:42 -0500 Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA05737 for FTPEXT Working Group ; Thu, 3 Oct 2002 10:24:53 -0400 (EDT) In-Reply-To: Your message of Thu, 3 Oct 2002 09:19:59 -0500 Message-ID: Date: Thu, 3 Oct 2002 09:34:42 -0500 X-OldDate: Thu, 3 Oct 2002 10:24:52 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Jeffrey Altman To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions > before you come to conclusions about mlst please read > > http://www.kermit-project.org/newftp.html > > in particular the section on NLST vs MLSD > > --- > > sorry I have been so quite recently but I spent a few weeks in and out > of surgery. > > I apologize for this e-mail going to the list. It was meant to be a private reply. I didn't notice the Reply-To: header .... From ftp-wg-owner@hethmon.com Thu Oct 3 10:45:07 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA13836 for ; Thu, 3 Oct 2002 10:45:06 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003095032-59770-10 ; Thu, 03 Oct 2002 09:50:32 -0500 Received: from web21101.mail.yahoo.com (web21101.mail.yahoo.com [216.136.227.103]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003095030-35758-9 ; Thu, 03 Oct 2002 09:50:31 -0500 Message-ID: <20021003144044.33309.qmail@web21101.mail.yahoo.com> Received: from [147.178.2.110] by web21101.mail.yahoo.com via HTTP; Thu, 03 Oct 2002 07:40:44 PDT MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 3 Oct 2002 09:50:31 -0500 X-OldDate: Thu, 3 Oct 2002 07:40:44 -0700 (PDT) Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Pat LaVarre To: ftp-wg@hethmon.com Subject: Ftp-WG: to mlst-16, an IESG(AD) directive > From: Robert Elz [kre@munnari.OZ.AU] > Subject: ... chmod bypass, we deprecate which? > ... NFS file handles ... ... > From: Robert Elz [kre@munnari.OZ.AU] > Subject: ... only Dos Eol excluded from pathnames? > ... weird characters appearing in filenames ... > about the same possibility of actually > being encountered in the wild as now does ... Thanks again for clueing me in. > an IESG(AD) directive... Should I know what an "IESG(AD) directive" is? http://www.google.com doesn't immediately, though clearly other folk on the web do e.g. "provide on demand the names of the quoted individuals to the ADs or the IESG if an AD or the IESG needs to cross reference or ask further questions." Thanks again in advance, curiously yours, Pat LaVarre __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com From ftp-wg-owner@hethmon.com Thu Oct 3 10:46:01 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA13899 for ; Thu, 3 Oct 2002 10:46:01 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003095123-58968-15 ; Thu, 03 Oct 2002 09:51:24 -0500 Received: from web21107.mail.yahoo.com (web21107.mail.yahoo.com [216.136.227.109]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003095122-33808-14 ; Thu, 03 Oct 2002 09:51:22 -0500 Message-ID: <20021003144135.21198.qmail@web21107.mail.yahoo.com> Received: from [147.178.2.110] by web21107.mail.yahoo.com via HTTP; Thu, 03 Oct 2002 07:41:35 PDT MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 3 Oct 2002 09:51:22 -0500 X-OldDate: Thu, 3 Oct 2002 07:41:35 -0700 (PDT) Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Pat LaVarre To: ftp-wg@hethmon.com Subject: Ftp-WG: nlst or mlsd via mget > From: Jeffrey Altman [mailto:jaltman@columbia.edu] > before you come to conclusions > about mlst please read > http://www.kermit-project.org/newftp.html > in particular the section on NLST vs MLSD Perhaps the "in particular" is: http://www.kermit-project.org/newftp.html#mget Interesting, thank you. I see an argument that MLSD is unusable for people, in part because it lacks wildcard support, so to retain the command-line usability of NLST we need to establish an MGET that may be NLST or MLSD, depending. Pat LaVarre __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com From ftp-wg-owner@hethmon.com Thu Oct 3 10:47:29 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA14015 for ; Thu, 3 Oct 2002 10:47:28 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003095255-59901-23 ; Thu, 03 Oct 2002 09:52:55 -0500 Received: from web21109.mail.yahoo.com (web21109.mail.yahoo.com [216.136.227.111]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003095253-34461-22 ; Thu, 03 Oct 2002 09:52:53 -0500 Message-ID: <20021003144306.46819.qmail@web21109.mail.yahoo.com> Received: from [147.178.2.110] by web21109.mail.yahoo.com via HTTP; Thu, 03 Oct 2002 07:43:06 PDT MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 3 Oct 2002 09:52:54 -0500 X-OldDate: Thu, 3 Oct 2002 07:43:06 -0700 (PDT) Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Pat LaVarre To: ftp-wg@hethmon.com Subject: Ftp-WG: Compressions and conversions > > Sun's `jar -xvf` and Microsoft's Win XP > > choked over these .Z and .gz compressions, > > but Mac OS X just plain worked. > Subject: Ftp-WG: Compressions and conversions > ... Winzip and several other tools > handle .Z quite comfortably, ... I'm thinking Windows doesn't work out of the box with .Z and .gz, and can't be quickly & freely upgraded to cope. Is that wrong? Last I checked, Winzip was now even less free than it used to be? Just now I tried renaming .Z and .gz to .zip, which in some versions of Windows invokes auto .zip handling ... here Windows says "invalid or corrupt" archive. Last time I inflated .Z on Windows, I think I downloaded, in multiple steps involving multiple megabytes, a version of Cygwin, possibly over the 19.2K line I had in the Grand Tetons. I hear a thing termed Mingw is now more in vogue: we wouldn't want technique learned to be valid for an extended period of time, would we. Once upon a time I found my way to a free zip effort, named something like InfoZip, out of which I heard the free Sun jar tool grew? InfoZip didn't work too well for me then: I commonly encountered .zip's it couldn't handle. As far as I know, all the legitimate distributions of the free Sun jar also involve multiple megabytes and trusting an installation utility. Currently on Windows I use a local copy of a 21 megabyte subset of a normal Sun JavaDevelopmentKit, with drag & drop installation. It's not that I can't defeat the popular compression schemes. I just can't do so easily, in a hassle free way I can share with technically illiterate friends across the globe. Pat LaVarre P.S. I tend to think the browser should cope, anyhow. Like I care how the data is compressed. __________________________________________________ Do you Yahoo!? New DSL Internet Access from SBC & Yahoo! http://sbc.yahoo.com From ftp-wg-owner@hethmon.com Thu Oct 3 11:34:27 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA18490 for ; Thu, 3 Oct 2002 11:34:27 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003103949-15951-10 ; Thu, 03 Oct 2002 10:39:49 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003103946-52488-9 ; Thu, 03 Oct 2002 10:39:47 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 119594 for ; Thu, 3 Oct 2002 11:29:46 -0400 (EDT) Message-Id: <4.3.2.7.2.20021003101650.02277d40@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <20021003144306.46819.qmail@web21109.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Thu, 3 Oct 2002 10:39:47 -0500 X-OldDate: Thu, 03 Oct 2002 10:30:25 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions At 09:52 AM 10/3/2002, you wrote: >I'm thinking Windows doesn't work out of the box with >.Z and .gz, and can't be quickly & freely upgraded to >cope. > >Is that wrong? > >Last I checked, Winzip was now even less free than it >used to be? http://www.winzip.com, click on "Download Evaluation Version" - I didn't see anything about it being any less free to try out - the nag screen is still there. Maybe you mean it's more expensive if you do decide to register? It's $29 - hardly "break the bank". >Just now I tried renaming .Z and .gz to .zip, which in >some versions of Windows invokes auto .zip handling >... here Windows says "invalid or corrupt" archive. Because it's not a .zip file. Microsoft's handling of zip files is only for zip files. >Last time I inflated .Z on Windows, I think I >downloaded, in multiple steps involving multiple >megabytes, a version of Cygwin, possibly over the >19.2K line I had in the Grand Tetons. I hear a thing >termed Mingw is now more in vogue: we wouldn't want >technique learned to be valid for an extended period >of time, would we. There's any number of uncompress tools available for Windows, both in GUIs (many zip, RAR, ACE, etc extractors will also handle other formats including Unix compress) and command lines - for instance, the GNUish project, and any number of DOS utilities found at archives such as Simtel, etc. Presumably there's similar tools for other platforms. But discussing how to find tools that you need is more appropriate on other forums than this one. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Thu Oct 3 20:44:07 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA09667 for ; Thu, 3 Oct 2002 20:44:07 -0400 (EDT) Date: Thu, 3 Oct 2002 20:44:07 -0400 (EDT) From: ftp-wg-owner@hethmon.com Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003194923-34948-10 ; Thu, 03 Oct 2002 19:49:23 -0500 Received: from hra.admin.uch.gr (hra.admin.uoc.gr [147.52.247.200]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021003194915-16401-9 ; Thu, 03 Oct 2002 19:49:16 -0500 Received: from cellnetwork.com [202.125.136.29] by hra.admin.uch.gr with ESMTP (SMTPD32-4.06) id AF45215B037A; Fri, 04 Oct 2002 03:39:01 +0100 Message-ID: <00002e926308$0000715f$00001a59@asys-h.de> rowes@fwi.com> , , , , , , , , , , , , , , , , , , , , , , MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Mailer: AOL 5.0 for Mac sub 7 Date: Thu, 3 Oct 2002 19:49:21 -0500 X-OldDate: Thu, 03 Oct 2002 20:37:45 -1600 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "DeForest Kelly" To: ,,,,,,,,,,,,,,,,,,,,,,, http://www.nucleardefense.net/ <- or CUT & PASTE the above link ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ in your browser URL bar! Q: GREAT, THANKS! Q: One more question, how do I get REMOVED from this DARN EMAIL LIST? A: That is NOT a problem. Click here -> mailto:info@nucleardefense.net?Subject=Remove <- And you will be removed within the legal period of 5 business days. ISAPP opp code djT*&278 From ftp-wg-owner@hethmon.com Fri Oct 4 11:10:58 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA07395 for ; Fri, 4 Oct 2002 11:10:58 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004102104-46982-9 ; Fri, 04 Oct 2002 10:21:04 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004102059-60502-5 ; Fri, 04 Oct 2002 10:21:00 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g94FB1C29714 for ; Fri, 4 Oct 2002 22:11:02 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g94FAd713937 for ; Fri, 4 Oct 2002 22:10:43 +0700 (ICT) In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <13935.1033744239@munnari.OZ.AU> Date: Fri, 4 Oct 2002 10:21:03 -0500 X-OldDate: Fri, 04 Oct 2002 22:10:39 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions Date: Thu, 3 Oct 2002 09:34:42 -0500 From: Jeffrey Altman Message-ID: | > before you come to conclusions about mlst please read | > | > http://www.kermit-project.org/newftp.html | > | > in particular the section on NLST vs MLSD | > | > --- | > | > sorry I have been so quite recently but I spent a few weeks in and out | > of surgery. | > | > | | I apologize for this e-mail going to the list. It was meant to be a | private reply. I didn't notice the Reply-To: header .... Actually, bringing Frank Da Cruz' thoughts out into the open is something that should be done anyway - he has been sending me (and Paul) private mail about some of the issues on his web page - we have been encouraging him to send them to the list instead, but so far, without success. kre From ftp-wg-owner@hethmon.com Fri Oct 4 11:18:28 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA07642 for ; Fri, 4 Oct 2002 11:18:28 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004102357-23483-18 ; Fri, 04 Oct 2002 10:23:57 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004102353-62319-13 ; Fri, 04 Oct 2002 10:23:55 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g94FDsC29786 for ; Fri, 4 Oct 2002 22:13:54 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g94FDa713950 for ; Fri, 4 Oct 2002 22:13:36 +0700 (ICT) In-Reply-To: <20021003144044.33309.qmail@web21101.mail.yahoo.com> References: <20021003144044.33309.qmail@web21101.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <13948.1033744416@munnari.OZ.AU> Date: Fri, 4 Oct 2002 10:23:56 -0500 X-OldDate: Fri, 04 Oct 2002 22:13:36 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: to mlst-16, an IESG(AD) directive Date: Thu, 3 Oct 2002 09:50:31 -0500 From: Pat LaVarre Message-ID: <20021003144044.33309.qmail@web21101.mail.yahoo.com> | Should I know what an "IESG(AD) directive" is? Possibly not unless you've been involved in this stuff for a while. It is when the IESG (or an Area Director) says "we won't allow this to be published unless you make this change". Sometimes they're for good reasons, sometimes not... But unless the request actually compromises the integrity of the work (which something like changing "amends" into "updates" certainly does not) they aren't worth arguing about. The IESG are the body who get (more or less) the final word on whether or not an internet draft turns into an RFC, and becomes a proposed standard. kre From ftp-wg-owner@hethmon.com Fri Oct 4 11:23:17 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA07795 for ; Fri, 4 Oct 2002 11:23:16 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004102846-46817-15 ; Fri, 04 Oct 2002 10:28:46 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004102844-57974-14 ; Fri, 04 Oct 2002 10:28:44 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 048852 for ; Fri, 4 Oct 2002 11:18:49 -0400 (EDT) Message-Id: <4.3.2.7.2.20021004102028.020ef130@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <13935.1033744239@munnari.OZ.AU> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Fri, 4 Oct 2002 10:28:45 -0500 X-OldDate: Fri, 04 Oct 2002 10:21:57 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions At 10:21 AM 10/4/2002, you wrote: >Actually, bringing Frank Da Cruz' thoughts out into the open is >something that should be done anyway - he has been sending me (and Paul) >private mail about some of the issues on his web page - we have been >encouraging him to send them to the list instead, but so far, without >success. Absolutely - after all, for those of us that _aren't_ in frequent communication with him, we're of the opinion that there's no earthly reason to hold up the RFC process on this draft, and we're upset that it's taken so long already. If there are significant problems to be addressed, then by all means the draft needs to be re-draughted until the problems go away. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Fri Oct 4 11:33:53 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA08151 for ; Fri, 4 Oct 2002 11:33:52 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004103922-59063-10 ; Fri, 04 Oct 2002 10:39:22 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004103917-61444-10 ; Fri, 04 Oct 2002 10:39:18 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g94FTPC00365 for ; Fri, 4 Oct 2002 22:29:25 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g94FT6713982 for ; Fri, 4 Oct 2002 22:29:06 +0700 (ICT) In-Reply-To: <20021003144135.21198.qmail@web21107.mail.yahoo.com> References: <20021003144135.21198.qmail@web21107.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <13980.1033745346@munnari.OZ.AU> Date: Fri, 4 Oct 2002 10:39:21 -0500 X-OldDate: Fri, 04 Oct 2002 22:29:06 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: nlst or mlsd via mget Date: Thu, 3 Oct 2002 09:51:22 -0500 From: Pat LaVarre Message-ID: <20021003144135.21198.qmail@web21107.mail.yahoo.com> | Interesting, thank you. I see an argument that MLSD | is unusable for people, in part because it lacks | wildcard support, so to retain the command-line | usability of NLST we need to establish an MGET that | may be NLST or MLSD, depending. Personally, I think he's gone overboard on options (but then again, kermit was always the ultimate in absolute flexibility, able to do just about anything, almost any conceivable way). The WG discussed wildcarding long long long ago, and decided that it belongs in the client. Current uses of wildcards in FTP depend upon the implementation of the original unix server (which simply ran a shell to run the ls command, passing through whatever arg NLST or LIST was sent - hence the ability to send "LIST -lt" not find out about a file called "-lt" but instead get a listing of the current directory sorted by modify time - and if the arg happened to be expanded by the shell, then that's what happened). None of this stuff is in any way standard, nor is it documented anywhere, and IMNSHO, should never be, it is an utter croc of ... NLST isn't even intended to be able to return data on normal files, The pathname should specify a directory or other system-specific file group descriptor; only directories (or other "file containers" for systems that have other kinds of filesystems, like "mini-disks"). It is quite clear from 959 that the intent is that the client do a NLST of a directory, get the results, and use those to select files to obtain for a "multiple get" or similar command ... You're not supposed to send "NLST m*" and then do a RETR on ever file name returned. Unix has contributed a lot to the internet, but its mangling of the FTP protocol isn't one of its brighter lights. The only issue Frank raises that is really relevant, is whether or not recursive listings should be implemented via a command, rather than requiring the client to do lots of MLSD commands, a new one for each directory. I don't think there was ever a WG position on that (that I remember anyway), beyond "not just yet please". That is, were someone to actually do the work of defining the format for a recursive listing response (something that can be properly parsed at the receiver), I don't expect it would get too many objections. On the other hand, implementing this at the client really isn't hard, and really doesn't need a new open file (file descriptor) for every directory encountered (meaning that there's a limit on the size, or depth, of the recursion imposed by the number of file descriptors available to the client implementation). That's just a cheap simple implementation technique, but others (such as doing breadth first recursion, instead of depth first) work just as well, and aren't hard to implement. kre From ftp-wg-owner@hethmon.com Fri Oct 4 12:34:15 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id MAA12265 for ; Fri, 4 Oct 2002 12:34:15 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004113942-26908-12 ; Fri, 04 Oct 2002 11:39:42 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021004113939-304-11 ; Fri, 04 Oct 2002 11:39:40 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g94GThC02236 for ; Fri, 4 Oct 2002 23:29:43 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g94GTN714346 for ; Fri, 4 Oct 2002 23:29:23 +0700 (ICT) In-Reply-To: <4.3.2.7.2.20021004102028.020ef130@208.55.91.110> References: <4.3.2.7.2.20021004102028.020ef130@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <14344.1033748963@munnari.OZ.AU> Date: Fri, 4 Oct 2002 11:39:41 -0500 X-OldDate: Fri, 04 Oct 2002 23:29:23 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: Compressions and conversions Date: Fri, 4 Oct 2002 10:28:45 -0500 From: Alun Jones Message-ID: <4.3.2.7.2.20021004102028.020ef130@208.55.91.110> | and we're upset that it's taken so long already. Yes... Some of that is my fault, for which I apologise (like this most recent draft should have been out about 6-7 weeks earlier), other delays have been elsewhere... kre From ftp-wg-owner@hethmon.com Tue Oct 8 14:53:41 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA26197 for ; Tue, 8 Oct 2002 14:53:40 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008140324-55445-12 ; Tue, 08 Oct 2002 14:03:25 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008135932-25415-11 ; Tue, 08 Oct 2002 13:59:33 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 1835081 for ; Tue, 8 Oct 2002 14:49:26 -0400 (EDT) Message-Id: <4.3.2.7.2.20021008105015.020305d8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 8 Oct 2002 13:59:41 -0500 X-OldDate: Tue, 08 Oct 2002 11:34:07 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. I get a lot of requests from users for the ability to "pop up a user / password dialog box". Obviously, this isn't anything to do with my server's implementation which, like other servers, _always_ greets with a 220, and responds to a USER command by asking for a PASS command. On a little investigation, it turns out to be a direct result of the FTP URLs (a pain in themselves), and the fact that there is currently no generic URL to say that the browser's user must be prompted for user name and password. I've given this only light consideration, and wonder if something like "ftp://@sitename" or "ftp://:@sitename" might be a good start - or perhaps "ftp://?@sitename". The first two obviously conflict with RFC 1739's statement that these specify an empty username and/or password. The second requires, essentially, that no user on any FTP server have the name "?", or that it should be escaped, something that potentially could cause backwards-compatibility issues. So, since these have some problems, can anyone suggest a form of generic FTP URL that could be used to say that the browser must ask its user for a name as well as a password? Or are the problems I've described simply too insignificant, and one of the above should be adopted? Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Tue Oct 8 15:25:29 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA27625 for ; Tue, 8 Oct 2002 15:25:27 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008142915-45116-10 ; Tue, 08 Oct 2002 14:29:17 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008141905-14957-15 ; Tue, 08 Oct 2002 14:19:07 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g98J8vIj006059; Tue, 8 Oct 2002 15:08:57 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g98J8sM2006056; Tue, 8 Oct 2002 15:08:54 -0400 (EDT) Message-ID: Date: Tue, 8 Oct 2002 14:19:18 -0500 X-OldDate: Tue, 8 Oct 2002 15:08:54 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: ftp-wg@hethmon.com Subject: Ftp-WG: MLSD details Hi all, I just now joined this group. I have been working to add the new FTP protocol features from RFC 2389 as well as most of what's in the Elz and Hethmon FTP Extensions Internet Draft to the Kermit FTP client: http://www.columbia.edu/kermit/ftpclient.html (for the next Kermit release) and found that adding MLSD support to a regular command-driven NLST-using FTP client can result in surprises for the user and/or complication of the user interface, as described here in some detail: http://www.columbia.edu/kermit/newftp.html I've been having private conversations on these topics with some of you, and have been encouraged to join the group and bring the issues into the open to see if anybody else believes they should be considered before the current draft advances. Therefore I invite you to read the document at the URL immediately above and comment to the list. Meanwhile I'll try to put together suggested amendments to the draft that might be helpful. Thanks. Frank da Cruz The Kermit Project Columbia University New York City http://www.columbia.edu/kermit/ From ftp-wg-owner@hethmon.com Tue Oct 8 15:37:35 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA28201 for ; Tue, 8 Oct 2002 15:37:34 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008144046-41256-13 ; Tue, 08 Oct 2002 14:40:48 -0500 Received: from watsun.cc.columbia.edu (watsun.cc.columbia.edu [128.59.39.2]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008142444-19411-11 ; Tue, 08 Oct 2002 14:24:48 -0500 Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id PAA14847; Tue, 8 Oct 2002 15:14:30 -0400 (EDT) In-Reply-To: Your message of Tue, 8 Oct 2002 13:59:41 -0500 Message-ID: Date: Tue, 8 Oct 2002 14:24:54 -0500 X-OldDate: Tue, 8 Oct 2002 15:14:30 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Jeffrey Altman To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. What if you simply provide an option to refuse "anonymous" as the username and return an appropriate error code? > I get a lot of requests from users for the ability to "pop up a user / > password dialog box". Obviously, this isn't anything to do with my > server's implementation which, like other servers, _always_ greets with a > 220, and responds to a USER command by asking for a PASS command. On a > little investigation, it turns out to be a direct result of the FTP URLs (a > pain in themselves), and the fact that there is currently no generic URL to > say that the browser's user must be prompted for user name and password. > > I've given this only light consideration, and wonder if something like > "ftp://@sitename" or "ftp://:@sitename" might be a good start - or perhaps > "ftp://?@sitename". The first two obviously conflict with RFC 1739's > statement that these specify an empty username and/or password. The second > requires, essentially, that no user on any FTP server have the name "?", or > that it should be escaped, something that potentially could cause > backwards-compatibility issues. > > So, since these have some problems, can anyone suggest a form of generic > FTP URL that could be used to say that the browser must ask its user for a > name as well as a password? Or are the problems I've described simply too > insignificant, and one of the above should be adopted? > > Alun. > ~~~~ > -- > Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at > 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com > Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to > Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. > > Jeffrey Altman * Sr.Software Designer Kermit 95 2.0 GUI available now!!! The Kermit Project @ Columbia University SSH, Secure Telnet, Secure FTP, HTTP http://www.kermit-project.org/ Secured with MIT Kerberos, SRP, and kermit-support@columbia.edu OpenSSL. From ftp-wg-owner@hethmon.com Tue Oct 8 15:50:01 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA28539 for ; Tue, 8 Oct 2002 15:49:59 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008145318-60953-16 ; Tue, 08 Oct 2002 14:53:21 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008143114-31823-12 ; Tue, 08 Oct 2002 14:31:16 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g98JKsIj008886 for ; Tue, 8 Oct 2002 15:20:54 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g98JKsoI008885 for FTPEXT Working Group ; Tue, 8 Oct 2002 15:20:54 -0400 (EDT) In-Reply-To: Your message of Tue, 8 Oct 2002 13:59:41 -0500 Message-ID: Date: Tue, 8 Oct 2002 14:31:20 -0500 X-OldDate: Tue, 8 Oct 2002 15:20:54 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. > So, since these have some problems, can anyone suggest a form of generic > FTP URL that could be used to say that the browser must ask its user for a > name as well as a password? Or are the problems I've described simply too > insignificant, and one of the above should be adopted? > Wouldn't the syntax be: ftp://[user[:password]]@host[:service][/path...] ? In which case if user is specified but no password, the client could prompt for the password. (Or, theoretically, also vice-versa :-) It's implemented like this in the Kermit ftp client. - Frank From ftp-wg-owner@hethmon.com Tue Oct 8 16:35:08 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA29879 for ; Tue, 8 Oct 2002 16:35:07 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008153842-64077-12 ; Tue, 08 Oct 2002 15:38:45 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008152040-35107-9 ; Tue, 08 Oct 2002 15:20:43 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 026017 for ; Tue, 8 Oct 2002 16:09:58 -0400 (EDT) Message-Id: <4.3.2.7.2.20021008145904.01e0baf8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 8 Oct 2002 15:20:49 -0500 X-OldDate: Tue, 08 Oct 2002 15:00:44 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. At 02:24 PM 10/8/2002, you wrote: >What if you simply provide an option to refuse "anonymous" as the >username and return an appropriate error code? I've suggested that to each of the users that want the pop-up, but there are some who want to have anonymous access _and_ user/password-based access. They want to have a URL that always pops up a dialog, even if the anonymous user is enabled at the FTP server. I know, waah, get over it, but I'm hoping that something more sensible could be achieved, and I know that there's been one or two browser authors posting here who might want to chime in. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Tue Oct 8 16:43:30 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA00184 for ; Tue, 8 Oct 2002 16:43:28 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008154654-63611-15 ; Tue, 08 Oct 2002 15:46:56 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008152906-34567-12 ; Tue, 08 Oct 2002 15:29:06 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 036595 for ; Tue, 8 Oct 2002 16:18:22 -0400 (EDT) Message-Id: <4.3.2.7.2.20021008151402.0274ec48@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 8 Oct 2002 15:29:13 -0500 X-OldDate: Tue, 08 Oct 2002 15:22:00 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. At 02:31 PM 10/8/2002, you wrote: > > So, since these have some problems, can anyone suggest a form of generic > > FTP URL that could be used to say that the browser must ask its user for a > > name as well as a password? Or are the problems I've described simply too > > insignificant, and one of the above should be adopted? > > >Wouldn't the syntax be: > > ftp://[user[:password]]@host[:service][/path...] > >? In which case if user is specified but no password, the client could >prompt for the password. (Or, theoretically, also vice-versa :-) > >It's implemented like this in the Kermit ftp client. That's correct, but the user is asking for a means to supply a URL that asks for _both_ the user name and password. "ftp://?@localhost", when fed to IE, goes away for a while (to send "USER ?" and probably a blank for the password), and then prompts with the box, but I was hoping to get something that could be agreed upon as a standard. Interestingly, "ftp://@localhost" is treated as the same as "ftp://localhost", which is a no-no according to RFC 1738. Same for "ftp://:@localhost" (no surprise). I guess in many ways, we're dealing with a screwed-up implementation of FTP to begin with, from companies (Netscape _and_ Microsoft are equally messed up) that see FTP as something that they'll support if they absolutely have to, but they aren't going to bother with doing anything properly if they can take a shortcut. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Tue Oct 8 17:48:11 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA01919 for ; Tue, 8 Oct 2002 17:48:06 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008165206-50331-10 ; Tue, 08 Oct 2002 16:52:11 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021008164610-26943-10 ; Tue, 08 Oct 2002 16:46:14 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g98La1Ij026344; Tue, 8 Oct 2002 17:36:01 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g98La00M026338; Tue, 8 Oct 2002 17:36:00 -0400 (EDT) In-Reply-To: Your message of Tue, 08 Oct 2002 09:52:21 -0500 Message-ID: Date: Tue, 8 Oct 2002 16:46:25 -0500 X-OldDate: Tue, 8 Oct 2002 17:31:11 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: Alun Jones Subject: Ftp-WG: NLST vs MLSD Replying to a reply from Alun Jones to a message from me about MGET... > At 10:56 AM 10/3/2002, you wrote: > >As specified, MLSD is sufficiently unnatural to have caused the following: > > > > 1. My first implementation simply assumed that the server would treat > > incoming filespecs just like NLST did. Yes, I should have read it > > more closely. > > NLST and MLST both specify a single filename that represents a directory > "or other system-specific file group descriptor" (you _could_, I suppose, > take that to mean a wild card specification, but that's setting your > reading to match your opinion, rather than the other way around.) > Right, everybody says this and I agree in principal, but the fact is that every command-line FTP client since the beginning of time has sent wildcards to the server in the NLST command, received back a list of matching files, and then RETR'd each file in the list, with no matching of the list against a locally-held copy of the wildcard, or any expecation that such matching was needed. Everybody says that Postel, et al, did not intend this, which might well be true, yet server-side expansion of NLST wildcards is current practice -- not just in Unix but as far as I can tell, everywhere: VMS, TOPS-20, VM/CMS, you name it. > > 2. My first implementation worked as expected against the two server > > implementations I could find: Ipswitch and NetBSD. Apparently the > > other implementors made the same assumptions I did. > > Apparently two other implementors added functionality that matched your > assumption. Tell me - what do they do when you provide a wildcard that > produces results in more than one directory? What do they list as cdir, > pdir, etc? Can you predict this and rely on the answer? > I doubt it! As is clearly evident in the real world; specs are one thing, implementations another. Since software development is uncontrolled, the only way to write a client that "just works" these days is to experiment with every existing server, catalog its quirks, and program around them. In any case, my concern is that the new spec conflicts with existing practice. For decades it has been possible for FTP clients to execute commands like: mget pics/mae-west-*.jpg or even: mget pics*/mae-west-*.jpg or: mget pics/*/mae-west-*.jpg and get at least a semi-expected result. This is about to change. > >In other words, server-side wildcard expansion is what people expect. > > Yes, but it's not what is documented. People expect that their computers > will understand what they mean rather than what they say, and they get > frustrated when the opposite is true. Expectations should be met where it > is reasonable (and possible) to do so, but features should not arrive > simply by accident - they should be specified. > Agreed, and I also agree the previous specification could not have been sufficiently clear if everybody misimplemented it. > >I had another minor quibble with them about the pitfalls of doing > >recursive MGETs under their model, but it's more theoretical than practical, > >and won't matter unless downloading a VERY deep directory (and, in a pinch, > >can be programmed around in the client, though I wouldn't like to do it). > > Recursive descent of subdirectories is a messy issue in any system. But > very necessary for mirroring and backup purposes. In such a situation, of > course, wild cards would likely be less used, and the directories would > usually be downloaded in full (in which case, the relative size of the > listing as compared to the files would generally be small). > I don't think it's our business to decide what recursion will be used for. Users can be creative; why limit their choices? I can think of all sorts of scenarios where I might want get a directory tree with certain files included (e.g. source code) but not other files (e.g. object modules). I also do not agree, as some suggest, that directory trees or other collections of files should just be "zipped up" and transferred all in one piece. Not all platforms support zip, gzip, bz2, tar, etc, and even when they do, the Zipped (or whatever) files, when unzipped on a different platform, are likely to have the wrong record format or character set. Thus the file transfer client/server pair have to deal with directory structure, file record format, the text/binary distinction, even text-file character sets themselves. > MLSD has the concept of "current directory", which certain wildcard > expansions make void. Now, we _could_ say that we only support wildcard > expansions of file names, but then one of the curious selling points of our > software in certain markets is that wildcard expansions happen at all > points along a file path. If I implement wildcard expansions in MLSD, I > want it to be done by following the standard, not by mimicking the way some > random operating system does it, and hope that this becomes the de-facto > standard, as it did with NLST. > Clearly it deserves some serious thought; that's what we're for. > >The real point is that "who expands wildcards" has a fundamental impact > >on the user, as noted in my paper. Once FTP servers and clients start > >supporting MLSD alongside NLST, countless FTP command-line client users > >will get nasty surprises and will have to be "educated" to accommodate > >themselves to new complexities dictated by network protocols that they > >should not have to worry about. > > I'd have thought that most command-line client users would be comfy with > LIST for regular directory listings, since that _is_ supposed to be a > user-readable format, not machine-readable, and that MGET would be > implemented through NLST. You obviously use other commands around your > MGET implementation to provide information such as the file size and date, > and I can appreciate that you might not want to download a list of many > thousands of files (though how many users have such a directory designed to > be accessible through FTP over slow modem lines?), or run these commands > several times. > Well, I'm totally nauseated at the idea of the FTP client itself using the LIST output from the server to construct a file list for MGET or for GUI clicking by parsing who-knows-how many platform-specific directory listing formats, especially the silly Unix one with its impossible date/time formats. The MLST format for machine-readable file lists should have been part of the FTP spec from the beginning. The question at hand is do we have to change the fundamental relationship between FTP client and server in order to get the very nice and handy MLST file-list format? (I'm alluding to, rather than restating, specific issues I listed in the "newftp" paper on the Kermit website.) > >Since server-side wildcard expansion has been the norm for decades, I think > >that any new protocol to replace NLST should be compatible in behavior by > >default in this respect. That's a basic principle of software engineering. > > It's never been discussed or debated, AFAICT. Server-side wildcard > expansion happened, not because the RFC mandated it, but because the FTP > server authors on Unix couldn't be bothered to repeat the 'ls' code, and > simply called out to the executable (which, incidentally, is why even a > chrooted FTP server on Unix _must_ have a 'bin' directory containing 'ls', > and an 'etc' directory containing 'passwd'). If wildcards are to be > supported, it _must_ be by specification, not by simple adoption of the > most popular accidental implementation. (We can all guess whose > implementation would be adopted in such a case, and it's unlikely to be the > best engineering solution.) > As noted in my paper, server-side wildcard expansion has certain benefits that we have all become accustomed to: . The size of the file lists returned by an MGET request are controlled by the user, by tuning the wildcard (in the "mae-west*.jpg" example above, suppose these pictures were in a directory that contained a million files, but you only wanted three of them). . The wildcard filespec to obtain a certain group of files is the same for all clients and users. Thus the FTP site admin can give a simple one-line formula for getting the files. They can even arrange their FTP site (e.g. by file naming conventions) that allow easy selection of groups of files by wildcard. . Certain platforms or FTP servers have unique capabilities that you might want to take advantage of if you know what they are. > >Client-side wildcard expansion is a big change and should be a conscious > >choice of the user, not a surprise. > > On the other hand, of course, it could make the user's experience more > comfortable - why should a user of operating system ABC be aware of > wildcard expansion syntax on every FTP system he uses? A standard > wild-card format could replace that, of course, and still allow the server > to do the expansion for the client. > That's a topic for another day. For now, there is no standard wildcard format, not even within one platform (with a few exceptions). In Unix, for example, every program that expands wildcards does it its own way, either with its own custom code, or by calling some nonstandard library or other. Thus even on the same computer, different applications use different wildcard syntax. > >To put it another way, if MLSD had been simple plug-in replacement for > >NLST that returned the file list in a better format, we'd have a better > >situation. As matters stand, it has too many "riders" attached to it to > >make it attractive to implementors, except maybe in the GUI world. > > I think that's what it was mostly designed for. Apparently there wasn't > enough representation in the command-line world. > Call me Mr. Command Line :-) > >There's nothing wrong with promoting GUI FTP as long as you don't do it > >by changing the procedural aspects in ways that make more advanced usage > >impractical, since after all, it's the advanced users who are responsible > >for all that content that mass-market consumers click on. > > Rather than have fiascos such as the RFC-standard FTP URL format that has > significant incompatibilities with the way _everyone_ has implemented it, > it's important to codify whatever's going on. Wildcards, where they are > supported, must be documented prior to implementation, not the other way > around. > Agreed. There should be no guesswork about who expands them. And heaven forbid we get into a situation where the client assumes the server does, and the server assumes the client does. As to their syntax and semantics, I think that's way beyond us. - Frank From ftp-wg-owner@hethmon.com Thu Oct 10 00:07:47 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA12724 for ; Thu, 10 Oct 2002 00:07:46 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021009231759-9925-10 ; Wed, 09 Oct 2002 23:17:59 -0500 Received: from stat.com (stat.com [207.211.1.10]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021009231749-49117-10 ; Wed, 09 Oct 2002 23:17:49 -0500 Received: from mail15a.boca15-verio.com [208.55.91.57] by stat.com (SMTPD32-7.07) id A52077A5007C; Wed, 09 Oct 2002 08:02:56 -0700 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id c53835 for ; Wed, 9 Oct 2002 11:02:52 -0400 (EDT) Message-Id: <4.3.2.7.2.20021009091954.02076c10@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 X-RBL-Warning: BADHEADERS: This E-mail was sent from a broken mail client [8004000e]. Date: Wed, 9 Oct 2002 23:17:50 -0500 X-OldDate: Wed, 09 Oct 2002 09:33:26 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 04:46 PM 10/8/2002, Frank Da Cruz wrote: > > Apparently two other implementors added functionality that matched your > > assumption. Tell me - what do they do when you provide a wildcard that > > produces results in more than one directory? What do they list as cdir, > > pdir, etc? Can you predict this and rely on the answer? > >I doubt it! As is clearly evident in the real world; specs are one thing, >implementations another. Since software development is uncontrolled, the >only way to write a client that "just works" these days is to experiment >with every existing server, catalog its quirks, and program around them. Does each MLSD output for a wildcard spec even produce a 'cdir' type? After all, the wild card isn't going to match a directory, usually, yet 7.3.1 of the MLST draft notes that an MLSD response MAY include one or more "cdir" type entries, probably because the authors anticipated that MLSD's argument would always be a directory that would be listed in that 'cdir' entry. "MLSD *.txt" could produce a line that's of 'cdir' type, and whose name definitely doesn't match "*.txt". In the instance that an FTP server allows "MLSD */*.txt", it may even be useful to define that a 'cdir' entry should be listed prior to the files that match the wildcard spec in its directory. So that if "*/*.txt" matches "a/1.txt", "a/2.txt" and "b/3.txt", you'd have a 'cdir' line that specified where 'a' is, followed by listings for "1.txt" and "2.txt", then a 'cdir' line for 'b', and a listing for "3.txt". This goes against the note in 7.3.1 that the 'cdir' entry can occur anywhere in the MLSD output, because it would require that the 'cdir' line precede all matched elements in that directory. Would it be worth specifying that all 'cdir' output lines must precede other lines relevant to that directory? Here's another thought - what should MLSD do if given a wildcard that expands into several names, some of which are directories, others of which are files? Should it list the details of the directories, or the contents of those directories? We either need to specify these sorts of answers, or require that MLSD not be used with wildcards. As you point out, it's too important to rely on people fudging it through by successively refining implementations. It needs to be specified this time. >In any case, my concern is that the new spec conflicts with existing >practice. For decades it has been possible for FTP clients to execute >commands like: > > mget pics/mae-west-*.jpg > >or even: > > mget pics*/mae-west-*.jpg > >or: > > mget pics/*/mae-west-*.jpg > >and get at least a semi-expected result. This is about to change. Hardly - NLST isn't going to go away. It is still expected that NLST will be used for MGET. What you're hoping to do is to _use_ MLSD to get facts about those files before transferring them, so that you don't have to do a SIZE, MDTM, etc, for each file that you transfer (you could just do an MLST, but I can see why you'd like an MLSD with wildcards). >That's a topic for another day. For now, there is no standard wildcard >format, not even within one platform (with a few exceptions). In Unix, for >example, every program that expands wildcards does it its own way, either >with its own custom code, or by calling some nonstandard library or other. Actually, to judge from the recent problems with "../*/../*" overloads, they mostly use glob() from libc. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Thu Oct 10 11:38:57 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA08346 for ; Thu, 10 Oct 2002 11:38:51 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021010104807-52075-8 ; Thu, 10 Oct 2002 10:48:09 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021010102547-22903-10 ; Thu, 10 Oct 2002 10:25:51 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9AFFQIj000280; Thu, 10 Oct 2002 11:15:26 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9AFFOHI000266; Thu, 10 Oct 2002 11:15:24 -0400 (EDT) In-Reply-To: Your message of Wed, 9 Oct 2002 23:17:50 -0500 Message-ID: Date: Thu, 10 Oct 2002 10:25:57 -0500 X-OldDate: Thu, 10 Oct 2002 11:15:24 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At Wed, 9 Oct 2002 23:17:50 -0500, Alun wrote: > Does each MLSD output for a wildcard spec even produce a 'cdir' type? > No. > After all, the wild card isn't going to match a directory, usually, yet > 7.3.1 of the MLST draft notes that an MLSD response MAY include one or > more "cdir" type entries, probably because the authors anticipated that > MLSD's argument would always be a directory that would be listed in that > 'cdir' entry. "MLSD *.txt" could produce a line that's of 'cdir' type, > and whose name definitely doesn't match "*.txt". > > In the instance that an FTP server allows "MLSD */*.txt", it may even be > useful to define that a 'cdir' entry should be listed prior to the files > that match the wildcard spec in its directory. > I suggested this too. It seems to me that the MLSD spec pushes a lot of work off on the client, and something doesn't feel right about that. One server, many clients -- why should all the clients have to deal with the fact that the server doesn't order its list in a useful way? > So that if "*/*.txt" > matches "a/1.txt", "a/2.txt" and "b/3.txt", you'd have a 'cdir' line that > specified where 'a' is, followed by listings for "1.txt" and "2.txt", then > a 'cdir' line for 'b', and a listing for "3.txt". > > This goes against the note in 7.3.1 that the 'cdir' entry can occur > anywhere in the MLSD output, because it would require that the 'cdir' line > precede all matched elements in that directory. Would it be worth > specifying that all 'cdir' output lines must precede other lines relevant > to that directory? > I have been advocating that. If "cdir" can't be taken as a heading, what's the point of including it in an MLSD listing if the client can't tell which entries it applies to? (The answer is evidently something like: "it can be used as a heading, in the sense that a GUI might put the cdir name in the title bar, but it isn't a heading in the sense that it comes at the head of the output") > Here's another thought - what should MLSD do if given a wildcard that > expands into several names, some of which are directories, others of which > are files? Should it list the details of the directories, or the contents > of those directories? > Since there are many possibilities, I think we might want to define a family of MLSx commands; one to do each thing. Or perhaps more rationally, to allow MLSD to be accompanied by options ("switches") because you quickly get into combinatorial problems if you have to define a separate command for each combination of options. Just off the top of my head, I think the options we need include: . Whether the server should interpret wildcards in the MLSD argument. . Whether the server should return a recursive listing. Well, if there are only two options, maybe it's not so bad: Interpret Wildcards Recurse MLSD No No MLSR No Yes MLSW Yes No MLSX Yes Yes So far I have dealt with recursion depth-first, by CWD'ing to any "dir" I run across, and CDUP'ing when done (recursively). If I had to deal with cdir's to recurse, I'm not sure how I'd do it, e.g. to do a breadth-first traversal (as Robert once suggested, to avoid the too-many-temp-files-open problem). After all, what does cdir tell you? Not much -- usually it's just "." or "/". Even if I know that foo is bar's parent, how do I get back back to foo from some arbitrary place without knowing foo's full path? In the general case, where a directory is too deep for the straightforward walk, I think the best approach is having MLSR or MLSX return filenames in TVFS notation, relative to the server's current directory at the time it received the MLSR or MLSX command. Then we need only one temp file and we don't have to clutter the connection with CWD or CDUP commands or do any bookkeeping. Thus perhaps MLSR and MLSX should be defined as returning only filenames, only in TVFS notation. As an aside, Kermit protocol has used TVFS-like notation for years to convey directory structure between client and server, and it works very nicely. For example, a VMS server (whose path syntax bears no resemblence to Unix's) can send a directory tree to a Windows client (Unix-like directory structure but with different dirseps), switching automatically between text and binary mode for each file. The same tree can be uploaded from Windows to another VMS computer with pretty decent results. (Obviously there's a lot I didn't say about preservation of VMS file properties, but that's irrelevant here.) But this does raise some interesting possibilities for MLSD facts: . Over the past few years, we have proven in the field (with Kermit software) that files can be categorized as text or binary for purposes of transfer by algorithm with near perfect accuracy. Of course the algorithm varies by platform, but that's as it should be. We use these same techniques in the Kermit FTP client when sending a group of files (MPUT). The client can automatically switch between "ASCII" (text) and binary mode for each file. But what about downloads? If the server could employ these same algorithms, then it could include a "text-or-binary" subfact with the "type" fact, similar to the MIME Content-Type value. This would be a hint to the client to use the appropriate mode for RETR, so as to avoid garbaging the file. (Just a hint -- of course the client can do as it wishes.) . For cross-platform (or even same-platform) directory-tree duplication, it might be desirable to have a standard notation for file permissions, as distinct from the "perm" fact, which applies only to the current user of the FTP connection. Kermit protocol has this and it works out nicely -- for example, we set the execute bit automatically when receiving executable files, and we don't open up private files to world access unintentionally just by transferring them to another computer. > >In any case, my concern is that the new spec conflicts with existing > >practice. For decades it has been possible for FTP clients to execute > >commands like: > > > > mget pics/mae-west-*.jpg > > ...(etc) > > mget pics/*/mae-west-*.jpg > > > >and get at least a semi-expected result. This is about to change. > > Hardly - NLST isn't going to go away. It is still expected that NLST will > be used for MGET. What you're hoping to do is to _use_ MLSD to get facts > about those files before transferring them, so that you don't have to do a > SIZE, MDTM, etc, for each file that you transfer (you could just do an > MLST, but I can see why you'd like an MLSD with wildcards). > My point was really this: the use of NLST versus MLSD should be transparent to the user. Thus the client software would pick the best method in each situation, according to the tradeoffs. But as the spec currently stands, there is no way to do that without risk. Thus the hideously complicated and obscure user interface described here: http://www.columbia.edu/kermit/newftp.html to allow the user to get out of trouble when the client's choices are not what the user needs. > >That's a topic for another day. For now, there is no standard wildcard > >format, not even within one platform (with a few exceptions). In Unix, > >for example, every program that expands wildcards does it its own way, > >either with its own custom code, or by calling some nonstandard library > >or other. > > Actually, to judge from the recent problems with "../*/../*" overloads, > they mostly use glob() from libc. > Glob is not universal, nor is ftw, nftw, nor fts, nor, for that matter, Unix. Internet standards are platform-independent and must not assume anything about what platforms are in play at the client or server end. By the way, since there is so much confusion over what NLST should do, I think there should be statement of clarification, to the effect that "in view of widespread, if not universal, current practice..." NLST is expected to expand wilcards and return a list of the files that match. And that wildcard syntax and semantics depend on the software (client, server, libraries, OS, whatever) -- there is no standard, no universal syntax. Unless we want to tackle that one too. Personally, I think it's better to "let the 100 flowers bloom" in this area. Or at least let sleeping dogs lie. - Frank From ftp-wg-owner@hethmon.com Sat Oct 12 11:23:36 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA27237 for ; Sat, 12 Oct 2002 11:23:35 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012103349-15378-8 ; Sat, 12 Oct 2002 10:33:50 -0500 Received: from stat.com (stat.com [207.211.1.10]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012103136-54631-9 ; Sat, 12 Oct 2002 10:31:37 -0500 Received: from ratree.psu.ac.th [202.12.73.3] by stat.com with ESMTP (SMTPD32-7.07) id A3154EDE0188; Sat, 12 Oct 2002 03:01:57 -0700 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g9C9wRM09427 for ; Sat, 12 Oct 2002 16:58:30 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g9C7REa17616 for ; Sat, 12 Oct 2002 14:27:14 +0700 (ICT) In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <17614.1034407634@munnari.OZ.AU> X-RBL-Warning: REVDNS: This E-mail was sent from a MUA/MTA 202.12.73.3 with no reverse DNS entry. Date: Sat, 12 Oct 2002 10:31:40 -0500 X-OldDate: Sat, 12 Oct 2002 14:27:14 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Date: Thu, 10 Oct 2002 10:25:57 -0500 From: Frank da Cruz Message-ID: | At Wed, 9 Oct 2002 23:17:50 -0500, Alun wrote: | > Does each MLSD output for a wildcard spec even produce a 'cdir' type? | > | No. You mean it wouldn't. As defined, MLSD is only for directories, the 'D' there is definitely "directory". | It seems to me that the MLSD spec pushes a lot of work off on the client, Absolutely. Which is where it ought to be. As a client author I know that means more work for you, but I'd much rather have you do the work, than have server implementations doing more work all the time. Busy servers already have too much to do, many would like to be able to offer more, but are already out of processing resources. On the other hand, most of the time, the client is just spinning, waiting for a file to transfer. | One | server, many clients -- why should all the clients have to deal with the | fact that the server doesn't order its list in a useful way? One reason is because there's no definition of what is "useful". I expect listings to be ordered by file size, by modification time, by file "type", by name, ... The server has no idea what will be useful, so either it has to be told, in which case it has to do the ordering every time - and in that scenario, just as above, it is better for the client to do it. Or it simply has to decide "this is the correct order", in which case clients need the ability to reorder anyway, to get the order that the user prefers. | If "cdir" can't be taken as a heading, what's | the point of including it in an MLSD listing if the client can't tell which | entries it applies to? It applies to all of the entries, by definition. The output is a listing of the contents of one directory, cdir gives the name of that directory. | (The answer is evidently something like: "it can be | used as a heading, in the sense that a GUI might put the cdir name in the | title bar, but it isn't a heading in the sense that it comes at the head of | the output") No, it isn't a heading at all. It (when TVFS is supported anyway) can be used to construct a path to a file in the directory, from the current working directory of the (instance of) the server. That is, it supplies information useful for naming the files listed. Sure, some GUI clients might decide that they can use it in a title bar, or something similar, but that's a by product, not the purpose. cdir not coming first is just because that's another constraint upon the server, which in some odd cisumstance, might mean that the server would need to generate the listing, beore transmitting any of it. Servers are busy. The less restrictions we impose upon them, the better. | . Whether the server should interpret wildcards in the MLSD argument. Not just whether, but also what he wildcards actually mean, otherwise your client is going to have to interpret the user's spec, and re-code it into some standard wildcard language. Personally, I believe that wildcards have no place at all in the FTP spec. They will cause far more problems than they'd ever solve. All we need is for clients to start acting more reaosnably. | . Whether the server should return a recursive listing. If this was to exist, it would certainly want to be a new command, rather than a switch, as the output format is necessarily going to need to differ (so it is possible to distinguish one directory from the next). Note that you cannot simply use a cdir line, require it to come first, and define that as the beginning of a new directory, or you would be unable to distinguish the case of a directory having multiple cdir lines (which is a very reasonable thing to do - giving both a fully qualified path, and a relative path from the PWD) from an empty directory, followed by another one. | After all, what does cdir tell you? Not much -- usually it's | just "." or "/". If you CWD first, and just "MLSD" all the time, then yes, "." is quite often going to be the result. But if you give an arg to MLSD, so you're referring to some other directory, then "." would simply be wrong (well, assuming traditional unix type semantics anyway, there's nothing in the MLSx spec that requires this, "." could easily be the name of some sub-directory of the current directory). | Even if I know that foo is bar's parent, how do I get | back back to foo from some arbitrary place without knowing foo's full path? There's no guaranteed way of doing that, other than establishing a new connection, in FTP. CDUP is optional. MLSx type pdir is optional. FTP servers aren't required to provide you with a way to get back out of a directory they have let you into. Once you're there, you can be confined there. When TVFS exists (which is probably going to be most of the time), that's why it makes more sense to construct paths, and use them, rather than CWD around all the time. This is true regardless of the MLSx spec. | In the general case, where a directory is too deep for the straightforward | walk, I think the best approach is having MLSR or MLSX return filenames in | TVFS notation, relative to the server's current directory at the time it | received the MLSR or MLSX command. That would be an option where TVFS exists. But what do we do with the servers that can't support TVFS? | Then we need only one temp file You only need one temp file anyway. You can easily seek around in that as you need to. Sure, seek pointers (and an accompanying size) are a little more overhead for the client, than a file descriptor, but they're much less overhead for the system that is running the client. | But what about downloads? If the server | could employ these same algorithms, then it could include a | "text-or-binary" subfact with the "type" fact, similar to the MIME | Content-Type value. This would be a hint to the client to use the | appropriate mode for RETR, so as to avoid garbaging the file. | (Just a hint -- of course the client can do as it wishes.) Sounds reasonable to me, write a definition for the fact type.... Once this draft gets published (eventually) we will have a fairly standardised method of adding new facts. Certainly it was never imagined that the set that is in the doc will cover everything (even if we are still hanging onto the completely useless, as far as I can tell, create-time ...) | . For cross-platform (or even same-platform) directory-tree duplication, | it might be desirable to have a standard notation for file permissions, | as distinct from the "perm" fact, which applies only to the current | user of the FTP connection. Im not sure how exactly that will work, as on almost all systems, the permissions vary from user to user. To describe anything more general than the current user, you need to be able to identify users in some way. And that then means some kind of general user namespace management problem. | Kermit protocol has this and it works out | nicely -- for example, we set the execute bit automatically when Yes, "executable" is perhaps one thing that is currently missing, though it is not clear to me whether executable is a permission, or a file type, or both. We could perhaps do with another fact type, perhaps attributes, to describe things that aren't really permissions, and aren't really types either. The binary/ascii file transfer mode hint, whether the file is an executable one (which is not the same as "the current user should be allowed to execute it") etc, could perhaps fit there. | My point was really this: the use of NLST versus MLSD should be transparent | to the user. To the user, yes. Not to the user's software though. | Thus the client software would pick the best method in each | situation, according to the tradeoffs. Yes. | But as the spec currently stands, | there is no way to do that without risk. There is no way at all to use NLST the way you do, at all, without risk. You have just decided that the risk is low enough that you have decided to ignore it. As server load gets higher, I fully expect to see servers start looking at the spec, seeing that they're not required to handle wildcards, and simply stop doing that (just as many that used to do on the fly gzip or ungzip, no longer do, because it just costs too much for the server, regardless of how much it would save the network or the client). | Thus the hideously complicated and | obscure user interface described here: | | http://www.columbia.edu/kermit/newftp.html | | to allow the user to get out of trouble when the client's choices are not | what the user needs. This is all because your'e attempting to optimise things which very rarely ever need optimising. Just fetching the directory listing, and doing your own wildcard processing would work fine in just about every case. It is likely to make your code bigger, and more complex, but someone has to do all the work, and the client is the place to do it, not the server. | Glob is not universal, nor is ftw, nftw, nor fts, nor, for that matter, | Unix. Internet standards are platform-independent and must not assume | anything about what platforms are in play at the client or server end. Absolutely. So, when I say "mget *.txt" I should get all the files that end in .txt on a dos or unix platform, but the file whose name is "*.txt" on a macintosh (I think there is no magic '*' there, at least before MacOS X) Then if I'm on a unix client and do 'mget [a-c]*' I should get all files that start with a b or c, on a dos client I should probably get an error as '[' means nothing to dos users (I believe), and also is illegal in file names. Anything other than this means that file names given to ftp don't mean the same as file names given to other commands on the system, and that's a very bad UI design. | By the way, since there is so much confusion over what NLST should do, I | think there should be statement of clarification, to the effect that "in | view of widespread, if not universal, current practice..." NLST is | expected to expand wilcards and return a list of the files that match. I would strongly object to that, and in fact, am quite likely to disable wildcard processing in my server. | And that wildcard syntax and semantics depend on the software (client, | server, libraries, OS, whatever) -- there is no standard, no universal | syntax. Unless we want to tackle that one too. Personally, I think it's | better to "let the 100 flowers bloom" in this area. Or at least let | sleeping dogs lie. You mean you're proposing to require a mechanism that no-one can use safely as no-one knows what it might mean? That would be an "interesting" design choice. kre From ftp-wg-owner@hethmon.com Sat Oct 12 14:11:28 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA28819 for ; Sat, 12 Oct 2002 14:11:27 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012131649-52759-9 ; Sat, 12 Oct 2002 13:16:49 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012131553-22650-5 ; Sat, 12 Oct 2002 13:15:54 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9CI5viD018040 for ; Sat, 12 Oct 2002 14:05:58 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9CI5uJU018028 for FTP Working Group ; Sat, 12 Oct 2002 14:05:56 -0400 (EDT) Message-ID: Date: Sat, 12 Oct 2002 13:16:07 -0500 X-OldDate: Sat, 12 Oct 2002 14:05:56 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTP Working Group Subject: Ftp-WG: help help From ftp-wg-owner@hethmon.com Sat Oct 12 14:48:01 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA29148 for ; Sat, 12 Oct 2002 14:47:59 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012135311-52021-11 ; Sat, 12 Oct 2002 13:53:12 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012134924-22880-5 ; Sat, 12 Oct 2002 13:49:25 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9CIdQiD020196; Sat, 12 Oct 2002 14:39:26 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9CIdQ96020195; Sat, 12 Oct 2002 14:39:26 -0400 (EDT) In-Reply-To: Your message of Sat, 12 Oct 2002 10:31:40 -0500 Message-ID: Date: Sat, 12 Oct 2002 13:49:37 -0500 X-OldDate: Sat, 12 Oct 2002 14:39:26 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD On Sat, 12 Oct 2002 10:31:40 -0500, Robert wrote: : At Thu, 10 Oct 2002 10:25:57 -0500, Frank wrote: : | It seems to me that the MLSD spec pushes a lot of work off on the : | client, : : Absolutely. Which is where it ought to be. As a client author I know : that means more work for you... : I'm sure you know my interest in this is not avoiding work. I've already done most of the work, so if nothing changes the Kermit FTP client will fit in, but how will users receive it? My concerns are: 1. That the spec be as clean, simple, and elegant as possible to promote the widest implementation; not just superficially (because it already is clean and elegant on the surface) but also at the deeper level where it must be implemented. 2. That it not force choices upon the user that s/he is not equipped do deal with. This not only frustrates users, but exerts back pressure on help desks. That is, it has human consequences. I understand there's a tradeoff between the complexity of the server and the client, but we're talking about the interface between them. The interface should be usable in a straightforward, obvious way or it won't take off and we'll still be parsing "ls -l" listings for another 30 years. : | . Whether the server should interpret wildcards in the MLSD argument. : : Not just whether, but also what he wildcards actually mean, otherwise your : client is going to have to interpret the user's spec, and re-code it into : some standard wildcard language. : Wildcard syntax and semantics are, by nature, up to the software that implements them. I don't believe this can be legislated, given the free-for-all nature of software development. You also can't legislate away 30 years of practice. The protocol specification might be cleaner and clearer by avoiding the issue, but neither users nor developers will be happy. Nor can you safely design a standard wildcard language because you can't enumerate all the known and future file-system and wildard syntaxes, and therefore can't avoid introducing conflicts and ambiguities. At some point, if only as a final resort, the user must be exposed to the server host's notation -- it can't be totally hidden without interfering with access. As noted previously and agreed by all, RFC959 was not clear in this area, This resulted in widespread if not universal server-side wildcard interpration. The cat has been loose and breeding for decades; you can't undo this. New servers that support MLSD as specified (and perforce, continue to also support NLST) will force arcane choices upon the user, as noted in my paper. We might consider it unlikely that a user might want to get 3 files out of a directory containing a very large number files, but that does not mean the protocol should ignore this case. It's same argument that was made when designing the address space of the early networks, which was outgrown faster than any expert foresaw (more than once). : Personally, I believe that wildcards have no place at all in the FTP spec. : They will cause far more problems than they'd ever solve. : You can't ignore them. RFC959 did and look what happened. The spec needs to say something about which side handles them and, since there are legitimate and even compelling arguments to made for each side, a choice should be offered. Suppose we have two commands, MLSD and MLSW that are implemented in every server. There is a use for each. GUI drag-and-drop FTP clients will choose MLSD; command-line clients (which we must not pooh-pooh) will tend to prefer MSLW because they are used by professionals who care about -- and need -- detailed control and automation capabilities. : Note that you cannot simply use a cdir line, require it to come first, and : define that as the beginning of a new directory, or you would be unable to : distinguish the case of a directory having multiple cdir lines (which is a : very reasonable thing to do - giving both a fully qualified path, and a : relative path from the PWD) from an empty directory, followed by another : one. : Then "cdir" needs a stricter definition; it should do one thing or the other. Add another fact if it's necessary to do both (and I believe it is; the simple relative form for GUI title bars or whatever, and the fully qualified form for use by the client in navigating the tree). In fact, this one of the most annoying things about FTP -- you can't find out (programmatically) the server's current directory. The PWD command is useless. CDIR should be an improvement. Ditto for PDIR. : | But what about downloads? If the server : | could employ these same algorithms, then it could include a : | "text-or-binary" subfact with the "type" fact, similar to the MIME : | Content-Type value. This would be a hint to the client to use the : | appropriate mode for RETR, so as to avoid garbaging the file. : | (Just a hint -- of course the client can do as it wishes.) : : Sounds reasonable to me, write a definition for the fact type.... : FTP protocol requires the client to control everything, but only the server knows about its own files. The server needs a way to suggest the appropriate STRU and TYPE for each file to the client. This could be done in various ways, but the cleanest is to add new facts: mode= (S, B, C) stru= (F, R, P) xtype= (A[ x], E, I, L x) "xtype" distinguishes this new fact from the existing "type" fact. If a value contains spaces (as TYPE values can), we have to work around the prohibition against facts containing spaces, such as replacing space by (say) an ASCII punctuation character, e.g.: xtype=L:8 xtype=A:C Presumably we won't have to confront "quoting hell" since MODE, STRU, and TYPE values should always be alphanumeric (but watch out; I don't see this stated explicitly anywhere). The wording should be to the effect that each of these facts can have any value defined for the corresponding FTP command, thus not committing ourselves to enumerate them, which would require this spec to change any time a new MODE, STRU, or TYPE value was added somewhere else. But it should list the existing values as examples. : | . For cross-platform (or even same-platform) directory-tree : | duplication, it might be desirable to have a standard notation for : | file permissions, as distinct from the "perm" fact, which applies : | only to the current user of the FTP connection. : : Im not sure how exactly that will work, as on almost all systems, the : permissions vary from user to user. To describe anything more general : than the current user, you need to be able to identify users in some way. : And that then means some kind of general user namespace management : problem. : We've had some experience with this in Kermit. Clearly you can't cover everything (ACLs spring to mind), and there's not much you could reasonably expect to do about namespace (file owner and group). But a simple model could accommodate most things that people care about: user and world read, write, delete, execute, list, and append permission. Maybe group too. This takes in VMS, Unix, DOS, TOPS-20 (gone but not forgotten because it pioneered many concepts some of which still need to catch on). In any case this is just information provided by the server to the client in a new fact, say XPERM. The client can use it however it wants. In the reverse direction it is also often desirable to set permissions of uploaded files. FTP protocol does not give a way to do that. Defining a standard notation for file permissions could open the door to this. : Yes, "executable" is perhaps one thing that is currently missing, though : it is not clear to me whether executable is a permission, or a file type, : or both. : It's a permission because it grants execute permission. In TOPS-20, a program could have execute but not read permission and you could still run it. But you couldn't copy it, save it, or look at its contents. Of course conceptually it's also a file type, and perhaps it might be useful to distinguish between (say) a shell script that has execute permission and an executable binary program (that might or might not have execute permission), but I'm not sure I see any need for this. Of course some file systems overload execute permission to save bits, e.g. a Unix directory with x permission means you can list its contents, but that's a separate concept. : We could perhaps do with another fact type, perhaps attributes, to : describe things that aren't really permissions, and aren't really types : either. ; We've done this in Kermit, but far from perfectly. The biggest problem comes in defining how to describe highly-structured files: fixed-length record files, variable-length record files (as in VMS, VOS, CMS), nonstream files (relative, indexed), various record formats (e.g. with carriage control). Ultimately what we settled on was a special OPTIONAL mode of file transfer in which the file's control block is sent (in a specially coded segment) along with the file. If the file transfer partner is on the same platform, it can use it to reconstruct the file; if it's not, it can archive the information in a way that can be used if the file is subsequently transferred to a computer with compatible FCBs. But in the general case, I think you've already defined most of the commonly used attributes: date, size, character-set, etc, and FTP's existing MODE, STRU, and TYPE commands cover most of the rest. (Any VMS FTP implementors read this?) : | But as the spec currently stands, : | there is no way to do that without risk. : : There is no way at all to use NLST the way you do, at all, without risk. : It's not me, it's everybody. : You have just decided that the risk is low enough that you have decided : to ignore it. As server load gets higher, I fully expect to see servers : start looking at the spec, seeing that they're not required to handle : wildcards, and simply stop doing that (just as many that used to do on the : fly gzip or ungzip, no longer do, because it just costs too much for the : server, regardless of how much it would save the network or the client). : You're overloading MLSD. It is at once a new file-listing format and a new discipline to reduce load on the server -- two unrelated concepts. It should be possible to get the new listing format without having a whole new set of rules about who does what. Perhaps what we need is a way for the server to set policy. First of all, we have to agree to accept the fact that NLST expands wildcards because it does; you can't change it. Then suppose the server supports MLSD, and announces this in its FEAT response. The client sends "NLST [abc]*.zip". The server says: 550 Too busy for NLST - try MLSD The client can switch to its MLSD handler and try again (sending "MLSD" and then holding "[abc]*.zip" locally for matching). : | Thus the hideously complicated and : | obscure user interface described here: : | : | http://www.columbia.edu/kermit/newftp.html : | : | to allow the user to get out of trouble when the client's choices are : | not what the user needs. : : This is all because your'e attempting to optimise things which very : rarely ever need optimising. : I've been in this business long enough to know that using words like "rarely" is tempting fate. As for the rest, we're going around in circles; I've said my piece and I don't care that much. If anybody else cares about these issues, please speak up; if not, don't. Meanwhile, in case I still have anybody's attention, here's another item that might be worth looking at: RFC959's STOU specification is inadequate. First, it does not accept an operand, which is unreasable. You should be able to "stou foo.bar" and have it stored as foo.bar if there is no conflict, or foo.bar.1 or whatever if there is, and in fact many clients and many servers do allow this, despite the spec. Second, and more serious, there is NO WAY for the client to find out the name picked by the server for the file. Why should we care? FTP is used increasingly for transaction processing, EDI, etc (for example at IBM Info Exchange and Advantis). "Atomic file movement" and feedback is essential for transaction processing. Summary: . Add MLSW, MLSR, MLSX. . Define a standard notation for permissions. . Add XTYPE, MODE, and STRU facts. . Add an XPERM fact for sending file permissions (as distinct from PERM). . Add a command (PERM?) for setting server file permissions. . Do something about STOU: augment its definition or define a new command. Oops, one more thing.... The SIZE command returns a different result depending on the prevailing TYPE. But the SIZE fact always returns the actual length of the file in octets, right? I just wanted to make sure this is intentional; in fact I prefer the latter behavior. I discovered that if you don't switch to TYPE I before sending a SIZE command then you can wind up waiting an awfully long time for the answer! - Frank P.S. Sorry for sending a "help" command to the list -- I meant to send it to the -request address... From ftp-wg-owner@hethmon.com Sat Oct 12 16:34:00 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA00413 for ; Sat, 12 Oct 2002 16:33:59 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012153842-46461-9 ; Sat, 12 Oct 2002 15:38:44 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012153515-17161-5 ; Sat, 12 Oct 2002 15:35:16 -0500 Received: (qmail 59655 invoked by uid 1016); 12 Oct 2002 20:25:39 -0000 Message-ID: <20021012202539.59654.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Sat, 12 Oct 2002 15:35:23 -0500 X-OldDate: 12 Oct 2002 20:25:39 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Frank da Cruz writes: > For decades it has been possible for FTP clients to execute commands like: > mget pics/mae-west-*.jpg And it's still possible. If you receive that command from the user, simply switch to the pics directory, do an NLST, extract the mae-west-* names, and retrieve those files one by one. What's the problem? Are you saying that your client tries to push the * processing over to the server? Sorry, but that isn't guaranteed by the FTP standard, and it isn't supported by all servers. You should be using the standard NLST mechanism instead. The UNIX ftpd and its offshoot wuftpd have, historically, accounted for most FTP servers on the Internet, but you shouldn't be relying on their wildcard quirks. There have always been servers that, quite reasonably and in full compliance with the protocol, don't imitate ftpd's wildcard processing. You would already know this if you had tested your client more thoroughly. By the way, this is orthogonal to the question of list format. > (in the "mae-west*.jpg" example > above, suppose these pictures were in a directory that contained > a million files, but you only wanted three of them). Reality check: What kind of idiot organizes his files that way? Do you realize how long it takes for a server to look through a million-file directory on a typical filesystem? Have you heard of subdirectories? ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Sat Oct 12 17:03:27 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA00854 for ; Sat, 12 Oct 2002 17:03:26 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012160812-62622-9 ; Sat, 12 Oct 2002 16:08:14 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012160324-33626-5 ; Sat, 12 Oct 2002 16:03:25 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 030174 for ; Sat, 12 Oct 2002 16:53:18 -0400 (EDT) Message-Id: <4.3.2.7.2.20021012152634.01ee2af8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <17614.1034407634@munnari.OZ.AU> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Sat, 12 Oct 2002 16:03:33 -0500 X-OldDate: Sat, 12 Oct 2002 15:29:32 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 10:31 AM 10/12/2002, Robert Elz wrote: >Then if I'm on a unix client and do 'mget [a-c]*' I should get all files >that start with a b or c, on a dos client I should probably get an error >as '[' means nothing to dos users (I believe), and also is illegal in file >names. Better still, at least as regards your argument: X:\>copy con [a-c].txt xx ^Z 1 file(s) copied. X:\>dir [a-c]* Volume in drive X has no label. Volume Serial Number is 0000-0000 Directory of X:\ 10/12/2002 03:27 PM 4 [a-c].txt 1 File(s) 4 bytes 0 Dir(s) 192,708,608 bytes free Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Sat Oct 12 17:18:24 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA01021 for ; Sat, 12 Oct 2002 17:18:24 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012162257-46489-5 ; Sat, 12 Oct 2002 16:22:59 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012161442-17139-12 ; Sat, 12 Oct 2002 16:14:43 -0500 Received: (qmail 64132 invoked by uid 1016); 12 Oct 2002 21:04:54 -0000 Message-ID: <20021012210454.64131.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Sat, 12 Oct 2002 16:14:55 -0500 X-OldDate: 12 Oct 2002 21:04:54 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Frank da Cruz writes: > widespread if not universal server-side wildcard Certainly not universal. High-quality clients don't rely on it. You are overgeneralizing when you claim that NLST expands wildcards. > You're overloading MLSD. It is at once a new file-listing format and a > new discipline to reduce load on the server -- two unrelated concepts. The discipline is not new. If you want to handle wildcards, you should be relying on the standard protocol that works with all servers, rather than on the behavior of certain FTP servers. Load is a red herring; the real issue is reliability. > At some point, if only as a final resort, the user must be exposed to > the server host's notation -- it can't be totally hidden without > interfering with access. False. For example, NFS and ftpfs users access files just fine without having any clue about the server's favorite wildcard notation. > We might consider it unlikely that a user might want to get 3 files out of a > directory containing a very large number files, but that does not mean the > protocol should ignore this case. It's same argument that was made when > designing the address space of the early networks, which was outgrown faster > than any expert foresaw (more than once). No, it isn't the same argument. Obvious difference #1: Address limits are much more serious than speed limits. Speed is multiplied by a renewable resource, namely time, while addresses are not. Obvious difference #2: Limits on the size of the entire system are much more serious than limits on local branching factors. I don't care whether I can put a million files into a single directory; I do care whether I can put a million files onto a single disk. > The server needs a way to suggest the > appropriate STRU and TYPE for each file to the client. No. It works much more smoothly to transfer a binary stream, along with enough out-of-band information for the client to understand the stream, as in HTTP. This is another good example of client-side processing being better than server-side processing. > FTP is used increasingly for transaction processing, EDI, etc (for > example at IBM Info Exchange and Advantis). Why? Do they have no clue how to design a sane protocol? ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Sat Oct 12 17:46:03 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA01440 for ; Sat, 12 Oct 2002 17:46:02 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012165005-49003-13 ; Sat, 12 Oct 2002 16:50:08 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012162910-19640-17 ; Sat, 12 Oct 2002 16:29:11 -0500 Received: (qmail 65946 invoked by uid 1016); 12 Oct 2002 21:19:06 -0000 Message-ID: <20021012211906.65945.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: <4.3.2.7.2.20021008105015.020305d8@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Sat, 12 Oct 2002 16:29:29 -0500 X-OldDate: 12 Oct 2002 21:19:06 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. Alun Jones writes: > there is currently no generic URL to > say that the browser's user must be prompted for user name and password. I don't understand. Why would anyone want a URL like that? Do you also want URLs that prompt for the file name? Is this supposed to be some sort of crude substitute for HTTP-based forms? ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Sat Oct 12 20:40:02 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA02923 for ; Sat, 12 Oct 2002 20:40:00 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012194331-61221-9 ; Sat, 12 Oct 2002 19:43:34 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012193326-32209-5 ; Sat, 12 Oct 2002 19:33:28 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 031401 for ; Sat, 12 Oct 2002 20:23:15 -0400 (EDT) Message-Id: <4.3.2.7.2.20021012153139.01ec8108@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Sat, 12 Oct 2002 19:33:34 -0500 X-OldDate: Sat, 12 Oct 2002 19:01:27 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 01:49 PM 10/12/2002, Frank da Cruz wrote: >We might consider it unlikely that a user might want to get 3 files out of a >directory containing a very large number files, but that does not mean the >protocol should ignore this case. Actually, yes, it may very well mean that. There are people out there, only a few, who want to use FTP to stream video. Does that mean that the general standard of FTP should be updated to include a command that lists the frame-rate of an MPG file? The more unlikely a usage is, the less it needs to be made a part of the general protocol. >You can't ignore them. RFC959 did and look what happened. The spec needs >to say something about which side handles them and, since there are >legitimate and even compelling arguments to made for each side, a choice >should be offered. The spec, as far as I was aware, _does_ say something about which side handles them: 2.2.2. Wildcarding ... Clients that desire some form of pattern matching functionality must obtain a listing of the relevant directory, or directories, and implement their own file name selection procedures. That sounds pretty clear. It may not be right, but it is currently present. I'd like to see how some of the MLST implementations you've cited that handle wildcards cope with extreme cases - how well thought-out they are. It may very well be the case that these MLST implementations are sufficiently broken that the advantage of wildcard support is lost among the disadvantage of unreliable output. If wildcards are to be supported, then these extreme cases must be supported _in_the_spec_. >Suppose we have two commands, MLSD and MLSW that are >implemented in every server. There is a use for each. GUI drag-and-drop >FTP clients will choose MLSD; command-line clients (which we must not >pooh-pooh) will tend to prefer MSLW because they are used by professionals >who care about -- and need -- detailed control and automation capabilities. If we're going to haggle about MLSW, then let's wait until we've got _this_ draft to RFC status; if MLSW is going to be a separate command, then it needs to stop holding up this draft, already long overdue in many respects. >Then "cdir" needs a stricter definition; it should do one thing or the >other. Add another fact if it's necessary to do both (and I believe it is; >the simple relative form for GUI title bars or whatever, and the fully >qualified form for use by the client in navigating the tree). Okay, so maybe the draft doesn't say that "cdir should not be used as a heading". It also doesn't say that it shouldn't be used as a dessert topping. The spec does say what cdir's purpose is, it doesn't say what its purpose isn't. Any 'cdir' form can be used by the client in navigating the tree, as "CWD " will get you to the directory listed, from the current directory. >In fact, this one of the most annoying things about FTP -- you can't find >out (programmatically) the server's current directory. The PWD command is >useless. CDIR should be an improvement. Since RFC 959, PWD is supposed to respond: 257"" However, maybe there is room for a record in MLST that specifies the absolute path, if one exists. In a TVFS file system, of course, that's _very_ easy to generate and recognise - it's any cdir entry where the name starts with "/". >FTP protocol requires the client to control everything, but only the >server knows about its own files. The server needs a way to suggest the >appropriate STRU and TYPE for each file to the client. This could be done >in various ways, but the cleanest is to add new facts: > > mode= (S, B, C) Why a fact for MODE? > stru= (F, R, P) No objections here, it would seem that this property would come directly from the filing system - of course, in many popular filing systems, it's going to be completely extraneous as it'll always be 'F', but there are systems that might want to communicate that a file is a 'record' format file. > xtype= (A[ x], E, I, L x) This is a tricky one to defend, though. How do you tell whether a file needs Image or ASCII mode? Reliably, I mean, and without using up a whole heap of resources by scanning every file just to produce a listing. >"xtype" distinguishes this new fact from the existing "type" fact. If a >value contains spaces (as TYPE values can), we have to work around the >prohibition against facts containing spaces, such as replacing space by >(say) an ASCII punctuation character, e.g.: > > xtype=L:8 > xtype=A:C Or, since the first parameter is a single character, omitting the space would do just as well. >In the reverse direction it is also often desirable to set permissions of >uploaded files. FTP protocol does not give a way to do that. Defining a >standard notation for file permissions could open the door to this. If this is done, of course, it's important to note that the permissions you get back out may not be exactly the same as the ones you put in. Ask to mark a file as readable but not executable, and what do you then do on a file system where the two facts are the same? Server-dependent? Probably, but it's likely that clients won't be happy to discover that their commands aren't interpreted exactly. >It's a permission because it grants execute permission. In TOPS-20, a >program could have execute but not read permission and you could still run >it. But you couldn't copy it, save it, or look at its contents. Why is it of importance to FTP, though? FTP is not about executing files. It's about reading and/or writing them, with a touch of deleting, renaming, etc. But there's no provision for execution (and it's a bad idea when there is!) >: There is no way at all to use NLST the way you do, at all, without risk. >: >It's not me, it's everybody. "Everybody" is using NLST in a way that is not documented. It's also liable to a few freaky behaviours. I can remember the first time someone complained to me that "MGET *" didn't get all the files. Well, no, because on Win16 (the platform I was working on at the time) doesn't match "*" to "every file", it specifically only matches "every file that doesn't have an extension". Wildcard expansion at the server is not as unambiguous and safe as all that. >You're overloading MLSD. It is at once a new file-listing format and a >new discipline to reduce load on the server -- two unrelated concepts. It >should be possible to get the new listing format without having a whole >new set of rules about who does what. The format, however, requires a new set of rules. The format as it stands is of no use when it comes to listing a wildcard that expands to match files in multiple directories - it's also of no use to a listing that matches files and directories. If MLSD should accomodate wildcards, then it needs a rewrite of the format; if the format stays as it is, then the rule must be that MLSD does not accomodate wildcards. >Perhaps what we need is a way for the server to set policy. First of all, >we have to agree to accept the fact that NLST expands wildcards because it >does; you can't change it. I don't think that's a good argument for you to be proposing. I'd rather approve a standard on the basis of "because it's useful to do so", instead of "because everyone expects it to behave in the way that one vendor implemented it". Need I raise the spectre of the "Big M" vendor as an example as to why it's a good idea to specify, rather than follow the most popular implementation? There are arguments why wildcard expansion is good. One is that it allows a server to sacrifice an amount of processor time to save an amount of bandwidth and transmission time. This may be a good trade, even for the server. An FTP server that's just an FTP server may very well spend most of its time simply accessing disk and network - not a lot of processor operations (unless it's encrypting, in which case it's probably spending a lot of time there). Now, that's a better argument than "we'll do it this way because it's the way it's been done so far". After all, the way it's been done so far is with an undefined de-facto "everyone does it like Unix" listing format :-) >Then suppose the server supports MLSD, and announces this in its FEAT >response. The client sends "NLST [abc]*.zip". The server says: > > 550 Too busy for NLST - try MLSD > >The client can switch to its MLSD handler and try again (sending "MLSD" >and then holding "[abc]*.zip" locally for matching). A client may well want to choose to evaluate wildcards locally for other reasons - for instance, it may know better how to expand certain combinations; "[abc]" as a file specification in Windows, for instance, means a file called "[abc]", not "a single letter file name, either a, b or c". Similarly, as someone noted earlier (possibly you?), there may be times when it is appropriate for the user who knows the wildcard syntax of the server, to ask the server to expand. >: | Thus the hideously complicated and >: | obscure user interface described here: If we can avoid words like "hideous" or "obscure", we might avoid making this into an overly emotional discussion, and get down to the issues at hand. Thanks. >: This is all because your'e attempting to optimise things which very >: rarely ever need optimising. >: >I've been in this business long enough to know that using words like >"rarely" is tempting fate. Okay, so let's look at the scenarios. You've suggested a possible situation that might cause server wild-card expansions to be a useful thing - does this occur frequently enough in practice that it's worth carrying the implementation over to a new command? As you've noted, every server currently does wildcard handling in NLST, so is there sufficient saving from current use of NLST wildcards to justify insisting on its retention? Does anyone here have means of gathering the sort of statistics that might be useful in determining this? >RFC959's STOU specification is inadequate. First, it does not accept an >operand, which is unreasable. You should be able to "stou foo.bar" and have >it stored as foo.bar if there is no conflict, or foo.bar.1 or whatever if >there is, and in fact many clients and many servers do allow this, despite >the spec. I'll admit it - even my own server allows a parameter for STOU. But part of the point of STOU is that we are expecting there to be a collision, in a significant number of cases. >Second, and more serious, there is NO WAY for the client to find >out the name picked by the server for the file. Why should we care? Apparently, because we haven't read RFC 1123 :-) 4.1.2.9 STOU Command: RFC-959 Section 4.1.3 The STOU command stores into a uniquely named file. When it receives an STOU command, a Server-FTP MUST return the actual file name in the "125 Transfer Starting" or the "150 Opening Data Connection" message that precedes the transfer (the 250 reply code mentioned in RFC-959 is incorrect). The exact format of these messages is hereby defined to be as follows: 125 FILE: pppp 150 FILE: pppp where pppp represents the unique pathname of the file that will be written. >Summary: > > . Add MLSW, MLSR, MLSX. In a separate draft, _please_. > . Define a standard notation for permissions. Wow. Good luck. > . Add XTYPE, MODE, and STRU facts. MODE? No. XTYPE? How to do it reliably? > . Add an XPERM fact for sending file permissions (as distinct from PERM). Should be handled by OS-specific facts. File permissions as they are on the file system are file-system dependent. > . Add a command (PERM?) for setting server file permissions. Again, this is an OS-specific issue. A SITE command should be used; _maybe_ a set of OS-dependent or FS-dependent commands. I don't think you'll get a generic format. > . Do something about STOU: augment its definition or define a new command. You know, I think you've got enough here that it's worth starting a new draft. >Oops, one more thing.... The SIZE command returns a different result >depending on the prevailing TYPE. But the SIZE fact always returns the >actual length of the file in octets, right? I just wanted to make sure >this is intentional; in fact I prefer the latter behavior. I discovered >that if you don't switch to TYPE I before sending a SIZE command then >you can wind up waiting an awfully long time for the answer! Not only is this intentional, but it is documented. Since MLSD doesn't generally precede a downloading of every file in the directory, it would be wasteful to calculate the transfer size of each file. SIZE, on the other hand, takes _one_ file name as parameter (hey, why not make that take wildcards? Just kidding!), and as such it's expected that the client user knows that the operation will take time. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Sat Oct 12 20:40:39 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA02938 for ; Sat, 12 Oct 2002 20:40:38 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012194421-60341-11 ; Sat, 12 Oct 2002 19:44:23 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012193352-31334-9 ; Sat, 12 Oct 2002 19:33:54 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 131401 for ; Sat, 12 Oct 2002 20:23:21 -0400 (EDT) Message-Id: <4.3.2.7.2.20021012190211.01ec83a8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <20021012202539.59654.qmail@cr.yp.to> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Sat, 12 Oct 2002 19:34:01 -0500 X-OldDate: Sat, 12 Oct 2002 19:03:19 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 03:35 PM 10/12/2002, D. J. Bernstein wrote: >Frank da Cruz writes: > > (in the "mae-west*.jpg" example > > above, suppose these pictures were in a directory that contained > > a million files, but you only wanted three of them). > >Reality check: What kind of idiot organizes his files that way? Do you >realize how long it takes for a server to look through a million-file >directory on a typical filesystem? Have you heard of subdirectories? Yes, I've seen it happen, I'm afraid. It was a file server for an NMR machine. An absolutely horrible, and totally inefficient layout, as you say. I don't think it's worth making an effort to support such bad behaviour. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Sat Oct 12 21:23:27 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id VAA03207 for ; Sat, 12 Oct 2002 21:23:27 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012202554-62926-5 ; Sat, 12 Oct 2002 20:25:59 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012200240-33889-14 ; Sat, 12 Oct 2002 20:02:45 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 084623 for ; Sat, 12 Oct 2002 20:51:28 -0400 (EDT) Message-Id: <4.3.2.7.2.20021012190501.01f0ace8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <20021012211906.65945.qmail@cr.yp.to> References: <4.3.2.7.2.20021008105015.020305d8@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Sat, 12 Oct 2002 20:03:05 -0500 X-OldDate: Sat, 12 Oct 2002 19:55:26 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. At 04:29 PM 10/12/2002, D. J. Bernstein wrote: >Alun Jones writes: > > there is currently no generic URL to > > say that the browser's user must be prompted for user name and password. > >I don't understand. Why would anyone want a URL like that? Do you also >want URLs that prompt for the file name? Is this supposed to be some >sort of crude substitute for HTTP-based forms? Funnily enough, it never occurred to me to ask why the customers wanted a URL like that. It just didn't seem like that crazy of a request to make. Perhaps it's for auditing access to a file that is shared between several users? So that you could use a web page that read something like "ftp://??@mysite/sharedfile.zip", and have a log of which of your users downloaded the file. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Sun Oct 13 00:48:46 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA05242 for ; Sun, 13 Oct 2002 00:48:45 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012235236-51593-9 ; Sat, 12 Oct 2002 23:52:40 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021012234122-22299-5 ; Sat, 12 Oct 2002 23:41:23 -0500 Received: (qmail 53304 invoked by uid 1016); 13 Oct 2002 04:31:34 -0000 Message-ID: <20021013043134.53303.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: <4.3.2.7.2.20021008105015.020305d8@208.55.91.110> <4.3.2.7.2.20021012190501.01f0ace8@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Sat, 12 Oct 2002 23:41:34 -0500 X-OldDate: 13 Oct 2002 04:31:34 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: Suggestion - blank userid in FTP URLs. Alun Jones writes: > Perhaps it's for auditing access to a file that is shared between > several users? Give the users a single shared FTP username with different passwords. Audit by password. No need to revise the FTP URL semantics. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Sun Oct 13 07:26:45 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id HAA17848 for ; Sun, 13 Oct 2002 07:26:44 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013063649-3811-10 ; Sun, 13 Oct 2002 06:36:50 -0500 Received: from mail.san.yahoo.com (mail.san.yahoo.com [209.132.1.30]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013063329-47475-11 ; Sun, 13 Oct 2002 06:33:39 -0500 Received: from Intellica (212.174.56.240) by mail.san.yahoo.com (6.5.029) id 3DA952A600000F20 for ftp-wg@hethmon.com; Sun, 13 Oct 2002 04:21:51 -0700 Message-ID: <001101c272aa$f5c4aff0$f038aed4@Intellica> References: <4.3.2.7.2.20021008105015.020305d8@208.55.91.110> <4.3.2.7.2.20021012190501.01f0ace8@208.55.91.110> <20021013043134.53303.qmail@cr.yp.to> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Date: Sun, 13 Oct 2002 06:33:47 -0500 X-OldDate: Sun, 13 Oct 2002 14:23:29 +0300 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "Fastream Technologies" To: "FTPEXT Working Group" Subject: Ftp-WG: Suggestion for DELTREE kind of command Content-Transfer-Encoding: 7bit Hello, Being the author of 200,000 client FTP client and server, I wonder if there is any particular reason ofr IETF to not include a command like the old Deltree command to FTP standard that does the traversing the contents and removing the sub directories/files of a directory on the server side? I think most of today's servers are powerful enough for such an operation support. What do you think? G. I. Ates Fastream Technologies http://fastream.com From ftp-wg-owner@hethmon.com Sun Oct 13 08:23:31 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA18528 for ; Sun, 13 Oct 2002 08:23:30 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013072825-827-9 ; Sun, 13 Oct 2002 07:28:26 -0500 Received: from mail.icydata.com (213.137.61.27 [213.137.61.27]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013072357-36031-9 ; Sun, 13 Oct 2002 07:24:00 -0500 Message-Id: <20021013072357-36031-9@mail.hethmon.com> Received: from kingrat [127.0.0.1] by mail.icydata.com [127.0.0.1] with SMTP (MDaemon.PRO.v5.0.5.R) for ; Sun, 13 Oct 2002 15:13:26 +0300 Comments: Authenticated sender is Organization: BG Universal Software MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT In-reply-to: <001101c272aa$f5c4aff0$f038aed4@Intellica> X-mailer: Pegasus Mail for Windows (v2.54) X-MDRemoteIP: 127.0.0.1 X-Return-Path: apl24win@yahoo.com X-MDaemon-Deliver-To: ftp-wg@hethmon.com Date: Sun, 13 Oct 2002 07:24:04 -0500 X-OldDate: Sun, 13 Oct 2002 15:13:24 +0200 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "Teodor Angeloff" To: ftp-wg@hethmon.com Subject: Ftp-WG: unsubscribe Content-Transfer-Encoding: 7BIT unsubscribe From ftp-wg-owner@hethmon.com Sun Oct 13 14:24:28 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA23152 for ; Sun, 13 Oct 2002 14:24:27 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013132936-52200-9 ; Sun, 13 Oct 2002 13:29:39 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013132636-23046-5 ; Sun, 13 Oct 2002 13:26:37 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9DIGeiD007778; Sun, 13 Oct 2002 14:16:40 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9DIGdiw007776; Sun, 13 Oct 2002 14:16:39 -0400 (EDT) In-Reply-To: Your message of Sat, 12 Oct 2002 19:33:34 -0500 Message-ID: Date: Sun, 13 Oct 2002 13:26:42 -0500 X-OldDate: Sun, 13 Oct 2002 14:16:39 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD D. J. Bernstein , Sat, 12 Oct 2002 15:35:23 -0500: : Frank da Cruz writes: : > For decades it has been possible for FTP clients to execute commands like: : > mget pics/mae-west-*.jpg : : And it's still possible. If you receive that command from the user, : simply switch to the pics directory, do an NLST, extract the mae-west-* : names, and retrieve those files one by one. What's the problem? : That the MLSD spec does not permit filenames or mixtures of paths and filenames in its argument; thus the user is not allowed to send this argument to the server in an MLSD command. Note that I don't advocate changing MLSD in this respect. But from the user's point of view, a command that once worked will no longer work and this will cause confusion, help desk involvement, broken scripts, who knows what else. I've suggested some approaches to preserving backwards compatibility as new servers are deployed. : Are you saying that your client tries to push the * processing over to : the server? : The Kermit FTP client already allows for every combination of NLST, MLSD, client-side wildcard interpreation, and server-side wildcard interpretation. I'm not here to avoid work; the work is done. I'm here to look at the implications of MLSD for the end user of command-line FTP clients. If you haven't seen it yet, please look at the table: http://www.columbia.edu/kermit/newftp.html#table : The UNIX ftpd and its offshoot wuftpd have, historically, accounted for : most FTP servers on the Internet, but you shouldn't be relying on their : wildcard quirks. There have always been servers that, quite reasonably : and in full compliance with the protocol, don't imitate ftpd's wildcard : processing. You would already know this if you had tested your client : more thoroughly. : I'm sure you're right but I have never seen them, neither clients nor servers. Here are four non-UNIX examples of servers that interpret wildcards that I was able to verify just this morning: IBM VM/CMS: FTPSERVE IBM VM/CMS Level 14, Service Level 802 DEC/Compaq/HP OpenVMS MultiNet FTP Server Process V4.4(16) UCX 5.1 FTP Server DEC TOPS-20 FTP Server Process 8(50)-1 I have never found a counterexample. In the absence of a complete census of FTP clients and servers, let us agree that clients and servers exist that rely on the client to send the wildcard to the server, and the server to interpret it. When a user of such a client encounters a server that does not intepret wildcards, it is impossible to request a selection of files from the server in a single command. An operation that was possible before is suddenly impossible. : > The server needs a way to suggest the : > appropriate STRU and TYPE for each file to the client. : : No. It works much more smoothly to transfer a binary stream, along with : enough out-of-band information for the client to understand the stream, : as in HTTP. This is another good example of client-side processing being : better than server-side processing. : If a text file is sent in binary mode to a radically different platform, the result is garbage (and vice-versa, of course). Each platform can't be expected to understand the file formats of every other platform. The data format on the wire must be a well-defined common intermediate representation, and luckily that's just how FTP already works. If the server can determine with some certainty whether a file is text or binary, why should it not divulge this information to the client, to whom it is vital? Alun Jones , Sat, 12 Oct 2002 19:33:34 -0500: : : >Then "cdir" needs a stricter definition; it should do one thing or the : >other. Add another fact if it's necessary to do both (and I believe it : >is; the simple relative form for GUI title bars or whatever, and the : >fully qualified form for use by the client in navigating the tree). : : Okay, so maybe the draft doesn't say that "cdir should not be used as a : heading". It also doesn't say that it shouldn't be used as a dessert : topping. : We're going in circles. The topic here was recursive downloads. Using MLSD as defined, it's easy to do them depth-first, but this opens the door to exceeding the limit on open files (yes, this could be programmed around but that's not the question). Some other form of tree traversal could be used if "cdir" (or some new fact) gave a full path that could be CWD'd to at a later time, regardless of context. Ditto, perhaps, for "dir" and "pdir". : The spec does say what cdir's purpose is, it doesn't say what its purpose : isn't. Any 'cdir' form can be used by the client in navigating the tree, : as "CWD " will get you to the directory listed, from the : current directory. : That's what I'd like to see, but it's not what I see in the existing server implementations. You just get "." or "foo", which are useless out of context. The draft merely says cdir "names the directory from which the contents of the listing were obtained" and that the name "MAY" be fully qualified, etc. Is it fully qualified or isn't it? : >FTP protocol requires the client to control everything, but only the : >server knows about its own files. The server needs a way to suggest the : >appropriate STRU and TYPE for each file to the client. This could be : >done in various ways, but the cleanest is to add new facts: : > : > mode= (S, B, C) : > : Why a fact for MODE? : Why not? If FTP has a MODE command with a selection of values, and if, in a certain setting, the value makes a difference, then it would be useful for the server to suggest the appropriate MODE for a given file (if the server knows it) so the user doesn't receive garbage. Again, I wonder if there are any VMS, CMS, or VOS people in this group -- platforms where files do not follow simple the Unix/DOS stream model. : > stru= (F, R, P) : : No objections here, it would seem that this property would come directly : from the filing system - of course, in many popular filing systems, it's : going to be completely extraneous as it'll always be 'F', but there are : systems that might want to communicate that a file is a 'record' format : file. : VMS and CMS are well-known high-profile examples of record-oriented file systems. : > xtype= (A[ x], E, I, L x) : : This is a tricky one to defend, though. How do you tell whether a file : needs Image or ASCII mode? Reliably, I mean, and without using up a whole : heap of resources by scanning every file just to produce a listing. : Who cares? Defining this as an optional fact that can be sent has no bearing on how it is done or if it is done. Windows might do it entirely by registry associations. VMS could do it from information in the FCB (fixed-block files are binary, variable-length-record files are text). Unix might do it by inspection. Years ago I, too, thought inspection was gross. Then I tried it. In Unix and Windows, the effect of looking at (say) the first 48K of a file is barely measurable (unless it's on a floppy disk or something). Kermit, when sending a file (with Kermit or FTP protocol) can categorize it with a great deal of reliability as: Binary 7-bit text (ASCII, ISO 646, Short KOI, ...) 8-bit text (ISO 8859-x, PC code page, KOI-8, HP-Roman8, JIS, EUC, ...) UTF-8 text UCS-2/UTF-16 text Big Endian UCS-2/UTF-16 text Little Endian (Of course users can turn this feature on and off in case it bites them; however, there has never been a complaint.) Incidentally, Kermit has a built-in DIRECTORY command that includes an option to show the "transfer mode" (from the list above) for each file listed. Even on relatively slow Unix platforms (e.g. old Sun-4's) there is no perceptible delay. [preserving permissions...] : Why is it of importance to FTP, though? FTP is not about executing files. : It's about reading and/or writing them, with a touch of deleting, : renaming, etc. But there's no provision for execution (and it's a bad : idea when there is!) : Symmetry. Users want "push" and "pull" with the same capabilities. They want to replicate file trees cross-platform with as many attributes intact as possible by uploading as well as by downloading. Clearly that requires a mapping because not all file systems support the same concepts, and because users should not be expected to know the commands, syntax, and codes of every other platform before they can do this. : >: | Thus the hideously complicated and : >: | obscure user interface described here: : : If we can avoid words like "hideous" or "obscure", we might avoid making : this into an overly emotional discussion, and get down to the issues at : hand. Thanks. : OK (but I was talking about my own server :-) : You know, I think you've got enough here that it's worth starting a new : draft. : That's fine. I would like to be sure the current draft does not close any doors on the issues raised here, and that some of its ambiguities are cleared up (e.g. the format of dir, cdir, and pdir types) before it advances, and maybe a few more file facts included. Perhaps the TYPE fact should become mandatory; what's the point of listing a file if the client can't tell from the list whether it's a file or directory? Unless anybody else has strong feelings to the contrary, I agree the rest can come later. - Frank From ftp-wg-owner@hethmon.com Sun Oct 13 17:54:32 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id RAA25104 for ; Sun, 13 Oct 2002 17:54:31 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013165911-61372-11 ; Sun, 13 Oct 2002 16:59:12 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013165259-32340-5 ; Sun, 13 Oct 2002 16:53:00 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 074016 for ; Sun, 13 Oct 2002 17:42:58 -0400 (EDT) Message-Id: <4.3.2.7.2.20021013162547.01ece868@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Sun, 13 Oct 2002 16:53:08 -0500 X-OldDate: Sun, 13 Oct 2002 16:47:03 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 01:26 PM 10/13/2002, Frank da Cruz wrote: >That the MLSD spec does not permit filenames or mixtures of paths and >filenames in its argument; thus the user is not allowed to send this >argument to the server in an MLSD command. Note that I don't advocate >changing MLSD in this respect. But from the user's point of view, a >command that once worked will no longer work and this will cause confusion, Why? You've given no indication as to why MGET - the "command that once worked" should use anything other than NLST. Hence, MGET will work exactly the same way it does now. >An operation that was possible >before is suddenly impossible. Only if you insist that MGET must use MLSD. Why not stick with NLST, as you are today? MLST / MLSD replaces only one (inappropriate) use of LIST, not NLST. >If a text file is sent in binary mode to a radically different platform, >the result is garbage (and vice-versa, of course). Incorrect. If a text file is sent in binary mode, the result may be a file where the line-ends are not in the local format, but the file is far from being garbage. Tools exist even on DOS - very simple tools - to convert "Mac" and "Unix" style line ends to "DOS" style, and vice-versa. >We're going in circles. The topic here was recursive downloads. No. The topic was wildcards in MLSD. There are wildcards that will produce matches that list the contents of one or more directories; there are wildcards that match some files and some directories. If MLSD is to match wildcards, it must be done reliably, and in a manner that can be replicated by client and server. >: Why a fact for MODE? >: >Why not? If FTP has a MODE command with a selection of values, and if, >in a certain setting, the value makes a difference, then it would be useful >for the server to suggest the appropriate MODE for a given file (if the >server knows it) so the user doesn't receive garbage. Because every mode can be used to transfer any file. Hence, MODE is entirely the choice of the client. >Years ago I, too, thought inspection was gross. Then I tried it. In Unix >and Windows, the effect of looking at (say) the first 48K of a file is >barely measurable (unless it's on a floppy disk or something). Are you the same Frank that was proposing the server manage a million files in a directory? >That's fine. I would like to be sure the current draft does not close any >doors on the issues raised here, and that some of its ambiguities are >cleared up (e.g. the format of dir, cdir, and pdir types) before it >advances, and maybe a few more file facts included. Perhaps the TYPE fact >should become mandatory; what's the point of listing a file if the client >can't tell from the list whether it's a file or directory? The point of making the TYPE fact (or any fact) optional, surely, is so that it doesn't add to the length of a listing if the client doesn't care to know it. The client, not the server, chooses which facts to ask for in its OPTS command. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Sun Oct 13 19:20:20 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA25935 for ; Sun, 13 Oct 2002 19:20:19 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013182456-46574-5 ; Sun, 13 Oct 2002 18:24:57 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013181520-17187-12 ; Sun, 13 Oct 2002 18:15:22 -0500 Received: (qmail 63968 invoked by uid 1016); 13 Oct 2002 23:05:19 -0000 Message-ID: <20021013230519.63967.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Sun, 13 Oct 2002 18:15:29 -0500 X-OldDate: 13 Oct 2002 23:05:19 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Frank da Cruz writes: > But from the user's point of view, a > command that once worked will no longer work Wrong. > the user is not allowed to send this argument to the server You are spouting nonsense. Users don't talk to servers. Users talk to clients. Clients talk to servers. Specifically: (1) The user tells your software ``mget pics/mae-west-*.jpg''. (2) Your software tells the server ``CWD pics'' and ``NLST''. (3) The server tells your software the filenames. (4) Your software extracts the filenames matching mae-west-*.jpg. (5) Your software retrieves those files, one by one. What's the problem? If that isn't how your client software works, your software is broken. You are relying on features that aren't guaranteed by the protocol and aren't provided by all servers. Fix your software. This has nothing to do with the filename list format. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Sun Oct 13 20:38:21 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id UAA26628 for ; Sun, 13 Oct 2002 20:38:19 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013194246-39114-9 ; Sun, 13 Oct 2002 19:42:48 -0500 Received: from watsun.cc.columbia.edu (watsun.cc.columbia.edu [128.59.39.2]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021013193526-17193-11 ; Sun, 13 Oct 2002 19:35:28 -0500 Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id UAA14602; Sun, 13 Oct 2002 20:25:14 -0400 (EDT) In-Reply-To: Your message of Sun, 13 Oct 2002 18:15:29 -0500 Message-ID: Date: Sun, 13 Oct 2002 19:35:36 -0500 X-OldDate: Sun, 13 Oct 2002 20:25:12 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Jeffrey Altman To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD > You are spouting nonsense. Users don't talk to servers. Users talk to > clients. Clients talk to servers. Specifically: > > (1) The user tells your software ``mget pics/mae-west-*.jpg''. > (2) Your software tells the server ``CWD pics'' and ``NLST''. > (3) The server tells your software the filenames. > (4) Your software extracts the filenames matching mae-west-*.jpg. > (5) Your software retrieves those files, one by one. > > What's the problem? > > If that isn't how your client software works, your software is broken. > You are relying on features that aren't guaranteed by the protocol and > aren't provided by all servers. Fix your software. This is not a question of what Kermit does, it is a question of what BSD FTP, Netscape, and IE do since they work in the manner that Frank is describing. Granted at the moment they clients do not support MLSD, but when kiddies get around to implementing MLSD they are going to try to do exactly the things that Frank describes. And when they do they are going to do it because many of the servers that are out there today allow them to misuse the protocol in the ways in which Frank has described. --- On a side note, I'm a bit embarrassed by the behavior of this working group. Kermit has been described by some as the second most portable C language application after "Hello World". This portability and its longevity have resulted in Frank gaining a unique experience in platform capabilities, protocol design, and interoperability. To hear his comments being rebutted with arguments that boil down to "you don't know what you are talking about" or "you can't have things both ways (large and small)" or "its preferable to choose solutions that create N x M situations" is more than a little sad. In many of the response I hear two unspoken assumptions: . clients are no longer small nor dumb . operating systems that do things differently from Unix / Windows will not be created or do not exist Neither one of these assumptions is true. What is true is that clients are becoming even more diverse today than they ever have been. They are both smaller and larger. They use both Unix / Windows style file systems and various object based storage systems. To assume that simply because PC and laptop clients are big enough to handle large amounts of client side processing that it is ok to push processing out to all clients is wrong. It is also wrong to assume that bandwidth to the endpoints can handle large quantities of raw unneeded data. PDAs, mobile devices, and next generation micro devices are just not going to have these capbilities. Someone asked "why IBM would use FTP as the front end for a transaction system?" They use it because in a large number of systems a transaction is a file. This is true is medical applications, financial applications, and military applications. Why are they using FTP? Because they need to move "files" between heterogeneous systems in a secure manner over public networks; AND they cannot control what the client device or operating system will be; therefore they must use an open widely implemented standard protocol. To say that FTP should not be used for this class of application is absurd. I realize that FTP is no longer considered a cool protocol. But it is one of the most widely implemented protocols in the world and those involved in supporting and extending it must keep the bigger picture in mind. Jeffrey Altman * Sr.Software Designer Kermit 95 2.0 GUI available now!!! The Kermit Project @ Columbia University SSH, Secure Telnet, Secure FTP, HTTP http://www.kermit-project.org/ Secured with MIT Kerberos, SRP, and kermit-support@columbia.edu OpenSSL. From ftp-wg-owner@hethmon.com Mon Oct 14 10:25:51 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA21699 for ; Mon, 14 Oct 2002 10:25:50 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021014093545-52173-9 ; Mon, 14 Oct 2002 09:35:46 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021014092942-22998-9 ; Mon, 14 Oct 2002 09:29:44 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9EEJciD009289; Mon, 14 Oct 2002 10:19:38 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9EEJcig009288; Mon, 14 Oct 2002 10:19:38 -0400 (EDT) In-Reply-To: Your message of Sun, 13 Oct 2002 16:53:08 -0500 Message-ID: Date: Mon, 14 Oct 2002 09:29:55 -0500 X-OldDate: Mon, 14 Oct 2002 10:19:37 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Alun Jones , Sun, 13 Oct 2002 16:53:08 -0500: : At 01:26 PM 10/13/2002, Frank da Cruz wrote: : >That the MLSD spec does not permit filenames or mixtures of paths and : >filenames in its argument; thus the user is not allowed to send this : >argument to the server in an MLSD command. Note that I don't advocate : >changing MLSD in this respect. But from the user's point of view, a : >command that once worked will no longer work and this will cause confusion, : : Why? You've given no indication as to why MGET - the "command that once : worked" should use anything other than NLST. Hence, MGET will work exactly : the same way it does now. : I guess I need to spell it out: 1. MGET works by getting a list of files from the server and then RETR'ing each file. 2. New clients will want to use MLSD because of its enhanced list format. 3. For backwards compatibility, clients will want to continue to have an MGET command with the same syntax as before. What is the new client supposed to do with an argument like "pics/m*.jpg"? 1. It can't include it as an MLSD argument because MLSD accepts only directory names. 2. But how is the client supposed to know whether it's a directory name when it's in the syntax of the server's file system? pics/m*.jpg (UNIX) pics:m+.jpg (AOS/VS) pics\m*.jpg (DOS) pics>m*.jpg (VOS) m* jpg 197 (CMS) [.pics]m*.jpg (VMS) pics:m*.jpg (also VMS) <.pics>m*.jpg (TOPS-20) [123,456]m*.jpg (TOPS-10) : >An operation that was possible before is suddenly impossible. : : Only if you insist that MGET must use MLSD. Why not stick with NLST, as : you are today? MLST / MLSD replaces only one (inappropriate) use of LIST, : not NLST. : If MLSD is not for use with MGET then what is it for? It is not a LIST replacement. LIST is for people, MLSD is for machines. I don't insist that MGET use only MLSD -- please go back and read: http://www.columbia.edu/kermit/newftp.html Sometimes NLST is better, sometimes MLSD is better. Thus a client can try to choose the best FTP command in each situation. But it can't always be right when the choice depends on the server's file system syntax, which the client can not be expected to know unless we reduce the world to Unix and Windows. Thus the user must be exposed to the details and tradeoffs of NLST versus MLSD. : >An operation that was possible before is suddenly impossible. : Anyway you took this statement out of context. It applies to the situation in which the client expects the server to expand wildcards (as virtually every client does) but the server interprets RFC959 strictly and does not accpet wildcards. : >If a text file is sent in binary mode to a radically different platform, : >the result is garbage (and vice-versa, of course). : : Incorrect. If a text file is sent in binary mode, the result may be a file : where the line-ends are not in the local format, but the file is far from : being garbage. Tools exist even on DOS - very simple tools - to convert : "Mac" and "Unix" style line ends to "DOS" style, and vice-versa. : If you restrict yourself to Unix and DOS that might be true. Once you allow for other platforms you find: . Variable-length record files with embedded record-length fields in a variety of formats (binary, ASCII numeric, packed decimal) and having different lengths (2 bytes, 4 bytes, ...) . Fixed-length records. . EBCDIC instead of ASCII (and remember there isn't just one EBCDIC, there are hundreds of them). . Some proprietary character set representing a non-English language. If you choose to ignore all but Unix and DOS file systems, you still can't dismiss line-end conflicts -- these affect real users every day, who don't have a clue what to do about them. It is the business of open standards -- and a founding principal of the ARPANET -- to put common intermediate representations on the wire. Once you start going down the road you're suggesting, you might as well abandon the Internet to a single vendor. For the purposes of cross-platform file interchange, the world has settled on distinguishing between text and binary files. It's not ideal but taking this concept further (e.g. converting database formats between one product and another) are beyond the scope of open standards. Thus text files are candidates for conversion to common format for transmission -- and that means both record format AND character set -- and binary files are transmitted without conversion. We have decades of experience to tell us which mode of transmission is appropriate for which file. : >We're going in circles. The topic here was recursive downloads. : : No. The topic was wildcards in MLSD. There are wildcards that will : produce matches that list the contents of one or more directories; there : are wildcards that match some files and some directories. If MLSD is to : match wildcards, it must be done reliably, and in a manner that can be : replicated by client and server. : Again, I'm not suggesting that MLSD accept wildcards; rather I'm proposing that MLSD-like commands be added that do so. Wildcards exist. As you pointed out, RFC959 deals with them, but in a way that has been rejected by virtually every developer. I agree it would be nice to have standard wildcard syntax and semantics -- but it would also be a large undertaking. As matters stand, server-side wildcard interpretation offers an "escape valve" to users who know what they are doing, that lets them accomplish effects that could not be achieved otherwise. In a word, server-side wildcards can be "useful". : >: Why a fact for MODE? : >: : >Why not? If FTP has a MODE command with a selection of values, and if, : >in a certain setting, the value makes a difference, then it would be : >useful for the server to suggest the appropriate MODE for a given file : >(if the server knows it) so the user doesn't receive garbage. : : Because every mode can be used to transfer any file. Hence, MODE is : entirely the choice of the client. : Right, but the server can suggest the most appropriate one, since it might know something the client doesn't. Stream mode is appropriate for stream files, block mode for structured files. : >Years ago I, too, thought inspection was gross. Then I tried it. In : >Unix and Windows, the effect of looking at (say) the first 48K of a file : >is barely measurable (unless it's on a floppy disk or something). : : Are you the same Frank that was proposing the server manage a million files : in a directory? : I'm the same Frank that has discovered that certain ideas have value in certain situations. Situations can exist where the benefits of the server offering useful information about a file outweigh the performance penalties. It is the users of FTP who should make the choice about the tradeoffs based on their needs at the time; it's not our business to make those decisions for them based on assumptions like "performance has precedence over utility." - Frank From ftp-wg-owner@hethmon.com Tue Oct 15 00:56:09 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id AAA11286 for ; Tue, 15 Oct 2002 00:56:08 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015000125-46950-8 ; Tue, 15 Oct 2002 00:01:26 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021014235851-17609-9 ; Mon, 14 Oct 2002 23:58:51 -0500 Received: (qmail 80535 invoked by uid 1016); 15 Oct 2002 04:49:15 -0000 Message-ID: <20021015044915.80534.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Mon, 14 Oct 2002 23:58:59 -0500 X-OldDate: 15 Oct 2002 04:49:15 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Jeffrey Altman writes: > when kiddies get around to implementing MLSD You are continuing to miss the point. The clients you're talking about are broken. They are handling wildcards in a way that * is not guaranteed to work by the FTP protocol and * _does not work_ with some deployed protocol-compliant servers. Those clients should be fixed. I've explained the correct way to handle wildcards. The correct way works with _all_ FTP servers. One example of a working client is ftpfs. > Netscape I'm curious: Since when did Netscape have multiple-file FTP retrieval? > It is also wrong to assume that bandwidth to > the endpoints can handle large quantities of raw unneeded data. You are talking about a ridiculously minor bandwidth issue. We could save more of the Internet's bandwidth by removing the ``P'' on the first line of every HTTP connection. Don't you have any real work to do? > To say that FTP should not be used for this class of application is absurd. FTP should not be used for this class of application. The protocol does not provide disk commitment, for example, so it can't be used for reliable transaction processing. If you aren't using FTP, but rather a private extension to FTP with some transaction features added, you're an incompetent protocol designer. Starting from FTP as the base for a new protocol is like starting from OS/360 as the base for a new operating system. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Tue Oct 15 01:12:33 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id BAA11524 for ; Tue, 15 Oct 2002 01:12:32 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015001734-46450-9 ; Tue, 15 Oct 2002 00:17:36 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015001309-17150-9 ; Tue, 15 Oct 2002 00:13:09 -0500 Received: (qmail 83194 invoked by uid 1016); 15 Oct 2002 05:03:33 -0000 Message-ID: <20021015050333.83193.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Tue, 15 Oct 2002 00:13:16 -0500 X-OldDate: 15 Oct 2002 05:03:33 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Frank da Cruz writes: > 2. But how is the client supposed to know whether it's a directory name > when it's in the syntax of the server's file system? Normal people use _one_ client on _one_ operating system to talk to a variety of servers. They find client-defined wildcard syntax stable and comprehensible, while server-defined wildcard syntax is a royal mess. People who bounce among clients have much bigger interface issues to deal with than the wildcard syntax. > What is the new client supposed to do with an argument like "pics/m*.jpg"? CWD pics and then NLST. This is what ftpfs does. The code is trivial, because ftpfs works with the wildcard handling in the client OS. > If MLSD is not for use with MGET then what is it for? It is not a LIST > replacement. LIST is for people, MLSD is for machines. You are, once again, confusing the client-server communication with the user-client communication. > As matters stand, server-side wildcard interpretation > offers an "escape valve" to users who know what they are doing, that lets > them accomplish effects that could not be achieved otherwise. You are, once again, falsely claiming that something is impossible, when the truth is merely that your broken client doesn't do it. Solution: Fix your client. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Tue Oct 15 06:23:43 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA25506 for ; Tue, 15 Oct 2002 06:23:41 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015052839-63232-11 ; Tue, 15 Oct 2002 05:28:40 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015052530-36445-10 ; Tue, 15 Oct 2002 05:25:32 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g9FAEuI12788 for ; Tue, 15 Oct 2002 17:14:57 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g9FAEjh16242 for ; Tue, 15 Oct 2002 17:14:49 +0700 (ICT) In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <16240.1034676885@munnari.OZ.AU> Date: Tue, 15 Oct 2002 05:25:39 -0500 X-OldDate: Tue, 15 Oct 2002 17:14:45 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Date: Mon, 14 Oct 2002 09:29:55 -0500 From: Frank da Cruz Message-ID: | 1. MGET works by getting a list of files from the server and then | RETR'ing each file. MGET is a local invention by clients (anticipated by the FTP spec for sure). How it works is up to the client. What you have described is the essence, without the detail. That is, it should get a list of files, pick the ones it wants, and RETR each of those. In fact, that's what the typical unix client does (with the human doing the picking). There's also often that broken wildcard nonsense, that works by chance (what's more, it can be useful, I often do "mget *" which fetches all the files in all direct subdirectories of the current directory, but no more than that ... but I know I'm abusing the hell out of the spec and relying on details of a dumb server implementation to make that work). But in any case, that is just additional filtering, the client still needs to be able to filter the list provided by the server. | 2. New clients will want to use MLSD because of its enhanced list format. They might, but that isn't what MLSD was intended for, | 2. But how is the client supposed to know whether it's a directory name | when it's in the syntax of the server's file system? It defines what is what in the syntax it accepts. That's all FTP has ever really allowed. If it doesn't define it itself, then it has to expose the "change directory and refer to file names only" model that the protocol exposes. With a TVFS server on the other hand, the client will be able to make more assumptions, but using a (I hope) well defined specification for exactly what means what. | If MLSD is not for use with MGET then what is it for? It is not a LIST | replacement. LIST is for people, MLSD is for machines. It is a LIST replacement. That was certainly its intent. It is a LIST replacement for when LIST output is needed by machines. That is, to provide a method other than "send LIST, hope the return format is ls -l style, or one of a few others I believe I understand, and parse it" that many programs engage in (mirror programs, GUI interfaces, and such). It was never really intended as a replacement for NLST. Though, I do see how it can be used that way, the way your web page sets out (to save the overhead of extra commands - particularly SIZE which should never be used for cosmetic purposes). But if you're going to use it that way, you really have to use it within its definition. I can't actually see any very compelling reason why a command line style, traditional FTP client, would ever need to use MLSD (I might have it do MLST to get size, mod time, ... before a RETR in one command/response). But GUI clients, that like to show users the entire directory, and let them click on files, are likely to use MLSD lots (rather than parsing LIST output which is what they currently do). | But it can't always be | right when the choice depends on the server's file system syntax, Best is to hide the server's file system syntax from users. De facto that's happened already, everyone now exposes FTP path names as URLs rather than other schemes (then they work in browsers, and other places, not only in (more or less) dedicated FTP clients). | which the client can not be expected to know unless we reduce the | world to Unix and Windows. It is precisely because we can't do that, and the client can't know either, that we have protocols. Those are the agreed conventions that everyone agrees to play by, so the client and the server can get things done, without either having to know anything much about the details of the other. For that to work, you really have to restrict yourself to playing within the rules. You can't just go and take advantage of some "extension" that some server happens to implement, or you're back with the problem of having to know which servers do, and which don't. Just as your implementation can't possibly cope with that (nor could any other) - nor can the human users, and it is unreasonable to ask them to have to figure out what all those wild syntaxes mean. | Thus the user must be exposed to the details and tradeoffs of NLST | versus MLSD. No, for your purposes, you can just ignore MLSD, I think, there's no requirement that a command line client use it. If you want to get to the ultimate in optimisation, then having all those switches for experts to use to save a minute or two here and there is fine, but it really isn't necessary. | It is the business of open standards -- and a founding principal of the | ARPANET -- to put common intermediate representations on the wire. Yes, I agree. That's why we put a single, common, intermediate representation of the directory listing on the wire. And why we don't just send wildcards at the server and hope it interprets them as we intended. | Again, I'm not suggesting that MLSD accept wildcards; rather I'm proposing | that MLSD-like commands be added that do so. Go ahead and propose that - but you're going to have to propose all of the details as well. And that includes the definition of the wildcard syntax (I haven't paid all that much attention, but it has always seemed to me that the NNTP people are spending more time on defining the wildcard stuff in their protocol (extenstions) than on the whole rest of the protocol) It also means defining the result format, which if it is to be compatible with MLSD, would require that you can handle multiple lines referring to the same filename, and distinguish them from a different filenames (and you can't use the unique fact for that, as that is for the underlying file, nor can you really use, I suspect, textual equivalence of the path name). | I agree it would be | nice to have standard wildcard syntax and semantics -- but it would also be | a large undertaking. Yes, large. But not just nice, essential. Just re-read what you wrote above... | It is the business of open standards -- and a founding principal of the | ARPANET -- to put common intermediate representations on the wire. which is exactly what would be required here. Wildcards "work" now, as much as they do, because everyone knows that they're breaching the standards, and hence whatever they do is OK. Mostly implementations just copy what they think some other implementation does, and/or what they believe to be the common subset that users actually use. That's fine as long as no-one can come to you and say "you're supposed to have implemented this, and you haven't". If we had a command that passed wildcards around, that is exactly what would happen. You can also rest upon my IETF experience, when I assure you that there's zero chance of getting something published as an RFC which says anything like "you send this undefined string, and an undefined list of files gets sent back to you". kre From ftp-wg-owner@hethmon.com Tue Oct 15 08:49:13 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id IAA29359 for ; Tue, 15 Oct 2002 08:49:12 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015075349-39066-9 ; Tue, 15 Oct 2002 07:53:51 -0500 Received: from watsun.cc.columbia.edu (watsun.cc.columbia.edu [128.59.39.2]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015074827-17154-12 ; Tue, 15 Oct 2002 07:48:28 -0500 Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id IAA04157; Tue, 15 Oct 2002 08:38:25 -0400 (EDT) In-Reply-To: Your message of Mon, 14 Oct 2002 23:58:59 -0500 Message-ID: Date: Tue, 15 Oct 2002 07:48:35 -0500 X-OldDate: Tue, 15 Oct 2002 8:38:24 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Jeffrey Altman To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD > > It is also wrong to assume that bandwidth to > > the endpoints can handle large quantities of raw unneeded data. > > You are talking about a ridiculously minor bandwidth issue. We could > save more of the Internet's bandwidth by removing the ``P'' on the first > line of every HTTP connection. Don't you have any real work to do? Its not about saving bandwidth on the Internet. Its about saving time and money on links which are much too slow to send large quantities of data. A large part of the world simply does not have the bandwidth you are accustomed to at a University in the USA. Even in the USA, the bandwidth over cellular networks is still severely limited. In most cases to 9600 cps or less. The cost of sending data on the networks is very high. In the third world, and even on most of Europe the cost for sending data is relative either to the amount of data sent or the time of the connection. Reducing the amount of data transfered is an issue in these applications. > > To say that FTP should not be used for this class of application is absurd. > > FTP should not be used for this class of application. The protocol does > not provide disk commitment, for example, so it can't be used for > reliable transaction processing. > > If you aren't using FTP, but rather a private extension to FTP with some > transaction features added, you're an incompetent protocol designer. > Starting from FTP as the base for a new protocol is like starting from > OS/360 as the base for a new operating system. IBM did not extend FTP. Their service works with any FTP client on any Operating System. That is why they are using FTP. Because it is everywhere. It is precisely because IBM did not want to be responsible for designing a new protocol and being forced to implement it on every possible operating system that they used FTP. The reality is that there are few guarrantees in network communications. NFS is not exactly the most reliable means of sharing file systems. Does that means that people do not use it? I think not. Jeffrey Altman * Sr.Software Designer Kermit 95 2.0 GUI available now!!! The Kermit Project @ Columbia University SSH, Secure Telnet, Secure FTP, HTTP http://www.kermit-project.org/ Secured with MIT Kerberos, SRP, and kermit-support@columbia.edu OpenSSL. From ftp-wg-owner@hethmon.com Tue Oct 15 10:05:34 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA02375 for ; Tue, 15 Oct 2002 10:05:32 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015091013-58635-9 ; Tue, 15 Oct 2002 09:10:15 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015090410-29492-9 ; Tue, 15 Oct 2002 09:04:12 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9FDs2A1000837 for ; Tue, 15 Oct 2002 09:54:02 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9FDs2SU000836 for FTPEXT Working Group ; Tue, 15 Oct 2002 09:54:02 -0400 (EDT) In-Reply-To: Your message of Tue, 15 Oct 2002 05:25:39 -0500 Message-ID: Date: Tue, 15 Oct 2002 09:04:20 -0500 X-OldDate: Tue, 15 Oct 2002 9:54:02 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD : Robert Elz ; Tue, 15 Oct 2002 05:25:39 -0500: : | Frank da Cruz ; Mon, 14 Oct 2002 09:29:55 -0500: : | 1. MGET works by getting a list of files from the server and then : | RETR'ing each file. : : MGET is a local invention by clients (anticipated by the FTP spec for sure). : This is what bothers me. Of course what you say is true, but I don't think it's a good idea to ignore implementation, existing practice, and widespread human behavior when modifying a key Internet protocol, if the modifications can affect all that. : | 2. But how is the client supposed to know whether it's a directory : | name when it's in the syntax of the server's file system? : : It defines what is what in the syntax it accepts. : How can it? Either the FTP client knows the file system syntax of the server host or it doesn't. In common practice, most clients simply assume the server to be Unix, which is no basis for an open protocol. If it doesn't know the host's syntax, it can't decompose strings like "[.blah]*.jpg" into CWD and NLST commands (or whatever). : With a TVFS server on the other hand, the client will be able to make : more assumptions, but using a (I hope) well defined specification for : exactly what means what. : This is what we have done in Kermit since 1997 and it works nicely. TVFS is a good and appropriate solution to cross-platform file reference and access. You can map any flat or tree-structured file system to it. Obviously other problems can come up too, like password-protected disks or directories, as in CMS or TOPS-20. Or conflicts between Space as a text character (as in Windows) and as a separator (as in CMS). Or the possibility that some file system uses "/" in identifiers. Mapping rules can be devised, but then of course users must be exposed to them. In other words, users are always going to have to know something about the platform on the other end of the connection, even if wildcards were abolished from the wire. : I can't actually see any very compelling reason why a command line style, : traditional FTP client, would ever need to use MLSD... : ...for your purposes, you can just ignore MLSD, I think, there's no : requirement that a command line client use it. : For recursive downloads. There's no other way to do it. : | It is the business of open standards -- and a founding principal of : | the ARPANET -- to put common intermediate representations on the : | wire. : : which is exactly what would be required here. Wildcards "work" now, : as much as they do, because everyone knows that they're breaching the : standards... : I used FTP for decades without knowing that. I don't believe I'm atypical in that respect. Some consideration has to be given to existing practice. : You can also rest upon my IETF experience, when I assure you that there's : zero chance of getting something published as an RFC which says anything : like "you send this undefined string, and an undefined list of files gets : sent back to you". : That's certainly a compelling argument. Yet it is not applied consistently. Even without wildcards, we have been sending undefined strings back and forth forever, and this will continue with MLSD, unless TVFS notation becomes mandatory. But if that happened MLSD but not for other commands (such as CWD), what would be the point? Again, undefined strings are the "escape valve" that lets users accomplish what they want when the protocol or software stands in their way. They let users to "go over the head" of the client and speak directly to the server. It's like the big boss having an open door policy. It's not necessarily a bad thing. But of course it should not be the only thing. Standard wildcard syntax would be nice -- as opposed to regexp syntax, which is similar but different. Something akin to C-shell wildcards might well cover most cases, but with some consideration to the way "." is a hard separator in most file systems, but not in Unix. And to the way Unix treats filenames that start with ".". And the device-directory distinction. And... and... Anyway, it might turn out that server-side wildcard interpretation is needed in practice, and if that demands a standard wildcard notation, then one will emerge. - Frank From ftp-wg-owner@hethmon.com Tue Oct 15 10:48:09 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA03847 for ; Tue, 15 Oct 2002 10:48:08 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015095245-16829-10 ; Tue, 15 Oct 2002 09:52:46 -0500 Received: from anvil.murkworks.com (anvil.murkworks.com [128.153.43.1]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015094247-57929-14 ; Tue, 15 Oct 2002 09:42:50 -0500 Received: from coal.murkworks.com (coal.murkworks.com [128.153.43.3]) by anvil.murkworks.com (8.9.1/8.9.1) with ESMTP id KAA00763 for ; Tue, 15 Oct 2002 10:19:10 -0400 (EDT) Received: from GIMPELSTIMER/SpoolDir by coal.murkworks.com (Mercury 1.44); 15 Oct 02 10:33:16 -0400 Received: from SpoolDir by GIMPELSTIMER (Mercury 1.44); 15 Oct 02 10:32:50 -0400 Organization: MurkWorks, Incorporated. MIME-Version: 1.0 Message-ID: <3DABEE69.29019.5AE2B66@localhost> References: Your message of Tue, 15 Oct 2002 05:25:39 -0500 In-reply-to: X-mailer: Pegasus Mail for Windows (v4.02-b11) Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Content-description: Mail message body Date: Tue, 15 Oct 2002 09:43:01 -0500 X-OldDate: Tue, 15 Oct 2002 10:32:42 -0400 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "Brad Clements" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Content-Transfer-Encoding: 7BIT On 15 Oct 2002 at 9:04, Frank da Cruz wrote: > Standard wildcard syntax would be nice -- as opposed to regexp syntax, > which is similar but different. Something akin to C-shell wildcards might > well cover most cases, but with some consideration to the way "." is a hard > separator in most file systems, but not in Unix. And to the way Unix > treats filenames that start with ".". And the device-directory > distinction. And... and... Anyway, it might turn out that server-side > wildcard interpretation is needed in practice, and if that demands a > standard wildcard notation, then one will emerge. > > - Frank I think this is a reasonable statement. Would you be willing to write up an "RFC" for "a standard server-side wildcard specification"? Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax AOL-IM: BKClements From ftp-wg-owner@hethmon.com Tue Oct 15 14:46:17 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA11515 for ; Tue, 15 Oct 2002 14:46:16 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015135138-61301-9 ; Tue, 15 Oct 2002 13:51:38 -0500 Received: from mail15b.boca15-verio.com (mail15b.boca15-verio.com [208.55.91.59]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015134900-32278-5 ; Tue, 15 Oct 2002 13:49:01 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15b.boca15-verio.com (RS ver 1.0.63s) with SMTP id 010346 for ; Tue, 15 Oct 2002 14:38:45 -0400 (EDT) Message-Id: <4.3.2.7.2.20021015131220.00ce5e20@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 15 Oct 2002 13:49:10 -0500 X-OldDate: Tue, 15 Oct 2002 13:41:21 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 09:04 AM 10/15/2002, Frank da Cruz wrote: >This is what bothers me. Of course what you say is true, but I don't >think it's a good idea to ignore implementation, existing practice, and >widespread human behavior when modifying a key Internet protocol, if the >modifications can affect all that. To an extent. MLST / MLSD was specifically supposed to ignore implementation, and existing practice, because what is there at present is wildly unreliable. How many versions of "ls -l" are there? That was what MLST and MLSD were supposed to replace - LIST, not NLST - and they were supposed to replace the LIST commands in some means other than declaring one or other "ls -l" to be the standard. Apparently, there is already implementation of wildcards in MLSD on two servers, and at least one client implementation - the question becomes what, if anything, should be done about that? Should we leave an undocumented (and counter-document) assumption in the works, or should we perhaps try and re-emphasize that an FTP client "MUST NOT" pass wildcards to MLSD? Sadly, I don't have access at present to a server that implements wildcards in MLSD so that I can discover and/or demonstrate the failings inherent in unplanned wildcard support in MLSD. As I mentioned earlier, certain assumptions inherent in the MLSD format break as soon as you allow wildcards, so MLSD either needs to change to support wildcards, or the use of wildcards in MLSD needs to be forbidden. >How can it? Either the FTP client knows the file system syntax of the >server host or it doesn't. In common practice, most clients simply assume >the server to be Unix, which is no basis for an open protocol. If it >doesn't know the host's syntax, it can't decompose strings like >"[.blah]*.jpg" into CWD and NLST commands (or whatever). Without TVFS, if a client user types "MGET foo", the client must "NLST foo", and then RETR everything that it sees in response. The same goes no matter what special characters other than "\r\n" are in 'foo'. A compliant MLSD should respond that there is no such directory as "[.blah]*.jpg", rather than try and list files matching the description. >In other words, users are always going to have to know something about >the platform on the other end of the connection, even if wildcards were >abolished from the wire. There's almost certainly room for a specification that says "this command takes a specification that refers to one or more files or directories, and lists the matching items in the following format". MLSD isn't it - so perhaps you'd like to write MLSW? >Again, undefined strings are the "escape valve" that lets users accomplish >what they want when the protocol or software stands in their way. They >let users to "go over the head" of the client and speak directly to the >server. It's like the big boss having an open door policy. It's not >necessarily a bad thing. But of course it should not be the only thing. And, where it is allowed, the results should be as useful as possible. If I ask a wildcard-capable MLSD implementation for "*", am I going to get the names of all the files and directories in the current directory, or am I going to get the names of all files in the current directory as well as the names of all files in every subdirectory? How, as a client, are you going to distinguish the files from one another? In MLSD as it is currently defined, you have no hope. So, either that support of wild-cards is going to be unreliable, or it's going to be to a standard that hasn't been documented. If it's a standard that hasn't been documented, then the implementors need to explain it, and how to rely on it. If it's unreliable, then it should be chopped out, and wildcards should be refused by the server. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Tue Oct 15 16:44:32 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id QAA15293 for ; Tue, 15 Oct 2002 16:44:31 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015154937-58921-9 ; Tue, 15 Oct 2002 15:49:39 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015154555-29754-5 ; Tue, 15 Oct 2002 15:45:56 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9FKZsCD026351 for ; Tue, 15 Oct 2002 16:35:54 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9FKZsYo026350 for FTPEXT Working Group ; Tue, 15 Oct 2002 16:35:54 -0400 (EDT) In-Reply-To: Your message of Tue, 15 Oct 2002 13:49:10 -0500 Message-ID: Date: Tue, 15 Oct 2002 15:46:05 -0500 X-OldDate: Tue, 15 Oct 2002 16:35:53 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Alun Jones ; Tue, 15 Oct 2002 13:49:10 -0500: : At 09:04 AM 10/15/2002, Frank da Cruz wrote: : >This is what bothers me. Of course what you say is true, but I don't : >think it's a good idea to ignore implementation, existing practice, and : >widespread human behavior when modifying a key Internet protocol, if the : >modifications can affect all that. : : To an extent. MLST / MLSD was specifically supposed to ignore : implementation, and existing practice, because what is there at present : is wildly unreliable. : From a theoretical standpoint, perhaps. But people have been using "mget " since time immemorial. As far as I know, they are not complaining. It does not seem to be a problem in search of a solution (and "if it's not broken...") : How many versions of "ls -l" are there? : Personally, I would never use LIST as a basis for MGET. I would never even dream of parsing host-dependent directory listings. I would rather use NLST to get the list of names, and then get whatever info about each file I could from the server by sending SIZE, MDTM, etc. But that is so gross that MLSD is far more attractive for this purpose. : That was what MLST and MLSD were supposed to replace - LIST, not NLST... : It would be silly to say MLSD, when available, can't replace NLST. If it's there, it will be used. Again, I agree MLSD is a fine LIST replacement, but (and this time let me set this off for emphasis)...: --> MLSD is the only way to accomplish recursive MGET <--- : Apparently, there is already implementation of wildcards in MLSD on two : servers, and at least one client implementation - the question becomes : what, if anything, should be done about that? : I think we should consider MLSD already specified except for some details, like the repertoire of facts and the ordering of the list. No wildcards, no filenames; the argument can be empty, or a directory name. (But you might want to say a bit more about exactly what format the directory name be in -- anything accepted by the host? TVFS? Relative but not absolute?... Can I send "MLSD somedisk:[dir1.dir2.dir3]" to a VMS server?) : Sadly, I don't have access at present to a server that implements : wildcards in MLSD... : There's a publicly accessible one at ftp.ipswitch.com (Robert says they will be fixing it to comply with the draft, but the last time I looked a few days ago, it still interpreted wildcards). : >... Either the FTP client knows the file system syntax of the server : >host or it doesn't. In common practice, most clients simply assume : >the server to be Unix, which is no basis for an open protocol. If it : >doesn't know the host's syntax, it can't decompose strings like : >"[.blah]*.jpg" into CWD and NLST commands (or whatever). : : Without TVFS, if a client user types "MGET foo", the client must "NLST : foo", and then RETR everything that it sees in response. : Whoa! In other words you want to say that MLSD "MUST NOT" be used to implement MGET. See what I said about that just above, especially the emphasized part. : There's almost certainly room for a specification that says "this command : takes a specification that refers to one or more files or directories, and : lists the matching items in the following format". MLSD isn't it - so : perhaps you'd like to write MLSW? : I'd *like* to :-) And I'd also like to write MLSR and MLSX. In other words all four combinations of who-interprets-wildcards versus whether-to-recurse. I'd also be willing write a proposal for a standard wildcard syntax if nobody else wants to take it on. I'd also like to keep my job, and those of my group. At the moment this is enough of an issue that I can't commit to doing anything useful within a specific time frame, as I normally would do. In any case, I would like to see at least some allusions to these issues in the draft so implementors have some background, guidance, and perhaps an inkling of "as-yet unresolved issues". - Frank From ftp-wg-owner@hethmon.com Tue Oct 15 18:16:33 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA17232 for ; Tue, 15 Oct 2002 18:16:32 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015172030-59000-5 ; Tue, 15 Oct 2002 17:20:33 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015171203-34271-9 ; Tue, 15 Oct 2002 17:12:04 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 045737; Tue, 15 Oct 2002 18:01:52 -0400 (EDT) Message-Id: <4.3.2.7.2.20021015163051.01f1a4f0@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 15 Oct 2002 17:12:13 -0500 X-OldDate: Tue, 15 Oct 2002 17:05:30 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 03:46 PM 10/15/2002, Frank da Cruz wrote: >Alun Jones ; Tue, 15 Oct 2002 13:49:10 -0500: >: To an extent. MLST / MLSD was specifically supposed to ignore >: implementation, and existing practice, because what is there at present >: is wildly unreliable. >: > >From a theoretical standpoint, perhaps. But people have been using >"mget " since time immemorial. As far as I know, they are not >complaining. It does not seem to be a problem in search of a solution >(and "if it's not broken...") Perhaps you misunderstood. MLST/MLSD is a replacement for LIST. Why you think I was commenting on "mget ", I can't imagine. I'm not DJB. What I was saying is grossly unreliable, and that MLSD hopes to fix, is the myriad of different formats returned by LIST - the format changes from one installation to the next (is your server running GNU's 'ls'? BSD's? Whose? Its own attempt to reproduce one of these?) mget, as you note, does seem to work with wildcards, in part because everyone copies Unix's format. However, even that's not exactly reliable, and many a server author comes across issues where his emulation of Unix's 'ls' is not quite the same as the 'real thing'. "mget *" versus "mget *.*" caused me quite a fuss in my early, Win16, days, when I insisted on interpreting the wildcards as DOS-style wildcards. Similarly with "mget */*", etc. So, yes, people are complaining, both when the wildcard spec matches that of the server's operating system, and when the wildcard spec is forced to match Unix's. However, as you say, it works for the most part, and doesn't need fixing. Why are you so intent to use MLSD as a replacement for NLST? >Personally, I would never use LIST as a basis for MGET. I would never >even dream of parsing host-dependent directory listings. I would rather >use NLST to get the list of names, and then get whatever info about each >file I could from the server by sending SIZE, MDTM, etc. But that is so >gross that MLSD is far more attractive for this purpose. A step partway between the two appears more appropriate. NLST, followed by MLST on each file that you are going to get, so that you can set the modification time, display progress information, etc, as you download. >It would be silly to say MLSD, when available, can't replace NLST. If it's >there, it will be used. Again, I agree MLSD is a fine LIST replacement, >but (and this time let me set this off for emphasis)...: > > --> MLSD is the only way to accomplish recursive MGET <--- And a recursive MGET is not going to benefit from server-side expansion of wildcards, unless I'm missing something obvious. "MGET/RECURSE *.txt" isn't going to pass "MLSD *.txt" and find any directories to recurse. The only way to get a listing of every directory that you'll be wanting to recurse into is to call MLSD with a parameter that allows you to list every directory within your source. In other words, you're just going to be passing in a directory name to list, which is what MLSD, as it is documented, expects to see. >I think we should consider MLSD already specified except for some details, >like the repertoire of facts and the ordering of the list. No wildcards, >no filenames; the argument can be empty, or a directory name. (But you >might want to say a bit more about exactly what format the directory name >be in -- anything accepted by the host? TVFS? Relative but not >absolute?... Can I send "MLSD somedisk:[dir1.dir2.dir3]" to a VMS server?) If the host doesn't support TVFS, you should be sending local format filespecs. That's fairly simple. As to what's displayed in the output from MLSD, it's not supposed to be read by the client and concatenated with anything else, so the server may choose to display only local format, only TVFS, or a combination of both. Of course, what you can't say (and only the server knows to prevent) is "MLSD somedisk:[dir1...]" to a VMS server, because that represents more than one directory. >There's a publicly accessible one at ftp.ipswitch.com (Robert says they >will be fixing it to comply with the draft, but the last time I looked a >few days ago, it still interpreted wildcards). Do you have a Unix one to point to? This one sidesteps the problem by matching wildcards only against the current directory - you can't do "MLSD */*.txt" or similar atrocities. >: Without TVFS, if a client user types "MGET foo", the client must "NLST >: foo", and then RETR everything that it sees in response. > >Whoa! In other words you want to say that MLSD "MUST NOT" be used to >implement MGET. See what I said about that just above, especially the >emphasized part. I think that NLST is adequate for MGET. You want to use MLSD, and you want to use wildcards in MGET. MLSD doesn't do wildcards, therefore is not appropriate for use in expanding wildcards for MGET; therefore it makes no sense to use MLSD for MGET. >I'd *like* to :-) And I'd also like to write MLSR and MLSX. In other >words all four combinations of who-interprets-wildcards versus >whether-to-recurse. Given the examples in the MLST draft refer to MLST / MLSD as "MLSx", perhaps that should be changed, or your choice of name should be revisited. >I'd also be willing write a proposal for a standard wildcard syntax if >nobody else wants to take it on. > >I'd also like to keep my job, and those of my group. At the moment this >is enough of an issue that I can't commit to doing anything useful within >a specific time frame, as I normally would do. In any case, I would like >to see at least some allusions to these issues in the draft so >implementors have some background, guidance, and perhaps an inkling of >"as-yet unresolved issues". We've all got day-jobs. Some of us have night-jobs, too, and raise families. It seems to me, and I know this may sound rude, that you're the one facing the problem. This group loves to take drafts and knock out the bad bits, but nobody wants to write the draft. I think a draft is required, if you want this problem to be solved, and it appears that you're the one interested in writing it. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Tue Oct 15 18:17:27 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA17284 for ; Tue, 15 Oct 2002 18:17:26 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015172120-59003-12 ; Tue, 15 Oct 2002 17:21:21 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015171221-34271-5 ; Tue, 15 Oct 2002 17:12:22 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 045737; Tue, 15 Oct 2002 18:01:52 -0400 (EDT) Message-Id: <4.3.2.7.2.20021015163051.01f1a4f0@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 15 Oct 2002 17:12:29 -0500 X-OldDate: Tue, 15 Oct 2002 17:05:30 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 03:46 PM 10/15/2002, Frank da Cruz wrote: >Alun Jones ; Tue, 15 Oct 2002 13:49:10 -0500: >: To an extent. MLST / MLSD was specifically supposed to ignore >: implementation, and existing practice, because what is there at present >: is wildly unreliable. >: > >From a theoretical standpoint, perhaps. But people have been using >"mget " since time immemorial. As far as I know, they are not >complaining. It does not seem to be a problem in search of a solution >(and "if it's not broken...") Perhaps you misunderstood. MLST/MLSD is a replacement for LIST. Why you think I was commenting on "mget ", I can't imagine. I'm not DJB. What I was saying is grossly unreliable, and that MLSD hopes to fix, is the myriad of different formats returned by LIST - the format changes from one installation to the next (is your server running GNU's 'ls'? BSD's? Whose? Its own attempt to reproduce one of these?) mget, as you note, does seem to work with wildcards, in part because everyone copies Unix's format. However, even that's not exactly reliable, and many a server author comes across issues where his emulation of Unix's 'ls' is not quite the same as the 'real thing'. "mget *" versus "mget *.*" caused me quite a fuss in my early, Win16, days, when I insisted on interpreting the wildcards as DOS-style wildcards. Similarly with "mget */*", etc. So, yes, people are complaining, both when the wildcard spec matches that of the server's operating system, and when the wildcard spec is forced to match Unix's. However, as you say, it works for the most part, and doesn't need fixing. Why are you so intent to use MLSD as a replacement for NLST? >Personally, I would never use LIST as a basis for MGET. I would never >even dream of parsing host-dependent directory listings. I would rather >use NLST to get the list of names, and then get whatever info about each >file I could from the server by sending SIZE, MDTM, etc. But that is so >gross that MLSD is far more attractive for this purpose. A step partway between the two appears more appropriate. NLST, followed by MLST on each file that you are going to get, so that you can set the modification time, display progress information, etc, as you download. >It would be silly to say MLSD, when available, can't replace NLST. If it's >there, it will be used. Again, I agree MLSD is a fine LIST replacement, >but (and this time let me set this off for emphasis)...: > > --> MLSD is the only way to accomplish recursive MGET <--- And a recursive MGET is not going to benefit from server-side expansion of wildcards, unless I'm missing something obvious. "MGET/RECURSE *.txt" isn't going to pass "MLSD *.txt" and find any directories to recurse. The only way to get a listing of every directory that you'll be wanting to recurse into is to call MLSD with a parameter that allows you to list every directory within your source. In other words, you're just going to be passing in a directory name to list, which is what MLSD, as it is documented, expects to see. >I think we should consider MLSD already specified except for some details, >like the repertoire of facts and the ordering of the list. No wildcards, >no filenames; the argument can be empty, or a directory name. (But you >might want to say a bit more about exactly what format the directory name >be in -- anything accepted by the host? TVFS? Relative but not >absolute?... Can I send "MLSD somedisk:[dir1.dir2.dir3]" to a VMS server?) If the host doesn't support TVFS, you should be sending local format filespecs. That's fairly simple. As to what's displayed in the output from MLSD, it's not supposed to be read by the client and concatenated with anything else, so the server may choose to display only local format, only TVFS, or a combination of both. Of course, what you can't say (and only the server knows to prevent) is "MLSD somedisk:[dir1...]" to a VMS server, because that represents more than one directory. >There's a publicly accessible one at ftp.ipswitch.com (Robert says they >will be fixing it to comply with the draft, but the last time I looked a >few days ago, it still interpreted wildcards). Do you have a Unix one to point to? This one sidesteps the problem by matching wildcards only against the current directory - you can't do "MLSD */*.txt" or similar atrocities. >: Without TVFS, if a client user types "MGET foo", the client must "NLST >: foo", and then RETR everything that it sees in response. > >Whoa! In other words you want to say that MLSD "MUST NOT" be used to >implement MGET. See what I said about that just above, especially the >emphasized part. I think that NLST is adequate for MGET. You want to use MLSD, and you want to use wildcards in MGET. MLSD doesn't do wildcards, therefore is not appropriate for use in expanding wildcards for MGET; therefore it makes no sense to use MLSD for MGET. >I'd *like* to :-) And I'd also like to write MLSR and MLSX. In other >words all four combinations of who-interprets-wildcards versus >whether-to-recurse. Given the examples in the MLST draft refer to MLST / MLSD as "MLSx", perhaps that should be changed, or your choice of name should be revisited. >I'd also be willing write a proposal for a standard wildcard syntax if >nobody else wants to take it on. > >I'd also like to keep my job, and those of my group. At the moment this >is enough of an issue that I can't commit to doing anything useful within >a specific time frame, as I normally would do. In any case, I would like >to see at least some allusions to these issues in the draft so >implementors have some background, guidance, and perhaps an inkling of >"as-yet unresolved issues". We've all got day-jobs. Some of us have night-jobs, too, and raise families. It seems to me, and I know this may sound rude, that you're the one facing the problem. This group loves to take drafts and knock out the bad bits, but nobody wants to write the draft. I think a draft is required, if you want this problem to be solved, and it appears that you're the one interested in writing it. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Tue Oct 15 19:07:40 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA18745 for ; Tue, 15 Oct 2002 19:07:39 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015181148-63967-9 ; Tue, 15 Oct 2002 18:11:52 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021015175535-38217-9 ; Tue, 15 Oct 2002 17:55:37 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 088270 for ; Tue, 15 Oct 2002 18:44:44 -0400 (EDT) Message-Id: <4.3.2.7.2.20021015174631.01f21a48@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <4.3.2.7.2.20021015163051.01f1a4f0@208.55.91.110> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Tue, 15 Oct 2002 17:55:45 -0500 X-OldDate: Tue, 15 Oct 2002 17:47:57 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Sorry for the double-post, there - my mail reader puts "you wrote" in the attribution line if I don't do "Reply To All", so I've got into the habit of hitting "Reply To All" in mailing lists. What I need to do, of course, is get into the habit of deleting the extra copy of the address from the headers before sending (goodness only knows why it shoves a copy into "To" and "Cc". Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Wed Oct 16 06:48:00 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id GAA24240 for ; Wed, 16 Oct 2002 06:47:59 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016055303-63122-9 ; Wed, 16 Oct 2002 05:53:05 -0500 Received: from ratree.psu.ac.th (202.12.73.3 [202.12.73.3]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016054949-36390-9 ; Wed, 16 Oct 2002 05:49:53 -0500 Received: from delta.cs.mu.OZ.AU (delta.coe.psu.ac.th [172.30.0.98]) by ratree.psu.ac.th (8.11.6/8.11.6) with ESMTP id g9GAcXR13085 for ; Wed, 16 Oct 2002 17:38:34 +0700 (ICT) Received: from munnari.OZ.AU (localhost [127.0.0.1]) by delta.cs.mu.OZ.AU (8.11.6/8.11.6) with ESMTP id g9GAcLh21338 for ; Wed, 16 Oct 2002 17:38:26 +0700 (ICT) In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <21336.1034764701@munnari.OZ.AU> Date: Wed, 16 Oct 2002 05:50:00 -0500 X-OldDate: Wed, 16 Oct 2002 17:38:21 +0700 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Robert Elz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Date: Tue, 15 Oct 2002 09:04:20 -0500 From: Frank da Cruz Message-ID: | This is what bothers me. Of course what you say is true, but I don't think | it's a good idea to ignore implementation, existing practice, and widespread | human behavior when modifying a key Internet protocol, if the modifications | can affect all that. Which modifications are we talking about here? Aside from the doc of the ancient commands that were never documented (which as best as we know are documented the way they're implemented) all that is being done is creating new commands. Nothing that currently exists (and works) is being eliminated. The draft doesn't say that NLST can't go on accepting wildcards if it wants to (but nor does it say that it should). Further, nothing here is mandatory, it is all optional, both for the servers to implement (though I hope that most will) and for clients to use (there, I'd hope that only clients that have a need for the new functions will use them, not just everyone "because it is there") | : | 2. But how is the client supposed to know whether it's a directory | : | name when it's in the syntax of the server's file system? | : It defines what is what in the syntax it accepts. | How can it? You're missing the point. What I said is that the client defines the syntax it accepts. How can it not? So, the client (user FTP implementation) tells its users "when you give me a string that looks like ... I will treat the yyyy part as a directory name, and the xxxx part as a file name in that directory). That's exactly what should always be done. Expecting every user to understand every syntax for every file system on the planet is absurd. | Either the FTP client knows the file system syntax of the | server host or it doesn't. It doesn't, and shouldn't need to. What it knows is what the FTP spec says can be passed, and in 959 that's just a directory name for CWD, and a file name for RETR (etc). So, the client provides some syntax for the user to enter file names and directory names, and it issues the correct commands to achieve the effect the user has requested. | In common practice, most clients simply assume | the server to be Unix, which is no basis for an open protocol. Of course not, that's what we're trying to get away from here. And that includes the expectation that the server implements some unix server idea of what '*' really means. | If it doesn't know the host's syntax, it can't decompose strings like | "[.blah]*.jpg" into CWD and NLST commands (or whatever). If the client says that the syntax in which it accepts paths is [.directory]filename then of course it can. The client is likely to do that if the native file system on the system it is implemented on uses that syntax, so the end-user can refer to files via FTP using the same general syntax as is used to refer to files on disks. On the other hand, a client on a unix system is likely to expect its users to input /directory/*.jpg which it (should) implement by doing CWD directory NLST RETR a.jpg RETR b.jpg (etc). That the server would have referred to those files as [.directory'a.jpg etc is completely irrelevant, unknown, and unimportant. Any modern client is also going to need to be able to understand and parse ftp://server-name/directory/a.jpg as well of course. This provides a universal file nameing format (which is what URLs and their kin were designed to do of course). Again, the client turns that into the appropriate CWD and RETR commands to fetch the file - regardless of the directory+file name syntax at the server. If the server happens to support TVFS, then the client gets more options in the way it passes the file name to the server. | Or the possibility | that some file system uses "/" in identifiers. Mapping rules can be | devised, but then of course users must be exposed to them. No they needn't. Old Mac filesystems (probably still current ones, but it has been a while) used ':' as the separator, and '/' as just a character. That means that the mac name volume:profit/loss account can be (easily) mapped into /volume/profit:loss account and the users don't have to know that at the server there's anything different. Of course, when advising someone of a name in the FTP namespace, its FTP name needs to be given, but that's not all that hard to deal with (only local users advertising files for FTP need be aware of that). | In other words, users are always going to have to know something about | the platform on the other end of the connection, even if wildcards were | abolished from the wire. We still don't have solutions to all the possible wild file types, so some knowledge may be required. But to just transfer bit images, and then send them back again, no knowledge of the server should be required. | : I can't actually see any very compelling reason why a command line style, | : traditional FTP client, would ever need to use MLSD... | For recursive downloads. There's no other way to do it. Recursive downloads aren't a requirement. Traditional FTP clients haven't offered any facility like that (though some have done it, by LIST parsing I guess). So, I guess yes, if you want to fetch everything (or some definable subset of everything) then I suppose MLSD is a reasonable way to do it. | I used FTP for decades without knowing that. I wasn't talking about users of FTP, but implementors. The user should be able to use wildcards - I have no problem with that at all. The client should make it all work (just as the unix shell makes it all work for unix command line users - and work the same way on unix filesystems, NFS filesystems (that might have almost any semantics imaginable) and even DOS filesystems (where wildcards typically get interpreted slightly differently by the native OS). But to do that, the client doesn't have to do it the stupid way. And client implementors should be aware that that way is unreliable, and unsafe. | That's certainly a compelling argument. Yet it is not applied consistently. If you mean hasn't been, then yes, I'd agree, almost anything used to be able to get published. These days, I suspect it is more consistent. | Even without wildcards, we have been sending undefined strings back and | forth forever, and this will continue with MLSD, unless TVFS notation | becomes mandatory. No, the strings there aren't undefined. Or not in the same way. The string given to RETR (etc) is a file name valid on the server. True we don't know precisely what is legal there, but we know what it means, it is a file name. And when we issue that name, the file named will be retrieved, or an error will occur. If you allow wildcards, without specifying what they mean, then you have no idea what will happen. That is, with MLSW assuming it is created, if I do MLSW * what happens? With no spec for what is a wildcard, and what is just a character, I might get a listing of the file/directory named '*' (or an error if there isn't one), or I might get a listing of all files with no "extensions" or I might get a list of everything (possibly even a list of everything including everything). I have no clue at all what will happen. I don't know whether '*' is a magic character or not. | But if that happened MLSD but not for other commands | (such as CWD), what would be the point? If TVFS is implemented, it is implemented for every command that accepts a file name. | Again, undefined strings are the "escape valve" that lets users accomplish | what they want when the protocol or software stands in their way. They let | users to "go over the head" of the client and speak directly to the server. But that is useless. What would your client do if when you issue RETR abc. The server just decided that names ending in '.' mean that you have abbreviated the name (etc.) and what you're asking for is for every file whose name starts with abc to be sent down the wire. There are sometimes good reasons to allow the user to bypass what the client things is correct, and talk to the server directly, but the client should at least be aware that's what the user is doing, and not be ambushed by something weird that the server implementor, and human client user just happen to have agreed will be treated strangely. | Anyway, it might turn out that server-side | wildcard interpretation is needed in practice, and if that demands a | standard wildcard notation, then one will emerge. Hmm... One hasn't (really) emerged after all these years. That suggests that there is no demand for a standard notation, and that suggests that server side wildcard isn't really needed in practice... kre From ftp-wg-owner@hethmon.com Wed Oct 16 11:36:57 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id LAA03643 for ; Wed, 16 Oct 2002 11:36:56 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016104120-58695-9 ; Wed, 16 Oct 2002 10:41:21 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016103419-29540-11 ; Wed, 16 Oct 2002 10:34:21 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9GFO8Tn028097 for ; Wed, 16 Oct 2002 11:24:09 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9GFO8L2028096 for FTPEXT Working Group ; Wed, 16 Oct 2002 11:24:08 -0400 (EDT) In-Reply-To: Your message of Tue, 15 Oct 2002 17:55:45 -0500 Message-ID: Date: Wed, 16 Oct 2002 10:34:28 -0500 X-OldDate: Wed, 16 Oct 2002 11:24:08 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD OK, I think we're finished here. At this point we are merely repeating ourselves. If I am able, I'll submit a new draft at some future time. - Frank From ftp-wg-owner@hethmon.com Wed Oct 16 13:22:31 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA06679 for ; Wed, 16 Oct 2002 13:22:30 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016122750-46496-9 ; Wed, 16 Oct 2002 12:27:51 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016122629-17205-8 ; Wed, 16 Oct 2002 12:26:30 -0500 Received: (qmail 56983 invoked by uid 1016); 16 Oct 2002 17:16:54 -0000 Message-ID: <20021016171654.56982.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Wed, 16 Oct 2002 12:26:42 -0500 X-OldDate: 16 Oct 2002 17:16:54 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Frank da Cruz writes: > OK, I think we're finished here. No. We've discovered that there's an interoperability failure between your client and some servers. The servers are complying with RFC 959, so the fault lies with your client. Your client has to be fixed. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Wed Oct 16 13:45:03 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id NAA07206 for ; Wed, 16 Oct 2002 13:45:02 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016125028-58742-8 ; Wed, 16 Oct 2002 12:50:29 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016124830-29599-9 ; Wed, 16 Oct 2002 12:48:31 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9GHcVTn011254 for ; Wed, 16 Oct 2002 13:38:31 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9GHcV8c011253 for FTPEXT Working Group ; Wed, 16 Oct 2002 13:38:31 -0400 (EDT) In-Reply-To: Your message of Wed, 16 Oct 2002 12:26:42 -0500 Message-ID: Date: Wed, 16 Oct 2002 12:48:37 -0500 X-OldDate: Wed, 16 Oct 2002 13:38:31 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD > Frank da Cruz writes: > > OK, I think we're finished here. > > No. We've discovered that there's an interoperability failure between > your client and some servers. The servers are complying with RFC 959, > so the fault lies with your client. Your client has to be fixed. > Please describe the problem. I believe the client as it stands handles any combination of server-side and client-side wildcard interpretation and NLST and MLSD, defaulting according to the principle of least astonishment: http://www.columbia.edu/kermit/ckc206.html It's still in Beta so if any problems do exist they can be fixed. - Frank From ftp-wg-owner@hethmon.com Wed Oct 16 14:10:04 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07924 for ; Wed, 16 Oct 2002 14:10:03 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016131518-58101-9 ; Wed, 16 Oct 2002 13:15:19 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016131127-33388-8 ; Wed, 16 Oct 2002 13:11:28 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 040753; Wed, 16 Oct 2002 14:01:21 -0400 (EDT) Message-Id: <4.3.2.7.2.20021016125225.01e2dfa0@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <20021016171654.56982.qmail@cr.yp.to> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Wed, 16 Oct 2002 13:11:36 -0500 X-OldDate: Wed, 16 Oct 2002 13:05:17 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 12:26 PM 10/16/2002, D. J. Bernstein wrote: >Frank da Cruz writes: > > OK, I think we're finished here. > >No. We've discovered that there's an interoperability failure between >your client and some servers. The servers are complying with RFC 959, >so the fault lies with your client. Your client has to be fixed. Actually, it seems that server _and_ client are complying with RFC 959. The client, in good faith, is passing what it believes is a "directory or other system-specific file group descriptor" given by its user, to the server. The server sees an NLST command that holds such a "directory or other system-specific file group descriptor", and responds with a set of file names. The client parses through each file name, and fetches them one by one. RFC 959 does not specify that the "system-specific file group descriptor" is only to mean a folder, directory, or other containing unit. It does not specify that it cannot be a wildcard. "*.txt" would seem to be a valid "system-specific file group descriptor", describing, as it does, a group of files in a system-specific manner. The original author may have intended it to mean "directory" only, but then why say "directory _or_ system-specific file group descriptor"? Sadly, at least one of the authors of RFC 959 is beyond our questions. Certainly it's risky passing a wildcard to the server - the risk is that the server says something akin to "I don't know what you're talking about", and nothing is transferred. But I don't think it's counter to RFC 959. If anything needs to be fixed to be compliant with RFC 959 (and I don't think it does), it's the user who thinks he can pass a wildcard spec in to his FTP client, and have it passed onto, and interpreted by, his server. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Wed Oct 16 14:10:48 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id OAA07943 for ; Wed, 16 Oct 2002 14:10:47 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016131548-58140-10 ; Wed, 16 Oct 2002 13:15:49 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016131129-33388-10 ; Wed, 16 Oct 2002 13:11:32 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 040753; Wed, 16 Oct 2002 14:01:21 -0400 (EDT) Message-Id: <4.3.2.7.2.20021016125225.01e2dfa0@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <20021016171654.56982.qmail@cr.yp.to> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Wed, 16 Oct 2002 13:11:37 -0500 X-OldDate: Wed, 16 Oct 2002 13:05:17 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 12:26 PM 10/16/2002, D. J. Bernstein wrote: >Frank da Cruz writes: > > OK, I think we're finished here. > >No. We've discovered that there's an interoperability failure between >your client and some servers. The servers are complying with RFC 959, >so the fault lies with your client. Your client has to be fixed. Actually, it seems that server _and_ client are complying with RFC 959. The client, in good faith, is passing what it believes is a "directory or other system-specific file group descriptor" given by its user, to the server. The server sees an NLST command that holds such a "directory or other system-specific file group descriptor", and responds with a set of file names. The client parses through each file name, and fetches them one by one. RFC 959 does not specify that the "system-specific file group descriptor" is only to mean a folder, directory, or other containing unit. It does not specify that it cannot be a wildcard. "*.txt" would seem to be a valid "system-specific file group descriptor", describing, as it does, a group of files in a system-specific manner. The original author may have intended it to mean "directory" only, but then why say "directory _or_ system-specific file group descriptor"? Sadly, at least one of the authors of RFC 959 is beyond our questions. Certainly it's risky passing a wildcard to the server - the risk is that the server says something akin to "I don't know what you're talking about", and nothing is transferred. But I don't think it's counter to RFC 959. If anything needs to be fixed to be compliant with RFC 959 (and I don't think it does), it's the user who thinks he can pass a wildcard spec in to his FTP client, and have it passed onto, and interpreted by, his server. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Wed Oct 16 15:03:19 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA09403 for ; Wed, 16 Oct 2002 15:03:18 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016140831-58815-11 ; Wed, 16 Oct 2002 14:08:32 -0500 Received: from watsol.cc.columbia.edu (watsol.cc.columbia.edu [128.59.39.139]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016140524-29701-8 ; Wed, 16 Oct 2002 14:05:25 -0500 Received: from watsol.cc.columbia.edu (localhost [127.0.0.1]) by watsol.cc.columbia.edu (8.12.3/8.12.3) with ESMTP id g9GItNTn019595 for ; Wed, 16 Oct 2002 14:55:23 -0400 (EDT) Received: (from fdc@localhost) by watsol.cc.columbia.edu (8.12.3/8.12.3/Submit) id g9GItNEk019594 for FTPEXT Working Group ; Wed, 16 Oct 2002 14:55:23 -0400 (EDT) In-Reply-To: Your message of Wed, 16 Oct 2002 13:11:37 -0500 Message-ID: Date: Wed, 16 Oct 2002 14:05:33 -0500 X-OldDate: Wed, 16 Oct 2002 14:55:23 EDT Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Frank da Cruz To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD By the way, readers of this list might want to take a look at C-Kermit 8.0.206: http://www.columbia.edu/kermit/ckc206.html (still in Beta) for their own purposes, because it's a useful tool for testing and stressing servers. It offers lots of options and combinations not supported by other clients, and it lets you watch what's happening if you tell it to: set ftp debug on As far as I know, it does not violate RFC959 or the current draft unless you out of your way to make it do so. Again, the commands shown in the table: http://www.columbia.edu/kermit/newftp.html#table allow you to force any combination of server-vs-client wildcard interpretation, NLST, and MLSD, and you can exercise it against any server. Thus, for example, it is possible to force it to send a wildcard as an MLSD argument (e.g. to test whether the server accepts and interprets it), although it will never do this by default. Note that this program is not only, or even primarily, an FTP client so it includes a lot of non-FTP related commands and functions. Most FTP commands work "naturally", in the sense that you can give the same commands at the C-Kermit> prompt that you can give to (say) the Unix FTP client, but to avoid ambiguity you can prefix any FTP command with the word "ftp". Complete documentation of the FTP client is here: http://www.columbia.edu/kermit/ckermit80.html#ftp (features new to the Beta version are in red). A scripting tutorial (which also illustrates many of the commands) is here: http://www.columbia.edu/kermit/ftpscripts.html This version of C-Kermit should be buildable on any Unix platform. - Frank From ftp-wg-owner@hethmon.com Wed Oct 16 18:16:06 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id SAA13640 for ; Wed, 16 Oct 2002 18:16:06 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016172032-49159-8 ; Wed, 16 Oct 2002 17:20:34 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016171454-19850-8 ; Wed, 16 Oct 2002 17:14:55 -0500 Received: (qmail 79221 invoked by uid 1016); 16 Oct 2002 22:05:13 -0000 Message-ID: <20021016220513.79220.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: <4.3.2.7.2.20021016125225.01e2dfa0@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Wed, 16 Oct 2002 17:15:04 -0500 X-OldDate: 16 Oct 2002 22:05:13 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD Alun Jones writes: > The client, in good faith, is passing what it believes is a "directory or > other system-specific file group descriptor" given by its user Incorrect. Perhaps you're not familiar with the programs in question. We're talking about the ``mget'' command supported by the BSD ftp program and its imitators. The mget argument is a wildcard, not a directory name. Or perhaps you're confused by the phrase ``other system-specific file group descriptor'' in RFC 959. That's simply recognizing the fact that some systems back then had other names for directories: disks, mounts, etc. You would have already realized that this doesn't include wildcards if you had read RFC 959 more carefully: the same phrase is used for CWD. Bottom line: Certain client implementors have led the users to believe that mget * will retrieve all files in the current directory, but they are not implementing this function correctly. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Wed Oct 16 19:22:50 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id TAA14844 for ; Wed, 16 Oct 2002 19:22:49 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016182702-63029-8 ; Wed, 16 Oct 2002 18:27:04 -0500 Received: from mail15a.boca15-verio.com (mail15a.boca15-verio.com [208.55.91.57]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021016181857-34012-8 ; Wed, 16 Oct 2002 18:18:58 -0500 Received: from www1525.boca15-verio.com (208.55.91.110) by mail15a.boca15-verio.com (RS ver 1.0.63s) with SMTP id 01313 for ; Wed, 16 Oct 2002 19:08:44 -0400 (EDT) Message-Id: <4.3.2.7.2.20021016175254.01e5a9a8@208.55.91.110> X-Sender: alun.texisc@208.55.91.110 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 In-Reply-To: <20021016220513.79220.qmail@cr.yp.to> References: <4.3.2.7.2.20021016125225.01e2dfa0@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop-Detect: 1 Date: Wed, 16 Oct 2002 18:19:05 -0500 X-OldDate: Wed, 16 Oct 2002 18:12:46 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: Alun Jones To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD At 05:15 PM 10/16/2002, D. J. Bernstein wrote: >Alun Jones writes: > > The client, in good faith, is passing what it believes is a "directory or > > other system-specific file group descriptor" given by its user > >Incorrect. > >Perhaps you're not familiar with the programs in question. We're talking >about the ``mget'' command supported by the BSD ftp program and its >imitators. The mget argument is a wildcard, not a directory name. The client has no way of knowing that. The client takes what is passed as a parameter to mget by the user, and sends it exactly in NLST. Then it takes the response and gets each file. You seem to think that the client somehow "knows" that the user's parameter is flying in the face of RFC 959. It knows no such thing. >Or perhaps you're confused by the phrase ``other system-specific file >group descriptor'' in RFC 959. That's simply recognizing the fact that >some systems back then had other names for directories: disks, mounts, >etc. You would have already realized that this doesn't include wildcards >if you had read RFC 959 more carefully: the same phrase is used for CWD. I've also taken it on previous occasions to mean other, more exotic, things, such as zip files or other archives (really, what's the difference between one of those and a directory?). A wild-card specification is no more or less valid, and a sufficiently perverse mindset could create an implementation wherein "CWD *.txt" might allow you to list/access only files containing ".txt" in their name. It _is_ a descriptor for a group of files. >Bottom line: Certain client implementors have led the users to believe >that mget * will retrieve all files in the current directory, but they >are not implementing this function correctly. Perhaps you can point me to the documentation that has led these users down the garden path. One could just as easily say that certain server implementors have done this, by accepting and expanding wild cards. Well, whatever, it's clear that there's a whole mess caused by the implementation of wild-card support in an environment that wasn't designed for that purpose. MLSD specifically forbids wild-cards for just that reason, it seems. However, there's no reason why an MLSW command couldn't be specified, that leaves the topic of wild-card expansion up to the server. Users aren't all that confused by existing wild-card techniques, and they won't be all that confused by an MLSW command being used for mget in the future, as long as the command is designed specifically to address that need. Alun. ~~~~ -- Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at 1602 Harvest Moon Place | http://www.wftpd.com or email alun@texis.com Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT. From ftp-wg-owner@hethmon.com Sat Oct 19 15:36:24 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id PAA06676 for ; Sat, 19 Oct 2002 15:36:23 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021019144828-8229-10 ; Sat, 19 Oct 2002 14:48:28 -0500 Received: from stoneport.math.uic.edu (stoneport.math.uic.edu [131.193.178.160]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021019144827-43058-9 ; Sat, 19 Oct 2002 14:48:27 -0500 Received: (qmail 63744 invoked by uid 1016); 19 Oct 2002 19:38:50 -0000 Message-ID: <20021019193850.63743.qmail@cr.yp.to> Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html. References: <4.3.2.7.2.20021016125225.01e2dfa0@208.55.91.110> <4.3.2.7.2.20021016175254.01e5a9a8@208.55.91.110> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Date: Sat, 19 Oct 2002 14:48:27 -0500 X-OldDate: 19 Oct 2002 19:38:50 -0000 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group From: "D. J. Bernstein" To: FTPEXT Working Group Subject: Ftp-WG: NLST vs MLSD > However, there's no reason why an MLSW command couldn't be specified, It's awful protocol design. Here's the picture: Protocol Servers Clients without mget Clients with mget -------- --------- -------------------- --------------------- standard no effort no effort parse wildcards, loop fantasy parse wildcards no effort loop your mix parse wildcards no effort parse wildcards, loop The first line is how the standard FTP protocol works. By moving the wildcard handling as close as possible to the user, it puts the burden on the smallest number of programs---the programs that actually care. The second line is how some confused implementors think FTP works. It puts a burden on every server, including servers that would otherwise be substantially simpler, for the benefit of some clients that have to be fairly complicated anyway. The third line, your MLSW proposal, is absurd. It has all the burdens of the first two lines without any of the benefits. > > The mget argument is a wildcard, not a directory name. > The client has no way of knowing that. The client's mget documentation states explicitly that wildcards are allowed. RTFM. > > Bottom line: Certain client implementors have led the users to believe > > that mget * will retrieve all files in the current directory, but they > > are not implementing this function correctly. > Perhaps you can point me to the documentation that has led these users down > the garden path. The Kermit FTP client mget documentation, for example, explicitly allows wildcards. Its implementation of this feature fails to interoperate with some compliant servers. ---D. J. Bernstein, Associate Professor, Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago From ftp-wg-owner@hethmon.com Mon Oct 21 23:51:24 2002 Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by ietf.org (8.9.1a/8.9.1a) with SMTP id XAA23494 for ; Mon, 21 Oct 2002 23:51:23 -0400 (EDT) Received: from mail.hethmon.com (mail.hethmon.com [208.171.56.195]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021021230330-8891-13 ; Mon, 21 Oct 2002 23:03:30 -0500 Message-Id: <20021021230330-8891-13@mail.hethmon.com> Received: from [SMTP.cybernetsurf.net[202.117.121.119]id162087ppp131].for.[cybersurf.net.tw[203.107.131.161]].Errors.to (d121101.ppp121.cyberway.com.sg [203.116.121.101]) by mail.hethmon.com (Hethmon Smtpd Perseus) id 20021021061906 ; Mon, 21 Oct 2002 23:03:28 -0500 Date: Mon, 21 Oct 2002 23:03:29 -0500 Sender: ftp-wg-owner X-Listname: ftp-wg@hethmon.com Reply-To: FTPEXT Working Group Subject: Ftp-WG: