Subject: Ftp-WG: Re: draft-ietf-ftpext-mlst-10 comments

From: "Brad Clements" <[email-masked]>
Date: Mon, 6 Mar 2000 13:35:49 -0500

On 7 Mar 2000, at 4:40, Robert Elz wrote:

>   | 1. with respect to FEAT and RESTart, should implementations that 
>   | support REST BLOCK say so in the feat request? Thats not mentioned | in
>   the draft.
> 
> Modes of REST other than STREAM mode have been defined in FTP since
> forever - if those transfer modes are supported at all, the REST
> ought to be supported with them.   FEAT is only intended to
> advertise features that have been added to FTP since 959 - the theory
> being that everyone supports all of 959 (which is nonsense of
> course, but at least it gives a clear delineation).

Ok. I just thought it might be more complete to mention that servers 
MAY include REST BLOCK and so forth in the FEAT list.

> Ending the fact list with semicolon (rather than space) came very
> late in the evolution of the draft, and was really done to ease
> generation of the fact list, rather than parsing of it.  I doubt
> that anyone even gave a second thought to the effect that would have
> on allowing spaces in the fact list.

Would it make sense to say that feature values that contain special 
characters should be URL encoded?

I know this is too late for this draft, but a follow-on clarification of how to 
handle feature values that contain reserved characters might be useful in 
the future. URL encoded does not excite me, but I think that there 
should be some definition of how to handle arbitrary feature values. If 
this can't get into this draft, is this still the best place to discuss it?

>   | 3. Servers will need to talk to new clients and old clients. If the
>   server | sends a UTF-8 pathname to a client that can't handle it, strange
>   things | may result. | | Should "new" clients specifically request UTF-8
>   pathnames be | enabled using a new option? 
> 
> Clients that use the MLST command are not old ones.   The mere act
> of doing that is asking for UTF-8 pathnames.   The listing commands
> are the only time that the server ever returns path names to the client,

Ok, so you're saying that PWD should not return UTF-8 paths?

I know the doc says that MLST must support UTF-8, but it does not 
mention "legacy" commands such as PWD.  Do you think this may 
leave the issue unclear?

Perhaps in the "pathnames" section it should say that servers and 
clients MUST use ASCII/EBCDIC names for all commands except for 
MLST. 

> Various FTP commands take pathnames as arguments, or return pathnames
>    in responses.  When the MLST command is supported, as indicated in
>    the response to the FEAT command [6], pathnames are to be transferred in
>    one of the following two formats.
> 
>         pathname       = utf-8-name / raw
>         utf-8-name     = <a UTF-8 encoded Unicode string>
>         raw            = <any string not being a valid UTF-8 encoding>

As I read this, I read it to mean that:

if the server includes MLST in the FEAT list, then pathnames (to any 
command or from any command) are either utf-8 or raw.

As I client, I'd infer that MLST appearing in the FEAT list allows me to 
use UTF-8 with any command.

As a server, I'd say that if my FEAT list would return MLST, then I can 
use UTF-8 anywhere as well and might receive UTF-8 pathnames 
anywhere.

I'm sure I've missed this somewhere else in the document (having read it 
rather quickly), but my concern is returning UTF-8 pathnames to old 
clients. (I happen to be operating in an environment where there is a 
difference).

---

Where can I find a client implementation that supports this draft?

Thanks,

-Brad

Brad Clements,                [email-masked]   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
netmeeting: ils://ils.murkworks.com               AOL-IM: BKClements

---

From: Robert Elz <[email-masked]>
Date: Tue, 7 Mar 2000 09:17:48 -0500

Date:        Mon, 6 Mar 2000 13:27:49 -0500
    From:        "Brad Clements" <[email-masked]>
    Message-ID:  <38C3B270.6016.2D969F38@localhost>

[Aside to list members: Brad asked Paul Hethmon and I a set of questions
about the draft - one of which was whether there is a list to discuss it.
I replied, answered most of the questions (I think), and included the
list address - Brad's message to the list was a reply to that reply of
mine - so in a sense, you're coming in in the middle of a brief discussion].

  | Ok. I just thought it might be more complete to mention that servers 
  | MAY include REST BLOCK and so forth in the FEAT list.

I'm not sure it would do a lot of good of itself.  Perhaps better
might be to indicate support of BLOCK mode at all - but that I don't
think is in scope for the current draft.

  | Would it make sense to say that feature values that contain special 
  | characters should be URL encoded?

The draft says somewhere that such things should be encoded (need to
be encoded).  It doesn't say how, as none of the facts so far defined
need any defined encoding of that form.  When a fact that would need
such encoding is defined, then the encoding method will need to be
defined as well - I would expect that a common method for encoding most
facts that might need it would be useful.  But I wouldn't necessarily agree
that one universal encoding method will necessarily be suitable for all
facts ever defined - an ecoding method suitable for a fact that will
occasionally contain an "unusual" character, isn't likely to be best
for encoding a fact that will typically contain mostly "unusual" characters,
and vice versa.

So, I'd prefer to leave that unspecified for now.

  | Ok, so you're saying that PWD should not return UTF-8 paths?

PWD I'd forgotten...   Or rather, I had forgotten until after I sent
the reply.   I'm not sure it how much it matters though.  In most
cases, the current directory won't be down a non-ascii pathname
unless the client has instructed the server to go there.  I know that
it is possible to log in directly to anywhere in the NFVS tree, and
so potentially get any path in the reply.  It would probably be wise
for servers, and server operators to avoid that happening, until at
least most clients understand more than ascii.

  | I know the doc says that MLST must support UTF-8, but it does not 
  | mention "legacy" commands such as PWD.  Do you think this may 
  | leave the issue unclear?

Actually, I thought it did say that all pathnames, in all commands,
would be affected.

  | Perhaps in the "pathnames" section it should say that servers and 
  | clients MUST use ASCII/EBCDIC names for all commands except for 
  | MLST. 

No, that would not be productive - having obtained a listing containing
non-ascii pathnames, how would the client ever manipulate the files
listed?

The FTP server really must deal with the file system it finds under it.
That's always been the case.   If there are are non-ascii file or directory
names, then the server really doesn't have a lot of choice, other than
I guess attempting to encode them somehow, and then deal with receiving
encoded filenames back.

  | As I read this, I read it to mean that:
  | 
  | if the server includes MLST in the FEAT list, then pathnames (to any 
  | command or from any command) are either utf-8 or raw.

The "pathname" in the grammar is relaly only intended to be useful
for this particular doc, 959 doesn't have a grammar.   But in spite of
that, I agree with the conclusion, if not the method of reaching it.

  | As I client, I'd infer that MLST appearing in the FEAT list allows me to 
  | use UTF-8 with any command.

I don't think you need to infer it, I think (I hope...) the draft
actually explicitly says that.

  | As a server, I'd say that if my FEAT list would return MLST, then I can 
  | use UTF-8 anywhere as well and might receive UTF-8 pathnames 
  | anywhere.

Yes.   But that you can does not mean that you necessarily automatically
should, if there is a reasonable alternative.  If there isn't, then you
really have no choice.   If the operators of the filestore have arranged
to implement it in a way that will cause problems for many clients, then
either the filestore will be ignored, or the clients will get upgraded,
depending upon just how desireable it is.

  | Where can I find a client implementation that supports this draft?

I have one, but "support" is truly rudimentary (it allowed be to
exercise the servers that are around, and get the examples that are
included in the draft).   It isn't something I would let a user near
though (or distribute).   Aside from that, I'm not sure.

Since it is possible for a server to support this doc without in any
way altering its appearance to old clients, I expect that updated servers
will appear first, and only when there are reasonable numbers of servers
around will clients that make use of them appear.

kre

---

From: "Brad Clements" <[email-masked]>
Date: Tue, 7 Mar 2000 09:46:06 -0500

On 7 Mar 2000, at 9:17, Robert Elz wrote:

> about the draft - one of which was whether there is a list to discuss it. I
> replied, answered most of the questions (I think), and included the list
> address - Brad's message to the list was a reply to that reply of mine - so
> in a sense, you're coming in in the middle of a brief discussion].

I'm now on this list. I'd like to apologize for "barging in" and copying the 
list in the middle of a conversation, sorry!

>   | Would it make sense to say that feature values that contain special |

> So, I'd prefer to leave that unspecified for now.

Ok. So if a feature appears that will be commonly used and if it contains 
"special characters" (ie, semi-colon or space) then the document 
describing that feature will also describe how it is encoded.

> least most clients understand more than ascii.

Well this is the crux of my problem. In the Novell NetWare environment, 
we can return "long" and "short" names. Short names (AKA DOS 
namespace) won't have non-ascii characters. However we have 
customers demanding support NOW for accented characters in the long 
namespace.

We really need to know if the client can accept UTF-8 in the PWD 
response.

If the client issues an MLSx command, we'll know that they can accept 
UTF-8 pathnames, so PWD will return UTF-8 paths.

Otherwise, we'll have to assume that it's an old client, and not return 
UTF-8 pathnames from PWD. If we encounter a path (which may be 
their starting directory after login) that has non-ascii characters, we have 
to return the short (DOS) form of the pathname.

I think this is not ideal.

> names, then the server really doesn't have a lot of choice, other than I
> guess attempting to encode them somehow, and then deal with receiving
> encoded filenames back.

Our server has a choice since each pathname is represented by at least 
one, but possibly five different namespaces. DOS, MAC, NFS, LONG, 
FTAM

Until we're certain that the client can accept a UTF-8 pathname, we'll be 
forced to use the DOS namespace if the pathname has non-ascii 
characters in the default namespace (ie, LONG)

>   | use UTF-8 anywhere as well and might receive UTF-8 pathnames |
>   anywhere.
> 
> really have no choice.   If the operators of the filestore have arranged to
> implement it in a way that will cause problems for many clients, then

If our server assumes that clients are "old" until identified as new, then 
PWD will return a DOS name if the LONG name has non-ascii 
characters. If the client supports UTF-8, we won't know this until they 
issue an MLSx command.

Most clients start with a SYST and PWD as their first commands, not 
LIST or NLST. Therefore the client, even if it supports UTF-8, MAY get 
the wrong pathname from its first PWD command.

Consider
C> PWD
S> /sys/usr/gunte
C> MLSD 
S> 220 .... 
C> PWD
S> /sys/usr/gunt<220><123>

totally made up, but you can see the problem..

I feel that this part of the draft leaves an ambiguity. What does PWD 
return before the client has issued an MLSx command?

I suppose this is too late to get into the draft, but it would be nice if 
there were a "CLIENT-FEATURE" command that the client could issue, 
at any point, that tells the server what features the client supports. 

In this case, UTF-8 is the only client-side feature that I can think of.

C> CFEA UTF-8
S> 220 Client Feature noted: UTF-8

> around will clients that make use of them appear.

Sure, but it would be nice to be able to test our server with a somewhat 
conforming client, rather than typing everything by-hand.

---

From: Alun Jones <[email-masked]>
Date: Tue, 7 Mar 2000 10:27:46 -0500

>customers demanding support NOW for accented characters in the long

Now, when you say "non-ascii" characters, do you truly mean that Novell's 
"DOS namespace" cannot contain characters outside the range 
32-127?  Certainly, my DOS machine can put those characters into files, as 
can be shown by simply typing "COPY CON <Alt-250>.txt", where "<Alt-250>" 
means holding down the Alt key, while typing "250" on the numeric keypad, 
then releasing the Alt key.  I get a nice little dot in the middle of the 
character - <Alt-160> produces an 'a' with a grave accent, and other 
sequences are possible.

>to return the short (DOS) form of the pathname.

If you have no reason to assume that the client is aware of UTF-8 encoding, 
then you have, as I see it, three choices:
1. Act like an RFC-959 compliant server and return 7-bit characters only.
2. Act like most FTP servers and pass the 8-bit character code in raw, and 
hope that the client can figure it out.
3. Return UTF-8 anyway - most clients are going to be capable of dealing 
with 8-bit character sequences, and UTF-8 is merely an encoding into an 
8-bit character sequence; further, where the characters used are 'regular' 
7-bit ASCII, the UTF-8 encoding is the same as the ASCII encoding.

Either of these options may require you encoding certain characters, but 
does not necessarily prohibit you from using the long filename - as long as 
the encoding is reversible, i.e. that the client can request the name 
provided in a PWD response, or an NLST/LIST listing, to fetch the file.

>characters in the default namespace (ie, LONG)

What does your FTP server do currently (i.e. assuming that this draft 
document had never been written)?

>characters. If the client supports UTF-8, we won't know this until they

And by that time, changing the output of some commands may cause problems 
as they will then disagree with information that the client has cached.

>return before the client has issued an MLSx command?

That's more an issue for the I18N document (*) to address than the MLST 
document; and it does so:

[From the book of RFC, chapter 2640, verse 4.1]
    A new command "LANG" is added to the FTP command set to allow
    server-FTP process to determine in which language to present server
    greetings and the textual part of command responses. The parameter
    associated with the LANG command SHALL be one of the language tags
    defined in RFC 1766 [RFC1766]. If a LANG command without a parameter
    is issued the server's default language will be used.

    Greetings and responses issued prior to language negotiation SHALL be
    in the server's default language. Paragraph 4.5 of [RFC2277] state
    that this "default language MUST be understandable by an English-
    speaking person". This specification RECOMMENDS that the server
    default language be English encoded using ASCII. This text may be
    augmented by text from other languages. Once negotiated, server-PI
    MUST return server messages and textual part of command responses in
    the negotiated language and encoded in UTF-8. Server-PI MAY wish to
    re-send previously issued server messages in the newly negotiated
    language.

In other words, all PWD and other responses must be in "English encoded 
using ASCII" unless the LANG command is issued.

Note that MLST's use of the UTF-8 specification is more to tie it in to the 
RFC 2640 document than it is to mandate sweeping changes to the FTP 
standard in that respect - it is RFC 2640 that mandates those changes.

>S> 220 Client Feature noted: UTF-8

This would be "LANG EN", selecting English as the language, or "LANG" - 
either will indicate to the server that you're able to understand UTF-8.

>conforming client, rather than typing everything by-hand.

And the client authors say "it would be nice to be able to test our client 
with a somewhat conforming server..." :-)  There's somewhat of a 
chicken-and-egg situation here, and as a server author, I do feel your pain 
- personally, I knocked up a quick and dirty hack to an existing FTP client 
code.  I wouldn't feel comfortable releasing it, of course, since it is so 
quick and dirty (everything's hard-coded).

Alun.
~~~~
(*) RFC 2640 makes occasional reference to the "HOST" command being defined 
in the MLST document - this, of course, is now no longer happening.
--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find it
1602 Harvest Moon Place | at web site http://www.wftpd.com or email
Cedar Park TX 78613     | us at [email-masked].  VISA / MC accepted.
Fax +1 (512) 378 3246   | NT based ISPs, be sure to read details of
Phone +1 (512) 378 3246 | WFTPD Pro, NT service version - $100.
*WFTPD and WFTPD Pro now available as native Alpha versions for NT*

---

From: Sergey Ayukov <[email-masked]>
Date: Tue, 7 Mar 2000 12:09:27 -0500

> clients MUST use ASCII/EBCDIC names for all commands except for 

While reading this part of the draft I had a feeling that it has too
clean-mind design. In other words, it totally ignores the current
situation (when 99% of clients/server speak national characters and not
Unicode; some clients (MSIE, Mozilla) do support Unicode but it is mostly
unused). Without real-world test implementation it is hard to predict what
was omitted and what was specified in unreasonable way. IMHO RFC2640 is an
example: I have pretty high doubts whether it will make into FTP
clients/servers. 

Note: you should probably view these comments as
'feelings-at-the-first-glance' and not a flamebait.

> use UTF-8 anywhere as well and might receive UTF-8 pathnames 

Such guessing (caused by vague standard) ought to provoke great mess! Am I
wrong?

> Where can I find a client implementation that supports this draft?

My FTP client does support some features found in the current draft (MLST)
but not the internationalization stuff: I just don't see the
demand. MLST/FEAT support is not enabled by default, you have to set an
option in nftp.ini. Critisms and comments about NFTP are welcome. Also I
suspect that ncftp by Mike Gleason should have MLST support since ncftpd
supports it (draft-05 compliance is claimed).

-- 
Dr. Sergey Ayukov       | NFTP:   Spell differently. Work faster.
http://www.ayukov.com   |
mailto:[email-masked]   | runs on:  BeOS * OS/2 * Unix * Windows

---

From: "Terje Flaarønning" <[email-masked]>
Date: Tue, 7 Mar 2000 14:35:18 -0500

>supports it (draft-05 compliance is claimed).

I've just added MLST/MLSD "support" to my server in the same matter
even tho the draft says UTF-8 must be supported. Until I see the demand
I'll probably do nothing about it.

...Terje...

Terje Flaarønning
-----------------
Tel#: +47-23 00 99 11	Terje Flaarønning
Mob#: +47-91 60 90 25	Bergensgt. 4
Fax#: +47-91 46 35 36	N-0468 Oslo
URL : http://home.eunet.no/~ill

---

From: Sergey Ayukov <[email-masked]>
Date: Thu, 9 Mar 2000 03:48:54 -0500

> Actually, I thought it did say that all pathnames, in all commands,

Sorry? To quote draft-10:

   Various FTP commands take pathnames as arguments, or return pathnames
   in responses.  When the MLST command is supported, as indicated in
   the response to the FEAT command [6], pathnames are to be transferred
   in one of the following two formats.

   Which format is used is at the option of the user-PI or server-PI
   sending the pathname. UTF-8 encodings [2] contain enough internal
   structure that it is always, in practice, possible to determine
   whether a UTF-8 or raw encoding has been used, in those cases where
   it matters.

This looks like raw pathnames are permitted? (Note: and only possibility
to find out whether this is a raw or UTF-8 string is guessing).

I'd like to comment this a little. But first some facts and opinions about
Unicode. I hope it won't trigger the discussion similar to one with Alex
Belits.

It is known that at least in several non-English speaking countries
(examples are Russia and Japan) Unicode in general and its implementations
such as UTF-8 are frowned upon by users. There are several reasons why
this has happened. The most important one is that many programs in wide
use don't support Unicode texts. For example, if I would want to view text
file with UTF-8 contents I'll either have to convert it to 866 or KOI8, or
find the software which does it automatically while viewing. On the other
hand, almost all software will display this file properly if it is already
in 866 or KOI8 encoding and proper fonts are installed -- even when this
software wasn't designed to work with 866 or KOI8. The second most
important reason is huge base of accumulated texts in national encodings.

However, this does not mean that Unicode is useless. It is far from that
because it allows to have single version of the program to work with many
languages and encodings without enormous amount of work which would be
required to support all national encodings.  That's why Unicode is mostly
pursued by large companies who produce multi-language products (to sell in
many countries) and have resources to implement Unicode support. The
examples are operating systems, wordprocessors and browsers. The most part
of the momentum gained by Unicode in Russia is IMHO due to the fact that
it is used/supported in newer versions of Microsoft Word and Outlook. But
since Word has its own document format Unicode continues to be pretty much
contained within Word :-)

Furthermore, wide acceptance of the Unicode would allow to get rid of
several variants of national encodings. For example, Russia uses 5 ones:
866 (DOS, OS/2), 1251 (Windows), KOI8 (Unix), ISO8859-5 (some Unixes, but
now pretty much unused), and Mac. This is great, convoluted mess.

Personally, I'd like to see Unicode to take over the world so that
everything would happen in it (and have 32 bit in byte so that ugly
workarounds like UTF-8 would be unnecessary). But right now I don't see it
happening; for example, my mail client (pine) cannot handle it. At such
level of acceptance IMHO it is unreasonable to demand Unicode support at
the protocol level.

Now let's return to our FTP protocol. Given the current situation (Unicode
is good but national encodings are more popular than it right now) I don't
see any point in tying MLSx support to UTF-8 support. These are in fact
totally independent things and I see no reasons to associate them with
each other. What bad things can happen from this tying? I can imagine two
ones. Firstly, server/client authors who don't want to add Unicode support
(due to real or perceived lack of demand) might choose not to implement
MLSx at all (to be on somewhat safer side of RFC conformance). Secondly,
MLSx could get implemented but without UTF-8 support (this is already
happening). Since draft-10 does not provide a way to find out whether
server really talks UTF-8, aware clients will have to guess. This is bad.

The way I like things to be in FTP protocol: make clients and servers
capable in telling/setting encoding used in filenames and commands,
including PWD, MLSx, LIST(!), DELE etc. I.e. server will have some default
encoding determined during the installation process. Typically that will
be encoding used in the underlying filesystem, but this can also be UTF-8,
or something else (e.g. Unix FTP server could be set up to automatically
translate from KOI8 to 866). When unaware clients connect to it they
operate in that encoding (while not actually knowing that). When aware
client connects to such server, it can either query the default (and if
default is reasonable for it, proceed) or request from server the list of
supported encodings, or simply try to set the encoding it wants to work in
(even if UTF-8 is not an encoding I use this term for all methods of
communicating non-ASCII information). This IMHO would allow for gradual
introduction of UTF-8 into FTP world. Unicode proponents might argue that
this will slow down acceptance of Unicode because servers are going to
talk national encodings first (since these are already in the filesystem,
and no conversion is necessary), but I don't think this is important.

Short summary: we shouldn't push Unicode acceptance, but provide an easy
way to add its support in inambiguous way when necessary. 

Q: Why we shouldn't require UTF-8 support in MLSx. 
A: 1) Because it will also require UTF-8 support in CWD/PWD/DELE/etc. 2)
Because there are certainly going to be a number of semi-broken
implementations which will not actually support Unicode but provide MLSx
interface. 3) Because Unicode has not gained widespread acceptance yet and
old-style national encodings must be permitted and method to determine the
encoding is needed.

Q: Why can't we require UTF-8 support in CWD/PWD/DELETE/etc.?
A: 1) Because it will break existing infrastructure. 2) Because it would
be chicken-and-egg problem: servers won't support it before clients, and
clients won't support it before servers (remember EPLF discussions!)

The RFC2640 is a special case. Sentences like "Another feature is that the
encoding rules make it very unlikely that a character sequence from a
different character set will be mistaken for a UTF-8 encoded character
sequence. Clients and servers can use a simple routine to determine if the
character set being exchanged is valid UTF-8" make me really wonder
whether this RFC pretends to be a standard because it expicitly suggests
guessing at the protocol level, and leaves no possibility to find out
whether guessing has been correct! ("There may be cases when the code set
/ encoding presented to the server or client cannot be determined. In such
cases the raw bytes SHOULD be used.") And as I already wrote requiring
UTF-8 is at least unreasonable at the current level of general Unicode
acceptance. The good thing is that at least FEAT option is included to
indicate UTF-8 support in servers, but clients cannot turn it off! In
other words, i18n-ed server suddenly breaks all existing clients when
dealing with non-ASCII filenames because UTF-8 support is required from
it, is turned on by default, and there's no way to turn it off. I don't
want to comment on the LANG command because I don't think it is important.

Finally, my proposed solutions:

-- problem posed by Brad Clements: support filenames encoded as in
underlying filesystem because users almost certainly are going to work
with them via other ways (not FTP) and consistency is a good thing.
RFC2640 is `clever' enough even to make this compliant with it! Provide an
installation option to turn on Unicode support for users who need to use
many languages at once and are willing to develop/search for new FTP
clients with Unicode which could appear in some (perhaps not very distant)
future;

-- mlst-draft: remove everything related to i18n.

---

From: "Brad Clements" <[email-masked]>
Date: Thu, 9 Mar 2000 08:39:35 -0500

With respect to Sergey's recent post.

I can't comment on the internationalization issues.

I do think that the pathname format being sent by the server, or the 
client, should be un-ambigious. 

The client should be able to tell the server "I'll accept UTF-8, I can send 
UTF-8" and the server should be able to say "I can send UTF-8, I can 
receive UTF-8". 

I know the LANG option was mentioned as a way to do some of this, 
but I think it would be good to eleminate "guess" and path-type 
determination based on inspection.

---

From: Sergey Ayukov <[email-masked]>
Date: Thu, 9 Mar 2000 08:48:31 -0500

On Thu, 9 Mar 2000, Brad Clements wrote:

> I know the LANG option was mentioned as a way to do some of this, but
> I think it would be good to eleminate "guess" and path-type

The LANG option (as defined in RFC2640) talks about server messages (such
as greetings, error messages, etc.) and does not affect the encoding of
the pathnames/filenames.