Subject: Is the user name always UTF-8?

From: "Jeff P. Van Dyke" <[email-masked]>
Date: Tue, 8 Aug 2000 08:07:56 -0600

During the working group meeting last week in Pittsburgh,
I asked why, in the "Password Authentication Method", the
password was encoded in UTF-8, but the "user name" was not.
It was explained why the user name could not be encoded
in UTF-8.

Now, it is possible, that my question was misunderstood
or that I misunderstood the answer, but in any event...

When I returned to the office, I discussed this one of
engineers and he pointed out that it is encoded UTF-8.

After reviewing the draft,

  http://www.ietf.org/internet-drafts/draft-ietf-secsh-userauth-07.txt

I found six uses (see below) of "user name".

Only the first one mentions that it is encoded in UTF-8.

If the "user name" should not be encoded in UTF-8, the
first one should be corrected.

If they all should be encoded in UTF-8, I think the
documentation would be clearer if each one of them
read:

   string    user name (in ISO-10646 UTF-8 encoding)

Jeff P. Van Dyke
[email-masked]

---

2.1.  Authentication Requests

...

  byte      SSH_MSG_USERAUTH_REQUEST
  string    user name (in ISO-10646 UTF-8 encoding)

4.  Public Key Authentication Method: publickey

...

  byte      SSH_MSG_USERAUTH_REQUEST
  string    user name

...

  byte      SSH_MSG_USERAUTH_REQUEST
  string    user name

5.  Password Authentication Method: password

...

  byte      SSH_MSG_USERAUTH_REQUEST
  string    user name
  string    service
  string    "password"
  boolean   FALSE
  string    plaintext password (ISO-10646 UTF-8)

...

  byte      SSH_MSG_USERAUTH_REQUEST
  string    user name

6.  Host-Based Authentication: hostbased

...

  byte      SSH_MSG_USERAUTH_REQUEST
  string    user name

---

From: Tero Kivinen <[email-masked]>
Date: Wed, 9 Aug 2000 16:55:11 +0300 (EET DST)

> documentation would be clearer if each one of them

I think we should encode all of them in UTF-8 ISO-10646. We might need
to add some text saying about the canonicalization process, i.e the
server is assumed to do canonicalization to any format it is using
internally. Client can send it out in any format it likes. I.e if
clinet sends either "Yl<o-umlaut>nen" or "Ylo<add umlaut to previous
letter>nen" the server should convert them to the format it is using
internally (lets say it is using iso latin 1, so it should convert
both of those to "Ylönen". We don't have that much problem there than
they have in the idn wg, because usernames are case sensitive, i.e we
don't need to specify how to lowercase them... 
-- 
[email-masked]                               Work : +358 303 9870
SSH Communications Security                  http://www.ssh.fi/
SSH IPSEC Toolkit                            http://www.ssh.fi/ipsec/

---

From: RJ Atkinson <[email-masked]>
Date: Thu, 10 Aug 2000 18:06:18 -0400

At 09:55 09/08/00 , Tero Kivinen wrote:
>both of those to "Ylönen". 

SSH would still need some form of canonicalisation rules for
on the wire purposes, however.  Or am I missing something obvious ?

There is likely some UNICODE Tech Report that could be used for
canonicalisation and cited in the applicable SSH RFCs.  My guess is
that the Application ADs have real opinions about internationalisation
and canonicalisation.  It might be easier to seek advice now
rather than have RFCs derail during IESG review later on, IMHO.

Ran
[email-masked]

---

From: [email-masked] (Niels M�ller)
Date: 11 Aug 2000 10:12:42 +0200

> > documentation would be clearer if each one of them

The draft-ietf-secsh-architecture-05.txt document says some more about
user names:

: The client and server user names are inherently constrained by what the
: server is prepared to accept.  They might, however, occasionally be
: displayed in logs, reports, etc.  They MUST be encoded using ISO 10646
: UTF-8, but other encodings may be required in some cases.
! Straight bit-wise binary comparison is RECOMMENDED.

> to add some text saying about the canonicalization process,

I agree, and I think the sentence marked with ! above should be
changed. And the "but other encodings may be required in some cases."
should be deleted, unless someone can provide a really good reason for
it.

> We don't have that much problem there than they have in the idn wg,
> because usernames are case sensitive, i.e we don't need to specify
> how to lowercase them...

Is anybody here updated on the idn wg?

/Niels

---

From: [email-masked] (Niels M�ller)
Date: 11 Aug 2000 11:07:55 +0200

> on the wire purposes, however.  Or am I missing something obvious ?

You could push canonicalisation to the recieving side, i.e. specify
that the reciever is required to handle all variants. If the reciever
implements this by canonicalization of received names,
canonicalization must be done *after* signature verification, but
that's quite obvious and should not be a problem.

At least I don't see that non-canonical strings on the wire causes any
fundamental problems, as long as the recievers do the right thing.

> rather than have RFCs derail during IESG review later on, IMHO.

I have the unicode-3.0 book in front of me. The problem is that there
are several ways to "canonicalize" unicode strings:

* Are short or long representations of characters like "ö" preferred?

* What to do with compatibility characters? (I'd think it's best to
  map them to non-compatibility characters as well, i.e. using the
  compatibility-equality rather than canonical-equality).

* What to do with hangul syllables. Compose or decompose?

So the details must be spelled out, somewhere.

And then there's utf-8, but it should be straight forward to say that
an implementation must always use the shortest utf-8 sequence
representing a unicode character (that's the usual way to use utf-8, I
guess, but utf-8 decoders in other contexts may be more forgiving).

Of course, it would be a good thing to use the same rules in other
protocols that have the same problem. An "IETF unicode and
canonicaliztion guidelines" rfc would be great...

/Niels

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 11 Aug 2000 16:41:28 +0300 (EET DST)

> that's quite obvious and should not be a problem.

The username is only a string of characters from the client point of
view. The server is the only one that tries to compare it to something 
it has in its internal database (password file etc). 

> fundamental problems, as long as the recievers do the right thing.

Correct. 

> So the details must be spelled out, somewhere.

I don't think so. The canonicalization problem is really an internal
issue for the server. If the server stores the usernames in the
password file as a Iso-latin1 strings, it needs to take the UTF-8
encoded unicode username and convert it to iso-latin1, and if there is 
no way to convert it to iso-latin1 then it just rejects the
authentication request...

So it is internal implementation issue, and there is no need to say
anything about it in the draft. 

> guess, but utf-8 decoders in other contexts may be more forgiving).

That doesn't affect the protocol either at all. Everything that is
signed, hashed etc in the protocol are done in the same octect string
representation they had on the wire and thats it. After that it is
servers duty to do whatever it likes to convert that username to the
format it is internally using.

Of course we might want to add that kind of text just to remove one
possible place which might cause interoperability problems. 
-- 
[email-masked]                               Work : +358 303 9870
SSH Communications Security                  http://www.ssh.fi/
SSH IPSEC Toolkit                            http://www.ssh.fi/ipsec/

---

From: [email-masked] (Niels M�ller)
Date: 11 Aug 2000 16:46:48 +0200

> authentication request...

Ok, this approach makes sense (btw, it's the way lsh works, except
that its conversion from utf8 to latin1 is broken).

> anything about it in the draft. 

The architecture draft says about user names:

: They MUST be encoded using ISO 10646 UTF-8, but other encodings may
: be required in some cases. It is up to the server to decide how to
: map user names to accepted user names. Straight bit-wise binary
: comparison is RECOMMENDED.

It's the last sentence that makes canonicalization an important issue.
The current spec can be interpreted as allowing the client to send
non-canonical usernames, *and* also allowing the server to perform
straight bit-wise comparisons when looking up user names. Which is
broken.

We have to require more work on one of the sides, i.e. either

(i) Require clients to send canonical strings only, so that the server
    can use plain bit-wise comparison, or

(ii) Allow clients to use any unicode representation they like, and
     require that the server treat all equivalent variants in the same
     way. 

Either approach is fine with me, but the spec has to specify one or
the other.

If we go with (i), the canonicalization rules has to be spelled out in
detail. With (ii), most of those details are irrelevant, but I still
think it is a good idea to specify if servers are expected to apply
canonical equivalence or compatibility equivalence.

I'll make a conrete proposal: Replace

  "Straight bit-wise binary comparison is RECOMMENDED."

with

  "A server (or in general, any receiver of an utf-8 string) SHOULD
   respect unicode character equivalence".

Please note that this affects also systems where only plain ascii.
user names are suported! Greping UNIDATA2.TXT, there are at least 69
"strange" characters that are _canonically_ equivalent to ascii
sequences. There are also 68 compatibility characters (tagged as
<compat> in the file), and even more if equivalences tagged as <font>,
<super>, <wide> etc are counted.

I'll not give the whole list, but I few examples for those of you that
are not familiar with unicode:

 0132;LATIN CAPITAL LIGATURE IJ;Lu;0;L;<compat> 0049 004A;;;;N;LATIN CAPITAL LETTER I J;;;0133;
 01F3;LATIN SMALL LETTER DZ;Ll;0;L;<compat> 0064 007A;;;;N;;;01F1;;01F2
 037E;GREEK QUESTION MARK;Po;0;L;003B;;;;N;;Erotimatiko;;;
 203C;DOUBLE EXCLAMATION MARK;Po;0;ON;<compat> 0021
 ;;;;N;;;;;
 20A8;RUPEE SIGN;Lt;0;ET;<compat> 0052 0073;;;;N;;;;;
 2116;NUMERO SIGN;Lt;0;ON;<compat> 004E 006F;;;;N;NUMERO;;;;
 212A;KELVIN SIGN;Lu;0;ON;004B;;;;N;DEGREES KELVIN;;;;
 2160;ROMAN NUMERAL ONE;No;0;L;<compat> 0049;;;1;N;;;;2170;

Of these examples, 037E and 212A are _canonically_ equvalent to ascii
characters, and the latter is a character that may well occur in a
user name or password.

/Niels

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 11 Aug 2000 16:41:28 +0300 (EET DST)

Correct.

---

From: [email-masked] (Niels M�ller)
Date: 11 Aug 2000 16:46:48 +0200

with

/Niels

---

From: Tero Kivinen <[email-masked]>
Date: Wed, 9 Aug 2000 16:55:11 +0300 (EET DST)