Subject: Fixing exchange of host keys in the SSH key exchange

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Wed, 22 Mar 2017 22:09:54 -0600

Hello everyone,

I’m wondering if anyone - OpenSSH and PuTTY in particular? - would be 
interested in implementing an extension to the way the initial SSH handshake 
is done, in a way that would allow the SSH server to require the client to 
communicate one or more fingerprints of the server’s host key(s), before 
exchanging KEXINIT.

This would have multiple benefits:

(1) Security benefit: although the client could still not verify the server’s 
signature during key exchange, or could do any number of other things 
incorrectly; under reasonable assumptions, if the client has to communicate 
one or more fingerprints of the server’s host key(s) before KEXINIT, this 
prevents the situation where the client will present the user with a host 
key verification dialog, and the user will just click through it.

(2) Security benefit: on servers that enable this policy, and prevent 
connections where the client does not communicate a correct server host key, 
drive-by password guessing becomes largely impossible, since unknown 
attackers do not know the server’s host key.

(3) Practical benefit: the server knows which ones of its host keys the 
client trusts, and can avoid the situation where introduction of a new host 
key, with no change to previous host keys, would break connections for 
existing clients that prefer to negotiate the algorithm of the new host key, 
but do not actually trust it. The server can avoid this by restricting its 
list of host key algorithms presented in KEXINIT to match what the client 
already trusts.

Implementation:

We have room for textual data that can be sent before the SSH version 
string. To implement this, the client would send one or more lines such as:

Expect-Key: <base64-sha256-fingerprint>\r\n

A server that implements this as a requirement would wait for the 
“Expect-Key” lines before sending its KEXINIT. If the client just sends a 
version string, without at least one correct Expect-Key, the server could 
disconnect - perhaps with an informative SSH_MSG_DISCONNECT message.

The rest of the protocol could remain unchanged. Server implementations that 
do not know about this extension, but correctly implement the spec, would 
ignore the unknown lines, and would be unaffected.

If the rest of the protocol is unchanged, the Expect-Key lines will NOT be 
verified as part of the SSH key exchange hash, which means a "helpful" 
man-in-the-middle (such as a proxy) could insert Expect-Key lines and cause 
the connection to work, when it otherwise wouldn't. This might be a feature, 
or a misfeature.

However, we could also change the protocol to include the Expect-Key lines 
in the SSH key exchange hash. In this case, the client would signal support 
by sending at least one Expect-Key line, and the server would signal support 
in another way. One way would be to include the name "expect-key" among its 
host key algorithms in KEXINIT.

If the client has sent at least one Expect-Key line, and the server includes 
the name "expect-key" in its host key algorithms, the Expect-Key lines would 
become part of the SSH key exchange hash.

denis

---

From: Mouse <[email-masked]>
Date: Thu, 23 Mar 2017 08:24:24 -0400 (EDT)

> Iâ??m wondering if anyone - OpenSSH and PuTTY in particular? - would
> be interested in implementing an extension to the way the initial SSH
> handshake is done, in a way that would allow the SSH server to
> require the client to communicate one or more fingerprints of the
> serverâ??s host key(s), before exchanging KEXINIT.

> (1) Security benefit: [...] if the client has to communicate one or
> more fingerprints of the serverâ??s host key(s) before KEXINIT, this
> prevents the situation where the client will present the user with a
> host key verification dialog, and the user will just click through
> it.

How?  As far as I can see, in order to do that, you have to move that
decision from the client to the server.  This carries its own prices.

> connections where the client does not communicate a correct server
> host key, drive-by password guessing becomes largely impossible,
> since unknown attackers do not know the serverâ??s host key.

Yes...but if the server does that, it also becomes impossible for the
client to do the now-usual "just accept the key on the first
connection" thing.  You have to provide the host key to the client by
some out-of-band means.

This of course could be an upside or a downside, depending.

> (3) Practical benefit: the server [...] can avoid the situation where
> introduction of a new host key, with no change to previous host keys,
> would break connections for existing clients that prefer to negotiate
> the algorithm of the new host key, but do not actually trust it.

True.

Another possible benefit would be automatic defeat of
host-key-gathering bots.  My logs are full of clients that connect, get
my host key, and then disconnect.  A few of them are honest about what
they're doing ("SSH-2.0-ZGrab ZGrab SSH Survey", or
"SSH-2.0-OpenSSH-keyscan"); others are...less so ("SSH-2.0-PUTTY").
Most just report a library version (eg, "SSH-2.0-sshlib-0.1").

Or, rather, clients used to be doing that; I now disconnect such
clients before giving them any host keys (I prefer to not talk about
the details of what I've configured my server to require to get it to
talk to a specific client).

> Expect-Key: <base64-sha256-fingerprint>\r\n

I strongly suggest that it instead be something like

Expect-Key: <hash-algorithm-name> <base64-hash>

so that when SHA256 finally shows weaknesses, it does not become a
major headache to switch hash algorithms.  If the server thinks the
hash algorithm is too weak, or doesn't recognize it at all, it can
ignore that line.  The client can of course send hashes of the same key
under multiple algorithms, if it wants.

I can't help wondering if perhaps this is time to use the uint32 in
SSH_MSG_KEXINIT that is "0 (reserved for future extension)", though,
rather than shoehorning it into something that's currently outside the
protocol (almost) entirely.  I see no guidance in 4253 for
implementation behaviour if that value isn't 0 - but see below.

Also, 4253 says the server MAY send text before its version number, but
is silent on the question of whether the client also may.  (I suspect
most server implementations share that code with their corresponding
client implementations, though.)

> However, we could also change the protocol to include the Expect-Key
> lines in the SSH key exchange hash. In this case, the client would
> signal support by sending at least one Expect-Key line, and the
> server would signal support in another way.  One way would be to
> include the name "expect-key" among its host key algorithms in
> KEXINIT.

The server has at least as much room to send text before its version
string as the client does.  It seems more elegant to me for the server
to signal support with something like

Accept-Expect-Key: yes

there - or perhaps to use that RFU uint32 in SSH_MSG_KEXINIT if the
client signals support (which would avoid the issue of an unaware
client printing that and thus confusing the user and possibly breaking
scripted uses, and also restrict nonzero values to implementations that
have already advertised awareness of this extension).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 24 Mar 2017 08:02:39 +0000

Mouse <[email-masked]> writes:

>Another possible benefit would be automatic defeat of host-key-gathering
>bots.  My logs are full of clients that connect, get my host key, and then
>disconnect.  A few of them are honest about what they're doing ("SSH-2.0-
>ZGrab ZGrab SSH Survey", or "SSH-2.0-OpenSSH-keyscan"); others are...less so
>("SSH-2.0-PUTTY"). Most just report a library version (eg, "SSH-2.0-
>sshlib-0.1").

I assume this is for weak-key-checking/key-sharing detection for research
purposes, or is there some malicious use for the info?

>SSH_MSG_KEXINIT that is "0 (reserved for future extension)", though, rather
>than shoehorning it into something that's currently outside the protocol
>(almost) entirely.  I see no guidance in 4253 for implementation behaviour if
>that value isn't 0 - but see below.

I think this was discussed in the context of SSH extensions and the conclusion
was that far too many things would break if this was nonzero.  So even though
it's marked as RFU, in practice it's "always set to zero".

Peter.

---

From: Mouse <[email-masked]>
Date: Fri, 24 Mar 2017 07:24:46 -0400 (EDT)

>> host-key-gathering bots.  [...]
> I assume this is for weak-key-checking/key-sharing detection for
> research purposes, or is there some malicious use for the info?

Well, I don't run one, and have never (knowingly) communicated with
anyone who does, so I don't actually know the intent behind them.  But
the way so many different hosts have been connecting and getting dumped
(and blocked at my border) after so long with the block in place makes
me think much of it is done by one or more botnets, which makes it
difficult for me to believe it's not malicious.  The few that are at
least a little honest about what they're doing (eg, ZGrab) have pretty
much gone away since the block went in.

Possible malicious use would be to break the host keys for hosts that
have too weak a key (whatever "too weak" means for the botnet herder in
question).

As for research, I don't consider it acceptable to co-opt others' hosts
for research without verifying their consent first, especially not when
it involves trawling for security-sensitive material.  I see no reason
anyone whom I don't want to actually authenticate to me should have a
copy of my host keys.

>> SSH_MSG_KEXINIT that is "0 (reserved for future extension)", [...]
> I think this was discussed in the context of SSH extensions and the
> conclusion was that far too many things would break if this was
> nonzero.  So even though it's marked as RFU, in practice it's "always
> set to zero".

Well, it seems to me that it could be used if the other end first
indicates support for whatever the use is.  There won't be many such,
since either that has to be done in the clear or it has to not apply to
the first kex, but this could be an example: the client indicates its
support before the server has to generate its KEXINIT packet.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 24 Mar 2017 07:22:37 +0000

denis bider (Bitvise) <[email-masked]> writes:

>prevents the situation where the client will present the user with a host key
>verification dialog, and the user will just click through it.

It also makes things a lot more painful for the user.  If you regularly need
to SSH into a dozen different devices to administer them, it gets even worse.

>attackers do not know the server’s host key.

This is really just an awkward form of port-knocking.  If you want to use a
port-knocking-style mechanism, you should probably do it properly.  If you
want to bind a client to a server ("only this client, only this server") then
there are better ways of doing it than this.

>Server implementations that do not know about this extension, but correctly
>implement the spec, 

How do we know what percentage of servers do this?  It's true that the spec
says you should expect other stuff in the textual data, but since pretty much
everything out there only ever sends the SSH ID string, it's quite likely that
lots of implementations will choke on unexpected extra lines of text.  Even in
cases where the code does try and handle non-SSH-ID lines, the absence of
anything that sends them means it's probably never been tested.

This doesn't seem like a good idea, it has the potential to break lots of
things, while what you're getting in return could be done better in other
ways.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Sat, 25 Mar 2017 09:56:38 +0000

Mouse <[email-masked]> writes:

>Well, it seems to me that it could be used if the other end first indicates
>support for whatever the use is.  There won't be many such, since either that
>has to be done in the clear or it has to not apply to the first kex, but this
>could be an example: the client indicates its support before the server has
>to generate its KEXINIT packet.

Hmm, so you need to signal that you want to use the RFU field to signal that
you want to do something else?

https://www.youtube.com/watch?v=AW2xyXl2tTI

Prepare to use the RFU field!

Peter.

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Sat, 25 Mar 2017 12:28:23 -0600

Mouse:

> > if the client has to communicate one or more fingerprints
> > of the server's host key(s) before KEXINIT, this prevents
> > the situation where the client will present the user with
> > a host key verification dialog, and the user will just
> > click through it.

> How?

A server that enforces this policy (not all would) will reject the 
connection outright if the client fails to send at least one correct 
Expect-Key. The client must therefore have it.

> Yes...but if the server does that, it also becomes
> impossible for the client to do the now-usual "just
> accept the key on the first connection" thing.

Preventing this, and requiring a stringent host key setup, is the central 
feature of this mechanism. It's an option that some of our users would like.

> You have to provide the host key to the client by
> some out-of-band means.

Yes. The idea is to make sure this actually happens.

> host-key-gathering bots.

Interesting. Yes, that would be a side effect for hosts that configure this 
policy.

> Expect-Key: <hash-algorithm-name> <base64-hash>

Agreed. Yes, definitely - for the reasons you described.

> I can't help wondering if perhaps this is time to use
> the uint32 in SSH_MSG_KEXINIT that is "0 (reserved for
> future extension)",

Unfortunately, there exist implementations that disconnect, or generate an 
invalid key exchange hash, if that value is not zero. When I last checked in 
2015, libssh was one.

> Also, 4253 says the server MAY send text before its
> version number, but is silent on the question of
> whether the client also may.

Hmm, keen observation. Of course, even if it did say otherwise, there could 
exist server implementations that do not handle this properly. It seems that 
testing would be in order.

> It seems more elegant to me for the server to signal
> support with something like
> Accept-Expect-Key: yes

Or maybe the server sends something like "Expect-Key: support|require". The 
format can vary between client and server. That might still not work if 
there are clients that violate 4253, though.

Another way that would work for sure would be:

- The server includes "expect-key" among its host key algorithms. The client 
never includes this algorithm. Instead it just sends signature algorithms 
that correspond to keys it knows for this server.

- If client sees "expect-key" among server's host key algorithms, then 
immediately after its own KEXINIT, it sends a newly defined message, 
SSH_MSG_EXPECT_KEYS, which contains fingerprints of host keys trusted for 
this server.

The server can now act according to its policy, based on whether it receives 
SSH_MSG_EXPECT_KEYS, and what it finds in it.

This approach has downsides compared to Expect-Key before version string:

- If client is required to send Expect-Key before version string, server can 
just be quiet and send nothing. The server can avoid being accurately 
fingerprinted. The server can't do this in the method that uses KEXINIT.

- The method that requires EXPECT_KEYS after KEXINIT complicates guessed key 
exchange, since the client does not know upfront if the server supports it.

Generally, if the client could send fingerprints without the server 
signaling anything, this would be superior. It would allow the server to not 
let itself be fingerprinted as accurately. (The OS can still be 
fingerprinted based on properties of its TCP stack, even if the SSH server 
application sends nothing.)

denis

----- Original Message -----
From: Mouse
Sent: Thursday, March 23, 2017 06:24
To: [email-masked]
Subject: Re: Fixing exchange of host keys in the SSH key exchange

True.

Accept-Expect-Key: yes

/~\ The ASCII   Mouse
\ / Ribbon Campaign

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Sat, 25 Mar 2017 13:09:55 -0600

Peter:

> > this prevents the situation where the client will present
> > the user with a host key verification dialog, and the user
> > will just click through it.
>
> It also makes things a lot more painful for the user.

It helps ensure secure host key configuration. There are administrators who 
would prefer this.

If this involves more pain than previously, the user was most likely not 
managing host key trust to the same standards.

> If you regularly need to SSH into a dozen different devices
> to administer them, it gets even worse.

This is meant for administrators who would want to enable this policy. This 
would be a server-side policy, so if you're administering machines, you 
could avoid configuring this policy.

> > drive-by password guessing becomes largely impossible,
> > since unknown attackers do not know the server’s host key.
>
> This is really just an awkward form of port-knocking.

Yes, and we have other mechanisms for this (e.g. protocol obfuscation). In 
this case, it's a nice effect you get for free, in case you enable this 
policy.

> > Server implementations that do not know about this extension,
> > but correctly implement the spec,
>
> How do we know what percentage of servers do this?

We don't, it would be necessary to test this.

Which means, probably, I would have to test this.

Mouse pointed out I was wrong about the spec. It turns out RFC 4253 provides 
for text before version string when sent by servers, not by clients. It's 
definitely necessary to test.

As discussed with Mouse, if this exact method doesn't work, there are 
alternatives. Server could send "expect-key" host key alg in KEXINIT, and 
client could send a new message SSH_MSG_EXPECT_KEYS after its KEXINIT. This 
would still solve the primary objective, but would not have all the same 
nice properties. E.g. the server could not use that to avoid fingerprinting.

> it has the potential to break lots of things

That seems to be the main potential downside of the method as presented.

But there are other ways to do it, and the suggested approach could be 
proven safe in testing.

denis

----- Original Message -----
From: Peter Gutmann
Sent: Friday, March 24, 2017 01:22
To: denis bider (Bitvise) ; [email-masked]
Cc: [email-masked] ; Simon Tatham
Subject: Re: Fixing exchange of host keys in the SSH key exchange

denis bider (Bitvise) <[email-masked]> writes:

>key

It also makes things a lot more painful for the user.  If you regularly need
to SSH into a dozen different devices to administer them, it gets even 
worse.

This is really just an awkward form of port-knocking.  If you want to use a
port-knocking-style mechanism, you should probably do it properly.  If you
want to bind a client to a server ("only this client, only this server") 
then
there are better ways of doing it than this.

How do we know what percentage of servers do this?  It's true that the spec
says you should expect other stuff in the textual data, but since pretty 
much
everything out there only ever sends the SSH ID string, it's quite likely 
that
lots of implementations will choke on unexpected extra lines of text.  Even 
in
cases where the code does try and handle non-SSH-ID lines, the absence of
anything that sends them means it's probably never been tested.

Peter.

---

From: Mouse <[email-masked]>
Date: Sat, 25 Mar 2017 08:27:37 -0400 (EDT)

>> (1) Security benefit: although [...]; under reasonable assumptions,
>> if the client has to communicate one or more fingerprints of the
>> server?s host key(s) before KEXINIT, this prevents the situation
>> where the client will present the user with a host key verification
>> dialog, and the user will just click through it.

How?  As far as I can see, this affects the user only in that "pick up
the host key on first connect" no longer works.  In particular...

> If you regularly need to SSH into a dozen different devices to
> administer them, it gets even worse.

...I can't see how it affects the user if the client already has a
correct host key for the server.

>> (2) Security benefit: [...]

Awkward?  _Less_ awkward in some respects - significantly easier to set
up in most cases, for example, with no need for anything client-side
beyond a client that knows the extension.  But weaker in other
respects; for example, it reveals the presence of an ssh server to
anyone, even someone who doesn't know the knock (well, probable
presence; nothing says someone can't use port 22 for something else).

>> Server implementations that do not know about this extension, but
>> correctly implement the spec, [...]

We don't, of course.  But note that the spec does not say that the
_client_ may send text before its version string, only that the
_server_ may.  (Or, at least, if it says that about the client, I
haven't found where.)

Furthermore, just because some implementations are broken (if the
consensus is that that constitutes brokenness, of course) is not a
reason to fail to call them on their brokenness.  I've run into ssh
servers that fall over hard when confronted with clients using the
string@domainname extension mechanism.  Should we stop using it as a
result?  No, we should get the relevant servers fixed (and, as an
interim measure for the immediate term, tell clients to not use
extensions when talking with such broken servers).

> It's true that the spec says you should expect other stuff in the
> textual data, but since pretty much everything out there only ever
> sends the SSH ID string, it's quite likely that lots of
> implementations will choke on unexpected extra lines of text.

Well, if the consensus is that the client is allowed to send text
there (if not, this becomes a private protocol change, not suitable for
interoperation use), then my stance is that that's their problem.  My
advice for users dealing with such servers would be "either get the
server fixed or turn off the extension when using it".

> Even in cases where the code does try and handle non-SSH-ID lines,
> the absence of anything that sends them means it's probably never
> been tested.

Then this suggestion has the additional feature that it will smoke out
such bugs!

Hmm, I think I'll give moussh a configuration option to send things
before the ID string, for exactly that reason.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Mouse <[email-masked]>
Date: Sat, 25 Mar 2017 22:43:35 -0400 (EDT)

>> SSH_MSG_KEXINIT that is "0 (reserved for future extension)",
> Unfortunately, there exist implementations that disconnect, or
> generate an invalid key exchange hash, if that value is not zero.

I'm not sure what my stance is on that.

In general, I take the stand that if there are implementations that
don't implement the spec correctly, that's their problem, not a reason
to avoid using that part of the spec.  But, here, I don't think there's
any current spec for what an implementation should do if it finds a
nonzero value there, so I'm not sure to what extent _any_ behaviour in
response to that constitutes misbehaviour.  In retrospect, I think
specifying that RFU zero without giving any explicit guidance for
behaviour upon finding it nonzero was a mistake.

>> Also, 4253 says the server MAY send text before its version number,
>> but is silent on the question of whether the client also may.
> Of course, even if it did say otherwise, there could exist server
> implementations that do not handle this properly.

Of course - but then it would clearly be a case of the servers in
question being broken (and my stance would be that it's the server's
problem - see above).  As it is, it's not clear.

> It seems that testing would be in order.

Probably.  But it seems to me that any implementation SHOULD have
client configuration knobs so that Expect-Key: is not sent unless the
server is expected to be prepared to handle it.

> Or maybe the server sends something like "Expect-Key:
> support|require".  The format can vary between client and server.
> That might still not work if there are clients that violate 4253,
> though.

That's the problem of any such clients.  IMO.  (As I think I said
upthread, I've encountered servers that misbehave - crash or something
equivalent from the client's point of view - when faced with the
string@domainname extension feature.  Is that a reason to not use it?)

> - The server includes "expect-key" among its host key algorithms.
> [...]

> - If client sees "expect-key" among server's host key algorithms,
> [...]

Yesss...though there may exist implementations that misbehave upon
seeing a non-extension algorithm name they do not recognize.  (My
stance, as above, would be that such implementations are broken and
deserve nothing but ridicule, but your remarks above seem to imply you
don't take that stance, or at least not as much as I do.)

> This approach has downsides compared to Expect-Key before version
> string: [...]

Both true, though the second one (that it interferes with guessed kex)
is weak in that all it means is that clients and servers using this
extension can't really do guessed kex.  But all guessed kex really buys
you is one network roundtrip fewer, and I submit that if that matters
then either you have a _really_ laggy network or your PK crypto is so
weak that your security is questionable anyway. :-)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Peter Gutmann <[email-masked]>
Date: Sun, 26 Mar 2017 08:53:18 +0000

Mouse <[email-masked]> writes:

>As far as I can see, this affects the user only in that "pick up the host key
>on first connect" no longer works.

It breaks TOFU.  Since this is how the vast majority of all users use SSH
(ref: https://www.usenix.org/system/files/login/articles/105484-Gutmann.pdf),
it means it would break SSH for them.  Conversely, it means the vast majority
won't use it.

>Then this suggestion has the additional feature that it will smoke out such
>bugs!

Trying to smoke out non-standards-compliant implementations at this point,
about twenty-odd after SSH2 started getting deployed, is probably a bit late
in the game.

Also, does this mean any implemenation that doesn't correctly implement a MUST
or MUST NOT can regarded as broken and discarded?

Peter.

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Sun, 26 Mar 2017 19:20:47 -0600

Peter:

> https://www.usenix.org/system/files/login/articles/
> 105484-Gutmann.pdf

This article is actually a strong argument in favor of Expect-Key, or 
something like it.

What you found is NOT that people use TOFU (Trust On First Use). They use 
TWAT - Trust Whatever, Any Time:

> I asked the IT support staff how many users had called
> or emailed to verify the SSH server key whenever it had
> changed in the past several years. They were unable to
> recall a single case, or locate any records, of any user
> ever verifying any SSH server key out-of-band

This is highly problematic because it means that, while SSH is supposed to 
defend against MITM in theory, it does not defend in practice.

OpenSSH already addresses this in their client, I believe, by making it 
really quite hard to verify a changed key. Many users further configure the 
OpenSSH client to require a key to be set up for the first connection, to 
begin with.

Expect-Key just extends that - which I think is a good practice - to all 
clients, in general, enforceable as a server-side policy. It defeats TWAT.

> Conversely, it means the vast majority won't use it.

Users will use what administrators set up for them. If admins don't want to 
set up Expect-Key, fine. But this gives them a tool.

denis

----- Original Message -----
From: Peter Gutmann
Sent: Sunday, March 26, 2017 02:53
To: Mouse ; [email-masked]
Subject: Re: Fixing exchange of host keys in the SSH key exchange

It breaks TOFU.  Since this is how the vast majority of all users use SSH
(ref: 
https://www.usenix.org/system/files/login/articles/105484-Gutmann.pdf),
it means it would break SSH for them.  Conversely, it means the vast 
majority
won't use it.

Also, does this mean any implemenation that doesn't correctly implement a 
MUST
or MUST NOT can regarded as broken and discarded?

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Mon, 27 Mar 2017 01:24:13 +0000

>TWAT - Trust Whatever, Any Time:

Damn, I could've used that in the article :-).

>defend against MITM in theory, it does not defend in practice.

Yeah, as the article points out, X.509 fans can point out that SSH is no
better, and SSH fans can point out that SSH is no worse.

Peter.

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Sun, 26 Mar 2017 19:08:31 -0600

Mouse:

> I've encountered servers that misbehave – crash or something
> equivalent from the client's point of view - when faced with
> the string@domainname extension feature.

I am flummoxed by the existence of such implementations. Surely, that would 
not even be compatible with OpenSSH?

Have you seen any of these recently? Was that perhaps a decade ago, when 
OpenSSH maybe didn't have a KEXINIT full of string@domain algorithms by 
default?

> My stance, as above, would be that such implementations are
> broken and deserve nothing but ridicule, but your remarks
> above seem to imply you don't take that stance, or at least
> not as much as I do.

I do take the stance that a broken implementation is broken, but this does 
not change that I'm usually expected to do something about it, and folks 
want me to work around it so it works.

For the most recent example, an older version of a popular library used to 
have the "maximum channel packet size" concept completely borked up. For a 
channel opened by the remote party, this library would overwrite its own 
maximum packet size with the remote one. This caused at least two different 
kinds of session-ending problems to arise.

The library fixed this problem years ago. But a customer wrote an 
application using the old version - it worked until then, why fix it?! - and 
deployed that on N installations. These installations now wouldn't work with 
our software, but are difficult to update. It was less costly if we 
implement a coping mechanism. So we did.

I'd like to think our implementation is close to defect-free, but chances 
are someone somewhere was tasked to implement a workaround for a bug that 
existed in our software, even if it had long since been fixed.

We judge our own software based on the next version we're about to release; 
and other people's software based on all versions that existed. :)

> Hmm, I think I'll give moussh a configuration option to
> send things before the ID string, for exactly that reason.

That would be amazing. I could implement it as an option, but unless it is 
on by default, there's no way we'll get feedback.

At this point, the most feasible option I see is the type of probing you 
said you don't like: blind connections to random IP addresses to produce 
statistics how often it works or not.

denis

----- Original Message -----
From: Mouse
Sent: Saturday, March 25, 2017 20:43
To: [email-masked]
Subject: Re: Fixing exchange of host keys in the SSH key exchange

/~\ The ASCII   Mouse
\ / Ribbon Campaign

---

From: Mouse <[email-masked]>
Date: Sun, 26 Mar 2017 08:10:53 -0400 (EDT)

>> As far as I can see, this affects the user only in that "pick up the
>> host key on first connect" no longer works.
> It breaks TOFU.

Yes.  As denis recently said, that's the major point of it: some -
probably a minority, but so what? - server admins _want_ to break TOFU,
to ensure that their users are doing more secure key handling.

Another possibility is that users _do_ use TOFU, but are allowed to
only over secure-enough networks.  This allows configurations like
"You're coming from the office LAN, you can TOFU - but when you come in
over the open Internet, you have to already have the host key", by
providing a mechanism to enforce the second half of it.

> Since this is how the vast majority of all users use SSH [...], it
> means it would break SSH for them.  Conversely, it means the vast
> majority won't use it.

Which is fine.  Having it available does not mean it should be the
default.

>> Then this suggestion has the additional feature that it will smoke
>> out such bugs!
> Trying to smoke out non-standards-compliant implementations at this
> point, about twenty-odd after SSH2 started getting deployed, is
> probably a bit late in the game.

Probably.  But better late than never.  Though, as remarked upthread,
it's not clear that breaking on pre-ID text in the client-to-server
direction _is_ a bug.

> Also, does this mean any implemenation that doesn't correctly
> implement a MUST or MUST NOT can regarded as broken and discarded?

Ideally.  But that's true of any spec.

Those ssh implementations I mentioned upthread that crashed upon seeing
string@domain extensions?  We didn't discard them; we worked around
them.  I would have preferred to replace them, but that was not
practical at the time (this years ago was at work and they were
embedded servers on, IIRC, network switches - I hope we filed bug
reports with the vendor(s) in question, but I'm not sure of even that
much).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Mouse <[email-masked]>
Date: Sun, 26 Mar 2017 23:27:29 -0400 (EDT)

>> I've encountered servers that misbehave â?? crash or something
>> string@domainname extension feature.
> I am flummoxed by the existence of such implementations.

So was I!

> Surely, that would not even be compatible with OpenSSH?

I don't know.  I suspect I tried OpenSSH against them, since I can't
imagine why I would have suspected it were DNS-based extensions causing
trouble without "works" and "fails" traces to compare, and OpenSSH is
the most plausible other implementation for me to have had at hand.
But it would have been OpenSSH of that era.  This was relatively long
ago; in particular, it was before 2008-07-03 - that's the oldest moussh
source tree I have at ready hand, and it's got -no-private (see below).

> Have you seen any of these recently?

No, but I haven't had much opportunity to, either.  (They were the SSH
listeners on devices such as switches.)

> Was that perhaps a decade ago, when OpenSSH maybe didn't have a
> KEXINIT full of string@domain algorithms by default?

That timing is about right.  I don't, and didn't, track what OpenSSH
does (resp. is doing), though.

> I do take the stance that a broken implementation is broken, but this
> does not change that I'm usually expected to do something about it,
> and folks want me to work around it so it works.

I sympathize.  Those switches (or whatever they were) are the reason
moussh has -no-private:

     -no-private
             Disables attempts to use private requests and algorithms (any-
             thing using the DNS extensibility mechanism, basically).  This
             should never be needed, but at least one commercial server is
             known to break when faced with such names.  [...]

> I'd like to think our implementation is close to defect-free, but
> chances are someone somewhere was tasked to implement a workaround
> for a bug that existed in our software, even if it had long since
> been fixed.

Probably.  Same goes for moussh, though it's more likely nobody has
cared enough since moussh surely has a much smaller user base.

> We judge our own software based on the next version we're about to
> release; and other people's software based on all versions that
> existed. :)

That is a common problem.  Stance.  Whatever you want to call it. :-)

> That would be amazing.  I could implement it as an option, but unless
> it is on by default, there's no way we'll get feedback.

I'll try to remember to mention it here when I finally get around to
adding it.  So far I've added it to nothing but my TODO list.

> At this point, the most feasible option I see is the type of probing
> you said you don't like: blind connections to random IP addresses to
> produce statistics how often it works or not.

Quite aside from whether it's moral to use others' servers for your own
purposes like that, that isn't going to tell you more than whether they
are willing to talk to you.  If they drop the connection, you will not
be in a position to tell whether it's because they crashed upon seeing
your pre-version text, or they're configured (as mine are) to refuse to
talk to third parties who don't exhibit whatever behaviour they're
configured to look for, or they crashed for an unrelated reason, or
what.  Also, would "fraction of random IP addresses with ssh servers
that don't crash upon seeing $THING" really be useful statistic, even
if were one you could get?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Mouse <[email-masked]>
Date: Sun, 26 Mar 2017 23:30:05 -0400 (EDT)

> This is highly problematic because it means that, while SSH is
> supposed to defend against MITM in theory, it does not defend in
> practice.

Not in the hands of "the masses", no.  But very little will.

It _does_ give the tools to those competent to use them; I think that's
about all any such protocol can really expect to do.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Peter Gutmann <[email-masked]>
Date: Mon, 27 Mar 2017 05:47:28 +0000

Mouse <[email-masked]> writes:

>(They were the SSH listeners on devices such as switches.)

It seems to be an unwritten law that big-iron networking hardware has to run
15-to-20-year-old unpatched copies of some version of SSH that the
switch/router vendor found in a dumpster somewhere.  I've seen late-1990s
ssh.com implementations as recently as a few years ago, which is why I have to
keep active old SSH bug-workarounds that should have been retired a decade or
more ago.  Being able to remotely crash a carrier-grade router [0] by sending
it an SSH option it didn't expect is a bit disturbing... this is why my code
grew a dump-full-packet-trace-to-debug-console option at some point in the
past.

Peter.

[0] I guess the SSH part is excluded from the carrier-grade categorisation.

---

From: Peter Gutmann <[email-masked]>
Date: Mon, 27 Mar 2017 05:55:04 +0000

Mouse <[email-masked]> writes:

>It _does_ give the tools to those competent to use them; I think that's about
>all any such protocol can really expect to do.

It also warns about key changes so you can take action if necessary, which is
something that SSL doesn't.  Bank of America web site now hosted in the
Ukraine on a Windows 7 Home Premium box?  Let's see, it has a $5.99 GoDaddy
certificate.  Seems legit [0].

(I've only ever encountered one SSL-using app that warns that the key/cert
you're getting now differs from the one you got last time.  I'm sure there are
more out there, but none of the mainstream stuff does it).

Peter.

[O] I know, I like to bash PKI, but with farcical behaviour like this it's
    hard not to.

---

From: Peter Gutmann <[email-masked]>
Date: Mon, 27 Mar 2017 06:21:51 +0000

>kinds of session-ending problems to arise.

It seems like every implementer has stories like this, but no-one can really
mention them in public because you don't want to embarrass a particular
vendor... would there be any interest in having a private list of email
addresses of people to exchange information like this with?  That way we could
compare notes on necessary fixes that otherwise would need to be rediscovered
for each new implementation.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Mon, 27 Mar 2017 23:57:34 +0000

denis bider (Bitvise) <[email-masked]> writes:

>The obstacle seems to be getting people together. Those of us who’ve been
>around for 15 years may be on this mailing list. I’m not sure if this is true
>for authors of newer implementations, who might benefit from this information
>most.

We can't solve every possible problem with incompatibilities, but we can at
least get good coverage of a lot of them.  I think there are quite a few
oddball SSH implementations whose developers have never been on this list and
who we can't get to, but it can at least help those on the list.

Just thinking about this a bit more, we'd maybe need two things, a means of
discussing quirks of other implementations, and an (informal) registry of SSH
ID strings and who to contact if you find a problem with that ID, because
that's been another problem, "I've found a bug with X, who do I report it
to?".  Going through standard tech-support channels often doesn't work because
you're not a customer and there's no obvious way to get past the front-end
people to talk to a developer.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 28 Mar 2017 01:16:04 +0000

denis bider (Bitvise) <[email-masked]> writes:

>This is currently the most comprehensive list of SSH implementations I’m
>familiar with:
> 
>http://ssh-comparison.quendi.de/
> 
>It’s done by Max Horn, who participated on this list as recently as in
>September, so he might be reading.

Wow, that's really impressive, because it'd involve either reading source code
or downloading and installing each one to scan it and see what it does.

Peter.

---

From: Mouse <[email-masked]>
Date: Mon, 27 Mar 2017 07:58:45 -0400 (EDT)

>> [SSH] _does_ give the tools to those competent to use them; I think
>> that's about all any such protocol can really expect to do.
> It also warns about key changes so you can take action if necessary,
> which is something that SSL doesn't.

Most implementations of SSH do; most implementations of SSL don't.
Neither protocol requires or forbids it (a protocol is not really in a
position to either compel or prohibit such a thing - fuzzy memory says
the SSH RFCs have some SHOULDs on the matter, though).

> (I've only ever encountered one SSL-using app that warns that the
> key/cert you're getting now differs from the one you got last time.
> I'm sure there are more out there, but none of the mainstream stuff
> does it).

Of course not.  That might confuse users!

> [O] I know, I like to bash PKI, but with farcical behaviour like this
> it's hard not to.

It's part of why I don't like SSL: it isn't really providing the
security it pretends to be providing.  The human-layer mechanisms don't
help; far too many people - users and service providers both - think of
security as a boolean: "It's Secure!".  Never mind that the correct
reaction to that is "...secure against what?".

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Mon, 27 Mar 2017 15:03:05 -0600

This medium lacks appropriate smileys to react to this text. :D

From: Peter Gutmann 
Sent: Sunday, March 26, 2017 23:47
To: Mouse ; [email-masked] 
Subject: Re: Fixing exchange of host keys in the SSH key exchange

Peter.

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Mon, 27 Mar 2017 15:18:10 -0600

That sounds like a good idea. I would be interested to follow and participate.

The obstacle seems to be getting people together. Those of us who’ve been around for 15 years may be on this mailing list. I’m not sure if this is true for authors of newer implementations, who might benefit from this information most.

From: Peter Gutmann 
Sent: Monday, March 27, 2017 00:21
To: denis bider (Bitvise) ; Mouse ; [email-masked] 
Subject: Re: Fixing exchange of host keys in the SSH key exchange

Peter.

---

From: Mouse <[email-masked]>
Date: Mon, 27 Mar 2017 18:04:59 -0400 (EDT)

>>> For the most recent example, an older version of a popular library
>>> used to have the "maximum channel packet size" concept completely
>>> borked up.  For a channel opened by the remote party, this library
>>> would overwrite its own maximum packet size with the remote one.
>> It seems like every implementer has stories like this, but no-one
>> can really mention them in public because you don't want to
>> embarrass a particular vendor...

Well, in many cases.  I, for example, am not at all chary about naming
OpenSSH as the implementation whose misfeature prompted me to add
-share-number to moussh (even the moussh manpage does so), and I'd name
the vendor of the devices whose implementation crashed when given
string@domianname extensions if I remembered it (and were sure I
rememebred it right).  But I can certainly understand at least a few of
the reasons not everyone is as willing as I am to do that.

However...

>> would there be any interest in having a private list of email
>> addresses of people to exchange information like this with?

...I don't see any need to name-and-shame on such a list.  It's the
misbehaviour, not whose implementation exhbits it, that matters for
implementation purposes.

There might perhaps be value in an implementation with options to make
it exhibit various kinds of misbehaviour, for testing against.  (I'm
tempted to turn moussh into such a one, but now is not a good time for
me to be taking on more timesinks.)

> That sounds like a good idea. I would be interested to follow and

Me too.

> The obstacle seems to be getting people together.  Those of us
> whoâ??ve been around for 15 years may be on this mailing list.  Iâ??m
> not sure if this is true for authors of newer implementations, who
> might benefit from this information most.

I'm not sure either.  Perhaps if the list were archived and the
archives were up somewhere on the Web, the new crowd who expects
everything to be Web this and Web that might be able to find it?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Mouse <[email-masked]>
Date: Mon, 27 Mar 2017 21:30:30 -0400 (EDT)

> Just thinking about this a bit more, we'd maybe need two things, a
> means of discussing quirks of other implementations, and an
> (informal) registry of SSH ID strings and who to contact if you find
> a problem with that ID, because that's been another problem, "I've
> found a bug with X, who do I report it to?".  Going through standard
> tech-support channels often doesn't work because you're not a
> customer and there's no obvious way to get past the front-end people
> to talk to a developer.

(Well, to be fair, _should_ they be interested in talking to a
non-customer?  Thirty years ago, most admins would have cared about the
problem for its own sake.  That, much as it pains me, is not today's
net.)

Yes, I would support - and participate in, provided it isn't done in a
way that ends up excluding me - such an effort.  There are a bunch of
other things that could be recorded about the various implementations,
such as what they support (not just crypto, as in the quendi.de list,
but other things as well).

It also might be interesting to do interop testing.  I know *I* would
be interested in interop tests - in both directions - with the various
other implementations.  There are a few things I know I don't support
and a bunch I'm fairly sure I do; I'd like to verify the support I
think I have in place, make sure I fail gracefully on the things I know
I don't do, and find out the status of anything in neither category.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Mon, 27 Mar 2017 18:22:28 -0600

This is currently the most comprehensive list of SSH implementations I’m familiar with:

It’s done by Max Horn, who participated on this list as recently as in September, so he might be reading.

Come to think of it – Max’s SSH comparison might be an excellent way to figure out how many servers will reject text from the client before the SSH version string. I understand that this might be quite a bit of work. However, Bitvise would be willing to sponsor such an effort. 

Max? Are you there? :-)

From: Peter Gutmann 
Sent: Monday, March 27, 2017 17:57
To: denis bider (Bitvise) ; Mouse ; [email-masked] 
Subject: Re: Fixing exchange of host keys in the SSH key exchange

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Thu, 30 Mar 2017 03:38:14 +0000

>misbehaviour, not whose implementation exhbits it, that matters for

It's not so much concerns about name-and-shame, it's that it's
impossible not to name vendors when you need to know whose SSH ID
to check for to add a workaround.  So it's pretty much something that 
can't be discussed in public for most people.

>Yes, I would support - and participate in, provided it isn't done in 
>a way that ends up excluding me - such an effort. 

I wasn't necessarily thinking a full email list, that's way too
organised, just a list of CC: addresses of people who'd be willing to
share info.  So far we've got you, me, and Denis AFAIK.

>It also might be interesting to do interop testing. 

Or just some agreement to run an instance of your implementation at
some fixed location so people could bounce messages off it.

Peter.

---

From: Darren Tucker <[email-masked]>
Date: Tue, 28 Mar 2017 16:56:39 +1100

On Tue, Mar 28, 2017 at 9:04 AM, Mouse <[email-masked]> wrote:

> -share-number to moussh (even the moussh manpage does so)

I was curious about what that was so I looked.  Quoting moussh(1):

     There is a misfeature (I would call it a bug, except that reading the
     source makes it clear it was done deliberately) in OpenSSH's server.
     (Similar issues may exist with others, but I have no knowledge of
them.)
     It gratuitously refuses to permit more than ten sessions per
connection.
     This means that using moussh's connection-sharing feature to connect to
     such a server will work fine until you try to open too many remote
login
     sessions, at which point you will get refusals from the remote server.
     Worst of all, OpenSSH does not provide any way for the server admin to
     raise this limit; it is hardwired into the code!

That last sentence is not accurate, OpenSSH has provided a MaxSessions
config option since the 5.1 (2008):
https://www.openssh.com/releasenotes.html#5.1

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

---

From: Mouse <[email-masked]>
Date: Thu, 30 Mar 2017 07:36:29 -0400 (EDT)

> impossible not to name vendors when you need to know whose SSH ID to
> check for to add a workaround.

Oh, hmm, true.  I wasn't thinking it through enough.

> organised,

:-)

> some fixed location so people could bounce messages off it.

That's pretty much what I was thinking - interop testing can also be
done informally. :-)  There are a lot of details, but I would expect
they can be worked out in most cases.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: "denis bider \(Bitvise\)" <[email-masked]>
Date: Thu, 30 Mar 2017 12:57:59 -0600

Also, unfortunately, the OpenSSH “MaxSessions” setting is often configured with the value 1.

This is really awful for a graphical client such as Bitvise SSH Client, which hopes to be able to open at least 2 channels in the same session (one for SFTP, one for terminal).

I’m not sure who sets that default to 1 – maybe it’s a distribution, or maybe some zealous administrator. However, I would suggest a sensible minimum might be at least 2 for servers that allow terminal shell and SFTP access.

denis

From: Darren Tucker 
Sent: Monday, March 27, 2017 23:56
To: Mouse 
Cc: [email-masked] 
Subject: Re: Implementation-hazards list [was Re: Fixing exchange of host keys in the SSH key exchange]

  [...]

     There is a misfeature (I would call it a bug, except that reading the
     source makes it clear it was done deliberately) in OpenSSH's server.
     (Similar issues may exist with others, but I have no knowledge of them.)
     It gratuitously refuses to permit more than ten sessions per connection.
     This means that using moussh's connection-sharing feature to connect to
     such a server will work fine until you try to open too many remote login
     sessions, at which point you will get refusals from the remote server.
     Worst of all, OpenSSH does not provide any way for the server admin to
     raise this limit; it is hardwired into the code!

That last sentence is not accurate, OpenSSH has provided a MaxSessions config option since the 5.1 (2008): https://www.openssh.com/releasenotes.html#5.1

--

---

From: Mouse <[email-masked]>
Date: Thu, 30 Mar 2017 15:54:52 -0400 (EDT)

> That last sentence is not accurate, OpenSSH has provided a
> MaxSessions config option since [] 5.1 (2008) [...]

I've updated moussh.1:

commit 58da35d27000668ec6c9125790c47cbfed430669
Author: Mouse <[email-masked]>
Date:   Tue Mar 28 08:21:05 2017 -0400

    Update the manpage description of the OpenSSH connection-limit misfeature.

diff --git a/moussh/moussh.1 b/moussh/moussh.1
index 2bfc901..e235526 100644
--- a/moussh/moussh.1
+++ b/moussh/moussh.1
@@ -747,7 +747,8 @@ to open
 .Ar N
 parallel connections to the server.  The sharing clients are then
 load-shared among the server connections.  This option is designed to
-work around a misfeature in OpenSSH; see the
+work around a misfeature in some OpenSSH versions, still useful with
+current versions in some circumstances; see the
 .Sx CONNECTION SHARING
 section for more.
 .It Fl just-die
@@ -1142,18 +1143,21 @@ option variables false.)
 There is a misfeature (I would call it a bug, except that reading the
 source makes it clear it was done deliberately) in OpenSSH's server.
 (Similar issues may exist with others, but I have no knowledge of
-them.)  It gratuitously refuses to permit more than ten sessions per
-connection.  This means that using
+them.)  It gratuitously refuses to permit more than some number of
+sessions per connection.  This number defaults to 10, and some versions
+(before 5.1, I believe) have it hardwired to 10 in the code.  (As far
+as I know no version has a way to remove the limit entirely, though for
+most purposes specifying a large number like 1000 is equivalent.)  This
+means that using
 .Nm moussh Ap s
 connection-sharing feature to connect to such a server will work fine
 until you try to open too many remote login sessions, at which point
-you will get refusals from the remote server.  Worst of all, OpenSSH
-does not provide any way for the server admin to raise this limit; it
-is hardwired into the code!
+you will get refusals from the remote server.
 .Pp
 .Nm
-contains a workaround for this: when establishing a connection-sharing
-server, you can specify
+contains a workaround for this, in case the server is a version old
+enough to have it hardwired or its admin doesn't want to raise the
+limit: when establishing a connection-sharing server, you can specify
 .Fl share-number
 (or the corresponding config-file variable) to make
 .Nm

/~\ The ASCII				  Mouse
\ / Ribbon Campaign