Subject: [Netconf] guidance on draft-kwatsen-reverse-ssh

From: Kent Watsen <[email-masked]>
Date: Tue, 12 Jul 2011 16:24:18 -0700

I'm writing this email for guidance on how to move this draft forward.

Per recommendation of NETCONF chairs and both the Ops and Security ADs, I have been reviewing draft-kwatsen-reverse-ssh with the SAAG and IETF-SSH lists, who currently review SSH-related RFCs.  After about a month of discussion, there seems to be four possible solutions (below), each with pros and cons.  Unfortunately, the SAAG and IETF-SSH lists seem to prefer solutions that I feel are excessively onerous to implement, which puts us into a virtual deadlock.

Before diving into the four solutions, let me first preface that the desire to support Reverse SSH is primarily for devices to be able to "call home" to an NMS that would then manage the device via the NetConf protocol though, of course, other network management protocols could be tunneled over the SSH connection as well.  This is "important" in that my company has been doing this for a number of years now and also because we've received requests from other WG members to submit a draft for it.  

And now, without further ado, here are the four possible solutions:

  1. out-of-band role-reversal

     Device initiates a TCP connection to an IANA-assigned port, sends
     a special SSH-role-reversal message, and then starts the SSH Server
     protocol (i.e. fork/exec `sshd -i`).  From the NMS perspective, upon
     accepting a connection on the IANA-assigned port, it receives the
     special SSH-role-reversal message and uses it to fetch credentials
     from its DB, and then starts the SSH Client protocol (SSH client 
     libraries available in many programming languages).  

     This solution is fully described by the draft-kwatsen-reverse-ssh-00 
     submission.  A variant of this solution has been in use at Juniper
     for ~5 years.  It does not require any modification to SSH 
     client/server implementations.  If we choose to promote this 
     solution, I recommend modifying the draft slightly to allow the MAC
     and host-key algorithms to be negotiated *exactly* like the SSH 
     protocol (essentially, merging in the best part of solution #2)

     I feel this is the best solution because it's simple to implement
     and interoperable with all SSH implementation we've tried it with.
     Also, this solution benefits in that, in all cases, the device is
     always running the SSH server and the NMS is always running the
     SSH client.  This keeps it simple in that how peers identify and
     authenticate each other is consistent regardless which peer 
     initiates the underlying TCP connection.

     However, the SAAG and IETF-SSH lists did not like this solution
     because they felt that the role-reversal should happen in-band to 
     the SSH protocol and also that there was no reason for another
     IANA-assigned port.  As maintainers of the SSH protocol, I can 
     understand why they feel that way, but taking a step back, what's
     wrong with the role-reversal happening a step before the 
     SSH-protocol starts?

  2. out-of-band reversal, but no special ssh-role-reversal message

     This solution is a lot like the first, but the special role-reversal
     message is replaced with heavy restrictions on which host-key
     algorithms are allowed to just those that relay identity as well
     as authentication (i.e. the x.509 algs from RFC6187 and a new
     family of algs based on HMAC I defined for the purpose).

     This solution is fully described by the draft-kwatsen-reverse-ssh-01 
     submission.  It requires modification to SSH client/server
     implementations - so that the server (the device) only advertizes
     the allowed algorithms and so that they're both aware of the new
     "hmac-*" family of algorithms.

     This solution has some pros and cons to the first solution.  On
     the positive side, leveraging the existing MAC and host-key 
     negotiation logic built into the SSH protocol is pretty slick.
     On the negative side, needing to modify the SSH client/server
     code support the x.509 and HMAC based keys would take some time
     and further, restricting which hosts keys can be advertized 
     to just these is somewhat uncomfortable (i.e. no ssh-rsa or 
     ssh-dsa allowed), unlike with solution #1.

 3. in-band solution, by allowing the SSH server to open channels

     In this solution, the device SSHs to the NMS (i.e. device is SSH
     client and the NMS is the SSH server).  The original thinking
     was that the device would connect to port 22, where the NMS would
     be running a generic SSH server (i.e. `sshd`), but further analysis
     suggests that it would be too difficult for the NMS to bind it's
     business-logic via a subsystem and instead should be a custom
     server built by linking to an SSH Server library and listening
     to some port other than 22.

     This solution would require non-trivial modification of SSH client
     and server implementations - to enable a server to open channels
     on the client.  As one SAAG member wrote "Frankly, you will probably
     have a hard time getting upstream OpenSSH to accept patches to 
     implement this or any other approach.  They're _very_ conservative
     about changes."

     This solutions has pros and cons as well.  On the positive side, it's
     the simplest solution at the RFC level - we only need to remove the
     SHOULD in RFC4254 section 6.1.  On the negative side, this solution
     may be difficult to implement, would require updating the SSH code
     on devices, and introduces new challenges in how the peers identify
     and authenticate each other.  Another big negative is that it's
     unlikely to be supported by OpenSSH because, let's face it, letting
     a server open channels to a client doesn't sound very secure  ;)

  4. in-band solution, based on remote port-forwarding

     This solution again has the device SSH-ing to the NMS, but this 
     time it sets up a remotely forwarded port that the NMS can then
     use to SSH back to the device on.

     This solution requires no modification to existing SSH software.
     It might be OK to use generic `sshd` listening on port 22, since
     the device only needs to send a signal to the NMS to connect back
     it on the forwarded-port it opened.

     The positives of this solution are its simplicity (not even an
     RFC is required).  The negatives are that how peers identify and
     authenticate each other would have to be solved for both directions.
     Another negative is that there could be collisions with the 
     remote-ports opened by the devices.  And yet another negative 
     is that there would be double encryption, wasting precious CPU
     cycles on both sides.

How to move forward?

   - first, it would be interesting to get some feedback on the various
     proposals from the NETCONF and OPSAWG WG members.   I realize that
     above are very high-level descriptions, but hopefully it's enough
     to get the gist of what's being proposed...

   - if it turns out that there is significant support for solution #1
     (or even #2),  then we might be able to take that back to the SAAG
     and IETF-SSH lists for their reconsideration.  Alternatively, perhaps
     either the OPSAWG or the NETCONF WG would be interested in picking
     up this I-D?  As a last-ditch effort, would it make sense to submit 
     it as an EXPERIMENTAL RFC? - would others who asked for this draft
     to be submitted even implement an EXPERIMENTAL RFC?

Thanks,
Kent

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 13 Jul 2011 06:47:11 +0200

> And now, without further ado, here are the four possible solutions:

[...]
 
>      to be submitted even implement an EXPERIMENTAL RFC?

There is some history of this discussion in the ISMS working group.
When ISMS did SNMP over SSH, we had a hard time dealing with
notifications and the Juniper approach was already put on the table at
that time as "running code that seems to work in practice to solve an
operational problem". As far as I recall, there was no doubt about the
operational problem and the need to solve it _but_ there were security
concerns brought forward. I would have to do several hours of reading
of ISMS archives in order to phrase them correctly. But simply put (as
far as I recall - and I don't really recall any details and so I might
be totally off), the concern had to do with something not truely
authenticated to tell a box to SSH somewhere which involves the usage
of identities with cryptographic keys. In ISMS, we ended up making the
notification originator the SSH client - and this caused quite some
costs since the amount of config increases and the identity to bind
access control to becomes different.

So in essence, I believe we have been for years at a deadlock
situation where operational people were clear that a solution for
devices to "call home" is clearly needed but the security people had
security concerns with the solutions presented and the solutions liked
more by security people to be operationally a pain. 

Perhaps one path forward is to have the operational people push a
solution that is implementable and solves an operational problem
(without creating a new operational problem) through the whole IETF
process forcing the security people to clearly document their security
concerns and then it can be seen whether that text all goes into the
Security Considerations and the protocol passes or the document stops
at the IESG. This is potentially a painful exercise.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Bert (IETF) Wijnen" <[email-masked]>
Date: Wed, 13 Jul 2011 09:42:14 +0200

On 7/13/11 1:24 AM, Kent Watsen wrote:

>       ssh-dsa allowed), unlike with solution #1.
So Kent, do I understand correctly from your email that the
SAAG and SSH people prefer solution 2?

If so, are they (I guess specifically the SSH guys) willing to work on the spec and
then implement this in SSH implementations/products/distributions?

Bert

---

From: Kent Watsen <[email-masked]>
Date: Wed, 13 Jul 2011 12:08:53 -0700

> SAAG and SSH people prefer solution 2?

Not as the -01 draft is written, which maps to solution #2, but it might be more palatable to them if we tweak the SSH handshake so that the device (the SSH client) indicates that it wants a role-reversal.  as it's written now, the server MUST listen on a port other than 22, with this change, it MAY listen on port 22.  
I've resisted this solution since it would require non-trivial modification to the various SSH client/server implementations.  The current -01 draft could be implemented with just minor tuning - to advertize support for the x.509 and HMAC based host-keys...

> If so, are they (I guess specifically the SSH guys) willing to work on the
> spec and then implement this in SSH implementations/products/distributions?

They are willing to work on the RFC, but they have no influence over implementations, specifically OpenSSH.  Case in point, they recently standardized x.509 host-keys (RFC 6187), but OpenSSH just released its own non-x.509 based solution (http://lwn.net/Articles/377703/)...

Thanks,
Kent

---

From: Kent Watsen <[email-masked]>
Date: Wed, 13 Jul 2011 12:26:13 -0700

> access control to becomes different.

As a security person myself, I'm fairly confident that the "Juniper approach" is OK.  We've also had numerous security folks look at it as well as the US government (it's FIPS certified) and Tier-1 SPs...

The only security issue raised so far by the SAAG and IETF-SSH lists was regarding the new HMAC family of host-key algorithms defined in the -01 draft.  By nature, it requires a symmetric key and they didn't see how a symmetric-key could be provided in a first-time "call-home" scenario.  Of course, the symmetric key can be provided to the device, either keyed in or read off a USB pen-drive...

> more by security people to be operationally a pain. 

Exactly.

> at the IESG. This is potentially a painful exercise.

As painful as it may be, I think this is our best-case scenario.  We could push it forward as an independent submission (EXPERIMENTAL?), but it would be much better if either the NETCONF and OPSAWG WGs could sponsor it.  Personally, I still agree that it's not NETCONF-specific and hence not for the NETCONF WG, but it makes sense for OPSAWG, if the WG agrees...

Thanks,
Kent

---

From: "t.petch" <[email-masked]>
Date: Tue, 19 Jul 2011 10:22:38 +0200

----- Original Message -----
From: "Kent Watsen" <[email-masked]>
To: "Juergen Schoenwaelder" <[email-masked]>;
<[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Wednesday, July 13, 2011 9:26 PM
> As a security person myself, I'm fairly confident that the "Juniper approach"
is OK.  We've also had numerous security folks look at it as well as the US
government (it's FIPS certified) and Tier-1 SPs...
>
> The only security issue raised so far by the SAAG and IETF-SSH lists was
regarding the new HMAC family of host-key algorithms defined in the -01 draft.
By nature, it requires a symmetric key and they didn't see how a symmetric-key
could be provided in a first-time "call-home" scenario.  Of course, the
symmetric key can be provided to the device, either keyed in or read off a USB
pen-drive...

Kent

My recollection is somewhat different to Juergen's.

Call home was requested by Eliot Lear but did not get much support in isms WG.

Sam Hartman, as AD, ruled it out of scope since the WG already had a lot to do,
the charter was to add security to SNMPv3 and not to introduce functions that
SNMPv3 did not have (like call home).  You will find this in the archives in
August 2005, particularly around August 18th.

For myself, I have always thought it obvious that the direction, client-server,
of
the TCP layer can and should be decoupled from that of the security layer, SSL,
SSH, TLS ... but whenever I have proposed that, I have not got much support:-(

Tom Petch

> As painful as it may be, I think this is our best-case scenario.  We could
push it forward as an independent submission (EXPERIMENTAL?), but it would be
much better if either the NETCONF and OPSAWG WGs could sponsor it.  Personally,
I still agree that it's not NETCONF-specific and hence not for the NETCONF WG,
but it makes sense for OPSAWG, if the WG agrees...
>
>
> Thanks,
> Kent
>
>
>
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 19 Jul 2011 12:24:54 +0200

> August 2005, particularly around August 18th.

Yes, ISMS was never chartered to work on call home. However, we had to
address the question how a notification originator sends notifications
to a notification receiver via SSH. And this is very similar and this
is the discussion I have been referring to in my previous email.

/js

---

From: Andy Bierman <[email-masked]>
Date: Tue, 19 Jul 2011 10:46:22 -0700

> is the discussion I have been referring to in my previous email.

Here is the original charter text from Eliot Lear:

    http://www.ietf.org/mail-archive/web/ietf/current/msg38521.html

Here are David Harrington's slides on why not ISMS:

    http://www.ietf.org/proceedings/64/slides/callhome-5.pdf

Here are the Callhome BoF minutes:

    http://www.ietf.org/proceedings/64/callhome.html

My recollection of the conclusion was that the problem space was not big enough,
and an SSH-specific solution was inappropriate.  A more general approach should
be investigated.

So almost 6 years have passed, and NETCONF vendors still want a call-home solution for SSH.
IMO, Ken's proposal should be published (assuming the security experts approve).
NETCONF notifications are rarely used.  The main reason given is that the client
connection maintenance is not worth the resources and coding effort.  I think
call-home for SSH could make NETCONF more deployable.

> /js

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 19 Jul 2011 10:52:57 -0700

Hi -

> From: "Juergen Schoenwaelder" <[email-masked]>
> To: "t.petch" <[email-masked]>
> Sent: Tuesday, July 19, 2011 3:24 AM
> Subject: Re: [Netconf] [OPSAWG] guidance on draft-kwatsen-reverse-ssh
...
> is the discussion I have been referring to in my previous email.
...

At the risk of an infinite loop...
At the SNMP level, wouldn't defining an appropriate notification type
for "call home" be sufficient?   It seems to me the necessary document
is one little notification type and a lot of applicability statement.

Randy

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 19 Jul 2011 11:14:23 -0700

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "t.petch" <[email-masked]>; "Kent Watsen" <[email-masked]>; <[email-masked]>; <[email-masked]>
> Sent: Tuesday, July 19, 2011 10:46 AM
...
...

It seems like there were two distinct problem spaces that folks had in mind
during the initial discussions.

One (I'll call it the "narrow" one) is the question of how a device new to the
network lets management know that it exists so that it can be (more fully)
configured. The other (I'll call it the "broad" one) includes the issues of
overcoming NAT.  The original proposed charter text seems consistent
with the narrow space.  Addressing it shouldn't be much work, but would
nonetheless be worthwhile.   The broad problem is a different matter.  David
Harrington's most persuasive arguments were addressed to it, rather than
the narrow problem.

The NETCONF connection maintenance issue Andy raises is really a third
issue, which wasn't germane to the isms discussions.  When talking about
user / vendor needs, it would probably be helpful to indicate which of these
three is "the problem" that needs to be addressed, and the extent to which
solving the others is necessary (or not).

Randy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 19 Jul 2011 20:17:45 +0200

> is one little notification type and a lot of applicability statement.

ISMS ended up not sending notifications over SSH sessions (or DTLS
session) initiated by a management system, this is very different from
how NETCONF notifications work. And the way access control is done in
SNMP land on outgoing notifications and the identity of what is being
authenticated by SNMP (transport) security models all adds its share
of complexity.

So yes, lets not get into a loop. Even though I understand how ISMS
ended up with what we have now, I do not consider the solution that
came out of ISMS a particularly practical solution and I can only
encourage people to try to solve "call home" in a more generic way.
Perhaps this time there is more success.

/js

---

From: Phil Shafer <[email-masked]>
Date: Tue, 19 Jul 2011 14:23:23 -0400

"Randy Presuhn" writes:
>overcoming NAT.

The terms "broad" and "narrow" seem meaningless in this discussion.
Can we choose better terms?  "new-device" and "reachability" are
perhaps more self-explanitory.

>The broad problem is a different matter.  David

Under "Alternative Proposals", he lists:

    Develop call-home as an add-on to existing transports, changing
    only the initiator

which is essentially what Kent's done.

So given that this is an existing problem for which there is no
current solution, what's the best path forward?  Are there issues
with Kent's draft?

Thanks,
 Phil

---

From: Kent Watsen <[email-masked]>
Date: Tue, 19 Jul 2011 13:13:07 -0700

> perhaps more self-explanitory.

Regarding the narrow/broad terminology, I tend to use "initial discovery" and "on-going management".  The draft enumerates motivation the for both cases.

> Are there issues with Kent's draft?

When considering this, please reference the -00 draft.  The -01 draft was put together to appease some SAAG/IETF-SSH list members, but it is unnecessarily more complex than the solution presented in the -00 draft.

That said, if there's support for the -00 draft, I recommend updating its bootstrap sequence to be more like SSH, by having the MAC and Host-Key algorithms negotiated.

Thanks,
Kent

---

From: Kent Watsen <[email-masked]>
Date: Tue, 19 Jul 2011 13:32:55 -0700

> My recollection of the conclusion was that the problem space was
> not big enough, and an SSH-specific solution was inappropriate.  
> A more general approach should be investigated.

If this is still a concern, we might be able to extend the protocol so that it could reverse *any* TCP-based client/server protocol (HTTP, POP, IMAP, etc.)  Is there a perceived need to reverse other protocols? - perhaps to resolve the NAT traversal case?

> ...assuming the security experts approve

So far their objections have primarily been that the solution proposed wasn't integrated into the SSH protocol enough.  There were few comments/concerns regarding security (see my July 12 email for details)

> The main reason given is that the client connection maintenance is
> not worth the resources and coding effort.

Just to clarify, the coding effort on the devices is really quite small.  All that is needed is a daemon to initiate the TCP connection, complete the bootstrap sequence, and then fork/exec `sshd -i` in exactly the same way that `inetd` does.  SSH itself can't differentiate how the underlying transport was established.  Juniper has C-based source code for doing this that we have provided to partners in the past, but we should be able to open source it in order to validate the draft, presuming it moves forward...

Thanks,
Kent

---

From: "t.petch" <[email-masked]>
Date: Wed, 20 Jul 2011 11:18:29 +0200

----- Original Message -----
From: "Kent Watsen" <[email-masked]>
To: "Phil Shafer" <[email-masked]>; "Randy Presuhn"
<[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Tuesday, July 19, 2011 10:13 PM

> Regarding the narrow/broad terminology, I tend to use "initial discovery" and
"on-going management".  The draft enumerates motivation the for both cases.
> When considering this, please reference the -00 draft.

How?  With the death of watersprings, I know of no way to get an obsoleted
draft; the IETF web site only gives me -01 which is what I would expect.  If you
really want us to consider the earlier version then I think that you should
re-issue it as -02.

And a URL would help.

Tom Petch

> The -01 draft was put together to appease some SAAG/IETF-SSH list members, but
it is unnecessarily more complex than the solution presented in the -00 draft.
>
> That said, if there's support for the -00 draft, I recommend updating its
bootstrap sequence to be more like SSH, by having the MAC and Host-Key
algorithms negotiated.
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: "t.petch" <[email-masked]>
Date: Wed, 20 Jul 2011 12:42:39 +0200

----- Original Message -----
From: "Kent Watsen" <[email-masked]>
To: "Andy Bierman" <[email-masked]>; "t.petch" <[email-masked]>;
<[email-masked]>; <[email-masked]>
Sent: Tuesday, July 19, 2011 10:32 PM

> A more general approach should be investigated.

If this is still a concern, we might be able to extend the protocol so that it
could reverse *any* TCP-based client/server protocol (HTTP, POP, IMAP, etc.)  Is
there a perceived need to reverse other protocols? - perhaps to resolve the NAT
traversal case?

Kent

As far as I can see, that is impossible with the architecture that the IETF
currently uses.

When a TCP connection is established, then the port (usually) determines the
'application'.
What happens next is up to the application.  It would have been immensely useful
had
there been a requirement for all applications (ie the layer above TCP/UDP/SCTP)
to start with a
negotiation, who is to be client, are we using security, ....?  Some
applications have
precisely this, but it is specific to the application.

So a generic call home has to be application specific, in which case it cannot
be
generic.

What we can do, and what you could have done, is define a new port which conveys
precisely that semantic, negotiation follows, or else role reversal follows,
become
the client to me instead of the server (which is part of what you have done).
But
it remains application specific and I cannot see any other solution.  Where the
application needs role reversal, allocate a new port; but I do not think that
that is
sufficiently complicated to warrant an I-D.

Tom Petch

> ...assuming the security experts approve

So far their objections have primarily been that the solution proposed wasn't
integrated into the SSH protocol enough.  There were few comments/concerns
regarding security (see my July 12 email for details)

> not worth the resources and coding effort.

Just to clarify, the coding effort on the devices is really quite small.  All
that is needed is a daemon to initiate the TCP connection, complete the
bootstrap sequence, and then fork/exec `sshd -i` in exactly the same way that
`inetd` does.  SSH itself can't differentiate how the underlying transport was
established.  Juniper has C-based source code for doing this that we have
provided to partners in the past, but we should be able to open source it in
order to validate the draft, presuming it moves forward...

Thanks,
Kent

---

From: Kent Watsen <[email-masked]>
Date: Wed, 20 Jul 2011 09:39:32 -0700

> draft; the IETF web site only gives me -01 which is what I would expect.  

I don't know what "watersprings" is, but whenever I look at drafts via
http://tools.ietf.org/html/<draft-name>, I'm always able to access all
previous drafts and even get diffs between them...

> And a URL would help.

Here's a directly link to the "-00" draft:

    http://tools.ietf.org/html/draft-kwatsen-reverse-ssh-00

Thanks,
Kent

---

From: "t.petch" <[email-masked]>
Date: Thu, 21 Jul 2011 18:27:07 +0200

----- Original Message -----
From: "Kent Watsen" <[email-masked]>
To: "t.petch" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Wednesday, July 20, 2011 6:39 PM

Kent

Thanks for that.

I was using the I-D search facility on the IETF web site - silly me - and it
only lists -00 even when I select current and expired I-Ds.

watersprings was a website that gave access to all IETF I-Ds going back further
than I even had need to go, via a very easy to use, so simple anyone could
navigate it interface.  Like all the best technology, it has been withdrawn:-(

Tom Petch

Thanks,
Kent