Subject: Re: [btns] Q: How to deal with connection latch breaks?

From: "Mike Eisler" <[email-masked]>
Date: Wed, 1 Jul 2009 17:37:10 -0700

Note: This is my first email to this list. The catalyst for this
email was that several documents that I'm authoring indirectly depend
on draft-ietf-btns-connection-latching moving forward ...

> The last DISCUSS on draft-ietf-btns-connection-latching-10
> concerns what the WG thinks is the best way for ULPs to
> handle connection latch transitions to the BROKEN state in
> the _absence_ of connection latching APIs for applications
> (or when apps are not aware of such APIs).

Isn't this DISCUSS specific to SCTP? Russ writes in the DISCUSS:

I am unsure that the SCTP section defines behavior which is consistent with
application expectations.  The last paragraph of 5.4 implies that the whole
connection terminates if one of the latches breaks.  This has an impact on
the
semantics of the application socket API.  While connection latching is
transparent when everything is working, there are new failures that ripple to
the application.  That is, the application will observe different behavior
on a
connection with and without latching.

My conclusion is that the API ought to provide information for the
application
about the connection latching, and it just does not seem to be there.  If you
can point me to a discussion of this topic on the WG mail list, then I'll
clear.  I'm not trying to alter consensus, but I do want to make sure that
this
topic was considered.

======================

The laat paragraph of section 5.4 says:

"When a connection latch transitions to the BROKEN state and the
   application requested (or system policy dictates it) that the
   connection be broken, then SCTP should inform the application, if
   there is a way to do so, or else it should wait, allowing SCTP path/
   endpoint failure detection (and/or application-layer keepalive
   strategy) to timeout the connection.  When a connection latch is
   administratively transitioned to the CLOSED state then SCTP should
   act as though an ABORT chunk has been received."

Note the inconsistency between what Russ wrote in his DISCUSS:

"The last paragraph of 5.4 implies that the whole
connection terminates if one of the latches breaks."

and this excerpt from the last paragraph of section 5.4:

"When a connection latch transitions to the BROKEN state and the
   application requested (or system policy dictates it) that the
   connection be broken,"

So actually the last paragraph is unequivocal: the connection
does terminate if two conditions are met:
- the latch goes to BROKEN
- the app or policy has previously dictated that a latch break produces
  a connection break.

Earlier in Section 5.4, two alternatives to mapping latches to multi-homed
SCTP connections:

"We can represent the multiplicity of SCTP association end-point
   addresses as a multiplicity of 5-tuples, each of which with its own a
   connection latch.  Alternatively we can extend the connection latch
   object to support a multiplicity of addresses for each end-point.
   The first approach is used throughout this document, therefore we
   will assume that representation."

Perhaps the second approach is what should be used for SCTP. However, as
the last sentence above notes, the first approach is used in the rest
of the document. I gather what Russ is saying is that approach might
not be appropriate for SCTP and he wants to know if the WG thoroughly
considered it.

In the event the WG did not thoroughly consider the implications
of multi-homed SCTP connections and latching, here is a pragmatic
suggestion:

- remove the SCTP discussion from the I-D. Most this means removing
  section 5.4, but there are a couple sentences that mention SCTP that
  would also need to be removed.

- if there is sufficient interest in SCTP and connection latching,
  channel binding, etc., then start a new work item to deal with
  the issue of connection latching with multi-homed connection-oriented
  transports.

-------------------------------------------------------

Nico wrote:

    * From: Nicolas Williams <Nicolas.Williams at sun.com>
    * To: btns at ietf.org
    * Date: Wed, 24 Jun 2009 15:17:07 -0500

The last DISCUSS on draft-ietf-btns-connection-latching-10 concerns what
the WG thinks is the best way for ULPs to handle connection latch
transitions to the BROKEN state in the _absence_ of connection latching
APIs for applications (or when apps are not aware of such APIs).

The two options are:

a) The ULP MAY/SHOULD/MUST pretend that the equivalent of a TCP reset
   occurred and the connection latch is transition to the CLOSED state
   (and destroyed/cleaned up);

b) The ULP MAY/SHOULD/MUST act as though bits aren't moving and let ULP
   and/or application-layer timeout processing decide if and when to
   close the connection (and underlying connection latch).

(b) means potentially hanging forever, but that's generally true now
with existing ULPs.

The I-D says (b), with "SHOULD".

I'd be just as happy with (a), SHOULD, (b) MAY.  I would not be happy
with (a), MUST, nor with (b), MUST.

Comments?

Nico
-- 

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Michael Richardson <[email-masked]>
Date: Sun, 12 Jul 2009 14:54:24 -0400

>>>>> "Mike" == Mike Eisler <[email-masked]> writes:
    >> concerns what the WG thinks is the best way for ULPs to handle
    >> connection latch transitions to the BROKEN state in the _absence_
    >> of connection latching APIs for applications (or when apps are
    >> not aware of such APIs).

    Mike> Isn't this DISCUSS specific to SCTP? 

  I don't think so. 
  I think it's just that SCTP tries to get around other network
connection failures by switching to working end-points, while TCP and
UDP do not, so those applications are already more suspectible to
network failures.

  (First, if you are using channel binding, then you have to be aware of
it, so it is possible that we haven't thought through the non-channel
binding case well enough)

    Mike> I am unsure that the SCTP section defines behavior which is
    Mike> consistent with application expectations.  The last paragraph
    Mike> of 5.4 implies that the whole connection terminates if one of
    Mike> the latches breaks.  This has an impact on the semantics of
    Mike> the application socket API.  While connection latching is
    Mike> transparent when everything is working, there are new failures
    Mike> that ripple to the application.  That is, the application will
    Mike> observe different behavior on a connection with and without
    Mike> latching.

  I agree that this is the case for SCTP.
  For TCP, it looks like a normal network failure, do you agree?

  I think that how the latch breaks when the SA breaks depends upon
whether there are in fact multiple SAs, (the simple NxM approach), or if
the connection latch object is represented as a the N+M object, with
underlying SAs negotiated as needed.

    Mike> Perhaps the second approach is what should be used for
    Mike> SCTP. However, as the last sentence above notes, the first
    Mike> approach is used in the rest of the document. I gather what
    Mike> Russ is saying is that approach might not be appropriate for
    Mike> SCTP and he wants to know if the WG thoroughly considered it.

  I think that we did not consider it enough.

    Nico> a) The ULP MAY/SHOULD/MUST pretend that the equivalent of a
    Nico> TCP reset occurred and the connection latch is transition to
    Nico> the CLOSED state (and destroyed/cleaned up);

    Nico> b) The ULP MAY/SHOULD/MUST act as though bits aren't moving
    Nico> and let ULP and/or application-layer timeout processing decide
    Nico> if and when to close the connection (and underlying connection
    Nico> latch).

    Nico> (b) means potentially hanging forever, but that's generally
    Nico> true now with existing ULPs.

    Nico> The I-D says (b), with "SHOULD".

    Nico> I'd be just as happy with (a), SHOULD, (b) MAY.  I would not
    Nico> be happy with (a), MUST, nor with (b), MUST.

    Nico> Comments?

  Tell me why the connection latch broke?

  I do not like layers that kill connections unlaterally because *they*
think they know what my application's timeout is.  

- -- 
]     Y'avait une poule de jammé dans l'muffler!!!!!!!!!        |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
]    h("Just another Debian GNU/Linux using, kernel hacking,    ruby  guy");  [

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 20 Jul 2009 11:00:59 -0700

On Sun, July 12, 2009 11:54 am, Michael Richardson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>

My point was that because SCTP gets around connection drops, and TCP
doesn't, the issue is specific to TCP. Apps that use TCP are used
to TCP dropping connections. Apps that use SCTP are not used a total
failure because of the "get around" behavior of SCTP.

>     Mike> observe different behavior on a connection with and without

Note that above is the content of the IESG DISCUSS from Russ.

>   For TCP, it looks like a normal network failure, do you agree?

I agree. Hence the DISCUSS looks specific to SCTP to me.

>   I think that we did not consider it enough.

Fair enough. So given that, perhaps connection latching for SCTP
should be deferred to another work item?
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Finger me for keys
>
> iQEVAwUBSloxWICLcPvd0N1lAQJ85Qf+MyRCh4x7Rt/EDvqh4haa2jaVqGnadnOA
> I3K2CbROKibly3QgRa6wN51nKWuPWyFcs1/dk8PzjRa6L/wnSL505cFTu30tVJ2j
> 89B1ykdodfXfTmyxR3j0rMysP13dHaiL+CizZiu/B9w/chhExYSuE/Jyw7SHy0uH
> jbBJV2lXHRgD8dWYQ1qey+fLmjwp/HhVtZm4+epMhlsECsFIGmZktkiW9UjeTakh
> 6rXJmmaF7lNAzq4eGJTKjNSQQSy+bq+b1UOR0sYJguNC0W8yH5xX/g3/lBLPxVH9
> yb+3X+6dUsW8zRXgJ9w7gSAhbJ0FvN7VxInZ9vzCkt97eLk7I37QnA==
> =Y4Mx
> -----END PGP SIGNATURE-----
>

---

From: Julien Laganier <[email-masked]>
Date: Mon, 20 Jul 2009 12:03:53 -0700

Hello,

On Mon, Jul 20, 2009 at 11:00 AM, Mike Eisler<[email-masked]> wrote:
>>     >> The last DISCUSS on draft-ietf-btns-connection-latching-10
>>     >> concerns what the WG thinks is the best way for ULPs to handle
>>     >> connection latch transitions to the BROKEN state in the _absence_
>>     >> of connection latching APIs for applications (or when apps are
>>     >> not aware of such APIs).
>>

I think that's indeed the case.

> should be deferred to another work item?

IIRC, as a result from requesting publication we were asked (during
IETF LC, maybe as a TSV directorate review) to cover SCTP and not only
TCP and UDP. So while we maybe didn't considered all the implications
of specifying SCTP behavior, it is my understanding that not covering
it is not an option on the table.

--julien

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 24 Jul 2009 13:01:55 -0700

Jilien, thanks for respondin. Inline ...
> it is not an option on the table.

OK.

So how about text (specific places to add this and wording, pending
if/when we have consensus) that says:

For UDP and TCP, the implementation SHOULD provide an unambiguous
indication that the connection drop was due to "unlatching" the
connection.

For SCTP, the implementation MUST provide an unambiguous indication that
the connection drop was due to "unlatching" the connection. Rationale:
SCTP is a transport that is "multi-homed"; connection drops should thus be
rare, and applications using SCTP will be better able to cope with a drop
across
all end-points with an unambiguous error indicating an unlatch.

(Obviously, I am not sure if the verb to "unlatch" is the correct
term here).

[My responses on this threa dhave been tardy due to holiday and getting
reading to go to Stockholm ... and will be boarding the plane for
Stockholm in about an hour].

>
> --julien
> _______________________________________________
> btns mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/btns
>

---

From: Nicolas Williams <[email-masked]>
Date: Sun, 26 Jul 2009 17:13:31 -0500

> Isn't this DISCUSS specific to SCTP? Russ writes in the DISCUSS:

Sortof.

Russ wrote, at one point:

| This is the point.  I do not think that hangs until killed or
| restarted should be on this menu.

with respect to the default handling of latch breaks when no additional
APIs are available (or used by the application).

IMO it's a generic issue.

My earlier opinion was that pretending that bits aren't moving is the
right thing to do because applications ought to have timeout handling,
if not even ULPs (think of SO_KEEPALIVE).  After all, latch break is not
fatal -- state transitions from BROKEN to ESTABLISHED are allowed.

But reasonable people can disagree about that.  A latch break is likely
to be permanent in practice, and in any case, acting as though the
connection has been reset (as in valid RST received) is no more or less
harmful than acting as though bits aren't moving.

Also, a latch break may be indicative of an attack, and so resetting the
connection may be a better action[*].

To a reply along those likes Russ wrote back:

| I'm happy with a SHOULD statement that the WG agrees is appropriate
| to handle the situation gracefully.

This thread is all about obtaining WG consensus on this issue, so we can
clear this DISCUSS.  All we need is consensus.

I'll be happy with either possible default behavior on the face of latch
breaks: act as if the connection was reset, or act as if bits aren't
moving.

I'd also be happy to require that implementors pick and implement one of
those two, without us giving a recommendation.  After all, neither
default affects interoperability, so why should we recommend or require
one or the other?

> I am unsure that the SCTP section defines behavior which is consistent
> with application expectations.  The last paragraph of 5.4 implies that
> the whole connection terminates if one of the latches breaks.  This
> has an impact on the semantics of the application socket API.  While
> connection latching is transparent when everything is working, there
> are new failures that ripple to the application.  That is, the
> application will observe different behavior on a connection with and
> without latching.

Surely SCTP, and any connection-oriented ULP, must have a way to reset
connections, just like TCP does.  A latch break causing a connection
reset would not be new behavior as far as applications go.

> application about the connection latching, and it just does not seem
> to be there.  If you can point me to a discussion of this topic on the
> WG mail list, then I'll clear.  I'm not trying to alter consensus, but
> I do want to make sure that this topic was considered.

APIs are nice, but existing apps won't use them until updated, and
anyways, connection latching adds value even without adding APIs, which
means we need a default response to latch breaks in the absence of new
APIs (either because not implemented or not used).

[*] Or not -- if the app is not doing authentication at all and the user
    is not doing leap-of-faith, then resetting the connection would be
    bad.  But then, the user can always interrupt the application and try
    again.

Nico
--

---

From: Michael Richardson <[email-masked]>
Date: Mon, 27 Jul 2009 12:59:01 -0400

>>>>> "Mike" == Mike Eisler <[email-masked]> writes:

    Mike> So how about text (specific places to add this and wording,
    Mike> pending if/when we have consensus) that says:

    Mike> For UDP and TCP, the implementation SHOULD provide an
    Mike> unambiguous indication that the connection drop was due to
    Mike> "unlatching" the connection.

    Mike> For SCTP, the implementation MUST provide an unambiguous
    Mike> indication that the connection drop was due to "unlatching"
    Mike> the connection. Rationale: SCTP is a transport that is
    Mike> "multi-homed"; connection drops should thus be rare, and
    Mike> applications using SCTP will be better able to cope with a
    Mike> drop across all end-points with an unambiguous error
    Mike> indicating an unlatch.

  So, if my stack said EUNLATCHED instead of ETIMEOUT, you'd be happy?

  (Of course, if you unplug all the "cables" to any ULP, and there is some
kind of make-dead process in the ULP, then you get ETIMEOUT eventually,
regardless of any wizardry that SCTP provides...)

---

From: Michael Richardson <[email-masked]>
Date: Mon, 27 Jul 2009 13:01:23 -0400

>>>>> "Nicolas" == Nicolas Williams <[email-masked]> writes:
    >> My conclusion is that the API ought to provide information for
    >> the application about the connection latching, and it just does
    >> not seem to be there.  If you can point me to a discussion of
    >> this topic on the WG mail list, then I'll clear.  I'm not trying
    >> to alter consensus, but I do want to make sure that this topic
    >> was considered.

    Nicolas> APIs are nice, but existing apps won't use them until
    Nicolas> updated, and anyways, connection latching adds value even
    Nicolas> without adding APIs, which means we need a default response
    Nicolas> to latch breaks in the absence of new APIs (either because
    Nicolas> not implemented or not used).

  So, errno value from write(2) is not a new API.
  A new errno value should provide enough information, I think.

  However, an application might know what to do with ETIMEOUT, while it
would not know what to do with EUNLATCHED or some such....

- --

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 27 Jul 2009 12:54:36 -0500

>   A new errno value should provide enough information, I think.

If an API's definition has a closed set of error codes, then no.  That
said, I believe that's not the case for write(2) and friends.  And, of
course, there may be implementors for whom it's not possible to add new
error codes for existing APIs, thus we must provide a recommendation or
requirement for such implementors.

> would not know what to do with EUNLATCHED or some such....

There's nothing unreasonable about returning ECONNRESET (an existing
error) when the latch breaks IF the application did not use any APIs to
indicate that it knows about latching.

If the errno set for I/O syscalls is not a closed set then there's also
nothing unreasonable about adding EUNLATCHED.

Note that if a latch never breaks then the connection should never
reset.  So EUNLATCHED would only distinguish latch breaks from peer
reboots -- not exactly Earth shaking for an application that is unaware
of latching in the first place.  But for apps that use new APIs
distinguishing latch breaks is important.

I think I/O syscalls return ETIMEDOUT when SO_KEEPALIVE is set and the
connection times out.  Again, distinguishing timeout from latch break in
the application-is-not-using-new-APIs case doesn't seem all that
important.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 30 Jul 2009 03:41:19 -0700

> would not know what to do with EUNLATCHED or some such....

Isn't the application the entity that decided to enable latching?  If so,
the application (or its author) needs to be prepared to deal with new
error indication.

If not, what is the model for enabling latching?
> iQEVAwUBSm3dYoCLcPvd0N1lAQLnNQf8CNCa4TWJY5I/TvtAH4+eyLbHzyO+I/dp
> /l5Zcs3Ld16qhYh9OeWogBT2wCeicdx3cL1PdoyibTOfMw/PPxUSY+HKWyR78Ky5
> afHA8JxRypvpLqULU7i2jCfwNJZbQi4wojPYl9oeQGAKKmwX3hsBQxKtWY5OWLh+
> ExHXfZpGfvauG+LVYHMSNATJEveESpgqRXQyxUqGEChblWVFANHjX/OLJTRCZ7vQ
> xXhI2rGIlBOwl9ssAx4vejDR0js8vQ98RXy7XiVJhGZq6+hklOuZfmW44T4iHUNk
> emB7EpEWwO7MshVPRCxARIXjICaG2xnQ0XR9ZwSvbtNMsS1vCGVeVg==
> =5max

---

From: "Laganier, Julien" <[email-masked]>
Date: Thu, 30 Jul 2009 04:30:21 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Mike Eisler
> Sent: Thursday, July 30, 2009 3:41 AM
> To: [email-masked]
> Subject: Re: [btns] Q: How to deal with connection latch breaks?
> 
> 

The current draft says " Implementations SHOULD create IPsec channels automatically by
default when the application does not explicitly request an IPsec channel.  Implemenations MAY provide a way to disable automatic creation of connection latches." Thus the application might not know what is EUNLATCHED.

Would it be possible that the application gets ETIMEOUT, unless it has used a BTNS connction latching API of some sort, in which case it presumably knows about latching and can be returned an EUNLATCHED?

--julien

---

From: Michael Richardson <[email-masked]>
Date: Thu, 30 Jul 2009 07:36:31 -0400

>> So, errno value from write(2) is not a new API.  A new errno
    >> value should provide enough information, I think.
    >> 
    >> However, an application might know what to do with ETIMEOUT,
    >> while it would not know what to do with EUNLATCHED or some
    >> such....

    Mike> Isn't the application the entity that decided to enable
    Mike> latching?  If so, the application (or its author) needs to be
    Mike> prepared to deal with new error indication.

  It may be enabled by system-policy, in particularly for the passive open
side ("bind()/accept()" side). It could also be enabled by the "inetd"
equivalent, and then pass that descriptor on.

- --