Subject: Close with unread data - (Was Re: RFC: Linger and Autoclose of associations)

From: Sridhar Samudrala <[email-masked]>
Date: Tue, 16 Aug 2005 17:05:35 -0700

On Tue, 2005-08-16 at 14:23 -0400, Randall Stewart wrote:
> Peter Lei (peterlei) wrote:
> > Anatoly Khusid wrote:
> > 
> >> I thought the default behavior for close() is to gracefully terminate 
> >> SCTP
> >> association?
> > 
> > 
> > Yes, it should be...
> 
> 
> I think this is true IF and ONLY IF the user has read
> all the data. If the socket is closed with data still unread
> then an ABORT is sent.. I believe this is true in TCP
> as well.. aka a RST is set if you close and did not clear
> your socket buffer... this is to alert the peer that you
> did not read all the data that was acknowledged.

Currently in lksctp, on close, we do a graceful shutdown when linger
option is not set even if there is unread data in the socket
buffer. This is definitely a bug.

I would like to know how other implementations handle this
case for 1-many style socket with multiple associations.
Do they have a per-association receive queues or walk through the
the single socket receive queue to identify the associations with
unread data?
The unread data may belong to only a subset of associations. So if we
are doing this correct, only those associations need to be aborted and
the remaining should go through a graceful shutdown.

Thanks
Sridhar

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 16 Aug 2005 22:51:24 -0600

Sridhar,

See RFC 2525, section 2.16 ("Failure to send a RST after Half Duplex
Close") and section 2.17 ("Failure to RST on close with data pending").
Note that the later (error in implementation for TCP) is less severe if
the former (error) is not present.  This has more to do with TCP
half-open connections that we tried to avoid with the design of SCTP.

Consider the same situation for an SCTP 1-to-1 connection: connect to an
SCTP FTP server, do one big GET, suspend the client until a zero window
is advertised, and then kill the client.  If the client does not ABORT
the connection (because there is unread data), but chooses graceful
shutdown instead, it will send SHUTDOWN and, if it implements the
shutdown guard timer, will start the shutdown guard timer to 5 RTOs.
The server will (hopefully) receive the SHUTDOWN but still has data to
send so it waits for outstanding data to be acknowledged.  It will do
heartbeats and zero window probes in this interval.  Any heartbeats and
zero window probes will be properly handled and the client will wait
until it receives SHUTDOWN-ACK or the timer expires.  In this case the
timer expires and the client sends ABORT and destroys the TCB.

So, if you do not abort on unread data, it will take a 1-to-1
association up to 5 * RTOs to disconnect.  With an RTO-min of 1 second,
this is at least 5 seconds.  If you ABORT with unread data, the
disconnect should be almost immediate (1/2 RTT, could be milliseconds).
This still has some impact on large server installations that will tie
up resources for longer than necessary.

As described for TCP in RFC 1122 section 4.2.2.13:

 "A host MAY implement a "half-duplex" TCP close sequence, so that an
 application that has called CLOSE cannot continue to read data from the
 connection.  If such a host issues a CLOSE call while received data is
 still pending in TCP, or if new data is received after CLOSE is called,
 its TCP SHOULD send a RST to show that data was lost."

So, even for SCTP it is a good idea on a 1-to-1 socket to ABORT the
association if there is unread data at close(), regardless of the
setting of linger.  Just s/TCP/SCTP/g;s/RST/ABORT/g in the above.

As regards linger, linger controls the behaviour when there is unsent
data, and also provides an ABORT call for TCP.

When the l_onoff option is set and the l_linger time is set to zero (0),
a RST is sent in TCP.  This is just the way that the RFC 793 ABORT
command is implemented on BSD sockets for TCP.  It is the only way for
the user to force an abortive disconnect using sockets for TCP.  (See,
for example, Stevens, "TCP Illustrated Volume 1", Aborting a Connection,
p.  247.)  So, when l_onoff is set and l_linger time is 0, the
connection is aborted regardless of whether there is and unsent data.

When l_onoff is set and l_linger time is a positive non-zero time,
close() blocks for l_linger time or until all unsent data has been sent.
This is clear from the POSIX close() reference page for sockets, that
says something like:

 "If fildes refers to a socket, close() shall cause the socket to be
 destroyed.  If the socket is in connection-mode, and the SO_LINGER
 option is set for the socket with non-zero linger time, and the socket
 has untransmitted data, then close() shall block for up to the current
 linger interval until all data is transmitted."

Also the POSIX "Sockets" section in "System Interfaces, General
Information" that says something like:

 "The SO_LINGER option controls the action of the interface when unsent
 messages are queued on a socket and a close() is performed.  The
 details of this option are protocol-specific.  The default value for
 SO_LINGER is zero, or off, for the l_onoff element of the option value
 and zero seconds for the linger time specified by the l_linger
 element."

What POSIX doesn't really say is what happens after the timeout, or if
linger is not set, and data remains to be transmitted.  But RFC 793 does,
somewhat for the CLOSE command:

 "... Closing connections is intended to be a graceful operation in the
 sends that outstanding SENDs will be transmitted (and retransmitted),
 as flow control permits, until all have been serviced.  Thus, it should
 be acceptable to make several SEND calls, followed by a CLOSE, and
 expect all the data to be sent to the destination.  It should also be
 clear that users should continue to RECEIVE on CLOSING connections,
 since the other side may be trying to transmit the last of its data.
 Thus, CLOSE means "I have no more to send" but does not mean "I will
 not receive any more."  It may happen (if the user level protocol is
 not well thought out) that the closing side is unable to get rid of all
 its data before timing out.  In this event, CLOSE turns into ABORT, and
 the closing TCP gives up."

So, it would appear that, for TCP at least, when the l_linger time
expires, and there remains data to be sent, a RST is sent.  The same
behaviour makes sense for SCTP too (but, of course, an ABORT is sent).

When linger is off, it appears that the thing to do is to ABORT if there
is still data to be sent.  If the association is waiting for a
SHUTDOWN-ACK, (i.e. FIN-WAIT) then the socket can be orphaned and
allowed to complete the graceful shutdown.  If it is closed, the point
is moot.

That is what I do for OpenSS7 implementation of 1-to-1.

For 1-to-many, I would think the analysis should be the same.  If the
application closes the socket with unread data, an ABORT should be sent
on all associations belonging to that socket.  If l_onoff is 1, and
l_linger is 0, an ABORT should be sent on all associations.  If l_onoff
is 1, and l_linger is some time period, close() should block for that
period of time, or until all data has been transmitted and SHUTDOWN or
SHUTDOWN-ACK sent on all associations still able to send data.  If the
timer expires, all associations with data still to be transmitted should
be aborted.  If l_onoff is 0, and there is data waiting to be
transmitted, all such associations should be aborted.

It think that 1-to-many should be treated as a bundle of 1-to-1
associations, and close() treated as though close() was called
simultaneously for each of the associations in the bundle.  But then
again, that is only if you consider 1-to-many to be "connection-
oriented" and not "connectionless".

Those are my thoughts...  ...hope it helps.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Sridhar Samudrala <[email-masked]>
Date: Wed, 17 Aug 2005 10:50:43 -0700

Brian,

On Tue, 2005-08-16 at 22:51 -0600, Brian F. G. Bidulock wrote:

> on all associations belonging to that socket.

I was referring to this specific case. linger is not set(l_onoff is 0).
Even if the unread data belongs to only a subset of the associations,
do you send ABORT on all the associations? 
Suppose i have a 1-many socket with 5 associations and there is single 
pending unread message belonging to assoc1, and the app does a close on
the socket, i think the right behaviour would be to send ABORT on assoc1
and SHUTDOWN on the remaining 4 associations? Do you agree with this?

Thanks
Sridhar

>   If l_onoff is 1, and

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 17 Aug 2005 20:05:16 +0200

At least I think that you only ABORT associations where unread
messages do exist. All other should use the SHUTDOWN procedure.

Best regards
Michael

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 17 Aug 2005 15:41:03 -0600

Sridhar,

Yes, that's more in fitting with the view of a 1-to-many being
treated as a bundle of 1-to-1's.

--brian