Subject: SHUTDOWN received in SHUTDOWN-PENDING state

From: Stephane Chazelas <[email-masked]>
Date: Wed, 8 Mar 2006 17:39:43 +0000

Hi guys,

I wanted to point out an unclear area in the RFC 2960:

What should happen if we receive a SHUTDOWN while in
the SHUTDOWN-PENDING state?

The RFC says we shouldn't leave that state until all our
outstanding data is acknowledged. But it also says that upon
receiving a SHUTDOWN (it doesn't mention the initial state), we
should enter the SHUTDOWN-RECEIVED state.

Those are conflicting statements.

Looking at three different implementations, I can see 3
different behaviors: lksctp (AFAICS) discards the shutdown
(which I think is a problem as the cTSN ack in it is not taken
into account, so it may be stuck there if the other end keeps
acknowledging DATAs with SHUTDOWNs as the RFC permits), BSD
treats it the same way as if it was received in the ESTABLISHED
state (it enters the SHUTDOWN-RECEIVED state) and SpiderSCTP
handles the cTSN ack, notifies the user (so that it stops
sending data which is probably meaningless in that state)
but stays in the SHUTDOWN-PENDING state (so won't send a
SHUTDOWN_ACK).

I think it would help if the RFC could be clarified as to what
to do in that case. Both the SpiderSCTP and BSD behaviors make
sense to me in that case.

SpiderSCTP will send a SHUTDOWN when all its data is
acknowledged (it decides to go with its own (the user's)
shutting down). Upon reception of that SHUTDOWN, the other end
will take it as a SHUTDOWN collision case and send a
SHUTDOWN_ACK back.

BSD will send a SHUTDOWN ack when all its data is acknowledged
(it kind of forgets about the user's shutdown and decides to go
with the peer's shutting down).

In both cases, the shutting down will perform OK without needing
any retransmission of shutdowns or shutdowns ACKs. I think
there's a problem in Linux approach, as in this scenario

User      Linux                  other
         
          DATA [TSN=1] -------->
                       <-------- SACK [cTSN=1]
          DATA [TSN=2] -------->
[shutdown] /* Linux entering shutdown pending state */
                       
         (discarded)   <-------- SHUTDOWN [cTSN=2]
[T3-rtx timer]
          DATA [TSN=2] -------->
         (discarded)   <-------- SHUTDOWN [cTSN=2]
[T3-rtx timer]
          DATA [TSN=2] -------->
         (discarded)   <-------- SHUTDOWN [cTSN=2]
[T3-rtx timer]
          DATA [TSN=2] -------->
         (discarded)   <-------- SHUTDOWN [cTSN=2]
	 ... aborting
	
(note that this is on quick code reading, not testing, I may be
wrong at interpreting how lksctp works)

Best regards,
Stephane

---

From: "Meyknecht, Bernward" <[email-masked]>
Date: Wed, 8 Mar 2006 19:31:12 +0100

Hi Stephane,

a good point, your email triggered me to check out, what my Siemens SCTP does
in that case: it behaves like SpiderSCTP with handling cTSN ack,
somehow the same as processing repeated Shutdown received in Shutdown-Received 
state: don't change the SCTP state, just process cTSN ack (the latter is a 
MUST for me).

So we have 4 implementations with 3 behaviors ;-)

I think both of the 2 working approaches from below should be mentioned as 
allowed in the RFC(bis).

Best regards,

Bernward

-----Ursprüngliche Nachricht-----
Von: Stephane Chazelas [mailto:[email-masked]] 
Gesendet: Mittwoch, 8. März 2006 18:40
An: SCTP Implementors
Betreff: SHUTDOWN received in SHUTDOWN-PENDING state

Hi guys,

User      Linux                  other
         
          DATA [TSN=1] -------->
                       <-------- SACK [cTSN=1]
          DATA [TSN=2] -------->

Best regards,
Stephane

---

From: "Peter Lei (peterlei)" <[email-masked]>
Date: Wed, 08 Mar 2006 12:39:18 -0600

Just a note of caution on reading the BSD implementation
code:

The SHUTDOWN_PENDING 'state' is a flag on the current state
and not actually a separate state-- it is set anytime a
shutdown is pending, but we still have data to send.  You
must look at the bottom bits (e.g. via SCTP_GET_STATE()) to
find out what state the assoc is actually in.

Thus, BSD does not "forget about the user's shutdown"-- if
there is data still to be sent, the pending flag is set.
No more user is accepted when this flag is set.  Since the
stack has not sent the shutdown yet, it is in the
ESTABLISHED+SHUTDOWN_PENDING state.  Since a shutdown was
received in this state, it will transition to the
SHUTDOWN_RECEIVED+SHUTDOWN_PENDING state, which is what I
think you are mis-interpreting.  The Cum-Ack in the shutdown
is processed in either case.

So, the on-the-wire behaviour I believe is not different
than the SpiderSCTP case-- the internal states in BSD may
be a bit more accurate, since you are not really in the
SHUTDOWN_SENT state if the stack had never sent one out
yet, but receive one from the peer first.  But in both
cases, the shutdown sequence will complete correctly. :-D

regards,
--peter

---

From: "Peter Lei (peterlei)" <[email-masked]>
Date: Wed, 08 Mar 2006 12:41:06 -0600

[Resending: our mailer doesn't like message signatures...]

regards,
--peter

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 8 Mar 2006 12:28:15 -0700

Peter,

OpenSS7 SCTP (both Sockets and STREAMS) behaves like BSD (sends
SHUTDOWN-ACK when data clears after receiving SHUTDOWN in
SHUTDOWN-PENDING).  I think the RFC is clear and that the the
only problem is lksctp implementation.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart <[email-masked]>
Date: Wed, 08 Mar 2006 19:07:56 -0500

> only problem is lksctp implementation.
I agree :-D

R

-- 
Randall Stewart
ITD - Cisco Systems Inc.
803-345-0369 <or> 815-342-5222 (cell)

---

From: Stephane Chazelas <[email-masked]>
Date: Thu, 9 Mar 2006 09:59:23 +0000

> only problem is lksctp implementation.
[...]

Hi Brian,

so what about the RFC saying that we shouldn't leave the
SHUTDOWN-PENDING state?

RFC> Upon receipt of the SHUTDOWN primitive from its upper layer, the
RFC> endpoint enters SHUTDOWN-PENDING state and remains there until all
RFC> outstanding data has been acknowledged by its peer.
[...]
RFC> Once all its outstanding data has been acknowledged, the endpoint
RFC> shall send a SHUTDOWN chunk to its peer including in the Cumulative
RFC> TSN Ack field the last sequential TSN it has received from the peer.
RFC> It shall then start the T2-shutdown timer and enter the SHUTDOWN-SENT
RFC> state.  If the timer expires, the endpoint must re-send the SHUTDOWN
RFC> with the updated last sequential TSN received from its peer.

If I understand the RFC correctly there, then we shouldn't leave
the SHUTDOWN-PENDING state upon receiving the SHUTDOWN unless
its cum-ack acknowledges all our outstanding data, and we
shouldn't send a SHUTDOWN_ACK, but a SHUTDOWN (as Siemens or
Spider SCTP do).

But then, there's that section about processing incoming
SHUTDOWNs that is not very clear either.

-- 
Stephane

---

From: Stephane Chazelas <[email-masked]>
Date: Thu, 9 Mar 2006 10:01:08 +0000

> find out what state the assoc is actually in.

Oh yes, sorry,

I didn't study the code in great details, I just noticed that
there was a macro for pending state similar to those for the
other states without noticing that this was actually treated
differently.

> is processed in either case.

Well, I think it resolves to the same behavior. I think it is
valid to consider PENDING as a separate state (most
implementations do, don't they) and to transition to
SHUTDOWN_RECEIVED in that case.  We will still retransmit our
pending data until it is acknowledged.

> than the SpiderSCTP case

I think it is, if I understand correctly, SpiderSCTP will send a
SHUTDOWN when all its DATA is acknowledged, while BSD will send
a SHUTDOWN_ACK when all its DATA is acknowledged.

> -- the internal states in BSD may
> yet, but receive one from the peer first.

SpiderSCTP doesn't enter that state upon receiving the
SHUTDOWN, it stays in the SHUTDOWN_PENDING state.

> But in both
[...]

That's what it looks like, but I think the RFC still needs
clarification.

Thanks,
Stephane

---

From: Randall Stewart <[email-masked]>
Date: Thu, 09 Mar 2006 13:11:00 -0500

Stephane:

I see no point in forcing one implementation to
change.. since changing the spec would do that.

Considering that the overall outcome is correct.. it
really does not matter IMO if you do what spider does
or do what BSD does..

SHUTDOWN-PENDING is IMO a sub-state.. a marker so you
know NOT to allow the user to send and at the same
time you allow your data to de-queue.

When you receive a SHUTDOWN from the other end and
you are still de-queuing your data,
at the end of that.. it matters not if you:

send a SHUTDOWN-ACK and then get back a SHUTDOWN-COMPLETE

<OR>

You send a SHUTDOWN, force the other end to send you
a SHUTDOWN-ACK and then you send a SHUTDOWN-COMPLETE.

The latter does send one more packet.. but you are talking
about a rare corner case.. so I am not concerned about
it and I really don't think it matters.. afterall the
graceful shutdown does complete properly in either
case.. which is the goal..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 9 Mar 2006 12:03:26 -0700

Randall,

I agree.

--brian

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 9 Mar 2006 20:10:54 +0100

ACK.

Best regards
Michael

---

From: "Peter Lei (peterlei)" <[email-masked]>
Date: Thu, 09 Mar 2006 13:55:20 -0600

Correct... this is handling of a collision case and both
scenarios results in a graceful shutdown (except for lksctp
which *may* have a problem).

--peter