Subject: iSCSI: Out Of Sequence due to null sequence with multiple connections.

From: Douglas Otis <[email-masked]>
Date: Thu, 22 Mar 2001 14:41:32 -0800

All,

Out of Sequence caused by multiple connections for iSCSI:

I attempted to discuss this issue at the IETF meetings but was tabled by
David Black for the Wednesday meeting.  At the beginning of the second
meeting, I was then told I should have attempted to discuss this issue at
the first meeting, and upon reminding him, I was then advised to take this
topic to the reflector.  So here it is and hopefully this will reach the
reflector. If this becomes repeated, my apologies.

Perhaps the inability of the iSCSI protocol to maintain sequence for
immediate commands is not considered important as was already mentioned, it
is seldom that it matters.  The point in time it is likely to matter however
is during these seldom events.  Should a command appear to hang and the
initiator attempts to bring the target into a known state, the likely method
per iSCSI would be to issue an immediate command that carries nulled
sequential information.  The management command could arrive before other
commands that it was attempting to cancel and as such fail to do so.  Rather
than bringing the target to a known state, the state becomes less known.

Solutions-
A) As suggested by David Black, deliver all messages to the target in
sequence and depend on the task attributes to sort out any problem.  This
would remove the use of the immediate command at the iSCSI level and thus
remove a need for the null CmdSN.

B) If the iSCSI sequencer is hung as a result of an unresolved hole in the
sequence or due to a resource limitation delivering to the target, use a
flag within transport to indicate the need for immediate processing but
assign CmdSN to the next sequential value.  Commands normally before the
immediate command pending a send to the target would then be rejected back
to the initiator to allow the initiator to resolve their status.

C) Place the immediate command on all connections and filter duplicates
based on the initiator tag.  From the duplicate packets, the relative
position of this management command could be deduced.

I favor B myself as it assures the acknowledgement of these immediate
commands as well as eliminates the need for sending duplicates.  David's
solutions assumes there will not be a need to deal with the iSCSI sequencer
in the event of a problem.  Sending commands back to the initiator by means
of rejection should allow a freeing of the iSCSI sequencer lock and ensures
the initiator can track the state of the target.  The present scheme does
not allow management nor provide for condition created by multiple
connections.

Doug

---

From: Matt Wakeley <[email-masked]>
Date: Thu, 22 Mar 2001 15:29:10 -0800

In simple terms, what Doug is asking for in (B) is an "immediate" command that
will go and abort all pending commands in the iSCSI queue that have not yet
been delivered to the SCSI layer due to missing commands sequence numbers.

I'm wondering how useful this is.  The initiator could simply look at the
ExpCmdSN from the target and resend the missing command(s) if the target
appears to be "stuck".  Given the way TCP works, the only time the target
could get "stuck" is if a TCP connection fails.  In which case the connection
would be re-established and the missing commands resent.

-Matt

---

From: Douglas Otis <[email-masked]>
Date: Thu, 22 Mar 2001 20:00:47 -0800

Matt,

If the sequencer gets stuck due to some target resource limitation, you are
now sitting behind a queue that you have no control over.  If you use the
immediate command as a bypass to this problem, its placement with respect to
pending commands is unknown.  Tearing down the entire session does not
resolve the state of the target with an ambiguity of the delivery from the
sequencer to the target still unresolved as well as commands in progress.
If the goal is to be sure of the state of the target, severing the session
does not accomplish this.  Assigning a sequence to an immediate command with
those pending at the sequencer to be rejected is a simple control to have in
place that will likely provide the needed actions to resolve a problem.
Disconnecting simply leaves the target state unknown permanently while
hoping the target is not sensitive to replaying commands.

Doug

---

From: Matt Wakeley <[email-masked]>
Date: Thu, 22 Mar 2001 20:14:46 -0800

Doug, re-read my message.  I did not say anything about disconnecting as a
means of recovery.

-Matt

---

From: Douglas Otis <[email-masked]>
Date: Thu, 22 Mar 2001 20:42:35 -0800

Matt,

"There will never be a time where the immediate command will be used to
reach around the sequencer and the only reason a sequencer could be stuck is
due to a hole that can be discovered and eventually filled" if I understand
your statement.

There are also resource limitations between the sequencer and the target
that may lead to a stall or a long term delay that needs to be halted.  As
the sequencer has exhausted these resources, it waits and waits.  If there
is a tag left open for the 'get me out of this' command, the sequencer needs
some means to understand when to employ this trump card.  An immediate
command seems like a good means of creating such a signal. The problem is
what is the final state of this action and is the reason for maintaining
sequence on all commands sent to the target.  Sorry, but I looked to the
next possible action and tried to answer that as well.  I tend to be long
winded I guess as well as long in the tooth.

Doug