Subject: Re: iSCSI - Change Proposal X bit

From: "Julian Satran" <[email-masked]>
Date: Wed, 24 Oct 2001 19:44:22 +0300

Santosh,

There is nothing in a command that arrives late on a link (as in the 
example in which it was sent redundantly)  to distinguish it from a new 
(valid) command.

This wraparound problem exists in all protocols - even in TCP, and we use 
the CmdSN per session in the same fashion TCP uses sequence numbers per 
connection - and it is solved in different ways (TCP uses time-stamps).

The NOP is meant to solve that wrap-around problem.

I am sure that when rereading the example you will see the issue.

Julo

Santosh Rao <[email-masked]>
Sent by: [email-masked]
24-10-01 18:29
Please respond to Santosh Rao

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     [email-masked]
        Subject:        Re: iSCSI - Change Proposal X bit

Julian,

Some comments on the below quoted scenarios :

>    session has 3 connections
>    on connection 1 I->T c1,c2,c3,C6
>    on connection 2 I->T c4,c5,c7,c8
>    Target receives 1,2,4,5,7,8 (miss 3 and 6) and acks 1 & 2
>    Initiator closes 1 and resends c3, c4, c5,c7,c8  on connection 2 and 
6
>    on connection 3
>    target receives all and starts executing and acks 8 on connection 3 
but
>    connection 2 stalls after c3 for a  LONG TIME
>    then (after 2 full sequence wraps) connection 2 is gets alive and
>    delivers c4,c5 etc (that are now valid)

When the target acks CmdSN 8 on connection 3, it has, in effect, sent
CmdSN ack's for CMdSNs 3,4,5,6,7,8. This implies that the commands with
CmdSN 3, 4, 5, 7, & 8 were received by the target on connection 2 and
their processing was commenced.

Hence, the following does not make sense :

>    connection 2 stalls after c3 for a  LONG TIME
>    then (after 2 full sequence wraps) connection 2 is gets alive and
>    delivers c4,c5 etc (that are now valid)

c4, c5, etc were already delivered to the target and are not being
re-delivered. There is no problem in this case. (??).

Take the next scenario :
> 2 connections:
> 
> connection1  I->T c3,c4,c5
>       status of 3 contains ack up to 6 and it and all other statuses are
> lost
> connection2  resend c3, c4 & c5 (no logout) and those are executed!

Since the initiator got CmdSN ack's upto 6, the initiator should not be
re-issuing these I/Os ??

I still don't see justification to require that initiators send a
immediate NOP-OUT in the manner being advocated.

On a more fundamental note, I see some issues with the initiator being
allowed to re-issue the commands on a different connection without
having first logged out the previous connection successfully. I see
nothing in the draft that suggests such behaviour, while at the same
time, it is not forbidden.

By resorting to command retries on a different connection in an attempt
to plug the hole, without first logging out the previous connection, the
initiator is susceptible to encountering I/O failure of that I/O due to
ULP timeout.

Here's the scenario why such recovery should not be allowed :
- Initiator sends CmdSN 3 on connection 1.
- No CmdSN updates for a while and initiator re-sends CmdSn 3 on
connection 2.
- At the same time, target has sent CmdSN ack's for CmdSN 3 on
connection 1.

- Initiator has transferred the command allegiance on its side from
connection 1 to connection 2 and is attempting the command on connection
2. However, the command does not go through, since the (ExpCmdSN,
MaxCmdSN) window has advanced and the trget discards the command.

- Target sends in data and/or R2T and/or status for CmdSN 3 on
connection 1. Since the initiator is not expecting any traffic for that
I/O on connection 1, it discards any PDUs received on that connection 1
for which no I/O state existed.

In the above scenario, initiator will never get a CmdSN ack on
connection 2 and will never be able to plug the hole despite repeated
retries, finally, causing a ULP timeout, followed by session recovery.

Given the above scenario, I suggest that the initiator must only
re-issue commands on the same connection, and can re-issue them on
another connection only following a successful logout.

Comments ?

Thanks,
Santosh

Julian Satran wrote:
> 
> Santosh,
> 
> The scenarios I am talking about are all derivatives of an initiator 
trying
> to plug-in holes and switching connections.
> As the initiator does know the "extent" of a hole it can send-out 
commands
> that he did not have to.
> I have sent the attached not to Mallikarjun  a while ago.  I think that
> there might be many of this kind.  I am also aware that X bit by itself
> might have some bad scenarios but the new proposal fixes them all.
> 
> Julo
> 
> _____________________________
> 
> Mallikarjun,
> 
> Take the following sequence scenario:
> 
>    session has 3 connections
>    on connection 1 I->T c1,c2,c3,C6
>    on connection 2 I->T c4,c5,c7,c8
>    Target receives 1,2,4,5,7,8 (miss 3 and 6) and acks 1 & 2
>    Initiator closes 1 and resends c3, c4, c5,c7,c8  on connection 2 and 
6
>    on connection 3
>    target receives all and starts executing and acks 8 on connection 3 
but
>    connection 2 stalls after c3 for a  LONG TIME
>    then (after 2 full sequence wraps) connection 2 is gets alive and
>    delivers c4,c5 etc (that are now valid)
> 
> That is not a very likely scenario, I admit, but it is possible.
> With X bit I could not find any such scenario since an X either follows 
a
> good one on the same connection or can be safely discarded.
> I suspect that there are some more scenarios that involve immediate
> commands or commands that carry their own ack in the status and are 
acked
> like:
> 
> 2 connections:
> 
> connection1  I->T c3,c4,c5
>       status of 3 contains ack up to 6 and it and all other statuses are
> lost
> connection2  resend c3, c4 & c5 (no logout) and those are executed!
> 
> I think we can avoid those be requiring a NOP exchange before reissuing 
a
> command on a new connection or reissue the command with a task 
management
> (that has an implied ordering) but why do it if X is an obvious and safe
> solution.
> 
> Julo
> 
> Regards,
> Julo
> 
> 
>                     "Mallikarjun
>                     C."                  To:     Julian 
Satran/Haifa/IBM@IBMIL
>                     <[email-masked]       cc:
>                     om>                  Subject:     Re: iscsi : X bit 
in SCSI Command PDU.
> 
>                     08-10-01 21:45
>                     Please respond
>                     to cbm
> 
> 
> 
> Julian,
> 
> We currently have the following specified in section 2.2.2.1 -
> 
> "The target MUST NOT transmit a MaxCmdSN that is more than
> 2**31 - 1 above the last ExpCmdSN."
> 
> It appears to me that the above is sufficient to ward off the
> accidents of the sort you describe.  Do you think otherwise?
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668 Hewlett-Packard, Roseville.
> [email-masked]
> 
> Julian Satran wrote:
> >
> > Mallikarjun,
> >
> > There is at least one theoretical scenario in which an "old" command
> > may appear in a "new window" and be reinstantiated.
> > At 10Gbs and several connection that does not take months. With X the
> > probability is far lower (not 0).   I have no other strong arguments
> > but I am still  thinking.  Matt Wakeley that insisted on it (against
> > me) had some other argument that I am trying to find (I am note
> > remembering).
> >
> > Julo
> >
> >   "Mallikarjun C."
> >   <[email-masked]>                  To:        Julian
> >                              Satran/Haifa/IBM@IBMIL
> >   08-10-01 20:39                     cc:
> >   Please respond to cbm              Subject:        Re: iscsi : X
> >                              bit in SCSI Command PDU.
> >
> >
> >
> > Julian,
> >
> > Now that you put me on the spot, :-), my response -
> >
> > Santosh argued with me privately that X-bit no longer serves a
> > useful purpose after the advent of task management commands to
> > reassign.  My response was that it never was a requirement per se,
> > but always a "courtesy" extended by the initiator to help the
> > target.  I also suggested that X-bit may be considered for its
> > usefulness in debugging.
> >
> > He still had some (very reasonable) comments for simplification
> > - the most appealing of which (to me) was the opportunity to do
> > away with the X-bit checking for *every* command PDU that the target
> > has to endure now.
> >
> > If I missed a legitimate use of X-bit, please comment. Do you
> > think it is a protocol requirement per se?  I couldn't justify
> > to myself so far (except the Login).
> >
> > Regards.
> > --
> > Mallikarjun
> >
> > Mallikarjun Chadalapaka
> > Networked Storage Architecture
> > Network Storage Solutions Organization
> > MS 5668 Hewlett-Packard, Roseville.
> > [email-masked]
> >
> >
> >
> > Julian Satran wrote:
> > >
> > > Santosh,
> > >
> > > I am not sure you went through all scenarios. A conversation with
> > your
> > > colleague - Mallikarjun - and getting through the state table may go
> > a
> > > long way to clarify the need for X.
> > >
> > > And I am sure that by now you found yourself several .
> > >
> > > Julo
> > >
> > >   Santosh Rao
> > >   <[email-masked]>                   To:        IPS Reflector
> > >   Sent by: [email-masked]  <[email-masked]>
> > >                                           cc:
> > >   06-10-01 01:56                          Subject:        iscsi : X
> > >   Please respond to Santosh Rao   bit in SCSI Command PDU.
> > >
> > >
> > >
> > > All,
> > >
> > > With the elimination of command relay from iscsi [in the interests
> > of
> > > simplification ?], I believe that the X bit in the SCSI Command PDU
> > > can
> > > also be removed. As it exists today, the X bit is only being used
> > for
> > > command restart, which is at attempt by the initiator to plug a
> > > potential hole in the CmdSN sequence at the target. It does this on
> > > failing to get an ExpCmdSN ack for a previously sent command within
> > > some
> > > timeout period.
> > >
> > > Given the above usage of command restart, no X bit is required to be
> > > set
> > > in the SCSI Command PDU when command re-start is done.
> > >
> > > Either :
> > > (a) the target had dropped the command earlier due to a digest
> > error,
> > > in
> > > which case, the command restart plugs the CmdSN hole in the target.
> > >
> > > [OR]
> > >
> > > (b) the target had received the command and was working on it, when
> > > the
> > > initiator timed out too soon and attempted a command restart to plug
> > > [what it thought was] a possible hole in the CmdSN sequence.
> > >
> > > In case (a), no X bit was required, since the target knows nothing
> > of
> > > the original command. In case (b), no X bit is required again, since
> > > the
> > > (ExpCmdSN, MaxCmdSN) window would have advanced and the target can
> > > silently discard the received retry and continue working on the
> > > original
> > > command received.
> > >
> > > Removal of the X bit in the SCSI Command PDU has the following
> > > benefits
> > > :
> > >
> > > a) The CmdSN rules at the target are simplified. No need to look at
> > X
> > > bit, only validate received CmdSN with (ExpCmdSN, MaxCmdSN) window.
> > >
> > > b) The reject reason code "command already in progress" can be
> > > removed.
> > > There's no need for this reject reason code anymore, since X bit
> > > itself
> > > is not required, and the targets can silently discard commands
> > outside
> > > the command window and continue to work on the original instance of
> > > the
> > > command already being processed at the target.
> > >
> > > c) Less work for the target and less resources consumed since it no
> > > longer needs to generate a Reject PDU of type "command in progress".
> > > It
> > > can just silently discard any command PDU outside the (ExpCmdSN,
> > > MaxCmdSN) window.
> > >
> > > d) Less code for the target, since it does not need :
> > > - any Reject code paths when it receives X bit command PDUs that are
> > > already in progress.
> > > - No special casing of CmdSN checking rules.
> > > - No overheads of verifying a received command based on its
> > initiator
> > > task tag, to check if the task is currently active, prior to sending
> > a
> > > Reject response with "command in progress".
> > >
> > > Comments ?
> > >
> > > Thanks,
> > > Santosh
> > >
> > > --
> > > ##################################
> > > Santosh Rao
> > > Software Design Engineer,
> > > HP-UX iSCSI Driver Team,
> > > Hewlett Packard, Cupertino.
> > > email : [email-masked]
> > > Phone : 408-447-3751
> > > ##################################
> 
> 
>                     Santosh Rao
>                     <santoshr@cup.       To:     IPS Reflector 
<[email-masked]>
>                     hp.com>              cc:
>                     Sent by:             Subject:     Re: iSCSI - Change 
Proposal X bit
>                     owner-ips@ece.
>                     cmu.edu
> 
> 
>                     23-10-01 22:50
>                     Please respond
>                     to Santosh Rao
> 
> 
> 
> Julian Satran wrote:
> >
> > However in order to drop "old" commands that might in the pipe on a
> > sluggish connection - removing the X bit will require the initiator to
> > issue an immediate NOP requiring a NOP response on every open 
connection
> > whenever CmdSN wraps around (becomes equal to InitCmdSN).
> 
> Julian,
> 
> Can you please explain further the corner case you are describing above
> ? Are you suggesting that special action should be taken every time
> CmdSN wraps around, in case there were holes in the CmdSN sequence at
> the wrap time ? Why is that ?
> 
> Here's my understanding of how this plays out :
> 
> Rule 1)
> The CmdSN management rules at the target should be handling CmdSN wrap
> case and the initiator cannot issue more than 2^32 -1 commands beyond
> the last ExpCmdSN update it has received from the target, since the
> target MUST NOT transmit a MaxCmdSN that is more than 2**31 - 1 above
> the last ExpCmdSN. (per Section 2.2.2.1)
> 
> Rule 2)
> Any holes that occur in the CmdSN sequence are attempted to be plugged
> by the initiator by re-issuing the original command. If the CmdSN never
> got acknowledged and the I/O's ULP timeout expired, the initiator MUST
> perform session recovery. (per Section 8.6)
> 
> Thus, going by the above 2 rules, if the CmdSN sequence wraps upto
> ExpCmdSN, the initiator will not be able to issue further commands,
> since the target will keep the CmdSN window closed. The window can only
> re-open when the CmdSN holes are plugged allowing ExpCmdSN and thereby,
> MaxCmdSN to advance.  (rule 1 above).
> 
> Under the above circumstances, the initiator will possibly try to plug
> the CmdSN hole by re-issuing the original command. It may do this 1 or
> more times before its ULP timeout expires. Either the holes get plugged
> and the windoe re-opens, or ULP timeout occurs without the corresponding
> CmdSN for that I/O having been acknowledged, resulting in session
> logout. (rule 2 above).
> 
> What is required over and beyond the above ? Why does removal of X-bit
> require an immediate NOP to be issued every time CmdSN wraps and a hole
> exists in the CmdSN sequence (??).
> 
> Regards,
> Santosh
> 
> --
> ##################################
> Santosh Rao
> Software Design Engineer,
> HP-UX iSCSI Driver Team,
> Hewlett Packard, Cupertino.
> email : [email-masked]
> Phone : 408-447-3751
> ##################################

-- 
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 25 Oct 2001 10:04:28 +0300

Santosh,

Some comments in text.  I suggest you discuss the scenarios with 
Mallikarjun or if you can't do that for some reason
please call me. We can't go on with this ad nauseam on the list.

Julo

Santosh Rao <[email-masked]>
Sent by: [email-masked]
24-10-01 18:29
Please respond to Santosh Rao

Julian,

6
but

+++ to understand fully the scenario make a timeline and observe taht 
after 3 and 6 where received 4-8 can be executed and their answers sent 
(connection 2 has not died). But c3 and the rest reach the pipe (depending 
on implementation this may happen after statuses have reached the 
initiator and TCP acks have been sent) and then the pipe stalls with the 
duplicates in the pipe without the target knowing about them+++

+++ the problem appears WHEN THE INITIATOR REISSUES THE COMANDS ON THE 
SAME CONNECTION +++

+++ That is not what the first scenario says. The second is trickier and 
involves the fact

+++ That is what the current text says explicitly.  Scenario 2 has not 
been properly stated. +++
Comments ?

Thanks,
Santosh

trying
commands
6
but
a
acked
a
management
Satran/Haifa/IBM@IBMIL
in SCSI Command PDU.
<[email-masked]>
Proposal X bit
connection

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 26 Oct 2001 15:50:10 -0700

Here's a private email I sent to Julian on this topic, FYI.
-- 
Mallikarjun 

Mallikarjun Chadalapaka

Subject: Re: iSCSI - Change Proposal X bit
From: "Mallikarjun C." <[email-masked]>
To: Julian Satran <[email-masked]>
Date: Thu, 25 Oct 2001 12:50:51 -0700

Julian,

We need to call out that all (X-bit) retries should
go out on the same connection, and I will make that 
explicit in the error recovery section.  In effect,
the "connection allegiance" should exist once the 
first copy of a command is sent.

In helping others walk through your scenario here in HP,
we all realized that the core solution to the problem 
is to mandate a complete execution of a numbered command 
(some I/O activity that is trackable) on each connection 
once at least every (2^32 -1) commands.  It may be a NOP, 
or any regular command.  So, here is my suggestion -

  An initiator MUST send at least one non-immediate command 
  on each of the connections participating in the session, 
  for every (2^31 -1) numbered commands.

Comments?
-- 
Mallikarjun 

Mallikarjun Chadalapaka

Julian Satran wrote:
> 
> Santosh,
> 
> The draft DOES NOT ALLOW commands to be reissued on another connection
> without an intervening logout.
> 
> Julo
> 
> 25-10-01 01:44
> The one scenario where I do see stale CmdSNs creating a problem is the
> following :
> 
> - 2 connections.
> - CmdSN 3 issued on connection 1.
> - Initiator does not get CmdSN ack for 1 and re-issues CmdSN 1 on
> connection 2.
> - CmdSN ack for 1 received on connection 2. All further I/O traffic on
> connection 2 only (nothing on connection 1) and CmdSN sequence wraps
> around.
> - Finally, connection 1 clears up and CmdSN 1 is delivered on connection
> 1.
> 
> In the above situation, the stale CmdSN 1 is a problem. However, the
> problem only occurs if the draft allows previously issued commands to be
> re-issued on a different connection without first logging out the
> previous connection.
> 
> Why is it not possible to disallow commands to be re-issued on a
> different connection unless the original connection used for that
> command was first logged out successfully ?
> 
> This would avoid this stale CmdSN on wrap around problem, as well as
> avoid sporadic ULP timeout of I/Os due to race condition b/n initiator
> re-issuing command on another connection and target sending CmdSN update
> on the first connection.
> 
> Thanks,
> Santosh
> 
> ps : I still don't see how your scenario below holds good. When the
> target ack's CmdSNs upto 8 on connection 3, it has received all CmdSNs.
> Hence, there can be no stalled commands on connection 2. However, a
> different scenario which does cause a problem is the one I describe
> above.
> 
> Julian Satran wrote:
> >
> > Santosh,
> >

---

From: "Julian Satran" <[email-masked]>
Date: Sat, 27 Oct 2001 12:12:37 +0300

Mallikarjun,

It is not clear to me what your response to the change proposal is. 
Restricting the command retries only on same connection is not strictly 
necessary.
If we explicitly say that sending it on another connection can be done 
only after a logout (and I think we say that) than
we could get rid of the task management reassign.

As for executing a non-immediate command in 2^31-1 that could be an 
acceptable cleanup solution.

Julo

"Mallikarjun C." <[email-masked]>
Sent by: [email-masked]
25-10-01 21:50
Please respond to cbm

Julian,

Comments?
-- 
Mallikarjun 

Mallikarjun Chadalapaka

Julian Satran wrote:
> > > > > colleague - Mallikarjun - and getting through the state table 
may
> go
> > > > >   06-10-01 01:56                          Subject:        iscsi 
:
> > > > > Given the above usage of command restart, no X bit is required 
to
> be
> > > > sluggish connection - removing the X bit will require the 
initiator
> to

---

From: "Mallikarjun C." <[email-masked]>
Date: Sat, 27 Oct 2001 13:14:31 -0700

Julian,

Perhaps I wasn't clear, but what I meant by "command retries" is what iSCSI
allows to plug holes (duplicate copies, currently indicated by X-bit).  I
agree
that a successful Logout should enable the reassignment to a different
connection,
so also would any Rejects of *command* PDUs from the target on a given
connection (the latter indicates that the task is not instantiated on the
target,
as on an immediate command for ex.).

I IRC, one of the reasons we came up with "task reassign" TMF was to ensure
that only the ITT is relied on for allegiance reassignment - so one does not
have
to either fill in the (old) CmdSN/flags/CDB/LUN etc., or reconfirm them on
the
target.   I wasn't sure if you meant to continue that idea.  One possibility
is to use
the X-bit as the "reassign" bit now - as a protocol by both the parties that
only
the ITT is valid and the rest can be ignored if the reassign-bit is set.
Other
obvious possibility is ofcourse to stay with the task reassign TMF (which we
mandate as immediate today).  Please comment.

Thanks for the agreement on the suggestion to require any non-immediate
command
every (2^31 -1) commands, in stead of only NOPs.

Regards.
--
Mallikarjun

Mallikarjun Chadalapaka