Subject: Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery

From: Santosh Rao <[email-masked]>
Date: Thu, 03 May 2001 17:39:15 -0700

Julian,

This is a problem that spans transports. FCP cannot deploy the ABTS-LS
abort protocol on a ULP timeout if the ABTS cannot be sent because the
target port is not responding with BB_Credit.

I don't believe that's reason enough to not extend "connection
allegiance" to include Abort Task. Sending an Abort Task on the same
connection as the command simplifies a number of issues related to the
task cleanup.

Regards,
Santosh

[email-masked] wrote:
> 
> Forcing abort task on the same connection severely limits the ability of
> sending an abort when the TCP windows is or gets closed.  And that is true
> for one or several adapters.
> 
> Julo
> 
> Santosh Rao <[email-masked]> on 28/04/2001 23:51:00
> 
> Please respond to Santosh Rao <[email-masked]>
> 
> To:   Julian Satran/Haifa/IBM@IBMIL
> cc:   [email-masked]
> Subject:  Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error recovery
> 
> Julian,
> 
> Thanks for the clarification. One basic question. Why don't we want to
> force Abort Task on the same connection as the command ? Extending
> connection allegiance to include the Abort Task would simplify the
> solutions to some of these issues and IMHO, it is not too stringent a
> requirement that the Abort Task be sent on the same connection.
> 
> After all, a portion of the I/O state would be resident in adapter
> firmware/hardware and sending Abort Task on the same connection would ease
> the abort process at the target by avoiding communicating across
> connections [NICs] for the clean up.
> 
> Thanks,
> Santosh
> 
> > We could add the RefCmdSN and it may help plug-in the hole but unless the
> > Abort is ussued on the same connection as the original command we can't
> be
> > sure that the old-one will not pop-up (as we enable ExpCmdSN to move on
> we
> > don't have even the 2**31-1 protection bracket :-)). Thus sending in
> > another nop/abort on the same connection is still required.
> >
> > To simplify the whole process I will:
> >
> >  a - add RefCmdSN to the Task Management
> >  b - add a command not received yet to the answers
> >
> >  c - add a part to 7 reading:
> >
> > 1.1  How to Abort Safely a Command that Was Not Received
> >
> >    To abort safely a task for which the task abort answer is "Command Not
> >    Received Yet" the initiator must issue another abort command on the
> same
> >    connection as the original command unless this connection was logged
> out
> >    in which case it may send it on any connection. The expected response
> >    for the second abort is Function Complete (if the command did not
> >    arrive) or "Task was not in task set".
> >
> >
> >    This convoluted scheme is necessary as the target does not know to
> what
> >    connection the hole is related and we don't want to force abort task
> to
> >    the same connection as the original command.
> >
> >
> >    Julo
> >
> > Santosh Rao <[email-masked]> on 27/04/2001 20:46:53
> >
> > Please respond to Santosh Rao <[email-masked]>
> >
> > To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
> > cc:   [email-masked]
> > Subject:  Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error
> recovery
> >
> >
> >
> >
> > Julian,
> >
> > The conclusion on this thread was that some text was to be added to the
> > spec to address this issue. The rev 06 does not have any text to this
> > effect. It would help to explicitly describe how initiators should plug
> > the hole in CmdSN when they do not intend to use "command retry".
> >
> > Also, regarding the use of NOP-OUT to fill the hole, why not just use
> > Abort Task for the same purpose ? Do we need a 2nd outbound PDU from the
> > initiator just to fill the hole ? When the initiator encounters a ULP
> > timeout, it would use an Abort Task to error the I/O. If the Abort Task
> > can contain the CmdSN of the original command being aborted, targets can
> > fill the hole based on that information, without requiring a second
> > outbound PDU from the initiator for this purpose.
> >
> > Some text in the draft on this subject would be helpful to implementors.
> >
> > - Santosh
> >
> >
> > [email-masked] wrote:
> > >
> > > Santosh,
> > >
> > > You had a possible answer from Matt.  However I agree that we might
> want
> > > to address this in text
> >
> >
> >
> > [email-masked] wrote:
> > >
> > > I the hole is in the command queue and the task is just aborted the
> > > response to the abort task
> > > will unveil the fact that it did not reach destination.
> > >
> > > Initiator can recover from there in several ways - clear the task set,
> > fill
> > > the hole with an iSCSI noop etc.
> > > The latter, I recall, Was sugested to you by Matt Wakeley a while ago.
> > >
> > > None of them require any changes in the spec.
> > >
> > > Julo
> > >
> > > Santosh Rao <[email-masked]> on 27/04/2001 04:56:26
> > >
> > > Please respond to Santosh Rao <[email-masked]>
> > >
> > > To:   Julian Satran/Haifa/IBM@IBMIL
> > > cc:   [email-masked]
> > > Subject:  Re: iSCSI : Bridging missing CmdSNs and Abort I/O Error
> > recovery
> > >
> > > Julian,
> > >
> > > Could you please clarify if the below issue is going to be addressed in
> > > the iSCSI draft, as was discussed earlier.
> > > (http://ips.pdl.cs.cmu.edu/mail/msg03155.html).
> > >
> > > Specifically, is the spec going to address the issue of how initiators
> > > can plug a hole in CmdSN sequence when they detect a ULP timeout and/or
> > > choose not to use "command retry".
> > >
> > > Regards,
> > > Santosh
> > >
> > > [email-masked] wrote:
> > > >
> > > > Santosh,
> > > >
> > > > You had a possible answer from Matt.  However I agree that we might
> > want
> > > to
> > > > address this in text although
> > > > a solution similar to that suggested by Matt should be by now obvious
> > to
> > > > every implementer - the target should leave a placeholder in the
> input
> > > > queue until the command after gets delivered.
> > > >
> > > > Julo
> > > >
> > > > Santosh Rao <[email-masked]> on 25/01/2001 21:38:04
> > > >
> > > > Please respond to Santosh Rao <[email-masked]>
> > > >
> > > > To:   IPS Reflector <[email-masked]>
> > > > cc:
> > > > Subject:  iSCSI : Bridging missing CmdSNs and Abort I/O Error
> recovery
> > > >
> > > > Julian & All,
> > > >
> > > > The draft is currently lacking a section that addresses abort I/O
> error
> > > > recovery. Specifically, how is CmdSN bridging issues to be handled in
> > > > the case where an initiator chooses not to retry an I/O [that failed
> on
> > > > a connection failure that affects the delivery of the command to the
> > > > target or a digest error at the target] because its ULP timer may
> have
> > > > expired.
> > > >
> > > > In such cases, the initiator can send an Abort Task to inform the
> > target
> > > > that the I.T.T is being aborted and its corresponding CmdSN can be
> > > > bridged, instead of having the target stall infinitely in its attempt
> > to
> > > > enforce ordering and await the missing CmdSN [which is'nt going to
> > > > arrive, because the initiator did not retry the command].
> > > >
> > > > Regards,
> > > > Santosh
> > > >
> > > >  - santoshr.vcf
> > >  - santoshr.vcf
> >  - santoshr.vcf
> >
> >
> >
> >
> 
> --
> #################################
> Santosh Rao
> Software Design Engineer,
> HP, Cupertino.
> email : [email-masked]
> Phone : 408-447-3751
> #################################

---

From: Santosh Rao <[email-masked]>
Date: Wed, 09 May 2001 09:37:41 -0700

> > >    in which case it may send it on any connection. 

Julian,

The above may not be sufficient, since a requirement for the Abort Task
completion is that both the initiator & target can safely re-use their
task tags and other resources following completion of the Abort Task. If
the Abort Task is sent on another connection than the connection used
for the original command [& that connection has not been logged out],
then, there may be some in-flight PDUs from the target to the initiator
for that command on the connection.

Sending an Abort Task on another connection may cause the initiator to
receive the Abort Task response indicating success on the other
connection which then results in initiator freeing up its task tag and
other resources used for that command. Thereafter, the stale PDUs on the
original connection can show up at the initiator, causing problems.

I would suggest that Abort Task be always sent on the same connection as
the command, unless the connection used for that command has been logged
out. Not doing so implies the initiator cannot safely re-use its task
tag and other resources that were tied up for that I/O.

Regards,
Santosh