Subject: iSCSI: I/O (command) recovery

From: Matt Wakeley <[email-masked]>
Date: Fri, 26 Jan 2001 10:25:39 -0800

There's been some confusion on how iSCSI commands that fail could be recovered
at the iSCSI layer without involving SCSI, so I thought I'd write up (my
understanding of) how this is done.

For a Target Read operation:

1- The initiator sends a read command to the target.

2- The target iSCSI sends the command to the SCSI layer.

3- The SCSI layer performs the read command and hands the read data and status
to the iSCSI layer. iSCSI then allocates resources and stores the read data
and the I/O status in the allocated resources.

4- The target iSCSI sends the read data and status to the initiator, but does
*not* deallocate the resources associated with the I/O.

5- When the initiator iSCSI successfully receives the data and status, it
acknowledges the receipt of the status back to the target iSCSI via the
ExpStatSN.

6- When the target iSCSI receives the ExpStatSN acknowledging the I/O, it then
deallocates the resources for the I/O.

If some error occurs such that the initiator iSCSI does not receive all the
data and status, then the initiator iSCSI performs whatever
link/connection/session recovery to re-establish communication with the
target.  At that point, the initiator iSCSI layer re-issues the I/O with the
retry bit set.

- At this point, the target iSCSI layer (*not* the SCSI layer) recognizes the
command is a duplicate (because the retry bit is set), and instead of sending
the command to the SCSI layer, it simply retransmits the buffered data and
status [continue at #5 above].

For a Target Write operation:

1- The initiator sends a write command to the target.

2- The target iSCSI sends the command to the SCSI layer.

3- The SCSI layer prepares for the write and asks the iSCSI for the data.
iSCSI then allocates resources for the I/O.

4- The target iSCSI sends R2T to the initiator.

5- The initiator sends the write data to the target.

6- The target iSCSI receives the data and sends it to the SCSI layer.

7- The SCSI layer performs the write and gives the status to the iSCSI layer.

8- The target iSCSI sends the status to the initiator (and saves it in the
iSCSI allocated resources for the I/O), but does *not* deallocate the
resources associated with the I/O.

9- When the initiator iSCSI successfully receives the status, it acknowledges
the receipt of the status back to the target iSCSI via the ExpStatSN.

10- When the target iSCSI receives the ExpStatSN acknowledging the I/O, it
then deallocates the resources for the I/O.

If some error occurs such that the initiator iSCSI does not receive the
status, then the initiator iSCSI performs whatever link/connection/session
recovery to re-establish communication with the target.  At that point, the
initiator iSCSI layer re-issues the I/O with the retry bit set.

- At this point, the target iSCSI layer (*not* the SCSI layer) recognizes the
command is a duplicate.

- If the target iSCSI has not received all the data for the R2T, then the
iSCSI layer re-issues the R2T [continue at #5 above].

- If the target iSCSI layer has sent the SCSI status, it simply retransmits
the status [continue at #9 above].

Now, this is just a basic overview on how it could be done.  I'm sure there
are variations on how it could be implemented in the target. Should this
example be put into the iSCSI document?

-Matt Wakeley
Agilent Technologies

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 14:24:30 -0800

Matt,

This is a good write-up and clearly outlines the behaviour. Just a couple of
comments below.

Regards,
Santosh

> *not* deallocate the resources associated with the I/O.

The draft is ambiguous about (4). It states that the numbering MUST be supported,
but the data and status recovery MAY be supported. This means initiators
& targets MUST use StatSN/ExpStatSN. However, the target is allowed to
de-allocate the resources associated with the I/O without waiting for ExpStatSN
ack.

> command is a duplicate (because the retry bit is set), and instead of sending

Again, the wording for the above in the draft could do with some clarity (Section
1.2.2.1). It is not clear from the draft whether CmdSNs that are outside the
(ExpCmdSN, MaxCmdSN) window are to be accepted if the retry bit is set.

My interpretation of :
 "- The target MUST silently ignore any command
     outside this range or duplicates within the range not flagged with
     the retry bit (the X bit in the opcode)."

is :
- ignore all commands outside the (ExpCmdSN, MaxCmdSN) range.
- ignore all duplicate commands within the range that are not flagged with the
   retry

So, if a command with the "retry" bit is set is received outside the (ExpCmdSN,
MaxCmdSN) window, this causes the target to drop the received retry. The wording
on this could do with further clarity.

> resources associated with the I/O.

Same comment as given for (4) under read operation.

---

From: Matt Wakeley <[email-masked]>
Date: Fri, 26 Jan 2001 15:00:44 -0800

> de-allocate the resources associated with the I/O without waiting for ExpStatSN

That's correct.  There is nothing in the document that states that retry must
be supported.  There should probably be a negotiation key that a target uses
to indicate if it supports retry or not.

> (ExpCmdSN, MaxCmdSN) window are to be accepted if the retry bit is set.

Well, in my mind, the retried command will either have a CmdSN of zero, or a
(new/different) valid value within the current range (the next available
CmdSN).  I don't think there should ever be duplicates.

> Same comment as given for (4) under read operation.

Same reply as for (4) under read....

-Matt