Subject: RE: iSCSI:flow control, acknowledgement, and a deterministic reco very

From: [email-masked]
Date: Sun, 8 Apr 2001 09:50:17 +0200

The main reason for selecting 0 and not a flag for immediate delivery was
to enable immediate delivery even when the command window is closed.

However we can achieve the same effect by using an immediate flag and
ausing the current CmdSN without advancing it.  With this we get a
reference to where in the stream the command where supposed to act if the
stream order is important. All commands that reached the target having
CmdSN less than the immediate CmdSN where sent before our command and all
those with a number equal or higher where sent after. Immediate commands
are the only ones that can have a CmdSN higher (by 1) that the allowed
window.

Julo

---

From: "Douglas Otis" <[email-masked]>
Date: Sun, 8 Apr 2001 10:10:45 -0700

Julian,

If you do not reject commands pending within the sequencer, then you will
need to log these out of sequence events to ensure the command taken out of
sequence will not cause a hole later.  You could allow an exception for
window size but if you are attempting to issue many of these urgent
commands, then a single exception does not provide much flexibility.

Should you reject all prior commands within the sequencer, there is no issue
regarding the logging of these out of sequence commands.  Should there be a
sequence of urgent commands, by flagging the first such command forces the
window to be cleared to allow these urgent commands their immediate
deployment.

This rejection technique also has the advantage of deleting command holes
without timeouts should there be a problem with a connection.  This
technique supplemented with SNACK appears to provide a rather immediate
recovery again without reliance on the SCSI layer.

As there should already be code ready to handle rejected commands, using
this mechanism does not add code and allows the transport to use this very
simple mechanism.  This also does not depend on the target within the SCSI
layer to handle these out of sequence events so there is less likelihood of
needing the SCSI layer tailored to use iSCSI.

While placing this immediate function into a flag ensures that such a
command is acknowledged eventually if rejection is not used and immediately
if commands are rejected, this flag however provides the vital information
of command placement to prevent such a command from applying after
subsequent commands for without this flag there is no assurance when such a
command is received.

Doug

---

From: "Douglas Otis" <[email-masked]>
Date: Sun, 8 Apr 2001 13:40:55 -0700

Following on,

As an overview, there is a change allowing commands trapped within the
sequencer to be rejected for replay at a new sequence.  There would need to
be a means of re-associating the PDU with new CmdSN or use the X-bit to
reissue.  The rejected sequences should become invalidated for only those
rejected due to an immediate command bypass and disallow the X-bit.  To
replay, these sequence associations would require reassignment.

The advantage would be:
  1) No need to score-board any sequencer bypass.
  2) Immediate opening of the window for urgent commands.
  3) Quick repairs for a connection problem.
  4) Immediate acknowledgement for all executed commands.
  5) No need to trap iSCSI errors within SCSI layer.

You would want to invalidate sequences only for commands rejected due to an
immediate command placement.  PDU rejected due to a digest error appears to
invalidate the sequence within the iSCSI proposal.  What is abort task
action, retire SCSI tag?  Is the target making assurances of a free tag?
Targets detecting data digest errors may abort tasks by issuing a Reject PDU
after waiting for all data if it does not support R2T. Is the only method of
control, a SCSI layer abort, and should this be an option?

Notes:
Ver 5 Page 80:

"6.2 Digest Errors

   When a target receives an iSCSI PDU with a header digest error or a
   payload digest error in an iSCSI PDU, it MUST answer with a Reject
   iSCSI PDU with a Reason-code of Header-Digest-error or Data-Digest-
   Error and discard the offending PDU.  If the error is a Data-Digest-
   Error in a Data-PDU, the target MUST either request retransmission
   with a R2T or answer with a Reject iSCSI PDU and abort the task.
   However if the error is detected while data from the initiator is
   still expected (the command PDU did not contain all the data and the
   target has not received a Data PDU with the final bit Set) the target
   MUST wait until it receives the a Data PDU with the F bit set before
   sending the Reject PDU.

   When an initiator receives an iSCSI PDU with a header digest error,
   it MUST discard it.  When an initiator receives any iSCSI PDU other
   than a data PDU, with a Data-Digest-Error, and this PDU is part of a
   task (has an Initiator Task Tag set), it MUST discard the PDU. It MAY
   restart the task (reissue the command with the same Initiator Task
   Tag and the X-bit set to 1).  If the reissued command is a SCSI
   command and it implies Read Data (Expected Data Length is not 0), the
   reissued command also includes the sequence number of the Next Data
   Packet expected by the initiator (0 if there was no data packet yet).

   When an initiator receives an iSCSI data PDU with a Data-Digest
   error, it must discard the PDU and it MUST either request the missing
   data PDUs through SACK or abort the task and terminate the command
   with an error.

6.3 Sequence Errors

   When an initiator receives an iSCSI data PDU with an out-of-order
   DataSN or a SCSI command response PDU with an EndDataSN implying
   missing data PDUs it MAY request the missing data PDUs through a data
   SACK PDU or handle this case as a connection failure.  In its turn,
   the target MUST either reject the SACK with a Reject PDU with a
   reason-code of Data-SACK-Reject or resend the data PDU.

   When an initiator receives an iSCSI status PDU with an out-of-order
   StatSN implying missing responses, it MUST either request the missing
   response PDUs through a status SACK or handle this case as a
   connection failure.  The target MUST reissue the missing responses.
   As a side effect of receiving the missing responses, the initiator
   may discover missing data PDUs. The initiator MUST NOT acknowledge
   (either explicitly through ExpStatRN or implicitly through a status
   SACK) the received responses until it has completed receiving all the
   data PDUs of a SCSI command."

Page 83:

"6.7.1.1 Recovery Within-connection

   At the initiator, the following cases lend themselves to within-
   connection recovery:

      (1)Lost iSCSI numbered Response recognized by either receiving
      it with a data digest error or receiving a Response PDU with a
      higher StatSN than expected. The initiator MAY request the
      missing responses through SACK, in which case the target MUST
      reissue them.
      (2)Requests not acknowledged for a long time. Requests are
      acknowledged explicitly through ExpCmdSN or implicitly by
      receiving data and/or status. The initiator MAY reissue non-
      acknowledged commands. The reissued, non-acknowledged commands
      MUST carry their original CmdSN and the X (retry) flag set to
      1.  Note that this is the only case in which the reissued
      command carries the same CmdSN as the "original".
      N.B. While the original connection for a command is still
      "active" (i.e., has not been logged-out or restarted), any
      command MUST be retried only on the original connection. After
      logging out the original connection, commands can be retried on
      a different connection, but must still carry the original
      CmdSN."

Doug