Subject: iSCSI: Target increment CmdSN on unretryable rejects

From: "Pittman, Joseph" <[email-masked]>
Date: Wed, 1 May 2002 11:37:57 -0700

I am working on an iSCSI target implementation, and I have a
a question about whether the target should increment CmdSN
under certain Reject conditions.

Draft 12 of the iSCSI Specification (draft-ietf-ips-iscsi-12.txt)
says the following:

  6.2 Usage of Reject PDU in Recovery

  ...
  If the task had never been active before the Reject (i.e. the
  Reject is on the command PDU), targets should not send any
  further responses since the command itself is being discarded.
  ...
  The CmdSN of the rejected command PDU MUST NOT be considered
  received by the target, (i.e. a command sequence gap must
  be assumed for the CmdSN)

This tells me that whenever I send a Reject as the only response
to a command, I should not increment CmdSN.

The specification also says the following:

  6.1.1 Usage of Retry

  By resending the same iSCSI command PDU ("retry")...
  an initiator attempts to "plug"... discontinuities in CmdSN
  ordering on the target end...

  When an iSCSI command is retried, the command PDU MUST carry
  the original Initiator Task Tag and the original operational
  attributes (e.g. flags, function names, LUN, CDB, etc.) as
  well as the original CmdSN.

This is the only discussion within the specification about the
steps the initiator takes to plug a command sequence gap.

So, as I understand the combined meaning of these two sections,
when an initiator receives a Reject as the only response to a
command, it must assume that the command was discarded, and it
must 'plug the CmdSN hole' by retrying the rejected request.

Now consider these two situations where a target may send a
Reject:

  1) Initiator issues Text command with a non-0xffffffff value
     for Target Transfer Tag (TTT).  But the target does not
     recognize or has discarded the state for the TTT value.
     So the target sends a Reject/0x9(Invalid PDU Field).

  2) Initiator issues a SNACK/Data to request retransmission
     of a DATA_IN.  But the ErrorRecoveryLevel=0 target does
     not support SNACK, so the target sends a
     Reject/0x3(Data-SNACK Reject), or a Reject/0x5(Cmd not supported)
     or a Reject/0x9(Invalid PDU Field).

In either case, the target is rejecting the command itself, so
according to section 6.1.2, the target should not increment CmdSN.
Instead, it should delay submission of subsequently received
commands with larger within-window CmdSN values, until the
initiator 'plugs the hole'.

But according to my understanding, the initiator can only plug the
hole by retransmitting the same command.  The target will only
reject this command again, so the hole won't really be 'plugged'
after all.

What am I missing here?

A) Is there a class of 'command rejects' for which CmdSN should
   be incremented?  Namely, those commands for which retrying the
   same command doesn't make sense?

B) Or perhaps the specification must allow an initiator to plug
   a CmdSN hole by sending a NOP.

I hope the answer is something along the lines of choice A.
Because, I would like to avoid, for now, the whole issue of
hole-plugging and out-of-order processing of incoming commands,
by taking advantage of

  - MaxConnections=1
  - guaranteed in-order per-connection delivery
  - escalating digest errors to session recovery

Any guidance would be greatly appreciated.
--
Joe Pittman

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 1 May 2002 22:00:52 +0300

comments in text - julo

"Pittman, Joseph" <[email-masked]>
05/01/2002 08:05 PM
Please respond to "Pittman, Joseph"

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     "[email-masked]'" <[email-masked]>
        Subject:        iSCSI: Target increment CmdSN on unretryable rejects

Julian,

  ...

+++ target is not incrementing just "taking note" and sending ExpCmdSN +++

  6.1.1 Usage of Retry

+++ not if it is an intermediate reject - like that of a data packet after 
the command got instantiated
For a new command that is true. +++

What am I missing here?

+++ the interdiction on pluging is to insure that the initiator knows what 
he does and has enough tools to keep the intended order. For the case you 
describe initiator may abort the "hole" (that is allowed) on the same 
connection as specified under task management. Enabling NOPs is not 
needed+++

  - MaxConnections=1

[email-masked]

---

From: "Pittman, Joseph" <[email-masked]>
Date: Wed, 1 May 2002 13:12:11 -0700

Julian,
Thanks for your quick response.  Can you help me by answering
the followup question below?

> +++ the interdiction on pluging is to insure that the initiator
>     knows what he does and has enough tools to keep the intended
>     order.  For the case you describe initiator may abort the
>     "hole" (that is allowed) on the same connection as specified
>     under task management.  Enabling NOPs is not needed +++

I see the following text in Draft12, section 9.6.1:

  For the ABORT TASK function:
    a) if the ITT identifies a valid task...
    b) if the ITT does not identify an existing task but if the
       CmdSN indicated by the RefCmdSN field in the task management
       function request is within the valid CmdSN window (between
       MaxCmdSN and ExpCmdSN), targets must consider the CmdSN
       received and return the "Function complete" response.
    c) if the ITT does not identify an existing task and ...
       outside the valid CmdSN window, ... "Task does not exist"

Is case b) above the piece of text within the specification that
describes 'aborting the hole', or is there some other discussion
of this?

To make sure I am clear on this, can you tell me whether the
following statements are accurate?

Consider this situation:

  - Target's ExpCmdSN=10, MaxCmdSN=19
  - Initiator sends 4 commands, numbered 10-13
  - Target processes command 10, recognizes a situation which
    warrants reject, and which will always warrant a reject
    no matter how many times it is retried (i.e. Text command
    with invalid TTT; SNACK for data when target does not support;
    or ANY Invalid PDU field-type error)

In this situation, my understanding is that the target must
behave as follows:

  - Delay submission of commands 11-13 until the command window
    hole at CmdSN=10 is plugged
  - If the initiator retries command 10, the target will send
    the same reject message again
  - If the initiator sends a non-immediate TaskMgmt ABORT_TASK
    command (CmdSN 14), to abort command 10, the target will NOT
    process command 14 because the hole has still not been filled
  - If the initiator sends an immediate TaskMgmt ABORT_TASK command
    to abort command 10, the target will process this request
    immediately, consider command 10 to have been sent (9.6.1 case b),
    close the hole, and respond to the immediate TaskMgmt command with
    Function Complete.  Then, the target will resume submitting
    the sequenced commands starting with 11 (since the hole at 10
    has been closed).

Is my understanding correct?
Thanks for your continued help.
--
Joe Pittman
[email-masked]

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 2 May 2002 00:00:18 +0300

comments in text - julo

"Pittman, Joseph" <[email-masked]>
Sent by: [email-masked]
05/01/2002 11:12 PM
Please respond to "Pittman, Joseph"

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     "[email-masked]'" <[email-masked]>
        Subject:        Re: iSCSI: Target increment CmdSN on unretryable rejects

Julian,

+++ yes +++

Consider this situation:

+++ yes +++