Subject: Re: iSCSI Requirements Draft - Informal WG Last Call

From: [email-masked]
Date: Mon, 16 Apr 2001 18:31:35 +0300

Ordered delivery of commands to ANY TYPE of devices will increase in
importance as network speeds increase and the need to hide latency
increases.

Today databases don't use queuing and rely and trickle the commands to
devices 1 by 1 to ensure atomicity and order.
As latency will become the determining factor in performance this is bound
to change.

SCSI has done an excellent job in defining the queueing mechanism. We have
to make it work with good performance in our environment.

Julo

Santosh Rao <[email-masked]> on 13/04/2001 04:33:45

Please respond to Santosh Rao <[email-masked]>

To:   [email-masked]
cc:   [email-masked]
Subject:  Re: iSCSI Requirements Draft - Informal WG Last Call

David & All,

I object to the following requirement :

" MUST support ordered delivery of SCSI commands from the initiator to
the
  target, to support SCSI Task Queuing. "

Ordered delivery is not a requirement for disk based applications and
non tagged queueing tape applications, which form the majority of
today's data traffic.

To impose strict ordering (even in the presence of errors ?) as a MUST
is penalizing the majority of today's data traffic that does not expect
ordering from the SCSI subsystem.

I am particularly concerned about the effect of the above requirement in
the presence of errors. Does iSCSI expect strict ordering to be
maintained even when individual I/O errors like ULP timeout occur ?

On a ULP timeout (caused by, say, a hole in CmdSN), the initiator may
choose not to retry the command, but instead, error it back to the ULP.
In such a case, it can plug the hole in CmdSN with a NOP-OUT.

The above requirement is not feasible to be met under such circumstances
and others similar to this. Mandating strict ordering on ULP timeouts
implies a session level error recovery on any individual I/O being
failed back from iSCSI to SCSI ULP. This is a very heavy hammer to use
as error recovery and should not be imposed.

The above requirement must be changed to :
" SHOULD support ordered delivery of SCSI commands from the initiator to
the
  target, to support SCSI Task Queuing. "

- Santosh

[email-masked] wrote:
>
> It is intended to submit draft-ietf-ips-iscsi-reqmts-02.txt
> as an Informational RFC. There is no formal requirement for
> a WG Last Call, but if you have any further substantive comments
> on the document please raise them on this list within the next
> two weeks, i.e. by April 27th at the latest.
>
> If you have typographical/editorial comments please send them
> direct to the document's author, Marjorie Krueger
> <[email-masked]>.
>
> Thanks,
> --David and Elizabeth, IPS WG co-chairs
 - santoshr.vcf

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 16 Apr 2001 14:10:07 -0700

Julian,

I agree that without serialization of commands and then mandating a
sequential delivery scheme, there is no means of ensuring sequential
delivery or allowing atomicity of command groups.  This sequential delivery
also allows rejection of stray commands that may have become trapped within
the network.  A problem of duplicating serialization for iSCSI "immediate"
is that now the receive window can be closed by a "non-immediate" command
carrying the same value.  You will need to adopt connection allegiance for
repeated sequences to avoid a closed window on "immediate" commands.  (It
would also be an improvement if this iSCSI "immediate" term did not overload
the meaning used in SCSI.)

Also, there would appear to be no succession limit imposed on "immediate"
commands.  Santosh will take this freedom to imply all commands can be
treated in this fashion.  If so, terminology that will impose First In First
Out treatment on multiple "immediate" commands that carrying the same
serialization will be needed and connection allegiance seems to provide this
feature.  For succession, perhaps unique serialization re-applies on
subsequent "immediate" commands to allow multiple "immediate" commands to be
serialized across multiple connections and to re-enable iSCSI flow control.
The problem this creates results in apparent skips for the sequencer
tracking only "non-immediate" commands.  (Rejecting commands bypassed is one
method to handle this problem.)  There would not need to be a
"non-immediate" command to cause this increment in the case of no
"non-immediate" commands ever presented.  This implies however two
sequencers are operating.  One for "immediate" and another for
"non-immediate."  In essence, not using "immediate" would be the same as
only using "immediate."  The "Casual" ordering sought by Santosh does not
seem possible within these constraints without an additional flag to
explicitly allow such treatment.  There would then need to be warnings about
SCSI Task Full flow control used in lieu of the iSCSI scheme.  How do you
envision the use of "immediate"?  The proposal does not provide much
clarification about potential limits on successive use.

The ability to flag these commands by the driver is left in doubt by this
proposal and termed out of scope.  Would it be prudent to adopt a scheme
where either Head of Queue or ACA tasks are marked for this treatment?
Depending on the implementation size of the buffers feeding the SCSI layer
on the target side, those commands bypassed can be problematic.  If the
definition of SCSI layer is to imply yet another queue where all commands
are held in sequence regardless of this iSCSI "immediate" flag, then
solutions are limited but then the function of the "immediate" flag is then
severely reduced as well.  If the "immediate" flag is still used within the
SCSI layer, then the definition of SCSI layer is a bit odd and should be
clarified.

You may wish to elaborate what you expect to be done in the case where tasks
to be acted upon are bypassed.  In my opinion, rejecting these commands back
to the initiator is the corrective action.  This rejection scheme also
allows "immediate" commands to carry their own serialization to benefit from
iSCSI flow control remaining valid, connection allegiance not a concern for
the initiator, the command window immediately opening, and "immediate"
commands then receive acknowledgement.  The only provision for rejection
would be that prior "immediate" commands would not be rejected back to the
initiator.

Ver 5-91, Pg 11,
   "If immediate delivery is used with task management commands, these
   commands may reach the SCSI target task management before the tasks
   they are supposed to act upon.  However, their CmdSN is a good
   reference for what commands the immediate command was supposed to act
   upon."

   "The target MUST NOT transmit a MaxCmdSN that is more than 2**31 - 1
   above the last ExpCmdSN.  For non-immediate commands, the CmdSN field
   can take any value from ExpCmdSN to MaxCmdSN. For immediate commands,
   the CmdSN field can take any value from ExpCmdSN to MaxCmdSN+1. The
   target MUST silently ignore any command outside this range or
   duplicates within the range that have not been flagged with the retry
   bit (the X bit in the opcode)."

Doug

---

From: Sandeep Joshi <[email-masked]>
Date: Tue, 17 Apr 2001 10:43:56 -0400

Julian,

I believe you mean "pipelined delivery" as opposed to "ordered
delivery".  The latter is more strict and will introduce pipeline
stalls which we would wish to avoid.  The flow control mechanism
is getting tied into introducing false sequentiality.

As an aside, a colleague here wanted to know if we had considered
having per-LUN ordering as opposed to per-session command ordering.
The pipeline could also slow down because of an aging disk or a 
heavily loaded disk.  I do see some discussions at the Haifa meeting
  http://www.ece.cmu.edu/~ips/meetingMinutes/06.20.2000.txt

Was it just added complexity which forced the per-session choice?

> Point 3.4.3 brings up ordering of commands per LUN or ordering of commands
> per session?
> MW - reorder requirements - second before the first one
> RH - does anybody disagree with 3.4.3?
> CS - No. It basically says that you support SCSI queuing
> JH - In the wide area, a method of pipelining commands and responses
> is required
> CS - the requirement is more complex than saying you just support
> SCSI queing.
> RH - delivering commands in order never hurts
> MT - Why keep order between logical units in commands?
> RH - SAM-2 does not require order between LUNs. However, it may make
> target implemetnation easier.

thanks,
-Sandeep

---

From: Santosh Rao <[email-masked]>
Date: Tue, 17 Apr 2001 12:06:56 -0700

Julian,

Ordered delivery can be achieved to better effect using the SAM-2 CRN
based ordering due to the following reasons :

1) CRN provides ordering on a per-lun basis and can be turned on and off
for a subset of I/Os to that LUN. This allows for flexible ordering
since ordering is a function of the I/O type from the application.
Applications that are doing READ only operations (like a search engine)
do not require any ordering. Ordering is required on metadata updates,
any form of synchronization I/Os, WRITEs interspersed with READS, etc.
Thus, an ordering solution should be flexible enough to be applied at
the scope of a subset of I/Os destined to a LUN.

Such an ordering scheme would also allow ordering to be turned on for
only tape applications if disk applications did not require ordering.

iSCSI's ordering solution does not provide this flexibility, whereas
usage of CRN would.

2) Such a fine granularity scope of ordering also minimizes the impact
of error recovery actions taken when loss of order occurs. The impact
with CRN would be a target-initiator handshake based on ACA + some
checkpoints to error back all the pending CRN enabled commands on that
LUN.

Comparing the equivalent error recovery in a CmdSN based iSCSI ordering
solution, the action taken would be to error back all the pending I/Os
destined to the entire session following loss of order. With high end
disk arrays having 1000+ LUN configurations, such error recovery is
extreme, [especially when ordering may have been desired by the appln
only on a small subset of I/Os to 1 LUN, and loss of ordering for the
remaining 999 LUNs was a don't care].

3) A CRN based ordering scheme works for all underlying SCSI transports
as opposed to CmdSN based ordering.

4) The generation of a stream of commands that expect strong ordering
will need to be accompanied by corresponding generation of a sequence
number at the same layer. (CRN would provide such a sequencing). Failure
to do so can result in silent loss of order that slips un-detected due
to potential points of failure in the stack b/n the SCSI ULP and the
physical bus/link. (ex : I/O failures within the HBA driver due to
resource allocation failures or other such conditions can cause loss of
order.).

Attempts to enforce ordering at multiple layers of the stack (CRN at the
ULP and CmdSN at the LLP), especially when CmdSN does not provide all
the benefits that CRN would provide is over-engineering the solution to
the ordering problem. It also impacts iSCSI performance.

- Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 17 Apr 2001 14:48:03 -0700

Santosh,

CRN does not resolve possibilities of trapped commands beyond the ULP
timeouts.  To resolve the trapped command problem, each command MUST pass
through the sequencer in order.  Stray commands are then identified by a
serialization beyond the current sequence point.  Any alternative scheme
would be complex.  This problem is not related to SCSI, it is an artifact of
IP networks with the potential for greatly delayed delivery.  The example
that I used was a technician temporarily unplugging the cable.

For security reasons, a command rejected for this reason should not be done
silently.  The iSCSI proposal should be changed to indicate that commands
serialized outside the current window will be reported.  The paragraph on
page 11 should be changed with an error code assigned to indicate the
direction of the error, before or beyond the command window.

Ver 5-92 Pg 11

If iSCSI were just a single connection protocol, you would have some basis
for your complaint.  The topic still not addressed by the "immediate"
command designation is with respect to limits placed on the number of
successive "immediate" commands to ensure enough resources are held in
reserve.

Doug