Subject: RE: iSCSI : Command Ordering Proposal.

From: Charles Monia <[email-masked]>
Date: Thu, 25 Jan 2001 23:16:46 -0800

Hi Santosh:

FWIW: The ordering constraints in SAM were established before the prospect
of deploying SCSI over relatively long latency, very high bandwidth
interconnects was close to a reality. IMHO therefore, the decision as to the
need for ordered delivery ought to be made on the basis of whether or not
that requirement is appropriate for the specific interconnect environment.

Anyhow (and with only a bit of irony), I'm not sure partial enforcement is
worth the trouble. In my opinion, you'd be better off eliminating the
requirement altogether since what's left may not be all that useful.

For one thing, an ordered command acts like a barrier seperating the
execution of queued commands received before and after it.  So, there is an
order dependancy between simple and ordered commands that would be hard, if
not impossible, to enforce with the new ordering rules.  For example, if I
send commands A B |C| D E, where |C| is ordered, I expect that D and E won't
complete before |C|.  If A B D and E arrive followed by |C|, there's no way
to obtain the correct behavior. Incidentally, similar constraints hold true
for "head of queue" tasks.

Also, to the extent that command ordering is useful, command sequence
numbers have been added to the CDB payload.  So devices would still have a
limited way to enforce command sequentiality at the LU level (above the
iSCSI layer).

Another issue is the problem of insuring that an ABORT TASK function is
processed in the correct order relative to the task being terminated. I
believe that can be handled by insuring that the task management function is
sent on the same TCP/IP connection that the task was bound to.  Of course,
there are other task management requests, such as ABORT TASK SET, for which
that approach doesn't work.  But those are edge conditions that may or may
not be worth considering (and, frankly, I don't feel much like arguing the
point).

As far as queue full and the other conditions you mention below, I was under
the impression that a SCSI change was being proposed to address the problems
you describe.  So those exceptions should not be among the reasons for
implementing the change you propose.

Charles
> -----Original Message-----
> From: Santosh Rao [mailto:[email-masked]]
> Sent: Thursday, January 25, 2001 6:15 PM
> To: IPS Reflector
> Subject: iSCSI : Command Ordering Proposal.
> 
> 
> Julian & All,
> 
> Proposal :
> =======
> CmdSN should be used to enforce ordering only when the 
> Ordered Task Tag
> attribute is set in the SCSI Command PDU.
> 
> Reference :
> =========
> Current iSCSI model for enforcing command ordering, based on striping
> commands across multiple TCP connections in an iSCSI Session 
> and using a
> 
> CmdSN to order the commands received out of order. Currently, CmdSN is
> used to enforce ordering for all commands.
> 
> Objective of multi-connection sessions :
> ==============================
> Multiple TCP connections per session were defined in order to
> allow for :
> a) bandwidth aggregation by the creation of a fat pipe between the
> target and initiator.
> b) to optimize against single connection failures.
> c) to allow for iSCSI layer failover mechanisms within a session.
> 
> 
> Problems with the current ordering model :
> ================================
> 1) When one TCP connection is faulty in an iSCSI session and 
> is causing
> command transmission failures, all the commands being sent on other
> connections in the session will stall [due to the CmdSN ordering]
> at the receiving iSCSI layer of  the target, until TCP
> timeouts + re-transmissions for the commands on the faulty
> connection are exhausted and a "link ping" of the faulty
> connection fails, thereby, detecting connection failure and initiating
> connection failure recovery.
> 
> While operating under such conditions, the commands sent on
> other TCP connections within the session will be stalled until the
> initiator commences connection failure recovery and re-sends the
> commands with their original CmdRN on a new connection.
> 
> This can result in a large number of stalled I/Os that can remain
> stalled for the period it takes to detect a faulty TCP connection and
> complete connection failure recovery.
> 
> 2) This strict command ordering introduces a lot of complexity into
> error recovery conditions like connection failures, QUEUE FULL, BUSY &
> CHECK CONDITION handling due to the attempts iSCSI makes to enforce
> strict ordering of command delivery.
> 
> 3) This model depends on the retry of ALL the failed commands on a new
> connection in order to bridge the missing CmdSNs at the target. If a
> subset of the failed commands were not to be retried [due to a retry
> policy that forbids this, such as non-idempotent media, or a
> ULP timeout],
> then, the CmdSNs at the target waiting for the out-of-order 
> commands to
> arrive, need bridging which results in additional complexity.
> 
> Requirements for Ordering :
> =====================
> 1) SCSI ordering is only required on a per-LUN basis.
> 
> 2) Strict SCSI ordering is only required when requested through the
> Ordered Task Tag attribute. For all the other types of tasks, 
> no command
> 
> ordering is required. Strict Command ordering is not enforced in other
> SCSI transport protocols. Command ordering in other SCSI transport
> protocols is violated when a QUEUE FULL, BUSY or CHECK
> CONDITION SCSI Status is returned. Command ordering can also
> be violated when the SCSI transport encounters transport failures
> or resource allocation failures on intermittent commands.
> 
> 3) SAM Section 4.6.2 assumes the service delivery subsystem to provide
> in-order delivery for convinience. "This assumption is only made to
> simplify
> description of behaviour and does NOT constitute a requirement."
> 
> Solution :
> ========
> iSCSI should only enforce strict ordering when requested to 
> do so by the
> 
> SCSI ULP through the Ordered Task Tag attribute.
> 
> Benefits :
> ========
> 1) Optimizes iSCSI performance by avoiding stalls on other TCP
> connections within a session when one connection is faulty.
> 
> 2) Simplifies error recovery on connection failure, digest error,
> CHECK CONDITION, QUEUE FULL and BUSY scsi status as
> well as initiator resource allocation errors.
> 
>  -------------------------------------------------------
> 
> Regards,
> Santosh
>

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 26 Jan 2001 11:19:17 -0800

Charles,

If you wish to use tape backup, you should reconsider a need for maintaining
sequences.  To allow a mixture of repeated command streams, this too is an
area where keeping things in order, even on the same TCP connection is a
challenge.  On a single TCP connection, at what point do you allow a
fail-over?  Or do you simply fail the operation?  As for task management, it
seems once those commands are in play, things have already gone astray.  For
recovery, this interface is highly time sensitive but you could not tell
from the proposal that time even matters.  Life is easier if you assume a
wait forever mode on any connection, but that is not the expectations of
most implementations.

Doug

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 15:16:36 -0800

Charles Monia wrote:

>   For example, if I

Charles,

SAM-2 Section 7 states :

"The rules for task set management apply only after the task is entered
into the task set."

This implies that the enforcement of the ordering of task tags in your above
example is NOT based on when the initiator sent the commands, but on when
these tasks enter the task set at the target.

i.e. The initiator may send A, B, |C|, D, E, where |C| is ordered.

The commands may arrive at the target in the order :
A, E, |C|, B, D in a multi-connection session.

The commands A and E are not subject to any form of ordering.
(being simple tag commands and having no ordered tag commands
ahead in the task set.)

The commands |C|, B & D are subject to the ordering that B & D cannot
be executed until C is first executed. Thereafter, B & D are subject to
simple tag rules.

I believe the confusion lies in whether ordered task tags imply end-to-end
ordering or ordering within the received task set at the target. the former
requires strict ordering on the link or re-ordering at the target, in the
absence
of strict link ordering. The latter does NOT require any link ordering or
re-ordering
at the target, since it only enforces ordering within the received task set.

Comments ?

Regards,
Santosh

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 17:25:00 -0800

I'd also like to point out that all this discussion on ordered task tags is
mostly of academic interest. How many implementations today use Ordered Task Tags
? I have'nt seen this in use in the implementations I've worked with.

Does anybody know of how widely used this ordered task tag feature is ? If it is
a totally un-used feature, we're adding all of this complexity to address
something that is un-used.

Regards,
Santosh