Subject: iSCSI : Command Ordering Proposal.

From: "David Peterson" <[email-masked]>
Date: Fri, 26 Jan 2001 09:19:07 -0600

Howdy Charles,
The CmdRN has been renamed to CmdSN (Command Sequence Number).
The previously reserved field (word 0, byte 2) is now the placeholder for
SAM's CRN.
Dave

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Charles Binford
> Sent: Friday, January 26, 2001 9:10 AM
> To: Ips (E-mail)
> Subject: RE: iSCSI : Command Ordering Proposal.
>
>
> I'm confused on a couple of points:
>
> - Are we talking about 'CmdSN' (a field that doesn't exist unless I missed
> an update since iscsi-03) or 'CmdRN'?
>
> - Yes, CRN was added to SAM, but I see nothing in iSCSI that utilizes it.
> Is this thread suggesting a new field be added?  I certainly don't see how
> the current CmdRN field could fill the roll of SAM's CRN and its current
> roll concurrently (different rules on initial value, when to reset, etc.)
>
> Seems to me like adding a new CRN field for command ordering (if the
> application so chooses) and re-defining the CmdRN field to only
> provide the
> flow control satisfies the requirements without overloading the
> functionality of a single field.
>
>
> Charles Binford
> Blue Spruce Networks
> office/cell: (316) 210-6404
> e-fax: (509) 756-4425
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> [email-masked]
> Sent: Friday, January 26, 2001 3:28 AM
> To: Santosh Rao
> Cc: [email-masked]
> Subject: Re: iSCSI : Command Ordering Proposal.
>
> Santosh Rao <[email-masked]> writes:
>
> > [1  <text/plain; us-ascii (7bit)>]
> > Julian & All,
> >
> > Proposal :
> > =======
> > CmdSN should be used to enforce ordering only when the Ordered Task Tag
> > attribute is set in the SCSI Command PDU.
>
>
> Santosh,
>
> I believe you may be missing a subtlety to the SCSI task queueing model.
> I may also be missing a subtlety of your argument and, if so,
> please excuse
> this e-mail.
>
> Consider three commands sent down a FIFO SCSI transport:
>
> 	1) Ordered Command
> 	2) Simple Command
> 	3) Simple command
>
> The ordered command is received first and then the two simple commands.
> The ordered command is guaranteed to be executed before either SIMPLE
> command.
>
> Relevant text from SAM-2 rev 14, sec 7.5.1
>
> "The [simple] task shall not enter the Enabled state until all
> older Head of
> Queue and older Ordered tasks have ended."
>
> Your proposed change means that CmdSN no longer is able express that
> the Ordered task is "older" that the "simple" task. Thus, an initiator
> might need to stop-and-wait before issuing #2 and #3.
>
> I say "might" because iSCSI now transports SCSI Command Reference
> Numbers (CRNs) in the command PDU. The CRN is a SAM-2 rev. 15 (and
> FCP) construct that tells us which commands are older than others. So,
> in some cases, the CmdSN may no longer be needed at all to tell which
> commands are "older" than others.
>
> Given that we have CRNs, why do we need CmdSNs? Well, CmdSNs provide
> a couple extra features:
> 	- ordering on task management (SAM doesn't defined
>           that CRNs are used with task management)
>         - ordering when application doesn't support CRNs
>        (though couldn't the iSCSI layer fabricate CRNs?)
> 	- flow control of the iSCSI target queue
>
> -Costa
>
>

---

From: [email-masked]
Date: Fri, 26 Jan 2001 19:12:19 +0200

Besides what Costa already stated vs. this will involve maintaining state
per LU - an expensive
thing we have rejected long ago.

Julo

Santosh Rao <[email-masked]> on 26/01/2001 04:15:15

Please respond to Santosh Rao <[email-masked]>

To:   IPS Reflector <[email-masked]>
cc:
Subject:  iSCSI : Command Ordering Proposal.

Julian & All,

Proposal :
=======

Reference :
=========
Current iSCSI model for enforcing command ordering, based on striping
commands across multiple TCP connections in an iSCSI Session and using a

CmdSN to order the commands received out of order. Currently, CmdSN is
used to enforce ordering for all commands.

Objective of multi-connection sessions :
==============================
Multiple TCP connections per session were defined in order to
allow for :
a) bandwidth aggregation by the creation of a fat pipe between the
target and initiator.
b) to optimize against single connection failures.
c) to allow for iSCSI layer failover mechanisms within a session.

Problems with the current ordering model :
================================
1) When one TCP connection is faulty in an iSCSI session and is causing
command transmission failures, all the commands being sent on other
connections in the session will stall [due to the CmdSN ordering]
at the receiving iSCSI layer of  the target, until TCP
timeouts + re-transmissions for the commands on the faulty
connection are exhausted and a "link ping" of the faulty
connection fails, thereby, detecting connection failure and initiating
connection failure recovery.

While operating under such conditions, the commands sent on
other TCP connections within the session will be stalled until the
initiator commences connection failure recovery and re-sends the
commands with their original CmdRN on a new connection.

This can result in a large number of stalled I/Os that can remain
stalled for the period it takes to detect a faulty TCP connection and
complete connection failure recovery.

2) This strict command ordering introduces a lot of complexity into
error recovery conditions like connection failures, QUEUE FULL, BUSY &
CHECK CONDITION handling due to the attempts iSCSI makes to enforce
strict ordering of command delivery.

3) This model depends on the retry of ALL the failed commands on a new
connection in order to bridge the missing CmdSNs at the target. If a
subset of the failed commands were not to be retried [due to a retry
policy that forbids this, such as non-idempotent media, or a
ULP timeout],
then, the CmdSNs at the target waiting for the out-of-order commands to
arrive, need bridging which results in additional complexity.

Requirements for Ordering :
=====================
1) SCSI ordering is only required on a per-LUN basis.

2) Strict SCSI ordering is only required when requested through the
Ordered Task Tag attribute. For all the other types of tasks, no command

ordering is required. Strict Command ordering is not enforced in other
SCSI transport protocols. Command ordering in other SCSI transport
protocols is violated when a QUEUE FULL, BUSY or CHECK
CONDITION SCSI Status is returned. Command ordering can also
be violated when the SCSI transport encounters transport failures
or resource allocation failures on intermittent commands.

3) SAM Section 4.6.2 assumes the service delivery subsystem to provide
in-order delivery for convinience. "This assumption is only made to
simplify
description of behaviour and does NOT constitute a requirement."

Solution :
========
iSCSI should only enforce strict ordering when requested to do so by the

SCSI ULP through the Ordered Task Tag attribute.

Benefits :
========
1) Optimizes iSCSI performance by avoiding stalls on other TCP
connections within a session when one connection is faulty.

2) Simplifies error recovery on connection failure, digest error,
CHECK CONDITION, QUEUE FULL and BUSY scsi status as
well as initiator resource allocation errors.

 -------------------------------------------------------

Regards,
Santosh

 - santoshr.vcf

---

From: Matt Wakeley <[email-masked]>
Date: Fri, 26 Jan 2001 10:20:54 -0800

One feature that command numbering was used for was to avoid the queue full
condition (because the initiator could not send commands past MaxCmdSN).

If you don't care about ordering, then either use a CmdSN of zero, or open
multiple "sessions" each with 1 TCP connection.  Then you could scatter your
commands across all the sessions.

-Matt

---

From: Charles Monia <[email-masked]>
Date: Fri, 26 Jan 2001 19:52:06 -0800

Hi Santosh:

See comments below.

> From: Santosh Rao [mailto:[email-masked]]
> Sent: Friday, January 26, 2001 3:17 PM
> To: Charles Monia
> Cc: IPS Reflector
> Charles Monia wrote:
> >   For example, if I
> > send commands A B |C| D E, where |C| is ordered, I expect 
> that D and E won't
> > complete before |C|.  If A B D and E arrive followed by 
> |C|, there's no way
> > to obtain the correct behavior. Incidentally, similar 
> constraints hold true
> > for "head of queue" tasks.
> >
> 
> Charles,
> 
> SAM-2 Section 7 states :
> 
> "The rules for task set management apply only after the task 
> is entered
> into the task set."
> 
> This implies that the enforcement of the ordering of task 
> tags in your above
> example is NOT based on when the initiator sent the commands, 
> but on when
> these tasks enter the task set at the target.
> 
> i.e. The initiator may send A, B, |C|, D, E, where |C| is ordered.
> 
> The commands may arrive at the target in the order :
> A, E, |C|, B, D in a multi-connection session.
> 
> The commands A and E are not subject to any form of ordering.
> (being simple tag commands and having no ordered tag commands
> ahead in the task set.)
> 
> The commands |C|, B & D are subject to the ordering that B & D cannot
> be executed until C is first executed. Thereafter, B & D are 
> subject to
> simple tag rules.
> 
> I believe the confusion lies in whether ordered task tags 
> imply end-to-end
> ordering or ordering within the received task set at the 
> target. the former
> requires strict ordering on the link or re-ordering at the 
> target, in the
> absence
> of strict link ordering. The latter does NOT require any link 
> ordering or
> re-ordering
> at the target, since it only enforces ordering within the 
> received task set.
> 
> Comments ?

Aside from the fact that the ordering semantics make no sense without this
property, the initiator's real-world expectation is that end-to-end ordering
is preserved. Also, as I mentioned earlier, without end-to-end ordering,
there is a potential for other undesirable side effects due to the
misordering of task management requests relative to commands in flight.

Charles

---

From: Charles Monia <[email-masked]>
Date: Fri, 26 Jan 2001 20:00:38 -0800

> Sent: Friday, January 26, 2001 5:25 PM
> To: IPS Reflector
> I'd also like to point out that all this discussion on 
> ordered task tags is
> mostly of academic interest. How many implementations today 
> use Ordered Task Tags
> ? I have'nt seen this in use in the implementations I've worked with.
> 
> Does anybody know of how widely used this ordered task tag 
> feature is ? If it is
> a totally un-used feature, we're adding all of this 
> complexity to address
> something that is un-used.
> 

The point isn't whether or not ordered task tags are used but whether or not
the ordering property is required.

---

From: [email-masked]
Date: Wed, 31 Jan 2001 08:37:36 +0200

I will spell-out more recovery scenarios in the next draft. Julo

Santosh Rao <[email-masked]> on 31/01/2001 04:11:29

To:   [email-masked]
cc:

[email-masked] wrote:

> Santosh,
>
> The only puzzling question is what kind of digest error would you detect
> for a command that the target has not yet started acting upon?

Julian,

The target has started acting on the command. However, there is nothing in
the standard that requires a ExpCmdSN ACK to accompany the inbound
Data PDUs for a READ command.

In the absence of this, targets are free to send in delayed ExpCmdSN
updates
and can commence transmission of Data PDUs [which are not yet ACKing
the CmdSN for the command in response to which the Data PDU is being
sent.]

Either :
a) An inbound Data PDU should be considered an implicit ack of the CmdSN
which was sent.

Or :
b) When initiators detect digest errors on an inbound Data PDUs
(header or digest), they should send the "retry" with a CmdSN of 0.

>  And even then if the task is recognized you could use a CmdSN of 0

> as the command is clearly active.

The above is sufficient. The draft should state either (a) or (b) in its
policy for initializing CmdSN on "retry" commands.

Regards,
Santosh