Subject: iSCSI : Command Ordering Proposal.

From: Santosh Rao <[email-masked]>
Date: Thu, 25 Jan 2001 18:15:15 -0800

Julian & All,

Proposal :
=======
CmdSN should be used to enforce ordering only when the Ordered Task Tag
attribute is set in the SCSI Command PDU.

Reference :
=========
Current iSCSI model for enforcing command ordering, based on striping
commands across multiple TCP connections in an iSCSI Session and using a

CmdSN to order the commands received out of order. Currently, CmdSN is
used to enforce ordering for all commands.

Objective of multi-connection sessions :
==============================
Multiple TCP connections per session were defined in order to
allow for :
a) bandwidth aggregation by the creation of a fat pipe between the
target and initiator.
b) to optimize against single connection failures.
c) to allow for iSCSI layer failover mechanisms within a session.

Problems with the current ordering model :
================================
1) When one TCP connection is faulty in an iSCSI session and is causing
command transmission failures, all the commands being sent on other
connections in the session will stall [due to the CmdSN ordering]
at the receiving iSCSI layer of  the target, until TCP
timeouts + re-transmissions for the commands on the faulty
connection are exhausted and a "link ping" of the faulty
connection fails, thereby, detecting connection failure and initiating
connection failure recovery.

While operating under such conditions, the commands sent on
other TCP connections within the session will be stalled until the
initiator commences connection failure recovery and re-sends the
commands with their original CmdRN on a new connection.

This can result in a large number of stalled I/Os that can remain
stalled for the period it takes to detect a faulty TCP connection and
complete connection failure recovery.

2) This strict command ordering introduces a lot of complexity into
error recovery conditions like connection failures, QUEUE FULL, BUSY &
CHECK CONDITION handling due to the attempts iSCSI makes to enforce
strict ordering of command delivery.

3) This model depends on the retry of ALL the failed commands on a new
connection in order to bridge the missing CmdSNs at the target. If a
subset of the failed commands were not to be retried [due to a retry
policy that forbids this, such as non-idempotent media, or a
ULP timeout],
then, the CmdSNs at the target waiting for the out-of-order commands to
arrive, need bridging which results in additional complexity.

Requirements for Ordering :
=====================
1) SCSI ordering is only required on a per-LUN basis.

2) Strict SCSI ordering is only required when requested through the
Ordered Task Tag attribute. For all the other types of tasks, no command

ordering is required. Strict Command ordering is not enforced in other
SCSI transport protocols. Command ordering in other SCSI transport
protocols is violated when a QUEUE FULL, BUSY or CHECK
CONDITION SCSI Status is returned. Command ordering can also
be violated when the SCSI transport encounters transport failures
or resource allocation failures on intermittent commands.

3) SAM Section 4.6.2 assumes the service delivery subsystem to provide
in-order delivery for convinience. "This assumption is only made to
simplify
description of behaviour and does NOT constitute a requirement."

Solution :
========
iSCSI should only enforce strict ordering when requested to do so by the

SCSI ULP through the Ordered Task Tag attribute.

Benefits :
========
1) Optimizes iSCSI performance by avoiding stalls on other TCP
connections within a session when one connection is faulty.

2) Simplifies error recovery on connection failure, digest error,
CHECK CONDITION, QUEUE FULL and BUSY scsi status as
well as initiator resource allocation errors.

 -------------------------------------------------------

Regards,
Santosh

---

From: [email-masked]
Date: 26 Jan 2001 01:28:16 -0800

Santosh Rao <[email-masked]> writes:

> [1  <text/plain; us-ascii (7bit)>]
> Julian & All,
> 
> Proposal :
> =======

Santosh,

I believe you may be missing a subtlety to the SCSI task queueing model.
I may also be missing a subtlety of your argument and, if so, please excuse
this e-mail.

Consider three commands sent down a FIFO SCSI transport:

	1) Ordered Command
	2) Simple Command 
	3) Simple command

The ordered command is received first and then the two simple commands.
The ordered command is guaranteed to be executed before either SIMPLE command.

Relevant text from SAM-2 rev 14, sec 7.5.1

"The [simple] task shall not enter the Enabled state until all older Head of Queue and older Ordered tasks have ended."

Your proposed change means that CmdSN no longer is able express that
the Ordered task is "older" that the "simple" task. Thus, an initiator
might need to stop-and-wait before issuing #2 and #3.

I say "might" because iSCSI now transports SCSI Command Reference
Numbers (CRNs) in the command PDU. The CRN is a SAM-2 rev. 15 (and
FCP) construct that tells us which commands are older than others. So,
in some cases, the CmdSN may no longer be needed at all to tell which
commands are "older" than others.

Given that we have CRNs, why do we need CmdSNs? Well, CmdSNs provide
a couple extra features:
	- ordering on task management (SAM doesn't defined
          that CRNs are used with task management)
        - ordering when application doesn't support CRNs
       (though couldn't the iSCSI layer fabricate CRNs?)
	- flow control of the iSCSI target queue

-Costa

---

From: "Charles Binford" <[email-masked]>
Date: Fri, 26 Jan 2001 09:10:15 -0600

I'm confused on a couple of points:

- Are we talking about 'CmdSN' (a field that doesn't exist unless I missed
an update since iscsi-03) or 'CmdRN'?

- Yes, CRN was added to SAM, but I see nothing in iSCSI that utilizes it.
Is this thread suggesting a new field be added?  I certainly don't see how
the current CmdRN field could fill the roll of SAM's CRN and its current
roll concurrently (different rules on initial value, when to reset, etc.)

Seems to me like adding a new CRN field for command ordering (if the
application so chooses) and re-defining the CmdRN field to only provide the
flow control satisfies the requirements without overloading the
functionality of a single field.

Charles Binford
Blue Spruce Networks
office/cell: (316) 210-6404
e-fax: (509) 756-4425

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
[email-masked]
Sent: Friday, January 26, 2001 3:28 AM
To: Santosh Rao
Cc: [email-masked]
Subject: Re: iSCSI : Command Ordering Proposal.

Santosh,

	1) Ordered Command
	2) Simple Command
	3) Simple command

The ordered command is received first and then the two simple commands.
The ordered command is guaranteed to be executed before either SIMPLE
command.

"The [simple] task shall not enter the Enabled state until all older Head of
Queue and older Ordered tasks have ended."

-Costa

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 14:53:26 -0800

> functionality of a single field.

This is exactly my point. Strict ordering should only be enforced when
driven by an application. (either through the use of Ordered Task Tags for
task set ordering or the use of CRN for end-to-end ordering).

The use of CmdSN to order all commands seems an over-kill when those
applications that did need ordering would use the Ordered Task Tag or
CRN, based on their need.

Strict ordering is only required for the aborts that follow their commands.

Also, it would be helpful to re-name the new 1-byte CmdRN in the
new iSCSI PDU to CRN to avoid confusion with the old semantics of
CmdRN. (which is now the CmdSN !!)

Regards,
Santosh

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 15:08:33 -0800

> might need to stop-and-wait before issuing #2 and #3.

Costa,

I need to add some more clarity on the discussion here. It was not my intent
to suggest that CmdSN NOT be used for simple task tags. My suggestion
was that the targets use CmdSN to enforce ordering only when an ordered
task tag enters the task set and for the tasks that are older than the ordered
task tag in the task set.

I see some confusion regarding the semantics of ordered task tag. SAM-2
Section 7 states :
"The rules for task set management (including those that govern the ordered
task tags) apply only after the task is entered into the task set."

This implies that ordering is enforced not from an end-to-end perspective,
but from the perspective of the tasks entering the task set at the target.

When end-to-end ordering is required, CRN should be used. Ordered Task
Tags do not imply end-to-end ordering (unless someone pulled that above
statement out of SAM-2).

Given the above, enforcing ordering of interleaved ordered tags and simple tags
within a task set becomes a 2-queue implementation along the following lines.

Consider 2 queues, unordered_q & ordered_q at the target.

A received task shall be placed at the tail of :

a) unordered_q if there are no pending tasks in the ordered_q and it has a
simple task tag.

b) ordered_q if it has an ordered task tag or (has a simple task tag and there
are pending tasks on the ordered_q).

If all older ordered task tag commands in the ordered_q have been
processed, the contiguous set of simple task tag commands that follow on
the ordered_q shall be processed as per the rules of simple task tag mgmt.

Any Head of Queue Tag or ACA Task Tag command shall be immediately
processed by the SCSI ULP.

This model ensures that a stream of "simple task tag" traffic only (which is the
majority of the cases toay, if not all) will not encounter stalled commands on other
TCP connections when one connection is faulty. It also is in line with the spirit of
the SAM ordering of task tags.

Anyone desiring end-to-end ordering uses CRN. Since I'm not proposing that
CmdSN NOT be initialized for simple task tag commands (only that they not be
used to enforce ordering), they can still be used for flow control that complements
the current QUEUE FULL based SCSI flow control mechanisms.

Regards,
Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 26 Jan 2001 17:34:52 -0800

Santosh,

If there is a task attribute, the complexity is when you allow CmdSN = 0;

Perhaps CmdSN = 0 should not be allowed or treated as just another in the
sequence.

One could view CmdSN as Client Sequence Number and StatSN as Server Sequence
Number where these change to CSN and SSN. :)  This gets away from the
duplexed meaning of the number as defined for CRN.  With the task attribute
and to be sure of the sequence number, there is little gained from the use
of CmdSN = 0 as an exception.  If a retry flag is set, the CmdSN below the
ExpCmdSN should be viewed as an exception much as Head of Queue should be an
exception for MaxCmdSN.  Both of these cases imply exceptional behavior by
the initiator in response to a problem.

Doug

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 17:56:13 -0800

> Perhaps CmdSN = 0 should not be allowed or treated as just another in the

The (CmdSN==0) does have its uses. One example is the use of NOP-OUT for
a "connection ping."

>  If a retry flag is set, the CmdSN below the

Accepting a command outside the (ExpCmdSN, MaxCmdSN) window with the retry
bit set then opens up windows for other types of stale and duplicate commands
to be
processed.

Here's the deal (as I view it) :

Commands are sent with the "retry" bit under 2 conditions :
- Connection Failures
- Digest Errors

In both these cases, an explicit handshake is required regarding the
(ExpCmdSN, MaxCmdSN) window, prior to starting to send "retry"
commands. This ensures that genuine "retry" commands do not slip
behind the (ExpCmdSN, MaxCmdSN) window.

In the case of a connection failure, the Logout Response provides this
handshake. For digest errors, there is no such hand-shake.
Hence, a command that encountered a digest error and was retried could
likely slip behind the (ExpCmdSN, MaxCmdSN) window and then be never
processed.

Regards,
Santosh

---

From: Santosh Rao <[email-masked]>
Date: Fri, 26 Jan 2001 17:42:56 -0800

Santosh Rao wrote:

> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The above should read "when an ordered task tag enters the task set and for simple
task tags that arrive after an ordered task tag in the task set, until the completion of
the ordered task tag."

- Santosh

---

From: Ralph Weber <[email-masked]>
Date: Sat, 27 Jan 2001 09:36:27 -0600

One of the masters from whom I learned programming repeatedly advised,
"Never under estimate the creativity of those who use your programs."

In this case, there are two architected capabilities:

  1) At the client-server level, there's Ordered Tags, and
  2) At the transport level, there's CRN ordering.

True, most applications will want to coordinate the usage of the
two.  In my opinion, it is less likely that applications will
want to disable CRN for Simple tags since I perceive the benefits
of transport ordering as having broader link reliability
implications.

I can more readily agree that use of an Ordered tag strongly
suggests use of CRN.  However, I once helped design an application
where only the Ordered Tag guarantee was needed.  Our cluster
application sought only to obtain the following guarantee from the
target: "When status is returned for this (ordered) command, all
commands received prior to this (ordered) command from all initiators
also have been completed."  By design, our application took care of
all the other ordering details and the our application had to do
that because it was dealing with multiple initiators.

Based on my experience, I'd ask you to ensure that the standards
contain all the tools application clients need but to refrain from
dictating how those tools are used.

Thanks.

Ralph...

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 29 Jan 2001 10:13:40 -0800

Santosh,

Rather than being clever with a special case setting the sequence number to
zero, the limits imposed on the sequence for things like Head of Queue or
NOP could be ignored and not be treated as violations.  This would allow all
such activity between the initiator and the target to remain in sequence.
Allowing this exception to a basic scheme of ensuring sequential
communication has several drawbacks such as lack of acknowledgement.  What
drawback is there in having the target making allowances for some commands
such as Head of Queue or NOP to exceed a maximum?

Doug

---

From: Matt Wakeley <[email-masked]>
Date: Mon, 29 Jan 2001 13:01:22 -0800

Retried commands should have already been queued and/or executed.  The goal of
the retry is simply to replay back to the initiator the results of the
execution. Therefore, there is no ordering requirement for "retried" commands
and CmdSN should always be zero.

-Matt

---

From: Matt Wakeley <[email-masked]>
Date: Mon, 29 Jan 2001 13:05:20 -0800

> task tag in the task set.

Without CmdSN or CRN you could not guarantee what simple tasks get into the
queue before or after the ordered task. CRN is new and in an unfinished
standard (SAM-2 r15).  Therefore, I believe you need CmdSN ordering to handle
"legacy" SCSI stacks that don't know this new feature.

-Matt

---

From: Santosh Rao <[email-masked]>
Date: Mon, 29 Jan 2001 13:38:23 -0800

Matt Wakeley wrote:

> Retried commands should have already been queued and/or executed.

Not necessarily. A TCP connection failure can affect the command PDU.
Similarly, the target may have detected a header or data digest error
on the SCSI Command PDU, sent a REJECT and the initiator would have
"discarded and retried" the task.

In both of the above cases, the command is not queued and/or being executed
at the target.

Such commands, when retried, need to enforce ordering. Hence, the need to
specify CmdSN in the "retry", if the CmdSN is yet to be acknowledged.
The difficulty lies in the establishment of a "sync point" with the ExpCmdSN,
prior to the initiation of these "retry" commands, so as to ensure that the "retry"

will not be discarded by the target, because the CmdSN slid behind the
(ExpCmdSN, MaxCmdSN) window.

The ExpCmdSN in the login response provides this "sync point" during connection
failure recovery. The problem is only with digest error "retries".

If one were to NOT apply the "discard and retry" policy on a header digest error,
[under the assumption that the I.T.T cannot be trusted on a header digest error],
the problem further reduces to the following cases :

a) data digest error detected by a target on a command PDU. (immediate data).
b) data digest error detected by an initiator on inbound Data PDUs of a READ
I/O, and the CmdSN for the command is yet to be acknowledged, but an ACK
may be in-flight behind the Data PDU.

Regards,
Santosh

> The goal of

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 29 Jan 2001 15:23:39 -0800

Matt,

Whether you use the CmdSN or the tag to associate the retry, the CmdSN is
still used as a means to handshake freeing of resources and not the tag.  It
seems like a cleaner solution not to include exceptions on maintaining the
sequence just to allow this type of non-sequential use.  Clearly there is an
alternative that has benefits. Simply have the target allow exceptions based
on the command or task attribute and not drop the ability to handshake.  I
am aware of the present concept, but you did not speak about any downside to
this alternative.

Doug

---

From: Matt Wakeley <[email-masked]>
Date: Mon, 29 Jan 2001 23:04:58 -0800

> Not necessarily. A TCP connection failure can affect the command PDU.

In which case the target will have "silently discarded" the PDU.  Then, it
will not acknowledge via ExpCmdSN any commands past the err'd command, so the
initiator can resend the command with the same CmdSN and enabling command
delivery to SCSI to continue.

> on the SCSI Command PDU, sent a REJECT

I thought any digest errors are silently discarded - therefore, there is no
reject.  Nevertheless, the ExpCmdSN is not incremented (as stated above), and
the initiator can then resend that command with the same CmdSN.

-Matt

> and the initiator would have "discarded and retried" the task.
>

---

From: Matt Wakeley <[email-masked]>
Date: Mon, 29 Jan 2001 23:08:22 -0800

Douglas Otis wrote:
> still used as a means to handshake freeing of resources and not the tag.

You are wrong.  It specifically states in the document in 1.2.2.1:

"CmdRNs are significant only during command delivery to the target.
Once the device serving part of the target SCSI has received a
command, CmdRN ceases to be significant."

Any resources the target allocates must be based on the task tag.

-Matt

>  It

---

From: Santosh Rao <[email-masked]>
Date: Tue, 30 Jan 2001 00:11:42 -0800 (PST)

Doug,

I'm not sure if we are speaking the same issue here. Can you
please elaborate further on your concerns. 

The handshake being discussed by me does NOT have to do with
freeing resources. It is a handshake b/n the target & 
initiator on the CmdSN/ExpCmdSN, so that "retry" commands
from initiator to target remain with the (ExpCmdSN, MaxCmdSN)
window.

Resources at the target end are de-allocated when it receives
an ExpStatSN for the StatSN. (IOW, resource de-allocation has 
to do with StatSN ACKs from the initiator, not CmdSN).
If a target were not to implement status recovery, resource
de-allocation can occur once a TCP ACK is received for the 
SCSI Response PDU. (if a mechanism were to exist to identify
that all the octets that constitute the Status PDU have been
ACK'ed.)

This also brings up an interesting issue which is :

If the target needs to hold onto its status PDU until at least
the TCP ACK is received for all of the bytes that constitute
the PDU and there is no easy mechanism to associate b/n
a TCP ACK and all bytes of the Status PDU being ACK'ed,
the simplest approach becomes de-allocation of status &
I/O resources based on ExpStatSN. 

If that is the case, the spec should change 
"targets MAY implement status recovery"
to "targets MUST hold onto status PDU until ExpStatSN" update
is received.

Regards,
Santosh

-- 
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : [email-masked]
Phone : 408-447-3751
#################################

---

From: Santosh Rao <[email-masked]>
Date: Mon, 29 Jan 2001 23:58:47 -0800 (PST)

Matt,

Comments in-line.

Regards,
Santosh

> delivery to SCSI to continue.

The above was stated to explain why "retry" commands need 
to use CmdSN [and cannot use a CmdSN of 0 as you were 
proposing].

The problem being originally described can occur when the 
initiator applies a "discard & retry" policy on detecting
a digest error at its end. In such a case, it would
"retry" the command, and if the CmdSN was not yet
acknowledged, would send the "retry" with the original
CmdSN.

Since the target continues to send PDUs to the initiator,
it can update the ExpCmdSN causing the "retry" to be 
discarded by the target, because its CmdSN has slid
behind the window.

The spec does not mandate that an update to ExpCmdSN
MUST accompany the Data PDUs for a command. IOW, a target
may lag behind in updates to ExpCmdSN while continuing to 
send Data PDUs for the received command.

> reject.  

Not when targets detect a digest error. Section 5.5 mandates targets
to send a REJECT with a reason of digest error. The "discard & retry"
policy is for initiators.

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 30 Jan 2001 13:42:10 -0800

Matt,

View iSCSI as isolated from either SCSI or TCP.  Mixing SCSI tags or TCP
Acks with Client/Server sequencing is confusing.  By resources, I was
focused only on the iSCSI function and not either TCP or SCSI.  From the
client's perspective, its PDU is sent only if acknowledged. This is where
you run into definitions of when is a PDU delivered to the target.  The same
should be true for the server and this send resource is my focus.  Rather
than fully using this handshake, a restart of the SCSI command is seen as
recovery but then we lose sense of sequence nor is there a  reliable link.

Doug

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 30 Jan 2001 13:42:04 -0800

Santosh,

I was trying to view iSCSI as isolated from either SCSI or TCP.  The mixing
of SCSI tags or TCP Acks together with Client/Server sequencing clouds this
conversation.  By resources, I was focused only on the iSCSI function and
not either TCP or SCSI.  SCSI uses tags, TCP uses its sequence number.
iSCSI uses CmdSN and StatSN.  An iSCSI PDU may require repeating if ExpCmdSN
does not acknowledge reception due to digest failure perhaps.  This is a
thorny issue as there is no sense of time to allow recovery.  If there is a
digest error, should the server tell the client via a type of selective
acknowledgement or by means of reporting a digest error on who knows what?

iSCSI v3:
   "It may happen that a target receives a message with a format error
   (inconsistent fields, reserved fields not 0, inexistent LUN etc.) or
   a digest error (invalid payload or header). The target returns the
   header of the message in error as the data of the response."

   "When a target receives an iSCSI data PDU with a data payload digest
   error, it MUST discard it and request retransmission with a R2T.
   When a target receives an iSCSI PDU with a header digest error or a
   payload digest error in anything but a data iSCSI PDU it MUST answer
   with a Reject iSCSI PDU with a Reject iSCSI PDU with a Reason-code of
   Digest-error."

Here is where a sequence is lost.

   "When an initiator receives an iSCSI data PDU with a data payload
   digest error or any other iSCSI PDU with a header or payload digest
   error it MUST discard it, and restart the task - the later provided
   it could recognize the Initiator Task Tag.  If the initiator can't
   recognize the Initiator Task Tag, (e.g., a header digest error) the
   initiators MUST logout the connection and restart it (including
   restarting all outstanding tasks)."

From the client's perspective, its PDU was sent only if acknowledged.  The
same should be true for the server.  Rather than using this level of
handshake, a restart of the SCSI command is seen as recovery but then we
lose any sense of sequence nor have these handshakes been fully utilized in
establishing a reliable link.  Using bad data to restart a command?  That
does not seem a means of ensuring reliability.  View iSCSI as a transport
independent of both TCP and SCSI and then we can make progress on
understanding how this should work.  A brew of all three is where we are
now.

Doug