Subject: RE: iSCSI:flow control, acknowledgement, and a deterministic reco very

From: [email-masked]
Date: Tue, 10 Apr 2001 13:57:22 -0400

Doug,

There are a couple of confusions in here:

TASK_SET_FULL *is* a flow control mechanism, albeit a
rather primitive one - the Initiator issues commands until
it gets a FULL response and hence knows the capacity
of the Target's queue.  The statement that the Target loses
resource control if it only uses TASK_SET_FULL is wrong,
although it is relying on code at or above the SCSI driver
on the Initiator to handle the flow control.

Santosh did not propose to remove CmdSN, although he
did point out an issue that needs attention - we need
to say something about the interaction of CmdSN with
TASK_SET_FULL since both mechanisms may be implemented.
In particular, it's possible to use CmdSN for delivery
ACKs and TASK_SET_FULL for flow control, although that's
a "peculiar" combination (and perhaps a SHOULD NOT?).

Beyond this, your entire second paragraph is well off
into the weeds.  Please be careful about taking the
discussion off onto wild tangents.

Thanks,
--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

From:	Douglas Otis [[email-masked]]
Sent:	Tuesday, April 10, 2001 2:09 AM
To:	[email-masked]
Cc:	Ips
Subject:	RE: iSCSI:flow control, acknowledgement, and a deterministic
recovery

Santosh,

In a change recommending a flag and a valid CmdSN and not a null value, I
included a flag named "Casual" to provide this mode of operation you
describe which still allows flow control.  I would not wish to permit this
Casual mode, but as David feels such lists or threads are easy to manage,
this should not represent any greater effort for feedback.  But then again,
if you are not using flow control, you are also not using any
acknowledgement other than an eventual SCSI status.  If there is no flow
control, this removes resource control and reception acknowledgement.  If
this is not needed, then such flow control should be removed for simplicity
as there then must always be an abundance by design of adequate recourses.
For cases where command sequence is important, CmdSN together with
acknowledgement must remain or only one command at a time can be issued.  I
can envision three modes, a Sequential mode, a Casual mode, and an Exigent
mode where the command is given head of sequence status (with rejection for
feedback simplicity ;).

You seem to be placing many things on the table for reconsideration.  Flow
control, acknowledgement, sequential delivery, and perhaps even a means of
moving a command up the queue feeding SCSI targets.  One command at a time
for sequential assurance is not impossible.  Is flow control needed?  Do you
care if there is any reception acknowledgement if all feedback is in the
SCSI layer?  It would seem in this barebones case, iSCSI adds little to
ensure integrity.  The FC encapsulation places a time stamp within the PDU.
Perhaps a time stamp should be employed as an alternative to sequence
control?  At least after a timeout, you are assured a missing command is
forever gone.  Otherwise, the check was to see that CmdSN is not too small
to ensure it is not a stray.

Doug

> Doug [& All],
>
> Some comments on this thread :
>
> 1) The immediate command feature can be exploited by initiators which
> are not required to provide strict command ordering for their SCSI
> sub-system. (The majority of today's scsi stacks). IOW, all commands can
> be sent with CmdSN=0 to indicate no ordering required.
>
> Any scheme to restrict the number of "immediate commands" that can be
> sent prohibits such a feature and is un-desirable.
>
> 2) The (ExpCmdSN, MaxCmdSN) based acknowledgement and flow control is a
> freebie that targets can use to implement additional flow control.
> There's nothing in the spec that mandates that a target MUST use this
> feature to throttle its window and apply flow control.
>
> Hence, any dependence and assumptions on the existence of CmdSN based
> flow control may be incorrect, since the target may be using QUEUE FULL
> (TASK SET FULL) to apply flow control.
>
> > I also suspect it is a mistake to allow commands
> > to remain trapped within the sequencer during emergency or
> abnormal events
> > signified by the use of these immediate commands.
>
> 3) If the initiator suspects that the CmdSN queue at the target is
> stuck, it can always use a task mgmt command or even an iSCSI login with
> the X (restart) bit to perform a cleanup of stuck commands at the
> target. There's no need to build in implicit assumptions that a CmdSN
> with the immediate flag result in a clean-up of previously pending
> commands at the CmdSN queue of the target.
>
> - Santosh
>
>
> Douglas Otis wrote:
> >
> > David,
> >
> > > > To not support rejects, you are then relying on the target to
> > > > handle these now out of sequence commands.
> > >
> > > Immediate commands are "out of sequence" by definition - immediate
> > > means deliver immediately without regard to CmdSN order.  Only the
> > > target can do this, so I don't see any problem with relying
> on the Target
> > > to do so.
> >
> > The commands that are NOW out of sequence are those commands
> bypassed by the
> > immediate command.  If this immediate command was Rewind, then
> a long list
> > of write commands become invalid as a new tape is now needed for those
> > operations.  You are expecting that the target to handle a long list of
> > invalid commands made possible by the iSCSI sequencer.
> Commands invalidated
> > by immediate commands become out of sequence commands.
> >
> > I'll reserve further comment until Julian has made some progress in
>
> >
> > Forgive my advocacy, but if one does not attempt to support a
> position, then
> > the subject is not explored.
> >
> > Doug

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 10 Apr 2001 14:55:08 -0700

David,

A limitation in the number of tasks for each or all targets would be a
control method or as a type of Xon/Xoff as with SEP, but not an iSCSI one as
it is now.  Santosh said all commands could use a null CmdSN in his first
statement.  Perhaps iSCSI should explicitly exclude this use.  This does
imply there is no acknowledgements, no flow control, and no sequential
delivery within iSCSI.

I pointed out in the second paragraph there is still a need to ensure timely
delivery or NO delivery.  That is not a property of IP.  With multiple
connections, if you are not going to use a valid CmdSN, or in his case a
null CmdSN for all commands, then there would be a requirement to include a
timestamp to meet a timely delivery requirement in the same manner as used
in the FC encapsulation.  He had made it clear he was not concerned about
sequential delivery and perhaps I made the rather absurd but valid point as
a result.

You have now offered the possibility of different flow control schemes.  I
started this effort to offer a solution for the existing scheme.  The flora
gets thick in this area it would appear.

Doug

---

From: Santosh Rao <[email-masked]>
Date: Tue, 10 Apr 2001 17:04:15 -0700

Douglas Otis wrote:
> 
> Santosh said all commands could use a null CmdSN in his first

Doug,

What you state above is no different than a traditional SCSI transport
implementation. The acknowledgements, flow control and sequential
delivery properties are dervied from TCP. iSCSI behaves as an
encapsulation only. Most host O.S. stacks and data applications have no
expectations of strict ordering from the scsi transport. The QUEUE FULL
has served as a flow control mechanism in the past. 

IOW, simple implementations may choose to derive benefits from existing
mature TCP and SCSI algorithms rather than re-invent & re-implement all
of the transport capabilities within iSCSI.

There is no need to preclude implementations from sending all commands
with a 0 CmdSN. 

As for your second conern regarding I/O timeouts, there is no need for
any timestamp. An I/O timeout is dealt with by an Abort Task. The abort
task response guarantees that the abort reached the target and pushed
all intermediate stale frames. Failure to complete Abort Task leads to
higher level error recovery (ex : Logout, or some higher form of task
mgmt).

- Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 10 Apr 2001 18:09:41 -0700

Santosh,

You deleted my point :(

With multiple connections, if you are not going to use a valid CmdSN, or in
your case a null CmdSN for all commands, then there would be a need to
include a timestamp to meet a timely delivery requirement in the same manner
as used in FC encapsulation.  IP can deliver over any time period.  A
command could arrive at any time with respect to other connections.  With
all of your feedback now from just the SCSI layer, the SCSI layer is likely
to have timed out and restarted and now stray commands finally make an
appearance (the technician re-inserted the cable).  What did that do?  Yes,
if this were on a single connection, then TCP could provide some assurances,
(ignoring digests errors) but you must not make that assumption nor can you
assume all disruptions are symmetric.

If I was making an iSCSI device and expected to have an ability to control
resources, then your all null CmdSN implementation has made my equipment
dysfunctional.  My suggestion did not prevent your approach if the Casual
flag were adopted and it also kept the documented scheme for flow control
functional.  It also assures stray commands over multiple connections would
be rejected!  If you think ordered delivery is not needed as most disk
vendors do, then you should object to the requirements document just posted.
Sequential SCSI layer delivery as a mandate provides the most information
concerning the state of the SAN.

Doug

---

From: Santosh Rao <[email-masked]>
Date: Thu, 12 Apr 2001 18:19:09 -0700

Doug,

The below snippet from my last mail answered your above concern. The
Abort Task is sent on the same connection as the command. (connection
allegiance applied to the abort task as well). The Abort task pushes the
stale data PDUs. There is no need for a timestamp on iSCSI PDUs.

> > As for your second concern regarding I/O timeouts, there is no need for

- Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 13 Apr 2001 11:20:55 -0700

Santosh,

I see a few problems with this approach.  Tasks as defined in iSCSI do not
maintain connection allegiance.  The driver binds all SCSI commands to their
connection for the most resent association.  Although there are several
places within the iSCSI proposal that make reference to a task having a
connection allegiance, this is in error.  Commands and not tasks carry such
allegiance.  Your recovery scheme will not allow a satisfactory recovery
with a sequential device.  In this case, repeating the command is not a
solution.  As a result, one connection falter and it will become a difficult
situation.  In addition, you have no clue from iSCSI your delivery status.
You do not know if you are waiting for the target or if you are waiting for
the connection.  Some sequential devices have rather long time-outs with
these complications of deducing status created by the multiple connections.

The application will not know about these connection allegiance problems.
The iSCSI layer does not define interaction to provide additional
application status to allow these applications to respond in a manner that
may aid this situation nor should such additional information be required.
With your scheme the SCSI driver must examine the content of these commands
to make a guess as to the connection allegiance assignments.  Now the driver
is expected to understand what the intended action is of this SCSI
management command.  What signal is used to indicate a need for the iSCSI
immediate treatment?  The only obvious seems to be the task attribute
argument.  With the way iSCSI has defined iSCSI immediate, I would expect
those commands to be treated in a LIFO rather than the normal FIFO fashion.

Doug

> Douglas Otis wrote:
> >
> > With multiple connections, if you are not going to use a valid
> > CmdSN, or in your case a null CmdSN for all commands, then there
> > would be a need to include a timestamp to meet a timely delivery
> > requirement in the same manner as used in FC encapsulation.  IP
> > can deliver over any time period.  A command could arrive at any
> > time with respect to other connections.  With all of your feedback
> > now from just the SCSI layer, the SCSI layer is likely to have timed
> > out and restarted and now stray commands finally make an appearance
> > (the technician re-inserted the cable).  What did that do?  Yes,
> > if this were on a single connection, then TCP could provide some
> > assurances, (ignoring digests errors) but you must not make that
> > assumption nor can you assume all disruptions are symmetric.
>
> Doug,
>

---

From: Santosh Rao <[email-masked]>
Date: Mon, 16 Apr 2001 11:13:04 -0700

Doug,

You seem to be referring to linked commands as a case wherein the
approach of Abort Task will not flush stale PDUs.

Linked Commands cannot work the way SCSI implementations are defined
today, since linked commands require the initiator task tag (I_T_L_x
nexus identifier in SAM-2 Execute Command terminology) to be generated
by the SCSI ULP. However, in practice, the Initiator Task Tag (or the FC
OX_ID) is typically generated in the SCSI LLP (or in some cases in the
adapter firmware). IOW, there is no common reference handle like the
task tag sent down from the ULP that allows for association of multiple
commands to a task in several/most implementations today.

When this is fixed up to get linked commands to work [& there exist
examples of its usage], there is no reason connection allegiance could
not be applied to all the commands within the task.

I fail to see why you think Abort Task will not work with sequential
devices (?).

- Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 16 Apr 2001 14:41:49 -0700

Santosh,

The iSCSI proposal ver 5-91 explicitly defines tasks and also includes the
option to allow linked commands to be sent across different connections.
Obviously, for sequential devices, particular attention must be paid to
command serialization as these commands tend to use relative addressing or
are dependent upon the successful completion of prior commands.  This
requirement is not helped with Auto-Sense and impels the need for a target
model change in SCSI.  An error injected into the SCSI layer as a result of
a network communication error will significantly reduce the utility of most
backup applications.  Such reliance on the SCSI layer to recover from such
uncertainty imposed as a result of the inability of the network transport to
do minimal handshakes and retries is the wrong approach.  Regardless, you
are burdening the driver with the duty of tracking a transient connection
allegiance status.  The latest version has improved language with the
exception of Pg. 16 in two places.

Ver 5-91
Pg. 15
"Connection allegiance is strictly per-command and not per-task."

Pg. 16
"tasks that have allegiance to the connection"
"all outstanding tasks that have allegiance to the connection to conclude
and send their status."

Doug

> > I see a few problems with this approach.  Tasks as defined in
> > iSCSI do not maintain connection allegiance.  The driver binds all
> > SCSI commands to their connection for the most resent association.
> > Although there are several places within the iSCSI proposal that
> > make reference to a task having a connection allegiance, this is
> > in error.  Commands and not tasks carry such allegiance.  Your
> > recovery scheme will not allow a satisfactory recovery with a
> > sequential device.  In this case, repeating the command is not a
> > solution.  As a result, one connection falter and it will become a
> > difficult situation.  In addition, you have no clue from iSCSI your
> > delivery status.  You do not know if you are waiting for the target
> > or if you are waiting for the connection.  Some sequential devices
> > have rather long time-outs with these complications of deducing
> > status created by the multiple connections.
> >
> > The application will not know about these connection allegiance
> > problems. The iSCSI layer does not define interaction to provide
> > additional application status to allow these applications to respond
> > in a manner that may aid this situation nor should such additional
> > information be required.  With your scheme the SCSI driver must
> > examine the content of these commands to make a guess as to the
> > connection allegiance assignments.  Now the driver is expected to
> > understand what the intended action is of this SCSI management
> > command.  What signal is used to indicate a need for the iSCSI
> > argument.  With the way iSCSI has defined iSCSI immediate, I
> > would expect those commands to be treated in a LIFO rather than the
> > normal FIFO fashion.
> >
> > Doug