Subject: Re: iSCSI & Linked Commands

From: [email-masked]
Date: Tue, 17 Apr 2001 15:48:52 +0200

Mark,

The argument about the HBAs generating tags is pretty weak as iSCSI will
have it's own HBA's and iSCSI will generate the tags in any implementation.
As for the utility - the sequential and conditional execution of the linked
commands is guaranteed regardless of delivery or queuing order.  The only
reason they might get obsolete is their inability to hide latency but I
don't see any compelling reason to have them unsupported by iSCSI.

Julo

Mark Mokryn <[email-masked]> on 17/04/2001 12:55:05

Please respond to Mark Mokryn <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  Re: iSCSI & Linked Commands

Julian,

Santosh is right. Linked commands require an identical I_T_L_x nexus,
but many Fibre Channel (and possibly) SCSI adapters generate a queue tag
on-board, with no possibility of host software control. On such
adapters, the generation of linked commands is impossible, and clearly
today's SCSI layers are aware of this.

This raises the entire issue of task management in iSCSI: Linked
commands are dated back to SCSI-2, where they indeed served a purpose.
In the SCSI bus protocol, the target controlled all SCSI bus phases
(following selection). Thus, in a linked command sequence, the target
may drive the command phase immediately following the status phase, thus
saving bus cycles (i.e. arbitration, selection, etc.). However, in the
serial protocols, I don't see how linked commands are of any use, since
there are no bus phases to save. In contrast with popular belief, linked
commands offer no atomicity. Even in SCSI bus protocol, a linked command
may be disconnected at any time (at the target's discretion), and a new
command (from any initiator) may be started. Linked commands have always
been optional, and indeed many target implementations today do not
support them. For instance, looking at the Shark SCSI reference manual,
according to the inquiry data, Shark does not support linked commands.

So, perhaps the wise thing to do is to not support linked commands in
iSCSI. It has always been an optional feature for logical units, and
today is outdated and often unsupported, both by targets and initiators.

-mark

[email-masked] wrote:
>
> Santosh,
>
> Sorry to interrupt this captivating thread. Why do you think linked
> commands won't work?
>
> Julo
>
> Santosh Rao <[email-masked]> on 16/04/2001 20:13:04
>
> Please respond to Santosh Rao <[email-masked]>
>
> To:   Douglas Otis <[email-masked]>
> cc:   Ips <[email-masked]>
> Subject:  Re: iSCSI:flow control, acknowledgement, and a deterministic
>       recovery
>
> Doug,
>
> You seem to be referring to linked commands as a case wherein the
> approach of Abort Task will not flush stale PDUs.
>
> Linked Commands cannot work the way SCSI implementations are defined
> today, since linked commands require the initiator task tag (I_T_L_x
> nexus identifier in SAM-2 Execute Command terminology) to be generated
> by the SCSI ULP. However, in practice, the Initiator Task Tag (or the FC
> OX_ID) is typically generated in the SCSI LLP (or in some cases in the
> adapter firmware). IOW, there is no common reference handle like the
> task tag sent down from the ULP that allows for association of multiple
> commands to a task in several/most implementations today.
>
> When this is fixed up to get linked commands to work [& there exist
> examples of its usage], there is no reason connection allegiance could
> not be applied to all the commands within the task.
>
> I fail to see why you think Abort Task will not work with sequential
> devices (?).
>
> - Santosh
>
> Douglas Otis wrote:
> >
> > Santosh,
> >
> > I see a few problems with this approach.  Tasks as defined in iSCSI do
> not
> > maintain connection allegiance.  The driver binds all SCSI commands to
> their
> > connection for the most resent association.  Although there are several
> > places within the iSCSI proposal that make reference to a task having a
> > connection allegiance, this is in error.  Commands and not tasks carry
> such
> > allegiance.  Your recovery scheme will not allow a satisfactory
recovery
> > with a sequential device.  In this case, repeating the command is not a
> > solution.  As a result, one connection falter and it will become a
> difficult
> > situation.  In addition, you have no clue from iSCSI your delivery
> status.
> > You do not know if you are waiting for the target or if you are waiting
> for
> > the connection.  Some sequential devices have rather long time-outs
with
> > these complications of deducing status created by the multiple
> connections.
> >
> > The application will not know about these connection allegiance
problems.
> > The iSCSI layer does not define interaction to provide additional
> > application status to allow these applications to respond in a manner
> that
> > may aid this situation nor should such additional information be
> required.
> > With your scheme the SCSI driver must examine the content of these
> commands
> > to make a guess as to the connection allegiance assignments.  Now the
> driver
> > is expected to understand what the intended action is of this SCSI
> > management command.  What signal is used to indicate a need for the
iSCSI
> > immediate treatment?  The only obvious seems to be the task attribute
> > argument.  With the way iSCSI has defined iSCSI immediate, I would
expect
> > those commands to be treated in a LIFO rather than the normal FIFO
> fashion.
> >
> > Doug
> >
> > > Douglas Otis wrote:
> > > >
> > > > With multiple connections, if you are not going to use a valid
> > > > CmdSN, or in your case a null CmdSN for all commands, then there
> > > > would be a need to include a timestamp to meet a timely delivery
> > > > requirement in the same manner as used in FC encapsulation.  IP
> > > > can deliver over any time period.  A command could arrive at any
> > > > time with respect to other connections.  With all of your feedback
> > > > now from just the SCSI layer, the SCSI layer is likely to have
timed
> > > > out and restarted and now stray commands finally make an appearance
> > > > (the technician re-inserted the cable).  What did that do?  Yes,
> > > > if this were on a single connection, then TCP could provide some
> > > > assurances, (ignoring digests errors) but you must not make that
> > > > assumption nor can you assume all disruptions are symmetric.
> > >
> > > Doug,
> > >
> > > The below snippet from my last mail answered your above concern. The
> > > Abort Task is sent on the same connection as the command. (connection
> > > allegiance applied to the abort task as well). The Abort task pushes
> the
> > > stale data PDUs. There is no need for a timestamp on iSCSI PDUs.
> > >
> > > > > As for your second concern regarding I/O timeouts, there is
> > > no need for
> > > > > any timestamp. An I/O timeout is dealt with by an Abort Task.
> > > The abort
> > > > > task response guarantees that the abort reached the target and
> pushed
> > > > > all intermediate stale frames. Failure to complete Abort Task
leads
> to
> > > > > higher level error recovery (ex : Logout, or some higher form of
> task
> > > > > mgmt).
> > >
> > > - Santosh
>  - santoshr.vcf

---

From: Santosh Rao <[email-masked]>
Date: Tue, 17 Apr 2001 11:19:06 -0700

Julian,

Mark has already replied to your question on why linked commands won't
work given today's implementations. In most cases, the initiator task
tag is either generated in the HBA driver or in the HBA firmware. 

Since there is no task tag generated from the SCSI ULP while handing an
I/O request to the SCSI LLP, making the association across different
commands within the same task (linked command) becomes impossible.

Is there any reason why iSCSI would want to apply connection allegiance
per command and not per task (?). If someone did use linked commands and
wanted a clean way of aborting the task on some error, an abort task
sent down the same connection as the rest of the task would ensure
flushing of stale PDUs. 

I agree that there is no need to say linked commands are unsupported
since this is a ULP feature and support or lack thereof is decided at
the initiator and target ULPs. 

- Santosh

---

From: Mark Mokryn <[email-masked]>
Date: Tue, 17 Apr 2001 23:52:39 +0200

Julian,

The HBA issue just underscores the fact that linked commands are simply
unused today. You can get sequential, conditional execution through
ordered commands. A precise emulation of the linked command sequence is
to send the second ordered command only after receiving status on the
first (we can even argue whether the ordered tags are necessary here or
anywhere else, but this is another issue). Thus, it appears to me that
linked commands are a redundant mechanism. The only implementation I
know of is on the AS/400 (if someone knows of others I'd like to know),
where they're used for skip reads and writes (i.e. LBA mask is sent with
first CDB, read/write with second CDB). But with all due respect to the
Rochester folks, if this is the chief implementation (plus these are
non-standard commands)... The reason I suggested to obsolete them is
that they're redundant, unused, and their benefit seems to be highly
questionable. Thus, linked commands (along with some other stuff) just
clutter the standards.

I believe linked commands were an attempt (and a weak one at that) to
mimic IBM channel command chaining. It's a poor effort, since another
task may be inserted into the task set (and possibly executed) at any
point between chain start and end. Mainframe command chaining works 
since the device (or at least device extent, with PAV) is locked once
the chain begins.

Now if you really want atomicity through chaining, then the way to go is
as you suggested earlier - to prefetch the commands. This is the way
it's done in Ficon - the entire chain is prefetched and then executed,
and only the final status counts. If recovery is needed, it is done at
the chain level.

-mark