Subject: RE: iSCSI & Linked Commands

From: Charles Monia <[email-masked]>
Date: Tue, 17 Apr 2001 13:07:25 -0700

Hi:

> Mark has already replied to your question on why linked commands won't
> work given today's implementations. In most cases, the initiator task
> tag is either generated in the HBA driver or in the HBA firmware. 

The fact that some adapter implementations on some interconnects may not
support linked commands does not grant a license to delete the capability
from the protocol specification.

Incidentally, in my opinion the problem of linked command support has more
to do with whether or not the adapter has enough intelligence to handle the
linked command semantics and retain its internal task state following the
receipt of status.  Making the tag accessible in some way is required if for
no other reason that to support the ABORT TASK function  

Charles

> -----Original Message-----
> From: Santosh Rao [mailto:[email-masked]]
> Sent: Tuesday, April 17, 2001 11:19 AM
> To: [email-masked]
> Subject: Re: iSCSI & Linked Commands
> 
> 
> Julian,
> 
> Mark has already replied to your question on why linked commands won't
> work given today's implementations. In most cases, the initiator task
> tag is either generated in the HBA driver or in the HBA firmware. 
> 
> Since there is no task tag generated from the SCSI ULP while 
> handing an
> I/O request to the SCSI LLP, making the association across different
> commands within the same task (linked command) becomes impossible.
> 
> Is there any reason why iSCSI would want to apply connection 
> allegiance
> per command and not per task (?). If someone did use linked 
> commands and
> wanted a clean way of aborting the task on some error, an abort task
> sent down the same connection as the rest of the task would ensure
> flushing of stale PDUs. 
> 
> I agree that there is no need to say linked commands are unsupported
> since this is a ULP feature and support or lack thereof is decided at
> the initiator and target ULPs. 
> 
> - Santosh
> 
> 
> [email-masked] wrote:
> > 
> > Mark,
> > 
> > The argument about the HBAs generating tags is pretty weak 
> as iSCSI will
> > have it's own HBA's and iSCSI will generate the tags in any 
> implementation.
> > As for the utility - the sequential and conditional 
> execution of the linked
> > commands is guaranteed regardless of delivery or queuing 
> order.  The only
> > reason they might get obsolete is their inability to hide 
> latency but I
> > don't see any compelling reason to have them unsupported by iSCSI.
> > 
> > Julo
> > 
> > Mark Mokryn <[email-masked]> on 17/04/2001 12:55:05
> > 
> > Please respond to Mark Mokryn <[email-masked]>
> > 
> > To:   Julian Satran/Haifa/IBM@IBMIL
> > cc:   [email-masked]
> > Subject:  Re: iSCSI & Linked Commands
> > 
> > Julian,
> > 
> > Santosh is right. Linked commands require an identical 
> I_T_L_x nexus,
> > but many Fibre Channel (and possibly) SCSI adapters 
> generate a queue tag
> > on-board, with no possibility of host software control. On such
> > adapters, the generation of linked commands is impossible, 
> and clearly
> > today's SCSI layers are aware of this.
> > 
> > This raises the entire issue of task management in iSCSI: Linked
> > commands are dated back to SCSI-2, where they indeed served 
> a purpose.
> > In the SCSI bus protocol, the target controlled all SCSI bus phases
> > (following selection). Thus, in a linked command sequence, 
> the target
> > may drive the command phase immediately following the 
> status phase, thus
> > saving bus cycles (i.e. arbitration, selection, etc.). 
> However, in the
> > serial protocols, I don't see how linked commands are of 
> any use, since
> > there are no bus phases to save. In contrast with popular 
> belief, linked
> > commands offer no atomicity. Even in SCSI bus protocol, a 
> linked command
> > may be disconnected at any time (at the target's 
> discretion), and a new
> > command (from any initiator) may be started. Linked 
> commands have always
> > been optional, and indeed many target implementations today do not
> > support them. For instance, looking at the Shark SCSI 
> reference manual,
> > according to the inquiry data, Shark does not support 
> linked commands.
> > 
> > So, perhaps the wise thing to do is to not support linked 
> commands in
> > iSCSI. It has always been an optional feature for logical units, and
> > today is outdated and often unsupported, both by targets 
> and initiators.
> > 
> > -mark
> > 
> > [email-masked] wrote:
> > >
> > > Santosh,
> > >
> > > Sorry to interrupt this captivating thread. Why do you 
> think linked
> > > commands won't work?
> > >
> > > Julo
> > >
> > > Santosh Rao <[email-masked]> on 16/04/2001 20:13:04
> > >
> > > Please respond to Santosh Rao <[email-masked]>
> > >
> > > To:   Douglas Otis <[email-masked]>
> > > cc:   Ips <[email-masked]>
> > > Subject:  Re: iSCSI:flow control, acknowledgement, and a 
> deterministic
> > >       recovery
> > >
> > > Doug,
> > >
> > > You seem to be referring to linked commands as a case wherein the
> > > approach of Abort Task will not flush stale PDUs.
> > >
> > > Linked Commands cannot work the way SCSI implementations 
> are defined
> > > today, since linked commands require the initiator task 
> tag (I_T_L_x
> > > nexus identifier in SAM-2 Execute Command terminology) to 
> be generated
> > > by the SCSI ULP. However, in practice, the Initiator Task 
> Tag (or the FC
> > > OX_ID) is typically generated in the SCSI LLP (or in some 
> cases in the
> > > adapter firmware). IOW, there is no common reference 
> handle like the
> > > task tag sent down from the ULP that allows for 
> association of multiple
> > > commands to a task in several/most implementations today.
> > >
> > > When this is fixed up to get linked commands to work [& 
> there exist
> > > examples of its usage], there is no reason connection 
> allegiance could
> > > not be applied to all the commands within the task.
> > >
> > > I fail to see why you think Abort Task will not work with 
> sequential
> > > devices (?).
> > >
> > > - Santosh
> > >
> > > Douglas Otis wrote:
> > > >
> > > > Santosh,
> > > >
> > > > I see a few problems with this approach.  Tasks as 
> defined in iSCSI do
> > > not
> > > > maintain connection allegiance.  The driver binds all 
> SCSI commands to
> > > their
> > > > connection for the most resent association.  Although 
> there are several
> > > > places within the iSCSI proposal that make reference to 
> a task having a
> > > > connection allegiance, this is in error.  Commands and 
> not tasks carry
> > > such
> > > > allegiance.  Your recovery scheme will not allow a satisfactory
> > recovery
> > > > with a sequential device.  In this case, repeating the 
> command is not a
> > > > solution.  As a result, one connection falter and it 
> will become a
> > > difficult
> > > > situation.  In addition, you have no clue from iSCSI 
> your delivery
> > > status.
> > > > You do not know if you are waiting for the target or if 
> you are waiting
> > > for
> > > > the connection.  Some sequential devices have rather 
> long time-outs
> > with
> > > > these complications of deducing status created by the multiple
> > > connections.
> > > >
> > > > The application will not know about these connection allegiance
> > problems.
> > > > The iSCSI layer does not define interaction to provide 
> additional
> > > > application status to allow these applications to 
> respond in a manner
> > > that
> > > > may aid this situation nor should such additional information be
> > > required.
> > > > With your scheme the SCSI driver must examine the 
> content of these
> > > commands
> > > > to make a guess as to the connection allegiance 
> assignments.  Now the
> > > driver
> > > > is expected to understand what the intended action is 
> of this SCSI
> > > > management command.  What signal is used to indicate a 
> need for the
> > iSCSI
> > > > immediate treatment?  The only obvious seems to be the 
> task attribute
> > > > argument.  With the way iSCSI has defined iSCSI 
> immediate, I would
> > expect
> > > > those commands to be treated in a LIFO rather than the 
> normal FIFO
> > > fashion.
> > > >
> > > > Doug
> > > >
> > > > > Douglas Otis wrote:
> > > > > >
> > > > > > With multiple connections, if you are not going to 
> use a valid
> > > > > > CmdSN, or in your case a null CmdSN for all 
> commands, then there
> > > > > > would be a need to include a timestamp to meet a 
> timely delivery
> > > > > > requirement in the same manner as used in FC 
> encapsulation.  IP
> > > > > > can deliver over any time period.  A command could 
> arrive at any
> > > > > > time with respect to other connections.  With all 
> of your feedback
> > > > > > now from just the SCSI layer, the SCSI layer is 
> likely to have
> > timed
> > > > > > out and restarted and now stray commands finally 
> make an appearance
> > > > > > (the technician re-inserted the cable).  What did 
> that do?  Yes,
> > > > > > if this were on a single connection, then TCP could 
> provide some
> > > > > > assurances, (ignoring digests errors) but you must 
> not make that
> > > > > > assumption nor can you assume all disruptions are symmetric.
> > > > >
> > > > > Doug,
> > > > >
> > > > > The below snippet from my last mail answered your 
> above concern. The
> > > > > Abort Task is sent on the same connection as the 
> command. (connection
> > > > > allegiance applied to the abort task as well). The 
> Abort task pushes
> > > the
> > > > > stale data PDUs. There is no need for a timestamp on 
> iSCSI PDUs.
> > > > >
> > > > > > > As for your second concern regarding I/O 
> timeouts, there is
> > > > > no need for
> > > > > > > any timestamp. An I/O timeout is dealt with by an 
> Abort Task.
> > > > > The abort
> > > > > > > task response guarantees that the abort reached 
> the target and
> > > pushed
> > > > > > > all intermediate stale frames. Failure to 
> complete Abort Task
> > leads
> > > to
> > > > > > > higher level error recovery (ex : Logout, or some 
> higher form of
> > > task
> > > > > > > mgmt).
> > > > >
> > > > > - Santosh
> > >  - santoshr.vcf
>

---

From: Santosh Rao <[email-masked]>
Date: Tue, 17 Apr 2001 15:43:00 -0700

> from the protocol specification.

Charles,

I was'nt saying that iSCSI must not support linked commands. See my

> the initiator and target ULPs.  

What I did say was that several/most implementations perform task tag
initialization at the LLP or further below rendering this feature
un-usable in several cases. 

The more interesting question is :
Is there any particular reason connection allegiance would rather be
enforced per command than per task ? (if linked commands are un-used,
this is a moot question....). If it were enforced per task, then, an
Abort Task would effectively flush all stale PDUs of that task ensuring
no stale PDUs in the network upon I/O termination at the initiator.

- Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 17 Apr 2001 22:03:06 -0700

Santosh,

Linked commands typically employ a feature that allows relative addressing
for sequential devices.  Link bits are still present within the CDB even for
fibre channel.  As only one link command is ever seen on the network at any
point in time, identify the most recent command related to a task and then
use this information to associate the Abort Task command possibly issued
when the ULP times out.  This assumes that Abort Task would be the command
used in this event and that the backup application is able to recover from
such.

For your scheme to work, log all command's connection allegiance, examine
all CDBs to determine their nature, and identify the related task based on
the content of a management command.  You are suggesting that the connection
will be flushed by your technique, but if not, tear down the connection?
Without any acknowledgement as to what was delivered into the SCSI layer,
this forces a wait for the ULP to timeout on each command.  Potentially this
invites a large list of timeouts to stumble over before recovery while each
timeout may be met with a management command.  One wonders if you have saved
any effort.  This is compared to checking the sequence of commands at the
receiver which also provides acknowledgement.  This means there is no extra
effort to ensure sequential delivery and a problem can be detected and
overcome without involving the ULP.

A result of not having any acknowledgement within the iSCSI transport would
be all problems are determined by timeouts within the ULP.  Each problem is
met with uncertainty of cause.  There would only be one command to flush per
such timeout however.  I was just trying to correct the language you used
with respect to command and not task allegiance.  You have however failed to
continue the operation of the sequential device based on a network problem.
In addition, you must ensure that the driver notices the intent of the
management command and finds the correct connection to send this command.
If this connector is indeed unplugged, then this command as well as the one
you are attempting to abort will timeout.  After two ULP timeouts, a
connection tear down will then create many more such events even if the
problem is not with the network.  Check by pinging first?  How long do you
wait for a ping to respond?

Without management being serialized, you could not be sure of their
placement relative to other connections.  All connections would need to stop
until this management command is acknowledged by the SCSI device and even
then these pending commands on other connections may not have already
arrived.  The first timeout caused some application to screech to a halt and
now the entire system will be required to wait for this recovery.  Should
this problem happen again, there will be uncertainty as to whether it is a
connection problem or a device problem.  A ping down the connection in
question may reveal a problem after an additional timeout. You have removed
communication with the sequencer with the exception of "rare" sequential
commands but recovery from a connection falter looks difficult, slow and
highly dependent upon the ULP being well written for this iSCSI environment.

Doug

---

From: "Dave Peterson" <[email-masked]>
Date: Wed, 18 Apr 2001 08:35:40 -0500

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Douglas Otis
> Sent: Wednesday, April 18, 2001 12:03 AM
> To: Santosh Rao; Charles Monia
> Cc: [email-masked]
> Subject: RE: iSCSI & Linked Commands
>
>
> Santosh,
>
> fibre channel.

Humm, I'm not aware of this feature for sequential devices.
Also, for FC-TAPE compliant devices command linking is prohibited.
Dave

---

From: "Douglas Otis" <[email-masked]>
Date: Wed, 18 Apr 2001 14:20:49 -0700

Dave,

You could be correct.  I have little knowledge about FC-TAPE.  At one time
there was a suggestion that intermediate status in FCP response and link
bits to make this accommodation.  I am not aware if this was ever
implemented.  Are you suggesting that linked commands be prohibited within
iSCSI as a result of it not being available within Fibre-Channel?  A locate
followed by a series of reads assumes a next block on a sequential device.
I see that as relative addressing.

Doug