Subject: RE: iSCSI Requirements Draft - Informal WG Last Call

From: [email-masked]
Date: Thu, 12 Apr 2001 22:09:12 -0400

Santosh,

Objection noted - could you attempt to write
alternative text that would respect the original
motivation to avoid reordering of commands in
most situations without causing the problems
that concern you in error cases?  Fibre Channel
may be a useful analogy, as FC delivers in order
as long as there isn't a frame drop or CRC
failure (both of which are very rare events).

Thanks,
--David

> -----Original Message-----
> From:	Santosh Rao [SMTP:[email-masked]]
> Sent:	Thursday, April 12, 2001 9:34 PM
> To:	[email-masked]
> Cc:	[email-masked]
> Subject:	Re: iSCSI Requirements Draft - Informal WG Last Call
> 
> David & All,
> 
> I object to the following requirement :
> 
> " MUST support ordered delivery of SCSI commands from the initiator to
> the 
>   target, to support SCSI Task Queuing. "
> 
> Ordered delivery is not a requirement for disk based applications and
> non tagged queueing tape applications, which form the majority of
> today's data traffic. 
> 
> To impose strict ordering (even in the presence of errors ?) as a MUST
> is penalizing the majority of today's data traffic that does not expect
> ordering from the SCSI subsystem.
> 
> I am particularly concerned about the effect of the above requirement in
> the presence of errors. Does iSCSI expect strict ordering to be
> maintained even when individual I/O errors like ULP timeout occur ? 
> 
> On a ULP timeout (caused by, say, a hole in CmdSN), the initiator may
> choose not to retry the command, but instead, error it back to the ULP.
> In such a case, it can plug the hole in CmdSN with a NOP-OUT.
> 
> The above requirement is not feasible to be met under such circumstances
> and others similar to this. Mandating strict ordering on ULP timeouts
> implies a session level error recovery on any individual I/O being
> failed back from iSCSI to SCSI ULP. This is a very heavy hammer to use
> as error recovery and should not be imposed.
> 
> The above requirement must be changed to :
> " SHOULD support ordered delivery of SCSI commands from the initiator to
> the 
>   target, to support SCSI Task Queuing. "
> 
> - Santosh
> 
> 
> 
> [email-masked] wrote:
> > 
> > It is intended to submit draft-ietf-ips-iscsi-reqmts-02.txt
> > as an Informational RFC. There is no formal requirement for
> > a WG Last Call, but if you have any further substantive comments
> > on the document please raise them on this list within the next
> > two weeks, i.e. by April 27th at the latest.
> > 
> > If you have typographical/editorial comments please send them
> > direct to the document's author, Marjorie Krueger
> > <[email-masked]>.
> > 
> > Thanks,
> > --David and Elizabeth, IPS WG co-chairs << File: Card for Santosh Rao >>
>

---

From: [email-masked]
Date: Wed, 18 Apr 2001 09:03:39 +0200

Sandeep,

Ordered vs. Pipelined - in delivery (not execution) terms are the same.
Yes we considered per LUN ordering and dropped as being excessively complex
to implement (the only thing that will require per-LUN state) and the total
ordering will be required for complete resets.

SAM still keeps the per-LUN ordering for CmdRN.

Julo

Sandeep Joshi <[email-masked]> on 17/04/2001 16:43:56

Please respond to Sandeep Joshi <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  Re: iSCSI Requirements Draft - Informal WG Last Call

Julian,

I believe you mean "pipelined delivery" as opposed to "ordered
delivery".  The latter is more strict and will introduce pipeline
stalls which we would wish to avoid.  The flow control mechanism
is getting tied into introducing false sequentiality.

As an aside, a colleague here wanted to know if we had considered
having per-LUN ordering as opposed to per-session command ordering.
The pipeline could also slow down because of an aging disk or a
heavily loaded disk.  I do see some discussions at the Haifa meeting
  http://www.ece.cmu.edu/~ips/meetingMinutes/06.20.2000.txt

Was it just added complexity which forced the per-session choice?

> Point 3.4.3 brings up ordering of commands per LUN or ordering of
commands
> per session?
> MW - reorder requirements - second before the first one
> RH - does anybody disagree with 3.4.3?
> CS - No. It basically says that you support SCSI queuing
> JH - In the wide area, a method of pipelining commands and responses
> is required
> CS - the requirement is more complex than saying you just support
> SCSI queing.
> RH - delivering commands in order never hurts
> MT - Why keep order between logical units in commands?
> RH - SAM-2 does not require order between LUNs. However, it may make
> target implemetnation easier.

thanks,
-Sandeep

[email-masked] wrote:
>
> Ordered delivery of commands to ANY TYPE of devices will increase in
> importance as network speeds increase and the need to hide latency
> increases.
>
> Today databases don't use queuing and rely and trickle the commands to
> devices 1 by 1 to ensure atomicity and order.
> As latency will become the determining factor in performance this is
bound
> to change.
>
> SCSI has done an excellent job in defining the queueing mechanism. We
have
> to make it work with good performance in our environment.
>
> Julo
>
> Santosh Rao <[email-masked]> on 13/04/2001 04:33:45
>
> Please respond to Santosh Rao <[email-masked]>
>
> To:   [email-masked]
> cc:   [email-masked]
> > --David and Elizabeth, IPS WG co-chairs
>  - santoshr.vcf

---

From: [email-masked]
Date: Wed, 18 Apr 2001 14:46:21 +0200

There where many reasons for going to a total ordering.
You will find them by looking up discussions starting with the Haifa
meeting last summer.

Julo

Santosh Rao <[email-masked]> on 17/04/2001 21:06:56

Julian,

Ordered delivery can be achieved to better effect using the SAM-2 CRN
based ordering due to the following reasons :

1) CRN provides ordering on a per-lun basis and can be turned on and off
for a subset of I/Os to that LUN. This allows for flexible ordering
since ordering is a function of the I/O type from the application.
Applications that are doing READ only operations (like a search engine)
do not require any ordering. Ordering is required on metadata updates,
any form of synchronization I/Os, WRITEs interspersed with READS, etc.
Thus, an ordering solution should be flexible enough to be applied at
the scope of a subset of I/Os destined to a LUN.

Such an ordering scheme would also allow ordering to be turned on for
only tape applications if disk applications did not require ordering.

iSCSI's ordering solution does not provide this flexibility, whereas
usage of CRN would.

2) Such a fine granularity scope of ordering also minimizes the impact
of error recovery actions taken when loss of order occurs. The impact
with CRN would be a target-initiator handshake based on ACA + some
checkpoints to error back all the pending CRN enabled commands on that
LUN.

Comparing the equivalent error recovery in a CmdSN based iSCSI ordering
solution, the action taken would be to error back all the pending I/Os
destined to the entire session following loss of order. With high end
disk arrays having 1000+ LUN configurations, such error recovery is
extreme, [especially when ordering may have been desired by the appln
only on a small subset of I/Os to 1 LUN, and loss of ordering for the
remaining 999 LUNs was a don't care].

3) A CRN based ordering scheme works for all underlying SCSI transports
as opposed to CmdSN based ordering.

4) The generation of a stream of commands that expect strong ordering
will need to be accompanied by corresponding generation of a sequence
number at the same layer. (CRN would provide such a sequencing). Failure
to do so can result in silent loss of order that slips un-detected due
to potential points of failure in the stack b/n the SCSI ULP and the
physical bus/link. (ex : I/O failures within the HBA driver due to
resource allocation failures or other such conditions can cause loss of
order.).

Attempts to enforce ordering at multiple layers of the stack (CRN at the
ULP and CmdSN at the LLP), especially when CmdSN does not provide all
the benefits that CRN would provide is over-engineering the solution to
the ordering problem. It also impacts iSCSI performance.

- Santosh

bound
have
 - santoshr.vcf