Subject: Re: iSCSI : digest error handling violates EMDP/InDataOrder

From: "John Hufferd" <[email-masked]>
Date: Mon, 7 May 2001 13:26:13 -0700

Santosh,
I have re-read 2.7.5 a number of times, and can not understand your point.
To me that section clearly states that the data can be sent out of order if
permitted by the EMDP bit.  But any individual bursts, must be in order
within that burst/R2T response.  The only thing that can be out of order,
is the data sent in different bursts, with respect to each other.  That
seems clear, so I must be missing some important point that you are trying
to make.  If you think I have this all wrong please let me know.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Internet address: [email-masked]

Santosh Rao <[email-masked]>@ece.cmu.edu on 05/03/2001 05:35:03 PM

Sent by:  [email-masked]

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  Re: iSCSI : digest error handling violates EMDP/InDataOrder

In previous exchanges in this thread, it was stated that EMDP only
governs data order within a sequence. This, to me [and others from FC
world, as was noted by Bob] reads as the definition of Random relative
Offset [as defined by bob].

However, Section 2.7.5 on Buffer Offset states that :
"Output data within a burst MUST be delivered in increasing buffer
offset order".

This contradicts the above by indicating that iSCSI forbids Random
Relative Offset.

So, does EMDP apply to WRITEs or does it not (?).

Also, does iSCSI allow data overlay or is this prohibited ? I'd suggest
the following be clearly described in the draft :

a) iSCSI defines EMDP to govern the order of data PDUs within a SCSI
command for both READ and WRITE operations. An EMDP setting of 0 implies
data overlay is forbidden.

b) The out-of-order transmission of a given sequence of WRITE data PDUs
in response to an R2T (the random relative offset property) be
explicitly dis-allowed by iSCSI [which is currently stated in Section
2.7.5].

Regards,
Santosh

[email-masked] wrote:
>
> Santosh,
>
> Let's take a systematic approach to it.
>
> Restriction on data ordering are required if the source or the
destination
> of the data is unable to deliver or take data data in any order other
that
> sequential.
>
> Semiconductor or other direct access memories don't  have this
restriction.
>
> Tapes and other sequential media do have this type of restriction and so
> some streaming devices.
>
> If the restricted device is a target of a SCSI operation with an
> unrestricted initiator then:
>
> a. on reads the target can always ship its data in sequential order
> b. on writes the target can  always request the data in sequential order
>
> However if the restricted device is an initiator then:
>
> a. on reads the initiator will request the target to send the data in
order
> b. on writes the restricted initiator will have to get the R2Ts in order
> from the target and will be able to support data recovery through an R2T
> only if it has enough buffered data.
>
> A restricted device will act as an initiator only if it becomes a third
> part copy manager (CM) in a third party operation an does copy from one
of
> its devices to another device.
>
> Introducing a new mode bit (as Robert Snively seems to suggest) will not
> change the fact that the restriction can't be upholded
> and do recovery unless the restricted initiator has enough memory.
>
> The spec should only specify a way to terminate a command in those
> conditions and leave it at that.
>
> I will change the wording of the DataOrder to make it clearer but I
> consider the whole issue entirely academic and overblown.
>
> Recall also that a CM implemented which such severe buffering
restrictions
> violates the basic SCSI assumption that a target is the data master.
>
> Regards,
> Julo
>
> Santosh Rao <[email-masked]> on 28/04/2001 00:03:26
>
> Please respond to Santosh Rao <[email-masked]>
>
> To:   Julian Satran/Haifa/IBM@IBMIL
> cc:   [email-masked], [email-masked]
> Subject:  Re: iSCSI : digest error handling violates EMDP/InDataOrder
>
> [email-masked] wrote:
> >
> > David,
> >
> > I read Bob's mail and my interpretation is similar to his. However I
> think
> > that SPC explicitly states that different transports are free to
> interpret
> > and make use of this page as they find appropriate.
> >
> > I have a hard time understanding Santosh's objection as it does not
refer
> > to the reason the EMDP is there but to the way it is written in FCP
(not
> > iSCSI).
>
> Julian,
>
> As has been stated earlier, EMDP allows control over the order in which
> the target requests outbound data or sends inbound data. EMDP can be
> used by initiators to control this order and turn off out-of-order R2T
> requests [as well as turn off out of order read data pdus].
>
> This is a useful control option and is already provided by other SCSI
> transports. What good reason exists to deny this provision in iSCSI ?
>
> Also, I have some concerns about the ambiguous definition of DataOrder.
>
> Per the spec :
> "DataOrder=<yes|no>
>
> The default is yes but targets MAY support no. No is used by iSCSI to
> indicate that the data PDUs can be in any order (EMDP = 1). Yes is used
> to indicate that incoming data PDUs have to be at continuously
> increasing addresses (EMDP = 0)."
>
> Based on the above definition wording :
>
> a) How is DataOrder interpreted for WRITE I/Os ?
> b) Is the ordering across the entire SCSI command or a subset of the I/O
> ? If so, what constitutes this subset ?
>
> Different implementors can arrive at different interpretations reading
> the above definition !
>
> - Santosh
>  - santoshr.vcf

---

From: Santosh Rao <[email-masked]>
Date: Mon, 07 May 2001 16:50:57 -0700

John,

The fundamental question is :
"Does EMDP apply to WRITE commands ?"

The answer for other SCSI transports is yes. [from my queries on t11 and
looking at SRP's defn.]

If EMDP is applicable to WRITEs, then, that does'nt come across clear
enough in the current iSCSI definition of this bit.

If it is applicable to WRITEs, does it control of the scsi command, or
order within a sequence (as in data PDUs sent in response to an R2T) ?
(This is'nt clear from the defn either.)

If EMDP does apply to WRITEs, earlier discussion in this thread that the
EMDP ordering applies within a sequence contradicts the following text
in Section 2.7.5 :
"Output data within a burst MUST be delivered in increasing buffer
offset order."

To summarize the changes required :

1. A clear defn for EMDP is required that states that EMDP applies to
both WRITEs and READs and governs the order of data within a scsi
command [not a sequence]. 

" When EMDP is set to 0 :
- For a READ, the target MUST send all the data PDUs within the scsi
command in continuous increasing order. 
- For a WRITE, the target MUST send R2T requests in continuous
increasing order.

When EMDP is set to 1 :
- For a READ, the data PDUs within a SCSI command may be delivered out
of order. 
- For a WRITE, the target may send R2T requests out of order."

2. Re-phrase the following in Section 6.2 :
"If the error is a Data-Digest-Error in a Data-PDU, the targets MUST
either request retransmission with a R2T or answer with a command
response PDU with a response-code  of delivery-subsystem-failure and
terminate the task."

to :

"If the error is a Data-Digest-Error in a data-PDU, the target MUST
perform either one of the following actions :
- Request re-transmission with a R2T, provided DataOrder was negotiated
to "No".
- Answer with a command response PDU with a response-code  of
delivery-subsystem-failure and terminate the task."

3. Specify clearly that the "Random relative Offset" feature is
prohibited by stating :
" All Data PDUs within a data sequence in response to an R2T MUST be
delivered in increasing buffer offset order."

Regards,
Santosh

John Hufferd wrote:
> is the data sent in different bursts, with respect to each other. 
> That seems clear, so I must be missing some important point that you are trying