Subject: iSCSI : digest error handling violates EMDP/InDataOrder

From: Santosh Rao <[email-masked]>
Date: Thu, 12 Apr 2001 17:54:28 -0700

Where :
=======

Section 6.2 (pg 80). Digest Errors
-----------------------------------
"If the error is a Data-Digest-Error in a Data-PDU, the target MUST
either request retransmission with a R2T or answer with a Reject iSCSI
PDU and abort the task."

Problem :
---------
On a Data digest error detected by a target, it MUST NOT request
re-transmission of the data PDU thru an R2T if the session login key
InDataOrder is set to yes. The current rev 05 draft violates
InDataOrder/EMDP settings by allowing a re-transmission of R2T by
target.

Scenario :
==========
initiator			target
---------			------
EMDP=0
InDataOrder=YES
(exp_off=0)
	   offset=0,len=64k <------ R2T
	
--------> data PDUs
(exp_off = 64K)
                              data digest error results in
			      an 8K PDU being dropped at offset 24K.

	  offset=24K,len=8K  <------ R2T for missing PDU.

exp_off != offset

- Santosh

---

From: [email-masked]
Date: Thu, 19 Apr 2001 08:44:53 +0200

Santosh,

No it is a mistake. But unlike FCP we will require ordering only within a
sequence (input data or unsolicited or answer to a R2T).

Thanks,
Julo

Santosh Rao <[email-masked]> on 18/04/2001 19:20:53

Please respond to Santosh Rao <[email-masked]>

To:
cc:   [email-masked]
Subject:  Re: iSCSI : digest error handling violates EMDP/InDataOrder

[email-masked] wrote:
>
> OK - I misread it. In any case we are not FCP and we don't violate iSCSI
> rules.

Julian,

What reasons exist to differ in EMDP behaviour b/n iSCSI and FCP ?

Also, a fundamental question is that the description only speaks about
incoming data PDUs. Are you implying that InDataOrder only control the
ordering for READ data PDUs ? If so, what is the mechanism to control
ordering for write data PDUs ?

It is a useful control option for initiators to negotiate that R2T
requests be made in increasing continuous buffer offset order and R2T
requests not be sent out of order. Does iSCSI allow this ?

- Santosh

"31 InDataOrder

   InDataOrder=<yes|no>

   No is used by iSCSI to indicate that the incoming data PDUs can be in
   any order (EMDP = 1). Yes is used to indicate that incoming data PDUs
   have to be at continuously increasing addresses (EMDP = 0).

   This also sets the Connect-Disconnect mode page EMDP bit.

   The default is yes but targets MAY support no. "

------------------------------------------------------------------------------------

> > FCP uses it like iSCSI - i.e. the order has to maintained within a
> sequence
>
> Not true. If you take a look at FCP-2 rev 04 Section 10.1.1.7
> description on EMDP, it explicitly states :
> "The EMDP bit does not affect the order of frames within a sequence".
>
> For a WRITE command, an EMDP setting of 0 implies that the buffer offset
> in R2T requests must be in continuous and increasing order whereas an
> EMDP setting of 1 implies the buffer offset in R2T can be out of order.
>
> For a READ command, an EMDP setting of 0 implies the buffer offset in
> READ data PDUs is in continuous and increasing order, whereas, an EMDP
> setting of 1 implies buffer offset in READ Data PDUs can be out of
> order.
>
> Based on the above rules, iSCSI is violating EMDP setting by its error
> recovery for data digest errors detected by targets on Data PDUs.
>
> - Santosh
 - santoshr.vcf

---

From: [email-masked]
Date: Thu, 26 Apr 2001 18:00:27 +0300

David,

I read Bob's mail and my interpretation is similar to his. However I think
that SPC explicitly states that different transports are free to interpret
and make use of this page as they find appropriate.

I have a hard time understanding Santosh's objection as it does not refer
to the reason the EMDP is there but to the way it is written in FCP (not
iSCSI).

I have trouble understanding why would a target that is reissuing an R2T
violate any rule. The ordering rules where meant to assure that targets can
do their work in a fashion that is optimal for them and they where not
meant to affect initiator operation - except perhaps to enable exception
checking.

The whole issue looks to me as overblown.

Regards,
Julo

[email-masked] on 24/04/2001 22:00:45

Please respond to [email-masked]

To:   [email-masked], [email-masked]
cc:
Subject:  RE: iSCSI : digest error handling violates EMDP/InDataOrder

Santosh's original issue was that R2Ts to request
retransmission of data (e.g., due to data CRC failure)
result in an initiator seeing what appear to be out
of order R2Ts due to the need to go back and get
the failed data retransmitted.

> InDataOrder is set to yes.

This key has been renamed to DataOrder in -06, and if
it's set to "yes" as currently defined, then (IMHO)
Santosh appears to be correct.  The Initiator is not
going to be expecting the R2T offset to step back to
pick up the missing data, and hence the Target MUST
Reject and Abort.

Beyond this, I take Bob Snively's mail as a suggestion
that we ought to split iSCSI DataOrder from SCSI's
EMDP, as FCP considers those to be separate concepts.
That seems like a reasonable approach.

Comments?

--David

> -----Original Message-----
> From:   Santosh Rao [SMTP:[email-masked]]
> Sent:   Tuesday, April 24, 2001 2:30 PM
> To:     [email-masked]
> Cc:     David Black
> What is the final resolution on this EMDP issue. IMHO, iSCSI must retain
> the EMDP semantics as defined in FCP, SRP. i.e. It controls the order of
> the data across the entire SCSI command. (which includes sending R2T
> requests in order, if EMDP was set to 1).
>
> Some additional thoughts on this topic are :
>
> 1) Is it worth a finer granularity of control wherein the initiator be
> allowed to negotiate with the target that R2T requests be sent in-order
> , while not imposing any constraints on the Read Data PDU order.
>
> 2) Should control be provided over a "Random Relative Offset" feature,
> as Bob describes it below, or is it to be assumed that iSCSI Data PDUs
> will always be in-order within a sequence ?
>
> 3) Speaking of sequence, this terminology has been often used in this
> thread. Where is the notion of a sequence defined in iSCSI ? What is the
> definition of an iSCSI sequence.
>
> - Santosh
>
> Robert Snively wrote:
> >
> > Seems to me that there are some unclarities in this area as well.
> >
> > There are really two pieces being discussed as one:
> >
> >         EMDP (a SCSI functionality)
> >
> >         Random relative offset (a transport functionality)
> >
> > EMDP is used to allow a target to request or deliver its data
> > out of order.  This is used for things like passing a stripe
> > segment from a RAID data extent as soon as it has been accumulated,
> > rather than waiting until all previous parts of the RAID data
> > extent have also been accumulated and delivered.  It is also used
> > for things like "start anywhere" reading of a disk track.
> >
> > It says nothing about the ordering of data within a PDU or sequence
> > which must be ordered according to the rules of the protocol.  Fibre
> > Channel allows the data within a sequence to be transmitted in order
> > or out of order by using the login parameter "random relative offset".
> > Almost all devices choose to login and require "continuously increasing
> > relative offset". << File: Card for Santosh Rao >>

---

From: [email-masked]
Date: Sat, 28 Apr 2001 16:14:10 +0300

Santosh,

Let's take a systematic approach to it.

Restriction on data ordering are required if the source or the destination
of the data is unable to deliver or take data data in any order other that
sequential.

Semiconductor or other direct access memories don't  have this restriction.

Tapes and other sequential media do have this type of restriction and so
some streaming devices.

If the restricted device is a target of a SCSI operation with an
unrestricted initiator then:

a. on reads the target can always ship its data in sequential order
b. on writes the target can  always request the data in sequential order

However if the restricted device is an initiator then:

a. on reads the initiator will request the target to send the data in order
b. on writes the restricted initiator will have to get the R2Ts in order
from the target and will be able to support data recovery through an R2T
only if it has enough buffered data.

A restricted device will act as an initiator only if it becomes a third
part copy manager (CM) in a third party operation an does copy from one of
its devices to another device.

Introducing a new mode bit (as Robert Snively seems to suggest) will not
change the fact that the restriction can't be upholded
and do recovery unless the restricted initiator has enough memory.

The spec should only specify a way to terminate a command in those
conditions and leave it at that.

I will change the wording of the DataOrder to make it clearer but I
consider the whole issue entirely academic and overblown.

Recall also that a CM implemented which such severe buffering restrictions
violates the basic SCSI assumption that a target is the data master.

Regards,
Julo

Santosh Rao <[email-masked]> on 28/04/2001 00:03:26

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked], [email-masked]
Subject:  Re: iSCSI : digest error handling violates EMDP/InDataOrder

Julian,

As has been stated earlier, EMDP allows control over the order in which
the target requests outbound data or sends inbound data. EMDP can be
used by initiators to control this order and turn off out-of-order R2T
requests [as well as turn off out of order read data pdus].

This is a useful control option and is already provided by other SCSI
transports. What good reason exists to deny this provision in iSCSI ?

Also, I have some concerns about the ambiguous definition of DataOrder.

Per the spec :
"DataOrder=<yes|no>

The default is yes but targets MAY support no. No is used by iSCSI to
indicate that the data PDUs can be in any order (EMDP = 1). Yes is used
to indicate that incoming data PDUs have to be at continuously
increasing addresses (EMDP = 0)."

Based on the above definition wording :

a) How is DataOrder interpreted for WRITE I/Os ?
b) Is the ordering across the entire SCSI command or a subset of the I/O
? If so, what constitutes this subset ?

Different implementors can arrive at different interpretations reading
the above definition !

- Santosh
 - santoshr.vcf