Subject: RE: iSCSI: DAP Retry comments

From: [email-masked]
Date: Wed, 10 Jul 2002 03:56:56 -0400

If the iSCSI retry timer and the SCSI ULP timeout are set
with appropriate coordination (e.g., in this case, 2 seconds
would be much better for iSCSI than 8), option A (send NOP
after 2 seconds to see if the command arrived) looks like
it works ok in the current spec.  B is allowed, and enables
a clever target to help out, but A should work regardless.

In a perfect world, we wouldn't have to deal with legacy
tape, and SCSI error recovery would be very efficient ...
but the world's not perfect.  It looks like the Retry and
ErrorRecoveryLevel > 0 recommendations are ok for legacy
tape and the like.

Thanks,
--David

> -----Original Message-----
> From: Dave Peterson [mailto:[email-masked]]
> Sent: Tuesday, July 09, 2002 12:22 PM
> To: [email-masked]; [email-masked]
> Subject: RE: iSCSI: DAP Retry comments
> 
> 
> Regarding Retry, it's not about the command executing twice.
> Below is a rehash from previous emails of the issues with Retry:
> *****
> Example scenario:
> 1. tape locate command is issued with a 10 second timer
> 2. tape command is dropped at the target due to a header digest error
> 3. having seen no response for the command after an iSCSI initiator
> determined timeout value, the initiator decides to retry the command
> 
> This example leaves a 2 second window for the response to the 
> second command
> to arrive before a ULP abort is sent.
> 
> A mechanism to determine whether or not the command arrived 
> at the target
> would be beneficial for the retry functionality to be useful.
> The mechanism should be initiated early enough in the ULP 
> timeout window to
> allow the iSCSI retried command the opportunity to complete.
> 
> Some options:
> 
> A. If the initiator issues a NOP-Out(immed=1), the target can 
> send back the
> expected CmdSN.
> 
> B. The target, upon a header digest error, sends back a 
> reject or NOP-In
> with
> the expected CmdSN. This would provide an indication to the 
> initiator that
> something happened and trigger the command retry, still 
> hopefully within the
> ULP timeout to allow for sucessful command completion.
> 
> C. Texting stating that an iSCSI initiator should (only) 
> perform a command
> retry when sufficient time remains for the command to 
> completed. But, this
> leads one down the path of device type specific behavior.
> 
> D. Remove the command retry functionality. I have yet to see 
> it actually
> being used. The use of command retry is really only 
> applicable when header
> digests are being used. Given TCP, probability of header 
> digest usage and
> occurance,
> and existing ULP tools for error detection and recovery, my 
> preference is to
> remove it.
> ****
> 
> My point is the following need to occur for the Retry 
> functionality to work:
> 1. header digest must be enabled
> 2. a header digest must be detected
> 3. the retried command must be issued early enough in the ULP 
> timeout window
> to allow completion
> 
> I don't see the Retry functionality being useful for disk devices and
> questionable for tape.
> Using a well-engineered (TCP/IP) network along with hopefully 
> available SCSI
> level tools is a better solution.
> 
> But I don't want to hold up the spec over this matter either, 
> we just won't
> use it.
> 
> Regarding SNACK, I don't really have a problem with it other 
> than again a
> well-engineered network should mitigate the need.
> Regarding connection allegiance reassignment, the 
> functionality is a bit
> more useful (if one supports more than 1 connection per 
> session), provided
> the reassignment completes in time to allow the command to 
> complete within
> the ULP timeout.
> 
> Regarding legacy tape devices, I expect them to be front 
> ended by a gateway.
> These devices will most likely not support any type of 
> transport level error
> detection and recovery making them incompatible/problem 
> childs with respect
> to the various iSCSI error recovery mechanisms.
> 
> Bottom line is engineer your network well and leave the error 
> detection and
> recovery to the SCSI level and above...dap
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]
> > Sent: Tuesday, July 09, 2002 1:51 AM
> > To: [email-masked]; [email-masked]
> > Subject: iSCSI: DAP Retry comments
> >
> >
> > > T	p 103	6.1.1 Usage of Retry and 6.7 SCSI Timeouts: the 
> semantics of
> > Retry
> > > remain broken rendering it useless for tape operation. 
> SCSI level error
> > > detection and recovery is the preferred mechanism. Refer 
> to previous
> > emails
> > > sent via the IPS reflector regarding this matter.
> >
> > Can you provide more information?  Command retry *never* results in
> > the command executing twice - both the original command and 
> the retry
> > have the same CmdSN, so the second one is dropped as a duplicate if
> > the first one was received correctly.  6.1.1 is very clear 
> that retry
> > MUST NOT be used if the command was received successfully 
> (acknowledged
> > by ExpCmdSN), and if it is used, the retried command PDU is silently
> > dropped.
> > iSCSI's ordered delivery requirement avoids the situation in which a
> > dropped command causes subsequent commands to mis-execute - if none
> > of the commands are marked for immediate delivery, iSCSI will stop
> > at the "hole" created by the dropped command, and wait for the retry
> > to plug the hole.
> >
> > > T	p 128	8.6 Considerations for State-dependent devices: last
> > paragraph:
> > > don't agree with the statement that error recovery at the 
> iSCSI level
> > > (specifically Retry in its current state) is advisable. Retry
> > at the SCSI
> > > level is feasible and is not difficult (i.e., READ 
> POSITION and LOCATE
> > > commands). This paragraph should be removed.
> >
> > Two questions:
> > - What about the SNACK and allegiance change mechanisms?
> > - What about the "legacy" tape devices (e.g., as discussed 
> in London)
> > 	that presumably don't implement those commands?  I believe this
> > 	text was originally intended to address this class of devices.
> >
> > Thanks,
> > --David
> > ---------------------------------------------------
> > David L. Black, Senior Technologist
> > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > +1 (508) 249-6449            FAX: +1 (508) 497-8018
> > [email-masked]       Mobile: +1 (978) 394-7754
> > ---------------------------------------------------
>

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 10 Jul 2002 08:10:33 +0300

Dave,

This note was helpful - even if I don't agree with all your points.
iSCSI recovery was designed having connection allegiance reassignment as
the main target. However reassignment requires all the other mechanisms for
it to work
properly - i.e., even if you don;t use them within connection you need them
during reassignment.

Julo

                      "Dave Peterson"                                                                                                       
                      <[email-masked]>          To:       <[email-masked]>, <[email-masked]>                                           
                      Sent by:                 cc:                                                                                          
                      [email-masked]        Subject:  RE: iSCSI: DAP Retry comments                                                      
                      .edu                                                                                                                  

                      07/09/2002 07:22                                                                                                      
                      PM                                                                                                                    
                      Please respond to                                                                                                     
                      "Dave Peterson"                                                                                                       

This example leaves a 2 second window for the response to the second
command
to arrive before a ULP abort is sent.

A mechanism to determine whether or not the command arrived at the target
would be beneficial for the retry functionality to be useful.
The mechanism should be initiated early enough in the ULP timeout window to
allow the iSCSI retried command the opportunity to complete.

Some options:

A. If the initiator issues a NOP-Out(immed=1), the target can send back the
expected CmdSN.

B. The target, upon a header digest error, sends back a reject or NOP-In
with
the expected CmdSN. This would provide an indication to the initiator that
something happened and trigger the command retry, still hopefully within
the
ULP timeout to allow for sucessful command completion.

C. Texting stating that an iSCSI initiator should (only) perform a command
retry when sufficient time remains for the command to completed. But, this
leads one down the path of device type specific behavior.

D. Remove the command retry functionality. I have yet to see it actually
being used. The use of command retry is really only applicable when header
digests are being used. Given TCP, probability of header digest usage and
occurance,
and existing ULP tools for error detection and recovery, my preference is
to
remove it.
****

My point is the following need to occur for the Retry functionality to
work:
1. header digest must be enabled
2. a header digest must be detected
3. the retried command must be issued early enough in the ULP timeout
window
to allow completion

I don't see the Retry functionality being useful for disk devices and
questionable for tape.
Using a well-engineered (TCP/IP) network along with hopefully available
SCSI
level tools is a better solution.

But I don't want to hold up the spec over this matter either, we just won't
use it.

Regarding SNACK, I don't really have a problem with it other than again a
well-engineered network should mitigate the need.
Regarding connection allegiance reassignment, the functionality is a bit
more useful (if one supports more than 1 connection per session), provided
the reassignment completes in time to allow the command to complete within
the ULP timeout.

Regarding legacy tape devices, I expect them to be front ended by a
gateway.
These devices will most likely not support any type of transport level
error
detection and recovery making them incompatible/problem childs with respect
to the various iSCSI error recovery mechanisms.

Bottom line is engineer your network well and leave the error detection and
recovery to the SCSI level and above...dap

> > T        p 103             6.1.1 Usage of Retry and 6.7 SCSI Timeouts:
the semantics of
> Retry
> > remain broken rendering it useless for tape operation. SCSI level error
> > detection and recovery is the preferred mechanism. Refer to previous
> emails
> the command executing twice - both the original command and the retry
> the first one was received correctly.  6.1.1 is very clear that retry
> MUST NOT be used if the command was received successfully (acknowledged
> > don't agree with the statement that error recovery at the iSCSI level
> > level is feasible and is not difficult (i.e., READ POSITION and LOCATE
> - What about the "legacy" tape devices (e.g., as discussed in London)