Subject: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues

From: Santosh Rao <[email-masked]>
Date: Wed, 24 Jan 2001 18:06:59 -0800

Julian & All,

There's a number of problems with the current iSCSI handling of digest
errors (Section 5.5) :

"When an initiator receives an iSCSI data PDU with a digest error or any

other PDU with a header or data digest error, it must discard and
re-start the task -provided it can recognize the Initiator Task Tag".

1) The initiator task tag cannot be trusted when a header digest error
is seen. What does the phrase "provided it can recognize the initiator
task tag" mean ?
How can an initiator reliably claim that the initiator task tag is
trustworthy ?

2) The use of a "discard and re-start" policy can cause some  problems.
The target is un-aware that the initiator has discarded the task and
will then receive a 2nd command with the same Initiator Task Tag and
CmdSN for a task already in progress, which can cause transmission of
duplicate sets of data on the same I.T.T.

Any policy of "discard and re-start" must be modified to "discard, abort

and then, re-start, and the draft MUST specify connection allegiance of
the command and its Abort Task".

3) The policy of "discard and restart" is also subject some race
conditions in the CmdSN sliding window. At the time the digest error was

detected at the initiator, the ExpCmdSN may not yet have acknowledged
that
command. This causes the initiator to restart the command with the same
Initiator Task Tag, CmdSN and "retry" bit set.

However, by the time the command gets to the target, the target may have

updated its ExpCmdSN window having sent a later PDU which updated the
ExpCmdSN. This results in the target discarding the received CmdSN since

it is outside the expected window of (ExpCmdSN, MaxCmdSN).

The "discard & restart" policy cannot be reliably used unless it is on a

connection failure, in which case, the failed connection is first logged

out and an updated ExpCmdSN is received on the logout response.
Following the logout response, the "retry" commands CmdSN can be
based on the ExpCmdSN.

In the case of digest errors, the CmdSN window can be moving forward
and an attempt to "discard and restart" will result in a re-send of a
CmdSN that is outside the CmdSN window, causing the target to
discard the retried command.

Regards,
Santosh

---

From: [email-masked]
Date: Fri, 26 Jan 2001 17:50:37 +0200

Santosh,

I think that you are misreading my comments.
When I say a may in one of my answers it does not mean that it has to
appear as such in the draft.

As for the interpretation of restart - once delivered a restarted command
has to do just that restart.
If for implementation reasons you want to take the abort path and then
restart  that is an implementation issue.

In any case it is not equivalent with sending an explicit task abort then
start the command as this is not an atomic sequence and may affect the
command ordering.

Julo

N.B. I really appreciate you keeping the list so alive during you education
session but I think that you can improve your hit ratio (i.e. getting to
real problems) if you'll discuss your points with your powerful "home-team"
(Randy, Pierre etc.).

Santosh Rao <[email-masked]> on 25/01/2001 21:17:33

Please respond to Santosh Rao <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues

Julian,

My responses enclosed inline.

Regards,
Santosh

[email-masked] wrote:

> <js> an initiator may choose to provide some redundancy in the tag itself
> </js>

All of these excessive "mays" only serve to complicate the standard and end
up
as un-used features. They are better off removed.

> <js> a restart is recognized as such through the restart bit. Why would
you
> require abort?

Where does the draft EXPLICITLY state that the "retry" bit implies an
implicit
abort of the command followed by re-start ?

In the absence of such explicit clarifications, the draft is opening up
opportunities for numerous creative mis-interpretations.

Despite this, there is still a window b/n the command being re-started at
the
initiator and the target receiving the command with "retry" bit set, when
stale frames continue to arrive at the initiator for that I.T.T. Therefore,
in
order to avoid stale frames from the previous command continuing to arrive
at
the initiator in this window, there needs to be an abort followed by a
re-start.

All of this complexity is only in the case of digest error handling [and
not
in the case of connection failures]. The draft can be rid of all this
complexity if digest errors were to be recovered at a connection level,
instead of this "discard & restart" policy which adds all these
complexities.

> please explain </js>
> <js> at command restart you never really rely on the CmdSN. you will want
> to check
> the Initiator Task Tag and accept it in the above case. </js>

The draft states the following on this subject :

Section 5.1
=========
-    the initiator will reissue all outstanding commands with their
      original Initiator Task Tag and their original CmdRN if they
      are not acknowledged yet or a CmdRN of 0 (not-numbered) if they
      were acknowledged; the retry (X) flag in the command PDU will
      be set

Section 1.2.2.1
===========
- The target MUST silently ignore any command
   outside this range or duplicates within the range not flagged with
   the retry bit (the X bit in the opcode).

This, to me, means :
- ignore all commands outside the (ExpCmdSN, MaxCmdSN) range.
- ignore all commands within the range that are not flagged with the retry
bit.

Can you please clarify the intent of the text ?

 - santoshr.vcf

---

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 26 Jan 2001 12:29:09 -0500

Regarding number 1) which refers to section 5.5 ...

why not just change the paragraph to say:

   When an initiator receives an iSCSI data PDU with a data payload
   digest error or any other iSCSI PDU with a header or payload digest
   error it MUST discard it, and restart the task - the later provided
   header does not have a digest error. If there is a header digest
   error the initiator may need to recognize and recover using an
   implementation specific scheme such as timing the request. Schemes
   may include loging out the connection and restarting it (including
   restarting all outstanding tasks).

Eddy

----Original Message Follows----
From: [email-masked]
To: [email-masked]
Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues
Date: Fri, 26 Jan 2001 17:50:37 +0200

Santosh,

Julo

Julian,

Regards,
Santosh

you

Section 5.1
=========

Section 1.2.2.1
===========

This, to me, means :

  - santoshr.vcf

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com

---

From: [email-masked]
Date: Fri, 26 Jan 2001 19:49:43 +0200

Santosh,

Education was not meant to be insulting and I am sorry if did sound to you.
However you are restarting discussion threads that where long settled
without
bringing new arguments. And I am not exactly thrilled in reliving the
thread.

Regards,
Julo

Santosh Rao <[email-masked]> on 26/01/2001 19:29:09

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked], [email-masked] (Santosh Rao)
Subject:  Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues

Julian,

I believe you are referring to the "Retry" bit , is that correct ? (There
is nothing in the draft that I can find called a Restart bit).

If you are referring to the retry bit, can you please point me to where
the draft EXPLICITLY states that a "retry" shall cause the target to
implicitly abort execution of the previous instance of that command ? In
the absence of such explicit wording in the draft, how does one derive
the conclusion you mention above ?

Also, I did'nt see your response to the below 2 issues raised :

> - ignore all duplicate commands within the range that are not
>   flagged with the retry bit.
>
Can you please clarify the intent of the text ? From the above it
does'nt sound like the draft specifies that CmdSNs received out of the
window are to be accepted if the Retry bit is set.

> There is still a window b/n the command being re-started at
> the initiator and the target receiving the command with "retry"
> bit set, when stale frames continue to arrive at the initiator for
> that I.T.T. Therefore, in order to avoid stale frames from the
> previous command continuing to arrive at the initiator in this window,
> there needs to be an abort followed by a re-start.

How is the above to be avoided ?

> N.B. I really appreciate you keeping the list so alive during you
> education session but I think that you can improve your hit ratio

Anyone would appreciate comments being restricted to technical responses
instead of such remarks. Perhaps, you may want to check the threads in
the past to see how many issues have been uncovered and how many times you
stood corrected.
(ex : faking of check conditions instead of using a service response,
response data and response length, DataRN, target originated connection
level error recovery thru the use of Async Messages, Abort Task response,
retries to tapes result in possible data corruption, bridging CmdSNs,
fragmentation & re-assembly issues, etc).

Wonder who's getting educated here. (?)

--
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : [email-masked]
Phone : 408-447-3751
#################################

---

From: "Eddy Quicksall" <[email-masked]>
Date: Sat, 27 Jan 2001 17:19:25 -0500

Actually, its not that bad. Parallel SCSI logic already considers a similar 
case. That is the case where one of the tagged commands never completes.

What happens is that one of the upper layers (sometimes the application and 
sometimes the operating system) times the I/O.

The I/O associated with this tag will timeout and the layer that is 
responsible takes corrective action. With a disk it is very simple ... just 
abort the I/O and re-issue it.

With something like a tape, the application has to determine if the tape 
needs to be backspaced or not and then re-issues the I/O.

In parallel SCSI, the driver or one of the upper layers will usually issue a 
Target Reset to get rid of all the tags (because it is assumed that the 
device has malfunctioned). This is not considered drastic in many systems 
and is very natural when testing RAID systems.

So ... for us, all we need to do is drop the packet.

In the iSCSI spec, we are basically saying you can rely on corrupted data 
and I just don't think that is a good idea.

Eddy

----Original Message Follows----
From: [email-masked]
To: [email-masked]
Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues
Date: Sat, 27 Jan 2001 22:24:31 +0200

If the header is a data header we can hardly trust the ULP to recognize the
error (he might be unaware
of a missing packet).  With data numbering this situation could have been
discovered at "status time".
The only thing we could do is restart all commands but this is equivalent
to a connection restart for all practical purposes.  Dropping data
numbering might have some more "side-effects" like this.
As the combination of values - tag, address, offset may stil let some
implementations to assume that they have
a correct task identifier I don't see a point in mandating a recovery
behavior and the implementer may choose to:

-retry/restart command
-logout drop and rebuild connection login and restart/retry
-abort all task sets (practically reset the target!) and report for all
commands a "delivery system failure" (kick-in the ULP recovery) and if you
suspect the link quality rebuild it; this later behavior means also that
you have to stop delivering anything on any link  to the target to avoid
out of order execution until you have finished the cleanup - pretty drastic

With data numbering recovery could have stayed within the confines of a
command even if a header was bad.
Perhaps we should leave the DataSN only as a sequencer so that at
status-time the initiator should be able to find if a data packet was
dropped (no ExpDataSN on a NOP).

Regards,
Julo

Michael Krause <[email-masked]> on 27/01/2001 04:59:12

Please respond to Michael Krause <[email-masked]>

 ><js> an initiator may choose to provide some redundancy in the tag itself

I'm aware of some techniques for inserting redundant information in tags
which limits the potential error exposure when a multi-bit error occurs,
however these are not fail-safe leading to potential incorrect operation -
perhaps benign in many cases; perhaps not in others. As such, if a header
digest error occurs, the PDU should be silently discarded and recovery
should be left to the ULP.  There is little to no value having two
mechanisms to solve the same problem.

Mike