Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues

From: [email-masked]
Date: Thu, 25 Jan 2001 19:40:12 +0200

answers in text - Julo

Santosh Rao <[email-masked]> on 25/01/2001 04:06:59

Please respond to Santosh Rao <[email-masked]>

To:   IPS Reflector <[email-masked]>
cc:
Subject:  iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues

Julian & All,

There's a number of problems with the current iSCSI handling of digest
errors (Section 5.5) :

"When an initiator receives an iSCSI data PDU with a digest error or any

other PDU with a header or data digest error, it must discard and
re-start the task -provided it can recognize the Initiator Task Tag".

1) The initiator task tag cannot be trusted when a header digest error
is seen. What does the phrase "provided it can recognize the initiator
task tag" mean ?
How can an initiator reliably claim that the initiator task tag is
trustworthy ?

<js> an initiator may choose to provide some redundancy in the tag itself
</js>

2) The use of a "discard and re-start" policy can cause some  problems.
The target is un-aware that the initiator has discarded the task and
will then receive a 2nd command with the same Initiator Task Tag and
CmdSN for a task already in progress, which can cause transmission of
duplicate sets of data on the same I.T.T.

Any policy of "discard and re-start" must be modified to "discard, abort

and then, re-start, and the draft MUST specify connection allegiance of
the command and its Abort Task".

<js> a restart is recognized as such through the restart bit. Why would you
require abort?

please explain </js>

3) The policy of "discard and restart" is also subject some race
conditions in the CmdSN sliding window. At the time the digest error was

detected at the initiator, the ExpCmdSN may not yet have acknowledged
that
command. This causes the initiator to restart the command with the same
Initiator Task Tag, CmdSN and "retry" bit set.

However, by the time the command gets to the target, the target may have

updated its ExpCmdSN window having sent a later PDU which updated the
ExpCmdSN. This results in the target discarding the received CmdSN since

<js> at command restart you never really rely on the CmdSN. you will want
to check
the Initiator Task Tag and accept it in the above case. </js>

it is outside the expected window of (ExpCmdSN, MaxCmdSN).

The "discard & restart" policy cannot be reliably used unless it is on a

connection failure, in which case, the failed connection is first logged

out and an updated ExpCmdSN is received on the logout response.
Following the logout response, the "retry" commands CmdSN can be
based on the ExpCmdSN.
<js> see above </js>
In the case of digest errors, the CmdSN window can be moving forward
and an attempt to "discard and restart" will result in a re-send of a
CmdSN that is outside the CmdSN window, causing the target to
discard the retried command.

<js> the target has enough information to discard only true replicas </js>
Regards,
Santosh

 - santoshr.vcf

---

From: Santosh Rao <[email-masked]>
Date: Thu, 25 Jan 2001 11:17:33 -0800

Julian,

My responses enclosed inline.

Regards,
Santosh

> <js> an initiator may choose to provide some redundancy in the tag itself

All of these excessive "mays" only serve to complicate the standard and end up
as un-used features. They are better off removed.

> <js> a restart is recognized as such through the restart bit. Why would you

Where does the draft EXPLICITLY state that the "retry" bit implies an implicit
abort of the command followed by re-start ?

In the absence of such explicit clarifications, the draft is opening up
opportunities for numerous creative mis-interpretations.

Despite this, there is still a window b/n the command being re-started at the
initiator and the target receiving the command with "retry" bit set, when
stale frames continue to arrive at the initiator for that I.T.T. Therefore, in
order to avoid stale frames from the previous command continuing to arrive at
the initiator in this window, there needs to be an abort followed by a
re-start.

All of this complexity is only in the case of digest error handling [and not
in the case of connection failures]. The draft can be rid of all this
complexity if digest errors were to be recovered at a connection level,
instead of this "discard & restart" policy which adds all these complexities.

> the Initiator Task Tag and accept it in the above case. </js>

The draft states the following on this subject :

Section 5.1
=========
-    the initiator will reissue all outstanding commands with their
      original Initiator Task Tag and their original CmdRN if they
      are not acknowledged yet or a CmdRN of 0 (not-numbered) if they
      were acknowledged; the retry (X) flag in the command PDU will
      be set

Section 1.2.2.1
===========
- The target MUST silently ignore any command
   outside this range or duplicates within the range not flagged with
   the retry bit (the X bit in the opcode).

This, to me, means :
- ignore all commands outside the (ExpCmdSN, MaxCmdSN) range.
- ignore all commands within the range that are not flagged with the retry
bit.

Can you please clarify the intent of the text ?

---

From: Matt Wakeley <[email-masked]>
Date: Thu, 25 Jan 2001 12:42:19 -0800

>       were acknowledged; the retry (X) flag in the command PDU will
I think you missed the word "duplicates"...

> Can you please clarify the intent of the text ?

-Matt

---

From: Matt Wakeley <[email-masked]>
Date: Thu, 25 Jan 2001 13:53:25 -0800

Santosh Rao wrote:
> Matt Wakeley wrote:
> Alright (so the cut & paste did'nt work)  ;-}
> 
> The issue under discussion is a retried command [and thereby, a duplicate
> CmdSN] that is OUTSIDE the (ExpCmdSn, MaxCmdSN) range. That falls under the
> first criteria.

And the point is?  What am I missing?

-Matt

> - Santosh

---

From: Santosh Rao <[email-masked]>
Date: Thu, 25 Jan 2001 13:21:56 -0800

- Santosh

---

From: Santosh Rao <[email-masked]>
Date: Thu, 25 Jan 2001 15:06:09 -0800

Matt,

Section 5.1
=========

Section 1.2.2.1
===========

This, to me, means :
- ignore all commands outside the (ExpCmdSN, MaxCmdSN) range.
- ignore all duplicate commands within the range that are not flagged with the
retry
bit. << "duplicate" added this time >>

---

From: "Eddy Quicksall" <[email-masked]>
Date: Thu, 25 Jan 2001 15:43:32 -0500

Unfortunatly, I must agree with this issue. You should never rely on
anything within the scope of a CRC error. So, there would be no way to
verify the tag.

Eddy

----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>
Sent: Thursday, January 25, 2001 12:40 PM
Subject: Re: iSCSI : Digest Error Problems & CmdSN/ExpCmdSN window issues

> <js> an initiator may choose to provide some redundancy in the tag itself
[ rest of original message deleted here]

---

From: Pierre Labat <[email-masked]>
Date: Thu, 25 Jan 2001 18:08:41 -0800

> > Initiator Task Tag, CmdSN and "retry" bit set.

The "retry" can work and works fine only in the case it "retry" a command
that was issued on a failed connection that has been logged out.
In the other cases you are in trouble because of the ghost IOs and "retry"
MUST be avoided.

>       were acknowledged; the retry (X) flag in the command PDU will

Duplicate means a command with the same CmdSN.

Regards,

Pierre

---

From: Michael Krause <[email-masked]>
Date: Fri, 26 Jan 2001 18:59:12 -0800

><js> an initiator may choose to provide some redundancy in the tag itself

I'm aware of some techniques for inserting redundant information in tags 
which limits the potential error exposure when a multi-bit error occurs, 
however these are not fail-safe leading to potential incorrect operation - 
perhaps benign in many cases; perhaps not in others. As such, if a header 
digest error occurs, the PDU should be silently discarded and recovery 
should be left to the ULP.  There is little to no value having two 
mechanisms to solve the same problem.

Mike