Subject: Re: iSCSI : Holes in StatSN

From: [email-masked]
Date: Wed, 31 Jan 2001 14:57:46 +0200

I've also realized this and included a "sack" mechanism that kicks-in as
soon as the initiator gets a hole in the status order.  ExpstatSN is still
needed to do a low cost bulk acknowledgement.

With a data sequence we may want to use a similar mechanism to ask for a
missed data block as soon as we see one of its successors or the status.

Julo

Santosh Rao <[email-masked]> on 31/01/2001 10:30:59

Please respond to Santosh Rao <[email-masked]>

To:   [email-masked] (ips)
cc:
Subject:  iSCSI : Holes in StatSN

Julian & All,

The StatSN ACK is of the form "ExpStatSN acknowledges all Status
upto the specified value", and hence, it cannot be sent until
all prior command status' have been received.

If StatSN 1 were yet to be received and StatSN 2 - 1000 were received
thereafter, they cannot be acknowledged by the initiator since
the current model does not allow an ExpStatSN ack of 2 - 1000,
without also ACKing 1.

Since StatSNs are assigned on a per-connection basis, this avoids
holes in StatSN received at the initiator. [since TCP orders
delivery to iSCSI within a connection]. However, this in itself
is insufficient and holes can still occur in the StatSN
sequence seen by the initiator, as explained below.

Holes in StatSN sequence are seen by an initiator when
it detects a digest error on the Status PDU
[and discards that PDU] , thereby, dropping such
Status'.

In such cases, the ExpStatSN acknowledgement is not straight
forward at the initiator and does involve some complexity to
keep track of possible holes in StatSN. Further, such holes
may never be filled, since the "retry" command only uses the
same CmdSN while the target sends its response on a different
StatSN.

In the presence of StatSN holes, [and given the current model
of ExpStatSN], initiators will need to score-board received
StatSNs prior to sending ExpStatSN acknowledgements.

A selective StatSN ack (i.e. ExpStatSN ACKs the specified StatSN)
is simpler to implement on the initiator side and allows for
quicker de-alloc of resources at the target end.
This could be considered as either a replacement for the existing
ExpStatSN model, or as a complement to it. (possibly indicated
by a SACK bit in the outbound PDUs containing the ExpStatSN).

Regards,
Santosh

ps :
There is an earlier thread that dates 10/26/00
and is titled "Re: iSCSI Error Recovery", which proposed StatSN per
connection as a solution to this problem, but the problem does not
go away with just setting StatSNs per connection, since holes still
occur on digest errors.

--
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : [email-masked]
Phone : 408-447-3751
#################################

---

From: "Douglas Otis" <[email-masked]>
Date: Wed, 31 Jan 2001 11:19:02 -0800

Julian,

If there is a complete implementation of client and server sequencing
without holes due to zero being a special case, then a sack mechanism should
suffice in allowing the removal of the 'retry' mechanism and muddling with
SCSI.  You would then avoid impacting SCSI with errors induced by the iSCSI
transport.  The interesting aspect of server or target side implementation
of a selective acknowledgement would be with the freedom allowed for the
connection the sack is sent.  As part of that selective acknowledgement, a
digest error detected would help avoid the reliance on a timeout.

Doug

---

From: Santosh Rao <[email-masked]>
Date: Wed, 31 Jan 2001 16:57:25 -0800

Julian,

Once the selective StatSN ACK mechanism kicks in, how is the initiator to
revert to the bulk StatSN ACK ? (i.e. when/how does an initiator realize that
the hole is filled ?) Or, does it only use selective StatSN ACK from there on
?

The difficulty lies in the fact that a hole created will never be filled
since "retry" will result in target sending back a subsequent Status PDU with
a different StatSN. However, the initiator does not know when to safely claim
that the hole is filled (by sending a bulk StatSN ACK), since there is no way
to detect this.

Regards,
Santosh

---

From: Santosh Rao <[email-masked]>
Date: Wed, 31 Jan 2001 17:53:13 -0800

Julian,

The missing data PDU is missing due to either a header format error, header
digest error or data digest error. [in all other cases, TCP ensures reliable
delivery].

In 2 of the above 3 cases, [header format error & header digest error] the
initiator CANNOT do a safe interpretation of the PDU header. Without
interpreting the PDU header, the initiator does NOT get the Initiator Task
Tag. Any request to re-send a particular data PDU MUST be qualified by :
I.T.T + missing_DataSN [+ T.T.T + CmdSN, optionally].

Since I.T.T. cannot be reliably determined in 2 of the 3 cases, such a
re-send request cannot be reliably achieved.

The alternate proposal that was made should be considered in its place, which
was to :
- dis-allow overlapped data xfer's
- initiators do a count check
- a command level retry is performed at the iSCSI layer on detecting an
underrun [due to a missing PDU].

On several ocassions, requests from different people have been made on this
list to dis-allow overlapped data xfers. Can a WG consensus be sought on this
issue to see if the benefits of allowing overlapped data xfer's offset its
complexities and justify its support ?

Regards,
Santosh