Subject: iSCSI: header digest error at initiator

From: Pierre Labat <[email-masked]>
Date: Tue, 09 Jan 2001 09:55:28 -0800

Julian,

In the section "4.5 Digest errors"
you say that if the initiator detects  an header digest error
in an incoming iSCSI PDU, the TCP connection must be
restarted.

Is it to address the following case or for something else?

1) the initiator issues a READ

2) one of the inbound data PDU has a header digest error

3) the initiator (as specified in "1.2.5 iSCSI Full Feature Phase":
   "Initiators MUST NOT perform any score boarding for data
   and the residual count calculation is to be performed by the
targets".)
    doesn't check the total length of the data received when the
completion
   comes. Because it trusts the target and TCP.

4) the initiator thinks the READ is ok and in fact it is not.

To solve that, you propose the drop of  the connection in 2) ?

Regards,

Pierre

---

From: Santosh Rao <[email-masked]>
Date: Tue, 09 Jan 2001 19:12:55 -0800

> 4) the initiator thinks the READ is ok and in fact it is not.

This sounds like  extreme behaviour to me. Why is iSCSI attempting to
prevent initiators from score-boarding data ? This is a standard
practice most parallel scsi and FC initiators would follow to detect
cases of I/O underrun. Not doing this will cause the initiator to trust
the target's resid value and if there was an underrun, reporting success
back to upper layers can result in data corruption.

What is the justification for NOT allowing initiators to score-board ?

> To solve that, you propose the drop of  the connection in 2) ?

If this is the intention of the recommended error recovery, it is the
result of not allowing score-boarding. By score-boarding an initiator
would detect an underrun and would just error the affected I/O back. In
this case, all outstanding tasks on that connection are being affected.

Both the Format Error and Digest Error handling seem too extreme. A
format error or digest error recovery should only involve the affected
task and none others.

Thanks,
Santosh

---

From: Matt Wakeley <[email-masked]>
Date: Mon, 15 Jan 2001 20:18:27 -0800

Santosh Rao wrote:
> would detect an underrun and would just error the affected I/O back.

Two comments here.  First, in your example, the initiator is inventing an
error that really didn't occur in the target.  The target completed the I/O
successfully, it was the transport that experienced an error, but you're
treating it like a target error.

Second, you say that initiators and targets routinely perform scoreboarding. 
How is this done today?  Buffer(s) are provided to an I/O chip.  The I/O chip
writes the data into the buffers.  It does not have the memory to determine
that each and every byte has been written to. So how is the initiator/target
supposed to be absolutely sure that every byte was written to?

> format error or digest error recovery should only involve the affected

-Matt Wakeley

---

From: Santosh Rao <[email-masked]>
Date: Tue, 16 Jan 2001 12:42:47 -0800

Matt,

My replies inline.

Regards,
Santosh

> treating it like a target error.

The LLP (initiator) would return an error to ULP indicating a transport error
(service response of "service delivery or target failure") occurred. [due to the
data underrun.]

> supposed to be absolutely sure that every byte was written to?

The initiator would use a check along the following lines  :

(total bytes xfer'ed as indicated by the chip ) =
(no. of bytes of data xfer specified by ULP) - (resid reported by the target in
SCSI Response PDU).

to verify that all the data the target sent is accounted for at the initiator
end.

---

From: Matt Wakeley <[email-masked]>
Date: Tue, 16 Jan 2001 14:17:59 -0800

> to verify that all the data the target sent is accounted for at the initiator

I think you need to use a different word than score boarding.

Score boarding means that you have a big "map" that describes every byte of
data you want to receive.  As each byte is received, you "check off" the
indicator for that byte that indicates that it was received.  When all the
indicators are checked off, you have all the data.  Score boarding is not a
mathematical calculation.

-Matt

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 16 Jan 2001 14:55:02 -0800

Santosh,

Do not just use byte counting to validate delivery as there may be overlap.
The word SHOULD and a real world of not all devices sending data according
to this SHOULD... requires that you MUST not assume data is consecutive and
not overlapping.  The non-consecutive nature may cause problems if you
simply overlay data digests rather than stripping it off before placement.
The problem with the overlap is obvious if you expect a byte count total to
have meaning.

From revision 3:
  "An R2T MAY be answered with one or more iSCSI Data-out PDU with a
   matching Target Transfer Tag. If an R2T is answered with a single
   Data PDU the Buffer Offset in the Data PDU MUST be the same as the
   one specified by the R2T and the data length of the Data PDU must not
   exceed the Desired Data Length specified in R2T. If the R2T is
   answered with a sequence of Data PDUs the Buffer Offset and Length
   must be within the range of those specified by R2T, the last PDU
   should have the F bit set to 1, the Buffer Offsets and Lengths for
   consecutive PDUs SHOULD form a continuous non-overlapping range and
   the PDUs should be sent in increasing offset order."

Doug