Subject: iSCSI error recovery

From: [email-masked]
Date: Fri, 2 Feb 2001 09:14:59 -0500

There's been a lot of discussion on the list about this
topic, including the proper use of CmdSN and StatSN
to respond to errors including those detected by CRC.
I hesitate to call consensus on any of this discussion
because it's based on a version of the draft (-03) that
is some combination of incomplete, incorrect, and/or
difficult to comprehend in this area.  Based on this
and the length/detail of the discussion, I suspect
that many people on the list are not completely aware
of the issues here, and calling consensus when a large
segment of the WG may not understand the issues is
unlikely to be productive.

The next version of the draft (-04) SHOULD contain a
much better section on error recovery with explanations
and examples that make it significantly easier to
understand.  Consensus calls on error recovery will not
be made before it appears.

Meanwhile, a few comments on ongoing issues:
- If a CRC error occurs, trying to use any data covered
	by that CRC is generally not a good idea in the
	absence of other measures (e.g., an error-correcting
	code covering the field(s) of interest).
- Given some of the CRC discussion, I think the conclusion
	in Orlando to have separate header and data CRCs is
	NOT the rough consensus of the WG.  We need to go
	back to a requirements discussion on this rather than
	debating exactly where to put the CRCs.  Would those
	envisioning middleboxes/gateways/etc. that would
	benefit from this sort of CRC separation please post
	short use cases/descriptions indicating the basic
	functionality of the box and which fields it needs
	to change (let's do this with reference to the header
	layout in -03 rather than subsequent changes)?  As
	part of the use case/description, please explain
	how/why Fibre Channel's single CRC covering both
	the frame header and data causes problems/difficulties.
- Mandating the use of IPsec or TLS solely to handle
	CRC-level integrity issues is overkill.

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 2 Feb 2001 09:01:16 -0800

> 	the frame header and data causes problems/difficulties.

With a single CRC covering both the frame header and payload, when a bad
frame is thrown away which is the only one in a sequence, the only way for
detecting the missing frame is timeout.  With separate CRCs for header and
payload, if the header CRC is still good, one can quickly inform the sender
about the error.  However, for iSCSI, the need for separate CRCs is not so
obvious in this context.  This is because the TCP header is still valid.
Hence, without relying on timeout, one can use on the TCP header to identify
the sender.  Needless to say, any attempt to use the contents covered by the
bad CRC is a cardinal sin for folks in storage industry.

Y.P. Cheng, CTO, ConnectCom Solutions Corp.

---

From: Santosh Rao <[email-masked]>
Date: Fri, 02 Feb 2001 11:28:27 -0800

YP,

Y P Cheng wrote:

> detecting the missing frame is timeout.

Not true for a Data PDU. An I/O timeout only occurs when the Status PDU is
discarded due to a CRC error [or format error]. Discarding a Data PDU [due to a
CRC error or format error] results in I/O underrun, [a terminology used to
describe initiator receiving less data than sent by target] and can be detected
by a count check . [using a DataSN based count as described by Julian in his
last clarification on this issue.]

If the Data PDU also contained the Status within, then discarding such a PDU
does cause an I/O timeout.

Regards,
Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 2 Feb 2001 12:00:03 -0800

David,

Although further refinement of how the initiator is expected to use or not
use StatSN will not modify complexity involved in recovering from a digest
error after some undefined timeout of expected status.  The initiator is to
guess a sequence hole belongs to some pending SCSI tag?  It is to decide if
it should apply a CmdSN based on this lack of status for a retry?  Is this
complexity is to allow non-sequential processing of commands irrespective of
delivery failures?  As you can see, I remain skeptical.

With respect to CRC or Checksums, you could also include a finer subdivision
than just header and data.  To be more explicit about potential options,
there could be static and dynamic fields where to assist handshakes being
debated, a separate checksum for these handshakes would have advantages.
You could then add routing information as yet another field if this
information is to be modified by middle tier equipment.  And finally, the
tag, CDB and related data and associated vectors.  So from my perspective
here is a breakdown:

1) Dynamic fields regarding handshakes. (Hopefully obsoleting retry scheme.)
2) Routing fields (LUN).
3) Static fields (All else).
4) SCSI Cmnd (attributes), CDB, Data.

Where would you wish to see a separation between 1:2, 2:3, or 3:4 or a
single Checksum?

Doug

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 2 Feb 2001 14:04:09 -0800

> > frame is thrown away which is the only one in a sequence, the
> > only way for detecting the missing frame is timeout.
>
> discarded due to a CRC error [or format error]. Discarding a Data
> PDU [due to a CRC error or format error] results in I/O underrun,
> [a terminology used to describe initiator receiving less data than
> sent by target] and can be detected by a count check .
> [using a DataSN based count as described by Julian in his
> If the Data PDU also contained the Status within, then discarding
> such a PDU does cause an I/O timeout.
> Santosh

Lets try a little more specific:

1. Bad command PDU    Thrown away by target     Initiator Timeout
2. Bad write data     Thrown away by target     Target Timeout
3. Bad read data      Thrown away by initiator  Initiator data underrun
4. Bad status PDU     Thrown away by initiator  Initiator Timeout

In case 1, in receiving another command frame, the target detects a missing
CmdSN and decides to wait for a while and then asks initiator to resend.

In case 2, the target can detect a missing DataSN except the very last one.

Case 3 is the specific case you are refrering to.

In case 4, in receiving another status frame, the initiator detects a
missing StatSN and decides to wait for a while and then asks target to
resend.

Y.P.