Subject: iSCSI: data ACKs

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 11 Oct 2001 15:13:35 -0700

Julian,

Consequent to the London consensus on keeping SNACKs in 
the protocol, I believe it is fair to enable targets desirous
of efficiently managing their buffers, by way of a data
acknowledgement request-response mechanism in iSCSI (under
certain constraints, as below) as earlier requested by
Matthew Burbridge on this list.  The current solution of 
allowing targets to re-access the medium to satisfy data 
transfer retransmission presents problems in the following 
cases -
        - any target in general that attempts to prevent
          reaccessing the medium in view of cache management 
          complexities, or wanting to conserve backend SCSI 
          bandwidth (only if the error rate is high).
	- tapes supporting queued commands, that want to 
          free up memory to work on subsequent commands.
	- iSCSI "middle boxes" supporting multiple tape 
	  devices behind, and wanting to decrease buffer
          requirements.
	- the SCSI-legitimate (but perhaps highly unlikely) 
          case of write-after-read to the same LBAs.

I propose the following means to enable a data ACK mechanism
	- target requests an ACK by setting a TBD bit
          (say A-bit, for ACK) on read data PDU.  It MAY
          set the A-bit once at most every MaxBurstSize 
          bytes, and MUST NOT do so any more frequently 
          than that.
        - an initiator generates a SNACK of type "ACK" (per
          Julian's earlier proposal) on reception of an 
          A-bit data PDU.  If there were pending data 
          retransmissions in the burst, the generation of 
          ACK is deferred till the holes are filled.
        - target processes the SNACK and frees up the 
          acknowledged buffers. 

The reason a new A-bit is proposed to be used (in stead of 
the F-bit) is that there may be unrelated reasons for setting 
the F-bit by a target (may be to enable opposite data 
flow for a bidirectional command, for ex.).

I believe that the implementation complexity in the above 
proposal is minimal, while the runtime cost is non-existent
for all "short" (< MaxBurstSize) transfers, and the same
for "long" transfers (> MaxBurstSize) is very reasonable.
You will note that the onus is on targets to generate ACK
requests and process ACKs, if they want to manage their 
buffers efficiently.  Finally, the ack scheme should be 
applicable only to sessions where the operational 
ErrorRecoveryLevel >= 1.

Comments?
-- 
Mallikarjun 

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668 Hewlett-Packard, Roseville.
[email-masked]

---

From: "Julian Satran" <[email-masked]>
Date: Sat, 13 Oct 2001 07:10:04 +0300

Excellent. Here are the proposed changes:
(See attached file: Changes-Data-ACK.txt)
Comments?

Julo

                    "Mallikarjun                                                              
                    C."                  To:     ips <[email-masked]>                        
                    <[email-masked]       cc:                                                  
                    om>                  Subject:     iSCSI: data ACKs                        
                    Sent by:                                                                  
                    owner-ips@ece.                                                            
                    cmu.edu                                                                   

                    12-10-01 00:13                                                            
                    Please respond                                                            
                    to cbm                                                                    

Julian,

Comments?
--
Mallikarjun

Mallikarjun Chadalapaka

Changes for Data ACK
______________________

The SCSI Data-in PDU for READ operations has the following format:

The SCSI Data-in PDU for READ operations has the following format:

Byte /    0       |       1       |       2       |       3       |
   /              |               |               |               |
  |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
  +---------------+---------------+---------------+---------------+
 0|1|1| 0x25      |F|A|0 0 0|O|U|S| Reserved      |Status or Rsvd |
  +---------------+---------------+---------------+---------------+
 4| Reserved      | DataSegmentLength                             |
  +---------------+---------------+---------------+---------------+
 8| Reserved                                                      |
  +                                                               +
12|                                                               |
  +---------------+---------------+---------------+---------------+
16| Initiator Task Tag                                            |
  +---------------+---------------+---------------+---------------+
20| Residual Count                                                |
  +---------------+---------------+---------------+---------------+
24| StatSN or Reserved                                            |
  +---------------+---------------+---------------+---------------+
28| ExpCmdSN                                                      |
  +---------------+---------------+---------------+---------------+
32| MaxCmdSN                                                      |
  +---------------+---------------+---------------+---------------+
36| DataSN                                                        |
  +---------------+---------------+---------------+---------------+
40| Buffer Offset                                                 |
  +---------------+---------------+---------------+---------------+
44| Reserved                                                      |
  +---------------+---------------+---------------+---------------+
48| Header Digest (if any)                                        |
  +---------------+---------------+---------------+---------------+
  / DataSegment (and digest if any)                               /
 +/                                                               /
  +---------------+---------------+---------------+---------------+

Status can accompany the last Data-in PDU if the command did not end with an exception.  Presence of status (and of a residual count) is signaled though the S flag bit.  Although targets MAY choose to send even non-exception status in separate responses initiators MUST support non-exception status in Data-In PDUs.

3.7.1 F (Final) Bit

For outgoing data, this bit is 1 for the last PDU of unsolicited data or the last PDU of a sequence answering an R2T.

For incoming data, this bit is 1 for the last input (read) data PDU of a sequence.  Input can be split in several sequences each one having it's own F bit. Splitting the data stream in sequences does not affect DataSN counting on Data-In PDUs. It MAY be used as a "change direction" indication for Bidirectional operations that need such a change.  A target that implements ErrorRecoveryLevel 1 or higher MUST use the F bit to indicate the end-of-sequence of Data-In PDUs is it is going to discard.

For Bidirectional operations, the F bit is 1 both for the end of the input sequences as well as the end of the output sequences. 

3.7.2 A (Acknowledge) bit

For sessions with ErrorRecoveryLevel 1 or higher the target sets this bit to 1 to indicate that it requests from the initiator a positive acknowledgement for the data received.  The target MAY set the A-bit to 1 once at most every MaxBurstSize bytes, and MUST NOT do so any more frequently than that.

On receiving a Data-In PDU with the A bit set to 1 the initiator MUST issued a SNACK of type DataACK.  If the initiator has detected holes in the input sequence, it MAY postpone issuing the SNACK of type ACKN until the holes are filled.

------------------------------

3.16  SNACK Request

Byte /    0       |       1       |       2       |       3       |
   /              |               |               |               |
  |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|
  +---------------+---------------+---------------+---------------+
 0|0|1| 0x10      |1|Rsrvd| Type  | Reserved                      |
  +---------------+---------------+---------------+---------------+
 4/ Reserved                                                      /
 +/                                                               /
  +---------------+---------------+---------------+---------------+
16| Initiator Task Tag or 0xffffffff                              |
  +---------------+---------------+---------------+---------------+
20| BegRun                                                        |
  +---------------+---------------+---------------+---------------+
24| RunLength                                                     |
  +---------------+---------------+---------------+---------------+
28| ExpStatSN                                                     |
  +---------------+---------------+---------------+---------------+
32| Reserved                                                      |
  +---------------+---------------+---------------+---------------+
36| ExpDataSN or Reserved                                         |
  +---------------+---------------+---------------+---------------+
32/ Reserved                                                      /
 +/                                                               /
  +---------------+---------------+---------------+---------------+
48

Support for SNACK is optional.

SNACK request is used to request retransmission of numbered-responses, data or R2T PDUs from the target.  The SNACK request indicates to the target the missed numbered-response or data run, where the run is composed of an initial missed StatSN, DataSN or R2TSN and the number of additional missed Status, Data or R2T PDUs (0 means only the initial).

The numbered-response or R2T requested by a SNACK have to be delivered as exact replicas of the ones the initiator missed including all its flags. 

The numbered Data-In PDUs individually requested by a SNACK have to be delivered as exact replicas of the ones the initiator missed including all its flags.  Data-In PDUs requested with RunLength 0 (meaning all after this number) may be different from the ones originally sent in order to reflect changes in MaxRecvPDULength. 

Any SNACK requesting a numbered-response, Data or R2T that was not sent by the target MUST be rejected with a reason code of "Invalid SNACK".

3.16.1 Type

This field encodes the SNACK function as follows:

0-Data/R2T SNACK - requesting retransmission of a Data-In or R2T PDU
1-Status SNACK - requesting retransmission of a numbered response
2-DataACK - positively acknowledges Data-In PDUs 

All other values are reserved.

Data/R2T SNACK for a command MUST precede status acknowledgement for the given command.

For a Data/R2T SNACK the Initiator Task Tag MUST be set to the Initiator Task Tag of the referenced Command. Otherwise, it is reserved.

For a Status SNACK the ExpDataSN field is reserved.

An iSCSI target that does not support recovery within connection MAY discard status SNACK. If the target supports command recovery within session it MAY discard the SNACK after which it MUST issue an Asynchronous Message PDU with an iSCSI event indicating "Request Logout".

If an initiator operates at ErrorRecoveryLevel 1 or higher it MUST issue a SNACK of type DataACK after receiving a Data-In PDU with the A bit set to 1.  However if the initiator has detected holes in the input sequence, it MAY postpone issuing the SNACK of type DataACK until the holes are filled. The DataACK is used to free resources at target and not to request or imply data retransmission.