Subject: [Ips] Updated text for MaxOutstandingUnexpectedPDUs for iSER

From: Mike Ko <[email-masked]>
Date: Thu, 11 Nov 2004 21:25:27 -0800

After thinking through the potential problem with overuse of the 
unidirectional NOP-out PDUs, I concluded that the situation where the 
initiator can send twice MaxOutstandUnexpectedPDUs cannot happen.  Even if 
the initiator abuses the use of the unidirectional NOP-out PDUs, it can 
only send MaxOutstandUnexpectedPDUs, but not twice the number.  The 
reasoning is as follows: 

For the unidirectional NOP-out, the CmdSN used is the next CmdSN and the 
variable is not advanced when the PDU is sent, and neither is ExpCmdSN. So 
for the target to respond with a PDU where ExpCmdSN is x+1 where x is 
CmdSN of the NOP-out PDU, it must be acknowledging a non-immediate PDU 
also with CmdSN = x.  This must be sent after all the NOP-outs with CmdSN 
= x have been sent, since this non-immediate PDU will advance CmdSN. 
Assuming the target has processed all the NOP-outs with the immediate flag 
set before the non-immediate PDU that follows, the receive buffers should 
be replenished.  So even if the initiator sends a huge number of NOP-outs 
later after receiving a PDU from the target with ExpCmdSN = x+1, it will 
be using CmdSN = x+1 for those NOP-outs and there is no ambiguity on the 
CmdSN that the ExpCmdSN is acknowledging.  So if the target has provisions 
for MaxOutstandUnexpectedPDUs number of receive buffers available, it 
should be able to handle the unexpected NOP-out PDUs.

For StatSN, the reasoning is similar since StatSN in the NOP-In is 
interpreted as the next StatSN. 

The following is the updated text for the new key, 
MaxOutstandingUnexpectedPDUs.  Note that the default for the key is TBD as 
David Black has suggested in the 61st IETF meeting.  The argument for the 
default being "None" is that it is the preferred value if the 
implementation uses Shared RQ, supports graceful handling, or can 
replenish receive buffers at wire speed.  Please respond if you have a 
suggestion for a different default value.

Mike
--------------  start of new key definition ---------------
Use: LO (leading only), Declarative

Senders: Initiator and Target

Scope: SW (session-wide)

Irrelevant when: RDMAExtensions=No

MaxOutstandingUnexpectedPDUs=<numerical-value-from-0-to-(2**64-1) | None>

Default is TBD

This key is used by the initiator and the target to declare the maximum 
number
of outstanding "unexpected" control-type PDUs that it can receive.  It is
intended to allow the receiving side to determine the amount of buffer
resources needed beyond the normal flow control mechanism available in 
iSCSI.
An initiator or target should select a value such that it would not impose 
an
unnecessary constraint on the iSCSI layer under normal circumstances.  If 
the
sending side fails to adhere to the declared limit as set by the receiving
side, it is up to the receiving side to determine ways of handling the 
overrun,
including dropping the connection.

The control-type PDUs that can be sent by an initiator to a target can be
grouped into the following categories:

1. Regulated:  Control-type PDUs in this category are regulated by the 
iSCSI
   CmdSN window mechanism and the immediate flag is not set.

2. Unregulated but Expected:  Control-type PDUs in this category are not
   regulated by the iSCSI CmdSN window mechanism but are expected by the
   target.

3. Unregulated and Unexpected:  Control-type PDUs in this category are not
   regulated by the iSCSI CmdSN window mechanism and are "unexpected" by 
the
   target.

For the control-type PDUs in the Regulated category, the queuing capacity
required of the iSCSI layer at the target is described in section 3.2.2.1 
of
[RFC3720].  For each of these control-type PDUs sent by the initiator, the
initiator MUST provision for the buffer resources required for the
corresponding control-type PDU to be returned from the target.  The 
following
is a list of the PDUs that can be sent by the initiator and the PDUs that 
are
sent by the target in response:

a. When an initiator sends a SCSI Command PDU, it expects a SCSI Response 
PDU
   from the target.  Alternatively, the target can respond with the Reject 
PDU
   before the task is active.

b. When the initiator sends a Task Management Function Request PDU, it 
expects
   a Task Management Function Response PDU from the target.

c. When the initiator sends a Text Request PDU, it expects a Text Response 
PDU
   from the target.

d. When the initiator sends a Login Request PDU, it expects a Login 
Response
   PDU from the target.

e. When the initiator sends a Logout Request PDU, it expects a Logout 
Response
   PDU from the target.

f. When the initiator sends a NOP-out PDU as a ping request with ITT /=
   0xffffffff and TTT = 0xffffffff, it expects a NOP-in PDU from the 
target
   with the same ITT and TTT as in the ping request.

For the control-type PDUs in the Unregulated but Expected category, the 
amount
of buffering resources required at the target can be predetermined.  The
following is a list of the PDUs in this category:

a. SCSI Data-out PDUs are used by the initiator to send unsolicited data. 
The
   amount of buffer resources required by the target can be determined 
using
   FirstBurstLength.  Note that SCSI Data-out PDUs are not used for 
solicited
   data since the R2T PDU which is used for solicitation is transformed 
into
   RDMA Read operations by the iSER layer at the target.  See section 
7.3.4.

b. A NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff is sent as a 
ping
   response by the initiator to the NOP-in PDU sent as a ping request by 
the
   target.

The number of PDUs in the Unregulated and Unexpected category which can be 
sent
by an initiator is controlled by MaxOutstandingUnexpectedPDUs declared by 
the
target.  After a PDU in this category is sent by the initiator, it is
outstanding until it is retired.  At any time, the number of outstanding 
PDUs
MUST not exceed MaxOutstandingUnexpectedPDUs.  The following is a list of 
the
PDUs in this category and the conditions for retiring the outstanding PDU:

a. For the PDUs listed in the Regulated category but with the immediate 
flag
   set, a PDU is outstanding until the target responds with the
   correspdonding response PDU.

b. SNACK PDUs are not needed in iSER (see section 7.3.11).  A SNACK PDU 
which
   is sent by the initiator anyway is outstanding until the target 
responds
   with a SCSI Response PDU for the referenced command.

c. For a NOP-out PDU with ITT = TTT = 0xffffffff, the PDU is outstanding 
until
   the target responds with a control-type PDU with ExpCmdSN set to at 
least
   x+1 where x is the CmdSN of the NOP-out PDU.

Control-type PDUs that can be sent by a target and are expected by the
initiator are listed in the Regulated category.  For the control-type PDUs
that are unexpected by the initiator, the number that can be sent by a 
target
is controlled by MaxOutstandingUnexpectedPDUs declared by the initiator. 
After
a PDU in this category is sent by a target, it is outstanding until it is
retired.  At any time, the number of outstanding PDUs MUST not exceed
MaxOutstandingUnexpectedPDUs.  The following is a list of the PDUs in this
category and the conditions for retiring the outstanding PDU:

a. For an Asynchronous Message PDU with StatSN = x, the PDU is outstanding
   until the initiator sends a control-type PDU with ExpStatSN set to at 
least
   x+1.

b. For a Reject PDU with StatSN = x which is sent after a task is active, 
the
   PDU is outstanding until the initiator sends a control-type PDU with
   ExpStatSN set to at least x+1.

c. For a NOP-in PDU with ITT = TTT = 0xffffffff, the PDU is outstanding 
until
   the initiator responds with a control-type PDU with ExpStatSN set to at
   least x+1 where x is the StatSN of the NOP-in PDU.

d. For a NOP-in PDU sent as a ping request with ITT = 0xffffffff and TTT 
/=
   0xffffffff, the PDU is outstanding until the initiator sends a NOP-out 
PDU
   with the same ITT and TTT as in the ping request.

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: William Studenmund <[email-masked]>
Date: Fri, 12 Nov 2004 12:27:55 -0800

> should be able to handle the unexpected NOP-out PDUs.

Just to make sure I'm understanding things, MaxOutstandUnexpectedPDUs 
limits the total number of un-ack'd PDUs (NOP-OUTs in this case), not 
the number per any specific CmdSN. Correct?

> MaxOutstandingUnexpectedPDUs=<numerical-value-from-0-to-(2**64-1) | 

Why 2**64-1? All the other counting in iSCSI uses 32-bit ints. 
Including the numbers we want to use for status acknowledgement. Thus I 
really don't see how we could need more than 2**32.

Take care,

Bill