Subject: RE: [Ips] Updated text for MaxOutstandingUnexpectedPDUs for iSER

From: [email-masked]
Date: Fri, 12 Nov 2004 16:20:56 -0700

Mike,

I think there is a problem with the assumption that advance of ExpCmdSN can be used to retire outstanding unexpected PDUs. The problem occurs when there are multiple connections in a session.

Description of how ExpCmdSN retires outstanding unexpected PDUs from Mike Ko's message:
"For the unidirectional NOP-out, the CmdSN used is the next CmdSN and the 
variable is not advanced when the PDU is sent, and neither is ExpCmdSN. So 
for the target to respond with a PDU where ExpCmdSN is x+1 where x is 
CmdSN of the NOP-out PDU, it must be acknowledging a non-immediate PDU 
also with CmdSN = x.  This must be sent after all the NOP-outs with CmdSN 
= x have been sent, since this non-immediate PDU will advance CmdSN. 
Assuming the target has processed all the NOP-outs with the immediate flag 
set before the non-immediate PDU that follows, the receive buffers should 
be replenished.  So even if the initiator sends a huge number of NOP-outs 
later after receiving a PDU from the target with ExpCmdSN = x+1, it will 
be using CmdSN = x+1 for those NOP-outs and there is no ambiguity on the 
CmdSN that the ExpCmdSN is acknowledging.  So if the target has provisions 
for MaxOutstandUnexpectedPDUs number of receive buffers available, it 
should be able to handle the unexpected NOP-out PDUs."

Let's look at a session with two connections. Call the connections c1 and c2.

The following are sent from iSCSI to the two connections (cmd(n) indicates a non-immediate command with CmdSN n. imm(n) indicates an immediate command carrying CmdSN n):

time    c1            c2   
 |    cmd(x-1)
 |    imm(x)
 V    imm(x)
      imm(x)
                    cmd(x)

Though all the imm(x) commands are sent by iSCSI before cmd(x), they may arrive at the target in a different order:
time    c1            c2   
 |    cmd(x-1)
 |    imm(x)
 V                  cmd(x)
      imm(x)
      imm(x)

This could happen for many reasons. They could have been delayed by congestion or a need to retransmit due to a lost packet. Perhaps the initiator's RNIC on c1 received a large RDMA  read request before it received some of the commands and is completing the transmission of the RDMA read response before it transmits them.

When the target receives cmd(x) on c2, it will advance ExpCmdSN and it has no way to know that there are outstanding unexpected PDUs. One can really only retire outstanding unexpected PDUs when ExpCmdSN is advanced past a CmdSN that was sent after them on the same connection - something that adds additional complexity to this process.

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Fri, 12 Nov 2004 16:33:09 -0800

Pat,

Note that for immediate PDUs sent by the initiator requiring a response 
from the target, which include all the PDUs in the regulated category 
except that the immediate flags are set, the PDUs are retired (not 
outstanding) when the target responds with the corresponding response PDU. 
 E.g., a Text Request PDU with the immediate flag set is not retired until 
the target responds with a Text Response PDU.  So the only PDU with the 
immediate flag set that can be sent by the initiator that does not require 
a response from the target is the NOP-out PDU with ITT = TTT = 0xffffffff. 
 This is the PDU which will be retired when the target responds with a PDU 
where ExpCmdSN = x+1 where x is the CmdSN of the NOP-out PDU.  The iSCSI 
spec said that "A NOP-Out may also be used to confirm a changed ExpStatSN 
if another PDU will not be available for a long time."  So for the 
initiator to send multiple unidirectional NOP-outs does not quite meet the 
spirit as intended in the iSCSI spec.  However, the iSCSI spec does not 
explicitly prohibit the behavior.

But your point is well taken that for multiple connections, the problem 
that I described in the 61st IETF meeting still exists.  We have several 
alternatives:

1. Require that this NOP-out PDU is retired only when the target responds 
with ExpCmdSN = x+1 or larger on the same connection.

2. Add a cautionary note to warn against the behavior of sending 
undirectional NOP-out floods.

3. Suggest that undirectional NOP-outs not be used to avoid the ambiguity. 
 Note that in the iSER spec, the recommended use for NOP-out is the 
bidirectional type.  See section 7.3.5 and 7.3.6.

I favor option 1, and suggesting option 3 at the same time for the 
implementation to avoid the complexity.

Mike
To:     <[email-masked]>, <[email-masked]>
cc:      
Subject:        RE: [Ips] Updated text for MaxOutstandingUnexpectedPDUs 
for iSER

Mike,

I think there is a problem with the assumption that advance of ExpCmdSN 
can be used to retire outstanding unexpected PDUs. The problem occurs when 
there are multiple connections in a session.

Description of how ExpCmdSN retires outstanding unexpected PDUs from Mike 
Ko's message:

Let's look at a session with two connections. Call the connections c1 and 
c2.

The following are sent from iSCSI to the two connections (cmd(n) indicates 
a non-immediate command with CmdSN n. imm(n) indicates an immediate 
command carrying CmdSN n):

time    c1            c2
|    cmd(x-1)
|    imm(x)
V    imm(x)
imm(x)
cmd(x)

Though all the imm(x) commands are sent by iSCSI before cmd(x), they may 
arrive at the target in a different order:
time    c1            c2
|    cmd(x-1)
|    imm(x)
V                  cmd(x)
imm(x)
imm(x)

This could happen for many reasons. They could have been delayed by 
congestion or a need to retransmit due to a lost packet. Perhaps the 
initiator's RNIC on c1 received a large RDMA  read request before it 
received some of the commands and is completing the transmission of the 
RDMA read response before it transmits them.

When the target receives cmd(x) on c2, it will advance ExpCmdSN and it has 
no way to know that there are outstanding unexpected PDUs. One can really 
only retire outstanding unexpected PDUs when ExpCmdSN is advanced past a 
CmdSN that was sent after them on the same connection - something that 
adds additional complexity to this process.