Subject: RE: [Ips] DA+iSER Flow Control, Limiting number of oustanding cmds attarget

From: "Alex Nezhinsky" <[email-masked]>
Date: Tue, 5 Oct 2004 17:04:56 +0200

> -----Original Message-----
> From: Mike Ko [mailto:[email-masked]] 
> Sent: Monday, October 04, 2004 9:00 PM
> To: Alex Nezhinsky
> Cc: Julian Satran; Caitlin Bestler; Mallikarjun C.; [email-masked]
> Subject: RE: [Ips] DA+iSER Flow Control, Limiting number of 
> oustanding cmds attarget

> From the target to the initiator, unsocited NOP-in are regulated by
NOP-out. 

I don't quite understand this statement. NOP-INs, which are unsolicited
can't be regulated by their replies. Do you mean that there can be only
one outstanding NOP-IN at a time? This would clearly solve the problem,
but I could not find such limitation in iSCSI specification. For
example, 10.4.11 says: "When MaxCmdSN changes at the target while the
target has no pending PDUs to convey this information to the initiator,
it MUST generate a NOP-IN to carry the new MaxCmdSN". This seems to
imply a possibility of sending a few NOP-INs without waiting for a
response NOP-OUT, because some resources may become free gradually.

> The unsolicited PDUs which are not covered by any existing flow
control are Async Messages.
> One of the solution proposed is to set a declared limit on the number
of such unsolicited requests.

Is it going to be declared in iSER spec as a const., as a
negotiable/declarative parameter?
Is it going to be declared in iSER spec at all, or rather left to
implementations?
Does this limit cover all the PDU types involved or is declared per
type?

> The next question is how does each side determine when is the 
> limit replenished.  One possible solution is as follows:
> 
> For the initiator, the ExpCmdSN from the target will be used 
> to advance the credit.  After sending PDUs with the immediate 
> flag set, when the initiator sends any iSCSI control-type PDU 
> with a new CmdSN without the immediate flag, if the target 
> responds with an iSCSI control-type PDU with ExpCmdSN set to 
> CmdSN + 1, then credit at the initiator is restored to its full value.
> 
> For the target, the ExpStatSN from the initiator will be used 
> to advance the credit.  When the initiator sends any PDU 
> which contains the ExpStatSN that acknowledges the StatSN in 
> the Async Message, the number of credits at the target used 
> up by that Async Message will be restored.

Sure, this works. These schemes, though, will force iSER DataMover into
a complex state tracking (just try to visualize the credit update
condition and the internal state varaibles involved in it). If a
separate credit counter is implemented, it only has to be updated from
any incoming PDU's header and checked+decremented upon every relevant
send opration.

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Tue, 5 Oct 2004 09:36:51 -0700

Alex,

My comments are in line preceded by [mk].

Mike
To:     "Mike Ko" <[email-masked]>
cc:     "Julian Satran" <[email-masked]>, "Caitlin Bestler" 
<[email-masked]>, "Mallikarjun C." <[email-masked]>, <[email-masked]> 
Subject:        RE: [Ips] DA+iSER Flow Control, Limiting number of 
oustanding cmds attarget

NOP-out.

[mk]
More correctly, I should state that the number of unsolicited requests 
from the target are regulated by the declared limit.  The credit 
replenishment rule (using ExpStatSN for example) applies in the general 
case.  However, for each NOP-in sent by the target which expects a 
corresponding NOP-out (TTT is set to a valid value), then when the 
initiator responds with the NOP-out, the iSER layer can use this to 
replenish the receive buffer. 
[mk]

[mk]
The intent is to have the initiator and the target declare to each other 
during the iSCSI Login the maximum number of unsolicited requests (as 
specified in my earlier e-mail) that the other side can send.
[mk]

Sure, this works. These schemes, though, will force iSER DataMover into
a complex state tracking (just try to visualize the credit update
condition and the internal state varaibles involved in it).

[mk]
I agree.  That's why John Hufferd suggesting using the Shared Receive 
Queue instead. 
[mk]

If a separate credit counter is implemented, it only has to be updated 
from
any incoming PDU's header and checked+decremented upon every relevant
send opration.

[mk]
Why would a separate credit counter create less "complex state tracking" 
problem than the one that relies on ExpCmdSN (or ExpStatSN)?
[mk]