Subject: Re: [Ips] iSER flow control options

From: "Alex Nezhinsky" <[email-masked]>
Date: Tue, 12 Oct 2004 13:55:13 +0200

Hi, all
Reading through the last messages in the "flow control" thread was quite
helpful, but I still have a few questions.

[Caitlin Bestler said> Shared Receive Queues are relevant here on two
fronts. First, iSCSI flow control is really more session oriented than
connection oriented. Pre-posting separate buffers to receive Cmd X on
*each* connection is clearly over-buffering. This may be valid, because
memory is cheap, but clearly it is not required to pre-commit duplicate
buffers for each connection in a session. <Caitlin Bestler]

This has alerted me with respect to the suggested credit update scheme: 
[Mike Ko said> One possible solution is as follows: For the initiator,
the ExpCmdSN from the target will be used to advance the credit... <Mike
Ko]

Such credit update scheme implies that iSER DataMover manages the
session context (connection-session bindings, and the state variables
incl. CmdSN,ExpCmdSN,MaxCmdSN). Attempting not to overcommit buffers
when posting based on MaxCmdSN brings up the same requirement. 
This information is routinely maintained by the iSCSI layer. There is no
API defined between the iSCSI and DataMover, which can tell a DataMover
about the session context information. If I understand it right, iSER as
a DataMover was intended to operate solely on connections without
bothering about sessions. Not only recovering session-scope information
from the passing PDUs is actually a duplication of the job already done
by iSCSI. iSER is not supposed to see the Login phase (where the session
ids and handles are exchanged) at all. Of course, connection-session
binding may be passed "somehow" (e.g. as faked keys of
Notice_Key_Values), but I guess that this wasn't the original intention.

Did I overlook anything? 
 
>From what has been said previously on similar issues, I can see two
replies to the above problem (if it is a problem):
1) Use S-RQ and all such problems are gone
2) At least in target, DataMover does not have to be "that" separated
from the iSCSI layer.

Well, I'll try to present our implementation issues, they may be of
value also as a "yet another use case" for iSER.

We are building a storage gateway between IB and FC. The current
incarnation is based on iSER. 
We have an operational (buffer management is the major remaining issue)
iSER DataMover linux kernel module, where iSER is built on top of IB
rather than iWARP. It is built as a DataMover "service provider",
capable to serve concurrently a number of iSCSI entities incl. initiator
and/or target-type. Such capability is useful as it enables us to have
the same code for both initiator and target platforms, streamlines
debugging and can enable easy extention to a router-like environment
where the same box acts both as initiator and target through different
interfaces.
We are in process of integration with the open-source linux-iscsi
initiator project, which is to act as an iSCSI client of the iSER
DataMover. 
I got into such level of detalization to stress that we already did not
opt for intimate integration between the iSER/iSCSI layers, but rather
tried to keep a reasonable level of separation between them. 

Being conscious about the ever-changing environment of RDMA transports,
we build iSER DataMover on top of DAPL, as DAT is more general than any
specific implementation of IB, iWARP etc. So we did know about the S-RQ
design option, it's only that it is still not fully supported by the
specs issued by the DAT collaborative. Though it surely will be
supported in the (hopefully near) future, I guess this is an example of
implementation which can't be based on S-RQ, at least for a time being.
I'd note also that nothing either in the DataMover Architecture or iSER
specs has warned us that the design should be based upon S-RQ, just as
noted by  [Mallikarjun C. said> I really would like us to view S-RQ as a
"dampener" in invoking #2 (graceful handling), not as a mandatory
feature that iSER requires - because it doesn't (though I as an
implementer would require S-RQ in my implementation to realize the
dampening).<Mallikarjun C.]

So it is currently a non S-RQ based implementation. As such, we are
running into a problem, how to manage the buffer credits (along the
lines of the latest discussion, plus see my question above).
Another, more subtle one is that the resulting design seems to differ
substantially from what would be necessary with a S-RQ implementation.
The fact that using a S-RQ can't be extended easily from a non S-RQ
implementation strikes me as non-elegant, to say the least.

I'd appreciate your comments and suggestions.

Alexander Nezhinsky

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Tue, 12 Oct 2004 10:50:38 -0700

Alex,

In response to your statement that "such credit update scheme implies that 
iSER DataMover manages the session context (connection-session bindings, 
and the state variables incl. CmdSN, ExpCmdSN, MaxCmdSN), bear in mind 
that it is the iSCSI layer that manages CmdSN, ExpCmdSN, MaxCmdSN, etc., 
not the iSER layer.  On the target side, for cases where the immediate 
flag is not set, the iSER layer must provision sufficient number of 
receive buffers to support the MaxCmdSN advertised by the iSCSI layer at 
the target.  For multiple connections per session, how the iSER layer at 
each connection handle the number of receive buffers required per 
connection is implementation dependent.  An SRQ can be used, but not 
required.  Similarly, on the initiator side, the iSER layer can determine 
the number of receive buffers required since most control-type PDUs sent 
by the target are responses to requests sent by the initiator side. 
Therefore the flow control issue being debated here refers to those PDUs 
which are not regulated by the CmdSN flow.

On the various schemes summarized by Mallikarjun, I agree with the general 
sentiment that option #3 (declared limit) should work, though it might not 
be the first choice for some.  However, by setting the declared limit to a 
high enough value (or "None"), the limit essentailly goes away and 
implementations that can support option #1 (iSER/iSCSI layer being able to 
replenish the buffers as fast as the packet arrival rate) or #2 (RNIC 
being able to support graceful handling) would not be impeded by the 
declared limit.  The key is to have the limit being declared instead of a 
stated value in the protocol (option #3a) to allow the implementation the 
flexibility, and to use existing iSCSI pacing mechanisms to advance the 
"credits" in order to avoid introducing yet another credit scheme into the 
iSER layer.

The use of the declared limit has been discussed last year and I have 
summarized the main points in my note of October 4 and 5.  To recap, from 
the initiator to the target, control-type PDUs with the immediate flag set 
and SNACK request are not regulated by the CmdSN mechanism.  (SNACKs are 
not used in iSER.)  From the target to the initiator, Async Messages and 
unsolicited NOP-in are sent not as a response to a request from the 
initiator.  These are the unregulated flows which fall under the declared 
limit.  I propose that we consider the following scheme to advance the 
credits set by the declared limit.  ExpCmdSN andExpStatSN are used as 
pacing mechanisms.

For the initiator, the ExpCmdSN from the target will be used to advance 
the credit.  After sending PDUs with the immediate flag set, when the 
initiator sends any iSCSI control-type PDU with a new CmdSN without the
immediate flag, if the target responds with an iSCSI control-type PDU with 
ExpCmdSN set to CmdSN + 1, then credit at the initiator is restored to its 
full value. 

For the target, the ExpStatSN from the initiator will be used to advance 
the credit.  For unsolicited PDUs sent by the target, if the StatSN of 
that PDU is acknowledged by ExpStatSN, then the credit is restored.  

Comments?

Mike
Sent by:        [email-masked]
To:     <[email-masked]>
cc:      
Subject:        Re: [Ips] iSER flow control options

Hi, all

Did I overlook anything?

Alexander Nezhinsky

---

From: John Hufferd <[email-masked]>
Date: Tue, 12 Oct 2004 11:57:28 -0700

I agree with both Mallikarjun, and Mike.  That is, my company will 
probably require vendors that OEM their RNIC to us, to provide "Graceful 
handling", and I expect that we and others will also exploit the value 
retained with non restricted iSCSI flow control and therefore will require 
the use of SRQs.  However, as I said in my Oct 6, 2004 message I believe 
that there can be an additional declared parameter (the Keyword=Value pair 
"MaxOutstandingUnexpectedPDUs") which can take on values of 1-?????, 
including "None" (which means non restricted  -- to be used in 
unrestricted iSCSI flow control).  This Keyword=Value pair gives non 
shared RQ implementations, and InfiniBand implementations, which do not 
yet have SRQs in their HW,  a reasonable way to be compatible with each 
other, yet not inhibit the creative implementation that can use the 
unrestricted iSCSI flow control. (Note: SRQ is now in the IB 
specifications.)
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Mike Ko <[email-masked]> 
Sent by: [email-masked]
10/12/2004 10:50 AM

To
"Alex Nezhinsky" <[email-masked]>
cc
[email-masked]
Subject
Re: [Ips] iSER flow control options

Alex,

iSER layer.

full value. 

Comments?

Mike

Hi, all

Did I overlook anything?

Alexander Nezhinsky