Subject: RE: [Ips] DA+iSER Flow Control, Limiting number of oustanding cmds at target

From: "Alex Nezhinsky" <[email-masked]>
Date: Sun, 3 Oct 2004 19:21:06 +0200

Mallikarjun, Mike, Julo,

Thanx for clarifiaction. Actually, I thought about the receive buffers
as connection-level resources, which must be allocated beforehand,
during Allocate_Connection_Resources, and  posted subsequently. 
I thought that treating these buffers as task-level resources opens a
possibility that their allocation fails at a later stage and the entire
connection may be closed. But of course, in this case the concerns that
I tried to raise are insubstantial.

Returning to the flow control issue -- a possible scheme for posting
receive buffers may be the following one. Each time the initiator send a
request PDU, it posts a receive buffer for the response (before
sending). Each time the target sends a PDU with an increased value of
MaxCmdSN, it posts enough recv buffers to accommodate with the longest
incoming command-related PDU sequence for the advertized CmdSN delta.
This addresses only the "normal path" PDUs.
Immediate commands and NOP-outs from initiator, NOP-Ins and AsyncMsgs
from the target are outside of the flow control offered by MaxCmdSN.
What is a feasible provision for the out-of band PDUs like these? In the
latest "Flow Ctrl" thread I saw remarks about a past discussion, where a
suggestion to limit the number of simultaneously issued PDUs of these
types had been dismissed. I could not find this discussion. Can you
shortly reproduce the reasoning and explain how do you see a possible
implementation? 

Alexander Nezhinsky.

> -----Original Message-----
> From: Mallikarjun C. [mailto:[email-masked]] 
> Sent: Wednesday, September 22, 2004 11:00 PM
> To: Alex Nezhinsky
> Cc: [email-masked]
> Subject: Re: [Ips] DA+iSER Flow Control,Limiting number of 
> oustanding cmds at target
> 
> Alex,
> 
>  > The only reason may be that the initiator does not know 
> the upper bound  > when it executes Allocate_Connection_Resources.
> 
> It appears that your concern is wrt sizing of per-task buffer 
> resources on the invocation of Allocate_Connection_Resources.
> 
> Note that section 9.4 in DA says the following:
> 
> "Note
> that the Datamover layer however does not allocate any 
> Datamover-specific task-level resources upon invocation of 
> this Primitive."
> 
> The task buffers are thus not related to 
> Allocate_Connection_Resources. 
>   The buffer management is specific to implementations.  DA 
> only specifies the architectural primitives that are 
> minimally required to allow the definition of a Datamover 
> protocol, but not everything for an implementation.
> 
> Thanks.
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions
> Hewlett-Packard MS 5668
> Roseville CA 95747
> cbm [at] rose.hp.com
> 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Julian Satran <[email-masked]>
Date: Mon, 4 Oct 2004 10:35:12 +0200

Alex,

The relevant text in RFC 3720 - 3.2.2.1 is:

  The number of commands used for immediate delivery is not limited and
   their delivery for execution is not acknowledged through the
   numbering scheme.  Immediate commands MAY be rejected by the iSCSI
   target layer due to a lack of resources.  An iSCSI target MUST be
   able to handle at least one immediate task management command and one
   immediate non-task-management iSCSI command per connection at any
   time.

I hope this helps.

Hag Sameach,
Julo

"Alex Nezhinsky" <[email-masked]> 
Sent by: [email-masked]
03/10/04 19:21

To
"Mallikarjun C." <[email-masked]>, "Mike Ko" <[email-masked]>, 
Julian Satran/Haifa/IBM@IBMIL
cc
[email-masked]
Subject
RE: [Ips] DA+iSER Flow Control, Limiting number of oustanding cmds at 
target

Mallikarjun, Mike, Julo,

Alexander Nezhinsky.

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 4 Oct 2004 05:18:37 -0500 (CDT)

>    immediate non-task-management iSCSI command per connection at any

The iSCSI target "MAY" be allowed to discard excess immediate commands,
but that doesn't help the RDMA layer that needs a buffer for an untagged
message, or it will be required to terminate the connection.

Because iSER maps both the flow-controlled and non-flow-controlled messages
to the same connection, lack of a buffer for an immediate command places
non-immediate messages at risk.

Obviously, creating an extra connection for infrequent commands is not
the correct solution.  But a correct solution does require that an iSER
implementation has *something* to enable it to determine how many
buffers to pre-post to the RDMAP layer.  It already has the answer for
non-immediate commands, which is the bulk of the traffic.  It is also
relatively easy to make some very good estimates on how many would
be required for immediate commands.

But interoperability of RFC compliant implementations should be based
upon their RFC compliance, not on compatible guestimations.

What I believe Alex is asking for, and I asked for earlier, is a basis for
making that estimate that is based on a specification, and not heuristic
measurements of N leading products.

--
Caitlin Bestler
http://asomi.com/

---

From: Julian Satran <[email-masked]>
Date: Mon, 4 Oct 2004 14:41:43 +0200

This is an iSER specific requirement and an iSER specific key may help. I 
am afraid however that even this might not be foolproof as an initiator 
may fill any number of buffers with non task management functions and then 
claim that the target is not iSCSI compliant as it does not accept an 
immediate task management function command.
Or even worse - as RDMA has no way of distinguishing immediate from 
non-immediate commands an initiator can clog the pipe with non-immediate 
commands before the target will feel abused.
Dropping the session when an initiator has sent more than the allowed 
number of commands+2 seems perfectly reasonable.

Julo

"Caitlin Bestler" <[email-masked]> 
04/10/04 12:18

To
Julian Satran/Haifa/IBM@IBMIL
cc
"Alex Nezhinsky" <[email-masked]>, [email-masked], "Mike Ko" 
<[email-masked]>, "Mallikarjun C." <[email-masked]>
Subject
RE: [Ips] DA+iSER Flow Control,      Limiting number of oustanding cmds 
attarget

Because iSER maps both the flow-controlled and non-flow-controlled 
messages
to the same connection, lack of a buffer for an immediate command places
non-immediate messages at risk.

--
Caitlin Bestler
http://asomi.com/