Subject: RE: [Ips] DA+iSER Flow Control, Limiting number of oustanding cmds attarget

From: "Alex Nezhinsky" <[email-masked]>
Date: Mon, 4 Oct 2004 17:47:55 +0200

> What I believe Alex is asking for, and I asked for earlier, is a basis
for
> making that estimate that is based on a specification, and not
heuristic
> measurements of N leading products.
Absolutely. 

Immediate commands is only one type of "out-of-band" PDUs.
NOPs is another one which is relevant in the context of flow cotrol
itself.
If the value of MaxCmdSN changes at the target when no PDUs are about to
be sent,
iSCSI suggests sending a NOP-IN in order to communicate the new MaxCmdSN
value.
So the initiator should be ready to get NOPs at any time. 

If response PDU is lost for whatever reason, the initiator will send
NOP-OUT,
this means that the target should post additional buffer for each
command it receives,
just in case.

Text requests are limited to 1 outstanding, while task mgmt requests are
not.
 
> Dropping the session when an initiator has sent more than the allowed
number 
> of commands+2 seems perfectly reasonable. 
Why not use a simple flow control scheme, in which, for example, iSER
header has 
one additional field designating the number of recv buffers posted in
addition to 
those implied by MaxCmdSn. That is, it's not just commands+2, but
F(commands)+X,
where X is communicated in each iSER header and F is a function specific
for initiator,
target. 
What the specification could also provide is, for example, a minimal
value for X,
similar to "MUST be able to handle at least one immediate task
management command and one
immediate non-task-management iSCSI command per connection at any time".

Alexander Nezhinsky
	________________________________

	From: Julian Satran [mailto:[email-masked]] 
	Sent: Monday, October 04, 2004 2:42 PM
	To: Caitlin Bestler
	Cc: Alex Nezhinsky; Mallikarjun C.; [email-masked]; Mike Ko;
[email-masked]
	Subject: RE: [Ips] DA+iSER Flow Control, Limiting number of
oustanding cmds attarget

	This is an iSER specific requirement and an iSER specific key
may help. I am afraid however that even this might not be foolproof as
an initiator may fill any number of buffers with non task management
functions and then claim that the target is not iSCSI compliant as it
does not accept an immediate task management function command. 
	Or even worse - as RDMA has no way of distinguishing immediate
from non-immediate commands an initiator can clog the pipe with
non-immediate commands before the target will feel abused. 
	Dropping the session when an initiator has sent more than the
allowed number of commands+2 seems perfectly reasonable. 
	
	Julo 
	
	________________________________
	
	"Caitlin Bestler" <[email-masked]> 

	04/10/04 12:18 

		To
		Julian Satran/Haifa/IBM@IBMIL 
		cc
		"Alex Nezhinsky" <[email-masked]>, [email-masked],
"Mike Ko" <[email-masked]>, "Mallikarjun C." <[email-masked]> 
		Subject
		RE: [Ips] DA+iSER Flow Control,      Limiting number of
oustanding cmds attarget
	
	Julian Satran said:
	> Alex,
	>
	> The relevant text in RFC 3720 - 3.2.2.1 is:
	>
	>   The number of commands used for immediate delivery is not
limited and
	>    their delivery for execution is not acknowledged through
the
	>    numbering scheme.  Immediate commands MAY be rejected by
the iSCSI
	>    target layer due to a lack of resources.  An iSCSI target
MUST be
	>    able to handle at least one immediate task management
command and one
	>    immediate non-task-management iSCSI command per connection
at any
	>    time.
	> I hope this helps.
	> Hag Sameach,
	> Julo

	The iSCSI target "MAY" be allowed to discard excess immediate
commands,
	but that doesn't help the RDMA layer that needs a buffer for an
untagged
	message, or it will be required to terminate the connection.
	
	Because iSER maps both the flow-controlled and
non-flow-controlled messages
	to the same connection, lack of a buffer for an immediate
command places
	non-immediate messages at risk.
	
	Obviously, creating an extra connection for infrequent commands
is not
	the correct solution.  But a correct solution does require that
an iSER
	implementation has *something* to enable it to determine how
many
	buffers to pre-post to the RDMAP layer.  It already has the
answer for
	non-immediate commands, which is the bulk of the traffic.  It is
also
	relatively easy to make some very good estimates on how many
would
	be required for immediate commands.
	
	But interoperability of RFC compliant implementations should be
based
	upon their RFC compliance, not on compatible guestimations.
	
	What I believe Alex is asking for, and I asked for earlier, is a
basis for
	making that estimate that is based on a specification, and not
heuristic
	measurements of N leading products.
	
	--
	Caitlin Bestler
	http://asomi.com/

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Mon, 4 Oct 2004 11:59:38 -0700

Alex,

This is the gist of the iSER "flow control" related discussion on the IPS 
forum in July/August of 2003.

>From the initiator to the target, for control-type PDUs where the 
immediate flag is not set, the CmdSN and MaxCmdSN mechanism in iSCSI 
already provides the flow control needed.  For unsolicited SCSI Data-out 
PDUs, FirstBurstLength can be used to estimate the amount of buffering 
required.  What remains that are not covered by any existing flow control 
are PDUs where the immediate flag is set or SNACK Request (which is not 
needed in iSER).

>From the target to the initiator, unsocited NOP-in are regulated by 
NOP-out.  The unsolicited PDUs which are not covered by any existing flow 
control are Async Messages.

One of the solution proposed is to set a declared limit on the number of 
such unsolicited requests.

The next question is how does each side determine when is the limit 
replenished.  One possible solution is as follows:

For the initiator, the ExpCmdSN from the target will be used to advance 
the credit.  After sending PDUs with the immediate flag set, when the 
initiator sends any iSCSI control-type PDU with a new CmdSN without the 
immediate flag, if the target responds with an iSCSI control-type PDU with 
ExpCmdSN set to CmdSN + 1, then credit at the initiator is restored to its 
full value.

For the target, the ExpStatSN from the initiator will be used to advance 
the credit.  When the initiator sends any PDU which contains the ExpStatSN 
that acknowledges the StatSN in the Async Message, the number of credits 
at the target used up by that Async Message will be restored.

Mike 
To:     "Julian Satran" <[email-masked]>, "Caitlin Bestler" 
<[email-masked]>
cc:     "Mallikarjun C." <[email-masked]>, "Mike Ko" 
<[email-masked]>, <[email-masked]> 
Subject:        RE: [Ips] DA+iSER Flow Control,      Limiting number of 
oustanding cmds attarget

for
heuristic
Absolutely.

number

Alexander Nezhinsky

Julo

04/10/04 12:18

To

limited and
the
the iSCSI
MUST be
command and one
at any

--
Caitlin Bestler
http://asomi.com/

---

From: John Hufferd <[email-masked]>
Date: Mon, 4 Oct 2004 12:23:03 -0700

The important thing to understand is that you need to queue enough buffers 
so that the suddenly chatty Initiator can proceed with out an impact to 
latency.   In most of the robust implementations, which will be 
controlling hundreds/thousands of logical connections, the approprate 
consideration is the fact that regardless of the number of logical 
connections, there can not be more data than the link is capable of 
handling.  Therefore, we designed the concept of Shared Receive Queue. The 
purpose was to have a method where the number of buffers that were needed 
to handle the link was all that was required.  Which of course meant that 
there needed to be an ongoing active buffer replenishment technique. Which 
is similar to what is done in other SCSI environments. 

When ever one attempts to work this problem with non shared receive 
queues, the problem gets out of hand very quickly.  With or without some 
additional form of end to end credit, the Total number of buffers that is 
needed becomes the sum of the buffers needed for all logical connections, 
since at any time they could burst (or desire to burst) at full link 
rates.  So the idea of having to add a few extra, for each connection 
seemed to be only useful for very limited implementations, and having flow 
control above what iSCSI offered seemed to be a needless impact on 
latency.  Even then the sum of all the credits (meaning backed by buffers) 
would be a lot more than what is needed to match the link speed.

So based on the experience of the iSER designers we did not like the 
latency impact or the total buffer impact of using NON Shared Receive 
Queues.  I believe your questions were in the area of attempting to 
implement iSER without Shared Receive Queues, and I feel that approach is 
problematical in itself, and will cause a much larger impact on the buffer 
space needed within the Storage Controller. 

On the other hand, if you have an implementation that only has a very few 
Logical connections, there is probably not an important problem of 
queueing a number of extra buffers.  Lets use a Tape Unit as an example, 
any specific drive needs to be able to have enough buffers to match the 
speed of the link with a single Initiator.  If you are at distance, and 
attempting to respond appropriately in some manner with credits, you may 
very well find that you either cause needless latency impact, or the 
number of buffers needed to avoid the latency, matches what you need for 
the maximum line rate anyway. 

When you consider what is needed both in the primary storage focus which 
is 100s - 1000s of connections, to the needs for an "at distance" small 
number of connections, the iSER spec came up with the right balance and 
additional flow control seems inappropriate.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

"Alex Nezhinsky" <[email-masked]> 
Sent by: [email-masked]
10/04/2004 08:47 AM

To
"Julian Satran" <[email-masked]>, "Caitlin Bestler" 
<[email-masked]>
cc
[email-masked], Mike Ko <[email-masked]>, "Mallikarjun C." 
<[email-masked]>
Subject
RE: [Ips] DA+iSER Flow Control, Limiting number of oustanding cmds 
attarget

for
heuristic
Absolutely. 

 
number 

Alexander Nezhinsky

                 From: Julian Satran [mailto:[email-masked]] 
                 Sent: Monday, October 04, 2004 2:42 PM
                 To: Caitlin Bestler
                 Cc: Alex Nezhinsky; Mallikarjun C.; [email-masked]; Mike 
Ko;
[email-masked]
                 Subject: RE: [Ips] DA+iSER Flow Control, Limiting number 
of
oustanding cmds attarget

                 This is an iSER specific requirement and an iSER specific 
key
may help. I am afraid however that even this might not be foolproof as
an initiator may fill any number of buffers with non task management
functions and then claim that the target is not iSCSI compliant as it
does not accept an immediate task management function command. 
                 Or even worse - as RDMA has no way of distinguishing 
immediate
from non-immediate commands an initiator can clog the pipe with
non-immediate commands before the target will feel abused. 
                 Dropping the session when an initiator has sent more than 
the
allowed number of commands+2 seems perfectly reasonable. 
 
                 Julo 

                 04/10/04 12:18 

                                 To
                                 Julian Satran/Haifa/IBM@IBMIL 
                                 cc
                                 "Alex Nezhinsky" <[email-masked]>, 
[email-masked],
"Mike Ko" <[email-masked]>, "Mallikarjun C." <[email-masked]> 
                                 Subject
                                 RE: [Ips] DA+iSER Flow Control, Limiting 
number of
oustanding cmds attarget
 
limited and
the
the iSCSI
MUST be
command and one
at any
                 > Julo

                 The iSCSI target "MAY" be allowed to discard excess 
immediate
commands,
                 but that doesn't help the RDMA layer that needs a buffer 
for an
untagged
                 message, or it will be required to terminate the 
connection.

                 Obviously, creating an extra connection for infrequent 
commands
is not
                 the correct solution.  But a correct solution does 
require that
an iSER
                 implementation has *something* to enable it to determine 
how
many
                 buffers to pre-post to the RDMAP layer.  It already has 
the
answer for
                 non-immediate commands, which is the bulk of the traffic. 
 It is
also
                 relatively easy to make some very good estimates on how 
many
would
                 be required for immediate commands.
 
                 But interoperability of RFC compliant implementations 
should be
based
                 upon their RFC compliance, not on compatible 
guestimations.
 
                 What I believe Alex is asking for, and I asked for 
earlier, is a
basis for
                 making that estimate that is based on a specification, 
and not
heuristic
                 measurements of N leading products.
 
                 --
                 Caitlin Bestler
                 http://asomi.com/

---

From: "Caitlin Bestler" <[email-masked]>
Date: Tue, 5 Oct 2004 01:45:21 -0500 (CDT)

John Hufferd said:
> The important thing to understand is that you need to queue
> enough buffers so that the suddenly chatty Initiator can proceed
> with out an impact to latency.   In most of the robust
> implementations, which will be controlling hundreds/thousands
> of logical connections, the approprate consideration is the
> fact that regardless of the number of logical connections,
> there can not be more data than the link is capable of
> handling.  Therefore, we designed the concept of Shared
> Receive Queue. The purpose was to have a method where the
> number of buffers that were needed to handle the link was
> all that was required.  Which of course meant that there
> needed to be an ongoing active buffer replenishment technique.
> Which is similar to what is done in other SCSI environments.
>
> When ever one attempts to work this problem with non shared
> receive queues, the problem gets out of hand very quickly.
> With or without some additional form of end to end credit,
> the Total number of buffers that is needed becomes the sum
> of the buffers needed for all logical connections, since at
> any time they could burst (or desire to burst) at full link
> rates.  So the idea of having to add a few extra, for each
> connection seemed to be only useful for very limited
> implementations, and having flow control above what
> iSCSI offered seemed to be a needless impact on latency.
> Even then the sum of all the credits (meaning backed by
> buffers) would be a lot more than what is needed to match
> the link speed.
>

There are two very distinct issues here.  One is the memory
allocation strategy that should be employed by a server that
supports a large number of clients.  The second is how to
determine what is a legal untagged message at the RDMA layer.

As Julian pointed out, the iSCSI layer can quickly discard
excessive asynchronous messages. The challenge is enabling
the iSER layer to provide buffers for them so that they
can be captured at the RDMA layer prior to their being
interpreted at the iSCSI layer.

Shared Receive Queues are relevant here on two fronts.
First, iSCSI flow control is really more session oriented
than connection oriented.  Pre-posting separate buffers to
receive Cmd X on *each* connection is clearly over-buffering.
This may be valid, because memory is cheap, but clearly it
is not required to pre-commit duplicate buffers for each
connection in a session.

But Shared Receive Queues are typically used to cover
multiple sessions, indeed frequently thousands of sessions.

As noted in the RDDP Security spec, when using a shared
resource across multiple sessions, the application must take
security precautions to prevent the shared resource from being
drained by a single malicious or malfunctioning client.

Basically, you MUST ensure that a single client sending
excess untagged messages will not drain the Shared Receive
Queue and result in connections for *other* sessions failing
for lack of a receive buffer.

Which means that use of a Shared Receive Queue does *nothing*
to eliminate the need to calculate a per-connection limit on
the maximum number of untagged messages -- it only means that
you do not need to pre-allocate that many buffers.

But any ULP that is compliant with the RDDP Security requirements
must supply implementations with enough information so that they
can enable the per-connection limits on untagged messages.  Those
limits might be a configuration field for a QP using an SRQ, or
they might be the size of the QP's Receive Queue.  It doesn't
matter what strategy any specific implementation uses, what
matters is that there needs to be a method of setting that
limit which has its justification in *specifications*, not
in measurements.

Otherwise, *each* connection must be enabled to receive buffers
at line rate.  And unless you can push the untagged messages to
the iSER module *at line rate* you don't have a workable solution.
Requiring the iSER module to filter untagged messages at line
rate would clearly make user-mode implementations infeasible,
and would prevent iSER from being implemented over general
purpose RDMA verbs.  *Either* of which would strike me as
justifying a change in the DataMover architecture to avoid
this problem.

--
Caitlin Bestler
http://asomi.com/

---

From: John Hufferd <[email-masked]>
Date: Tue, 5 Oct 2004 00:46:53 -0700

This is not a general User State application it is a very specific Kernel 
Mode operation.  This is Storage I/O, and the server you are talking about 
is a very specially constructed kernel application that follows the rules 
of SCSI, and the operations that run in a storage controller manages a 
large cache of memory that is divided up between the what is Going out and 
coming in.  And this is both going out and coming in the communication 
link and going out and coming in from the SCSI LUN.  iSCSI/iSER is just a 
part of the buffering and caching that is going on.  There is various 
different algorithms to deal with Cache, both write through and write 
behind and algorithms to select the right buffer locations for new data 
(from either side) and algorithms to decide which Cache buffers need to be 
written to disk, etc.  The Caches/Buffers are usually all managed via 
chains, and what comes in as an iSCSI buffer becomes a Cache buffer and 
then a LUN I/O buffer, then a free buffer, then perhaps placed again on 
the iSER buffer chain or on the LUN read chain, etc.  You can almost talk 
about a Storage Controller as "Buffers are US".  There can not be a 
problem for a storage controller to service the line rate of the 
communication link, or the storage controller will not operate correctly, 
and that type of operation is fundamental to a storage controller. 
Therefore to talk of this as if the storage controller is some general 
user App is just not correct.  This type of buffer balancing is how 
storage controllers have always acted, and balancing buffers and caches in 
and out is not a problem.  We do it all the time.  The flow control of 
iSCSI is all that is needed when we are working with Shared RQs, and are 
managing the buffers ourselves.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

10/04/2004 11:45 PM

To
John Hufferd/San Jose/IBM@IBMUS, [email-masked]
cc
"Alex Nezhinsky" <[email-masked]>, "Mike Ko" <[email-masked]>, 
"Julian Satran" <[email-masked]>, "Mallikarjun C." 
<[email-masked]>
Subject
RE: [Ips] DA+iSER Flow Control,     Limiting number of oustanding cmds 
attarget

--
Caitlin Bestler
http://asomi.com/

---

From: "Caitlin Bestler" <[email-masked]>
Date: Tue, 5 Oct 2004 03:10:42 -0500 (CDT)

> managing the buffers ourselves.

I disagree.  I believe it is fundamentally incorrect to propose a
protocol as an IETF standard based upon such implementation assumptions.

Especially when those assumptions are not stated.

If that is the justification, then it the DataMover architecture
at the minimimum must make it clear that the iSCSI layer MUST
accept pushed iSCSI PDUs at line rate.

That would make it a correct specification, in the sense of
consistency.  But I still do not think it would be wise.
One of the goals of iSER should be to enable it to be
implemented over RDMA RNICs using their *existing* local
interfaces.  If an intelligent NIC is going to support a
special local interface specifically for iSCSI it might
as well be iSCSI, not iSER.  The goal is to reduce the
amount of application-specific logic in the NIC, not to
create a third interface in addition to those supporting
generic RDMA and non-RDMA iSCSI.

--
Caitlin Bestler
http://asomi.com/

---

From: John Hufferd <[email-masked]>
Date: Tue, 5 Oct 2004 09:38:38 -0700

Caitlin,
I have no idea why you say that we need something special in the RNIC that 
is used for iSCSI/iSER.  That is NOT the case, and I did not intend to 
suggest anything like that.  The only issue you keep bringing up is with 
buffering, which is fully handled by the Storage Controllers and Shared 
RQs.  And the point of all the implementations is how fast we can service 
the data coming on the link.  I do not think we need to say anything like 
the Goal is to operate at link speed!  The purpose of SRQs is to permit 
application such as iSER to be able to respond at link speed, while 
keeping the use of pre posted buffers to a minimum.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

10/05/2004 01:10 AM

To
John Hufferd/San Jose/IBM@IBMUS
cc
"Caitlin Bestler" <[email-masked]>, "Alex Nezhinsky" <[email-masked]>, 
"Mallikarjun C." <[email-masked]>, [email-masked], "Julian Satran" 
<[email-masked]>, "Mike Ko" <[email-masked]>
Subject
RE: [Ips] DA+iSER Flow Control,          Limiting number of oustanding 
cmdsattarget

--
Caitlin Bestler
http://asomi.com/

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 5 Oct 2004 19:07:50 +0200

On Oct 5, 2004, at 6:38 PM, John Hufferd wrote:

>
> Caitlin,
> I have no idea why you say that we need something special in the RNIC 
> that is used for iSCSI/iSER.
> That is NOT the case, and I did not intend to suggest anything like 
> that.  The only issue you keep
> bringing up is with buffering, which is fully handled by the Storage 
> Controllers and Shared RQs.
>  And the point of all the implementations is how fast we can service 
> the data coming on the link.
> I do not think we need to say anything like the Goal is to operate at 
> link speed!  The purpose of
> SRQs is to permit application such as iSER to be able to respond at 
> link speed, while keeping
> the use of pre posted buffers to a minimum.

The de facto standard for the local interface to an RNIC requires 
pre-posting
of receive buffers for untagged messages, either specifically to the 
endpoint
(the receive queue) or to a shared receive queue supporting a group of 
endpoints.

The supply of available buffers is finite.  Once a buffer is claimed, 
it will be
delivered to the local ULP (iSER in this case).  The buffer can only be
repopulated to the buffer pool after it is processed.

If there are no session limits on an individual session understood at
the RDMA layer, then one remote connection can drain the pool at
wire speed.

The iSER layer cannot return buffers to the buffer pool than it can
accept them.   If the only break on a runaway connection is the
iSER layer (or even beyond that the iSCSI layer) then they must
be able to accept iSCSI PDUs at line rate or a remote connection
will be able to drain the Shared Receive Queue.

SRQs have per-endpoint limits designed to short circuit such
attacks.  But you have to have a value to set, and it is a limit
on the number of outstanding untagged messages.  The RDMA
layer has no way to distinguish between valuable messages
that the iSCSI must process (or terminate the session) and
those that the iSCSI layer is allowed to throw away.

This is basic math.  If a single connection can drain the SRQ
at line rate (and that's the maximum number of *messages*
possible at line rate) then the iSCSI layer must accept the
iSCSI PDUs at line rate -- OR there has to be a method of
stating that a given connection has send too many untagged
messages.

That's a credit limit.  If you don't allow there to be a credit
limit then the only credit limit is the *message* rate of the wire.
Either that or you leave all sessions vulnerable to a bug
in one client (or even one malicious attack).  I do not believe
requiring iSCSI to accept iSCSI PDUs at wire rate is feasible
(it is also impossible on a PCI-X bus), and not addressing
the security issue is unacceptable.

The objection that has been repeatedly raised was that there
was no basis for setting that credit limit based upon specifications.

---

From: John Hufferd <[email-masked]>
Date: Tue, 5 Oct 2004 11:34:19 -0700

Caitlin,
I am having difficulty following your issues.  But I think it now has 
something to do with the fact that the code in the iSCSI part of the 
iSCSI/iSER process will not be able to determine that its partner is over 
subscribing its links. 

Since we are talking about a single Kernel application that is only 
divided into logical modules, which we decide to call iSER and the non 
iSER part of iSCSI.  All the flow control of iSCSI is still in effect, and 
will react to a given buffer within nano seconds of its data being 
received via the SRQ buffers.  The iSCSI code will then determine if the 
peer is doing something wrong (like exceed its iSCSI set limits).  If the 
iSCSI set limits are exceeded, then the iSCSI part of the code will 
terminate the connection.  If the iSCSI set limits have not been exceeded, 
that peer can burst on the link as much as it can, that is not a problem, 
and I do not like the idea of adding yet another set of flow controls 
above what iSCSI has. 

With a well run buffer chain management this is not a problem.  iSCSI will 
see the problem almost immediately and make the needed corrections.  Yes 
there is some design and implementation art to making sure that the SRQ 
buffer chain management, the general Cache buffer chain management, iSER 
and iSCSI work well together, but that is where the Intellectual Property 
of each vendor comes in.  And each vendor will claim, as they always do, 
that they handle it better than every other vendor.  But there is no real 
problem of having the iSCSI part of the code be the point where over 
subscription of the link is determined and handled.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Caitlin Bestler <[email-masked]> 
10/05/2004 10:07 AM

To
[email-masked], John Hufferd/San Jose/IBM@IBMUS
cc

Subject
Re: [Ips] DA+iSER Flow Control,          Limiting number of oustanding 
cmdsattarget

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 5 Oct 2004 22:33:26 +0200

On Oct 5, 2004, at 8:34 PM, John Hufferd wrote:

> something to do with the fact that
> the code in the iSCSI part of the iSCSI/iSER process will not be able 
> to determine that its partner
> is over subscribing its links.  
>

I am not thinking of the problem as a single software entity.  I am
focusing on being a provider of a Datamover Architecture service.

I have no guarantee that the Consumer of my service is
aware that they must reap completions fast enough to
prevent the buffer pool from being drained.

Further the Datamover specification could easily be
read so that they would think it is *my* responsibility
to apply back-pressure on the peer to prevent the
packets from being sent.  The *only* form of back
pressure available over RDMA is dropping the
connection.  I do not want to drop connections
without justification, such as violation of a credit
limit.  Dropping a connection when there has not
been a violation or an error is *not* providing a
reliable transport service.

This is a particular concern with RDMA because
it is possible to implement RDMA in a way where
the TCP bottlenecks are not totally eliminated.
So a competitor's product might conceal this
problem with the "solution" of failing to eliminate
the TCP bottleneck completely enough.  Without
a clear statement of responsibility for flow control
being at the iSCSI layer my customer may conclude
that the problem is *my* implementation.

---

From: John Hufferd <[email-masked]>
Date: Tue, 5 Oct 2004 15:20:32 -0700

Caitlin,
I guess the problem is that we are looking at a completely different model 
of what is going on with the iSCSI/iSER application.  I do not think that 
we defined iSCSI/iSER with the thought that they were somehow disjoint 
models.  In fact I believe that the name iSCSI/iSER kind of describes how 
we anticipated the protocol would be used. 

Having a Data Mover Architecture Service, was not something that we had in 
mind.  iSCSI/iSER is the service.  That is, it is a kernel mode Device 
Driver, which handles specific logical TCP/RNIC ports.  When a SRQ is 
used, it is used by the iSCSI/iSER service, and was not envisioned to be 
managed by some Data Mover Service that would need to respond to multiple 
iSCSI Device Drivers.  If for some reason a storage controller or OS 
vendor feels they need to have multiple iSCSI/iSER Device Drivers, the SRQ 
would apply to each specific one.  That is, each iSCSI/iSER Device Driver 
would have its own SRQ. 

Therefore, the Architecture was defined with the idea that iSCSI & iSER 
were "joined at the hip" (sort of speak).  If an implementation chooses to 
separate the iSER from the iSCSI, and if the remaining iSCSI Device Driver 
is not capable of seeing its buffers fast enough to stop the connection 
when it goes beyond the iSCSI flow control, then that is simply a bug in 
the implementation.  Even with that type of an implementation the iSCSI 
Kernel mode Device Driver should be able to inspect the Buffers within 
nano seconds of the data arrival.  Again, I do not see the problem.

Now, just so you understand why I object to the issue of additional flow 
control on top of iSCSI, you need to reference what has to be done with 
Credit protocols like FC which need to have extra HW boundary boxes to 
"spoof" the credits in order to have the protocol go across very large 
distances.  iSCSI was design to travel over large distances, without such 
"spoofing" techniques, and I object to additional Flow control, especially 
since it is not obvious that it is a hard requirement and also since I do 
not see the case that a Storage Controller needs "Beyond iSCSI" Flow 
Control. 
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

10/05/2004 01:33 PM

To
John Hufferd/San Jose/IBM@IBMUS
cc
[email-masked]
Subject
Re: [Ips] DA+iSER Flow Control, Limiting number of oustanding cmdsattarget

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 6 Oct 2004 05:02:55 +0200

On Oct 6, 2004, at 12:20 AM, John Hufferd wrote:

>
> Caitlin,
> I guess the problem is that we are looking at a completely different 
> model of what is going on with the
> iSCSI/iSER application.  I do not think that we defined iSCSI/iSER 
> with the thought that they were somehow
> disjoint models.  In fact I believe that the name iSCSI/iSER kind of 
> describes how we anticipated the protocol
> would be used.
>

But that is how the DataMover Architecture is specified.  It is a clean 
layer.
Responsibilities must be separated in specifications, even if 
implementations
choose to be integrated.  Isn't that fundamental to the process of 
setting standards?
>
>
> Therefore, the Architecture was defined with the idea that iSCSI & 
> iSER were "joined at the hip" (sort of speak).
> If an implementation chooses to separate the iSER from the iSCSI, and 
> if the remaining iSCSI Device Driver is not
> capable of seeing its buffers fast enough to stop the connection when 
> it goes beyond the iSCSI flow control, then
> that is simply a bug in the implementation.  Even with that type of an 
> implementation the iSCSI Kernel mode Device
> Driver should be able to inspect the Buffers within nano seconds of 
> the data arrival.  Again, I do not see the problem.
>

The pre-existing iWARP interfaces are *not* "joined at the hip" with 
their users. One of the goals is
to use enable iSCSI to be implemented over the general purpose iWARP 
service.  Doing that
requires The Upper Layer Protocol to provide for flow control of 
untagged messages, and to
protect against attacks of resources shared across session lines.

The pre-existing interfaces are *not* designed to push deliverable 
packets to the ULP on
demand, but rather to generate completions of pre-posted receive 
operations, which the
ULP consumes at its own pace.  Further, each connection has a credit 
limit, violation of
which will result at the minimum in an alarm being raised allowing for 
prompt termination
of a runaway or malicious client.

If the iSCSI layer is *not* using the per-connection limit, and is 
using a shared resource
to service multiple clients that have no basis for shared trust it is 
violating the RDMAP/RDDP
security requirements for ULPs.

So in order to comply with the Security *requirements* for using iWARP, 
iSER MUST have
sufficient information to set per-connection limits on untagged 
messages.

It is also basic theory.  You cannot provide a reliable transport 
service (in the absence
of a lower layer error) unless you can block sends or push receives.  
The DataMover
architecture fails to define this, but does require it to be reliable.

> Now, just so you understand why I object to the issue of additional 
> flow control on top of iSCSI,
> you need to reference what has to be done with Credit protocols like 
> FC which need to have extra
> HW boundary boxes to "spoof" the credits in order to have the protocol 
> go across very large distances.
>  iSCSI was design to travel over large distances, without such 
> "spoofing" techniques, and I object t
>  additional Flow control, especially since it is not obvious that it 
> is a hard requirement and also since I
> do not see the case that a Storage Controller needs "Beyond iSCSI" 
> Flow Control.  

Absolutely no new wire protocol is needed.  The only thing that is 
required
is to *document* the an allowance for the packets that iSCSI currently
does not flow control.

iSCSI is relying on the TCP bottleneck to flow control these.  When TCP
buffer credits have been exhausted the sender is prevented from sending
more data without breaking the connection. The DataMover Architecture
needs to be defined so it works over transports that do not provide this
bottleneck "feature".  That includes iSER, and it would include other
possible DataMover architectures.

Stating that the the DataMover architecture user may be required
to pre-supply buffers would solve the broad computer science
problem.  It would not solve the security vulnerability of enabling
expedited handling of a resource drain caused by a single
malfunctioning or malicious client.

Failure to address this issue would make DA and iSER proposed
standards that *fail* to comply with the security requirements of
the protocol they claim to be utilizing.  Obviously the IETF cannot
hand-inspect applications to ensure they comply with ULP
requirements, but at the minimum other IETF drafts should
be compliant with IETF requirements.

---

From: John Hufferd <[email-masked]>
Date: Wed, 6 Oct 2004 00:45:43 -0700

I guess I still do not see the security exposure for a kernel device 
driver that can inspect the buffers within nano seconds of the data 
arrival and determine if the packet is valid, and if an invalid set of 
buffers is received, terminating the connection.  This puts the security 
issue right on the shoulders of the iSCSI device driver that can tell that 
the buffers are invalid.  I do not see the issue.  A run away connection 
that invalidly uses up all the bandwidth is just not a real problem if the 
buffers are inspected as they come in by the iSCSI device driver, and the 
connection dropped when an error is detected.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

10/05/2004 08:02 PM

To

---

From: "Caitlin Bestler" <[email-masked]>
Date: Wed, 6 Oct 2004 03:42:49 -0500 (CDT)

> connection dropped when an error is detected.

That division of responsibility should certainly be allowed.
It would require the iSER layer to reap completions almost
instantly, and to pass all completed iSCSI PDUs to the iSCSI
layer almost instantly. The iSCSI layer would have to keep up,
period.  It could do so by tossing packets and terminating
iSCSI sessions.

The current draft does not provide language that would justify
that division of responsibilities.  That should be fixed.

I also believe, although I really don't care that strongly
about it, that a properly designed wire protocol should not
be dependent on a specific implementation strategy.

That means that there MUST be a method by which it is impossible
for a single malfunctionign and/or malicious client to drain
shared resources and thereby cause a failure affecting other
clients.  Requiring the iSCSI layer to at least accept packets
at line rate, or to refrain from using shared resources, would
indeed accomplish that.  But it unnecessarily constrains the
range of possible implementations.

Specifying a cap on the number of "extra" untagged messages,
either as a ULP constant or a session parameter,would enable
the use of per-connection safety mechanisms associated with
SRQs.  That would enable a wide range of possible implementations
to use SRQs to implement iSCSI/iSER.

As previously noted, pre-allocating a fully committed buffer
for each possible untagged message on each connection is not
a prudent method of allocating memory for an iSCSI daemon.
Enabling the widest possible range of implementations to
use SRQs when implementing iSER is a desirable goal.

Implementations which are indeed capable of having the iSCSI
layer accept iSCSI PDUs at line rate would never have to use
this information.  At worst they are exchanging one extra piece
of data during session setup.  That is much to ask if it enables
a wider range of possible implementations.

One of the benefits of RDMA is that when it is used properly
in can enable daemons to be implemented in user-space that
previously could only have been attempted in the kernel.
That is a desirable feature.  iSER should not be designed
on the assumption that it is unobtainable.

--
Caitlin Bestler
http://asomi.com/

---

From: John Hufferd <[email-masked]>
Date: Wed, 6 Oct 2004 16:27:05 -0700

Caitlin,
Perhaps we are talking past each other.  My point has been and still is 
that I would not like to see additional flow control used with iSCSI/iSER 
that would prevent the "at distance" operations from being as efficient as 
current iSCSI.  Now having said that, I believe that the point that Mike 
Ko said in his previous note is that all we need to do is to cap the 
number of special responses that are beyond the normal expected iSCSI 
windowing process.  I think that is not really needed for the correct use 
of SRQ, and I believe you concurred on that point.  However, I believe 
that it may be of value to someone that wants to implement iSCSI/iSER 
without SRQ, or perhaps even in your hypothetical generic Data Mover 
Service.  So my interpretation of what Mike was saying is as follows 
(which if correct I can accept):

1. Each iSCSI/iSER side (peer) will declare to the other (for this 
discussion lets call this Keyword=Value pair 
"MaxOutstandingUnexpectedPDUs") a value that represents the number of 
messages that can occur beyond what is handled by the normal iSCSI Flow 
Control.  The values may be either numeric. or "None" where None means no 
limit specified.  I would even suggest that "None" should be the default.
2. From the Target side, this value (that it would declare), is the Max 
number of Outstanding Immediate Commands, and/or Outstanding SNACK 
commands that it could receive (SNACK should not be an important 
consideration, but the spec says that even if SNACKs are not needed, if 
SNACKs are issued by the Initiator they are passed on, so this value would 
cover SNACKs also).
3. From the Initiator side, this value (that it would declare), is the Max 
number of outstanding Async Messages and/or NOPs that it can receive from 
the Target.

There would be no need to have any additional flow control, and the NON 
SRQ Target implementations could ensure that the number of buffers that 
they queue will be at least sufficient to handle the iSCSI Command Window, 
plus the number of Declared "MaxOutstandingUnexpectedPDUs".   And the 
initiator would queue the number of buffers that will be sufficient to 
handle the iSCSI Command Window responses, plus the number of Declared 
"MaxOutstandingUnexpectedPDUs".

Now, as I said before, I believe it is best handled by an SRQ approach 
that does not limit the flow of messages "at Distance", and that where 
ever possible the peers should operate with the value of "NONE" (no 
limits) and that is why I think it should be the default. 

The reason that this approach is acceptable to me is because the use of 
Immediate commands (and SNACKs) or the arrival of Async Message PDUs from 
the Target, should be rare enough to not inhibit the normal iSCSI flow 
control.  However, we should have a caution stated somewhere, that this 
might have an important Error Recovery impacts when dealing with ERL=2 and 
we are in an at distance situation.  That is, when a link fails (which may 
happen more than we like "at distance") the ERL=2 mode says that the Task 
Management (Immediate) Commands must reassign the Tasks to a surviving 
connection.  It is very likely that we may see hundreds of commands that 
are subject to this action and since we have only a limited amount of time 
to make this switch,  it could be that the distance and limit on 
Outstanding Immediate commands could lengthen this recovery to a point 
that the recovery time out is exceeded.  (This would not be a problem with 
the SRQ design that I talked about previously.)

Caitlin, with your acceptance that the No Limit SRQ approach that I 
specified, and with the above proposal, which I believe may cover your 
additional issues, I think we maybe close to closure on this.

However, some of the folks on this reflector have told me that they prefer 
a resolution that requires the "Graceful handling" of RQ overruns by the 
RNIC. There are several approaches to do that, and quite frankly I prefer 
that over the above.  But perhaps we need to see if we at least can agree 
on the above and then discuss the other options.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

10/06/2004 01:42 AM

To
John Hufferd/San Jose/IBM@IBMUS
cc
"Caitlin Bestler" <[email-masked]>, [email-masked]
Subject
Re: [Ips] DA+iSER Flow Control,      Limiting number of oustanding 
cmdsattarget

--
Caitlin Bestler
http://asomi.com/

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 06 Oct 2004 18:01:08 -0700

I finally managed to catch up on this thread, let me attempt to 
summarize what I think are the options for iSER flow control.  I would 
like to get more comments to assess the WG position, perhaps also from 
the co-chairs on this topic.  I hope this long email acts as a helpful 
summary.

Let me ignore S-RQ for a moment and consider a simple RQ.  Let's even 
assume the target had adequately provisioned receive buffers (CmdSN 
window + 2 buffers, i.e. no overcommitting).  In this simplistic case, 
if the initiator sends too many immediate commands, we know that the 
target would run out of receive buffers.  Now the options that I see to 
deal with this scenario are (you may add more if I missed any):

1. Replenish buffers faster than the arrival of the unexpected messages 
(iSCSI layer may later discard the "too many immediate commands" if it 
deems so).
2. Require the RNIC to do graceful handling of lack of receive buffers 
on these exceptions.
3. Define a new iSCSI MaxOutstandingUnexpectedPDUs key-based limit 
scheme to deal with these exceptions (that John describes below).
4. Define an iSER end2end flow control protocol (such as the one iSER 
design team originally considered).
5. Mandate an S-RQ as the mandatory RNIC feature (so there's more 
flexibility for overprovisioning across multiple connections).
6. Allow the connection to be dropped because the buffers are overrun.

#1 may be feasible (and every implementation tries to do it anyway), but 
basing the protocol on that assumption doesn't strike me as a 
"standards" way of specifying a protocol.

#2 is quite nice from iSCSI & iSER's perspective.  We should be able to 
cast any immediate/un-flow-controlled iSCSI PDU as an "exception" (which 
it is) message (in DDP parlance) and thus get around adding any more 
protocol restrictions in iSER.  The major consideration here is the 
feasibility (from an RNIC vendor perspective) of realizing such a 
graceful handling.

#3 seems like a workable option if we can cleary specify it.  It however 
does add more protocol complexity to iSCSI, and makes some exception 
management cases harder to deal with (when the limit is not "None") as 
John points out below.

#4 is clearly expensive although we know that it can work.  This should 
be the last resort.

#5 is a second-order choice in that it may still require help from other 
options.  The main considerations here are: making normative references 
to the Verbs spec [ or defining the S-RQ text in abstract terms (i.e. 
stripping off references to WQE, CQE, RQ, CQ, RI etc.) - which again 
doesn't strike me as a "standards" way of defining a protocol ], the 
acceptance (from an RNIC vendor perspective) on requiring it.  We still 
need to  address a single connection case from the specification 
perspective (i.e. consider an embedded target supporting a single 
iSCSI/iSER connection).

#6 is also feasible (I think Julian too hinted at this in an email). 
Clearly, an initiator sending too many immediate commands or a target 
sending too many Async Messages is suspect. Problem though is that this 
is a major functional difference between how an iSCSI system would 
behave, and an iSCSI/iSER system would behave (thus potentially exposing 
the dichotomy to layers further up in the stack).  The other issue is 
that a genuine exception could be punished with an even bigger one - 
leading to termination of several I/Os.

The way I see it, the current iSER spec depends on #2, #1, #5 and #6. 
Note that while #2 alone is necessary and sufficient for operation 
without connection drops, others work to minimize the probability of 
invoking graceful handling (#2) but together ultimately allow a 
connection failure if the graceful handling service cannot be maintained 
by the RNIC after several DDP exceptions.  In the opinion of iSER 
authors, this integrated approach is quite reasonable.

I realize that an argument can be made for #3 (which Caitlin & Alex did 
as I understand the exchanges so far).  More comments from others would 
help the authors understand the WG position on this.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

John Hufferd wrote:
> that I would not like to see additional flow control used with 
> iSCSI/iSER that would prevent the "at distance" operations from being as 
> efficient as current iSCSI.  Now having said that, I believe that the 
> point that Mike Ko said in his previous note is that all we need to do 
> is to cap the number of special responses that are beyond the normal 
> expected iSCSI windowing process.  I think that is not really needed for 
> the correct use of SRQ, and I believe you concurred on that point. 
>  However, I believe that it may be of value to someone that wants to 
> implement iSCSI/iSER without SRQ, or perhaps even in your hypothetical 
> generic Data Mover Service.  So my interpretation of what Mike was 
> saying is as follows (which if correct I can accept):
> 
> Control.  The values may be either numeric. or "None" where None means 
> no limit specified.  I would even suggest that "None" should be the 
> default.
> SNACKs are issued by the Initiator they are passed on, so this value 
> would cover SNACKs also).
> 3. From the Initiator side, this value (that it would declare), is the 
> Max number of outstanding Async Messages and/or NOPs that it can receive 
> from the Target.
> 
> they queue will be at least sufficient to handle the iSCSI Command 
> Window, plus the number of Declared "MaxOutstandingUnexpectedPDUs".   
> And the initiator would queue the number of buffers that will be 
> sufficient to handle the iSCSI Command Window responses, plus the number 
> of Declared "MaxOutstandingUnexpectedPDUs".
> 
> Immediate commands (and SNACKs) or the arrival of Async Message PDUs 
> from the Target, should be rare enough to not inhibit the normal iSCSI 
> flow control.  However, we should have a caution stated somewhere, that 
> this might have an important Error Recovery impacts when dealing with 
> ERL=2 and we are in an at distance situation.  That is, when a link 
> fails (which may happen more than we like "at distance") the ERL=2 mode 
> says that the Task Management (Immediate) Commands must reassign the 
> Tasks to a surviving connection.  It is very likely that we may see 
> hundreds of commands that are subject to this action and since we have 
> only a limited amount of time to make this switch,  it could be that the 
> distance and limit on Outstanding Immediate commands could lengthen this 
> recovery to a point that the recovery time out is exceeded.  (This would 
> not be a problem with the SRQ design that I talked about previously.)
> 
> However, some of the folks on this reflector have told me that they 
> prefer a resolution that requires the "Graceful handling" of RQ overruns 
> by the RNIC. There are several approaches to do that, and quite frankly 
> I prefer that over the above.  But perhaps we need to see if we at least 
> can agree on the above and then discuss the other options.
> .
> .
> John L. Hufferd
> *"Caitlin Bestler" <[email-masked]>*
> 
> 10/06/2004 01:42 AM
> 
> 	
> To
> ------------------------------------------------------------------------
>

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 7 Oct 2004 11:55:26 +0200

On Oct 7, 2004, at 3:01 AM, Mallikarjun C. wrote:

> summarize what I think are
> the options for iSER flow control.  I would like to get more comments 
> to assess the WG
> position, perhaps also from the co-chairs on this topic.  I hope this 
> long email acts as a
> helpful summary.
>
> assume the target
> had adequately provisioned receive buffers (CmdSN window + 2 buffers, 
> i.e. no overcommitting).
> In this simplistic case, if the initiator sends too many immediate 
> commands, we know that the
> target would run out of receive buffers.  Now the options that I see 
> to deal with this scenario
> are (you may add more if I missed any):
>
> 1. Replenish buffers faster than the arrival of the unexpected 
> messages (iSCSI layer may later
>     discard the "too many immediate commands" if it deems so).
> scheme to deal with
>     these exceptions (that John describes below).

> #1 may be feasible (and every implementation tries to do it anyway), 
> but basing the protocol on that
> assumption doesn't strike me as a "standards" way of specifying a 
> protocol.

I think the operative word here is "try".  What I am particularly 
concerned about is that for both TCP
and SCP transport level flow control is based on the availability of 
receive buffering and network
congestion.

A TCP connection or SCTP association in use for RDDP effectively has no 
buffer limits, assuming
it can forward DDP Segments to the RDDP layer at wire speed.  RDDP is 
responsible for placing
the buffers when presented, or terminating the stream.  It has no 
back-pressure mechanism.

However, many implementations will not tailor the calculation of the 
buffer window for RDMA,
but rather use the general purpose algorithm.  That can result in false 
blocking of placable
traffic, but this failure will not be apparent when users are used to 
TCP bottlenecking.

Existing iSCSI implementations in particular could easily be relying on 
TCP bottlenecks.
When they test on an implementation that does not tailor the TCP window 
on RDMA connections
to "no system buffering, hence no buffer limits" they will not detect 
that the problem is still there.
Later they will test on an implementation that *does* fully eliminate 
the TCP bottleneck, and
promptly declare that "this RNIC breaks my iSCSI" and demand the vendor 
"fix it".

>
> #2 is quite nice from iSCSI & iSER's perspective.  We should be able 
> to cast any
> immediate/un-flow-controlled iSCSI PDU as an "exception" (which it is) 
> message
> (in DDP parlance) and thus get around adding any more protocol 
> restrictions in iSER.
> The major consideration here is the feasibility (from an RNIC vendor 
> perspective) of
> realizing such a graceful handling.

That is moving the "take it or flush it" interface up from the LLP to 
RDDP interface to
the RDMAP to ULP interface.  It is a good idea, one that local 
interfaces *should*
offer.  But we all know that it isn't part of the de facto standard. It 
is one of many
defects in that interface, but I would still oppose adopting any ULP 
that cannot
be implemented on top of commonly deployed verbs.

>
> #3 seems like a workable option if we can cleary specify it.  It 
> however does add
> more protocol complexity to iSCSI, and makes some exception management 
> cases
> harder to deal with (when the limit is not "None") as John points out 
> below.

There is also "3a" which is to make a default limit that is specified 
in the protocol itself.

I believe the goal should be that no application that is generating 
these messages for
their intended purposes should ever have to check the limit.  The 
receiver SHOULD
(or perhaps even MUST) set it high enough so that *only* a 
malfunctioning or malicious
client could exceed it.

The flow control can be fairly simple to implement however, anytime the 
flow control
limit would have been hit, it is for an unanticipated message that the 
target iSCSI side
would be free to throw away anyway.  The sender could simply do so 
immediately.
But I do think it would be better to define the credit limit such that 
it would be impossible
to violate it when generating messages legitimately.

The goal here is to enable the receiving side to identify a runaway 
malfunctioning or
malicious client that is attacking shared resources *before* it results 
in a legitimate
message from another client being dropped.

Even without SRQs, it is not that difficult for the receiving iSER 
layer to allow sufficient
extra receive buffers to deal with all *legitimate* iSCSI packets.  It 
is also certainly
preferable for the receive side to have some extra buffers available 
rather than
requiring non-malfunctioning/non-abusive senders to check their credit 
limit
before sending any of the "unaccounted" iSCSI PDUs.  These messages were
excluded from the normal iSCSI flow control precisely because they were 
not
anticipated to be high volume.

So there are two ways to determine when a connection is sending 
illegitimate
iSCSI packets:  place a credit limit that legitimate applications will 
not violate
(even without thinking about it) or have the iSER layer examine packets 
at
wire speed.

>
> #4 is clearly expensive although we know that it can work.  This 
> should be the last resort.
>
> #5 is a second-order choice in that it may still require help from 
> other options.  The main
> considerations here are: making normative references to the Verbs spec 
> [ or defining the
> S-RQ text in abstract terms (i.e. stripping off references to WQE, 
> CQE, RQ, CQ, RI etc.)
> which again doesn't strike me as a "standards" way of defining a 
> protocol ], the acceptance
> (from an RNIC vendor perspective) on requiring it.  We still need to  
> address a single
> connection case from the specification perspective (i.e. consider an 
> embedded target
> supporting a single iSCSI/iSER connection).

An SRQ does not eliminate the vulnerability.  In fact it is necessary 
for it to exist.  A runaway
client connected to a simple RQ can only tear down its own connection.  
It does not impact
other clients.

With an SRQ, you either have a per-connection limit/alarm or you are 
dependent on wire-speed
draining of completions by the ULP (the iSER layer).  That case is 
essentially option 1,
enabling the per-connection limit/alarm requires having a number to set 
it with which
leads back to option 3 (or 3a).

An argument for 3, as opposed to 3a, is to specifically allow "None". 
But the iSER layer
should only uses "None" when it is committing to drain completions at 
wire speed *or*
that the pool of buffers used for untagged messages is not shared with 
any other iSCSI
session.

> sending too many Async Messages is suspect. Problem though is that 
> this is a major functional difference between how an iSCSI system 
> would behave, and an iSCSI/iSER system would behave (thus potentially 
> exposing the dichotomy to layers further up in the stack).  The other 
> issue is that a genuine exception could be punished with an even 
> bigger one - leading to termination of several I/Os.
>
> Note that while #2
> alone is necessary and sufficient for operation without connection 
> drops, others work to
> minimize the probability of invoking graceful handling (#2) but 
> together ultimately allow
> a connection failure if the graceful handling service cannot be 
> maintained by the RNIC
> after several DDP exceptions.  In the opinion of iSER authors, this 
> integrated approach is quite reasonable.
>
Option 2 is not a real solution.  It is simply attempting to shove the 
excess messages into
buffers that are accounted for (either "system buffers" or by abusing 
the network itself as a
buffer forcing false retransmits).   But ultimately it is just hoping 
that the problem goes away
because it does nothing to ensure that only the connection causing the 
problem will
have the corrective action taken against it.

> I realize that an argument can be made for #3 (which Caitlin & Alex 
> did as I understand the
> exchanges so far).  More comments from others would help the authors 
> understand the WG position on this.
>

I believe the RDDP WG position on this is clear.  In my opinion option, 
only options 1 and 3
comply with the RDDP Security requirements for use of shared receive 
resources.

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 08 Oct 2004 11:50:37 -0700

Caitlin,

#1: I do not see any major disagreement here.

#2: You say "It is a good idea, one that local interfaces *should*
offer.  But we all know that it isn't part of the de facto standard."
I don't understand this.  I presume you're referring to Verbs in the 
second setence, but #2 has nothing to do with Verbs as far as I can 
tell.  This can be done totally under the covers by the RNIC.  The 
latest DDP spec does allow (or even encourage) this behavior.

#3: I think you are agreeing with this option.

in the protocol itself.

I wish we know the "right" limit, but we don't.  So I don't see any 
qualitative difference from current iSER's approach.

 >The

IMHO, to meet this requirement, the number of posted "unexpected 
buffers" would have to be so high that it becomes very expensive and 
thus leads implementers to wanting #2 anyway.

Besides, in a qualitative sense, let me point out that this option has 
the exact *same* "best effort" flow control nature associated with 
today's iSER spec approach (i.e. neither is "positive" flow control, 
without help from #2), except this will turn out be more demanding on 
buffers.  I'm not sure why you don't prefer the current approach but 
recommend this one.

#4: No one wants it.

#5: I don't see any major disagreement.  We both agree that this is not 
a self-sufficient option for different reasons.

#6: I presume agreement.

 > buffer forcing false retransmits).

I disagree.  This is exactly the same thing host-based overcommitting 
TCP/IP implementations do.  If the stack runs out of mbufs, the frame is 
dropped on the floor and recovered later.  Killing a TCP connection 
under the same exception (buffer overflow) scenario when it runs on an 
RNIC is ridiculous, that's why the RDDP WG allowed the DDP spec to 
change to accommodate graceful handling.

 >But ultimately it is just hoping

Disagree again, even more.  iSCSI/iSER layers of course *do not* just 
sit around hoping that the problem goes away.  They do inspect the PDUs, 
confirm the legitimacy (or otherwise), replenish buffers or kill the 
connection - all at the application level in an aggressive fashion. 
It's just that RNIC clients don't need premature "help" from the RNIC in 
killing the connection.  That's what #2 is all about.

 > comply with the RDDP Security requirements for use of shared receive

I really would like us to view S-RQ as a "dampener" in invoking #2 
(graceful handling), not as a mandatory feature that iSER requires - 
because it doesn't (though I as an implementer would require S-RQ in my 
implementation to realize the dampening).

The net is that I see the real major disagreement is only wrt #2 - *not* 
on #2's desirability, but wrt iSER relying on #2.

Thanks.
-- 
Mallikarjun

Mallikarjun Chadalapaka

Caitlin Bestler wrote:

> There is also "3a" which is to make a default limit that is specified in 
> the protocol itself.
> 
> Even without SRQs, it is not that difficult for the receiving iSER layer 
> to allow sufficient
> excluded from the normal iSCSI flow control precisely because they were not
> (even without thinking about it) or have the iSER layer examine packets at
> wire speed.
> 
>>

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 9 Oct 2004 17:06:50 +0200

On Oct 8, 2004, at 8:50 PM, Mallikarjun C. wrote:

> second setence,
> but #2 has nothing to do with Verbs as far as I can tell.  This can be 
> done totally
> under the covers by the RNIC.  The latest DDP spec does allow (or even 
> encourage)
> this behavior.

I was envisioning a synchronous exception callback from the RI to a 
privileged
application that said "ALERT imminent connection failure, provide a 
buffer NOW
or else!"

What you were thinking of is a non-standard interface between RDDP and 
the LLP.
The RDDP spec *allows* an implementation to offer such a mechanism as a
*fault recovery* feature.  It is *not* something that any application 
should *rely*
upon, and it is *not* something that can be required of the RDDP layer.

First, it is absolutely impossible when RDDP is implemented cleanly
on top of the LLP.  Even in hardware implementations this can be
the case, and will indeed be common if the RNIC also supplies TOE
services.

> qualitative difference from current iSER's approach.
It would be a published standard that implementers could use as 
justification.
This is a minimal solution that is vital for creating a level playing 
field.  But the
explicitly negotiated limit has advantages for the cases John 
identified.
> > excess messages into  buffers that are accounted for (either "system
> > buffers" or by abusing the network itself as a buffer forcing false 
> retransmits).
>
> TCP/IP implementations do.
> If the stack runs out of mbufs, the frame is dropped on the floor and 
> recovered later.  Killing a TCP
> connection under the same exception (buffer overflow) scenario when it 
> runs on an RNIC is ridiculous,
> that's why the RDDP WG allowed the DDP spec to change to accommodate 
> graceful handling.
>

As noted earlier, the RDDP WG also said that the RDDP Consumer could not
rely upon the existence of such a mechanism.  It only agreed to remove 
the
requirement that RDDP MUST terminate the connection in response.  
Eliminating
the requirement that the Highway Patrol shoot everyone going over the 
speed
limit is quite different from repealing the speed limit.

If you send more untagged buffers than you are allowed to your 
connection
MAY be terminated.  You MUST NOT complain about it.  If you advertise
X credits you are expected to do so in good faith, and are expected to
have those buffers available when needed.  You are not supposed to
over-advertise and then rely on the network to sweep your mistake
under the rug.  That's like a bank saying failure to give a depositor
back their money is all right because they can always get Welfare.
Statistically under-allocating advertised buffers is fine and proper,
but whether you are working at the LLP or RDMAP layers the expectation
is that the network is not going to pay for your mistaken estimates.

>
>
> The net is that I see the real major disagreement is only wrt #2 - 
> *not* on #2's desirability, but wrt iSER relying on #2.
>

Exactly,  #2 is an implementation strategy that should not be 
forbidden, but neither should it be relied upon.