Subject: RE: [Ips] iSER flow control options (Was: Re: DA+iSER Flow Control, Limiting number of oustanding cmdsattarget)

From: "Carrier, John" <[email-masked]>
Date: Thu, 7 Oct 2004 05:08:00 -0700

I have nothing really to add to the conversation, other than I agree with Caitlin's analysis.   

The iSER design team relied on the RNIC to handle RQ overruns gracefully.  But for SRQs, the verbs were modified to provide a mechanism for the RNIC to report to the ULP when a connection over-consumed SRQ resources.  Since verbs are currently out of the scope of the IETF, this leaves iSER vulnerable as Caitlin has described.

I appreciate JohnH's concerns about the effects credit-based flow control could have on iSCSI at a distance, but that argument just highlights how iSCSI relies on system buffers consumed by the transport for its flow control.  An RNIC isn't designed for this level of resource availability. 

--jc

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Caitlin Bestler
> Sent: Thursday, October 07, 2004 2:55 AM
> To: Mallikarjun C.
> Cc: [email-masked]
> Subject: Re: [Ips] iSER flow control options (Was: Re: DA+iSER Flow
> Control,Limiting number of oustanding cmdsattarget)
> 
> 
> 
> On Oct 7, 2004, at 3:01 AM, Mallikarjun C. wrote:
> 
> > I finally managed to catch up on this thread, let me attempt to 
> > summarize what I think are
> > the options for iSER flow control.  I would like to get 
> more comments 
> > to assess the WG
> > position, perhaps also from the co-chairs on this topic.  I 
> hope this 
> > long email acts as a
> > helpful summary.
> >
> > Let me ignore S-RQ for a moment and consider a simple RQ.  
> Let's even 
> > assume the target
> > had adequately provisioned receive buffers (CmdSN window + 
> 2 buffers, 
> > i.e. no overcommitting).
> > In this simplistic case, if the initiator sends too many immediate 
> > commands, we know that the
> > target would run out of receive buffers.  Now the options 
> that I see 
> > to deal with this scenario
> > are (you may add more if I missed any):
> >
> > 1. Replenish buffers faster than the arrival of the unexpected 
> > messages (iSCSI layer may later
> >     discard the "too many immediate commands" if it deems so).
> > 2. Require the RNIC to do graceful handling of lack of 
> receive buffers 
> > on these exceptions.
> > 3. Define a new iSCSI MaxOutstandingUnexpectedPDUs key-based limit 
> > scheme to deal with
> >     these exceptions (that John describes below).
> > 4. Define an iSER end2end flow control protocol (such as 
> the one iSER 
> > design team originally
> >     considered).
> > 5. Mandate an S-RQ as the mandatory RNIC feature (so there's more 
> > flexibility for overprovisioning
> >     across multiple connections).
> > 6. Allow the connection to be dropped because the buffers 
> are overrun.
> >
> >
> 
> > #1 may be feasible (and every implementation tries to do it 
> anyway), 
> > but basing the protocol on that
> > assumption doesn't strike me as a "standards" way of specifying a 
> > protocol.
> 
> I think the operative word here is "try".  What I am particularly 
> concerned about is that for both TCP
> and SCP transport level flow control is based on the availability of 
> receive buffering and network
> congestion.
> 
> A TCP connection or SCTP association in use for RDDP 
> effectively has no 
> buffer limits, assuming
> it can forward DDP Segments to the RDDP layer at wire speed.  RDDP is 
> responsible for placing
> the buffers when presented, or terminating the stream.  It has no 
> back-pressure mechanism.
> 
> However, many implementations will not tailor the calculation of the 
> buffer window for RDMA,
> but rather use the general purpose algorithm.  That can 
> result in false 
> blocking of placable
> traffic, but this failure will not be apparent when users are used to 
> TCP bottlenecking.
> 
> Existing iSCSI implementations in particular could easily be 
> relying on 
> TCP bottlenecks.
> When they test on an implementation that does not tailor the 
> TCP window 
> on RDMA connections
> to "no system buffering, hence no buffer limits" they will not detect 
> that the problem is still there.
> Later they will test on an implementation that *does* fully eliminate 
> the TCP bottleneck, and
> promptly declare that "this RNIC breaks my iSCSI" and demand 
> the vendor 
> "fix it".
> 
> >
> > #2 is quite nice from iSCSI & iSER's perspective.  We 
> should be able 
> > to cast any
> > immediate/un-flow-controlled iSCSI PDU as an "exception" 
> (which it is) 
> > message
> > (in DDP parlance) and thus get around adding any more protocol 
> > restrictions in iSER.
> > The major consideration here is the feasibility (from an 
> RNIC vendor 
> > perspective) of
> > realizing such a graceful handling.
> 
> That is moving the "take it or flush it" interface up from the LLP to 
> RDDP interface to
> the RDMAP to ULP interface.  It is a good idea, one that local 
> interfaces *should*
> offer.  But we all know that it isn't part of the de facto 
> standard. It 
> is one of many
> defects in that interface, but I would still oppose adopting any ULP 
> that cannot
> be implemented on top of commonly deployed verbs.
> 
> >
> > #3 seems like a workable option if we can cleary specify it.  It 
> > however does add
> > more protocol complexity to iSCSI, and makes some exception 
> management 
> > cases
> > harder to deal with (when the limit is not "None") as John 
> points out 
> > below.
> 
> There is also "3a" which is to make a default limit that is specified 
> in the protocol itself.
> 
> I believe the goal should be that no application that is generating 
> these messages for
> their intended purposes should ever have to check the limit.  The 
> receiver SHOULD
> (or perhaps even MUST) set it high enough so that *only* a 
> malfunctioning or malicious
> client could exceed it.
> 
> The flow control can be fairly simple to implement however, 
> anytime the 
> flow control
> limit would have been hit, it is for an unanticipated message 
> that the 
> target iSCSI side
> would be free to throw away anyway.  The sender could simply do so 
> immediately.
> But I do think it would be better to define the credit limit 
> such that 
> it would be impossible
> to violate it when generating messages legitimately.
> 
> The goal here is to enable the receiving side to identify a runaway 
> malfunctioning or
> malicious client that is attacking shared resources *before* 
> it results 
> in a legitimate
> message from another client being dropped.
> 
> Even without SRQs, it is not that difficult for the receiving iSER 
> layer to allow sufficient
> extra receive buffers to deal with all *legitimate* iSCSI 
> packets.  It 
> is also certainly
> preferable for the receive side to have some extra buffers available 
> rather than
> requiring non-malfunctioning/non-abusive senders to check 
> their credit 
> limit
> before sending any of the "unaccounted" iSCSI PDUs.  These 
> messages were
> excluded from the normal iSCSI flow control precisely because 
> they were 
> not
> anticipated to be high volume.
> 
> So there are two ways to determine when a connection is sending 
> illegitimate
> iSCSI packets:  place a credit limit that legitimate 
> applications will 
> not violate
> (even without thinking about it) or have the iSER layer 
> examine packets 
> at
> wire speed.
> 
> >
> > #4 is clearly expensive although we know that it can work.  This 
> > should be the last resort.
> >
> > #5 is a second-order choice in that it may still require help from 
> > other options.  The main
> > considerations here are: making normative references to the 
> Verbs spec 
> > [ or defining the
> > S-RQ text in abstract terms (i.e. stripping off references to WQE, 
> > CQE, RQ, CQ, RI etc.)
> > which again doesn't strike me as a "standards" way of defining a 
> > protocol ], the acceptance
> > (from an RNIC vendor perspective) on requiring it.  We 
> still need to  
> > address a single
> > connection case from the specification perspective (i.e. 
> consider an 
> > embedded target
> > supporting a single iSCSI/iSER connection).
> 
> An SRQ does not eliminate the vulnerability.  In fact it is necessary 
> for it to exist.  A runaway
> client connected to a simple RQ can only tear down its own 
> connection.  
> It does not impact
> other clients.
> 
> With an SRQ, you either have a per-connection limit/alarm or you are 
> dependent on wire-speed
> draining of completions by the ULP (the iSER layer).  That case is 
> essentially option 1,
> enabling the per-connection limit/alarm requires having a 
> number to set 
> it with which
> leads back to option 3 (or 3a).
> 
> An argument for 3, as opposed to 3a, is to specifically allow "None". 
> But the iSER layer
> should only uses "None" when it is committing to drain completions at 
> wire speed *or*
> that the pool of buffers used for untagged messages is not 
> shared with 
> any other iSCSI
> session.
> 
> >
> > #6 is also feasible (I think Julian too hinted at this in 
> an email). 
> > Clearly, an initiator sending too many immediate commands 
> or a target 
> > sending too many Async Messages is suspect. Problem though is that 
> > this is a major functional difference between how an iSCSI system 
> > would behave, and an iSCSI/iSER system would behave (thus 
> potentially 
> > exposing the dichotomy to layers further up in the stack).  
> The other 
> > issue is that a genuine exception could be punished with an even 
> > bigger one - leading to termination of several I/Os.
> >
> > The way I see it, the current iSER spec depends on #2, #1, 
> #5 and #6. 
> > Note that while #2
> > alone is necessary and sufficient for operation without connection 
> > drops, others work to
> > minimize the probability of invoking graceful handling (#2) but 
> > together ultimately allow
> > a connection failure if the graceful handling service cannot be 
> > maintained by the RNIC
> > after several DDP exceptions.  In the opinion of iSER authors, this 
> > integrated approach is quite reasonable.
> >
> Option 2 is not a real solution.  It is simply attempting to 
> shove the 
> excess messages into
> buffers that are accounted for (either "system buffers" or by abusing 
> the network itself as a
> buffer forcing false retransmits).   But ultimately it is just hoping 
> that the problem goes away
> because it does nothing to ensure that only the connection 
> causing the 
> problem will
> have the corrective action taken against it.
> 
> > I realize that an argument can be made for #3 (which Caitlin & Alex 
> > did as I understand the
> > exchanges so far).  More comments from others would help 
> the authors 
> > understand the WG position on this.
> >
> 
> I believe the RDDP WG position on this is clear.  In my 
> opinion option, 
> only options 1 and 3
> comply with the RDDP Security requirements for use of shared receive 
> resources.
> 
> 
> _______________________________________________
> Ips mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ips
> 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips