Subject: Re: [Ips] I-D ACTION:draft-ietf-ips-iwarp-da-00.txt - Flow Control

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 21 Sep 2004 22:14:22 -0700

Mike is right about the "demand" part, I wasn't planning on adding the 
suggested text verbatim for that reason ("along these lines").  The text 
I had planned to add is the following:

"In order to meet the functional requirements listed in this section, 
certain Datamover protocols may require pre-posted buffers from the 
local iSCSI protocol layer via mechanisms outside the scope of this 
document and in some implementations, the absence of such buffers may 
result in a connection failure.  Datamover protocols may also realize 
these functional requirements via methods not explicitly listed in this 
document."

Regarding implications to iSER especially, I do not think there are any. 
  Wearing the iSER co-author's hat, I think iSER has the right approach 
to flow control.  I thought the previous iSER flow control threads 
generally agreed that it's the case.

And finally, I do not believe this is a change per se either.  The 
architecture draft from the beginning intended to leave the flexibility 
of a specific approach to each Datamover protocol spec.  The new text is 
only making that intent explicit.  For the same reason, I personally 
think we wouldn't lose anything important if the new text isn't added.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

Mike Ko wrote:

> I am uncomfortable with adding the statement that "the Datamover MAY 
> demand pre-posted buffers for all iSCSI PDUs".  What mechanisms will be 
> available to the datatmover to allow it to determine how many pre-posted 
> buffers it should "demand" and when it should "demand" the buffers?  Are 
> we revisiting the idea of setting limits on the number of async messages, 
> immediate commands, etc.?  Before we start changing the Datamover 
> Architecture, I think we ought to understand the implication of the 
> changes first.
> 
> Mike
> Sent by:        [email-masked]
> To:     [email-masked]
> cc:      
> Subject:        Re: [Ips] I-D ACTION:draft-ietf-ips-iwarp-da-00.txt - Flow 
> Control
> 
> 
> 
> 
>>Of course a Datamover could be using a transport with flow-control, in
>>which
>>case it will not have to demand pre-posted buffers. But it should be 
> 
> made
> 
>>clear that the Datamover MAY demand pre-posted buffers for all iSCSI
>>PDUs and that absence of a buffer MAY result in a connection failure.
> 
> 
> Okay, I can add some text to the next draft along these lines unless
> someone objects.
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions
> Hewlett-Packard MS 5668
> Roseville CA 95747
> cbm [at] rose.hp.com
> 
> 
> 
> 
> Caitlin Bestler wrote:
> 
> 
>>On Sep 15, 2004, at 12:27 AM, Mallikarjun C. wrote:
>>
>>
>>>Caitlin,
>>>
>>>
>>>
>>>>In my opinion the absence of enabling language
>>>>would actually forbid some of the solutions you listed.
>>>
>>>It was not intended to.  I am open to making changes if you can
>>>identify the specific
>>>DA language that actually forbids the options that I listed.
>>
>>
>>The term guarantee mandates that barring factors outside of its control,
>>such as
>>a transport layer error, that the Datamover MUST implement algorithms
>>that will
>>actually guarantee delivery.
>>
>>There are only two ways to accomplish that: stalling transmits, and
>>demanding
>>that the iSCSI layer take deliver of iSCSI PDUs on demand (that is the
>>interface
>>at the receiving end is a push rather than a pull), or that there be
>>flow control
>>which will prevent requests from being sent (either by blocking until
>>they can
>>be sent, or by denying a request and making the consumer try again 
> 
> later).
> 
>>A TCP transport accomplishes the pacing at the sending end, that is the
>>sender
>>is either blocked or temporarily denied (depending on whether the socket 
> 
> is
> 
>>blocking or non-blocking).
>>
>>But that is not how any existing iWarp local interface works.  They 
> 
> assume
> 
>>that the poster has knowledge that there is a buffer available at the
>>other end,
>>and that if the poster is wrong in that assumption it is a ULP error
>>that warrants
>>breaking the connection. (Yes, the implementation is allowed to try to
>>recover
>>from the fault, but it is an application fault and the application is
>>not guaranteed
>>that their mistake will be covered).
>>
>>So an iWarp Datamover is incapable of guaranteeing delivery unless the
>>caveat
>>is added that there is a valid reason to believe that there is a waiting
>>buffer at
>>the other end. Shifting the language to reflect that reality won't have
>>that much
>>impact, because iSCSI already requires the sender to have one form of
>>credit
>>or another for nearly all iSCSI PDUs.
>>
>>The other possible solution is to require to deliver the iSCSI PDU to
>>the iSCSI
>>layer without a receiving buffer.  That is, the iSCSI layer MUST accept
>>callbacks
>>that deliver iSCSI PDUs at line rate.  And if *it* wants to drop the
>>connection
>>because it doesn't have a buffer for them, then it isn't the Datamover 
> 
> that
> 
>>is failing to deliver on its guarantee.
>>
>>That option is actually compatible with the wire protocol, but it is not
>>compatible
>>with any of the published APIs or verbs for iWarp (or InfiniBand for
>>that matter).
>>
>>
>>One possible change would be to clarify the receiving end's 
> 
> responsibility
> 
>>to provide receiving buffers for all legitimate iSCSI PDUs that could
>>arrive
>>on the connection -- and that failure to do so MAY result in Connection
>>Failure.
>>That is, the Datamover *only* guarantees delivery under certain 
> 
> conditions,
> 
>>and guarantees that delivery will be made *or* a connection failure will 
> 
> be
> 
>>reported.  Further connection failuers will be due to a problem outside
>>of its control (transport error or failure of the receiver to supply a
>>buffer).
>>
>>Of course a Datamover could be using a transport with flow-control, in
>>which
>>case it will not have to demand pre-posted buffers. But it should be 
> 
> made
> 
>>clear that the Datamover MAY demand pre-posted buffers for all iSCSI
>>PDUs and that absence of a buffer MAY result in a connection failure.
>>
>>That is a *very* different, but more applicable, requirement than simply
>>stating the Datamover "guarantees' delivery.

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Caitlin Bestler" <[email-masked]>
Date: Wed, 22 Sep 2004 02:18:02 -0500 (CDT)

> think we wouldn't lose anything important if the new text isn't added.

That does clarify the responsibility for me, and I agree that it
matches the draft's intent (my objection was that I thought the
prior wording actually implied a requirement that was not intended).

One possible clarification would be to state that the Datamover
Consumer may/should assume that it's peer is following all iSCSI
flow control when determining how many buffers to pre-post.

It may also be worth noting that other Datamover architectures may
block on or temporarily reject sending of iSCSI PDUs. That is, they
may be using traditional socket semantics.

A robust iSCSI design would be prepared for either Datamover option.

--
Caitlin Bestler
http://asomi.com/