Subject: Re: sctp_peeloff()

From: Michael Tuexen <[email-masked]>
Date: Thu, 11 Dec 2003 22:47:15 +0100

Caitlin,
see my comments below.
Best regards
Michael

On Dec 11, 2003, at 10:24 PM, Caitlin Bestler wrote:

>
> On Dec 11, 2003, at 10:42 AM, Kacheong Poon wrote:
>
>> Michael Tuexen wrote:
>>
>>> But my main point is: for supporting different receive buffers
>>> for different streams you also need different flow controls for
>>> different streams. Do you agree with this statement?
>>
>> It really depends.  I think what Caitlin's point is that an app
>> does not have control over on putting data from one particular
>> stream to one particular pool of buffers.  For example, in the current
>> socket API, an app gives a buffer the recvmsg().  It does not know
>> which stream's data will be put into that buffer.  With stream-socket,
>> the app can control this as the buffer given to recvmsg() will
>> only be filled with data from a particular stream.  This has nothing
>> to do with flow control.  The flow control issue comes up at the
>> protocol level.
>>
>>> Then there is the next problem: Can we extend SCTP (or even do we
>>> need to extend) to provide a per stream flow control. Some guys
>>> thought about this already and the result was, that it is not
>>> possible to do as an extension. Of course, SCTP could have been
>>> designed to provide that feature. But it was not done that way.
>>
>> Again, the API is not about changing SCTP.  This needs to be
>> emphasized.
>
>
> Absolutely correct. The issues here have nothing to do with the
> end-to-end semantics SCTP, but rather with the semantics of the
> socket interface.
>
> Michael' key objection, as I understand it, is that any stream
> specific receive will lead to laggard messages that cannot be
> delivered. Fundamentally, the API layer is expected to deliver
> the message with the oldest TSN first because that will enable
> advancing the receive window.
>
> That is a valid concern. But you can achieve stream-specific
> buffering without causing the receive window to be tardy.
>
> RDDP's use of SCTP (for RDMA services) relies on per stream
> buffering for untagged messages and strictly upon ULP flow
> control. SCTP flow control is used solely to manage network
> congestion, and theoretically for SCTP layer buffering
> (although the RDDP adaptation of SCTP uses only unordered
> delivery, thereby enabling the SCTP layer to function
> without its own buffers. It can place directly from the
> IP layer buffers to the ULP buffers).
>
> Therefore the key conflict here is not between per-stream
> receive buffering and SCTP, but between per-stream receive
> buffering and normal socket semantics.
>
> Applications have gotten used to having the socket interface
> provide critical ULP flow control: pause receiver until
> message from peer arrives, pause transmitter until peer
> is ready to receive more messages.
>
> Inherently, the socket layer will hold any messages that
> it allowed to be sent that the ULP is not ready to receive.
>
> Michael is assuming that because SCTP does not have per-
> stream flow control that per stream buffering will not work.
> What is more accurate, is that it means that inferring
> per-stream flow control from the sockets interface will
> not work.
>
> To successfully provide per-stream buffering, an API must
> have the following rules:
>
> 	The receiver, through some mechanism, must have
> 	informed the sender of the maximum number of messages
> 	that can be in-flight. For most RDMA applications this
> 	is a pre-arranged or negotiated number of credits. Credits
> 	are reduced when a message is sent, and incremented when
> 	the receiver replies. For non-RDMA applications it would
> 	probably work to increase credits as prior messages were
> 	acked.
What does 'the receiver replies' means. Sending a SACK? I have
the impression that you are running a thin layer on top of SCTP
doing the ACKs, is that right?
>
> 	When any authorized message is received, the API layer
> 	MUST be able to forward it to a ULP buffer immediately.
> 	This implies a callback that says "here, this messge is
> 	yours -- do something with it NOW", and/or pre-posting of
> 	receive buffers.
>
> 	With pre-posting,  an unauthorized message MAY be discarded,
> 	placing the stream in a "terminated" state. Essentially, if
> 	the ULP fails to provide sufficient buffers the API layer
> 	would substitute the bit bucket as the destination.
>
> The key is that advance the receive window must never be held
> up because the application failed to ask for buffer on a specific
> stream. I do not see how you can extend the socket paradigm
> to match these rules, but APIs based upon callbacks and/or RQ/CQs
> would handle this easily.
>
> Obviously all streams within an association should be related.
> They are sharing the association's bandwidth, and probably resources
> that do not share across process boundaries easily. But there are
> still many examples of applications that could naturally be using
> different streams to concurrently receive data into different
> target buffers. FTP, HTTP, email, streaming media, SAN and NAS
> come to mind quickly.
>
>