Subject: Re: sctp_peeloff()

From: Caitlin Bestler <[email-masked]>
Date: Thu, 11 Dec 2003 15:24:27 -0600

On Dec 11, 2003, at 10:42 AM, Kacheong Poon wrote:

> Michael Tuexen wrote:
>
>> But my main point is: for supporting different receive buffers
>> for different streams you also need different flow controls for
>> different streams. Do you agree with this statement?
>
> It really depends.  I think what Caitlin's point is that an app
> does not have control over on putting data from one particular
> stream to one particular pool of buffers.  For example, in the current
> socket API, an app gives a buffer the recvmsg().  It does not know
> which stream's data will be put into that buffer.  With stream-socket,
> the app can control this as the buffer given to recvmsg() will
> only be filled with data from a particular stream.  This has nothing
> to do with flow control.  The flow control issue comes up at the
> protocol level.
>
>> Then there is the next problem: Can we extend SCTP (or even do we
>> need to extend) to provide a per stream flow control. Some guys
>> thought about this already and the result was, that it is not
>> possible to do as an extension. Of course, SCTP could have been
>> designed to provide that feature. But it was not done that way.
>
> Again, the API is not about changing SCTP.  This needs to be
> emphasized.

Absolutely correct. The issues here have nothing to do with the
end-to-end semantics SCTP, but rather with the semantics of the
socket interface.

Michael' key objection, as I understand it, is that any stream
specific receive will lead to laggard messages that cannot be
delivered. Fundamentally, the API layer is expected to deliver
the message with the oldest TSN first because that will enable
advancing the receive window.

That is a valid concern. But you can achieve stream-specific
buffering without causing the receive window to be tardy.

RDDP's use of SCTP (for RDMA services) relies on per stream
buffering for untagged messages and strictly upon ULP flow
control. SCTP flow control is used solely to manage network
congestion, and theoretically for SCTP layer buffering
(although the RDDP adaptation of SCTP uses only unordered
delivery, thereby enabling the SCTP layer to function
without its own buffers. It can place directly from the
IP layer buffers to the ULP buffers).

Therefore the key conflict here is not between per-stream
receive buffering and SCTP, but between per-stream receive
buffering and normal socket semantics.

Applications have gotten used to having the socket interface
provide critical ULP flow control: pause receiver until
message from peer arrives, pause transmitter until peer
is ready to receive more messages.

Inherently, the socket layer will hold any messages that
it allowed to be sent that the ULP is not ready to receive.

Michael is assuming that because SCTP does not have per-
stream flow control that per stream buffering will not work.
What is more accurate, is that it means that inferring
per-stream flow control from the sockets interface will
not work.

To successfully provide per-stream buffering, an API must
have the following rules:

	The receiver, through some mechanism, must have
	informed the sender of the maximum number of messages
	that can be in-flight. For most RDMA applications this
	is a pre-arranged or negotiated number of credits. Credits
	are reduced when a message is sent, and incremented when
	the receiver replies. For non-RDMA applications it would
	probably work to increase credits as prior messages were
	acked.

	When any authorized message is received, the API layer
	MUST be able to forward it to a ULP buffer immediately.
	This implies a callback that says "here, this messge is
	yours -- do something with it NOW", and/or pre-posting of
	receive buffers.

	With pre-posting,  an unauthorized message MAY be discarded,
	placing the stream in a "terminated" state. Essentially, if
	the ULP fails to provide sufficient buffers the API layer
	would substitute the bit bucket as the destination.

The key is that advance the receive window must never be held
up because the application failed to ask for buffer on a specific
stream. I do not see how you can extend the socket paradigm
to match these rules, but APIs based upon callbacks and/or RQ/CQs
would handle this easily.

Obviously all streams within an association should be related.
They are sharing the association's bandwidth, and probably resources
that do not share across process boundaries easily. But there are
still many examples of applications that could naturally be using
different streams to concurrently receive data into different
target buffers. FTP, HTTP, email, streaming media, SAN and NAS
come to mind quickly.