Subject: Re: sctp_peeloff()

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 11 Dec 2003 15:52:36 -0700

Caitlin,

Per-stream flow control can be effectively performed by both the
receiver and the transmitter.  How each proportions its receive
and transmit buffer is up to it.  The end-point implementation
can even track the number of oustanding bytes per stream and
proportion its local transmit buffer in accordance with the
remote end's receive buffer if it wants (full flow control).

The only protocol difficulty I know of is when messages are
being fragmented and MUST be sent with consecutive TSNs,
blocking all other streams off of the wire and the rwnd,
introducing HOL blocking again.  Relaxing that constraint, as is
easily done for most implementations, would free the protocol
from this defficiency in design.

If you don't permit a read from a stream, you will just force
the user to flock and peek the socket, something like:

mutex-lock-
loop: no-wait-poll flock -peek- -read-if-correct-sid- else
      cond-signal funlock cond-wait-mutex-unlock wakeup-mutex-locked

Which could more easily be performed in the kernel, after all
socket sendmsg recvmsg functions are supposed to be thread-safe
as well a respecting mandatory POSIX file locking.

Well, if it won't be done by the kernel, the sockets library
will wind up doing it.  But why do it so different than TCP or
standard SOCK_SEQPACKET?

--brian

On Thu, 11 Dec 2003, Caitlin Bestler wrote:

> 
> On Dec 11, 2003, at 10:42 AM, Kacheong Poon wrote:
> 
> > Michael Tuexen wrote:
> >
> >> But my main point is: for supporting different receive buffers
> >> for different streams you also need different flow controls for
> >> different streams. Do you agree with this statement?
> >
> > It really depends.  I think what Caitlin's point is that an app
> > does not have control over on putting data from one particular
> > stream to one particular pool of buffers.  For example, in the current
> > socket API, an app gives a buffer the recvmsg().  It does not know
> > which stream's data will be put into that buffer.  With stream-socket,
> > the app can control this as the buffer given to recvmsg() will
> > only be filled with data from a particular stream.  This has nothing
> > to do with flow control.  The flow control issue comes up at the
> > protocol level.
> >
> >> Then there is the next problem: Can we extend SCTP (or even do we
> >> need to extend) to provide a per stream flow control. Some guys
> >> thought about this already and the result was, that it is not
> >> possible to do as an extension. Of course, SCTP could have been
> >> designed to provide that feature. But it was not done that way.
> >
> > Again, the API is not about changing SCTP.  This needs to be
> > emphasized.
> 
> 
> Absolutely correct. The issues here have nothing to do with the
> end-to-end semantics SCTP, but rather with the semantics of the
> socket interface.
> 
> Michael' key objection, as I understand it, is that any stream
> specific receive will lead to laggard messages that cannot be
> delivered. Fundamentally, the API layer is expected to deliver
> the message with the oldest TSN first because that will enable
> advancing the receive window.
> 
> That is a valid concern. But you can achieve stream-specific
> buffering without causing the receive window to be tardy.
> 
> RDDP's use of SCTP (for RDMA services) relies on per stream
> buffering for untagged messages and strictly upon ULP flow
> control. SCTP flow control is used solely to manage network
> congestion, and theoretically for SCTP layer buffering
> (although the RDDP adaptation of SCTP uses only unordered
> delivery, thereby enabling the SCTP layer to function
> without its own buffers. It can place directly from the
> IP layer buffers to the ULP buffers).
> 
> Therefore the key conflict here is not between per-stream
> receive buffering and SCTP, but between per-stream receive
> buffering and normal socket semantics.
> 
> Applications have gotten used to having the socket interface
> provide critical ULP flow control: pause receiver until
> message from peer arrives, pause transmitter until peer
> is ready to receive more messages.
> 
> Inherently, the socket layer will hold any messages that
> it allowed to be sent that the ULP is not ready to receive.
> 
> Michael is assuming that because SCTP does not have per-
> stream flow control that per stream buffering will not work.
> What is more accurate, is that it means that inferring
> per-stream flow control from the sockets interface will
> not work.
> 
> To successfully provide per-stream buffering, an API must
> have the following rules:
> 
> 	The receiver, through some mechanism, must have
> 	informed the sender of the maximum number of messages
> 	that can be in-flight. For most RDMA applications this
> 	is a pre-arranged or negotiated number of credits. Credits
> 	are reduced when a message is sent, and incremented when
> 	the receiver replies. For non-RDMA applications it would
> 	probably work to increase credits as prior messages were
> 	acked.
> 
> 	When any authorized message is received, the API layer
> 	MUST be able to forward it to a ULP buffer immediately.
> 	This implies a callback that says "here, this messge is
> 	yours -- do something with it NOW", and/or pre-posting of
> 	receive buffers.
> 
> 	With pre-posting,  an unauthorized message MAY be discarded,
> 	placing the stream in a "terminated" state. Essentially, if
> 	the ULP fails to provide sufficient buffers the API layer
> 	would substitute the bit bucket as the destination.
> 
> The key is that advance the receive window must never be held
> up because the application failed to ask for buffer on a specific
> stream. I do not see how you can extend the socket paradigm
> to match these rules, but APIs based upon callbacks and/or RQ/CQs
> would handle this easily.
> 
> Obviously all streams within an association should be related.
> They are sharing the association's bandwidth, and probably resources
> that do not share across process boundaries easily. But there are
> still many examples of applications that could naturally be using
> different streams to concurrently receive data into different
> target buffers. FTP, HTTP, email, streaming media, SAN and NAS
> come to mind quickly.
> 
> 

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦