Subject: Re: sctp_peeloff()

From: Caitlin Bestler <[email-masked]>
Date: Fri, 19 Dec 2003 11:31:18 -0600

On Dec 17, 2003, at 6:17 PM, Gabriel Kaelin wrote:

> Michael Tuexen wrote:
>> let us see if people want this feature in the API.
>
> I'm not an implementer of SCTP, but I'm currently studying the 
> applicability of SCTP
> to the Coda file system. To be able to peel off streams sounds 
> interesting from a
> performance point of view, because Coda clients and servers run in 
> user space:
>
> There is a lot of meta data being transported, but also files. Files 
> don't need the
> application level attention during transfer, therefore it would be 
> nice if everything
> could happen in kernel space, e.g. by using sendfile(). However, this 
> requires a dedicated
> socket, so to peel one stream temporarily off would really be nice.
>
> In this case I can't see a risk of getting stuck in rwnd issues. Well, 
> this is a selfish
> point of view, but maybe you agree, that this is a good application of 
> this stream peeling off.
> Besides, the whole discussion about giving more responsibility to the 
> application layer is
> the fundamental question about mechanism vs. policy, isn't it? Me 
> personally, I'm for mechanism,
> it usually has a longer life than policy. ;-)
>

Generally, sharing an association across process boundaries is nearly 
as unwise as sharing
the space for a general purpose heap across process boundaries. If all 
of the applications
behave you do indeed have some savings, but that is a very big if.

But, if the association belongs to the "file system", and what you want 
to do is target
data chunks by stream directly into user specified buffers (for zero 
copy), then I would
see per-stream buffers and SCTP as being useful. The key is that once 
ordered messages
are involved, relying on *multiple* distinct user processes to dequeue 
them is at
the minimum very risky.

Even dealing only with unordered chunks, the DDP mapping to SCTP was 
envisioning a single
association per user process. That can still be quite useful. If the 
file system client
is an email or web daemon it is quite likely to have multiple 
concurrent file transfers
pending for a variety of clients. I'm not certain if there is enough of 
a benefit to
do this with just SCTP, as opposed to using RDMA over SCTP. But there 
is definitely
a benefit to any client that has multiple pending file transfers to 
using SCTP with
per-stream buffering. Being forced to use intermediate buffers could 
even make the
solution less efficient that the use of multiple TCP connections.