Subject: Re: sctp_peeloff()

From: Gabriel Kaelin <[email-masked]>
Date: Fri, 19 Dec 2003 18:00:52 -0500

I realized that I forgot to send it to the mailing list as well...
-----
I put my comments below...

Caitlin Bestler wrote:
> On Dec 17, 2003, at 6:17 PM, Gabriel Kaelin wrote:
> 
>> Michael Tuexen wrote:
>>
>>> let us see if people want this feature in the API.
>>
>>
>> I'm not an implementer of SCTP, but I'm currently studying the applicability of SCTP
>> to the Coda file system. To be able to peel off streams sounds interesting from a
>> performance point of view, because Coda clients and servers run in user space:
>>
>> There is a lot of meta data being transported, but also files. Files don't need the
>> application level attention during transfer, therefore it would be nice if everything
>> could happen in kernel space, e.g. by using sendfile(). However, this requires a dedicated
>> socket, so to peel one stream temporarily off would really be nice.

>
> Generally, sharing an association across process boundaries is nearly as unwise as sharing
> the space for a general purpose heap across process boundaries. If all of the applications
> behave you do indeed have some savings, but that is a very big if.

In the general case, I agree with you. In Coda, the association would be
governed only by Coda's RPC implementation, which allows concurrent RPCs in one
single process. The file transfers, which are considered as RPC side effects,
happen under tight control of RPC2 as well. The idea would be, to use for
example a kernel thread dedicated to feeding or reading from the peeled off socket.

> But, if the association belongs to the "file system", and what you want to do is target
> data chunks by stream directly into user specified buffers (for zero copy), then I would
> see per-stream buffers and SCTP as being useful. The key is that once ordered messages
> are involved, relying on *multiple* distinct user processes to dequeue them is at
> the minimum very risky.

I can imagine that per-stream buffers could allow me to move large data
transfers mostly into kernel space, especially if I can provide user specified
per-stream send buffers as well. Unordered packets could still take the standard
path for delivery to the application.

> 
> Even dealing only with unordered chunks, the DDP mapping to SCTP was envisioning a single
> association per user process. That can still be quite useful. If the file system client
> is an email or web daemon it is quite likely to have multiple concurrent file transfers
> pending for a variety of clients. I'm not certain if there is enough of a benefit to
> do this with just SCTP, as opposed to using RDMA over SCTP. But there is definitely
> a benefit to any client that has multiple pending file transfers to using SCTP with
> per-stream buffering. Being forced to use intermediate buffers could even make the
> solution less efficient that the use of multiple TCP connections.