Subject: Transport neutral connections and SCTPSCT

From: Caitlin Bestler <[email-masked]>
Date: Fri, 9 May 2003 13:36:32 -0500

Some background:

The Direct Access Transport Collaborative
(www.datcollaborative.org) defines transport neutral APIs
for RDMA capable fabrics.

The IETF's Remote Direct Data Placement (RDDP) working group
is in the process of defining RDMA over IP, with mappings
for SCTP and MPA/TCP.

I am trying to determine how to map the DAT Collaborative's
connect method to SCTP. The goal is to have a natural method
for the ULP to specify their remote end that can be
implemented naturally and efficiently over InfiniBand,
TCP/IP and SCTP.

To support both InfiniBand and TCP/IP, the consumer
specifies the remote IA Address (An IP Address or an IB GID)
and a 64-bit "Connection Qualifier". The entire 64 bits is
used as Service ID for InfiniBand, the least significant 16
bits is used as the TCP port number for TCP/IP.

The question is, how should those 64 bits be mapped for
SCTP? Or more precisely, what if any methods are people
already using to create "connections" within an existing
SCTP association?

My assumption is that there are many multi-threaded
applications that open several "connections" to the same
server. A primary example would be a web or email server
that makes use of a file server. Each thread servicing an
exteranl client would have its own "connection" to the file
server.

RDDP defines that an "RDDP Stream" is carried over a lower
layer protocol stream (LLP Stream), such as a TCP connection
or a matched pair of SCTP streams.

So when mapped to TCP, each "connection" would in fact be a
distinct TCP connection -- complete with all of the
associated overhead, its own slow startup, etc. [This
is preaching to the choir, isn't it?]

When mapped to SCTP it  would be desirable to have each
"connection" be a matched pair of SCTP streams. The question
is, how should this be specified?

Ultimately an SCTP "LLP Stream" would be fully identified
by the following, once established:

    The identifiers of the association:
        source/dest IP Address lists, source/dest port #s.
    The Stream ID.
    
That can be mapped to the API. But does it always make sense
to specify the entire set. Typically, TCP applications only
specify the remote IP Address and port -- nothing local.

A few possible approaches:

    The active side specifies the remote IP Address and 
    the port number. If an association already exists
    a new port number is allocated, if not the association
    is established. This is *very* TCP compatible.
    
    The active side specifies the source port, destination
    port and stream ID within the 64 bit Connection
    Qualifier. The source address can be bound just as
    for existing protocols. This is very flexible, but
    doesn't seem very friendly for application threads
    who just want to open "a connection" to the file server.
    
    The association is established to a fixed port number.
    The first message sent on any stream is the equivalent
    of an InfiniBand CM Connection Request or Reply.
    Meaning it would simply pass the 64 bit Connection
    Qualifier to the other side for matching. This is
    very InfiniBand compatible.
    
Any opinions on the relative merits of the above would be
greatly appreciated. Any other solutions that others have
deployed to sharing associations across multiple threads
that are not naturally pre-configured would also be very
helpful.

Note: the RDDP mapping to SCTP uses only unordered delivery.
Therefore there is no real cost to reserving as many streams
as might be needed when the Association is created.

Caitlin Bestler
http://asomi.com/CaitlinBestler/