Subject: [Ips] Open-iSCSI High-Performance Initiator for Linux/Unix

From: Dmitry Yusupov <[email-masked]>
Date: Wed, 09 Mar 2005 08:09:46 -0800

This is to announce Open-iSCSI project: High-Performance iSCSI Initiator
for Linux. Though OpenSolaris port is a work in progress.

MOTIVATION
==========

Our initial motivations for the project were: (1) implement the right
user/kernel split, and (2) design iSCSI data path for performance.
Recently we added (3): get accepted into the mainline kernel.
 
As far as user/kernel, the existing iSCSI initiators bloat the kernel
with ever-growing control plane code, including but not limited to:
iSCSI discovery, Login (Authentication and Operational), session and
connection management, connection-level error processing, iSCSI Text,
Nop-Out/In, Async Message, iSNS, SLP, Radius... Open-iSCSI puts the
entire control plane in the user space. This control plane talks to the
data plane via well defined interface over the netlink transport.
 
(Side note: prior to closing on the netlink we considered: sysfs, ioctl,
and syscall. Because the entire control plane logic resides in the user
space, we needed a real bi-directional transport that could support
asynchronous API to transfer iSCSI control PDUs: Login, Logout, Nop-in,
Nop-Out, Text, Async Message.

Performance.
This is the major goal and motivation for this project. As it happens,
iSCSI has to compete with Fibre Channel, which is a more entrenched
technology in the storage space. In addition, the "soft" iSCSI
implementation have to show good results in presence of specialized
hardware offloads.

Our today's performance numbers are: 

- 450MB/sec Read on a single connection (2-way 2.4Ghz Opteron, 64KB
block size);

- 320MB/sec Write on a single connection (2-way 2.4Ghz Opteron, 64KB
block size);

- 50,000 Read IOPS on a single connection (2-way 2.4Ghz Opteron, 4KB
block size).

Prior to starting from-scratch the data path code we did evaluate the
sfnet Initiator. And eventually decided against patching it. Instead, we
reused its Discovery, Login, etc. control plane code.
Technically, it was the shortest way to achieve the (1) and (2) goals
stated above. We believe that it remains the easiest and the most
practical thing on the larger scale of: iSCSI for Linux.

STATUS
======

There's a 100% working code that interoperates with all (count=5) iSCSI
targets we could get our hands on.

The software was tested on AMD Opteron (TM) and Intel Xeon (TM). 

Code is available online via either Subversion source control database
or the latest development release (i.e., the tarball containing Open-
iSCSI sources, including user space, that will build and run on kernels
starting 2.6.10).

        http://www.open-iscsi.org
 
Features:

    - highly optimized and small-footprint data path;
    - multiple outstanding R2Ts;
    - thread-less receive;
    - sendpage() based transmit;
    - zero-copy header processing on receive;
    - no data path memory allocations at runtime;
    - persistent configuration database;
    - SendTargets discovery;
    - CHAP;
    - DataSequenceInOrder=No;
    - PDU header Digest;
    - multiple sessions;
    - MC/S (note: disabled in the patch);
    - SCSI-level recovery via Abort Task and session re-open.

TODO
====

The near term plan is: test, test, and test. We need to stabilize the
existing code, after 5 months of development this seems to be the right
thing to do.

Other short-term plans include:

a) process community feedback, implement comments and apply patches;
b) cleanup user side of the iSCSI open interface; use API calls (instead
of directly constructing events); 
c) eliminate runtime control path memory allocations (for Nop-In, Nop-
Out, etc.);
d) implement Write path optimizations (delayed because of the self-
imposed submission deadline);
e) oProfile the data path, use the reports for further optimization;
f) complete the readme.

Comments, code reviews, patches - are greatly appreciated!

THANKS
======

Special thanks to our first reviewers: Christoph Hellwig and Mike
Christie.

Special thanks to Ming Zhang for help in testing and for insightful
questions.

Regards,

Alex Aizman & Dmitry Yusupov

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips