Subject: [Ips] [iSCSI] iSER on InfiniBand

From: John Hufferd <[email-masked]>
Date: Thu, 4 Nov 2004 11:23:45 -0800

Folks,
I am aware of a lot of discussions that have been going on recently 
regarding iSCSI/iSER operating on InfiniBand Networks.  There has been 
some authors that have been working to create some proposed wordage for 
the iSER document that would permit support of iSER by an InfiniBand 
Network.  Unfortunately, the work to propose some new wordage did not 
complete in time to avoid the IETF draft black-out period.  However, there 
was a document that was created that explained what was being discussed 
and why, and the WEB link to that document (on Julian's Web site) is 
included here:

http://www.haifa.il.ibm.com/satran/ips/iSER-in-an-IB-network-V9.pdf

I encourage you to take a look at this document.  It should also be noted 
that because of this work dealing with InfiniBand it was also possible to 
define how an SCTP implementation could also be created that could be 
conformant to some new proposed wordage for the iSER Draft (to be released 
some time after the Black-Out period).

Below is the introduction to the paper that was used as a base for some 
new proposed wordage.  (Please understand, this is not meant to do 
anything other than to start the discussions, I expect there will be a lot 
of additional work to clarify issues, and craft the most approprate 
wordage for the iSER document.)

Introduction

There has been great interest in defining how the iSCSI/iSER (iSCSI 
Extensions for RDMA) can be made to operate over an InfiniBand network 
instead of a previously defined protocol called SRP (SCSI RDMA Protocol). 
The SRP protocol lacked Storage Management and Discovery processes. 
Therefore, great interest in iSCSI/iSER (iSER for short) has developed 
since it has the management and discovery structure defined, built, or 
being built for iSCSI. This will also consolidate the efforts of both 
Ethernet and InfiniBand communities, and reduce the number of Storage 
protocols a user has to learn and maintain.
 
Several things are needed in order for an iSCSI/iSER over an InfiniBand 
(iSER/IB) network to operate.  They are:

1. Being able to exploit the fact that all the IB Initiator and Target 
nodes can also have an IP Transport (IP-over-IB a.k.a. IPoIB) and 
associated IP Addresses for standard iSCSI discovery and for address 
resolution.

2. A way for an iSCSI/iSER on IB Initiator to Login directly to an 
InfiniBand attached iSER storage node (without having to connect and send 
Login PDUs via TCP/IP).

3.  A way for an IB Initiator to choose, if it wishes, directly attached 
IB iSER Storage Nodes in preference to an iSER to iSCSI Gateway, or in 
preference to using TCP over IPoIB (Internet Protocol over InfiniBand) 
connection.

4. A way for iSER to operate on InfiniBand Channel Adapters supporting 
versions prior to 1.2 that does not support what is called a Zero Based 
Tagged Offset (ZBTO) or SendInvSE Messages. (Though the IB specification 
has been updated to support the new RDMA features that were introduced by 
iWARP, there are likely to be a number of systems that are only using IB 
hardware that only meet the pre version 1.2 specifications.  This would 
mean that they would not support the features called ZeroBasedTO (called 
in InfiniBand ZBVA) or SendInvSE Messages.)

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 08 Nov 2004 08:31:07 -0800

John and all,

- Agree with and support the basic notion being advanced here -
   it is ideal to have one iSCSI-based transport for RDMA and non-RDMA
   transports, and one iSCSI/iSER-based transport for iWARP and IB
   storage access.

- Suggest that this work should go into separate draft(s) because it
   appears to describe different modes of operation as I see them (more
   on this below), and allows significantly different behaviors from
   the current mandatory behavior required in the iSER draft. E.g.,
	- Support for non-SendInvSE operation.
	- Support for non-zero-based-TO operation.

   Also separate drafts would allow the current iSER/iWARP draft to
   proceed to its milestone dates as planned.

- Assuming that S-RQ is a high-want for iSER targets, one should
   consider if it's indeed worth designing for the old IB ecosystem.
   If we assume SendInvSE & zb-TO support in IB, the iSER/IB semantics
   would be substantially closer to iSER/iWARP semantics during runtime.

- I am not an IB expert, but it seems to me that the draft is
   outlining multiple modes of operation.  Is it fair to say that
   modes b, c and d below could support gateways?
	a) iSCSI/iSER/IB end node talking to iSCSI/iSER/IB end node.
	b) iSCSI-3720 PDUs in IB Send Messages.  This is currently
            proposed only for login phase, but appears to be usable
            even beyond to talk to an iSCSI/iSER gateway.
	c) iSCSI/IB end node talking to iSCSI/TCP end node.  As I
            understand this, this should be iSCSI-3720 over IPoIB with
            no changes to peer iSCSI layers.
	d) iSCSI/iSER/IB end node talking to iSCSI/iSER/iWARP end node.

- Should iSER/RDMA capabilities be reported in discovery?  Should
   IB-support?  How about the iSCSI MIB?

- Finally, the draft makes an interesting observation which was brought
   up earlier as well - devising an integrated approach for iSER/SCTP
   and iSER/IB.  This notion should be considered further.  It might
   also decouple iSER from the current indirect MPA dependency.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

John Hufferd wrote:

> complete in time to avoid the IETF draft black-out period.  However, 
> there was a document that was created that explained what was being 
> discussed and why, and the WEB link to that document (on Julian's Web 
> site) is included here:
> 
> I encourage you to take a look at this document.  It should also be 
> noted that because of this work dealing with InfiniBand it was also 
> possible to define how an SCTP implementation could also be created that 
> could be conformant to some new proposed wordage for the iSER Draft (to 
> be released some time after the Black-Out period).
> 
> anything other than to start the discussions, I expect there will be a 
> lot of additional work to clarify issues, and craft the most approprate 
> *Introduction*
> instead of a previously defined protocol called SRP (SCSI RDMA 
> Protocol). The SRP protocol lacked Storage Management and Discovery 
> processes.  Therefore, great interest in iSCSI/iSER (iSER for short) has 
> developed since it has the management and discovery structure defined, 
> built, or being built for iSCSI. This will also consolidate the efforts 
> of both Ethernet and InfiniBand communities, and reduce the number of 
> Storage protocols a user has to learn and maintain.
>  
> InfiniBand attached iSER storage node (without having to connect and 
> send Login PDUs via TCP/IP).
> 
> has been updated to support the new RDMA features that were introduced 
> by iWARP, there are likely to be a number of systems that are only using 
> IB hardware that only meet the pre version 1.2 specifications.  This 
> would mean that they would not support the features called ZeroBasedTO 
> (called in InfiniBand ZBVA) or SendInvSE Messages.)
> 
> 
> .
> .
> John L. Hufferd
> ------------------------------------------------------------------------
>

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 12 Nov 2004 15:30:17 -0600

Eliminating the TCP dependencies in iSER would be a good idea.

Once a connection between two RDMA Endpoints is established,
iSER can be defined as a series of RDMA Messages.  As such
they are dependent only on RDMAP, and not on the underlying
LLP. Adapting to other IP based transports would be easy,
such as SCTP, and should be supported.  It also enables
non-IP transports, which is something that I do not believe
anyone would object to, especially if a clarifying draft
establishes how to do iSER over generic RDMA services
without having to specify the mapping of generic RDMA
services to any specific non-IP transport.

The only missing service required is support of an initial
message before endpoints are selected (the equivalent of the
MPA Request and Reply messages and their private data).

The key requirement is specifying that the private data
contain enough information to enable RDMA endpoints to
be selected.  After that remaining iSCSI negotiations
can be completed using Send messages.

Private data exchange prior to pairing RDMA endpoints
is already supported for both MPA/TCP and SCTP.  The
essential similarity is already documented in the RDDP
applicability draft.

If a "non-streaming mode" method of establishing an
iSCSI connection is defined that is compatible with
the existing private data exchange followed by Send
messages for the remaining negotiations it will enable
other IP transports, such as SCTP, and do so in a manner
that is fully compatible with existing RDMA APIs such
as DAPL and IT-API.

It would also make it easy to port "non-TCP-dependent"
iSER to non-IP transports, should anyone wish to do
that outside the scope of the IETF.

Eliminating iSERs dependency on a single RDMA transport
is something that should be done anyway.  The fact that
doing so would be useful for InfiniBand ports is certainly
not a reason to avoid doing something that should have
been done in the first place.