Subject: RE: [Ips] [iSCSI] iSER on InfiniBand

From: "Yaron Haviv" <[email-masked]>
Date: Sat, 6 Nov 2004 02:08:33 +0200

John and Folks,

The proposal seems rather complete and straight forward 

I believe running iSER over InfiniBand in addition to iWarp serves the
iSCSI cause 

It allows iSCSI to span from the lowest end with pure software
implementations to the highest end with 30Gb/s InfiniBand hardware 

Managing all of it with ubiquitous IP semantics, and with common
iSCSI/iSNS/SLP infrastructure 

The InfiniBand community can also accelerate the iSER creation and
adoption 

By pooling in more RDMA developers, and leveraging on existing HW and SW

It looks like the required changes in the current iSER spec are only few
and are around the negotiation of Send/SendInvSE and ZBTO 

And specifying that a Login can travel over the IB Reliable connection
rather than over TCP

The Transport preference looks like something that should also be
defined some where 

I believe its not covered for Ethernet as well today, I haven't seen a
definition of when iSCSI decides on using the RDMAExtensions option or
not (probably an implementation specific issue), and what happens when
we have both a NIC and an RNIC, do we decide base on Subnet Masks, ..

So it can either be part of iSER or specified somewhere else 

If people don't want to change much in the iSER Draft, Maybe it's
possible to add just the capability negotiation piece (SendInv, ZBTO),
and general comments on iSER usability over InfiniBand, and put all the
rest in an external draft that can be discussed separately 

I can think of some use for non ZBTO (VA Based) for iWarp applications
as well

All in all it's a good draft and good idea to add iSER binding to
InfiniBand 

Yaron

Voltaire, CTO

www.voltaire.com <http://www.voltaire.com/> 

________________________________

From: [email-masked] [mailto:[email-masked]] On Behalf Of
John Hufferd
Sent: Thursday, November 04, 2004 9:24 PM
To: [email-masked]
Subject: [Ips] [iSCSI] iSER on InfiniBand

Folks, 
I am aware of a lot of discussions that have been going on recently
regarding iSCSI/iSER operating on InfiniBand Networks.  There has been
some authors that have been working to create some proposed wordage for
the iSER document that would permit support of iSER by an InfiniBand
Network.  Unfortunately, the work to propose some new wordage did not
complete in time to avoid the IETF draft black-out period.  However,
there was a document that was created that explained what was being
discussed and why, and the WEB link to that document (on Julian's Web
site) is included here: 

http://www.haifa.il.ibm.com/satran/ips/iSER-in-an-IB-network-V9.pdf 

I encourage you to take a look at this document.  It should also be
noted that because of this work dealing with InfiniBand it was also
possible to define how an SCTP implementation could also be created that
could be conformant to some new proposed wordage for the iSER Draft (to
be released some time after the Black-Out period). 

Below is the introduction to the paper that was used as a base for some
new proposed wordage.  (Please understand, this is not meant to do
anything other than to start the discussions, I expect there will be a
lot of additional work to clarify issues, and craft the most approprate
wordage for the iSER document.) 

Introduction

There has been great interest in defining how the iSCSI/iSER (iSCSI
Extensions for RDMA) can be made to operate over an InfiniBand network
instead of a previously defined protocol called SRP (SCSI RDMA
Protocol). The SRP protocol lacked Storage Management and Discovery
processes.  Therefore, great interest in iSCSI/iSER (iSER for short) has
developed since it has the management and discovery structure defined,
built, or being built for iSCSI. This will also consolidate the efforts
of both Ethernet and InfiniBand communities, and reduce the number of
Storage protocols a user has to learn and maintain. 
  
Several things are needed in order for an iSCSI/iSER over an InfiniBand
(iSER/IB) network to operate.  They are: 

1. Being able to exploit the fact that all the IB Initiator and Target
nodes can also have an IP Transport (IP-over-IB a.k.a. IPoIB) and
associated IP Addresses for standard iSCSI discovery and for address
resolution. 

2. A way for an iSCSI/iSER on IB Initiator to Login directly to an
InfiniBand attached iSER storage node (without having to connect and
send Login PDUs via TCP/IP). 

3.  A way for an IB Initiator to choose, if it wishes, directly attached
IB iSER Storage Nodes in preference to an iSER to iSCSI Gateway, or in
preference to using TCP over IPoIB (Internet Protocol over InfiniBand)
connection. 

4. A way for iSER to operate on InfiniBand Channel Adapters supporting
versions prior to 1.2 that does not support what is called a Zero Based
Tagged Offset (ZBTO) or SendInvSE Messages. (Though the IB specification
has been updated to support the new RDMA features that were introduced
by iWARP, there are likely to be a number of systems that are only using
IB hardware that only meet the pre version 1.2 specifications.  This
would mean that they would not support the features called ZeroBasedTO
(called in InfiniBand ZBVA) or SendInvSE Messages.) 

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Dror Goldenberg <[email-masked]>
Date: Tue, 9 Nov 2004 20:17:18 +0200

> -----Original Message-----
> From: Mallikarjun C. [mailto:[email-masked]] 
> Sent: Monday, November 08, 2004 6:31 PM
> To: [email-masked]
> Subject: Re: [Ips] [iSCSI] iSER on InfiniBand
> 
 
<snip>

> - Suggest that this work should go into separate draft(s) because it
>    appears to describe different modes of operation as I see 
> them (more
>    on this below), and allows significantly different behaviors from
>    the current mandatory behavior required in the iSER draft. E.g.,
> 	- Support for non-SendInvSE operation.
> 	- Support for non-zero-based-TO operation.
> 
>    Also separate drafts would allow the current iSER/iWARP draft to
>    proceed to its milestone dates as planned.
> 
> - Assuming that S-RQ is a high-want for iSER targets, one should
>    consider if it's indeed worth designing for the old IB ecosystem.
>    If we assume SendInvSE & zb-TO support in IB, the iSER/IB semantics
>    would be substantially closer to iSER/iWARP semantics 
> during runtime.

Hi Mallikarjun,

The IB Verbs Extensions (IB 1.2) have been defined in groups, where each
group is optional to implement in an IB 1.2 compliant Channel Adapter.
The groups of interest are:
(1) SRQ - Shared Receive Queue
(2) BMM - Base Memory Management - includes Send&Invalidate (and Fast
Register)
(3) ZBVA - Zero Based Regions
An implementation may support any combination of these groups. Therefore
it is very likely that you can find an implementation that supports SRQ, but
does not support ZBVA and Send&Invalidate. Therefore I see a good value in
iSER supporting hybrid modes (especially implementation that have SRQ but
do not support other extensions).

BTW, the reason that the IB Verbs Extensions were defined in optional groups
is because vendors believed that some of the extensions can be implemented
even by the old IB parts through a firmware upgrade or through simple design
changes. Implementation of all the features, usually require a longer
development
cycle, and therefore delays the deployment of products which strongly rely 
on the new features for their wire protocol.

-Dror