Subject: Re: iSCSI: SCSI Iniitator port: a simple approach, acceptable?

From: "Jim Hafner" <[email-masked]>
Date: Thu, 11 Oct 2001 14:33:52 -0700

Santosh,

I agree we haven't really solved the problem exactly (about what has to be
configured per HBA, though I hinted at an implementation of portal group
tag assignment that is based on network stuff, that might make it easier.

It's your last comment that I was expecting to get solicited.  But I'll ask
why is it so important to have several sessions in parallel?
If I can have multiple connections per session, doesn't that suffice for
multiplexing?  If the target says only one connection per session, then
isn't that a hint that it has limited network resources and it would be bad
form to build too many sessions just to bypass this?

I'm not strongly advocating this, playing devil's advocate.

Jim Hafner

Santosh Rao <[email-masked]>@cup.hp.com on 10/11/2001 02:10:56 pm

Sent by:  [email-masked]

To:   Jim Hafner/Almaden/IBM@IBMUS
cc:   [email-masked]
Subject:  Re: iSCSI: SCSI Iniitator port: a simple approach, acceptable?

Jim,

I am not in favor of the below proposal and would rather see the ISID
model, for the reason you mention under "Cons".

Up until now, the concept of "initiator portal groups" has not held any
significant importance in the protocol, unlike target portal groups
which is formal information exchanged b/n initiator and target, and
which governs the session end points.

How is the initiator portal group now going to be communicated to the
iscsi HBAs ? In essence, we've just moved the piece of information that
needs to be co-ordinated across HBAs, from being an ISID to being a
portal group tag.

For single connection sessions, it is essential that several sessions be
allowed in parallel b/n portal groups.

Regards,
Santosh

Jim Hafner wrote:
>
> Folks,
>
> I'm curious to know from this group whether the following is an
acceptable
> alternative to the complex issues we're battling with the definition and
> behavior of SCSI Initiator Port (viz a viz, ISID, key=value,etc.).
>
> Make the SCSI Initiator Port = initiator portal group, named by iSCSI
Name
> + portal group tag (which would be presented during login).
>
> Pros:
> a) the model is completely symmetric on the initiator and the target side
> b) namespace is managed by any mechanism to identify the portal group
tag.
> (For example, the tag could be the position in the SCSI port/module load
> sequence (/dev/scsi[x]) OR one could derive the portal group tag from
> either a MAC or from ipaddress of one of the interfaces in the portal
> group).
> c) the drivers present exactly one channel (SCSI initiator port) for each
> initiator portal group to the upper layers in the OS (how that is
> named/identified to the upper layers may be independent of how it is
named
> on the wire, i.e., independent of the portal group tag).
> d) completely static/stable configuration on each boot (though it may
> change, as it can today, between boots if components fail or are changed
in
> any way).
>
> Cons:
> a)  we *prohibit more than one session* between an initiator portal group
> and a target portal group (to avoid parallel nexus).  This has the side
> consequence of making it harder to be a bad net-citizen by opening too
many
> connections through the same set of network interfaces (you open what you
> want within a session only).
>
> I've always avoided this approach as I expected the "cons" would not be
> acceptable, but I've never actually asked this of the group.
>
> So, what's the 'acceptability' of this approach?
>
> Jim Hafner
>
> Jim Hafner/Almaden/IBM@[email-masked] on 10/11/2001 08:58:41 am
>
> Sent by:  [email-masked]
>
> To:   [email-masked], [email-masked]
> cc:
> Subject:  Re: iSCSI: ISID progress
>
> David,
>
> More comments in line
>
> Jim Hafner
>
> [email-masked] on 10/10/2001 03:48:55 pm
>
> To:   Jim Hafner/Almaden/IBM@IBMUS, [email-masked]
> cc:
> Subject:  iSCSI: ISID progress
>
> This message is shorter than my last one, so that's
> at least a superficial indication of progress ;-).
>
> Jim and Julian had four comments/objections to the
> notion of using a text key to indicate iSCSI Initiator
> Port Name.  Summarizing and responding:
>
> Jim (a): Optional use of port name is ok as far as SAM-2 is
>      concerned, but is odd.
>
> Indeed it is odd, but given the choice between an odd
> mapping of SAM-2 to iSCSI and allowing odd behavior of
> iSCSI implementations, I'll take the former.
> <JLH>
> I'm not sure what "odd behavior of iSCSI implementations" you're
referring
> to here.
> </JLH>
>
> Jim (b): Would require at least as much coordination above
>      the session level of iSCSI as ISIDs.
>
> That would be incorrect.  128 bits is sufficient to eliminate
> coordination.  The reference for this is an expired Internet-
> Draft, draft-leach-uuids-guids-01.txt, that can still be found
> on the web at:
>
> http://casbah.org/cbRFC/misc/draft-leach-uuids-guids-01.txt
> http://globecom.net/ietf/draft/draft-leach-uuids-guids-01.html
>
> I'm not seriously proposing that port name generation be done
> in this fashion, but rather providing a widely used counter-
> example to Jim's statement.  Note that a network interface
> MAC is likely to be available to many iSCSI implementations.
> <JLH>
> I glanced through that draft and certainly don't think it is the right
> approach to this problem.  However, if you/we believe that by simply
making
> the equivalent of the ISID field or portname text value big enough, we
can
> find a solution to the "coordination problem", why can't we just
*require*
> that sessions have a SCSI port name (extension to iSCSI Name) that is
long
> enough to solve the weak or no coordination problem, instead of making
this
> "TBD"?
>
> I've argued that having the SCSI Portname is valuable.  I've argued that
> there wasn't any net value in putting the name in a key because I already
> had the ISID field as part of the "name space for the session endpoint".
> But if the claim is simply the ISID isn't big enough, then I don't have a
> problem with effectively making them bigger (either in the header or in a
> key value).   I've even suggested one way to do that by embedding the
> initiator portal group tag in the value.  But another way is to embed the
> OUI of the HBA (except for the cases where there isn't an HBA) or embed
one
> of the MACs of one of the nics (except for the cases where there isn't a
> nic, e.g., dialup), etc.  But we'd have to solve all the possible cases
> (all of which the NDT had to deal with for iSCSI Names in the first
place).
>
> So, are we at the point where (for this alternative proposal):
> (a) we just need a bigger (than 2byte ISID) field for the SCSI portname
> extension
> (b) we just need an algorithm that lets each component that wants to
> generate such a name can do so without collision concerns?
> </JLH>
>
> Jim (c): How to describe model when the text key is optional?
>      "Is it that all initiator session endpoints that don't
>       provide this text key have *implicit* unique names and
>       only when the text key is presented does the name get
>       explicit (and then possibly not be unique)? In that case,
>       the key would have to be supplied in login next to the
>       InitiatorName.
>
> Yes and yes when it's used, in that order.  Jim's comment (a)
> about this being odd applies.
>
> Julian: ... and have as much chances to blow a session as ISID
>
> That would also be incorrect.  As previously stated, ISID conflict
> is fatal to one of the sessions involved (one cannot change the
> ISID and continue), and can occur in any system that opens parallel
> sessions.  This text key conflict need not be fatal (one can
> change the text key and continue negotiation) and can only occur
> in systems that want to use the new port-spanning persistent
> reservation functionality, as other systems won't use the text
> key.  Also see the draft noted in response to Jim (b) above;
> Julian's "have as much chance" language is incorrect.
> <JLH>
> I think this is somewhat misleading about the cost ("fatal").  If we
could
> find a reasonable model for "reject: ISID in use" (rather than "blow away
> the old session"), then the "fatal to one of the sessions" is a nit.
This
> would happen on the first exchange of a connection so starting over isn't
> any big deal (we may not even have to require connection drop, just
restart
> of the initial message with new ISID).  The same would happen with the
key
> as that would have to be somewhere in the pre-full feature phase (but in
> this case it could happen anywhere in that phase).
> </JLH>
>
> As previously noted, I can also deal with requiring conservative
> reuse of ISIDs as a means to address this situation.
>
> Thanks,
> --David
>
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> [email-masked]       Mobile: +1 (978) 394-7754
> ---------------------------------------------------

--
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: Santosh Rao <[email-masked]>
Date: Thu, 11 Oct 2001 14:54:17 -0700

Jim,

I suspect you already know the answers to your questions ;) We've been
through these questions already in the past. 

isci needs to continue to support operation of wedge drivers. There will
exist both initiator and target implementations that do only single
connection sessions. Such implementations can attempt to establish
sessions to each TgtAddr advertised by the target, and several of these
TgtAddr may belong to the same target portal group.

Also, if a target supports session spanning across all of its controller
ports, it may well export a single target portal group. Similarly, an
initiator with no specific constraints (such as different vendor type
implementations) has no reason to define more than 1 initiator portal
group and may well be oblivious to the very concept of an initiator
portal group. In such cases, the initiator will create multiple sessions
from a single initiator portal group to multiple TgtAddr's within a
single Target Portal Group. 

I believe it was the ISID based initiator port model that prevented the
"parallel nexus" from occurring in the above scenario.

- Santosh

---

From: Santosh Rao <[email-masked]>
Date: Fri, 12 Oct 2001 16:09:49 -0700

Somesh,

I don't understand your concern voiced below. 

A session spanning multiple adapters requires the host resident session
manager to co-ordinate assignment of initiator task tag, cmdsn,
exp_cmdsn & max_cmdsn, [over and beyond the usual initiator name &
ISID/TSID].

The initiator task tag & cmdsn can be passed down from the session
manager to the HBA in a mailbox command, as a part of issuing a new
iscsi command, and the (exp_cmdsn, max_cmdsn) values can be updated to
the host resident session manager by the HBA as a part of conveying a
mailbox command completion. There are no extra host interrupts in normal
I/O paths due to the presence of a host resident session manager.

If you're referring to locking overheads, then, these can be optimized
out.

I don't see why a host resident session manager affects performance of a
multi-connection session ? 

Regards,
Santosh

Somesh Gupta wrote:
> 
> Jim,
> 
> Multiple connections, especially when spread across multiple
> adapters will give performance that will be much less than
> that of multiple sessions in parallel (of course when there
> are multiple connections/session over multiple adapters - the
> adapters have to be able to take the session id from the
> host software - I obviously don't get the point of the debate).
> 
> There is no way a common synchronization point for
> command sequencing and flow control at the host level will not
> cause sub-standard performance. I have argued this point
> many times before, only to have it be brushed aside. Of course,
> only time will tell.
> 
> Somesh
>

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 12 Oct 2001 15:31:25 -0700

Jim,

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Jim Hafner
> Sent: Thursday, October 11, 2001 2:34 PM
> To: Santosh Rao; [email-masked]

---

From: Santosh Rao <[email-masked]>
Date: Fri, 12 Oct 2001 17:19:59 -0700

Somesh Gupta wrote:

>   You sort of answered the question yourself. As a cache line (containing
>   common numbers) ping-pongs
>   between processors that handle interrupts for different adapters, its
>   adds significant overhead. A 2GHz processor in a CC-Numa box sharing
>   cache lines between multiple processors?

I think the above can be optimized out by some proper design. I don't
think locking or cache thrashing are significant performance inhibitors
that cannot be worked around in this case. 

> 
>   This does not include the cost/complexity of managing multiple queues,
>   (and additional cache lines to share)

What multiple queues ? (I don't see any major issues here either.)

>   and the fact that the "rare" retransmit has the potential to slow
>   everything down to the speed of the slowest connection.

But, this is not a drawback of multi-connection session. It is a
drawback due to the fact that the iscsi protocol chose to impose strict
command ordering across the session [,which is not really required. But,
I won't start that thread again !].

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 12 Oct 2001 16:53:43 -0700

> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, October 12, 2001 4:10 PM
> To: [email-masked]
> Cc: [email-masked]

  We will see. I don't think this debate is going to be solved
  by discussion. People will even obtain different results from
  experiments.

---

From: "Somesh Gupta" <[email-masked]>
Date: Fri, 12 Oct 2001 17:18:08 -0700

Santosh,

As I said, this is an endless debate. I don't think
everything can be optimized out of the code path.
I apologise in advance for not responding any more
to this thread.

Somesh

> Sent: Friday, October 12, 2001 5:20 PM