Subject: iSCSI: ISID issue summary

From: [email-masked]
Date: Tue, 9 Oct 2001 21:36:42 -0400

For those who haven't been following the long intricate
email discussion between Jim Hafner and myself, let me
try to explain the issue.

SCSI architecture (SAM2) is based on an intuition that SCSI
ports are fixed (e.g., physical) things that have names.  So
far, the existence of the names has been an intellectual
curiosity as they haven't mattered to any visible SCSI
functionality.  Single port Initiators predominate in 
both Parallel SCSI and Fibre Channel (e.g., a dual ported
Fibre Channel HBA is usually treated as two SCSI Initiators,
not a single dual-ported one).

Jim tells us that T10 is about to define persistent reserve
functionality that depends on the identity of the SCSI Initiator
Port - in particular that under some circumstances, persistent
reservations will be associated with the Initiator Port independent
of the Target Port.  Persistent reservations are currently bound
to the Initiator Port and Target Port (as well as the LU they
affect).  For the rest of this message, I'm going to assume
that we want to support this functionality.  Assuming this and
that I got the description in this paragraph right ...

 ... I need to take a slight detour into pragmatics.  Those
who configure storage networks rapidly discover the value
of consistency and matching configurations.  Multi-path I/O
configurations tend to want to see access to the same set of LUs
consistently across a set of interfaces (i.e., Ports).  Between
suitable configuration and the aggressive setup of connections
to all possible targets, all the required connections come into
existence as part of the boot process.  Applying
this experience to the new persistent reservation functionality,
one would expect persistent reservations that are bound only to
an Initiator Port to be independent of the Target Port, in that
the reservation should span connections to all of the relevant
Target Ports and those connections should come into existence
as part of the boot process.

Turning to iSCSI, the situation gets complicated by the
interaction of two factors:
- Parallel sessions are permitted down to the interface
	level (e.g., more than one iSCSI session may exist
	that uses IP addresses A and B to connect iSCSI Initiator
	I to iSCSI Target T).
- SCSI disallows parallel nexii (i.e., more than one session
	between the same two SCSI Ports).
If SCSI Ports are mapped to hardware entities such as network
interfaces in iSCSI, the opportunity for parallel sessions
between the same pair of network interfaces vanishes.  In order
to avoid that outcome, iSCSI Initiator Ports have to be
mapped to software entities. To date, the ISID has been
used to identify those software entities, and the only rule
on their allocation is:

     ISID RULE: Between a given iSCSI Initiator and iSCSI Target Portal 
     Group (SCSI target port), then can be only one session with a given 
     ISID (identifying a SCSI initiator port).  See 3.12.8. 

In order to get the expected outcome of the new functionality
being defined by T10, the same ISID has to be used to establish
all the connections that the persistent reservation is supposed
to span.  Unfortunately, that's above and beyond what iSCSI
requires - the relevant text about doing this in -08 is:

        The "ISID rule" does not preclude 
        the use of the same ISID from the same iSCSI Initiator with
        different Target Portal Groups on the same iSCSI target or
        on other iSCSI targets 

"does not preclude" is rather weak language for something that's
essential to making a piece of SCSI functionality work.  If an
iSCSI implementation uses a different ISID for every session
(which is also not precluded by the current text), then persistent
reservations will never span Targets or Target Ports for that
implementation despite T10's best efforts and intentions.  IMHO,
allowing that outcome is *wrong* (and this is the source of the
difference of opinion between Jim and myself).

The correction to this involves what Jim has called "conservative
reuse" of ISIDs.  This is something like for each ISID that an
implementation uses, a connection with that ISID is made to
every possible Target Portal Group of every possible Target.  I
suspect a longer explanation than this will be needed, including
a discussion of the desirability of avoiding a situation in which
the same ISID is used by two iSCSI implementations that are part
of the same Initiator and set up sessions concurrently.

IMHO, if we want to support persistent reservations at this stage
that span target ports, we need to replace the "does not preclude"
language with a REQUIREMENT for conservative reuse to avoid a
situation in which SCSI functionality that works consistently with
other transports works inconsistently with iSCSI (i.e., doesn't
work with iSCSI implementations that don't do conservative reuse).

Stepping back from the assumption that support for port-spanning
persistent reservations is required, I observe that the conservative
reuse rule impacts all implementations, even those that will never
use such persistent reservations because ISID is a mandatory part
of the iSCSI protocol.  If a text key were used instead, only those
Initiators that want to support this functionality on which T10
is "crossing a few t's and dotting a few i's" are affected (the
text key is optional).  The text key would be subject to a
conservative reuse rule similar to the one that would be needed
for ISIDs, and once T10 finishes the i's and t's, we can precisely
reference the SCSI features (persistent reservation expansion)
that require use of this text key.

In addition, text keys have much better alternatives for dealing
with conflicts than ISIDs - if a text key conflicts, it is possible
to continue  the negotiation with a different text key, whereas
ISID conflict is fatal to one of the two sessions involved.
As Jim has previously observed, changing the text key (e.g.,
generate a large random number in response to a conflict),
results in a situation that can be dealt with by software that
uses persistent reservations, although this departure from
conservative reuse should only occur as a consequence of some
sort of configuration mistake or bug.  At the tail end of this
reasoning chain is the observation that use of this text key
leaves the ISID without value/purpose, and hence would call
for its removal (or perhaps designation as a reserved field).

Putting on my WG chair hat for a moment, I can accept either
alternative outlined above:
- Add conservative reuse to the ISID rule.
- Use a text key for iSCSI port identification.
But I'm not comfortable with the current situation in which iSCSI
implementations are permitted to break T10-defined SCSI features
that will work as expected in other SCSI transports.

I hope this now makes sense to more people than just Jim and I,
and I apologize for the length of this message, as this is a
somewhat subtle issue.

Comments?
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 10 Oct 2001 17:39:14 +0300

... and have as much chances to blow a session as ISID.

Julo

Jim Hafner/Almaden/IBM@IBMUS
Sent by: [email-masked]
10-10-01 16:27
Please respond to Jim Hafner

        To:     [email-masked], [email-masked]
        cc: 
        Subject:        Re: iSCSI: ISID issue summary

David,

An excellent summary of the issues.

A couple of things I want to point out about the "text key" proposal, 
however.
a) The proposal sounds to me like iSCSI will have "optional SCSI Port 
names". Only when the key is used is the SCSI port named and when the key 
is not used, there is no such name. That's probably an acceptable 
situation as far as SAM-2 is concerned, but it certainly is odd.
b) The proposal would require at least as much coordination above the 
"session manager" level as managing ISIDs, though admittedly, it need not 
be implemented in all cases.
c) I'm concerned how to describe the model when names are optional, so as 
to avoid the parallel nexus problem. Is it that all initiator session 
endpoints that don't provide this text key have *implicit* unique names 
and only when the text key is presented does the name get explicit (and 
then possibly not be unique)? In that case, the key would have to be 
supplied in login next to the InitiatorName.

Jim Hafner

Sent by: [email-masked] 
To: [email-masked]
cc: 
Subject: iSCSI: ISID issue summary

Comments?
--David

---

From: "Jim Hafner" <[email-masked]>
Date: Wed, 10 Oct 2001 07:27:47 -0700

David,

A couple of things I want to point out about the "text key" proposal,
however.
a) The proposal sounds to me like iSCSI will have "optional SCSI Port
names".  Only when the key is used is the SCSI port named and when the key
is not used, there is no such name.  That's probably an acceptable
situation as far as SAM-2 is concerned, but it certainly is odd.
b) The proposal would require at least as much coordination above the
"session manager" level as managing ISIDs, though admittedly, it need not
be implemented in all cases.
c) I'm concerned how to describe the model when names are optional, so as
to avoid the parallel nexus problem.  Is it that all initiator session
endpoints that don't provide this text key have *implicit* unique names and
only when the text key is presented does the name get explicit (and then
possibly not be unique)? In that case, the key would have to be supplied in
login next to the InitiatorName.

Jim Hafner

[email-masked]@ece.cmu.edu on 10/09/2001 06:36:42 PM

To:    [email-masked]
cc:

Comments?
--David

---

From: "Elliott, Robert" <[email-masked]>
Date: Wed, 10 Oct 2001 19:19:22 -0500

David Black wrote:

> functionality.

Not quite true.  The Fibre Channel port's Worldwide_Name -
what SAM-2 now calls the initiator port name - is used 
today for persistent reservations.  SPC-2 has this wording:

  For those SCSI protocols for which the initiator port's 
  world wide identification is available to the device 
  server the initiator port's world wide identification 
  shall be used to determine if the initiator identifier 
  has changed.

FCP-2 indicates that the Worldwide_Name of the Fibre Channel 
port serves as the "initiator port's world wide 
identification."  (that phrase will change to "initiator 
port name" as Jim's (accepted) T10 name proposal is 
digested.)

>                Single port Initiators predominate in 

According to the SAM-2 multiport model, you can upgrade that
"usually" to "always."  The key to the multiport model is:

	Initiator = initator port
	Target = target port

In FCP-2, initiator ports equal Fibre Channel ports.  There's 
no way to treat two physical ports as the same initiator
without violating the standards.

The multiport model defines "initiator device" as the object
that contains multiple initiator ports.  The initiator device 
has neither an address nor a name in any existing protocol.
There are no commands that rely on it.  Jim wanted to start
using that concept for iSCSI access controls.

> affect).  For the rest of this message, I'm going to assume

Again, the reason for this is the multiport model that says
initiator = initiator port and target = target port.

> as part of the boot process.

That's the goal of one of the T10 proposals.  If initiators
are identified only by their port addresses, it's not safe.  
The two target ports could be on different fabrics, where
initiators are different but happen to have the same
addresses.  On a fabric where the initiator port has a name
that is known to be worldwide unique, this becomes a
non-issue if the initiator port name is used instead of the
initiator port address.

>   ISID RULE: Between a given iSCSI Initiator and iSCSI 
>   Target Portal Group (SCSI target port), then can be only
>   one session with a given ISID (identifying a SCSI 
>   initiator port).  See 3.12.8. 

There's a bit more than that.  The "iSCSI Name" has to
be included with the ISID to identify the initiator port.
Jim wanted the iSCSI name to be the "initiator device name."

> work with iSCSI implementations that don't do conservative reuse).

As Nick noted, the problem is where do you store these ISIDs?
For that matter, where do you store the iSCSI name?  With
Fibre Channel, the Worldwide_name is in a ROM associated with
the port.  With iSCSI you have no guarantee of hardware.  You
can usually find an Ethernet MAC address and could base a
name off of that.  However, there's no guarantees.

For the SCSI RDMA Protocol for InfiniBand, we chose:
  initiator port name = 64-bit worldwide identifier from
                        the InfiniBand channel adapter plus
                        64 extra bits
All software using the same InfiniBand channel adapter have to
coordinate usage of the extra bits, but single-OS single-driver
systems can just fill them with zeros.

> of the iSCSI protocol.

Persistent reservations, access controls, extended copy,
third-party XOR commands, and the supporting alias commands
are all potential users of port names today.  Protocol bridges
and management tools may also benefit from using names 
rather than addresses.

>                       If a text key were used instead, only those

This just seems to make the names more vague and volatile.

Are the iSCSI name and ISID used as part of authentication?
How does a device driver prove it has the right to use
a certain iSCSI name/ISID?  How does it prevent some other
software from using its own iSCSI name/ISID?

> and I apologize for the length of this message, as this is a

If we didn't have ISIDs we'd still have the same debate about
managing the iSCSI names.  Two software implementations
on the same machine could choose the same iSCSI name and
conflict - the same problems result.

Jim's partitioning lets the OS pick an iSCSI name (unique 
from any other OS on the fabric, with any luck) and requires 
the OS coordinate ISIDs for any iSCSI drivers sharing
that iSCSI name.  A dedicated iSCSI HBA could also hand
out ISIDs for drivers using it.  This should limit the 
scope of conflicts to one system rather than the whole 
fabric.  It'd be helpful to have standards for the OS 
or HBAs to solve this, but that seems outside the scope 
of the iSCSI protocol spec.

> --David
---
Rob Elliott, Compaq Server Storage
[email-masked]

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 21 Oct 2001 01:21:18 +0300

There is going to be a "conservative reuse" rule. Stay tuned.  Julo

                    "Rod Harrison"                                                          
                    <rod.harrison@wind       To:     Julian Satran/Haifa/IBM@IBMIL          
                    river.com>               cc:                                            
                                             Subject:     RE: iSCSI: ISID issue summary     
                    15-10-01 13:10                                                          
                    Please respond to                                                       
                    "Rod Harrison"                                                          

Julian,

           I don't understand what you are considering "excessive and
restrictive."

           I'm requesting that we don't do something that would prevent
designing an iSCSI HBA to work with existing SCSI HBA drivers, for
example requiring that the host side driver be involved in every
session set-up.

           I have no issue with the host being required to provide some
coordinating entity if multiple sessions are being spanned over
multiple iSCSI HBAs, either from the same or different vendors. I
would just like us not to place this restriction on simpler
deployments like one iSCSI HBA, or more than one with no session
spanning. In the latter cases I believe David's ISID range idea would
be sufficient.

           Does this explain my position better, or is there something you
still
object to?

           - Rod

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Sunday, October 14, 2001 9:21 PM
To: Rod Harrison
Subject: RE: iSCSI: ISID issue summary

I think that this would be both excessive and unwarranted. And there
is no
reason to associate it with the wire protocol.
And I don't think that David Black had this type of excessive
restriction
in mind in the "conservative reuse" rule.

Julo

                    "Rod Harrison"
                    <rod.harrison@wind       To:
<[email-masked]>,
                    river.com>
<[email-masked]>, <[email-masked]>
                    Sent by:                 cc:
                    [email-masked].       Subject:     RE: iSCSI:
ISID issue summary
                    edu

                    13-10-01 13:25
                    Please respond to
                    "Rod Harrison"

           I am in favour of the 'statically configure the ISIDs along
with
the
iSCSI name and write the "conservative reuse" rule into the iSCSI
protocol specification' approach.

           One thing I would like to see happen here is to allow iSCSI
HBAs
to
be used with existing SCSI drivers. Many vendors have well tested
drivers in the field, and it is currently a relatively easy job to
make an iSCSI HBA look like an existing SCSI HBA. Doing so is a time
to market win and should also slightly ease customers new technology
fears. There is a clear advantage to being able to tell customers that
they can fit an iSCSI HBA into systems alongside their existing SCSI
HBAs and have them "just work" after some initial configuration.
Setting ISID ranges can easily be done as part of this initial
configuration, as can setting the iSCSI name, IP address, DHCP use, or
whatever else an iSCSI HBA needs to know.

           I'm not against having to have some coordinating entity on
the
host
as long as that is optional, and preferably only needed for more
complex installations.

           - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
[email-masked]
Sent: Saturday, October 13, 2001 2:49 AM
To: [email-masked]; [email-masked]
Subject: RE: iSCSI: ISID issue summary

Rob,

> For that matter, where do you store the iSCSI name? [..snip..]
> With iSCSI you have no guarantee of hardware.

Software-based implementations will have access to a file or
registry or something like that.  Boot involves BIOS nonsense
and pain of varying forms and has access to BIOS parameter
store (may be battery-backed RAM rather than flash) if needed.
Booting over a software implementation can be very difficult
- in essence, the iSCSI software gets added to the main system
BIOS (not something to be done lightly, especially as there
are existing protocols that can retrieve a boot image over IP).

> systems can just fill them with zeros.

But SRP only needs to coordinate within "the same InfiniBand channel
adapter".  iSCSI currently requires the ISID to be coordinated across
adapters/network interfaces.  That's an important difference.

> and management tools may also benefit from using names

I'll take that as a strong hint from the T10 direction that this
set of issues cannot be deferred (matching Jim's comments).

> software from using its own iSCSI name/ISID?

Only the iSCSI name is used, and iSCSI inband authentication
(which requires state on the target) is used to deal with both
of the latter questions.

> conflict - the same problems result.

Actually, the iSCSI name has to be statically configured -
it's somewhat akin to an IP address in that using the same
one twice on two different independent systems will have
unintended (and potentially unpleasant) results.  Picking
the name dynamically is not anticipated because that name
is expected to show up in the configuration of LUN masking/
mapping access controls on the Targets.

> out ISIDs for drivers using it.

With the exception of having "the OS pick an iSCSI name",
that's correct.  The problem is one system with two HBAs
that don't know about each other.  Also, unlike InfiniBand,
one can expect an iSCSI HBA to have a dedicated driver.

> This should limit the

A solution that works here is to statically configure the
ISIDs along with the iSCSI name and write the "conservative
reuse" rule into the iSCSI protocol specification.

Thanks,
--David

> -----Original Message-----
> From: Elliott, Robert [mailto:[email-masked]]
> Sent: Wednesday, October 10, 2001 8:19 PM
> To: [email-masked]
> Subject: RE: iSCSI: ISID issue summary
>
>
> David Black wrote:
>