Subject: RE: iSCSI: "conservative reuse" requirement

From: [email-masked]
Date: Thu, 3 Jan 2002 11:08:34 -0500

> Thanks for the message.  Let me comment on what I agree with first.
>
> >if "conservative reuse" does not
> > happen (breaking shared persistent reservations), the reason that
> > it does not happen MUST be externally configurable as opposed to
> > being an unchangeable internal characteristic of the implementation.
> 
> Agreed.  If I understand you correctly, you're concerned that NICs
> that are hard-coded not to allow single initiator port abstraction (i.e.
that
> don't do conservative reuse) will break the new shared persistent
reservations.
> I agree that it currently is an issue, I propose that we have a NIC
requirement
> in the implementation notes (like the node name configurability
requirement)
> that says something like - Every iSCSI NIC MUST provide a means of
> enabling a strict "conservative reuse" of ISIDs across different target
portal
> groups.

Good.  Once that's done, what's the value in permitting operating modes
other than "conservative reuse"?  The risk of multiple operating modes is
that there'll be:
- The mode that works, and
- The mode that complies with the standards.
Having only one mode avoids this problem.  Don't laugh too loudly -
versions of this situation are occurring in Fibre Channel switches
today.

> With that said, here's the idea (repeated in several places) 
> that I have a problem with -
> > every possible connection is
> > attempted, and the reasons that not every connection is established
> > are external (and configurable, modulo physical constraints).
> 
> First off, I am not sure what you meant by "connection" here, and
"presented"
> in your previous message.  I took it that you meant "establishing a SCSI
nexus".
> If that's indeed so, the assertion that it's only the external factors
that decide
> nexii is not always true.  Consider the case of an FC initiator 
> port discovering target ports using an FC Name Server, and an iSCSI
initiator
> port discovering target ports using a "SendTargets" command.  In either
case,
> the initiator SCSI port may not really establish a nexus with a discovered
> target SCSI port, unless the application (or
> even the SCSI class driver) really wants to do an I/O (starting with an
open()
> system call in Unix systems).

I strongly disagree with that.  Virtually all commercial OS implementations
of SCSI do aggressive establishments of the nexii as part of the boot
sequence,
as SCSI code tends to expect a "bus walker" to find devices in the boot
sequence, and make sure they work.  Put a trace analyzer to work on a boot
and watch all the TEST UNIT READY and similar commands flow by.  Is anyone
making significant changes to this "bus walker" boot paradigm out there?

There's also a subtle issue to watch out for here in that this delays
error discovery - if the shared persistent reservations are being
used by cluster software, discovering that the alternate path nexus can't
be established when trying to fail over to it is way too late.  For things
like quorum volumes, I'm fairly sure that cluster software checks all the
access paths at initialization time, but can someone familiar with cluster
software comment on whether the alternate access paths needed for failover
of application volumes are checked in advance, vs. relying on the OS to
complain if the SCSI connection to the device didn't set up correctly?

> But in any case, it appears to be in the implementation territory.

That's something I can agree with - namely that when nexii are established
is an orthogonal issue to the session identifiers used to establish
them, although I would caution implementers that there's a lot of code
out there that operates under the "bus walker" paradigm and hence wants
to see nexii (sessions) established aggressively.  In a code stack that
doesn't operate under the "bus walker" paradigm, "conservative reuse"
is still applicable, as it need only constrain what ISIDs have to be
used, and not when they are used.

Taking a whack at requirements, I think there are at least two of them:
- MUST support "conservative reuse" without requiring any iSCSI sessions
	to be terminated in order to set up the required sessions (i.e.,
	one can take the ISID from an existing session and a target portal
	group to which that ISID is not in use and set up an iSCSI session
	with that ISID without having to terminate any existing session for
	ISID usage reasons - this is tricky to state precisely when the
	possibility of allowing sessions to be set up on the fly well after
	booting is allowed).
- SHOULD use "conservative reuse" in setting up sessions from the same
	Initiator Portal Group (set of IP addresses) to different Target
	Portal Groups (i.e., reuse an existing ISID in preference to
	creating a new one).
I presume the "mapping police" will let us know if Initiator Portal Group
is not the right term for the second requirement ;-).

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 249-6449 *NEW*      FAX: +1 (508) 497-8500
[email-masked]         Cell: +1 (978) 394-7754
---------------------------------------------------

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 03 Jan 2002 11:08:14 -0800

I think we're getting close to wrapping this up.

>Once that's done, what's the value in permitting operating modes
> other than "conservative reuse"?

I thought we already discussed this - that of two initiator NICs
each establishing a session (acting as independent initiator ports)
to a different target portal group.

>Is anyone

Not that I know of.  But keep in mind that the nexii established 
for the boot discovery process may not have to be kept around
persistently by initiators (as a matter of fact, I know that HP-UX
does not).  There are also other cases - DLKM drivers, dynamically
discovered targets etc - which do not have nexii from boot time.
But again, I prefer to leave it there since this is a little orthogonal 
to the current issue.

> - SHOULD use "conservative reuse" in setting up sessions

I guess this is okay, the other alternative/complement is wording 
(proposed earlier in this thread) that focuses on where conservative 
reuse is a MUST.

>         Portal Groups

In the sense you used "initiator portal group" here, it ought to be
defined
(otherwise, simply use "initiator node" in your sentence) as "the set of 
initiator portals sharing the same ISID Type & ISID Naming Authority
values
and sharing one ISID Qualifier name space, and that allow a session to
span 
an arbitrary subset of the portals."

IOW, if two NICs A and B from the same vendor (so same Type and Naming 
Authority) are present in a host (so share the ISID Qualifier space,
perhaps 
with a s/w "ISID allocator" in the host or whatever), but can not span a 
session across them, then they are technically two independent
"initiator 
portal groups".

Thanks.
-- 
Mallikarjun 

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668	Hewlett-Packard, Roseville.
[email-masked]