Subject: iSCSI: handling of persistent reserves during initiator reboot

From: "Thomas McSweeney" <[email-masked]>
Date: Tue, 24 Apr 2001 10:50:36 -0400

I'd like to move this discussion from the iSCSI MIB exploder to the IPS
exploder, since it has implications on the iSCSI spec.

John Hufferd appended a question asking whether the target would reset its
session counters under the following conditions:

>And if the Host was rebooted but comes back to the Target
>with the same iSCSI name and ISID, of a broken connection --
>which has outstanding persistent reserves -- then the
>Target must respond with the corresponding TSID of the
>interrupted Session.   This means that the Session has
>continued, and the connection has also been continued.

My response was:

I disagree.  Certainly, it is a new connection, so at the target the old
connection object would disappear and be replaced.  I think it is a new
session also - spec section 1.2.3 states that Login with a null TSID is for
a new session.  Whether or not the old session is cleaned up is still a
matter of debate...  Section 6.7.3 mentions "Login with an implied Logout",
which seems to apply to this case.  However, that depends on how the
"multiple sessions between one initiator and one target" issue is resolved.

If the initiator retained enough info about the I-T nexus despite the
reboot to send the correct non-null TSID on the (re)Login, then it should
have saved all its counters too.  The target does not reset its session
object counters due to within-session connection recovery.

John replied:

>I did not say that the initiators saved the TSID, just their own ISID. The
>point is that it is a SAM-2 requirement that if Persistent Reserves are
>outstanding, the Initiator should be able to clean them up or continue to
>work with them after a boot.  This will mean that the target will have to
>detect that  Persistent Reserves are outstanding, and keep approprate
>information around for the reconnection (with the InitiatorName=xxxxxxx,
>and ISID of zzzzzzz).  The Target will then need to respond --- when a new
>session is being created for InitiatorName=xxxxxxx, and ISID or zzzzzzzz
>--- with the TSID used with the broken connection.  In this way the
>initiator can clean up or continuation of the Persistent Reserves.

>Persistent Reserves are a bit different then any other State that is kept
>by the Target.

I believe the Login which the rebooted initiator sends has to be for a
"leading connection", so it can learn the value of negotiated iSCSI
parameters like MaxConnections and FMarker that are only exchanged during
initial session activation.  So to me it's a new session.  It shouldn't
matter to the underlying SCSI resources whether the TSID matches the
previous session or not.   I'm not sure how the target should handle the
outstanding persistent reserves - I assume they can be moved to the new
session.  Now it's time for the real experts to chime in...

I'd like to see a new section added to Chapter 6 of the spec to describe
the handling of outstanding persistent reserves, since it is a special
recovery case.

As for the SNMP objects, I think the counts will be reset (old session
instance disappears, new session instance initialized to zero).  This would
make the target's counts consistent with the initiator's.  We plan to
discuss the tracking of long-term (across multiple sessions) usage per
initiator in Nashua.

Tom McSweeney
iSCSI Development, Storage Systems Group, IBM
Email: [email-masked]
Phone: (USA) 919-254-5634  (tie line: 444-5634)
Fax:   (USA) 919-254-0391  (tie line: 444-0391)

---

From: "Jim Hafner" <[email-masked]>
Date: Tue, 24 Apr 2001 15:44:54 -0700

Sandeep,

Suppose an initiator has a persistent reservation for a logical unit on
some target device.  This initiator has two choices:
a) a requirement, as you suggest, to remember some initiator state (the
ISID it used for the particular target device and the reservation key that
it was using, the latter in any case) OR
b) it doesn't recover the reservation "for free" but must use a heavier
hammer (like prempt and abort) to recover the reservation

As in another part of my note, a host rebooting is only one scenario where
this reservation auto-recovery should occur (as seen from the SCSI layers).
For all the other cases, the host doesn't loose any state (it never went
down) so things are OK.

For a host rebooting, a lot has to do with why it shutdown in the first
place and whether it expects to get back its reservations with minimal
effort.   One can argue that in the case of reboot, the host will start
from scratch for everything and use the bigger hammer.  Or one can argue
that if it really cared, it can preserve this state information.  I don't
think this is a major requirement either way.

As for iSCSI-FC bridges, I'm not thought about the requirements.  A very
large lot depends on the model that that bridge implements:
a) how much does the bridge attempt to be a "front" for an FC device and
tunnel initiator state through it
b) how much does the bridge act as an independent iSCSI target and just
aglomerate all (some of) the logical units on the FC network behind it.

For case (b), the bridge is an iSCSI target device and therefore needs to
support all target requirements.
When I look at case (a), I seem to always come to the conclusion that such
bridges cannot be very stateless and that this extra burden is relatively
minimal.  For example, the bridge will need to something non-trivial to
restore initiator state through itself when (a) changes occur on the FC
side, (b) the target resets completely, (c) the IP side resets (sessions
crash and burn for IP or link layer errors).  Adding an ISID to the "saved"
state along with the iSCSI Initiators Name doesn't sound like a whole lot
of extra stuff.

Jim Hafner

Sandeep Joshi <[email-masked]>@research.bell-labs.com on
04-24-2001 09:51:15 AM

Sent by:  [email-masked]

To:   Jim Hafner/Almaden/IBM@IBMUS
cc:   [email-masked]
Subject:  Re: iSCSI: handling of persistent reserves during initiator
      reboot

Jim Hafner wrote:
>
> Tom,
>
> Here's my two cents. Apologies for being long-winded.  In short, I think
I
> agree with most of your sentiment.
>
> 1) In order to accommodate Persistent Reservations (and a few other SCSI
> things), I think it is important to have some mechanism for an initiator
to
> reestablish some nexus state information through logout/login.  The
current
> thinking in N&DT is that the initiator should reuse it's old ISID and the
> target should respond with the old TSID (thereby rebuilding the nexus).
> Note, the initiator starts with TSID=0, as if it was a new session.  The
> target that remembers the old ISID/TSID pair (can reuse the old TSID).
Keep
> in mind that that target is already remembering some state information
(the
> reservation) about the ISID/TSID (and Names) relationship in the nexus,
so
> this isn't a big deal.

If I am not mistaken, isnt this also adding to initiator state ?
Initiators now have to remember the ISIDs used with every target
(or have only one!)

And doesnt this also add state to iSCSI-FC bridges...?

Regards,
-Sandeep