Subject: RE: iSCSI: Need to Kill Session from Surviving Node

From: "John Hufferd" <[email-masked]>
Date: Thu, 15 Mar 2001 02:16:48 -0800

Target Reset have value, but not as an in-line command.  It is a management
function, not a recovery function from a single initiator, and as such
should be up to the implementation to see how the accomplish the Target
Reset via there own admin facility.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Internet address: [email-masked]

"Mallikarjun C." <[email-masked]>@ece.cmu.edu on 03/14/2001 07:31:29 PM

Please respond to [email-masked]

Sent by:  [email-masked]

To:   [email-masked] ()
cc:   [email-masked]
Subject:  RE: iSCSI: Need to Kill Session from Surviving Node

Somesh,

Please refer to the following URL (and emails around that) for
a long discussion on this topic.  Both flavors of resets have value,
and I agree with Julian that iSCSI should support both.

http://ips.pdl.cs.cmu.edu/mail/msg00740.html

--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668   Hewlett-Packard, Roseville.
[email-masked]

>Do you need both a "warm-reset" and a "cold-reset"?
>The "warm-reset" seems to be an over-optimization.
>
>Somesh
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]]On Behalf Of
>> Mallikarjun C.
>> Sent: Wednesday, March 14, 2001 12:16 PM
>> To: [email-masked]
>> Subject: Re: iSCSI: Need to Kill Session from Surviving Node
>>
>>
>> >There is a related problem that I think we need to address, and
>> that is --
>> >the case where the Storage Controller is an Active-Active Fail-Over
unit
>> >with very little state shared between the Storage Controller Nodes.  (I
>> >think a number of you folks have told me that you are planning such a
>> >design.)  This would mean that a specific initiator OS (WWUI) would
have
>> >separate sessions between the Nodes.  At fail-over, the
>> Take-Over Node will
>> >want to cause the Initiator, to Stop messing around with the Failing
node
>> >ASAP, and start using the existing session, or start a new
>> session with the
>> >Take_Over Node.  The Initiator, will normally just time-out the TCP/IP
>> >connection, and then turn the recovery over to the SCSI Layer. Then,
when
>> >SCSI retries, iSCSI will either use the Session to the Surviving Node
or
>> >Create a New Session to that Surviving Node.  The problem is, that this
>> >will take a longer time then a system might want.
>> >
>> >So the question: is there a way for a surviving Target Node to tell the
>> >Initiator to "Kill" the Session to the Failing Target Node?
>> >
>> >Comments?
>>
>> Assuming that the failing Node is still able to execute commands,
>> a cold target reset from the Initiator to the failing node should "kill"
>> all the sessions to it.
>>
>> The answer to two hosts configured in a failover configuration sharing
>> a single Storage Controller Node is also the same.
>>
>> If it is active-active configuration, it appears to me that there's a
>> need to somehow prompt the Initiator to doing the reset on a third party
>> (Async PDU?).
>>
>> This is a reason why we'd still need target reset in iSCSI.
>> --
>> Mallikarjun
>>
>>
>> Mallikarjun Chadalapaka
>> Networked Storage Architecture
>> Network Storage Solutions Organization
>> MS 5668     Hewlett-Packard, Roseville.
>> [email-masked]
>>
>> >
>> >.
>> >.
>> >.
>> >John L. Hufferd
>> >Senior Technical Staff Member (STSM)
>> >IBM/SSG San Jose Ca
>> >(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
>> >Internet address: [email-masked]
>> >
>> >
>> >Julian Satran/Haifa/IBM@[email-masked] on 03/14/2001 08:31:19 AM
>> >
>> >Sent by:  [email-masked]
>> >
>> >
>> >To:   [email-masked]
>> >cc:
>> >Subject:  Re: iSCSI: Logout needs ISID
>> >
>> >
>> >
>> >
>> >
>> >Mark,
>> >
>> >If you have only one connection you are supposed to use a Login with
the
>> >restart bit one - and achieve a similar effect as a Login/Logout - i.e.
>> >keep the session alive.  Even this might be a problem for a
>> target that is
>> >so poor it has only one socket (even SNMP won't work there).
>> For this case
>> >the only way out is to drop the connection and hope the target will
hear
>> >you (it probably won't -:)). The comment is the draft is a memento for
>> >implementers to keep looking for new connections always (even for a one
>> >connection target) but probably it does not come through clear enough.
>> >
>> >Regards,
>> >Julo
>> >
>> >Mark Bakke <[email-masked]> on 14/03/2001 18:17:14
>> >
>> >Please respond to Mark Bakke <[email-masked]>
>> >
>> >To:   Julian Satran/Haifa/IBM@IBMIL
>> >cc:   [email-masked]
>> >Subject:  Re: iSCSI: Logout needs ISID
>> >
>> >
>> >
>> >
>> >Julian-
>> >
>> >A target that does not support multiple connections per session
>> >will return "Initiator SID Error" when the login attempt is made.
>> >In this case, there is no way to log in, so there's no way to
>> >log out the old connection.  The initiator would be stuck waiting
>> >until the target's side of the connection times out and goes
>> >away, which could be a very long time.
>> >
>> >There is an open question within the MIB team about whether anyone
>> >needs to be able to kill connections or sessions from SNMP; however,
>> >I don't think that anyone will want to use SNMP as part of error
>> >recovery.
>> >
>> >--
>> >Mark
>> >
>> >[email-masked] wrote:
>> >>
>> >> Josh,
>> >>
>> >> No command, including logout, can be sent without a login.
>> >> The complete scenario is:
>> >>
>> >> -open a new connection
>> >> -login in the same session as the old connection (this has the ISID &
>> >TSID)
>> >> -logout the old connection
>> >> -recover commands
>> >>
>> >> You are not supposed to be able to kill a session from outside
>> (at least
>> >> not an iSCSI defined mode).
>> >> You will need management support for that (SNMP?)
>> >>
>> >> Regards,
>> >> Julo
>> >>
>> >> Joshua Tseng <[email-masked]> on 14/03/2001 17:09:48
>> >>
>> >> Please respond to Joshua Tseng <[email-masked]>
>> >>
>> >> To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
>> >> cc:
>> >> Subject:  RE: iSCSI: Logout needs ISID
>> >>
>> >> Julian,
>> >>
>> >> In 2.14, you state that a logout command can be
>> >> sent on a different connection to destroy a single-
>> >> connection iSCSI session.  If you have multiple
>> >> sessions ongoing, and the logout command is sent
>> >> on a different connection than the one used
>> >> to support the session that is to be killed, then
>> >> you will need ISID to distinguish the session that
>> >> you want killed.
>> >>
>> >> Regards,
>> >> Josh
>> >>
>> >> > -----Original Message-----
>> >> > From: [email-masked] [mailto:[email-masked]]
>> >> > Sent: Tuesday, March 13, 2001 9:57 PM
>> >> > To: [email-masked]
>> >> > Subject: Re: iSCSI: Logout needs ISID
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Josh,
>> >> >
>> >> > There is something I am missing. The Logout can be issued
>> >> > only after Login
>> >> > (authentication and the rest).
>> >> > The session is then implied.
>> >> >
>> >> > Regards,
>> >> > Julo
>> >> >
>> >> > Joshua Tseng <[email-masked]> on 14/03/2001 03:56:45
>> >> >
>> >> > Please respond to Joshua Tseng <[email-masked]>
>> >> >
>> >> > To:   Julian Satran/Haifa/IBM@IBMIL
>> >> > cc:   [email-masked]
>> >> > Subject:  iSCSI:  Logout needs ISID
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Julian,
>> >> >
>> >> > Section 2.14 states that the logout command may
>> >> > be sent on a second connection to clean up the
>> >> > a single connection iSCSI session.  If the reason
>> >> > code is 0 (close the session), then ISID is needed
>> >> > to identify the iSCSI session to close.
>> >> >
>> >> > Josh
>> >> >
>> >> >
>> >> >
>> >
>> >--
>> >Mark A. Bakke
>> >Cisco Systems
>> >[email-masked]
>> >763.398.1054
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>
>_________________________________________________________
>Do You Yahoo!?
>Get your free @yahoo.com address at http://mail.yahoo.com
>
>

---

From: "Elliott, Robert" <[email-masked]>
Date: Thu, 15 Mar 2001 11:43:28 -0600

There is a T10 proposal being developed by Ken Moe at Sun (originally by Bob
Snively) titled Asymmetric SCSI Behavior (00-232r6) that deals with
controller fail-over.  It describes how a controller with multiple target
ports connected to the same logical unit can report the state of those
target ports (active, standy, or unavailable) with a new REPORT TARGET
GROUPS command.  It lets an initiator initiate state changes using a new SET
TARGET GROUPS command.

If the fabric path to a controller's target port fails, then I think you
would be more concerned with getting the path running again than with
reusing some now unusable target port resources.  It's like a RAID system
during a disk rebuild - you are operating without redundancy and are subject
to a single point of failure until you fix the problem.  

During this time the logical unit resources can be recovered.  CLEAR TASK
SET and LOGICAL UNIT RESET task management functions work.  If persistent
reservations is being used, the PREEMPT AND ABORT service action can clear
out the tasks from the nexus that is no longer available.

I think it's a mistake to affect logical units you don't own, which a TARGET
RESET would do.  Some other operating system might be using them and expect
a different error recovery procedure.

> From: John Hufferd [mailto:[email-masked]]
> Sent: Wednesday, March 14, 2001 11:21 PM
> To: [email-masked]
> Wait a minute, I did not expect my Surviving Node question to somehow
> trigger the discussion of whether the Target Reset should be issued by
> iSCSI Initiators.   That is a completely different Pit.
> 
> My question was, when the Storage Controller fails over to a surviving
> NODE, how do we tell the Initiator that is waiting peacefully for the
> TCP/IP connection to time-out.  To quickly give-up and let 
> the initiator's
> SCSI layer, retry the operations, so that the iSCSI layer can 
> start either
> sending the SCSI cmds to the Surviving Node, or starts a new 
> Session with
> the Surviving Node.
> 
> This has nothing to do with the Initiator issuing Target Resets.
> 
> 
> 
> .
> .
> .
> John L. Hufferd
> "Somesh Gupta" <[email-masked]>@ece.cmu.edu on 03/14/2001 
> 08:25:58 PM
> 
> Please respond to <[email-masked]>
> 
> To:   <[email-masked]>, <[email-masked]>
> cc:   <[email-masked]>
> Mallikarjun,
> I am not disagreeing with the desire to do a graceful
> termination. However, when recovery states at the TCP
> level, and at the iSCSI level get intertwined with this,
> it is probably better to focus on a clean (possibly graceful)
> termination and a clean bringup.
> 
> Somesh
> 
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Mallikarjun C.
> > Sent: Wednesday, March 14, 2001 7:31 PM
> > To: [email-masked]
> > Cc: [email-masked]
> > >> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> > >> Mallikarjun C.
> > >> >the case where the Storage Controller is an Active-Active
> > Fail-Over unit
> > >> >design.)  This would mean that a specific initiator OS (WWUI)
> > would have
> > >> >want to cause the Initiator, to Stop messing around with the
> > Failing node
> > >> >connection, and then turn the recovery over to the SCSI
> > Layer. Then, when
> > >> >SCSI retries, iSCSI will either use the Session to the
> > Surviving Node or
> > >> >If you have only one connection you are supposed to use a
> > Login with the
> > >> >the only way out is to drop the connection and hope the
> > target will hear
> > >> >you (it probably won't -:)). The comment is the draft 
> is a memento
> for

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 15 Mar 2001 10:25:38 -0800

Elliott,
I think what you said is right (Don't use Target Reset).  That as I have
stated should be an Admin action.  OK,

What I was trying to get at is ---  trying to enable the Surviving Node
(which has no TCP/IP state from the Failing node) to get the Initiator,
which had a previous Session with the Failing Node to take actions right
away, and not wait for its TCP/IP connections with the Failed Node to Time
Out.  (This requires some sort of a message from the Surviving node back to
an Initiator.) This is not quite what we have defined into iSCSI.

Since, I preceived that a number of companies were building such devices it
seemed that a method for accomplishing such a Fail-Over, in a timely manor
might be a common requirement.  If so, did anyone have any bright
ideas......

.
.
.
John L. Hufferd

"Elliott, Robert" <[email-masked]>@ece.cmu.edu on 03/15/2001
09:43:28 AM

To:   [email-masked]
cc:

There is a T10 proposal being developed by Ken Moe at Sun (originally by
Bob
Snively) titled Asymmetric SCSI Behavior (00-232r6) that deals with
controller fail-over.  It describes how a controller with multiple target
ports connected to the same logical unit can report the state of those
target ports (active, standy, or unavailable) with a new REPORT TARGET
GROUPS command.  It lets an initiator initiate state changes using a new
SET
TARGET GROUPS command.

during a disk rebuild - you are operating without redundancy and are
subject

I think it's a mistake to affect logical units you don't own, which a
TARGET
RESET would do.  Some other operating system might be using them and expect
a different error recovery procedure.

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 15 Mar 2001 22:26:10 -0800

OK, let me see if I can stretch it a bit more.  The key to what you said,
is for the Initiator needs to be sending something to the target, not just
waiting.  So if the Initiator always sent a Nop-Out after some period of
time of inactivity from the target, (hopefully a lot smaller time then the
TCP/IP Time Out) then the Fail-Over node would be able to cause the failure
reasonably quickly by returning ICMP Destination Unreachable.

If I have it right, it seems straight forward, but of course you need to
take over the IP Address.

.
.
.
John L. Hufferd

[email-masked]@ece.cmu.edu on 03/15/2001 06:26:44 PM

To:   John Hufferd/San Jose/IBM@IBMUS, [email-masked]
cc:   [email-masked]
Subject:  RE: iSCSI: Need to Kill Session from Surviving Node

> What I was trying to get at is ---  trying to enable the
> Surviving Node (which has no TCP/IP state from the Failing
> node) to get the Initiator, which had a previous Session with
> the Failing Node to take actions right away, and not wait
> for its TCP/IP connections with the Failed Node to Time Out.
> (This requires some sort of a message from the Surviving
> node back to an Initiator.) This is not quite what we have
> defined into iSCSI.

Since this is a TCP problem (get the connection to close),
iSCSI may not need to do anything ... what about failing over
the IP address and responding with ICMP Destination Unreachable
to anything that arrives on that failed-over IP address?
Implementations of this probably need to support multiple
IP addresses on a single network interface, which is not a
big deal.

--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------