Subject: Re: iSCSI: response to second login (with same ISID)

From: "Jim Hafner" <[email-masked]>
Date: Mon, 21 May 2001 12:49:00 -0700

Santosh,

Yes, that is exactly the context.   No concurrent sessions from the same
iSCSI initiator to the same iSCSI target with the same ISID.

And Yes, the recovery for persistent reservations is exactly the reason for
that rule.

You give valid arguments for option 2.

Jim Hafner

Santosh Rao <[email-masked]>@ece.cmu.edu on 05-21-2001 12:33:43 PM

Sent by:  [email-masked]

To:   [email-masked]
cc:
Subject:  Re: iSCSI: response to second login (with same ISID)

Jim Hafner wrote:

> The current thinking is that an iSCSI initiator must use a different ISID
> for each session it opens with a specific iSCSI target.

Jim,

Are you referring to multiple concurrent sessions from an initiator node
to a given target port above in saying that different ISIDs must be used
?

In the case where the initiator re-establishes a session to the same
target port after a power cycle, it is desirable to re-use the same
ISID, to allow for tracking of persistent reservation like properties,
correct ? (Just checking to make sure the ISID semantics did'nt change
since the Nashua discussion.)

> We need to decide on the iSCSI target's response when the iSCSI initiator
> attempts to break this rule.  The choices are:
> 1) REJECT the login with some error code (that the iSCSI initiator can
> understand or infer to mean "ISID in use") and leave the pre-existing
> session alone
> 2) CLOSE the pre-existing session (and abort all it's tasks) and then
> process the new login fresh.
>
> Personally, I have no strong feeling either way, but lean towards option
> (1).  It's simple, clean and doesn't involve any interruption.

I'd vote for (2) for the following reasons :

a) It is possible for initiators to use the ISID as a scheme to
establish persistence of their O.S. device files. In doing so,
initiators would attempt to obtain the same ISID for a given session to
a given target ports across boots. (and rightly so, for enabling targets
to track persistent reservation like properties, etc).

In the event where the initiator went through a non-orderly shutdown
which did not close all sessions, it would attempt to re-login on
re-boot with the same ISID. If the target rejected this re-login, per
(1) (citing an invalid ISID), the initiator would no longer be able to
maintain persistence of the session to an ISID.

b) Using (2) allows the hosts to hint to the target that the previous
session is now stale and thus triggers clean-up of stale session
resources.

Regards,
Santosh

---

From: "KRUEGER,MARJORIE (HP-Roseville,ex1)" <[email-masked]>
Date: Mon, 21 May 2001 18:47:36 -0400

I can't find Santosh original mail on this subject- I assume you are talking
about the case where TSID=0,ISID=dup session number?  So it wouldn't be a
recovery attempt.  I'd think #2 would result in unintended behavior - acts
like 2nd way to close a session and is a side effect that the initiator
shouldn't intend.  I feel #1 is the clean way to handle this case.

Marj
 
> Folks,
> 
> Santosh has requested a response to this question and I don't 
> think I've
> noticed an answer.  It's one of those "dot the i" things that 
> we need to
> complete.
> 
> Anybody else got any opinions?
> 
> Jim Hafner
>

---

From: Gary Kotzur <[email-masked]>
Date: Mon, 21 May 2001 13:56:14 -0700

I agree with your choice of number one, it is cleaner and the initiator can
"correct" his login after getting the error code.  

Gary Kotzur
Principal Member of Technical Staff
MaXXan Systems, Inc.
[email-masked]
Ph: 832-436-1012
Cl: 281-814-9396
Fx: 832-436-1099

This email message is for the sole use of the intended recipient(s) and may
contain confidential information. Any unauthorized use; review, disclosure
or distribution is prohibited. If you are not the intended recipient, please
contact the sender by reply email to [email-masked] and destroy all
copies of the original message.

-----Original Message-----
From: Jim Hafner [mailto:[email-masked]]
Sent: Monday, May 21, 2001 11:15 AM
To: [email-masked]
Subject: iSCSI: response to second login (with same ISID)

Folks,

Santosh has requested a response to this question and I don't think I've
noticed an answer.  It's one of those "dot the i" things that we need to
complete.

Jim Hafner

---

From: "Jim Hafner" <[email-masked]>
Date: Thu, 24 May 2001 08:12:10 -0700

Steph, Santoch, Nick, and All,

As the one who posted this question in the first place, I think it's time
for me to chime back in.

I see valid arguments for both behaviors and Nick supporting a feature to
enable that.

There are two scenarios we're dealing with.
(1) Initiator (for whatever reason) unknowingly reuses an active ISID.
Nick's example of the user application using a generic NIC is a good one (I
programmed a Java Initiator, and that's about as high up in the application
stack as you can get!).
(2) Error scenario of some sort, where the initiator and target don' agree
on state for the "live ISID".

The former begs for "login reject, ISID in use".  The latter begs for error
recovery.

The way I see it, however, is that the second case should be dealt with in
the context of error recovery (independent of the ISID question).  If we
don't have any way to cleanup a target's session without having a live
connection in that session to do so, then we're missing a major piece of
the iSCSI puzzle.  We have to have a mechanism for the target to clean up
session resources that are lost to the host.  Isn't there some timeout the
target can rely on here?  Isn't there a mechanism for the initiator to say
"I lost my brains on that session, so you can clean up"?  (Isn't this last
what Nick's Login PDU bit would say?)

So, I'd recommend we use "login reject, ISID in use" for the reused ISID.
We solve the "error case" as an error case (who's doing these scenarios?).
In the error case, the "login reject" is one form of error detection (there
are others).   Santosh argues that login shouldn't take too long, but error
recovery can take as long as it should to clean things up, even if it has
to go through "login->reject->cleanup->login" during boot.

One final note. The comparison with FCP behavior is only partially valid.
The problem is that FC login occurs at an extremely low level of the wire
protocol where the NICs have control of things in firmware (typically), or
at least down in the lowest level of the kernel.   In other words, that
process is owned by one self-contained system component.  For iSCSI, this
login, as has been noted, can occur anywhere and everywhere
(simultaneously) in the application stack.  So, I wouldn't necessarily
recommend that we model FC behavior here.

Jim Hafner

---

From: [email-masked]
Date: Fri, 25 May 2001 07:36:56 +0300

An amazingly long thread.

The basic assumptions of the thread have been that:

a - a resource (device) is allocated by reserve/release
b -  reservations are associated with a INAME+ISID as the "reserver name"
and therefore those have to be unique in a initiator target pair.

Assuming that everything else is running fine (initiators are able to
login), IMHO this looks entirely as a matter of recovery.  If a target
senses an attempt to create a new session with the same INAME+ISID as an
existing session it should:

    1-Determine if the old session is active (currently transmitting and
receiving or send a polling NOP-In and wait for
         an answer)
      If it is then it has to reject the new session
    2-If it does not it has to clean the old session and establish a new
session.

I also assume that well designed targets will determine that a session is
dead well before a new login forces them
to do so.

The larger question (that Marjorie alluded to)  is whether SCSI is designed
to allow user level access and employ only reservations for coordination
and if it is not conceivable that several sessions can own a reservation
(as several threads in process can get the same lock).  And if reservation
uniqueness within a given initiator should not be maintained by some SCSI
mechanism (a key that is already in SCSI) as in some networks I might have
now a thing running over FCP and after a while a session running on a
backup dialup iSCSI link.

Julo

"Martin, Nick" <[email-masked]> on 24-05-2001 09:38:17

Please respond to "Martin, Nick" <[email-masked]>

To:   "'Santosh Rao'" <[email-masked]>, [email-masked]
cc:
Subject:  RE: iSCSI: response to second login (with same ISID)

[The message I am replying to is below this one.  A similar one was posted
while I was composing this reply.]

Santosh,

If the iSCSI HBA is exposing SCSI targets to the OS, then the OS's SCSI
target driver (or class driver if I have my Microsoft terminology correct)
can support "exclusive access" semantics which can protect the current user
of a tape device from other users on the same system attempting to use the
same device through the same driver, or can allow a disk device (and the
session for it) to be shared by multiple users when that is appropriate.
If
the target device supports some form of SCSI reserve/release, then this
would protect the device from access via a second session from anywhere.
In
the target, iSCSI could also refuse a second concurrent session login to
the
same tape target protecting initiators which do not reserve/release.  (The
trick is: does the target think this is a new session using the same ISID
(reject), or an attempt to clean-up and re-establish an existing session
(reset, accept).)

Think about an iSCSI enabled version of tar or cpio (simple applications
used for tape backups).  These could run from user space over a TCP
connection on a standard NIC using an OS that has no awareness of iSCSI.
If
a second copy of the program is started, how is it to know that something
is
already using the same tape device?  How is it to know the ISID used for
that other copy?  How is it to know whether ANY particular ISID is really
currently in use, or what it is being used for?  How can we prevent the
second copy from unknowingly stealing the device from the first by simply
using the same ISID?  The second user thinks everything is fine until the
first user restarts his command that just failed.  In this model there is
no
HBA driver or target driver.  There is nothing that knows about both
sessions being attempted except the target.

You may think this is a bit far fetched, but I almost have such a program.
I expect other folks will be clever enough to think of other ways to use
this "user mode access" capability of TCP/IP to use remote iSCSI devices
without kernel co-operation.  Can we enable them to peacefully co-exist
with
each other?  Can they peacefully co-exist with kernel mode drivers?

There may be other scenarios such as administrators dynamically adding and
removing iSCSI devices to an OS while a system is running rather than at
boot time (think 24x7 operations, what choice to they have?).  If the
ISID's
are going to have to be manually configured, then there is a high
likelihood
of human error causing disruptions.  If we can enable non-disruptive
automatic ISID probing to find an available ISID during device
configuration
or initialization, this seems safer.  Once we pick an ISID, we should still
remember it across re-boot, to avoid repeated probing, and to assist manual
clean-up after ungraceful shutdown.  We also need the "logout this old
session" capability both for recovery after an initiator crash or error
which failed to logout, and for when the initiator realizes it has no
remaining working TCP connections to an existing session and wants to be
sure to free target resources and start fresh.

So, I think there are valid arguments for both behaviors 1 and 2, and we
need to enable the initiator to select the behavior appropriate for each
application.

Stated another way, we need to allow for the case when the initiator is
sure
that this ISID is unique and it takes full responsibility for all
consequences, and for the case where the initiator can not be sure this
ISID
is unique and wants there to be no side effects for others if it turns out
not to be unique.

Thanks,
Nick
-----Original Message-----
From: Santosh Rao [mailto:[email-masked]]
Sent: Wednesday, May 23, 2001 8:36 PM
To: Martin, Nick
Cc: [email-masked]
Subject: Re: iSCSI: response to second login (with same ISID)

"Martin, Nick" wrote:

>
> I know from personal experience how easy it is for an authorized user to
> unintentionally start a second copy of a backup program, while the tape
> drive is still in use.  This should be a harmless mistake.  What kind of
> traffic cop would prevent this request from making it to the iSCSI target
> and terminating the session already in progress?  (Rhetorical question,
> please do not describe for me such a mechanism ;)

Nick,

I know you don't want an answer to your rhetorical question ;-}, but, I
could'nt resist one quick query here.

In the scenario you describe, is it not the case that HBA drivers would
only send out a Login on the first open of that target, and on
subsequent opens would only bump an open_count or a ref_count of some
kind rather than send a login on the wire on every open ?

If that is correct, then, the scenario you describe would still cause
disruption even with the modified semantics, only, when the WRITE is
issued by the 2nd execution of the backup application (?).

Regards,
Santosh

---

From: "Martin, Nick" <[email-masked]>
Date: Fri, 25 May 2001 15:22:11 -0500

Julian,

I do not know how long it would take a target using a ping to determine that
an old session is really dead.  Some folks think it would be too long, thus
they want to be able to "login unconditionally".  Primarily I thought this
would be used for initiators rebooting after ungraceful shutdown.  (For
those who can otherwise reboot quickly, but would stall on each iSCSI
session not already recovered by the target.)

I do not expect to want to use a "login unconditionally" option.  (I can
handle rejection ;)
The scary thing to me was if an initiator could not try an ISID which was in
use (intentionally, accidentally, or erroneously) without disrupting
(logging out) its current user (a.k.a. option 2).  
It appears that if given a "login unconditionally" option, some folks would
use it exclusively.

If the initiator can specify the maximum time it would take a target to
notice dead connections and sessions, then the argument that waiting for
target recovery of the session would take too long presumably goes away.

I am thinking about login negotiated parameters like MinIdleTime and
MaxIdleTime values in seconds.  These could specify respectively the
connection idle interval after which a NO-OP (keep-alive) should be sent and
expected, and the time with no traffic received on this connection which
should be regarded as an implicit logout request due to connection failure.
(This could cause an async event if detected by the target and a working
connection remains in this session.)

Better names can be chosen.  Reasonable defaults might be 10 seconds and 60
seconds.  There should be a recommendation like MaxIdleTime should always be
at least 3 times MinIdleTime.  Such parameters would probably exist within
the Target and the Initiator, even if they are not negotiated or exchanged.
The intent is that an Initiator that cares, can limit Target connection
recovery delays.

These parameters set by the Initiator for the Target, would be known to
both.  A broken connection could be detected by each at approximately the
same time.  The keep-alive NO-OP's could be pings so that both directions
would be kept alive by a single exchange.  If the Initiator wants to be
responsible for keep-alives, it should do them before the specified
MinIdleTime, if it wants the Target to do them, it may need to do them also,
slightly after the specified MinIdleTime, but well before MaxIdleTime.

My networking is weak, so I am not sure whether there should be a separate
(shorter) time specified for the maximum time to wait for the reply to a
ping, or how long to wait to retry it.  I am not clear on the interactions
with TCP and its timeouts and retries.  Is there a maximum time a ping could
take and still get through?  Is there a maximum time a busy iSCSI target or
initiator would be expected to delay before sending a reply to a ping?

Does this sound like a good idea at all or not?

Thanks,
Nick

---

From: [email-masked]
Date: Sat, 26 May 2001 15:42:08 +0300

Martin,

I agree that it would be bad if at login an initiator would have to wait
for a long time for the target
to detect if the old session has gone away.   But there is nothing in  the
current spec that will
prevent a good target implementation doing what you describe whithout any
negotiation.

Regards,
Julo

"Martin, Nick" <[email-masked]> on 25-05-2001 23:22:11

To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
cc:
Subject:  RE: iSCSI: response to second login (with same ISID)

Julian,

I do not know how long it would take a target using a ping to determine
that

The scary thing to me was if an initiator could not try an ISID which was
in

connection idle interval after which a NO-OP (keep-alive) should be sent
and

seconds.  There should be a recommendation like MaxIdleTime should always
be

MinIdleTime, if it wants the Target to do them, it may need to do them
also,

with TCP and its timeouts and retries.  Is there a maximum time a ping
could

Thanks,
Nick

---

From: "Lakshmi Ramasubramanian" <[email-masked]>
Date: Sat, 26 May 2001 09:27:54 -0700

Even if the target is capable of supporting multiple initiators, won't
it cause problems
with devices such as disk - say, filesystem from the two initiators'
side attempt to write 
to the same sectors (even not intentionally) would cause data
corruption. Is iSCSI layer
suppose to guard against this type of device sharing? 

But, if the target serializes access to the device by reserve\release,
it would avoid
conflict. Is that how it's suppose to work? 

Thanks!
 -lakshmi

-----Original Message-----
From: Santosh Rao [mailto:[email-masked]] 
Sent: Friday, May 25, 2001 6:09 PM
To: Lakshmi Ramasubramanian
Cc: [email-masked]
Subject: Re: iSCSI: response to second login (with same ISID)

Why would the target want to reject the login, unless it is not capable
of [or does not wish to] speak to multiple initiators ?

In the case of most targets, they should be capable of handling multiple
initiators and therefore, should accept the login.

If the target is incapable of supporting multiple initiators or does not
wish to allow multiple initiators (ex : iSCSI tape drive preventing
concurrent access), then, it can reject the login with a status
class/response of 0x0205. (Target Conflict).

- Santosh

Lakshmi Ramasubramanian wrote:
> 
> How is a target driver suppose to respond to a login from an 
> initiator, when another initiator already has a session going on with 
> the given target? Is the target
> suppose to reject the second login (say, with an error "device busy"
or
> something simialar)?

---

From: "Martin, Nick" <[email-masked]>
Date: Tue, 29 May 2001 01:32:22 -0500

Julian,

Yes, there is nothing preventing a target from being "well behaved", there
is also nothing to assure the initiator that any particular target will be
"well behaved".  At the moment, I think we do not have a definition of "well
behaved" in terms of how long it should take a target to discover loss of
connection to an initiator.  Thus some folks are sure it will be "too long"
and that we need a way to circumvent waiting for it.  IMHO We should either
agree in advance on such parameters for all targets (if we could), or allow
all initiators to set, or at least query some behavior defining parameter
values used by their targets.

How often should a "well behaved" iSCSI target test each of its connections?
How long should a target wait for the response to a "NOP-OUT" (a.k.a. iSCSI
ping) before presuming the connection failed?  Is more than one attempt
required?  If this behavior should be configurable, should it be
configurable by the administrator of the target, or the administrator of the
initiator, or negotiated?  (For leased storage, these two administrators may
not even work for the same employer.)  To whom should it be visible/known?

I presume any connection is considered tested&OK for N seconds after the
latest valid PDU arrives.  Is a value for N discussed?  How long before or
after N seconds of inactivity should we begin testing the connection?  Does
this apply equally to targets and initiators listening on their iSCSI
connections?

If I interpret correctly, we presently have some parameters (negotiated) for
how long after detecting a connection failure, the target must preserve the
session state to allow the initiator to attempt to reconnect.

It may be that something already dictates "it will take 2.5 to 3.0 minutes"
(seconds? hours?) from the moment of initiator cease function until target
detection of connection failure, and this is an inherent characteristic of
iSCSI.  If this should be the case, then some folks will say those times are
fine with me, others may say that's too long.  Those who can not live with
our protocol can deploy something else.  However, I do not see what is
currently dictating how long this should or could take.  I hope to not
exclude applications with reasonably strict recovery time requirements.  I
also hope not to ping every second on every idle iSCSI connection only to
hasten recovery for a few applications.

Thanks,
Nick
-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Saturday, May 26, 2001 7:42 AM
To: [email-masked]
Subject: RE: iSCSI: response to second login (with same ISID)

Martin,

Regards,
Julo

Julian,

Thanks,
Nick