Subject: Re: [Ips] iSCSI: When does a connection become part of a session?

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 11 Jan 2004 21:20:06 -0800

Fellow iSCSI Architects,

        I wanted to bring up a few points in regards to Joseph Pitmann's
2nd topic: "when should Connection Reinstatement occur?", how it applies
to the discussion below, and the implementation method we use that does
not require connection count to exceed MaxConnections even in the single
connection ErrorRecoveryLevel=2 case, and hence makes the passage in
section 3.2.1 not apply.

"For error recovery purposes, targets and initiators that support a
single active connection in a session SHOULD support two connections
during recovery."

First a couple prerequisites on both sides:

Initiator:

In an ErrorRecoveryLevel=2 single or multiple connection session, when a
connection fails the initiator will attempt to perform connection
reinstatement on the failed CID before creating a connection with a new
CID.

Target:

The login process controls how many login attempts are possible on a
given session to prevent a DoS attack.

End prerequisites.

Now, the following order of steps on the target allows MaxConnections to
be inforced while still allowing a single connection (as well as
multiple, but the main concern Mallikarjun mentioned was the single
connection example) ErrorRecoveryLevel=2 session to perform Connection
Recovery.

In a single connection ErrorRecoveryLevel=2 connection, CID 0 goes into
cleanup state from the initiators perspective yet remains active on the
Target.  The Initiator sends a non zero TSIH login request (CSM-I) with
CID 0 to perform Session Continuation by way of Connection Reinstatement
(and hence Implict Logout) of the failed connection still active from
the Target's perspective (CSM-C).

Connection Reinstatement is performed after SecurityNegoitation phase
has completed successfuly but before MaxConnections is checked. The
ExpStatSN of the initial login request is checked against commands of
the connection that was just forced to shutdown (CSM-C) to release those
acknowledged by the initiator before the connection failed, then finally
the MaxConnection key is checked.  At this point we continue to
LoginOperationalNegotiation stage.

This is the implementation we have used for a period of time and think
is the correct method to avoid having to exceed the negoitated
MaxConnections key.  We have come to the conclusion based primarly on
the following point that Pat and Mallikarijun mentioned:

>> However note that a failed connection
>> reinstatement causes T15 which moves the session from LOGGED_IN to
>> CLEANUP_WAIT so a failed attempt would still disrupt an active
>> connection.

> True, that was intended.  However, there are two state
> machines here (CSM-C and CSM-I, borrowing 7.2 terminology),
> and note that Pat's clarification applies to CSM-C (i.e.
> the state machine representing the old instance).  The CSM-I,
> the new state machine the login is happening on, OTOH,
> takes the T7 transition from IN_LOGIN to FREE.

> A "failed connection reinstatement", however, should have
> been defined in the spec (as Pat says).  For a connection
> reinstatement effort to be considered "failed" and thus
> cause a state change to CSM-C on the target end, the effort
> should have progressed (and failed) beyond the SecurityNegotiation
> stage.  Ditto for session reinstatement.

Taking this into account, I see the above order of steps as the desired
implementation because when a connection makes it past the point of
successful SecurityNeogitation, the existing CID (CSM-C) will be forced
into shutdown regardless if CSM-I makes it into Full Feature Phase or
not.

The other implemenation example(s) that has been discussed so far that
requires the connection count to exceed MaxConnections in a single (or
multple) connection ErrorRecoveryLevel=2 session roughly follow these
steps:

Connection Reinstatement happens when the last login response is sent
before moving to Full Feature Phase.  The MaxConnection key is checked
either before or after security negotiation phase, requiring the
connection count to temporarily exceed MaxConnections in order to
perform Connection Reinstatement before moving to FFP.  This is of
course legal but IMHO exceeding MaxConnections is generally frowned
upon. 
  
As mentioned in the prerequisities, an Initiator in a single or multiple
connection ErrorRecoveryLevel=2 session should attempt connection
reinstatement before starting a new connection with a different CID.
So basically in ErrorRecoveryLevel=2, Session Continuation is achieved
by performing continuous Connection Reinstatement on failed CIDs and NOT
by starting new connections with different CIDs. This is the method that
our ErrorRecoveryLevel=2 multiple connection session iSCSI Initiator
uses that will be open sourced in the near future.

Thanks!

-- 
Nicholas A. Bellinger <[email-masked]>

On Thu, 2004-01-08 at 17:00, [email-masked] wrote:
> On Thu, 8 Jan 2004, Mallikarjun C. wrote:
> 
> >  >Also, our target expects StatSN (and
> >  > ExpStatSN) to be handled correctly even in login. As StatSN is
> >  > connection-specific, I see no session nor security problems
> >  > with this. :-)
> >
> > Agreed.
> >
> >  > Does anyone see a security implication of the target leaking
MaxCmdSN and
> >  > ExpCmdSN?
> >
> > I think inadvertent revealing of current ExpCmdSN even before the
> > security negotiation stage allows an attacker to gain insights
> > into traffic volumes - he can periodically use this to get
> > snapshots of the ExpCmdSN value.
> > I think I agree with you and Joseph (private EMail) that it would be
> advantageous to not expost ExpCmdSN & MaxCmdSN. Unfortunately the spec
> requires their inclusion at present. :-|
> 
> >  >We take a third approach. If we don't already have MaxConnections
+ 2
> >  > connections in FFP and in login, we let a new login proceed.
> >
> > OK, this is one implementation approach.  I see this as a
> > superset of the required behavior I described.
> 
> Yes, it's a superset.
> 
> Oh, as for why it's + 2 as opposed to + 1 is a bit of
over-conservativism
> on my part. :-)
> 
> >  >If we didn't
> >  > do something like this, then if you had a session with
MaxConnections
> >  > connections, you could not do any ERL 2 recovery.
> >
> > Not necessarily.  I'd think if a TCP connection for CID=x is dead
from
> > the initiator's perspective, the initiator can Logout CID=x on a
>
> I had in mind the case where you had MaxConnections connections and
they
> all died. Yes, it's extreme.
> 
> > connection with CID=y, and do a task reassign for all tasks active
> > on CID=x.  If MaxConnections=1 and connection recovery is supported
> > and the target thinks the connection is still up though initiator
> > thinks it's dead, it would have to adopt an approach like yours.
> > My recollection is that the iSCSI spec in 3.2.1 says the following
> > for the same reason.
> >
> >     "For error recovery purposes, targets and initiators
> >           that support a single active connection in a session
> >           SHOULD support two connections during recovery."
> 
> Yep. That's why we came up with permitting a little more than
> MaxConnections connections.
> 
> Take care,
> 
> Bill
> 
> _______________________________________________
> Ips mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: [email-masked]
Date: Tue, 13 Jan 2004 09:20:07 -0800 (PST)

> reinstatement on the failed CID before creating a connection with a new

Well, that's not the only thing the initiator can do. You're talking about
adding a more-narrow restriction on the initiator than the spec says;
you're talking about not permitting CSM-E.

> LoginOperationalNegotiation stage.

I do not see what is gained by having connection reinstatement happen
between the Security and Operational phases. You _still_ have to support a
connection that hasn't passed through security (you have to allocate
resources to support security), and you have also muddled the waters for a
connection that bypasses the security phase all together (which is legal).
I think it's easier to just wait for FFP to happen.

> into shutdown regardless if CSM-I makes it into Full Feature Phase or

To be honest, I think I'd just can (throw out) the concept of
"reinstatement failed" for the target. Waiting for FFP seems much cleaner
and simpler. The only reason I'd see that a target would fail the
transition after starting it would be some sort of internal error.

I think the state makes a lot of sense for an initiator, for which it
knows that it is trying to do the reinstatement.

All we're doing by having this state in the target is having the
reinstatement code let a misbehaving initiator (I mean come on, how do you
screw up reinstatement operational processing, if you aren't being
stupid? :-) trigger state transitions. As the spec was vague about this
point, it seems valid to define the transition in the target such that it
won't fail.

> course legal but IMHO exceeding MaxConnections is generally frowned

I disagree with this statement. MaxConnections is to protect resources.
What resources do you need in the operational phase that you don't need in
the security phase? For both, you need the negotiating engine, and you
need to know what you're negotiating. So I see no advantage to looking at
MaxConnections at Security->Operational. Either you look at tcp_accept()
time (when you're allocating connection resources), or you look after
negotiations are done.

Also, as above, connections can bypass the security phase altogether. So
it strikes me as much cleaner to think about connections as in login or
done with it. Doing otherwise (looking in the middle) seems messy.

Finally, the "having two in error recovery" is, as far as I can tell, to
permit people to do CSM-E cleanup. So when we do the connection counting
won't matter; to do CSM-E, you will still need to let the new connection
get to FFP so that it can clean the old one up.

> uses that will be open sourced in the near future.

While that would work, you are adding a requirement to the spec that is
not there. The encouragement (SHOULD) to being able to exceed
MaxConnections during error recovery indicates that the spec authors
really wanted to be able to log in a different connection and do an
explicit Logout on the old connection.

Also, I think you've muddled two things in the above. You're talking about
doing the connection count at the transition between security and
operational phases, and you talk about CSM-I being prefered. I think
prefering CSM-I is what lets you get around MaxConnections=1, and that
looking in the middle of login isn't helping there.

Take care,

Bill

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 13 Jan 2004 22:48:00 -0800

> you're talking about not permitting CSM-E.

First off, lets both agree that CSM-E is an explict logout by way of a
REMOVECONNFORRECOVERY Logout Request, and has an nothing explicty to do
with exceeding MaxConnections. I have not mentioned anything about not
permitting CSM-E, what I am discussing is the implementation method and
very clear case where it is not needed in a single connection
ErrorRecoveryLevel=2 session in the context of enforcing the negoitated
MaxConnections key.  The fact that this method works regardless of
ErrorRecoveryLevel is a key factor into why it is a simpler and
effective method of achiving our desired goal of interoperable
implementations.

The fact of the matter is when a target follows the aforementioned steps
of checking MaxConnections after Connection Reinstatement occurs, there
are very few (one that I can think of) situations where creating a new
connection with a different CID is adventagous to performing Session
Continuaion by way of continous Connection Reinstatement, and hence
making CSM-E a not apply.

The only example that I have encountered requires user intervention in a
'traditional iSCSI' single connection ErrorRecoveryLevel=2 session is as
follows.

Initiator: CID 0 to IP address 0 on Communication (Ethernet) Port 0
fails. Connection Reinstatement to IP address 0 fails because IP address
0 no longer exists or Communcation Port 0 is no longer active.

With User Intervention: While the Initiator is attempting to perform
Connection Reinstatement on CID 0 unsuccessfuly and before the session
falls back to Session Reinstatement, the User attempts to add another
connection with CID 1 to IP address 1 on Communcation Port 1 with the
same Target Portal Group Tag where communcation is active. 
In the example where the Initiator is temporarily allowed to exceed
MaxConnections, a explict logout (CSM-E) is sent on CID 1 to shutdown
CID 0 and then returns to a connection count that does not exceed
MaxConnections.  In the example where MaxConnections is enforced the
Login Request for CID 1 by User is rejected, Connectection Reinstatement
on CID 0 times out and the session falls back to Session Reinstatement.
(ie: the 'Without User Intervention' case)

Without User Intervention: After the Connection Reinstatement attempt on
CID 0 times out, the Initiator falls back to Session Reinstatement and
into Target Portal Group failover looking for a IP address on a
Communcation Port that it can perform Session Reinstatement into.  Once
Session Reinstatement is performed, the connection still active from the
Target's prespective is forced to shutdown.

If there is any other situation(s) where MaxConnections must be exceeded
in order to continue on with Connection Recovery while following the
single Initiator prerequsite of performing Session Continuation by way
of continous Connection Reinstatement, and the aforementioned method of
performing Connection Reinstatement before checking MaxConnections after
SecurityNegotiation is complete in _any_ ErrorRecoveryLevel, I would
like to hear them now.

> I think it's easier to just wait for FFP to happen.

The key advantage to this whole discussion is when the Initiator does
Session Continuation by way of continious Connection Reinstatement, and
the Target performs Connection Reinstatement after SecurityNegoitation
is complete, MaxConnection can be enforced.

As for MaxConnections, I have always interpreted its meaning as the
maximum allowable Full Feature Phase connections in an iSCSI Session. 
Its value has no bearing on connections that have not passed into Full
Feature Phase state.  This is the reason for the above Target
Prerequsite for controlling resources and avoiding DoS attacks.

Additonally, it is obvious that when AuthMethod is negoitiated to None,
or the Initiator decides to skip SecurityNegotiation all together with a
Login Request containing CSG=1 and NSG=[1,3], the Connection
Reinstatement and MaxConnections checks occur upon receipt of the
initial Login Request that indicates its desire to skip CSG=0.  This is
why I explictly said after SecurityNegotiation and not during. 

> point, it seems valid to define the transition in the target such that it

I see the primary reason for enforcing the concept of 'reinstatement
failed' is the simple fact that a Initiator would not be attempting
Connection Reinstatement on a given CID unless it had already failed
from its perspective.  There is not a single reason once a Connection
Reinstatement attempt has passed SecurityNegotiation on a target, that
CSM-C should not be forced to shutdown given this very clear point.

This state is not about accomidating broken initiators, this state is a
subtle indication about what is the correct point during the login
procedure to perform Connection Reinstatement.  That point being after
sucessfuly SecurityNegotitation, or upon reciept of a logout request
containing CSG=1 and NSG=[1,3]

> get to FFP so that it can clean the old one up.

I stand firm to the points made above:

The negotiated MaxConnections keys sole purpose is the control the
maximum allowed Full Feature Phase connections in an iSCSI session, and
not to control resource allocation during login.  I have never seen
MaxConnections discussed in a pre-FFP resource allocation capacity until
this thread.  Pre-FFP resource allocation should be controlled by an
external mechinism as was mentioned in the Target's prerequsites.

In the previous points regarding Connection Reinstatement and
MaxConnections being checked after SecurityNegotation is complete (or
upon the initial login request with CSG=1, NSG[1,3]).  The check of
MaxConnections is all that occurs and the connection count is not
incremented until Full Feature Phase occurs.

As for the third paragraph quoted above, all of the examples and
concerns have been addressed at the top of this reply.  There is no
logical reason why an Initiator should start an new connection with a
different CID when Connection Reinstatement is performed before
MaxConnections checked.

The only two examples where CSM-E is required in a single connection
ErrorRecoveryLevel=2 session if when a new connection with a different
CID is started instead of performing Connection Reinstatement (which
ErrorRecoveryLevel=2 Initiators do this as normal operation?), and in
the case of a Target that performs Connection Reinstatement before Full
Feature Phase, and after MaxConnections is checked and hence exceed. 
Using CSM-E in the latter case for normal single connection
ErrorRecoveryLevel=2 operation is accomidating a Target that chooses to
use a implementation method that goes against the 'failed connection
reinstatement' stage mentioned above.    

> looking in the middle of login isn't helping there.

This is not a requirement, but a simple and straightforward
implementation method that allows the negotiated MaxConnections key to
be enforced.  Considering there is only a single, unlikely, and
interrupted by user situation where starting a new connection with a
different CID is adventagous to performing Session Continuation by way
of Connection Reinstatement in both single connection
ErrorRecoveryLevel=2 and non ErrorRecoveryLevel=2 sessions, it
demonstrates our implementation method to be the simplest while still
enforcing negotiated SessionWide keys, and conforming to the iSCSI
specification.

I have to reinterate the same challenge above given the following
scenario:

Initiator: Performs continuous Session Continuation by way of Connection
Reinstatement.

Target: Connection Reinstatement occurs and then MaxConnections is
checked after SecurityNegotiation is complete, or upon reciept of a
Login Request containing CSG=1, NSG=[1,3]. The 'failed connection
reinstatement' state is enforced as there is no single reason to keep
CSM-C around when the Initiator has attempted Connection Reinstatement.

What other recovery situations invoke Session Continuation by starting a
new connection with a different CID as opposed to continous Connection
Reinstatement other than the single user invoked example mentioned above
that would cause a non-zero TSIH login attempt to be rejected due to
exceeding MaxConnections when Connection Reinstatement and
MaxConnections are checked in the order I have proposed?

Any takers? :-)

---

From: [email-masked]
Date: Wed, 14 Jan 2004 07:40:27 -0800 (PST)

> > get to FFP so that it can clean the old one up.

[snip]

> different CID when Connection Reinstatement is performed before

The "logical" reason for the initiator to choose CSM-E recovery is that
that is what the initiator author(s) chose to do. Just because it is not
your choice does not mean it won't be someone else's choice.

You say that doing things this way would permit always enforcing
MaxConnections. You are correct. However to make a difference in the
targets (to have a strong impact), targets have to be coded to ignore the
SHOULD in the spec concerning MaxConnections & error recovery. That's
changing the spec; I don't think your discussion above is an appropriate
technical reason for a target to ignore the spec.

> be enforced.

As above, enforcing MaxConnections like this breaks a SHOULD in the spec.
I do not think you have given a sufficient justification for a target to
get around the SHOULD. This discussion may have lead to the SHOULD's
removal had it been made earlier in the standards process, but for now we
have a SHOULD.

Take care,

Bill

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 14 Jan 2004 22:12:10 -0800

> your choice does not mean it won't be someone else's choice.

No, the "logical" reason is that no-one can name a single reason why
CSM-E would occur during normal operation in a single connection
ErrorRecoveryLevel=2 session in the first place.  Furthermore I have yet
to encounter an ErrorRecoveryLevel=2 Initiator that takes this approach,
or any other technical reason why needing to have two approaches other
than to give Initiator authors more 'choice' ensures interoperability.

If imposing a small restriction (how hard can it be to use a CID from a
failed connection as opposed to assigning a new one?) ensures
interopability between ErrorRecoveryLevel=2 implementations than it is a
small price to pay to avoid single connection ErrorRecoveryLevel=2
implementations from potentially falling back to Session Reinstatement
aka 'Degraded Mode' every time.

Considering the passage in section 3.2.1 that you continuously sight as
a reason to prevent a simpler and more effective method of ensuring
interoperability is a SHOULD, there is a very obvious interoptability
hole that needs to be closed.

> technical reason for a target to ignore the spec.

This SHOULD is the interoperability problem for ErrorRecoveryLevel=2
implementations I am trying to close.  Consider the two following
implementation scenraios operating in a multi-vendor enviroment given
the following methods of iSCSI vendor X and vendor Y:

iSCSI Target Vendor X: Enforces a MaxConnections check after Connection
Reinstatement occurs. 

iSCSI Target Vendor Y: Allows MaxConnections to be exceeded at any time.

Scenario 0:

iSCSI Initiator Vendor A: Performs Session Continuation by way of
continutions Connection Reinstatement (CSM-I).

Interoptiablity Issues:

iSCSI Target Vendor X:  None

iSCSI Target Vendor Y:  None.

Scenario 1:

iSCSI Initiator Vendor B: Performs Session Continuation by way of
creating a new connection with a different CID, and logging out the
failed connection with a explict logout request. (CSM-E).

Interoptiablity Issues:

iSCSI Target Vendor X: MaxConnections is enforced as section 3.2.1
defines a SHOULD instead of a MUST.  The login attempt is with rejected
with a Status Class/Detail of 0x0206.

iSCSI Target Vendor Y: None.

> removal had it been made earlier in the standards process, but for now we

It comes down to one of two options the working group has in order to
prevent the clear interoperability issue between ErrorRecoveryLevel=2
implementations described above.

0: Change the SHOULD to a MUST in section 3.2.1 and require all Targets 
to exceed MaxConnections for the sole purpose of creating more 'choice'
for Initiator Authors.  Additionally, not a _single_ technicial reason
has been raised that explains why CSM-E should ever happen during normal
operation in a single connection ErrorRecoveryLevel=2 session.

OR
 
1: Require ErrorRecoveryLevel=2 Initiator Implementations to perform
Connection Resinstatement (CSM-I) before attempting to start a new
connection with a different CID (CSM-E).  This allows targets to enforce
MaxConnections.

Additionally I believe it has been proven beyond a reasonable doubt that
the correct location to perform Connection Reinstatement on a Target is
after a succcessful SecurityNegotiation or upon reciept of a Login
Request containing CSG=1 and NSG=[1,3], and before MaxConnections is
checked.  I believe the 'reinstatement failed' state mentioned in the
previous replys is a clear indication of why this is the appropriate
place, and when used in conjunction with the second option mentioned
above allows for interoperable ErrorRecoveryLevel=2 implementations with
the desired side effect of enforcing the negotiated MaxConnections key.

Consensus Please?

---

From: [email-masked]
Date: Thu, 15 Jan 2004 14:13:58 -0800 (PST)

> than to give Initiator authors more 'choice' ensures interoperability.

With the "logic" you are showing above, no one will ever convince you of a
thing - you seem to have already convinced yourself.

> hole that needs to be closed.

From RFC 2119, 'SHOULD This word, or the adjective "RECOMMENDED", mean
that there may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.'

"I think it would be cleaner to never exceed MaxConnections ever" does not
strike me, personally, as a valid reason to violate this SHOULD. The fact
the text is there and survived the many-year development process indicates
that the WG didn't find it a concern. I also did not find it terribly
difficult to implement.

You speak of a better way of ensuring interoperability and a hole that
needs closing. The hole is not the spec, it is a target that chose to
ignore a SHOULD. That target should receive the blunt of concern regarding
interop issues; it made a decision against a recomendation in the spec.

Also, you are talking about ERL=2. What about ERL=0 or ERL=1? Here, CSM-E
is the best choice for recovery, as it gives you final status for the
tasks; you know which tasks were completed during the connection failures
and which got killed.

> with a Status Class/Detail of 0x0206.

Vendor X distrgarded a SHOULD. Problems may result.

And you still have session recovery, so you're not out of recovery
options. Thus the initiator and target can still interoperate, even if
it's not as nicely as you'd wish.

> connection with a different CID (CSM-E).  This allows targets to enforce

Actually, there is a third option. You have a configuration that contains
a target which is violating a SHOULD. You should ether accept that you may
have issues, and/or you should raise the issue with the target vendor.

The reason, as I understand it, that this section is a SHOULD and not a
MUST is so that iSCSI can support (for some value of support)
exceptionally simplistic targets. Say one that only only only supported
one connection and thus one session.

> the desired side effect of enforcing the negotiated MaxConnections key.

I doubt you'll get concensus. Concensus is everyone coming together to
work something out. The tone of your notes does not seem condusive to you
shifting position. In fact, the notes sound much more like you are trying
to bludgeon us into agreeing with you rather than actually discussing
things.

Take care,

Bill