Subject: Re: CSM [Ips] iSCSI: ErrorRecoveryLevel=2 Interoperability Issue

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 15 Jan 2004 20:22:23 -0800

On Thur, 15 Jan 2005, [email-masked] wrote:

> On Wed, 14 Jan 2004, Nicholas A. Bellinger wrote:
> > On Wed, 2004-01-14 at 07:40, [email-masked] wrote:
> > >
> > >
> > > The "logical" reason for the initiator to choose CSM-E recovery is that
> > > that is what the initiator author(s) chose to do. Just because it is not
> > > your choice does not mean it won't be someone else's choice.
> >
> > No, the "logical" reason is that no-one can name a single reason why
> > CSM-E would occur during normal operation in a single connection
> > ErrorRecoveryLevel=2 session in the first place.  Furthermore I have yet
> > to encounter an ErrorRecoveryLevel=2 Initiator that takes this approach,
> > or any other technical reason why needing to have two approaches other
> > than to give Initiator authors more 'choice' ensures interoperability.

> With the "logic" you are showing above, no one will ever convince you of a
> thing - you seem to have already convinced yourself.

I can be convinced by a solid technical reason. I have yet to see a single
response other than 'more choice for Initiator Authors' to any of the
previous challenges regarding this issue.

> > If imposing a small restriction (how hard can it be to use a CID from a
> > failed connection as opposed to assigning a new one?) ensures
> > interopability between ErrorRecoveryLevel=2 implementations than it is a
> > small price to pay to avoid single connection ErrorRecoveryLevel=2
> > implementations from potentially falling back to Session Reinstatement
> > aka 'Degraded Mode' every time.
> >
> > Considering the passage in section 3.2.1 that you continuously sight as
> > a reason to prevent a simpler and more effective method of ensuring
> > interoperability is a SHOULD, there is a very obvious interoptability
> > hole that needs to be closed.

> "I think it would be cleaner to never exceed MaxConnections ever" does
>not strike me, personally, as a valid reason to violate this SHOULD.
> The fact the text is there and survived the many-year development
> process indicates that the WG didn't find it a concern. I also did not
> find it terribly difficult to implement.
>
> You speak of a better way of ensuring interoperability and a hole that
> needs closing. The hole is not the spec, it is a target that chose to
> ignore a SHOULD. That target should receive the blunt of concern regarding
> interop issues; it made a decision against a recomendation in the spec.

This has nothing to do with difficultly of implementing.  As I have stated
in previous emails, there is a situation where by a user involked action in
single connection ErrorRecoveryLevel=2 where temporarly exceeding
MaxConnections should be allowed when the IP Address and Communication Port
of CSM-I is no longer active, and hence preventing Session Reinstatement from
having to occur. This is the only situation where exceeding MaxConnections is
useful and is a situation we currently support in our Initiator.

If a SHOULD causes ErrorRecoveryLevel=2 implementations to fall back to
Session Reinstatement every time, then there is a problem.  The SHOULD needs to
be changed to a MUST.   

> Also, you are talking about ERL=2. What about ERL=0 or ERL=1? Here, CSM-E
> is the best choice for recovery, as it gives you final status for the
> tasks; you know which tasks were completed during the connection failures
> and which got killed.

I have already covered all non ERL=2 situations in previous emails.
We can agree that the one half of CSM-E, an explict logout by way of a Logout
Request with ReasonCode=2 is strictly an ErrorRecoveryLevel=2 feature.  Lets
focus on the other half of CSM-E, a Logout Request with ReasonCode 1.

Saying that an Explict Logout with ReasonCode 1 in CSM-E is the 'best choice'
for recovery because it contains the ExpStatSN of the Connection to be closed so
it can be determined which 'tasks were completed during the connection failures
and which got killed' is a incorrect.  An implict Logout by way of CSM-I
contains the very same ExpStatSN value. (See section 10.12.9)

If this was an incorrect assumption as to why you deam CSM-E as the 'best choice'
for non ERL=2 implementations to perform Session Continuation, I am waiting to be
corrected. 

> > > You say that doing things this way would permit always enforcing
> > > MaxConnections. You are correct. However to make a difference in the
> > > targets (to have a strong impact), targets have to be coded to ignore the
> > > SHOULD in the spec concerning MaxConnections & error recovery. That's
> > > changing the spec; I don't think your discussion above is an appropriate
> > > technical reason for a target to ignore the spec.
> > >
> >
> > This SHOULD is the interoperability problem for ErrorRecoveryLevel=2
> > implementations I am trying to close.  Consider the two following
> > implementation scenraios operating in a multi-vendor enviroment given
> > the following methods of iSCSI vendor X and vendor Y:
> >
> > iSCSI Target Vendor X: Enforces a MaxConnections check after Connection
> > Reinstatement occurs.
> >
> > iSCSI Target Vendor Y: Allows MaxConnections to be exceeded at any time.
> >
> > Scenario 0:
> >
> > iSCSI Initiator Vendor A: Performs Session Continuation by way of
> > continutions Connection Reinstatement (CSM-I).
> >
> > Interoptiablity Issues:
> >
> > iSCSI Target Vendor X:  None
> >
> > iSCSI Target Vendor Y:  None.
> >
> > Scenario 1:
> >
> > iSCSI Initiator Vendor B: Performs Session Continuation by way of
> > creating a new connection with a different CID, and logging out the
> > failed connection with a explict logout request. (CSM-E).
> >
> > Interoptiablity Issues:
> >
> > iSCSI Target Vendor X: MaxConnections is enforced as section 3.2.1
> > defines a SHOULD instead of a MUST.  The login attempt is with rejected
> > with a Status Class/Detail of 0x0206.
>
> Vendor X distrgarded a SHOULD. Problems may result.

Again, this SHOULD needs to be a MUST to prevent a interoperability issue

> And you still have session recovery, so you're not out of recovery
> options. Thus the initiator and target can still interoperate, even if
> it's not as nicely as you'd wish.

This is obvious, as quoted in the 2nd paragraph of my reply:

"from potentially falling back to Session Reinstatement aka
 'Degraded Mode' every time."

> > iSCSI Target Vendor Y: None.
> >
> > It comes down to one of two options the working group has in order to
> > prevent the clear interoperability issue between ErrorRecoveryLevel=2
> > implementations described above.
> >
> > 0: Change the SHOULD to a MUST in section 3.2.1 and require all Targets
> > to exceed MaxConnections for the sole purpose of creating more 'choice'
> > for Initiator Authors.  Additionally, not a _single_ technicial reason
> > has been raised that explains why CSM-E should ever happen during normal
> > operation in a single connection ErrorRecoveryLevel=2 session.
> > 
> > OR
> >
> > 1: Require ErrorRecoveryLevel=2 Initiator Implementations to perform
> > Connection Resinstatement (CSM-I) before attempting to start a new
> > connection with a different CID (CSM-E).  This allows targets to enforce
> > MaxConnections.

> Actually, there is a third option. You have a configuration that contains
> a target which is violating a SHOULD. You should ether accept that you may
> have issues, and/or you should raise the issue with the target vendor.
>

I have no issues whatsoever.  It is the oppososing point of view here that
seems to be encourging a interoperability hole caused by a vauge SHOULD to
be left open.

> The reason, as I understand it, that this section is a SHOULD and not a
> MUST is so that iSCSI can support (for some value of support)
> exceptionally simplistic targets. Say one that only only only supported
> one connection and thus one session.

Section 3.2.1:

"targets and initiators that support a single active connection in a
 session SHOULD support two connections during recovery."

The key word being 'during recovery'. Well what type of recovery could
this mean?

Session Recovery: This is the most basic form of recovery that
'exceptionally simplistic' implementations would support.  This form
has no bearing on extra connection(s) needing to be started..

Leaving the 'exceptionally simplistic' field we have Session
Continuation:

Connection Reinstatement (both ERL=2 and non ERL=2): During normal
operation, the SHOULD does not apply.

Explict Logout with ReasonCode 1 in non ERL=2: During normal operation
the SHOULD applies.

Explict Logout with ReasonCode 1 in ERL=2: Not used in the context
of recovery, see ReasonCode 2.

Explict Logout with ReasonCode 2 in non ERL=2: Illegal, see Logout
Response ResponseCode 2.

Explict Logout with ReasonCode 2 in ERL=2: During normal operation the
SHOULD applies in a Single Connection Session Only.

Considering 'exceptionally simplistic' really only means Session
Recovery, I see the above interpretation as being incorrect. Under the
assumption that a implementation that supports more than Session Recovery
(ie: Session Continuation) is beyond the realm of simplistic, it works
out to:

"targets and initiators that support a single active connection in a
 session and support session continuation (see Section 5.3.6), MUST support
 two connections during continuation."

This means that _only_ when MaxConnections=1 (as with the original passage)
is it legal to exceed MaxConnections to 2 when Session Continuation is
supported on the Target.  Exceeding MaxConnections when MaxConnections!=1 is
clearly a violation of the above as well as orignal passage.

> > Additionally I believe it has been proven beyond a reasonable doubt that
> > the correct location to perform Connection Reinstatement on a Target is
> > after a succcessful SecurityNegotiation or upon reciept of a Login
> > Request containing CSG=1 and NSG=[1,3], and before MaxConnections is
> > checked.  I believe the 'reinstatement failed' state mentioned in the
> > previous replys is a clear indication of why this is the appropriate
> > place, and when used in conjunction with the second option mentioned
> > above allows for interoperable ErrorRecoveryLevel=2 implementations with
> > the desired side effect of enforcing the negotiated MaxConnections key.
> >
> > Consensus Please?
>
> I doubt you'll get concensus. Concensus is everyone coming together to
> work something out. The tone of your notes does not seem condusive to you
> shifting position. In fact, the notes sound much more like you are trying
> to bludgeon us into agreeing with you rather than actually discussing
> things.

If bludgeoning involves making solid technical point after technical point
including unearthing an interoperability issue, then I am guilty. (I would be
more than happy to list them all again if need be :-)

This is not about who is right or wrong, but about closing an issue that
exists because of a SHOULD.  My only concern is with fostering an enviroment
where ErrorRecoveryLevel=2 implementations run at ErrorRecoveryLevel=2, and
not in some type of degraded mode.  The latter is a very real possibly if the
vauge SHOULD in the final sentance of section 3.2.1 is not changed to a clear
MUST.

Please, comments from someone besides Bill and I? Mallikarjun?

-- 
Nicholas A. Bellinger <[email-masked]>

P.S: Or we can simply follow the 2nd option made in my previous reply that
makes the this whole debate invalid in all but a single user involked
case. :-))

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 16 Jan 2004 16:53:07 -0800

> Please, comments from someone besides Bill and I? Mallikarjun?

I don't claim to have followed this thread very closely,
nor I may be able to.  With that, a couple of quick comments.

- A target normally supporting only one connection, in
   order to cater to a connection recovery attempt via
   implicit logout from an initiator seeing original connection
   to have failed on its end, has to implement most of the
   functionality required to support the same via an explicit
   logout with a new CID.  IOW, target "almost" has to
   support two connections - if it promises to concurrently
   handle two active CSMs at somepoint (which is what
   connection recovery is).

- Allowing an explicit logout approach from the initiator
   has the additional benefit that an initiator can choose
   to use a different Logout reason code from that implied
   based on operational ErrorRecoveryLevel (look at the
   table at the end of 10.14).  E.g., despite an operationl
   ErrorRecoveryLevel=2, an initiator may simply close the
   failed connection (i.e. Reason code=1) via an explicit
   logout (i.e. CSM-E way).

- OTOH, the above flexibility wasn't deemed critical
   to mandate multi-connection sessions for the first time
   in the spec - hence the SHOULD.

- When the original text was written, the idea is that a
   "connection reinstatement" is a _successful_ act of
   replacing a connection with a new one where the new one
   inherits all the open tasks.  So, I'm at a loss to
   follow what you mean by "doing connection reinstatement
   after SecurityNegotiation stage" - reinstatement can't
   be "done" without a successful FFP connection.  I see
   no reason to change the original notion.

The net is that I do not see an interoperability problem,
nor anything that needs changing in the spec.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 16 Jan 2004 20:15:25 -0800

>    handle two active CSMs at somepoint (which is what

No disagreement here.  The main topic for up for disscussion
was when it is legal to exceed MaxConnections.  The last
sentance of section 3.2.1 seems to imply that this is only
permitted when MaxConnections=1, and be bumped up to
MaxConnections=2 for Session Continuation purposes.  If this
is a correct assumption, then I think changing a vauge 'in error
recovery.' SHOULD into a clear MUST along the following
lines does not break simple implementations that only support
Session Recovery:

"targets and initiators that support a single active connection
 in a session and support session continuation (see Section 5.3.6),
 MUST support two connections during continuation."

This seems to be along the lines of what you mention above,
and has the desired side effect of closing a interoperabililty
issue on Targets that ignore a SHOULD, support Session Continuation,
but do not support exceeding MaxConnections in any ErrorRecovery.
The undesired outcome (espically in ErrorRecoveryLevel=2), is
Initiators who perfer CSM-E to CSM-I having to falling back to
Session Recovery every time.

Also, there is the first paragraph of section 5.3.4:

"In sessions with a single connection, this may imply the opening of
 a second connection with the sole purpose of cleaning up the first.
 Targets MUST support opening a second connection even when they do 
 not support multiple connections in Full Feature Phase if
 ErrorRecoveryLevel is 2 and SHOULD support opening a second connection
 if ErrorRecoveryLevel is less than 2."

This is more along of lines of what I think is correct, but
seems to be at odds with the SHOULD from section 3.2.1.  Also
since its only mentioned in the context of Connection Reinstatement
it many not fall under the guise of CSM-E to the reader.

>    failed connection (i.e. Reason code=1) via an explicit

I assume that a Logout Request with ReasonCode=2 must still
follow an Logout Request with ReasonCode=1 if any active
commands acknowledged by ExpCmdSN are intended to be reassigned
to another opreational connection? I also assume its legal to only
send a ReasonCode=1 if the Initiator intends to perform Session
Continuation and the failed connection only contains commands greater
than or equal to ExpCmdSN that are going to be retried, or no commands
at all?  If not, could you please give an example otherwise?
 
>    in the spec - hence the SHOULD.

Please see above.

>    no reason to change the original notion.

By "doing connection reinstatement after SecurityNegotiation stage",
I mean checking for a active CID matching the implict logout
request's CID, and shutting down the CID active from the target's
prespective (CSM-C) at this point.  The advantage is that when CSM-C is
shutdown upon completion of SecurityNegotiation or upon reciept of a
Login Request containing CSM=1 and NSG=[1,3], yet before MaxConnections
is checked, MaxConnections can be enforced.

This opposed to performing the shutdown of the active CID before sending
the last Login Request before moving to FFP, and hence exceeding
MaxConnections in order to allow the implict logout to continue.

I give the 'reinstatement failed' state previously mentioned as
reasoning for the former case.  That logic dictates CSM-C must be failed
when CSM-I makes it past the point of SecurityNegotition, but not
necessarily into FFP.  I believe this state shows why it is correct to
shutdown CSM-C after completion of SecurityNegotiation and any 
MaxConnections check.

Additionaly, when this implementation method is referenced with the
above noted SHOULD and MUST from section 5.3.4 it makes both not apply,
or at least not in the context of Connection Reinstatement.  Of course
that is an implementation specific issue. :-)

> nor anything that needs changing in the spec.

The ErrorRecoveryMan has spoken. :-)

Thanks!

---

From: "Ken Sandars" <[email-masked]>
Date: Tue, 20 Jan 2004 16:19:58 -0000

Hi Mallikarjun,

I'm seeking a clarification on the effects of connection 
reinstatement at ERL 2 via CSM-I:

>    inherits all the open tasks.

Do the open tasks on the CSM-C connection go into the
non-allegiant state, or do they get treated as though each
received an implicit TMF-"task reassign" to the new connection?

Thanks,
Ken Sandars
Elipsan UK

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 20 Jan 2004 10:18:42 -0800

Ken,

Thank you for pointing out a mis-statement, I meant
the inheritance of the CID, not the open tasks.

The open tasks can only be reassigned via explicit
task reassigns (or be allowed to time out).  BTW, this
is also true for the explicit logout (CSM-E).

Thanks.

Mallikarjun

---

From: Julian Satran <[email-masked]>
Date: Tue, 20 Jan 2004 16:54:22 -0800

Reassignment has to be explict - Julo

"Ken Sandars" <[email-masked]> 
Sent by: [email-masked]
20/01/2004 08:19

To
"'Mallikarjun C.'" <[email-masked]>, <[email-masked]>
cc

Subject
RE: CSM [Ips] iSCSI: ErrorRecoveryLevel=2 Interoperability Issue

Hi Mallikarjun,

Thanks,
Ken Sandars
Elipsan UK