Subject: iSCSI question

From: Shahram Davari <[email-masked]>
Date: Wed, 7 Aug 2002 07:17:06 -0700

Hi,

I have a question regarding the hierarchy of error recovery.
Section 6.13 mentions the hierarchy as:

2: Connection recovery
1: Digest failure recovery
0: Session recovery

And it states that the higher levels are a superset of the
lower levels and that the level of complexity increases from 0->1->2.

Couple of questions:

1) How is digest failure recovery done? by retransmission of PDUs?
2) Why is the connection recovery a superset of session recovery
and more complex?
3) It seems to me the order should be:

2: Session recovery
1: Connection recovery
0: Digest failure recovery

I appreciate any insight.

Thanks,
-Shahram

---

From: Shahram Davari <[email-masked]>
Date: Wed, 7 Aug 2002 08:09:31 -0700

Julian,
 
Thanks. I have read that section but it is not very clear.
 
I also agree that Connection recovery requires everything in command recovery.
But what about session recovery? isn't it a superset of both connection and command recovery?
 
Yours,
-Shahram

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:03 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: Re: iSCSI question

Sharam, 

You may want to go over the recovery chapter. 
It has detailed answers to all your questions. 
The superset/subset is based on functions you need for the next level. 

Session recovery drops real recovery to SCSI. 
Command recovery recovers from individual command errors without 
changing connection and the highest enable you to switch to a new connection and 
continue commands there. 

2 requires everything in 1. 

Julo 

	Shahram Davari <[email-masked]> 
Sent by: [email-masked] 

08/07/2002 05:17 PM 

        To:        [email-masked] 
        cc:         
        Subject:        iSCSI question 

Hi,

2: Connection recovery

Couple of questions:

2: Session recovery
1: Connection recovery

Thanks,
-Shahram

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 7 Aug 2002 18:33:58 +0300

Session recovery is in fact leaving all recovery to SCSI - it drops 
everything and creates a new session.
As for you comment on the clarity of chapter 5 at this stage it makes 
sense to be either specific
or keep this type of comment out of this context.

Julo

08/07/2002 06:09 PM

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     [email-masked], [email-masked]
        Subject:        RE: iSCSI question

Julian,

I also agree that Connection recovery requires everything in command 
recovery.
But what about session recovery? isn't it a superset of both connection 
and command recovery?
 
Yours,
-Shahram
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:03 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: Re: iSCSI question

Sharam, 

Session recovery drops real recovery to SCSI. 
Command recovery recovers from individual command errors without 
changing connection and the highest enable you to switch to a new 
connection and 
continue commands there. 

Julo 

        To:        [email-masked] 
        cc:         
        Subject:        iSCSI question 

Hi,

2: Connection recovery

Couple of questions:

2: Session recovery
1: Connection recovery

Thanks,
-Shahram

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 7 Aug 2002 18:03:08 +0300

Sharam,

Julo

        To:     [email-masked]
        cc: 
        Subject:        iSCSI question

Hi,

2: Connection recovery

Couple of questions:

2: Session recovery
1: Connection recovery

Thanks,
-Shahram

---

From: Shahram Davari <[email-masked]>
Date: Wed, 7 Aug 2002 08:43:55 -0700

Julian,
 
To start a new session you need to start new connections and you need to support
the PDU recovery. So how is that a subset of PDU and connection recovery?

-Shahram
 
(I will explain the detailed clarity issues in another email)

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:34 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Session recovery is in fact leaving all recovery to SCSI - it drops everything and creates a new session. 
As for you comment on the clarity of chapter 5 at this stage it makes sense to be either specific 
or keep this type of comment out of this context. 

Julo 

08/07/2002 06:09 PM 

Julian, 

  
Yours, 
-Shahram 
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:03 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: Re: iSCSI question

Sharam, 

Julo 

08/07/2002 05:17 PM 

       To:        [email-masked] 
       cc:         
       Subject:        iSCSI question 

Hi,

2: Connection recovery

Couple of questions:

2: Session recovery
1: Connection recovery

Thanks,
-Shahram

---

From: [email-masked]
Date: Thu, 8 Aug 2002 12:19:17 -0600

Shahram,
 
Wen you start a new session, you don't recover any PDUs. All the iSCSI state died with the old session. iSCSI doesn't know the new session had any relationship to the old session.
 
As Julian said, recovery at that point is up to the SCSI layer above iSCSI. It is up to SCSI to retry any commands that it wants to retry. When SCSI retries a command, iSCSI doesn't know it is a retry. To the iSCSI layer it is just like any other SCSI command it receives.
 
Pat
 
-----Original Message-----
From: Shahram Davari [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 8:44 AM
To: 'Julian Satran'
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Julian,

-Shahram

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:34 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Julo 

08/07/2002 06:09 PM 

Julian, 

  
Yours, 
-Shahram 
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:03 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: Re: iSCSI question

Sharam, 

Julo 

08/07/2002 05:17 PM 

       To:        [email-masked] 
       cc:         
       Subject:        iSCSI question 

Hi,

2: Connection recovery

Couple of questions:

2: Session recovery
1: Connection recovery

Thanks,
-Shahram

---

From: Shahram Davari <[email-masked]>
Date: Thu, 8 Aug 2002 13:40:59 -0700

Amir,
 
Which one requires more CPU processing?
 
-Shahram

-----Original Message-----
From: Amir Shalit [mailto:[email-masked]]
Sent: Thursday, August 08, 2002 4:39 PM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Shahram,
 
1) Connection reassignment requires more than PDU retransmission to handle corner cases
 
2) Error recovery level 1&2 requires more buffering which takes space and lowers performance. If
   you support error recovery level 1 you might as well support level 2.
 
Amir

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Shahram Davari
Sent: Thursday, August 08, 2002 11:27 AM
To: [email-masked]'; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Pat,
 
Thanks. I understand your point. Although terminating a session may be easy, but, starting a new session requires new login, parameter exchange, new connections establishment, authentication, etc. So I wonder how is this any simpler than a simple PDU retransmit?
 
Yours,
-Shahram

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, August 08, 2002 2:19 PM
To: Shahram Davari; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Shahram,

 
Pat
 
-----Original Message-----
From: Shahram Davari [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 8:44 AM
To: 'Julian Satran'
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Julian,

-Shahram

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:34 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI question

Julo 

08/07/2002 06:09 PM 

Julian, 

  
Yours, 
-Shahram 
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, August 07, 2002 11:03 AM
To: Shahram Davari
Cc: [email-masked]; [email-masked]
Subject: Re: iSCSI question

Sharam, 

Julo 

08/07/2002 05:17 PM 

       To:        [email-masked] 
       cc:         
       Subject:        iSCSI question 

Hi,

2: Connection recovery

Couple of questions:

2: Session recovery
1: Connection recovery

Thanks,
-Shahram

---

From: [email-masked]
Date: Thu, 8 Aug 2002 15:12:48 -0600

Hi Shahram,

There isn't any session "re-establishment" code. There is session establishment code that has to be there for one to do anything and the recovery code would have something to decide to close the session and and whether to try to start the session establishment code.

The question is how often will PDU recovery work. Look at various problems:

A) The path breaks and routing doesn't switch things to another path. The TCP connection goes down. Recovery requires session or connection recovery and iSCSI knows it can't do path recovery.

B) If there is a short lived problem in the path (noise burst, congestion drop, physical layer training, quick path change), then TCP will detect the drops and recover. The iSCSI layer won't see it.

C) If the iSCSI payload is damaged when it is not protected by CRC and the TCP checksum doesn't catch the error, then iSCSI would see a PDU loss (assuming either that digests are on or that the error caused an error in a header field that prevents processing). PDU recovery can work for this situation.

D) Another possibility is that there is a protocol state problem that has occurred at one end of the connection (could be due to an implementation bug or a soft memory error changing state information). PDU recovery wouldn't work here; depending on the exact problem it may require session recovery or connection recovery may work. However, the iSCSI initiator probably doesn't know whether it is in this case or case C so if it supports path recovery, it may try it.

The worth of PDU recovery depends partly on how often case C occurs. If 99% of the failures are A and B, 1% are C and 9% are D, then one may be better off to go straight to connection or session recovery when iSCSI detects a need to recover. It isn't worth extra code, cost, and complexity to recover faster in 1% of the cases.

Even if 5% of failure are due to C and 5% are due to D, it may not be desireable to attempt path recovery. Path recovery could recover the C failures quickly but it will delay recovery for the D failures because their resolution will wait for Path recovery to be tried and time out. It can be better to decrease the maximum delay for recovery rather than to speed some cases at the cost of delaying the slower recoveries

Regards,
Pat

-----Original Message-----
From: Shahram Davari [mailto:[email-masked]]
Sent: Thursday, August 08, 2002 1:21 PM
To: 'Bill Studenmund'
Cc: [email-masked]'; [email-masked]; [email-masked];
[email-masked]

Hi Bill,

Thanks for your reply. Please see my comments below:

1) I doubt that the session re-establishment code is simpler than PDU recovery code.
But think what you are saying is that, you need to have the session recovery anyway, so the PDU recovery is extra code.
2) Even if that is the case, it has nothing to do with the error recovery hierarchy.
The error recovery hierarchy must show what recovery should be tried before the other ones. In other words it has to show how the recovery escalates.
3) I think it should escalate as following: PDU -> Connection -> Session

Yours,
-Shahram  

> From: Bill Studenmund [mailto:[email-masked]]
> Sent: Thursday, August 08, 2002 3:59 PM
> To: Shahram Davari
> Cc: [email-masked]'; [email-masked]; 
> [email-masked];
> [email-masked]
> On Thu, 8 Aug 2002, Shahram Davari wrote:
> 
> > Pat,
> >
> 
> > Thanks. I understand your point. Although terminating a 
> session may be
> > easy, but, starting a new session requires new login, parameter
> > exchange, new connections establishment, authentication, etc. So I
> > wonder how is this any simpler than a simple PDU retransmit?
> 
> It's simpler in terms of the code in both the initiator and target.
> 
> That's how it's simpler. :-)
> 
> Take care,
> 
> Bill
>

---

From: Shahram Davari <[email-masked]>
Date: Thu, 8 Aug 2002 14:21:51 -0700

Paul,

> But that's not what "hierarchy" refers to here.
> 
> The hierarchy is one of increased capability, not increased
> desperation.  Session recovery is the minimum required; the additional
> levels are optional capabilities in addition to the minimum.  Each
> higher level in the hierarchy is a superset of the one below.

It all depends on the definition of these recovery classes.

1) If they are defined in a superset/subset fashion, then I agree
that level of complexity increases as: Session->PDU->connection.
Then I suggest changing texts in other parts of the draft such as
section 5.14 to indicate that if you have the capabilities of
class X, then you don't need to escalate to lower classes, because
class X already has those capabilities itself. Also I suggest
changing the hierarchy figure as following:

                             +
                            / \
                           / 2 \      
                          +-----+
                         /  1,2  \     
                        +---------+
                       /   0,1,2   \   
                      +-------------+

2) If they are defined as disjoint classes, then the hierarchy for
complexity makes no sense. Rather you need a hierarchy for escalation
or transition.

Based on the emails that I have received so far it seems that the intent is the former definition.

Yours,
-Shahram

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 9 Aug 2002 11:02:49 +0300

You have repaetedly said this and your position was respectfully noted.
Why do you bring this up again.  The list has somehow agreed that closed 
issues stay closed unless
you have new technical argument - and you don't bring here any.
Please refrain!

Julo
----- Forwarded by Julian Satran/Haifa/IBM on 08/09/2002 10:54 AM -----

"Somesh Gupta" <[email-masked]>
08/09/2002 03:43 AM

        To:     "Mark Bakke" <[email-masked]>, Julian Satran/Haifa/IBM@IBMIL
        cc:     "Shahram Davari" <[email-masked]>, <[email-masked]>, 
<[email-masked]>
        Subject:        RE: iSCSI question

Additionally (and with all due respect to the protocol
inventors), the recovery levels do not take into account
the problems most likely to be encountered.

As someone had previously pointed out, recovery from
failed connections is likely to be most used
(and should be disjoint from recovery within a
command due to digest errors which is probably
most useful for legacy tape). The person referred to
it as recovery level 0.5.

Even though the initiator has significant flexibility in
the model to use, the target is fairly hosed in this regard.

But time to close the spec.

Somesh

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Mark Bakke
> Sent: Wednesday, August 07, 2002 10:03 AM
> To: Julian Satran
> Cc: Shahram Davari; [email-masked]; [email-masked]
> Julian-
> Your comment is accurate, but I just want to make sure that the
> readers take "last resort" properly.  From a technical, iSCSI
> point of view, it's the last resort.
> 
> However, from a practical (implementation/deployment) point of
> view, session recovery is not all that drastic.  We have been
> shipping iSCSI drivers and targets that do session recovery for
> quite a while, and it works just quite well for disks (the OS
> SCSI layer handles any retries), and for tape applications that
> support re-positioning, which are becoming more common.  We
> have spent a significant amount of time testing this both in
> our lab and in the field.
> 
> <SOAPBOX>
> Once tape device and driver vendors support SSC-2, there should be
> no need to do more than session recovery, except perhaps for
> some performance advantages environments where a lot of connection
> recovery needs to take place.  But for most iSCSI environments,
> doing more than session recovery is not necessary.
> </SOAPBOX>
> 
> --
> Mark
> 
> Julian Satran wrote:
> > 
> > Session recovery means just creating a NEW session and 
> forgetting about all old commands.
> > It is the last resort recovery where everything else fails and 
> as such it is the most basic
> > function - that anybody has to have.
> > 
> > Julo
> > 
> >   08/07/2002 06:43 PM
>     [email-masked],
> > Session recovery is in fact leaving all recovery to SCSI - it 
> drops everything and creates a new
> > session.
> >                                                            To: 
>       Julian
> Satran/Haifa/IBM@IBMIL
>       [email-masked],
>        Subject:        iSCSI
>  question
> -- 
> Mark A. Bakke
> Cisco Systems
> [email-masked]
> 763.398.1054

---

From: Shahram Davari <[email-masked]>
Date: Fri, 9 Aug 2002 06:47:35 -0700

Pat,

Thanks for your excellent description. I now fully understand the
issue.

However, I disagree that the text is clear on the difference between
error recover classes and error recovery levels. As a proof, look at figure 2, in section 5.15:

  +-------------------+--------------------------------------------+
  |ErrorRecoveryLevel |  Associated Error recovery capabilities    |
  +-------------------+--------------------------------------------+
  |        0          |  Session recovery class                    |
  |                   |  (Section 5.14.4 Session Recovery)         |
  +-------------------+--------------------------------------------+
  |        1          |  Digest failure recovery (See Note below.) |
  +-------------------+--------------------------------------------+
  |        2          |  Connection recovery class                 |
  |                   |  (Section 5.14.3 Connection Recovery)      | 
  +-------------------+--------------------------------------------+

In which, each recovery level is associated with only one recovery class.

Yours,
-Shahram

> Sent: Thursday, August 08, 2002 5:58 PM
> To: Shahram Davari; [email-masked]
> Cc: [email-masked]
> There are two recovery heirarchies. You seem to be confusing 
> error recovery level with error recovery class.
> 
> One is the heirarchy for implementation requirements. With 3 
> forms of recovery, there are 8 possible combinations but to 
> simplify interoperability and to specify minimum acceptable 
> operation, we don't want to allow all eight. Session 
> qualifies two ways to be level 0 of this heirarchy - it is 
> simplest to implement and it should be able to recover from 
> any recoverable error. 
> 
> Note that these are the error recovery levels that are 
> described in 5.15. A sessions recovery level indicates the 
> set of recovery classes it is capable of using. The recovery 
> levels are defined such that a lower recovery level includes 
> a subset of the recovery classes available at a higher recovery level.
> 
> The other heirarchy is the order in which the available 
> classes are applied once an error occurs. This is the subject 
> of 5.14. The classes are disjoint. 
> 
> Once an error occurs, the device will choose an error 
> recovery class from the set of recovery classes in its 
> recovery level. If that recovery class fails, it may try the 
> next higher class.
> 
> Error recovery level 0 includes the Session recovery class.
> Error recovery level 1 includes the two classes useful for  
> Digest failure recovery (Within-Command and 
> Within-Connection) plus the Session recovery class from level 0.
> Error recovery level 2 includes  Connection recovery class 
> plus the classes included in level 1. 
> 
> I think the text already makes this clear and the pyramid 
> correctly represents this, but perhaps the chart below the 
> period could add "plus the capablilities of 
> ErrorRecoveryLevel n" to the boxes for Associated Error 
> recovery capablities for levels 1 and 2 ("n" = 0 and 1, respectively).
> 
> I hope this helps remove some of the confusion.
> 
> Pat
> 
> Sent: Thursday, August 08, 2002 2:22 PM
> To: 'Paul Koning'
> Cc: [email-masked]

---

From: [email-masked]
Date: Fri, 9 Aug 2002 11:10:06 -0600

Shaharam,

I agree that the table by itself is not clear, but that the text of the section clearly states things. That is why I suggested a change to that table:

  +-------------------+--------------------------------------------+
  |ErrorRecoveryLevel |  Associated Error recovery capabilities    |
  +-------------------+--------------------------------------------+
  |        0          |  Session recovery class                    |
  |                   |  (Section 5.14.4 Session Recovery)         |
  +-------------------+--------------------------------------------+
  |        1          |  Digest failure recovery (See Note below.) |
  |                   |  plus all ErrorRecoverLevel 0 capabilities |  
  +-------------------+--------------------------------------------+
  |        2          |  Connection recovery class                 |
  |                   |  (Section 5.14.3 Connection Recovery)      | 
  |                   |  plus all ErrorRecoverLevel 1 capabilities |
  +-------------------+--------------------------------------------+

Pat

-----Original Message-----
From: Shahram Davari [mailto:[email-masked]]
Sent: Friday, August 09, 2002 6:48 AM
To: [email-masked]'; [email-masked]
Cc: [email-masked]
Subject: RE: iSCSI question

Pat,

Yours,
-Shahram

---

From: Shahram Davari <[email-masked]>
Date: Fri, 9 Aug 2002 10:27:20 -0700

Pat,

Seems alright.

-Shahram

> Sent: Friday, August 09, 2002 1:10 PM
> To: Shahram Davari; [email-masked]; [email-masked]
> Cc: [email-masked]
> I agree that the table by itself is not clear, but that the 
> text of the section clearly states things. That is why I 
> suggested a change to that table:
>

---

From: "Julian Satran" <[email-masked]>
Date: Sat, 10 Aug 2002 00:04:11 +0300

it is alredy in - julo

[email-masked]
08/09/2002 08:10 PM

        To:     [email-masked], [email-masked], 
[email-masked]
        cc:     [email-masked]
        Subject:        RE: iSCSI question

Shaharam,

I agree that the table by itself is not clear, but that the text of the 
section clearly states things. That is why I suggested a change to that 
table:

Pat

-----Original Message-----
From: Shahram Davari [mailto:[email-masked]]
Sent: Friday, August 09, 2002 6:48 AM
To: [email-masked]'; [email-masked]
Cc: [email-masked]
Subject: RE: iSCSI question

Pat,

However, I disagree that the text is clear on the difference between
error recover classes and error recovery levels. As a proof, look at 
figure 2, in section 5.15:

Yours,
-Shahram