Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

From: [email-masked]
Date: Thu, 5 Sep 2002 02:50:51 -0400

This thread's taken a while to sort through, but there
does appear to be a problem in here.

> If the missing command was non-immediate, then yes, the plug-in avoids
> having to resend the missing command.  But, if the missing command was
> immediate, then there is no hole to plug.  In this case, the spec requires
> the target to plug a hole which does not exist, which is the problem that
I
> am trying to point out.

I'm working off of -15.  Section 2.2.2.1 contains the following statements:

	CmdSN always contains the number to be 
      assigned to the next Command PDU.

   Commands meant for immediate delivery are marked with an immediate 
   delivery flag; they MUST also carry the current CmdSN.

So suppose CmdSN is 7, and the initiator sends an immediate command
with CmdSN 7 and then decides to abort it, and sends TASK ABORT as
an immediate command (CmdSN is 7 and not advanced).  Suppose the TASK
ABORT crosses the response to successful completion of the immediate
command on the wire.  At this point the following text in Section
9.6.1 applies:

     b)  if the Referenced Task Tag does not identify an existing task 
     but if the CmdSN indicated by the RefCmdSN field in the Task Man-
      agement function request is within the valid CmdSN window (between 
      MaxCmdSN and ExpCmdSN), targets must consider the CmdSN received 
      and return the "Function complete" response.

7 is still the CmdSN of the next non-immediate command to be sent,
so it is within the window (ExpCmdSN is also 7).  Following these
directions, the Target considers CmdSN 7 to be received, and advances
its window.  Now, when the initiator sends its next non-immediate
command (CmdSN=7), its CmdSN is outside the window, and the target
bit-buckets the command instead of executing it.

This went awry because the task to be aborted was (implicitly)
identified by its CmdSN, and not its Task Tag resulting in an attempt
to abort an immediate command actually clobbering the next non-
immediate one.  I think Tony gets credit for finding another problem.

While I'm in here, it also appears that the text describing the
mapping of iSCSI task management response codes to SCSI service
responses needs to change to map 1 (Task does not exist) to
FUNCTION COMPLETE rather than FUNCTION REJECTED to line up with
Section 6.2 of SAM-2.

Thanks,
--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 249-6449            FAX: +1 (508) 497-8018
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 5 Sep 2002 08:46:31 -0400

Can we say that aborts with the immediate bit only abort commands with the
immediate bit? And then say that the window is not advanced under that
situation?

Eddy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 2:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

I

Thanks,
--David

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 5 Sep 2002 16:22:36 +0300

No need - it works always as it is. An immediate is aborted based on ITT 
or not at all.
Add to it the fact that they are on the same connection and immediates are 
not reassigned
and everything works AS IS.
What am I missing?

Julo

Eddy Quicksall <[email-masked]>
05/09/02 16:02

        To:     Julian Satran/Haifa/IBM@IBMIL, Eddy Quicksall <[email-masked]>
        cc:     "[email-masked]'" <[email-masked]>, [email-masked], 
[email-masked], [email-masked]
        Subject:        RE: iSCSI: aborting an immediate command with ABORT TASK

How about another bit in the abort that says it is aborting an immediate 
command and hence does not advance the window?
 
Eddy
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 9:00 AM
To: Eddy Quicksall
Cc: [email-masked]'; [email-masked]; [email-masked]; 
[email-masked]

I don't think we want to. The TM made immediate are done so with good 
reason - to allow expedited consideration. 
And there are no known scenarios (to us) in which it does not work. 

Julo 

Eddy Quicksall <[email-masked]> 
Sent by: [email-masked] 
05/09/02 15:46 
        
        To:        "[email-masked]'" <[email-masked]>, 
[email-masked] 
        cc:        [email-masked] 
        Subject:        RE: iSCSI: aborting an immediate command with 
ABORT TASK 

Eddy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 2:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

I
> am trying to point out.

I'm working off of -15.  Section 2.2.2.1 contains the following 
statements:

Thanks,
--David

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 5 Sep 2002 16:00:20 +0300

Julo

        To:     "[email-masked]'" <[email-masked]>, [email-masked]
        cc:     [email-masked]
        Subject:        RE: iSCSI: aborting an immediate command with ABORT TASK

Eddy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 2:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

I

Thanks,
--David

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 5 Sep 2002 09:20:34 -0400

If the Abort Task had its RefCmdSN == 7, then wouldn't this cause the
problem David and Tony point out (considering everything David said)?
 
Eddy

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 8:27 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

The next command can't be clobered. Abort Task acts on command  up to 6
(included) but not 7- and it has to be on the same 
connection as the aborted command. That insures that it will abort the first
immediate but not the non-immediate after. 

As for the mapping - you are right - I'll mark it for the final edit
(whenever that happens). 

Julo 

	[email-masked] 

05/09/02 09:50 

        To:        [email-masked] 
        cc:        [email-masked] 
        Subject:        RE: iSCSI: aborting an immediate command with ABORT
TASK 

I

Thanks,
--David

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 5 Sep 2002 09:38:25 -0400

I don't see that fact when I read the spec. 9.5.5 RefCmdSN implies that the
RefCmdSN may be used if the Referenced Task Tag (ITT) is not with the target
anymore. And, I think that is David and Tony's point.
 
I may have missed it someplace ... can you please point out where it says or
implies "an immediate is aborted based on ITT"?
 
Eddy

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 9:23 AM
To: Eddy Quicksall
Cc: [email-masked]'; Eddy Quicksall; [email-masked];
[email-masked]; [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

No need - it works always as it is. An immediate is aborted based on ITT or
not at all. 
Add to it the fact that they are on the same connection and immediates are
not reassigned 
and everything works AS IS. 
What am I missing? 

Julo 

05/09/02 16:02 

        To:        Julian Satran/Haifa/IBM@IBMIL, Eddy Quicksall
<[email-masked]> 
        cc:        "[email-masked]'" <[email-masked]>,
[email-masked], [email-masked], [email-masked] 
        Subject:        RE: iSCSI: aborting an immediate command with ABORT
TASK 

Eddy 
-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 9:00 AM
To: Eddy Quicksall
Cc: [email-masked]'; [email-masked]; [email-masked];
[email-masked]

I don't think we want to. The TM made immediate are done so with good reason
- to allow expedited consideration. 
And there are no known scenarios (to us) in which it does not work. 

Julo 

05/09/02 15:46 

Eddy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 2:51 AM
To: [email-masked]
Cc: [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

I

Thanks,
--David

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 5 Sep 2002 16:55:51 +0300

----- Forwarded by Julian Satran/Haifa/IBM on 05/09/02 16:54 -----

Julian Satran
05/09/02 16:53

        To:     <[email-masked]>
        cc: 
        From:   Julian Satran/Haifa/IBM@IBMIL
        Subject:        RE: iSCSI: aborting an immediate command with ABORT TASK

Again no - there was no hole filed and the second 7 gets through. Julo 

"Tony Battersby" <[email-masked]>
05/09/02 16:23
Please respond to tonyb

        To:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
        cc:     <[email-masked]>
        Subject:        RE: iSCSI: aborting an immediate command with ABORT TASK

> (included) but not 7- and it has to be on the same connection as the
> aborted command. That insures that it will abort the first immediate but
> not the non-immediate after.

Ok, here's the gotcha:

The initiator sends a non-immediate command with CmdSN 7 on a different
connection before sending the immediate ABORT TASK for the first immediate
command.  The ABORT TASK is sent on the same connection as the immediate
command with CmdSN 7 to be aborted, but a different connection than the
non-immediate command with CmdSN 7.  The ABORT TASK will carry a CmdSN of 
8,
and the target may receive the ABORT TASK before it receives the
non-immediate command with CmdSN 7, because they are sent on different
connections.  If this happens, the target considers CmdSN 7 received and
then discards the non-immediate command with CmdSN 7 when it finishes
receiving it on the other connection.

Tricky, tricky, tricky...

Tony

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 5 Sep 2002 10:51:27 -0400

Paul,

Its a race having to do with sending the command response and receiving the
abort ... not between getting the command and the abort request.

I-T> immediate cmdSN=7
T-I> some delay but finally command response for above w/ExpCmdSN=7
(a race here due to processing at the initiator)
I-T> before getting the response, sends abort for 7
(target advances ExpCmdSN to 8)
I-T> non-immediate CmdSN=7
(outside of the command window, target dumps the command)

The directions say to consider the command received (that would mean the
non-immediate command) and that would be command 7. So you would advance
ExpCmdSN to 8. When non-immediate command 7 arrives, it will be outside of
the window and will be thrown out.

This is all considering ErrorRecoveryLevel 0.

Maybe I have missed something so please point out where I'm wrong.

Eddy

-----Original Message-----
From: Paul Koning [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 9:49 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

>>>>> "Julian" == Julian Satran <[email-masked]> writes:

 Julian> The next command can't be clobered. Abort Task acts on
 Julian> command up to 6 (included) but not 7- and it has to be on the
 Julian> same connection as the aborted command. That insures that it
 Julian> will abort the first immediate but not the non-immediate
 Julian> after.

If the abort has to be sent on the same connection as the command
being aborted, then it cannot overtake an immediate command preceding
it.  At least not in the network stack...  So that seems to eliminate
the case that David was worrying about. 

    paul

---

From: "Tony Battersby" <[email-masked]>
Date: Thu, 5 Sep 2002 10:31:57 -0400

Let me break it down:

From the initiator's perspective:
1) Send an immediate command with ITT = 22 and CmdSN = 7 on connection A
2) Send a non-immediate command with CmdSN = 7 on connection B
3) Send an immediate ABORT TASK command with Referenced Task Tag = 22 and
RefCmdSN = 7 and CmdSN = 8

From the target's perspective:
4) ExpCmdSN is 7
5) Receive the immediate command with ITT = 22 and CmdSN = 7 on connection A
6) Send the response to the immediate command from step 5 and free its
resources
7) Receive the immediate ABORT TASK command with Referenced Task Tag = 22
and RefCmdSN = 7 and CmdSN = 8 on connection A
8) Since no task exists with ITT = 22, and RefCmdSN is within the valid
CmdSN window, and RefCmdSN is less than the CmdSN of the ABORT TASK command,
consider CmdSN 7 received and set ExpCmdSN to 8
9) Receive the non-immediate command with CmdSN = 7 on connection B and
silently discard it because it is outside the valid CmdSN window.

In summary:
10) The end result is that the target aborted the wrong command.

Specifically, which one of these steps can't happen, and why?  AFAIK, this
_can_ happen in my current target iSCSI implementation.  If I am missing
something, then my implementation may be broken and I would like to fix it;
if I am not missing something, then there may be a real but fixable problem
with the way ABORT TASK works.

Tony

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 9:56 AM
To: [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

Ok, here's the gotcha:

non-immediate command with CmdSN 7.  The ABORT TASK will carry a CmdSN of 8,

Tony

---

From: Robert Snively <[email-masked]>
Date: Thu, 5 Sep 2002 08:38:29 -0700

This has become a remarkably complex structure.
FCP took a somewhat simpler approach to the TASK ABORT
synchronization problem.

In FCP, the principal was that the ABORT TASK chase
the targeted command through the chain of host adapter,
transport media, target/logical unit, and back again, clearing
up whatever residues were encountered along the way.

That way, ordering was never an issue, because the command
was either completed before an ABORT TASK ever started, or
the ABORT TASK tracked it down and aborted the artifacts
associated with the command wherever they were found.  The
inclusion of the HBA in the chain eliminated any possibility
of the command escaping after the ABORT TASK was transmitted.
It also eliminated ordering issues associated with transport
paths, since any residue left in the transport path was
deprived of context and discarded if it popped out later.

Bob

> From: Eddy Quicksall [mailto:[email-masked]]
> Sent: Thursday, September 05, 2002 7:51 AM
> To: 'Paul Koning'

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 5 Sep 2002 13:52:38 -0400

In SCSI host adapters this race also exists ... the way that is handled is
the host adapter does not see the task and just responds with "task not
fund". The driver doesn't have a problem because the only way the task could
not be found is if it already completed.
 
iSCSI has introduced this problem by 9.6.1 saying that if the task does not
exist, it is considered to have existed ("Function complete").
 
I think we should just say "task not found". The initiator then sees the
whole story and can work it out.
 
However, the condition that Tony points out may be an additional case that I
am not addressing.
 
Eddy

-----Original Message-----
From: Robert Snively [mailto:[email-masked]]
Sent: Thursday, September 05, 2002 11:38 AM
To: [email-masked]
Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

Bob 

> From: Eddy Quicksall [ mailto:[email-masked]
<mailto:[email-masked]> ] 
> From: Paul Koning [ mailto:[email-masked] <mailto:[email-masked]> ]

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 5 Sep 2002 17:41:50 +0300

I think that whole confusion is the result of some missing words.
The missing words are the that you cannot have RefCmdSN reffer to something
equal or higher than the ABORT TASKS CmdSN.
It is implied in the section 2 text (about the scope of commands) but not
explicit. I will add a word about ASAP.

the text I suggest for 9.5.5 is:

       For the ABORT TASK function, initiators MUST always set this to the
       CmdSN of the task identified by the Referenced Task Tag field.
       Targets must use this field as described in section 9.6.1 when the
       task identified by the Referenced Task Tag field is not with the
       target. RefCmdSN MUST be lower than the Task Management Request
       CmdSN field (in serial arithmetic sense).

As for immediate commands and RefCmdSN - my comment was that a missed
immediate command is missed forever!
The RefCmdSN will not help you when it is missing since an immediate commad
does not generate a hole!

Julo

                      Eddy Quicksall                                                                                                            
                      <eddy_quicksall@i        To:       Julian Satran/Haifa/IBM@IBMIL                                                          
                      vivity.com>              cc:       "[email-masked]'" <[email-masked]>, [email-masked], [email-masked], 
                                                [email-masked]                                                                           
                      05/09/02 16:38           Subject:  RE: iSCSI: aborting an immediate command with ABORT TASK                               

I don't see that fact when I read the spec. 9.5.5 RefCmdSN implies that the
RefCmdSN may be used if the Referenced Task Tag (ITT) is not with the
target anymore. And, I think that is David and Tony's point.

I may have missed it someplace ... can you please point out where it says
or implies "an immediate is aborted based on ITT"?

Eddy
      -----Original Message-----
      From: Julian Satran [mailto:[email-masked]]
      Sent: Thursday, September 05, 2002 9:23 AM
      To: Eddy Quicksall
      Cc: [email-masked]'; Eddy Quicksall; [email-masked];
      [email-masked]; [email-masked]
      Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

      No need - it works always as it is. An immediate is aborted based on
      ITT or not at all.
      Add to it the fact that they are on the same connection and
      immediates are not reassigned
      and everything works AS IS.
      What am I missing?

      Julo

   Eddy Quicksall                                                          
   <eddy_quicksall@iviv         To:        Julian Satran/Haifa/IBM@IBMIL,  
   ity.com>             Eddy Quicksall <[email-masked]>        
                                cc:        "[email-masked]'"         
                        <[email-masked]>, [email-masked],            
   05/09/02 16:02       [email-masked], [email-masked]       
                                Subject:        RE: iSCSI: aborting an     
                        immediate command with ABORT TASK                  

      How about another bit in the abort that says it is aborting an
      immediate command and hence does not advance the window?

      Eddy
      -----Original Message-----
      From: Julian Satran [mailto:[email-masked]]
      Sent: Thursday, September 05, 2002 9:00 AM
      To: Eddy Quicksall
      Cc: [email-masked]'; [email-masked]; [email-masked];
      [email-masked]
      Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

      Julo
                                                                           
   Eddy Quicksall                                                          
   <[email-masked]        To:        "[email-masked]'"    
   m>                         <[email-masked]>, [email-masked] 
   Sent by:                                                                
   [email-masked]             cc:        [email-masked]            
                                     Subject:        RE: iSCSI: aborting   
                              an immediate command with ABORT TASK         
   05/09/02 15:46                                                          

      Can we say that aborts with the immediate bit only abort commands
      with the
      immediate bit? And then say that the window is not advanced under
      that
      situation?

      Eddy

      -----Original Message-----
      From: [email-masked] [mailto:[email-masked]]
      Sent: Thursday, September 05, 2002 2:51 AM
      To: [email-masked]
      Cc: [email-masked]
      Subject: RE: iSCSI: aborting an immediate command with ABORT TASK

      I

          agement function request is within the valid CmdSN window
      (between

      Thanks,
      --David

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 5 Sep 2002 19:26:28 +0300

Forget the previous text - we wont the clarification to be added for the
"plugging side" (the target) and the we suggest changing the last bullet b
in 9.6.1
to:

          If the Referenced Task Tag does not identify an existing task,
          but if the CmdSN indicated by the RefCmdSN field in the Task
          Management function request is within the valid CmdSN window and
          less than the CmdSN of the Task Management function request
          itself, then targets must consider the CmdSN received and return
          the "Function complete" response.

 We thought this was obvious from the overview text but it apparently is
 not.

 Julo

The current wording IS CORRECT!

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 5 Sep 2002 22:05:06 +0300

In addition in order to eliminate the "corner case" I mentioned earlier
(and that is the only ambiguity we know about) - that a TASK ABORT may
abort
both an immediate and a non-immediate that have the same number and are
both lost even though only the immediate was intended I added the following
text to 9

       If an ABORT TASK is issued for a task created by an immediate
       command then RefCmdSN MUST be that of the Task Management request
       itself (i.e. CmdSN and RefCmdSN are equal; otherwise RefCmdSN MUST
       be set to the CmdSN of the task to be aborted (lower than CmdSN).

 and made 9.5.5 to raed:

   RefCmdSN

       If an ABORT TASK is issued for a task created by an immediate
       command then RefCmdSN MUST be that of the Task Management request
       itself (i.e. CmdSN and RefCmdSN are equal).

       For an ABORT TASK of a task created by non-immediate command
       RefCmdSN MUST be set to the CmdSN of the task identified by the
       Referenced Task Tag field. Targets must use this field as described
       in section 9.6.1 when the task identified by the Referenced Task Tag
       field is not with the target.

       Otherwise, this field is reserved.

    Regards,
    Julo

----- Forwarded by Julian Satran/Haifa/IBM on 05/09/02 21:56 -----
                                                                                                                                                
                      Julian Satran                                                                                                             
                                               To:       [email-masked]                                                                        
                      05/09/02 19:26           cc:                                                                                              
                                               From:     Julian Satran/Haifa/IBM@Haifa/IBM@IBMIL                                                
                                               Subject:  RE: iSCSI: aborting an immediate command with ABORT TASK                               

 Julo