Subject: Fw: [Ips] iSCSI: abort task set

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 30 Jan 2004 09:42:24 -0500

Mallikarjun,

I am really wondering what the target should do. Can you give some advice?

It is clear what to do if the TMF is not immediate... the problem is: If it is immediate, how can the target determine when "all affected tasks in the task set" have been received? The target can't simply wait for all earlier CmdSN's to be received or the target can deadlock if multiple connections.

Eddy

----- Original Message ----- 
From: Eddy Quicksall 
To: [email-masked] 
Sent: Wednesday, January 28, 2004 4:39 PM
Subject: [Ips] iSCSI: abort task set

I've always wondered about the following paragraph but never asked.

10.6.2 says:
The target:

a) Receives the ABORT TASK SET/CLEAR TASK SET request.

b) Waits for all target transfer tags to be responded to

and for all affected tasks in the task set to be

received.

How can the target determine that all affected tasks in the task set have been received? Does this simply mean that the target must sequence the TMF just as any other command even if the TMF is immediate?

What about if TST is 000b? How do I determine when some other initiator has sent all affected tasks?

Eddy

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 30 Jan 2004 10:41:53 -0800

Eddy,

Sorry for not responding sooner, I was checking my old
notes (but couldn't find relevant ones so far).

The authors' advice on this is the following -

The wait for affected tasks can be done (need to
pick a point in time as the reference for determining
the "affected" tasks, regardless of the # of sessions).
However, an alternative that's also legal is given below.

Bullet (b) should have been:

b) Waits for all target transfer tags to be responded to
    and for all affected tasks in the task set to be
    received (alternatively, the target may plug the CmdSN
    gaps for unreceived commands just as if an abort request
    is received for each individual affected task, refer
    section 6.9 "SCSI timeouts")

Note that the wait on StatSN ack is however
mandatory.

It is good to get this alternative/clarification into
the final RFC.

Thanks.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

Eddy Quicksall wrote:

> It is clear what to do if the TMF is not immediate... the problem is: If 
> it is immediate, how can the target determine when "all affected tasks 
> in the task set" have been received? The target can't simply wait for 
> all earlier CmdSN's to be received or the target can deadlock if 
> multiple connections.
>  
> Eddy
>  
> From: Eddy Quicksall <mailto:[email-masked]>
> To: [email-masked] <mailto:[email-masked]>
> How can the target determine that all affected tasks in the task set 
> have been received? Does this simply mean that the target must sequence 
> the TMF just as any other command even if the TMF is immediate?
>  

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: [email-masked]
Date: Fri, 30 Jan 2004 11:33:55 -0800 (PST)

>     section 6.9 "SCSI timeouts")

Abort Task Set has to plug the whole CmdSN window on each session?? That
sounds excessive. Plugging the CmdSN hole for an Abort Task seems fine,
since the initiator tells the target explicitly which CmdSN to abort (and
also knows what the highest CmdSN used so far is). But with the Task Set
commands, we _don't_ know the highest CmdSN the initiator has used, only
the highest CmdSN we've seen and MaxCmdSN.

Also, for the TST 000b case, what does that mean for other initiators?
Their entire CmdSN window gets filled? Consider an initiator that was
quiescent. If I'm understanding the window-filling correctly, its entire
CmdSN window gets slammed shut. Since its quescent, how will it know its
window got closed? Any non-immediate command it sends will just disapear
as the CmdSN window was filled. It will also see the CmdSN window just
move if the target sends in a PDU. It won't understand why the window
moved of its own accord until it gets a SCSI abort UA. ??

Finally, since iSCSI tasks don't get passed to the SCSI layer until the
CmdSN window is satisfied, are CmdSN-pending tasks really in the SCSI task
set?

Or did I misunderstand something? Please tell me I did as the above looks
messy. :-)

> Note that the wait on StatSN ack is however

Is there any sort of time out?

> It is good to get this alternative/clarification into

Take care,

Bill

---

From: "Ken Sandars" <[email-masked]>
Date: Fri, 30 Jan 2004 19:58:43 -0000

Hi Bill,

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of [email-masked]
> Sent: 30 January 2004 19:34
> To: Mallikarjun C.
> Cc: Eddy Quicksall; [email-masked]
> Subject: Re: Fw: [Ips] iSCSI: abort task set
> 
> 
> > the "affected" tasks, regardless of the # of sessions). However, an 
> > alternative that's also legal is given below.
> >
> Abort Task Set has to plug the whole CmdSN window on each 
> session?? That sounds excessive. Plugging the CmdSN hole for 
> an Abort Task seems fine, since the initiator tells the 
> target explicitly which CmdSN to abort (and also knows what 
> the highest CmdSN used so far is). But with the Task Set 
> commands, we _don't_ know the highest CmdSN the initiator has 
> used, only the highest CmdSN we've seen and MaxCmdSN.
> 

For the session the TMF was issued in, the CmdSN of the TMF
command is where it's up to.  However I think this plugging
breaks down for a multiple-LUN target. Can the target assume
the missing tasks are for the affected LUN?

> Also, for the TST 000b case, what does that mean for other 
[snip]

Nothing - fallback to to the 'pick a point in time' method.

Hope that helps,

Ken Sandars
Elipsan UK

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 30 Jan 2004 12:10:20 -0800

The term "plugging the CmdSN gap" doesn't mean
shutting the CmdSN window down.  If the target
received a CmdSN X but has not yet received a
CmdSN Y within the CmdSN window (where Y < X,
in Serial Number Arithmetic), then Y is the
"CmdSN gap" (not the valid window > X).  To
summarize, a "CmdSN gap" must meet two conditions
a) target deterministically knows it's already
issued by the initiator, and, b) it has not yet
arrived.

In this example, an abort operation for CmdSN=Y
simply "plugs" the CmdSN gap, i.e. the CmdSN is
considered received and aborted from now on.  This
lets the window slide (standard behavior today).
There's no reason to disallow the same behavior
when multiple tasks are being aborted.

Plugging the CmdSN gaps at a point in time
is not a complex operation.  And it doesn't affect
quiescent initiators at all.

 > CmdSN window is satisfied, are CmdSN-pending tasks really in the SCSI 
task

I assme you mean "commands are reordered" when you
say "CmdSN window is satisfied".  One could make an
argument that the tasks wating in the transport are
not in a SCSI task set.  But we decided way back that
iSCSI should hide the queueing that it introduced in
the transport, so application clients won't receive
some task responses after the task set is aborted
(also check the command ordering draft for a related
discussion).

 > Is there any sort of time out?

Not that the spec specifies.  Standard error recovery
is applicable on timeouts.

Hope this helps.

M

---

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 30 Jan 2004 15:52:32 -0500

When you say "pick a point in time" do you simply mean to use the CmdSN of
the ABORT TASK SET TMF for the given initiator and TST=001b? Or are you
applying this phrase to other initiators when TST is 000b?

Eddy

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 30 Jan 2004 12:59:39 -0800

>However I think this plugging

No, not always.  A very good point.  Even though the
suggested "alternative" is an optimization for
tight-knit (i.e. rototilled) targets, it isn't generally
usable for multi-LU, strict-layered, targets.  Now
I recall this is why we didn't put this "alternative"
in the spec to begin with (haven't found those notes...).

So I now believe the "alternative" text in paranetheses
that was suggested is at best a fit for implementers
notes, if at all.

To answer Eddy's original question in light of this,
while the standard error recovery is applicable on
timeouts, the wait for all TTT-reponses, CmdSN gaps
and StatSN acks is necessary.

Thanks.
-- 
Mallikarjun

Mallikarjun Chadalapaka

---

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 30 Jan 2004 16:15:38 -0500

> the wait for all ... CmdSN gaps ... is necessary

Does this mean that if an ABORT TASK SET is immediate. it can't be executed
as immediate? (i.e., it must wait for all CmdSN's less than the ABORT TASK
SET CmdSN to be received).

If so, and the missing CmdSN's are from a connection that has stalled
indefinatly (e.g. a stuck router), how long should the target wait?
Time2Retain (actually, I don't think that is what Time2Retain is for)?

Eddy

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 30 Jan 2004 13:16:27 -0800

The same method applies for either case, the TST field is
however not relevant unless it is CLEAR TASK SET.

The "point in time" should preferably immediately follow
the reception of the TMF.

BTW, now that I think more about it, the "alternative"
text below is to be ignored - rototilled or otherwise.

M

Eddy Quicksall wrote:

>>>I am really wondering what the target should do. Can you give some
> 
> advice?
>

---

From: "Ken Sandars" <[email-masked]>
Date: Fri, 30 Jan 2004 21:31:42 -0000

Hi Eddy,

> Behalf Of Eddy Quicksall
> Sent: 30 January 2004 21:16
> To: Mallikarjun C.
> Cc: [email-masked]
> Does this mean that if an ABORT TASK SET is immediate. it 
> can't be executed as immediate? (i.e., it must wait for all 
> CmdSN's less than the ABORT TASK SET CmdSN to be received).

Sort of like the spec should mandate ABORT TASK SET and 
CLEAR TASK SET MUST NOT be sent as immediate? I'd buy an
iSCSI spec like that ;-)

> 
> If so, and the missing CmdSN's are from a connection that has 
> stalled indefinatly (e.g. a stuck router), how long should 
> the target wait? Time2Retain (actually, I don't think that is 
> what Time2Retain is for)?

I'm thinking that's an initiator problem ;-)

Targets do the receive data sequence timeout thang on TTT, but
isn't it up to the initiators to work out the news when ExpCmdSN isn't
changing regardless of what the target sends back (in say NOP-IN pdus)?

Cheers,
Ken Sandars
Elipsan UK

> >  > breaks down for a multiple-LUN target. Can the target 
> assume  > the 
> > missing tasks are for the affected LUN?
> >
> > No, not always.  A very good point.  Even though the suggested 
> > "alternative" is an optimization for tight-knit (i.e. rototilled) 
> > targets, it isn't generally usable for multi-LU, strict-layered, 
> > targets.  Now I recall this is why we didn't put this "alternative"
> > >>From: [email-masked] [mailto:[email-masked]] On Behalf Of 
> > >>[email-masked]
> However, an 
> session?? 
> > >>That sounds excessive. Plugging the CmdSN hole for an Abort Task 
> > >>seems fine, since the initiator tells the target explicitly which 
> > >>CmdSN to abort (and also knows what the highest CmdSN used so far 
> > >>is). But with the Task Set commands, we _don't_ know the highest 
> > >>CmdSN the initiator has used, only the highest CmdSN 
> we've seen and 
> > >>MaxCmdSN.
> > >>
> > >
> > >
> > > For the session the TMF was issued in, the CmdSN of the 
> TMF command 
> > > is where it's up to.  However I think this plugging 
> breaks down for 
> > > a multiple-LUN target. Can the target assume the missing 
> tasks are 
> > > for the affected LUN?
> > >
> > >
> > >

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 30 Jan 2004 13:32:43 -0800

> SET CmdSN to be received).

Yes.  10.5.1 already states for this reason:

If the task management request is marked for immediate delivery, it
must be considered immediately for execution, but the operations
involved (all or part of them) may be postponed to allow the target
to receive all relevant tasks.

> Time2Retain (actually, I don't think that is what Time2Retain is for)?

No, it is not that.  This is in the implementation territory,
and I would simply say that standard error recovery should be
used for non-responding connections with implementation-defined
timeouts.

Mallikarjun

---

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 30 Jan 2004 16:43:53 -0500

Ok ... got it. It turns out that my NOP-In's will catch a stuck router (I
forgot about that).

Eddy

---

From: [email-masked]
Date: Fri, 30 Jan 2004 14:54:48 -0800 (PST)

> quiescent initiators at all.

I agree that the act of plugging gaps is not hard. My question though is
about ambiguity as to what gaps to plug. :-) My original concern was about
a command CmdSN=Z where X (received) < Z. That's why I was talking about
plugging the whole command window. And on other sessions (TST=000b), since
we don't have a CmdSN on the command to tell us if there are any commands
'Z', we do have an ambiguity.

As Ken mentioned, we have a bigger concern.  For the example above, how do
we know if CmdSN=Y is for the LU on which we're aborting the set?

> say "CmdSN window is satisfied".  One could make an

No, I meant the commands that are queued so they don't get reordered. :-)

> (also check the command ordering draft for a related

Well, we trade one problem (responses after aborting the task set since
the commands were queued and went forward after the abort) for another
(ambiguities as to which unreceived tasks should be aborted).

I was about to suggest that for multi-LU targets we could just create
CmdSN-plugs that had an explicit LUN. If a command comes in with the same
CmdSN as a plug but for a different LUN, we would silently unplug & let
the command through. Unfortunately it won't work.

The problem though is that we could have a situation where the command
window moves before a lost command is resent. In that case the initiator
thinks a command (for a different LUN than the abort task set) has been
received since we moved ExpCmdSN, but the target doesn't have it, since it
thought it was aborted.

While I agree that we should hide iSCSI's command queuing from end
clients, I don't think we need to hide it from the initiator.

To that end, I propose that we:

1) Change (or clarify) ABORT TASK SET and CLEAR TASK SET to only apply to
commands that have been "delivered for execution"; for commands with
CmdSNs that have been successfully received. Commands held to prevent
out-of-order SCSI delivery are NOT touched.

2) mandate an initiator's behavior when presented with an ABORT TASK SET
or CLEAR TASK SET operation to be: a) issue ABORT TASKs for all issued
commands not then acknowledged by the target (*) and then b) issue the
ABORT or CLEAR TASK SET.

(*) Specifically commands with CmdSN >= last received ExpCmdSN. This may
mean aborting immediate commands if there were any in the window.

This way all the tasks the initiator thinks are outstanding on the LU get
aborted and no others. We hide iSCSI's queuing from end clients yet leave
no opportunity for tasks to just be dropped. As for other sessions
(TST=000b), the "Abort" happened at a time when only the commands
delivered for execution were in the task set.

> is applicable on timeouts.

So just to be clear, say one connection in the session is down. Or in the
TST=000b case a different session accessing this LU is ill. We will end up
waiting for error recovery to happen before responding to the TMR? Which
can be tens of seconds if the connection or session times out through
Time2Retain? Just making sure...

> Hope this helps.

It does. I think this discussion is productive. I know my concerns have
been refined because of it.

Take care,

Bill