Subject: [Ips] Fast multi-task abort model

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 03 Nov 2005 17:32:00 -0800

As promised earlier, this memo describes a possible new iSCSI multi-task 
abort model that supports a much more expeditious termination of 
affected tasks.  I gave it a decent attention, but have not thought 
through all the corner cases fully, so please give it a critical read. 
If we agree adopting something like this,  consider if it should be an 
additional behavior when a new key "FastMultiTaskAbort=Yes" is 
negotiated by both parties.

So where does the "fast" in FastMultTaskAbort come from?

- Eliminates the CmdSN wait on third-party sessions
- Eliminates the TTT wait on issuing & third-party sessions
- Eliminates the wait on third-party StatSN acks for the issuing
   session

And:
- Should work identically for iSCSI-3720 & iSCSI/iSER
- No new SCSI-level requirements (e.g. TAS=1)

However, the downside is that the new model is more complex.

The model description follows.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
StorageWorks Division
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

The following iSCSI protocol actions happen on an initiator iSCSI layer
after issuing a multi-task abort TMF - i.e. one of Abort Task Set, Clear
Task Set, LU Reset, Target Cold Reset, and Target Warm Reset TMFs:

1. The initiator iSCSI layer must not send any more Data-Out PDUs
    for affected tasks on the issuing connection of the issuing iSCSI
    session once the TMF is sent to the target.

2. Initiator iSCSI layer should receive any responses that the target
    may provide for some tasks among the affected tasks (may process them
    as usual because they are guaranteed to have chronologically
    originated before the TMF response).

3. Initiator iSCSI layer must respond to the Async Message PDU with
    AsyncEvent=5 by taking two steps in this order:
	a) stop Data-Out transfers on that connection for all active
            TTTs for the affected LUN quoted in the Async Message PDU.
	b) acknowledge the StatSN of the Async Message PDU via a
            Nop-Out PDU with ITT=0xffffffff (i.e. non-ping flavor),
            while copying the LUN field from Async Message to Nop-Out.

4. Initiator iSCSI layer receives the task management response
    concluding all the tasks in the set of affected tasks.

The following iSCSI protocol actions happen on a target iSCSI layer
receiving a multi-task abort TMF - i.e. one of Abort Task Set, Clear
Task Set, LU Reset, Target Cold Reset, and Target Warm Reset TMFs:

0. Based on the CmdSN ordering, target iSCSI layer waits for all
    SCSI Command PDUs of the affected tasks to be received on the
    issuing session.  On third-party affected sessions, only the
    instantiated tasks have to be considered for the purpose of
    determining the affected tasks - i.e. no waiting is needed on
    third-party sessions.  In the case of target-scoped requests (i.e.
    target resets), all the commands that are not yet received on
    the issuing session in the command stream however can be considered
    to have been received with no command waiting period - i.e. the
    entire CmdSN space upto the CmdSN of the task management function
    can be "plugged".

1. Target iSCSI layer initiates the termination proceedings on all
    affected tasks immediately - this may involve interacting with the
    local SCSI layer.

2. Target iSCSI layer leaves all active "affected TTTs" (i.e. active
    TTTs associated with "affected tasks") valid along with any buffer
    allocations for the TTTs intact.

3. Target iSCSI layer generates an Asynchronous Message PDU with a new
    AsyncEvent=5 that says "all active tasks for LU with matching LUN
    field in the Async Message PDU are being terminated" on:
	a) each connection of each third-party session that at least one
            affected task is allegiant to, and
	b) each connection except the non-issuing connection of the
            issuing session that at least one affected task is allegiant
            to.

    If there are multiple affected LUs (say due to a target reset),
    then one Async Message PDU must be sent for each such LU on each
    affected connection.

4. Once the local SCSI layer had terminated the SCSI tasks and handed
    a TMF Response to it, the target iSCSI layer takes note of last-sent
    and unaknowledged StatSN on each of the connections in the iSCSI
    session(s) sharing the affected tasks, and waits for acknowledgement
    of each such StatSN (may solicit for acknowledgement by way of a
    Nop-In).  If any new task responses for the affected LU(s) are
    meanwhile received from the SCSI layer while waiting for StatSN
    acknowledgement(s), those Response PDUs  the first SCSI Response
    PDU of which is presumably carrying the UA notification - must
    be held and queued at the iSCSI layer.

5. For each of the affected sessions, the target iSCSI layer waits
    independently for the StatSN acknowledgements.
	a) As all StatSNs are acknowledged on the issuing session,
            the target iSCSI layer sends the TMF Response in an iSCSI
            PDU to the issuing initiator.  All task response PDUs held
	   back at the iSCSI layer in Step#4, if any, are simultaneously
	   eligible for being placed on the wire as appropriate.
	b) As all StatSNs are acknowledged on a third-party session,
	   all task response PDUs held back at the iSCSI layer in
            Step#4, if any, are simultaneously eligible for being placed
	   on the wire as appropriate.

6. Technically, the task termination is complete in Step#5. Data
    transfers corresponding to terminated tasks may however still be in
    progress by Step#6.  In the case of iSCSI/iSER, these transfers
    would be into tagged buffers with STags not owned by any active
    tasks.  Target iSCSI layer must free up the affected TTTs and the
    corresponding buffers as it receives the associated Nop-Out
    acknowledgement that the initiator generated in Step#3b (initiator
    actions).  A target may on an implementation-defined internal timeout
    choose to drop the connections that it does not receive the expected
    Nop-Out acknowledgement on and reclaim the buffer and TTT resources.

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: William Studenmund <[email-masked]>
Date: Tue, 15 Nov 2005 14:41:34 -0800

On Nov 3, 2005, at 5:32 PM, Mallikarjun C. wrote:

> As promised earlier, this memo describes a possible new iSCSI 
> multi-task abort model that supports a much more expeditious 
> termination of affected tasks.  I gave it a decent attention, but have 
> not thought through all the corner cases fully, so please give it a 
> critical read. If we agree adopting something like this,  consider if 
> it should be an additional behavior when a new key 
> "FastMultiTaskAbort=Yes" is negotiated by both parties.

I think some of the ideas in this model should be dependent on a new 
key, but to the extent other ideas represent clarification of RFC 3720, 
I'm up for just implementing them.

My thoughts are that the use of the new AsyncEvent probably needs to be 
negotiated. However I am all for removing the inter-nexus delay 
requirement w/o requiring any negotiation.

> The model description follows.

Sounds fine, and I like the fact that it will work well with other 
transports. While I thought of a way that might work with iSER and TCP, 
I strongly doubt my thoughts would work with iSER and SCTP, so 
something else is needed.

>    entire CmdSN space upto the CmdSN of the task management function

I assume that, for a target-scoped request, other sessions would also 
be able to "plug" the entire CmdSN space? Oh, hmm...  I'm not sure 
about non-SCSI tasks on other iSCSI sessions... Never mind, we probably 
should NOT assume "pluggability" as I just mentioned.

Even though I reject the idea above, I'm leaving the text in so other 
folks can see why the idea won't work. :-)

>    affected tasks immediately - this may involve interacting with the

I like this step even for the non-FastAbort case.

>    be held and queued at the iSCSI layer.

:-) Isn't that text really assuming TAS set to 0? :-)

If TAS is set to 1, do we really need to wait for StatSN 
acknowledgement? Each task will return status so we will know what was 
aborted and what wasn't.

> 	   on the wire as appropriate.

Sounds fine modulo my question above.

I would like to see this step be applicable even w/o the FastTaskAbort 
option; I'd like us to amend RFC 3720 to use this delay model.

>    Nop-Out acknowledgement on and reclaim the buffer and TTT resources.

I thought that if we were doing FastAbort, that we wouldn't exit step 5 
for a given session until we receive an ack for the AsyncEvent, and 
thus all we need to do with #6 is just finish the cleanup implied in 
that ack. ?? Or is that what you're saying? :-)

Sounds good.

Take care,

Bill

---

From: Julian Satran <[email-masked]>
Date: Tue, 29 Nov 2005 17:08:45 +0200

Mallikarjun & All,

Sorry for the late answer but here are my 2 Cents:

-CmdSN wait on third party sessions is a mistake and if required we can 
fix it in an errata (It does not even appear on my copy of 
draft20+errata).
-TTT wait elimination requires a handshake and this could be an Async 
message but it can be also forcing TAS=1 and mandating the status.  I 
agree that your Asynch message is more ellegant but does it warrant a new 
version of the RFC? 
-StatSN wait for 3rd party falls in the same category as the first (CmdSN 
waiting) - can be fixed in errata.

Julo

"Mallikarjun C." <[email-masked]> 
Sent by: [email-masked]
04-11-05 03:32

To
IPS <[email-masked]>
cc

Subject
[Ips] Fast multi-task abort model

And:

Mallikarjun Chadalapaka

                 a) stop Data-Out transfers on that connection for all 
active

3. Target iSCSI layer generates an Asynchronous Message PDU with a new
    AsyncEvent=5 that says "all active tasks for LU with matching LUN
    field in the Async Message PDU are being terminated" on:
                 a) each connection of each third-party session that at 
least one
            affected task is allegiant to, and
                 b) each connection except the non-issuing connection of 
the
            issuing session that at least one affected task is allegiant
            to.

5. For each of the affected sessions, the target iSCSI layer waits
    independently for the StatSN acknowledgements.
                 a) As all StatSNs are acknowledged on the issuing 
session,
            the target iSCSI layer sends the TMF Response in an iSCSI
            PDU to the issuing initiator.  All task response PDUs held
                    back at the iSCSI layer in Step#4, if any, are 
simultaneously
                    eligible for being placed on the wire as appropriate.
                 b) As all StatSNs are acknowledged on a third-party 
session,
                    all task response PDUs held back at the iSCSI layer in
            Step#4, if any, are simultaneously eligible for being placed
                    on the wire as appropriate.

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 29 Nov 2005 17:08:45 -0800 (PST)

Sorry for the delay in responding, and thanks for the
review.  Comments below.

>requirement w/o requiring any negotiation.

Those two sentiments may be in conflict with each
other.  But I understand.

>something else is needed.

Not sure what's meant...

>:-) Isn't that text really assuming TAS set to 0? :-)

No.  The "new task responses" are chronologically
later than the 
affected tasks.  Think of them as "responses for
commands to the affected LU that were issued after the
TMF was issued".  This is a corner case, included only
for the paranoid.

>acknowledgement? Each task will return status so we
will know what was 

Waiting for StatSN ack is mandatory to satisfy the
"single flow" abstraction.  We do not want the TMF
Response to be received before the successful task
responses, *if any*, from the affected task set are
received by an initiator.  

TAS=1 addresses an entirely different matter - that of
TASK ABORTED status for the terminated tasks (*not*
the successful tasks).

>that ack. ?? Or is that what you're saying? :-)

Note that Step#3 is *sending* Async PDU only, an ack
is not necessary to complete the TMF.  The ack is a
"lazy ack" - i.e. may be received after the TMF is
finished.  Because the data transfers are decoupled
from the TMF processing, the task cleanup/TMF
completion happens much faster in this proposal. 
Step#6 is really an internal book-keeping operation
within the target to free up the TTTs, and associated
buffers.

Thanks.

Mallikarjun

--- William Studenmund <[email-masked]>
wrote:

>> termination of affected tasks.  I gave it a decent
attention, but 
>have 
>> The following iSCSI protocol actions happen on an
initiator iSCSI 
>layer
>> after issuing a multi-task abort TMF - i.e. one of
Abort Task Set, 
>> Clear
>>    may provide for some tasks among the affected
tasks (may process 
>> them
>>    the issuing session in the command stream
however can be 
>considered
>> 3. Target iSCSI layer generates an Asynchronous
Message PDU with a 
>new
>>            issuing session that at least one
affected task is 
>allegiant
>>    a TMF Response to it, the target iSCSI layer
takes note of 
>last-sent
>>    session(s) sharing the affected tasks, and waits
for 
>acknowledgement
session,
simultaneously
>>            Step#4, if any, are simultaneously
eligible for being 
>placed
>>    transfers corresponding to terminated tasks may
however still be 
>in
>>    actions).  A target may on an
implementation-defined internal 
>> timeout
>>    choose to drop the connections that it does not
receive the 
>expected
>>    Nop-Out acknowledgement on and reclaim the
buffer and TTT 
>resources.

--
Mallikarjun

Mallikarjun Chadalapaka

		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 29 Nov 2005 17:34:05 -0800 (PST)

Hi Julian,

Thanks for the review.

I agree with you that CmdSN & StatSN ack wait on 3rd
party sessions wasn't the original intent, and both
should be errata noted in the implementer's guide.

The third major source of delay comes from the TTT
wait - it's a cross-nexus delay and RFC 3720 requires
it for completing the original TMF.  The Async-NopIn
exchange enables us to take this wait out of the
"performance path" by giving the target iSCSI layer a
definitive event to free up the TTTs in a lazy
fashion.

I don't like mandating TAS=1 for several reasons - a)
it's a SCSI feature, b) it's not a default control
mode page setting, c) it makes it harder to build
iSCSI-to-XYZ bridges, d) puts iSCSI in a minor
competitive disadvantage wrt other transports, e) it
does not address the TTT problem with a
multi-connection issuing session.  So I am glad you
like the Async PDU approach.

I would let David comment on the RFC revision part -
but given all the other things we have been collecting
in the iSCSI implementer's guide and given the guide
will be published as an RFC that "updates" RFC 3720, I
think it should be OK to include the new behavior in
the guide.

Thanks!

Mallikarjun

--- Julian Satran <[email-masked]> wrote:

> initiator iSCSI layer
> Abort Task Set, Clear
> tasks (may process them
> however can be considered
> Message PDU with a new
> affected task is allegiant
>             to.
> 
> takes note of last-sent
> for acknowledgement
>     of each such StatSN (may solicit for
> acknowledgement 
=== message truncated ===>
_______________________________________________

__________________________________ 
Yahoo! Music Unlimited 
Access over 1 million songs. Try it free. 
http://music.yahoo.com/unlimited/

---

From: William Studenmund <[email-masked]>
Date: Tue, 29 Nov 2005 18:47:14 -0800

>> something else is needed.

I had some half-formed ideas that, well, I don't think will work with 
iSER or SCTP. I haven't explicitly mentioned them as I think they won't 
work. I indirectly mention them to say that I was trying to think of 
other things and didn't come up with much. :-)

> received by an initiator.

I understand why we want the wait for StatSN ack for the session 
issuing the TMF. I think that's fine.

However I still don't understand why StatSN ack matters for the 
inter-nexus case.

Oh, let me try that again. I still don't understand why it matters to 
make the TMF session wait for the completion of StatSN ack on all the 
other sessions. I agree that each of the other sessions & connections 
should go through the StatSN Ack wait in and of itself. I just don't 
understand why the TMF session has to wait for them to finish.

There are always delays (or potential delays) and races, and it is hard 
for an initiator to say "this happened exactly then." For instance, 
with MC/S, we have no way to interlock status delivery between 
different connections. So task A on connection 1 could finish and be 
issued status at the target before task B on connection 2, but due to 
network congestion/traffic flow, the status for task B could be 
delivered to the initiator before that of task A. We consider that to 
be OK. I'm seeing the inter-nexus case as a between-session extension 
of the same.

As long as we can say that there was one common instant in time when we 
separated all impacted tasks into already-done, aborted, and 
active-after-abort-done and further that the TMF response is generated 
ONLY after that instant, do we really need to do more?

As I asked before, how would this inter-nexus StatSN wait work if one 
of the initiators failed, and as such will never be acknowledging 
StatSN?

In my experience, I've seen initiators just stop receiving iSCSI 
packets. The initiator host is healthy, ping works, keep-alives are 
happy, but the initiator logic just isn't doing anything. I think a LUN 
Reset is a great thing to do in this case to clean everything up. But 
if I have to wait for said dead initiator, how do I ever get anything 
done?

> TASK ABORTED status for the terminated tasks (*not*

Ok, I see my confusion.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 29 Nov 2005 19:24:54 -0800 (PST)

>I indirectly mention them to say that I was
> trying to think of 

OK, :-)  But please do let me know if you spot a
problem with the proposed model wrt some aspect of
SCTP.

>I still don't understand
> why it matters to 
> should go through the StatSN Ack wait in and of
> itself.

I think we're in serious agreement.  This is exactly
what I wrote in the summary notes for the original
proposal - that StatSN ack wait for third-party
sessions is out (and also to Julian earlier today) -
each session's StatSN ack wait is on its own.

Mallikarjun

wrote:

> >> My thoughts are that the use of the new
> AsyncEvent
> > probably needs to be
> >> I strongly doubt my thoughts would work with iSER
> and
> > SCTP, so
> >> acknowledgement? Each task will return status so
> we
> > will know what was
> >
> >> I thought that if we were doing FastAbort, that
> we
> > wouldn't exit step 5
> >> for a given session until we receive an ack for
> the
> > AsyncEvent, and

--
Mallikarjun

Mallikarjun Chadalapaka