Subject: Re: [Ips] Fast multi-task abort model

From: William Studenmund <[email-masked]>
Date: Thu, 1 Dec 2005 09:03:55 -0800

On Nov 29, 2005, at 5:34 PM, Mallikarjun C. wrote:

> Hi Julian,
>
> Thanks for the review.
>
> I agree with you that CmdSN & StatSN ack wait on 3rd
> party sessions wasn't the original intent, and both
> should be errata noted in the implementer's guide.
>
> The third major source of delay comes from the TTT
> wait - it's a cross-nexus delay and RFC 3720 requires
> it for completing the original TMF.  The Async-NopIn
> exchange enables us to take this wait out of the
> "performance path" by giving the target iSCSI layer a
> definitive event to free up the TTTs in a lazy
> fashion.

Why does it need to be cross-nexus, other than the fact that RFC 3720 
says it should be?

My thought is that we just revise it to not be cross-nexus. Yes, a 
given session is in the "waiting for things to stop" state until the 
TTTs clear, but as long as the "instant of death" I mentioned in my 
other mail has happened, I don't think that the TMF needs to wait.

> I don't like mandating TAS=1 for several reasons - a)
> it's a SCSI feature, b) it's not a default control
> mode page setting, c) it makes it harder to build
> iSCSI-to-XYZ bridges, d) puts iSCSI in a minor
> competitive disadvantage wrt other transports, e) it
> does not address the TTT problem with a
> multi-connection issuing session.  So I am glad you
> like the Async PDU approach.

I agree with (c) and (e), and they are enough to make me not like the 
TAS=1-required idea.

> I would let David comment on the RFC revision part -
> but given all the other things we have been collecting
> in the iSCSI implementer's guide and given the guide
> will be published as an RFC that "updates" RFC 3720, I
> think it should be OK to include the new behavior in
> the guide.

Just to be clear, I like both the new abort proposal (AsyncEvent 5) and 
also removing all long-wait inter-nexus delays. Obviously the 
abort/reset will have to have started on all nodes before the TMF is 
returned. :-)

Take care,

Bill
_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 1 Dec 2005 16:01:01 -0800 (PST)

> Why does it need to be cross-nexus, other than the

There's a good rationale behind RFC 3720 text:

TMF cannot complete until all affected tasks are
terminated on the target per SAM.  Receiving stale
data after tasks are terminated is a problem.  Waiting
for all active TTTs of all affected tasks (same and/or
other nexus) to finish was thus the adopted approach
to avoid stale data for any affected task.

What the new proposal does is to make receiving stale
data after task terminations a non-problem - via a
separate accounting scheme and new target semantics.

Mallikarjun

--- William Studenmund <[email-masked]>

> abort/reset will have to have started on all nodes

		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

---

From: Julian Satran <[email-masked]>
Date: Fri, 2 Dec 2005 21:17:20 +0200

Regrdless of the TMF completion - discarding old TTT tagged data buffers 
is problems third party target sessions will face unless they know for 
sure that none can be expected in a safe way. 

Julo

William Studenmund <[email-masked]> 
Sent by: [email-masked]
01-12-05 19:03

To
"Mallikarjun C." <[email-masked]>
cc
IPS <[email-masked]>
Subject
Re: [Ips] Fast multi-task abort model

Take care,

Bill

---

From: William Studenmund <[email-masked]>
Date: Mon, 5 Dec 2005 21:13:45 -0800

On Dec 2, 2005, at 11:17 AM, Julian Satran wrote:

> Regrdless of the TMF completion - discarding old TTT tagged data 
> buffers is problems third party target sessions will face unless they 
> know for sure that none can be expected in a safe way.

Agreed. And unless the new abort model is used, I agree that each 
session will have to wait for all the TTTs on it to drain before it 
can, in the TAS=0 case, start accepting commands again.

I _think_ we all agree on what each session must do once a reset 
happens. I think the only disagreement is how far along the actions on 
third-party sessions must be before the TMF can complete.

I'm going to reply to David's EMail and see if I can explain the core 
of my concern. Then maybe folks can explain how I'm wrong. ;-)

Take care,

Bill

---

From: William Studenmund <[email-masked]>
Date: Mon, 5 Dec 2005 21:14:24 -0800

> to avoid stale data for any affected task.

Shouldn't we leave how the target avoids/copes with stale data to the 
target?

I guess I don't see how receiving "stale" data after a task has been 
terminated is a problem. Specifically I don't see how it's a problem to 
receive data you knew was in-flight at the moment the termination 
happened. You have your tcb structure (however you have it laid out), 
and you know what TTs are associated with it. Mark the tcb as 
terminated, then wait for the TTs to complete. When they are done, 
throw the whole thing out.

> separate accounting scheme and new target semantics.

And I think this is a good thing. However I still think we need to 
abandon the inter-nexus TT wait for the old scheme.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 6 Dec 2005 15:12:58 -0800 (PST)

>Mark
> the tcb as 

Reasonable, and this is in abstract what the new
proposal does.

RFC 3720 does not explicitly say "wait for TTTs to
complete even after TMF is completed".  The only
default interpretation of the 3720 text today would be
to invalidate the TTTs along with the buffer
associations when a task is "terminated" and the TMF
completion is generated.  An invalid TTT/STag is in
fact a problem for at least two reasons - 

a) All Data-Out PDU (with the now invalid TTTs) are
protocol errors that should be Reject'ed.
b) Any DDP tagged data segment to an invalid STag
immediately takes the connection down (for
iSCSI/iSER).

What I am suggesting in the new proposal is that there
be an explicit event to conclude the "waiting for TTTs
to complete" for a target at the iSCSI control
protocol level (not at the Datamover level).  That new
event is the reception of the Nop-Out ack'ing the new
Async PDU.  And that brings us to the new proposal.

I think the 3720 text is self-consistent (although it
isn't the most efficient).  Simply scratching the TTT
cross-nexus requirement off the current text, IMHO, is
not the right thing. We need to specify the new
initiator and target semantics in the absence of that
requirement, and that's what the fast multi-task abort
proposal attempts.

Mallikarjun

wrote:

> >> Why does it need to be cross-nexus, other than
> the
> >> fact that RFC 3720
> >> says it should be?
> >

__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com

---

From: William Studenmund <[email-masked]>
Date: Wed, 7 Dec 2005 15:09:50 -0800

> fact a problem for at least two reasons -

If the target breaks the buffer associations, does it really have to 
invalidate the TTT?

We're dragging in a lot of details that feel like they are 
implementation specific. Our target has no trouble with the idea that a 
task is dead yet has active TTTs, the contents of which will never be 
used. We just let the TTTs drain off and clean up the task. We do this 
as you've done a good job of explaining to me that things are REALLY 
messy if we don't. :-)

As long as data received "after" (in terms of some atomic state change) 
the termination will be discarded, does the termination act really care 
how how the cleanup gets done?

Put another way, I think it's fine for a task to be terminated at the 
SCSI level yet still have things going on (being cleaned up) at the 
iSCSI level. As long as cleanup transfers aren't actually placed to the 
SCSI device, I don't see why a target can't do this.

I understand that the STags won't be able to go away. But as long as 
the target device doesn't use the buffers the STag goes into, is there 
a problem? Thus if a target breaks buffer associations, we should be 
fine, shouldn't we?

> Async PDU.  And that brings us to the new proposal.

I understand that. But that's orthogonal to what I'm very concerned 
about. I'm not concerned that session A waits for its TTTs, nor that 
session B waits for its TTTs, nor that session C waits for its TTTs, 
but that session D has to wait for all of them to complete a TMF.

I think that session A waiting for TTTs or using the new Async PDU is 
fine. As for session B and session C.

> requirement, and that's what the fast multi-task abort

We have a denial of service window. I don't see how we can get rid of 
it w/o ripping out the cross-nexus TTT wait.

I also don't understand why it's ok to remove the inter-nexus wait for 
the new technique and not for the old?

The only alternatives I see are to either migrate everyone to the new 
Async PDU or to warn implementors and administrators that failover can 
take a long time on iSCSI.

The latter will be a real shame. My understanding is that FibreChannel 
doesn't have such delays, so we'll be at a competitive disadvantage.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 8 Dec 2005 10:45:06 -0800 (PST)

> If the target breaks the buffer associations, does

We may be using different terminologies.....

I think of:
"invalidating the TTT" = "breaking the buffer
association" = "not having a buffer behind a TTT"

And that can be a serious problem in iSCSI-3720, and
fatal in iSCSI/iSER.

>I think it's fine for a task to be
> terminated at the 
> iSCSI level.

Precisely.  That's the fast multi-task abort model.

I don't see a disagreement wrt the preferred behavior.
 Only on the mechanics of how to phase it in.  I think
if the inter-nexus TTT wait requirement from the 3720
text is deleted, we have to add several careful
sentences on how it would still work well with both
iSCSI-3720 and iSCSI/iSER.  By then, we would have
suggested enough deltas to protocol behavior that it
would be very close to the new proposal.

Note that this is different from scratching the other
two waits - inter-nexus CmdSN gap wait, inter-nexus
StatSN ack wait.  They can be simply dropped without
any additional mandates.

>either migrate
> everyone to the new 
>.... so we'll be at a
> competitive disadvantage.

Yes, multi-task abort affecting multiple nexuses
(i.e., excluding Abort Task Set) may indeed take a
long time in iSCSI today.  And yes, it is a
competititve disadvantage.  That is what prompted
several on the list to point out the current problems,
and prompted me to propose the new model.  We will
have seriously improved the situation in a few months.

Mallikarjun

wrote:

> >> terminated, then wait for the TTs to complete.
> When
> >> they are done,

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

---

From: William Studenmund <[email-masked]>
Date: Thu, 8 Dec 2005 17:53:59 -0800

> We may be using different terminologies.....

Probably.

> And that can be a serious problem in iSCSI-3720, and

Because of that concern, I would like to suggest we refine our 
terminology, and make a distinction that hasn't been made before.

I suggest we consider "breaking the buffer association" at the SCSI 
layer to be separate from breaking it at the iSCSI level. All the task 
state diagram in SAM-2 (and later SAM) care about is the SCSI layer.

Specifically, I think it'd be fine for an abort to leave the TTT's 
buffer around and simply do nothing with it once the TTT is done. Thus 
all the PDUs flying at the target still find a destination (either in 
TCP iSCSI or iSER).

> would be very close to the new proposal.

Kind of. I agree we will need to add text, but I don't think it will be 
that bad.

I see the inter-nexus-TTT-fixup idea as an intermediate step between 
what we have now and the new multi-task abort model. We add the idea of 
not waiting for inter-nexus cleanup to finish (we do wait for it to 
start!) and we add the idea that data arriving after cleanup started to 
not get used by the SCSI layer. The new multi-task abort model then 
adds a method by which we can cleanly fix multiple tasks at once.

> StatSN ack wait.  They can be simply dropped without

I still don't see how dropping the TTT wait will hurt more than the 
others. I'm really trying to, but can't.

I fully agree I'm proposing a notable protocol change. :-) The reason 
I'm contemplating it, much less strongly suggesting it, is that 1) I 
think the wait we have isn't workable. Something needs to be done. 2) 
I'm not really sure how initiators will notice the change. Since the 
on-the-wire PDUs won't change, I suspect many initiators will never 
notice.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 9 Dec 2005 19:05:33 -0800 (PST)

Bill,

> others. I'm really trying to, but can't.

OK.  Let me attempt to be more specific.  Here are
some spec implications if we eliminate the inter-nexus
TTT wait from RFC 3720 text.  All these must be
specified in addition to deleting the TTT wait.
 
- Do not deallocate the buffers assigned to the
currently active TTTs at the time of task termination.

- Do not assign the outstanding TTT-LUN pairs to other
tasks on the matching session-connection until a "TMF
TTT-Buffer release event" although tasks that own the
TTTs are terminated.

- Do not Reject any Data-Out PDUs with an invalid
ITT-TTT combination if the combination has a buffer
assignment.  Place the inbound data in the assigned
buffer and ignore the data.

- The outstanding TTTs and the assigned buffers can be
freed only upon the corresponding "TMF TTT-Buffer
release event".  The event can be one of:
	a) Last Data-Out PDU with the F-bit set for that 
           TTT-LUN combination (not an ideal fit for
iSCSI/iSER)
	b) Implementation-specific timeout
	c) Connection/session termination

And on the initiator side, we have to specify that an
initiator must continue to respond to an outstanding
TTT via Data-out PDUs even after the tasks are
terminated - i.e. even after TMF Response with
"success" is received.

These are fairly invasive changes to current model. 
These *can* however be done.  Assume we have them all
specified as RFC 3720 "errata", how far off would then
the new multi-task abort model be?  I see only two
additional items remaining - sending an Async PDU, and
specifying one additional TMF TTT-Buffer Release
event:"d) Receiving a Nop-Out ack for the Async PDU"

Given all this, I still think the best course of
specifying this whole area is:

(1) Make two specific deltas to RFC 3720 model as
"errata" 
           (a) Delete 3rd-party CmdSN gap wait
           (b) Delete 3rd-party StatSN ack wait
    This would be the new default iSCSI behavior.

(2) Define the new multi-task abort model separately. 
This
    includes the above two deltas and the TTT-wait
delta.
    This will be the operational behavior only if 
    "FastMultiTaskAbort=Yes" is negotiated.

Mallikarjun

wrote:

> >> If the target breaks the buffer associations,
> does
> >> it really have to
> >> invalidate the TTT?
> >

---

From: William Studenmund <[email-masked]>
Date: Fri, 9 Dec 2005 21:06:19 -0800

> TTT-Buffer release event" although tasks that own the

These details assume a specific implementation. I favor mentioning them 
as a suggestion, but different implementations may or may not need 
them.

For instance, an implementation that deals with mbuf chains may not 
need the buffers anymore. Just notice that the PDU is destined for a 
dying TTT and drop the mbuf chain.

> buffer and ignore the data.

All we really need is for the PDUs to not be rejected. How exactly the 
target throws away the data is up to it. :-)

> 	c) Connection/session termination

As above, this is an implementation detail. A target can just as 
reasonably close all connections after sending a "closing connection, 
you can immediately log back in" event. It's not as efficient, but it 
works. :-)

I do, however, like the idea of mentioning an implementation-specific 
timeout.

> terminated - i.e. even after TMF Response with

I'm sorry, I only have been concerned about the inter-nexus delay. I 
think it's fine for the aborting/terminating session to have to be all 
cleaned up before proceeding. Thus the aborting session will not see 
the TMF response until after all its tasks have stopped.

Hmmm... Let me be firmer. I think we should leave the intra-nexus delay 
in place. Thus no session should have to sustain tasks after a 
"succesful" TMF Response.

> event:"d) Receiving a Nop-Out ack for the Async PDU"

This is a good question.

The difference is that, at least as I envision things, the initiators 
will need no changes. Initiators designed for the old model will still 
work with this intermediate one. Also, we will not be changing the 
on-the-wire packets.

I like the new model, I just think that we need to fix the old one in 
addition to adding it.

>     "FastMultiTaskAbort=Yes" is negotiated.

Well, we disagree. :-) What does everyone else think?

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 20 Dec 2005 15:51:46 -0800 (PST)

What I outlined calls out the RFC3720-legal
implementation aspects that will now be illegal (if we
"fix" the 3720-model beyond simple errata).  So some
implementation-specifics do become important in this
discussion.

> Well, we disagree. :-) What does everyone else

Yes, at this point, it'd be good to get feedback from
others on the spec approach.  My last email summarizes
the tactical approach I intend to take for the next
draft revision.

Thanks.

Mallikarjun

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 07 Dec 2005 12:43:54 -0800

The discussion prompted me to take a closer look at
the fast multi-task abort model. In principal, I agree
that the new async PDU helps the target and initiator
reach a common understanding of which tasks have
been cleared, and stop processing them.

However, I think the language of the proposal puts
too much knowledge in the target transport layer -
the language implies that the target transport layer
is managing the task set for a LUN - it is responsible
for determining which initiators have active tasks on
a specific LUN, and which tasks these are, what their
disposition is - and for sending a TMF response (and which
TMFs are supported and what the back-end impact of each is).

It is not so simple since the values of the TAS, TST and
Qerr bits really determine the actions.

There is some unavoidable interaction/modification of
the SCSI layer to handle this situation correctly -
because only the SCSI layer knows whether for the cleared tasks
a. the initiator is really not expecting anything
   (the nexus is the same as which received the tmf command)
b. should be sent an explicit status of ABORTED BY ANOTHER
   INITIATOR
c. No impact
d. Needs to be cleaned up through the asynch notification
   mechanism (different nexus, TST=0:Qerr=01b;TAS=0)
c. Whether the initiator is likely to be dead (LUN Reset
   with reserve/release model or equivalent behavior in
   persistent reservation model)

In other words, the asynch PDU must be explicitly triggered
by the SCSI layer on each impacted Nexus - if adding the
asynch PDU becomes an issues, we can use a new response
code which would be sent for every command. In any case,
the statsn acknowledgement rule should help the initiator
and target flush all impacted commands in the pipe, and
have a common understanding of what is impacted.

Bill makes a very valid point about the real world situation
of clusters. Shouldn't the SCSI layer be able to send a
response for the TMF when it has triggered the asynch PDU on
each of the Nexuses (is it Nexi?). The acknowledgement or
time-out is only for the purposes of recovering the buffers?

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of Mallikarjun C.
>Sent: Tuesday, December 06, 2005 3:13 PM
>To: IPS
>Subject: Re: [Ips] Fast multi-task abort model
>
>
>>Mark
>> the tcb as 
>Yahoo! DSL - Something to write home about. 
>Just $16.99/mo. or less. 
>dsl.yahoo.com 
>
>