Subject: [Ips] Clearing the outstanding TTTs

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 24 Oct 2005 12:49:19 -0700

Based on the recent list discussion, the implementer's draft requires 
the task set target behavior also for LU and target resets.  IOW, the 
new draft requires that the target must wait for all outstanding TTTs to 
be retired before the TMF is acted on.

Bill Galloway has pointed out to me that this language appears to 
conflict with RFC 3720 clearing effects table.  The RFC table (page 250, 
bottom of the table) currently says that all incomplete TTTs are cleared 
on any of the reset flavors.

I believe that this is a consistency issue that needs to be addressed in 
the new draft (sigh....), so in the absence of any objections, I would 
like to change the cell entries (in the IT column) for the three reset 
rows to "N", meaning TTTs are *not* automatically cleared.  They will 
however be "cleared" by regular means, i.e. on initiator's Data-Out 
sequence.

FYI.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
StorageWorks Division
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: William Studenmund <[email-masked]>
Date: Tue, 25 Oct 2005 14:13:04 -0700

On Oct 24, 2005, at 12:49 PM, Mallikarjun C. wrote:

> new draft requires that the target must wait for all outstanding TTTs 
> to be retired before the TMF is acted on.

To be honest, I am still concerned about this behavior. I don't fully 
see how a sudden abort will severely disrupt an initiator. Yes, there 
will be confusion, but the initiator should be able to cope.

My main concern is that I see the reset operations as being mostly used 
when something is wrong. Be it either failover or where something VERY 
ODD has happened and the initiator wants to reset everything to a known 
state.

As such, there may not be an initiator to retire these TTTs. In the 
case of failover, the initiator(s) other than the one issuing the TMF 
may be off-line or be suffering from a network partition issue. In the 
case of great confusion, an initiator may not know it needs to issue 
data to retire a TTT as it and the target are not on the same page 
anymore.

We have spoken of using timeouts as a way to address this. However this 
case would need timeouts to function correctly in what I see as a 
common use case, which we really haven't done before.

I do see one way a sudden abort could confuse a target, and I can offer 
text to address it. The case I see is where the sudden abort triggers 
SCSI status, the SCSI task is retired at the target, and then all of 
the in-flight Data-Out PDUs generate Rejects. If the SCSI status also 
cleared the task at the initiator, then the initiator will see rejects 
for Data-Outs for a task that has ended. Or worse yet, the ITT will get 
reused, and the initiator will see rejects for the old task and think 
they are for the new one.

My suggestion is that if a target immediately aborts a task (doesn't 
wait for TTT retirement) and there were any outstanding Target Tasks, 
the target MUST NOT consider the ITT unused until the initiator 
acknowledges receipt of the SCSI Command Response. Further, the target 
SHALL ignore all Data-Out PDUs for the task received after the abort. 
The initiator MUST NOT acknowledge receipt of status until it has 
stopped sending data to the target.

The biggest thing I see this text doing is it disentangles servicing 
the TMF and cleaning up state with the initiators. All the tasks will 
be aborted before the TMF is responded to, but the TMF need not wait 
for retirement of the TTTs.

> conflict with RFC 3720 clearing effects table.  The RFC table (page 
> 250, bottom of the table) currently says that all incomplete TTTs are 
> cleared on any of the reset flavors.
>
> I believe that this is a consistency issue that needs to be addressed 
> in the new draft (sigh....), so in the absence of any objections, I 
> would like to change the cell entries (in the IT column) for the three 
> reset rows to "N", meaning TTTs are *not* automatically cleared.  They 
> will however be "cleared" by regular means, i.e. on initiator's 
> Data-Out sequence.

I agree that the two should agree. If the wg doesn't like what I 
suggest above, please make this change.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 25 Oct 2005 17:20:47 -0700

Bill,

Thanks for the feedback.

 > acknowledges receipt of the SCSI Command Response.

That assumes though that there would be a Response.  In fact, the 
default in the case of task set aborts (unless the TAS bit is set to 1) 
is not to return a status for the aborted tasks.

 >Further, the target

The problem is that targets cannot really ignore the PDUs per se.  They 
have to be received, parsed and then discarded and/or rejected. In the 
case of iSCSI/iSER, a Data-Out  PDU (i.e. RDMA Read Response)to an 
invalid TTT (i.e. STag) will take the connection down.  Even in 
iSCSI-aware NICs, I assume receiving large Data-Out PDUs with stale TTTs 
is very disruptive to the buffer management.

 > stopped sending data to the target.

The problem we are grappling with is not so much the sending of data on 
the initiator end, but of the data in flight.  In multi-connection 
sessions (or worse, in multi-session TMF situations - consider Clear 
Task Set or LU Reset), there is really no good way to account for all 
Data-Out PDUs in flight unless the last PDU in each sequence is received.

Mallikarjun

Mallikarjun Chadalapaka

William Studenmund wrote:
> As such, there may not be an initiator to retire these TTTs. In the case 
> of failover, the initiator(s) other than the one issuing the TMF may be 
> off-line or be suffering from a network partition issue. In the case of 
> great confusion, an initiator may not know it needs to issue data to 
> retire a TTT as it and the target are not on the same page anymore.
> 
> case would need timeouts to function correctly in what I see as a common 
> use case, which we really haven't done before.
> 
> SCSI status, the SCSI task is retired at the target, and then all of the 
> in-flight Data-Out PDUs generate Rejects. If the SCSI status also 
> The biggest thing I see this text doing is it disentangles servicing the 
> TMF and cleaning up state with the initiators. All the tasks will be 
> aborted before the TMF is responded to, but the TMF need not wait for 
> retirement of the TTTs.
> 
> I agree that the two should agree. If the wg doesn't like what I suggest 
> above, please make this change.
> 
> Take care,
> 
> Bill

---

From: William Studenmund <[email-masked]>
Date: Wed, 26 Oct 2005 14:09:30 -0700

On Oct 25, 2005, at 5:20 PM, Mallikarjun C. wrote:

> default in the case of task set aborts (unless the TAS bit is set to 
> 1) is not to return a status for the aborted tasks.

Ahhh.. Yes. I was assuming a response. In the case of TAS set to 0, 
what you describe is probably best.

> The problem is that targets cannot really ignore the PDUs per se.  
> They have to be received, parsed and then discarded and/or rejected. 
> In the case of iSCSI/iSER, a Data-Out  PDU (i.e. RDMA Read Response)to 
> an invalid TTT (i.e. STag) will take the connection down.  Even in 
> iSCSI-aware NICs, I assume receiving large Data-Out PDUs with stale 
> TTTs is very disruptive to the buffer management.

Discarded is another way to say what I had in mind for ignore. All of 
the TTTs are flagged to discard arriving data, and do so either until 
the task is done or the initiator acknowledges the task has been 
aborted.

> The problem we are grappling with is not so much the sending of data 
> on the initiator end, but of the data in flight.  In multi-connection 

Agreed. My thought though is that with the "MUST" above is that 
acknowledging the abort is a definite, implicit acknowledgement of the 
termination of all TTTs on the original task.

For non-iSER, since the data are processed in-order, the 
acknowledgement serves as a barrier in the TCP stream. No data are 
in-flight for that task after the ack.

I'm glad you mentioned multi-session TMF situations as they are what 
drives my thoughts and concerns. :-)

I'm glad to keep talking. There may be another way to address my 
concerns, and I'd love to find it.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 31 Oct 2005 15:27:07 -0800

Bill,

I hope you are not suggesting conditioning iSCSI protocol behavior based 
on SCSI TAS bit setting.

Also, the process of discarding the Data-out PDUs itself transiently 
consumes some buffers.  And bad things happen for iSCSI/iSER if the 
tagged buffers are not available.  I do not want us to spec different 
task set behaviors for iSCSI-3720 and iSCSI/iSER.

Finally, we have to specify an approach that works for multi-connection 
sessions (where a Response/Ack etc. on one connection says nothing about 
data in flight on others), as well as with operations affecting SCSI 
objects accessible via multiple sessions (clear task set, LU reset etc.).

I would be willing to consider any proposal that meets these 
requirements.  The Implementers' guide details one such (arrived at 
after a lot of discussion on the list).  I am however not sure I 
understand your concerns per se.

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

William Studenmund wrote:
> Ahhh.. Yes. I was assuming a response. In the case of TAS set to 0, what 
> you describe is probably best.
> 
>> >Further, the target
> the task is done or the initiator acknowledges the task has been aborted.
> 
> For non-iSER, since the data are processed in-order, the acknowledgement 
> serves as a barrier in the TCP stream. No data are in-flight for that 
> task after the ack.
>

---

From: William Studenmund <[email-masked]>
Date: Mon, 31 Oct 2005 17:21:01 -0800

On Oct 31, 2005, at 3:27 PM, Mallikarjun C. wrote:

> Bill,
>
> I hope you are not suggesting conditioning iSCSI protocol behavior 
> based on SCSI TAS bit setting.

I am proposing a MAY. The main condition for the MAY is that there be a 
way to know if the initiator is aware of the abort. I do not see how it 
is onerous that TAS bit setting, which is one of the controls dictating 
if the initiator is aware of the abort, comes into play. iSCSI has 
spoken about things that SCSI has to support, such as autosense, so we 
have a precedent of one layer influencing another.

Since iSCSI is, strictly speaking, following SAM-2, we could use an 
Asynchronous Event containing a Unit Attention of COMMANDS CLEARED BY 
ANOTHER INITIATOR. We then use acknowelgement of the Async Event as 
indicating that all tasks on all sessions have stopped; unfortunately I 
doubt this will work for MC/S.

We already say you MUST use autosense and you SHOULD use ACA, maybe 
this discussion indicates LUs SHOULD initialize TAS to 1 on reset.

> task set behaviors for iSCSI-3720 and iSCSI/iSER.

I don't think iSCSI-3720 and iSCSI/iSER will have different task set 
behaviors with this proposal. I agree we should not lightly do that...

The idea is to give the initiator a second way to indicate that it 
agrees the task is dead. The motivation is that it is a way that does 
not require the Task Management Function to wait for the initiator to 
agree the task is dead; between the TMF deciding the task is dead and 
this acknowledgement, any received data will be discarded.

As for non-TCP transports, I think I can think of a few ways to make 
this work. These thoughts are still very rough. They would depend on 
the target's behavior, but since this is a MAY that the target can 
choose, it just avoids the choice if it can't make things work.

> Finally, we have to specify an approach that works for 
> multi-connection sessions (where a Response/Ack etc. on one connection 
> says nothing about data in flight on others), as well as with 
> operations affecting SCSI objects accessible via multiple sessions 
> (clear task set, LU reset etc.).

The multiple sessions case is EXACTLY why I want this.

As for MC/S, I agree that one connection says nothing about data in 
flight on others. But a given command is supposed to be allegiant to 
only one connection at a time, so that the target receiving StatSN 
acknowledgement of a tasks's (aborted) status will say something about 
all of the data for that given task. Iterate over all the tasks, and we 
will know that no data are in flight.

> understand your concerns per se.

If initiator 2 is using TMF LU Reset to clean up after initiator 1 
because initiator 1 caught on fire and exploded, how will initiator 1 
ever clear its outstanding TTTs?

That's my concern - waiting for the TTTs to clear assumes that they 
will actually clear.

I feel that these TMF actions are exactly the thing you want to use 
when things are broken and the TTTs WON'T clear, yet making them wait 
for clearing actually makes that impossible.

To fix this, we either depend on the target timing out the TTTs, the 
target does something we don't talk about yet (like close all the 
connections, thus terminating in-flight data), or the TMF LU reset 
waits forever (or until the repair person comes out and replaces 
initiator 1). We give no guidelines to targets or initiators about 
timeouts (which I think is a fine thing), yet I think this is a case 
where sanity and interoperability would really need it.

My desire is to provide a way, within the spec, to permit the TMF 
actions to complete in a timely manner and also maintain the 
requirement that no status for these aborted commands be generated 
after the TMF completes. And for all of this to happen when one or more 
of the initiators is slow or dead.

Another case where this can arise is when the iSCSI session goes over 
the public internet. We have had a few inter-continental deployments, 
and they give a different view on what exactly is timely behavior.

If we are in a data center, with good connectivity and active hosts, I 
agree that what you have proposed in this document is very nice. It's 
politer than just aborting, and if TAS is 0, it can be very important. 
My concern is what do we do if we have inactive hosts and/or bad 
connectivity.

Thank you for continuing to respond even when you don't understand why 
I'm making a fuss. I appreciate it. Also, you are more clever about 
these issues than I am, and a solution to this issue with your input 
will probably be much better than any without. ;-)

Take care,

Bill