Subject: RE: iscsi: rev 05 2.5.1 requires ACA support

From: [email-masked]
Date: Thu, 15 Mar 2001 21:29:02 +0200

Charles,

After an error if you have commands in flight you want them all dropped
until you specifically reset the ACA and restart the queue (prevent things
to be executed out of order).

Julo

"Charles Binford" <[email-masked]> on 15/03/2001
20:35:26

Please respond to "Charles Binford" <[email-masked]>

To:   "IPS Reflector" <[email-masked]>
cc:
Subject:  RE: iscsi: rev 05 2.5.1 requires ACA support

Ralph, you missed a reference in iSCSI where ACA is required.  In 7.2 (p
86)
it says:

  "ACA helps preserving ordered command execution
   in presence of errors. As iSCSI can have many commands
   in-flight between initiator and target iSCSI mandates
   support for ACA."

I agree with Ralph that iSCSI should not be mandating ACA support.
However,
I don't really understand what the problem/scenario is that ACA is supposed
to be solving.  Can someone please explain that to me?  Before I start
throwing more darts at ACA I'd like to better understand the perceived
value.

Charles Binford
Blue Spruce Networks
office: (316) 315-0382 / cell: (316) 210-6404
e-fax: (509) 756-4425

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Ralph Weber
Sent: Monday, March 12, 2001 10:17 PM
To: IPS Reflector
Subject: iscsi: rev 05 2.5.1 requires ACA support

The following two 2.5.1 statements require the use
of ACA and there is no way for an initiator that
doesn't implement ACA to stop them.

   For the <Clear Task Set>, ...
   The target MUST then enter the ACA state for any
   initiator for which it had pending tasks.

   In addition, for the <Target Warm Reset> the target
   enters the ACA state on all sessions and all LUs on
   which an AE was sent.

Have you all signed the operating system vendors up for
rewrites of their class drivers to support ACA?  If not
anticipate some serious deadlocks when iSCSI is first
attempted with a non ACA class driver, since the driver
will never clear these ACA conditions and no progress
will be made after that.

Alternatively, a way can be found to give the initiator
control over the "MUST" statements.

Thanks.

Ralph...

---

From: [email-masked]
Date: Fri, 16 Mar 2001 09:34:18 +0200

We are aware of the support for ACA missing from some drivers.
The situation is even exacerbated by the fact that certain exceptions that
are not errors per-se
will require ACA to be fired to accomodate for commands in flight (like
reservations, busy, task-set-full).

However actions at the initiator can be fine-tuned with the NACA bit in CDB
and the ACA atrribute for the task.

Julo

[email-masked] on 16/03/2001 03:46:29

Please respond to [email-masked]

To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
cc:
Subject:  RE: iscsi: rev 05 2.5.1 requires ACA support

> to be executed out of order).

The T10 folks will have to go after this one in more detail, but Julian's
above statement is correct *only* if the commands in flight depend on
the one that caused the error (i.e. executing them out of order will cause
problems).  This is generally not the case for disks where the usual
practice is to enforce command execution order dependencies
(e.g., database write ordering) in the operating system and applications
by waiting for responses (yes it's possible to do better, but lots of
existing software doesn't).  The result is that commands in
flight can be executed in arbitrary order with arbitrary ones of them
failing without causing further difficulties.  As Ralph has pointed out,
most hosts do not use ACA for disk-based storage, and if iSCSI
always does ACA, this will cause nasty integration issues.

Just to stir the pot further, I believe Fibre Channel provides a negative
example, because if FC drops a frame (which is not a good idea,
but can still happen), the FC component that dropped the frame has no
clue about what ACA is, or how to get the target (which is not the
same piece of hardware) to enter ACA state.  Both Class 2 and Class 3
are vulnerable to this.

Tapes are another matter - do we still have a tape expert on the list?

I thought where we were headed on ACA was something along the
lines of:
- Targets MUST implement.
- Initiators MAY use.
- Initiator support for ACA is NOT REQUIRED.
which would imply a text key for Initiators to tell Targets
whether ACA behavior is expected.  Did I miss something?

--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

---

From: [email-masked]
Date: Fri, 16 Mar 2001 09:21:53 -0500

If that's the case, then the wording that Ralph
pointed out needs to be modified to indicate that
ACA is used only when appropriate.

--David

> From:	[email-masked] [SMTP:[email-masked]]
> Sent:	Friday, March 16, 2001 2:34 AM
> To:	[email-masked]

---

From: "Prasenjit Sarkar" <[email-masked]>
Date: Fri, 16 Mar 2001 09:24:59 -0800

Hi David and Julian,

To impart some finality into the discussion, can you summarize for
the benefit of the mailing list what is distinctive about iSCSI that ACA
must be made mandatory for targets. I'm not trying to start an argument
here, but a few words of motivation might help us to understand
as to why iSCSI-ACA must be made manadatory (required in most situations)
versus optional (implement where needed).

Regards,
Prasenjit

   Prasenjit Sarkar
   Research Staff Member
   IBM Almaden Research
   San Jose

                    Black_David@em                                                            
                    c.com                To:     Julian Satran/Haifa/IBM@IBMIL,               
                    Sent by:              [email-masked]                                     
                    owner-ips@ece.       cc:                                                  
                    cmu.edu              Subject:     RE: iscsi: rev 05 2.5.1 requires ACA    
                                          support                                             
                                                                                              
                    03/16/2001                                                                
                    06:21 AM                                                                  

--David

---

From: [email-masked]
Date: Fri, 16 Mar 2001 19:43:21 -0500

> as to why iSCSI-ACA must be made manadatory (required in most
> situations) versus optional (implement where needed).

Actually there are (at least) two issues underlying
Prasenjit's simple question -- this whole area of
target response to errors feels like the old Adventure
"maze of little twisty passages, all different" :-).

Issue 1: Should ACA be REQUIRED, RECOMMENDED, or
	OPTIONAL to implement?
	- Independently for Initiators and Targets.
Issue 2: Should ACA be REQUIRED, RECOMMENDED, or
	OPTIONAL to use?
	- Basically an Initiator issue, as SCSI puts the
		Initiator in charge of deciding whether to
		use ACA.

I'm not sure Julian and I see eye-to-eye on this, so
here are my current opinions:

Issue 1, Initiators - ACA should be OPTIONAL to implement
Issue 1, Targets - ACA should be REQUIRED or RECOMMENDED
	to be implemented (note, this is a slight change from
	my earlier email).
Issue 2 - ACA should OPTIONAL to use, Initiator makes the
	choice in each case.

So, Prasenjit's question to me is "Why should ACA be
	REQUIRED or RECOMMENDED to implement in targets?".

ACA is specified in SAM, and T10 therefore requires that
any transport mapping of SAM specify how ACA is implemented.
Hence it has to be specified for iSCSI, lest we get dinged
by T10 for failing to handle all SCSI features.

This leaves the choice of REQUIRED, RECOMMENDED or OPTIONAL
for target implementation.  My concern is based on looking
forward to the point in time where we work on moving iSCSI
from Proposed Standard to Draft Standard.  At that
juncture, features that aren't implemented will be removed
from the specification (retaining feature X requires at
least two independently developed interoperable
implementations of X).  If ACA is removed at that point,
iSCSI will get dinged by T10 for no longer being a
complete mapping of SCSI, which would be bad.

Hence, the word we choose should lead to multiple
implementations of ACA that can be tested for
interoperability, even if ACA is rarely used in practice.
The removal of features for which interoperability
cannot be demonstrated as part of advancing from
Proposed Standard to Draft Standard is distinctive
to iSCSI by comparison to other SCSI transport mappings.

OPTIONAL strikes me as the wrong choice.  I'd like to
avoid a situation in which iSCSI can't move to Draft
Standard because nobody's implemented an OPTIONAL to
implement feature -- IMHO, that sounds really silly.
REQUIRED seems most likely to lead to implementations.
RECOMMENDED would also be ok.

Sorry to have to give such a long answer to a short
question,

--David

---

From: Charles Monia <[email-masked]>
Date: Fri, 16 Mar 2001 20:03:14 -0800

Hi:

I though the action items in the following extracts from the T10 CAP minutes
were intended to address this issue.  What happened?

From T10 SCSI Commands, Architecture, & Protocol Working Group Meeting
Minutes --
January 17, 2001 T10/01-007r1

"4.4 Queue Locking for Busy, Task-Queue-Full, & Reservation Active [Satran]

"Julian Satran presented a discussion regarding the lack of ACA-like
interlocks on the BUSY, TASK SET FULL, and RESERVATION CONFLICT status codes
(01-048r0). The group tentative agreed to resolve the issue by providing an
initiator selectable way to make BUSY, TASK SET FULL, and RESERVATION
CONFLICT status codes translate to CHECK CONDITION status with informative
sense data. Ed Gardner agreed incorporate the tentative resolution in the
next revision of his proposal (see agenda item 5.5)."

"5.5 Unit attention issue (00-359) [Gardner]

"Ed Gardner described a problem where in autosense clears a unit attention
condition (00-359r1). Basically, the combination of commands in flight and
autosense results in a condition where the initiator may not be able to stop
commands that should be stopped based on unit attention conditions. Ed asked
for recommendations on how the
initiator should indicate that the new feature was being requested
(recognizing that there must be a way to maintain the current behavior).
Options discussed were Control byte (plus INQUIRY bit), Control mode page,
and protocol specific. The group recommended using the Control mode page and
Ed agreed to bring a specific proposal based\ on that recommendation to the
next meeting."

Charles

> From: Charles Binford [mailto:[email-masked]]
> Sent: Friday, March 16, 2001 11:57 AM
> To: [email-masked]
> (sorry this got kind of lengthy - but I do have proposed 
> solution to what I
> believe are the problems below)
> 
> Julian,
> I don't think iSCSI needs to look at the NACA bit in the CDB 
> - I don't think
> iSCSI should care about ACA at all.
> 
> In my 10+ years of working with SCSI disk products I've yet 
> to see a host
> driver that cared about ordering.  As David said below, if there is an
> ordering dependency then the application waits for completion 
> of the first
> I/O before releasing the second.  One gets the best 
> performance from a SCSI
> block I/O target (from a single disk to a large RAID 
> controller) by issuing
> queued commands with simple tags and allowing the target to 
> freely re-order
> the commands internally.  When an error happens it doesn't matter what
> commands are in-flight because they are all independent I/Os. 
>  This is the
> way high performance SCSI disks and disk subsystems work.  It 
> would be a
> mistake (in my opinion) for iSCSI to come in and mandate the extra
> complexity of ACA to handle a corner case most of the SCSI world it is
> trying to penetrate doesn't care about.
> 
> Granted, tape and other sequential devices are a different 
> story.  However,
> I participated heavily in T11 for 2 to 3 years helping develop a error
> recovery scheme that would allow SCSI tape devices to work 
> with FC.  The
> bottom line, however, was that while we talked about it over 
> and over, no FC
> tape vendor had a device that supported queuing.  If you 
> don't queue, then
> nothing is in-flight when an error occurs.  (streaming is 
> accomplished by
> caching in the target).  After this drug on for a couple of 
> years and the
> complexities of queuing to sequential devices became evident, 
> a new project
> was started in T10 to develop a new tape command set that 
> included relative
> block addresses in the CDB - making the new tape model very similar to
> disk - each command can be interpreted independently and 
> precise delivery
> order is not a fundamental requirement.
> 
> ===================

---

From: [email-masked]
Date: Sat, 17 Mar 2001 18:42:52 +0200

Prasenjit,

I think that David gave you a fair summary of the situation.  The only
think that I am uneasy about is that
we (iSCSI) have direct way of controlling it.  We require a target to
support ACA because we know that
on iSCSI connections there may be commands in flight. We have nothing to do
with the way the SCSI initiator is going to use this (if at all). An iSCSI
initiator is unaware of the ACA support.  It would have been cleaner if SAM
had something to say about high latency delivery subsystems and the
requirements those create on targets and initiators.

Regards,
Julo

From: Prasenjit Sarkar@IBMUS on 16/03/2001 19:24

To:   [email-masked]
cc:   [email-masked], Julian Satran/Haifa/IBM@IBMIL, [email-masked]
From: Prasenjit Sarkar/Almaden/IBM@IBMUS
Subject:  RE: iscsi: rev 05 2.5.1 requires ACA support  (Document link:
      Julian Satran - Mail)

Hi David and Julian,

Regards,
Prasenjit

   Prasenjit Sarkar
   Research Staff Member
   IBM Almaden Research
   San Jose

|--------+------------------------>
|        |          Black_David@em|
|        |          c.com         |
|        |          Sent by:      |
|        |          owner-ips@ece.|
|        |          cmu.edu       |
|        |                        |
|        |                        |
|        |          03/16/2001    |
|        |          06:21 AM      |
|        |                        |
|--------+------------------------>
  >----------------------------------------------------|
  |                                                    |
  |      To:     Julian Satran/Haifa/IBM@IBMIL,        |
  |       [email-masked]                              |
  |      cc:                                           |
  |      Subject:     RE: iscsi: rev 05 2.5.1 requires |
  |       ACA support                                  |
  |                                                    |
  |                                                    |
  >----------------------------------------------------|

--David