Subject: RE: [Ips] Response Fence Flag

From: "Robert Snively" <[email-masked]>
Date: Thu, 4 Jan 2007 10:01:36 -0800

The problem is that the definition of "previous" in this context is most
likely flawed.
 
The "Response Fence" is merely an internal and untestable state machine
input that
attempts to order the transmission of responses in the target port's
queue.
It  does nothing to improve the overall ordering of the responses at the
application
client.
 
Drivers for generic SCSI transports must always be aware that requested
operations
report completion with a more or less arbitrary time relationship to
each other and must be
designed to tolerate that.  
 
Using LOGICAL UNIT RESET as an example, the operation does not act only
on
the device server.  As it passes down through the initiator stack, it
may clean out
previously offered commands that have not yet been transported.  Then as
it
passes up through the target stack, it may clean out previously offered
commands
that have not yet been operated on.  As it passes through the device
server
execution path, it may clean out previously offered commands that have
begun
execution and some that have completed execution without yet responding.
As its 
response passes back down the target stack, it may clean up commands
that have completed for which the response has not yet been transmitted,
and as
it passes back up the initiator stack, it may clean up commands for
which the
response has been transmitted, but not yet been made available to the
driver or
application.  SAM-4 attempts to define the passage through the target
port,
through the device server, and back to the target port.  However, beyond
the
initiator port is outside the SAM-4 scope, and even where it is defined,
the "shall ensure"s are unlikely to be testable and in some cases not
implementable. 
 
Will I sometimes get command complete indications for commands after
a LOGICAL UNIT RESET response is received?  You bet.  Response Fence
doesn't
change that.
 
Will I sometimes get command aborted indications for commands after a 
LOGICAL UNIT RESET is response is received?  You bet.  Response Fence
doesn't change that.
 
Is this a problem?  It never has been in the past, why is it now?
 
Can you test the corner conditions for a fence operation?  I doubt it.
The
existence of the "Response Fence" argument is internal to the target
implementation and invisible to testing.
 
Bob

________________________________

From: Elliott, Robert (Server Storage) [mailto:[email-masked]] 
Sent: Wednesday, January 03, 2007 3:35 PM
To: [email-masked]
Subject: RE: [Ips] Response Fence Flag

SAM-3 and SAM-4 require all transport protocols to tag the TMF responses
with the requests:

"Each SCSI transport protocol shall allow a Received Task Management
Function Executed confirming completion of the requested task to be
associated with the corresponding Send Task Management Request."

Although iSCSI was based on SAM-2, it complies with that rule. The Fence
concept is asking for more than that - it wants the target to be able to
ensure that all previous commands/TMFs are complete before delivering a
particular TMF response (e.g., for a LOGICAL UNIT RESET).  Since iSCSI
doesn't have ACKs, the target must wait for the next PDU from the
initiator with an updated ExpStatSN.

06-341r0 discussed an alternative - add a Delivery Result output to the
Send Command Complete and Task Management Function Executed
confirmations (as previously proposed in 04-072r0 for a different
purpose). This would let the device server/task manager wait for all the
previous commands/TMF responses to complete (be ACKed) before proceeding
to make additional calls (e.g., Task Management Function Executed for a
LOGICAL UNIT RESET).

However, iSCSI allows the target port to send a NOP-IN to force delivery
of a NOP-OUT with ExpStatSN, rather than passively wait for a PDU to
show up.  The device server/task manager must instruct the target port
when to do this, and the Request Fence argument serves that purpose.
Target ports using protocols without such a force mechanism are still OK
- they will just wait for protocol-specific delivery confirmations (e.g.
ACKs).

-- 
Rob Elliott, [email-masked] 
Hewlett-Packard Industry Standard Server Storage Advanced Technology 
https://ecardfile.com/id/RobElliott 

________________________________

	From: Robert Snively [mailto:[email-masked]] 
	Sent: Wednesday, January 03, 2007 4:46 PM
	To: Elliott, Robert (Server Storage); [email-masked]
	Subject: RE: [Ips] Response Fence Flag

	I need a little tutorial here.  Note that this is an
architectural hack that
	is very "un-SCSI" and is not required if the response is
qualified with identification
	information associated with the original request.  I know that
such qualification
	exists for commands, but does it exist for task management
requests in
	iSCSI?  
	 
	I know that it does exist for parallel SCSI (which is strictly
interlocked)
	and for SAS and Fibre Channel SCSI, making such "fences"
unrequired by
	those technologies.
	 
	Note that Rob's previous revision of 06-341 is available on
www.t10.org.
	 
	I would hate to see such a hack creep into the SCSI
architecture.
	 
	Bob

________________________________

	From: Elliott, Robert (Server Storage) [mailto:[email-masked]] 
	Sent: Wednesday, January 03, 2007 1:56 PM
	To: [email-masked]
	Subject: RE: [Ips] Response Fence Flag

	If T10 agrees, T10 proposal 06-341 will add it to SAM-4.
	 
	-- 
	Rob Elliott, [email-masked] 
	Hewlett-Packard Industry Standard Server Storage Advanced
Technology 
	https://ecardfile.com/id/RobElliott 

________________________________

		From: Eddy Quicksall
[mailto:[email-masked]] 
		Sent: Wednesday, January 03, 2007 2:00 PM
		To: [email-masked]
		Subject: [Ips] Response Fence Flag

		Section 3.3.1 talks about a Response Fence Flag:
		 
		SCSI protocol layer instructs the SCSI transport layer
of a 
		"Response Fence" associated with the response in
question when 
		the "Send Command Complete" protocol data service
(SAM-2, clause 
		5.4.2) ...
		 
		But I don't see any reference to that in SAM-2.
		 
		Is this strictly an iSCSI flag? Where is the flag
specified?
		 
		Eddy

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Eddy Quicksall" <[email-masked]>
Date: Fri, 5 Jan 2007 11:58:47 -0500

>From an earlier email I think that Response Fence is only a proposal in T10 (http://www.t10.org:80/doc06.htm). If so shouldn't iSCSI wait a bit until this has been ratified?

Eddy
  ----- Original Message ----- 
  From: Robert Snively 
  To: Elliott, Robert (Server Storage) ; [email-masked] 
  Sent: Thursday, January 04, 2007 1:01 PM
  Subject: RE: [Ips] Response Fence Flag

  The problem is that the definition of "previous" in this context is most likely flawed.

  The "Response Fence" is merely an internal and untestable state machine input that
  attempts to order the transmission of responses in the target port's queue.
  It  does nothing to improve the overall ordering of the responses at the application
  client.

  Drivers for generic SCSI transports must always be aware that requested operations
  report completion with a more or less arbitrary time relationship to each other and must be
  designed to tolerate that.  

  Using LOGICAL UNIT RESET as an example, the operation does not act only on
  the device server.  As it passes down through the initiator stack, it may clean out
  previously offered commands that have not yet been transported.  Then as it
  passes up through the target stack, it may clean out previously offered commands
  that have not yet been operated on.  As it passes through the device server
  execution path, it may clean out previously offered commands that have begun
  execution and some that have completed execution without yet responding.  As its 
  response passes back down the target stack, it may clean up commands
  that have completed for which the response has not yet been transmitted, and as
  it passes back up the initiator stack, it may clean up commands for which the
  response has been transmitted, but not yet been made available to the driver or
  application.  SAM-4 attempts to define the passage through the target port,
  through the device server, and back to the target port.  However, beyond the
  initiator port is outside the SAM-4 scope, and even where it is defined,
  the "shall ensure"s are unlikely to be testable and in some cases not implementable. 

  Will I sometimes get command complete indications for commands after
  a LOGICAL UNIT RESET response is received?  You bet.  Response Fence doesn't
  change that.

  Can you test the corner conditions for a fence operation?  I doubt it.  The
  existence of the "Response Fence" argument is internal to the target
  implementation and invisible to testing.

  Bob

------------------------------------------------------------------------------
  From: Elliott, Robert (Server Storage) [mailto:[email-masked]] 
  Sent: Wednesday, January 03, 2007 3:35 PM
  To: [email-masked]
  Subject: RE: [Ips] Response Fence Flag

  SAM-3 and SAM-4 require all transport protocols to tag the TMF responses with the requests:

  "Each SCSI transport protocol shall allow a Received Task Management Function Executed confirming completion of the requested task to be associated with the corresponding Send Task Management Request."

  Although iSCSI was based on SAM-2, it complies with that rule. The Fence concept is asking for more than that - it wants the target to be able to ensure that all previous commands/TMFs are complete before delivering a particular TMF response (e.g., for a LOGICAL UNIT RESET).  Since iSCSI doesn't have ACKs, the target must wait for the next PDU from the initiator with an updated ExpStatSN.

  06-341r0 discussed an alternative - add a Delivery Result output to the Send Command Complete and Task Management Function Executed confirmations (as previously proposed in 04-072r0 for a different purpose). This would let the device server/task manager wait for all the previous commands/TMF responses to complete (be ACKed) before proceeding to make additional calls (e.g., Task Management Function Executed for a LOGICAL UNIT RESET).

  However, iSCSI allows the target port to send a NOP-IN to force delivery of a NOP-OUT with ExpStatSN, rather than passively wait for a PDU to show up.  The device server/task manager must instruct the target port when to do this, and the Request Fence argument serves that purpose.  Target ports using protocols without such a force mechanism are still OK - they will just wait for protocol-specific delivery confirmations (e.g. ACKs).

   

----------------------------------------------------------------------------

    I need a little tutorial here.  Note that this is an architectural hack that
    is very "un-SCSI" and is not required if the response is qualified with identification
    information associated with the original request.  I know that such qualification
    exists for commands, but does it exist for task management requests in
    iSCSI?  

    I know that it does exist for parallel SCSI (which is strictly interlocked)
    and for SAS and Fibre Channel SCSI, making such "fences" unrequired by
    those technologies.

    Note that Rob's previous revision of 06-341 is available on www.t10.org.

    I would hate to see such a hack creep into the SCSI architecture.

    Bob

----------------------------------------------------------------------------
    From: Elliott, Robert (Server Storage) [mailto:[email-masked]] 
    Sent: Wednesday, January 03, 2007 1:56 PM
    To: [email-masked]
    Subject: RE: [Ips] Response Fence Flag

--------------------------------------------------------------------------
      From: Eddy Quicksall [mailto:[email-masked]] 
      Sent: Wednesday, January 03, 2007 2:00 PM
      To: [email-masked]
      Subject: [Ips] Response Fence Flag

      SCSI protocol layer instructs the SCSI transport layer of a 
      "Response Fence" associated with the response in question when 
      the "Send Command Complete" protocol data service (SAM-2, clause 
      5.4.2) ...

      Is this strictly an iSCSI flag? Where is the flag specified?

      Eddy

------------------------------------------------------------------------------