Subject: [Ips] Re: Last Call Response Fence resolution

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 26 Jan 2007 14:11:46 -0800 (PST)

David,

OK.  Didn't realize you were concerned about "flag", we may be converging.  Please take a look at the new text below.....

Mallikarjun

----- Original Message ----
From: "[email-masked]" <[email-masked]>
To: [email-masked]; [email-masked]
Sent: Thursday, January 25, 2007 6:44:53 PM
Subject: RE: Last Call Response Fence resolution (Was Re: [Ips] iSCSIImplementer's Guide - WG Last Call status, 2nd try)

Mallikarjun,

> I am trying to preserve the notion of an iSCSI-defined & 
> iSCSI-scoped Response Fence in the text (you seem to be OK 
> with it).  This gives us a convenient shorthand to refer to a 
> set of semantics.  Besides, if/when T10 approves the response 
> fence concept for SAM-4, iSCSI implementers will know what to 
> map it to for iSCSI.  As you'll see, I made it very clear in 
> the new text that this is not an attempt to change SAM in 
> this draft (which was of course never the intent).

I agree in principle with the intent where "Response Fence" is
iSCSI-scoped and defined "behavior."  OTOH, calling it a "flag"
is a problem.  I would prefer that all text anticipating
addition of a "flag" to SAM-4 be removed, and in particular
the following text:

> the following 
> semantics are defined for a "Response Fence" flag if such a 
> flag were to qualify SCSI completion messages as they are 
> handed off from the SCSI protocol layer to the iSCSI layer.  
> Note however that [SAM2] does not model such a flag in its 
> protocol interface model, so the "Response Fence" flag is an 
> iSCSI-defined artifact for descriptive convenience of the 
> mandatory iSCSI response ordering semantics.

which is just asking for an IETF Last Call comment or AD question
about why this circumlocution was used when it suffices to define
the "Response Fence" behavior without ever mentioning a non-
existent flag.  Section 3.3.3 doesn't need the flag - it can
refer to the behavior and say what it MUST be applied to.

If Rob succeeds in getting T10 to put a Response Fence flag
into SAM-4, we can deal with that flag in the draft that
deals with the impacts of SAM-3 and SAM-4 on iSCSI, which
will have a number of other things to deal with.  Before anyone
gets excited about this, I will state my view that the IETF
will need to ensure that all such impacts are specified in a
fashion that is backwards compatible with RFC-3720-only
implementations.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

3.3 Model Assumptions for Response Ordering
Whenever an iSCSI session is composed of multiple connections, the Response PDUs (task responses or TMF responses) originating in the target SCSI layer are distributed onto the multiple connections by the target iSCSI layer according to iSCSI connection allegiance rules.  This process generally may not preserve the ordering of the responses by the time they are delivered to the initiator SCSI layer.  Since ordering is not expected across SCSI responses anyway, this approach works fine in the general case.  However to address the special cases where some ordering is desired by the SCSI layer, the following "Response Fence" semantics are defined with respect to handling SCSI response messages as they are handed off from the SCSI protocol layer to the iSCSI layer. 
 
3.3.1 Model Description
Target SCSI protocol layer hands off the SCSI response messages to the target iSCSI layer by invoking the "Send Command Complete" protocol data service ([SAM2], clause 5.4.2) and "Task Management Function Executed" ([SAM2], clause 6.9) service.   On receiving the SCSI response message, iSCSI layer exhibits the Response Fence behavior for certain SCSI response messages (section 3.3.3 describes the specific instances where the semantics must be realized).  Whenever the Response Fence behavior is required for a SCSI response message, the target iSCSI layer MUST ensure that the following conditions are met in delivering the response message to the initiator iSCSI layer:
(1) Response with Response Fence MUST chronologically be delivered after all the "preceding" responses on the I_T_L nexus, if the preceding responses are delivered at all, to the application client on the initiator. 
(2) Response with Response Fence MUST chronologically be delivered prior to all the "following" responses on the I_T_L nexus. 
The “preceding” and “following” notions refer to the order of hand-off of a response message from the target SCSI protocol layer to the target iSCSI layer.

3.3.2 iSCSI Semantics with the Interface Model
Whenever the TaskReporting key (section 9.1) is negotiated to ResponseFence or FastAbort for an iSCSI session and the Response Fence behavior is required for a SCSI response message, the target iSCSI layer MUST perform the actions described in this section for that session.:
a) If it is a single-connection session, no special processing is required.  Standard SCSI Response PDU build and dispatch process happens. 
b) If it is a multi-connection session, target iSCSI layer takes note of last-sent and unacknowledged StatSN on each of the connections in the iSCSI session, and waits for acknowledgement (SHOULD solicit for acknowledgement by way of a Nop-In) of each such StatSN to clear the fence.  SCSI response with the Response Fence flag must be sent to the initiator only after receiving acknowledgements for each of the unacknowledged StatSNs.
c) Target iSCSI layer must wait for an acknowledgement of the SCSI Response PDU that carried the response which the target SCSI layer marked with the Response Fence flag.  The fence must be considered cleared after receiving the acknowledgement.
d) All further status processing for the LU is resumed only after clearing the fence.  If any new responses for the I_T_L nexus are received from the SCSI layer before the fence is cleared, those Response PDUs must be held and queued at the iSCSI layer until the fence is cleared.

3.3.3 Current List of Fenced Response Use Cases

--- No changes ----

____________________________________________________________________________________
Be a PS3 game guru.
Get your game face on with the latest PS3 news and previews at Yahoo! Games.
http://videogames.yahoo.com/platform?platform=120121

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Robert Snively" <[email-masked]>
Date: Fri, 26 Jan 2007 14:30:08 -0800

In the text below, there is one thing I am not sure
I understand correctly.  The text in 3.3.1, item 1 (and
implicitly item 2) says that "MUST be chronologically
delivered after all the preceding responses... to the
application client on the initiator.

Similarly in 3.3.2 item b) says "target iSCSI layer 
takes note of last-sent and unacknowledged StatSN 
on each of the connections in the iSCSI session, and 
waits for acknowledgement (SHOULD solicit for 
acknowledgement by way of a Nop-In) of each such 
StatSN to clear the fence."

Implicit in 3.3.2 is knowledge of the timing of delivery
to the iSCSI layer, not necessarily to the "application
client".  I do not believe there is any defined time marker
or pollable status available to a target that indicates when 
delivery to the "application client" actually occurs that
could be used to satisfy 3.3.1.  This is the
generic ordering question at the root of my continued
concern about this approach.

Since software constituting the application client (including
kernel, OS, file system, application, etc.) must be designed
to be tolerant of this ambiguity in any case, I believe we
are spending a lot of effort creating an ordered case where
the rare cases of misordering are already managed behavior.

Bob

-----  original text offered by Mallikarjun  -----------------
 
3.3.1 Model Description
Target SCSI protocol layer hands off the SCSI response messages to the
target iSCSI layer by invoking the "Send Command Complete" protocol data
service ([SAM2], clause 5.4.2) and "Task Management Function Executed"
([SAM2], clause 6.9) service.   On receiving the SCSI response message,
iSCSI layer exhibits the Response Fence behavior for certain SCSI
response messages (section 3.3.3 describes the specific instances where
the semantics must be realized).  Whenever the Response Fence behavior
is required for a SCSI response message, the target iSCSI layer MUST
ensure that the following conditions are met in delivering the response
message to the initiator iSCSI layer:
(1) Response with Response Fence MUST chronologically be delivered after
all the "preceding" responses on the I_T_L nexus, if the preceding
responses are delivered at all, to the application client on the
initiator. 
(2) Response with Response Fence MUST chronologically be delivered prior
to all the "following" responses on the I_T_L nexus. 
The "preceding" and "following" notions refer to the order of hand-off
of a response message from the target SCSI protocol layer to the target
iSCSI layer.

3.3.2 iSCSI Semantics with the Interface Model
Whenever the TaskReporting key (section 9.1) is negotiated to
ResponseFence or FastAbort for an iSCSI session and the Response Fence
behavior is required for a SCSI response message, the target iSCSI layer
MUST perform the actions described in this section for that session.:
a) If it is a single-connection session, no special processing is
required.  Standard SCSI Response PDU build and dispatch process
happens. 
b) If it is a multi-connection session, target iSCSI layer takes note of
last-sent and unacknowledged StatSN on each of the connections in the
iSCSI session, and waits for acknowledgement (SHOULD solicit for
acknowledgement by way of a Nop-In) of each such StatSN to clear the
fence.  SCSI response with the Response Fence flag must be sent to the
initiator only after receiving acknowledgements for each of the
unacknowledged StatSNs.
c) Target iSCSI layer must wait for an acknowledgement of the SCSI
Response PDU that carried the response which the target SCSI layer
marked with the Response Fence flag.  The fence must be considered
cleared after receiving the acknowledgement.
d) All further status processing for the LU is resumed only after
clearing the fence.  If any new responses for the I_T_L nexus are
received from the SCSI layer before the fence is cleared, those Response
PDUs must be held and queued at the iSCSI layer until the fence is
cleared.

---

From: [email-masked]
Date: Fri, 26 Jan 2007 17:57:36 -0500

Bob,

> concern about this approach.

For 3.3.2, I don't think that's not correct, at least not to
the degree you've stretched it.  What's going on is an assumption
that any local delivery event of interest will have occurred
well before the round trip from initiator to target (acknowledge
status) and back (target does something as a consequence of that
acknowledgement).  Based on that assumption, the target's record
of acknowledged status is being used to drive the sequencing.

If one is in a situation where the assumption is false (e.g.,
app client is a low priority user process for which the OS does
not maintain an ordered event queue, allowing event reordering
over periods of time on the order of an iSCSI round trip),
then I would actually recommend that multi-connection iSCSI
sessions not be used, because the corresponding reordering
of commands is going to cause target-side holes in the CmdSN
sequence.  I also believe the example I used is indicative
of an OS shortcoming, and is not at all typical.  For typical
situations, I believe the assumption holds, the fence property
is useful, and the "pollable status" that drives everything is
the target's tracking of the acknowledged StatSN's.  iSCSI
doesn't use time markers.

Thanks,
--David

> -----Original Message-----
> From: Robert Snively [mailto:[email-masked]] 
> Sent: Friday, January 26, 2007 5:30 PM
> To: Mallikarjun C.; [email-masked]
> Subject: RE: [Ips] Re: Last Call Response Fence resolution
>

---

From: [email-masked]
Date: Fri, 26 Jan 2007 20:56:32 -0500

I think that text is ok, as the remaining SAM2 references serve
as justification for talking about the interface between SCSI
and iSCSI.

Thanks,
--David

> From: Mallikarjun C. [mailto:[email-masked]] 
> Sent: Friday, January 26, 2007 5:12 PM
> To: [email-masked]
> Subject: [Ips] Re: Last Call Response Fence resolution
> 
> David,
> 
> OK.  Didn't realize you were concerned about "flag", we may 
> be converging.  Please take a look at the new text below.....
> 
> Mallikarjun
> 
> Subject: RE: Last Call Response Fence resolution (Was Re: 
> [Ips] iSCSIImplementer's Guide - WG Last Call status, 2nd try)
> 
> 
> Mallikarjun,
> 
> Whenever an iSCSI session is composed of multiple 
> connections, the Response PDUs (task responses or TMF 
> responses) originating in the target SCSI layer are 
> distributed onto the multiple connections by the target iSCSI 
> layer according to iSCSI connection allegiance rules.  This 
> process generally may not preserve the ordering of the 
> responses by the time they are delivered to the initiator 
> SCSI layer.  Since ordering is not expected across SCSI 
> responses anyway, this approach works fine in the general 
> case.  However to address the special cases where some 
> ordering is desired by the SCSI layer, the following 
> "Response Fence" semantics are defined with respect to 
> handling SCSI response messages as they are handed off from 
> the SCSI protocol layer to the iSCSI layer. 
>  
> 3.3.1 Model Description
> Target SCSI protocol layer hands off the SCSI response 
> messages to the target iSCSI layer by invoking the "Send 
> Command Complete" protocol data service ([SAM2], clause 
> 5.4.2) and "Task Management Function Executed" ([SAM2], 
> clause 6.9) service.   On receiving the SCSI response 
> message, iSCSI layer exhibits the Response Fence behavior for 
> certain SCSI response messages (section 3.3.3 describes the 
> specific instances where the semantics must be realized).  
> Whenever the Response Fence behavior is required for a SCSI 
> response message, the target iSCSI layer MUST ensure that the 
> following conditions are met in delivering the response 
> (1) Response with Response Fence MUST chronologically be 
> delivered after all the "preceding" responses on the I_T_L 
> nexus, if the preceding responses are delivered at all, to 
> the application client on the initiator. 
> (2) Response with Response Fence MUST chronologically be 
> delivered prior to all the "following" responses on the I_T_L nexus. 
> The "preceding" and "following" notions refer to the order of 
> hand-off of a response message from the target SCSI protocol 
> layer to the target iSCSI layer.
> 
> ResponseFence or FastAbort for an iSCSI session and the 
> Response Fence behavior is required for a SCSI response 
> message, the target iSCSI layer MUST perform the actions 
> described in this section for that session.:
> a) If it is a single-connection session, no special 
> processing is required.  Standard SCSI Response PDU build and 
> dispatch process happens. 
> b) If it is a multi-connection session, target iSCSI layer 
> takes note of last-sent and unacknowledged StatSN on each of 
> the connections in the iSCSI session, and waits for 
> acknowledgement (SHOULD solicit for acknowledgement by way of 
> a Nop-In) of each such StatSN to clear the fence.  SCSI 
> response with the Response Fence flag must be sent to the 
> initiator only after receiving acknowledgements for each of 
> the unacknowledged StatSNs.
> c) Target iSCSI layer must wait for an acknowledgement of the 
> SCSI Response PDU that carried the response which the target 
> SCSI layer marked with the Response Fence flag.  The fence 
> must be considered cleared after receiving the acknowledgement.
> d) All further status processing for the LU is resumed only 
> after clearing the fence.  If any new responses for the I_T_L 
> nexus are received from the SCSI layer before the fence is 
> cleared, those Response PDUs must be held and queued at the 
> iSCSI layer until the fence is cleared.
>