Subject: Last Call Response Fence resolution (Was Re: [Ips] iSCSI Implementer's Guide - WG Last Call status, 2nd try)

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 25 Jan 2007 18:18:07 -0800 (PST)

David, 

I understand your feedback on section 3.3.1.  I have made a handful of changes to the text, new text is attached below for your review.

I am trying to preserve the notion of an iSCSI-defined & iSCSI-scoped Response Fence in the text (you seem to be OK with it).  This gives us a convenient shorthand to refer to a set of semantics.  Besides, if/when T10 approves the response fence concept for SAM-4, iSCSI implementers will know what to map it to for iSCSI.  As you'll see, I made it very clear in the new text that this is not an attempt to change SAM in this draft (which was of course never the intent).

Mallikarjun

----- Original Message ----
From: "[email-masked]" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Sent: Wednesday, January 24, 2007 5:34:41 PM
Subject: [Ips] iSCSI Implementer's Guide - WG Last Call status, 2nd try
.......

--- c) The Response Fence

......

Unfortunately, I have to object to Section 3.3.1, which appears
to be trying to modify SAM-2 to add the Response Fence flag.
That can't be done in IETF, and any T10 modification would be
to SAM-4, whereas iSCSI is still based on SAM-2.  I think the
mention of SAM-2 and model of the interface between SCSI and
a SCSI transport needs to be removed from Section 3.3.1.  The
items numbered (1) and (2) in that section look ok (and in
fact useful to keep) - the problematic text is that which
suggests the existence of a "Response Fence flag" in the
abstract interface between SCSI and the SCSI transport (e.g.,
iSCSI) along with the mention of "SCSI protocol interface
model" in Section 3.3.  The upshot will be that Response
Fence behavior will be specified in a fashion that makes it
only applicable to the cases spelled out in Section 3.3.3.

...........

3.3 Model Assumptions for Response Ordering
Whenever an iSCSI session is composed of multiple connections, the Response PDUs (task responses or TMF responses) originating in the target SCSI layer are distributed onto the multiple connections by the target iSCSI layer according to iSCSI connection allegiance rules.  This process generally may not preserve the ordering of the responses by the time they are delivered to the initiator SCSI layer.  Since ordering is not expected across SCSI responses anyway, this approach works fine in the general case.  However to address the special cases where some ordering is desired by the SCSI layer, the following semantics are defined for a "Response Fence" flag if such a flag were to qualify SCSI completion messages as they are handed off from the SCSI protocol layer to the iSCSI layer.  Note however that [SAM2] does not model such a flag in its protocol interface model, so the "Response Fence" flag is an iSCSI-defined artifact for descriptive convenience of the mandatory iSCSI response ordering semantics.  

3.3.1 Model Description
Target SCSI protocol layer hands off the SCSI completion messages to the target iSCSI layer by invoking the "Send Command Complete" protocol data service ([SAM2], clause 5.4.2) and "Task Management Function Executed" ([SAM2], clause 6.9) service.   On receiving the SCSI completion message, iSCSI layer assumes that a Response Fence flag optionally qualifies the SCSI completion message (section 3.3.3 describes the specific instances where the flag qualifies the completion message).  Whenever the Response Fence flag is present, the target iSCSI layer MUST ensure that the following conditions are met in delivering the response message to the initiator iSCSI layer:
(1) Response with Response Fence MUST chronologically be delivered after all the "preceding" responses on the I_T_L nexus, if the preceding responses are delivered at all, to the application client on the initiator. 
(2) Response with Response Fence MUST chronologically be delivered prior to all the "following" responses on the I_T_L nexus. 
The “preceding” and “following” notions refer to the order of hand-off of a response message from the target SCSI protocol layer to the target iSCSI layer.

3.3.2 iSCSI Semantics with the Interface Model
Whenever the TaskReporting key (section 9.1) is negotiated to ResponseFence or FastAbort for an iSCSI session, the target iSCSI layer MUST perform the actions described in this section on all tasks related to that session.  A target iSCSI layer MUST do the following whenever Response Fence flag qualifies a SCSI completion message handed down from the target SCSI layer:
a) If it is a single-connection session, no special processing is required.  Standard SCSI Response PDU build process happens. 
b) If it is a multi-connection session, target iSCSI layer takes note of last-sent and unacknowledged StatSN on each of the connections in the iSCSI session, and waits for acknowledgement (SHOULD solicit for acknowledgement by way of a Nop-In) of each such StatSN to clear the fence.  SCSI response with the Response Fence flag must be sent to the initiator only after receiving acknowledgements for each of the unacknowledged StatSNs.
c) Target iSCSI layer must wait for an acknowledgement of the SCSI Response PDU that carried the response which the target SCSI layer marked with the Response Fence flag.  The fence must be considered cleared after receiving the acknowledgement.
d) All further status processing for the LU is resumed only after clearing the fence.  If any new responses for the I_T_L nexus are received from the SCSI layer before the fence is cleared, those Response PDUs must be held and queued at the iSCSI layer until the fence is cleared.

3.3.3 Current List of Fenced Response Use Cases

----No changes ---

____________________________________________________________________________________
Don't get soaked.  Take a quick peak at the forecast
with the Yahoo! Search weather shortcut.
http://tools.search.yahoo.com/shortcuts/#loc_weather

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: [email-masked]
Date: Thu, 25 Jan 2007 21:44:53 -0500

Mallikarjun,

> I am trying to preserve the notion of an iSCSI-defined & 
> iSCSI-scoped Response Fence in the text (you seem to be OK 
> with it).  This gives us a convenient shorthand to refer to a 
> set of semantics.  Besides, if/when T10 approves the response 
> fence concept for SAM-4, iSCSI implementers will know what to 
> map it to for iSCSI.  As you'll see, I made it very clear in 
> the new text that this is not an attempt to change SAM in 
> this draft (which was of course never the intent).

I agree in principle with the intent where "Response Fence" is
iSCSI-scoped and defined "behavior."  OTOH, calling it a "flag"
is a problem.  I would prefer that all text anticipating
addition of a "flag" to SAM-4 be removed, and in particular
the following text:

> the following 
> semantics are defined for a "Response Fence" flag if such a 
> flag were to qualify SCSI completion messages as they are 
> handed off from the SCSI protocol layer to the iSCSI layer.  
> Note however that [SAM2] does not model such a flag in its 
> protocol interface model, so the "Response Fence" flag is an 
> iSCSI-defined artifact for descriptive convenience of the 
> mandatory iSCSI response ordering semantics.

which is just asking for an IETF Last Call comment or AD question
about why this circumlocution was used when it suffices to define
the "Response Fence" behavior without ever mentioning a non-
existent flag.  Section 3.3.3 doesn't need the flag - it can
refer to the behavior and say what it MUST be applied to.

If Rob succeeds in getting T10 to put a Response Fence flag
into SAM-4, we can deal with that flag in the draft that
deals with the impacts of SAM-3 and SAM-4 on iSCSI, which
will have a number of other things to deal with.  Before anyone
gets excited about this, I will state my view that the IETF
will need to ensure that all such impacts are specified in a
fashion that is backwards compatible with RFC-3720-only
implementations.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: Mallikarjun C. [mailto:[email-masked]] 
> Sent: Thursday, January 25, 2007 9:18 PM
> To: [email-masked]
> Subject: Last Call Response Fence resolution (Was Re: [Ips] 
> iSCSIImplementer's Guide - WG Last Call status, 2nd try)
> 
> David, 
> 
> I understand your feedback on section 3.3.1.  I have made a 
> handful of changes to the text, new text is attached below 
> for your review.
> 
> I am trying to preserve the notion of an iSCSI-defined & 
> iSCSI-scoped Response Fence in the text (you seem to be OK 
> with it).  This gives us a convenient shorthand to refer to a 
> set of semantics.  Besides, if/when T10 approves the response 
> fence concept for SAM-4, iSCSI implementers will know what to 
> map it to for iSCSI.  As you'll see, I made it very clear in 
> the new text that this is not an attempt to change SAM in 
> this draft (which was of course never the intent).
> 
> Mallikarjun
> 
> Whenever an iSCSI session is composed of multiple 
> connections, the Response PDUs (task responses or TMF 
> responses) originating in the target SCSI layer are 
> distributed onto the multiple connections by the target iSCSI 
> layer according to iSCSI connection allegiance rules.  This 
> process generally may not preserve the ordering of the 
> responses by the time they are delivered to the initiator 
> SCSI layer.  Since ordering is not expected across SCSI 
> responses anyway, this approach works fine in the general 
> case.  However to address the special cases where some 
> ordering is desired by the SCSI layer, the following 
> semantics are defined for a "Response Fence" flag if such a 
> flag were to qualify SCSI completion messages as they are 
> handed off from the SCSI protocol layer to the iSCSI layer.  
> Note however that [SAM2] does not model such a flag in its 
> protocol interface model, so the "Response Fence" flag is an 
> iSCSI-defined artifact for descriptive convenience of the 
> mandatory iSCSI response ordering semantics.  
> 
> 3.3.1 Model Description
> Target SCSI protocol layer hands off the SCSI completion 
> messages to the target iSCSI layer by invoking the "Send 
> Command Complete" protocol data service ([SAM2], clause 
> 5.4.2) and "Task Management Function Executed" ([SAM2], 
> clause 6.9) service.   On receiving the SCSI completion 
> message, iSCSI layer assumes that a Response Fence flag 
> optionally qualifies the SCSI completion message (section 
> 3.3.3 describes the specific instances where the flag 
> qualifies the completion message).  Whenever the Response 
> Fence flag is present, the target iSCSI layer MUST ensure 
> that the following conditions are met in delivering the 
> response message to the initiator iSCSI layer:
> (1) Response with Response Fence MUST chronologically be 
> delivered after all the "preceding" responses on the I_T_L 
> nexus, if the preceding responses are delivered at all, to 
> the application client on the initiator. 
> (2) Response with Response Fence MUST chronologically be 
> delivered prior to all the "following" responses on the I_T_L nexus. 
> The "preceding" and "following" notions refer to the order of 
> hand-off of a response message from the target SCSI protocol 
> layer to the target iSCSI layer.
> 
> Whenever the TaskReporting key (section 9.1) is negotiated to 
> ResponseFence or FastAbort for an iSCSI session, the target 
> iSCSI layer MUST perform the actions described in this 
> section on all tasks related to that session.  A target iSCSI 
> layer MUST do the following whenever Response Fence flag 
> qualifies a SCSI completion message handed down from the 
> target SCSI layer:
> a) If it is a single-connection session, no special 
> processing is required.  Standard SCSI Response PDU build 
> process happens. 
> b) If it is a multi-connection session, target iSCSI layer 
> takes note of last-sent and unacknowledged StatSN on each of 
> the connections in the iSCSI session, and waits for 
> acknowledgement (SHOULD solicit for acknowledgement by way of 
> a Nop-In) of each such StatSN to clear the fence.  SCSI 
> response with the Response Fence flag must be sent to the 
> initiator only after receiving acknowledgements for each of 
> the unacknowledged StatSNs.
> c) Target iSCSI layer must wait for an acknowledgement of the 
> SCSI Response PDU that carried the response which the target 
> SCSI layer marked with the Response Fence flag.  The fence 
> must be considered cleared after receiving the acknowledgement.
> d) All further status processing for the LU is resumed only 
> after clearing the fence.  If any new responses for the I_T_L 
> nexus are received from the SCSI layer before the fence is 
> cleared, those Response PDUs must be held and queued at the 
> iSCSI layer until the fence is cleared.
>