Subject: Re: [Ips] Response Fence Flag

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 4 Jan 2007 11:10:19 -0800 (PST)

Bob,

Response Fence prompts an end-to-end acknowledgement, it is not "merely an internal" artifact with no consequence.  This e2e ack can happen in a SCSI transport protocol specific way.

Response ordering is an issue for a very small % of task/TMF responses on a multi-connection iSCSI session, due to the network skew between the different connections participating in one I_T nexus.  IPS WG discussed this during the Dallas IETF and subsequently on the mailing list, there is already consensus that it is an issue.  I also understand that iSCSI is not alone here.

Please see the conference proceedings slides from the Dallas IETF for additional information:
http://www3.ietf.org/proceedings/06mar/slides/ips-0.pdf

Section 3.3 of http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-impl-guide-03.txt discusses the semantics and section 3.3.3 lists specific cases that we know of today that require such a response ordering.  Section 3.3.2 defines what the "transport protocol specific" way to realize e2e ack for iSCSI is.

If you would like to chat offline, feel free to give Rob or myself a call.

Thanks.

Mallikarjun

----- Original Message ----
From: Robert Snively <[email-masked]>
To: "Elliott, Robert (Server Storage)" <[email-masked]>; [email-masked]
Sent: Thursday, January 4, 2007 10:01:36 AM
Subject: RE: [Ips] Response Fence Flag

The problem is that the definition of "previous" in this context is most likely flawed.
 
The "Response Fence" is merely an internal and untestable state machine input that
attempts to order the transmission of responses in the target port's queue.
It  does nothing to improve the overall ordering of the responses at the application
client.
 
Drivers for generic SCSI transports must always be aware that requested operations
report completion with a more or less arbitrary time relationship to each other and must be
designed to tolerate that.  
 
Using LOGICAL UNIT RESET as an example, the operation does not act only on
the device server.  As it passes down through the initiator stack, it may clean out
previously offered commands that have not yet been transported.  Then as it
passes up through the target stack, it may clean out previously offered commands
that have not yet been operated on.  As it passes through the device server
execution path, it may clean out previously offered commands that have begun
execution and some that have completed execution without yet responding.  As its 
response passes back down the target stack, it may clean up commands
that have completed for which the response has not yet been transmitted, and as
it passes back up the initiator stack, it may clean up commands for which the
response has been transmitted, but not yet been made available to the driver or
application.  SAM-4 attempts to define the passage through the target port,
through the device server, and back to the target port.  However, beyond the
initiator port is outside the SAM-4 scope, and even where it is defined,
the "shall ensure"s are unlikely to be testable and in some cases not implementable. 
 
Will I sometimes get command complete indications for commands after
a LOGICAL UNIT RESET response is received?  You bet.  Response Fence doesn't
change that.
 
Will I sometimes get command aborted indications for commands after a 
LOGICAL UNIT RESET is response is received?  You bet.  Response Fence
doesn't change that.
 
Is this a problem?  It never has been in the past, why is it now?
 
Can you test the corner conditions for a fence operation?  I doubt it.  The
existence of the "Response Fence" argument is internal to the target
implementation and invisible to testing.
 
Bob

From: Elliott, Robert (Server Storage) [mailto:[email-masked]] 
Sent: Wednesday, January 03, 2007 3:35 PM
To: [email-masked]
Subject: RE: [Ips] Response Fence Flag

SAM-3 and SAM-4 require all transport protocols to tag the TMF responses with the requests:
"Each SCSI transport protocol shall allow a Received Task Management Function Executed confirming completion of the requested task to be associated with the corresponding Send Task Management Request."
Although iSCSI was based on SAM-2, it complies with that rule. The Fence concept is asking for more than that - it wants the target to be able to ensure that all previous commands/TMFs are complete before delivering a particular TMF response (e.g., for a LOGICAL UNIT RESET).  Since iSCSI doesn't have ACKs, the target must wait for the next PDU from the initiator with an updated ExpStatSN.
06-341r0 discussed an alternative - add a Delivery Result output to the Send Command Complete and Task Management Function Executed confirmations (as previously proposed in 04-072r0 for a different purpose). This would let the device server/task manager wait for all the previous commands/TMF responses to complete (be ACKed) before proceeding to make additional calls (e.g., Task Management Function Executed for a LOGICAL UNIT RESET).
However, iSCSI allows the target port to send a NOP-IN to force delivery of a NOP-OUT with ExpStatSN, rather than passively wait for a PDU to show up.  The device server/task manager must instruct the target port when to do this, and the Request Fence argument serves that purpose.  Target ports using protocols without such a force mechanism are still OK - they will just wait for protocol-specific delivery confirmations (e.g. ACKs).
-- 
Rob Elliott, [email-masked] 
Hewlett-Packard Industry Standard Server Storage Advanced Technology 
https://ecardfile.com/id/RobElliott 

From: Robert Snively [mailto:[email-masked]] 
Sent: Wednesday, January 03, 2007 4:46 PM
To: Elliott, Robert (Server Storage); [email-masked]
Subject: RE: [Ips] Response Fence Flag

I need a little tutorial here.  Note that this is an architectural hack that
is very "un-SCSI" and is not required if the response is qualified with identification
information associated with the original request.  I know that such qualification
exists for commands, but does it exist for task management requests in
iSCSI?  
 
I know that it does exist for parallel SCSI (which is strictly interlocked)
and for SAS and Fibre Channel SCSI, making such "fences" unrequired by
those technologies.
 
Note that Rob's previous revision of 06-341 is available on www.t10.org.
 
I would hate to see such a hack creep into the SCSI architecture.
 
Bob

From: Elliott, Robert (Server Storage) [mailto:[email-masked]] 
Sent: Wednesday, January 03, 2007 1:56 PM
To: [email-masked]
Subject: RE: [Ips] Response Fence Flag

If T10 agrees, T10 proposal 06-341 will add it to SAM-4.
 
-- 
Rob Elliott, [email-masked] 
Hewlett-Packard Industry Standard Server Storage Advanced Technology 
https://ecardfile.com/id/RobElliott 

From: Eddy Quicksall [mailto:[email-masked]] 
Sent: Wednesday, January 03, 2007 2:00 PM
To: [email-masked]
Subject: [Ips] Response Fence Flag

Section 3.3.1 talks about a Response Fence Flag:
 
SCSI protocol layer instructs the SCSI transport layer of a 
"Response Fence" associated with the response in question when 
the "Send Command Complete" protocol data service (SAM-2, clause 
5.4.2) ...
 
But I don't see any reference to that in SAM-2.
 
Is this strictly an iSCSI flag? Where is the flag specified?
 
Eddy
_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 5 Jan 2007 10:08:09 -0800 (PST)

Not really.  Current draft text is intentionally written to not have any dependencies on T10 dynamics.  The point is that iSCSI needs such a notion for succinctly describing the proper iSCSI protocol actions in a few places - ACA, TMF, Persistent reserve/Abort to name a few.  We certainly hope it will be approved by T10 and be a part of SAM-4 soon, but that isn't required per se for describing what iSCSI needs for its correct behavior.

IPS WG has adopted what it needs in the past - staying ahead of T10 review/approval cycle if necessary.  I_T nexus loss notification, iSCSI target/port naming, clearing effects are a few I recall.

Mallikarjun

----- Original Message ----
From: Eddy Quicksall <[email-masked]>
To: Robert Snively <[email-masked]>; "Elliott, Robert (Server Storage)" <[email-masked]>; [email-masked]
Sent: Friday, January 5, 2007 8:58:47 AM
Subject: Re: [Ips] Response Fence Flag

From an earlier email I think that Response Fence is only a proposal in T10 (http://www.t10.org:80/doc06.htm). If so shouldn't iSCSI wait a bit until this has been ratified?
 
Eddy
----- Original Message ----- 
From: Robert Snively 
To: Elliott, Robert (Server Storage) ; [email-masked] 
Sent: Thursday, January 04, 2007 1:01 PM
Subject: RE: [Ips] Response Fence Flag

 

 

 
Bob

 

 
Bob

 

 

 
Eddy

---

From: "John Hufferd" <[email-masked]>
Date: Fri, 5 Jan 2007 13:01:06 -0800

Mallikarjun,

I must admit, that I have been a bit confused with the direction of the
conversation here.

Therefore, I went back and reviewed the charts from Dallas.  And as I
remembered, the initial focus was to address the issue of Multiple
Connections per Session (MC/S) (as stated on chart 4 - "Non-issue for
single-connection iSCSI sessions").  So I think I have missed the step
where we have morphed the discussion into one dealing with multiple
sessions.  (I am not sure how that happened, or if I miss-read the
charts from Dallas, or have not followed the discussion adequately.)

If we are attempting to define two different issues, one with MC/S and
one with Multiple Session from different Initiators, I think it would be
useful to break down the conversation into Topic A - MC/S and Topic B
Multiple Sessions.  It is possible that one solution will addresses
both, but I for one think I am hearing arguments that might be
appropriate for Topic B, while I am thinking about its applicability to
Topic A.

Perhaps, you could address the issue as either being all about MC/S or
explicitly state that it is intended to affect Multiple Sessions also,
and then address the issues and solution for each separately.  For
example, I believe Robert was addressing the issue from a view of
Multiple Sessions and if we only intended to address MC/S then I expect
the response might be somewhat different.

Anyway, if you could clear-up some of this, I think it would be useful
(at least to me).

.

.

.

John L Hufferd

Sr. Executive Director of Technology

Brocade Communications Systems, Inc

[email-masked]

Office Phone: (408) 333-5244; eFAX: (408) 904-4688

Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606

________________________________

From: Mallikarjun C. [mailto:[email-masked]] 
Sent: Friday, January 05, 2007 10:08 AM
To: [email-masked]
Subject: Re: [Ips] Response Fence Flag

Not really.  Current draft text is intentionally written to not have any
dependencies on T10 dynamics.  The point is that iSCSI needs such a
notion for succinctly describing the proper iSCSI protocol actions in a
few places - ACA, TMF, Persistent reserve/Abort to name a few.  We
certainly hope it will be approved by T10 and be a part of SAM-4 soon,
but that isn't required per se for describing what iSCSI needs for its
correct behavior.

IPS WG has adopted what it needs in the past - staying ahead of T10
review/approval cycle if necessary.  I_T nexus loss notification, iSCSI
target/port naming, clearing effects are a few I recall.

Mallikarjun

----- Original Message ----
From: Eddy Quicksall <[email-masked]>
To: Robert Snively <[email-masked]>; "Elliott, Robert (Server
Storage)" <[email-masked]>; [email-masked]
Sent: Friday, January 5, 2007 8:58:47 AM
Subject: Re: [Ips] Response Fence Flag

>From an earlier email I think that Response Fence is only a proposal in
T10 (http://www.t10.org:80/doc06.htm <http://www.t10.org/doc06.htm> ).
If so shouldn't iSCSI wait a bit until this has been ratified?

Eddy

	----- Original Message ----- 

	From: Robert Snively <mailto:[email-masked]>  

	To: Elliott, Robert (Server Storage) <mailto:[email-masked]>  ;
[email-masked] 

	The problem is that the definition of "previous" in this context
is most likely flawed.

	The "Response Fence" is merely an internal and untestable state
machine input that

	attempts to order the transmission of responses in the target
port's queue.

	It  does nothing to improve the overall ordering of the
responses at the application

	client.

	Drivers for generic SCSI transports must always be aware that
requested operations

	report completion with a more or less arbitrary time
relationship to each other and must be

	Using LOGICAL UNIT RESET as an example, the operation does not
act only on

	the device server.  As it passes down through the initiator
stack, it may clean out

	previously offered commands that have not yet been transported.
Then as it

	passes up through the target stack, it may clean out previously
offered commands

	that have not yet been operated on.  As it passes through the
device server

	execution path, it may clean out previously offered commands
that have begun

	execution and some that have completed execution without yet
responding.  As its 

	response passes back down the target stack, it may clean up
commands

	that have completed for which the response has not yet been
transmitted, and as

	it passes back up the initiator stack, it may clean up commands
for which the

	response has been transmitted, but not yet been made available
to the driver or

	application.  SAM-4 attempts to define the passage through the
target port,

	through the device server, and back to the target port.
However, beyond the

	initiator port is outside the SAM-4 scope, and even where it is
defined,

	the "shall ensure"s are unlikely to be testable and in some
cases not implementable. 

	Will I sometimes get command complete indications for commands
after

	a LOGICAL UNIT RESET response is received?  You bet.  Response
Fence doesn't

	change that.

	Will I sometimes get command aborted indications for commands
after a 

	LOGICAL UNIT RESET is response is received?  You bet.  Response
Fence

	doesn't change that.

	Is this a problem?  It never has been in the past, why is it
now?

	Can you test the corner conditions for a fence operation?  I
doubt it.  The

	existence of the "Response Fence" argument is internal to the
target

	Bob

________________________________

	SAM-3 and SAM-4 require all transport protocols to tag the TMF
responses with the requests:

	"Each SCSI transport protocol shall allow a Received Task
Management Function Executed confirming completion of the requested task
to be associated with the corresponding Send Task Management Request."

	Although iSCSI was based on SAM-2, it complies with that rule.
The Fence concept is asking for more than that - it wants the target to
be able to ensure that all previous commands/TMFs are complete before
delivering a particular TMF response (e.g., for a LOGICAL UNIT RESET).
Since iSCSI doesn't have ACKs, the target must wait for the next PDU
from the initiator with an updated ExpStatSN.

	06-341r0 discussed an alternative - add a Delivery Result output
to the Send Command Complete and Task Management Function Executed
confirmations (as previously proposed in 04-072r0 for a different
purpose). This would let the device server/task manager wait for all the
previous commands/TMF responses to complete (be ACKed) before proceeding
to make additional calls (e.g., Task Management Function Executed for a
LOGICAL UNIT RESET).

	However, iSCSI allows the target port to send a NOP-IN to force
delivery of a NOP-OUT with ExpStatSN, rather than passively wait for a
PDU to show up.  The device server/task manager must instruct the target
port when to do this, and the Request Fence argument serves that
purpose.  Target ports using protocols without such a force mechanism
are still OK - they will just wait for protocol-specific delivery
confirmations (e.g. ACKs).

	-- 
	Rob Elliott, [email-masked] 
	Hewlett-Packard Industry Standard Server Storage Advanced
Technology 
	https://ecardfile.com/id/RobElliott 

________________________________

		I need a little tutorial here.  Note that this is an
architectural hack that

		is very "un-SCSI" and is not required if the response is
qualified with identification

		information associated with the original request.  I
know that such qualification

		exists for commands, but does it exist for task
management requests in

		iSCSI?  

		I know that it does exist for parallel SCSI (which is
strictly interlocked)

		and for SAS and Fibre Channel SCSI, making such "fences"
unrequired by

		those technologies.

		Note that Rob's previous revision of 06-341 is available
on www.t10.org <http://www.t10.org/> .

		I would hate to see such a hack creep into the SCSI
architecture.

		Bob

________________________________

		From: Elliott, Robert (Server Storage)
[mailto:[email-masked]] 
		Sent: Wednesday, January 03, 2007 1:56 PM
		To: [email-masked]
		Subject: RE: [Ips] Response Fence Flag

		-- 
		Rob Elliott, [email-masked] 
		Hewlett-Packard Industry Standard Server Storage
Advanced Technology 
		https://ecardfile.com/id/RobElliott 

________________________________

			From: Eddy Quicksall
[mailto:[email-masked]] 
			Sent: Wednesday, January 03, 2007 2:00 PM
			To: [email-masked]
			Subject: [Ips] Response Fence Flag

			SCSI protocol layer instructs the SCSI transport
layer of a 
			"Response Fence" associated with the response in
question when 
			the "Send Command Complete" protocol data
service (SAM-2, clause 
			5.4.2) ...

			Is this strictly an iSCSI flag? Where is the
flag specified?

			Eddy

---

From: "John Hufferd" <[email-masked]>
Date: Mon, 8 Jan 2007 12:21:32 -0800

Mallikarjun,

OK, so I would like us to first cover Topic A (your case 1).

As I understand it you are trying to make TMC and ACA operate correctly
on a MC/S Session.  

And, if I understand your Slide 5, et.al. correctly then the iSCSI code
can ASSUME a Fence has been issued for all TM and ACA operations and
react as you have specified.  In that specific case there is no need for
Fence parameter in the Target SCSI to iSCSI interaction (at least at
this time).

So if that is the case, then for all current operations we are covered
without additional Target SCSI parameters.  Is that correct?

Now when you use the words Third Party Session, I believe that could
mean Session that are iSCSI or Non iSCSI (e.g. Fibre Channel), I also
think it could include iSCSI Sessions on other portal groups connected
to the same Target. 

The thing that is concerning me is that some of the words which are
included in the document do not factor in the fact that any specific
iSCSI Target portal group may not have a clue about what is happening at
another iSCSI Target portal group, or at a Fibre Channel portal that may
also be attached to the same Target/LUN.

The way I read your document, I thought that it implied that the iSCSI
Transport would know about things that are happening on other Portal
Groups that may be iSCSI or some other Transport technology.

For instance in section 4.1.3.a (under the Target iSCSI Layer) you state
"...SHOULD NOT wait for new commands on third-party affected sessions -
only the instantiated tasks have to be considered for the purpose of
determining the affected tasks."  Other than the fact that it does not
matter what the other Session do, since you ignore it, the thought
pattern is that you will know that you are ignoring it.
 
Also in section 4.1.3 d (under the Target iSCSI layer) you state
"...MUST generate an Asynchronous Message PDU with AsyncEvent=5 (section
8.1) on: 
          i) each connection of each third-party session that at 
            least one affected task is allegiant to, and 
          ii) each connection except the non-issuing connection of the 
            issuing session that has at least one allegiant affected 
            task. "
 
I think that implies that one iSCSI portal group has knowledge of what
is happening in other portal groups, and that extends to other
Technology attachments such as Fibre Channel.
 
If you are stating that it is up to the target SCSI layer to issue the
"Fence parameter" appropriately such that the effect is what you stated
in 4.1.3 (under the Target iSCSI Layer) then I might feel that I am
coming closer to understanding your point.
 
Otherwise, I just do not see how that is practical since even the iSCSI
target portal groups may be implemented in separate iSCSI Offload
Adapters.  And I really do not know how they will know what is happening
on the Fibre Channel connections. 

When I combine the above thought with the statements that were made by
Robert Snively, which in effect states that all the actions and
responses arrivals are not predictable (at least across different
Sessions), that seems to me to mean that the appearance of UAs, caused
by another Session, may occur at any time with relation to what is going
on at any other Session.  Therefore, I can not yet grasp the need for
ordering Asynchronous Messages among the MC/S connections.  I am trying
to see what the initiator surprise will be, and why that will cause a
problem, if the Asynchronous Messages occur in some undefined order.
However, at this time I am left with an unease that not only are some of
the above document statements not implementable, I am wondering it they
are even important.

Hopefully, you can understand my dilemma on understanding what you are
proposing, and can explain what I am missing.

.

.

.

John L Hufferd

[email-masked]

From: Mallikarjun C. [mailto:[email-masked]] 
Sent: Friday, January 05, 2007 2:45 PM
To: IPS
Subject: Fw: [Ips] Response Fence Flag

Trying again after bouncing (>50KB), tail is now clipped....

----- Forwarded Message ----
From: Mallikarjun C. <[email-masked]>
To: [email-masked]
Sent: Friday, January 5, 2007 2:40:56 PM
Subject: Re: [Ips] Response Fence Flag

Hi John,

As you correctly point out, there are two distinct but related topics
here.

(1) proper response ordering across participating connections in a
multi-connection session, for a handful of task/TMF responses

(2) proper way to terminate and signal tasks when actions on one session
can impact multiple initiators

(1) is all about response fence, it is covered separately in section 3.3
of IG draft.  That's what this email thread started off with.

(2) covers cases that IG had called as as "multi-task abort" cases that
typically have a shared task set across sessions, or are the result of a
Logical Unit Reset TMF.  Section 4.1 of IG adresses (2).

So how are (1) and (2) related?

- Some cases covered under (2) overlap with (1), but some cases in (2)
are for single connection sessions.  E.g. LU Reset on a single
connection session impacting several 3rd party sessions

- Some cases under (1) overlap with (2), but some cases in (1) have
nothing to do with other sessions.  E.g. establishment of ACA on a
non-shared task set

- Section 4.1 (i.e. text for (2)) uses response fence (i.e. text for
(1)) whenever multi-connection sessions are involved in multi-task abort
situations.

I hope that clarifies it.  Feel free to ask if that is not clear.  The
net is that the response fence is the underpinning notion to describe
the correct iSCSI behavior in a few cases and some of those cases are
about task terminations across third-party sessions.

Mallikarjun

----- Original Message ----
From: John Hufferd <[email-masked]>
To: Mallikarjun C. <[email-masked]>; [email-masked]
Sent: Friday, January 5, 2007 1:01:06 PM
Subject: RE: [Ips] Response Fence Flag

Mallikarjun,

Therefore, I went back and reviewed the charts from Dallas .  And as I
remembered, the initial focus was to address the issue of Multiple
Connections per Session (MC/S) (as stated on chart 4 - "Non-issue for
single-connection iSCSI sessions").  So I think I have missed the step
where we have morphed the discussion into one dealing with multiple
sessions.  (I am not sure how that happened, or if I miss-read the
charts from Dallas , or have not followed the discussion adequately.)

.

.

.

John L Hufferd

[email-masked]

Mallikarjun

----- Original Message ----
From: Eddy Quicksall <[email-masked]>
To: Robert Snively <[email-masked]>; "Elliott, Robert (Server

Eddy

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 8 Jan 2007 17:20:51 -0800 (PST)

John,

On your topic A, section 3.3.3 lists the current list of response fence cases where the fence is "assumed" to be present.  So for the time being, iSCSI "knows" when to fence.  Further, note that reponse fence is limited to *one I_T_L nexus* only - so by definition, there cannot be any cross-nexus ordering requirements out of a response fence, and not even cross I_T_L nexus linkages.

On your topic B, I can now see where some of the confusion is - wherever we say "third-party session" in the draft, we mean a "third-party iSCSI session" unless otherwise qualified.  I need to add a sentence to that effect to the definition of "third-party" in section 1.1, sorry about that ambiguity.  All verbiage that is meant for non-iSCSI I_T nexuses is clearly marked as such in the draft.  

How broad is the notion of "third-party" - across portal groups? across transports? - is something I cannot directly answer.  It is really a question of target design - I'd say whenever actions on an iSCSI session "affect" tasks (per definition of 4.1.1) on a different session, it is a third-party session for that action.  E.g., if one iSCSI target portal group (which is really a different I_T nexus from the issuing nexus) truly has no clue about a Clear Task Set on the issuing iSCSI target portal group, then it must not have had a shared task set at the LU level, shared jointly between the two portal groups.  In such a case, the first session is *not* a third-party session for the purpose of Clear Task Set on the second session.  If OTOH, they *do* share a task set at the SCSI LU level, then the first session becomes a third party session wrt the Clear Task Set on the second session per the intent of the draft.  Note that there is absolutely nothing new in this draft about
 the functional requireements on third-party sessions.  Cross-nexus clearing effects, UAs etc. are really SCSI territory.  The draft does not require any cross-portal group knowledge beyond what is required for 3720-compliant implementations (in fact, this draft frees up the TTT linkage we had before) - the 4.1.3 text you quote can be implemented either with a shared SCSI LU task set (in which case the notion of "affected" has to be propagated via the SCSI layer), or with a completely offloaded SCSI+iSCSI portal group (in which case all third-party sessions are local to the hardware).  Again, I see no new issues here.

The Nop-Out Messages are "a lazy signal" for a target to optimally cleanup any residual task state after the task was terminated, therefore Async Messages and Nop-Out Messages again absoluly have no cross-nexus linkages (and Async+Nop-Out exchange, BTW, is not related to response fence as you may have already sensed, it is related to multi-task abort handling).

Hope that clarifies some.  As always, feel free to ask if not, or give me a call.

Thanks.

Mallikarjun

----- Original Message ----
From: John Hufferd <[email-masked]>
To: Mallikarjun C. <[email-masked]>; IPS <[email-masked]>
Sent: Monday, January 8, 2007 12:21:32 PM
Subject: RE: [Ips] Response Fence Flag

Mallikarjun,
OK, so I would like us to first cover Topic A (your case 1).
As I understand it you are trying to make TMC and ACA operate correctly on a MC/S Session.  
 
And, if I understand your Slide 5, et.al. correctly then the iSCSI code can ASSUME a Fence has been issued for all TM and ACA operations and react as you have specified.  In that specific case there is no need for Fence parameter in the Target SCSI to iSCSI interaction (at least at this time).
 
So if that is the case, then for all current operations we are covered without additional Target SCSI parameters.  Is that correct?
 
Now when you use the words Third Party Session, I believe that could mean Session that are iSCSI or Non iSCSI (e.g. Fibre Channel), I also think it could include iSCSI Sessions on other portal groups connected to the same Target. 
 
The thing that is concerning me is that some of the words which are included in the document do not factor in the fact that any specific iSCSI Target portal group may not have a clue about what is happening at another iSCSI Target portal group, or at a Fibre Channel portal that may also be attached to the same Target/LUN.
 
The way I read your document, I thought that it implied that the iSCSI Transport would know about things that are happening on other Portal Groups that may be iSCSI or some other Transport technology.
 
For instance in section 4.1.3.a (under the Target iSCSI Layer) you state “…SHOULD NOT wait for new commands on third-party affected sessions - only the instantiated tasks have to be considered for the purpose of determining the affected tasks.“  Other than the fact that it does not matter what the other Session do, since you ignore it, the thought pattern is that you will know that you are ignoring it.
  
Also in section 4.1.3 d (under the Target iSCSI layer) you state “…MUST generate an Asynchronous Message PDU with AsyncEvent=5 (section 8.1) on: 
            task. “
  
I think that implies that one iSCSI portal group has knowledge of what is happening in other portal groups, and that extends to other Technology attachments such as Fibre Channel.
  
If you are stating that it is up to the target SCSI layer to issue the “Fence parameter” appropriately such that the effect is what you stated in 4.1.3 (under the Target iSCSI Layer) then I might feel that I am coming closer to understanding your point.
  
Otherwise, I just do not see how that is practical since even the iSCSI target portal groups may be implemented in separate iSCSI Offload Adapters.  And I really do not know how they will know what is happening on the Fibre Channel connections. 
 
When I combine the above thought with the statements that were made by
 Robert Snively , which in effect states that all the actions and responses arrivals are not predictable (at least across different Sessions), that seems to me to mean that the appearance of UAs, caused by another Session, may occur at any time with relation to what is going on at any other Session.  Therefore, I can not yet grasp the need for ordering Asynchronous Messages among the MC/S connections.  I am trying to see what the initiator surprise will be, and why that will cause a problem, if the Asynchronous Messages occur in some undefined order.  However, at this time I am left with an unease that not only are some of the above document statements not implementable, I am wondering it they are even important.
 
Hopefully, you can understand my dilemma on understanding what you are proposing, and can explain what I am missing.
 
.
.
.
John L Hufferd

----- Forwarded Message ----
From: Mallikarjun C. <[email-masked]>
To: [email-masked]
Sent: Friday, January 5, 2007 2:40:56 PM
Subject: Re: [Ips] Response Fence Flag
Hi John,
 
As you correctly point out, there are two distinct but related topics here.
 
(1) proper response ordering across participating connections in a multi-connection session, for a handful of task/TMF responses
(2) proper way to terminate and signal tasks when actions on one session can impact multiple initiators
 
(1) is all about response fence, it is covered separately in section 3.3 of IG draft.  That's what this email thread started off with.
 
(2) covers cases that IG had called as as "multi-task abort" cases that typically have a shared task set across sessions, or are the result of a Logical Unit Reset TMF.  Section 4.1 of IG adresses (2).

- Some cases covered under (2) overlap with (1), but some cases in (2) are for single connection sessions.  E.g. LU Reset on a single connection session impacting several 3rd party sessions
 
- Some cases under (1) overlap with (2), but some cases in (1) have nothing to do with other sessions.  E.g. establishment of ACA on a non-shared task set
 
- Section 4.1 (i.e. text for (2)) uses response fence (i.e. text for (1)) whenever multi-connection sessions are involved in multi-task abort situations.
 
I hope that clarifies it.  Feel free to ask if that is not clear.  The net is that the response fence is the underpinning notion to describe the correct iSCSI behavior in a few cases and some of those cases are about task terminations across third-party sessions.
 
Mallikarjun

----- Original Message ----
From: John Hufferd <[email-masked]>
To: Mallikarjun C. <[email-masked]>; [email-masked]
Sent: Friday, January 5, 2007 1:01:06 PM
Subject: RE: [Ips] Response Fence Flag
Mallikarjun,
I must admit, that I have been a bit confused with the direction of the conversation here.
 
Therefore, I went back and reviewed the charts from Dallas .  And as I remembered, the initial focus was to address the issue of Multiple Connections per Session (MC/S) (as stated on chart 4 – “Non-issue for single-connection iSCSI sessions”).  So I think I have missed the step where we have morphed the discussion into one dealing with multiple sessions.  (I am not sure how that happened, or if I miss-read the charts from Dallas , or have not followed the discussion adequately.)
 
If we are attempting to define two different issues, one with MC/S and one with Multiple Session from different Initiators, I think it would be useful to break down the conversation into Topic A – MC/S and Topic B Multiple Sessions.  It is possible that one solution will addresses both, but I for one think I am hearing arguments that might be appropriate for Topic B, while I am thinking about its applicability to Topic A.
 
Perhaps, you could address the issue as either being all about MC/S or explicitly state that it is intended to affect Multiple Sessions also, and then address the issues and solution for each separately.  For example, I believe Robert was addressing the issue from a view of Multiple Sessions and if we only intended to address MC/S then I expect the response might be somewhat different.
 
Anyway, if you could clear-up some of this, I think it would be useful (at least to me).
 
.
.
.
John L Hufferd

 
 
Mallikarjun

----- Original Message ----
From: Eddy Quicksall <[email-masked]>
To: Robert Snively <[email-masked]>; "Elliott, Robert (Server Storage)" <[email-masked]>; [email-masked]
Sent: Friday, January 5, 2007 8:58:47 AM
Subject: Re: [Ips] Response Fence Flag
 
Eddy