Subject: RE: [Ips] iSCSI cmd ordering and tagged task

From: "Elliott, Robert (Server Storage)" <[email-masked]>
Date: Thu, 20 Nov 2003 09:04:48 -0600

Although you can view iSCSI's sequence numbers as interfering with an
unordered use of SIMPLE task attribute, it has the benefit of making the
ORDERED and HEAD OF QUEUE task attributes usable.  Although SCSI
architecture does not strictly require ordering, all SCSI transport
protocols but one (Fibre Channel FCP, class 3, without command reference
numbers) do ensure ordered delivery.  Although it's not so important for
disks, it is very helpful for tapes.
 
-- 
Rob Elliott, [email-masked] 
Hewlett-Packard Industry Standard Server Storage Advanced Technology 
https://ecardfile.com/id/RobElliott 

	-----Original Message-----
	From: dan yo [mailto:[email-masked]] 
	Sent: Wednesday, November 19, 2003 6:28 PM
	To: [email-masked]; [email-masked]
	Subject: Re: [Ips] iSCSI cmd ordering and tagged task

	I agree, network latency is insignificant in comparision to disk
latency.  My real concern is the complexity in dealing with session wide
variables (cmdSN is one) in a device that supports multiple connections
per session.  An example is the case where each connection (of a
multi-connections session) terminates on different ingress (host facing)
port of a gateway (iSCSI to FC, or iSCSI to SATA).  As much as possible,
I want to route the I/O from each ingress port directly to the egress
port (target facing) without any inter-ingress port synchronization or
lookup.   The SCSI 'Simple' task seem to be an ideal candidate for that,
but the cmdSN ordering gets in the way. 
	Thanks,
	-Dan   
	
	Julian Satran <[email-masked]> wrote:

		I do not think your concern is valid.  The skew
introduced by multiple connections - when those operate correctly - is
minimal. 
		The iSCSI ordering requirement serves mainly to detect
missing commands due to digest errors (not caught by TCP) and not to
impose a SCSI ordering. 
		It helps also avoiding having the initiator overrun the
target beyond with more than the later can digest without using slow
interlock. 
		
		Julo 

		
dan yo <[email-masked]> 
Sent by: [email-masked] 

19/11/2003 07:03 

To
[email-masked] 
cc
Subject
[Ips] iSCSI cmd ordering and tagged task	

		It seems that the iSCSI requirement of preserving
command ordering would reduce the benefit of the "tagged" tasks feature
in SCSI.  For instance, if a host has several I/O tasks to send to the
target and that the ordering is not important, it may want to tag them
as "Simple" and let the target rearrange them in a way that is optimal
to disk access.  However, when these tasks arrive out of sequence at the
iSCSI target side (different from the order in which they enter the
iSCSI layer at the initiator side), they will be held up to wait for
their predecessor to arrive before they can be delivered to the SCSI
target.  This will a bigger issue with the multiple connections per
session scenario in iSCSI where commands from different connections will
get out of order more often than not.&n! bsp; Is my concern valid? 
		Thanks, 
		Dan 

  _____  

		Do you Yahoo!?
		Protect your identity with Yahoo! Mail AddressGuard
<http://antispam.yahoo.com/whatsnewfree>  

  _____  

	Do you Yahoo!?
	Protect your identity with Yahoo! Mail AddressGuard
<http://antispam.yahoo.com/whatsnewfree>

---

From: dan yo <[email-masked]>
Date: Thu, 20 Nov 2003 09:20:26 -0800 (PST)

Pardon me for thinking out loud.  Then does it make sense for the iSCSI initiator to turn on the "Immediate bit" for SIMPLE tasks?  I haven't thought through the side effect, but this seems to accomplish what I want, i.e. to allow each connection of a multiple connection session operate independently (no need to do cmdSN synchronization with other connections) and therefore easier to implement cut through switching in an iSCSI to FC gateway.
Thanks,
-Dan  

"Elliott, Robert (Server Storage)" <[email-masked]> wrote:
Although you can view iSCSI's sequence numbers as interfering with an unordered use of SIMPLE task attribute, it has the benefit of making the ORDERED and HEAD OF QUEUE task attributes usable.  Although SCSI architecture does not strictly require ordering, all SCSI transport protocols but one (Fibre Channel FCP, class 3, without command reference numbers) do ensure ordered delivery.  Although it's not so important for disks, it is very helpful for tapes.

-----Original Message-----
From: dan yo [mailto:[email-masked]] 
Sent: Wednesday, November 19, 2003 6:28 PM
To: [email-masked]; [email-masked]
Subject: Re: [Ips] iSCSI cmd ordering and tagged task

I agree, network latency is insignificant in comparision to disk latency.  My real concern is the complexity in dealing with session wide variables (cmdSN is one) in a device that supports multiple connections per session.  An example is the case where each connection (of a multi-connections session) terminates on different ingress (host facing) port of a gateway (iSCSI to FC, or iSCSI to SATA).  As much as possible, I want to route the I/O from each ingress port directly to the egress port (target facing) without any inter-ingress port synchronization or lookup.   The SCSI 'Simple' task seem to be an ideal candidate for that, but the cmdSN ordering gets in the way. 
Thanks,
-Dan   

I do not think your concern is valid.  The skew introduced by multiple connections - when those operate correctly - is minimal. 
The iSCSI ordering requirement serves mainly to detect missing commands due to digest errors (not caught by TCP) and not to impose a SCSI ordering. 
It helps also avoiding having the initiator overrun the target beyond with more than the later can digest without using slow interlock. 

Julo 

[email-masked] cc

It seems that the iSCSI requirement of preserving command ordering would reduce the benefit of the "tagged" tasks feature in SCSI.  For instance, if a host has several I/O tasks to send to the target and that the ordering is not important, it may want to tag them as "Simple" and let the target rearrange them in a way that is optimal to disk access.  However, when these tasks arrive out of sequence at the iSCSI target side (different from the order in which they enter the iSCSI layer at the initiator side), they will be held up to wait for their predecessor to arrive before they can be delivered to the SCSI target.  This will a bigger issue with the multiple connections per session scenario in iSCSI where commands from different connections will get out of order more often than not.&n! bsp; Is my concern valid? 
Thanks, 
Dan 

---------------------------------
Do you Yahoo!?
Protect your identity with Yahoo! Mail AddressGuard 

---------------------------------
Do you Yahoo!?
Protect your identity with Yahoo! Mail AddressGuard

---------------------------------
Do you Yahoo!?
Free Pop-Up Blocker - Get it now

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 20 Nov 2003 10:57:11 -0800

Dan,

	Session wide ordering is a requirement for even single connection
ErrorRecoveryLevel=0 implementations.  Please see section 3.2.2.1 for
the MUST requirement.  Not supporting CmdSN/ExpCmdSN/MaxCmdSN ordering
requirements is a definate violation of the iSCSI specification in a
simple ErrorRecoveryLevel=0 implementation,  let alone a multiple
connection ErrorRecoveryLevel=1 and above implementation.

One of the key goals of RFC 3347 and the iSCSI draft is to allow iSCSI
vendors to create interoptable high speed implemenations.  Leaving out
basic parts of the specification to avoid doing the necessary work to
make the implementation is question interoptable with industry standard
implemenations is a bad position to put ones self into.

Frankly, I will continue to advicate the point that only
ErrorRecoveryLevel=1 and above implementations will be allowed use
multiple connection sessions.  Any iSCSI target supporting multiple
connection sessions in a ErrorRecoveryLevel=0 implemenation is broken
IMHO and does so to support broken iSCSI ErrorRecoveryLevel=0 initiator
implemenations and vendors who believe they can get away with only doing
parts of ErrorRecoveryLevel=1 without doing a complete
ErrorRecoveryLevel=1 implementation.

Negoitating the ErrorRecoveryLevel=1 key during the leading connection
of a full feature phase iSCSI login means the supporting the nine (9)
requirements of within-command and within-connection recovery from
sections 6.1.4.1 and 6.1.4.2,  negoitating the ErrorRecoveryLevel=1 key
without implementing these features is a clear violation of the
specification.

In order to support the nine (9) requirements of ErrorRecoveryLevel=1
means supporting Fixed Interval Markers or some other framing mechnisim
to recover from header digest failures where the offset to the next
iSCSI Opcode cannot be reliably determined. (See section 6.1.4.1,
Initiator and Target case C, and section 3.2.8)  Also in the
within-command recovery cases from section 6.1.4.1, DataPDUInOrder=No
and DataSequenceInOrder=No cases must also be considered.

Thanks!

-- 
Nicholas A. Bellinger <[email-masked]>

On Thu, 2003-11-20 at 09:20, dan yo wrote:
> Pardon me for thinking out loud.  Then does it make sense for the
> iSCSI initiator to turn on the "Immediate bit" for SIMPLE tasks?  I
> haven't thought through the side effect, but this seems to accomplish
> what I want, i.e. to allow each connection of a multiple connection
> session operate independently (no need to do cmdSN synchronization
> with other connections) and therefore easier to implement cut
> through switching in an iSCSI to FC gateway.
> Thanks,
> -Dan  
> 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 20 Nov 2003 12:08:33 -0800

In addition to Julian's and Rob's responses, which
are right on, let me add one other note.

This is a concern periodically expressed - that a
multi-connection sesion has this serious problem of
the network skew across multiple connections that
slows down command reordering at the target.  The
response to that is - one does not have to use
a multi-connection session unless one wanted *one*
I_T nexus across all of them with all the implicit/expclicit
expectations that go with it (that Rob refers to below),
including ordered delivery.  A perfectly reasonable
alternative for some configurations may be to use
multiple I_T nexuses between the two network entities -
each on a different TCP connection.

One could then argue that command ordering is a
problem, period - even for single connection sessions.
I think  Julian already answered that - CmdSN is
is also an error detection mechanism that ensures
that all iSCSI sessions meet another expectation of an
I_T nexus - that of reliable delivery.  Finally,
CmdSN is also a flow control mechanism that is required
for the benefit of targets regardless of the number
of connections.  You can attempt to bypass the CmdSN
"constraints" by immediate commands, but then you also
bypass all these benefits for which it was designed.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

Elliott, Robert (Server Storage) wrote:

>     latency.  My real concern is the complexity in dealing with session
>     wide variables (cmdSN is one) in a device that supports multiple
>     connections per session.  An example is the case where each
>     connection (of a multi-connections session) terminates on different
>     ingress (host facing) port of a gateway (iSCSI to FC, or iSCSI to
>     SATA).  As much as possible, I want to route
>     the I/O from each ingress port directly to the egress port (target
>     facing) without any inter-ingress port synchronization or
>     lookup.   The SCSI 'Simple' task seem to be an ideal candidate for
>     that, but the cmdSN ordering gets in the way. 
>     Thanks,
>     -Dan  
> 
>         I do not think your concern is valid.  The skew introduced by
>         multiple connections - when those operate correctly - is minimal.
>         The iSCSI ordering requirement serves mainly to detect missing
>         commands due to digest errors (not caught by TCP) and not to
>         impose a SCSI ordering.
>         It helps also avoiding having the initiator overrun the target
>         beyond with more than the later can digest without using slow
>         interlock.
> 
>         Julo
> 
> 
> 
>         It seems that the iSCSI requirement of preserving command
>         ordering would reduce the benefit of the "tagged" tasks feature
>         in SCSI.  For instance, if a host has several I/O tasks to send
>         to the target and that the ordering is not important, it may
>         want to tag them as "Simple" and let the target rearrange them
>         in a way that is optimal to disk access.  However, when these
>         tasks arrive out of sequence at the iSCSI target side (different
>         from the order in which they enter the iSCSI layer at the
>         initiator side), they will be held up to wait for their
>         predecessor to arrive before they can be delivered to the SCSI
>         target.  This will a bigger issue with the multiple connections
>         per session scenario in iSCSI where commands from different
>         connections will get out of order more often than not.&n! bsp;
>         Is my concern valid?
>         Thanks,
>         Dan
> 
>         ------------------------------------------------------------------------
>         Do you Yahoo!?
>     ------------------------------------------------------------------------
>     Do you Yahoo!?

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 20 Nov 2003 13:27:20 -0800

Mallikarjun,

	I have come to the conclusion based upon the following examples:

0) The handling of out of order StatSNs and their eventual delayed
execution on the Initiator are strictly a within-connection recovery
feature.  Why is one out of order execution mechinism defined as
ErrorRecoveryLevel=1 while another is allowed for ErrorRecoveryLevel=0?
It could be argued that since CmdSNs are expected to reguarly be out of
order as they arrive, as compared to an out of order StatSN is usually
attributed to a header or data digest failure in with a response pdu or
datain pdu carrying a status bit.  Nevertheless it is an asymmetry. :-)

1) The CmdSN timer on the Initiator that checks what CmdSNs have been
acknowledged and possibly issue retries of non-immediate commands is
listed as a within-connection recovery feature in section 6.1.4.2. 
Granted it could be argued that this is a implementation issue,  but
really any implemenation will have something along these lines (a CmdSN
timer to either retry the CmdSN in ERL>=1 or fail the session in ERL=0)
to determine when a non-immediate CmdSN has been lost for various
reasons.

Aside from pointing what I see as minor inconsistencies, I fear that it
becomes very easy for vendors to choose what multiple connection
funtionality not to implement at ErrorRecoveryLevel=0 and leave a window
open for serious interoptabilty issues.  From the point of view of
someone who as brought Initiator, Target and Repeater stacks to
ErrorRecoveryLevel=2 in parallel, it became very obvious that
ErrorRecoveryLevel=1 was designed to make implementations as through as
possible and force vendors to seriously take into consideration all the
needed extra funtionality that comes with multiple connection sessions.

One issue that immediatly comes to mind is the logging out of additional
connections on the fly with a logout request with either the same or
differing CID than the connection it arrived on.  It is pure speculation
as to if vendors who ultimately decide to take the multiple connection
per session at ErrorRecoveryLevel=0 route will implement any of this
functionality.  IMHO, from the various inquries on the reflector
regarding multiple connections per session and ErrorRecoveryLevel=0, I
seriously doubt to see any of this additional required functionality
being implemented.

Thanks! 

On Thu, 2003-11-20 at 12:31, Mallikarjun C. wrote:
> Nick,
> Why?
> The iSCSI semantics wrt multi-connection sessions are
> always true (really relevant based on the operational value
> of the MaxConnections key), not tied to the operational
> ErrorRecoveryLevel key.
> 
> M
> 
> 
> Nicholas A. Bellinger wrote:
> 
> > Dan,
> >

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 20 Nov 2003 15:03:08 -0800

Let me attempt to clarify this if I can.

ErrorRecoveryLevel key as the name implies
exclusively deals with error management.
An OOO command arrival to the extent it happens
due to network skew is *not* an error and so
its handling is not governed by the ErrorRecoveryLevel
key.

All the semantics defined in the spec for
multi-connection sessions have to be followed
regardless of ErrorRecoveryLevel (i.e. even at = 0).
If an implementation supports MaxConnections > 1
but not support related iSCSI spec semantics,
then it isn't compliant to the spec.

Regarding the extent of support for Logout
flavors, I cannot speak for specific implementations,
but let me note that except for recovery logout
which is dependent on ErrorRecoveryLevel=2 support,
all other semantics are mandatory to support.

M