Subject: RE: iSCSI: CmdSN and Retry

From: "Y P Cheng" <[email-masked]>
Date: Thu, 1 Feb 2001 12:07:49 -0800

Julo wrote:
> Except for sending the status - an executing command helds-up the LU queue
> and makes the "local" recovery simpler than clearing the LU queue and
> resending the commands.
Santosh wrote:
> This is correct, in the case where the target does NOT implement
> data/status recovery. i.e. Assume ordering is required, 2 commands , say,
> 1 & 2, executed in order at the target. Now, if 1 encountered a digest
error
> or format error at the initiator, and was re-sent with the "retry" bit
> AND the target were to NOT implement data/status recovery, it would result
> in target executing 1, 2, 1. This may be a problem and canot be addressed,
> unless iSCSI mandates data/status recovery.

I certainly understand the need of doing data/status recovery and the
argument of "local" recovery being simpler.  However, this comes with heavy
cost of performance when pipelined design demands 100,000 IOs per second on
a network with long delay.  For services and responses happening a few times
per second, it is OK to hold on the resources until we are certain the ACK
is returned.  However, in the example above, after completing command 1 if a
target can't start command 2 until the status for 1 is ACK'ed, the wait can
be 100 milliseconds on a network with long delay.  The wait make it
impossible to have large number of IOs in the pipeline. By mandating
data/status recovery in iSCSI, we change the pipelined command execution to
interlock handshakes.  As I have said in the previous email, an initiator
will never send an command which depends on the success of a previous
command.  This fact makes the pipeline execution in a target possible.

On a separate note, I really respect Santosh's fine-tooth analysis of the
iSCSI draft.  But, in his arguments the fact that SCSI has been functional
for the last 20 years was badly ignored.  The CmdSN, DataSN, and StatSN
allow iSCSI to detect missing PDUs and to quickly ask for retransmit.  They
should not be used to enforce sequentiality to slow things down.  SCSI
already has the semantics of ordered execution that requires the help of
CmdSN when multiple TCP connections are used. However, using StatSN to
mandate data/status retry pays a great performance price. Both overlapped
and out-of-order data transfers are allowed in SCSI (Check out the Modify
Data Pointer extended message).  SCSI works fine without mandating
non-overlapping transfers or data/status recovery.  Retry can be done in a
simple and clean manner without introducing complicated semantics for CmdSN,
DataSN, and StatSN.  Note, if we must retry more than once in a million IOs,
something is wrong of the infrastructure.  Therefore, let the pipeline flow
quickly and don't optimize the retry.

As long as we separate the TCP, iSCSI, and SCSI ULP layers cleanly -- for
which this WG has done a good job -- SCSI will continue to work.  Without
wasting more bandwidth on this subject, I will be willing to discuss the
SCSI retry implementations with anyone offline.

---

From: Santosh Rao <[email-masked]>
Date: Thu, 1 Feb 2001 15:17:36 -0800 (PST)

> argument of "local" recovery being simpler.  However, this comes with 
> heavy cost of performance when pipelined design demands 100,000 IOs per 
> second on a network with long delay.  For services and responses 
> happening a few times per second, it is OK to hold on the resources 
> until we are certain the ACK is returned.  However, in the example 
> above, after completing command 1 if a target can't start command 2 
> until the status for 1 is ACK'ed, the wait can
> data/status recovery in iSCSI, we change the pipelined command 
> execution to interlock handshakes.  
> As I have said in the previous email, an initiator

YP,

I agree with the above and have been saying the same thing in a seperate
thread. (See : "Command Ordering Proposal"
http://ips.pdl.cs.cmu.edu/mail/msg03171.html)

(The corner cases stated simply point out issues with the SNs as they exist
in the draft today.) The bigger question of why iSCSI is attempting to
provide ordering when its peers do not is still un-answered.

iSCSI is trying to enforce strict ordering which is not required due to the
following reasons :

1) Other SCSI Transport Protocols (FC, pSCSI) do not guarantee strict
ordering across commands. 

2) Most (All ?) operating system SCSI Stacks enforce ordering above the
SCSI ULP and do not depend on the SCSI Stack to provide end-to-end
ordering across commands. (The exception to this is ordering of commands
followed by their aborts).

3) In the absence of (1) & (2) above, applications cannot depend on SCSI
transport protocols to provide strict end-to-end ordering across commands,
since they are transport agnostic and do not wish to base ordering
assumptions on the transport they operate on.  (i.e. appln code enforces
ordering if it operates on FC and pSCSI and uses iSCSI ordering when it
runs above iSCSI !)

4) Error recovery scenarios needed to maintain ordering across transport
errors, resource allocation errors, errors at the target, digest errors
and format errors becomes extremely complex, stalls all pipelined
operations and in some cases cannot resolve the out-of-order situation
induced unless the target executes 1 command at a time and ONLY starts on
the next command after a StatSN ACK for the previous one.

5) Strict ordering results in performance penalties such as :
a) non-concurrent command execution at the targets.
b) stalled TCP connections in a session when a connection turns faulty.
c) all commands need to be stalled and re-initiated on any I/O error in an
attempt to maintain ordering.
d) Flow Control mechanisms like QUEUE FULL error recovery will change to
complete oscillating algorithms with the initiator completely stopping all
further I/Os until order is restored in the sequence they were originally
sent.

6) Strict ordering is not required in the majority if not all cases, since
most (all ?) O.S' do not enforce strict ordering or do anything 
special in their error recovery to  maintain strict ordering.

7) Several targets treat simple and ordered tags in a similar manner
without differentiating and providing strict ordering for ordered tag
commands.

8) Majority of the I/O traffic (if not all) is simple tag commands and all
this ordering and error recovery for ordering will impact performance when
ordering was never required in the first place.

9) The use of ordering within a SCSI stack is not prevalent. IN those few
places where it is known to be in use, this is a static characteristic of
that SCSI stack, and such stacks can use single-connection sessions to
achive their ordering.

IOW, the majority of stacks do NOT require strict ordering. Hence, iSCSI
should optimize for this most common case and NOT penalize it. 

> for the last 20 years was badly ignored.  

YP, the corner cases simply bring out problems in the draft as they exist
today. I have been consistently raising the bigger question about why
iSCSI is attempting to differ from its peers in issues like ordering,
overlapped data xfer's, etc. SCSI has been functioning over the last 20
years without applns depending on SCSI stacks for strict ordering or
deploying un-used features like overlapped data xfers. 

Applns will continue NOT to depend on scsi transports for the above since
they are written to be transport agnostic, and are based on lowest common
denominator support. (i.e. assume no ordering from O.S SCSI Stack).

> However, using StatSN to
> mandate data/status retry pays a great performance price. 

Agreed. 

> Both overlapped
> non-overlapping transfers or data/status recovery.  

The Modify Data Pointers needs to be explicitly enabled through Mode
Select and I can't recollect seeing any instances of its usage.
As for FCP, overlapped data xfer's are not allowed. period.

Regards,
Santosh

-- 
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : [email-masked]
Phone : 408-447-3751
#################################

---

From: Santosh Rao <[email-masked]>
Date: Thu, 1 Feb 2001 15:26:01 -0800 (PST)

> Unless the transport maintains order, there will be a requirement to wait
> for each command to complete to assure the sequence.  

Doug, 

The majority of SCSI stacks today DO function as stated above.
The layer above ULP enforces the ordering and awaits completion of 1 I/O
prior to posting the subsequent I/O , PROVIDED those 2 I/Os required
ordering to be maintained.

> This makes 
> compliance to SCSI impossible and not a means of improving the pipe-line.  
Are you saying that the majority of stacks today are not 
SCSI compliant (?)

YP raises a valid concern in that if strict ordering is expected from the
transport, it should be capable of providing this 100 %. (98% strict
ordering is'nt good enough.). To attempt to provide 100% guarantee of
strict ordering is going to prevent concurrent I/O processing at the
target at a session level (NOT even a LUN level, which is the granularity
where one would expect ordering, if at all !).

Regards,
Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 1 Feb 2001 14:23:20 -0800

Y.P.

Any TCP bandwidth is limited to .93 MSS/(RTT * loss_rate^-2) so your
assumption of one in a million at 100 milli-seconds limits at Fast Ethernet
rates.  There will be significant overhead on long pipes dealing with TCP
handshakes and any additional handshake will not add significantly to this
resource overhead.  The more immediate these handshakes, the less resources
consumed with respect to any needed replays to satisfy the transport.  I do
not agree procrastination that depends on rediscovery of indexes unrelated
to these handshakes to be more effective in allowing pipe-line execution.
Unless the transport maintains order, there will be a requirement to wait
for each command to complete to assure the sequence.  This makes compliance
to SCSI impossible and not a means of improving the pipe-line.  An immediate
handshake within the transport layer dealing with digest errors is the best
means of improving performance.  Reliance on the SCSI tag to recover from a
failure deduced from a dropped transport sequence is not a clean separation
of layers.  The amount of resources saved by deleting this positive
relationship is dwarfed by the window size required for such a fat pipe.

Doug

Should each IO represent

> execution to

---

From: "Y P Cheng" <[email-masked]>
Date: Thu, 1 Feb 2001 15:08:31 -0800

Doug,

I think you have misunderstood what I said about error recovery in different
layers.   I specifically have stated that we should separate the three
layers: 1) Transport layer provides in order delivery on Internet, 2) iSCSI
keeps the in order among multiple connections, and 3) SCSI has its own in
order command semantics.  Frame retransmit and congestion control belong to
the transport.  Digest error retransmit belongs to iSCSI layer.  Finally,
SCSI itself has done command level recovery very well for the last 20 years.
We should mix iSCSI and SCSI retries.  I want to be able to pipe 100,000
frames down the wire without letting retry and ACKs getting into the way.
In my previous writings, I have stated that for TCP to support 10Gb/100Ms
network with a huge reassembly buffer, message framing and TCP/RDMA must be
used.  Whether the maximum TCP bandwidth supports such a high IO rate is
beyond the Charters of this WG.  I have no problem in using SCTP, if this is
what it would take for the 10Gb/100Ms network.

Y.P.

---

From: Santosh Rao <[email-masked]>
Date: Thu, 1 Feb 2001 16:36:14 -0800 (PST)

> An extra handshake that requires a small delay should there
> be a non-sequential event would occur orders of magnitude less often than
> similar waits for TCP.  As this must be a rare event, the wait will be
> insignificant.  

Doug,

The non-sequential event may be rare. However, the handshake should occur
on EVERY I/O to safeguard against that rare event. [if 100% ordering is
attempted.]. 

Strict ordering of 2 commands, say,  A followed by B, imply that : 

A target cannot execute B until the initiator acknowledges Status 
PDU for A. (This implies sequential operation on I/Os on the task set
similar to untagged targets, only greater, because this behaviour is per
session and not even per LUN.)

Since iSCSI applies strict ordering to all commands (well, all non-0 CmdSN
commands), the above behaviour will be required on all I/Os within a
session.

Regards,
Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 1 Feb 2001 16:15:26 -0800

Santosh,

TCP mandates sequential delivery.  As was often stated on this reflector,
framing would be to place data payloads but would not impact the sequence of
any operation.  An extra handshake that requires a small delay should there
be a non-sequential event would occur orders of magnitude less often than
similar waits for TCP.  As this must be a rare event, the wait will be
insignificant.  Assurance of sequential delivery improves reliability and
does not require command-by-command pacing to provide expected SCSI
operation that otherwise is easily done with parallel SCSI configurations.
The buffer size will be nominally one packet larger to handle such extra
handshake and again insignificant.  I see nothing to justify complexity
offered by convoluted recovery for a digest error.  SCTP offers a means of
delivering objects out-of-sequence and then perhaps flagging which objects
can violate SCSI's sense of ordering could be done to help facilitate long
pipes.  If this ever becomes a significant factor, the bandwidth will be so
small as to be of little concern anyway.

Doug

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 1 Feb 2001 17:35:27 -0800

Santosh,

1. iSCSI is based on TCP, a protocol that mandates sequential delivery.
2. TCP provides insignificant bandwidth if there is a high error rate.
3. Digest errors will happen at the 16-bit checksum undetected error rate.
4. Higher latency of networks makes such command-by-command constraints less
effective than a transport that assures sequencing.
5. There are many cases (not most) where order is significant. You could
make a better case for ensuring sequencing at the transport level than
negligible benefits derived from non-sequential processing of commands
following a digest error.

> on EVERY I/O to safeguard against that rare event. [if 100% ordering is

The only potential delay would be with respect to any skew between
connections.  If the PDUs are delivered in order, as they would be normally,
then there would be zero delay in the processing of the information.  The
pending handshake would only require the sender to retain a structure that
relates the sequence value to the PDU sent.  This handshake is trivial,
already within the structures, and only requires the return of an additional
packet before this structure can be removed.  Yes every I/O would be
examined to ensure its sequence but the same check is made to acknowledge
now.  Normally this results in no extra operation or delay.

> session and not even per LUN.)

As status should enjoy the same independent assurance of delivery, SCSI
would not be impacted by a transport error as you imply nor would there be a
need to execute command-status-command-status. If you could not assure
reliable sequential delivery, then should such become important, then indeed
you would be required to execute command-status-command-status.  The benefit
from using the handshake would be to prevent this requirement.  Digest
errors will happen at the 16-bit checksum undetected error rate.  We are
taking about a flea of a flea with respect to any loss of performance.

> commands), the above behaviour will be required on all I/Os within a

This is the claim but a claim that can not be assured due to the lack of
integrity in the current handshake.

Doug