Subject: RE: iSCSI:flow control, acknowledgement, and a deterministic reco very

From: [email-masked]
Date: Fri, 6 Apr 2001 09:11:26 -0400

Let me see if I can distill out the issue in Doug's long
message on this subject ...

When a CmdSN of zero is used to mark a command for immediate
delivery, the CmdSN-based acknowledgement and windowing
mechanisms don't apply to that command.  This means that
any command sent for immediate delivery (CmdSN = 0):
(A) Cannot recover from a CRC (digest) error via CmdSN-based
	retransmit.  The Initiator can still time this out,
	but that seems like a poor way to initiate recovery
	of something that was supposed to be done immediately.
(B) Does not have its use of resources (e.g. command buffer)
	on the target controlled by the CmdSN windowing
	mechanism, complicating target resource management.
	Targets have better control over their resources if
	all inbound commands use one set of resources,
	described/controlled by the CmdSN window, rather
	than having to set aside separate resources just
	in case the initiator sends an immediate command. 
The alternate proposal that Doug envisions is to apply
CmdSN (and hence its error recovery and resource management)
to immediate commands, and use a flag  bit elsewhere in
the header to indicate that the command is to be delivered
to SCSI immediately by iSCSI rather than waiting for
"missing" commands to show up.  That seems reasonable,
and comments are invited.  One of the things that this
does is transfer the responsibility to keep some space
in the CmdSN window open for immediate commands (and
determine how much is appropriate) from the Target to
the Initiator - all other things being equal, this
is the right direction to move functionality in a SCSI
system.

Note Well: This concept of "immediate" delivery is an
iSCSI concept that affects only the iSCSI CmdSN sequence
- this does not affect TCP's "deliver in order" behavior.

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

> -----Original Message-----
> From:	Douglas Otis [SMTP:[email-masked]]
> Sent:	Thursday, April 05, 2001 11:34 PM
> To:	Ips
> Cc:	David Black; Elizabeth G Rodriguez (Elizabeth)
> Subject:	iSCSI:flow control, acknowledgement, and a deterministic
> recovery
> 
> Ver 5 Pg. 10
> 
>    "Command numbering is session-wide and is used for ordered command
>    delivery over multiple connections.  It can also be used as a
>    mechanism for command flow control over a session."
> 
> "1.2.2.1 Command Numbering and Acknowledging
> 
>    iSCSI supports ordered command delivery within a session.  All
>    commands (initiator-to-target) are numbered."
> 
>    "Commands in transit from the initiator SCSI layer to the SCSI target
>    layer are numbered by iSCSI; the number is carried by the iSCSI PDU
>    as CmdSN (Command-Sequence-Number).  The numbering is session-wide.
>    All iSCSI PDUs that have a task association carry this number. CmdSNs
>    are allocated by the initiator iSCSI within a 32-bit unsigned counter
>    (modulo 2**32).  The value 0 is reserved and used to mean immediate
>    delivery. Comparisons and arithmetic on CmdSN SHOULD use Serial
>    Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32.
> 
>    Not covered in this document are he means by which the SCSI layer may
>    request immediate delivery for a command or by which iSCSI will
>    decide by itself to mark a PDU for immediate delivery."
> 
> 
> Not all commands are serialized as a result of serial number 0
> representing
> a special case for immediate delivery.  This has an impact on flow
> control,
> acknowledgement, and a deterministic recovery in the face of an error
> situation.  With the exception of those commands with null serialization,
> all commands MUST be sequenced at the point of network aggregation
> described
> here as a PDU sequencer that issues commands to the SCSI target.  This
> focal
> point is likely to find situations where its normal operation is
> curtailed.
> 
> 	1) Prolonged device operation resulting in a resource constraint.
> 	2) Digest Error causing a missing sequence.
> 	3) Connection loss causing a missing sequence.
> 
> The technique of using a null sequence to bypass the sequencer has some
> inherent problems.  In first case, those queued commands MAY become
> invalid
> following management that terminates prolonged operation with a command
> that
> has bypassed the sequencer.  Those invalidated commands queued within the
> sequencer can not be cancelled in an orderly manner within the existing
> scheme.  This sequencer MUST be used as defined in the iSCSI proposal and,
> as a result, these queued commands are beyond SCSI and iSCSI controls.
> Issuing a null sequence command followed by a replicate serialized command
> will have differing results depending on the target but will not result in
> a
> deterministic treatment of these pending commands.
> 
> The use of the sequencer bypass technique (null serialization) should
> signal
> an extreme measure where logically, those commands being bypassed become
> suspect.  The conservative approach to this situation would be to reject
> all
> bypassed commands.  As a result of this conservative behavior, a technique
> that does not use a null sequence would be to institute a flag such as
> "Exigent" that signals an extreme condition exists and that all pending
> commands within the sequencer are to be rejected.
> 
> Note:
> In addition to checking for the next PDU sequence, the sequencer should be
> checking for PDUs with a serialization number that are prior to the
> desired
> next sequence.  This examination would look something like this:
> 
> if ( (sequencer CmdSN - next sequence CmdSN ) > 2^(SERIAL_BITS - 1))
> 	{
> 	reject_pdu(CmdSN, SEQUENCER_INVALIDATION);
> 	}
> 
> if (sequencer CmdSN == next sequence CmdSN)
> 	{
> 	send_pdu(CmdSN);
> 	next sequence++;
> 	}
> 
> Upon receipt of a PDU flagged as "Exigent", the 'next sequence' value
> immediately becomes the serial number of this command as well as ExpCmd
> advancing to this value plus one.  The effect of this is to have the
> sequencer reject all pending commands up to the "Exigent" command.  This
> has
> the benefit of removing these now suspect commands as well as allowing
> this
> highly urgent command to be sent to the target immediately without
> accidentally affecting subsequent commands as is now possible.
> 
> As it is now, one other possible use of a null sequence would be to always
> bypass the sequencer as perhaps the sequencer is viewed as unnecessary.
> The
> use of zero to allow commands to bypass the sequencer then represents a
> problem with respect to resource management as now the flow control scheme
> no longer works.  If bypassing the sequencer is the desired behavior and
> this command will not impact validity of those commands serialized prior
> to
> this command, then this PDU flagged "Casual" would allow this command to
> be
> issued directly to the target.  These commands should still include a
> serial
> number to allow flow control and acknowledgement to remain functional.
> The
> task of acknowledgement would be to comprise a min-max list of those
> commands sent and to look for the highest sequential value.
> 
> In the case of a lost connection, waiting to time-out on those holes
> created
> within the sequencer would be one method of handling this situation.  If
> there was a means of rejecting those commands within the sequencer using
> an
> "Exigent" command would mean there are no holes left to fill.  Those
> commands received would be rejected and the initiator could then resend
> these commands on a different connection without stumbling through a
> process
> of repeated timeouts.  Should the method used to recover from a digestion
> error mean terminating the connection, then those commands can also be
> quickly shifted over to a new connection.
> 
> This does not resolve all issues created in a error event, but it does
> provide a simpler solution for most of those events concerning the
> sequencer
> and gets rid of the special case handing of the command serial number.
> Not
> having flow control, acknowledgement, and a method of dealing with queued
> commands appear as a serious flaw in the present protocol.  I hope I have
> presented this in a clear manner and I am interested in finding how others
> deal with these situations.
> 
> Doug

---

From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>
Date: Fri, 6 Apr 2001 15:21:24 +0100

Alternatively, only allow one command with CmdSN=0 outstanding at anyone
time.  This has the advantage of only having to reserve one buffer at the
target,  the retry bit could still be used if the command or response has a
digest error, and finally its simple!.  The obvious disadvantage is that
there can only be one message is progress at a time - is this an issue?

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010
E-mail: [email-masked]

> From: [email-masked] [mailto:[email-masked]]
> Sent: 06 April 2001 14:11
> To: [email-masked]; [email-masked]
> Subject: RE: iSCSI:flow control, acknowledgement, and a deterministic
> reco very
> 
>

---

From: [email-masked]
Date: Sat, 7 Apr 2001 11:27:48 +0200

Doug & David,

Commands can be recovered based on their ITT (that is the main identifier).

Nevertheless the idea of a separate immediate delivery flag is appealing.

The idea was raised many months ago and I will have to review some old
notes to find why we dropped it in the first place (except that FCP at the
time and SCSI now use the same marking - i.e. 0 for immediate).

Julo

[email-masked] on 06/04/2001 15:11:26

Please respond to [email-masked]

To:   [email-masked], [email-masked]
cc:
Subject:  RE: iSCSI:flow control, acknowledgement, and a deterministic reco
      very

Thanks,
--David

---

From: [email-masked]
Date: Mon, 9 Apr 2001 11:11:44 -0400

> The main reason for selecting 0 and not a flag for immediate delivery was
> to enable immediate delivery even when the command window is closed.
> 
> However we can achieve the same effect by using an immediate flag and
> using the current CmdSN without advancing it.  With this we get a
> reference to where in the stream the command where supposed to act if the
> stream order is important.

I think this causes resource management complications on the target
because reusing the CmdSN for an immediate command requires a new
command buffer, unless we allow the target to arbitrarily reject/abort some
command to make room, which seems wrong (the immediate command
may not be a task management command, and even then arbitrary
rejects/aborts may not be desirable). 

How is this better than requiring the Initiator to not permit the command
window to close (i.e., the Initiator always keeps one slot in the window
open for one immediate command, or N slots for N immediate commands,
or no slots if it wants to live dangerously)?  I think existing (SCSI and
FC)
Initiators have to do this sort of resource management anyway, as I don't
believe immediate commands are exempt from TASK_SET_FULL.

There appears to be a "no free lunch" principle here in that some piece
of the iSCSI Initiator-Target system somewhere has to be cognizant
of the fact that resources for immediate commands are being 
reserved on the target, and Initiator control of the command window
(coupled with a requirement that MaxCmdSN never decrease) seems
to be the easiest way to get there. 

--David

---

From: Charles Monia <[email-masked]>
Date: Tue, 10 Apr 2001 13:16:26 -0700

> TASK_SET_FULL *is* a flow control mechanism, albeit a
> rather primitive one - the Initiator issues commands until
> it gets a FULL response and hence knows the capacity
> of the Target's queue.

Using task set full to probe in this way is a guess at best. It's certainly
not behavior specified by any standard. Getting a task full after "N"
commands does not guarantee that the LU will have room for "N" commands in
the future. For efficiency reasons, logical units are loath to make any such
resource guarantee to individual initiators.  Earlier postings have
discussed this at some length.

> Santosh did not propose to remove CmdSN, although he
> did point out an issue that needs attention - we need
> to say something about the interaction of CmdSN with
> TASK_SET_FULL since both mechanisms may be implemented.
> In particular, it's possible to use CmdSN for delivery
> ACKs and TASK_SET_FULL for flow control, although that's
> a "peculiar" combination (and perhaps a SHOULD NOT?).
> 

The only thing to be said is that there is no interaction.  For example, an
iSCSI gateway fronting a bunch of FC devices has no real way to tell how
many commands each device has room for.  So, a logical unit may return a
TASK_SET_FULL response even though the CmdSN window was wide open.
Naturally, the converse is also true. I.e., the CmdSN window may close even
though logical units have room for additional commands.

Charles

---

From: [email-masked]
Date: Thu, 12 Apr 2001 04:30:20 -0400

Summarizing the state of this dicussion:

(1) The idea for use of regular CmdSN numbering and
an "immediate" flag for immediate commands instead of
a zero CmdSN appears to have received a positive reception,
although the details of how CmdSN is used for immediate
commands are still open (use number in sequence vs.
duplicate what would be the next number).  Since
immediate commands are currently exempt from the
requirement to deliver from iSCSI to SCSI in order,
the default way to do this would be to exempt them
from the "MUST deliver in order" text that Doug
repeatedly quotes.

(2) It looks like some words are in order to advise
implementers to watch out for TASK_SET_FULL interactions
with CmdSN-based windowing possibly including restrictions
on issuing TASK_SET_FULL when the CmdSN window is still
open.  While the TASK_SET_FULL usage I described isn't
always present or used, there are Initiators and Targets
that behave in that fashion and hence we need some words
to advise those who plan to carry that mechanism forward
in order to avoid surprises when SCSI code (or code above
it) receives a TASK_SET_FULL.

Now the open issue:

(3) My understanding of the outstanding issue is the
rejection mechanism whereby arrival of an immediate
command with a CmdSN higher than the CmdSN of the last
command delivered to SCSI would then cause rejection
(return to sender) of all undelivered commands with
lower CmdSNs.  I think Doug is asking that support for
this mechanism be available but not be required.

The overall goal for this is apparently:

> My concern is regarding the affect [on] existing
> interfaces where there is not this potential for
> a large queue of commands that can not be purged
> except through action at the target at a much
> later point in time.

> I am asking for predictable behavior in the event
> of the sequencer being bypassed.

First of all, we need realistic motivating scenarios for
this.  The Rewind scenario isn't realistic, as Doug has
agreed that it's an application bug.  Beyond this, the
statement that existing interfaces cannot build up a large
queue of unpurgeable commands is false - if the application
issues a bunch of writes just after the Rewind, the same
situation results, and again the application is buggy.

The original motivating scenarios involved task management
commands, but I thought the following comment from Bob
Snively closed out those scenarios:

  At present, SCSI applications do not have a clear
  guarantee of the order between task management functions
  and the processing or delivery of any particular task.
  In fact, the concept of an ORDERED attribute applied
  to a task management function is unknown.  As a result,
  SCSI drivers have to be aware of the implications.

I'm assuming that Bob's comment applies to tape as well
as disk (tape experts, please correct this if it's wrong).

Absent a plausible description of what's broken, it's
hard to justify designing a fix.  There are already
situations in which SCSI behavior is unpredictable,
so a general appeal to the goodness of predictability
does not suffice - it needs to be accompanied by one
or more realistic scenarios in which such a fix would
make a visible difference.

Thanks,
--David