Subject: RE: iSCSI:flow control, acknowledgement, and a deterministic reco very

From: [email-masked]
Date: Mon, 9 Apr 2001 15:08:18 -0400

> Ver 5 Page 11:
> 
>    "The iSCSI target layer MUST deliver the commands to the SCSI target
>    layer in the order specified by CmdSN."
> 
> I am not making an assumption about there being a CmdSN sequence function.
> It is a MUST within the proposal!  Yes, you can bridge holes within CmdSN
> sequence using pointers and that would be expected and I described this as
a
> list.  This list still must be sorted between sequential and immediate
> especially when multiple connections are used to determine a next in
> sequence.

Aha ... here's a problem.  The text Doug quotes can't apply to existing
immediate
commands, because the current use of a zero CmdSN would require them to be
delivered before all other commands - what if the first immediate command
shows
up after 3000 non-immediate commands?  If we adopt the proposal to use
regular
CmdSNs for immediate commands, we'll need some text indicating that the
above MUST does NOT apply to immediate commands - i.e., iSCSI is expected
to deliver them to SCSI immediately after TCP delivers them to iSCSI no
matter
what sate the CmdSN window is in.  There is no need for rejection -
if worst comes to worst, a dummy command whose "deliver to SCSI"
operation is "do nothing" will keep Doug's sequencer happy even though
the actual command was delivered out of order.

> This list still must be sorted between sequential and immediate
> especially when multiple connections are used to determine a next in
> sequence.

That's one option.  Another is to have two lists so the immediate commands
don't get mixed with those for ordered delivery.  As I said, this is within
the realm of implementation choices.

> You would not want the creation of these lists to be blocking
> until the next in sequence is seen as you seem to imply.

I suspect Doug has misread what I wrote - most of the time, there's
no need to block for "next in sequence". OTOH, there is a "no free lunch"
principle in here that says Target resources are limited - if the Target is
temporarily out of resources, not much is going to happen until it gets
resources, and in that case, blocking is a reasonable first step.
Rejecting perfectly valid commands just to free resources doesn't seem
like the first thing that a Target should do (e.g., vs. waiting for SCSI to
finish something).

> You view this as a pointer list, again I agree.  The use of rejection
still greatly
> simplifies this process.

I still believe that "greatly simplifies" is implementation-specific until
someone
else speaks up in support of it.

> You are wrong about the use of CmdSN being limited to just
> the flow control window.

That's incorrect.  Once the command is ready for delivery to SCSI,
CmdSN is not needed to track it if there's some other way to do so.
"Deliver in CmdSN order" doesn't require that an implementation use
CmdSN to track that order, and the fact that immediate commands
put holes in this ordering may lead an implementation to use
something else.

> To not support rejects, you are then relying on the target to handle these
> now out of sequence commands.  

Immediate commands are "out of sequence" by definition - immediate
means deliver immediately without regard to CmdSN order.  Only the
target can do this, so I don't see any problem with relying on the Target
to do so.

>  In addition, that there be spare overhead allotted to handle these
> undetermined number of "immediate" commands.  

That's open to debate and depends on whether we require the CmdSN
to be within the window (as originally proposed) or allow one (more
than one?) beyond the window to be used (as Julian has proposed).

> You will then also not see any immediate acknowledgement of these
> "immediate" commands sent to the target unlike ALL other such commands.

In most of the cases, the TCP ACK will indicate that the bytes got there.
If there's an iSCSI CRC failure, then we're back to the CmdSN sequencing
to catch it and clean up, and that might not be "immediate" - but is that
going
to happen often enough to be worth any additional effort (rejects, immediate
ACKs, something else?) to optimize for performance?  I'm sceptical,
especially if a Cmd SACK/SNACK is proposed to correct this.

> Can you answer how many "immediate" commands are allowed?

Initiator determines this by appropriate management of usage of the CmdSN
window.
If it keeps N slots open in the window, N immediate commands can be sent
immediately.

> When would you expect these "immediate" commands to be acknowledged to
> allow more such commands?

When the resources from executing an immediate command are no longer
needed on the Target, MaxCmdSN would advance to allow an additional command
in, even if ExpCmdSN has not moved.  The Initiator has to spend some of its
time dealing with keeping ExpCmdSN moving (i.e., resend the command
at ExpCmdSN) because Targets are unlikely to support arbitrarily large
windows.

--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 9 Apr 2001 13:20:45 -0700

David,

> > To not support rejects, you are then relying on the target to
> > handle these now out of sequence commands.
>

The commands that are NOW out of sequence are those commands bypassed by the
immediate command.  If this immediate command was Rewind, then a long list
of write commands become invalid as a new tape is now needed for those
operations.  You are expecting that the target to handle a long list of
invalid commands made possible by the iSCSI sequencer.  Commands invalidated
by immediate commands become out of sequence commands.

I'll reserve further comment until Julian has made some progress in
addressing these concerns.  I also suspect it is a mistake to allow commands
to remain trapped within the sequencer during emergency or abnormal events
signified by the use of these immediate commands.

Forgive my advocacy, but if one does not attempt to support a position, then
the subject is not explored.

Doug

---

From: Santosh Rao <[email-masked]>
Date: Mon, 09 Apr 2001 18:40:21 -0700

Doug [& All],

Some comments on this thread :

1) The immediate command feature can be exploited by initiators which
are not required to provide strict command ordering for their SCSI
sub-system. (The majority of today's scsi stacks). IOW, all commands can
be sent with CmdSN=0 to indicate no ordering required.

Any scheme to restrict the number of "immediate commands" that can be
sent prohibits such a feature and is un-desirable.

2) The (ExpCmdSN, MaxCmdSN) based acknowledgement and flow control is a
freebie that targets can use to implement additional flow control.
There's nothing in the spec that mandates that a target MUST use this
feature to throttle its window and apply flow control.

Hence, any dependence and assumptions on the existence of CmdSN based
flow control may be incorrect, since the target may be using QUEUE FULL
(TASK SET FULL) to apply flow control.

> I also suspect it is a mistake to allow commands

3) If the initiator suspects that the CmdSN queue at the target is
stuck, it can always use a task mgmt command or even an iSCSI login with
the X (restart) bit to perform a cleanup of stuck commands at the
target. There's no need to build in implicit assumptions that a CmdSN
with the immediate flag result in a clean-up of previously pending
commands at the CmdSN queue of the target.

- Santosh

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 9 Apr 2001 23:09:01 -0700

Santosh,

In a change recommending a flag and a valid CmdSN and not a null value, I
included a flag named "Casual" to provide this mode of operation you
describe which still allows flow control.  I would not wish to permit this
Casual mode, but as David feels such lists or threads are easy to manage,
this should not represent any greater effort for feedback.  But then again,
if you are not using flow control, you are also not using any
acknowledgement other than an eventual SCSI status.  If there is no flow
control, this removes resource control and reception acknowledgement.  If
this is not needed, then such flow control should be removed for simplicity
as there then must always be an abundance by design of adequate recourses.
For cases where command sequence is important, CmdSN together with
acknowledgement must remain or only one command at a time can be issued.  I
can envision three modes, a Sequential mode, a Casual mode, and an Exigent
mode where the command is given head of sequence status (with rejection for
feedback simplicity ;).

You seem to be placing many things on the table for reconsideration.  Flow
control, acknowledgement, sequential delivery, and perhaps even a means of
moving a command up the queue feeding SCSI targets.  One command at a time
for sequential assurance is not impossible.  Is flow control needed?  Do you
care if there is any reception acknowledgement if all feedback is in the
SCSI layer?  It would seem in this barebones case, iSCSI adds little to
ensure integrity.  The FC encapsulation places a time stamp within the PDU.
Perhaps a time stamp should be employed as an alternative to sequence
control?  At least after a timeout, you are assured a missing command is
forever gone.  Otherwise, the check was to see that CmdSN is not too small
to ensure it is not a stray.

Doug