Subject: iSCSI:out-of-order notification proposal

From: Sandeep Joshi <[email-masked]>
Date: Fri, 20 Apr 2001 12:43:02 -0400

This is a proposal to allow the initiator to inform the target 
if out-of-order execution within the command stream is possible.

The target execution can rotate between "in-cmdSN-order" and 
"out-of-order" during runtime as informed by the initiator..

Appreciate comments on subtleties that I may have missed.

thanks,
-Sandeep

http://ips.pdl.cs.cmu.edu/mail/msg03152.html
   by Costa provides a good summary of the issue at hand.

http://ips.pdl.cs.cmu.edu/mail/msg04255.html
  David provides the "new" reqts.  In particular, this one

   > MUST specify the ability to preserve ordered delivery
   > of SCSI commands even in the presence of transport
   > errors.  A mechanism MUST be provided to allow
   > Initiators and Targets to negotiate this preservation
   > on a per-session or finer granularity basis

Note :
======
1) This does not rely on SCSI cmdRN, but operates at iSCSI level.
2) Flow control using cmdSN works as designed.
3) This solution is not a per-session negotiation option but can be
   disabled and re-enabled again at "runtime" by the initiator if it
   notices that Ordered/HOQ tasks (or any other need) have entered
   the iSCSI command stream which is being dispatched.

Problem:
=========
In case of out-of-order arrival or digest errors, it is NOT possible
to know if the initiator had sent an ordered command before the one
which was received.

Solution:
=========
To notify target of presence of in-flight ordered commands, we set
a flag on *every* PDU following the ordered command *until* the target
moves it expCmdSN above the cmdSN of the ordered command.  The
expCmdSN indicates target has found the ordered command.

( Those familiar with "ECN over IP" (Floyd, et al) may see this is
similar to how a congestion bit keeps being set until the sender acks
that it has received the notification)

Figuratively:
=============
So, assume there was a new "strict_order" flag in the BHS.
In the figure below, braces shows value of (cmdSN, strict_order)

Initiator                       Target
---------                       -------

simple cmd (cmdSN=100, strict_order=0) ->
simple cmd (101, 0) ->
simple cmd (102, 0) ->

   these may get reordered or have digest errors ->

                        target executes as they arrive
                             exec simple cmd(102, 0)
                             exec simple cmd(100, 0)
                             exec simple cmd(101, 0)

Now say the initiator wants to send a HOQ task.
It sets strict_order=1 on all PDUs

ordered cmd (103, 1) ->
simple  cmd (104, 1) ->
simple  cmd (105, 1) ->
                        in case of reordering or digest errors
                        target must wait & execute in cmdSN order!
simple  cmd (106, 1) ->
simple  cmd (107, 1) ->

         <---- now target sends expCmdSN=103

This implies target has seen command(cmdSN=103) and target will do the
appropriate ordering and delivery to SCSI layer.  This is left to
the target implementation to tackle.

Initiator checks (expCmdSN >= cmdSN of ordered cmd) and then resets
strict_order to zero for all subsequent PDUs.

simple  cmd (108, 0) ->
simple  cmd (109, 0) ->

Other issues:
=============
If the basic scheme is ok, then we could later tackle other questions
"what about multiple ordered commands" and the like..

---

From: [email-masked]
Date: Sat, 21 Apr 2001 16:53:20 +0200

What exactly does it buy you? (that is not already in).

And FWIW David (B) raised a question related to the requirements doc.

Julo

Sandeep Joshi <[email-masked]> on 20/04/2001 18:43:02

Please respond to Sandeep Joshi <[email-masked]>

To:   [email-masked]
cc:
Subject:  iSCSI:out-of-order notification proposal

thanks,
-Sandeep

Note :
======

Problem:
=========

Solution:
=========

Figuratively:
=============

Initiator                       Target
---------                       -------

ordered cmd (103, 1) ->
simple  cmd (104, 1) ->
simple  cmd (105, 1) ->

simple  cmd (108, 0) ->
simple  cmd (109, 0) ->

Other issues:
=============

---

From: [email-masked]
Date: Mon, 23 Apr 2001 11:37:49 +0200

Sandeep,

If you take a look at what is already - look also a 7.3 - you will find
that what you want to do can be achieved with what is already in - even if
the exact mechanism is not completely spelled out (the draft is not an
implementation).

Regards,
Julo

Sandeep Joshi <[email-masked]> on 21/04/2001 21:55:14

To:   [email-masked]
cc:
Subject:  Re: iSCSI:out-of-order notification proposal

Answering two responses in one email.  This would be mighty
easier if there was a whiteboard around here :-)

> What exactly does it buy you? (that is not already in).

The ability to execute subsequent PDUs in the presence of holes.
(Holes due to multiple connections OR due to skipping onto next marker)

If the hole is due to a REAL ordering issue (e.g. ORdered task
was lost) then the target will pause for the hole to fill.

But if the hole is due to simple task, then the target
can continue execution of the new commands it is receiving.

Currently, target will *always* pause due to cmdSN ordering,
which is somewhat like a primitive microprocessor which wont
reorder its instruction stream, though its feasible.

> And FWIW David (B) raised a question related to the requirements doc.

My mistake, the rephrasing of requirements seemed to match the intent
of previous discussions on why total cmdSN ordering is overkill.
That was the only point i was trying to reinforce.

"Mallikarjun C." wrote:
>
> Sandeep,
>
> Some comments on your proposal.
>
> - The "strict_order" flag that you mention appears to carry
>   information that's already contained by the SCSI task attributes
>   (ATTR field).

But now the information is also contained in PDUs following it.
(e.g. A simple command PDU may also have strict_order=1)

If an Ordered task PDU is lost, the target *knows* there is a REAL
ordering issue (later PDUs have info) so it must wait for the hole
to be filled.

>
> - This proposal requires the target iSCSI layer to look at more
>   variables (SCSI-level information in that) for making its
>   sequencing decisions.

Only one more bit.. to decide if the command can be passed
to SCSI layer now or later in cmdSN sequence.

To build an efficient SCSI transport, some hints would need
to be passed between layers.

>
> - Simple commands (intended to be) received prior to an Ordered
>   command must be executed before the Ordered one.  If you lose a
>   Simple command, you must wait for it before you act on the Ordered
>   that you received out-of-order.

Good pt.  This, I forgot to mention below.  The target must clear
out all PDUs received with strict-order=0 before commencing
execution of PDUs received with strict-order=1.

>
> This appears more an implementaion optimization where desired than
> something we want to get into the draft.  You are proposing that
> even in the absence of errors, iSCSI should make ordering decisions
> based on SCSI task attributes.  I disagree with that.

The optimization needs a bit in the PDU, otherwise I wouldnt
have gone to the trouble of writing this out!  And you would agree
that there are a bunch of TCP optimizations in various drafts.

Lastly, what is the point in building a sync-and-steering layer to
skip ahead in the command stream if you cannot execute that next command
?
With this little more work, you could as well continue execution.

> --
> Mallikarjun
>
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668 Hewlett-Packard, Roseville.
> [email-masked]
>