Subject: RE: iSCSI: Out of order commands

From: "Julian Satran" <[email-masked]>
Date: Wed, 7 Nov 2001 07:59:48 +0200

Rod,

I stated repeatedly that this causes deadlock and adds nothing in terms of 
performance.
It is explicitly stated for unsolicited data and it was just an overlook 
that it was never made explicit for
commands.

Julo

"Rod Harrison" <[email-masked]>
07-11-01 06:16
Please respond to "Rod Harrison"

        To:     "Santosh Rao" <[email-masked]>, <[email-masked]>
        cc:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
        Subject:        RE: iSCSI: Out of order commands

                 It seems to me that if a target offers a command window 
greater than
one it is buying into the complexity associated with supporting that
window.

                 There is very little difference at the target between 
buffering a
command that arrives out of order and a buffering a write command that
arrives without all the payload. In both cases other commands may
arrive which cannot be committed to the SCSI layer. If a target
doesn't want to be in the business of command queuing it has the
option of offering a command window of one.

                 Implementing a command queue is a much simpler 
proposition at the
target than at the initiator. The target is in complete control of the
command window and can therefore simply use a static array to hold
command descriptors. The initiator has no such luxury since it has no
a priori knowledge of the command window size, indeed the command
window size can change dynamically. For the initiator to support
ordered command queuing it must use an ordered list which can be
expensive, especially when we consider the CPU power that will
typically be available to an iSCSI HBA. Negotiating the command window
size as part of login would make this more palatable for an initiator,
but I suspect targets wouldn't want to commit to an unchanging command
window.

                 We've been debating the merits of an initiator sending 
out of order
commands which is perhaps beyond the scope of where we should be. The
cost to a target implementer is negligible and there is a potential
benefit to an initiator implementer, so why should we prohibit this
behaviour?

                 - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Santosh Rao
Sent: Tuesday, November 06, 2001 6:16 PM
To: [email-masked]
Cc: Rod Harrison; Julian Satran; [email-masked]
Subject: Re: iSCSI: Out of order commands

Mallikarjun,

Some comments below.

Regards,
Santosh

"Mallikarjun C." wrote:
>
> Rod and Julian,
>
> This has been an interesting thread of discussion.  Some
> comments -
>
> 1.My first reaction was - allowing out-of-order command
>   transmission on the same connection deprives targets of
>   an implementation choice.  Targets which support only
>   single-connection sessions and only support session
>   recovery (reasonable assumptions in my mind) can no
>   longer afford *not to* implement a command scoreboard.

Even a single connection target *MUST* implement a scoreboard. The
reason being that it can see out-of-order arrival of commands due to
commands being dropped on digest errors. In such a case, it must block
further command processing until holes are filled.

Thus, there is no getting away from implementing a sequencer at the
target. Given this, I think it is unreasonable to restrict initiator
implementation flexibility by imposing a strict ordering requirement
within the connection.

> 2.Any end-node efficiency that is sought to be achieved
>   by transmitting CmdSNs out-of-order from the initiator
>   would be lost on the other end-node, since the target
>   now must wait for re-ordering the commands.

It has to handle this situation anyway to deal with holes caused by
digest errors. This scenario occurs even with initiators that issue
commands in order.

>
> 3.The flipside is that out-of-order transmission saves
>   link badwidth (albeit at the expense of end-node efficiency),
>   compared to idling the link waiting for outbound DMA.
>   We have to determine if this is a reasonable trade-off.
>
> 4.I can see Rod's point that prefetching all immediate
>   data can be a burden on the NIC resources.  But, two
>   questions -
>         - could the NIC not use unsolicited separate data
>           PDUs in these cases? [ I realize that InitialR2T
>           has to be "no" to let it happen... ]
>         - could the NIC have a memory architecture that
>           allows data prefetching for the next command (so
>           this is a non-issue from the protocol perspective)?
>           This scheme incurs one DMA delay for every new
>           burst of commands.
>
> 5.Another (perhaps radical at this point) option is to do
>   away with immediate unsolicited data, to stick only with
>   separate unsolicited data.  I would personally be okay
>   with the choice, particularly if this feature (that
>   helps software implementations) starts making hardware
>   design complicated/expensive.
>
> So, to summarize -
>
> option                         immediate         allow
>                                data in spec?     out-of-order?
>
> (A) (5) above                  no                no
> (B) No real reason to do this. no                yes
> (C) (4) above                  yes               no
> (D) pros & cons (1), (2) & (3) yes               yes
>
> >From the arguments I heard so far, I am leaning towards
> option A, and option C in that order.
>
> Comments?
> --
> Mallikarjun
>
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668 Hewlett-Packard, Roseville.
> [email-masked]
>
> Rod Harrison wrote:
> >
> > Julian,
> >
> >         I don't understand what you are proposing here, what do
you mean by
> > "multiplexed" DMA?
> >
> >         The problem is that the DMAs take some time, the more
there are
> > queued the longer the last DMAs queued take to complete. Some
commands
> > require DMAs to complete before they can be sent, i.e. Writes with
> > immediate data, some commands do not, i.e. Reads and writes with
no
> > immediate data. The iSCSI HBA wants to be able to send commands as
> > soon a possible, which for a read after a write can be before the
> > write's DMA has completed. Maintaining an ordered queue for
commands
> > to be sent on the HBA is expensive and redundant since the target
> > already knows how to queue commands before committing them to its
SCSI
> > layer.
> >
> >         The iSCSI HBA and its host driver are not at liberty to
change the
> > order of commands from the OS, but the DMAs those commands need
are
> > unlikely to complete in the same order, and as I mentioned some
> > commands need no DMA. If the HBA can't send commands out of CmdSN
> > order it has to maintain an ordered queue of commands waiting to
be
> > sent, and potentially buffer a lot of data. For an HBA this makes
> > immediate data almost impossible to support.
> >
> >         I don't see the problem with allowing out of order
commands given
> > that the target already has to deal with very similar problems. I
> > think we are getting in to the area of implementation choices
here,
> > which is inappropriate for a specification.
> >
> >         - Rod
> >
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On
Behalf Of
> > Julian Satran
> > Sent: Monday, November 05, 2001 10:06 PM
> > To: [email-masked]
> > Subject: Re: iSCSI: Out of order commands, was current UNH
Plugfest
> >
> > Rod,
> >
> > I don't see any reason why DMA operations cant be "multiplexed"
with
> > commands.
> > If you have scheduled a long outbound DMA you are doomed
regardless of
> > the
> > command ordering.
> > And if you have scheduled DMA operations piecemeal then you can
insert
> > your commands in correct order.
> >
> > Julo
> >
> > "Rod Harrison" <[email-masked]>
> > 05-11-01 20:48
> > Please respond to "Rod Harrison"
> >
> >         To:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
> >         cc:
> >         Subject:        iSCSI: Out of order commands, was current
UNH
> > Plugfest
> >
> >                  [ Subject changed ]
> >
> > Julian,
> >
> >                  The ordering difference is introduced between the
> > host
> > side driver
> > and the iSCSI HBA. The host side driver must present SCSI commands
to
> > the HBA in the order they are received from the OS to prevent read
> > after write dependency failures. The HBA might reorder the
commands
> > depending on when DMA completes. The reordering can't be done
ahead of
> > time in the host driver since it doesn't know how long each DMA
might
> > take. As long as the HBA assigns CmdSN in the order it receives
> > commands the desired host ordering is preserved.
> >
> >                  - Rod
> >
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On
Behalf Of
> > Julian Satran
> > Sent: Monday, November 05, 2001 12:35 AM
> > To: [email-masked]
> > Subject: RE: iSCSI: current UNH Plugfest
> >
> > Rod,
> >
> > I all examples give the point I find hard to understand is why is
the
> > ordering on the wire different from the presentation order to the
> > initiator.  You can get as many overlaps as you want by presenting
the
> > commands to the initiator in the desired order.
> > What we are considering here is the case in which you want to ship
in
> > an
> > order different than the one you present the commands.
> >
> > Julo
> >
> > "Rod Harrison" <[email-masked]>
> > Sent by: [email-masked]
> > 04-11-01 04:42
> > Please respond to "Rod Harrison"
> >
> >         To:     "Barry Reinhold" <[email-masked]>, "Dave
> > Sheehy"
> > <[email-masked]>, "IETF IP SAN Reflector"
> > <[email-masked]>
> >         cc:
> >         Subject:        RE: iSCSI: current UNH Plugfest
> >
> > Barry,
> >
> >                  In general I agree but I don't think this is as
much
> > of a
> > corner case
> > as it at first appears. Targets will have code very similar to
that
> > needed to handle out of order commands to deal with digest errors.
> > Targets also need to queue commands whilst waiting for both
solicited
> > and unsolicited data to arrive. Queuing out of order commands
seems
> > little extra work.
> >
> >                  From an initiators point of view there are
> > efficiency,
> > and probably
> > performance gains to be had from sending commands out of order.
Bob
> > Russell gave the example of a read being sent whilst write data
DMA is
> > happening, and a similar situation can arise with DMA for writes
> > overtaking that of earlier writes if the initiator has multiple
DMA
> > engines. In this case the initiator might be forced to let the
wire go
> > idle if it can't send the data from completed DMAs as soon as
> > possible.
> >
> >                  We already have a command queue at the target to
> > enforce
> > correct
> > serialisation of commands, doing the same thing at the initiator
is
> > redundant.
> >
> >                  Finally, I don't believe we should be writing a
> > standard
> > to work
> > around poor coding and test coverage, especially at the cost of
> > potential efficiency gains.
> >
> >                  I agree with Dave and Santosh that commands being
> > sent
> > out of order
> > on a single session should be allowed by the standard.
> >
> >                  - Rod
> >
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On
Behalf Of
> > Barry Reinhold
> > Sent: Friday, November 02, 2001 5:24 PM
> > To: Dave Sheehy; IETF IP SAN Reflector
> > Subject: RE: iSCSI: current UNH Plugfest
> >
> > Using features such as out of order command delivery on a
connection
> > tend to
> > be the sort of things that lead to interoperability problems. It
is
> > unexpected and probably going to hit poorly tested code paths even
if
> > the
> > standard is written to allow it.
> >
> > >-----Original Message-----
> > >From: [email-masked] [mailto:[email-masked]]On
Behalf
> > Of
> > >Dave Sheehy
> > >Sent: Friday, November 02, 2001 4:19 PM
> > >To: IETF IP SAN Reflector
> > >Subject: Re: iSCSI: current UNH Plugfest
> > >
> > >
> > >
> > >> 3. Can commands be sent out of order on the same connection?
> > >>
> > >>    The behavior of targets is clearly specified in Section
2.2.2.3
> > on
> > >>    page 25 of draft 8, which says:
> > >>      "Except for the commands marked for immediate delivery the
> > iSCSI
> > >>      target layer MUST eliver the commands for execution in the
> > order
> > >>      specified by CmdSN."
> > >>
> > >>    Section 2.2.2.3 on page 26 of draft 8 also says:
> > >>      "- CmdSN - the current command Sequence Number advanced by
1
> > on
> > >>      each command shipped except for commands marked for
immediate
> > >>      delivery."
> > >>    but the meaning of the term "shipped" is vague, and does not
> > >> necessarily
> > >>    require that the PDUs arrive on the other end of a TCP
> > connection
> > >>    in the same order that the CmdSN values were assigned to
these
> > PDUs.
> > >>
> > >>    Some initiators have been designed to send commands out of
CmdSN
> > >>    order on one connection.  Consider the situation where there
is
> > only
> > >>    one connection and a high-level dispatcher creates a PDU for
a
> > SCSI
> > >>    command that involves writing immediate data to the target.
> > This PDU
> > >>    is enqueued to a lower-level layer which has to setup,
start,
> > and
> > >>    wait-for a DMA operation to move the immediate data into an
> > onboard
> > >>    buffer before the PDU can be put onto the wire.  While this
is
> > >>    happening, the dispatcher creates another unrelated PDU for
a
> > SCSI
> > >>    read command (for example), and when this PDU is passed to
the
> > >>    lower-level layer it can be sent immediately, ahead of the
> > previous
> > >>    write PDU and therefore out of order on this connection.
> > >>
> > >>    The standard clearly allows this to happen if the two PDUs
were
> > sent
> > >>    on different connections, and seems to imply that this can
also
> > happen
> > >>    when the two PDUs are sent on the same connection.
> > >>
> > >>    The suggestion is to put in the standard an explicit
statement
> > that
> > >>    this is allowed or not allowed, as appropriate.
> > >>
> > >>    If this is allowed, such a statement would avoid the
erroneous
> > >>    assumption being made by some target implementers that
within a
> > single
> > >>    connection, commands will arrive in order.
> > >>
> > >>    If this is not allowed, such a statement would avoid the
> > erroneous
> > >>    assumption being made by some initiator implementers that
within
> > a
> > >>    single connection, commands can be put on the wire out of
order.
> > >>
> > >> +++
> > >>
> > >> will add an explicit statement saying that this behaviour is
> > forbidden.
> > >> 2.2.2.1 will contain:
> > >>
> > >> On any given connection, the iSCSI initiator MUST send the
> > >commands in the
> > >> order specified by CmdSN.
> > >>
> > >> +++
> > >
> > >Why do you feel this behavior should be forbidden? Targets
already
> > have to
> > >order commands across the session. I don't see why it's a problem
to
> > extend
> > >that to the connection as well. I, for one, believe we should
take
> > >a liberal
> > >stance on this.
> > >Dave Sheehy

--
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: Dave Sheehy <[email-masked]>
Date: Thu, 08 Nov 2001 17:19:17 PST

Julo,

> Targets will most likely advertise a total (command and data) window 
> larger than they can accommodate on any long haul link.  With the current 
> ordering rules nothing bad will happen.

Any target that does that deserves what it gets IMHO.

> The whole discussion thread seems also related to some perceived gain from 
> relaxing this restriction - but no one was able to show a single scenario 
> showing a real gain.

Rod Harrison (I believe) described the scenario quite well. I recomend that
you search the archive for his message.

Dave Sheehy

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 8 Nov 2001 18:07:02 -0800

I understand what you are saying but that is a typical buffer manager
problem which most systems work with today.  I have usually resolved this
in the past with a statistical probability analyses, often that is adjusted
in real time.  Again, I see nothing unusual here.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Somesh Gupta" <[email-masked]> on 11/08/2001 05:58:03
PM

Please respond to <[email-masked]>

To:   John Hufferd/San Jose/IBM@IBMUS
cc:   "Robert D. Russell" <[email-masked]>, Julian
      Satran/Haifa/IBM@IBMIL, <[email-masked]>
Subject:  RE: iSCSI: Out of order commands

John,

> From: John Hufferd [mailto:[email-masked]]
> Sent: Thursday, November 08, 2001 5:19 PM
> To: [email-masked]
> Cc: Robert D. Russell; Julian Satran; [email-masked]
> I understand your point, however, their are several approaches that I and
> others have used to mitigate this issue.  We face that problem with Multi
> Processing all the time.
>
> One technique that I have seen use that is similar to what you describe
is
> to allocate a set of buffers to each processor (in this case an HBA), and
> when they are used up, and only then, the go back to the Buffer Mother,
to
> allocate another Buffer Set.  In this way the Processors (HBA) run mostly
> independent, but once in awhile get a new supply.  There are other
> techniques also, but that one clearly works wells and is very simple.

  I agree with the mechanism above. So here is where the problem is.
  Let us take a simplification of M adapters and 1 connection per
  adapter. Let us also assume that the N buffers are given to each
  adapter.

  Total number of buffers used - M * N
  Buffers available to each adapter - N

  How large of a command window should be advertised in the case of
  a session wide window.

  If the window is N, then we are under-utilizing the buffers but we
  will never drop commands.

  If we advertise M * N, and the initiator decides to send commands
  in an unbalanced manner (because 1 connection is working better),
  then we have a potential problem where the one adapter uses up
  its allocation - In the meantime the system will provide it some
  more (but not enough),
  and there is an imbalance leading to a situation where commands
  may have to be dropped. (this could also happen if the system
  did not allocate buffers evenly e.g. in a scenario where the
  adapters are not using buffers at the same rate).

  Now we can try to find a balance in the middle, but again there
  is a loss of efficiency on one axis or the other.

>
> Therefore, I do not see the Session Window as a problem in the protocol.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: [email-masked]
>
>
> "Somesh Gupta" <[email-masked]> on 11/08/2001
04:57:40
> PM
>
> Please respond to <[email-masked]>
>
> To:   John Hufferd/San Jose/IBM@IBMUS
> cc:   "Robert D. Russell" <[email-masked]>, Julian
>       Satran/Haifa/IBM@IBMIL, <[email-masked]>
> I think we may be talking about different things here.
> Yes, there are memories that are "on the iSCSI HBA"
> (in the case where everything is on a single board,
> this would be the memories that are controlled by
> only the iSCSI ASIC or whatever) - and these memories
> are not shared among multiple HBAs. These may be used
> for any book-keeping purposes, contexts, eddy buffering
> etc etc by the adapter.
>
> Then there is memory pool that is owned by the "system".
> Adapters will DMA commands/data/responses directly into
> and out of this memory pool which is managed by the "system".
> However to prevent adapters from running over each other,
> and over the "system" processor, producer/consumer
> mechanisms have to be implemented.
>
> This is necessary to maintain order on the system. So
> on a long-term basis, the memory in the system is shared
> across the HBAs - absolutely yes. But at any given
> instance in time, it is partitioned among various
> usages.
>
> So all in all I agree with you. The only point of difference
> is that it is not possible that a given buffer (part of
> a buffer pool) is accessible to multiple adapters at
> the same instance in time. It is a necessary condition to
> prevent corruption and chaos.
>
> > From: John Hufferd [mailto:[email-masked]]
> > Sent: Thursday, November 08, 2001 4:40 PM
> > To: [email-masked]
> > Cc: Robert D. Russell; Julian Satran; [email-masked]
> > I guess I disagree with this design approach.  The buffers on the iSCSI
> > HBAs are for Eddy buffers.  Once you have the iSCSI Header
> should be able
> > to place the data directly into a Shared buffer.  If you have a session
> > that spans HBAs, that should work just fine.  But the buffers match the
> > Session Window, and the Buffers need to be shared across the
> HBAs, not in
> > the HBAs.  Again, the HBAs should just have the Eddy Buffers.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136, Cell: (408) 499-9702
> > Internet address: [email-masked]
> >
> >
> > "Somesh Gupta" <[email-masked]> on 11/08/2001
> 04:31:29
> > PM
> >
> > Please respond to <[email-masked]>
> >
> > To:   John Hufferd/San Jose/IBM@IBMUS
> > cc:   "Robert D. Russell" <[email-masked]>, Julian
> >       Satran/Haifa/IBM@IBMIL, <[email-masked]>
> > John,
> > You are correct if all the connections are on
> > a single adapter. In this case the buffer pool
> > can be shared among all the connections on the
> > adapter.
> >
> > However this is not true if the connections are
> > on different adapters (the point being discussed).
> > Think of there being a big buffer pool in the board
> > into which multiple adapters/chips are "plugged" in.
> > However, at any given point in time, these buffers
> > are allocated (handed over through command queues or
> > whatever) to each of the individual adapters. And this
> > is where the problem arises.
> >
> > The assumption that adapters can share a typical host
> > command buffer pool on their own without host intervention
> > is incorrect.
> >
> > Somesh
> >
> > > From: John Hufferd [mailto:[email-masked]]
> > > Sent: Thursday, November 08, 2001 4:20 PM
> > > To: [email-masked]
> > > Cc: Robert D. Russell; Julian Satran; [email-masked]
> > > Most folks I know have a buffer that applies to all the
> > > connections, not to
> > > just one buffer per connection.  I fail to see the problem for most
> > > implimentations.  The window should apply to the session.
> > >
> > > .
> > > .
> > > .
> > > John L. Hufferd
> > > Senior Technical Staff Member (STSM)
> > > IBM/SSG San Jose Ca
> > > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > > Home Office (408) 997-6136, Cell: (408) 499-9702
> > > Internet address: [email-masked]
> > >
> > >
> > > "Somesh Gupta" <[email-masked]>@ece.cmu.edu on
> > > 11/08/2001 02:22:30 PM
> > >
> > > Please respond to <[email-masked]>
> > >
> > > To:   "Robert D. Russell" <[email-masked]>
> > > Comments below
> > > > From: Robert D. Russell [mailto:[email-masked]]
> > > > Sent: Thursday, November 08, 2001 11:46 AM
> > > > To: Somesh Gupta
> > > > Cc: Julian Satran; [email-masked]
> > > > Somesh:
> > > > > > It seems to me that if a target can only handle x new commands,
> > > > > > then its queueing capacity is x, and in the SCSI Response PDU
it
> > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with
the
> > > > > > formula in section 2.2.2.1.  This in turn controls the number
of
> > > > > > commands the initiator can send, and thus prevents the incoming
> > > > > > commands from overfilling the target's queue.
> > > > >
> > > > >   This is actually a weakness of the multiple connections over
> > > > >   multiple adapters model (it is not related to ordering).
> > > > >   The target advertises a command window on a session wide basis.
> > > > >   At the same time, it has to provide the resource to the adapter
> > > > >   to be able to DMA those commands to. Since there is no
> restriction
> > > > >   on which connection the commands may be received, either the
> > > > >   target has to allocate the max resources needed to each adapter
> > > > >   (thus committing n times the resources actually required), or
> > > > >   has to "lie" (it would not be a complete lie) which could
> > > > >   result in running out of resources. One way to fix that would
be
> > > > >   to have the credit on a per connection basis.
> > > >
> > > > If I understand you correctly, you are saying that deadlock can
> > > > occur even if we enforce ordered delivery by initiators and even
> > > > if the advertisements are correct and even if both parties obey
> > > > the advertisements.
> > >
> > >   I would let Julian enlighten us on that, or whether it just
> > >   leads to wholesale command drops and retries, or TASK SET FULL
> > >   is returned (is TASK SET FULL a valid response in this case
> > >   for a LINKED task when it is not the first command?)
> > >
> > > >
> > > > In other words, doesn't this mean that the standard can lead to a
> > > > trap in implementing targets and that, in fact, the advertisements
> > > > really should be on a per connection basis?
> > > > However, what would be the consequences for initiators if
> > > > the advertisements were on a per-connection basis?
> > >
> > >   There is really no issue with adveritisement on a per
> > >   connection basis. It leads to a protocol change. However, it
> > >   also allows a much more performant flow of commands.
> > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Bob Russell
> > > > InterOperability Lab
> > > > University of New Hampshire
> > > > [email-masked]
> > > > 603-862-3774
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 9 Nov 2001 09:39:21 +0200

The data generated in response to an R2T for C1 do not have to be ordered.
The data can be multiplexed with commands. Ordering applies only for
commands, unsolicited data and the order in which R2T are satisfied.
The ordering rules exist also for targets. A target is not assumed to send
a data or status PDU out of order.

Julo

                    "FRYMAN,ROBERT                                                            
                    (A-Roseville,ex1)       To:     "'Barry Reinhold'"                        
                    "                        <[email-masked]>, Rod Harrison           
                    <robert_fryman@ag        <[email-masked]>, "Robert D.         
                    ilent.com>               Russell" <[email-masked]>, Somesh Gupta    
                                             <[email-masked]>             
                    08-11-01 19:03          cc:     Julian Satran/Haifa/IBM@IBMIL,            
                    Please respond to        [email-masked]                                  
                    "FRYMAN,ROBERT          Subject:     RE: iSCSI: Out of order commands     
                    (A-Roseville,ex1)                                                         
                    "                                                                         

> Thus the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry
> about the fact that C1 generates data in response to the R2T.
> However the command sequence of C1, C3, C2, C4 forces the iSCSI
> layer to buffer C3. This is significant to a hardware accelerated
> iSCSI layer.

This is significant to a target hardware accelerated iSCSI layer.  By
forcing the initiator to order the iSCSI commands, you move the
significants
from the target to the initiator, but it is still a complexity added to a
hardware accelerator.

The problem I have is that we are trying to add a requirement that only
effects a specific area, but adds complexity to a larger area.  If we say
that initiators must order commands on a single connection, then doesn't
this mean that we have just added complexity and slowdown to the enterprise
storage area?

In the Fibre channel realm, a standard base configuration was to have a
target (JBOD, array, etc.) with 2 ports, and an initiator with two ports.
These would be connected on two seperate physical connections, so if a
problem happens on one, the other will take over.  We allow the same thing
in iSCSI, plus specified how the customer can use both connections to
increase bandwidth as long as both connections work.  It seems to me that
any customer that wants a fault tolerant solution, and gets increased
bandwidth for free due to the ability to have multiple connections, would
take advantage of that.

This being the case, these targets have to have reordering capability.  If
we force the initiators to send the comands in order, we just added
complexity and slowdown that adds no benifit.

As I see it (and I could be totally off base here), we have a choice of
complexity and slowdown in the enterprise market, or adding a little
complexity to simple targets.  Am I flawed in this thinking?

Scott Fryman
Agilent Technologies

> From: Barry Reinhold [mailto:[email-masked]]
> Sent: Thursday, November 08, 2001 6:19 AM
> To: Rod Harrison; Robert D. Russell; Somesh Gupta
>          I can see your point, from the perspective of a target
> that is a box
> combining an iSCSI entity with a SCSI entity. However, from
> the perspective
> of an "iSCSI only" entity, delivery does not have to be
> delayed until C1
> completes. In this case you only need to ensure that the commands are
> delivered in order to the SCSI layer. The SCSI layer has to
> address the
> semantics of what the operation entails in terms of command
> execution. Thus
> the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry
> about the fact
> that C1 generates data in response to the R2T. However the
> command sequence
> of C1, C3, C2, C4 forces the iSCSI layer to buffer C3. This
> is significant
> to a hardware accelerated iSCSI layer.
>
> >From: Rod Harrison [mailto:[email-masked]]
> >Sent: Wednesday, November 07, 2001 8:34 PM
> >To: Barry Reinhold; Robert D. Russell; Somesh Gupta
> >        I'm curious, where do you see the extra complexity between the
> >following?
> >
> >        Assuming all the following commands are writes ...
> >
> >c1, c3, c2, c4 and
> >
> >c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate.
> >
> >        In both cases the target has to queue commands 2, 3,
> and 4 whilst
> >waiting for its R2T on c1 to be satisfied.
> >
> >        Even if the target doesn't support any unsolicited data
> it still has
> >to queue commands 2, 3, and 4. I can't see how the target can avoid
> >queuing commands, in which case the order of arrival seems to make
> >little difference.
> >
> >        - Rod
> >
> >Sent: Wednesday, November 07, 2001 3:10 PM
> >To: Robert D. Russell; Somesh Gupta
> >Bob,
> >        I can't speak for targets, but OOO commands on a session with a
> >single
> >connection sure increases the complexity of the code path we take.
> >        To me the issue is still that these types of situations
> tend to be
> >poorly
> >tested and lead to interoperability issues. If we do go down this
> >path, the
> >spec. should make it very clear that this is expected behavior. Some
> >statement with a shall in it should be written (for the receiver), so
> >that
> >there is a conformance test items to pass.
> >
> >>Robert D. Russell
> >>Sent: Wednesday, November 07, 2001 4:13 PM
> >>To: Somesh Gupta
> >>Somesh, Julian:
> >>You state that dealing with OOO commands on the target
> >>will add substantial complexity on the target.
> >>Do you have any basis for that claim?  My impression from the
> >>plugfest is that most targets are already dealing with
> >>it.  Perhaps we need to hear from someone who is actually
> >>building a target for which this would be a real problem.
> >>
> >>If anything, what we are hearing from people who really
> >>are building initiators is that dealing with the requirement
> >>to send commands in order will introduce substantial complexity
> >>on the initiator.
> >>
> >>So why should we be saving complexity on (hypothetically) simple
> >>targets yet requiring complexity on real initiators?
> >>
> >>As far as the deadlock issue is concerned, if the only way
> >>that deadlock can occur with OOO commands on the same
> >>connection is during the use of immediate data (which is I
> >>think what Julian was saying), then shouldn't we change
> >>the standard to just say that -- if the initiator sends
> >>commands out of order on a single connection, then immediate
> >>data MUST NOT be used on that connection in order to avoid deadlock.
> >>
> >>This gives everybody what they want, since initiators who find
> >>it beneficial to deliver commands OOO will just negotiate
> >>immediate data off.  Those who really want to use immediate data
> >>will have to ensure that commands are sent in order.
> >>The tradeoff then becomes an implementation issue, not a
> >>standards issue, which is where it belongs.
> >>
> >>
> >>Bob Russell
> >>InterOperability Lab
> >>On Wed, 7 Nov 2001, Somesh Gupta wrote:
> >>
> >>> I think we should either have it as a MUST or not require
> >>> it (at both ends to get the real benefit). SHOULD is one
> >>> of those things that leads to implementation
> >>> burden and confusion, without perhaps the feature being
> >>> used. There are implementation as well as protocol
> >>> considerations mixed in here.
> >>>
> >>> If we are to remove the restriction, we should (SHOULD)
> >>> get the maximum benefit from it, rather than to
> >>> accomodate an implementation choice. Out of sequence
> >>> commands, combined with the possibility of digest errors,
> >>> will add substantial complexity on the target side,
> >>> without corrosponding benefit in performance. If we change
> >>> this to SHOULD, we should also relax the requirement
> >>> to present commands on the target side to a SHOULD.
> >>>
> >>>
> >>>
> >>> > Sent: Wednesday, November 07, 2001 10:00 AM
> >>> > To: [email-masked]
> >>> > I did not see a SINGLE performance improvement that results from
> >OOO
> >>> > shipping.
> >>> > I would be bad engineering to give away the "no-deadlock"
> >mechanism we
> >>> > have now for nothing.
> >>> > I have also the impression that the point about deadlock that I
> >keep
> >>> > repeating is ignored or not understood.
> >>> > As we stand today commands can be shipped with Immediate data
> >>or without
> >>> > and an implementer determined
> >>> > to squeeze maximum bandwidth and overlap command start with
> >>delivery will
> >>> > choose not to work with immediate data
> >>> > (as you have pointed out) while a low performance software
> >>implementation
> >>> > will use immediate data to minimize CPU cycles
> consumed.  However
> >both
> >>> > will be guaranteed to work without deadlock as source and sink
> >use the
> >>> > same ordering.
> >>> > Recovery is still a low probability event and should be handled
> >with a
> >>> > different set of considerations in mind.
> >>> > As for the strictness of the recommendation - yes we
> could settle
> >on
> >>> > SHOULD.
> >>> >
> >>> > Julo
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > "Mallikarjun C." <[email-masked]>
> >>> > 07-11-01 19:41
> >>> > Please respond to cbm
> >>> >         To:     Santosh Rao <[email-masked]>,
> >[email-masked]
> >>> >         cc:
> >>> > Santosh,
> >>> > I have only one comment on your responses.
> >>> >
> >>> > I made two convenient assumptions if you noticed, :-), one of
> >which
> >>> > is that target forces session recovery on *any* error that it
> >sees
> >>> > (ErrorRecoveryLevel=0) - including a dropped command due to a
> >digest
> >>> > error.  With that assumption, a target can afford not to
> >implement
> >>> > a scoreboard.
> >>> >
> >>> > As I said in a private note, I guess what primarily bothers me
> >about
> >>> > OOO commands on a connection is that it requires the receiver to
> >>> > undo this "optimization" on its end - most notably on a single
> >>> > connection.  TCP experts may comment on how/if they dealt with a
> >>> > similar issue.
> >>> >
> >>> > OTOH, you had some valid comments on exceptions to ordering
> >during
> >>> > connection recovery.  Perhaps we can move on by making Julian's
> >>> > proposed stipulation a SHOULD....
> >>> > --
> >>> > Mallikarjun
> >>> >
> >>> >
> >>> > Mallikarjun Chadalapaka
> >>> > Santosh Rao wrote:
> >>> > > Thus, there is no getting away from implementing a
> sequencer at
> >the
> >>> > > > >         I don't understand what you are proposing here,
> >>what do you
> >>> > mean by
> >>> > > > > "multiplexed" DMA?
> >>> > > > >
> >>> > > > >         The problem is that the DMAs take some time, the
> >>more there
> >>> > are
> >>> > > > > immediate data, some commands do not, i.e. Reads and
> >>writes with no
> >>> > > > > write's DMA has completed. Maintaining an ordered queue
> >>for commands
> >>> > > > > order of commands from the OS, but the DMAs those
> >>commands need are
> >>> > > > > commands need no DMA. If the HBA can't send
> commands out of
> >CmdSN
> >>> > > > > order it has to maintain an ordered queue of commands
> >>waiting to be
> >>> > > > > sent, and potentially buffer a lot of data. For
> an HBA this
> >makes
> >>> > > > >         I don't see the problem with allowing out of
> >>order commands
> >>> > given
> >>> > > > > think we are getting in to the area of implementation
> >>choices here,
> >>> > > > > From: [email-masked]
> >>[mailto:[email-masked]]On Behalf
> >>> > Of
> >>> > > > > Julian Satran
> >>> > > > > Subject: Re: iSCSI: Out of order commands, was current
> >>UNH Plugfest
> >>> > > > >
> >>> > > > > Rod,
> >>> > > > >
> >>> > > > > I don't see any reason why DMA operations cant be
> >>"multiplexed" with
> >>> > > > > commands.
> >>regardless
> >>> > > > > And if you have scheduled DMA operations
> piecemeal then you
> >can
> >>> > insert
> >>> > > > > after write dependency failures. The HBA might reorder
> >>the commands
> >>> > > > > depending on when DMA completes. The reordering can't be
> >>done ahead
> >>> > of
> >>> > > > > time in the host driver since it doesn't know how
> long each
> >DMA
> >>> > might
> >>> > > > > From: [email-masked]
> >>[mailto:[email-masked]]On Behalf
> >>> > Of
> >>> > > > > Julian Satran
> >>> > > > > as it at first appears. Targets will have code very
> >>similar to that
> >>> > > > > and unsolicited data to arrive. Queuing out of order
> >>commands seems
> >>> > > > > little extra work.
> >>> > > > >
> >>> > > > > performance gains to be had from sending commands out of
> >>order. Bob
> >>> > > > > Russell gave the example of a read being sent whilst
> >>write data DMA
> >>> > is
> >>> > > > > overtaking that of earlier writes if the initiator has
> >>multiple DMA
> >>> > > > > engines. In this case the initiator might be forced to
> >>let the wire
> >>> > go
> >>> > > > > serialisation of commands, doing the same thing at the
> >>initiator is
> >>> > > > > redundant.
> >>> > > > >
> >>> > > > > From: [email-masked]
> >>[mailto:[email-masked]]On Behalf
> >>> > Of
> >>> > > > > Barry Reinhold
> >>> > > > > Using features such as out of order command delivery on
> >>a connection
> >>> > > > > tend to
> >>> > > > > be the sort of things that lead to interoperability
> >>problems. It is
> >>> > > > > >From: [email-masked]
> >>[mailto:[email-masked]]On Behalf
> >>> > > > > Of
> >>> > > > > >Dave Sheehy
> >>> > > > > >>      "- CmdSN - the current command Sequence Number
> >>advanced by 1
> >>> > > > > on
> >>> > > > > >>    in the same order that the CmdSN values were
> >>assigned to these
> >>> > > > > PDUs.
> >>> > > > > >>
> >>> > > > > >>    one connection and a high-level dispatcher creates
> >>a PDU for a
> >>> > > > > SCSI
> >>> > > > > >>    is enqueued to a lower-level layer which has to
> >>setup, start,
> >>> > > > > and
> >>> > > > > >>    buffer before the PDU can be put onto the wire.
> >>While this is
> >>> > > > > >>    happening, the dispatcher creates another
> >>unrelated PDU for a
> >>> > > > > SCSI
> >>> > > > > >>    read command (for example), and when this PDU is
> >>passed to the
> >>> > > > > >>    The standard clearly allows this to happen
> if the two
> >PDUs
> >>> > were
> >>> > > > > sent
> >>> > > > > >>    The suggestion is to put in the standard an
> >>explicit statement
> >>> > > > > that
> >>> > > > > >>    If this is allowed, such a statement would avoid
> >>the erroneous
> >>> > > > > >>    assumption being made by some target implementers
> >>that within
> >>> > a
> >>> > > > > single
> >>> > > > > >Why do you feel this behavior should be forbidden?
> >>Targets already
> >>> > > > > have to
> >>> > > > > >that to the connection as well. I, for one, believe we
> >>should take
> >>> > > > > >a liberal
> >>> > > > > >stance on this.
> >>> > > > > >
> >>> > > > > >Dave Sheehy
> >>> > > > > >
> >>> > >
> >>> > > --

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 9 Nov 2001 11:51:47 +0200

A careful reading of the SCSI documents will clarify that SCSI does not 
require in order delivery of commands but
the standards assume that commands will be delivered in order.  iSCSI 
takes in fact a similar stance.
The standard described mechanisms assume that the hand-over is in order 
and all other mechanisms
will work under this assumption.  However iSCSI has no mechanism to 
enforce this (hand-over in or out-of-order)
is a local issue at the target and an implementer may choose to do 
whatever it deems appropriate.

For a regular stream of SCSI commands things will probably work fine.
Things will break when commands like as clear task set will end up 
clearing even commands supposed to reach the target after the clear - but 
with enough sophistication even this case can be handled.

Julo

08-11-01 03:57

        To:     <[email-masked]>, "Barry Reinhold" 
<[email-masked]>, "Robert D. Russell" <[email-masked]>
        cc:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
        Subject:        RE: iSCSI: Out of order commands

Somesh,

                 The spec prohibits processing commands simultaneously, if 
by
processing you mean handing off to the SCSI layer. If by processing
simultaneously you mean preparing data for committing to SCSI when the
CmdSN allows then surely reception of commands out of order is a no
brainer.

                 By the way, I am in support of allowing the target to 
make the
decision on when to commit the SCSI layer rather than having to use
the CmdSN. That decision can be made on the basis of device type, LUN,
and queue tag. Forcing CmdSN order of committal to SCSI has always
seemed overly restrictive to me.

                 - Rod

-----Original Message-----
From: Somesh Gupta [mailto:[email-masked]]
Sent: Wednesday, November 07, 2001 5:46 PM
To: Rod Harrison; Barry Reinhold; Robert D. Russell
Cc: Julian Satran; [email-masked]
Subject: RE: iSCSI: Out of order commands

Rod,

Sorry to butt in, but targets are not doing serialized
processing. They will process many many commands
simultaneously.

We had prior discussions on this and unfortunately
we agreed on ordered delivery at the iSCSI level. The
real boost in performance is to have ordering semantics
at a higher layer, so that the user code can determine
whether it needs ordering or not.

As an example, in the general/normal case of a computer
talking to a disk, the ordering rule could be relaxed
completely.

Somesh

> Sent: Wednesday, November 07, 2001 5:34 PM
>                In both cases the target has to queue commands 2, 3, and 
4 whilst
>                Even if the target doesn't support any unsolicited data 
it still
has
tend to be
> >> > will use immediate data to minimize CPU cycles consumed.
However
> >> > As for the strictness of the recommendation - yes we could
settle
> >> > > Even a single connection target *MUST* implement a
scoreboard.
> The
> >> > > Even a single connection target *MUST* implement a
scoreboard.
> The
> >> > > Thus, there is no getting away from implementing a sequencer
at
> >> > > > > to be sent on the HBA is expensive and redundant since
the
> target
> >> > > > > already knows how to queue commands before committing
them
> to its
> >> > > > > unlikely to complete in the same order, and as I
mentioned
> some
> >> > > > > commands need no DMA. If the HBA can't send commands out
of
> >> > > > > sent, and potentially buffer a lot of data. For an HBA
this
> >> > > > > And if you have scheduled DMA operations piecemeal then
you
> >> > > > > time in the host driver since it doesn't know how long
each
> >> > > > > I all examples give the point I find hard to understand
is
> why is
> >> > the
> >> > > > > ordering on the wire different from the presentation
order
> to the
> >> > > > >         To:     "Barry Reinhold"
<[email-masked]>,
> "Dave
> >> > > > >                  In general I agree but I don't think
this
> is as
> >> > > > > idle if it can't send the data from completed DMAs as
soon
> as
> >> > > > > >>    The standard clearly allows this to happen if the
two
> >> > > > > >>    If this is not allowed, such a statement would
avoid
> the
> >> > > > > erroneous
> >> > > > > >>    assumption being made by some initiator
implementers

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 9 Nov 2001 16:08:53 +0200

Nothing wrong with your reasoning. Only that a with ordering in place a 
designer might commit for it's cumulative set of resources (TCP windows + 
application buffers) while for unordered delivery it can commit only the 
application resources.

And I did not hear back from you about any specific reasons to do so 
(neither initiator nor targets get gaster).

Julo

"Robert D. Russell" <[email-masked]>
08-11-01 17:50
Please respond to "Robert D. Russell"

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     [email-masked]
        Subject:        RE: iSCSI: Out of order commands

Julian:

> However allowing initiators to ship them out of order creates a 
> potential deadlock that does not appear otherwise.
> 
> If a target is missing a command in a queue (and there are no errors) 
then
> this command is bound to be first on some connection under the current 
set 
> of rules.
> 
> If we allow OOO shipping then the missing command can be somewhere 
> "inside" the window on some connection and if the target has just filled 

> his queue and has room in the staging buffer only for the command it is 
> waiting for and that command happens to be the first to pass to SCSI you 

> have a deadlock.

I must be understanding something.

Your example is certainly correct if the target has no control
over the number of commands sent to it by the initiator.
But the target does control this number through the MaxCmdSN field.
For the scenario you are describing to occur, wouldn't it be
necessary for the target to advertize a MaxCmdSN value bigger than
it actually has resources to handle?

It seems to me that if a target can only handle x new commands,
then its queueing capacity is x, and in the SCSI Response PDU it
should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the
formula in section 2.2.2.1.  This in turn controls the number of
commands the initiator can send, and thus prevents the incoming
commands from overfilling the target's queue.

So isn't the deadlock caused by a broken target?
Isn't the target advertizing a queueing capacity greater than it
actually has and isn't that the cause of the deadlock?

An alternative explanation of the deadlock might be that the
target is advertizing the correct MaxCmdSN, but the initiator is
sending commands beyond what it is allowed to send.

However, in this case the target should silently ignore any
non-immediate command outside the allowed range.
For the deadlock to occur, wouldn't the target have to be queuing
commands outside the allowed range instead of ignoring them?

So in this case too, the target is broken and that is what causes
the deadlock.

Where am I going wrong in this reasoning?

Thanks,
Bob Russell
InterOperability Lab