Subject: Re: iSCSI: Out of order commands

From: "Julian Satran" <[email-masked]>
Date: Wed, 7 Nov 2001 19:59:36 +0200

Mallikarjun,

I did not see a SINGLE performance improvement that results from OOO 
shipping. 
I would be bad engineering to give away the "no-deadlock" mechanism we 
have now for nothing.
I have also the impression that the point about deadlock that I keep 
repeating is ignored or not understood.
As we stand today commands can be shipped with Immediate data or without 
and an implementer determined
to squeeze maximum bandwidth and overlap command start with delivery will 
choose not to work with immediate data
(as you have pointed out) while a low performance software implementation 
will use immediate data to minimize CPU cycles consumed.  However both 
will be guaranteed to work without deadlock as source and sink use the 
same ordering.
Recovery is still a low probability event and should be handled with a 
different set of considerations in mind.
As for the strictness of the recommendation - yes we could settle on 
SHOULD.

Julo

"Mallikarjun C." <[email-masked]>
Sent by: [email-masked]
07-11-01 19:41
Please respond to cbm

        To:     Santosh Rao <[email-masked]>, [email-masked]
        cc: 
        Subject:        Re: iSCSI: Out of order commands

Santosh,

I have only one comment on your responses.

> Even a single connection target *MUST* implement a scoreboard. The
> reason being that it can see out-of-order arrival of commands due to
> commands being dropped on digest errors. In such a case, it must block
> further command processing until holes are filled.

I made two convenient assumptions if you noticed, :-), one of which
is that target forces session recovery on *any* error that it sees 
(ErrorRecoveryLevel=0) - including a dropped command due to a digest 
error.  With that assumption, a target can afford not to implement 
a scoreboard.

As I said in a private note, I guess what primarily bothers me about 
OOO commands on a connection is that it requires the receiver to
undo this "optimization" on its end - most notably on a single 
connection.  TCP experts may comment on how/if they dealt with a 
similar issue. 

OTOH, you had some valid comments on exceptions to ordering during 
connection recovery.  Perhaps we can move on by making Julian's 
proposed stipulation a SHOULD....
-- 
Mallikarjun 

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668          Hewlett-Packard, Roseville.
[email-masked]

Santosh Rao wrote:
> 
> Mallikarjun,
> 
> Some comments below.
> 
> Regards,
> Santosh
> 
> "Mallikarjun C." wrote:
> >
> > Rod and Julian,
> >
> > This has been an interesting thread of discussion.  Some
> > comments -
> >
> > 1.My first reaction was - allowing out-of-order command
> >   transmission on the same connection deprives targets of
> >   an implementation choice.  Targets which support only
> >   single-connection sessions and only support session
> >   recovery (reasonable assumptions in my mind) can no
> >   longer afford *not to* implement a command scoreboard.
> 
> Even a single connection target *MUST* implement a scoreboard. The
> reason being that it can see out-of-order arrival of commands due to
> commands being dropped on digest errors. In such a case, it must block
> further command processing until holes are filled.
> 
> Thus, there is no getting away from implementing a sequencer at the
> target. Given this, I think it is unreasonable to restrict initiator
> implementation flexibility by imposing a strict ordering requirement
> within the connection.
> 
> > 2.Any end-node efficiency that is sought to be achieved
> >   by transmitting CmdSNs out-of-order from the initiator
> >   would be lost on the other end-node, since the target
> >   now must wait for re-ordering the commands.
> 
> It has to handle this situation anyway to deal with holes caused by
> digest errors. This scenario occurs even with initiators that issue
> commands in order.
> 
> >
> > 3.The flipside is that out-of-order transmission saves
> >   link badwidth (albeit at the expense of end-node efficiency),
> >   compared to idling the link waiting for outbound DMA.
> >   We have to determine if this is a reasonable trade-off.
> >
> > 4.I can see Rod's point that prefetching all immediate
> >   data can be a burden on the NIC resources.  But, two
> >   questions -
> >         - could the NIC not use unsolicited separate data
> >           PDUs in these cases? [ I realize that InitialR2T
> >           has to be "no" to let it happen... ]
> >         - could the NIC have a memory architecture that
> >           allows data prefetching for the next command (so
> >           this is a non-issue from the protocol perspective)?
> >           This scheme incurs one DMA delay for every new
> >           burst of commands.
> >
> > 5.Another (perhaps radical at this point) option is to do
> >   away with immediate unsolicited data, to stick only with
> >   separate unsolicited data.  I would personally be okay
> >   with the choice, particularly if this feature (that
> >   helps software implementations) starts making hardware
> >   design complicated/expensive.
> >
> > So, to summarize -
> >
> > option                         immediate         allow
> >                                data in spec?     out-of-order?
> >
> > (A) (5) above                  no                no
> > (B) No real reason to do this. no                yes
> > (C) (4) above                  yes               no
> > (D) pros & cons (1), (2) & (3) yes               yes
> >
> > >From the arguments I heard so far, I am leaning towards
> > option A, and option C in that order.
> >
> > Comments?
> > --
> > Mallikarjun
> >
> > Mallikarjun Chadalapaka
> > Networked Storage Architecture
> > Network Storage Solutions Organization
> > MS 5668 Hewlett-Packard, Roseville.
> > [email-masked]
> >
> > Rod Harrison wrote:
> > >
> > > Julian,
> > >
> > >         I don't understand what you are proposing here, what do you 
mean by
> > > "multiplexed" DMA?
> > >
> > >         The problem is that the DMAs take some time, the more there 
are
> > > queued the longer the last DMAs queued take to complete. Some 
commands
> > > require DMAs to complete before they can be sent, i.e. Writes with
> > > immediate data, some commands do not, i.e. Reads and writes with no
> > > immediate data. The iSCSI HBA wants to be able to send commands as
> > > soon a possible, which for a read after a write can be before the
> > > write's DMA has completed. Maintaining an ordered queue for commands
> > > to be sent on the HBA is expensive and redundant since the target
> > > already knows how to queue commands before committing them to its 
SCSI
> > > layer.
> > >
> > >         The iSCSI HBA and its host driver are not at liberty to 
change the
> > > order of commands from the OS, but the DMAs those commands need are
> > > unlikely to complete in the same order, and as I mentioned some
> > > commands need no DMA. If the HBA can't send commands out of CmdSN
> > > order it has to maintain an ordered queue of commands waiting to be
> > > sent, and potentially buffer a lot of data. For an HBA this makes
> > > immediate data almost impossible to support.
> > >
> > >         I don't see the problem with allowing out of order commands 
given
> > > that the target already has to deal with very similar problems. I
> > > think we are getting in to the area of implementation choices here,
> > > which is inappropriate for a specification.
> > >
> > >         - Rod
> > >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]]On Behalf 
Of
> > > Julian Satran
> > > Sent: Monday, November 05, 2001 10:06 PM
> > > To: [email-masked]
> > > Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest
> > >
> > > Rod,
> > >
> > > I don't see any reason why DMA operations cant be "multiplexed" with
> > > commands.
> > > If you have scheduled a long outbound DMA you are doomed regardless 
of
> > > the
> > > command ordering.
> > > And if you have scheduled DMA operations piecemeal then you can 
insert
> > > your commands in correct order.
> > >
> > > Julo
> > >
> > > "Rod Harrison" <[email-masked]>
> > > 05-11-01 20:48
> > > Please respond to "Rod Harrison"
> > >
> > >         To:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
> > >         cc:
> > >         Subject:        iSCSI: Out of order commands, was current 
UNH
> > > Plugfest
> > >
> > >                  [ Subject changed ]
> > >
> > > Julian,
> > >
> > >                  The ordering difference is introduced between the
> > > host
> > > side driver
> > > and the iSCSI HBA. The host side driver must present SCSI commands 
to
> > > the HBA in the order they are received from the OS to prevent read
> > > after write dependency failures. The HBA might reorder the commands
> > > depending on when DMA completes. The reordering can't be done ahead 
of
> > > time in the host driver since it doesn't know how long each DMA 
might
> > > take. As long as the HBA assigns CmdSN in the order it receives
> > > commands the desired host ordering is preserved.
> > >
> > >                  - Rod
> > >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]]On Behalf 
Of
> > > Julian Satran
> > > Sent: Monday, November 05, 2001 12:35 AM
> > > To: [email-masked]
> > > Subject: RE: iSCSI: current UNH Plugfest
> > >
> > > Rod,
> > >
> > > I all examples give the point I find hard to understand is why is 
the
> > > ordering on the wire different from the presentation order to the
> > > initiator.  You can get as many overlaps as you want by presenting 
the
> > > commands to the initiator in the desired order.
> > > What we are considering here is the case in which you want to ship 
in
> > > an
> > > order different than the one you present the commands.
> > >
> > > Julo
> > >
> > > "Rod Harrison" <[email-masked]>
> > > Sent by: [email-masked]
> > > 04-11-01 04:42
> > > Please respond to "Rod Harrison"
> > >
> > >         To:     "Barry Reinhold" <[email-masked]>, "Dave
> > > Sheehy"
> > > <[email-masked]>, "IETF IP SAN Reflector"
> > > <[email-masked]>
> > >         cc:
> > >         Subject:        RE: iSCSI: current UNH Plugfest
> > >
> > > Barry,
> > >
> > >                  In general I agree but I don't think this is as 
much
> > > of a
> > > corner case
> > > as it at first appears. Targets will have code very similar to that
> > > needed to handle out of order commands to deal with digest errors.
> > > Targets also need to queue commands whilst waiting for both 
solicited
> > > and unsolicited data to arrive. Queuing out of order commands seems
> > > little extra work.
> > >
> > >                  From an initiators point of view there are
> > > efficiency,
> > > and probably
> > > performance gains to be had from sending commands out of order. Bob
> > > Russell gave the example of a read being sent whilst write data DMA 
is
> > > happening, and a similar situation can arise with DMA for writes
> > > overtaking that of earlier writes if the initiator has multiple DMA
> > > engines. In this case the initiator might be forced to let the wire 
go
> > > idle if it can't send the data from completed DMAs as soon as
> > > possible.
> > >
> > >                  We already have a command queue at the target to
> > > enforce
> > > correct
> > > serialisation of commands, doing the same thing at the initiator is
> > > redundant.
> > >
> > >                  Finally, I don't believe we should be writing a
> > > standard
> > > to work
> > > around poor coding and test coverage, especially at the cost of
> > > potential efficiency gains.
> > >
> > >                  I agree with Dave and Santosh that commands being
> > > sent
> > > out of order
> > > on a single session should be allowed by the standard.
> > >
> > >                  - Rod
> > >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]]On Behalf 
Of
> > > Barry Reinhold
> > > Sent: Friday, November 02, 2001 5:24 PM
> > > To: Dave Sheehy; IETF IP SAN Reflector
> > > Subject: RE: iSCSI: current UNH Plugfest
> > >
> > > Using features such as out of order command delivery on a connection
> > > tend to
> > > be the sort of things that lead to interoperability problems. It is
> > > unexpected and probably going to hit poorly tested code paths even 
if
> > > the
> > > standard is written to allow it.
> > >
> > > >-----Original Message-----
> > > >From: [email-masked] [mailto:[email-masked]]On Behalf
> > > Of
> > > >Dave Sheehy
> > > >Sent: Friday, November 02, 2001 4:19 PM
> > > >To: IETF IP SAN Reflector
> > > >Subject: Re: iSCSI: current UNH Plugfest
> > > >
> > > >
> > > >
> > > >> 3. Can commands be sent out of order on the same connection?
> > > >>
> > > >>    The behavior of targets is clearly specified in Section 
2.2.2.3
> > > on
> > > >>    page 25 of draft 8, which says:
> > > >>      "Except for the commands marked for immediate delivery the
> > > iSCSI
> > > >>      target layer MUST eliver the commands for execution in the
> > > order
> > > >>      specified by CmdSN."
> > > >>
> > > >>    Section 2.2.2.3 on page 26 of draft 8 also says:
> > > >>      "- CmdSN - the current command Sequence Number advanced by 1
> > > on
> > > >>      each command shipped except for commands marked for 
immediate
> > > >>      delivery."
> > > >>    but the meaning of the term "shipped" is vague, and does not
> > > >> necessarily
> > > >>    require that the PDUs arrive on the other end of a TCP
> > > connection
> > > >>    in the same order that the CmdSN values were assigned to these
> > > PDUs.
> > > >>
> > > >>    Some initiators have been designed to send commands out of 
CmdSN
> > > >>    order on one connection.  Consider the situation where there 
is
> > > only
> > > >>    one connection and a high-level dispatcher creates a PDU for a
> > > SCSI
> > > >>    command that involves writing immediate data to the target.
> > > This PDU
> > > >>    is enqueued to a lower-level layer which has to setup, start,
> > > and
> > > >>    wait-for a DMA operation to move the immediate data into an
> > > onboard
> > > >>    buffer before the PDU can be put onto the wire.  While this is
> > > >>    happening, the dispatcher creates another unrelated PDU for a
> > > SCSI
> > > >>    read command (for example), and when this PDU is passed to the
> > > >>    lower-level layer it can be sent immediately, ahead of the
> > > previous
> > > >>    write PDU and therefore out of order on this connection.
> > > >>
> > > >>    The standard clearly allows this to happen if the two PDUs 
were
> > > sent
> > > >>    on different connections, and seems to imply that this can 
also
> > > happen
> > > >>    when the two PDUs are sent on the same connection.
> > > >>
> > > >>    The suggestion is to put in the standard an explicit statement
> > > that
> > > >>    this is allowed or not allowed, as appropriate.
> > > >>
> > > >>    If this is allowed, such a statement would avoid the erroneous
> > > >>    assumption being made by some target implementers that within 
a
> > > single
> > > >>    connection, commands will arrive in order.
> > > >>
> > > >>    If this is not allowed, such a statement would avoid the
> > > erroneous
> > > >>    assumption being made by some initiator implementers that 
within
> > > a
> > > >>    single connection, commands can be put on the wire out of 
order.
> > > >>
> > > >> +++
> > > >>
> > > >> will add an explicit statement saying that this behaviour is
> > > forbidden.
> > > >> 2.2.2.1 will contain:
> > > >>
> > > >> On any given connection, the iSCSI initiator MUST send the
> > > >commands in the
> > > >> order specified by CmdSN.
> > > >>
> > > >> +++
> > > >
> > > >Why do you feel this behavior should be forbidden? Targets already
> > > have to
> > > >order commands across the session. I don't see why it's a problem 
to
> > > extend
> > > >that to the connection as well. I, for one, believe we should take
> > > >a liberal
> > > >stance on this.
> > > >
> > > >Dave Sheehy
> > > >
> 
> --
> ##################################
> Santosh Rao
> Software Design Engineer,
> HP-UX iSCSI Driver Team,
> Hewlett Packard, Cupertino.
> email : [email-masked]
> Phone : 408-447-3751
> ##################################

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 7 Nov 2001 11:20:39 -0800

I think we should either have it as a MUST or not require
it (at both ends to get the real benefit). SHOULD is one
of those things that leads to implementation
burden and confusion, without perhaps the feature being
used. There are implementation as well as protocol
considerations mixed in here.

If we are to remove the restriction, we should (SHOULD)
get the maximum benefit from it, rather than to
accomodate an implementation choice. Out of sequence
commands, combined with the possibility of digest errors,
will add substantial complexity on the target side,
without corrosponding benefit in performance. If we change
this to SHOULD, we should also relax the requirement
to present commands on the target side to a SHOULD.

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Julian Satran
> Sent: Wednesday, November 07, 2001 10:00 AM
> To: [email-masked]

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 07 Nov 2001 13:39:05 -0800

Somesh,

I see a SHOULD encouraging efficient implementations in
this case, and I think it's desirable to do that.  As 
you yourself argued in your last email, this allows
implementations to optimize for the most-likely case.

IMHO, a MUST here would be going too far (connection recovery
scenario pointed out by Santosh) unless we start special
casing error recovery cases/multi-connection cases.  In
effect, that again comes down to a SHOULD for a general
n-connection session case.

Regards.
-- 
Mallikarjun 

Mallikarjun Chadalapaka

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 7 Nov 2001 13:30:48 -0800

> From: Robert D. Russell [mailto:[email-masked]]
> Sent: Wednesday, November 07, 2001 1:13 PM
> To: Somesh Gupta
> Cc: Julian Satran; [email-masked]
> Subject: RE: iSCSI: Out of order commands
>
>
> Somesh, Julian:
>
> You state that dealing with OOO commands on the target
> will add substantial complexity on the target.
> Do you have any basis for that claim?  My impression from the
> plugfest is that most targets are already dealing with
> it.  Perhaps we need to hear from someone who is actually
> building a target for which this would be a real problem.

  Since we are making related products, I can assure you
  that this adds complexity. Mallikarjun also added the
  target perspecitive.

>
> If anything, what we are hearing from people who really
> are building initiators is that dealing with the requirement
> to send commands in order will introduce substantial complexity
> on the initiator.

  I think you heard from one (WindRiver). We don't have a
  problem and it does not add "substantial complexity".

>
> So why should we be saving complexity on (hypothetically) simple
> targets yet requiring complexity on real initiators?
>
> As far as the deadlock issue is concerned, if the only way
> that deadlock can occur with OOO commands on the same
> connection is during the use of immediate data (which is I
> think what Julian was saying), then shouldn't we change
> the standard to just say that -- if the initiator sends
> commands out of order on a single connection, then immediate
> data MUST NOT be used on that connection in order to avoid deadlock.
>
> This gives everybody what they want, since initiators who find
> it beneficial to deliver commands OOO will just negotiate
> immediate data off.  Those who really want to use immediate data
> will have to ensure that commands are sent in order.
> The tradeoff then becomes an implementation issue, not a
> standards issue, which is where it belongs.

  An initiator that has a problem with this model can avoid
  sending immediate data.
>
>
> Bob Russell
> InterOperability Lab
> University of New Hampshire
> [email-masked]
> 603-862-3774

  Bob,

  Sorry to be so blunt, but I say this because of your insinuations
  here. The only one in the debate not building a product
  seems to be you. But I do not disagree with your right
  to add meaningful points as you have done in the past.

  Somesh
>
>
> On Wed, 7 Nov 2001, Somesh Gupta wrote:
>

---

From: "Robert D. Russell" <[email-masked]>
Date: Wed, 7 Nov 2001 16:13:26 -0500

Somesh, Julian:

Bob Russell
InterOperability Lab

---

From: "Rod Harrison" <[email-masked]>
Date: Wed, 7 Nov 2001 13:11:08 -0800

Julian,

	The deadlock situation you outlined is not related to out of order
commands, it arises because the target implementation is broken. A
target that advertises a command window larger than the number of
commands it can support with the intent of leveraging TCP buffering is
making big, and invalid assumptions about the initiator. The target
has no idea how long it might take to receive / generate the data
associated with commands. For example, if a target offers a command
window of 10 with the intent of processing 8 commands at a time and
having the other two in flight it will deadlock if the oldest command
needs an R2T. This happens even if the commands are sent in order. It
is never acceptable for a target to stop reading its TCP stream unless
its command window is full since there may be unread DATA-OUTs needed
to allow commands to be committed to SCSI.

	There are very real performance gains associated with sending out of
order commands, the simplest of which was outlined by Bob Russell when
this thread started. We do not have a zero latency network so queuing
as close as possible the ultimate data sink will always be a win.

	- Rod

-----Original Message-----

Mallikarjun,

I did not see a SINGLE performance improvement that results from OOO
shipping.
I would be bad engineering to give away the "no-deadlock" mechanism we
have now for nothing.
I have also the impression that the point about deadlock that I keep
repeating is ignored or not understood.
As we stand today commands can be shipped with Immediate data or
without
and an implementer determined
to squeeze maximum bandwidth and overlap command start with delivery
will
choose not to work with immediate data
(as you have pointed out) while a low performance software
implementation
will use immediate data to minimize CPU cycles consumed.  However both
will be guaranteed to work without deadlock as source and sink use the
same ordering.
Recovery is still a low probability event and should be handled with a
different set of considerations in mind.
As for the strictness of the recommendation - yes we could settle on
SHOULD.

Julo

Santosh,

Mallikarjun Chadalapaka

mean by
are
commands
SCSI
change the
given
Of
of
insert
UNH
to
of
might
Of
the
the
in
much
solicited
is
go
Of
if
2.2.2.3
immediate
CmdSN
is
were
also
a
within
order.
to

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 7 Nov 2001 13:37:26 -0800

Mallikarjun,

Again, a simple target does not bear this burden.
Only targets that do accept striping of commands
have to deal with this.

It is also a performance inhibitor on the target
side unless we also add SHOULD to out of delivery.
You may have shipped the command faster, but it
then just sits somewhere till the holes are filled.
(and of course the processing and data structure
requirements that it implies). Sort of like speeding
to the next red light.

Somesh

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Mallikarjun C.
> Sent: Wednesday, November 07, 2001 1:39 PM
> To: [email-masked]
> Cc: [email-masked]

---

From: "Dave Francheski" <[email-masked]>
Date: Wed, 7 Nov 2001 15:39:05 -0800

Hi all,

This may not be the proper forum for this question,
but I have exhausted almost all other options at this point.

Does anybody know of a Windows based (e.g., Windows 2000)
version of an iSCSI initiator implemented to Revision 6
(or greater) of the iSCSI standard?
   
I have Cisco's version of the iSCSI Initiator for
Windows but it is implemented to the first original
draft of iSCSI, and doesn't interoperate with other
more recent iSCSI target implementations.

Thankyou for any and all support you can provide.

regards,
-Dave

David Francheski
Seven Systems Technologies
575 Menlo Drive Suite 2
Rocklin CA
916-577-1281
[email-masked]

---

From: "Barry Reinhold" <[email-masked]>
Date: Wed, 7 Nov 2001 18:09:36 -0500

Bob,
	I can't speak for targets, but OOO commands on a session with a single
connection sure increases the complexity of the code path we take.
	To me the issue is still that these types of situations tend to be poorly
tested and lead to interoperability issues. If we do go down this path, the
spec. should make it very clear that this is expected behavior. Some
statement with a shall in it should be written (for the receiver), so that
there is a conformance test items to pass.

>Robert D. Russell
>Sent: Wednesday, November 07, 2001 4:13 PM
>To: Somesh Gupta

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 7 Nov 2001 17:46:13 -0800

Rod,

Sorry to butt in, but targets are not doing serialized
processing. They will process many many commands
simultaneously.

We had prior discussions on this and unfortunately
we agreed on ordered delivery at the iSCSI level. The
real boost in performance is to have ordering semantics
at a higher layer, so that the user code can determine
whether it needs ordering or not.

As an example, in the general/normal case of a computer
talking to a disk, the ordering rule could be relaxed
completely.

Somesh

> From: Rod Harrison [mailto:[email-masked]]
> Sent: Wednesday, November 07, 2001 5:34 PM
> To: Barry Reinhold; Robert D. Russell; Somesh Gupta
> 	I'm curious, where do you see the extra complexity between the
> following?
> 
> 	Assuming all the following commands are writes ...
> 
> c1, c3, c2, c4 and
> 
> c1 with no payload, c2 + immediate, c3 + immediate, c4 + immediate.
> 
> 	In both cases the target has to queue commands 2, 3, and 4 whilst
> waiting for its R2T on c1 to be satisfied.
> 
> 	Even if the target doesn't support any unsolicited data it still has
> to queue commands 2, 3, and 4. I can't see how the target can avoid
> queuing commands, in which case the order of arrival seems to make
> little difference.
> 
> 	- Rod
> 
> Sent: Wednesday, November 07, 2001 3:10 PM
> To: Robert D. Russell; Somesh Gupta

---

From: "Rod Harrison" <[email-masked]>
Date: Wed, 7 Nov 2001 17:34:14 -0800

Barry,

c1, c3, c2, c4 and

	- Rod

-----Original Message-----

Bob,
	I can't speak for targets, but OOO commands on a session with a
single
connection sure increases the complexity of the code path we take.
	To me the issue is still that these types of situations tend to be
poorly
tested and lead to interoperability issues. If we do go down this
path, the
spec. should make it very clear that this is expected behavior. Some
statement with a shall in it should be written (for the receiver), so
that
there is a conformance test items to pass.

Of

---

From: "Rod Harrison" <[email-masked]>
Date: Wed, 7 Nov 2001 17:57:56 -0800

Somesh,

	The spec prohibits processing commands simultaneously, if by
processing you mean handing off to the SCSI layer. If by processing
simultaneously you mean preparing data for committing to SCSI when the
CmdSN allows then surely reception of commands out of order is a no
brainer.

	By the way, I am in support of allowing the target to make the
decision on when to commit the SCSI layer rather than having to use
the CmdSN. That decision can be made on the basis of device type, LUN,
and queue tag. Forcing CmdSN order of committal to SCSI has always
seemed overly restrictive to me.

	- Rod

-----Original Message-----
From: Somesh Gupta [mailto:[email-masked]]
Sent: Wednesday, November 07, 2001 5:46 PM
To: Rod Harrison; Barry Reinhold; Robert D. Russell
Cc: Julian Satran; [email-masked]
Subject: RE: iSCSI: Out of order commands

Rod,

Somesh

Of
> >> > I did not see a SINGLE performance improvement that results
from
> OOO
> >> > will use immediate data to minimize CPU cycles consumed.
However
> both
> >> > As for the strictness of the recommendation - yes we could
settle
> >> > > Even a single connection target *MUST* implement a
scoreboard.
> The
> >> > > Even a single connection target *MUST* implement a
scoreboard.
> The
> >> > > Thus, there is no getting away from implementing a sequencer
at
> >> > > > > to be sent on the HBA is expensive and redundant since
the
> target
> >> > > > > already knows how to queue commands before committing
them
> to its
> >> > > > > unlikely to complete in the same order, and as I
mentioned
> some
> >> > > > > commands need no DMA. If the HBA can't send commands out
of
> >> > > > > sent, and potentially buffer a lot of data. For an HBA
this
> makes
> >> > > > > And if you have scheduled DMA operations piecemeal then
you
> can
> >> > > > > time in the host driver since it doesn't know how long
each
> DMA
> >> > > > > I all examples give the point I find hard to understand
is
> why is
> >> > the
> >> > > > > ordering on the wire different from the presentation
order
> to the
> >> > > > >         To:     "Barry Reinhold"
<[email-masked]>,
> "Dave
> >> > > > >                  In general I agree but I don't think
this
> is as
> >> > > > > idle if it can't send the data from completed DMAs as
soon
> as
> >> > > > > >>    The standard clearly allows this to happen if the
two
> PDUs
> >> > > > > >>    If this is not allowed, such a statement would
avoid
> the
> >> > > > > erroneous
> >> > > > > >>    assumption being made by some initiator
implementers

---

From: "Barry Reinhold" <[email-masked]>
Date: Thu, 8 Nov 2001 08:06:14 -0500

Bob,

---

From: "Barry Reinhold" <[email-masked]>
Date: Thu, 8 Nov 2001 09:19:24 -0500

Rod,
	I can see your point, from the perspective of a target that is a box
combining an iSCSI entity with a SCSI entity. However, from the perspective
of an "iSCSI only" entity, delivery does not have to be delayed until C1
completes. In this case you only need to ensure that the commands are
delivered in order to the SCSI layer. The SCSI layer has to address the
semantics of what the operation entails in terms of command execution. Thus
the iSCSI layer delivers C1, C2, C3, and C4. It doesn't worry about the fact
that C1 generates data in response to the R2T. However the command sequence
of C1, C3, C2, C4 forces the iSCSI layer to buffer C3. This is significant
to a hardware accelerated iSCSI layer.

>Sent: Wednesday, November 07, 2001 8:34 PM

---

From: "Eddy Quicksall" <[email-masked]>
Date: Thu, 8 Nov 2001 18:07:04 -0500

Rob,

You will not know the tag type of a command that is not there yet. That is
why you must only submit to the SCSI layer in CmdSN order (at least that is
why I thought that was in the spec).

E.g., C0 and C1 may be a simple tag, C2 may be an ordered tag and C3 may be
a simple tag. C0 and C1 can execute in any order but, because C2 is ordered,
C3 must execute after C2.

Therefore, C3 must wait for C2 and CmdSN regiments that.

Eddy

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Wednesday, November 07, 2001 8:58 PM
To: [email-masked]; Barry Reinhold; Robert D. Russell
Cc: Julian Satran; [email-masked]
Subject: RE: iSCSI: Out of order commands

Somesh,

        - Rod

-----Original Message-----
From: Somesh Gupta [mailto:[email-masked]]
Sent: Wednesday, November 07, 2001 5:46 PM
To: Rod Harrison; Barry Reinhold; Robert D. Russell
Cc: Julian Satran; [email-masked]
Subject: RE: iSCSI: Out of order commands

Rod,

Somesh

Of
from
However
settle
scoreboard.
scoreboard.
at
the
them
mentioned
of
this
you
each
is
order
this
soon
two
avoid
implementers