Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest

From: "Julian Satran" <[email-masked]>
Date: Tue, 6 Nov 2001 08:06:06 +0200

Rod,

I don't see any reason why DMA operations cant be "multiplexed" with 
commands.
If you have scheduled a long outbound DMA you are doomed regardless of the 
command ordering.
And if you have scheduled DMA operations piecemeal then you can insert 
your commands in correct order.

Julo

"Rod Harrison" <[email-masked]>
05-11-01 20:48
Please respond to "Rod Harrison"

        To:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
        cc: 
        Subject:        iSCSI: Out of order commands, was current UNH Plugfest

                 [ Subject changed ]

Julian,

                 The ordering difference is introduced between the host 
side driver
and the iSCSI HBA. The host side driver must present SCSI commands to
the HBA in the order they are received from the OS to prevent read
after write dependency failures. The HBA might reorder the commands
depending on when DMA completes. The reordering can't be done ahead of
time in the host driver since it doesn't know how long each DMA might
take. As long as the HBA assigns CmdSN in the order it receives
commands the desired host ordering is preserved.

                 - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Julian Satran
Sent: Monday, November 05, 2001 12:35 AM
To: [email-masked]
Subject: RE: iSCSI: current UNH Plugfest

Rod,

I all examples give the point I find hard to understand is why is the
ordering on the wire different from the presentation order to the
initiator.  You can get as many overlaps as you want by presenting the
commands to the initiator in the desired order.
What we are considering here is the case in which you want to ship in
an
order different than the one you present the commands.

Julo

"Rod Harrison" <[email-masked]>
Sent by: [email-masked]
04-11-01 04:42
Please respond to "Rod Harrison"

        To:     "Barry Reinhold" <[email-masked]>, "Dave
Sheehy"
<[email-masked]>, "IETF IP SAN Reflector"
<[email-masked]>
        cc:
        Subject:        RE: iSCSI: current UNH Plugfest

Barry,

                 In general I agree but I don't think this is as much
of a
corner case
as it at first appears. Targets will have code very similar to that
needed to handle out of order commands to deal with digest errors.
Targets also need to queue commands whilst waiting for both solicited
and unsolicited data to arrive. Queuing out of order commands seems
little extra work.

                 From an initiators point of view there are
efficiency,
and probably
performance gains to be had from sending commands out of order. Bob
Russell gave the example of a read being sent whilst write data DMA is
happening, and a similar situation can arise with DMA for writes
overtaking that of earlier writes if the initiator has multiple DMA
engines. In this case the initiator might be forced to let the wire go
idle if it can't send the data from completed DMAs as soon as
possible.

                 We already have a command queue at the target to
enforce
correct
serialisation of commands, doing the same thing at the initiator is
redundant.

                 Finally, I don't believe we should be writing a
standard
to work
around poor coding and test coverage, especially at the cost of
potential efficiency gains.

                 I agree with Dave and Santosh that commands being
sent
out of order
on a single session should be allowed by the standard.

                 - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Barry Reinhold
Sent: Friday, November 02, 2001 5:24 PM
To: Dave Sheehy; IETF IP SAN Reflector
Subject: RE: iSCSI: current UNH Plugfest

Using features such as out of order command delivery on a connection
tend to
be the sort of things that lead to interoperability problems. It is
unexpected and probably going to hit poorly tested code paths even if
the
standard is written to allow it.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf
Of
>Dave Sheehy
>Sent: Friday, November 02, 2001 4:19 PM
>To: IETF IP SAN Reflector
>Subject: Re: iSCSI: current UNH Plugfest
>
>
>
>> 3. Can commands be sent out of order on the same connection?
>>
>>    The behavior of targets is clearly specified in Section 2.2.2.3
on
>>    page 25 of draft 8, which says:
>>      "Except for the commands marked for immediate delivery the
iSCSI
>>      target layer MUST eliver the commands for execution in the
order
>>      specified by CmdSN."
>>
>>    Section 2.2.2.3 on page 26 of draft 8 also says:
>>      "- CmdSN - the current command Sequence Number advanced by 1
on
>>      each command shipped except for commands marked for immediate
>>      delivery."
>>    but the meaning of the term "shipped" is vague, and does not
>> necessarily
>>    require that the PDUs arrive on the other end of a TCP
connection
>>    in the same order that the CmdSN values were assigned to these
PDUs.
>>
>>    Some initiators have been designed to send commands out of CmdSN
>>    order on one connection.  Consider the situation where there is
only
>>    one connection and a high-level dispatcher creates a PDU for a
SCSI
>>    command that involves writing immediate data to the target.
This PDU
>>    is enqueued to a lower-level layer which has to setup, start,
and
>>    wait-for a DMA operation to move the immediate data into an
onboard
>>    buffer before the PDU can be put onto the wire.  While this is
>>    happening, the dispatcher creates another unrelated PDU for a
SCSI
>>    read command (for example), and when this PDU is passed to the
>>    lower-level layer it can be sent immediately, ahead of the
previous
>>    write PDU and therefore out of order on this connection.
>>
>>    The standard clearly allows this to happen if the two PDUs were
sent
>>    on different connections, and seems to imply that this can also
happen
>>    when the two PDUs are sent on the same connection.
>>
>>    The suggestion is to put in the standard an explicit statement
that
>>    this is allowed or not allowed, as appropriate.
>>
>>    If this is allowed, such a statement would avoid the erroneous
>>    assumption being made by some target implementers that within a
single
>>    connection, commands will arrive in order.
>>
>>    If this is not allowed, such a statement would avoid the
erroneous
>>    assumption being made by some initiator implementers that within
a
>>    single connection, commands can be put on the wire out of order.
>>
>> +++
>>
>> will add an explicit statement saying that this behaviour is
forbidden.
>> 2.2.2.1 will contain:
>>
>> On any given connection, the iSCSI initiator MUST send the
>commands in the
>> order specified by CmdSN.
>>
>> +++
>
>Why do you feel this behavior should be forbidden? Targets already
have to
>order commands across the session. I don't see why it's a problem to
extend
>that to the connection as well. I, for one, believe we should take
>a liberal
>stance on this.
>
>Dave Sheehy
>

---

From: Santosh Rao <[email-masked]>
Date: Tue, 06 Nov 2001 11:52:12 -0800

Julian,

There's further reasons why iscsi must not mandate that the initiator
issue commands in order within a given connection :

1) The cmdsn assignment may be done in a host resident session manager
in an architecture that allows for multi-connection sessions that span
HBAs.
In such a model, the initiator may issue commands from the host driver
to the HBA in cmdsn order. However, within the HBA firmware, some
command[s] may be errored back to the host either due to resource
reasons or link down or mailbox command incorrectly initialized.
Requiring that the initiator issue commands in order implies that the
initiator HBA firmware needs to implement some sort of sequencing scheme
to ensure that it does not issue a lower cmdsn.

There's no need to add any of the above complexity into initiator HBA
firmware, since command ordering is already being implemented at the
target end. What we've ended up doing is inserting sequencing logic at
both ends in attempting to enforce compliance to this rule !

2) On a connection recovery, re-issued commands [on a cmdsn ack timeout]
may be issued on a different connection, thus, causing out of order
cmdsn's to be issued on a specific connection. There's no way for the
initiator HBA firmware to know that this is a re-issued command and
detect that this specific out of order cmdsn is to be allowed.

3) Initiator driver algorithms are quite flexible in their ordering
schemes today and imposing strict ordering at the initiator is beyond
what is required for the wire protocol. The layer that hands out cmdsn's
to outgoing commands may be a higher layer driver in the iscsi stack
(common across multiple vendor specific drivers) and re-ordering of
commands may occur in the lower layers of the iscsi stack.

For all of the above reasons, in addition to HBA optimizations that Rod
has voiced, I suggest that NO ordering requirement be imposed on
initiators within a specific connection. The cmdsn ordering scheme is
mandatory to implement/use and that is more than sufficient.

Regards,
Santosh

-- 
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: "Rod Harrison" <[email-masked]>
Date: Tue, 6 Nov 2001 11:28:06 -0800

Julian,

	I don't understand what you are proposing here, what do you mean by
"multiplexed" DMA?

	The problem is that the DMAs take some time, the more there are
queued the longer the last DMAs queued take to complete. Some commands
require DMAs to complete before they can be sent, i.e. Writes with
immediate data, some commands do not, i.e. Reads and writes with no
immediate data. The iSCSI HBA wants to be able to send commands as
soon a possible, which for a read after a write can be before the
write's DMA has completed. Maintaining an ordered queue for commands
to be sent on the HBA is expensive and redundant since the target
already knows how to queue commands before committing them to its SCSI
layer.

	The iSCSI HBA and its host driver are not at liberty to change the
order of commands from the OS, but the DMAs those commands need are
unlikely to complete in the same order, and as I mentioned some
commands need no DMA. If the HBA can't send commands out of CmdSN
order it has to maintain an ordered queue of commands waiting to be
sent, and potentially buffer a lot of data. For an HBA this makes
immediate data almost impossible to support.

	I don't see the problem with allowing out of order commands given
that the target already has to deal with very similar problems. I
think we are getting in to the area of implementation choices here,
which is inappropriate for a specification.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Julian Satran
Sent: Monday, November 05, 2001 10:06 PM
To: [email-masked]
Subject: Re: iSCSI: Out of order commands, was current UNH Plugfest

Rod,

I don't see any reason why DMA operations cant be "multiplexed" with
commands.
If you have scheduled a long outbound DMA you are doomed regardless of
the
command ordering.
And if you have scheduled DMA operations piecemeal then you can insert
your commands in correct order.

Julo

        To:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
        cc:
        Subject:        iSCSI: Out of order commands, was current UNH
Plugfest

                 [ Subject changed ]

Julian,

                 The ordering difference is introduced between the
host
side driver

                 - Rod

-----Original Message-----

Rod,

Julo

Barry,

                 - Rod

-----Original Message-----

Of
on
iSCSI
order
on
connection
PDUs.
only
SCSI
This PDU
and
onboard
SCSI
previous
sent
happen
that
single
erroneous
a
forbidden.
have to
extend

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 06 Nov 2001 17:17:50 -0800

Rod and Julian,

This has been an interesting thread of discussion.  Some
comments -

1.My first reaction was - allowing out-of-order command
  transmission on the same connection deprives targets of
  an implementation choice.  Targets which support only 
  single-connection sessions and only support session 
  recovery (reasonable assumptions in my mind) can no 
  longer afford *not to* implement a command scoreboard.

2.Any end-node efficiency that is sought to be achieved
  by transmitting CmdSNs out-of-order from the initiator
  would be lost on the other end-node, since the target
  now must wait for re-ordering the commands.  

3.The flipside is that out-of-order transmission saves 
  link badwidth (albeit at the expense of end-node efficiency),
  compared to idling the link waiting for outbound DMA. 
  We have to determine if this is a reasonable trade-off.

4.I can see Rod's point that prefetching all immediate 
  data can be a burden on the NIC resources.  But, two
  questions -
	- could the NIC not use unsolicited separate data
          PDUs in these cases? [ I realize that InitialR2T
          has to be "no" to let it happen... ]
	- could the NIC have a memory architecture that 
          allows data prefetching for the next command (so
          this is a non-issue from the protocol perspective)?
          This scheme incurs one DMA delay for every new 
          burst of commands.

5.Another (perhaps radical at this point) option is to do 
  away with immediate unsolicited data, to stick only with
  separate unsolicited data.  I would personally be okay
  with the choice, particularly if this feature (that 
  helps software implementations) starts making hardware
  design complicated/expensive. 

So, to summarize -

option                         immediate         allow
                               data in spec?     out-of-order?

(A) (5) above                  no                no
(B) No real reason to do this. no                yes
(C) (4) above                  yes               no
(D) pros & cons (1), (2) & (3) yes               yes

From the arguments I heard so far, I am leaning towards
option A, and option C in that order.

Comments?
-- 
Mallikarjun 

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668	Hewlett-Packard, Roseville.
[email-masked]

---

From: Santosh Rao <[email-masked]>
Date: Tue, 06 Nov 2001 18:15:56 -0800

Mallikarjun,

Some comments below.

Regards,
Santosh

>   longer afford *not to* implement a command scoreboard.

Even a single connection target *MUST* implement a scoreboard. The
reason being that it can see out-of-order arrival of commands due to
commands being dropped on digest errors. In such a case, it must block
further command processing until holes are filled.

Thus, there is no getting away from implementing a sequencer at the
target. Given this, I think it is unreasonable to restrict initiator
implementation flexibility by imposing a strict ordering requirement
within the connection. 

>   now must wait for re-ordering the commands.

It has to handle this situation anyway to deal with holes caused by
digest errors. This scenario occurs even with initiators that issue
commands in order.

---

From: "Rahul Bhagwat" <[email-masked]>
Date: Wed, 7 Nov 2001 08:13:54 +0530

Hi,

Comment below

> implementation flexibility by imposing a strict ordering requirement

If a target supports only session recovery, then a digest (header) error
means we have to drop the connection (Such a simple target would not
even probably support markers). I think for data digest errors command
can be treated as received (Command SN window moves).

---

From: "Rod Harrison" <[email-masked]>
Date: Tue, 6 Nov 2001 20:16:26 -0800

It seems to me that if a target offers a command window greater than
one it is buying into the complexity associated with supporting that
window.

	There is very little difference at the target between buffering a
command that arrives out of order and a buffering a write command that
arrives without all the payload. In both cases other commands may
arrive which cannot be committed to the SCSI layer. If a target
doesn't want to be in the business of command queuing it has the
option of offering a command window of one.

	Implementing a command queue is a much simpler proposition at the
target than at the initiator. The target is in complete control of the
command window and can therefore simply use a static array to hold
command descriptors. The initiator has no such luxury since it has no
a priori knowledge of the command window size, indeed the command
window size can change dynamically. For the initiator to support
ordered command queuing it must use an ordered list which can be
expensive, especially when we consider the CPU power that will
typically be available to an iSCSI HBA. Negotiating the command window
size as part of login would make this more palatable for an initiator,
but I suspect targets wouldn't want to commit to an unchanging command
window.

	We've been debating the merits of an initiator sending out of order
commands which is perhaps beyond the scope of where we should be. The
cost to a target implementer is negligible and there is a potential
benefit to an initiator implementer, so why should we prohibit this
behaviour?

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Santosh Rao
Sent: Tuesday, November 06, 2001 6:16 PM
To: [email-masked]
Cc: Rod Harrison; Julian Satran; [email-masked]
Subject: Re: iSCSI: Out of order commands

Mallikarjun,

Some comments below.

Regards,
Santosh

---

From: "Sukanta Ganguly" <[email-masked]>
Date: Tue, 06 Nov 2001 22:17:09 -0800

Rod,
  I agree with the philosophy of letting the target have the capability to decide on the command window. The initiator just follows suit. Stating that the initiator will not generate holes on its own. The holes will only occur due to loss of packets in which case TCP retransmission will handle the resends and hence the initiaTOR is not even aware of the holes being generated during the transport mechanism.

Even if the initiator decides to handle multiple command windows, the target will still have to do it. So loading the initiator with extra work does not buy any benefits.

SG

*********** REPLY SEPARATOR  ***********

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 07 Nov 2001 09:41:05 -0800

Santosh,

I have only one comment on your responses.

> further command processing until holes are filled.

I made two convenient assumptions if you noticed, :-), one of which
is that target forces session recovery on *any* error that it sees 
(ErrorRecoveryLevel=0) - including a dropped command due to a digest 
error.  With that assumption, a target can afford not to implement 
a scoreboard.

As I said in a private note, I guess what primarily bothers me about 
OOO commands on a connection is that it requires the receiver to
undo this "optimization" on its end - most notably on a single 
connection.  TCP experts may comment on how/if they dealt with a 
similar issue. 

OTOH, you had some valid comments on exceptions to ordering during 
connection recovery.  Perhaps we can move on by making Julian's 
proposed stipulation a SHOULD....
-- 
Mallikarjun 

Mallikarjun Chadalapaka

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 7 Nov 2001 10:02:19 -0800

Santosh,

Considering what the likelyhood (or lack thereof)
of digest errors, its occurance can be treated in
quite a different way, then if out of order
command transmission is allowed by the initiator.
This premise allows a completely different set
of design choices.

Of course, the ultimate would be to do away with
ordering at either end, and let the upper layer
deal with this issue. The upper layer may or may
not want any ordering at either end, and this is
quite a reasonable scenario in the normal case of
a computer communicating with a disk drive (array).

Somesh

---

From: Santosh Rao <[email-masked]>
Date: Wed, 07 Nov 2001 18:41:50 -0800

Mallikarjun , Julian & All,

A target *MUST* implement command ordering even if it is only supporting
single connection sessions and is only using session recovery.

Such targets will still see holes in CmdSN sequence on a protocol error
and need to stall the processing of subsequently received commands.

There is no getting away from implementing command ordering at the
target unless iscsi makes command ordering itself optional to implement
and negotiate this at login. 

Given the above, iscsi MUST NOT attempt to enforce upon initiators a
requirement to transmit commands in order within a given connection.
There is no tangible requirement for this and implementation flexibility
should be allowed as long as a scheme exists to maintain end-to-end
ordering.

Regards,
Santosh