Subject: Re: iSCSI: current UNH Plugfest

From: Dave Sheehy <[email-masked]>
Date: Fri, 02 Nov 2001 13:18:34 PST

> 3. Can commands be sent out of order on the same connection?
> 
>    The behavior of targets is clearly specified in Section 2.2.2.3 on
>    page 25 of draft 8, which says:
>      "Except for the commands marked for immediate delivery the iSCSI
>      target layer MUST eliver the commands for execution in the order
>      specified by CmdSN."
> 
>    Section 2.2.2.3 on page 26 of draft 8 also says:
>      "- CmdSN - the current command Sequence Number advanced by 1 on
>      each command shipped except for commands marked for immediate
>      delivery."
>    but the meaning of the term "shipped" is vague, and does not 
> necessarily
>    require that the PDUs arrive on the other end of a TCP connection
>    in the same order that the CmdSN values were assigned to these PDUs.
> 
>    Some initiators have been designed to send commands out of CmdSN
>    order on one connection.  Consider the situation where there is only
>    one connection and a high-level dispatcher creates a PDU for a SCSI
>    command that involves writing immediate data to the target.  This PDU
>    is enqueued to a lower-level layer which has to setup, start, and
>    wait-for a DMA operation to move the immediate data into an onboard
>    buffer before the PDU can be put onto the wire.  While this is
>    happening, the dispatcher creates another unrelated PDU for a SCSI
>    read command (for example), and when this PDU is passed to the
>    lower-level layer it can be sent immediately, ahead of the previous
>    write PDU and therefore out of order on this connection.
> 
>    The standard clearly allows this to happen if the two PDUs were sent
>    on different connections, and seems to imply that this can also happen
>    when the two PDUs are sent on the same connection.
> 
>    The suggestion is to put in the standard an explicit statement that
>    this is allowed or not allowed, as appropriate.
>  
>    If this is allowed, such a statement would avoid the erroneous
>    assumption being made by some target implementers that within a single
>    connection, commands will arrive in order.
> 
>    If this is not allowed, such a statement would avoid the erroneous
>    assumption being made by some initiator implementers that within a
>    single connection, commands can be put on the wire out of order.
> 
> +++ 
> 
> will add an explicit statement saying that this behaviour is forbidden.
> 2.2.2.1 will contain:
> 
> On any given connection, the iSCSI initiator MUST send the commands in the 
> order specified by CmdSN.
> 
> +++

Why do you feel this behavior should be forbidden? Targets already have to
order commands across the session. I don't see why it's a problem to extend
that to the connection as well. I, for one, believe we should take a liberal
stance on this.

Dave Sheehy

---

From: Santosh Rao <[email-masked]>
Date: Fri, 02 Nov 2001 14:06:19 -0800

I agree with Dave. I think a wire protocol is stepping out of bounds in
imposing this kind of restrictions on implementations. As long as an end
to end ordering scheme exists and is mandatory to implement/use, I don't
see why ordering within the initiator stack must be mandated ?

- Santosh

-- 
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: "Barry Reinhold" <[email-masked]>
Date: Fri, 2 Nov 2001 20:23:59 -0500

Using features such as out of order command delivery on a connection tend to
be the sort of things that lead to interoperability problems. It is
unexpected and probably going to hit poorly tested code paths even if the
standard is written to allow it.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Dave Sheehy
>Sent: Friday, November 02, 2001 4:19 PM
>To: IETF IP SAN Reflector
>Subject: Re: iSCSI: current UNH Plugfest
>
>
>

---

From: "Rod Harrison" <[email-masked]>
Date: Sat, 3 Nov 2001 18:42:13 -0800

Barry,

	In general I agree but I don't think this is as much of a corner case
as it at first appears. Targets will have code very similar to that
needed to handle out of order commands to deal with digest errors.
Targets also need to queue commands whilst waiting for both solicited
and unsolicited data to arrive. Queuing out of order commands seems
little extra work.

	From an initiators point of view there are efficiency, and probably
performance gains to be had from sending commands out of order. Bob
Russell gave the example of a read being sent whilst write data DMA is
happening, and a similar situation can arise with DMA for writes
overtaking that of earlier writes if the initiator has multiple DMA
engines. In this case the initiator might be forced to let the wire go
idle if it can't send the data from completed DMAs as soon as
possible.

	We already have a command queue at the target to enforce correct
serialisation of commands, doing the same thing at the initiator is
redundant.

	Finally, I don't believe we should be writing a standard to work
around poor coding and test coverage, especially at the cost of
potential efficiency gains.

	I agree with Dave and Santosh that commands being sent out of order
on a single session should be allowed by the standard.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Barry Reinhold
Sent: Friday, November 02, 2001 5:24 PM
To: Dave Sheehy; IETF IP SAN Reflector
Subject: RE: iSCSI: current UNH Plugfest

Using features such as out of order command delivery on a connection
tend to
be the sort of things that lead to interoperability problems. It is
unexpected and probably going to hit poorly tested code paths even if
the
standard is written to allow it.

---

From: "Dave Peterson" <[email-masked]>
Date: Sat, 3 Nov 2001 21:47:31 -0600

There is no reason why the Initiator needs to send commands in CmdSN order.
The target should be able to queue up any out of order commands to the
queuing capacity. As stated below the Target is responsible for delivering
the commands for execution in the order specified by CmdSN.

Dave

> Sent: Friday, November 02, 2001 7:24 PM