Subject: Addition of CmdSN in Data-out PDU

From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>
Date: Mon, 8 Oct 2001 11:30:56 +0100

Please can the CmdSN be added to the Data-out PDU to improve implementations
when receiving unsolicited Data-out PDUs:

Currently, implementations have to search the command queue list for the
associated command using the ITT as the search criteria.  For solicited
Data-out PDUs this is not an issue as they contain the Target Task Tag.
However, for unsolicated Data PDUs (Target Task Tag = 0xFFFFFFFF) the
locating of the associated command can be greatly enhanced by adding the
Associated CmdSN to Data-out PDU.

Cheers

Matthew Burbridge
Senior Development Engineer
NIS-Bristol
Hewlett Packard
Telnet: 312 7010
E-mail: [email-masked]

---

From: "Binford, Charles" <[email-masked]>
Date: Fri, 12 Oct 2001 09:49:52 -0400

I have not verify version 8 is still the same, but version 07-97 had a rule
that unsolicited data MUST be sent in the same order as the commands.  

Thus, there is no need for a search on the ITT.  The target just needs to
keep of linked list of I/Os waiting on unsolicited data.  New commands are
added to the tail, any unsolicited data *should* be associated with the I/O
at the head of the list.  The ITT is used as a sanity check and you're done.

What am I missing?

Charles Binford
Pirus Networks
316.315.0382 x222

-----Original Message-----
From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
[mailto:[email-masked]]
Sent: Thursday, October 11, 2001 3:21 AM
To: 'Paul Koning'
Cc: [email-masked]
Subject: RE: Addition of CmdSN in Data-out PDU

Since I started this thread I feel I must at least contribute!

The reason why I proposed putting CmdSN (actually it should be RefCmdSN) in
the Data-out PDUs was to enable the target to have a faster search to
associate unsolicited data out PDUs with its SCSI Command PDU.  Solicited
Data-out PDUs do not require this as they have a Target Task Tag.

If all Command PDUs were queued then I believe this would work just fine.
However, as Santosh correctly pointed out they are not and without repeating
what he said this mechanism would not work for immediate command PDUs.

I am sure that particular implementations could make this work but the
underlying argument is that it needs to work and be useful to all
implementations.  The only benefit I now see of having CmdSN in the data PDU
is as a check as implementations must (and can only) use the initiator task
tag to associated the Data-out PDU with the command PDU.  Therefore, IMO it
is not a good enough reason for having CmdSN in the Data-out PDUs simply for
a consistency check.

The benefit of having the data sent unsolicited to minimise if not eradicate
round trip times far out weighs the overhead in having to perform a search
on receipt of unsolicited data. If we could have developed a well defined
mechanism to overcome this overhead then all well and good and that is what
I attempted.  Still, if someone can do this and the solution is simple and
straight forward then I am sure that it will have my backing but until then
...

Cheers

Matthew Burbridge

-----Original Message-----
From: Paul Koning [mailto:[email-masked]]
Sent: Wednesday, October 10, 2001 5:07 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: Addition of CmdSN in Data-out PDU

Excerpt of message (sent 10 October 2001) by Julian Satran:
> Inconsistency can be legitimate. CmdSN is ephemeral. It can be reused, it 
> may have large holes and using it in an implementation is as bad as a 
> hashed index.

Not true.

CmdSN values are sequential, by definition.  Yes, clearly there will
be small holes because commands complete out of order.  But "large"
holes are unlikely.

In any case, the target has control over that.  I can use an array
whose size is given by the number of pending commands times a
correction factor to account for the likely density of holes.  Then
MaxCmdSN would be updated based on two considerations: the ability to
handle more pending commands, and the need to keep the distance
between oldest (lowest) still active CmdSN and MaxCmdSN bounded by the
size of the lookup array.

So having CmdSN in the DataOut PDU allows this approach, thereby
replacing a hash lookup on a rapidly changing ID space by a simple
array indexing operation.  Without CmdSN, you're forced to use a
mechanism that has a lot more overhead (in the insert/remove or in the
lookup, depending on the mechanism chosen).

	paul

---

From: "Julian Satran" <[email-masked]>
Date: Sat, 13 Oct 2001 23:09:14 +0300

Robert,

The reason for the rule is limiting restrictions to what is logically 
mandated and leaving all the rest to the implementer.
The ordering rule is there to help avoid the trivial deadlock where 
commands start asking for solicited data and the windows is closed by 
unsolicited data  and having to resort to dropping data to reopen the TCP 
window.

Having data immediately follow the command is admisible but some 
implementer might choose to have and launch as many commands as possible 
to get the data transfer overlap with positioning operations.

The targets are supposed to consider out of order delivery of data a 
protocol error.

Regards,
Julo

John Hufferd/San Jose/IBM@IBMUS
Sent by: [email-masked]
13-10-01 02:01
Please respond to John Hufferd

        To:     Robert Snively <[email-masked]>
        cc:     "[email-masked]'" 
<[email-masked]>, "BURBRIDGE,MATTHEW 
(HP-UnitedKingdom,ex2)" <[email-masked]>, "'Binford, Charles'" 
<[email-masked]>, [email-masked]
        Subject:        RE: Addition of CmdSN in Data-out PDU

Robert,
I think that an Initiator being able to send a waiting Read command,
without having to wait for many large write segments -- that are being 
sent
(as unsolicited data) -- is very useful.  And that would mean, the
unsolicited data is waiting to be sent until the Read Commands are sent.
This might be a very frequent case.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: [email-masked]

Robert Snively <[email-masked]>@ece.cmu.edu on 10/12/2001 03:56:06 
PM

Sent by:  [email-masked]

To:   "[email-masked]'"
      <[email-masked]>, "BURBRIDGE,MATTHEW
      (HP-UnitedKingdom,ex2)" <[email-masked]>, "'Binford,
      Charles'" <[email-masked]>, [email-masked]
cc:
Subject:  RE: Addition of CmdSN in Data-out PDU

I have two small questions that would help me understand this.

How do you know that unsolicited data is expected?  By nature,
unsolicited data is received as a "maximum surprise" which can
only be determined after unpacking and parsing at least a
part of the command structure, possibly even including the
SCSI command (although there are some hints contained in the
command header information).

How can you constrain a generic system to posting the
unsolicited data in the same order that the commands were
emitted?  In general, I would have expected the system to
be emitting command followed immediately by the corresponding
unsolicited data.  If that is not the case, it means that there
was a delay in obtaining the unsolicited data for transfer and
that the delay was sufficient to allow the insertion of commands.
If the delay is that large (and probably variable), the enforcement
of transfer of unsolicited data in the same order as the
commands are emitted seems to me to be a significant challenge,
and certainly shouldn't be required as normal behavior.  While it
would make things simpler for targets (already challenged by
unsolicited data), it seems to me that it would make things
much more complex for initiators.

Bob

> From: Somesh Gupta [mailto:[email-masked]]
> Sent: Friday, October 12, 2001 2:51 PM
> To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Binford, Charles';
> [email-masked]
> Matthew,
> Since the unsolicited data does not follow the
> command, you need the link list. But since the
> unsolicited data must be sent in the same order
> as the commands, a link list is enough.
>
> Let us say that you have 8 commands. The ones
> for which we expect unsolicted data are marked
> as Cnn(ud). And I have marked the unsolicted
> data PDUs as UD(nn). The (nn) with data implies
> that it is implicit and not actually carried with
> the PDU itself.
>
> C01(ud) C02(ud) C03(ud) C04 C05 C06(ud) UD(01) --->
> --> C07(ud) UD(02) UD(03) D04 D04 D05 D05 UD(06) --->
> --> UD(07) C08(ud) UD(08) --->
>
> After the target receives the command C01, C02, and C03
> for which it expects unsolicited data, it puts them in
> a link list. It also receives C04 and C05 for which
> unsolicited data is not expected and they don't go
> on the list. It then receives C06 for which unsolicited
> data is expected, and it is added to then end of the list.
> Then an unsolicited data PDU is received. It must go
> with the command at the head of the list which is C01.
> Use the ITT to make sure and you can then take C01 off
> the list and so on.
>
> Somesh
>
>
> > From: [email-masked]
> [mailto:[email-masked]]On Behalf Of
> > BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
> > Sent: Friday, October 12, 2001 7:53 AM
> > To: 'Binford, Charles'; [email-masked]
> > Charles,
> > As you have described the spec states that "that
> unsolicited data MUST be
> > sent in the same order as the commands".  This is not the same as
> > unsolicited data must follow the command associated with
> it:  For example:
> >
> > (Cx = SCSI Command PDU, Dx = The unsolicited data PDUs. The
> x in all the
> > example can be the ITT. It is not the CmdSN.
> >
> > This is allowed:
> >
> >   C1 C2 C3 D1 D2 C4 D3 D4
> >
> > and the target will have to use the ITT to associate the
> data with the
> > command.
> >
> > Matthew
> >
> >
> > From: Binford, Charles [mailto:[email-masked]]
> > Sent: Friday, October 12, 2001 2:50 PM
> > To: [email-masked]

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 14 Oct 2001 05:38:36 +0300

Eddy,

That was good advise and free.

Julo

"Eddy Quicksall" <[email-masked]>
Sent by: [email-masked]
11-10-01 18:12
Please respond to "Eddy Quicksall"

        To:     "'Sandeep Joshi'" <[email-masked]>, "'BURBRIDGE,MATTHEW 
\(HP-UnitedKingdom,ex2\)'" <[email-masked]>
        cc:     "'Paul Koning'" <[email-masked]>, <[email-masked]>
        Subject:        RE: Addition of CmdSN in Data-out PDU

You can't assume the ITT takes a specific order.

Eddy

-----Original Message-----
From: Sandeep Joshi [mailto:[email-masked]]
Sent: Thursday, October 11, 2001 10:30 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
Cc: 'Paul Koning'; [email-masked]
Subject: Re: Addition of CmdSN in Data-out PDU

If we use the fact that the ITT-ordering of the unsolicited
Data-out PDUs is identical to the ITT-ordering of the commands
sent *within* a connection, then it should be possible to 
resolve the Data-out PDUs in a constant-time operation[*1]
No hash and no cmdSN is required keep a per-connection
chain within the command queue and look at its head.

[*1]There are a couple of exceptions, due to the leeway the
    standard provides the initiator on Data-out PDUs.

-Sandeep

---

From: Robert Snively <[email-masked]>
Date: Mon, 15 Oct 2001 13:59:32 -0700

Can they send enough commands to close the window just
as fatally?

> From: Julian Satran [mailto:[email-masked]]
> Sent: Saturday, October 13, 2001 1:09 PM
> To: John Hufferd
> Cc: 'Binford, Charles'; [email-masked]; BURBRIDGE,MATTHEW
> (HP-UnitedKingdom,ex2); [email-masked]; Robert Snively;
> [email-masked]'

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 21 Oct 2001 05:55:47 +0300

Robert,

Closing the window is a common event on TCP connection.
The deadlock can occur only if a command sends its data while the target 
requests data for an earlier command.

Julo

Robert Snively <[email-masked]>
Sent by: [email-masked]
15-10-01 22:59
Please respond to Robert Snively

        To:     Julian Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS
        cc:     "'Binford, Charles'" <[email-masked]>, [email-masked], 
"BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>, 
[email-masked], Robert Snively <[email-masked]>, 
"[email-masked]'" 
<[email-masked]>
        Subject:        RE: Addition of CmdSN in Data-out PDU