Subject: RE: Unsolicited Data.

From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>
Date: Thu, 7 Jun 2001 08:39:59 +0100

Hi Stephen, Sanjeev, and other interested parties,

I whole heartily agree with both of your comments and see the benefit of
unsolicited data in removing the R2T induced latency for both small writes
and large writes on systems where such latency is an issue.  However, for
large writes the unsolicited portion of the data need not be all the data in
the transfer but enough to overcome the initial latency.

My main concern is that although the initiator and target have negoiated to
use unsolicited data (InitialR2t=no), in the spec it appears to be optional.
Therefore how does the target know that unsolicited data will be sent.
Currently it does not and the target will have to set a timer that expires
if no unsolicited data is forthcoming which will incur an even more
unacceptable latency.

I see that there are two options:
1)State in the spec that if unsolicited data has been negotiated then the
initiator MUST honour it. (My preferred)
2)Have a flag in the SCSI Command PDU indicating that data is expected (No
thanks). 

Thanks

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010
E-mail: [email-masked]

-----Original Message-----
From: Stephen Bailey [mailto:[email-masked]]
Sent: Thursday, June 07, 2001 2:20 AM
To: [email-masked]
Subject: Re: Unsolicited Data. 

Sanjeev,

> why would initiator choose not to send unsolicited immediate data when
> it has negotiated for IntialR2T= Yes??

I assume you mean InitialR2T=no.

I would do this because I think that the real benefit of unsolicited
data is for writes where ALL the write data can be sent unsolicited,
and I'm happy to have the target schedule the transfer of (ALL) data
for larger writes.  Of course, others would say I'm a moron.

Assuming the overwhelming majority of storage use is file access,
small writes are exclusively file system metadata writes.  Unsolicited
data is a huge win for metadata writes because the latency of these
metadata writes is frequently not covered by transferring other data
(within a single access thread).  On the other hand, the file system
(usually) generates enough concurrent demand for real data writes that
the flow control (R2T) latency is well hidden.

You might argue that if you're running a file system optimized fors
today's SANs over a link with a HUGE bandwidth * delay product
(greater than the expected demand created by the file system), sending
large amounts of unsolicited write data will substantially improve the
link utilization.  That is true, but I don't think many targets are
going to offer unsolicited data bursts remotely that large.

In a hardware accelerate implementation, unsolicited data will be
handled differently than solicited data.  The target can chose where
the solicited data is delivered, but unsolicited data arrives through
a general-delivery path.  Therefore, if the offered unsolicited data
burst size approaches the regular burst size, an implementation will
need TWICE as much buffer memory (half in the general-delivery area
and half in the flow-controlled buffer area) to support the same
operation load.  Maybe this tradeoff is OK, but given the cost
sensitivity of storage targets, I doubt it.

As always, I welcome comments from those who see things differently.

Steph

---

From: Stephen Bailey <[email-masked]>
Date: Sat, 09 Jun 2001 23:56:44 -0400

Matthew,

> However, for large writes the unsolicited portion of the data need
> not be all the data in the transfer but enough to overcome the
> initial latency.

I was probably unclear (well, OK, completely incoherent) in my
previous message.  Lemmie try it again.

My point about large writes is that there are usually many of them
outstanding concurrently.  That's what file systems live for.  The
initial latency of the next write is covered by the terminal data
transfer of the current write.

As long as there's bandwidth * delay worth of total outstanding
writes, you don't need unsolicited data.  If you have an obscene
bandwidth * delay, you might realistically expect that current file
systems WON'T give you that much outstanding demand, so saving the
startup latency with unsolicited data will improve performance
somewhat.  But not much.  I'm claiming that's not a case worth
optimizing.

Steph