Subject: RE: Unsolicited Data.

From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" <[email-masked]>
Date: Thu, 7 Jun 2001 14:08:47 +0200

Matthew,

among the two options you gave I would like to comment on the second

It is possible for initiator **not** to send any unsolicited data and send
the rest of data as solicited data by the use of bit 7 in Flags and Task
Attributes. This is posted here http://ips.pdl.cs.cmu.edu/mail/msg04647.html

The new 2.3.1 is:

2.3.1     Flags and Task Attributes

      The flags for a SCSI Command are:

      b7   (F) set to 1 when the immediate data that accompany the command
      are all the unsolicited data associated with this command. For a
      write, if Expected Data Transfer Length is larger than the Length the
      target may solicit additional data through R2T.

About the option no. 1 Stephen has already explained. Refer to the very
specific paragraph below of his email

In a hardware accelerate implementation, unsolicited data will be
handled differently than solicited data.  The target can chose where
the solicited data is delivered, but unsolicited data arrives through
a general-delivery path.  Therefore, if the offered unsolicited data
burst size approaches the regular burst size, an implementation will
need TWICE as much buffer memory (half in the general-delivery area
and half in the flow-controlled buffer area) to support the same
operation load.  Maybe this tradeoff is OK, but given the cost
sensitivity of storage targets, I doubt it.

I would like to comment one thing here that in the current iSCSI specs
implementation the complete data for a specific command Including (Cmd PDUS
, Data PDUs and Response PDUs) follow the same path. So the whole of data
will follow the general delivery path (same connection) whether solicited or
unsolicited. Is it still going to help if Unsolicited data is not chosen to
be sent while IntialR2t=No has been negotiated???? Or do I have
misunderstanding with a general delivery path here.

Regards,
Sanjeev

-----Original Message-----
From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
[mailto:[email-masked]]
Sent: Thursday, June 07, 2001 12:40 AM
To: [email-masked]
Subject: RE: Unsolicited Data. 

Hi Stephen, Sanjeev, and other interested parties,

I whole heartily agree with both of your comments and see the benefit of
unsolicited data in removing the R2T induced latency for both small writes
and large writes on systems where such latency is an issue.  However, for
large writes the unsolicited portion of the data need not be all the data in
the transfer but enough to overcome the initial latency.

My main concern is that although the initiator and target have negoiated to
use unsolicited data (InitialR2t=no), in the spec it appears to be optional.
Therefore how does the target know that unsolicited data will be sent.
Currently it does not and the target will have to set a timer that expires
if no unsolicited data is forthcoming which will incur an even more
unacceptable latency.

I see that there are two options:
1)State in the spec that if unsolicited data has been negotiated then the
initiator MUST honour it. (My preferred)
2)Have a flag in the SCSI Command PDU indicating that data is expected (No
thanks). 

Thanks

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010
E-mail: [email-masked]

-----Original Message-----
From: Stephen Bailey [mailto:[email-masked]]
Sent: Thursday, June 07, 2001 2:20 AM
To: [email-masked]
Subject: Re: Unsolicited Data. 

Sanjeev,

> why would initiator choose not to send unsolicited immediate data when
> it has negotiated for IntialR2T= Yes??

I assume you mean InitialR2T=no.

I would do this because I think that the real benefit of unsolicited
data is for writes where ALL the write data can be sent unsolicited,
and I'm happy to have the target schedule the transfer of (ALL) data
for larger writes.  Of course, others would say I'm a moron.

Assuming the overwhelming majority of storage use is file access,
small writes are exclusively file system metadata writes.  Unsolicited
data is a huge win for metadata writes because the latency of these
metadata writes is frequently not covered by transferring other data
(within a single access thread).  On the other hand, the file system
(usually) generates enough concurrent demand for real data writes that
the flow control (R2T) latency is well hidden.

You might argue that if you're running a file system optimized fors
today's SANs over a link with a HUGE bandwidth * delay product
(greater than the expected demand created by the file system), sending
large amounts of unsolicited write data will substantially improve the
link utilization.  That is true, but I don't think many targets are
going to offer unsolicited data bursts remotely that large.

In a hardware accelerate implementation, unsolicited data will be
handled differently than solicited data.  The target can chose where
the solicited data is delivered, but unsolicited data arrives through
a general-delivery path.  Therefore, if the offered unsolicited data
burst size approaches the regular burst size, an implementation will
need TWICE as much buffer memory (half in the general-delivery area
and half in the flow-controlled buffer area) to support the same
operation load.  Maybe this tradeoff is OK, but given the cost
sensitivity of storage targets, I doubt it.

As always, I welcome comments from those who see things differently.

Steph

---

From: [email-masked]
Date: Sun, 10 Jun 2001 14:31:49 +0300

I've published the new text to this list a while ago. Please look-up the
archives. Julo

"BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]> on
06-06-2001 17:35:38

Please respond to "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)"
      <[email-masked]>

To:   "[email-masked]'" <[email-masked]>
cc:
Subject:  RE: Unsolicited Data.

The current spec states that the F bit is "set to 1 when the immediate data
that accompany the command are all the data associated with this command.
It
is an error if the Length and Expected Length do not match and this bit is
set to 1".

I interpret this as there is no more data to follow and not that the
initiator has opted not to use the unsolicited data feature.  Therefore the
spec needs to be modified to indicate that if unsolicited data has been
negotiated (i.e. InitialR2T=no), then the initiator MUST send unsolicited
data of length = min( FirstBurstSize, ExpectedTransferLength ) minus any
immediate data sent).

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010

-----Original Message-----
From: Ayman Ghanem [mailto:[email-masked]]
Sent: Tuesday, June 05, 2001 6:31 PM
To: [email-masked]
Subject: RE: Unsolicited Data.

If the initiator decides not to send immediate or unsolicited data when it
has negotiated to do so, then the initiator must set the F-bit in the
command PDU. This prompts the target to send R2T.

I still agree that the spec should indicate that the initiator MUST use the
resources it has negotiated. If it has negotiated the option to send
immediate data or unsolicited data then it should do that to the limits
allowed. If it has negotiated a PDU length, it must not send data PDUs less
than the negotiated limit except for last one. While most implementation
may
do that for performance reasons, I would prefer defining this in the spec.

-Ayman

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
> Sent: Tuesday, June 05, 2001 11:58 AM
> To: [email-masked]'
> Subject: Unsolicited Data.
>
>
> I'm not sure if this has been discussed before but it is causing some
> confusion.  The statement below implies that if immediate data has been
> negotiated then the initiator MAY use it.
>
> "If ImmediateData is set to no and InitialR2T is set to no then the
> initiator MUST NOT send unsolicited immediate data but MAY send one
> unsolicited burst of Data-OUT PDUs."
>
> Therefore the target must wait for the initial burst of unsolicited data
> before issuing the first R2T (if there is subsequent data).  If the
> initiator decides not to send it then the target must timeout and
> issue the
> R2T for the initial data.  Can the spec be changed to state that if
> unsolicited data has been negotiated, then the initiator MUST use it.
>
> Thanks
>
> Matthew Burbridge
> NIS-Bristol
> Hewlett Packard
> Telnet: 312 7010

---

From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>
Date: Mon, 11 Jun 2001 17:56:38 +0100

Hi Julian

Sorry about harking on with this one but I still don't feel that it has been
resolved.

By the new text are you referring to
http://ips.pdl.cs.cmu.edu/mail/msg04647.html?

If so it does not take into account the issue that I am trying to resolve:
If an initiator and target have negotiated to use unsolicited data
PDUs(InitialR2t=no), how does an target know that an initiator is going to
sent unsolicited SCSI Data PDUs when it is optional to send unsolicited data
PDUs?

Mallikarjun has suggested a solution which will work providing that
immediate data and unsolicited Data PDUs can not be used together (which is
what Rev 06 states). Am I correct it believe this is still the case - I seem
to remember a thread suggesting otherwise.

The other option, which is less flexible is that if both sides have
negotiated to use unsolicited Data PDUs (InitialR2t=no), then it MUST always
send unsolicited data which needs to be reflected in the specification.

Cheers

Matthew

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Sunday, June 10, 2001 12:32 PM
To: [email-masked]
Subject: RE: Unsolicited Data.

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010

-----Original Message-----
From: Ayman Ghanem [mailto:[email-masked]]
Sent: Tuesday, June 05, 2001 6:31 PM
To: [email-masked]
Subject: RE: Unsolicited Data.

-Ayman

---

From: [email-masked]
Date: Sun, 17 Jun 2001 09:47:58 +0300

The F-flag is used to signal the end-of-unsolicited (+- somew unclarity
removed in 07).

Julo

07-06-2001 10:39:59

To:   [email-masked]
cc:

large writes the unsolicited portion of the data need not be all the data
in

use unsolicited data (InitialR2t=no), in the spec it appears to be
optional.

Thanks

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010

-----Original Message-----
From: Stephen Bailey [mailto:[email-masked]]
Sent: Thursday, June 07, 2001 2:20 AM
To: [email-masked]
Subject: Re: Unsolicited Data.

Sanjeev,

Steph

---

From: [email-masked]
Date: Sun, 17 Jun 2001 08:21:27 +0300

It may vary from command to command. A simple cdesign may decide to send
immediate data only when it will contain all it has to send and send in
separate PDUs if it has more (solicited or unsolicited).
I saw no real reason to enforce a behavior or other.

Julo

"Sanjeev Bhagat \(TRIPACE/Zoetermeer\)" <[email-masked]> on 07-06-2001
02:05:39

Please respond to "Sanjeev Bhagat \(TRIPACE/Zoetermeer\)"
      <[email-masked]>

To:   [email-masked]
cc:

Hello,

All that is said below is ok but I cannot understand one thing here
----- why would initiator choose not to send unsolicited immediate data
when
it has negotiated for IntialR2T= Yes??

Sanjeev

----- Original Message -----
From: "Ayman Ghanem" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, June 06, 2001 6:48 PM
Subject: RE: Unsolicited Data.

> That section was modified. Julian posted the revised text here:
>
> http://ips.pdl.cs.cmu.edu/mail/msg04647.html
>
> -Ayman
>
>
> > Sent: Wednesday, June 06, 2001 9:36 AM
> > To: [email-masked]'
> > that accompany the command are all the data associated with this
> > command. It
> > than the negotiated limit except for last one. While most
> > implementation may