Subject: ISCSI: Immediate data extending beyond a single PDU

From: "Barry Reinhold - Lamprey" <[email-masked]>
Date: Thu, 1 Feb 2001 15:04:28 -0500

Julian,

My understanding of your email in conjunction with the current draft is:

The initiator may send zero or more immediate data bytes in the CMD PDU, and
may send additional unsolicited DATA PDUs with further immediate data, with
the
last PDU having the F bit set. The total amount of immediate data to be
limited
by the value negotiated at login.

There are two issues I have with this:

First, the target does not know how much immediate data has being sent.
There
is no indication in the CMD frame to indicate that more immediate data is to
follow (whether or not the CMD frame contains any immediate data). The
initiator is free to send any amount, from zero up to the negotiated value.
Therefore, the target will likely send an R2T to the host at whatever
boundry
it believes is present - which may cause a retransmission of much or all of
the
immediate data.

Second, the additional unsolicited PDU's do not contain a target
task/transfer
tag. As such, in the mainline processing, you are perhaps forcing the target
to
look up the Initiator Task Tag in its list of active i/o's. This may exact a
heavy performance penalty as optimizing this lookup maybe cost-prohibitive.
The
only other scenarios in which the target had to perform this lookup was in
low-occurance error/abort conditions.

Are there negative implications to keeping immediate data to a single PDU?

-- Barry

[email-masked] wrote:

> Barry,
>
> Immediate data is "attached to the command". If your unsolicited data (the
> so called initial burst) is larger that what you are ready to commit to a
> single PDU you can send several PDUs with the  last having the F bit.
After
> that the target can decide that it wants more and send an R2T or send
> status.
>
> You are no supposed to send both immediate and separate PDU.
>
> I admit that the text is bit confusing (in 1.2.5)  and there is an error
> also in the appendix.
> 2.7.1 talk about unsolicited data (not immediate) (the context is data
> PDUs).
>
> The new version of 1.2.5 now reads:
>
>    Unsolicited data can be sent as part of an iSCSI command PDU
("immediate
>    data") or an in separate iSCSI data PDUs.  An initiator may send
>    unsolicited data either as immediate (up to the negotiated maximum PDU
>    size - DataPDULength - disconnect-reconnect mode page) or in a separate
>    PDU sequence (up to the negotiated limit - FirstBurstSize -
>    disconnect-reconnect mode page). All subsequent data have to be
>    solicited.  The maximum size of an individual data PDU or the
>    immediate-part of the initial unsolicited burst as well as the initial
>    burst size MAY be negotiated at login.
>
> Thanks,
> Julo
>
> "Barry Reinhold - Lamprey" <[email-masked]> on 29/01/2001
16:34:30
>
> Please respond to "Barry Reinhold - Lamprey" <[email-masked]>
>
> To:   Julian Satran/Haifa/IBM@IBMIL
> cc:   "Jon Sreekanth" <[email-masked]>, "James Smart"
>       <[email-masked]>
> Subject:
>
> Issue:
>
> I am unclear on how to respond to an iSCSI SCSI command PDU when there is
> immediate data but not enough to meet the requirements of the IO operation
> as given in the expected data transfer length field. Should an R2T be
> issued
> or not?
>
> Background:
>
> Clause 2.7.1 in 03 reads:
>
> "2.7.1 F (Final) bit
>
>    This bit is 1 for the last PDU of immediate data or the last PDU of a
>    sequence answering a R2T."
>
> This suggests that immediate data can extend beyond the iSCSI SCSI CMD PDU
> (there is no final bit on the command PDU)but the closet thing we have to
a
> definition for immediate data, which is in clause 1.2.5 below, suggests
> that
> immediate data is limitted to the command PDU:
>
> "Outgoing SCSI data (initiator to target - user data or command
>    parameters) will be sent as either solicited data or unsolicited
>    data.  Solicited data are sent in response to Ready To Transfer (R2T)
>    PDUs. Unsolicited data can be part of an iSCSI command PDU
>    ("immediate data") or an iSCSI data PDU.  An initiator may send
>    unsolicited data (immediate or in a separate PDU) up to the
>    negotiated limit (initial burst size - mode page 02h). All subsequent
>    data have to be solicited.  The maximum size of an individual data
>    PDU or the immediate-part of the initial unsolicited burst MAY be
>    negotiated at login (maximum connect size - mode page 02h)."
>
> Unsolicited data is bounded in a number of ways, but most significant to
> this issue is in the description of the login key useR2t which states:
>
> "Note than only the first
>    outgoing data item (either immediate data or a separate PDU) can be
>    sent unsolicited by a R2T."
>
> Given the above it is unclear to me if the concept of immediate data is:
>
> 1. Write data that can be sent without an R2T (unsolicited write data)and
> starts in the command PDU.
> 2. Write data that is entirely contained within the command PDU. (my
> initial
> concept of immediate data)
> 3. The non-zero portion of data, in a possible multi-PDU write operation,
> that is contained in the iSCSI SCSI command PDU. Where the remaining PDUs
> of
> the write operation must be solicited with an R2T.
>
> Given the current definition of the Final bit and the current limits on
> unsolicited data it is unclear to me what a target should do when
receiving
> an iSCSI SCSI command PDU that has immediate data in it, but not
sufficient
> data to meet the expected data transfer length specificied in the command.
> Does an R2T go out of not?
>
> Barry Reinhold
> 603-659-0885

Barry Reinhold
603-659-0885

---

From: [email-masked]
Date: Fri, 2 Feb 2001 13:41:10 +0200

Barry,

There is a misunderstanding:

You can send EITHER immediate data OR a sequence of unsolicited PDUs.

Immediate data was conceived for short writes - for which an initiator does
not want to do
two socket operations or 2 HBA operations.

Larger unsolicited bursts can be sent with a sequence and then we felt that
you won't gain too much by having the first one glued to the command.

Julo

"Barry Reinhold - Lamprey" <[email-masked]> on 01/02/2001 22:04:28

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   "ISCSI" <[email-masked]>
Subject:  ISCSI: Immediate data extending beyond a single PDU

Julian,

The initiator may send zero or more immediate data bytes in the CMD PDU,
and
may send additional unsolicited DATA PDUs with further immediate data, with
the
last PDU having the F bit set. The total amount of immediate data to be
limited
by the value negotiated at login.

is no indication in the CMD frame to indicate that more immediate data is

Second, the additional unsolicited PDU's do not contain a target
task/transfer
tag. As such, in the mainline processing, you are perhaps forcing the
target
to
look up the Initiator Task Tag in its list of active i/o's. This may exact
a
heavy performance penalty as optimizing this lookup maybe cost-prohibitive.
The
only other scenarios in which the target had to perform this lookup was in
low-occurance error/abort conditions.

-- Barry

After
("immediate
16:34:30
a
receiving
sufficient

Barry Reinhold
603-659-0885

---

From: [email-masked]
Date: Mon, 5 Feb 2001 18:50:17 +0200

Barry,

You are suggesting support of immediate data as separate from the support
for unsolicited data as a target option?

That could be accomodated by stating that support for immediate data is
not implied by R2T NO.

If there are no strong objections on the list I am ready to remove/correct
the offending statement in the appendix.

However we will maintain only one initial burst per command (either
immediate or in several separate PDUs).

Regards,
Julo

"Barry Reinhold" <[email-masked]> on 05/02/2001 17:11:36

Please respond to "Barry Reinhold" <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
cc:
Subject:  RE: ISCSI: Immediate data extending beyond a single PDU

Julian,
     I  think one could carry on a debate about what the draft currently
states
about the coupling of immediate and unsolicited data - but it is probably
more productive to figure out if we want to have both the concept of
immediate data and unsolicited data and then worry about the wording in the
draft.
     I am of the opinion that immediate data (data within a single PDU) is
a
useful distincition from unsolicited data (iSCSI DATA PDU sent without
R2T).
The reasoning behind this is that there are a number of write IOs that can
be satisfied by providing a buffer of 8K or less. If this data is contained
in the command PDU the context can be created for the command, and then the
data just happens to be sitting there with the context to process it. Thus
the target can send the iSCSI response PDU without creating any special
context or mechansims - should have nice low latency characteristics for
small writes.
     For devices that don't want to use the initiator task tag for
establishing
context of an iSCSI DATA PDU, immediate data is a nice thing. An iSCSI DATA
PDU without a target task tag may not be such a nice thing.
     Question here - What would you lose if you supported only immediate
data?
It seems like this meets some of the common application requirements that I
am aware of and is friendly to what I think is the more common approach to
target processing of DATA PDUs.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>[email-masked]
>Sent: Sunday, February 04, 2001 5:20 AM
>To: [email-masked]
>Subject: RE: ISCSI: Immediate data extending beyond a single PDU
>
>
>
>
>Barry,
>
>I think that the current draft does not allow you to accept immediate but
>not unsolicited.
>You can either allow unsolicited (including immediate) or request always
>R2T.
>
>And yes - if the expected length is larger that what you sent as
>unsolicited (even if the unsolicited was less than the limit) the target
is
>supposed to send R2T (there is only one unsolicited burst per command).
>
>The limit for immediate data is implicit - the max-PDU length and is
>less-then-or-equal the first burst.
>
>Your proposal is interesting but I doubt that it is very useful.
>
>Unsolicited data help avoid an RTT on writes but require resources to be
>allocated (that is why a target may not accept them).
>
>Immediate data is an "optimized" form of unsolicited data in which the
>command and data are collapsed to one unit. Nevertheless the immediate
data
>are unsolicited and require resources to be allocated ahead of time. If
you
>have larger amounts to send than what fits into one PDU the "collapse
gain"
>is minimal and does not justify the added complexity of 2 options.
>
>Regards,
>Julo
>
>
>
>"Barry Reinhold" <[email-masked]> on 02/02/2001 21:06:29
>
>Please respond to "Barry Reinhold" <[email-masked]>
>
>cc:
>Subject:  RE: ISCSI: Immediate data extending beyond a single PDU
>
>
>
>
>Julian,
>     I agree with the conceptual idea of immediate data, but further
>refinement
>is required. As a receiver, if I have allowed immediate data, but not
>unsolicited data, and I get a write command with an expected data transfer
>length that is greater then the amount of data in the command PDU, do I
>send
>an R2T or not (or from the sender's point of view does he wait for an R2T
>to
>complete the data transfer?) The draft is not clear on this, yet it is an
>externally visible behavior that needs to be defined for interoperability.
>
>I believe that what we need here is a refinement of two distinct concepts,
>that of immediate data and that of unsolicited data. Here is my take at
it:
>
>1. Immediate data - Data that is sent within a single iSCSI SCSI command
>PDU. Immediate data shall be fully contained within a single iSCSI command
>PDU, and shall be limitted by the negotiated value of ImmediateData. The
>size of Immediate data is not subject to the limits of FirstBurst.
>
>2. Unsolicited data - Data that may be transfered to a target without an
>explicit R2T being sent to request its transfer. Unsolicited data is sent
>in
>iSCSI SCSI data PDUs in which the target transfer tag is invalid. The
>maximum amount of unsolicited data is negotiated between the initiator and
>target through the FirstBurst key. If unsolicited data is supported by the
>target, upon receipt of a write command, the target shall not send an R2T
>until an iSCSI Data PDU with the Final bit set has been received. After
>reception of an iSCSI Data PDU with the Final bit set R2Ts shall be sent
by
>the target to request transmission of any additional data associated with
>this command. The initiator shall not send any additional unsolicited data
>for this command.
>
>Interaction between Immediate data and Unsolicited data -
>
>Immediate data and Unsolicited data are features that may be selected
>independently of one another. Support of either is optional.
>
>Note: Based on the current specification of FirstBurst, support for
>unsolicited data is not optional. One must select between a range of
>1-65536. I think we would want to make this optional and indicate non
>support for unsolicited data by setting FirstBurst to 0.
>
>
>>-----Original Message-----
>>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>>[email-masked]
>>Sent: Friday, February 02, 2001 6:41 AM
>>To: [email-masked]
>>Subject: Re: ISCSI: Immediate data extending beyond a single PDU
>>
>>
>>
>>
>>Barry,
>>

---

From: [email-masked]
Date: Thu, 8 Feb 2001 08:30:30 +0200

Pierre,

I am not sure I see where the problem is.

Both today and with Barry's suggested addition you can send EITHER
immediate or a regular unsolicited burst. In case you choose to do the
later the end is marked by the F bit. In case you choose immediate it is
only 1 PDU.  Expected Data Length is for the whole operation (not only the
first burst).

Barry suggested that we restrict also, at target choice, the type of
unsolicited data to immediate
(we either let UseR2T:yes and ImmediateData:yes mean only immediate data or
better UseR2T:no ImmediateData:ONLY) for the convenience of some targets.

In all the cases the target knows when the FirstBurst is over (now and with
Barry's proposed change).

Am I missing something?

Regards,
Julo

Pierre Labat <[email-masked]> on 07/02/2001 20:56:57

Please respond to Pierre Labat <[email-masked]>

To:   [email-masked]
cc:

is
> supposed to send R2T (there is only one unsolicited burst per command).

Julian, Barry,

I think there is a problem, as stated Barry, if the initiator decides to
send unsolicited data but less than the Firstburst and
the expected length is greater than the Firstburst. In this case the
initiator

after sending some unsolicited data waits for an R2T to continue
and the target waits for a First burst load of data before sending
an R2T.

Solution
======
To avoid that, it should be explicitely stated (may be it's what
is assumed but i can't read it) that when the initiator sends
unsolicited non immediate data, it must send unsolicited data
up to Firstburst or Expected data length whichever is less.

Regards,

Pierre

<stuff deleted>

---

From: [email-masked]
Date: Sat, 10 Feb 2001 00:12:02 +0200

Pierre,

Barry did not ask for unsolicited data to be eliminated - he rather asked
for immediate to be enabled independent of enabling unsolicited (i.e., you
can have either both - not in the same command obviously - or only
immediate). Do you object to this too?

Regards,
Julo

Pierre Labat <[email-masked]> on 09/02/2001 18:56:05

To:   [email-masked]
cc:

Robert Snively wrote:

> I do not see any reason to create the constraint that Pierre
> suggests.  The goal is to negotiate a maximum acceptable buffer
> space always available for unsolicited data up to the length Firstburst.
> If the initiator chooses not to send that much data in the first
> transfer for reasons of its own, it should be allowed to.  The
> performance penalty for the affected transfer is small and does not
> affect over-all throughput.
>
> There should be no architectural limitation demanding that the first
> burst of a transfer greater than Firstburst in length is always equal
> to Firstburst.  In particular, if it isn't, what are you going to do
about
> it, post an error?
> Bob

Julian, Robert,

I missed the "F" bit in the data PDU. With that, the target
knows when to send the R2T if the unsolicited data
is less than the FirstBurst. Hence, as Julian said
there is no problem for that.
However on the other point, i am against the removal
of the possibility to send unsolicited DATA PDU
(and use only immediate data for unsolicited) because
in the case where small iSCSI PDUs are used, it
may not be possible to send all the unsolicited data
in the command PDU.

Regards,

Pierre