Subject: RE: ISCSI: Immediate data extending beyond a single PDU

From: [email-masked]
Date: Sun, 4 Feb 2001 12:19:57 +0200

Barry,

I think that the current draft does not allow you to accept immediate but
not unsolicited.
You can either allow unsolicited (including immediate) or request always
R2T.

And yes - if the expected length is larger that what you sent as
unsolicited (even if the unsolicited was less than the limit) the target is
supposed to send R2T (there is only one unsolicited burst per command).

The limit for immediate data is implicit - the max-PDU length and is
less-then-or-equal the first burst.

Your proposal is interesting but I doubt that it is very useful.

Unsolicited data help avoid an RTT on writes but require resources to be
allocated (that is why a target may not accept them).

Immediate data is an "optimized" form of unsolicited data in which the
command and data are collapsed to one unit. Nevertheless the immediate data
are unsolicited and require resources to be allocated ahead of time. If you
have larger amounts to send than what fits into one PDU the "collapse gain"
is minimal and does not justify the added complexity of 2 options.

Regards,
Julo

"Barry Reinhold" <[email-masked]> on 02/02/2001 21:06:29

Please respond to "Barry Reinhold" <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:
Subject:  RE: ISCSI: Immediate data extending beyond a single PDU

Julian,
     I agree with the conceptual idea of immediate data, but further
refinement
is required. As a receiver, if I have allowed immediate data, but not
unsolicited data, and I get a write command with an expected data transfer
length that is greater then the amount of data in the command PDU, do I
send
an R2T or not (or from the sender's point of view does he wait for an R2T
to
complete the data transfer?) The draft is not clear on this, yet it is an
externally visible behavior that needs to be defined for interoperability.

I believe that what we need here is a refinement of two distinct concepts,
that of immediate data and that of unsolicited data. Here is my take at it:

1. Immediate data - Data that is sent within a single iSCSI SCSI command
PDU. Immediate data shall be fully contained within a single iSCSI command
PDU, and shall be limitted by the negotiated value of ImmediateData. The
size of Immediate data is not subject to the limits of FirstBurst.

2. Unsolicited data - Data that may be transfered to a target without an
explicit R2T being sent to request its transfer. Unsolicited data is sent
in
iSCSI SCSI data PDUs in which the target transfer tag is invalid. The
maximum amount of unsolicited data is negotiated between the initiator and
target through the FirstBurst key. If unsolicited data is supported by the
target, upon receipt of a write command, the target shall not send an R2T
until an iSCSI Data PDU with the Final bit set has been received. After
reception of an iSCSI Data PDU with the Final bit set R2Ts shall be sent by
the target to request transmission of any additional data associated with
this command. The initiator shall not send any additional unsolicited data
for this command.

Interaction between Immediate data and Unsolicited data -

Immediate data and Unsolicited data are features that may be selected
independently of one another. Support of either is optional.

Note: Based on the current specification of FirstBurst, support for
unsolicited data is not optional. One must select between a range of
1-65536. I think we would want to make this optional and indicate non
support for unsolicited data by setting FirstBurst to 0.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>[email-masked]
>Sent: Friday, February 02, 2001 6:41 AM
>To: [email-masked]
>Subject: Re: ISCSI: Immediate data extending beyond a single PDU
>
>
>
>
>Barry,
>
>There is a misunderstanding:
>
>You can send EITHER immediate data OR a sequence of unsolicited PDUs.
>
>Immediate data was conceived for short writes - for which an initiator
does
>not want to do
>two socket operations or 2 HBA operations.
>
>Larger unsolicited bursts can be sent with a sequence and then we felt
that
>you won't gain too much by having the first one glued to the command.
>
>
>Julo
>
>"Barry Reinhold - Lamprey" <[email-masked]> on 01/02/2001
22:04:28
>
>Please respond to "Barry Reinhold - Lamprey" <[email-masked]>
>
>To:   Julian Satran/Haifa/IBM@IBMIL
>cc:   "ISCSI" <[email-masked]>
>Subject:  ISCSI: Immediate data extending beyond a single PDU
>
>
>
>
>Julian,
>
>My understanding of your email in conjunction with the current draft is:
>
>The initiator may send zero or more immediate data bytes in the CMD PDU,
>and
>may send additional unsolicited DATA PDUs with further immediate data,
with
>the
>last PDU having the F bit set. The total amount of immediate data to be
>limited
>by the value negotiated at login.
>
>There are two issues I have with this:
>
>First, the target does not know how much immediate data has being sent.
>There
>is no indication in the CMD frame to indicate that more immediate data is
>to
>follow (whether or not the CMD frame contains any immediate data). The
>initiator is free to send any amount, from zero up to the negotiated
value.
>Therefore, the target will likely send an R2T to the host at whatever
>boundry
>it believes is present - which may cause a retransmission of much or all
of
>the
>immediate data.
>
>Second, the additional unsolicited PDU's do not contain a target
>task/transfer
>tag. As such, in the mainline processing, you are perhaps forcing the
>target
>to
>look up the Initiator Task Tag in its list of active i/o's. This may exact
>a
>heavy performance penalty as optimizing this lookup maybe
cost-prohibitive.
>The
>only other scenarios in which the target had to perform this lookup was in
>low-occurance error/abort conditions.
>
>Are there negative implications to keeping immediate data to a single PDU?
>
>-- Barry
>
>
>[email-masked] wrote:
>
>> Barry,
>>
>> Immediate data is "attached to the command". If your unsolicited data
>(the
>> so called initial burst) is larger that what you are ready to commit to
a
>> single PDU you can send several PDUs with the  last having the F bit.
>After
>> that the target can decide that it wants more and send an R2T or send
>> status.
>>
>> You are no supposed to send both immediate and separate PDU.
>>
>> I admit that the text is bit confusing (in 1.2.5)  and there is an error
>> also in the appendix.
>> 2.7.1 talk about unsolicited data (not immediate) (the context is data
>> PDUs).
>>
>> The new version of 1.2.5 now reads:
>>
>>    Unsolicited data can be sent as part of an iSCSI command PDU
>("immediate
>>    data") or an in separate iSCSI data PDUs.  An initiator may send
>>    unsolicited data either as immediate (up to the negotiated maximum
PDU
>>    size - DataPDULength - disconnect-reconnect mode page) or in a
>separate
>>    PDU sequence (up to the negotiated limit - FirstBurstSize -
>>    disconnect-reconnect mode page). All subsequent data have to be
>>    solicited.  The maximum size of an individual data PDU or the
>>    immediate-part of the initial unsolicited burst as well as the
initial
>>    burst size MAY be negotiated at login.
>>
>> Thanks,
>> Julo
>>
>> "Barry Reinhold - Lamprey" <[email-masked]> on 29/01/2001
>16:34:30
>>
>> Please respond to "Barry Reinhold - Lamprey" <[email-masked]>
>>
>> To:   Julian Satran/Haifa/IBM@IBMIL
>> cc:   "Jon Sreekanth" <[email-masked]>, "James Smart"
>>       <[email-masked]>
>> Subject:
>>
>> Issue:
>>
>> I am unclear on how to respond to an iSCSI SCSI command PDU when there
is
>> immediate data but not enough to meet the requirements of the IO
>operation
>> as given in the expected data transfer length field. Should an R2T be
>> issued
>> or not?
>>
>> Background:
>>
>> Clause 2.7.1 in 03 reads:
>>
>> "2.7.1 F (Final) bit
>>
>>    This bit is 1 for the last PDU of immediate data or the last PDU of a
>>    sequence answering a R2T."
>>
>> This suggests that immediate data can extend beyond the iSCSI SCSI CMD
>PDU
>> (there is no final bit on the command PDU)but the closet thing we have
to
>a
>> definition for immediate data, which is in clause 1.2.5 below, suggests
>> that
>> immediate data is limitted to the command PDU:
>>
>> "Outgoing SCSI data (initiator to target - user data or command
>>    parameters) will be sent as either solicited data or unsolicited
>>    data.  Solicited data are sent in response to Ready To Transfer (R2T)
>>    PDUs. Unsolicited data can be part of an iSCSI command PDU
>>    ("immediate data") or an iSCSI data PDU.  An initiator may send
>>    unsolicited data (immediate or in a separate PDU) up to the
>>    negotiated limit (initial burst size - mode page 02h). All subsequent
>>    data have to be solicited.  The maximum size of an individual data
>>    PDU or the immediate-part of the initial unsolicited burst MAY be
>>    negotiated at login (maximum connect size - mode page 02h)."
>>
>> Unsolicited data is bounded in a number of ways, but most significant to
>> this issue is in the description of the login key useR2t which states:
>>
>> "Note than only the first
>>    outgoing data item (either immediate data or a separate PDU) can be
>>    sent unsolicited by a R2T."
>>
>> Given the above it is unclear to me if the concept of immediate data is:
>>
>> 1. Write data that can be sent without an R2T (unsolicited write
data)and
>> starts in the command PDU.
>> 2. Write data that is entirely contained within the command PDU. (my
>> initial
>> concept of immediate data)
>> 3. The non-zero portion of data, in a possible multi-PDU write
operation,
>> that is contained in the iSCSI SCSI command PDU. Where the remaining
PDUs
>> of
>> the write operation must be solicited with an R2T.
>>
>> Given the current definition of the Final bit and the current limits on
>> unsolicited data it is unclear to me what a target should do when
>receiving
>> an iSCSI SCSI command PDU that has immediate data in it, but not
>sufficient
>> data to meet the expected data transfer length specificied in the
>command.
>> Does an R2T go out of not?
>>
>> Barry Reinhold
>> 603-659-0885
>
>Barry Reinhold
>603-659-0885
>
>
>
>

---

From: "Barry Reinhold" <[email-masked]>
Date: Mon, 5 Feb 2001 10:11:36 -0500

Julian,
	I  think one could carry on a debate about what the draft currently states
about the coupling of immediate and unsolicited data - but it is probably
more productive to figure out if we want to have both the concept of
immediate data and unsolicited data and then worry about the wording in the
draft.
	I am of the opinion that immediate data (data within a single PDU) is a
useful distincition from unsolicited data (iSCSI DATA PDU sent without R2T).
The reasoning behind this is that there are a number of write IOs that can
be satisfied by providing a buffer of 8K or less. If this data is contained
in the command PDU the context can be created for the command, and then the
data just happens to be sitting there with the context to process it. Thus
the target can send the iSCSI response PDU without creating any special
context or mechansims - should have nice low latency characteristics for
small writes.
	For devices that don't want to use the initiator task tag for establishing
context of an iSCSI DATA PDU, immediate data is a nice thing. An iSCSI DATA
PDU without a target task tag may not be such a nice thing.
	Question here - What would you lose if you supported only immediate data?
It seems like this meets some of the common application requirements that I
am aware of and is friendly to what I think is the more common approach to
target processing of DATA PDUs.

>Sent: Sunday, February 04, 2001 5:20 AM
>To: [email-masked]

---

From: Pierre Labat <[email-masked]>
Date: Wed, 07 Feb 2001 11:04:41 -0800

Barry,

Initiators that want to use small iSCSI PDUs (less than 8k) cannot send the 8k
data as immediate. They have to send it in several small unsolicited data PDUs.
It 's why we must keep this alternative to immediate data.

Regards,

Pierre

<stuff deleted>

---

From: Pierre Labat <[email-masked]>
Date: Wed, 07 Feb 2001 10:56:57 -0800

> supposed to send R2T (there is only one unsolicited burst per command).

Julian, Barry,

I think there is a problem, as stated Barry, if the initiator decides to
send unsolicited data but less than the Firstburst and
the expected length is greater than the Firstburst. In this case the initiator

after sending some unsolicited data waits for an R2T to continue
and the target waits for a First burst load of data before sending
an R2T.

Solution
======
To avoid that, it should be explicitely stated (may be it's what
is assumed but i can't read it) that when the initiator sends
unsolicited non immediate data, it must send unsolicited data
up to Firstburst or Expected data length whichever is less.

Regards,

Pierre

<stuff deleted>