Subject: Re: [Ips] New iSCSI key

From: John Hufferd <[email-masked]>
Date: Sat, 16 Oct 2004 00:15:41 -0700

Comments below.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Mike Ko/Almaden/IBM
10/15/2004 05:34 PM

To
John Hufferd/San Jose/IBM@IBMUS
cc
[email-masked]
Subject
Re: [Ips] New iSCSI key

John,

The initiator should never see an R2T since an R2T is transformed into an 
RDMA Read Request by the target and is never sent as a control-type PDU. 
Neither the DA nor the iSER spec ever mentioned sending an R2T as a 
control-type PDU.  That being the case, the initiator should never be 
sending SCSI Data-out PDUs that are "solicited" by an R2T.  Conversely, a 
target will not be "expecting" control-type PDUs containing "solicited" 
data since it never sends an R2T as is.  But since a SCSI Data-out PDU 
containing legitimate unsolicited data is sent as a control-type PDU, we 
allow the iSER layer to send *all* SCSI Data-out PDUs as control-type 
PDUs, even when they contain "solicited" data. 
<Huff> That makes no since.  It can not be solicited data if there is no 
way to solicit it. <Huff>

However in view of MaxOutstandingUnexpectedPDU, what happens if the limit 
is smaller than the amount of "solicited" data that the iSCSI layer wants 
to send? 
<Huff> Again this makes no sense, because there is no way to solicit the 
data.  <Huff>

Using ExpCmdSN to determine if the PDUs are outstanding or not might not 
work since the target might not advance ExpCmdSN frequent enough for all 
the "solicited" data to be transferred. 
<Huff> If there is no way to solicit the data then there should be no fear 
about getting solicited data. <Huff>

We would need a different mechanism to replenish the credit.  But sending 
"solicited" data as control-type PDUs is an error since it is not 
"solicited" by an R2T. 
<Huff> Agreed <Huff>

Should we make life complicated in order to accommodate an error case?
<Huff> No! <Huff>

For SCSI Data-in and R2T, neither the DA nor the iSER spec mentioned 
allowing the target to send them as control-type PDUs.  On SCSI Data-out 
PDUs, the DA spec mentioned the following:

Section 7.1 "the iSCSI layer does not request a SCSI Data-Out PDU carrying 
solicited data for transmission across the Datamover Interface per this 
architecture"

Section 7.1.1 "a SCSI Data-Out PDU for solicited data is never requested 
for transmission across the Datamover Interface at the initiator"

Section 11.4.5.1 "the solicited SCSI Data-out arriving on the target is 
not notified to the iSCSI layer using the
Control_Notify Primitive because the solicited SCSI Data-out was not sent 
by the initiator iSCSI layer as control-type PDUs"

So to be consistent with the DA spec, I propose that we also disallow the 
sending of "solicitied" data as control-type PDUs.  If we accept this 
correction, then the SCSI Data-out handling for 
MaxOutstandingUnexpectedPDUs can be simplified.  Also the last paragraph 
in section 7.3.4 will be updated to remove the reference to allowing 
"solicited" data being sent as control-type PDUs.
<Huff> I believe the current wordage is simply a piece of left over text 
from an early approach, which we failed to remove.  So yes, I agree that 
we should disallow the sending of "solicited" data as a control-type PDU.

Comments?

Mike
From:   John Hufferd on 10/15/2004 04:18 PM

To:     Mike Ko/Almaden/IBM
cc:     [email-masked] 
Subject:        Re: [Ips] New iSCSI key 

Mike,
I think we are getting closer, but since if Data-Outs ever gets solicited 
by an R2T, even though it shouldn't, it is by definition expected.  So 
since the only other case for Data-Out PDUs is for unsolicited Data that 
can be anticipated, as you pointed out, via the FirstBurstLength, I just 
do not see Data-Out PDUs being in the same class as what we have been 
discussing, and think we should not even factor it into the unexpected 
classification.  In other words just drop point 2.  However, the text you 
have at the beginning of that paragraph, "Unsolicited data in a SCSI write 
operation are sent by the iSCSI layer at the initiator using SCSI Data-out 
PDUs.  The iSER layer at the target can use
FirstBurstLength to determine the amount of buffering required."  should 
some how be placed somewhere else in the draft.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Mike Ko/Almaden/IBM
10/15/2004 01:23 PM

To
John Hufferd/San Jose/IBM@IBMUS
cc
[email-masked]
Subject
Re: [Ips] New iSCSI key

John,
Here is the replacement paragraph for the SCSI Data-out PDU:

2. Unsolicited data in a SCSI write operation are sent by the iSCSI layer 
at
the initiator using SCSI Data-out PDUs.  The iSER layer at the target can 
use
FirstBurstLength to determine the amount of buffering required.  For 
solicited
data, an R2T PDU sent by the iSCSI layer at the target is transformed by 
the
iSER layer at the target into an RDMA Read operation (see section 7.3.6).
Therefore the iSCSI layer at the initiator should not be issuing SCSI 
Data-out
PDUs for solicited data.  If a SCSI Data-out PDU containing solicited data 
is
requested for transmission by the iSCSI layer at the initiator anyway, the 
iSER
layer may send it as a control-type PDU (see section 7.3.4).  Since it is
"unexpected", the number that can be sent by the initiator falls under the
MaxOutstandingUnexpectedPDU declared by the target.  The PDU is 
outstanding
until the target responds with a control-type PDU with ExpCmdSN set to X + 
1
where X is the CmdSN of the referenced command.

Mike
Sent by:        [email-masked]
To:     John Hufferd/San Jose/IBM@IBMUS
cc:     [email-masked] 
Subject:        Re: [Ips] New iSCSI key

John,

Would the following replacement text for the sentence "Solicited data are
normally handled ..." better?
"The R2T mechanism is normally used to handle solicited data and the SCSI
Data-out PDU should not be used for this purpose (see section 7.3.4)."

Mike
From:   John Hufferd on 10/15/2004 10:22 AM

To:     Mike Ko/Almaden/IBM
cc:     [email-masked]
Subject:        Re: [Ips] New iSCSI key

Mike,
I think you covered my concerns, but perhaps we should actually use words
in the draft similar to what you have used here for the unsolicited
Data-Out PDUs.  I just thought that the words you proposed for the draft
were a little bit confusing, since we never have talk about how an iSCSI
layer requests Data-Out PDUs, etc.  I think your words in this note were
more to the point.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Mike Ko/Almaden/IBM
10/15/2004 10:01 AM

To
John Hufferd/San Jose/IBM@IBMUS
cc
[email-masked]
Subject
Re: [Ips] New iSCSI key

John,

My comments are embedded.

Mike
To:     Mike Ko <[email-masked]>
cc:     [email-masked]
Subject:        Re: [Ips] New iSCSI key

Mike,
First I think there is a Typo near the end of your note when you talk
about NOP-Out PDU where you might have meant Nop-In.  That is, where you
say "...Otherwise, a NOP-out PDU is
considered "unexpected",...." perhaps you should say Nop-In.

[mk[
Good catch.  I will fix that.
]mk]

Second I think we need to discuss the unsolicited Data-Out PDU.  Please
expand on the concept of the use of Data-Out PDU for solicited Data in
iSER.

[mk[
SCSI Data-out PDU should not be used for solicited data even though it is
supported in the iSER draft.  Since this is not a typical usage, the
burden is left to the iSER/iSCSI layer to keep track of the number of
"unexpected" PDU that it can send.  What specifically do you want to
expand on this topic?
]mk]

Third we need to discuss or expand on considering the Reject PDU an
unsolicited PDU.   Perhaps you meant Rejects for PDUs that are not for the
Command itself.

[mk[
I will replace the phrase "A Reject PDU is considered "unexpected"" with
the following:
For a task which has never been active before the Reject PDU is sent, the
target should not send any further responses because the command itself is
discarded (see section 6.3 of RFC3720).  Therefore, this type of Reject
PDU is expecetd.  All other Reject PDUs are "unexpected".
]mk]
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Mike Ko <[email-masked]>
Sent by: [email-masked]

10/14/2004 09:36 PM

To
<[email-masked]>
cc

Subject
[Ips] New iSCSI key

If we do decide on adding a new iSCSI key to limit the number of
"unexpected"
or "unanticipated" PDUs, I offer the following text to be considered for
adding
to section 6 of the iSER draft.

Currently the key is tentatively called "MaxOutstandingUnexpectedPDUs".
Unless
there are other suggestions, this name will be used.

Use: LO (leading only), Declarative
Senders: Initiator and Target
Scope: SW (session-wide)
Irrelevant when: RDMAExtensions=No
MaxOutstandingUnexpectedPDUs=<numerical-value-from-0-to-(2**64-1) | None>
Default is None

This key is used by the initiator and the target to declare the maximum
number
of outstanding "unexpected" control-type PDUs that it can receive.  It is
intended to allow the receiving side to determine the amount of buffer
resoruces needed beyond the normal flow control mechanism available in
iSCSI.
An initiator or target that cannot function properly without any limit on
MaxOutstandingUnexpectedPDU (which is the default) should choose a value
such
that it would not impose an unnecessary constraint on the iSCSI layer
under
normal circumstances.  The proper value to use is implementation
dependent.  If
the sending side fails to adhere to the declared limit as set by the
receiving
side, it is up to the receiving side to determine ways of handling the
overrun,
including dropping the connection.

For control-type PDUs sent by an initiator to a target, where the
immediate
flag is not set, the command numbering mechanism regulates the maximum
number
of PDUs that the target can expect.  The exceptions include the SNACK PDU,

the
SCSI Data-out PDU, and any PDU with the immediate flag set.

1. The SNACK PDU is not needed in iSER (see section 7.3.11).  But if one
is
requested for transmission by the iSCSI layer at the initiator, it will be

sent
by the iSER layer as a control-type PDU.  Since it is "unexpected", the
number
that can be sent by the initiator falls under the
MaxOutstandingUnexpectedPDU
declared by the target.  The PDU is outstanding until the target responds
with
a control-type PDU with ExpCmdSN set to X + 1 where X is the CmdSN of the
referenced command.

2. For unsolicited SCSI Data-out PDU, the target can use FirstBurstLength
to
determine the amount of buffering required.  Solicited data are normally
handled using the R2T mechanism (see section 7.3.4).  However, if a SCSI
Data-out PDU containing solicited data is requested for transmission by
the
iSCSI layer at the initiator, the iSER layer may send it as unsolciited
data.
Since it is "unexpected", the number that can be sent by the initiator
falls
under the MaxOutstandingUnexpectedPDU declared by the target.  The PDU is
outstanding until the target responds with a control-type PDU with
ExpCmdSN set
to X + 1 where X is the CmdSN of the referenced command.

3. A PDU with the immediate flag set is considered "unexpected", and the
number
that can be sent by the initiator falls under the
MaxOutstandingUnexpectedPDU
declared by the target.  The PDU is outstanding until the target responds
with
a control-type PDU with ExpCmdSN set to X + 1 where X is the CmdSN of the
PDU
with the immediate flag set.

Control-type PDUs sent by a target as a response to a request from an
initiator
are expected by the initiator and buffer resources required can be
provisioned.
The exceptions include the SCSI Data-in PDU, the R2T PDU, the Asynchronous
Message PDU, the Reject PDU, and the NOP-in PDU.

1. A SCSI Data-in PDU is a data-type PDU and is transformed into RDMA
Write
operations (see section 7.3.5).

2. An R2T PDU is a data-type PDU and is transformed into RDMA Read
operations
(see section 7.3.6).

3. An Asynchronous Message PDU is considered "unexpected", and the number
that
can be sent by the target falls under the MaxOutstandingUnexpectedPDU
declared
by the initiator.  The PDU is outstanding until the initiator sends a
control-type PDU with ExpStatSN set to X + 1 where X is the StatSN of the
Asychronous Message PDU.

4. A Reject PDU is considered "unexpected", and the number that can be
sent by
the target falls under the MaxOutstandingUnexpectedPDU declared by the
initiator.  The PDU is outstanding until the initiator sends a
control-type PDU
with ExpStatSN set to X + 1 where X is the StatSN of the Reject PDU.

5. A NOP-in PDU sent as a response by the target to a NOP-out PDU from the
initiator is expected by the initiator.  Otherwise, a NOP-out PDU is
considered
"unexpected", and the number that can be sent by the target falls under
the
MaxOutstandingUnexpectedPDU declared by the initiator.  The PDU is
outstanding
until the initiator sends a control-type PDU with ExpStatSN set to X + 1
where
X is the StatSN of the NOP-in PDU.

Mike

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 18 Oct 2004 11:12:58 -0700

I'm all for developing this option (this thread discusses #3 in the 
original list of options) but I really would like more feedback on other 
options as well.  Developing this option is a good thing since doing so 
allows us to appreciate the (non)complexity of #3 compared to other 
candidates.

With that caveat, a few comments on this thread.

I agree that the current 7.3.4 discussion of the "sending solicited PDUs 
for data that was never solicited" case is just confusing.  I recommend 
deleting the last two sentences in the last para of 7.3.4 (starting from 
"However...").  In the current architecture, if the TTT happens to be 
non-0xffffffff, it's an initiator code bug that will be suitably 
responded to by the target.

If we do this, then I presume the #2 item in Mike's I->T list drops out.

A general suggestion about all items is: factor out the repeating 
sentence to the top of the list ("Since it is "unexpected",.....").

Recommend deleting #1 & #2 in the T->I list that simply explain 
data-type PDUs, but describe them as "exceptions" to the control-type 
PDU approach.  I think this is confusing and besides, covered elsewhere.

#4 on the T->I list discusses Reject.  I presume this is for Rejects on 
non-command PDUs since Rejects on commands are "expected" in a way (at 
least the buffer room is available on the initiator).  If you agree with 
this, suggest noting this "non-command" nature explicitly.

I assume #5 is changing per John's comments.

Thanks.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

John Hufferd wrote:

> *Mike Ko/Almaden/IBM*
> 	Re: [Ips] New iSCSI keyLink 
> <Notes:///8725644B00277CD1/A24CF79FA927570285256356005827C4/FB555F8427E661F588256F2E007F4CE6>
> 
> 
> 	
> 
> 
> 
> 
> John,
> 
> The initiator should never see an R2T since an R2T is transformed into 
> an RDMA Read Request by the target and is never sent as a control-type 
> PDU.  Neither the DA nor the iSER spec ever mentioned sending an R2T as 
> a control-type PDU.  That being the case, the initiator should never be 
> sending SCSI Data-out PDUs that are "solicited" by an R2T.  Conversely, 
> a target will not be "expecting" control-type PDUs containing 
> "solicited" data since it never sends an R2T as is.  But since a SCSI 
> Data-out PDU containing legitimate unsolicited data is sent as a 
> control-type PDU, we allow the iSER layer to send *all* SCSI Data-out 
> PDUs as control-type PDUs, even when they contain "solicited" data.  
> However in view of MaxOutstandingUnexpectedPDU, what happens if the 
> limit is smaller than the amount of "solicited" data that the iSCSI 
> layer wants to send?  
> <Huff> If there is no way to solicit the data then there should be no 
> fear about getting solicited data. <Huff>
> 
> We would need a different mechanism to replenish the credit.  But 
> sending "solicited" data as control-type PDUs is an error since it is 
> not "solicited" by an R2T.  
> <Huff> Agreed <Huff>
> 
> Section 7.1 "the iSCSI layer does not request a SCSI Data-Out PDU 
> carrying solicited data for transmission across the Datamover Interface 
> per this architecture"
> 
> Control_Notify Primitive because the solicited SCSI Data-out was not 
> sent by the initiator iSCSI layer as control-type PDUs"
> 
> So to be consistent with the DA spec, I propose that we also disallow 
> the sending of "solicitied" data as control-type PDUs.  If we accept 
> this correction, then the SCSI Data-out handling for 
> Subject:        Re: [Ips] New iSCSI key  Link 
> <Notes:///882563E0007789EC/A24CF79FA927570285256356005827C4/FB555F8427E661F588256F2E007F4CE6> 
> 
> 
> Mike,
> I think we are getting closer, but since if Data-Outs ever gets 
> solicited by an R2T, even though it shouldn't, it is by definition 
> expected.  So since the only other case for Data-Out PDUs is for 
> unsolicited Data that can be anticipated, as you pointed out, via the 
> FirstBurstLength, I just do not see Data-Out PDUs being in the same 
> class as what we have been discussing, and think we should not even 
> factor it into the unexpected classification.  In other words just drop 
> point 2.  However, the text you have at the beginning of that paragraph, 
> "Unsolicited data in a SCSI write operation are sent by the iSCSI layer 
> at the initiator using SCSI Data-out PDUs.  The iSER layer at the target 
> can use
> *Mike Ko/Almaden/IBM*
> 	Re: [Ips] New iSCSI keyLink 
> <Notes:///8725644B00277CD1/A24CF79FA927570285256356005827C4/46DCAAEBC453D43D85256F2E00640D67>
> 
> 
> 	
> 
> 
> 
> 
> John,
> 2. Unsolicited data in a SCSI write operation are sent by the iSCSI 
> layer at
> the initiator using SCSI Data-out PDUs.  The iSER layer at the target 
> can use
> PDUs for solicited data.  If a SCSI Data-out PDU containing solicited 
> data is
> requested for transmission by the iSCSI layer at the initiator anyway, 
> the iSER
> MaxOutstandingUnexpectedPDU declared by the target.  The PDU is outstanding
> until the target responds with a control-type PDU with ExpCmdSN set to X 
> + 1
> ------------------------------------------------------------------------
>

---

From: Mike Ko <[email-masked]>
Date: Mon, 18 Oct 2004 15:06:23 -0700

Mallikarjun,

Here is the revised write up for the new key based on feedback from you 
and John.  Note that there are also minor changes not covered by the 
feedback:

1. The "unexpected" PDU is outstanding until ExpStatSN is *at least* X + 1 
where X is the StatSN of the "unexpected" PDU (was "until ExpStatSN is X + 
1").

2. A SNACK PDU is outstanding until the target sends a SCSI Response PDU 
for the referenced command.

Start of key description:
-----------
Use: LO (leading only), Declarative

Scope: SW (session-wide)

Default is None

resources needed beyond the normal flow control mechanism available in 
MaxOutstandingUnexpectedPDUs (which is the default) should choose a value 

For control-type PDUs sent by an initiator to a target, where the 
immediate
flag is not set, the command numbering mechanism regulates the maximum 
number
of PDUs that the target can expect.  For iSCSI Data-out PDUs which are 
used by
the iSCSI layer at the initiator to send unsolicited data, the iSER layer 
at
the target can use FirstBurstLength to determine the amount of buffering
required.  For the "unexpected" PDUs, the MaxOutstandingUnexpectedPDUs key
declared by the target is used to limit the number of oustanding 
"unexpected"
control-type PDUs which can be sent by the initiator.  The "unexpected" 
PDUs
include the following:

1. A PDU with the immediate flag set.  The PDU is outstanding until the 
target
responds with a control-type PDU with ExpCmdSN set to at least X + 1 where 
X is
the CmdSN of the PDU with the immediate flag set.

2. A SNACK PDU.  Note that SNACK PDUs are not needed in iSER (see section
7.3.11).  A SNACK PDU which is sent anyway by the initiator is outstanding
until the target responds with a SCSI Response PDU for the referenced 
command.

Control-type PDUs sent by a target as a response to a request from an 
initiator
are expected by the initiator and buffer resources required can be 
provisioned.
Note that the SCSI Data-in PDU and the R2T PDU are data-type PDUs (see 
section
7.1).  For the "unexpected" control-type PDUs, the 
MaxOutstandingUnexpectedPDUs
key declared by the initiator is used to limit the number of outstanding
"unexpected" control-type PDUs which can be sent by the target.  An
"unexpected" PDU is outstanding until the initiator sends a control-type 
PDU
with ExpStatSN set to at least X + 1 where X is the StatSN of the 
"unexpected"
PDU.  The "unexpected" PDUs include the following:

1. An Asynchronous Message PDU

2. A Reject PDU except when it is sent before a task is active since the
command itself is discarded and the target should not send any further
responses (see section 6.3 of RFC3720).

3. A NOP-in PDU except when it is sent as a response to a NOP-out PDU.

---------------------
end of key description.

For section 7.3.4, the last paragraph will be replaced by the following 
text:

"Note that for solicited data, the SCSI Data-out PDUs are not used since 
R2T
PDUs are not delivered to the iSCSI layer at the initiator; instead R2T 
PDUs
are transformed by the iSER layer at the target into RDMA Read operations 
(see
section 7.3.6)."

Mike
To:     [email-masked]
cc:      

Thanks.
--
Mallikarjun

Mallikarjun Chadalapaka

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 18 Oct 2004 18:38:33 -0500

On Oct 18, 2004, at 5:06 PM, Mike Ko wrote:
>   If the sending side fails to adhere to the declared limit as set by 
> the
> receiving side, it is up to the receiving side to determine ways of 
> handling the
> overrun, including dropping the connection.
>

It may be worth noting that the sending side MAY enforce the
limit by blocking or refusing to send a PDU that would have
exceeded the limit. This would be the equivalent of the local
back-pressure applied by TCP when transport buffering is exhausted.

Or then, it might belong more in the DataMover Architecture draft.

---

From: Mike Ko <[email-masked]>
Date: Mon, 18 Oct 2004 17:24:25 -0700

Caitlin,

It seems to me that we can treat this new key for "unexpected" PDUs 
similar to the way MaxCmdSN is described in iSCSI.  Rather than specifying 
all the possible ways the sender can do to meet the requirement, we can 
just leave it up to implementation.

Mike
To:     [email-masked], Mike Ko <[email-masked]>
cc:      
Subject:        Re: [Ips] New iSCSI key