Subject: [Ips] New iSCSI key

From: Mike Ko <[email-masked]>
Date: Thu, 14 Oct 2004 21:36:43 -0700

If we do decide on adding a new iSCSI key to limit the number of 
"unexpected"
or "unanticipated" PDUs, I offer the following text to be considered for 
adding
to section 6 of the iSER draft.

Currently the key is tentatively called "MaxOutstandingUnexpectedPDUs". 
Unless
there are other suggestions, this name will be used.

Use: LO (leading only), Declarative
Senders: Initiator and Target
Scope: SW (session-wide)
Irrelevant when: RDMAExtensions=No
MaxOutstandingUnexpectedPDUs=<numerical-value-from-0-to-(2**64-1) | None>
Default is None

This key is used by the initiator and the target to declare the maximum 
number
of outstanding "unexpected" control-type PDUs that it can receive.  It is
intended to allow the receiving side to determine the amount of buffer
resoruces needed beyond the normal flow control mechanism available in 
iSCSI.
An initiator or target that cannot function properly without any limit on
MaxOutstandingUnexpectedPDU (which is the default) should choose a value 
such
that it would not impose an unnecessary constraint on the iSCSI layer 
under
normal circumstances.  The proper value to use is implementation 
dependent.  If
the sending side fails to adhere to the declared limit as set by the 
receiving
side, it is up to the receiving side to determine ways of handling the 
overrun,
including dropping the connection.

For control-type PDUs sent by an initiator to a target, where the 
immediate
flag is not set, the command numbering mechanism regulates the maximum 
number
of PDUs that the target can expect.  The exceptions include the SNACK PDU, 
the
SCSI Data-out PDU, and any PDU with the immediate flag set.

1. The SNACK PDU is not needed in iSER (see section 7.3.11).  But if one 
is
requested for transmission by the iSCSI layer at the initiator, it will be 
sent
by the iSER layer as a control-type PDU.  Since it is "unexpected", the 
number
that can be sent by the initiator falls under the 
MaxOutstandingUnexpectedPDU
declared by the target.  The PDU is outstanding until the target responds 
with
a control-type PDU with ExpCmdSN set to X + 1 where X is the CmdSN of the
referenced command.

2. For unsolicited SCSI Data-out PDU, the target can use FirstBurstLength 
to
determine the amount of buffering required.  Solicited data are normally
handled using the R2T mechanism (see section 7.3.4).  However, if a SCSI
Data-out PDU containing solicited data is requested for transmission by 
the
iSCSI layer at the initiator, the iSER layer may send it as unsolciited 
data.
Since it is "unexpected", the number that can be sent by the initiator 
falls
under the MaxOutstandingUnexpectedPDU declared by the target.  The PDU is
outstanding until the target responds with a control-type PDU with 
ExpCmdSN set
to X + 1 where X is the CmdSN of the referenced command.

3. A PDU with the immediate flag set is considered "unexpected", and the 
number
that can be sent by the initiator falls under the 
MaxOutstandingUnexpectedPDU
declared by the target.  The PDU is outstanding until the target responds 
with
a control-type PDU with ExpCmdSN set to X + 1 where X is the CmdSN of the 
PDU
with the immediate flag set.

Control-type PDUs sent by a target as a response to a request from an 
initiator
are expected by the initiator and buffer resources required can be 
provisioned.
The exceptions include the SCSI Data-in PDU, the R2T PDU, the Asynchronous
Message PDU, the Reject PDU, and the NOP-in PDU.

1. A SCSI Data-in PDU is a data-type PDU and is transformed into RDMA 
Write
operations (see section 7.3.5).

2. An R2T PDU is a data-type PDU and is transformed into RDMA Read 
operations
(see section 7.3.6).

3. An Asynchronous Message PDU is considered "unexpected", and the number 
that
can be sent by the target falls under the MaxOutstandingUnexpectedPDU 
declared
by the initiator.  The PDU is outstanding until the initiator sends a
control-type PDU with ExpStatSN set to X + 1 where X is the StatSN of the
Asychronous Message PDU.

4. A Reject PDU is considered "unexpected", and the number that can be 
sent by
the target falls under the MaxOutstandingUnexpectedPDU declared by the
initiator.  The PDU is outstanding until the initiator sends a 
control-type PDU
with ExpStatSN set to X + 1 where X is the StatSN of the Reject PDU.

5. A NOP-in PDU sent as a response by the target to a NOP-out PDU from the
initiator is expected by the initiator.  Otherwise, a NOP-out PDU is 
considered
"unexpected", and the number that can be sent by the target falls under 
the
MaxOutstandingUnexpectedPDU declared by the initiator.  The PDU is 
outstanding
until the initiator sends a control-type PDU with ExpStatSN set to X + 1 
where
X is the StatSN of the NOP-in PDU.

Mike

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Andre Hedrick <[email-masked]>
Date: Thu, 14 Oct 2004 23:07:26 -0700 (PDT)

Mike,

Not to be cheeky, but would it be possible for everyone to support and
send all the current keys first before adding another?

Cheers,

Andre

---

From: Mike Ko <[email-masked]>
Date: Thu, 14 Oct 2004 23:37:51 -0700

Andre,

I am not sure I follow your comment.  Can you be more specific?

Mike
To:     Mike Ko <[email-masked]>
cc:     [email-masked] 
Subject:        Re: [Ips] New iSCSI key

Mike,

Cheers,

Andre

---

From: John Hufferd <[email-masked]>
Date: Fri, 15 Oct 2004 00:13:38 -0700

Mike,
First I think there is a Typo near the end of your note when you talk 
about NOP-Out PDU where you might have meant Nop-In.  That is, where you 
say "...Otherwise, a NOP-out PDU is 
considered "unexpected",...." perhaps you should say Nop-In.

Second I think we need to discuss the unsolicited Data-Out PDU.  Please 
expand on the concept of the use of Data-Out PDU for solicited Data in 
iSER.

Third we need to discuss or expand on considering the Reject PDU an 
unsolicited PDU.   Perhaps you meant Rejects for PDUs that are not for the 
Command itself.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

Mike Ko <[email-masked]> 
Sent by: [email-masked]
10/14/2004 09:36 PM

To
<[email-masked]>
cc

Subject
[Ips] New iSCSI key

the

sent

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 15 Oct 2004 04:38:30 -0500

On Oct 14, 2004, at 11:36 PM, Mike Ko wrote:

> "unexpected" or "unanticipated" PDUs, I offer the following text to
> be considered for adding to section 6 of the iSER draft.
>
> Unless there are other suggestions, this name will be used.
>

Having "None" be the default makes sense in the immediate
context: that is if your peer does not inform you of a limit there
is none.

However "None" should not be the default assumed by the
user of a DataMover service. The IETF RDDP WG has decided
that an SRQ user must not be forced to take per-connection
overrun alerts on its own as the only available remedy to
prevent an SRQ-drain denial-of-service attack.  The mechanisms
that would enable the hardware or Privileged Resource Manager
to guard against these attacks require a finite number.

That immediately leads to the question as to how this limit is
communicated between the DataMover service and the user
of that service.  Possibilities include:
	- DataMover publishes a value, user lives with it.
	- User publishes a value, DataMover lives with it.
	- User requests a value, DataMover indicates what
	   it can/is willing to provide.

Of course this is a conceptual interface only.  There is
no obligation to implement the DataMover as a distinct
layer that is separate from the iSCSI code.

My other question is whether there is already language
that adequately defines when an unexpected PDU is
no longer "outstanding".  I have been assuming that
when any later numbered PDU has been acked that
it is no longer outstanding.  Is there already language
that covers that adequately?

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 15 Oct 2004 04:51:35 -0500

> send all the current keys first before adding another?

If you are not implementing iSER this key is irrelevant.
If you have implemented iSER along the lines that the drafters
envisioned then you simply use the "None" default.  Otherwise
you are merely publishing the allowance you already calculated
in determining how to configure and populate your SRQ.

A given implementation probably already has a maximum
number of outstanding unanticipated PDUs.  For one thing
it should have understood that the receiving iSCSI peer
was already at liberty to discard most of them, and likely
would when the number got large.

In any event, if you know that number and the allowance
published by your peer is greater than you do not have
to do *anything*.

Otherwise you simply need to keep track of the number
of unanticipated buffers you have outstanding, and refrain
from sending more.  Since that is already done for anticipated
buffers it would not seem to be that difficult to implement.

Keep in mind that your peer SHOULD provide enough
extra credits that a typical implementation will not be
required to do any actual counting.

It might make sense to create an official "safe harbor".
Any implementation that knows it would never generate
more than X outstanding unanticipated buffers will NEVER
have to implement any flow control logic for unanticipated
buffers.

An implementation is also at liberty to simply ignore the
limit, but it does so at the risk that sending too many
unanticipated messages will result in the connection
being broken -- which is EXACTLY what the state of
iSER would be if this key is not added.

So adding this key does not have to represent more
complex logic for anyone, but it does allow implementations
to have better control over avoiding overruns caused by
unanticipated messages when sent over transports
that lack TCP's bottleneck "feature".

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 15 Oct 2004 06:11:23 -0500

> layer that is separate from the iSCSI code.
Also, a DataMover may offer a blocking or balking interface
for submitting iSCSI PDUs.  If that is the nature of the local
interface, then any publishing of the limit is optional, and
the semantics are that *any* iSCSI PDU that the Datamover
considers premature may result in blocking of the caller
or in the put being temporarily disallowed (i.e., the caller
gets a "try again later" error).

That is essentially the semantics offered by a TCP transport.
It is hard to anticipate when you will hit the transport layer
buffering limits.  When you do you block, or you are told
to try again later.

---

From: Mike Ko <[email-masked]>
Date: Fri, 15 Oct 2004 09:34:58 -0700

Caitlin,

My comments are embedded.

Mike
cc:     <[email-masked]> 

[mk[
My first thought was that the default value should apply for most 
implementations so that the key need not be declared during the login 
process.  For implementations that support graceful handling or where the 
iSER/iSCSI layer can replenish the buffers as fast as the packet arrival 
rate, this is indeed the case.  For other implementations, the value to 
use could be highly dependent on the implementation, making a generally 
acceptable default difficult to define.  But I could be wrong, and I 
welcome any suggestion as to what this default value should be.
]mk]

[mk[
Will respond in your follow-up note.
]mk]

[mk[
There were discussions on the IPS forum that mention the use of additional 
explicit credit scheme (Alex Nezhinsky) and uncertainty as to when an 
"unexpected" PDU is no longer outstanding, or "retired" (Pat Thaler).  The 
text was written to address these concerns.  I don't have any problem 
adding references to appropriate sections in RFC3720 that describe when an 
unexpected PDU is no longer outstanding.
]mk]

---

From: Mike Ko <[email-masked]>
Date: Fri, 15 Oct 2004 09:38:52 -0700

Caitlin,

It seems to me how the iSER layer and the iSCSI layer exchange information 
on the value of MaxOutstandingUnexpectedPDU is implementation dependent 
and need not be specified in the iSER draft.

Mike
To:     [email-masked]
cc:

---

From: Mike Ko <[email-masked]>
Date: Fri, 15 Oct 2004 10:01:43 -0700

John,

Mike

Mike, 

[mk[
Good catch.  I will fix that.
]mk]

[mk[
SCSI Data-out PDU should not be used for solicited data even though it is 
supported in the iSER draft.  Since this is not a typical usage, the 
burden is left to the iSER/iSCSI layer to keep track of the number of 
"unexpected" PDU that it can send.  What specifically do you want to 
expand on this topic?
]mk]

[mk[
I will replace the phrase "A Reject PDU is considered "unexpected"" with 
the following:
For a task which has never been active before the Reject PDU is sent, the 
target should not send any further responses because the command itself is 
discarded (see section 6.3 of RFC3720).  Therefore, this type of Reject 
PDU is expecetd.  All other Reject PDUs are "unexpected".
]mk]
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked] 

10/14/2004 09:36 PM 

To
<[email-masked]> 
cc

Subject
[Ips] New iSCSI key

the

sent

Mike