Subject: RE: [Ips] WG Last Call Review: iSER

From: [email-masked]
Date: Mon, 13 Jun 2005 10:43:38 -0400

Mike,

There are a couple of problems in what you propose:

> (3) Section 3.3., item 5. says "MUST successfully negotiate
> the TargetRecvDataSegmentLength key".  Again there's a "takes
> two to negotiate" problem, and the "failed to successfully
> negotiate" case needs to be specified.
> 
> [mk] The old items 4 and 5 are now replaced by the new item 4 
> with the following text:
> 
> "The iSCSI Layers at both the initiator and the target MUST 
> support the keys as defined in section 6 on Login/Text Operational Keys.  
> If used as specified, these keys MUST NOT be answered with NotUnderstood
> and the semantics as defined MUST be followed for each iSER-assisted 
> connection."

That would be ok for the case where there are iSER systems on both
ends of a connection, but your new text indirectly forbids connecting
an iSER implementation to a non-iSER one (i.e., Traditional iSCSI),
as this would violate the first MUST (at least one of the systems
will not support the new keys).  This text (and all of the negotiation
text) needs to be rephrased to apply to a single system, and specify
requirements on that system - if iSER is to be used, then the following
keys MUST be offered or responded to in negotiation *and* if this
negotiation fails for any reason (e.g., a NotUnderstood response, or
lack of an offered key that would be necessary to use iSER), then
iSER MUST NOT be used, and traditional iSCSI is used instead.  It
is important to specify the fallback-to-Traditional-iSCSI cases in
the negotiation requirements, as appealing to the covering "in order
to use iSER" text is not enough because it doesn't cover what happens
when negotiation fails.  This needs to be done at least in Section 5.1
also.

> (10) Section 8.4 could result in a situation where an iSER Target
> cannot send a Reject PDU that an iSCSI Target would be
> able send.  It needs to explain what to do in that situation.
> 
> [mk] The following text has been added in section 8.4:
> (Implementation note:  If the number of outstanding unexpected 
> control-type PDUs equals MaxOutstandingUnexpectedPDUs, the target will not

> be able to send another unexpected control-type PDU until some outstanding

> ones have been retired.  To ensure that the initiator has a means of 
> retiring outstanding PDUs, the target should reserve the last unexpected 
> control-type PDU allowable by MaxOutstandingUnexpectedPDUs as a NOP-in 
> ping request with ITT = 0xffffffff and TTT != 0xffffffff to allow the 
> initiator to return the NOP-out ping response with the 
> current ExpStatSN.)

While that helps, it still doesn't respond to the comment's request to
explain what an iSER target should do when it wants to send a Reject PDU
but cannot due to this limitation.  Is the target required to remember
the PDU that caused the reject until it gets enough space to send a
Reject?  How much effort does it put into obtaining permission to send
the Reject before doing something else, and what is that something else?

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: Mike Ko [mailto:[email-masked]] 
> Sent: Monday, June 13, 2005 1:10 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [Ips] WG Last Call Review: iSER
> 
> David,
> 
> Thanks for the review.  Please see my comments inline. 
> 
> Mike
> Sent by:        [email-masked]
> To:     [email-masked]
> cc:      
> Subject:        [Ips] WG Last Call Review: iSER
> 
> 
> I have reviewed the iSER draft, draft-ietf-ips-iser-03.txt,
> and in addition to editorial issues that I will send
> directly to Mike, I found a number of technical issues:
> 
> (1) Section 3.3 has a structural problem in that the first
> sentence says "should satisfy" and the remaining sentences
> to be satisfied are expressed as "MUST" requirements.  The
> net result is that all the requirements in that section are
> "should" requirements, which is not what was intended.
> The first sentence needs to be rewritten to say something
> to the effect of "In order to use iSER, the following MUST
> be done:" .
> 
> [mk] ok
> 
> (2) Section 3.3, item 1. says "MUST negotiate the
> RDMAExtensions key to "Yes" "  Since the key is negotiated
> (not declared) and it takes two to negotiate, that requirement
> cannot be achieved by an implementation if the other side
> doesn't cooperate.  This needs to be rewritten to talk about
> offering the key as "Yes" and replying as "Yes" and that
> if the result is not "Yes" then iSER is not used (traditional
> iSCSI is used instead).
> 
> [mk] ok
> 
> (3) Section 3.3., item 5. says "MUST successfully negotiate
> the TargetRecvDataSegmentLength key".  Again there's a "takes
> two to negotiate" problem, and the "failed to successfully
> negotiate" case needs to be specified.
> 
> [mk] The old items 4 and 5 are now replaced by the new item 4 
> with the 
> follwowing text:
> 
> "The iSCSI Layers at both the initiator and the target MUST 
> support the 
> keys as defined in section 6 on Login/Text Operational Keys.  
> If used as 
> specified, these keys MUST NOT be answered with NotUnderstood and the 
> semantics as defined MUST be followed for each iSER-assisted 
> connection."
> 
> (4) Section 5.1, last two paragraphs have similar problems in
> that they specify required negotiation behavior without
> describing what to do when the other side doesn't cooperate
> or do what it is supposed to.  Those failure cases need to
> be covered.
> 
> [mk] The new text for comment #3 should address this issue.
> 
> (5) Section 5.1.2, in the Note at the very end of the section
> about two steps needing to be atomic, "must" should be "MUST".
> 
> [mk] ok
> 
> (6) Section 5.1.3 is missing specification and explanation of
> version negotiation and version numbers in the iSER Hello
> Exchange.  The error case instructions for when version
> negotiation fails need to be copied here in addition to their
> current location in Section 10.1.3.2 .
> 
> [mk] ok
> 
> (7) Section 7.3.9 - I believe sending of Login Request or Login
> Response in Datamover mode is always an error.  This section
> should say so and say what to do about it.
> 
> [mk] ok
> 
> (8) Sections 7.3.11 and 10.2.1 - The specified SNACK handling
> does not work because the instruction in 10.2.1 to handle SNACK
> as per RFC 3720 would require retransmission of non-existent
> Data and R2T PDUs.  I can see a couple of possible corrective
> approaches:
> a) Change Section 7.3.11 to prohibit (MUST NOT) sending SNACK
> PDUs (instead of the current SHOULD NOT) and change
> Section 10.2.1 to define reception of any SNACK PDU as
> an error.
> b) Put limits on what SNACK PDUs can be used to recover
> (Status only?) in Section 7.3.11, and modify Section
> 10.2.1 to say how to recover that and what to do when
> a prohibited SNACK is received.
> 
> [mk] The updated iSER draft will follow the suggestion in 8a.  Also 
> updated text in section 3.3 and section 8.3 on SNACK handling.
> 
> (9) Section 8.1 uses the phrase "buffer underrun" for an "out
> of buffers" condition.  That should be "buffer exhaustion".
> 
> [mk] ok
> 
> (10) Section 8.4 could result in a situation where an iSER Target
> cannot send a Reject PDU that an iSCSI Target would be
> able send.  It needs to explain what to do in that situation.
> 
> [mk] The following text has been added in section 8.4:
> (Implementation note:  If the number of outstanding unexpected 
> control-type PDUs equals MaxOutstandingUnexpectedPDUs, the 
> target will not 
> be able to send another unexpected control-type PDU until 
> some outstanding 
> ones have been retired.  To ensure that the initiator has a means of 
> retiring outstanding PDUs, the target should reserve the last 
> unexpected 
> control-type PDU allowable by MaxOutstandingUnexpectedPDUs as 
> a NOP-in 
> ping request with ITT = 0xffffffff and TTT != 0xffffffff to allow the 
> initiator to return the NOP-out ping response with the 
> current ExpStatSN.)
> 
> (11) Section 11 on Security Considerations is weak, but the
> overall approach of appealing to security considerations in
> other specifications is probably ok.  This section needs to
> be strengthened by saying that since iSER is still functionally
> iSCSI, all of the iSCSI security considerations in RFC 3720
> apply, and it should also reference the discussion of iSCSI
> security in RFC 3723.
> 
> [mk] This section now references RFC3723 and the security 
> drafts for RDDP. 
>  Both documents have also been added to the list of references.
> 
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
> 
> _______________________________________________
> Ips mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ips
> 
> 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Mon, 13 Jun 2005 16:18:28 -0700

David,

On rereading section 3, bullet 1 should have been separated out from the 
other bullets.  Bullet 1 addressed the issue where an iSER implementation 
attempts a connection with a non-iSER one, and it mandates that 
iSER-assisted mode must not be used if RDMAExtensions is negotiated to 
"No".  But the remaining bullets deal with the situation assuming 
RDMAExtensions have been negotiated to "Yes".  If the connection is to 
operate in iSER-assisted mode, then the rest of the bullets (except bullet 
1) must be followed.  If any of the other bullets are not followed, then 
the connection cannot really fall back to traditional iSCSI mode since 
RDMAExtensions have been negotiated to "Yes".  Not implementing those 
other bullets in an iSER-assisted mode is an error.  So here is the 
proposed new text for section 3.3:

---  start of replacement section 3.3
To operate in an iSER-assisted mode, the iSCSI Layers at both the 
initiator and the target MUST negotiate the RDMAExtensions key (see 
section 6.3) to "Yes" on the leading connection.  If the RDMAExtensions 
key is not negotiated to "Yes", then iSER-assisted mode MUST NOT be used. 
If the RDMAExtensons key is negotiated to "Yes" but the invocation of the 
Allocate_Connection_Resources Operational Primitive to the iSER layer 
fails, the iSCSI layer MUST fail the iSCSI Login process or terminate the 
connection as appropriate.  See section 10.1.3.1 for details.

If the RDMAExtensions key is negotiated to "Yes", the iSCSI Layer MUST 
satisfy the following protocol usage requirements from the iSER protocol:

1.      The iSCSI Layer at the initiator MUST set ExpDataSN = 0 in Task 
Management Function Requests for Task Allegiance Reassignment for 
read/bidirectional commands, so as to cause the target to send all 
unacknowledged read data.

2.      The iSCSI Layer at the target MUST always return the SCSI status 
in a separate SCSI Response PDU for read commands, i.e., there MUST NOT be 
a "phase collapse" in concluding a SCSI Read Command.

3.      The iSCSI Layers at both the initiator and the target MUST support 
the keys as defined in section 6 on Login/Text Operational Keys.  If used 
as specified, these keys MUST NOT be answered with NotUnderstood and the 
semantics as defined MUST be followed for each iSER-assisted connection.

4.      The iSCSI Layer at the initiator MUST NOT issue proactive (based 
on time-outs) SNACKs for PDUs that it presumes are lost.

---  end of replacement section 3.3

For section 8.4 (now section 8.1.2), if the iSER target wants to send a 
Reject but couldn't do so because of the limits imposed by 
MaxOutstandingUnexpectedPDUs, the PDU will be discarded.  To the 
initiator, this is similar to receiving a Reject PDU with a digest error.  
Section 6.7 of RFC3720 states that "When an initiator receives any iSCSI 
PDU with a payload digest error, it MUST discard the PDU."  Furthermore, 
"No further action is necessary for initiators if the discarded PDU is an 
unsolicited PDU (e.g., Async, Reject).  Task timeouts as in the initiator 
waiting for a command completion, or process
timeouts, as in the target waiting for a Logout, will ensure that the 
correct operational behavior will result in these cases despite the 
discarded PDU."

It is incumbent upon the initiator to declare a reasonable limit on 
MaxOutstandingUnexpectedPDUs to avoid the situation where the target has 
to discard a PDU due to lack of resources on the initiator's side.  As 
stated in section 8.1, shared buffer pools can always be used to minimize 
the risk.  I can modify the new text in section 8.1.2 as follows:

"If the number of outstanding unexpected control-type PDUs equals 
MaxOutstandingUnexpectedPDUs, the target will not be able to send another 
unexpected control-type PDU until some outstanding ones have been retired. 
 In this case the unexpected PDU will be discarded and not sent by the 
target.  Task timeouts as in the initiator waiting for a command 
completion, or process timeouts, as in the target waiting for a Logout, 
will ensure that the correct operational behavior will result in these 
cases despite the discarded PDU.

(Implementation note:  To ensure that the initiator has a means to inform 
the target that outstanding PDUs have been retired, the target should 
reserve ...  with the current ExpStatSN.)"

Mike
To:     [email-masked]
cc:     [email-masked] 
Subject:        RE: [Ips] WG Last Call Review: iSER

Mike,

Thanks,
--David

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 13 Jun 2005 18:29:02 -0700

Hi Mike,

Some comments on the text discussed in this thread.

7.3.11

Get rid of " In particular, the Proactive (Time out) SNACK MUST NOT be
issued."  It is redundant now with the MUST NOT that precedes it.

3.3 (new replacement text)

Change bullet 4 to:

4.      The iSCSI Layer at the initiator MUST NOT issue SNACKs for PDUs.

8.1.2 (new text)

Suggest changing the text to:

"If the number of outstanding unexpected control-type PDUs equals
MaxOutstandingUnexpectedPDUs, the target iSCSI layer will not be
able to send another unexpected control-type PDU until some outstanding
ones have been retired.  In this case the the iSCSI layer SHOULD
discard the unexpected PDU and allow the initiator iSCSI layer
to respond to the resulting error (the discard in the target iSCSI
layer manifests as as a lost PDU on the initiator).  Task or sequence
timeouts - as in the initiator waiting for a command completion -
will ensure that the correct operational behavior will result in these
cases despite the discarded PDU."

10.2.1 (new text)

Suggest changing it to:
"As described in section 7.3.11, SNACK PDUs are not used by the iSCSI
  Layer in the iSER-assisted mode."

11 (new text)

Suggest changing it to:
Since iSER-asisted iSCSI protocol is still functionally iSCSI from a
security considerations perspective, all of the iSCSI security
requirements as described in [RFC3720] and [RFC3723] apply.  "

Thanks.

Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
StorageWorks Division
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

Mike Ko wrote:
>>control-type PDUs equals MaxOutstandingUnexpectedPDUs, the target will 
> 
> not
> 
> 
>>be able to send another unexpected control-type PDU until some 
> 
> outstanding
> 
>

---

From: Mike Ko <[email-masked]>
Date: Tue, 14 Jun 2005 10:06:36 -0700

Mallikarjun,

Thanks for your review.  I have incorporated all of your suggestions in 
the iSER draft except for the revised text for section 8.1.2.  I have some 
problem with the discarded PDU being treated as a lost PDU by the 
initiator.  Since the target can always send the SCSI Response PDU 
(because it is expected and not bound by the MaxOutstandingUnexpectedPDUs 
limit), this will appear to the initiator as an out of order StatSN and 
section 6.8 of RFC3720 requires that "it MUST address the one or more 
missing status PDUs as described in Section 6.7 Digest Errors."  In 
section 6.7 of RFC3720, if the initiator receives a PDU with a payload 
digest error, as long as there is no header digest error, the initiator 
need not take any action if the PDU is an unexpected PDU (e.g., Async, 
Reject).  This is the desired result.  But a lost PDU would appear as a 
header digest error and RFC3720 stipulates that the iSCSI layer can close 
the connection.  This is not the desired result.  In other words, the 
initiator can live with losing a Reject PDU if it knows it is a Reject PDU 
that is lost.  Since the target knows that it is a Reject (or Async) PDU 
that it wants to send and it is the lack of buffer resources at the 
initiator that is preventing it from sending the PDU, a less destructive 
way of handling this corner case is to discard the PDU and NOT advance 
StatSN.  So here is my suggested revised text for section 8.1.2:

"If the number of outstanding unexpected control-type PDUs equals 
MaxOutstandingUnexpectedPDUs, the iSCSI layer at the target will not be 
able to send another unexpected control-type PDU until some outstanding 
ones have been retired.  In this case the the iSCSI layer at the target 
SHOULD discard the unexpected PDU and if the PDU is discarded, it SHOULD 
NOT advance StatSN.  Task timeouts as in the initiator waiting for a 
command completion, will ensure that the correct operational behavior will 
result in these cases despite the discarded PDU."

Mike

Hi Mike,

7.3.11

Change bullet 4 to:

8.1.2 (new text)

10.2.1 (new text)

Suggest changing it to:

11 (new text)

Suggest changing it to:

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

---

From: Mike Ko <[email-masked]>
Date: Tue, 14 Jun 2005 10:15:09 -0700

Mallikarjun,

Should change "SHOULD NOT advance StatSN" to "MUST NOT advance StatSN".

Mike
Sent by:        [email-masked]
To:     "Mallikarjun C." <[email-masked]>
cc:     [email-masked] 
Subject:        Re: [Ips] WG Last Call Review: iSER

Mallikarjun,

Mike

Hi Mike,

7.3.11

Change bullet 4 to:

8.1.2 (new text)

10.2.1 (new text)

Suggest changing it to:

11 (new text)

Suggest changing it to:

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 14 Jun 2005 11:57:06 -0700

Mike,

 >In
 > Reject).

You may have discovered a piece of somewhat misleading text for the 
implementer's guide.

The 6.7 text clearly says that the PDU with a payload digest error must 
be discarded - the StatSN of a discarded PDU must not be considered as 
received.  This is analogous to the target throwing away the entire 
command PDU (including the CmdSN) even when there's a payload digest 
error on the immediate data (6.7 describes this case as well).

OTOH, I see that the "no further action is necessary" piece of text is 
misleading at best, since it could imply that no status SNACKs are 
necessary when in fact they are.

I am adding it to my to-do list for the new draft.

 >Since the target knows that it is a Reject (or Async) PDU
 > StatSN.

I am inclined to suggest that we make no exceptions to cover for a 
seriously corner case, making implementations' life even more complex.

At this point, I think there are two legitimate Options for the iSCSI 
Layer to deal with this corner case -

A) Do not generate any unexpected PDUs if there's no currently
    available credit.

B) Generate the PDU as always.
	(i) Wait for the credit to become available.
	(ii) Don't wait, discard the generated PDU immediately.
	(iii) A combination thereof, wait for sometime and discard
               based on implementation considerations, e.g. resource
               pressure.

IMHO, Option A should never consume a StatSN and Option B (all its 
sub-cases) should always consume a StatSN.  I prefer this approach 
because in an Option B implementation, depending on the implementation, 
the next target-to-initiator PDU "in the line" may have already taken 
the "next number" of StatSN.

In the replacement text I suggested earlier, I was leaning towards 
"B-(ii) with StatSN consumed" semantics.  Rethinking this now, I see 
that there are wire protocol implications depending on the Option.  One 
can argue that A is better, and I'd be personally agreeable to it.

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

Mike Ko wrote:
>>other bullets.  Bullet 1 addressed the issue where an iSER
> 
> implementation
> 
>>operate in iSER-assisted mode, then the rest of the bullets (except
> 
> bullet
> 
>>key is not negotiated to "Yes", then iSER-assisted mode MUST NOT be
> 
> used.
> 
>>If the RDMAExtensons key is negotiated to "Yes" but the invocation of
> 
> the
> 
>>fails, the iSCSI layer MUST fail the iSCSI Login process or terminate
> 
> the
> 
>>satisfy the following protocol usage requirements from the iSER
> 
> protocol:
> 
>>in a separate SCSI Response PDU for read commands, i.e., there MUST NOT
> 
> be
> 
>>3.      The iSCSI Layers at both the initiator and the target MUST
> 
> support
> 
>>the keys as defined in section 6 on Login/Text Operational Keys.  If
> 
> used
> 
>>initiator, this is similar to receiving a Reject PDU with a digest
> 
> error.
> 
>>"No further action is necessary for initiators if the discarded PDU is
> 
> an
> 
>>unsolicited PDU (e.g., Async, Reject).  Task timeouts as in the
> 
> initiator
> 
>>stated in section 8.1, shared buffer pools can always be used to
> 
> minimize
> 
>>MaxOutstandingUnexpectedPDUs, the target will not be able to send
> 
> another
> 
>>unexpected control-type PDU until some outstanding ones have been
> 
> retired.
> 
>>(Implementation note:  To ensure that the initiator has a means to
> 
> inform
>

---

From: Mike Ko <[email-masked]>
Date: Tue, 14 Jun 2005 14:08:21 -0700

Mallikarjun,

For any of the option B you have listed, StatSN is advanced.  If the PDU 
is discarded, it then creates a sequence error.  Since the iSER draft 
states that SNACK PDUs are not used, creating a sequence error on purpose 
can be problematic for the initiator.  Since you are personally agreeable 
to option A, I suggest we stick with this simpler and cleaner solution for 
this corner case.  The wording "SHOULD discard" in my revised text could 
be misconstrued to imply option B (ii).  To make it clear that it is 
option A that is the intended course of action, the revised text in 
section 8.1.2 now reads as follows: 

"If the number of outstanding unexpected control-type PDUs equals 
MaxOutstandingUnexpectedPDUs, the iSCSI Layer at the target MUST NOT 
generate and send any unexpected PDU even if the PDU is used to indicate 
an iSCSI error condition (e.g., Asynchronous Message, Reject).  Task 
timeouts as in the initiator waiting for a command completion, will ensure 
that correct operational behavior will result in these cases despite the 
missing PDU."

Mike
To:     Mike Ko <[email-masked]>
cc:     [email-masked] 
Subject:        Re: [Ips] WG Last Call Review: iSER

Mike,

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

unexpected

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 15 Jun 2005 15:13:20 -0700

Mike,

I am generally ok with this except for editorials (suggested text 
below).  It had also occurred to me that the initiator-to-target 
transfer has the same upper bound issue, the draft should address that 
as well.

"While the number of outstanding unexpected control-type PDUs equals
MaxOutstandingUnexpectedPDUs, the iSCSI Layer at the target MUST NOT
generate any unexpected PDUs which otherwise it would have generated, 
even if its intent is to indicate an iSCSI error condition (e.g., 
Asynchronous Message, Reject).  Task timeouts as in the initiator 
waiting for a command completion or other connection and session level 
exceptions will ensure that correct operational behavior will result in 
these cases despite not generating the PDU."

I have a different editorial/technical on 8.1.1 -
"At any time, the number of outstanding PDUs MUST not exceed 
MaxOutstandingUnexpectedPDUs."

should be:
"At any time, the number of outstanding unexpected PDUs (i.e. immediate 
commands, and one-way NOP-Outs) MUST NOT exceed the target-advertised 
value of MaxOutstandingUnexpectedPDUs."

And a similar comment on similar text in 8.1.2.

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

Mike Ko wrote:
>>the iSER draft except for the revised text for section 8.1.2.  I have 
> 
> some
> 
>>(because it is expected and not bound by the 
> 
> MaxOutstandingUnexpectedPDUs
> 
>>header digest error and RFC3720 stipulates that the iSCSI layer can 
> 
> close
> 
>>initiator can live with losing a Reject PDU if it knows it is a Reject 
> 
> PDU
> 
>>command completion, will ensure that the correct operational behavior 
> 
> will
> 
>>>>support the keys as defined in section 6 on Login/Text Operational 
> 
> Keys.
> 
>>>>If used as specified, these keys MUST NOT be answered with 
> 
> NotUnderstood
>