Subject: iSCSI Plugfest at UNH

From: "Robert D. Russell" <[email-masked]>
Date: Wed, 18 Jul 2001 18:46:46 -0400

From:  Bob Russell     [email-masked]
        Bill Lynn       [email-masked]
        Kalman Meth     [email-masked]
        Barry Reinhold  [email-masked]

 Points that arose during the UNH Plugfest (July 16-20, 2001)
 where the Standard is unclear.

 (not in any particular order)

--------------------------------------------------------------------------------

 1. Exactly what should happen if operational parameters are sent by the
    initiator during the security phase of login?

    In section 4.2 the standard says:
    "-The iSCSI parameter negotiation (non-security parameters)
     SHOULD start only after security is established.  This should be
     performed using text commands."

    and:
    "When establishing the security context, any
    operational parameters sent before establishing a secure
    context MAY be ignored by both the target and the initiator."

    There is a problem with the meaning of "ignore" -- should the target:

        1.  not respond to these operational keys at all, or
        2.  respond with key=reject, or
        3.  respond with a valid value and then just not use these values.
            In this case, how does the initiator know whether or not the target
            is ignoring these parameters or not?

    A final note:  The standard seems to allow the initiator to
    send operational parameters when establishing the security context,
    get back a valid value from the target, and then ignore the entire
    negotiation anyway.  Is this legal?  In this case, how does
    the target know that the initiator is ignoring these parameters?

--------------------------------------------------------------------------------

 2. In section 2.8.3 the standard says "Many key=value pairs can be included
    in the Text block by separating them with null (0x00) delimiters."

    This implies that the last (or only) key=value pair is NOT terminated by
    a null delimiter.  However, all C programs naturally treat key=value pairs
    as C strings which are always terminated by a null (0x00) delimiter.
    Therefore, to avoid a source of errors and to simplify implementations,
    please change the standard to require all key=value pairs MUST be
    TERMINATED by null (0x00) delimiters.

    NOTE: if this change is NOT made, then the standard needs to address the
    issue of what to do when a receiver receives a set of key=value pairs
    that is terminated by a final NULL.  Should this be treated as a format
    error, or does that final NULL separate the last key from an empty key
    which the receiver should just ignore?

    The group consensus was unanimous that all key=value pairs MUST be
    TERMINATED by null (0x00) delimiters.

--------------------------------------------------------------------------------

 3. Many key parameters can take only "yes" or "no" as valid values.  However,
    section 1.2.4 "Text Mode Negotiation" discusses only "list" negotiation
    and "numerical" negotiation -- there is no discussion of "binary" or
    "boolean" negotiation.  However, the examples never show the use of a
    list to negotiate any of these parameters.  There are then two
    interpretations of how to negotiate these parameters:

    (a) Use the existing "list" negotiation mechanism for parameters that
        can take only "yes" or "no" as valid values.

        (1) If an initiator has a key named xxx for which it would prefer a
            "yes" setting, but can also operate correctly with a "no" setting,
            then the initiator must offer "xxx=yes,no" to the target and then
            the target must respond with "xxx=yes" if it wants to use the
            feature, "xxx=no" if it does not use the feature, "xxx=reject"
            if it does not support the feature, or "xxx=NotUnderstood" if it
            does not understand the key name xxx.

        (2) If an initiator has a key named xxx for which it can only utilize
            a "yes" setting, it offers the target "xxx=yes".  The target
            must respond with "xxx=yes" if it can also use this feature,
            "xxx=reject" if it either cannot use or does not support this
            feature, or "xxx=NotUnderstood" if it does not understand the
            key name xxx.  The reply "xxx=no" is explicitly disallowed.

        This is a very simple way to do this negotiation, and does not require
        any new mechanism in either the standard or an implementation that
        already does list negotiation.  However, the standard should
        explicitly show an example similar to one of those just shown to make
        it clear that this is the way to do it.

    (b) Invent a new "binary" or "boolean" negotiation.  This would require
        new wording in the standard and probably new code in addition to
        the "list" and "numeric" negotiation implementation code.

        In particular, with this method there is no obvious way for the
        initiator to indicate to the target that it can use either "yes" or
        "no" but would prefer "no", for example, because sending the
        string "xxx=no" could be interpreted either as "only no" or
        "no prefered but yes is also ok".  This gets complicated by
        the default key values.  If the required default is "yes", then
        offering "no" would appear to mean "no is prefered bu yes is also ok".
        If the required default is "no", then offering "no" would not appear
        to be necessary, (but is currently not explicitly prohibited and
        may be useful for implementations that simply send all keys) and
        would appear to mean "only no".

    The group consensus seemed to be that solution (a) is by far the simplest
    and therefore is prefered.

--------------------------------------------------------------------------------

 4. It is not clear from the standard whether or not the value in the
    CmdSN field of an initial login command for a session (i.e., TSID=0) is
    "consumed" by its appearance in the login command or not.  For example,
    suppose an intiator sends a login command with TSID=0 and CmdSN=123
    and then waits for a response from the target.  Should the Login Response
    from the target contain ExpCmdSN=123 or ExpCmdSN=124?  Similarly, should
    the next command sent by the initiator after this login contain CmdSN=123
    or CmdSN=124?

    The consensus seemed to be that ExpCmdSN=124 in the Login Response and
    CmdS124 in the next command, but the standard needs to make it explicit,
    perhaps with an example such as the one just given.

    This is being dealt with in a mailing list thread.

--------------------------------------------------------------------------------

 5. Analagous with point 4, it is not clear how InitStatSN is handled
    by the login phase.  If a login response contains a value of 123
    in its InitStatSN field, is the value of the next response from the
    same target 123 or 124?  For consistency with the handling of CmdSN
    it would appear that 124 is correct.  However, this situation may
    be different because the field is called "InitStatSN", not "StatSN"
    as it is in other responses.

    At the plugfest the consensus was to use the value 124 for now,
    but that 123 MIGHT be better long term.

    A second issue is the fact that there can be two login responses
    in response to a login (the "login partial response" and the
    "login final response"), each of which carries an InitStatSN field.
    It is not clear if the value of the InitStatSN field in a login
    final response needs to be part of the sequence established by
    the InitStatSN field in a preceding login partial response, or whether
    the login final response can restart the sequence to an arbitrary value
    supplied in its InitStatSN field.  The standard should make this
    explicit.

    At the plugfest the consensus was that the final response CAN
    restart the sequence.

    This is being dealt with in a mailing list thread.

--------------------------------------------------------------------------------

 6. During text mode negotiation, if a format error occurs, what is the proper
    response?  Section 6.1 "Format Errors" does not handle all cases.

    (a) During login, suppose the initiator offers a list and the target
        responds with a value that is not in the list (and is not "reject"
        or "Not Understood").  Is the initiator required to close the
        connection or can it just internally recover from the error
        (perhaps reporting the error through the "appropriate service
        response") and continue with the login?

    (b) During full-feature phase, suppose the initiator sends a text
        command that offers a list and the target responds with a value that
        is not in the list (and is not "reject" or "NotUnderstood").
        Section 7.3 deals only with the situation when there is an
        outstanding task related to the PDU, but in this case there is
        no outstanding task.  Therefore, is the proper response to just
        report the error through an appropriate service response
        and then ignore it?

--------------------------------------------------------------------------------

 8. The last example in Appendix A "iSCSI Security and Integrity"
    illustrates a special case that should be elimiated.
    The last sentence of this example says:

    "Note that no SecurityContextComplete=yes is required since no
    security mechanism was chosen."

    But it would be simpler if the exchange of "SecurityContextComplete=yes"
    keys were required in this case.

    The reason they should be required is the following:

    Appendix B. starts by saying that "the parameters (keys)
    negotiated for security are:
        - Digests (HeaderDigest, DataDigest)
        - Authentication method (AuthMethod)"

    Thus as soon as the Initiator offers either the HeaderDigest or
    the DataDigest key, the security negotiation phase has been entered.

    Section 4.2 "Security and Integrity Negotiation" clearly indicates
    that the security sub-phase ends only when both sides have
    correctly executed the "SecurityContextComplete=yes" handshake.
    There is NO qualification in section 4.2 about the outcome of
    the negotiation -- i.e., one would reasonably conclude that
    selecting no digests and no authorization still requires the handshake,
    because the final selections were acheived through negotiation, and the
    handshake marks the end of the security negotiation subphase.

    The phrase in the standard that makes this example "legal" is in
    section 4.3, which says:
    "Operational parameter negotiation during the login MAY be done:
        . . .
    - starting immediately after the Login response with Final bit
    0 if the initiator does offer security/integrity options but
    the target chose none."

    This clause overrides the rules of section 4.2 with a special case.
    Why bother with this special case?  It complicates the logic in
    implementations, requires extra wording in the standard, and appears
    to offer no benefit.  If it stays, there should be some indication
    in section 4.2 about it, or better yet, combine these sections so
    all this information is together in one place.  As it is now, this is
    an error trap for implementors of both targets and initiators.

--------------------------------------------------------------------------------

 9. the SCSI Command (section 2.3) contains R and W bits and an
    "Expected Data Transfer Length".  The standard says in section 2.3.1:
    "b6 (R) set to 1 when input data is expected
     b5 (W) set to 1 when output data is expected"

     and in section 2.3.5:
     "the Expected Data Transfer Length field states the number of bytes
     of data involved in this SCSI operation."
     It then goes on to describe 2 cases:
        - W=1 with R=0,
        - W=0 with R=1.

     The problem is that some SCSI commands involve no data transfer
     (Test Unit Ready is an example).  So in this case, the
     Expected Data Transfer Length field is set to 0.
     How should the R and W bits be set?
        - Should they both be set to 0?
        - are they "don't care" bits so that their value can be anything?
          (In the Test Unit Ready example, some implementations will set
          the R bit because the SCSI implementation interfacing with the
          iSCSI code uses a single bit to differentiate between read and
          write commands, and considers this a read command).

     The standard should make this explicit.

--------------------------------------------------------------------------------

 10.Another problem related to setting the F bit on SCSI commands.
    Section 2.3 says:

    "b7 (F) set to 1 when no unsolicited SCSI Data-Out PDUs follow
     this PDU.  For a write, if Expected Data Transfer Length is
     larger than the Length the target may solicit additional data
     through R2T."

    This does not explicitly say what happens on a read or on a
    command other than a write (point 9 above) or on a write with
    an expected data transfer length of 0.  These commands do
    not allow unsolicited SCSI Data-Out PDUs to follow them, so one
    could infer that the F bit should be set to 1.  However, the present
    wording can also be interpreted that because unsolicited SCSI
    Data-Out PDUs are not allowed to follow these commands, this
    explanation does not apply to them.

    The standard should explicitly state what happens in the case of
    these commands, rather than leave it up to an implication.

--------------------------------------------------------------------------------

 11.In section 1.2.3 iSCSI Login there is the paragraph:

    "The login PDU includes a session ID that is composed of an initiator
    part ISID and a target part TSID.  For a new session, the TSID is
    null.  As part of the response, the target generates a TSID.
 >> Session specific parameters can be specified only for the first login of a
 >> session (TSID null) (e.g., the maximum number of connections that can
 >> be used for this session).  Connection specific parameters, if any,
 >> can be specified for any login.  Thus, a session is operational once
    it has at least one connection."

    The wording of the 2 sentences indicated by ">>" needs to be changed
    to explicitly indicate login phase, since as now worded in could
    be literally interpreted to mean login command only.  Proposed
    rewording for those sentences:
    "Session specific parameters can be specified only during the login phase
    begun by a login command containing a null TSID (e.g., the maximum
    number of connections that can be used for this session).  Connection
    specific parameters, if any, can be specified during the login phase
    begun by any login command.

--------------------------------------------------------------------------------

 12.If unsolicited data is negotiated, it appears that it still does not
    have to be used.  In practice this can lead to performance inefficiencies.

    Consider the following:
        - Unsolicited data has been negotiated to YES but immediate data
          is NO (although a problem still exists even if this is YES).

        - The initiator sends a SCSI command with F=0, W=1 and a large
          Expected Transfer Length field (greater than FirstBurstSize).
        
        - When the target receives this command, it knows that unsolicited
          data follows (because F=0), but does NOT know how much (there
          is a maximum determined by the negotiated FirstBurstSize, but there
          is no requirement that the initiator actually send this much).
          Therefore, the target cannot at this time send out an R2T to get
          the "rest" of the data --  it must wait to receive data PDUs
          until it gets the F=1 set on one of these data PDUs, and then
          compute the amount of data to ask for in the R2T.  This may
          introduce needless delay.

          To avoid this situation, the standard should either:
          - require that when unsolicited data is to be sent, that the
            amount be min(Expected Transfer Length, FirstBurstSize),
          - alternatively, provide a field in the SCSI command that
            allows the initiator to indicate how much unsolicited data
            follows.

--------------------------------------------------------------------------------

 13.Some keys are dependent on other keys. For example, the keys that
    determine the actual marker intervals (RFMarkInt, SFMarkInt) have
    meaning only if markers are in fact being used, as determined by
    the FMarker key.  Typically an initiator will offer these
    three keys in the same message.  If the target responds with
    FMarker=no, does it need to respond to the RFMarkInt and SFMarkInt
    keys, or can it just omit those from the response, since whatever
    values they contain will be meaningless anyway?

--------------------------------------------------------------------------------

 14.The login phase involves many combinations and permutations that
    need to be clarified through a state diagram incorporated into
    the standard.  One example is the situation that arises when an
    initiator sends a login command with F=1 and operational parameters
    but no security parameters (because the initiator does not want to
    utilize any security).  The target, however, wishes to use security.
    There appear to be several valid responses the target can give:

    (a) It can reply with a login final response, F=1, Reject Code of
        201 (Authentication failed) and then disconnect.
    (b) It can reply with a login partial response, F=0, status code
        1 (if the login included the ITN) or 2 (if login did not include
        ITN) and offer its own security parameters.

    The consensus seems to be that login in its current form is too
    complex and needs to be simplified considerably.

    One possibility would be to standardize one or a small number of simple,
    "fast-path" logins that involve no (or very limited) negotiation and
    that together would cover the majority of situations.

--------------------------------------------------------------------------------

 15.The issue of case sensitivity in names arose, and is being dealt with
    by a mailing list thread.  There also seems to be cases where the
    Naming and Discovery document is in conflict with the iSCSI Standard.

--------------------------------------------------------------------------------

 16.The issue of using a target name of "iscsi" for both discovery sessions
    and "simple" devices arose, and is being dealt with by a mailing list
    thread and new text in draft 6-98, which was not available at the time
    of the plugfest.

--------------------------------------------------------------------------------

 17.Can the option "none" be offered as the first element in a key list
    if that is in fact the prefered option?  The present wording in
    section 1.2.4 says nothing about where in the list it can be offered,
    so by implication it can be anywhere.  However, the standard should be
    explicit on this point, since all examples (and apparently some
    earlier drafts required) indicate "none" as the last option offered.

--------------------------------------------------------------------------------

 18.The login process is incredibly difficult to understand from the
    current standard.  There are three major obstacles:

    (1) The information is spread out in at least 5 major sections of
        the document (section 1.2.3, section 2.3 and 2.4, section 4,
        Appendix A, and Appendix D.)

    (2) There is no state diagram to bring all this information together.

    (3) It is too complex in its current form.

--------------------------------------------------------------------------------

 19.In Appendix A the Standard defines the crc-32C generator polynomial
    to be used by iSCSI but gives no example of its use.  On the mailing list
    several people worked out an example with actual data and gave the actual
    result of the algorithm.  Also on the mailing list several people
    contributed code to perform this computation.  This information (the
    detailed worked out example and the code) should be made part of the
    standard in order to eliminate any sources of confusion and to allow
    implementors to check their own implementations against a reference.

--------------------------------------------------------------------------------

 20.This is just an observation -- the iSCSI response sent by a target
    contains a Data Segment Length field and requires the SCSI response
    to carry sense and response data in the PDU itself.  Any I/O data
    produced by the SCSI command itself should be carried in a separate
    Data-In command, not in the iSCSI response.  This should be made
    explicit in the standard, especially because in earlier drafts it
    was apparently possible to do this.  The group at the plugfest
    unanimously agrees that SCSI I/O data not be allowed in the iSCSI
    response, but would just like it to be stated explicitly in the standard.

--------------------------------------------------------------------------------

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 19 Jul 2001 19:34:27 +0300

+++ in text Thanks, Julo

"Robert D. Russell" <[email-masked]> on 19-07-2001 17:58:47

Please respond to "Robert D. Russell" <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   ips <[email-masked]>
Subject:  Re: iSCSI Plugfest at UNH

Julian:

Thank you for changing the last example in Appendix A to require the
SecurityContextComplete=yes handshake, and adding the phrase to 4.2:
"The Security Context Complete handshake MUST be performed if any of
the negotiating parties has offered a security/integrity item (even if it
is not selected)."

As part of this same change, you need to also delete the 3rd point in
section 4.3 quoted below (deletion marked with X):

>
>    Operational parameter negotiation during the login MAY be done:
>
>     - starting with the Login command if the initiator does not offer
>     any security/ integrity option
>     - starting immediately after the security/integrity negotiation if
>     the initiator and target perform such a negotiation
>  X  - starting immediately after the Login response with Final bit 0 if
>  X  the initiator does offer  security/integrity options but the target
>  X  chose none.

------
+++ Done +++
>
> +++ OK  (that is from the plugfest or personal?)+++

It acually came up -- someone was arguing that as stated he had to include
all session sepcific parameters in the first login command (literally).

-------

refering to section 1.2.4:

>  What do we have add. I assume that if we add more text the amount of
>  overlooked text will only increase +++

What about the wording:

   In "list" negotiation, the offering party sends for each key a list of
   values (which may include "none") in its order of preference.

+++ if that helps +++
Bob

---

From: [email-masked] (Sandeep Joshi)
Date: Thu, 19 Jul 2001 13:17:39 -0400 (EDT)

> > +++ Should we really? A good driver will do what it can to optimize its use
> > of resources while getting maximum performance. IMHO a warning to
> > implementers will be enough and I will try to put it in.++++

Julian,

Matthew Burnbridge explained this well at the plugfest meeting.

The problem is that the target does not know if its dealing
with a good initiator or a bad one.   The protocol is introducing
needless network delay (wait states!).

Why wait for the final bit if the target has enough memory to 
handle all the data right away (and could send R2Ts *NOW*) ?   

Regards,
-Sandeep

> > +++ Should we really? A good driver will do what it can to optimize its use
> > of resources while getting maximum performance. IMHO a warning to
> > implementers will be enough and I will try to put it in.++++