Subject: iSCSI: current UNH Plugfest

From: "Robert D. Russell" <[email-masked]>
Date: Mon, 29 Oct 2001 19:44:12 -0500

Attached are some of the issues that arose during the
first day of the iSCSI plugfest at UNH on Monday 29-Oct-2001.

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

----------------------------------------------------------------------------

1. Situation: the very first command sent on a new connection is a
   login.  However, the I bit is not set.  What should the target do?

   Interpretation 1:
   According to section 8.3 page 127 of draft 8,
   "Explicit violations of the PDU layout rules stated in this document
   are format errors."
   "When a target or an initiator receives an iSCSI PDU with a format
   error, it MUST reset all transport connections in the session
   immediately ..."
   This is a format error so section 8.3 applies.  Therefore, the target
   should simply close the connection without sending anything to the
   initiator.

   Interpretation 2:
   This is a login command that contains an error caused by the
   initiator.  Therefore, the target should send back a login response
   with a status code of 0x0200 and then close the TCP connection.

2. Situation: on the first login command in operational parameter negotiation
   stage, the initiator sends the keys InitialR2T=yes and ImmediateData=no
   and the target agrees with these values.  Therefore, according to the
   table on page 182 of draft 8, both sides agree that unsolicited data
   is disallowed.  These keys are then followed, either later in the keys
   associated with this same login command or in a subsequent login command
   belonging to the same login phase, with the key FirstBurstSize=6.
   What should the target do?

   Interpretation 1:
   FirstBurstSize indicates "the maximum amount of unsolicited data an
   iSCSI initiator may send to the target during the execution of a
   single SCSI command, in units of 512 bytes."  However unsolicited data
   is now disallowed, so FirstBurstSize is irrelevant.  Therefore, no
   response is necessary -- the key is just ignored.

   Interpretation 2:
   In section 2.2.4 on page 30 of draft 8 it says "For numerical
   negotiations, the responding party MUST respond with the required key."
   Since FirstBurstSize is a numerical key, the target MUST respond with
   a numeric value ("reject" is not allowed for numerical keys), even if
   the value is irrelevant and will not be used.

3. Situation: As the first command on a new connection the initiator
   sends a valid login command with SessionType=discovery, CSG=0, NSG=3,
   and T=1.  The target responds with CSG=0, NSG=1 and T=1.  So both
   sides are now in the operational parameter negotiation stage.
   Question: what parameters can they negotiate in this stage for a
   discovery session, and what should be done if other, irrelevant,
   parameters are negotiated?

   Interpretation 1:
   The only relevant operational parameters in a discovery session seem to
   be those related to markers (FMarker, RFMarkInt, SFMarkInt), error
   recovery (ErrorRecoveryLevel) and vendor specific keys.
   If this is true, what should one side do when the other side attempts
   to negotiate other operational parameters in a discovery session?

   Interpretation 2:
   Even the relevant operational parameters in a discovery session given
   above have very marginal utility.  It would be easier to just
   disallow all operational parameter negotiation in a discovery session.

4. Contradiction in draft 8.
   In section 3.13.5 on page 92 it says that in a login response:
   "for a non-zero Status-Class the T, CSG & NSG fields MUST be 0."

   In section 5.1 on page 112 it says that in a login response with
   login reject (i.e., Status-Class non-zero): "The T bit of the
   response MUST be set to 1 and the CSG and NSG are ignored."

   One of these has to be changed.  Note also that the second statement
   just given (from page 112) is inconsistent with the statement 
   in section 3.12.4 on page 87 that for login requests:
   "the next stage value is valid only when the T bit is 1 and is
   reserved otherwise."  One would expect the standard to have the
   same rule for the way the T, NSG and CSG fields are handled in
   both login request and login response.

5. Clarification suggested:
   In section 3.13.4 on page 91 of draft 8 it says:  "For the first
   Login Response (the response to the first Login Request) this is
   the starting status Sequence Number for the connection (the next
   response of any kind will carry this number + 1)."
   If the login phase requires more than one login-request/login-response
   exchange, this rule implies that the second (third, fourth, ...)
   login-response should be incrementing StatSN.
   Nobody was doing this, although it seems clear that the standard requires
   it.  Perhaps there needs to be a slight addition to the wording of this
   section so it reads:
   "For the first Login Response (the response to the first Login Request)
   this is the starting status Sequence Number for the connection (the next
   response of any kind, including the next login response if any in the
   same login phase, will carry this number + 1)."

6. Situations where frequent errors or misunderstandings occured, although
   the draft seems clear:
   1.  - StatSN must be incremented in Login Phase (point 5 above)
   2.  - CmdSN must not be incremented CmdSN in Login Phase
   3.  - it is okay to transmit the first command of the Login Phase
         with the CSG set to 1. It is also okay for the target to reject it
         and to close the connection.  
   4.  - If the Initiator transmits its first command of the Login Phase
         with the T bit set, its okay for the target to transmit a response
         with T = 0, even though there is no example in the draft
         demonstrating it. 
   5.  - Stage Transitions from 0 to 3 are valid according to the spec
         although this is not specifically stated in the draft.

---

From: "Robert D. Russell" <[email-masked]>
Date: Tue, 30 Oct 2001 19:06:03 -0500

Attached are some new issues that arose during the iSCSI plugfest at
UNH on Tuesday 29-Oct-2001.
(Note: these issues do not take into account any modifications or
clarifications that occured in the standard due to the issues raised
on Monday.)

Bob Russell
InterOperability Lab

-----------------------------------------------------------------------------

1. Situation: as the first command on a new TCP connection, the initiator
   sends a login with T=1, CSG=1, NSG=3, and valid InitiatorName, TargetName,
   and SessionType keys.  However, there is also a valid key having an
   invalid value, such as MaxConnections=abcd (i.e., not a number after the
   '=') or MaxConnections4 (i.e., missing the '=').
   What should the target do?

   Interpretation 1:

   According to section 3.10.4 page 82 of draft 8 (page 83 of draft 8a),
      "Any other key not understood by the target may be ignored by the
      target without affecting basic function.  However the Text Response
      for a key that was not understood MUST be key=NotUnderstood."

   Two things have to be clarified:
   1. Does this section also apply to keys received in a login?
   2. Can "NotUnderstood" also apply to "values of keys" that are not
      understood, even if the key word itself is understood?

   If the answer to these 2 of these questions is "yes", then the appropriate
   response would seem to be for the target to just ignore the key and send
   back MaxConnections=NotUnderstood as part of its next login response.

   Interpratation 2:

   According to section 8.7 page 129 of draft 8 (page 130 of draft 8a),
      "A negotiation failure is considered one or both of the following:
      - None of the choices or the stated value is acceptable to one
      negotiating side. ..."
   Clearly this stated value ("abcd") is not acceptable to the target.
   Therefore, the following rule on page 129, draft 8 (page 130, draft 8a)
   applies:
      "- During Login, any failure in negotiation MUST be considered
      as the login process failure and the connection must be dropped."

   Therefore, the target should just drop the connection without sending
   any login response back to the initiator.

   Interpretation 3:

2. Situation: on the first login command in operational parameter negotiation
   stage, the initiator sends no operational keys, thereby telling the target
   that it accepts all the default values for these keys.  However, the target
   wants to negotiate the value of MaxConnections, so in the login response
   it sends back "MaxConnections=3" (for example).  Should the initiator
   send a response to this key or not? 

   The statement in section 2.2.4 on page 30 of draft 8 and 8a:
   "For numerical (and single literal) negotiations, the responding party MUST
   respond with the required key.(...)"
   makes it clear that the responding party MUST respond.  However, in this
   situation, it is not clear who the responding party is.

   Interpretation 1:

   By not explicitly sending this key in the login command, the initiator
   is implicitly offering the default value and therefore is the offering
   party and the target is the responding party.  The conclusion is that
   the initiator does not have to send a response to this key from the
   target.

   Interpretation 2:

   The target is the offering party because it is the party that explicitly
   stated the key for the first time during these negotiations.  The
   conclusion is that the initiator MUST send a response to this key from the
   target.

   NOTE 1: If interpretation 1 is correct, it would seem to imply that the
   target MUST respond to every key whether or not it is present in the
   login from the initiator, even if it does not want to change the default
   value.  The reason is that a missing key is an implicit offer of the
   default value, and the responding party MUST respond.  Is this a correct
   interpretation?

   NOTE 2: The following statements in section 2.2.4 page 29 of draft 8a:

        Originator-> <key>=<valuex>
        Responder-> <key>=<valuey>|NOtUnderstood

   seem to imply that the originator is the party (initiator or target) that
   explicitly offers a key, and that omitting a key is not an implicit offer
   of that key with the default value.  However, even in the revised draft 8a
   there is no definition of "Originator" and/or "Responder" that would
   make this clear.  Adding to the standard these definitions, and an
   explicit statement that "a missing key does not constitute an implicit
   offer of the default" would help eliminate misunderstandings.  In
   addition, including an example of this situation (where an initiator
   omits a key and the target offers the key) would be a big help.

3. Some of the login phase examples given in Appendix A of both draft 8 and
   8a do not follow the rule in section 3.12.4 page 87 of draft 8 (page 88
   of draft 8a):
      "The next stage value is valid only when the T bit is 1 and is
      reserved otherwise."
   and the rule in section 3 page 48 of draft 8 (page 49 of draft 8a):
      "Any reserved fields and values MUST be 0 unless specified otherwise."

   If these rules are applied, all examples having T=0 should also
   have NSG=0.  Presently all of them with T=0 also have NSG=1 or NSG=3.

4. Situation: The initiator and target have successfully completed the
   login phase for a discovery session and are now in full feature phase.
   The initiator sends a text command containing the single key:
   "SendTargets=".  What response is expected from the target?

   Interpretation 1:

   According to the explanation on page 188 of draft 8 (page 189 of
   draft 8a):
   "If no target name is specified, the session should respond with
   addresses only for the target to which the session is logged in.
   This MUST be supported on operational sessions, and MAY NOT
   return targets other than the one to which the session is logged in."

   However, for a discovery session there is no target per se (the
   initiator does not specify a TargetName= during login), so the
   target therefore follows the rule on page 188 of draft 8 (page 189
   of draft 8a):
   "A SendTargets response MAY contain no target names, if there are no
   targets for the requesting initiator to access."
   and sends back a Text Response with no keys in it.

   Interpretation 2:

   In a discovery session, the key "SendTargets=" makes no sense and should
   be treated by the target in the same manner as the key "SendTargets=all".

5. Some common error situations:

   1) - when a SCSI Response contains a CHECK CONDITION (Status=0x02),
      some targets are not including the SenseLength as the first 2
      bytes in the data segment.  Although the format of the data segment
      is clear from the diagram in section 3.4.6 on page 62 of draft 8
      (page 63 of draft 8a), the last entry in the diagram for the SCSI
      Response PDU on page 58 of draft 8 (page 59 of draft 8a) is
      misleading because it mentions only the Sense Data and Response
      Data and omits the Sense Length.  It would therefore be helpful
      if the last entry in the diagram on page 58 were changed to explicitly
      reference the diagram on page 62, as in:

         Data Segment -- see section 3.4.6 (optional)

   2) - after sending a CmdSN on an initial login, some initiators are
      incrementing it before sending their first non-immediate command.
      (i.e., if the initial login contains CmdSN=123, they are sending
      CmdSN=124 on the first non-immediate command after the login phase).
      Section 3.12.10 on page 89 of draft 8 (page 90 of draft 8a) is
      clear that in this example the first non-immediate command should
      carry CmdSN=123, not 124.  This was an issue at the July plugfest
      and apparently some implementations have not been updated to conform
      to the draft 8 standard in their handling of CmdSN.

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 31 Oct 2001 10:41:08 +0200

Good point - Julo

Steve Senum <[email-masked]>
Sent by: [email-masked]
30-10-01 21:05
Please respond to Steve Senum

        To:     ietf-ips <[email-masked]>
        cc: 
        Subject:        Re: iSCSI: current UNH Plugfest

Julian,

Julian Satran wrote:

> For numerical and single literal negotiations, the responding party MUST
> respond with the required key and the value it selects, based on the
> selection rule specific to the key, becomes the negotiation result.  If 
a
> key is irrelevant in the current negotiation, the responder may use the
> special numeric constant 0r.
> Selection of a value not admissible under the selection rules is
> considered a protocol error and handled accordingly.

I don't see the point or using '0r' instead of
'irrelevant' like the list and boolean types do.
Even numeric types have to check for 'NotUnderstood',
a decidedly non numeric value.

Steve Senum

---

From: "Robert D. Russell" <[email-masked]>
Date: Wed, 31 Oct 2001 17:39:11 -0500

Attached are the new issues that arose during the iSCSI plugfest
at UNH on Wednesday 31-Oct-2001.

on Monday or Tuesday.)

Bob Russell
InterOperability Lab

1. Are receivers (initiator or target) REQUIRED to check that reserved
   bits and/or fields are set to 0?

   Section 3 on page 48 of draft 8 says:
     "Any bits not defined MUST be set to zero.  Any reserved fields and
     values MUST be 0 unless specified otherwise."

   and Section 8.3 on page 127 of draft 8 says:
     "Explicit violations of the PDU layout rules stated in this document
     are format errors.  This when detected, usually indicates a major
     implementation flaw in one of the parties.

     When a target or an initiator receives an iSCSI PDU with a format
     error, it MUST reset all transport connections in the session
     immediately and escalate the format error to session recovery
     (section 8.11.4)."

   According to these rules, a PDU with reserved bits and/or fields that
   are not set to 0 violates the PDU layout rules.  Therefore, if an
   initiator or target receives such a PDU, it should immediately close
   all connections in the session and go to session recovery.

   Clearly a format error has extremely severe consequences!

   Although all vendors are setting reserved bits and fields to 0 on
   PDUs they are sending, many are NOT checking PDUs they are receiving
   to see if these bits and fields are set to 0.  Basically, vendors are
   saying "who cares if reserved bits and/or fields in incoming PDUs are
   not zero?  We do not want to take the time to do this checking, and
   there is no benefit to doing it.  As long as the non-reserved bits and
   fields are set properly, we can and should proceed.  Any time devoted
   to doing this checking is wasted in 99+% of the cases, and in the
   (unlikely) case that a non-zero bit or field is found, the
   consequences are too severe."

   There should be some statement in the standard to clarify what checking
   is required and what is optional.

2. A similar situation arises with respect to checking the consistency
   of fields such as Version-max, Version-min and Version-active in Login
   Requests and Login Responses.

   For example, consider the Version-max field.

   Section 3.12.5 says:
     "All Login requests within the Login phase MUST carry the same
     Version-max."

   All vendor initiators are setting Version-max correctly on all
   login requests they are sending, but many vendor targets are NOT
   checking received login requests to ensure that this rule is enforced.
   In particular, many targets simply use the Version-max and Version-min
   on the first login request they receive on a new connection, and then
   they ignore these fields on all subsequent login requests in the same
   login phase.

   Strictly speaking, a change in the Version-max field during the login
   phase constitutes a protocol error according to section 8.8 on page 130
   of draft 8:

     "All violations of iSCSI PDU exchange sequences specified in this
     draft are also protocol errors.  This category of errors can be
     addressed only by fixing the implementations; iSCSI defines Reject
     and response codes to enable this".

   Therefore the target should send back a login response with a status
   of 0x0200 and then close the connection.

   However, Section 3.12.5 also says:
     "The target MUST use the value presented with the first login request."

   This rule seems to imply that the value CAN change, because if it cannot
   change, then it doesn't matter which one of the login requests it is
   taken from, they are all the same anyway.

   The suggestion is to keep the requirement that the target MUST use the
   value presented on the first login request, but to allowed the target
   to ignore the value presented on all subsequent login requests in the
   same login phase.  A similar rewording should be done for the other
   fields.

3. Can commands be sent out of order on the same connection?

   The behavior of targets is clearly specified in Section 2.2.2.3 on
   page 25 of draft 8, which says:
     "Except for the commands marked for immediate delivery the iSCSI
     target layer MUST eliver the commands for execution in the order
     specified by CmdSN."

   Section 2.2.2.3 on page 26 of draft 8 also says:
     "- CmdSN - the current command Sequence Number advanced by 1 on
     each command shipped except for commands marked for immediate
     delivery."
   but the meaning of the term "shipped" is vague, and does not necessarily
   require that the PDUs arrive on the other end of a TCP connection
   in the same order that the CmdSN values were assigned to these PDUs.

   Some initiators have been designed to send commands out of CmdSN
   order on one connection.  Consider the situation where there is only
   one connection and a high-level dispatcher creates a PDU for a SCSI
   command that involves writing immediate data to the target.  This PDU
   is enqueued to a lower-level layer which has to setup, start, and
   wait-for a DMA operation to move the immediate data into an onboard
   buffer before the PDU can be put onto the wire.  While this is
   happening, the dispatcher creates another unrelated PDU for a SCSI
   read command (for example), and when this PDU is passed to the
   lower-level layer it can be sent immediately, ahead of the previous
   write PDU and therefore out of order on this connection.

   The standard clearly allows this to happen if the two PDUs were sent
   on different connections, and seems to imply that this can also happen
   when the two PDUs are sent on the same connection.

   The suggestion is to put in the standard an explicit statement that
   this is allowed or not allowed, as appropriate.
   
   If this is allowed, such a statement would avoid the erroneous
   assumption being made by some target implementers that within a single
   connection, commands will arrive in order.

   If this is not allowed, such a statement would avoid the erroneous
   assumption being made by some initiator implementers that within a
   single connection, commands can be put on the wire out of order.

4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize)
   now allow: "A value of 0 indicates no limit."

   Is this useful?  Does it buy anything?

   The difficulties implementers are having with this are:

   1) It is a special case.
   2) It causes discontinuous ranges (for example, [0,64..2**24])
   3) It violates the min/max function normally used for the key.
   4) There is always a limit anyway.

   Consider FirstBurstSize, which can have a value that is described
   as "<0|number-64-2**24>", and for which the minimum of the 2 numbers
   is selected.

   I one side offers 0 to mean unlimited, and the other side
   has a limit, it will reply with that limit, say 128 Kbytes.
   Therefore, the result is not min(0, 128K) but rather max(0, 128K).
   The statement in the standard that "the minimum of the 2 numbers is
   selected" is therefore wrong when one of the numbers is 0.

   Furthermore, when an initiator or target receives an offer for one
   of these keys, it cannot simply check that the offered value is
   legal by testing it against some minimum and maximum.  It must first
   check for 0 and then only if that check shows the value is non-zero
   can it do the min/max range check for legality (i.e., 64-2**24).

   Finally, there is always a limit. If nothing else it is the
   limit imposed by the 24-bit DataSegmentLength field of the PDU
   requesting the transfer.  It is useless to specify a FirstBurstSize
   (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because
   the largest possible DataSegmentLength in any PDU that can use
   this value is 2**24-1.

   The suggestion is to just eliminate this special case of 0 and require
   that the range 64-to-(2**24-1) be used instead -- it has exactly the
   same effect in all cases, it is easier to describe in the standard
   because it avoids all the extra words, and it is easier to code
   because it avoids all the special cases.

   NOTE: the standard should specify the limit in the ranges for
   MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead
   of 2**24.  The number 2**24 cannot be represented in the 24-bit
   DataSegmentLength field and therefore can never be used.

5. This is a suggestion for a minor rewording in the standard to avoid
   misunderstandings.

   In Appendix E on page 188 of draft 8 it says:

     "The response to this command is a text response containing a list of
     targets and their addresses.  Each target is returned as a target
     record.  A target record begins with the TargetName text key,
     followed by a list of TargetAddress text keys, ..."

   In fact, there are situations where there are no targets and/or no
   addresses.  These situations are clearly defined in the draft after
   the sentences quoted above, but it would help if those sentences
   at least hinted at the possibility that the lists could be empty
   or might not contain addresses.  A possible rewording would be:

     "The response to this command is a text response containing a list of
     zero or more targets and, optionally, their addresses.  Each target
     is returned as a target record.  A target record begins with the
     TargetName text key, followed by a list of zero or more
     TargetAddress text keys, ..."

6. This is a suggestion for another very minor rewording in the standard.

   At the end of section 2.2.3 on page 29 of draft 8 it says:

     "Before full feature phase is established, only Login PDUs are
     allowed. ..."

   The suggested rewording is:

     "Before full feature phase is established, only Login Request and
     Login Response PDUs are allowed. ..."

---

From: "Anshul Chadda" <[email-masked]>
Date: Wed, 31 Oct 2001 19:11:27 -0500

My intent in making the MUST requirement is to make target send the sense
data along if it is setting the CHECK CONDITION in SCSI Response PDU (to
support and enable AutoSense).

 If we are making the iSCSI standard tight enough to make target "support
and enable AutoSense" (Clause 3.4.6, para 1 in draft 8), why not make a
requirement for target to send sense data if CHECK Condition is set (to make
fully tight).

 I am not even addressing situations that can happen on the initiator side.
Also, it is upto the initiator to handle a situation where SCSI Response PDU
has CHECK CONDITION set but no sense data with it. There are many reasons
why this can happen, one reason can be data digest error as you pointed out.
And the MUST requirement should not make the initiator assume things if the
SCSI Response PDU has the CHECK condition set.

 Regards,
Anshul

 ----- Original Message -----
 From: "Santosh Rao" <[email-masked]>
 To: <[email-masked]>
 Sent: Wednesday, October 31, 2001 4:41 PM
 Subject: Re: iSCSI: current UNH Plugfest

> > Robert Snively wrote:
 > >
 > > Sense Data will never be generated in a data segment to any command
 except
 > > REQUEST SENSE.  In that case, CHECK CONDITION will never
 > > be indicated unless the REQUEST SENSE command failed in
 > > some way, and (if autosensing is used), the CHECK CONDITION
 > > will be associated with its own set of sense information in
 > > the response PDU.
 >
 > The SCSI sense data is contained in the data segment of the SCSI
 > Response pdu. Hence, it is possible for the initiator to receive the
 > header of the scsi response pdu containing the SCSI Status, but not the
 > data segment (sense data) of the scsi response pdu, due to a data digest
 > error.
 >
 > - Santosh
 >
> > --
> > ##################################
> > Santosh Rao
> > Software Design Engineer,
> > HP-UX iSCSI Driver Team,
> > Hewlett Packard, Cupertino.
> > email : [email-masked]
> > Phone : 408-447-3751
> > ##################################
>