Subject: Re: iSCSI: plugfest4 issues

From: "Julian Satran" <[email-masked]>
Date: Tue, 30 Jul 2002 03:19:21 +0300

Comments in text - thanks - Julo

"Robert D. Russell" <[email-masked]>
07/30/2002 02:49 AM

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     [email-masked]
        Subject:        iSCSI: plugfest4 issues

Julian:

iSCSI plugfest 4 got underway today at UNH with 26 companies 
participating,
all testing at draft 13 of the standard.

Two issues came up today:

1. There appears to be a minor inconsistency in wording in draft 15:

Section 11.4 TargetName labels the Use of TargetName as:
  "ALL by target".

The very last sentence in this same section says:
 "The TargetName key may also be returned by the "SendTargets" text
 request (which is its only use when issued by a target)."

These 2 statements are inconsistent, because ALL implies there are other
times the target can send a TargetName key.

Either the "ALL" is wrong or the "only" is wrong.

I believe it is probably the word "only" that is wrong, and that it
should probably be replaced with a word like "principal", since, in
fact, the principal use of the TargetName key (but not its only use)
is as a reply to the "SendTargets".

+++ The only is lower-case and is meant to say that this is the only use 
the target has for the key
ALL is meant to show that the target can use it in all stages I will see 
how I can rephrase it and set is FFPO +++

2. Section 4.3 is clear that unless explicitly stated otherwise, keys can
only be sent during a specific stage.  What does not seem to be stated
anywhere is what should a receiver do if it receives a key in the wrong
stage.  Is this a "protocol error", or can the receiver just silently
throw the erroneous key way or, if the key is an operational key received
during security stage, can the receiver simply postpone responding to it
until after the transition into operational stage?  Also, in this
latter case, what happens if the key is also sent (again) during the
(proper) operational stage?  Is that an attempt at renegotiation?

I believe the simplest solution would be for the standard to explicitly
state in section 4.3 that sending a key outside the stage in which it
is allowed is a "protocol error", and that during login the Initiator
MUST close the connection and that the Target MUST send a Login Reject
and then close the connection (i.e., wording along the lines of that
in section 4.4 in the paragraph detailing what happens when there is
an attempt to negotiate a parameter more than once).
+++ will add a statement ++++
Thank you for your consideration.

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

---

From: "Robert D. Russell" <[email-masked]>
Date: Tue, 30 Jul 2002 19:43:05 -0400 (EDT)

Julian:

Four issues came up today at the iSCSI plugfest:

1. A question about whether or not the Residual Count field and the
   appropriate O and U bits need to be computed on all SCSI Response
   PDUs, regardless of the values in the Status and/or Response fields.

   One point of view says that the Residual Count field and the O and U
   bits appear to be strictly iSCSI values that are derived by the
   iSCSI target layer from the ExpectedDataTransferLength field of the
   SCSI Command PDU and the DataSegmentLength fields of the DataIn or
   DataOut PDUs sent as part of this command.  Therefore ,the iSCSI
   target always computes a Residual Count value without regard to the
   Status and/or Response fields, since these are SCSI values.

   The other point of view says that the Residual Count field, and the
   O and U bits, need only be set when the Status and Response fields
   indicate that the command was completed at the target with a GOOD
   Status, and the target does not have to compute or set the Residual
   Count field and the O or U bits for other values of the Status and/or
   Response fields.

   Which is it?  In any case, could this be clarified somewhere in the
   standard, most likely in section 9.4.4.

2. The last paragraph of section 2.2.3 says:

   "Before the Full Feature Phase is established, only Login Request and
   Login Response PDUs are allowed. Any other PDU, when received at ini-
   tiator or target, is a protocol error and MUST result in the connec-
   tion being terminated. ..."

   The question is the following:  is this rule literally true for the
   target (i.e., can the target disconnect as soon as it receives a
   non-Login PDU from the initiator) or does the target have to first
   send a Login Response with Login reject PDU before disconnecting, as
   it does for all other errors detected by the target during Login
   Phase (according to section 4.3.1)?

   A related question is: does the target take the same action when
   the very first PDU it receives on a new TCP connection is not a
   Login Request PDU?

   If the target has to send the Login reject PDU before disconnecting,
   then the last paragraph of section 2.2.3 should be reworded along
   the following lines (modeled after the last paragraph of section 4.3):

   "Before the Full Feature Phase is established, only Login Request
   and Login Response PDUs are allowed.  If the target receives any PDU
   other than Login Request, it must send a Login reject (code 0x020b)
   and then disconnect.  If the initiator receives any PDU other than
   Login Response, it MUST drop the connection. ..."

   This wording would also appear to cover the case of when the very first
   PDU a target receives on a new TCP connection is not a Login Request.

3. Section 4.2 says:

  "All keys in Chapter 11, except for the X- extension format, MUST be
  supported by iSCSI initiators and targets and MUST NOT be answered with
  NotUnderstood."

  Two questions arose with this statement:

  1. Shouldn't it say "All keys in Chapter 11 and Appendix A, ..." in order
     to include the Marker keys within the scope of this rule?

  2. Does this literally mean that targets have to support initiator-only
     keys (and initiators support target-only keys)?
     For example, suppose a target sends an InitiatorAlias key, which is
     supposed to be sent only by Initiators.  Does the target have to
     make an extra check to recognize that this is a "key in Chapter 11"
     that is used incorrectly, (so that it does not respond with
     NotUnderstood) or can it just treat it like it would any other key
     it cannot handle, for example, X-InitiatorAlias, and respond with
     NotUnderstood?

     This last question is actually a special case of a more general
     question, and that is: "How much checking does a receiver have to
     do?" just to generate a "proper error response"?
     Section 9 explicitly says that "Any compliant receiver MUST ignore
     any bit not defined and all reserved fields unless specified
     otherwise."  This is the expression of the general philosophy
     "conservative in what you send, liberal in what you accept" and
     eliminates a lot of unnecessary checking.  A similar general
     philosophy applied to other checking would be beneficial,
     although it may be hard to formulate it.

4. This is not a question but rather an observation for implementers.

   Not setting the I bit in a NOP-Out ping request can lead to unusual
   situations.  Consider the situation where a session has multiple
   connections and the initiator periodically sends NOP-Out ping
   requests on each connection just to ensure that the target is
   still alive.  If these NOP-Out requests are not immediate, and
   one of them gets lost (due to a bad checksum, or because that
   connection really is no longer alive), then none of the following
   NOP-Out requests on the other connections can be processed by the
   target, because that would be processing them out of CmdSN order,
   and as a result it would appear to the initiator that all
   connections are bad instead of just one!

   It appears that similar situations may arise when Task Management
   Function commands are sent without setting the I bit -- the target
   would be prevented from processing any of them that follow a
   command that is lost.

Bob Russell
InterOperability Lab

---

From: Luben Tuikov <[email-masked]>
Date: Tue, 30 Jul 2002 23:54:49 -0400

>    PDUs, regardless of the values in the Status and/or Response fields.

In which case if a command fails completely for whatever reason,
then you'd set the residual to equal edtl. This doesn't sound right.
 
>    standard, most likely in section 9.4.4.

Shortly, both.

Residuals are sometimes set also by the low-level device drivers,
this should be reported by the target.

The target can add a residual of its own. If the device
controller doesn't support clustering of requests, then the data
to be transferred is limited by the maximum scatter-gather table
times the sector size. Then the residual added to any future
residual (say by the LLDD or the controller or the device server)
is request_size-max_sg*sector_size. This should be reported by
the target to the initiator.

As to WRITE, you could use 9.4.6.2 Sense Data.
You cannot start SCSI WRITE for 10 bytes when you have 8 bytes.

Regarding response code, I wouldn't presume that a block OS layer
(or OS SCSI layer) would use residual counts when SCSI service
response is SERVICE DELIVERY or TARGET FAILURE, since SAM-3
specifies that ``All output parameters are invalid.''

CHECK CONDITION status code is also not a good candidate, since
nothing in the sense codes table specifies that residual counts
should be valid or looked up.

It would seem appropriate to set residual counts only when
the command completed or partially so. The status codes for
this condition are unambiguous.

The initiator shouldn't be too smart with residual counts
and try to retransmit, or it would lock up OS's ULP.
 
>    the very first PDU it receives on a new TCP connection is not a

For security/bandwidth/DOSA reasons the target should immediately
close the connection. No explanation is due for non-complient
initiators since the target cannot establish if they are initiators
in the first case (port scanners, security breakers, etc.).

That is, before FFP, the target should be ``trigger-happy'' to close
the connection. In FFP, it depends on the Recovery Level.
 
>      it cannot handle, for example, X-InitiatorAlias, and respond with

In login phase, I personally close the connection. (period)
This somehow simplifies the logic (or frees up more space for
other login related logic) and makes the target more secure.
 
>      otherwise."

This simply means that a receiver should _only_ check the relevant
information in a complient PDU format. I.e. if a reserved bit is not
0 as it should be, it's better neglected rather than the connection
compromized.

It shouldn't be generalized, for security reasons.

>    would be prevented from processing any of them that follow a

Right!
Perfect analysis which shows that request Nop-Outs should have I=1.

The logic for TMF requests is as follows:
TMF req. always carry I=1. This basically means that all recovery
and action that could have been taken at the initiator was actually
taken and we want immediate assistance/action by the target.

That is, TMF requests should be attempted at the Initiator first,
and if not successful or the nature of the task can only be performed
at the Target, then a TMF request should be sent with I=1. This also
solves other problems...

-- 
Luben

---

From: Paul Koning <[email-masked]>
Date: Wed, 31 Jul 2002 09:36:15 -0400

>>>>> "Robert" == Robert D Russell <[email-masked]> writes:

 Robert> Julian:

 Robert> Four issues came up today at the iSCSI plugfest:

 Robert> ...2. The last paragraph of section 2.2.3 says:

 Robert> "Before the Full Feature Phase is established, only Login
 Robert> Request and Login Response PDUs are allowed. Any other PDU,
 Robert> when received at ini- tiator or target, is a protocol error
 Robert> and MUST result in the connec- tion being terminated. ..."

 Robert> The question is the following: is this rule literally true
 Robert> for the target (i.e., can the target disconnect as soon as it
 Robert> receives a non-Login PDU from the initiator) or does the
 Robert> target have to first send a Login Response with Login reject
 Robert> PDU before disconnecting, as it does for all other errors
 Robert> detected by the target during Login Phase (according to
 Robert> section 4.3.1)?

 Robert> A related question is: does the target take the same action
 Robert> when the very first PDU it receives on a new TCP connection
 Robert> is not a Login Request PDU?

I like Luben's argument that a summary disconnect with no messages is
a good solution.  Apart from the security argument, I don't like
spending lots of effort coding up extra error paths that are only used
when the other side commits a gross protocol error.

 Robert> 3. Section 4.2 says:

 Robert> "All keys in Chapter 11, except for the X- extension format,
 Robert> MUST be supported by iSCSI initiators and targets and MUST
 Robert> NOT be answered with NotUnderstood."

 Robert> Two questions arose with this statement:

 Robert> 1. Shouldn't it say "All keys in Chapter 11 and Appendix A,
 Robert> ..." in order to include the Marker keys within the scope of
 Robert> this rule?

 Robert> 2. Does this literally mean that targets have to support
 Robert> initiator-only keys (and initiators support target-only
 Robert> keys)?  For example, suppose a target sends an InitiatorAlias
 Robert> key, which is supposed to be sent only by Initiators.  Does
 Robert> the target have to make an extra check to recognize that this
 Robert> is a "key in Chapter 11" that is used incorrectly, (so that
 Robert> it does not respond with NotUnderstood) or can it just treat
 Robert> it like it would any other key it cannot handle, for example,
 Robert> X-InitiatorAlias, and respond with NotUnderstood?

Similar reasoning as above: such an event is a protocol error.
"NotUnderstood" seems like a good response.  Disconnecting is also
fine in my view.

     paul

---

From: "Rod Harrison" <[email-masked]>
Date: Wed, 31 Jul 2002 10:10:47 -0500

With a tape in variable block mode a GOOD SCSI status and short read,
i.e. underflow is possible. Similarly with mode sense, or inquiry
etc., and a buffer that is too small there would be GOOD status and an
overflow indication. So the target does need to set the resid count
and bits when there is GOOD status.

	- Rod

				--- >8 ---

[snip ]

1. A question about whether or not the Residual Count field and the
   appropriate O and U bits need to be computed on all SCSI Response
   PDUs, regardless of the values in the Status and/or Response
fields.

   One point of view says that the Residual Count field and the O and
U

   Count field and the O or U bits for other values of the Status
and/or
   Response fields.

[snip]

---

From: "Robert D. Russell" <[email-masked]>
Date: Wed, 31 Jul 2002 19:18:08 -0400 (EDT)

Julian:

Five issues came up today, Wednesday, 31-July-2002, at the iSCSI
plugfest.

1. We have hit a situation where a lot of people are not interoperating,
   and this has to do with the use of "Irrelevant" as a key response
   value.  In section 4.2 the standard says only: "If a specific key is
   not relevant for the current negotiation, the responder may answer
   with the constant "Irrelevant" for all types of negotiation. ...".

   The problem is that nowhere in the standard is it further defined
   when a key is relevant and when it is not.  It is left up to each
   individual implementation, and as a result, different implementers
   are coming to different conclusions in the same circumstances.  Thus,
   they do not interoperate.

   A few implementations are simply saying that all keys are always
   relevant, and never accept "Irrelevant" as an answer in any situation.
   That is the extreme, but their argument is that it requires extra
   logic to decide when a key is relevant and when it is not, and hence
   it is simpler not to do this.

   More common are implementations that have simply not thought of all
   the possible situations in which a key has become irrelevant.

   We have a choice on how to proceed.

   The simplest alternative would be to eliminate "Irrelevant" as a
   response, since it is never necessary to use this in a response,
   yet it always requires extra logic on the receiving side to deal
   with it.

   An alternative if we decide to keep "Irrelevant" as a response is to
   have the standard specify as part of the definition of each key in
   Section 11 and Appendix A those situations in which that key may be
   considered irrelevant.  This could be listed as an attribute
   "Irrelevant when:" immediately following the current "Scope:"
   attribute.  This attribute is in addition to the integrity rules
   which already exist.

   The following is a list of all the keys that I am aware of that can
   accept a response of Irrelevant, and the situations in which this
   can legitimately occur (numbers indicate the section where the key is
   defined):

   11.2  MaxConnections -- Irrelevant when: SessionType=Discovery
   11.10 InitialR2T -- Irrelevant when: SessionType=Discovery
   11.11 BidiInitialR2T -- Irrelevant when: SessionType=Discovery
   11.12 ImmediateData -- Irrelevant when: SessionType=Discovery
   11.10 MaxBurstLength -- Irrelevant when: SessionType=Discovery
   11.10 FirstBurstLength -- Irrelevant when: SessionType=Discovery
         -- also irrelevant when ( InitialR2T=Yes and ImmediateData=No )
   11.18 MaxOutstandingR2T -- Irrelevant when: SessionType=Discovery
   11.19 DataPDUInOrder -- Irrelevant when: SessionType=Discovery
   11.20 DataSequenceInOrder -- Irrelevant when: SessionType=Discovery
   A.3.2 OFMarkInt -- Irrelevant when: OFMarker=No
         IFMarkInt -- Irrelevant when: IFMarker=No

2. Section 2.2.2.1 states:
   "Commands meant for immediate delivery are marked with an immediate
   delivery flag; they also carry CmdSN.  CmdSN does not advance for
   commands marked for immediate delivery."

   and 2 paragraphs later in the same section:
   "If immediate delivery is used with task management commands, these
   commands may reach the target before the tasks on which they are sup-
   posed to act.  For this reason the task management command MUST carry
   the current CmdSN as a marker of their position in the stream of com-
   mands."

   Given the first statement quoted above, why is this last sentence
   needed?  It is causing some confusion because it makes it sound
   as if the task management command is somehow an exception because
   it MUST carry the current CmdSN (implying that maybe other
   immediate commands do not have to carry the current CmdSN).

   To eliminate the confusion, but to keep the same intention, a
   suggested rewording would be:

   To the first paragraph quoted above, change the phrase "they also
   carry CmdSN" to: "they MUST also carry the current CmdSN".

   In the second paragraph quoted above, either change the second
   sentence (the one starting "For this reason...") to: "However,
   their CmdSN is a marker of their position in the stream of
   commands." (which is the wording that was used previously in
   draft 12), or just eliminate the second sentence entirely, since
   this topic is discussed later in section 2.5.1.3 and again in
   section 9.5.1.

3. Section 9.13.3 states:
   "For a new session, the target MUST generate a non-zero TSIH and
   return it in the Login Final-Response (see Section 4.3 Login Phase).
   In all other cases, this field should be set to the TSIH provided
   by the initiator in the Login Request."

   The phrase "in all other cases" is ambiguous.  Some companies are
   interpreting it to mean "sessions other than a new session", while
   others are interpreting it to mean "Login Responses other than
   the Login Final-Response in a new session".

   So what is the intent?  On a new session, clearly the target MUST
   return the non-zero TSIH in the Login Final-Response, but can it
   also return a non-zero TSIH in the first, second, ... Login Partial
   Responses that precede the Login Final-Response?

   If the intent for new sessions is to have the TSIH set to 0 in Login
   Partial-Responses and only change to a non-zero TSIH in the Login
   Final-Response, then a suggested rewording to make this clear would be:

   "Except for the Login Final-Response in a new session, this field
   should be set to the TSIH provided by the initiator in the Login
   Request.  For a new session, the target MUST generate a non-zero
   TSIH and return it ONLY in the Login Final-Response (see Section
   4.3 Login Phase)."

   If the intent is to allow, but not require, the target to return
   the non-zero TSIH in the first, second, ... Login Partial
   Responses in a new session, then a suggested rewording to make
   this clear would be:

   "During Login Phases that do not establish a new session, this
   field should be set to the TSIH provided by the initiator in
   the Login Request.  For a new session, the target MUST generate
   a non-zero TSIH and MAY return it in a Login Partial-Response
   but MUST return it in the Login final-Response (see Section 4.3
   Login Phase)."

   This last interpretation still leaves open the question of whether
   the target is allowed to toggle back and forth between 0 and the
   newly-generated TSIH within sequences of Login Partial-Responses.

4. There has been some misunderstanding with the phrase "not advanced".
   For example, in section 9.8.2 StatSN it says:  "The StatSN field will
   contain the next StatSN.  The StatSN for this connection is not
   advanced."

   In spite of the presence of the word "next", some implementations are
   interpreting this to mean that this PDU will always contain the same
   value for StatSN as was sent in the previous response.  Perhaps it
   would help if the words "after this PDU is sent" were added to the
   last sentence quoted above so that it would read: "The StatSN for
   this connection is not advanced after this PDU is sent."

   Other examples where this wording could be added to help clarify
   when the advancing is performed (or not performed) are:

   section 2.2.2.1: "Commands meant for immediate delivery are marked with
   an immediate delivery flag; they also carry the next CmdSN.  CmdSN does
   not advance after commands marked for immediate delivery are sent."

   section 2.2.2.1: "- CmdSN - the current command Sequence Number,
   advanced by 1 after each command shipped except for commands marked
   for immediate delivery."

   section 2.2.2.3: "For input (read) data PDUs, DataSN starts with 0 for
   the first data PDU of an input command and advances by 1 after each
   data PDU is sent.  For output data PDUs, DataSN starts with 0 for the
   first data PDU of a sequence (the initial unsolicited sequence or any
   data PDU sequence issued to satisfy an R2T) and advances by 1 after
   each data PDU is sent.  R2Ts are also sequenced per command.  For
   example, the first R2T has an R2TSN of 0 and the R2TSN advances by 1
   after each R2T is sent."

   section 9.11.5: "The target StatSN register is advanced after each
   Text Response is sent."

   section 9.18.1: "However, the I bit must be set to 1 and the CmdSN
   is not advanced after this PDU is sent."

   section 9.19.2: "However, when the Initiator Task Tag is set to
   0xffffffff,  StatSN for the connection is not advanced after this
   PDU is sent."

5. There has been major disagreement about correct digest values.

   A number of implementations differ only due to a Big-Endian/
   Little-Endian problem that we hope to be able to resolve over
   the next few days.  Differences between other implementations
   show no discernible pattern.  Of course, all implementations
   claim to compute the proper answers to the test cases given in
   the standard.  The difficulty is that these test cases are not
   sample pdus and therefore never actually occur during operation.

   A recommendation would be to include in the standard a simple
   sample PDU (one that could actually be sent during a session) and
   give its correctly computed CRC value.  For example, a NOP-OUT
   PDU with I=1, DataSegmentLength=0, Lun=0, InitiatorTaskTag and
   Target Transfer Tag both = 0xffffffff, CmdSN = 1234, and
   ExpStatSN=567 could be legal, and most implementations could probably
   arrange fairly simply for such a PDU to actually be sent by an
   initiator.  By publishing the correct CRC value for such a pdu,
   all questions about who is right and who is wrong are immediately
   solved, and new implementations can be developed more accurately.

Bob Russell
InterOperability Lab

---

From: Luben Tuikov <[email-masked]>
Date: Wed, 31 Jul 2002 22:51:02 -0400

>    Responses that precede the Login Final-Response?

The intent is that for a new session the initiator MUST set
the TSIH to the reserved value of 0, during login. During login
the target will copy TSIH from the request to the response,
effectively setting it to 0.

On the last login response (the one which completes the switch
to FFP, I<-T), the target will set TSIH to a valid non-reserved
value (which it generated whenever).
From that point on the initiator should always set TSIH
to that value. (FFP, connection reinstatement, etc.)

Though not explicit, compiling everything mentioning TSIH
in the draft, evident this becomes.

>    this connection is not advanced after this PDU is sent."

Actually the above two sentences are correct and in harmony.

Anything SN in the draft, at any point in time, contains the
next value to be assigned. This is the intent, as counting starts
from 0 and setting xxxxSN = 0, works out as intended. This is also
deduced from looking at the equation which gives the queue size.

So particulartly for 9.8.2 (R2T) since no status is reported
one does (w.l.g.) packet->statsn = conn->statsn;
but when status is reported (w.l.g) MOV_AND_INC_SN(packet->statsn, conn->statsn),
which does an assignment (packet->statsn = conn->statsn) and increments
conn->statsn (over the reserved value of course) somewhat atomically.

The only difference is incrementing _after_ assignment, assignment
always being first.

Similarly for I->T, when I=0 and I=1, regarding cmdsn, etc.
 
>    0xffffffff,  StatSN for the connection is not advanced after this

Which makes sense, since if ITT = the reserved value, then
the Target is initiating the Nop sequence and it is NOT
reporting status.

-- 
Luben