Subject: RE: iSCSI: current UNH Plugfest

From: "Eddy Quicksall" <[email-masked]>
Date: Thu, 1 Nov 2001 08:14:48 -0500

I am reluctant to say this because I think most people think the
initiator/target must check for correctness ... but, it is my feeling that
that job should be up to a basher program. The target should not be in the
business of diagnosing the initiator. The only time a target should check a
field is when it could crash the system or data. Some format errors may have
no consequences whatsoever.

Eddy

-----Original Message-----
From: Robert D. Russell [mailto:[email-masked]]
Sent: Wednesday, October 31, 2001 05:39 PM
To: [email-masked]
Subject: Re: iSCSI: current UNH Plugfest

Attached are the new issues that arose during the iSCSI plugfest
at UNH on Wednesday 31-Oct-2001.

(Note: these issues do not take into account any modifications or
clarifications that occured in the standard due to the issues raised
on Monday or Tuesday.)

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

------------------------------------------------------------------------
----

1. Are receivers (initiator or target) REQUIRED to check that reserved
   bits and/or fields are set to 0?

   Section 3 on page 48 of draft 8 says:
     "Any bits not defined MUST be set to zero.  Any reserved fields and
     values MUST be 0 unless specified otherwise."

   and Section 8.3 on page 127 of draft 8 says:
     "Explicit violations of the PDU layout rules stated in this
document
     are format errors.  This when detected, usually indicates a major
     implementation flaw in one of the parties.

     When a target or an initiator receives an iSCSI PDU with a format
     error, it MUST reset all transport connections in the session
     immediately and escalate the format error to session recovery
     (section 8.11.4)."

   According to these rules, a PDU with reserved bits and/or fields that
   are not set to 0 violates the PDU layout rules.  Therefore, if an
   initiator or target receives such a PDU, it should immediately close
   all connections in the session and go to session recovery.

   Clearly a format error has extremely severe consequences!

   Although all vendors are setting reserved bits and fields to 0 on
   PDUs they are sending, many are NOT checking PDUs they are receiving
   to see if these bits and fields are set to 0.  Basically, vendors are
   saying "who cares if reserved bits and/or fields in incoming PDUs are
   not zero?  We do not want to take the time to do this checking, and
   there is no benefit to doing it.  As long as the non-reserved bits
and
   fields are set properly, we can and should proceed.  Any time devoted
   to doing this checking is wasted in 99+% of the cases, and in the
   (unlikely) case that a non-zero bit or field is found, the
   consequences are too severe."

   There should be some statement in the standard to clarify what
checking
   is required and what is optional.

2. A similar situation arises with respect to checking the consistency
   of fields such as Version-max, Version-min and Version-active in
Login
   Requests and Login Responses.

   For example, consider the Version-max field.

   Section 3.12.5 says:
     "All Login requests within the Login phase MUST carry the same
     Version-max."

   All vendor initiators are setting Version-max correctly on all
   login requests they are sending, but many vendor targets are NOT
   checking received login requests to ensure that this rule is
enforced.
   In particular, many targets simply use the Version-max and
Version-min
   on the first login request they receive on a new connection, and then
   they ignore these fields on all subsequent login requests in the same
   login phase.

   Strictly speaking, a change in the Version-max field during the login
   phase constitutes a protocol error according to section 8.8 on page
130
   of draft 8:

     "All violations of iSCSI PDU exchange sequences specified in this
     draft are also protocol errors.  This category of errors can be
     addressed only by fixing the implementations; iSCSI defines Reject
     and response codes to enable this".

   Therefore the target should send back a login response with a status
   of 0x0200 and then close the connection.

   However, Section 3.12.5 also says:
     "The target MUST use the value presented with the first login
request."

   This rule seems to imply that the value CAN change, because if it
cannot
   change, then it doesn't matter which one of the login requests it is
   taken from, they are all the same anyway.

   The suggestion is to keep the requirement that the target MUST use
the
   value presented on the first login request, but to allowed the target
   to ignore the value presented on all subsequent login requests in the
   same login phase.  A similar rewording should be done for the other
   fields.

3. Can commands be sent out of order on the same connection?

   The behavior of targets is clearly specified in Section 2.2.2.3 on
   page 25 of draft 8, which says:
     "Except for the commands marked for immediate delivery the iSCSI
     target layer MUST eliver the commands for execution in the order
     specified by CmdSN."

   Section 2.2.2.3 on page 26 of draft 8 also says:
     "- CmdSN - the current command Sequence Number advanced by 1 on
     each command shipped except for commands marked for immediate
     delivery."
   but the meaning of the term "shipped" is vague, and does not
necessarily
   require that the PDUs arrive on the other end of a TCP connection
   in the same order that the CmdSN values were assigned to these PDUs.

   Some initiators have been designed to send commands out of CmdSN
   order on one connection.  Consider the situation where there is only
   one connection and a high-level dispatcher creates a PDU for a SCSI
   command that involves writing immediate data to the target.  This PDU
   is enqueued to a lower-level layer which has to setup, start, and
   wait-for a DMA operation to move the immediate data into an onboard
   buffer before the PDU can be put onto the wire.  While this is
   happening, the dispatcher creates another unrelated PDU for a SCSI
   read command (for example), and when this PDU is passed to the
   lower-level layer it can be sent immediately, ahead of the previous
   write PDU and therefore out of order on this connection.

   The standard clearly allows this to happen if the two PDUs were sent
   on different connections, and seems to imply that this can also
happen
   when the two PDUs are sent on the same connection.

   The suggestion is to put in the standard an explicit statement that
   this is allowed or not allowed, as appropriate.

   If this is allowed, such a statement would avoid the erroneous
   assumption being made by some target implementers that within a
single
   connection, commands will arrive in order.

   If this is not allowed, such a statement would avoid the erroneous
   assumption being made by some initiator implementers that within a
   single connection, commands can be put on the wire out of order.

4. Three numeric keys (MaxRecvPDULength, MaxBurstSize, FirstBurstSize)
   now allow: "A value of 0 indicates no limit."

   Is this useful?  Does it buy anything?

   The difficulties implementers are having with this are:

   1) It is a special case.
   2) It causes discontinuous ranges (for example, [0,64..2**24])
   3) It violates the min/max function normally used for the key.
   4) There is always a limit anyway.

   Consider FirstBurstSize, which can have a value that is described
   as "<0|number-64-2**24>", and for which the minimum of the 2 numbers
   is selected.

   I one side offers 0 to mean unlimited, and the other side
   has a limit, it will reply with that limit, say 128 Kbytes.
   Therefore, the result is not min(0, 128K) but rather max(0, 128K).
   The statement in the standard that "the minimum of the 2 numbers is
   selected" is therefore wrong when one of the numbers is 0.

   Furthermore, when an initiator or target receives an offer for one
   of these keys, it cannot simply check that the offered value is
   legal by testing it against some minimum and maximum.  It must first
   check for 0 and then only if that check shows the value is non-zero
   can it do the min/max range check for legality (i.e., 64-2**24).

   Finally, there is always a limit. If nothing else it is the
   limit imposed by the 24-bit DataSegmentLength field of the PDU
   requesting the transfer.  It is useless to specify a FirstBurstSize
   (or MaxRecvPDULength or MaxBurstSize) any bigger than that, because
   the largest possible DataSegmentLength in any PDU that can use
   this value is 2**24-1.

   The suggestion is to just eliminate this special case of 0 and
require
   that the range 64-to-(2**24-1) be used instead -- it has exactly the
   same effect in all cases, it is easier to describe in the standard
   because it avoids all the extra words, and it is easier to code
   because it avoids all the special cases.

   NOTE: the standard should specify the limit in the ranges for
   MaxRecvPDULength, MaxBurstSize, and FirstBurstSize as 2**24-1 instead
   of 2**24.  The number 2**24 cannot be represented in the 24-bit
   DataSegmentLength field and therefore can never be used.

5. This is a suggestion for a minor rewording in the standard to avoid
   misunderstandings.

   In Appendix E on page 188 of draft 8 it says:

     "The response to this command is a text response containing a list
of
     targets and their addresses.  Each target is returned as a target
     record.  A target record begins with the TargetName text key,
     followed by a list of TargetAddress text keys, ..."

   In fact, there are situations where there are no targets and/or no
   addresses.  These situations are clearly defined in the draft after
   the sentences quoted above, but it would help if those sentences
   at least hinted at the possibility that the lists could be empty
   or might not contain addresses.  A possible rewording would be:

     "The response to this command is a text response containing a list
of
     zero or more targets and, optionally, their addresses.  Each target
     is returned as a target record.  A target record begins with the
     TargetName text key, followed by a list of zero or more
     TargetAddress text keys, ..."

6. This is a suggestion for another very minor rewording in the
standard.

   At the end of section 2.2.3 on page 29 of draft 8 it says:

     "Before full feature phase is established, only Login PDUs are
     allowed. ..."

   The suggested rewording is:

     "Before full feature phase is established, only Login Request and
     Login Response PDUs are allowed. ..."

---

From: "Bill Strahm" <[email-masked]>
Date: Thu, 1 Nov 2001 08:55:57 -0800

Usually the "Conservative in what you send, Liberal in what you accept"
policy is used...

In otherwords, The sender MUST set to 0 (or some other value) The receiver
MUST ignore the value...

This allows for some tweaking of the implementation, if I control both ends
I might set a reserved value to 1, then I know something... If I receive a
reserve value set to 1 and I don't do anything the other end knows it is not
talking to itself (this can even be a versioning thing as well)

Now, we need to be VERY careful in defining this, do we plan on having
Protocol V1 endpoints talk to Protocol V2 endpoints, what does that mean...
is it possible, is it desirable ? will there be extensions ???

If you can truly answer NO to all of those things, I would argue for
REMOVING the reserved fields (if possible), if not, the MUST set, MUST
ignore policy seems better

Bill
+========+=========+=========+=========+=========+=========+=========+
Bill Strahm     Software Development is a race between Programmers
Member of the   trying to build bigger and better idiot proof software
Technical Staff and the Universe trying to produce bigger and better
[email-masked] idiots.
(503) 601-0263  So far the Universe is winning --- Rich Cook

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Eddy Quicksall
Sent: Thursday, November 01, 2001 5:15 AM
To: [email-masked]
Subject: RE: iSCSI: current UNH Plugfest

Eddy

-----Original Message-----
From: Robert D. Russell [mailto:[email-masked]]
Sent: Wednesday, October 31, 2001 05:39 PM
To: [email-masked]
Subject: Re: iSCSI: current UNH Plugfest

Bob Russell
InterOperability Lab



   Section 3.12.5 says:





   1) It is a special case.

---

From: Paul Koning <[email-masked]>
Date: Thu, 1 Nov 2001 13:38:40 -0500

> Usually the "Conservative in what you send, Liberal in what you accept"

That's a very important principle...

> In otherwords, The sender MUST set to 0 (or some other value) The receiver

That's the way to express the principle.  But the text quoted in the
earlier notes does not express it the same way.

In particular "... are format errors.  This when detected..." implies
that receivers may or may not detect format errors.  In other words,
it implies -- but does NOT state explicitly -- that receivers MAY
check reserved fields for zeroness.

> ignore policy seems better

I agree strongly.  There is a lot of experience in the community on
what helps and what hurts convenient version upgrade.  In particular,
there is a LOT of evidence that ANY checking of reserved fields is a
nasty thing.  Unless you plan never to go beyond V1, you really need
to require the rule Bill stated, i.e., receivers MUST ignore the
contents of reserved fields -- they are NOT allowed to "verify" them.

So the spec needs to be clarified to say that random values in
reserved fields are NOT format errors and receivers MUST NOT treat
them as such.

	 paul

---

From: "Robert D. Russell" <[email-masked]>
Date: Thu, 1 Nov 2001 18:31:48 -0500

Attached are the new issues that arose during the iSCSI plugfest at
UNH on Thursday 1-Nov-2001.

Bob Russell
InterOperability Lab

----------------------------------------------------------------------

There were no new issues today!  Lots of people were busy sending
data and interoperating with each other!  There were, however, a
few suggestions/requests for clarifications in the standard:

1. Since all Login Requests MUST be issued as immediate requests,
the diagram in section 3.12 should show bit 6 of byte 0 as "1",
not "I", and section 3.12.2 should simply be eliminated.

2. The first paragraph of Appendix D should describe the special
role of the first Login request on the first connection of a new
session.  In particular, there are 3 keys that MUST be sent in
that Login PDU (InitiatorName, SessionType if it is discovery,
TargetName if the session is normal).  This is, in fact, yet
another classification that is currently included within LO, but
LO is too general, since LO allows keys to appear at any time
during the leading login phase, and these keys must appear on the
very first login PDU in that leading login phase.

3. A target cannot release its resources for a command until it
receives an ExpStatSN back from the initiator that acknowledges
receipt of the StatSN carried in the SCSI Response to that command.
However, if the initiator does not have any more I/O to perform,
that ExpStatSN will not come back to the target, and the target
may end up holding on to those resources indefinitely.  This is a
situation where initiator non-action can impact target performance,
and could even lead to denial of service attacks if several
initiators were to do this simultaneously to the same target.

The standard provides the NOP-In/Out mechanism to deal with this:

The last paragraph of section 2.2.2 says:
  During periods when traffic on a connection is unidirectional,
  iSCSI NOP-Out/In PDUs may be utilized to synchronize the command
  and status ordering counters of the target and initiator.

and section 3.18 says:
  A NOP-Out may also be used to confirm a changed ExpStatSN if
  there is no other PDU to carry it for a long time.

The question is, is there a recommended mechanism to trigger these
NOP-pings?

Clearly the target could set a timer, and if ExpStatSN is not
received back at the end of the time interval, the target could
send a NOP-in as a ping in order to get the initiator to send back
a NOP-out with the updated ExpStatSN.

Also clearly the initiator could set a timer, and if it has no
traffic to send during the time interval, the initiator could send
a NOP-out to deliver the updated ExpStatSN to the target.
This approach is probably not as reliable as the first, since
a) if the initiator does not do it, only the target gets hurt, and
b) if the connection drops, the target will never receive the NOP-out.
On the other hand, it may be more efficient for the initiator to do
this, since a typical target may be dealing with hundreds of
connections, and the initiator only a few.

Which is the best procedure?  Are there other ways to do this?
What is the recommended way, or is there a reason for the standard
not to recommend a triggering mechanism, in which case, could
the standard suggest methods or provide guidance, especially if there
are better, less obvious ways to do it?

---

From: "Ron Cohen" <[email-masked]>
Date: Fri, 2 Nov 2001 14:59:18 -0000

The problem with accepting values in reserved bits is that when the reserved
bit is no longer reserved (evolution in the standard) it gives the
impression to the initiator that a target implements the new feature when it
may actually only be ignoring it.

Ron

---

From: "Bill Strahm" <[email-masked]>
Date: Fri, 2 Nov 2001 09:23:49 -0800

In many cases that is acceptable...  Imagine what would happen if Router
vendors threw out packets with headers that set reserved bits to anything
but 1.  Anyone that wanted to play with DiffServ would have to buy new
routers that implemented the functionality of the TOS bits...  This when it
is perfectly acceptable to just ignore them and send packets on without
applying any extra functionality - What a pain.

Some of the upgrade problem can be handled with a new version number (if you
don't speak the new version, I will drop the packet) but there are MANY
extensions that I can think of where using a bit or two of a reserved field
would allow potentially better performance if the endpoint knows what to do,
and no real degradation of service if it doesn't know what to do...  Why
should I have to rev the version number to handle this case ?

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Ron Cohen
Sent: Friday, November 02, 2001 6:59 AM
To: ips; Paul Koning
Subject: Re: iSCSI: current UNH Plugfest: Reserved bits

Ron

---

From: Paul Koning <[email-masked]>
Date: Fri, 2 Nov 2001 18:01:37 -0500

> may actually only be ignoring it.

Yes, if you don't do some other things that are needed.

You MUST have protocol version numbers in the initial messages.
Otherwise you are utterly lost.

With that, you know what version the other end has.

The key point is that requiring reserved fields to be ignored is that
you can rely on the fact that new things can be asked for in reserved
fields with the knowledge that old implementations will ignore those
things.

Quite often this is what you want.  Occasionally it is not; for those
cases you make use of the version number.

In any case, what I was describing is long established practice in
many protocols that have successfully navigated the version number
migration path.

	  paul

---

From: Tow Wang <[email-masked]>
Date: Tue, 27 Nov 2001 15:55:02 -0800

Hi all.

I fully agree with the suggestions for issue 4. If we really need to reserve a
value to mean "unlimited" or "infinite", could we use a value of all bits set to
one to indicate this? (Specifically, 0xFFFFFF or 0xFFFFFFFF would be great for
most 32-bit processors and above.) This would make arithmetic comparisons more
intuitive, IMHO.

"Robert D. Russell" wrote:

>    The suggestion is to just eliminate this special case of 0 and require

--
Tow Wang
Adaptec Software Engineer       Telephone: 408 957 4838
Mail stop 46                               800 869 8883 x4838
691 South Milpitas Boulevard    Fax:       530 686 8023
Milpitas, California 95035      E-mail:    [email-masked]