Subject: Re: iscsi : iscsi parameter default values

From: "John Hufferd" <[email-masked]>
Date: Mon, 1 Oct 2001 15:42:38 -0700

I think your logic reverses if the support wanted is yes and the target is
supporting yes.

If they say No they clearly do not care about extra handshakes sense R2T is
all they have and it requires extra handshakes.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: [email-masked]

Santosh Rao <[email-masked]>@ece.cmu.edu on 10/01/2001 09:18:23 AM

Sent by:  [email-masked]

To:   [email-masked]
cc:
Subject:  Re: iscsi : iscsi parameter default values

All,

As the one who started this thread, please allow me to raise one
important consideration to this discussion.

This discussion has gone down the path of whether immediate data boosts
performance and is implementable or whether it is so demanding that it
is not feasible to implement.

I believe the decision on what the default value must assume for this
parameter must be based on what is going to be the simplest for the
login process and which setting minimizes the overheads involved for
completion of login.

The reason for applying the above criteria is that it provides the
benefit of completing login negotiation in a single handshake and in
most cases, where initiator goes with the defaults, allows no
negotiation to be required whatsoever. [in the case where no security is
being negotiated.]. This has the significant benefit of boosting iscsi
interoperability, since it has been seen that login is one of the most
painful areas of iscsi.

With the above in mind, let us consider the 2 cases, Immediate data
default to "yes" & "no" (assuming neither side is interested in
security) :

In the first case (default immediate data="yes') :
--------------------------------------------------
- Initiator wishes to use immediate data. Target does not support it.

I -> T : (no key is sent. default assumed for immediate data).
      (CSG=Operational, NSG=FullFeaturePhase). T=1

T -> I : ImmediateData="no" (tgt does not support imm data.)
      (CSG=Operational,NSG=Operational). T=0

I -> T : (CSG=Operational, NSG=FullFeaturePhase). T=1
         (no keys to negotiate.
          negotiation completed at initiator.
          only phase change taking place.)

T -> I : (CSG=Operational, NSG=FullFeaturePhase). T=1
         (target completes phase change.
          both sides move to FFP.)

Thus, in the above model, there's an extra dummy login handshake, when
the default ImmediateData is "yes" but targets don't support it for the
sole purpose of completing the login phase change.

Now, take the case of default ImmediateData set to "no" :
---------------------------------------------------------
- Initiator wants to use immediate data. target does not support it.
(same scenario as above).

I -> T : ImmediateData="yes"
         (CSG=Operational. NSG=FullFeaturePhase.). T=1

T -> I : ImmediateData="no"
      (CSG=Operational. NSG=FullFeaturePhase). T=1

- Initiator does not want to use immediate data.

I -> T : (no key is sent. defaults assumed).
         (CSG=Operational. NSG=FullFeaturePhase). T=1

T -> I : (targets can accept all defaults since they
       are conservative.)
      (CSG=Operational, NSG=FullFeaturePhase). T=1

Thus, as you can see from the above, the login process involves the
least number of exchanges when the default for ImmediateData is chosen
to be "no", as opposed to a default of "yes".

It DOES NOT matter whether some believe ImmediateData is useful and
feasible or some believe it is useful but not feasible, or yet some
others believe it is not useful.

What MUST govern this decision is which default allows for the simplest
login operation.

From the above, it seems to me like a default of "no" for ImmediateData
allows login to be completed in a single handshake in the case where
initiators wishes to use immediate data or not, and in the case where
target supports immediate data or not.

Therefore, I request the group to please consider setting the
ImmediateData default to "no" and vote for the setting of "no". This
decision benefits BOTH the camp of immediate data users and non-users,
since both benefit from minimized steps in the login process.

Thanks,
Santosh

--------------------------------------------------------------------------

Robert Snively wrote:
>
> Julian,
>
> For SCSI Write operations, ImmediateData = yes is
> the most demanding mode for the target.  If I understand
> it correctly, it essentially over-rides the R2T state,
> allowing the first burst to be included as immediate data.
> In SCSI, except for special cases like this, the target
> is the device in charge of data transfers.
>
> This mode requires the target to have a guaranteed collection of
> received buffer space adequate to receive all possible
> outbound queued operations from all possible initiators
> through all possible target sessions (which may sum to
> 1000s of output operations), regardless of what other
> buffer-intensive operations may be going on in the device.
>
> It then requires the device to provide association with each
> of those commands instead of just the commands it has elected
> to activate.  Without immediate data, the command buffer can be
> separated and separately managed from the data buffer,
> limiting buffer requirements.
>
> It then requires the device to operate in a non-zero-copy
> mode (which raises buffer utilization and increases latency)
> while the device analyzes the command to determine what should
> be done with the data.  It further requires the subsequent
> data (if there is more than one PDU to be transmitted) to
> be separately managed, and perhaps actually stored in a
> separate operation if the buffer must be cleared to make
> space available for it.  It further requires the target to
> analyze the data for completeness before going on to determine
> what to do with it.
>
> Sure, it is easy for the initiator, but so is everything else
> except out-of-order reads.
> It is the target you are stressing with this.
>
> For local sub-LANs, and depending on the actual buffering
> available in the target, the performance may actually be lower with
> ImmediateData allowed, because zero copy behavior of the
> target to non-volatile media is not available, which raises
> buffer utilization.
>
> Have I missed something that would change my mind?
>
> Bob
>
> > -----Original Message-----
> > From: Julian Satran [mailto:[email-masked]]
> > Sent: Friday, September 28, 2001 10:38 AM
> > To: [email-masked]
> > Subject: RE: iscsi : iscsi parameter default values
> >
> >
> >
> >
> > Robert,
> >
> > I disagree that Immediate is "the most demanding" mode.  That
> > is not my
> > experience and not what I heard from most of the developers.  On the
> > contrary immediate data do not require a separate indexing
> > operation on the
> > target (as non-immediate unsolicited do) and that was the base for the
> > consensus to have them negotiated separately.
> >
> > For the software initiator it is the most inexpensive way to
> > perform short
> > data transfers (4-8k) typical for major applications like
> > Database access
> > and so it is for lightweight target.
> >
> > Julo
> >
> >
> >
> >
> >
> >                     Robert Snively
> >
> >                     <rsnively@broc       To:     John
> > Hufferd/San Jose/IBM@IBMUS, Julian
> >                     ade.com>
> > Satran/Haifa/IBM@IBMIL
> >                                          cc:
> > [email-masked]
> >                     28-09-01 17:25       Subject:     RE:
> > iscsi : iscsi parameter default values
> >                     Please respond
> >
> >                     to Robert
> >
> >                     Snively
> >
> >
> >
> >
> >
> >
> >
> >
> > I vote no as the default value on ImmediateData.
> >
> > A default value of yes on ImmediateData requires implementation
> > of the most complex and demanding mode of operation as the
> > default.  SCSI has traditionally made its default behavior the
> > simplest and most encompassing, setting more sophisticated
> > behavior by subsequent agreement.  While it may be your earnest
> > desire to encourage the implementation of this function, it
> > is not appropriate to demand that as the default behavior
> > of all devices.  In particular, there is no special benefit
> > to providing ImmediateData in low-cost local sub-lans.
> >
> > If you want to encourage it in a profile, fine, but demanding
> > it as the default in the core standard is not appropriate.
> >
> > Note that the behavior of SCSI is traditionally managed
> > entirely by the target.  As such, there has never before now
> > been a requirement for the target to, as a default, accept
> > any PDU except a command or task management function
> > that was not explicitly solicited.  That is one of the mechanisms
> > that assists SCSI in achieving a low-overhead zero copy
> > capability while operating with a large number of initiators
> > and with deep command queues.
> >
> > Bob Snively                        e-mail:    [email-masked]
> > Brocade Communications Systems     phone:  408 487 8135
> > 1745 Technology Drive
> > San Jose, CA 95110
> >
> >
> >
> > > -----Original Message-----
> > > From: John Hufferd [mailto:[email-masked]]
> > > Sent: Friday, September 28, 2001 1:33 AM
> > > To: Julian Satran
> > > Cc: [email-masked]
> > > Subject: RE: iscsi : iscsi parameter default values
> > >
> > >
> > >
> > > I also agree with this.  It should be yes.
> > >
> > > .
> > > .
> > > .
> > > John L. Hufferd
> > > Senior Technical Staff Member (STSM)
> > > IBM/SSG San Jose Ca
> > > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > > Home Office (408) 997-6136
> > > Internet address: [email-masked]
> > >
> > >
> > > Julian Satran/Haifa/IBM@[email-masked] on 09/27/2001 09:50:21 AM
> > >
> > > Sent by:  [email-masked]
> > >
> > >
> > > To:   [email-masked]
> > > cc:
> > > Subject:  RE: iscsi : iscsi parameter default values
> > >
> > >
> > >
> > >
> > > The one that I have trouble living with is ImmediateData.
> > > This is important
> > > for low-end desktops and hardly matters for large boxes.
> > > As such I would suggest it stays as yes.
> > >
> > > Julo
> > >
> > >
> > >
> > >                     "Eddy
> > >                     Quicksall"            To:     "'Santosh Rao'"
> > > <[email-masked]>,
> > >                     <EQuicksall@med        <[email-masked]>
> > >                     iaone.net>            cc:
> > >                     Sent by:              Subject:     RE:
> > > iscsi : iscsi
> > > parameter default values
> > >                     [email-masked]
> > >                     mu.edu
> > >
> > >
> > >                     27-09-01 17:22
> > >                     Please respond
> > >                     to "Eddy
> > >                     Quicksall"
> > >
> > >
> > >
> > >
> > >
> > > I like your defaults below.
> > >
> > > But, section 5 says:
> > >
> > >  The initial Login request MUST include the InitiatorName and
> > >  SessionType key=value pairs.
> > >
> > > Since SessionType is REQUIRED, naming a default would imply a
> > > possible typo
> > > in the spec.
> > >
> > > Eddy
> > >
> > > -----Original Message-----
> > > From: Santosh Rao [mailto:[email-masked]]
> > > Sent: Wednesday, September 26, 2001 10:29 PM
> > > To: [email-masked]
> > > Subject: iscsi : iscsi parameter default values
> > >
> > >
> > > All,
> > >
> > > With the issue of mode page vs. login keys having [almost]
> > drawn to a
> > > close, I would like to
> > > raise the below issues again, on the subject of default
> > > values for login
> > > keys and other iscsi
> > > parameters :
> > >
> > >
> > >    * In keeping with traditional use of scsi mode pages,
> > iscsi should
> > > not specify any default
> > >      settings for any mode pages that continue to exist for iscsi.
> > > "Default values" for mode
> > >      pages are target specific and should not be bound to
> > the protocol
> > > draft.
> > >
> > >    * MORE IMPORTANTLY, I read the default for EMDP as being set to 1
> > > :-<  This implies that
> > >      targets must be always prepared to deal with out of
> > > order data and
> > > initiators must be
> > >      prepared to deal with out of order data, unless the initiator
> > > performs a mode select to
> > >      disable it. [which defeats all the previous advantages
> > > gained thru
> > > mandating use of login
> > >      keys for other negotiations.]. A default, if it were to exist,
> > > should be 0. (in-order, by
> > >      default).
> > >
> > >    * Conservative specification of defaults for login keys along the
> > > following lines :
> > >                             MaxConnections = 1
> > >                             FMarker = "no"
> > >                             InitialR2T = "yes"
> > >                             BidiInitialR2T = "yes"
> > >                             ImmediateData = "no"
> > >                             DataPDULength = 16
> > >                             MaxOutstandingR2T = 1
> > >                             DataPDUInOrder = "yes"
> > >                             ErrorRecoveryLevel = 0
> > >                             SessionType = "normal"
> > >
> > >    * Should the iscsi protocol require a "Lun Control Mode
> > Page"? IOW,
> > > is an EnableCRN bit
> > >      required at the transport layer ? If the device server
> > capability
> > > is to be negotiated , I
> > >      suggest this bit be moved to a SCSI ULP Mode Page such as the
> > > "Control Mode Page", through a
> > >      T10 change as a part of the SCSI changes being driven by iscsi.
> > >
> > > Comments ?
> > >
> > > Thanks,
> > > Santosh
> > >
> > >
> > > Santosh Rao wrote:
> > >
> > > > There are the separate issues of :
> > > >
> > > >    * iscsi's specification of defaults for its mode pages
> > > which is not
> > > in line with mode page
> > > >      usage. This impacts the target's ability to enforce
> > > values other
> > > than the protocol
> > > >      specified default, if the initiator were to not use mode
> > > sense/select.
> > > >
> > > >    * default settings for login keys.
> > > >
> > > >    * Is there a need for the "LUN Control Mode Page" and whether
> > > "Enable CRN" should be in a
> > > >      transport specific mode page ?
> > > >
> > > > which need to be driven to closure as well.
> > > >
> > > > Regards,
> > > > Santosh
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >

--
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: Santosh Rao <[email-masked]>
Date: Mon, 01 Oct 2001 16:15:43 -0700

> I think your logic reverses if the support wanted is yes and the target is

How so ? 

Here are the scenarios when initiator wants to use immediate data and
target supports it, first with default ImmediateData set to "yes" and
then, with default Immediatedata set to "no". In both cases, a single
login handshake occurs. The only difference being that when
ImmediateData defaults to "yes", the key need not be sent in either
direction, whereas when it defaults to no, the ImmediateData key travels
in both directions in the login payload.

(FFP = FullFeaturePhase).

Default ImmediateData="yes"
---------------------------
Initator wants immediate data & targets supports it.
----------------------------------------------------

I -> T : (no key sent).
         (CSG=Operational, NSG=FFP). T=1

T -> I : (no key sent).
         (CSG=Operational, NSG=FFP). T=1

Default ImmediateData="no"
--------------------------
Initiator wants immediate data and target supports it.
-----------------------------------------------------------

I -> T : ImmediateData="yes"
         (CSG=Operational, NSG=FFP). T=1

T -> I : Immediatedata="yes"
         (CSG=Operational, NSG=FFP). T=1 (tgt has no more keys to send.)

Same number of handshakes in both cases. The only difference is that the
key is not sent in the first case, and key is explicitly exchanged in
the 2nd case.

> all they have and it requires extra handshakes.

Hmm.... The way I see it, an initiator that does not use ImmediateData
is a simple initiator and is more interested in seeing a simpler login
(completed in a single login req/rsp) than one that does support
ImmediateData. 

Those implementations that can do the extra work of supporting
ImmediateData can well do the extra work of negotiating for its use.

Regards,
Santosh

---

From: Santosh Rao <[email-masked]>
Date: Tue, 02 Oct 2001 09:48:04 -0700

Eddy,

My comments inline.

Thanks,
Santosh

Eddy Quicksall wrote:
> 
> Would all cases look like these? (If I made a mistake, please correct it).
> 
> Note that I am attempting to use the rule specified as:
> 
>  The negotiation MAY
>  proceed only up to the point where both parties can unequivocally
>  compute the result; continuing beyond this point is OPTIONAL.
> 
> And my interpretation of the rule is that when a responder gets a binary
> value, it can compare that against the default and can therefore compute the
> result and therefore does not need to respond if the result is what he
> wants.
> 
> Am I correct in my interpretation?
> 
> Eddy
> 
> ===========================
> Initiator wants immediate data & target supports it.
> ---------------------------------------------------
> 
> I -> T : (no key sent).
> yes * yes = yes

Correct. 

> 
> Initiator does not want immediate data & target supports it.
> I -> T : ImmediateData="no".
> no * yes = no

Correct. 

> 
> Initiator wants immediate data & target does not support it.
> T -> I : ImmediateData="no".
> yes * no = no

IMO, the above is in violation of the below rule found in Section 5.1 :

"An initiator that can operate without security and with all the
operational parameters taking the default values issues the Login with
the T bit set to 1, the CSG set to LoginOperationalNegotiation and NSG
set to FullFeaturePhase. If the target is also ready to forego security
and LoginOperationalNegotiation the Login response is empty and has T
bit set to 1, the CSG set to LoginOperationalNegotiation and NSG set to
FullFeaturePhase in the next stage. "

Thus, when a target cannot accept the defaults, it must float its own
values for the keys and in this case, it cannot perform a phase change,
[since it did not forego operational negotiation.]

Hence, per the current draft wording, I believe the above case would be
:
> T -> I : ImmediateData="no".
>          (CSG=Operational, NSG=Operational). T=0
  I -> T : (no key sent)
> yes * no = no

However, I do believe the above rule is flawed and should be modified. A
target must be allowed to perform a phase change any time it has no more
operational keys to negotiate and this should apply regardless of
whether defaults are sent or keys are explicitly sent. This should not
be dependent on whether the target had to forego operational
negotiation, whether it sent an empty login response or sent keys back ,
etc.

> 
> Initiator does not want immediate data & does not target support it.
> -------------------------------------------------------------------
> 
> I -> T : ImmediateData="no".
> no * yes = no

Correct. 

> ==========================
> Initiator wants immediate data & target supports it.
> yes * no = no

If the target supports immediate data it would send back a key
indicating the same.

Thus, the above should be :

T -> I : ImmediateData="yes"
	 CSG=Operational, NSG=FFP). T=1 (tgt has no more keys to send.)

yes * yes = yes

Note the difference in the target's phase change since in this case, all
key exchanges were explicit and the above "flawed" rule did not apply. I
suggest we make the modification suggested to the rule to enforce more
consistent target behaviour. 

> 
> Initiator does not want immediate data & target supports it.
> --------------------------------------------------------------
> 
> I -> T : (no key sent).
> T -> I : (no key sent)
> no * no = no

Correct.

> Initiator wants immediate data & target does not support it.
> I -> T : ImmediateData="yes".
> yes * no = no

Correct.

> 
> Initiator does not want immediate data & does not target support it.
> -------------------------------------------------------------------
> 
> I -> T : (no key sent).
> no * no = no

Correct.

On a seperate note, I believe the following changes are required in the
login process :

1) All these rules about both sides having to compute the result of the
negotiation are complicating the login procedure. Why not just follow
the standard practice of letting the target pick the negotiation result
and send it back in its response ? This allows for several advantages
like less code (only 1 side computing the result), ease of debugging
(target state is known to initiator), etc.

2) (1) also simplifies the case where responder silently drops a key
instead of responding to each key that was sent. I think a response from
the target for each key sent by the initiator is more explicit and
simpler. 

3) The draft should state that the initiator is the originator and the
target is the responder for ALL security and operational keys. (either
due to the use of defaults or mandated keys).

> Sent: Monday, October 01, 2001 7:16 PM
> To: John Hufferd
> Cc: [email-masked]

---

From: "Eddy Quicksall" <[email-masked]>
Date: Tue, 2 Oct 2001 12:08:17 -0400

The negotiation MAY

Eddy

I -> T : (no key sent).

T -> I : (no key sent).

yes * yes = yes

T -> I : (no key sent).

no * yes = no

I -> T : (no key sent).

yes * no = no

T -> I : (no key sent).

no * yes = no

T -> I : (no key sent).

yes * no = no

I -> T : (no key sent).

T -> I : (no key sent)

no * no = no

T -> I : (no key sent).

yes * no = no

I -> T : (no key sent).

T -> I : (no key sent).

no * no = no

Eddy

-----Original Message-----
From: Santosh Rao [mailto:[email-masked]]
Sent: Monday, October 01, 2001 7:16 PM
To: John Hufferd
Cc: [email-masked]
Subject: Re: iscsi : iscsi parameter default values

How so ?

I -> T : (no key sent).

T -> I : (no key sent).

Regards,
Santosh