Subject: Re: ISCSI: CmdSN in non-leading login

From: "Julian Satran" <[email-masked]>
Date: Sat, 11 May 2002 01:06:33 +0300

Logins are all immediate. CmdSN is the current one and not advancing by 
the sender and should not be checked by the receiver.
Nothing special needs to be mentioned.

Julo

"Mark S. Edwards" <[email-masked]>
Sent by: [email-masked]
05/10/2002 12:51 PM
Please respond to "Mark S. Edwards"

        To:     [email-masked]
        cc: 
        Subject:        ISCSI: CmdSN in non-leading login

I am looking for clarification with regard the value that an initiator 
should put in the CmdSN field of a login request for a non-leading 
connection.

There is a certain ambiguity about 9.12.8.  The text describes how in a 
leading connection, the target takes must accept and use a non-zero CmdSN, 

but it does not explain what should be present for a non-leading 
connection.

There are a couple of problems here for the target.  At what point is a 
non-leading connection considered to be part of the session ?  Is it the 
moment that the login request is received with a non-zero TSIH ?  Or is it 

only when the non-leading login succeeds and it enters full feature phase 
?

So, should an initiator set the CmdSN in the first login request to zero 
and only synchronise with the session command stream after full feature 
phase is established ?  This is my preferred option.

What happens if the initiator tries using the current session command 
sequence number is that whilst the login negotiation occurs, other 
connections within the session can be issuing new commands, so by the time 

that the login is finished the CmdSN exchanged in the initial request is 
invalid anyway.

I would like to see something along the lines that for a non-leading 
connection, the CmdSN field MUST be zero and that the connection can not 
be 
considered part of the session until full feature phase is entered, at 
which point any commands issued on the connection are now synchronised 
with 
the session command sequence number as observed by all other existing 
connections on the session.

regards,

Mark.

---

From: "John Hufferd" <[email-masked]>
Date: Fri, 10 May 2002 17:38:06 -0700

The text as it is written has not been updated since we at one point had
the Login command marked as and Immediate command.  Therefore, it always
had to pick up the next value that would be assigned to the next non
immediate command.  Also since it was an immediate command the ExpCmdSN was
not advanced so it did not effect the commands in the rest of the session.
They would continue with the next CmdSN as usual.  Login is still not
suppose to advance the ExpCmdSN, so there should still not be a problem.

When we removed the flag that said that it was an immediate command, the
wording in the draft about the Initial Login was not adjusted.

I suggest that the only thing that needs to be changed is the following:
At the end of the first paragraph under 9.12.8, we should add, the words
"Similar rules exist for the Leading Login for a non leading connection,
that is, the next CmdSN value is used for the Leading Login of the non
leading connection, and that same number is used with all the subsequent
Login PDUs on that connection. But at completion of the Login of the
connection, the ExpCmdSN is not advanced and the subsequent commands take
the next CmdSN value in the session."

This is what we were all doing, and simply did not change anything with the
Immediate bit was dropped from the Login PDU.  I suggest we leave it with
that operational intent, since it was so long ago that we establish this
way of operating I can not remember exactly why, but would not like to be
surprised sometime in the future, and I do not see why we would have to
change it.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Mark S. Edwards" <[email-masked]>@ece.cmu.edu on 05/10/2002 10:38:07
AM

To:    John Hufferd/San Jose/IBM@IBMUS
cc:    <[email-masked]>
Subject:    Re: ISCSI: CmdSN in non-leading login

John,

If an initiator opens a new connection on a session and in the first login
request uses the current session CmdSN, the lifetime of that CmdSN
continues all the way through until the completion of the first
non-immediate command on that new connection.

This means that the initiator has to reserve the use of that CmdSN for the
first non-immediate command on that particular session.  That's stupid.

Especially bearing in mind that it could be pumping commands like crazy
down other open connections in the session.

There is also another side effect, if the login takes any length of time,
like computing a DH or having to make external calls to a Radius or a
Kerberos server, then because this CmdSN is outstanding, the other
connections in the session will block as soon as the command window is
filled.

The final sentence in 9.12.8 is meaningless for a non-leading
connection.  The target either has to block the whole session until the
login is completed, or run the risk of corrupting session state.

Mark

At 10:14 AM 5/10/2002 -0700, John Hufferd wrote:

>I do not see the issue, it works as is, it does not effect the first non
>immediate command unless it completes as a authorized connection. Again, I
>do not see the issue, in fact it works as is.
>
>.
>.
>.
>John L. Hufferd
>Senior Technical Staff Member (STSM)
>IBM/SSG San Jose Ca
>Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
>Home Office (408) 997-6136, Cell: (408) 499-9702
>Internet address: [email-masked]
>
>
>"Mark S. Edwards" <[email-masked]>@ece.cmu.edu on 05/10/2002 09:47:53
>AM
>
>To:    Bill Studenmund <[email-masked]>
>cc:    <[email-masked]>
>Subject:    Re: ISCSI: CmdSN in non-leading login
>
>
>
>At 09:13 AM 5/10/2002 -0700, Bill Studenmund wrote:
> >On Fri, 10 May 2002, Mark S. Edwards wrote:
> >
> > > moment that the login request is received with a non-zero TSIH ?  Or
is
>it
> > > only when the non-leading login succeeds and it enters full feature
>phase ?
> >
> >I think the target shouldn't let anything from a new connection
influence
> >things until after at least the security phase has been passed.
Otherwise
> >we have a security hole. After operational negotiation would probably be
> >best.
>
>Good, as far as I can see, a new connection should not be allowed to
>influence any session context until the connection reached full-feature
>phase.  But this is a problem if we have to follow the CmdSN rules of
>9.12.8.   See next comment.
>
>
> > > connection, the CmdSN field MUST be zero and that the connection can
>not be
> >I thought login counted as a command, so it got its own command number,
in
> >the stream of all other commands. ??
>
>
>For a leading connection the CmdSN, whether zero or non-zero, is regarded
>as a primer with which to set the initial session command sequence number,
>all the login requests exchanged in the negotiation, no matter how many
>carry the same CmdSN as does the first non-immediate command.  In other
>words, the first pdu in a leading connection does influence session state.
>
>So if we agree that a non-leading connection can not influence session
>state until full-feature phase, then we have to also state that the rule
in
>9.12.8 about the first non-immediate command carrying the same CmdSN as
the
>initial login request can not work for a non-leading connection.  In this
>case, the first non-immediate command must be set from the next logical
>value in the session context.  I would like the spec to have this added
and
>to explicitly say that the CmdSN in a login request for a non-leading
>connection is ignored by the target.  I'd really prefer that the actual
>value be zero, but that's not necessary from a protocol perspective if the
>target ignores the field.
>
>Mark.

---

From: "John Hufferd" <[email-masked]>
Date: Fri, 10 May 2002 18:57:04 -0700

I have a correction to the previous note (below).  We removed the Immediate
bit but there is clear statement that "Login requests are always considered
as immediate.".  (This means they always get the next CmdSN but do not
advance the ExpCmdSN.)  The rest of the message is correct, and the
suggested update of the wording, is approprate.  It will not, however,
change the way things work.
.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]
---------------------- Forwarded by John Hufferd/San Jose/IBM on 05/10/2002
06:49 PM ---------------------------

John Hufferd
05/10/2002 05:38 PM

To:    "Mark S. Edwards" <[email-masked]>
cc:    <[email-masked]>
From:  John Hufferd/San Jose/IBM@IBMUS
Subject:    Re: ISCSI: CmdSN in non-leading login  (Document link: John
       Hufferd)

.
.
.
John L. Hufferd

John,

Mark

is
influence
Otherwise
in
in
the
and

---

From: "Julian Satran" <[email-masked]>
Date: Sat, 11 May 2002 11:03:15 +0300

John,

The current text starts by stating that login requests are immediate.

It then states:

CmdSN is either the initial command sequence number of a session (for the 
first Login request of a session - the "leading" login) or the command 
sequence number in the command stream (e.g., if the leading login carries 
the CmdSN 123 all other Login requests carry the CmdSN 123 and the first 
non-immediate command also carries the CmdSN 123). 

It has all the information needed.
I am puzzled by your comments.

Julo

John Hufferd/San Jose/IBM@IBMUS
Sent by: [email-masked]
05/11/2002 03:38 AM
Please respond to John Hufferd

immediate command.  Also since it was an immediate command the ExpCmdSN 
was

This is what we were all doing, and simply did not change anything with 
the

.
.
.
John L. Hufferd

John,

Mark

At 10:14 AM 5/10/2002 -0700, John Hufferd wrote:

> > > There are a couple of problems here for the target.  At what point 
is
a
is
influence
Otherwise
> > > connections within the session can be issuing new commands, so by 
the
>time
> > > that the login is finished the CmdSN exchanged in the initial 
request
in
in
the
and

---

From: Eddy Quicksall <[email-masked]>
Date: Sat, 11 May 2002 13:50:52 -0400

It seems like the current wording is already clear. Maybe an example would
be appropriate in appendix B.

Eddy

-----Original Message-----
From: John Hufferd [mailto:[email-masked]]
Sent: Friday, May 10, 2002 9:57 PM
To: Mark__S.__Edwards"
<[email-masked]/@boulder.ibm.com"@twestrelay04.boulder.ibm.com
Cc: [email-masked]
Subject: Re: ISCSI: CmdSN in non-leading login

John Hufferd
05/10/2002 05:38 PM

.
.
.
John L. Hufferd

John,

Mark

is
influence
Otherwise
in
in
the
and

---

From: "John Hufferd" <[email-masked]>
Date: Sat, 11 May 2002 12:25:22 -0700

Julian,  (watch out long note to follow)
Sometimes our messages pass by each other in the night.  I had corrected my
note to indicate that I had overlooked the very obvious statement that we
kept in the draft that the Login command was immediate.  But the rest of
the note I think is correct.

That is, the command that follows the completion of the non
leading-connection Login should take its CmdSN from the next command
sequence number in the "Session" stream, not the CmdSN of the Login for
that connection.

Here is the thinking,

1. The second connection is started while commands are flowing on the
Leading connection,
2. The CmdSN in the stream when the Login for the second connection is
started is lets say 123.
3. Since the Login command is immediate, and does not change the ExpCmdSN
then one of three things can happen
    a. Initiator uses the same CmdSN for the next command that it sends on
the first connection, etc. (123, 124, 125, etc.)
    OR
    b. Initiator uses the next CmdSN 124 on the leading connection and
thereby creates a delivery hole, for the target until the CmdSN 123 is
finally received on the secondary connection and then all the commands
123----125, etc, are delivered
    c. No commands are sent on any connection until the Login of the
secondary connection is completed and the next command is given the CmdSN
of 123
4. If 3a is done, then the first command after the delivery will use the
next CmdSN in the stream and no blockage would have occurred.
5. If 3b is done there will be a blockage of the delivery of commands until
the secondary Login in completed
6. If 3c is done, then the first command after the delivery will use the
same CmdSN (123) as the Login, but the commands will have a blockage at the
initiator until the secondary Login is completed.

So I believe, that IF the subsequent commands in a secondary connection has
to have the same CmdSN as the Initial Login of that  connection, either all
commands in the entire session must stop being sent on other connections,
or commands will not be delivered until the delivery hole is closed at the
end of the secondary Login.

If we were to handle the Login for secondary connections like other
immediate commands,  the 3a. path would be done (the ExpCmdSN would not be
advanced, but other commands will continue and they will advance the
ExpCmdSN as normal, and when the next  non immediate command is sent on the
same connection as the immediate command, it is simply given the very next
CmdSN from the session stream.

The problem of treating the secondary connection Login exactly like the
leading Connection, when it comes to the next command, is that, unlike the
leading connection, other things continue to move on the other connections.

Therefore, in my opinion the correct way of interpreting the current
wordage in 9.12.8 is at the point it says:
  "....and the first non-immediate command also carries the CmdSN 123.)"
we should read that as "... the next non-immediate command IN THE SESSION
also carries the CmdSN 123)".  In this way the words mean the same for both
the leading connection and secondary connection.

This is how I believed the implementations have been built.

Anyway, that was the essences of my words where I suggested:

Now perhaps we can have some better words, perhaps the words but clearly
the current word have confused some folks.  Perhaps even me :-)

.
.
.
John L. Hufferd

Julian Satran/Haifa/IBM@[email-masked] on 05/11/2002 01:03:15 AM

To:    John Hufferd/San Jose/IBM@IBMUS
cc:    [email-masked], "Mark S. Edwards" <[email-masked]>,
       [email-masked]
Subject:    Re: ISCSI: CmdSN in non-leading login

John,

It then states:

Julo

    John                                                                   
    Hufferd/San                   To:        "Mark S. Edwards"             
    Jose/IBM@IBMU         <[email-masked]>                            
    S                             cc:        <[email-masked]>             
    Sent by:                      Subject:        Re: ISCSI: CmdSN in      
    owner-ips@ece         non-leading login                                
    .cmu.edu                                                               
                                                                           
    05/11/2002                                                             
    03:38 AM                                                               
    Please                                                                 
    respond to                                                             
    John Hufferd                                                           

.
.
.
John L. Hufferd

John,

Mark

is
influence
Otherwise
in
in
the
and

---

From: "John Hufferd" <[email-masked]>
Date: Sat, 11 May 2002 22:53:15 -0700

Julian,
You need to make it clear that the Initiator CmdSN that follows the
secondary connection Login, is the next CmdSN in the session.  The current
text make it sounds that the next non immediate command on the same
connection as the secondary connection Login, must have the same CmdSN as
that Login.  And that is NOT the case.

The problem words are
"(e.g., if the leading login carries the CmdSN 123 all other Login requests
carry the CmdSN 123 and the first non-immediate command also carries the
CmdSN 123.)"

If you don't like the other words I suggested before then at least change
the statement to:
"(e.g., if the leading login carries the CmdSN 123 all other Login requests
carry the CmdSN 123 and the first non-immediate command in the session also
carries the CmdSN 123.)"

But I think my other words are better :-)

.
.
.
John L. Hufferd

Julian Satran@IBMIL
05/11/2002 09:02 PM

To:    John Hufferd/San Jose/IBM@IBMUS@IBMDE
cc:    [email-masked], "Mark S. Edwards" <[email-masked]>,
       [email-masked]
From:  Julian Satran/Haifa/IBM@IBMIL
Subject:    Re: ISCSI: CmdSN in non-leading login  (Document link: John
       Hufferd)

John,

My only point was that there is enough information in the section to
describe correct behavior.
And immediate commands do not create any blockage regardless of their
CmdSN.

I am still trying to understand what your point was (is).

Julo

                      John                                                                                             
                      Hufferd@IBMUS            To:      Julian Satran/Haifa/IBM@IBMIL                                  
                                               cc:      [email-masked], "Mark S. Edwards" <[email-masked]>,      
                      05/11/2002 10:25         [email-masked]                                                   
                      PM                       From:    John Hufferd/San Jose/IBM@IBMUS                                
                                               Subject: Re: ISCSI: CmdSN in non-leading login(Document link: Julian    
                                               Satran - Mail)                                                          

                                                                                                                       

Here is the thinking,

.
.
.
John L. Hufferd

John,

It then states:

Julo

    John                                                                   
                                                                           
    05/11/2002                                                             
    03:38 AM                                                               
    Please                                                                 
    respond to                                                             
    John Hufferd                                                           

.
.
.
John L. Hufferd

John,

Mark

is
influence
Otherwise
in
in
the
and

---

From: "John Hufferd" <[email-masked]>
Date: Sun, 12 May 2002 10:52:45 -0700

That works for me.
Thanks.

.
.
.
John L. Hufferd

Julian Satran@IBMIL
05/12/2002 04:20 AM

John,

Here is the text I suggest for 9.12.8

   CmdSN

       CmdSN is either the initial command sequence number of a session
       (for the first Login request of a session - the "leading" login) or
       the command sequence number in the command stream if the login is
       for a new connection in an existing session.

       Examples:

          - A leading login phase - if the leading login carries the CmdSN
            123 all other login requests in the same login phase carry the
            CmdSN 123 and the first non-immediate command in
            FullFeaturePhase also carries the CmdSN 123.

          - A non-leading login phase - the current CmdSN at the time the
            first login on the connection is issued is 500 - the login
            request carries CmdSN=500 the second login request carries a
            CmdSN not lower than 500 (higher if non-immediate requests
            where issued in the session between the first and the second
            request in the new login phase) etc..

       If the login request is a leading login request the target MUST use
       the value presented in CmdSN as the target value for ExpCmdSN.

 Regards,
 Julo

---

From: Eddy Quicksall <[email-masked]>
Date: Sun, 12 May 2002 20:41:35 -0400

Julian,
 
I like your 1st paragraph but I think it so clear that the examples are not
necessary.
 
Eddy

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Sunday, May 12, 2002 7:24 AM
To: John Hufferd
Cc: [email-masked]; Mark S. Edwards; [email-masked]
Subject: Re: ISCSI: CmdSN in non-leading login

John, 

CmdSN 

CmdSN is either the initial command sequence number of a session (for the
first Login request of a session - the "leading" login) or the command
sequence number in the command stream if the login is for a new connection
in an existing session. 

Examples: 

- A leading login phase - if the leading login carries the CmdSN 123 all
other login requests in the same login phase carry the CmdSN 123 and the
first non-immediate command in FullFeaturePhase also carries the CmdSN 123. 

- A non-leading login phase - the current CmdSN at the time the first login
on the connection is issued is 500 - the login request carries CmdSN=500 the
second login request carries a CmdSN not lower than 500 (higher if
non-immediate requests where issued in the session between the first and the
second request in the new login phase) etc.. 

If the login request is a leading login request the target MUST use the
value presented in CmdSN as the target value for ExpCmdSN. 

Regards, 
Julo

---

From: Eddy Quicksall <[email-masked]>
Date: Mon, 13 May 2002 17:44:40 -0400

From Julian's re-write, it seems that login requests would just carry the
current CmdSN. For a non-leading login, it would follow that the CmdSN can
increase if there are other non-immediate commands in progress in the
session. The leading login's CmdSN would not increase because it would be
impossible to have any commands in progress.

For the leading login, there is a special case where the CmdSN is used as
the first ExpCmdSN.

That way, nothing gets "stuck" and I see very clean implementation.

Eddy

-----Original Message-----
From: Bill Studenmund [mailto:[email-masked]]
Sent: Monday, May 13, 2002 1:57 PM
To: Julian Satran
Cc: John Hufferd; [email-masked]; Mark S. Edwards;
[email-masked]

> And immediate commands do not create any blockage regardless of their

Don't they though? Say I have a queue depth of 2, and I start a secondary
login with a CmdSN of 1000. I can then send other immediate commands at
1000, and a non-immediate command at 1000. I can then also send a
non-immeidate command at 1001. But can I send a command at 1002 before the
login has finished? If we increase the queue depth, there will still be
some other command number that would overflow it.

Say the login takes a while. Won't the login pin the queue at 1002 until
it's done? Now let the login take a while, it requires talking to say a
low kerberos server or doing a DH computation. If we really pay attention
to CmdSN, then the slow login process can effectively take the device
off-line while it completes, since we can't enqueue new commands.

Worse yet, say the secondary connection is NOT from our initiator, but
from an imposter. All the rogue has to do is get the TSIH right & guess a
CmdSN, and stall at trying to add another connection. Say do leading
negotiation, but just never send any security negotiations. The queue will
eventually become pinned until the target times out. Nice little DOS
attack; all you have to do is get the TSIH right & guess a CmdSN. You
don't have to do any security negotiation.  :-)

I'd like to suggest that we just ignore CmdSN on non-leading sessions,
beyond the fact that a well-behaved initiator should pick a CmdSN in its
current command window. We certainly shouldn't pay attention to it before
security negotiations have completed. :-) Oh also no commands should be
sent over the new connection with a CmdSN lower than the one used in the
new connection.

Take care,

Bill

---

From: "John Hufferd" <[email-masked]>
Date: Mon, 13 May 2002 13:20:11 -0700

Perhaps our notes passed in the night, but that is what I attempted to say
in my last note, your comments are correct.

.
.
.
John L. Hufferd

"THALER,PAT (A-Roseville,ex1)" <[email-masked]>@ece.cmu.edu on
05/13/2002 12:05:53 PM

To:    Bill Studenmund <[email-masked]>, Julian
       Satran/Haifa/IBM@IBMIL
cc:    John Hufferd/San Jose/IBM@IBMUS, [email-masked], "Mark S. Edwards"
       <[email-masked]>, [email-masked]
Subject:    RE: ISCSI: CmdSN in non-leading login

John,

I don't understand your point. An outstanding immediate command doesn't
affect the progress of ExpCmdSN or MaxCmdSN. In the situation you describe,
the target would act on the non-immediate commands 1000 and 1001 when it
got them. It would then increase MaxCmdSN as it empties those non-immediate
commands from its queue.

The login immediate command with CmdSN would be processed when it arrives
regardless of the current non-immediate command's CmdSN. Any immediate
command can arrive with a CmdSN lower than ExpCmdSN or higher than
MaxCmdSN. The use of CmdSN in immediate commands is to allow the immediate
command to have a relationship to the non-immediate commands - e.g. CLEAR
TASK SET.

Pat

-----Original Message-----
From: Bill Studenmund [mailto:[email-masked]]
Sent: Monday, May 13, 2002 10:57 AM
To: Julian Satran
Cc: John Hufferd; [email-masked]; Mark S. Edwards;
[email-masked]

Take care,

Bill