Subject: [iSCSI]: Key negotiation procedure proposal

From: Luben Tuikov <[email-masked]>
Date: Thu, 23 May 2002 09:35:28 -0400

I'm proposing the following key negotiation procedure. It
preserves the no-renegotiation of keys rule and all the
rules of login request and response currently used in the
draft (12-91). It applies to any kind of value (list,
boolean, integer, etc.).

Simple implementations:

Simple implementations SHOULD close the connection right
after Reject has been communicated. This ensures that the
reason for closing has been communicated to the peer.

Regular implementations:

Core rule: A negotiation of a key=value pair is
complete/finished when both the Originator and Responder
have sent their values (non-reserved keywords).

The core rule implies the the following:

If a Responder replies with Reject, then the Originator
SHOULD NOT renegotiate the rejected key.

If a Responder replies with Reject, it SHOULD sent its value
of that key on the next reply to the Originator.

If an Originator finds the response to an offered key=value
pair to be unacceptable, it SHOULD send Reject and close the
connection.

Please note that the core rule above ensures that both the
Originator and Responder _know_ each other's values if the
connection is closed due to Reject. This will give the user
of each node more information to find out what went wrong.
That is, this kind of negotiation is exhaustive.

Let O: denote Originator and R: the Responder.

Example 1:
----------
O: v = x  -->
         <--   R: v = Reject

O: ""     -->
         <--   R: v = y

O: 1) v = y is OK, continue as normal,
   2) v = y is unacceptable,
   
   v = Reject -->
   close the connection (reason just given).

Example 2:
----------
O: v = x  --> 
         <--   R: v = y

O: 1) v = y is OK, continue, as normal,

   2) v = y is unacceptable,
   
   v = Reject -->

   close the connection (reason just given).

-- 
Luben

---

From: Luben Tuikov <[email-masked]>
Date: Thu, 23 May 2002 12:57:42 -0400

Bill Studenmund wrote:
> 
> My only concern with this is that we now have to keep more state around;
> we have to remember to send the key next negotiation, and the other side
> has to know to send a blank message as part of that next negotiation.

Yes, but as we already know negotiations are stateful!
I.e. you have to remember what you sent!

State: This can be accomplished by a linked list with status
in each element.
 
> Instead, why not just send it later in the first message?

Because the Originator needs to know that its offer was
Rejected. If it is a simple implementation it will close
the connection with TCP FIN, else it will stick around
for the rest of the negotiation, i.e. send more (other)
key=value pairs or sent empty PDU as per the protocol.
 
> We 1) make the negotiations happen faster, and 2) can keep less state
> around (O doesn't need to send "", and R doesn't need to remmeber to send
> v = y next time).

The Originator _has_ to send something as are the rules
of the protocol. This is very well described in the protocol.
It may send OTHER variables for negotiation or just and empty
PDU.

As I mentioned the Originator has to know its offer failed
so that it can then decide if it wants to close the connection
with TCP FIN, or stick around (simple implementation vs. complete one)
and send more (other) key=value pairs for negotiation.
 
> All the implementations I've seen break the login text up one key/value
> pair at a time, so seeing the same key twice won't cause a problem AFAICT.

But this is not the rule, it is the exception.

I negotiate all keys I can send in one packet!

Please also note that the proposed negotiation procedure
takes care of MULTIPLE key=value pairs being sent in each PDU,
and their responses.

This is important as not all implementations will
send one variable at a time! I.e. to conserve
time and bandwidth!

Those negotiation procedure rules allow for the connection
to survive even when all of the Originator keys have been Rejected,
but the Responder sent back values acceptable by the Originator.
I.e. complete ineroperability.

-- 
Luben
P.S. Those rules are the minimum consistent set of
a larger set of rules which allow complete interoperability.
Any mathematicians here? Bill, I suspect?

---

From: Bill Studenmund <[email-masked]>
Date: Thu, 23 May 2002 09:21:06 -0700 (PDT)

> Let O: denote Originator and R: the Responder.

i.e.

O: v = x  -->
         <--   R: v = Reject
         <--   R: v = y

>    close the connection (reason just given).

Except for the above, looks good.

Take care,

Bill

---

From: Bill Studenmund <[email-masked]>
Date: Thu, 23 May 2002 11:08:22 -0700 (PDT)

> I.e. you have to remember what you sent!

True, but you are proposing negotiation add an extra state, "partly sent."
i.e. we've sent a Reject, and need to send a value next time.

Before all we needed to remember is what we want to negotiate, what we
have negotiated, and what we'll accept for new negotiations. Now we have
to remember that we are in the middle of a reject/reoffer.

> State: This can be accomplished by a linked list with status

It can, but we don't have to, so why should we?

> key=value pairs or sent empty PDU as per the protocol.

That statement doesn't make sense.

The originator WILL know its offer was rejected; there's a "key=Reject" in
the login payload. I don't understand how the fact that's a reject isn't
clear to the originator.

I'm suggesting that we not wait for a future packet pair to send the value
we would be happy with. As you so strongly pointed out, we can have
multiple key=value tuples, why not repeat this key twice, once to indicate
reject, and once to indicate our value. Thus the originator gets all the
info it needs very quickly, we save a packet pair, and we don't need to
add more state.

i.e. addlen = snprintf(buffer, "%s=Reject\000%s=%s", key, key, newval);
to generate the text response for this negotiation step.

> and send more (other) key=value pairs for negotiation.

You snipped my diagram, so I can't explain how we took exactly different
spins on it.

As above, I wasn't talking about two packets from the responder, but two
key/value pairs that share the same key in the same text payload of one
login response.

> I negotiate all keys I can send in one packet!

So do I.

> takes care of MULTIPLE key=value pairs being sent in each PDU,

And my suggestion was to make use of that ability to make it so we don't
have the extra pair of packets you described. :-)

> I.e. complete ineroperability.

And I think it's a good idea. It also clarifies the use of Reject.

With the tweak I suggested above, it just happens faster. :-)

Take care,

Bill

---

From: Bill Studenmund <[email-masked]>
Date: Thu, 23 May 2002 11:50:12 -0700 (PDT)

> I.e. complete ineroperability.

One thing I wanted to make clear. I like Luben's idea.

The main thing I like about it is we have ambiguity in how we handle
negotiations now, and this change 1) removes the ambiguity, and 2)
encorporates elements that others seem to want in negotiation.

My one thought though is we can, as mentioned in other messages, close
Luben's Reject behavior more quickly; we can do it and keep total
negotiations down to one login pdu each direction. :-)

Take care,

Bill

---

From: Luben Tuikov <[email-masked]>
Date: Thu, 23 May 2002 15:00:02 -0400

> That statement doesn't make sense.

I didn't see that you meant to send the 
Responder's offer along with the responder's Reject.
I see what you mean.
 
> info it needs very quickly, we save a packet pair, and we don't need to

This certianly sounds reasonable. I'm happy with that.
So to make it more formal, the second implied rule should
read:

If a Responder replies with Reject, it SHOULD sent its value
of that key on the same reply to the Originator.

To recap:
---------------------
Simple implementations:

Regular implementations:

If a Responder replies with Reject, it SHOULD send its value
of that key in the same reply PDU to the Originator after the
key=Reject pair.

------------------

Well people, what do you think?

Bill, thanks for your analysis and support.

-- 
Luben

---

From: Bill Studenmund <[email-masked]>
Date: Thu, 23 May 2002 12:07:57 -0700 (PDT)

> of that key on the same reply to the Originator.

Sounds good. I think it's fairly clean, and direct for implementors to
work with. :-)

> Bill, thanks for your analysis and support.

No problem.

Take care,

Bill

---

From: Martins Krikis <[email-masked]>
Date: Thu, 23 May 2002 12:38:34 -0700 (PDT)

Luben,

I'm afraid I'm agreeing with Bill or Paul more here
than with you. 

The simple version looks perfectly acceptable.
I would not mandate closing the connection, since
I am also perfectly happy considering (on both
sides) the Reject-ed key to be "negotiated to
the current value" (i.e., negotiated, but the
value doesn't change on commit). But even
if closing is mandated, that's fine with me.
In fact, it is simpler so should be preferable.

If we go into the more complex version, where
you propose to let the originator know what
values would have made the responder happy... hmm...
First of all what you propose looks like
renegotion to me. It's now originating from the other
side, so it is kind-of justifyable and explainable.
However, just like Bill I don't like having to
send the key again in the next sequence. It does
require some new values for the state of the key.
Something like "received and rejected, but not yet
sent". Every new possibility for a state means
more checks upon reception and possibly even when
sending each key. Repeating the same key in the same
PDU sounds better, since then no new state is
necessary (sent+received) suffices. Or does it?
What if there is no space to send the key again
in the same PDU and it has to be left for the 
next time? Then it does need this extra state
again. Also, if it does get Reject-ed by the
other side (original Originator, now Responder),
we don't want to treat it as a no-renegotiation
violation, (unless we'll be closing the connection),
so we may need state.

To put it briefly, I think it still is too
complicated. The less state we can get away
with, the better. 

Speaking of mathematics, we have to express clearly
what the requirements are, then we can start proving
minimal sets of rules to satisfy those requirements.
Do we have a requirement to let the other side know
what values we would have accepted? I'm saying, let's
just Reject what we don't like, close the connection
if we wish, and if the other side doesn't like it,
its admin can give our admin a call and find what
values we like.

> > Example 2:

I don't quite get this one. How can y be unacceptable?
Did it violate the selection rules? If so, it's a
protocol error. We can just close the connection
or send the Reject PDU (not a key=Reject thing),
or send a Login Response with bad status, then close
the connection, it all depends of who we are.

In summary, I think I'm happy with what's currently
in 12-92, i.e., a key used twice by the same side
is a no-renegotiation violation.

  Martins Krikis, Intel Corp.

Disclaimer: these opinions are mine and may not be
            those of my employer

__________________________________________________
Do You Yahoo!?
LAUNCH - Your Yahoo! Music Experience
http://launch.yahoo.com

---

From: "Randy Jennings" <[email-masked]>
Date: Thu, 23 May 2002 13:50:59 -0700

A minor note on a weird possibility.

There is a remote chance that the PDU with both keys could span PDU's by
adding up to more than the MaxPDULength or whatever it ends up being called.
As such, I do not think you could mandate it be in the same PDU, but that it
be next in the Login stream.  I think this falls under, "I know what you
meant, just say it."

For example:
O ----> key=[some really long and obnoxious value that just barely fits in
the PDU size]/000

     <--------T
key=Reject/000
key=[some really long and obnoxious value that does not fit in the PDU size]

O --------> ""

     <--------T
[the remnants]/000

At least, that is how I remember it working.

Sincerely,
Randy Jennings

Data Transit

---

From: Bill Studenmund <[email-masked]>
Date: Thu, 23 May 2002 14:10:44 -0700 (PDT)

> be next in the Login stream.  I think this falls under, "I know what you

True, and I thought I was addressing that, but the words didn't do what I
meant.

I wanted to say that the second response should be later in the same login
negotiation blob. i.e. that which can span multiple PDUs.

Do we have a term for that?

Take care,

Bill

---

From: Luben Tuikov <[email-masked]>
Date: Thu, 23 May 2002 17:49:37 -0400

> I'm afraid I'm agreeing with Bill or Paul more here

In other words you'd use ``simple implementation''
rule. That is fine.
 
> renegotion to me.

It is NOT, since the Originator is NOT allowed
to send the same key twice, see implied rule 1.
(If a Responder replies with Reject, then the Originator
SHOULD NOT renegotiate the rejected key.)

> send the key again in the next sequence.

This has already been finalized -- see the
mailing list.

> next time?

You could put it right after: e.g.
[BHS+[opt]]key=Reject\000key=reponder_value\000...

> Then it does need this extra state

Please note when this happens, there's no
choice but to close the connection!
I.e. both sides rejected each other's
values -- this is the core rule, and the result
of rejection on both sides == non-operability.
 
Yes, you do need state at the originator.

> complicated. The less state we can get away

Well this is implementation issues.

I think that the negotiation schematic is pretty
simple, easiliy transferable into if () {} else {} etc.
 
> minimal sets of rules to satisfy those requirements.

I think that all here in this mailing list know
what the requrements are -- interoperability.

> if we wish,

There is no other choice Martins -- is it?
(Interoperability...)

> I don't quite get this one. How can y be unacceptable?

Say, its a mininum of some values and the Originator
found y to be too small.

Anyway, we have to cover all.

> is a no-renegotiation violation.

Currently it looks like ``simple implementations''
with the added bonus of loops.

First, Originator shouldn't ``guess'' and keep sending
same_key=value pairs indefinitely.

Second, in 4.2.2, 12-92, third change-bar from top
has a inconsistenly problem (contradiction)
as has been shown in this mailing list many times before.
Depending on MAY 1) Reject, 2) send admissible value
plus the next sentence ``not admissible by the
selection rules...'' we get to a contradicion
of negotiation...

?

-- 
Luben

---

From: Martins Krikis <[email-masked]>
Date: Thu, 23 May 2002 15:52:35 -0700 (PDT)

Luben,

I'm not sure whether what you are proposing is or
is not an improvement to the current situation,
but unless it is perfect, we have no chance of
changing anything. So here are my comments.

> In other words you'd use ``simple implementation''

No. I'm not that kind of guy. I'd make an
implementation that could behave either as the
simple one or as the complex one, depending on
how it's configured to behave. :-)

> SHOULD NOT renegotiate the rejected key.)

But the Responder may, yes I know, I understand.
All I was trying to say is that after a key
has traveled in both direction seeing it travel
again _looks_ like a renegotiation. Period.
That's true even for my (later and not too serious)
key=? reinstantement proposal. I understand
that you could define what a renegotiation is
and exempt this case because it hasn't yet
traveled in that direction together with an
unreserved value. But then we need more state
than now. So, I understand that you don't call
this a renegotiation, but I don't consider it
beautiful either. Sorry.

> This has already been finalized -- see the

Mails crossed over the wires. I can't help the lag.

> [BHS+[opt]]key=Reject\000key=reponder_value\000...

Of course I would put it right after. Why would I
wait? But that does not guarantee that it will fit.
So what now? Adjusting the state, of course.

> of rejection on both sides == non-operability.

I know! But I should just close the connection
now and log that I have met an incompatible Originator

instead of sending 
X-vendor-blah-login-reject-reason=duplicate_key
and then close it. Subtle nuanses maybe, but my
admin may like to know.

> Yes, you do need state at the originator.

A super-nice implementation needs it on both sides. 
And more than currently. That's the reason I'm not
yet crazy about this proposal. Once the protocol
can work with as much or less state, I'll be more
agreeable.

> simple, easiliy transferable into if () {} else {}

I think it is more complicated than the current
situation, where I've got already plenty of all
kinds of flags. I'm reluctant to introduce more.

> what the requrements are -- interoperability.

If we're speaking mathematics, we will need to
get more precise.

> There is no other choice Martins -- is it?

Other choice than closing? Yes---we can keep it
open and consider it negotiated for commit
purposes, yet the value won't change. Other
choice than interoperability? No, I guess
there isn't. But interoperability can be easier
achieved with sides not sending junk to the
other side, than requiring rejecting sides
to show what they would have liked...

> Say, its a mininum of some values and the Originator

It shouldn't find it too small. I don't think that's
allowed. For those kind of things we have ranges.
If using a range in negotiation is not allowed, 
then a compliant implementation should be happy
with any number that can be obtained by applying
the selection rule (min or max, so far) to the
number it supplied and to any number in the acceptable
range.

Alright, so here we have a chance to get more words
in the draft (whichever way it is).

> with the added bonus of loops.

Your example of loops (unless there have been more)
was based on the possibility of renegotiation.
The way I understand it (confirmed by a private
email from Julian) renegotiation is now disallowed.
Yes, even for Rejects, Irrelevants and NotUnderstoods.
The phrase that Pat suggested got added in 12-92,
except the last sentence. I have asked Julian why.

Plus, loops aren't as tragic as is complicating
the protocol. I don't mind having a configurable
counter that puts an upper bound on the number
of negotiation rounds. That's one variable per
all keys, instead of additional if/else cases
to be done for each key. 

But the way I see it, we cannot have loops anymore.
So what we have is like your "simple version",
except that closing the connection is not mandated. 
And simplicity is good. If we
mandate closure, I'm fine with that too. In fact,
I think it is the best, but not by an amount
worth fighting for.

> same_key=value pairs indefinitely.

He no longer can. That would be a renegotiation.

> selection rules...'' we get to a contradicion

Yes, true, not the clearest text. However, I
actually object to the whole idea of cutting any
slack to non-conforming Originators by not Reject-ing
their keys and sending a "good value" instead.
I don't think it is good. So I'm for the removal of
this possibility first, and only if somebody
convinces me about the value of it, then for
making the text more precise.
Take care,

---

From: Santosh Rao <[email-masked]>
Date: Tue, 28 May 2002 12:38:56 -0700

>  However, I

I agree with all the points Martins Krikis makes above. I strongly
oppose the proposal made by Luben since it is too complex and it
complicates the base negotiation scheme in an attempt to accomodate
non-interoperable implementations.

In order to communicate to the other side as to what was the problem
with the login negotiation, there are several better/simpler ways to do
this than to have to send the key back with the value the target would
have supported. This does amount to re-negotiation and causes added
complexity on both sides.

If the target does not negotiate what the initiator requested, log an
error message, issue a reject response and close the connection. The
support personnel who diagnose the problem will be able to root cause
the problem from the error log. Alternatively, consider enhancing the
iscsi MIB statistics to provide better fault isolation data, so that the
MIB statistics can be used to determine the root cause of the problem.

There is no need to complicate the base login negotiation protocol in an
attempt to deal with such non-interoperable implementations.

This is not an issue of simple vs. complex implementations. Its a choice
that should be made regarding what is the simplest way to provide a
mechanism to fault isolate non-interoperable implementations.

Also, in such cases, the market will decide which implementations will
sell and those target implementations that cannot support what the
initiator is negotiating will not sell in that server/OS market. The
initiator is seldom going to change its negotiation options in an
attempt to be able to interoperate with that target.

Lastly, I think we ought to be freezing the spec and stop making changes
that are'nt bug fixes. If we don't do that, we are never going to make
it to RFC and ship some iscsi products. 

- Santosh

-- 
The world is so fast that there are days when the person who says 
it can't be done is interrupted by the person who is doing it.
	~ Anon