Subject: RE: iSCSI: Negotiation clarifications still needed

From: [email-masked]
Date: Fri, 24 May 2002 15:16:53 -0600

Martins,

Comments referenced by the same items Martins used.

1. Julian sent an email saying he would put the text 
I proposed in (though the text you quoted is not the 
whole text).

2. I think that the principle we have been using on 
text negotiation was that each key negotion is a 
separate item. Your proposal would be counter to that 
and I don't think it would be an improvement. The
target should be allowed to respond to any complete
key-value pair it has received. When a key-value
pair is straddling the PDU bondary, then it shouldn't
respond to that key until the complete key-value pair
has been received.

There is one potential corner case issue that should
to be covered. Targets can initiate keys. If key-value
pairs didn't straddle PDU boundaries, then ensuring 
that there is a clear originator for each offered key
is easy. You can originate any key that you haven't 
received an offer of from your partner. But now keys
can straddle PDUs. If the text between the last separater
and the end of the PDU is Ma, then you don't know what
key your partner has started to offer. If the partner 
was starting to offer MaxBurstSize and you offer it in
the next PDU of the exchange, both sides may think they 
are originator.

I suggest one of the following
a) don't allow keys to straddle PDU boundaries.
b) don't allow originating a key when the last login 
PDU ended in a partial key.
c) don't allow offering a key where the start of the key
matches a partial key at the end of the last login 
PDU.

3. Yes, we noticed a little while ago that losing a
packet at the end of negotiation could hang things up
though the concern is mainly for a full feature phase
negotiation. Looking at 6.8, any timeout during
negotiation causes the login and its TCP connection to
be terminated. The whole negotiation process (see the 
point about origninators in 2) depends upon a one-by-one
exchange of PDUs. PDU loss has to terminate it. 

Therefore, the target commits to the end of login
as described for T5 target. It has sent the final 
login response with a status of zero. Moves to S5.
If the login response doesn't get to the initiator,
then either the initiator will close the connection
due to the timeout. Since the target is in S4, loss 
of the transport connection will cause it to go to
S8 and R1 of the cleanup state machine. It presumably
will not take the M2 transition because the intiator
isn't going to do cleanup for a connection it thinks
wasn't in full feature phase. It will take M1 due to
timeout - not elegant but good enough.

The concern was Full Feature Phase negotation. Until 
negotiation ends, it can be reset and no values change.
When the target sends the last Text Response PDU, then
it thinks negotiation has ended and it applies the
new values. If that PDU doesn't reach the initiatior,
then it terminates the entire negotiation and continues
to use the old values. The two ends are using different
values.

We decided to not raise this as an issue because it is
such a corner case - we are operating over a reliable
connection so PDUs shouldn't be lost (unless the whole
path goes down in which case it doesn't matter). Also,
there are few values exchanged during full feature so
it isn't worthwhile to add complexity. 

Regards,
Pat

-----Original Message-----
From: Martins Krikis [mailto:[email-masked]]
Sent: Friday, May 24, 2002 12:39 PM
To: Julian Satran
Cc: [email-masked]
Subject: iSCSI: Negotiation clarifications still needed

The previous thread went on too long, but since it
has now quieted down, I'll conjecture that the
following are the aspects that still need to be
addressed.

1. Not everybody seemed to have noticed that it is
   NOT legal to send the same key again, if it
   has once been negotiated (including negotiations
   that end with a reserved value (Reject, 
   Irrelevant or NotUnderstood).

I think it would benefit the draft to add the
sentence that Pat proposed to paragraph 5 or page
72 in 12-92: "Sending the key again would be a
re-negotiation". That I think would make it crystal
clear.

2. When the key=value pairs that Originator is
   sending are broken across multiple PDUs, it
   is not clear whether the Responder may start
   responding to the keys as soon as it receives
   them or whether it should send blank PDUs back
   (as in the example on page 164 of 12-92) until
   it gets a PDU, the data part of which ends in
   a NUL byte (thus signaling that there are no
   broken key=value pairs at the end of it).

I am proposing that the draft should make it explicit
that only blank PDUs are to be sent. This allows
decoupling of key=value generation from
their encapsulation in PDUs (i.e., the generating
logic need not worry about whether a key=value
pair will fit and go out in this PDU or has to
be retained to go out in the next). I can explain
in detail why this is important (it has to do
with teh possibility of receiving the "just-about
outgoing" keys) but I'm keeping this "brief".

Furthermore, it is my feeling that instead of
checking the last bytes of a PDU for NUL, it
would be better if the end-of-data was marked by
a flag in the header. This way encapsulation will
be simpler---just put as much data in the PDU as
fits there and raise the flag if it isn't all,
instead of checking whether it ends in a NUL and
possibly shortening data to make it not end like
that.

3. There is an opinion that on page 73 of 12-92,
   the phrase that says "or the responder may
   select an admissible value" is in contradiction
   to the very next sentence. There is also an
   opinion that this phrase is entirely unnecessary
   and detrimental to achieving broad 
   interoperability (I call it "cutting slack to
   misbehaving or incompatible originators").

I don't have a suggestion since I consider the
"feature" that this phrase allows of little 
importance to a properly built iSCSI node.

4. This is new. When doing Text Request/Response
negotiations (i.e., in FFP), it seems 
that the Initiator commits to the new values
when it receives a response from the Target with
the F bit set. It is unclear when the Target should
commit. Should it switch to using the new values
as soon as it sends its response with the F-bit
set, or should it do so only when it knows that
the Initiator received its response? 

Commiting right away is simpler and since responses
with F-bits set have TTT=0xffffffff and thus
may not be reset, sounds plausible. If the
values have importance on the next reception,
it may also be important to commit timely. However,
what if the Initiator doesn't get this response? 
Target now has committed, Initiator hasn't. Committing
later puts the burden on Initiator to send something
effectively telling "I've received your final
response". Otherwise the Target will time out and
not commit. This response can get lost too.

Basically, it is beginning to look a bit like
(what was it called?) "distributed consensus problem"?
I think it goes like this:

Two generals that are on oposite sides of the
enemy want to synchronize their attack, and
start sending messengers through with messages
like 
  "attack at dawn->", 
  "<-ok, attack at dawn",
  "I know you know we attack at dawn->",
  "<-, I know you know I know we attack at dawn",
  etc., etc., ...
But at no point can they commit yet...

Is anybody else worried about this?
Anyway, so when should a target commit? Page 83
of 12-92 is the relevant reference.

Thanks,

  Martins Krikis

Disclaimer: these opinions are my own and may
            not be those of my employer.

__________________________________________________
Do You Yahoo!?
LAUNCH - Your Yahoo! Music Experience
http://launch.yahoo.com

---

From: Martins Krikis <[email-masked]>
Date: Fri, 24 May 2002 15:29:23 -0700 (PDT)

> I proposed in (though the text you quoted is not the

Julian put in part of the text (in 12-92). I quoted 
the part that he didn't put in. I'm wondering why.
Julian, I asked you once personally, and asked on
the list again. Why don't you put the remaining part
of Pat's text in so that there are no doubts
about this issue?

> and I don't think it would be an improvement.

I thought so too until yesterday, in fact my
implementation was NOT sending just blank PDUs,
but I realized it had a subtle bug, having to do
with what you describe as a "corner case" below.

> The

of course, since it may not even know for sure
what the straddling key or value is...

> received an offer of from your partner.

That's roughly why I made the proposal to look for
a "data-end" flag.

> But now keys

And that is bad. Also, if we don't consider
MaxBurstSize as offered, then the side that put 
"Ma" in the PDU is in a somewhat uneasy situation for
that key's state... it cannot consider it sent,
however, presumably it had some value ready for
it, so it had gone through some logic for that key.
It just becomes a clumsy situation.

> a) don't allow keys to straddle PDU boundaries.

This would work, but is not as clean as having
a data-end flag introduced, which signals that
now the other side should respond. 

With your proposal the Originator
when it prepares each key and value must check
that the key and '=' fits entirely, otherwise
it must put it back and likely go through the
process again in the next round. Here we have
the key + value generation logic tightly coupled
with the iSCSI encapsulator (which knows how much
is fitting).

> PDU ended in a partial key.

Do you mean don't allow originating any key at
all or do you mean the same as in (c) below?
If the former, then this is practically the
same as what I'm proposing, except my restriction
would be in effect if any part of key=value pair
is straddled. That requires just checking for NUL-s
at the end of the PDU and not looking farther back 
for '='-s. Still an even cleaner aproach is
a new flag in the header.

> matches a partial key at the end of the last login 

This would work too. But is not a very clean
aproach to the problem. Before considering
whether to originate a key we have to compare it
to the incomplete key that's left over from the
just received PDU. Furthermore, the side that
sent the incomplete key ideally should remember
that it sent an incomplete key and check incoming
keys against the incomplete one to catch the other
side in cheating...

My point really is that ideally we want to see
something like this:

Initiator:

  key1=value1
  key2=value2
  key3=value3
  ...
  key17=value17

Target:

  key1=value1
  ...
  key17=value17
  key18=value18
  ...
  key34=key34

Initiator:
  
  key18=value18
  ...
  key34=key34

Done.

We don't want to worry about how this splits up
in PDUs. Just like when I'm writing to a BSD-socket
I don't care in which IP datagram each byte travels.
And if I'm trying to send a certain block of data
and expect the other side to answer once it receives
it all, I don't like it interrupting my thought
and beginning to answer little by little after
each IP packet received. 

The point is that negotiation would be more natural
if it where "block-based", where each side can
send its block of data to the other side without
being interrupted with partial answers and new
keys requiring answers. Right now we've made PDU 
the block, but that is not convenient because PDU
size is limited, and thus not all data (that a
node may have worked hard preparing) may fit.
Because it all doesn't fit, the other side may
start originating the same data that we would
have sent if only the PDU had been larger. We
get into extra complications trying to avoid
the problem of both sides thinking that they
originated something.

Perhaps an even better analogy is the IP protocol
itself. IP datagrams can get fragmented en route.
Yet I know of no protocols that would work on
individual fragments of them, they all first
assemble the whole IP datagram, then see what
can be done with it. Same thing here. An iSCSI
node likely has a bunch of key=value pairs
that it wants to send to the other side and it
is an extra hassle to keep checking how many 
will fit and how many won't, then adjusting
state for those, etc., etc. It is way cleaner
to just pass all data down to the encapsulating
layer, knowing that the other side won't do a thing
before all this data hasn't been received and
reassembled there.

I'm saying my proposal is cleaner. Your proposals
use bandwidth more efficiently but are more complex.
This is not a performance critical area of the
specification, nor are we even worrying about a
very likely case. I honestly believe that typically
most everything that a node wants to send will fit.
Thus, I think the cleaner aproach should be preferred.

> 3. Yes, we noticed a little while ago that losing a

This was my #4, BTW, but that's OK.

> exchange of PDUs. PDU loss has to terminate it. 

I'm beginning to skip and snip text here because
I was explicitly worried about the Text 
Request/Response case, not Logins.

> new values. 

So you're suggesting that the target commits to the
new values as soon as it sends the last Text
Response PDU. This is what I've been doing, but from
other posts on the list I know that other people
preferred to wait till the next PDU from the
initiator.

> If that PDU doesn't reach the
> initiatior,
> then it terminates the entire negotiation and

just to be precise, it = initiator.

> continues

Yes, that's what I'm worried about and don't see
a good way out. If the workgroup is aware that
this is possible and is happy to live with it, 
then I can probably leave the issue alone.

> it isn't worthwhile to add complexity. 

OK, I'll take your word for it.

Thanks,

  Martins Krikis, Intel Corp.

Disclaimer: these opinions are mine and may not
            be those of my employer

---

From: Bill Studenmund <[email-masked]>
Date: Fri, 24 May 2002 17:41:57 -0700 (PDT)

On Fri, 24 May 2002 [email-masked] wrote:

** note there's a proposed experiment at the end. Please try it and see
what you think of how your code does :-) **

> respond to that key until the complete key-value pair

Has anyone actually operated negotiations in the mode where keys don't
fully fit and there are multiple keys to deal with?

I'd expect not since the default PDU data payload is 8k, and I don't think
we really have 8k worth of negotiation parameters. While we do have
security parameters which can get over 8k, the security negotiations I saw
(when scanning the spec) only had one key in flight at a time (thus we
don't get into this case).

To be honest, what Martins proposed is exactly what I read the spec to
indicate! :-) It's also what I've coded, and was pretty easy to do.

The example showing the spec text negotiation certainly indicated that's
the way to do things. :-|

The reason I really like what Martins has suggested (and which we talked
about in private EMail) is that my head hurts when I think about how to
build code to deal with what you describe.

Here's the case that makes my head hurt:

Initiator builds up a set of keys and starts to negotiate them. Due to the
fact it had too much text, it can't send it all in one go. So it builds up
the first PDU, and sends it. It now 1) must remember where it is (so it
can finish in the next PDU), and 2) deal with the key negotiations that
come back from the target in response to the first PDU.

When the reply comes back from the target, the initiator has both handle
the keys the target sends back and the unsent ones it has. You partially
cover that below by talking about not letting the target initiate keys
when we had a partial send. But there's another case too:

Say the response from the target ALSO doesn't fit in one PDU. Yes, the
target's not going to respond to the key=value pair that wasn't complete,
but say its response to everything else was bigger than the initiator's
representation of said keys. For this case to happen, the target's
response has to grow more than the first bit of the PDU-spanning key/value
pair.

The case where I can see this happening is we had a mix of boolean-or keys
and other keys. And for these bolean-or keys, the initiator wanted NO but
the target wanted YES. So the target's response is bounded to be bigger.
Or maybe there were some Rejects in there when the offer was less than 7
bytes long.

So now we have TWO PDU-spanning key sets. And the target didn't originate
anything. :-)

Yes, this is a rare case. But it can happen. And, for iSCSI to be robust,
we need to prevent it.

When I think about this, my head spins.

It seems much simpler to me to let one side finish sending before the
other starts responding. That way we won't end up with having to deal with
any corner cases. :-)

> the next PDU of the exchange, both sides may think they

Yeah, that's the other corner case.

What Martins suggests just seems easier to me. We decouple negotiation
parsing from sending/receiving negotiation packets. The only time we hand
data to the nego. parser, all the key/value pairs are there. We then
generate all the response we want to send. If needed we send said over
multiple PDUs.

Among other things, we DON'T need to even deal with split keys at all;
they get reassembled before we look at them!

Here's a sample logic flow:

dealing_with_input_login_cmd/rsp()
{
    if (have_more_to_send_from_last_time) {
	send a PDU containing more data
	if (this_PDU_finishes_data_flow)
		clear have_more_to_send_from_last_time;
	return;
    }

    if (PDU.len != 0) {
	read PDU's data to buffer
	if (this_pdu_isn't_last)
		return
    }

    Process_input_buffer_and_generate_response

    Send a PDU containing data
    if (not_all_data_sent)
	set have_more_to_send_from_last_time;
}

The same logic flow works for both text and login, and avoids LOTS of
headaches.

*** Experiement ***

Here's a simple experiment to try. Try cranking down the default PDU
length your implementation uses to something rediculously small, like 64
bytes, and see what happens when you negotiate a login. Are you happy with
the results?

Take care,

Bill