Subject: Re: iSCSI: keys/parameter dependence

From: "Robert D. Russell" <[email-masked]>
Date: Sun, 2 Jun 2002 05:27:44 -0400 (EDT)

Julian:

The following paragraph from an e-mail from Martins Krikis
made me realize that the definition of the new C bit
is not correct, because it goes much further
than is necessary to solve the "split key" problem,
and as a result, it opens up a whole new set of problems,
complicates what was previously a simple situation, and
breaks a lot of already running and tested code.

> "(sent) after" isn't defined. It is unclear
> whether "in higher numbered bytes within the
> same PDU" qualifies as "after" or whether only
> "in a PDU sent at a later time" would. It's
> probably irrelevant, since due to the introduction
> of the C-bit, parameters can be accumulated and
> processed one "batch" at a time, without any
> specific order within the "batch" and they will
> quite likely not be processed PDU by PDU.
> Therefore, the text about them being sent in
> "the same command" is likely irrelevant, too,
> since many implementation simply won't check that.
>

I don't believe the intention of the recent discussion
that led to the introduction of the C bit was, in fact,
to allow "batching" or to throw out "ordering" of the
sending of keys.  However, the definition in Draft 12-95
sections 9.10.2  and 9.11.2 seems to now allow batching:
(I agree with John Huffered that Martins' comments about
"ordering" are not correct in any case.)

Section 9.10.2 C (Complete) Bit:
"When set to 1, indicates that the text (set of key=value pairs) in
this Text Request is not complete (it will be continued on a
subsequent Text Request); otherwise, it indicates that this
Text Request ends a set of key=value pairs."

(Section 9.11.2 is the same, but with Response substituted for Request.)

The problem I see with this definition of the C bit is that
the sending side can force the receiving NOT to process
a PDU containing key=value pairs, even if all those key=value
pairs are complete within the PDU.  Therefore, the receiver
is forced to save this PDU (and any number of following PDUs)
until the sender finally sends C = 0.  In the worst case, the
sender could send one key=value pair per PDU with the C bit set
to 0 and the receiver could not reply to any of these until it
got one with C bit set to 1.
As the e-mail from Martins indicates, some people are already
taking this to mean that parameter negotiations will "normally"
be batched.  Up until the introduction of the C bit, PDUs and keys
in them were always processed as received, and most code already does
this, since it is the most logical and efficient way to do it.
We should still encourage this as the "normal" way of negotiating.

I believe the problem we were trying to solve by introducing
the C bit was the one where a key was split over a PDU boundary
-- we were not trying to redesign the negotiation mechanism,
which was discussed on the mailing list long ago and has been
working in a lot of running code for the past two plugfests.

Could we therefore please change the definition of the C bit in
section 9.10.2 to read:
"When set to 1, indicates that the text in this Text Request
contains a single key=value pair, and that pair is incomplete
within this PDU."

(Section 9.11.2 should have the same change, but with Response
instead of Request)

I believe this corresponds more closely with the description
in section 4.2 Text Mode Negotiation, where the C bit (flag) is
clearly tied to the idea of split keys only:
"Key=value" pairs may span PDU boundaries.  An initiator or target
having sent partial key=value text within a PDU indicates that more
text follows by setting the C flag bit in the Text Request or
Text Response to 1."

Perhaps an additional sentence could also be added to this:
"The following PDUs should only contain text that completes
the partial key=value text until after a PDU with C set to 1
has been sent, at which time new key=value pairs can be sent
in the next PDU."

I'm not happy about seeing such a major change (C flag bit)
introduced so late in the process to solve an "almost" non-existant
problem (split keys).  It is "almost" non-existand because of
the following considerations (some of which were already
mentioned on the list and are just summarized here):
During the login process, the PDU size is 8192 bytes.
Therefore, except for the long keys during negotiation
that follows a chosen AuthMethod, there is never a need to split
keys across pdu boundaries during login --
even if you negotiate every possible login key
(except those special keys that occur in security protocols)
to its maximum length, you do not reach the total of 8192 bytes.
It is not even close!  Therefore, there is no need for any
implication that the C bit should appear during this process.
When those long security keys are needed, the negotiation process is
in a 1 key over, 1 key back exchange mode, so there is no "batching"
possible and the split key should in fact be the only key in the pdu.
Finally, during text exchanges, there only 6 keys that can be
legally sent (plus the X-extension key): SendTargets, TargetName,
TargetAlias, TargetAddress, InitiatorAlias, and MaxRecvPDULength.
None of these can legally have a value greater than 255 bytes,
which is well less than the minimum PDU size (512 bytes).
So there is no need to split keys over boundaries during text
negotiation, except perhaps for the X-extension keys.
My proposed definition of the C bit as applying only to split keys
therefore only applies to X-extension keys during text negotiation.
Since we have had no experience with X- keys, and cannot forsee their
use, anything we introduce only for them should be the minimum possible.

In summary, I would like to see as little disruption as possible
to the key negotiation scheme which has been stable for a long time,
especially since anything we introduce will be needed only in very
limited circumstances.  I believe my modification to the definition of
the C flag accomplishes that, without introducing any new, false
perceptions about "batching" and "ordering".

Thank you for your consideration.

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

---

From: Martins Krikis <[email-masked]>
Date: Mon, 3 Jun 2002 13:30:05 -0700 (PDT)

(I've thrown some text out to keep it shorter)

> breaks a lot of already running and tested code.

The C-bit actually simplifies a previously very
complicated situation. As to "running and tested",
I don't think many people have encountered split
PDUs in operational stage or FFP Text negotiations
yet.

> sections 9.10.2  and 9.11.2 seems to now allow

Those, who prefer the "batch-processing" approach
fought for the C-bit, so I would say that it was
introduced in order to allow it.

Regarding ordering, however, it had never been
defined, and I would hate to see it introduced.
In my opinion, it is best to treat all operational
parameters as independent and negotiate them as
such. Just before the commit 
(i.e., turning on the T or F bit), one can do an
all-encompassing consistency check and reset
the negotiations if the values violate the laws
imposed on them, or offer some more keys to solve
any such problems, if this is still possible.
I could even imagine negotiations
succeeding but laws 
(e.g. FirstBurstSize <= MaxBurstSize) not being
broken when sending data... OK, the last thing
might technically be forbidden at the moment
but it is not that unreasonable. It would be
something like 
FirstBursSize = min(FirstBurstSize, MaxBurstSize)
done after the negotiation end, and then you can
forget about dependencies.I think it is way
easier than worrying about key ordering every
time something is sent or received.

> "ordering" are not correct in any case.)

I'd have to know which particular ones are we
talking about in order to clarify my position.

> until the sender finally sends C = 0.  

"You may process, just please don't send anything
back yet, because I'm not finished sending stuff
to you. The small PDU size wrt the abnormally large
data amounts that I have to send (all very important
and urgent) are making it hard for me to fit it all 
in a single PDU."

> In the worst
> case, the

If this is really a problem, I'm happy to retreat
my position to one of the other schemes, going
as far as allowing the responding side to send
responses and originate key declarations, but
not to originate keys that are not declarations.
I am really fond of doing batch-processing of the
keys and I would hate to have it broken again.

> be batched. 

I'm planning to batch my parameters. But I don't
care how the other side processes them, as long as
it sticks to the (new) protocol and doesn't interrupt
me when I have the C-bit still set.

> Up until the introduction of the C bit,
> PDUs and keys

How could you possibly know what "most code" does?
I would even conjecture that most code isn't out
there open for examination or experimentation with.

Most code that I've seen does batch-processing.
I must admit, the batch is typically one PDU and
the code can break if keys are getting split.
The C-bit allows a trivial update of that code
to deal with split keys as well.

> the C bit was the one where a key was split over a

sure

> -- we were not trying to redesign the negotiation

and we didn't really. It's still essentially the
same, one side sends some pairs, the other side
answers. But PDU boundaries no longer mess up this
simple scheme.

> working in a lot of running code for the past two

Did you see split-keys outside of security stage? 
What happened, if so?

> contains a single key=value pair, and that pair is

I could see it anyway when I find no terminating '\0'.
But how would this change help with the original
split-key problem (both sides feeling like 
originators)? Do you propose to send complete
pairs separated from split-pairs? And the sender
to check for each key=value whether it will fit
entirely or not? If so, please see the very
lengthy discussion which got us to the C-bit.

> problem (split keys). 

Yes it is "almost" non-existant. That's why I asked
above, have you seen split-keys outside security
stage or not?

> It is "almost" non-existand
> because of

OK, disallow it completely. Just don't let some
split-keys make both sides feel like originators,
nor ask me to check each key=value pair I'm preparing
for fitting or not fitting.

> possible and the split key should in fact be the

Sure, although nothing prevents me from declaring
that my 1 key=value is my "whole batch" and apply
batch-processing to this pair.

> which is well less than the minimum PDU size (512

And how many instances of TargetName and TargetAddress
can the SendTargets command provoke from the other
side? I think it can easily overflow the 8192 bytes.

> negotiation, except perhaps for the X-extension

One could get away without splitting keys ever
(but not necessarily without splitting values).
However, a scheme like that means that the sender
has to check whether a newly prepared pair fits
or doesn't (definition of "fits" varies depending
on whether the key may be split or not).
Again, the lengthy previous discussions were all
about why this would be a bad requirement to have.

> to the key negotiation scheme which has been stable

I think the C-bit just achieved that. A "batch"
is a "virtual PDU" (no length restrictions).
Once you get a "batch", you're welcome to send
the response batch, just as previously done with PDUs.
Except that nobody has to worry about split keys
anymore.

> I believe my modification to
> the definition of

It's nice that batching is now possible. Having
any ordering (within a "batch") is entirely 
unnecessary, however, IMO.

Martins Krikis, Intel Corp.

Disclaimer: these opinions are mine and may not be
            those of my emplyer

__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

---

From: "Robert D. Russell" <[email-masked]>
Date: Tue, 4 Jun 2002 02:46:46 -0400 (EDT)

Martins:

Comments below on all your e-mails in this one reply.

> As to "running and tested",

Agreed.

> introduced in order to allow it.

I missed that point earlier.

> defined, and I would hate to see it introduced.

It was defined before draft 8, but was taken out when that
draft came in.  Unfortunately, I did not take it out of
my memory.  I agree that it should not be reintroduced.

> such.

Agreed.

> Just before the commit

What you are saying seems to imply that C=0 does NOT
require the receiver to reply to keys received up to that point
-- it could send another empty PDU, or more likely, it could send
new offers of its own.  I agree that there is nothing in the draft
that says when replies to keys should be sent, only that they
MUST be sent (sometime).

For example, consider a login negotiation of operational parameters
in which the initiator sends a login pdu containing key=value offers
and the C bit 0.  The target responds with a login response pdu
containing key=value offers of its own (offering different keys)
but no replies to any of the offers it received in the login.
The initiator can then reply to the target's new offers, or it
can decide not to reply, and instead send additional new offers
or even an empty pdu, (C bit 0 in both alternatives).  And now
the target could do as it did before, not replying to any offers
but either sending back new offers or sending back an empty login
response pdu (again C bit 0 in both alternatives).  This could go
on indefinitely.

It would seem that the initiator might try to force replies by
setting T=1 to force an end-of-stage transition.  However, the target
can refuse to make the transition and can reply with T=0 and still
no replies to the keys it was offered.

This is admittedly a rather far-out example, because presumably both
initiator and target want to end the negotiations as soon as possible.
My point only is that I do not see anything in the draft that says
when the replies to keys have to be sent, only several references
that there MUST be a reply to every key offered (eventually).

Thoughts, comments?

> forget about dependencies. I think it is way

The dependency can be accounted for when generating the reply.
For example,

reply.FirstBurstSize = min(offer.FirstBurstSize, reply.MaxBurstSize)

That way nothing is broken when sending data.

> side? I think it can easily overflow the 8192 bytes.

Yes it can, but there was already a mechanism in place to deal
with this.  In fact, this brings up an interesting point.
Presumably the C bit has to be used with replies sent to
SendTargets (or any other offer that might generate a long reply),
since the C bit is in the Text Response PDU used to send these replies.

Refering to sections 9.10.2 and 9.10.4:  If the target generating a
long reply has more text to send than will fit in one PDU, then it
should indicate this by setting C = 1.  Setting C = 1 also forces F=0,
and this in turn forces the Target Transfer Tag to be something
other than 0xffffffff.  When there is no more text to send,
the target sets C = 0 and F = 1 and TTT=0xffffffff in the last
text response pdu it sends to the initiator.  There is no
situation in which C = 1 and F = 1 can occur (since this is
explicitly stated in 9.10.2 as being an error), nor is there a
situation in which C = 0 and F = 0 should occur (because C = 0
means "I'm done sending stuff" and F = 0 means "I'm not done sending
stuff").

Is this the way you interpret the merging of the C bit with the
long text reply mechanism?  If there is a consensus on this,
perhaps the wording in the draft in section 9.10.4 should include
the (required) settings of the C bit whenever it mentions the
corresponding settings of the F bit and TTT field.

> > -- FirstBurstSize and MaxBurstSize are dependent
> > because of the
> > requirement in section 11.15: "FirstBurstSize MUST
> > NOT exceed
> > MaxBurstSize."  See my e-mail response to Mike for
> > details
> > on that dependence.
>
> I saw it. I consider it unbelievably ugly to have to
> look at the values in order to figure out ordering
> requirements. I prefer ignoring ordering completely
> and check for overall consistency before commit.

You are correct, the values can be figured out without
resorting to an ordering -- my mistake.

> Nowhere does it say anything about the order.

Agreed.

> So, are you really proposing a requirement that
> we must look at the values in order to figure out in
> which order to send keys? That is so UGLY, it is
> hard to believe that this is happening.

Please forget I even suggested it.

> > The existence of a dependency between OFMarker and
> > OFMarkInt, and between
> > IFMarker and IFMarkInt, is implied by the statements
> > in section A.3.2:
> > "When the interval is unacceptable the responder
> > answers with
> > "Reject".  Reject is resetting the marker function
> > in the spcified
> > direction (Output or Input) to No."
>
> No it isn't. IMO, it is resetting the marker interval
> to its previous value, which is likely the default
> value. I believe it's perfectly legal to negotiate
> the marker interval but to not turn on marker use,
> or to turn on the use but stay with current (default)
> values for the interval. If one side can't  tolerate
> the other's  Reject (i.e., can't live with the
> default value), it is welcome to bail and try next
> time w/o markers. BTW, we could use 0 here as a
> special value, meaning that markers are not in use,
> then we wouldn't need the boolean keys for
> markers.
>
> > This last sentence should probably be reworded to
> > say:
> > "A response value of "Reject" to an OFMarkInt
> > (IFMarkInt) key resets the
> > corresponding OFMarker (IFMarker) key value to
> > "No"."
>
> No, thank you. I would prefer if Reject meant the
> same as with the other keys, i.e., negotiation
> failed for this key, let's stick with the old
> value, or bail if we must.

Either the sentence needs to be reworded so it is proper English
or it should be taken out of the draft.  I take it you are advocating
its removal?

Bob Russell