Subject: Re: iSCSI:Can each do there own thing?

From: Martins Krikis <[email-masked]>
Date: Wed, 29 May 2002 18:03:19 -0500

This isn't a very interesting reply, but the next mail is a 
a very important read, IMHO. Please read that one carefully,
a compromise is included there.

> > I may have this wrong, but the focus seems to be on the receiver of the
> > spanned PDU, and the issues it has in keeping state etc.
> 
> For me, the focus is on both. 

Yes, in particular, I think sending is harder to implement w/o the
guarantee of only empty PDUs coming back.

> You are right that the receiver is easy to
> fix; just send empty PDUs and process at the end. The one thing we would
> need to add to the spec is a sentance saying you can't nail the other side
> for a negotiation error (for not answering keys) if you sent a split PDU
> or they sent a split PDU.

I don't think that there is a requirement to answer all received keys
as soon as possible. They may not all fit, so this is not even possible
in general. And split PDUs may be unavoidable, too. 

What we need added to the draft is some restrictions that would make it
impossible for both sides to feel like they originated a key. Currently,
if one side sends part of the key, it may feel well entitled to consider
that key as sent, but the other side is not mandated to consider this
partially received key as received, so it may send it too---with disastrous
results for the negotiation.

> The other messy part (not hard, just messy) is for the sender. And this
> part is why I personally favor empty PDUs. The sender has to be able to
> process new input packets before it has finished sending everything. i.e.
> it has to be able to deal with the other side interrupting it while it is
> speaking.

Yes, and for this reason it cannot simply do what many consider the
simplest aproach---create all the key=value pairs that would be nice
to process in one batch, and pass them down to the encapsulation layer
without worrying how many will fit in the first PDU.

> If we forbid the receiver from originating new keys when it has received a
> split PDU, we make life easier on the sender, but it still has to be
> interruptable.

By forbidding origination on split PDUs we avoid the possibility of
both sides feeling like originators. It is a step in the right direction.
In order to deal with interruptability, we need to forbid sending
anything until the other side hasn't finished. (For better definitions
please see the next mail from me.)

> > Now the rules we have now in the current draft level 12-94,  some seem to
> > say, are workable.  

Well, there is not enough of those rules, hence the above mentioned
possibility of both sides feeling like originators for the same key.
But there is work in progress here, so 12-95 may have the missing rule(s).

> > But if someone sends empty PDUs, etc., that seems to
> > work too, and some folks (at least 2) think that is a good idea.

Guarantee of receiving only empty PDUs is more important than ability
to send them (that already exists).

> > So are we at a point that says that empty PDUs are OK, and if used, you
> > need to remember less stuff, but if you want to remember state, then go
> > ahead, but be sure you do everything correctly?

Without guarantees of empty PDUs there is more work at sending even for
an implementation which is "empty-PDU-favoring".

> > Let me ask two question then:
> >
> > 1. Is what I summarized correct with respect to the receiver?  If not
> > please itemize, the items that are broken, so that we can work on one at a
> > time.
> 
> I believe that is correct with respect to the receiver.

So do I. Receiver is not the hard part, because it is easy to be
"polite towards the sender" and send only empty PDUs. And with any
scheme you have to notice that you're forbidden to do something.

> The one thing new I mentioned is the fact you can't claim negotiation
> error when not all the keys you offered were returned *if* eithe you or
> the other side sent a split PDU. i.e. if one or both sides are still
> talking, you can't tell yet.

Again, I don't think it is an error to send a belated (many PDUs later)
response to a key, as long as it is sent in the same negotiation
sequence. It is also not an error to send empty PDUs (as many times
as one wants w/o any reason whatsoever). And both these things are
fine in my eyes, so I may not be understanding the issue here...

> > 2. Is there an important issue with the offering side, that is not covered
> > with the current spec. and will things work on the offering side if the
> > Null PDU us sent from the receiver?   Again, if no, please itemize.
> 
> As above, I think it's more what has to be in the sender if the receiver
> can send a non-empty PDU.

True. The sender has to be aware of whether each new key=value pair
fits in the PDU completely, partially or not at all. It cannot work
on a batch of pairs and not worry about PDU boundaries.

> > I get the feeling, and I maybe wrong here, that the various sides are
> > trying to evangelize each other, even if the other side's approach works.
> > I am not a big fan of evangelizing at this point.

I'm religiously illiterate, so I don't even know (or care to know) what
I was just accused of doing, but I'm rather sure I was doing it. In my
eyes, however, I was simply defending a simpler scheme. But in order
to try to be fair I have composed an analysis of all schemes. Read the
next mail please, this is long as it is.

> > I am asking for a compromise that you folks can agree with that says
> > something like, "yea, you can do it that way you silly so & so, but my way
> > works too perhaps better", and the spec covers both implementations.  This
> > might not be possible but I am not sure we are converging.  And we need to
> > within a day or so.

I think the way spec gets clarified/restricted will pretty much tell
whether the simpler batch-processing approach is possible. But with
a minor change in what is called a split-PDU there actually is a hope
for batch-processing implementations even with the scheme that seemed
to prohibit them, so there is a hope for a compromise.

> I think with the adjustments above (no offers when you got a split PDU and
> wait until no split PDUs to determine if a lack of replies is an error)
> would be sufficient. I think sending empty PDUs would be easier, but the
> above would suffice.

I don't understand why lack of replies would be considered an error
under any conditions.

Take care,

Martins Krikis, Intel Corp.

Disclaimer: these opinions are mine and may not be those of my employer

---

From: [email-masked]
Date: Wed, 29 May 2002 17:15:47 -0600

John,

1. Yes there is one item that is broken:
It is offers that require a choice one way or the other. One could build
a buffer blindly of responses and declarations and either partition it into
PDUs and start sending or wait until one thought one had received all the 
offers from the other side. 

If one wants to be able to build a buffer that includes offers, chop it
into PDUs and send, one needs everyone to use a mechanism to pass the 
offer token back and forth. One could say that an implementation without
the offer token MAY send PDUs with declarations and responses to offers.
(Aside: Am I the only one who thinks it invites confusion to have "response" mean
a PDU type and a key-value pair type that may be in either PDU type?)

It isn't clear to me whether the mechanism should be a new flag or whether
modifying the meaning of the T bit could be used. I think the former would
be cleaner and there is no reason to scrimp on a bit. I would like to go 
over proposed text to the draft for that before we commit.

I don't think that the last receive PDU ending in a partial key-value
pair should be used for the hand-off because it would require the sender
to be careful where it chops when breaking the buffer into PDUs. People
didn't want to have to ensure that they didn't break a key-name and this
would be the same type of thing.

2. Null PDUs are not a problem. With the current protocol, one may get
null PDUs and there are even times when one has to respond with null 
PDUs - receiving a long security key, receiving the response to a 
SendTargets and any time one has nothing to send but the received
PDU doesn't have T set. Therefore null PDUs have to be tolerated
regardless of what we do here.

I think there is little technical difference between the two implementation
choices (assuming that the hand-off process for the offer token doesn't
get messy) but for interoperability one needs to make a choice between them
for offers.

Pat

-----Original Message-----
From: John Hufferd [mailto:[email-masked]]
Sent: Tuesday, May 28, 2002 8:33 PM
To: Martins Krikis
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: iSCSI:Can each do there own thing?

I am starting a new thread.

I am not sure that I have gotten all the nuances of the various proposals
here, but ...

I think I read, that if you receive a spanned PDU, that some want to keep
some state and continue processing, while other are saying that it is
easier if only an empty PDU is returned, until the complete PDU with
nothing spanning is received.

Now the rules we have now in the current draft level 12-94,  some seem to
say, are workable.  But if someone sends empty PDUs, etc., that seems to
work too, and some folks (at least 2) think that is a good idea.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

Martins Krikis <[email-masked]>@ece.cmu.edu on 05/28/2002 07:53:23 PM

Sent by:    [email-masked]

To:    [email-masked], [email-masked], [email-masked]
cc:
Subject:    RE: iSCSI: Negotiation clarifications still needed

> It's to avoid corner cases while doing what you say above, "preparing
> its
> strings and chopping them into PDUs."
>
> The simplest thing in my mind is to:
>
> 1) read the other side's offers into a buffer
>
> 2) parse that buffer generating responses to another buffer
>
> 3) send the output buffer
>
> Does that make sense? You don't have to agree, just does it make sense?
>
> The problem I see with what Pat described is that if you are in
> operational negotiation and have more than 1 PDU worth of data, you
> will
> be receiving input while you're in step 3. You have to be able to
> re-enter
> steps 1 & 2 before you can finish step 3.
>
> <PAT> I think that is over stating the complexity. During negotiation,
> the other side can only have one PDU outstanding at a time. That is
> explictly
> stated for TextNegotiation PDUs. For LoginPDUs, it isn't explicit but
> since
> they are immediate and the initiator can only count on the target
> having one
> immediate buffer for the connection it can't send another LoginRequest
> until
> it gets the LoginResponse without risking that a PDU will be dropped.
> Also
> allowing anything other than one oustanding LoginPDU at a time would
> open
> the possibility that both sides offered the same key. Therefore, a
> target
> won't get the next PDU until after it sends the response PDU (or
> vice-versa
> for an initiator). A PDU won't come in while a PDU is being sent from
> the
> outgoing negotiation buffer.

Pat, I must interrupt you here. (A subliminal part of this exercise
is to show how annoying interruptions are :-).) In order to argue your
point I think you're intentionally trying to misunderstand Bill.
He didn't talk about getting a new PDU while sending his PDU.
He talked about getting an incoming non-empty PDU while
sending his data _buffer_. This buffer could be much
larger than a PDU. You cannot deny that this may happen
with the scheme that you're arguing for. It cannot happen with the
scheme that we're proposing, since only empty PDUs would be coming
back while we're sending our (possibly much larger than PDUs) data
buffers.

> Once you send the PDU, there may still be some data left in the output
> buffer but it doesn't seem complex to switch at that point to waiting
> to
> receive and parse the next received PDU.

If you have noticed that a PDU is full, it is certainly possible to
switch to a receiving mode. Yet it is even simpler not to have to notice.

> One had better be able to switch
> from one process to another anyway as most implementations will handle
> multiple connections which can be in diffent phases.

I don't see the relevance of this argument.

> If you are willing
> to buffer all your output into a buffer and wait to send it until the
> other
> side has stopped sending new parameters, then all you need to do to
> change
> that to an implementation that sends what it has in the output buffer
> as
> it goes along is to have pointer into the buffer that indicates the
> next
> byte to be sent. <PAT>

Not true, because as I'll start sending it, the other side may
send me new offers, thus preempting me from making similar offers
myself and instead requiring _responses_.

> The bit about sending empty PDUs is an effort to avoid that. We are
> either
> in a sending-negotiation-text mode, or a parsing mode. We don't have to
> worry about strings being partially there, since we don't process until
> they are. We also don't have to worry about if our key/value pairs
> cross a PDU. The parser/negotiatior is MUCH simpler; it doesn't have to
> deal with all of that.
>
> Letting the target answer while the initiator is trying to send an
> extended PDU adds the following complications (AFAICT):
>
> 1) initiator has to make sure a key and the '=' don't get split on a
> PDU
> boundry.
>
> <PAT> One doesn't have to do that anyway. We can chose the rule to not
> offering a new (non-declarative key) when one has received a partial
> key
> which seemed to be acceptable to a number of people. <PAT>

First I haven't yet seen anybody but you and myself speak about the
declarative/non-declarative variation of this theme yet, so I wouldn't
call it "acceptable to a number of people" yet, at least not if
you include the parenthesized part. Overall, however,
a truly aware Sender most likely will check whether a key and =
got split or not. Because if it was split, and the other side
originates a new key, it is a violation. It would be nice to catch it.
And it definitely must check whether key=value fit fully or not.

> 2) if an initiator fills up a PDU, it has to stop processing & wait for
> the response. It also has to remember where it is so that it can finish
> sending that key/value next time.
>
> <PAT> It has to remember where it was and finish the key/value next
> time
> anyway as it can't send again until it gets a response. <PAT>

The difference is that in the scheme we're proposing we'd leave the
unsent key=value pairs sitting in the "leftover" buffer, and happily mark
all those keys individually as "sent". So we'll consider them
processed, even though they may not have been put on the wire yet.
With your scheme, you may not mark them as sent, so there is also
little point having them stay in the buffer. The only thing that
can safely stay in the buffer is the remainder of the last key=value pair,
if it was sent partially.

> 3) The target has to detect a split key/value pair on in-bound, and a)
> store the received text, and b) remember not to offer said key or any
> others.
>
> <PAT> detecting an incomplete key is reasonably easy. And the target
> can
> chose to do the simplist thing which is not to offer new keys when a
> partial key is outstanding. <PAT>

Yes, easy. Still harder than never dealing with an incomplete key at all.

> 4) The target ALSO has to make sure ITS response doesn't fill up a PDU.
> If
> it does, it has to remember where IT is and that it has more to send.
>
> <PAT> In your approach the target can have a buffer with more than
> one PDU worth of response in it. Once it starts sending from that
> buffer
> it has to remember where it is and that it has more to send. This is
> not any different. There is a buffer with pointers into for end and
> last byte sent (or next byte to send). <PAT>

Yes, it is different, because keys can be marked "sent" and happily
wait their physical send-out in the buffer. With your scheme you
can't put them in a buffer and forget, because incoming data may
force you to go back and start mangling the values you had assigned
them...

> That's a lot of complexity and saved state. At least in my mind.
>
> <PAT> I don't see much complexity difference here.

There is quite some difference in complexity, here. I think you're
trying not to see it.

<snip>

> <PAT> I agree. This means that plug fests probably haven't hit these
> cases. <PAT>

That's why there is the famous Bill's experiment---set the PDU size
to 64 bytes and try negotiating.

> Does the statement of the perceived problem make sense, even if the
> opinion of the WG is to not use the empty PDUs? I would feel much
> comfortable with deciding to not use empty PDUs if folks said they
> understand the above point and they choose to address the problems
> differently.
>
> <PAT> I want to point out that there are times in the protocol when
> one will have to send an empty PDU. For instance, when getting a
> security key that is longer than a the Max PDU size (or even less
> than that - the sender isn't required to send Max size PDUs) or when
> a device has no key-value pairs to send but its partner hasn't set
> the F bit. The issue is whether to require that there be a process
> for handing the right to originate keys (or in your proposal to send
> keys at all) keys from initiator to target and back which adds
> its own complexity. I think the complexity will be
> of the same order as the existing situation. <PAT>

I think in both cases we do have a process to hand the right to
originate/send to the other side and back and forth again. We are
not trying to take this process away. The question is should the
"handing over of the right (to originate/send)" happen after
each PDU sent or should it happen after a data _buffer_ (possibly
containing more key=value pairs than fit in a PDU) sent?

Again a simple comparison to real life... Take the first text
in this message written by me. How would you have liked to delete
everything beneath it and to send the rest of your comments
in a new message? Again, include only as little as there is
until my next comment. That's when I'd interrupt you again
and potentially mess with what you were planning to say.
Pretty frustrating, wouldn't it be? OK, to be fair, I couldn't
do it as often as I could want to, since you'd be entitled to one
PDU worth of uninterrupted text in iSCSI. But it isn't as clean
as "letting the speaker finish" (not the lecture, just the
"complete thought").

Disclaimer: these opinions are mine and may not be those of my employer.