Subject: RE: iSCSI: Iterating long text responses

From: "Rod Harrison" <[email-masked]>
Date: Mon, 2 Jul 2001 16:24:32 +0100

We have a concern over option #5, the outline below assumes
that it is safe to issue the same text command more than
once with no side effects. This does seem safe for the
current use of SendTargets, however it might not be if this
mechanism is extended for other long response text commands.

	Our preference would be for option #3 with #4 a close
second. Number 4 has the disadvantage that the initiator has
to deal with a stream of unsolicited text responses with no
idea of when they might stop.

	Perhaps #3 with the following modifications might suffice.

	Add an offset to the initiator R2T (iR2T?) allowing the
target to choose to either buffer the whole response or
regenerate it on the fly. The target should know whether the
data can be regenerated safely and so can make this
decision.

	Add a final bit to the iR2T allowing the initiator to stop
the response transfer at any time. Mandate that the
initiator must either receive all the data or send an iR2T
with final bit set to indicate no more data is required.
This prevents a target getting stuck with the response data
in a buffer forever.

	A typical transfer might be.

	I->T Text command (F bit set)
	T->I Text response (F bit clear)
	I->T iR2T incl. valid offset and length (F bit clear)
	T->I Text response (F bit clear)

	....

	Sequence either ends with

	T->I Text response (F bit set)

	or

	I->T iR2T Offset 0, length 0, (F bit set)

	Since the first text response is unsolicited all current
text command implementations can remain unchanged. Only if
the target responds with the F bit clear does the initiator
need to change.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On
Behalf Of
Mark Bakke
Sent: Friday, June 29, 2001 9:18 PM
To: IPS
Subject: iSCSI: Iterating long text responses

Initiator developers-

   Please respond to the questions at the end.

Thanks,

Mark

The current iSCSI draft allows text command and response
PDUs of up to 4096 bytes.  While we don't see any real
problems for the command PDU size, commands such as
SendTargets
can easily exceed the response size.

There are several ways we can fix this.  The first two
solutions
require no differences in the current iSCSI text command and
response; the latter three involve the use of the F bit.

1. Assume that such commands are done on a "special"
connection
   or are handled completely in software, and allow its
response
   PDU to be as large as it needs to be.

   This one appears too restrictive to be a solid solution.
It
   will also weaken any data digests done on the longer PDU.

2. Create an iterative SendTargets key (and do the same for
any
   other text commands that need this), that would allow the
   initiator to request the "next target" or "next address".

   This would work, but would require any new command that
needed
   a large response to implement an iterator.  It also means
that
   each part of the response from the iterator would have to
fit
   in the 4k PDU.

The remainder of these require that only one text command
sequence
be outstanding on a connection at a given time.  They use
the F
bit to indicate the last PDU of the sequence.  Note that
connection
allegiance is assumed for the entire sequence, and text
commands
are never retried on another connection; a new command is
issued
instead.

3. Do a text-response R2T, where the initiator controls when
the
   next partial response is sent.  This would be more
generic:

   I->T  Text Command (F bit set)
   T->I  Text Response (first PDU, F bit cleared)
   I->T  Text Command (with some indicator that we want
more)
   T->I  Text Response (next PDU, F bit cleared)

   ...

   I->T  Text Command (with indicator that we want more)
   T->I  Text Response (last PDU, F bit set)

   The main problem with this is that the target must keep
track
   of the state of its responses; if the initiator just
stops asking,
   it may have to keep a buffered response around until it
times out,
   the connection is dropped, or another text command comes
along.

4. Allow multiple response PDUs to a single text command:

   I->T  Text Command (F bit set)
   T->I  Text Response (first PDU, F bit cleared)
   T->I  Text Response (next PDU, F bit cleared)

   ...

   T->I  Text Response (last PDU, F bit set)

   Basically, we are doing (3) without the R2T.  The
initiator,
   upon sending the text command, must be prepared either to
   accept as many PDUs as come back, or throw them away if
it
   can't handle them.  This solution is a lot like #1, but
with
   the response spread across 4k PDUs.

   Also note that this (and the following scheme) avoid the
problem
   of the target keeping state; it sends ALL of the response
PDUs
   without the initiator asking for them.

5. Do #4, but allow the initiator to specify the amount of
data
   it is willing to accept:

   I->T  Text Command (F bit set, AcceptResponseLength=4096)
   T->I  Text Response (first PDU, F bit set,
TotalResponseLength=12288)

   In the above example, we have created a new text command
field:

      AcceptResponseLength

   And in the text response PDU:

      TotalResponseLength

   The initiator indicated it was willing to accept a 4k
response.
   The target returned the first 4k in the text response,
but set
   the F bit since it was at its limit.  It also returned a
   TotalResponseLength field.  Since this was greater than
the
   AcceptResponseLength, the initiator knows the amount of
buffer
   space it will need to get the full response.  It can then
choose
   whether it will re-send the command, and if so, can give
it a
   large enough response length:

   I->T  Text Command (F bit set,
AcceptResponseLength=12288)
   T->I  Text Response (first PDU, F bit clear)
   T->I  Text Response (next PDU, F bit clear)
   T->I  Text Response (last PDU, F bit set,
TotalResponseLength=12288)

   Note that most initiators will probably send an
AcceptResponseLength
   of the largest response they would normally accept, and
that most
   target responses will fit in one or a few PDUs anyway.

   #5 is really a compromise between #3 and #4; the target
has the
   benefit of being less statefull, and the initiator has
the benefit
   of controlling the amount of data it receives.

I would like to recommend either #4 or #5.  I think that #5
is
probably the safest solution, but #4 may not be a problem
for anyone.

Assuming that none of the implementors of initiators have a
problem
with #4, I would pick that.

If they do have a problem with it, we should go with #5.

Targets probably don't care much whether we do #4 or #5.

Initiator developers-

  Please indicate which solution (#4 or #5) appeals to you.

--
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: Mark Bakke <[email-masked]>
Date: Thu, 05 Jul 2001 09:11:27 -0500

Rod-

You are correct that this assumes that the initiator can do
the same text command multiple times.  I think that the real
question is whether attempting to allow non-idempotent text
commands, and providing easier mechanisms for doing them is
a requirement.

If so, your addition to #3 would work.

If not, it would place unnecessary burden on the target.

At the moment, I can't think of much that would need to be
added to iSCSI from a (non-negotiation) text command point
of view.  In fact, we are supposed to be careful to limit
what can be done in this way, since most other functionality
we can envision should be left to other standard protocols.

Any thoughts on specific uses for non-idempotent text
commands?

--
Mark

Some more comments: 

> mechanism is extended for other long response text commands.

True; SendTargets doesn't care.  However, we seem to have a
general consensus that the use of text commands should be
carefully limited; what are the real chances that some
functionality comes along that we really need, and that does
not fit in?  I don't see that really happening.

> idea of when they might stop.

Actually, an initiator can do the same thing with #4 as with #5;
it could issue the text command, and just keep track of the number
of bytes of response that it was unable to keep.  It could then
allocate enough space, and re-do the request.  #5 is just a cleaner
mechanism to do this.

At any rate, the burden is on the initiator as to whether it
wants to make use of these mechanisms.

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: "Rod Harrison" <[email-masked]>
Date: Thu, 5 Jul 2001 20:36:10 +0100

Mark,

	It was more of a general concern for the iterative process
than thoughts of a specific non-idempotent text command. If
text commands are ever used in an IOCTL like capacity we
might run into problems, but as you say there are probably
better ways to do that sort of thing, like Mode Select.

	I would still prefer to have the initiator able to pace the
receipt of, and prematurely end, large text responses but I
understand this may be too expensive for some targets.

	Requiring the initiator to allocate an arbitrarily large
buffer for #5 is a little concerning, especially since it
seems that could be larger than the negotiated max PDU size.
For that reason I would prefer option #4 over #5, but I
think we'll need to add a sequence number to the text
response to allow the initiator to detect out of order PDUs.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, July 05, 2001 3:11 PM
To: Rod Harrison
Cc: IPS
Subject: Re: iSCSI: Iterating long text responses

Rod-

--
Mark

Some more comments:

> idea of when they might stop.

Actually, an initiator can do the same thing with #4 as with
#5;
it could issue the text command, and just keep track of the
number
of bytes of response that it was unable to keep.  It could
then
allocate enough space, and re-do the request.  #5 is just a
cleaner
mechanism to do this.

>    I->T  Text Command (F bit set,
AcceptResponseLength=4096)

--
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: Stephen Bailey <[email-masked]>
Date: Fri, 06 Jul 2001 11:05:54 -0400

>   Please indicate which solution (#4 or #5) appeals to you.

Well, this makes a hearty assumption....

I believe 4 completely misses the point.  The initiator will still
have no control over the resource requirements to swallow the target's
responses.  This is bad both for the initiator and the target since it
can flow-control the connection.  For iSCSI to work well, you must
make sure that the connection is kept `lively' (connection-level flow
control is not engaged).

Solution #5 is better, but you still have an irritating problem
dealing with what happens when the response length is larger than your
reserved resources.

I believe in #2.  If we specify that each text PDU results in at most
one response PDU, that gives you effective control over the resources
required for text messages.  Then, on a case-by-case basis, you build
a protocol for addressing this limitation within its confines.  In the
case if SendTargets, we use an iterator.

In the case of each X-* operation, the protocol is chosen to meet the
needs of the operation.  The reason why you don't want to solve this
problem `in general' (which we haven't actually done with any of these
solutions anyway) for everybody is because `everybody' has different
needs.  Some actions of X-* are NOT going to be idempotent, and the
vendor (or whomever) may want to build an exactly-once protocol which
works across connections, and on recovered sessions.  Other cases,
might want at-most-once semantics, and yet other cases, unreliable,
best-effort.

It's too painful (and pointless) for us to build an exactly-once
protocol for text messages when the only specified application is
SendTargets, which doesn't need exactly-once semantics.

What you're trying to provide is the equivalent of general IP
fragmentation.  For application purposes (as opposed to network
purposes), IP fragmentation just gets in the way.  Provide a basic
datagram capability the upper layer can build everything else as
necessary.

If the application heavy data transfer services, it should use SCSI
directly.  This is already a tradition within SCSI architecture.

Steph

---

From: Mark Bakke <[email-masked]>
Date: Mon, 09 Jul 2001 15:58:03 -0500

Steph-

Comments are below.

--
Mark

> make sure that the connection is kept `lively' (connection-level flow

Absolutely true.

> dealing with what happens when the response length is larger than your

Also true, but at least the initiator is in control of the amount
of data it receives; the target must not send more than was requested,
and simply indicates the amount of data the would have been available
had the initiator been able to receive it.

> datagram capability the upper layer can build everything else as

Your arguments for #2 do make sense.

However, The reason I had favored #5 over #2 is that I felt 5 would be
simpler; subject to occasional flow control for a large response
(which I hope will be rare), the target does not have to keep
track of where it was during the response.  Sending messages
fragmented in this way is very simple, although it might make
receiving and piecing them together more difficult.  This didn't
really bother me, though; I'd rather put more of the burden
on the initiator than on the target for anything discovery-
related, since the target is not in control of when and how often
this happens.

Anyway, #2 would work fine as well, but again, I'd really like
to avoid keeping state on the target.

--
Mark

> directly.  This is already a tradition within SCSI architecture.

Agreed.

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: Mark Bakke <[email-masked]>
Date: Mon, 09 Jul 2001 14:37:10 -0500

Rod-

I don't see where option #4 is any better than option #5,
since 5 is the same as 4, but with the initiator able to
restrict the amount of data returned.  The response is still
sent back in multiple text response PDUs, just as in #4.

I don't think that out-of-order PDUs are a problem, since
the text command would have connection allegiance.  I think
that we should explicitly state that text commands are not
to be recovered if something goes wrong.  That would eliminate
the need for a text response sequence number.

--
Mark

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: Stephen Bailey <[email-masked]>
Date: Mon, 09 Jul 2001 21:11:33 -0400

Mark,

> to avoid keeping state on the target.

You have to keep state on lots of state at this granularity (per
session/connection) on the target anyway.  All this would require is
an integer index into your target list table (which you obviously have
in some form to respond to SendTargets at all...)

One might suggest a variant of #5 with an initial offset argument to
the request, so you could get `what's left', but that really does
create a state problem.  With the simple iterator at least you know
that every return is (or was) a complete target description.

But then again, what do I know?  [this list is like driving, cf. Repo
Man]

Certainly doing #5 would not leave a criminal hole in the spec.
Neither would it probably be used in general by X-* operations.

Steph

---

From: Mark Bakke <[email-masked]>
Date: Wed, 11 Jul 2001 10:30:28 -0500

> in some form to respond to SendTargets at all...)

True; it just seemed better (well, certainly easier) to avoid the
problem of iterating SendTargets at by forcing the long response
into the current protocol with multiple PDUs.  I would imagine that
most responses will fit in a small number of PDUs anyway.

> that every return is (or was) a complete target description.

If we had to do that, I'd rather do the iterator.

> Neither would it probably be used in general by X-* operations.

It looks like most folks are comfortable implementing #5 for
SendTargets, even if it doesn't solve all of the possible problems
of X-* stuff.  Are you planning to use X-* commands?

BTW, if anyone is planning to make heavy use of X-* text commands, it
may be worth speaking up now, to make sure we do this right.

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054