Subject: Re: iSCSI: Iterating long text responses

From: Mark Bakke <[email-masked]>
Date: Thu, 05 Jul 2001 14:31:24 -0500

Santosh-

I was making the assumption that a text command stayed on a single
connection, and response PDUs from the target would be in-order,
and that if the connection failed, the target would throw away any
results it hadn't yet sent, and the initiator would just completely
re-do the command.  This meant we wouldn't need an offset, since
the initiator would have to receive them in order anyway.  However,
it certainly wouldn't hurt to add the offset.  I guess I am just
concerned that if we do, then someone will want to make use of the
field to be able to send text responses out of order, or to recover
them on another connection, and so on, so I would rather leave them
out unless we really need them.  How badly are they needed?

--
Mark

Santosh Rao wrote:
> 
> Mark,
> 
> IMO, alternative #4 is sufficient and is semantically aligned with the
> FC-GS-2/GS-3 GID_FT model.
> 
> It does require modification of the Text Response PDU to provide a
> Relative Offset (RO) field indicating the RO of the text response
> PDU from the start of the text response payload, in order to allow initiators
> to correctly re-assemble the received response PDUs.
> 
> Regards,
> Santosh
> 
> Mark Bakke wrote:
> >
> > Initiator developers-
> >
> >    Please respond to the questions at the end.
> >
> > Thanks,
> >
> > Mark
> >
> > The current iSCSI draft allows text command and response
> > PDUs of up to 4096 bytes.  While we don't see any real
> > problems for the command PDU size, commands such as SendTargets
> > can easily exceed the response size.
> >
> > There are several ways we can fix this.  The first two solutions
> > require no differences in the current iSCSI text command and
> > response; the latter three involve the use of the F bit.
> >
> > 1. Assume that such commands are done on a "special" connection
> >    or are handled completely in software, and allow its response
> >    PDU to be as large as it needs to be.
> >
> >    This one appears too restrictive to be a solid solution.  It
> >    will also weaken any data digests done on the longer PDU.
> >
> > 2. Create an iterative SendTargets key (and do the same for any
> >    other text commands that need this), that would allow the
> >    initiator to request the "next target" or "next address".
> >
> >    This would work, but would require any new command that needed
> >    a large response to implement an iterator.  It also means that
> >    each part of the response from the iterator would have to fit
> >    in the 4k PDU.
> >
> > The remainder of these require that only one text command sequence
> > be outstanding on a connection at a given time.  They use the F
> > bit to indicate the last PDU of the sequence.  Note that connection
> > allegiance is assumed for the entire sequence, and text commands
> > are never retried on another connection; a new command is issued
> > instead.
> >
> > 3. Do a text-response R2T, where the initiator controls when the
> >    next partial response is sent.  This would be more generic:
> >
> >    I->T  Text Command (F bit set)
> >    T->I  Text Response (first PDU, F bit cleared)
> >    I->T  Text Command (with some indicator that we want more)
> >    T->I  Text Response (next PDU, F bit cleared)
> >
> >    ...
> >
> >    I->T  Text Command (with indicator that we want more)
> >    T->I  Text Response (last PDU, F bit set)
> >
> >    The main problem with this is that the target must keep track
> >    of the state of its responses; if the initiator just stops asking,
> >    it may have to keep a buffered response around until it times out,
> >    the connection is dropped, or another text command comes along.
> >
> > 4. Allow multiple response PDUs to a single text command:
> >
> >    I->T  Text Command (F bit set)
> >    T->I  Text Response (first PDU, F bit cleared)
> >    T->I  Text Response (next PDU, F bit cleared)
> >
> >    ...
> >
> >    T->I  Text Response (last PDU, F bit set)
> >
> >    Basically, we are doing (3) without the R2T.  The initiator,
> >    upon sending the text command, must be prepared either to
> >    accept as many PDUs as come back, or throw them away if it
> >    can't handle them.  This solution is a lot like #1, but with
> >    the response spread across 4k PDUs.
> >
> >    Also note that this (and the following scheme) avoid the problem
> >    of the target keeping state; it sends ALL of the response PDUs
> >    without the initiator asking for them.
> >
> > 5. Do #4, but allow the initiator to specify the amount of data
> >    it is willing to accept:
> >
> >    I->T  Text Command (F bit set, AcceptResponseLength=4096)
> >    T->I  Text Response (first PDU, F bit set, TotalResponseLength=12288)
> >
> >    In the above example, we have created a new text command field:
> >
> >       AcceptResponseLength
> >
> >    And in the text response PDU:
> >
> >       TotalResponseLength
> >
> >    The initiator indicated it was willing to accept a 4k response.
> >    The target returned the first 4k in the text response, but set
> >    the F bit since it was at its limit.  It also returned a
> >    TotalResponseLength field.  Since this was greater than the
> >    AcceptResponseLength, the initiator knows the amount of buffer
> >    space it will need to get the full response.  It can then choose
> >    whether it will re-send the command, and if so, can give it a
> >    large enough response length:
> >
> >    I->T  Text Command (F bit set, AcceptResponseLength=12288)
> >    T->I  Text Response (first PDU, F bit clear)
> >    T->I  Text Response (next PDU, F bit clear)
> >    T->I  Text Response (last PDU, F bit set, TotalResponseLength=12288)
> >
> >    Note that most initiators will probably send an AcceptResponseLength
> >    of the largest response they would normally accept, and that most
> >    target responses will fit in one or a few PDUs anyway.
> >
> >    #5 is really a compromise between #3 and #4; the target has the
> >    benefit of being less statefull, and the initiator has the benefit
> >    of controlling the amount of data it receives.
> >
> > I would like to recommend either #4 or #5.  I think that #5 is
> > probably the safest solution, but #4 may not be a problem for anyone.
> >
> > Assuming that none of the implementors of initiators have a problem
> > with #4, I would pick that.
> >
> > If they do have a problem with it, we should go with #5.
> >
> > Targets probably don't care much whether we do #4 or #5.
> >
> > Initiator developers-
> >
> >   Please indicate which solution (#4 or #5) appeals to you.
> >
> > --
> > Mark A. Bakke
> > Cisco Systems
> > [email-masked]
> > 763.398.1054
> --
> #################################
> Santosh Rao
> Software Design Engineer,
> HP, Cupertino.
> email : [email-masked]
> Phone : 408-447-3751
> #################################

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054

---

From: Santosh Rao <[email-masked]>
Date: Thu, 5 Jul 2001 12:21:06 -0700 (PDT)

Mark,

Regards,
Santosh

---

From: Santosh Rao <[email-masked]>
Date: Thu, 05 Jul 2001 14:06:45 -0700

Mark Bakke wrote:
> re-do the command. 

Mark,

The CRC error checking and the Text command processing may be handled in
different layers
of the iSCSI stack (CRC in hardware, text command in firmware or
software), with CRC errors causing a discard of the errant PDU alone.
Thus, upon receipt of the final Text Response PDU the layer handling the
Text command may not be aware that 1 PDU of the returned set of
responses was lost.

Some ways to address this would be :
a. usage of a RO in each Text Response PDU.
b. a count of the number of Text Response PDUs sent in the final Text
Response ("f=1"). 
c. A sequence count applied to all multi-PDU text sequences.

In addition, the spec needs to explicitly word the "connection
allegiance" description to apply to all iSCSI commands, and not just
SCSI commands.

> This meant we wouldn't need an offset, since
> the initiator would have to receive them in order anyway.

The offset is required to ensure integrity of the response received.

Regards,
Santosh

>  However,

---

From: "Lawrence J. Lamers" <[email-masked]>
Date: Fri, 06 Jul 2001 08:05:13 -0700

Mark, Santosh,

The behavior in #5 is what most SCSI device guys are used to.  It is 
similar to the way MODE SENSE and other commands work that return parameter 
lists.  I would vote for #5.

Regards,

==========================================
Lawrence J. Lamers
email:  [email-masked]
Cell Phone:  (408) 234-0071
Home Phone:  (408) 578-1709

---

From: Mark Bakke <[email-masked]>
Date: Mon, 09 Jul 2001 15:44:07 -0500

Santosh-

Good point; I guess a relative offset would be useful for an
implementation that does this.  It would be used only for error
checking, and not for recovery.  I like this better than the
other two alternatives you had mentioned.  So I have to take back
my response to Rod as well.

The new wording of the second paragraph of 1.2.5 should take care
of text commands:

   If an iSCSI request is issued over one TCP connection, the 
   corresponding response or requested PDU MUST be sent over the same 
   connection. We call this "connection allegiance". 

Anyway, if we end up with #5, I will suggest the addition of
a relative offset in the text response as well.

--
Mark

-- 
Mark A. Bakke
Cisco Systems
[email-masked]
763.398.1054