Subject: RE: sector alignment for DataOut PDUs?

From: Robert Snively <[email-masked]>
Date: Mon, 4 Mar 2002 07:29:11 -0800

> > There has always been a feeling that it was nice to do things
> > on convenient boundaries, including memory page boundaries and
> > device physical block boundaries, but SCSI has long since 
> > elected to perform any operation on almost any boundary.  
> SCSI drives
> > and related operating systems have commonly used block sizes of 512
> > bytes and 520 bytes and less commonly of other values.  
> > 
> > The only requirement we were able to enforce in previous
> > SCSI protocols is that all but the last PDU of a transfer
> > were required to be on 4 byte boundaries.  We were also
> > able to enforce the maximum values and a requirement that
> > the transferred data count exactly match the requested byte
> > count.
> 
> I don't see that last point.  Certainly not for unsolicited data --
> and for R2T I can find no stated requirement to send exactly the
> requested count either.  Or did you mean the total for the entire
> operation as opposed to the individual bursts?
>  

I haven't reviewed all of iSCSI's rules with respect to this
particular issue.  There appears to be an assumption of 
non-congested processor-sized buffers that allow unsolicited
write data transfer.  That may be a convenient assumption for
some devices, but a problem for those that want more control
of their buffering.  In general, the requests for write data 
are part of the logical unit's write buffer management
and data striping management.  If you don't supply
exactly what is asked for, you foul up the buffer management of the
target, sometimes causing pathological performance problems.

> > I would strongly suggest that you not bother to enforce any
> > other boundary restrictions.
> > 
> > I believe Eddy was right when he asked:
> > 
> > 	Given that the TCP issues of reassembly 
> > 	are so much more complicated than
> > 	the concatenation issues necessary for 
> > 	target data alignment, is there
> > 	really a problem here?
> > 
> > My answer to him would be, no, there is not a problem.
> 
> I answered it before when Eddy said that, but I'll summarize it again:
> I'm not talking about a problem, I'm talking about an opportunity for
> simplifying the target and making it more efficient.
> 
> Given alignment, each iSCSI PDU that carries data can be sent to disk
> by itself, because it corresponds exactly to one or more disk blocks.
> Without alignment, doing writes when the data arrives is still
> possible, but it clearly adds complexity because PDU boundaries don't
> match disk block boundaries.
> 
> I made the proposal because it clearly helps the target and appears to
> add no significant burden on the initiator.  The feedback to date
> indicates that this is indeed the case.
> 
> Are you saying "I don't care one way or the other" or are you saying
> "I feel this is a bad idea because it creates problems you haven't
> thought of"?  I'm reading your comment as the former; if you meant the
> latter, could you elaborate?

I think I am saying that I care, and that it is an idea that
will trap you into limiting simplifications.  In general, the 
initiator's data flow engine has no knowledge of the logical 
unit's physical block structure.  In addition, the page structure
from which you are obtaining data in the host will usually have boundaries
that are unrelated to the logical unit's physical block structure.
While it is perfectly possible to align PDU's with blocks, it
requires SCSI ULP information to be forced in to the lower level
data transfer hardware.  This is a significant inconvenience and
will cost performance and/or complexity.  Note that the blocks
in a logical unit are not all required to be the same size, and in
tape drives and some optical devices are typically not the same size.

I still think it is best for the logical unit to ask for what it wants,
then expect to get it.  The logical unit is the only device in
the system that really knows what it needs and when it needs it.
That is one of the important SCSI principles, allowing the logical unit
to take over all the media dependent functions from the initiator
and the host programming.

---

From: Paul Koning <[email-masked]>
Date: Mon, 4 Mar 2002 14:07:06 -0500

>>>>> "Robert" == Robert Snively <[email-masked]> writes:

 >> > There has always been a feeling that it was nice to do things >
 >> on convenient boundaries, including memory page boundaries and >
 >> device physical block boundaries, but SCSI has long since >
 >> elected to perform any operation on almost any boundary.  SCSI
 >> drives > and related operating systems have commonly used block
 >> sizes of 512 > bytes and 520 bytes and less commonly of other
 >> values.  > > The only requirement we were able to enforce in
 >> previous > SCSI protocols is that all but the last PDU of a
 >> transfer > were required to be on 4 byte boundaries.  We were also
 >> > able to enforce the maximum values and a requirement that > the
 >> transferred data count exactly match the requested byte > count.
 >> 
 >> I don't see that last point.  Certainly not for unsolicited data
 >> -- and for R2T I can find no stated requirement to send exactly
 >> the requested count either.  Or did you mean the total for the
 >> entire operation as opposed to the individual bursts?
 >> 

 Robert> I haven't reviewed all of iSCSI's rules with respect to this
 Robert> particular issue.  There appears to be an assumption of
 Robert> non-congested processor-sized buffers that allow unsolicited
 Robert> write data transfer.  That may be a convenient assumption for
 Robert> some devices, but a problem for those that want more control
 Robert> of their buffering.  In general, the requests for write data
 Robert> are part of the logical unit's write buffer management and
 Robert> data striping management.  If you don't supply exactly what
 Robert> is asked for, you foul up the buffer management of the
 Robert> target, sometimes causing pathological performance problems.

I agree that this is a concern.  But the iSCSI spec allows initiators
to do what you don't want them to do.

 >> ...
 >> Given alignment, each iSCSI PDU that carries data can be sent to
 >> disk by itself, because it corresponds exactly to one or more disk
 >> blocks.  Without alignment, doing writes when the data arrives is
 >> still possible, but it clearly adds complexity because PDU
 >> boundaries don't match disk block boundaries.
 >> 
 >> I made the proposal because it clearly helps the target and
 >> appears to add no significant burden on the initiator.  The
 >> feedback to date indicates that this is indeed the case.
 >> 
 >> Are you saying "I don't care one way or the other" or are you
 >> saying "I feel this is a bad idea because it creates problems you
 >> haven't thought of"?  I'm reading your comment as the former; if
 >> you meant the latter, could you elaborate?

 Robert> I think I am saying that I care, and that it is an idea that
 Robert> will trap you into limiting simplifications.  In general, the
 Robert> initiator's data flow engine has no knowledge of the logical
 Robert> unit's physical block structure.  In addition, the page
 Robert> structure from which you are obtaining data in the host will
 Robert> usually have boundaries that are unrelated to the logical
 Robert> unit's physical block structure.  While it is perfectly
 Robert> possible to align PDU's with blocks, it requires SCSI ULP
 Robert> information to be forced in to the lower level data transfer
 Robert> hardware.  This is a significant inconvenience and will cost
 Robert> performance and/or complexity.  Note that the blocks in a
 Robert> logical unit are not all required to be the same size, and in
 Robert> tape drives and some optical devices are typically not the
 Robert> same size.

 Robert> I still think it is best for the logical unit to ask for what
 Robert> it wants, then expect to get it.  The logical unit is the
 Robert> only device in the system that really knows what it needs and
 Robert> when it needs it.  That is one of the important SCSI
 Robert> principles, allowing the logical unit to take over all the
 Robert> media dependent functions from the initiator and the host
 Robert> programming.

I think we are in agreement.

As for the point about the initiator data flow engine having no
knowledge of the target LU block structure, that is true.  However, if
you want to require the initiator to send exactly the amount that the
target is asking for (for example, by changing the description of R2T
to say that the amount of data sent by the initiator "MUST be exactly
equal to the Desired Data Transfer Length specified in the R2T" rather
than "MUST not exceed the Desired Data Transfer Length specified in
the R2T" as in the current text), that too imposes a constraint on the
initiator's data flow engine by the target. 

That constraint isn't quite identical to the one needed to support the
data PDU alignment, but it's pretty similar.  Either way, the
initiator lost some of its freedom to send exactly what it wants to
send.

Also, the arguments for doing it are the same in both cases: giving
the target some control over how the initiator breaks up the outgoing
data, so the target can arrange its buffering and memory management
efficiently.

You're right that this proposal isn't particularly applicable to
tapes.  I don't want to claim that every target will want to use the
proposed alignment capability -- only that there are enough targets
that will get a significant benefit from it to make the feature worth
having. 

	paul