Subject: RE: sector alignment for DataOut PDUs?

From: Paul Koning <[email-masked]>
Date: Fri, 1 Mar 2002 12:12:50 -0500

I like the way you've filled in the details on this.

It would make sense for targets to negotiate burst size limits, and
send requested length in R2T PDUs, that are multiples of the specified
DataPDUAlignment.  Is it necessary to specify that explicitly?  I
think it would make sense to do it, even though the current (rev 10 at
least) rules don't make it necessary).

       paul

---

From: "Martin, Nick" <[email-masked]>
Date: Fri, 1 Mar 2002 23:33:57 -0600

Paul,

Yes, it makes sense to select other values to have the same alignment, and most implementations would do so anyway.  I think all of the currently specified values limiting data size are likely (not required) to be powers of 2 from 2**9 (512) to 2**18 (256K).  I don't think anything breaks if they are not multiples of DataPDUAlignment which is likely (not required) to be a power of 2 from 2**0 (1) to 16 (64K).  If we make that a requirement and an something does not follow it, then what?  Do we want to get into specifying those kind of error cases?  I think this falls under "be conservative in what you send and liberal in what you accept" common sense rules.  If it does not break anything, why enforce it?  The benefits of doing it are clear enough.  

I thought about having DataPDUAlignment just specify the power of 2 (range 0 to 23), but I don't think restricting it in this way is called for.  It could make the length calculations slightly more efficient, but that just doesn't seem very important to me.

We will get into trouble of some of the other limits are less than DataPDULength because the only integer multiple allowed would be 0, potentially causing an infinite sequence of 0 length PDU payloads.  I will study how to correct this.

Thanks,
Nick
-----Original Message-----
From: Paul Koning [mailto:[email-masked]]
Sent: Friday, March 01, 2002 11:13 AM
To: Martin, Nick
Cc: [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

       paul

---

From: "THALER,PAT (A-Roseville,ex1)" <[email-masked]>
Date: Mon, 4 Mar 2002 10:34:07 -0700

All,

Putting in a requirement for sector alignment will make implementation of
bridges difficult. I've been looking through the Fibre Channel specs and I
cannot find any requirement that transfers be in block size multiples. If we
put a sector alignment requirement in, then bridges would have to buffer
data that arrived in a non-multiple until further data arrived to add to it.
They would also have to keep an accounting to know whether a transfer was
the last and therefore should be allowed to go even if it isn't a multiple
of block size. This would complicate bridging excessively.

Regards,
Pat

-----Original Message-----
From: Robert Snively [mailto:[email-masked]]
Sent: Monday, March 04, 2002 7:29 AM
To: Paul Koning; [email-masked]
Cc: [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

> > There has always been a feeling that it was nice to do things
> > on convenient boundaries, including memory page boundaries and
> > device physical block boundaries, but SCSI has long since 
> > elected to perform any operation on almost any boundary.  
> SCSI drives
> > and related operating systems have commonly used block sizes of 512
> > bytes and 520 bytes and less commonly of other values.  
> > 
> > The only requirement we were able to enforce in previous
> > SCSI protocols is that all but the last PDU of a transfer
> > were required to be on 4 byte boundaries.  We were also
> > able to enforce the maximum values and a requirement that
> > the transferred data count exactly match the requested byte
> > count.
> 
> I don't see that last point.  Certainly not for unsolicited data --
> and for R2T I can find no stated requirement to send exactly the
> requested count either.  Or did you mean the total for the entire
> operation as opposed to the individual bursts?
>  

I haven't reviewed all of iSCSI's rules with respect to this
particular issue.  There appears to be an assumption of 
non-congested processor-sized buffers that allow unsolicited
write data transfer.  That may be a convenient assumption for
some devices, but a problem for those that want more control
of their buffering.  In general, the requests for write data 
are part of the logical unit's write buffer management
and data striping management.  If you don't supply
exactly what is asked for, you foul up the buffer management of the
target, sometimes causing pathological performance problems.

> > I would strongly suggest that you not bother to enforce any
> > other boundary restrictions.
> > 
> > I believe Eddy was right when he asked:
> > 
> > 	Given that the TCP issues of reassembly 
> > 	are so much more complicated than
> > 	the concatenation issues necessary for 
> > 	target data alignment, is there
> > 	really a problem here?
> > 
> > My answer to him would be, no, there is not a problem.
> 
> I answered it before when Eddy said that, but I'll summarize it again:
> I'm not talking about a problem, I'm talking about an opportunity for
> simplifying the target and making it more efficient.
> 
> Given alignment, each iSCSI PDU that carries data can be sent to disk
> by itself, because it corresponds exactly to one or more disk blocks.
> Without alignment, doing writes when the data arrives is still
> possible, but it clearly adds complexity because PDU boundaries don't
> match disk block boundaries.
> 
> I made the proposal because it clearly helps the target and appears to
> add no significant burden on the initiator.  The feedback to date
> indicates that this is indeed the case.
> 
> Are you saying "I don't care one way or the other" or are you saying
> "I feel this is a bad idea because it creates problems you haven't
> thought of"?  I'm reading your comment as the former; if you meant the
> latter, could you elaborate?

I think I am saying that I care, and that it is an idea that
will trap you into limiting simplifications.  In general, the 
initiator's data flow engine has no knowledge of the logical 
unit's physical block structure.  In addition, the page structure
from which you are obtaining data in the host will usually have boundaries
that are unrelated to the logical unit's physical block structure.
While it is perfectly possible to align PDU's with blocks, it
requires SCSI ULP information to be forced in to the lower level
data transfer hardware.  This is a significant inconvenience and
will cost performance and/or complexity.  Note that the blocks
in a logical unit are not all required to be the same size, and in
tape drives and some optical devices are typically not the same size.

I still think it is best for the logical unit to ask for what it wants,
then expect to get it.  The logical unit is the only device in
the system that really knows what it needs and when it needs it.
That is one of the important SCSI principles, allowing the logical unit
to take over all the media dependent functions from the initiator
and the host programming.

---

From: [email-masked]
Date: Mon, 4 Mar 2002 14:53:57 -0700

Rod,

I don't think a maximum PDU size would have made sense for v08 or earlier
either.

When the size was negotiated, the resulting maximum receive PDU would be the
minimum of the two offered key values. If the target wanted 512 and the
initiator was offering 493, then the target would have had to accept 493. To
let a target choose a size that makes the data convenient for its buffers,
then one would have to have a list negotiation rather than numeric.

Such a requirement would be very burdensome for bridges which can't force
the data to be delivered to them into neat sizes.

Regards,
Pat Thaler

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Tuesday, February 26, 2002 6:08 PM
To: Martin, Nick; Paul Koning; [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

Nick,

	This makes sense for unsolicited data but it doesn't stop the
initiator sending more than one "weird" sized DATA-OUT PDU in
response to an R2T, even if the R2T is for a multiple of 512
bytes.

	Strengthening the wording to make sending "full" PDUs a
requirement is not really practical in the post v08 world where
we don't have a negotiated PDU size. It would have made sense
when the max PDU size was agreed upon but given the current
scheme of allowing the receiver to name an arbitrary sized
maximum receive PDU it would place an unreasonable burden on the
initiator.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On
Behalf Of
Martin, Nick
Sent: Wednesday, February 27, 2002 12:14 AM
To: Paul Koning; [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

Paul,

For all data carrying PDUs except the last in a sequence, the
sender is
expected to send maximum sized PDU allowed.  When the negotiated
maximum
is a multiple of 512, this effect is what you request.

I thought this was a requirement, but I did not find it as such
in draft
10.  I did find this:

: 8.5 Unsolicited Data and Performance
: Unsolicited data on write are meant to reduce the effect of
latency on
: throughput (no R2T is needed to start sending data). In
addition,
: immediate data are meant to reduce the protocol overhead (both
bandwidth
: and execution time).
: However, negotiating an amount of unsolicited data for writes
and
: sending less than the negotiated amount when the total data
amount to
: be sent by a command is larger than the negotiated amount may
negatively
: impact performance and may not be supported by all the targets.

This is a warning that an initiator sending less than the
negotiated
maximum when the expected data transfer is greater than the
maximum a)
may reduce performance and b) may not be supported by all
targets.

IMHO, it makes more sense to include stronger wording encouraging
maximum negotiated length transfers rather than to add another
parameter
requiring PDU breaks on different boundaries.

If such initiator behavior may not be supported by all targets,
then the
initiators SHOULD NOT do it.

A disk target which can not handle such behavior is forgiven in
advance
;).

Thanks,
Nick

> Sent: Tuesday, February 26, 2002 3:51 PM
> To: [email-masked]
> Subject: sector alignment for DataOut PDUs?
>
>
> In the past, a number of login parameters were specified as
multiples
> of 512 bytes, which makes a lot of sense for disk targets (but
not so
> much for tape targets).  With the spec as it stands now, you
can
> negotiate pretty much arbitrary burst sizes etc.  And in any
case, the
> sender can pick "strange" PDU sizes even if the negotiated
sizes are
> multiples of 512, because after all those are limits, not
required
> sizes.
>
> It would be very attractive for disk targets to be able to
specify
> that they require DataOut PDUs to be multiples of 512 bytes in
length.
> That way, any PDU would correspond exactly to one or more
sectors,
> rather than potentially having several PDUs straddling sector
> boundaries as is currently permitted.  This could be done by a
Boolean
> key (512 byte alignment, yes or no).  (An integer key would
allow
> other values than 512, but I'm not sure that such flexibility
is a
> whole lot more useful.)
>
> Basically, the effect of this feature would be to tell the
initiator
> that it must send DataOut PDUs and immediate data whose length
is
> always a multiple of 512.  Obviously, targets can't ask for
that
> unless the devices on that target already come with such a
limitation.

---

From: "Martin, Nick" <[email-masked]>
Date: Mon, 4 Mar 2002 19:43:33 -0600

Pat,

Are you considering a bridge between iSCSI initiator and FC target, or the reverse?

For the former case, the bridge is an iSCSI target.  The buffer size reported to the iSCSI initiator as DataPDUAlignment would be one which simplifies command and memory management within the bridge.  If the bridge can conveniently manage its memory in 1 byte increments, then report 1 byte which does not restrict the initiator.  If it prefers 1024 byte increments, it can send that to the iSCSI initiator.

For the latter case, the bridge is an iSCSI initiator.  Perhaps the iSCSI target has requested DataPDUAlignment such as 1024 bytes, but the FC initiator sends unaligned data, or perhaps 512 byte data.  This is the same situation the iSCSI target is in with no data alignment in iSCSI.  Data has arrived, but nothing can be done with it until more data arrives to fill the next transfer unit.

This is perhaps acceptable for some store-and-forward "bridge" designs, but rules out some "switch" or other low latency pass-through middle device designs.  When the alignment requirement can not be enforced for the FC or other protocol device on the other side, store-and-forward buffering would be required.

So we come to, DataPDUAlignment as a requirement is good for some designs and bad for others.  I believe that most initiators, even without this parameter=value, will break data PDUs on convenient boundaries.  If an initiator does not do so, the target needs to do the bookkeeping.  When the target implements bookkeeping and the initiator sends aligned data, the target operates efficiently.  When the initiator does not send conveniently aligned data, the target must not fail.

DataPDUAlignment could be advisory in nature.  The target advises the initiator of the transfer increment which will be optimal for the target.  When the initiator can not (or does not) break its transfers on these boundaries, the target may be less efficient.  No requirement is removed from the target, so no development work is saved developing or testing it.  The target is required to accept data PDUs of any otherwise legal length.  No new requirement is imposed on the initiator, nor on a bridge excepting to accept this parameter.  However, iSCSI initiators can now be given enough information to enable them to avoid unaligned data transfers to the target.  Targets will be able to operate at maximum efficiently when the initiators utilize this information.

If it needs to be advisory in nature, therefore optional to implement, I suggest that it could be tabled so as not to interfere with progress toward iSCSI 1.0.  On the other hand, being optional to implement, only required to understand, it is a small impact to implementations currently underway.

As someone else mentioned, there are targets whose block size is not a power of two.  Without such a parameter=value, aligned transfers to these targets could be rare. 

Thanks,
Nick

> From: THALER,PAT (A-Roseville,ex1) [mailto:[email-masked]]
> Sent: Monday, March 04, 2002 11:34 AM
> To: Robert Snively; Paul Koning
> Cc: [email-masked]

---

From: "Julian Satran" <[email-masked]>
Date: Tue, 5 Mar 2002 08:13:32 +0200

Bob,

I completely agree with your line of reasoning.  In SCSI the target is the 
master of the transfer and gets what it wants and it manages its resources 
as well as it can.  Buffer management is handled by both sides expressing 
their own requirements with the PDU length negotiation. 

I do not think that this is a thread that we would like to linger on.

Julo

Robert Snively <[email-masked]>
Sent by: [email-masked]
04-03-02 17:29
Please respond to Robert Snively

        To:     Paul Koning <[email-masked]>, [email-masked]
        cc:     [email-masked]
        Subject:        RE: sector alignment for DataOut PDUs?

---

From: Robert Snively <[email-masked]>
Date: Wed, 13 Mar 2002 13:16:25 -0800

Why would an initiator provide any different amount of data than
that requested by an R2T?  By the SCSI SAM definitions, it always
has that data available to send.  

> However, if
> you want to require the initiator to send exactly the amount that the
> target is asking for (for example, by changing the description of R2T
> to say that the amount of data sent by the initiator "MUST be exactly
> equal to the Desired Data Transfer Length specified in the R2T" rather
> than "MUST not exceed the Desired Data Transfer Length specified in
> the R2T" as in the current text), that too imposes a constraint on the
> initiator's data flow engine by the target. 

Yes, I absolutely believe that the description of R2T should be changed
to read (for write data):

	"MUST be exactly equal to the Desired Data Transfer 
	Length specified in the R2T"

The target should never ask for more data than is available or
can be transferred according to any constraining parameters, and
should be considered guilty of a protocol error if it does.  

I would have to be convinced to think that this is not a last call issue.

Bob

> Sent: Monday, March 04, 2002 11:07 AM
> To: [email-masked]
> Cc: [email-masked]
> >>>>> "Robert" == Robert Snively <[email-masked]> writes:
> 
>  >> > There has always been a feeling that it was nice to do things >
>  >> on convenient boundaries, including memory page boundaries and >
>  >> device physical block boundaries, but SCSI has long since >
>  >> elected to perform any operation on almost any boundary.  SCSI
>  >> drives > and related operating systems have commonly used block
>  >> sizes of 512 > bytes and 520 bytes and less commonly of other
>  >> values.  > > The only requirement we were able to enforce in
>  >> previous > SCSI protocols is that all but the last PDU of a
>  >> transfer > were required to be on 4 byte boundaries.  We were also
>  >> > able to enforce the maximum values and a requirement that > the
>  >> transferred data count exactly match the requested byte > count.
>  >> 
>  >> I don't see that last point.  Certainly not for unsolicited data
>  >> -- and for R2T I can find no stated requirement to send exactly
>  >> the requested count either.  Or did you mean the total for the
>  >> entire operation as opposed to the individual bursts?
>  >> 
> 
>  Robert> I haven't reviewed all of iSCSI's rules with respect to this
>  Robert> particular issue.  There appears to be an assumption of
>  Robert> non-congested processor-sized buffers that allow unsolicited
>  Robert> write data transfer.  That may be a convenient assumption for
>  Robert> some devices, but a problem for those that want more control
>  Robert> of their buffering.  In general, the requests for write data
>  Robert> are part of the logical unit's write buffer management and
>  Robert> data striping management.  If you don't supply exactly what
>  Robert> is asked for, you foul up the buffer management of the
>  Robert> target, sometimes causing pathological performance problems.
> 
> I agree that this is a concern.  But the iSCSI spec allows initiators
> to do what you don't want them to do.
> 
>  >> ...
>  >> Given alignment, each iSCSI PDU that carries data can be sent to
>  >> disk by itself, because it corresponds exactly to one or more disk
>  >> blocks.  Without alignment, doing writes when the data arrives is
>  >> still possible, but it clearly adds complexity because PDU
>  >> boundaries don't match disk block boundaries.
>  >> 
>  >> I made the proposal because it clearly helps the target and
>  >> appears to add no significant burden on the initiator.  The
>  >> feedback to date indicates that this is indeed the case.
>  >> 
>  >> Are you saying "I don't care one way or the other" or are you
>  >> saying "I feel this is a bad idea because it creates problems you
>  >> haven't thought of"?  I'm reading your comment as the former; if
>  >> you meant the latter, could you elaborate?
> 
>  Robert> I think I am saying that I care, and that it is an idea that
>  Robert> will trap you into limiting simplifications.  In general, the
>  Robert> initiator's data flow engine has no knowledge of the logical
>  Robert> unit's physical block structure.  In addition, the page
>  Robert> structure from which you are obtaining data in the host will
>  Robert> usually have boundaries that are unrelated to the logical
>  Robert> unit's physical block structure.  While it is perfectly
>  Robert> possible to align PDU's with blocks, it requires SCSI ULP
>  Robert> information to be forced in to the lower level data transfer
>  Robert> hardware.  This is a significant inconvenience and will cost
>  Robert> performance and/or complexity.  Note that the blocks in a
>  Robert> logical unit are not all required to be the same size, and in
>  Robert> tape drives and some optical devices are typically not the
>  Robert> same size.
> 
>  Robert> I still think it is best for the logical unit to ask for what
>  Robert> it wants, then expect to get it.  The logical unit is the
>  Robert> only device in the system that really knows what it needs and
>  Robert> when it needs it.  That is one of the important SCSI
>  Robert> principles, allowing the logical unit to take over all the
>  Robert> media dependent functions from the initiator and the host
>  Robert> programming.
> 
> I think we are in agreement.
> 
> As for the point about the initiator data flow engine having no
> knowledge of the target LU block structure, that is true.  However, if
> you want to require the initiator to send exactly the amount that the
> target is asking for (for example, by changing the description of R2T
> to say that the amount of data sent by the initiator "MUST be exactly
> equal to the Desired Data Transfer Length specified in the R2T" rather
> than "MUST not exceed the Desired Data Transfer Length specified in
> the R2T" as in the current text), that too imposes a constraint on the
> initiator's data flow engine by the target. 
> 
> That constraint isn't quite identical to the one needed to support the
> data PDU alignment, but it's pretty similar.  Either way, the
> initiator lost some of its freedom to send exactly what it wants to
> send.
> 
> Also, the arguments for doing it are the same in both cases: giving
> the target some control over how the initiator breaks up the outgoing
> data, so the target can arrange its buffering and memory management
> efficiently.
> 
> You're right that this proposal isn't particularly applicable to
> tapes.  I don't want to claim that every target will want to use the
> proposed alignment capability -- only that there are enough targets
> that will get a significant benefit from it to make the feature worth
> having. 
> 
> 	paul
> 
>