Subject: RE: sector alignment for DataOut PDUs?

From: "Buck Landry" <[email-masked]>
Date: Tue, 26 Feb 2002 23:15:00 -0600

>>> Also, I think we should consider making the "alignment" any
multiple of 512 the target wants
<<<

Well.. if we put no restrictions on it, and I decided my target should only handle one data-out pdu per scsi write cmd, I might abuse this and force an alignment of (512 * ReallyHugeNumber).  This might demand more resources or complexity from the initiator than is wished for, although I can't think of any reasons off the top of my head (perhaps I should think harder).  OTOH I am fully aware that the target is the entity that is supposed to be cheap n' easy to implement...

Perhaps you could limit the alignment size by the target's MaxRecvPDULength?  Or...  

I believe this "multiple-of" idea also needs to be reconciled with MaxBurstSize.  If MaxBurstSize is negotiated by the initiator to 512 bytes, then by my understanding the target still must send 512-byte R2Ts for writes (and hence, expect 512-byte data-out pdus), even if the target's sector size is 4k.

--buck 

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Tuesday, February 26, 2002 5:51 PM
To: Paul Koning
Cc: [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

	I was talking about DATA-OUT PDUs.

	I would suggest we also restrict any target that wants to
negotiate for the 512 byte DATA-OUT option to not doing anything
weird with FirstBurstSize or MaxRecvPDULength. I think it would
unreasonable to allow a target to require 512 byte aligned
DATA-OUTs and have those other two parameters not multiples of
512 bytes.

	How do we handle the case where the iSCSI target node has
multiple LUNs and not all of them are disks? Perhaps the rule for
transfers lengths that are not multiples of 512 bytes works OK
here?

	Also, I think we should consider making the "alignment" any
multiple of 512 the target wants, not just 512. Otherwise it
seems to me you are back where you started if you have a disk
with 4096 byte sectors and the initiator chooses to send you
DATA-OUT PDUs that are not a multiple of 4k.

	- Rod

-----Original Message-----
From: Paul Koning [mailto:[email-masked]]
Sent: Tuesday, February 26, 2002 11:05 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

>>>>> "Rod" == Rod Harrison <[email-masked]> writes:

 Rod> I have no objection in principle to this but I have a
concern
 Rod> over non-media access commands. Things like mode select
 Rod> payloads, i.e. defect lists etc are not necessarily 512
byte
 Rod> multiples. The wording would need to allow "odd" sizes when
the
 Rod> SCSI transfer length is not a 512 byte multiple.

Are those DataOut or DataIn?  The target can arrange inbound PDU
sizes
to be whatever multiple is helpful if it wants to, that isn't the
case
I was talking about.

In any case, if there are cases where the total transfer length
for
data out isn't a multiple of 512, then the rule would have to
allow
the last transfer to have an odd length.  The earlier transfers
could
still be constrained if the constraint is enabled.

      paul

---

From: "THALER,PAT (A-Roseville,ex1)" <[email-masked]>
Date: Wed, 27 Feb 2002 12:34:45 -0700

I agree with Rod. If we changed the rule to require full PDUs, then we would
have to also change the negotiation for the related keys.

Pat

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Wednesday, February 27, 2002 11:26 AM
To: Paul Koning; [email-masked]
Cc: [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

Paul,

	I strongly object to your alternative of tightening the spec
wording to force all PDUs to be max sized except the last. As I
mentioned in an earlier email, since draft v09 we don't have a
negotiated PDU size we have arbitrary maximums set by the
receivers. Forcing an initiator to send "full" PDUs is
unreasonable since the target may set a huge MaxRecvPDULength.

	The intent of the current wording was, if I remember correctly,
to have the initiator always send as much unsolicited data as
possible. This was to avoid the target having to wait for the F
bit on the unsolicited data before issuing R2T(s) for the
remaining payload. Note that unsolicited data, and its maximum
size, are negotiated parameters and that that current SHOULD is
quite different from forcing initiators to always send "full"
DATA-OUT PDUs.

	Please note that I don't object to your original proposal, just
to the alternative which I believe would be a huge functional
change.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On
Behalf Of
Paul Koning
Sent: Wednesday, February 27, 2002 5:20 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: sector alignment for DataOut PDUs?

>>>>> "Eddy" == Eddy Quicksall <[email-masked]>
writes:

 Eddy> Given that the TCP issues of reassembly are so much more
 Eddy> complicated than the concatenation issues necessary for
target
 Eddy> data alignment, is there really a problem here?

 Eddy> If there is a problem, can someone explain that in more
detail?

I'd describe it as an opportunity for optimization.

Yes, at the TCP level your segments can be strange sizes.  My
goal is
to make things better once you get up to the iSCSI level, and
you're
looking at an iSCSI PDU.

Right now, the initiator is allowed to send data out in strange
sized
chunks.  There's a requirement for 4 byte multiples but nothing
stronger than that.

If you're implementing a disk target, what you want to do is to
take
in the arriving stream of data from the initiator (DataOut or, if
applicable, immediate data) and send that off to disk as it
arrives.
If data out boundaries are 512 byte multiples, this is
straightforward: each time an iSCSI PDU arrives, you can send a
disk
write down your local disk I/O stack.  But as matters stand now,
it's
more complex because the data out may end in the middle of a disk
block, so the bookkeeping for what disk I/Os go with what iSCSI
PDUs
gets a lot more painful.

To add to this, it seems entirely likely that initiators will use
512
byte multiples, especially if you negotiate burst sizes and max
pdu
sizes that are 512 byte multiples.  But the current wording in
the
spec doesn't require that.  (It does hint that targets may object
if
you don't do it that way -- which makes no sense at all unless
there's
a MUST for initiators to do what targets are allowed to expect.)

So that's why I made the proposal.  An alternative would be to
tighten
the spec so it requires all but the last PDU to be exactly the
size of
the negotiated max size that applies.  That already appears to be
the
intent, so it may be the easier change.

	paul

---

From: "Martin, Nick" <[email-masked]>
Date: Wed, 27 Feb 2002 19:40:33 -0600

Paul,

As Rod Harrison points out, MaxRecvPDULength is not negotiated, it is
"declared" and is likely to be different for Initiator and Target.  I
had missed the implication of this change which occurred at draft 09.

I can now appreciate your earlier proposal.  

To be fair to all targets, we should allow targets with arbitrary sized
sectors/blocks to request the same kind of boundary alignment.  Today we
already have a need to support 512, 1K, 2K, and 4K sectors for disk, and
anything from 512 up to at least 64K blocks for tape (including
non-powers of 2 like 10K).  [Ouch, I just realized that the tape block
size can be changed mid-session, the most likely time being after a
media change.  The tape folks may not be able to use of this in the same
way, but let's not prevent them from trying.]  Further we do not really
want iSCSI initiators (or their drivers) to have to guess or issue SCSI
commands to learn the sector size.  The size needs to be specified by
the target during operational parameter negotiation if it is requesting
sector/block aligned PDU payloads.  

I actually do not think unaligned payloads are likely, but there is
nothing to prevent them.  We may simplify some target designs by
eliminating the requirement to support them.

Certainly, we want to eliminate loopholes which allow compliant but
non-interoperable implementations.

Please consider whether the following proposal meets all of your
requirements.

Proposed addition to chapter 12 to enable requiring sector aligned PDU
payloads:

"
DataPDUAlignment

Use: LO
Senders: Target and Initiator
Scope: SW

DataPDUAlignment=<number-512-to-(2**24-1)>

The default is 0.

The initiator and target negotiate the alignment boundary on which data
transfers may be continued to a subsequent PDU.

The value 0 means no boundary alignment limitation is imposed on data
carrying PDUs.

The minimum of two non-zero values is selected.  If one value is zero,
then the other value is selected.

This is intended primarily for a target to force an initiator to send
PDU data buffers whose boundaries match physical media boundaries such
as disk sector size.

When DataPDUAlignment is non-zero, for any PDU other than the last in a
sequence, the PDU data segment length (DataSegmentLength) is required to
be an integer multiple of DataPDUAlignment less than or equal to
MaxRecvPDULength.
"

Thanks,
Nick

> Sent: Wednesday, February 27, 2002 12:57 PM
> To: Martin, Nick
> Cc: [email-masked]
> >>>>> "Nick" == Martin, Nick <[email-masked]> writes:
> 
>  Nick> Paul, For all data carrying PDUs except the last in a sequence,
>  Nick> the sender is expected to send maximum sized PDU allowed.  When
>  Nick> the negotiated maximum is a multiple of 512, this effect is
>  Nick> what you request.
> 
>  Nick> I thought this was a requirement, but I did not find it as such
>  Nick> in draft 10. ...
> 
>  Nick> IMHO, it makes more sense to include stronger wording
>  Nick> encouraging maximum negotiated length transfers rather than to
>  Nick> add another parameter requiring PDU breaks on different
>  Nick> boundaries.
> 
> That sounds like a fine suggestion, if it gives the target enough
> power to achieve the alignment I'm looking for.  Strengthening the
> text you quoted is a start, but other places would have to be
> strengthened as well -- for example the rules on responding to an R2T.
> 
>  Nick> If such initiator behavior may not be supported by all targets,
>  Nick> then the initiators SHOULD NOT do it.
> 
> That's not sufficient.  You need "MUST NOT".  The reason: if you say
> "should not" then you can build a conforming initiator that cannot
> interoperate with a conforming target.  All correctly written protocol
> specs have the property that conformance implies interoperability.
> (Unfortunately, few specs pass this test... :-( )  So a target can be
> allowed to rely on full size PDUs only if initiators are required (not
> merely encouraged) to send full size PDUs.
> 
>        paul
> 
>

---

From: "Martin, Nick" <[email-masked]>
Date: Thu, 28 Feb 2002 18:33:38 -0600

Martins,

Comments imbedded.

> From: Martins Krikis [mailto:[email-masked]]
> Sent: Thursday, February 28, 2002 2:02 PM
> To: [email-masked]
> I like the DataPDUAlignment parameter.
> (Just mandating to send maximum amount of data
> that's allowed, while being a good thing,
> still cannot guarantee the right alignment
> if those limits are not the right multiples.)
> 
> But I don't see why the selection function should be
> the minimum of two numbers. That way the outcome may
> be completely inapropriate for the target. 
> I think that whatever the target declares should be
> the final value. So the selection function should
> be called, IMHO, "target said so".
+++
Yes, I see your point.
+++
> 
> Also, (I'm quoting Nick):
> >  be an integer multiple of DataPDUAlignment
> > less than or equal to MaxRecvPDULength.
> 
> In case of Data_out pdu-s even the last one should
> be aligned, shouldn't it? Since all PDU-s have
> brought data aligned on block-size, if the last
> one does the same, we guarantee that the total
> data transfer length is a multiple of block-size,
> which certainly must be so, right? So far I don't
> see a guarantee that it will be, but I feel it
> must match the length carried by CDB which is
> expressed as the number of blocks...
+++
For disks, yes this will be true.  For devices accepting variable length
records such as tape, there is no restriction on the total size.  My
intention is that only the last buffer required by the transfer would
not be an integer multiple.

I now picture a slightly more general case where the DataPDUAlignment
value requested represents the buffer memory management granularity of
the target.  This might be a convenient multiple of sector size other
than one.  Again in this case, only the final transfer would not fill
each DataPDUAlignment sized buffer allocated to receive it.

Finally, it has been suggested (by Eddy Quicksall) that allowing any
value including 1, but not zero eliminates special case processing for
0.  The value 1 means one byte alignment which is no restriction.  Any
value convenient to the target could be selected, or the target can
leave it unspecified which means it does not require alignment.
+++
> 
> Finally, speaking about MaxRecvPDULength, this
> really is the last of operational parameters that
> is not negotiated, but instead declared, right? That
> makes it different from others. Plus, the same key
> means different things depending on which direction
> it is sent. While I'm not as upset with this as I
> was about RFMarkInt, wouldn't it look cleaner if
> we used two different keys, MaxORecvPDULength and
> MaxIRecvPDULength? We could even say that they
> are negotiated with the very special selection
> functions "initiator said so" and "target said so",
> respectively...
> 
> Any comments?
> 
>   Martins Krikis, Intel Corp.
> 
> Disclaimer: These opinions are my own and may not
>             reflect those of my employer.
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Greetings - Send FREE e-cards for every occasion!
> http://greetings.yahoo.com
>

---

From: "Martin, Nick" <[email-masked]>
Date: Thu, 28 Feb 2002 18:51:08 -0600

Aryan,

In reply to your question "But what about those implementations which
would like to change the size after few transfers? Should they look/wait
for something in the spec/draft or try their own implementation?"

If you control both ends of the wire, you can add vendor specific X-
type of parameters to do what ever you want, but if you want to
interoperate with everyone, you need to raise your voice in support of
adding something like DataPDUAlignment to the spec.  Until it is
included, you can not prevent unaligned data transfer sizes.  (If you
have no desire to prevent such transfers, this parameter will not help
you.)

If you are for example a tape drive and you want to change the value on
the fly, then this parameter would need to be supported in text mode
negotiation during Full Feature Phase.  There are additional
implications if it can change with commands in flight.  I did not
explicitly state whether this was or was not permitted in my proposal.
I have not noticed any other parameters which explicitly can be changed
during FPP, although early on, I think that was one important reason to
have Text PDUs.

Also note that the scope of effect is a single value for the session
which includes the target and all of its LUs.  There did not seem to be
precedent for anything but Session or Connection scope.  It would
perhaps more logically have LU scope (no pun intended).

For a disk target, a single value is probably sufficient.  All sectors
on all LUs are (almost certainly) the same size.  For a multi LU tape
library, although each drive may be using a different (fixed or
variable) block size, there is perhaps still some value in specifying a
power of two PDU alignment to simplify buffer management.

If a target sets this value to its data buffer memory management
granularity, this could allow simplification or efficiency within the
target.  The granularity of target data buffer memory management is not
likely to change dynamically.  Efficiency of memory management in the
target is a main benefit.  Being to schedule device writes as each data
PDU arrives is another.  This is possible when no device block will span
PDUs.

Thanks,
Nick
> From: Ajit Aryan [mailto:[email-masked]]
> Sent: Thursday, February 28, 2002 1:42 AM
> To: Martin, Nick
> Subject: Re: sector alignment for DataOut PDUs?
> 
> 
> Nick,
> That looks fine.
> But what about those implementations which would
> like to change the size after few transfers?
> Should they look/wait for something in the spec/draft
> or try their own implementation?
> 
> Aryan
> 
> Martin, Nick wrote:
> 
> >Paul,
> >