Subject: RE: iSCSI: frame formats

From: Charles Monia <[email-masked]>
Date: Mon, 2 Apr 2001 14:21:57 -0700

Hi:

While we're on the subject of padding: Another (non-ip) transport has the
rule that padding is only allowed on the last data PDU of a command.  That
is, the initial and intermediate data transfer PDUs must not be padded.

I believe this simplifies implementations doing cut-through placement into
word-aligned buffers. 

Charles

> -----Original Message-----
> From: Barry Reinhold [mailto:[email-masked]]
> Sent: Monday, April 02, 2001 11:59 AM
> To: Rod Harrison; [email-masked]
> Subject: RE: iSCSI: frame formats 
> 
> 
> Rod,
> 	I'm pretty sure the we decided to go with bytes at the 
> 50th IETF meeting
> for data length. No one felt that a max data size of 16 megs 
> in an iSCSI PDU
> was an issue (remember this is not max SCSI data transfer 
> size, it is max
> data bytes in an iSCSI PDU).
> 	I do think we need to word the padding with a bit of 
> care, as I can
> envision people doing this in at least two different ways 
> that would not
> interoperate.
> 	The interoperability issues on the padding question 
> boils down to: "Where
> in the TCP stream do I put the Data Digest if the number of 
> bytes I have to
> send is not a multiple of 4?"
> 	My expectation was that the transmitter would be null 
> padding the data
> portion of the iSCSI PDU to a word boundary, then sticking on 
> the digest.
> Thus the padding is actually in the TCP stream. For example 
> if <d> = a data
> octet, <p> = pad octet, and <dd> = data digest octet, then 
> the TCP stream
> for a 6 byte data transfer would look as follows:
> <octets in the header - end modulo 4> <d> <d> <d> <d> <d> <d> 
> <p> <p> <dd>
> ....
> 	The receiver would get the value "6" in the data length 
> portion of the
> header. After pulling out the header, the receiver would pull 
> out 8 bytes of
> "data + pad" and then get the digest.
> 	The other way to do this to have a "virtual pad" such 
> that padding is
> created by the receiver when construction the iSCSI PDU from 
> the TCP stream.
> The padding is never actually in the TCP stream itself.
> 	I do not think this is as helpful, but whatever we do, 
> the spec. should
> address this minor detail so we don't trip over it.
> 
> 
> >-----Original Message-----
> >From: [email-masked] [mailto:[email-masked]]On 
> Behalf Of
> >Rod Harrison
> >Sent: Monday, April 02, 2001 1:44 PM
> >To: [email-masked]
> >Subject: RE: iSCSI: frame formats
> >
> >
> >Stephen,
> >
> >	I don't agree with your concerns about the maximum PDU
> >size. I think few targets, or indeed initiators, will be
> >interested in negotiating PDU sizes anywhere near this
> >large, so big transfers will have to be fragmented anyway.
> >
> >	However, I share your concern about the padding. I don't
> >really see why we are considering it. If one has a local
> >alignment issue it can, and I believe should, be taken care
> >of locally and not in the specification. There are several
> >easy ways of handling this sort of thing; insert and remove
> >the pad locally; separate header and payload buffers
> >allowing each to be naturally aligned, etc.
> >
> >	If we have to pad then every read of non-header data will
> >have to involve a rounding calculation on the length, and
> >then perhaps a second read to discard the pad if the
> >underlying buffer is the exact size of the data. Possibly
> >the same on send, if the data buffer is the exact size the
> >transmit code can't just 'go off the end,' it will have to
> >send the data, and then fake up some pad and make another
> >send.
> >
> >	Am I missing something here, why do we care about padding?
> >
> >	- Rod
> >
> >-----Original Message-----
> >From: [email-masked] [mailto:[email-masked]]On
> >Behalf Of
> >Stephen Bailey
> >Sent: Monday, April 02, 2001 3:18 PM
> >To: [email-masked]
> >Subject: Re: iSCSI: frame formats
> >
> >
> >Sandeep,
> >
> >> DataLen will now be max 8M/4M but then we dont wish to
> >have large
> >> iSCSI PDUs in any case.
> >
> >This max size is getting the the point where I'm sure it'll
> >be an
> >irritant.
> >
> >I would like (but, in fact, a sure way to guarantee that it
> >won't
> >happen is for me to like it :^) to view iSCSI PDUs as the
> >expected
> >grain size at which you might have software involvement in
> >an
> >otherwise hardware-driven iSCSI implementation.
> >
> >For example, when a target is returning read data for
> >multiple
> >outstanding reads, it might want to return a bit at a time
> >from each,
> >and each `bit' should be an iSCSI PDU.  Clearly sending
> >these bits
> >will be a software-level decision.  That certainly was the
> >rational
> >for allowing multiple FCP DATA PDUs per read operation, and
> >I naively
> >assumed similar logic was being applied here.
> >
> >The alternative is to say that the hardware will do iSCSI
> >PDU
> >chunking, but if that's the case, I expect that the header
> >is, well, a
> >bit bulky.
> >
> >I'm also incredibly unexcited about having data length be a
> >multiple
> >of 4 bytes (if that's still in the cards).  There operations
> >within
> >the SCSI command set which return arbitrary length data.
> >There are
> >perfectly nominal cases where you get less data than you
> >requested
> >(e.g. inquiry & request sense).  Furthermore, the SCSI
> >architecture
> >does not prohibit this, even though certain commands do, so
> >it is not
> >for iSCSI to say anything about this one way or the other.
> >
> >The problem with handling lengths that include padding comes
> >when
> >you're trying to move the data into a buffer which is a
> >non-multiple
> >length.  For example, if I ask for 22 bytes of inquiry data
> >(with a 22
> >byte buffer), what can I say, at the time a PDU arrives that
> >has 24
> >bytes?  It might have 21 or 22 bytes (or perhaps even,
> >erroneously 23
> >or 24 bytes).  A data residual coming later will tell the
> >software how
> >much was actually there, but it can't tell the hardware.
> >The typical
> >expectation of this type of transfer is that it will only
> >overwrite
> >bytes of the buffer that are actually transferred, but
> >having padded
> >lengths will not allow this.
> >
> >The completely standard solution to carrying arbitrary data
> >of
> >arbitrary length in an aligned transfer unit is to pad the
> >transfer
> >unit but report the exact (shorter) length.  Another
> >solution, used by
> >FC, is to carry a pad length.  In iSCSI, why bother---you've
> >just
> >reintroduced added the 2 bits you were trying to remove?
> >
> >The data length scenario is not comparable to IP header
> >lengths, where
> >what is being carried is not arbitrary data.
> >
> >Certainly, for iSCSI additional header segments (AHSs) you
> >could
> >arguably use this cell length technique, since we can
> >control what
> >we're carrying (AHSs that need exact byte lengths will have
> >to be
> >internally self-describing) but frankly, I still think it's
> >a bad
> >idea.
> >
> >I can't understand why we're messing around with all these
> >tricky
> >`solutions' to standard problems.  We should avoid the
> >temptation to
> >get cute, and wholesomely provide the same capabilities as
> >any other
> >SCSI transport.  Specifically:
> >  o allow long PDUs
> >  o carry exact data lengths
> >
> >Steph
> >
>

---

From: [email-masked]
Date: Tue, 3 Apr 2001 03:16:39 +0200

Steph,

Very thoughtful. The why did you not speak up for Format 1 - it has no
tricks.

And BTW the length (data) as it stands now includes only the effective
bytes - padding is implicit.
Not so for AHS - but there the private length could end-up being bytes with
padding implied.

Julo

Stephen Bailey <[email-masked]> on 02/04/2001 16:17:33

Please respond to Stephen Bailey <[email-masked]>

To:   [email-masked]
cc:

Sandeep,

> DataLen will now be max 8M/4M but then we dont wish to have large
> iSCSI PDUs in any case.

This max size is getting the the point where I'm sure it'll be an
irritant.

I would like (but, in fact, a sure way to guarantee that it won't
happen is for me to like it :^) to view iSCSI PDUs as the expected
grain size at which you might have software involvement in an
otherwise hardware-driven iSCSI implementation.

For example, when a target is returning read data for multiple
outstanding reads, it might want to return a bit at a time from each,
and each `bit' should be an iSCSI PDU.  Clearly sending these bits
will be a software-level decision.  That certainly was the rational
for allowing multiple FCP DATA PDUs per read operation, and I naively
assumed similar logic was being applied here.

The alternative is to say that the hardware will do iSCSI PDU
chunking, but if that's the case, I expect that the header is, well, a
bit bulky.

I'm also incredibly unexcited about having data length be a multiple
of 4 bytes (if that's still in the cards).  There operations within
the SCSI command set which return arbitrary length data.  There are
perfectly nominal cases where you get less data than you requested
(e.g. inquiry & request sense).  Furthermore, the SCSI architecture
does not prohibit this, even though certain commands do, so it is not
for iSCSI to say anything about this one way or the other.

The problem with handling lengths that include padding comes when
you're trying to move the data into a buffer which is a non-multiple
length.  For example, if I ask for 22 bytes of inquiry data (with a 22
byte buffer), what can I say, at the time a PDU arrives that has 24
bytes?  It might have 21 or 22 bytes (or perhaps even, erroneously 23
or 24 bytes).  A data residual coming later will tell the software how
much was actually there, but it can't tell the hardware.  The typical
expectation of this type of transfer is that it will only overwrite
bytes of the buffer that are actually transferred, but having padded
lengths will not allow this.

The completely standard solution to carrying arbitrary data of
arbitrary length in an aligned transfer unit is to pad the transfer
unit but report the exact (shorter) length.  Another solution, used by
FC, is to carry a pad length.  In iSCSI, why bother---you've just
reintroduced added the 2 bits you were trying to remove?

The data length scenario is not comparable to IP header lengths, where
what is being carried is not arbitrary data.

Certainly, for iSCSI additional header segments (AHSs) you could
arguably use this cell length technique, since we can control what
we're carrying (AHSs that need exact byte lengths will have to be
internally self-describing) but frankly, I still think it's a bad
idea.

I can't understand why we're messing around with all these tricky
`solutions' to standard problems.  We should avoid the temptation to
get cute, and wholesomely provide the same capabilities as any other
SCSI transport.  Specifically:
  o allow long PDUs
  o carry exact data lengths

Steph

---

From: [email-masked]
Date: Tue, 3 Apr 2001 12:19:41 +0200

This detail is already fixed in 06 -:) Julo

"Barry Reinhold" <[email-masked]> on 02/04/2001 20:59:29

Please respond to "Barry Reinhold" <[email-masked]>

To:   "Rod Harrison" <[email-masked]>, [email-masked]
cc:
Subject:  RE: iSCSI: frame formats

Rod,
     I'm pretty sure the we decided to go with bytes at the 50th IETF
meeting
for data length. No one felt that a max data size of 16 megs in an iSCSI
PDU
was an issue (remember this is not max SCSI data transfer size, it is max
data bytes in an iSCSI PDU).
     I do think we need to word the padding with a bit of care, as I can
envision people doing this in at least two different ways that would not
interoperate.
     The interoperability issues on the padding question boils down to:
"Where
in the TCP stream do I put the Data Digest if the number of bytes I have to
send is not a multiple of 4?"
     My expectation was that the transmitter would be null padding the data
portion of the iSCSI PDU to a word boundary, then sticking on the digest.
Thus the padding is actually in the TCP stream. For example if <d> = a data
octet, <p> = pad octet, and <dd> = data digest octet, then the TCP stream
for a 6 byte data transfer would look as follows:
<octets in the header - end modulo 4> <d> <d> <d> <d> <d> <d> <p> <p> <dd>
....
     The receiver would get the value "6" in the data length portion of the
header. After pulling out the header, the receiver would pull out 8 bytes
of
"data + pad" and then get the digest.
     The other way to do this to have a "virtual pad" such that padding is
created by the receiver when construction the iSCSI PDU from the TCP
stream.
The padding is never actually in the TCP stream itself.
     I do not think this is as helpful, but whatever we do, the spec.
should
address this minor detail so we don't trip over it.

>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Rod Harrison