Subject: Re: frame formats

From: [email-masked] (Sandeep Joshi)
Date: Sun, 1 Apr 2001 15:19:37 -0400 (EDT)

To steal & reintroduce that idea, is it possible to borrow 
a bit or two from the DataLen field to parity-check the AHSLen ?   

Adding an entire BHS header digest, when all you need to verify 
is that AHS-length field, seemed overkill.   On the other hand,
Format-1's chief disadvantage was that it required some processing 
(QL) to locate either/both the length fields.   This has complicated
the choice between all these 3 formats resulting in the current vote.

Borrowing the bits for AHS parity from DataLen solves this problem.

DataLen will now be max 8M/4M but then we dont wish to have large 
iSCSI PDUs in any case.  Btw, I assume Next DataLen AHS does not 
exist in the new setup ?

-Sandeep

> Well, perhaps I was just not quick enough.  I thought I would let this
> settle out a bit before I added my two cents.
>  
> If you all remember, some folks on this reflector gave Julian a hard time
> because you would have to use a length field that you were not sure was OK,
> if you had a digest error and wanted to jump forward to the next, etc. etc.
> etc.  I am sure you all remember this.  OK, now that Julian proposed a
> parity way to ensure that you could trust the length field, some of the
> parties, have now, I think, voted for format #2.  Unless you want now to
> reconsider your vote, we should stop giving Julian a hard time about the
> length not being ensured correct in the presents of a Digest Error.
>  
> Either drop the session, or use the length to see if you can get somewhere,
> search for the next marker etc.  All the stuff you said you did not like
> before.  OK, now you have format 2, but lets not go over that old ground
> now that you have decided against the parity.
> 
> 
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Internet address: [email-masked]
>  
>  
> Julian Satran/Haifa/IBM@[email-masked] on 03/30/2001 08:51:50 AM
>  
> Sent by:  [email-masked]
>  
>  
> To:   [email-masked]
> cc:
> Subject:  frame formats
>  
>  
>  
>  
>  
> Dear colleagues,
>  
> It look like Format-2 is selected by popular vote.
>  
> Julo
>  
>  
>

---

From: Stephen Bailey <[email-masked]>
Date: Mon, 02 Apr 2001 10:17:33 -0400

Sandeep,

> iSCSI PDUs in any case.

This max size is getting the the point where I'm sure it'll be an
irritant.

I would like (but, in fact, a sure way to guarantee that it won't
happen is for me to like it :^) to view iSCSI PDUs as the expected
grain size at which you might have software involvement in an
otherwise hardware-driven iSCSI implementation.

For example, when a target is returning read data for multiple
outstanding reads, it might want to return a bit at a time from each,
and each `bit' should be an iSCSI PDU.  Clearly sending these bits
will be a software-level decision.  That certainly was the rational
for allowing multiple FCP DATA PDUs per read operation, and I naively
assumed similar logic was being applied here.

The alternative is to say that the hardware will do iSCSI PDU
chunking, but if that's the case, I expect that the header is, well, a
bit bulky.

I'm also incredibly unexcited about having data length be a multiple
of 4 bytes (if that's still in the cards).  There operations within
the SCSI command set which return arbitrary length data.  There are
perfectly nominal cases where you get less data than you requested
(e.g. inquiry & request sense).  Furthermore, the SCSI architecture
does not prohibit this, even though certain commands do, so it is not
for iSCSI to say anything about this one way or the other.

The problem with handling lengths that include padding comes when
you're trying to move the data into a buffer which is a non-multiple
length.  For example, if I ask for 22 bytes of inquiry data (with a 22
byte buffer), what can I say, at the time a PDU arrives that has 24
bytes?  It might have 21 or 22 bytes (or perhaps even, erroneously 23
or 24 bytes).  A data residual coming later will tell the software how
much was actually there, but it can't tell the hardware.  The typical
expectation of this type of transfer is that it will only overwrite
bytes of the buffer that are actually transferred, but having padded
lengths will not allow this.

The completely standard solution to carrying arbitrary data of
arbitrary length in an aligned transfer unit is to pad the transfer
unit but report the exact (shorter) length.  Another solution, used by
FC, is to carry a pad length.  In iSCSI, why bother---you've just
reintroduced added the 2 bits you were trying to remove?

The data length scenario is not comparable to IP header lengths, where
what is being carried is not arbitrary data.

Certainly, for iSCSI additional header segments (AHSs) you could
arguably use this cell length technique, since we can control what
we're carrying (AHSs that need exact byte lengths will have to be
internally self-describing) but frankly, I still think it's a bad
idea.

I can't understand why we're messing around with all these tricky
`solutions' to standard problems.  We should avoid the temptation to
get cute, and wholesomely provide the same capabilities as any other
SCSI transport.  Specifically:
  o allow long PDUs
  o carry exact data lengths

Steph

---

From: "Barry Reinhold" <[email-masked]>
Date: Mon, 2 Apr 2001 14:59:29 -0400

Rod,
	I'm pretty sure the we decided to go with bytes at the 50th IETF meeting
for data length. No one felt that a max data size of 16 megs in an iSCSI PDU
was an issue (remember this is not max SCSI data transfer size, it is max
data bytes in an iSCSI PDU).
	I do think we need to word the padding with a bit of care, as I can
envision people doing this in at least two different ways that would not
interoperate.
	The interoperability issues on the padding question boils down to: "Where
in the TCP stream do I put the Data Digest if the number of bytes I have to
send is not a multiple of 4?"
	My expectation was that the transmitter would be null padding the data
portion of the iSCSI PDU to a word boundary, then sticking on the digest.
Thus the padding is actually in the TCP stream. For example if <d> = a data
octet, <p> = pad octet, and <dd> = data digest octet, then the TCP stream
for a 6 byte data transfer would look as follows:
<octets in the header - end modulo 4> <d> <d> <d> <d> <d> <d> <p> <p> <dd>
....
	The receiver would get the value "6" in the data length portion of the
header. After pulling out the header, the receiver would pull out 8 bytes of
"data + pad" and then get the digest.
	The other way to do this to have a "virtual pad" such that padding is
created by the receiver when construction the iSCSI PDU from the TCP stream.
The padding is never actually in the TCP stream itself.
	I do not think this is as helpful, but whatever we do, the spec. should
address this minor detail so we don't trip over it.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Rod Harrison
>Sent: Monday, April 02, 2001 1:44 PM
>To: [email-masked]
>Subject: RE: iSCSI: frame formats
>
>
>Stephen,
>
>	I don't agree with your concerns about the maximum PDU
>size. I think few targets, or indeed initiators, will be
>interested in negotiating PDU sizes anywhere near this
>large, so big transfers will have to be fragmented anyway.
>
>	However, I share your concern about the padding. I don't
>really see why we are considering it. If one has a local
>alignment issue it can, and I believe should, be taken care
>of locally and not in the specification. There are several
>easy ways of handling this sort of thing; insert and remove
>the pad locally; separate header and payload buffers
>allowing each to be naturally aligned, etc.
>
>	If we have to pad then every read of non-header data will
>have to involve a rounding calculation on the length, and
>then perhaps a second read to discard the pad if the
>underlying buffer is the exact size of the data. Possibly
>the same on send, if the data buffer is the exact size the
>transmit code can't just 'go off the end,' it will have to
>send the data, and then fake up some pad and make another
>send.
>
>	Am I missing something here, why do we care about padding?
>
>	- Rod
>
>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On
>Behalf Of
>Stephen Bailey
>Sent: Monday, April 02, 2001 3:18 PM
>To: [email-masked]
>Subject: Re: iSCSI: frame formats
>
>
>Sandeep,
>

---

From: "Rod Harrison" <[email-masked]>
Date: Mon, 2 Apr 2001 18:43:56 +0100

Stephen,

	- Rod

-----Original Message-----

Sandeep,

> DataLen will now be max 8M/4M but then we dont wish to
have large

This max size is getting the the point where I'm sure it'll
be an
irritant.

I would like (but, in fact, a sure way to guarantee that it
won't
happen is for me to like it :^) to view iSCSI PDUs as the
expected
grain size at which you might have software involvement in
an
otherwise hardware-driven iSCSI implementation.

For example, when a target is returning read data for
multiple
outstanding reads, it might want to return a bit at a time
from each,
and each `bit' should be an iSCSI PDU.  Clearly sending
these bits
will be a software-level decision.  That certainly was the
rational
for allowing multiple FCP DATA PDUs per read operation, and
I naively
assumed similar logic was being applied here.

The alternative is to say that the hardware will do iSCSI
PDU
chunking, but if that's the case, I expect that the header
is, well, a
bit bulky.

I'm also incredibly unexcited about having data length be a
multiple
of 4 bytes (if that's still in the cards).  There operations
within
the SCSI command set which return arbitrary length data.
There are
perfectly nominal cases where you get less data than you
requested
(e.g. inquiry & request sense).  Furthermore, the SCSI
architecture
does not prohibit this, even though certain commands do, so
it is not
for iSCSI to say anything about this one way or the other.

The problem with handling lengths that include padding comes
when
you're trying to move the data into a buffer which is a
non-multiple
length.  For example, if I ask for 22 bytes of inquiry data
(with a 22
byte buffer), what can I say, at the time a PDU arrives that
has 24
bytes?  It might have 21 or 22 bytes (or perhaps even,
erroneously 23
or 24 bytes).  A data residual coming later will tell the
software how
much was actually there, but it can't tell the hardware.
The typical
expectation of this type of transfer is that it will only
overwrite
bytes of the buffer that are actually transferred, but
having padded
lengths will not allow this.

The completely standard solution to carrying arbitrary data
of
arbitrary length in an aligned transfer unit is to pad the
transfer
unit but report the exact (shorter) length.  Another
solution, used by
FC, is to carry a pad length.  In iSCSI, why bother---you've
just
reintroduced added the 2 bits you were trying to remove?

The data length scenario is not comparable to IP header
lengths, where
what is being carried is not arbitrary data.

Certainly, for iSCSI additional header segments (AHSs) you
could
arguably use this cell length technique, since we can
control what
we're carrying (AHSs that need exact byte lengths will have
to be
internally self-describing) but frankly, I still think it's
a bad
idea.

I can't understand why we're messing around with all these
tricky
`solutions' to standard problems.  We should avoid the
temptation to
get cute, and wholesomely provide the same capabilities as
any other
SCSI transport.  Specifically:
  o allow long PDUs
  o carry exact data lengths

Steph

---

From: Sandeep Joshi <[email-masked]>
Date: Mon, 02 Apr 2001 13:21:38 -0400

comments in text..

-Sandeep

> This max size is getting the the point where I'm sure it'll be an

Your point about maxSize limitations set me thinking..

Hardware folks dont like parsing down the AHS to find the length 
field so perhaps putting jumbo-lengths in AHS is unattractive.

So what is the level of HW complexity in processing the SCSI PDUs which 
already have 6/10/12/16 byte commands, and consequently put the 
length field in *variable* locations ?   Can iSCSI adopt a similar
scheme to future-proof it from maxSize limitations ?

Roughly, a few extra states get added into the FSM to process variable 
length BHS based on the opcode, and you may need an extra register or
two.
But in hardware this can still get done in parallel.  Any thoughts ?

Needless to say, a SCSI-like variable BHS kills this 2-reads thingy that 
software implementors have been talking about.

> for iSCSI to say anything about this one way or the other.

If I am not mistaken, dataLen value will be exact (not the padded)
and it will be in units of bytes (not words - this mistake was 
pointed out by someone at the IETF).

---

From: Stephen Bailey <[email-masked]>
Date: Mon, 09 Apr 2001 12:24:09 -0400

> No one felt that a max data size of 16 megs in an iSCSI PDU was an
> issue

I did, I was just exhausted.  Furthermore, the size the PDU could
handle shrank (from 64M to 16M) right there before our ears in a split
second.

The `question' was also not asked in a form which I could effectively
answer or discuss.  There were two proposals presented, and they had
very different characteristics.  I figured I would refrain from
comment until I a) understood the proposals better b) got a sense of
which way we might be going.

I have a hard time swallowing a PDU that won't accomodate the data
from even a READ10 or WRITE10.

If people expect to tile each TCP segment with an iSCSI PDU, I think
that's a mistake.  The headers are too bulky for that.  If they think
they're going to gain something meaningful in hardware implementation
complexity (e.g. bounding the amount of reassembly buffering) by
assuming small multiples of segment size PDUs, I think that's a
mistake too.  I'd like to see a show of hands of people who have
actually implemented this approach in hardware.  Anybody?

If you (eventually) accept this, then you're back to the model that
there's a lower layer which is providing your data steering on a
per-segment basis, and your iSCSI PDU is the granularity at which the
iSCSI implementation delivers data to this layer.  It is clearly a
granularity at which software (transfer scheduling) occurs. 

At 10 Gbit/s, a 16 MB PDU is only 16 MS worth of data.

Steph