Subject: iSCSI: Towards a more effective PDU format

From: "Robert D. Russell" <[email-masked]>
Date: Mon, 12 Mar 2001 11:40:01 -0500

All,

Some people are objecting to having 2 (there is no need for SEVERAL,
just 2) header digests.  But to have only 1 header digest for
variable length headers leads to the following problem:

If the header is variable length, how does the receiver find the
header digest WITHOUT using the header data that is supposed to
be protected by the digest (since that data might be unreliable)?

The only answer I see is that the receiver MUST know where the digest
is located in the PDU (WITHOUT looking at the header data first).

And the only way that can be done is to put the digest at a FIXED
offset from the start of the PDU.

>From this it follows that the digest cannot be at the end of a
variable-length header -- the receiver will NOT know where to find it!

However, if the digest is NOT at the end of the header, then 2
possibilities arise:

1) If there is only one digest for ALL the header data, then some
   of that data must FOLLOW the digest.

2) If a digest must always follow the data it protects, then
   there has to be a second digest that follows and protects the
   data not covered by the first digest.

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

---

From: "Robert D. Russell" <[email-masked]>
Date: Mon, 12 Mar 2001 15:01:39 -0500

All:

Here is a proposal for an iSCSI PDU format without WN Next-Qualifiers:

1. Every PDU must start with a 48-byte Basic Header Segment (BHS).
   Assuming header digests are in use, this BHS is immediately
   followed by a header digest that covers it.

2. Every BHS contains 2 fields at fixed locations:
   a) AHS-length, containing the total length of all Additional Header
      Segments (AHS) that follow.  This field is 0 if there are no
      AHSs.
   b) DATA-length, containing the total length of all Data that follows
      the AHSs.  This field is 0 if there is no data.

3. If the AHS-length field in the BHS is non-zero, all the AHSs
   immediately follow the header digest of the BHS.  An additional
   digest immediately follows that last AHS and covers all the AHSs.

4. If the DATA-length field in the BHS is non-zero, all the data
   immediately follows the AHS digest.  A data digest immediately
   follows the data.

5. If space within the BHS is an issue, the AHS-length field could
   be restricted to a single byte, and could be in units of 4-byte
   words.  This would allow up to 1020 bytes of AHSs to follow a
   BHS, and this seems to be more than enough for the uses foreseen
   (so far, only extended CDB and bidi-read info).

6. Each AHS should start with a word containing a TYPE field and a
   LENGTH field.  The TYPE field should be enumerated rather than
   bit-field encoded, for easier decoding and future expansion.
   The LENGTH field is the number of bytes of additional information
   in this AHS that follows this word.

Advantages of this proposal.

1. The receiver always knows where to find digests in a reliable manner,
   without having to use unreliable data to find them.

2. Because the BHS is fixed length, the receiver can reliably obtain the
   entire BHS and its header digest in a single "read" operation.

3. Because the receiver gets the AHS-length field from the BHS, it can
   reliably obtain the entire set of AHSs and the header digest covering
   them in a single "read" operation.

4. Because the receiver gets the DATA-length field from the BHS, it can
   reliably  obtain all the data and the data digest covering it
   in a single "read" operation.

5. Since each AHS begins with a word containing the TYPE and LENGTH in
   fixed positions, an unknown TYPE (i.e., a type introduced in the
   future that is received by a legacy receiver) does NOT result in
   loss of synchronization during header processing -- the receiver
   knows the length of this AHS in any case, and can just skip over it.

Disadvantages of this proposal.

1. A second header digest is required, but only if one or more AHSs are
   present.  In practice this should be a rare event.

Proposed iSCSI PDU Format

     +-------------+
     | 48-byte BHS |
     +-------------+
     | BHS digest  |
   --+-------------+--
     |    AHS 1    |\
     | - - - - - - | \
     |    AHS 2    |  \
     | - - - - - - |   > total length in AHS_length field in BHS
     |    . . .    |  /
     | - - - - - - | /
     |    AHS n    |/
     +-------------+
     | AHSs digest |
   --+-------------+--
     |             |\
     |    Data     | > total length in DATA_length field in BHS
     |             |/
     +-------------+
     | Data digest |
     +-------------+

Bob Russell
InterOperability Lab

---

From: [email-masked]
Date: Sat, 17 Mar 2001 10:31:10 +0200

Bob,

I will present at Minneapolis a survey of possibilities. Both your and
Barry Reynhold's proposal will be listed. You will have a chance to  say
what you have to say.

I have also a modified structure that enables one digest and a total AHSs
length (as I mentioned earlier).

I am not ignoring your proposal but I have a flight ahead and some more
things to do during my remaining weekend hours.

Regards,
Julo

"Robert D. Russell" <[email-masked]> on 16/03/2001 00:06:20

Please respond to "Robert D. Russell" <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   [email-masked]
Subject:  Re: iSCSI: Towards a more effective PDU format

Julo:

Even if we give up on the idea of having a second header digest, I still
believe that some changes should to be made to the PDU Format proposed in
version 5 to make it simpler, more efficient to process, and more robust.

I would therefore propose eliminating the use of the current WN
Next-Qualifier
scheme and replacing it as follows (this is just a slight modification of
my
earlier proposal in order to eliminate the second header digest).

1. Every PDU must start with a 48-byte Basic Header (BHS) that can be read
   in a single "read" operation.  The format of this is identical to that
   proposed in section 2.2.4 of version 5, but with the addition of 2
   fields (4-bytes) as described next.

   a) AHS_length, containing the total length of all Additional Header
      Segments (AHS) that follow.  This field is 0 if there are no AHSs.

   b) DATA_length, containing the total length of all data that follows
      the AHSs.  This field is 0 if there is no data.

3. If the AHS_length field in the BHS is non-zero, all the AHSs immediately
   follow the BHS in the PDU.  The AHS_length value gives the total number
of
   bytes in all the AHSs, allowing a single "read" operation to read them
all.
   (The total header consists of 48+AHS_length bytes.)

4. If header digests are in use, one header digest covering the BHS and
   all the AHSs (if any) immediately follows the last AHS.  This digest
   does NOT require a separate read, since the first read (of the BHS)
   should be for 48+(length-of-header-digest) bytes.  If the AHS_length
   field is zero, there are no more reads.  If the AHS_length field is
   non-zero, exactly that many more bytes need to be read in a second read.
   These additional bytes are appended to those obtained in the first read.
   The header digest will always be the last (length-of-header-digest)
bytes.

4. If the DATA-length field in the BHS is non-zero, all the data
   immediately follows the header digest.  If data digests are in use,
   a data digest immediately follows the data.

5. To save space within the BHS, the AHS-length field can be restricted to
a
   single byte, and can be in units of 4-byte words.  This allows up to
1020
   bytes of AHSs to follow a BHS, which seems to be more than enough for
the
   uses foreseen (so far, only extended CDB and bidi-read info).
Currently,
   byte 2 is unused (reserved) in all PDU types except Login and Login
   response, where byte 6 is unused.  Therefore, the AHS-length field can
   be added without increasing the BHS size of 48 bytes.

7. The DATA_length field should be a 4-byte field added to the 44-byte
   BHS of the current section 2.2.4 to give a new BHS of 48-bytes.
   However, since the version 5 44-byte header is always preceded by
   a 4-byte WN Next-Qualifier that would no longer be needed, there is
   no net change in the effective BHS size of 48-bytes.

Advantages of this proposal over the current WN Next-Qualifer of version 5.

1. Because the receiver gets the AHS-length field from the BHS, it can
obtain
   the entire set of ALL AHSs that follow the BHS in a single "read"
operation
   (the current WN Next-Qualifier scheme requires a separate "read" for
EACH
   additional header segment after the BHS).

2. By limiting the total size of all AHSs to 1020 bytes, a receiver can
   preallocate a fixed-length buffer of (48 + 1020 + size-of-header_digest)
   bytes for header processing (the current WN Next-Qualifier scheme has
   no limit on either total header size nor individual AHS sizes).

3. Since each AHS begins with a word containing the TYPE and LENGTH in
   fixed positions, an unknown TYPE (i.e., a type introduced in the
   future that is received by a legacy receiver) does NOT result in
   loss of synchronization during header processing -- the receiver
   knows the length of this AHS in any case, and can just skip over it
   (the current WN Next-Qualifier type determines how the length field
   is to be interpreted -- there is no "general rule", so unknown types
   mean loss of synchronization).

4. The format of the AHS is the familiar Type/Length/Value (T/L/V)
structure
   (the current WN Next-Qualifier is T+1/L+1/V, but this does not seem to
   provide any benefit and only adds confusion and complexity -- see
e-mails
   from David Black and Barry Reinhold).

5. The AHS TYPE is enumerated (the current WN Next-Qualifier is bit
encoded,
   again without seeming to provide any benefit -- see e-mail from David
Black).

6. Because the BHS always contains a 4-byte DATA_length field, the maximum
data
   segment size is 4 Gigabytes (the current WN Next-Qualifier scheme, with
   the long data header removed, limits the data segment size to 16
Megabytes).

7. The "Header Digest Present" and "Data Digest Present" bits have been
   completely eliminated (the purpose of these in the current WN
Next-Qualifier
   scheme was never explained, but they appear to be useless -- the
presence
   or absence of digests has to be negotiated, and once negotiated, all
   PDUs must obey the negotiated decision).

I see no disadvantages of this proposal relative to the current WN scheme.

Since there is only one header digest, both schemes suffer the disadvantage
of using unreliable data to find the header digest, which can lead to
unnecessary blocking on "reads".

     +------------------------+
     |     required BHS       | > fixed length of 48 bytes
     +------------------------+
     |     optional AHS 1     |\
     | - - - - - - - - - - -  | \
     |     optional AHS 2     |  \
     | - - - - - - - - - - -  |   > total length in AHS_length field in BHS
     |        . . . .         |  /
     | - - - - - - - - - - -  | /
     |     optional AHS n     |/
     +------------------------+
     | optional header digest | -- covers preceding (48 + AHS_length) bytes
     +------------------------+
     |                        |\
     |     optional data      | > total length in DATA_length field in BHS
     |                        |/
     +------------------------+
     |  optional data digest  | -- covers preceding (DATA_length) bytes
     +------------------------+

Thanks,

Bob Russell
InterOperability Lab

On Tue, 13 Mar 2001 [email-masked] wrote:

>
>
> Bob,
>
> Interesting.  This is close to one of the variants I had for IETF-50 (a
> clear header).
> The only adavantage it has is that you are able to read all the AHSs in
one
> read.
> The (admitedly academic) disadvantage it has is that you are limited in
the
> size of extensions and have some redundancy.
> The basic issue that was raised - and there is no simple way out - is
that
> once you have lost a block (BHS in your suggested layout) you are out of
> synch.  There is no simple way around it (at least not one that can be
> solved by changing layout) and an added digest (only the code or silicon
to
> account for it) is IMHO not warranted by what you gain.  If you are
willing
> to spend silicon or code on this the making header failures less probable
> and recovering if you fail by dropping the connection is a better bet
> (using the redundancy for a coding gain).  But this involves some
> complexity too.
>
> Julo
>