Subject: Re: iSCSI: Towards a more effective PDU format

From: [email-masked]
Date: Tue, 13 Mar 2001 08:23:06 +0200

Bob,

Interesting.  This is close to one of the variants I had for IETF-50 (a
clear header).
The only adavantage it has is that you are able to read all the AHSs in one
read.
The (admitedly academic) disadvantage it has is that you are limited in the
size of extensions and have some redundancy.
The basic issue that was raised - and there is no simple way out - is that
once you have lost a block (BHS in your suggested layout) you are out of
synch.  There is no simple way around it (at least not one that can be
solved by changing layout) and an added digest (only the code or silicon to
account for it) is IMHO not warranted by what you gain.  If you are willing
to spend silicon or code on this the making header failures less probable
and recovering if you fail by dropping the connection is a better bet
(using the redundancy for a coding gain).  But this involves some
complexity too.

Julo

"Robert D. Russell" <[email-masked]> on 12/03/2001 22:01:39

Please respond to "Robert D. Russell" <[email-masked]>

To:   [email-masked]
cc:   [email-masked], "Ashish P. Palekar"
      <[email-masked]>, Anshul Chadda <[email-masked]>,
      Narendran Ganapathy <[email-masked]>
Subject:  Re: iSCSI: Towards a more effective PDU format

All:

Here is a proposal for an iSCSI PDU format without WN Next-Qualifiers:

1. Every PDU must start with a 48-byte Basic Header Segment (BHS).
   Assuming header digests are in use, this BHS is immediately
   followed by a header digest that covers it.

2. Every BHS contains 2 fields at fixed locations:
   a) AHS-length, containing the total length of all Additional Header
      Segments (AHS) that follow.  This field is 0 if there are no
      AHSs.
   b) DATA-length, containing the total length of all Data that follows
      the AHSs.  This field is 0 if there is no data.

3. If the AHS-length field in the BHS is non-zero, all the AHSs
   immediately follow the header digest of the BHS.  An additional
   digest immediately follows that last AHS and covers all the AHSs.

4. If the DATA-length field in the BHS is non-zero, all the data
   immediately follows the AHS digest.  A data digest immediately
   follows the data.

5. If space within the BHS is an issue, the AHS-length field could
   be restricted to a single byte, and could be in units of 4-byte
   words.  This would allow up to 1020 bytes of AHSs to follow a
   BHS, and this seems to be more than enough for the uses foreseen
   (so far, only extended CDB and bidi-read info).

6. Each AHS should start with a word containing a TYPE field and a
   LENGTH field.  The TYPE field should be enumerated rather than
   bit-field encoded, for easier decoding and future expansion.
   The LENGTH field is the number of bytes of additional information
   in this AHS that follows this word.

Advantages of this proposal.

1. The receiver always knows where to find digests in a reliable manner,
   without having to use unreliable data to find them.

2. Because the BHS is fixed length, the receiver can reliably obtain the
   entire BHS and its header digest in a single "read" operation.

3. Because the receiver gets the AHS-length field from the BHS, it can
   reliably obtain the entire set of AHSs and the header digest covering
   them in a single "read" operation.

4. Because the receiver gets the DATA-length field from the BHS, it can
   reliably  obtain all the data and the data digest covering it
   in a single "read" operation.

5. Since each AHS begins with a word containing the TYPE and LENGTH in
   fixed positions, an unknown TYPE (i.e., a type introduced in the
   future that is received by a legacy receiver) does NOT result in
   loss of synchronization during header processing -- the receiver
   knows the length of this AHS in any case, and can just skip over it.

Disadvantages of this proposal.

1. A second header digest is required, but only if one or more AHSs are
   present.  In practice this should be a rare event.

Proposed iSCSI PDU Format

     +-------------+
     | 48-byte BHS |
     +-------------+
     | BHS digest  |
   --+-------------+--
     |    AHS 1    |\
     | - - - - - - | \
     |    AHS 2    |  \
     | - - - - - - |   > total length in AHS_length field in BHS
     |    . . .    |  /
     | - - - - - - | /
     |    AHS n    |/
     +-------------+
     | AHSs digest |
   --+-------------+--
     |             |\
     |    Data     | > total length in DATA_length field in BHS
     |             |/
     +-------------+
     | Data digest |
     +-------------+

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

---

From: "Robert D. Russell" <[email-masked]>
Date: Tue, 13 Mar 2001 12:36:31 -0500

Julo:

Of course you are correct that once the BHS is lost, you are out of synch
and have to drop the connection.  However, there is more to this issue,
and that is how quickly you can detect the failure and start recovery.

In both my proposal and the current WN Next-Qualifier scheme,
a PDU that is shortened due to loss of data can lead to a situation
where the receiver blocks waiting for the rest of the PDU to
arrive before it can be processed.  The only way to detect this
would seem to be to put timers on every read, so the read does
not block indefinitely.  When the timer goes off, the connection
has to be dropped.

However, in the WN Next-Qualifier scheme, the receiver can also block
in situations where data is simply corrupted but not dropped, which is
probably a more common situation.  This will NOT happen in my proposal.

For example, suppose there is a PDU consisting of a header with
some of the extensions (extended CDB and/or bidi-read), but no
data.  Further, suppose the length field in one of the WN items gets
corrupted, so that its value becomes longer than it should be and
points off the end of the header and all its extensions.  The receiver
cannot know that this length has been corrupted, since it has not yet
gotten to the data digest, and will therefore block on a read waiting
for the rest of that WN item to arrive, but it never will.

This cannot happen in my proposal, because the total length of the AHSs
part of the PDU is given as the AHS_length field in the fixed-length
BHS, and if that length field is corrupted then the data digest for
the BHS will immediately detect that.  It is true that the response
will be to drop the connection, but it happens right away, without
blocking on a read until a timeout occurs.  Similarly, if the
AHS_length field in the BHS is ok, but the length field in one of
the AHS items itself is corrupted, that will also be known immediately
in my proposal, because the data digest at the end of the AHSs part
of the PDU is checked BEFORE using any of the length fields in the
AHS items themselves.

This points out the weakness of the WN Next-Qualifier scheme --
that the receiver has to use unreliable data to decode the header
EVEN IF A DIGEST IS USED.  This NEVER happens in my proposal.

Thanks for your consideration,

Bob Russell
InterOperability Lab

---

From: "Robert D. Russell" <[email-masked]>
Date: Thu, 15 Mar 2001 17:06:20 -0500

Julo:

Even if we give up on the idea of having a second header digest, I still
believe that some changes should to be made to the PDU Format proposed in
version 5 to make it simpler, more efficient to process, and more robust.

I would therefore propose eliminating the use of the current WN Next-Qualifier
scheme and replacing it as follows (this is just a slight modification of my
earlier proposal in order to eliminate the second header digest).

1. Every PDU must start with a 48-byte Basic Header (BHS) that can be read
   in a single "read" operation.  The format of this is identical to that
   proposed in section 2.2.4 of version 5, but with the addition of 2
   fields (4-bytes) as described next.

   a) AHS_length, containing the total length of all Additional Header
      Segments (AHS) that follow.  This field is 0 if there are no AHSs.

   b) DATA_length, containing the total length of all data that follows
      the AHSs.  This field is 0 if there is no data.

3. If the AHS_length field in the BHS is non-zero, all the AHSs immediately
   follow the BHS in the PDU.  The AHS_length value gives the total number of
   bytes in all the AHSs, allowing a single "read" operation to read them all.
   (The total header consists of 48+AHS_length bytes.)

4. If header digests are in use, one header digest covering the BHS and
   all the AHSs (if any) immediately follows the last AHS.  This digest
   does NOT require a separate read, since the first read (of the BHS)
   should be for 48+(length-of-header-digest) bytes.  If the AHS_length
   field is zero, there are no more reads.  If the AHS_length field is
   non-zero, exactly that many more bytes need to be read in a second read.
   These additional bytes are appended to those obtained in the first read.
   The header digest will always be the last (length-of-header-digest) bytes.

4. If the DATA-length field in the BHS is non-zero, all the data
   immediately follows the header digest.  If data digests are in use,
   a data digest immediately follows the data.

5. To save space within the BHS, the AHS-length field can be restricted to a
   single byte, and can be in units of 4-byte words.  This allows up to 1020
   bytes of AHSs to follow a BHS, which seems to be more than enough for the
   uses foreseen (so far, only extended CDB and bidi-read info).  Currently,
   byte 2 is unused (reserved) in all PDU types except Login and Login
   response, where byte 6 is unused.  Therefore, the AHS-length field can
   be added without increasing the BHS size of 48 bytes.

7. The DATA_length field should be a 4-byte field added to the 44-byte
   BHS of the current section 2.2.4 to give a new BHS of 48-bytes.
   However, since the version 5 44-byte header is always preceded by
   a 4-byte WN Next-Qualifier that would no longer be needed, there is
   no net change in the effective BHS size of 48-bytes.

Advantages of this proposal over the current WN Next-Qualifer of version 5.

1. Because the receiver gets the AHS-length field from the BHS, it can obtain
   the entire set of ALL AHSs that follow the BHS in a single "read" operation
   (the current WN Next-Qualifier scheme requires a separate "read" for EACH
   additional header segment after the BHS).

2. By limiting the total size of all AHSs to 1020 bytes, a receiver can
   preallocate a fixed-length buffer of (48 + 1020 + size-of-header_digest)
   bytes for header processing (the current WN Next-Qualifier scheme has
   no limit on either total header size nor individual AHS sizes).

3. Since each AHS begins with a word containing the TYPE and LENGTH in
   fixed positions, an unknown TYPE (i.e., a type introduced in the
   future that is received by a legacy receiver) does NOT result in
   loss of synchronization during header processing -- the receiver
   knows the length of this AHS in any case, and can just skip over it
   (the current WN Next-Qualifier type determines how the length field
   is to be interpreted -- there is no "general rule", so unknown types
   mean loss of synchronization).

4. The format of the AHS is the familiar Type/Length/Value (T/L/V) structure
   (the current WN Next-Qualifier is T+1/L+1/V, but this does not seem to
   provide any benefit and only adds confusion and complexity -- see e-mails
   from David Black and Barry Reinhold).

5. The AHS TYPE is enumerated (the current WN Next-Qualifier is bit encoded,
   again without seeming to provide any benefit -- see e-mail from David Black).

6. Because the BHS always contains a 4-byte DATA_length field, the maximum data
   segment size is 4 Gigabytes (the current WN Next-Qualifier scheme, with
   the long data header removed, limits the data segment size to 16 Megabytes).

7. The "Header Digest Present" and "Data Digest Present" bits have been
   completely eliminated (the purpose of these in the current WN Next-Qualifier
   scheme was never explained, but they appear to be useless -- the presence
   or absence of digests has to be negotiated, and once negotiated, all
   PDUs must obey the negotiated decision).

I see no disadvantages of this proposal relative to the current WN scheme.

Since there is only one header digest, both schemes suffer the disadvantage
of using unreliable data to find the header digest, which can lead to
unnecessary blocking on "reads".

     +------------------------+
     |     required BHS       | > fixed length of 48 bytes
     +------------------------+
     |     optional AHS 1     |\
     | - - - - - - - - - - -  | \
     |     optional AHS 2     |  \
     | - - - - - - - - - - -  |   > total length in AHS_length field in BHS
     |        . . . .         |  /
     | - - - - - - - - - - -  | /
     |     optional AHS n     |/
     +------------------------+
     | optional header digest | -- covers preceding (48 + AHS_length) bytes
     +------------------------+
     |                        |\
     |     optional data      | > total length in DATA_length field in BHS
     |                        |/
     +------------------------+
     |  optional data digest  | -- covers preceding (DATA_length) bytes
     +------------------------+

Thanks,

Bob Russell
InterOperability Lab

---

From: "Rod Harrison" <[email-masked]>
Date: Fri, 16 Mar 2001 21:42:00 -0000

I rather like the new header format proposed here, although
I would prefer to see a separate digest for the AHS
segments. I like that the new format allows the next BHS to
be found if anything except the current BHS is damaged,
which is not the case with the current format since a
damaged AHS length causes loss of sync. If a BHS is damaged
I think recovery is going to be expensive and including more
bytes in the digest covering the BHS increases the
likelihood of having to perform that recovery, even though
the BHS itself might not be damaged. How expensive is it to
add the second digest covering all the AHS segments? I'm
guessing that the second digest could be calculated using
the first as a starting point. I don't see a down side of
mandating that any AHS digest will be of the same type as
the BHS digest.

	On a related point, if a BHS is damaged I would prefer to
see the connection immediately dumped rather than any
re-sync activity performed. I don't want to get into the
debate about the merits versus risks of re-syncing but I
think the fact that it has been talked about for so long
shows that the risks are not obvious. Either way I think
there is a more fundamental reason to prefer connection
dumping. In order to re-sync there must be a data stream, if
a BHS is damaged during a period of light activity there may
be a significant delay before any more data is sent. In the
limit the next PDU may be a retry of the damaged PDU by the
iSCSI or host SCSI layers. Those timeouts are going to be in
seconds or more likely tens of seconds, whereas dumping the
connection will immediately cause the recovery to begin. The
time to make a new connection will be significantly shorter
than the worst case re-sync time. Further, unless we can
guarantee that re-syncing will always succeed there will
have to be a point where we give up and dump the connection
anyway. Also, even if we do manage to re-sync we still incur
a timeout for the damaged PDU to be resent. Again, if the
connection is dumped that timeout doesn't need to happen.

	I should point out that when I say dumped here I mean a
graceful TCP close.

	- Rod Harrison

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On
Behalf Of
Robert D. Russell
Sent: Thursday, March 15, 2001 10:06 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: iSCSI: Towards a more effective PDU format

Julo:

Even if we give up on the idea of having a second header
digest, I still
believe that some changes should to be made to the PDU
Format proposed in
version 5 to make it simpler, more efficient to process, and
more robust.

I would therefore propose eliminating the use of the current
WN Next-Qualifier
scheme and replacing it as follows (this is just a slight
modification of my
earlier proposal in order to eliminate the second header
digest).

1. Every PDU must start with a 48-byte Basic Header (BHS)
that can be read
   in a single "read" operation.  The format of this is
identical to that
   proposed in section 2.2.4 of version 5, but with the
addition of 2
   fields (4-bytes) as described next.

   a) AHS_length, containing the total length of all
Additional Header
      Segments (AHS) that follow.  This field is 0 if there
are no AHSs.

   b) DATA_length, containing the total length of all data
that follows
      the AHSs.  This field is 0 if there is no data.

3. If the AHS_length field in the BHS is non-zero, all the
AHSs immediately
   follow the BHS in the PDU.  The AHS_length value gives
the total number of
   bytes in all the AHSs, allowing a single "read" operation
to read them all.
   (The total header consists of 48+AHS_length bytes.)

4. If header digests are in use, one header digest covering
the BHS and
   all the AHSs (if any) immediately follows the last AHS.
This digest
   does NOT require a separate read, since the first read
(of the BHS)
   should be for 48+(length-of-header-digest) bytes.  If the
AHS_length
   field is zero, there are no more reads.  If the
AHS_length field is
   non-zero, exactly that many more bytes need to be read in
a second read.
   These additional bytes are appended to those obtained in
the first read.
   The header digest will always be the last
(length-of-header-digest) bytes.

4. If the DATA-length field in the BHS is non-zero, all the
data
   immediately follows the header digest.  If data digests
are in use,
   a data digest immediately follows the data.

5. To save space within the BHS, the AHS-length field can be
restricted to a
   single byte, and can be in units of 4-byte words.  This
allows up to 1020
   bytes of AHSs to follow a BHS, which seems to be more
than enough for the
   uses foreseen (so far, only extended CDB and bidi-read
info).  Currently,
   byte 2 is unused (reserved) in all PDU types except Login
and Login
   response, where byte 6 is unused.  Therefore, the
AHS-length field can
   be added without increasing the BHS size of 48 bytes.

6. Each AHS should start with a word containing a TYPE field
and a
   LENGTH field.  The TYPE field should be enumerated rather
than
   bit-field encoded, for easier decoding and future
expansion.
   The LENGTH field is the number of bytes of additional
information
   in this AHS that follows this word.

7. The DATA_length field should be a 4-byte field added to
the 44-byte
   BHS of the current section 2.2.4 to give a new BHS of
48-bytes.
   However, since the version 5 44-byte header is always
preceded by
   a 4-byte WN Next-Qualifier that would no longer be
needed, there is
   no net change in the effective BHS size of 48-bytes.

Advantages of this proposal over the current WN
Next-Qualifer of version 5.

1. Because the receiver gets the AHS-length field from the
BHS, it can obtain
   the entire set of ALL AHSs that follow the BHS in a
single "read" operation
   (the current WN Next-Qualifier scheme requires a separate
"read" for EACH
   additional header segment after the BHS).

2. By limiting the total size of all AHSs to 1020 bytes, a
receiver can
   preallocate a fixed-length buffer of (48 + 1020 +
size-of-header_digest)
   bytes for header processing (the current WN
Next-Qualifier scheme has
   no limit on either total header size nor individual AHS
sizes).

3. Since each AHS begins with a word containing the TYPE and
LENGTH in
   fixed positions, an unknown TYPE (i.e., a type introduced
in the
   future that is received by a legacy receiver) does NOT
result in
   loss of synchronization during header processing -- the
receiver
   knows the length of this AHS in any case, and can just
skip over it
   (the current WN Next-Qualifier type determines how the
length field
   is to be interpreted -- there is no "general rule", so
unknown types
   mean loss of synchronization).

4. The format of the AHS is the familiar Type/Length/Value
(T/L/V) structure
   (the current WN Next-Qualifier is T+1/L+1/V, but this
does not seem to
   provide any benefit and only adds confusion and
complexity -- see e-mails
   from David Black and Barry Reinhold).

5. The AHS TYPE is enumerated (the current WN Next-Qualifier
is bit encoded,
   again without seeming to provide any benefit -- see
e-mail from David Black).

6. Because the BHS always contains a 4-byte DATA_length
field, the maximum data
   segment size is 4 Gigabytes (the current WN
Next-Qualifier scheme, with
   the long data header removed, limits the data segment
size to 16 Megabytes).

7. The "Header Digest Present" and "Data Digest Present"
bits have been
   completely eliminated (the purpose of these in the
current WN Next-Qualifier
   scheme was never explained, but they appear to be
useless -- the presence
   or absence of digests has to be negotiated, and once
negotiated, all
   PDUs must obey the negotiated decision).

I see no disadvantages of this proposal relative to the
current WN scheme.

Since there is only one header digest, both schemes suffer
the disadvantage
of using unreliable data to find the header digest, which
can lead to
unnecessary blocking on "reads".

     +------------------------+
     |     required BHS       | > fixed length of 48 bytes
     +------------------------+
     |     optional AHS 1     |\
     | - - - - - - - - - - -  | \
     |     optional AHS 2     |  \
     | - - - - - - - - - - -  |   > total length in
AHS_length field in BHS
     |        . . . .         |  /
     | - - - - - - - - - - -  | /
     |     optional AHS n     |/
     +------------------------+
     | optional header digest | -- covers preceding (48 +
AHS_length) bytes
     +------------------------+
     |                        |\
     |     optional data      | > total length in
DATA_length field in BHS
     |                        |/
     +------------------------+
     |  optional data digest  | -- covers preceding
(DATA_length) bytes
     +------------------------+

Thanks,

Bob Russell
InterOperability Lab