Subject: Re: iSCSI: Towards a more effective PDU format

From: [email-masked]
Date: Sat, 10 Mar 2001 19:16:33 +0200

Barry,

Thanks for outline.
I will consider it as one of the variants I am going through to asses
clarity.

There are two things that are against it:

1- several digests
2- if I don't gather (how?) from BHS that TLVs are there then I have to do
2 reads for each additional segment - and besides this you are using a
field before knowing it is correct.

We could however rearrange in any fashion the BHS and NWs to make it look
very close to what you propose and still have one read per segment and data
used only after checking.

Regards,
Julo

"Barry Reinhold - Lamprey POP3" <[email-masked]> on 10/03/2001
04:02:28

Please respond to "Barry Reinhold - Lamprey POP3" <[email-masked]>

To:   "ISCSI" <[email-masked]>
cc:   "Robert Russell" <[email-masked]>, "Ashish A. Palekar"
      <[email-masked]>, "Jon Sreekanth" <[email-masked]>,
      "James Smart" <[email-masked]>
Subject:  iSCSI: Towards a more effective PDU format

All,

I know there are well defined feelings about how a PDU should be
structured,
and not everyone is going to be in the same camp on this issue. However, I
am concerned that the current PDU format with the WN field has not really
achieved the goals that I, for one, would like to see it achieve. So, let
me
list a few objectives for the PDU format and then suggest one possible
solution.

Objectives for the iSCSI PDU (that could be improved upon).

1. Simplicity - The PDU layout should be as straight forward as possible.
Interoperability starts with simplicity. When the logical function is
simple, documenting proper usage is a whole lot easier. When documentation
is clear in a standard, the time to interoperability and market acceptance
is reduced.

2. Efficient operation in the common execution path. I think "rough
consensus" can probably be achieved in terms of what parts of an iSCSI
header are going to be needed in the normal flow and what parts are
exception cases. If we have consensus on this we should be able to agree on
where to make things easy and where to add complexity.

3. Data integrity - If a header digest is going to be provided, the
information it covers should be checked against the digest before being
used.

I think we can improve the current PDU format, in terms of these
objectives,
by doing the following:

1. Providing a fixed size header that contains the most commonly used
fields.
2. Placing less commonly used fields in an optional portion of the header.
3. Providing a header digest over the fixed portion of the header and a
separate digest for the optional portion of the header.

Basic approach:

1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It is
always the same size and is followed by a header digest that covers the
"BHS", if header digests are being used. The fact that header digests are
being used within a TCP byte stream is not encoded into the frame, but is
obtained from state associated with the session. This allows the location
of
the header digest to be determined without using any information in the
header. Only when the integrity of the header has been validated is the
information in it used.

2. The header contains information on the size of the complete PDU, and a
length field for any additional headers. The integrity of this information
has been validated as it is part of the "BHS". The location of the
additional headers, if they exist (in most cases they will not) is after
the
header digest. Thus additional headers can be located using validated
information.

3. Each additional header is encoded using a standard TLV (type, length,
value) format. In TLV format the fixed size type indicator tells how to
interpret the data. The fixed size length field says how long the data is,
and the variable sized value field contains the data. If an application
doesn't understand the type, it can maintain sync. in the byte stream by
skipping to the length field and then skipping the value.

4. The complete set of TLV values are covered by a distinct digest. This
digest is considered a part of the header digest in terms of iSCSI option
negotiation but is physically distinct. Although this requires another
digest calculation its usage is expected to be fairly low and it is most
likely calculated over a small number of bytes.

5. The data and data digest follow the TLV digest.

In terms of layout, what you might have is the following:

================= Start of iSCSI PDU =============================

     This portion of the PDU contains the information in
     the "BHS" plus:

     1. The length field (4 bytes)
     2. The length of the TLV information

--------------- End of the fixed size header ---------------------
--------------- Start of the header digest (optional) ------------

--------------- End of the header digest -------------------------
--------------- Start of TLV information (optional) --------------

    (Additional header fields go here in a variable length space)

--------------- End of TLV information ---------------------------
------ Start of "TLV header" digest (if header digest present)----

--------------- End of "TLV header" digest -----------------------
--------------- Start of data ------------------------------------

--------------- End of data --------------------------------------
--------------- Start of data digest (optional) ------------------

=============== End of data digest and PDU =======================

[David B: This looks better in a .pdf....are we allowed to use pdfs on this
reflector?]

I believe this to be a simple yet efficient structure for common usage.
The optional headers require more involved processing, but allow
incremental
advancement of the TCP byte stream and avoids the loss of synchronization
that might take place in other formats.
The header can be verified, before being used.

[Note 1: This PDU format does not attempt to address the problem of how to
recover when there is a digest error. I believe this to be a distinct
problem, though the PDU format could provide some additional level of
robustness via redundancy of information at known locations.]

[Note 2: Julian, If more detailed information on the PDU format is needed I
would be glad to present it. However I think that most of the significant
issues are addressed by this outline. The other items don't appear, to me,
to be as significant.]

Barry Reinhold
[email-masked]
603-659-0885

---

From: "Barry Reinhold" <[email-masked]>
Date: Sat, 10 Mar 2001 15:55:06 -0500

Julian,
	On the two drawbacks:

1. Agreed, there are two header digests and this isn't desirable. However, I
suspect the TLVs will seldom be used so at least it is out of the fast path.
[BTW - Can some digest expert suggest how bad this is vs. a single longer
digest?]

2. I'm not sure I fully understand what you are asking under two, so I'm
going to answer the question of "How would you read the TLV w/o using any
unverified info." If this is an answer to the wrong question, my apologies.
You'll have to ask again.

The TLV length field is obtained from the BHS header as, or after, it has
been verified. This TLV length field gives the size of the TLV data
(surprise) that has to be buffered in reading the TCP stream to get to the
TLV digest. The TLV digest is checked against the TLV data. The TLV's are
then parsed. (Humm - I guess I have an assumption in here that TLV data is
not *really* big, such that buffering the whole set of TLVs is a reasonable
thing to do)
I don't see where one ends up using unverified fields. I would expect to
read a fixed N bytes from the TCP stream (through the header digest), verify
the header, use the fixed offset of the "TLV length field" that is in the
header, which has been verified, to find if I had TLV fields. If no, I'm on
to the data. If yes I read the TCP stream with the number of bytes given by
"TLV length field" + sizeof(TLV digest), verify the TLV fields, and then
parse the TLVs as needed.

A plus is that there is very little work to do for the fast path.
Another plus is that the TLV mechanism allows vendor options, and/or future
extensions with well understood backward compatibility behavior.
A third plus is that there should be no special cases relative to how the
PDU format has you read the TCP stream...read header, optionaly read TLV
stuff, read data. This should be friendly to hardware accelerated stuff.
A negative is that TLV parsing is sorta slow if you have complicated TLV
structures such as in DOCSIS. Our TLVs are very simple....so far at least.

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>[email-masked]
>Sent: Saturday, March 10, 2001 12:17 PM
>To: [email-masked]
>Subject: Re: iSCSI: Towards a more effective PDU format
>
>
>
>
>Barry,
>
>Thanks for outline.