Subject: iSCSI: Markers

From: "John Hufferd" <[email-masked]>
Date: Sun, 6 Jan 2002 13:57:56 -0800

When we were in Salt Lake City, we decided to hold a discussion on the
Reflector regarding Markers.  I think it is probably time to hold that
discussion.  Therefore, I will start it out.

First I want to thank Julian for going to the trouble to document the
proposal for COWS.  This now gives us two Marker proposals to consider.
They are the Fixed Interval Markers (FIM) and the Constant Overhead Word
Stuffing (COWS) proposals.

Having said that, lets examine each one and determine their usefulness.

1. Markers (FIM or COWS) are only needed by Hardware HBAs that are
attempting to limit the amount of Reassembly Buffers they need on-board.
This will always be a probabilistic determination by the vendors, but the
purpose of Markers are to hold down the amount of total RAM needed,
especially when factored into a probabilistic determination.
2. The use of Markers are negotiated per connection, and per direction.  It
is therefore possible for a software implementation to send markers, but
not have to accept them.
3. When CRC Digest are negotiated to be used, it does no good to send out
Markers since the hardware side will have to have Reassembly Buffers to
compute the CRC Digest.  Therefore, Markers do not help anything when CRCs
are used on a connection.
4. FIM can be implemented easily in software, and with an out going
Scatter/Gather technique could send PDUs across a network to a Hardware HBA
destination without requiring additional moves or touching each byte.
5. Software iSCSI implementations should always negotiate NOT to receive
markers since Markers do not help the SW in any way.
6. Hardware HBAs can also implement FIM easily, and HW can do it easily in
both directions.  Therefore, if FIM is the approved Marker proposal, a HW
HBA should send FIM whenever it sends iSCSI Commands and/or Data to other
HW HBAs (assuming iWARP is not an available option), and the other HW HBAs
wants them.
7. I do not foresee many installations using CRC Digests with Laptops and
Desktops, since the overhead will probably be noticeable, and most
installations do not judge Desktop and Laptop data to be key corporate
assets (I am not saying that opinion is right, or that it is Universal,
just that is a prevalent opinion).
8. A single software node would not be fast enough to cause a significant
load or a significant  memory requirement for the Reassembly Buffers.
However, the combination of many (perhaps Desktops and Laptops spread
across a real or virtual campus) can bring on a large load and the need for
many Reassembly Buffers.  The resultant amount of small Reassembly Buffers
could make the Total requirement very High (thus raising the cost of the
HBA).
9. Given the need for Desktops and Laptops to avoid noticeable degradation,
and the probability that they will not be using CRC, means that the use of
FIM is a compatible option when working across a network from a SW Node to
an iSCSI HW HBA implemented Storage Controller.
10. COWS was also designed to be easily implemented in Software and in
Hardware.
11. COWS will require the iSCSI Software implementations to touch each
byte, as it looks for matches to its Framing Pattern within the PDUs.
12. When CRC-32 digests are being computed, the Software will have to touch
ever byte anyway, so the additional overhead to look for matches to the
Framing Pattern is negligible.  On the other hand FIM also works well with
SW CRC-32 Digest computation.
13. But the premies I set above (which is clearly up to debate), is that
Desktops and Laptops will not usually compute the CRC Digest.
14. COWS has the ability to have its pointing Markers, which point to the
Framing Pattern Matches, point either forward or backwards.  When being
processed by software, the direction does not matter.  So since the Markers
should only be used on outgoing PDUs, the approprate direction of the
pointers is what ever the destination needs.
15. Many vendors, especially at the higher speeds, are implementing a pipe
line design.  The pipeline receive process, will most likely want the COWS
pointers to be forward pointing (pointing in the direction of the end of
the PDU).  Since the SW does not care, forward pointers should be fine.
16. If the HW Pipelining iSCSI HBA is to send COWS Markers to another HW
Pipelining HBA, there may be a problem.  The outgoing pipeline would prefer
to have the pointers pointing backwards, but the receiving pipeline would
prefer to have the pointers going forwards.  Therefore with COWS, either
the sending or the receiving HW Pipelining HBA will be unhappy and pipeline
design will get very much more complicated.
17. Everyone likes iWARP better, but it requires changes to the Host SW
TCP/IP stacks in the case of SW Initiators.  Since these SW installations
are probably going to be Desktops and Laptops, these systems will very
likely have some non current version of the OS, (such as windows 2k, ME,
etc.).  Hence it is unlikely that they will have iWARP in the majority of
the SW TCP/IP Stacks).
18. If, for some reason, a Server uses Software iSCSI, it will probably
have the latest and greatest OS software, and if that is true, and if iWARP
is approved by the IETF then it will probably have the TCP/IP extensions
that are required by iWARP.  It will therefore use iWARP instead of any
Marker solution.
19. If the Server uses Software iSCSI,  and if iWARP is not approved by the
IETF, then COWS or FIM are both  reasonable choices  for the Server, if the
Server also uses the CRC option.  If CRC is not used on the Server, FIM is
a better match with the Server SW iSCSI.
19. iWARP is probably going to stay in IETF "experimental mode" for some
time into the future.  And without iWARP being specified by iSCSI as -- at
least, "MAY Implement" and "MAY use", the HW to HW mode will still need a
Marker solution.
20. Therefore, we are left with Markers as the only technique we can count
on to aid and assist the iSCSI HW HBAs, keep their "on-board" RAM
requirements to a minimum.

Based on the above I come to the following conclusions:
1. Markers are needed.
2. FIM works best with the probable SW environment, and has the lowest
overhead (since they will probably not use CRC Digests).
3. Since if CRC is used, the HW needs its Reassembly Buffers anyway, so
there is no reason to use COWS, or any type of Marker.
4. FIM can work well between HW HBAs without significant impact to pipeline
design.

Therefore, I recommend that the Draft continue to define FIM.  Also, I
recommend that the FIM type of Markers be "MUST Implement" on the Out-Going
direction, and "MAY Implement" on the incoming direction".   Further, I
recommend the Draft say that FIM "MAY be used" in any direction, depending
on the negotiations.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

---

From: Stephen Bailey <[email-masked]>
Date: Wed, 09 Jan 2002 08:24:32 -0500

> assuming iWARP is not an available option

What John is referring to as iWarp is actually `TUF' (TCP ULP
Framing).  iWarp is a putative RDMA on IP transports protocol, which
is not what we're talking about.

All the TCP framing proposals we're discussing have:

  1) an in-stream format

To this, TUF adds:

  2) a TCP sender segmentation behavior

The in-stream formats so far are:
  o fixed-interval markers (FIM)
  o random key/length headers (TUF's FPDU)
  o COWS header + marker knockout chain

In addition to Julian's COWS proposal, Costa & I also did one:

  http://www.cs.uchicago.edu/~steph/draft-bailey-tsvwg-cows-00.txt

It's substantially similar to Julian's, with a couple differences:

  o pointers are only forward
  o payload length is 16-bit length (trivial to make it up to 2^30-1 tho)
  o payload size is byte-granular
  o marker is defined a priori (I like Julian's idea of being able
    to exchange markers if desired---we'd add that)

Our COWS was defined as an alternative to (replacement of) TUF's
current in-stream format.  Compelling benefits of the COWS in-stream
format from our standpoint are:

  o the SAME in-stream format is works with or without TCP sender
    segmentation support 
  o no chance of a false positive from key/length validation

The first point is particularly important.  COWS can be used to
validate the PDU contaiment property if the TCP sender is
TUF-compliant, and it can be used for marker-like (better, actually)
resynchronization if the TCP sender is stock.  Before the COWS
proposal, we always had two different in-stream representations
(markers, & key/lengths) for different purposes.

The primary disadvantage of the COWS in-stream format is:

  o prohibits 0-touch software senders

I had suggested that if everybody liked COWS, we would update TUF to
use it, and even if TUF remains experimental, iSCSI could define the
same in-stream format for its framing (replacing markers).  That way,
there would really be only ONE in-stream format in the universe,
though it might be redundantly defined in two places.

For iSCSI, a single in-stream format that supports both modified and
stock sender approachs seems nice.  Implementations can easily migrate
from stock to modified TCP senders (ask one of the authors of the TUF
draft if you don't understand why modified TCP sender segmentation is
good).

For the rest of the world it would be nice (essential in my opinion)
if the same framing technique could be used for iSCSI and other
protocols (e.g. iWarp).  System vendors that support iSCSI just might
want to support NAS too, which could mean DAFS (on iWarp).  Chip
vendors CERTAINLY want to maximize return from their TOE investment
which means being able support various accelerated protocols with the
same hardware.  Wouldn't it be stupid if we had a proliferation of
framing alternatives just because the world originally seemed flat?

The tradeoff of 0-touch versus a single format clearly has powerful
arguments pulling either way.  I'm personally ambivalent.  Mostly, I
want to make sure everybody in the iSCSI community is sufficiently
informed about what's at stake.

Steph

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 9 Jan 2002 22:40:17 -0800

COWS used in conjunction with "framing" allows for
the "packetization" of TCP, and a lot of good things follow
from that.

COWS and framing allow complete processing of the segment
as a single entity - one PDU, processed in a pipelined
without recursion, and done. Markers enable no such
capability. They actually make things worse by having to
be inserted or removed as extra bytes in the stream.

Markers and CRC interact very poorly since the markers
are part of the data stream, but not part of the CRC.

Markers can only be processed in the context of the
connection state. COWS could actually help locate the
state.

There seem to be two main motivations to push markers

1. laptops and desktops using unchanged TCP/IP stacks
   and running iSCSI.
   a. when do we think customers will really deploy iSCSI
	at desktops/laptops without security and make their
	entire storage vulnerable? If of course, security is
	deployed, then COWS should be minor additional burden
   b. If we have lots of spare cycles, does it matter anyway
   c. even if the first generation does not have the capability,
	the second generation of iSCSI implementation may implement
	the TCP changes required for security - especially if it
	mandatory

2. decide something now and sooner rather than better. Given
   the substantial lack of evidence whether markers are useful
   at all, and the significant promise of COWS/ULP framing,
   I think it would be ill-advised to try to push this one.

I think we should take this as part of our next phase. If
we do want something now, I vote for COWS/ULP. Even if it
is on the experimental track, we can always specify it as
part of iSCSI.

COWS/ULP really allows the cheaper and more widely applicable
adapters that we would all like to see.

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stephen Bailey
> Sent: Wednesday, January 09, 2002 5:25 AM
> To: [email-masked]
> Subject: Re: iSCSI: Markers 
> 
>