Subject: RE: iSCSI: Markers

From: "John Hufferd" <[email-masked]>
Date: Thu, 10 Jan 2002 01:06:10 -0800

Folks we can not add TUFs to our iSCSI protocol since it requires TCP/IP
changes.  Our draft says that we WILL NOT work on Modifications to internet
transport protocols,  Hence,  bringing TUFs into our draft is not doable.

I think I made the point about Desktops and laptops on my kick off note,
they use NFS today and think things are fine.  They will accept this in the
future. (Right or Wrong).

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Somesh Gupta" <[email-masked]>@ece.cmu.edu on
01/09/2002 10:40:17 PM

Please respond to <[email-masked]>

Sent by:    [email-masked]

To:    <[email-masked]>
cc:
Subject:    RE: iSCSI: Markers

COWS used in conjunction with "framing" allows for
the "packetization" of TCP, and a lot of good things follow
from that.

COWS and framing allow complete processing of the segment
as a single entity - one PDU, processed in a pipelined
without recursion, and done. Markers enable no such
capability. They actually make things worse by having to
be inserted or removed as extra bytes in the stream.

Markers and CRC interact very poorly since the markers
are part of the data stream, but not part of the CRC.

Markers can only be processed in the context of the
connection state. COWS could actually help locate the
state.

There seem to be two main motivations to push markers

1. laptops and desktops using unchanged TCP/IP stacks
   and running iSCSI.
   a. when do we think customers will really deploy iSCSI
 at desktops/laptops without security and make their
 entire storage vulnerable? If of course, security is
 deployed, then COWS should be minor additional burden
   b. If we have lots of spare cycles, does it matter anyway
   c. even if the first generation does not have the capability,
 the second generation of iSCSI implementation may implement
 the TCP changes required for security - especially if it
 mandatory

2. decide something now and sooner rather than better. Given
   the substantial lack of evidence whether markers are useful
   at all, and the significant promise of COWS/ULP framing,
   I think it would be ill-advised to try to push this one.

I think we should take this as part of our next phase. If
we do want something now, I vote for COWS/ULP. Even if it
is on the experimental track, we can always specify it as
part of iSCSI.

COWS/ULP really allows the cheaper and more widely applicable
adapters that we would all like to see.

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stephen Bailey
> Sent: Wednesday, January 09, 2002 5:25 AM
> To: [email-masked]
> Subject: Re: iSCSI: Markers
>
>
> > assuming iWARP is not an available option
>
> What John is referring to as iWarp is actually `TUF' (TCP ULP
> Framing).  iWarp is a putative RDMA on IP transports protocol, which
> is not what we're talking about.
>
> All the TCP framing proposals we're discussing have:
>
>   1) an in-stream format
>
> To this, TUF adds:
>
>   2) a TCP sender segmentation behavior
>
> The in-stream formats so far are:
>   o fixed-interval markers (FIM)
>   o random key/length headers (TUF's FPDU)
>   o COWS header + marker knockout chain
>
> In addition to Julian's COWS proposal, Costa & I also did one:
>
>   http://www.cs.uchicago.edu/~steph/draft-bailey-tsvwg-cows-00.txt
>
> It's substantially similar to Julian's, with a couple differences:
>
>   o pointers are only forward
>   o payload length is 16-bit length (trivial to make it up to 2^30-1 tho)
>   o payload size is byte-granular
>   o marker is defined a priori (I like Julian's idea of being able
>     to exchange markers if desired---we'd add that)
>
> Our COWS was defined as an alternative to (replacement of) TUF's
> current in-stream format.  Compelling benefits of the COWS in-stream
> format from our standpoint are:
>
>   o the SAME in-stream format is works with or without TCP sender
>     segmentation support
>   o no chance of a false positive from key/length validation
>
> The first point is particularly important.  COWS can be used to
> validate the PDU contaiment property if the TCP sender is
> TUF-compliant, and it can be used for marker-like (better, actually)
> resynchronization if the TCP sender is stock.  Before the COWS
> proposal, we always had two different in-stream representations
> (markers, & key/lengths) for different purposes.
>
> The primary disadvantage of the COWS in-stream format is:
>
>   o prohibits 0-touch software senders
>
> I had suggested that if everybody liked COWS, we would update TUF to
> use it, and even if TUF remains experimental, iSCSI could define the
> same in-stream format for its framing (replacing markers).  That way,
> there would really be only ONE in-stream format in the universe,
> though it might be redundantly defined in two places.
>
> For iSCSI, a single in-stream format that supports both modified and
> stock sender approachs seems nice.  Implementations can easily migrate
> from stock to modified TCP senders (ask one of the authors of the TUF
> draft if you don't understand why modified TCP sender segmentation is
> good).
>
> For the rest of the world it would be nice (essential in my opinion)
> if the same framing technique could be used for iSCSI and other
> protocols (e.g. iWarp).  System vendors that support iSCSI just might
> want to support NAS too, which could mean DAFS (on iWarp).  Chip
> vendors CERTAINLY want to maximize return from their TOE investment
> which means being able support various accelerated protocols with the
> same hardware.  Wouldn't it be stupid if we had a proliferation of
> framing alternatives just because the world originally seemed flat?
>
> The tradeoff of 0-touch versus a single format clearly has powerful
> arguments pulling either way.  I'm personally ambivalent.  Mostly, I
> want to make sure everybody in the iSCSI community is sufficiently
> informed about what's at stake.
>
> Steph
>

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 10 Jan 2002 10:25:40 -0800

I think the objections are to more to changing TCP wire
protocol (i.e. header) than to the implementation.

Also with NFS, there are two things.
1. There is some built-in containment in NAS which does
provide some protection again "buggy" implementation/administration/
deliberate access to the wrong data. SANs as a rule require the
clients to be much better behaved.
2. The security problems are recognized and being addressed in
newer revs - also a sign that protocols can evolve. Would it
be better to have everything in up-front - of course. But in
this case, the evidence isn't there - and to make a choice
just to accomodate - "software implementation but no rolling
of the COWS in the copy or checksum loop, and which does not
do CRC" and "in a hurry"??

Somesh

> John Hufferd
> Sent: Thursday, January 10, 2002 1:06 AM
> To: [email-masked]
> Cc: [email-masked]

---

From: Jonathan Stone <[email-masked]>
Date: Thu, 10 Jan 2002 11:41:33 -0800

In message <[email-masked]>,

>do CRC" and "in a hurry"??

Somesh,

Part of the message seems to've suffered packet drop.

I can, and do, change my TCP implementation.  The gotcha is that
folding another operation into the existing loops in my TCP protocol--
whether it's COBS or COWS or a CRC -- *IS* changing my TCP
implementation.  Even if it doesn't change the on-the wire semantics,
its still a change to my TCP.

In fact, it's easier for me to change my TCP to make it guarantee to
preserve any segment boundaries between iSCSI PDUs[*], than it is to
fold iSCSI-PDU processing into TCP processing.

[*] when sending/retransmitting, that is.
If we could guarantee that, hardware-accelerated receiers could use a
TCP header option to indicate start-of-frame in this segment, and be
done with it.  I'd even to go to bat with the transport-area
representative, arguing that the ratio of iSCSI PDUs to expected drops
is ``small enough'' that using that portion of the limited TCP option
space for non-SACK usage was in fact a good trade-off.  (My
understanding is that it's not really necessary to mark *all* PDU
boundaries, just to mark enough of them to permit a reassembly buffer
significantly smaller than BW*RTT, which therefore doesn't require
lots of external SRAM; while still allowing direct data placement.)

I understand that  door is well and truly closed, though.

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 10 Jan 2002 12:12:30 -0800

> From: Jonathan Stone [mailto:[email-masked]]
> Sent: Thursday, January 10, 2002 11:42 AM
> Cc: John Hufferd; [email-masked]
> In message 
> <[email-masked]>,
> "Somesh Gupta" writes:
> 

  Yes and at least the IETF should not object to that part. John was
  referring to the fact that iSCSI charter states no changes to
  TCP (and by that I mean protocol). Even markers require changes
  to TCP on the receiver end, and fairly heavy at that.

> fold iSCSI-PDU processing into TCP processing.

  By PDU processing do you imply
  doing the word stuffing processing? It should be something
  that can be rolled into the copy loop (if iSCSI is in user space
  and one of the applications that John referred to in the past)
  or into the checksum loop (if iSCSI is in the kernel space), or
  into the CRC loop.
 
  NOTE: If the data is touched once anyway, the cost of an additional
  touch is fairly minimal because everything is in the processor cache.

  The worst case is an accelerated NIC which does everything except
  COWS and CRC - the sender uses iSCSI, does not to do CRC, but
  does do COWS. This requires an extra touch.

> I understand that  door is well and truly closed, though.

  Actually as someone pointed out at the last IETF, the timestamp
  option wastes a couple of bytes(?). Seems like an opportunity
  without adding overhead. That is really the right way but people
  point at all sorts of misbehaving middle boxes.

  The advantage with COWS is that detection of alignment is
  deterministic.

---

From: Jonathan Stone <[email-masked]>
Date: Thu, 10 Jan 2002 12:46:16 -0800

In message <[email-masked]>,
"Somesh Gupta" writes:

>  doing the word stuffing processing? 

Yes, that's part of it.

>  It should be something
  [...]

This is still a change to the TCP _implementation_. Only the
implementor of the TCP and the kernel/user boundary can do it, and it
has to be tied, somehow, to the particular socket/fd/whatever is using
not just TCP, but iSCSI-over-TCP.

The issue I see here isn't IETF mandates: its that reasonable-quality
software implementations require changes to code which may well be
beyond the control (or even the access) of the iSCSI implementor.

>  touch is fairly minimal because everything is in the processor cache.

No, it isn't, not always.  I can think of at least three SIGCOM/TON
papers, measuring data on top-end 1996 vintage machines, on which this
assumption has been conclusively shown to be wrong. That
price/performance point is still quite viable.  Torsten Braun and
Christoph Diot had one paper in SIGCOMM 95 and later in ToN; I dont
recall the other one (my Stanford office papers are all packed).

I can't speak to COWS. I have done a COBS implementation, and the
byte-insertion meant that COBS dirtied every cache line; it turned out
simpler and faster to just do another copy.

[...]

[header options]

My understanding is that the limited option space, and the rather
large demand on it which SACK can generate in wide-area links
(transatlantic can regularly show 50% drop rate!), means our friendly
transport representatives are reluctant to admit further use of TCP options.

As I said, I'm willing to examine the numbers with the people I know.
I'd be for it, myself.