Subject: RE: new iSCSI draft - 02.txt

From: "David Peterson" <[email-masked]>
Date: Wed, 3 Jan 2001 08:47:07 -0600

I believe there was agreement to remove the Urgent-Pointer framing mechanism
but don't recall any agreement to replace it with an in-stream marker. For a
software implementation it would be hard to support this type of framing
mechanism. I believe a TCP option indicating the message boundry or a
fixed-length PDU at a granularity to minimize overhead are much better
solutions and are workable for both software and hardware implementations. I
have not seen an agenda but I would hope this issue would be discussed at
the upcoming meeting in Orlando.
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
[email-masked]
Sent: Saturday, December 30, 2000 11:11 AM
To: [email-masked]
Subject: new iSCSI draft - 02.txt

Dear colleagues,

I've just submitted to the Internet-Drafts repository and to our list
archive (at CMU) a new
version of the draft.

Changes (from 2b!):

   framing by Urgent-Pointer replaced by framing through in-stream marker
   editorials and typos (not completed yet)
   simpler digests
   digest recovery and some clarifications on iSCSI specific errors (more
   to come)

I will have version 03 with many more editorial changes before the
intermediate meeting.

You can see the draft at:

http://www.haifa.il.ibm.com/satran/draft-ietf-ips-iSCSI-02.txt

Regards,
Julo

---

From: "Sukha Ghosh" <[email-masked]>
Date: Wed, 3 Jan 2001 19:52:02 -0500

I agree with you fully on this. Also I like the idea of fixed length PDU
better than a new TCP option indicating message boundary.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
David Peterson
Sent: Wednesday, January 03, 2001 9:47 AM
To: Ips@Ece. Cmu. Edu
Subject: RE: new iSCSI draft - 02.txt

-----Original Message-----

Dear colleagues,

Changes (from 2b!):

Regards,
Julo

---

From: Matt Wakeley <[email-masked]>
Date: Wed, 03 Jan 2001 17:25:30 -0800

> better than a new TCP option indicating message boundary.

Why do you like fixed length PDUs?  SCSI commands are really short.  Do you
really want to waste a big PDU to send a small payload?   Or do you want to
use small PDUs and incur a large header vs data ratio?

-Matt Wakeley
Agilent Technologies

---

From: Matt Wakeley <[email-masked]>
Date: Wed, 03 Jan 2001 17:21:22 -0800

> the upcoming meeting in Orlando.

What is so difficult for software to insert a few extra bytes in the byte
stream?  It's simply a layering problem:

iSCSI layer <-> marker layer <-> TCP

Normally, the marker layer simply transfers the bytes from the iSCSI layer to
the TCP. Every X number of bytes, the marker layer inserts the marker into the
byte stream.

Since your software will probably not benefit from the receipt of the markers,
it would negotiate not to receive the markers.  It would only send markers
*IF* the remote node requested them to be sent.

-Matt Wakeley
Agilent Technologies

---

From: Matt Wakeley <[email-masked]>
Date: Wed, 03 Jan 2001 18:22:54 -0800

David Peterson wrote:
> 
> I believe a TCP option indicating the message boundry ... are much better
> solutions and are workable for both software and hardware implementations.

Dave,

I would *LOVE* to have a TCP option to indicate message boundaries. However,
an implementation cannot rely on the new Option being implemented.  Besides,
the IETF is opposed to any changes to TCP since they've invented the as yet
unproven SCTP.

The marker scheme is the quickest and cleanest way to achieve the framing
requirements iSCSI needs.

-Matt

---

From: Glen Turner <[email-masked]>
Date: Thu, 04 Jan 2001 15:31:31 +1030

Sukha Ghosh wrote:
> 
> Also I like the idea of fixed length PDU better than a new TCP
> option indicating message boundary.

A fixed length PDU doesn't solve the fundamental problem.  In a
stream of bytes you still can't tell where the PDU starts
unless you assume PDUs pack nicely into packets and frames
and the iSCSI process has special knowledge of the data
stream (such as TCP options).

Without altering TCP (which this IESG is opposed to) the
options are an adaption layer containing either a flag
byte and escapes (as in SLIP) or a counting scheme (such
as computing a CRC which becomes 0 when the PDU is complete).

Objections that byte stuffing to mark the PDU boundary
discourages host-based implementations are somewhat difficult
to understand.  Firstly, the feature can, and should, be
negotiatable in both directions.  A host processor implementation
can chose not to negotiate the reception of packets with
byte stuffing, at the cost of incurring TCP head-on-line
blocking due to late packets.

Secondly, the decision to include a iSCSI-level checksum (to avoid
surprisingly common corruption by DMA chips and the like, see the
SIGCOMM2000 Proceedings) means that the data has to be scanned
by the CPU anyway.  Any other implementation of the iSCSI-level
checksum gives no protection.

The real objection is that removing the byte-stuffing requires
a buffer copy, not just a scan.  The second objection is that
the overhead from SLIP-style byte stuffing has a pathological case.

The counting schemes do not need a buffer copy (as you can arbitrarily
drop, say, the last 8 octets that were required to drive the CRC
to 0 and retreive the original data).  However, these schemes are
unavoidably computationally expensive, perhaps more so than a buffer
copy (although RISC processors do about 100 instructions per
memory cycle, so this assumption needs testing).

A modification of "Consistent Overhead Byte Stuffing" (IEEE/ACM
Transactions on Networking, Vol 7, No 2, 1999) seems to hold
a neat solution to the pathological case of SLIP.  It offers a
neat way of identifying packet boundaries that has a bound on
the growth in the packet size.  However, like SLIP it also
requires that a buffer copy occur to undo the detection of the
packet boundary.

If the WG find it best to deal with TCP as a stream of octets
with zero knowledge of the underlying packet boundaries, options
and sequence numbers then an adaption layer for TCP with a
COBS-based scheme to identify packet boundaries seems to have the
most promise.

Regards,
Glen

-- 
 Glen Turner                                 Network Engineer
 (08) 8303 3936      Australian Academic and Research Network
 [email-masked]          http://www.aarnet.edu.au/
--
 The revolution will not be televised, it will be digitised

---

From: Kevin Quick <[email-masked]>
Date: Thu, 4 Jan 2001 10:06:55 -0600 (CST)

: From: Matt Wakeley <[email-masked]>
: 
: David Peterson wrote:
: > 
: > I believe there was agreement to remove the Urgent-Pointer framing mechanism
: > but don't recall any agreement to replace it with an in-stream marker. For a
: > software implementation it would be hard to support this type of framing
: > mechanism. I believe a TCP option indicating the message boundry or a
: > fixed-length PDU at a granularity to minimize overhead are much better
: > solutions and are workable for both software and hardware implementations. I
: > have not seen an agenda but I would hope this issue would be discussed at
: > the upcoming meeting in Orlando.
: > Dave
: 
: What is so difficult for software to insert a few extra bytes in the byte
: stream?  It's simply a layering problem:
: 
: iSCSI layer <-> marker layer <-> TCP
: 
: Normally, the marker layer simply transfers the bytes from the iSCSI layer to
: the TCP. Every X number of bytes, the marker layer inserts the marker into the
: byte stream.
: 
: Since your software will probably not benefit from the receipt of the markers,
: it would negotiate not to receive the markers.  It would only send markers
: *IF* the remote node requested them to be sent.
: 
: -Matt Wakeley
: Agilent Technologies

I would think that there would have to be a number of changes to existing
TCP implementations to utilize TCP-option-based framing:

* Modification of the transmit request interface to indicate frame
  boundaries
* Modification of the TCP transmit segmentation algorithm to recognize
  and preserve frame boundaries
* Modification of TCP slow-start window sizing to recognize and preserve
  frame boundaries
* Modification of MTU path discovery, with possible rejection of some
  frame transmit requests at the transmit request interface if the frame
  exceeds allowable boundaries.
* Possible delivery failures if the path is silently changed or
  multiplexed after initial discovery?
* Possible failures if "legacy" nodes either preserve option data but
  re-segment the traffic, or discard the "new" option information
  entirely.
  ** Firewalls, data-caching nodes, layer-3 routers, and other
     store-and-forward nodes would be susceptible.
* Modification of the receive path socket management to preserve frame
  boundaries.
* If any node re-transmits any portion of the data stream (due to NAK or
  ACK-timeout) then the retransmit logic must be modified to identify the
  frame boundaries in the retransmitted portion, recalculate the TCP
  options for that retransmitted portion (since the headers may occur at a
  new position relative to the datastream), and then perhaps resynchronize
  at the original location when the remote ACKs the rest of the previously
  transmitted window.

Some or all of the above may not be typical situations, or even generally
"seen", but since we are defining a standard, it's important to define
that standard relative to what the existing standards *allow*, not
relative to how current sw/hw is *implemented*.  [Note: I'm not familiar
enough with the details of the standards myself to know if the concerns
above are warranted... I'm trying to point out potential issues here for
consideration by more knowledgeable experts.]

As Matt points out above, in-band markers are unequivocably perserved as a
consequence of being part of the data stream, and can be handled easily by
a layer above the existing TCP management at the endpoints only.

Additionally, an in-band marker can forward reference the id of the next
command in the stream, thereby allowing receive hardware to prefetch DMA
buffer information for that next command.  This usage is congruent with
the high-performance considerations of this general effort.  Note that
this enhances both loss-less and loss-error transfers, since a marker
allows resynchronization only at the *next* message anyhow.

Regards,
  Kevin

---

From: "David Peterson" <[email-masked]>
Date: Fri, 5 Jan 2001 15:40:07 -0600

The problem is: at that layer there is simply not a byte stream. The data
is contained in socket buffers which are most likely mbuf clusters for most
implementations. To "simply insert a few extra bytes" requires mbuf
manipulation.
Mbuf manipulation can processing intensive. Depending on how the mbuf data
is
packaged a small DMA operation may occur for these few extra bytes also.
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Matt Wakeley
Sent: Wednesday, January 03, 2001 7:21 PM
To: IPS Reflector
Subject: Re: new iSCSI draft - 02.txt

> solutions and are workable for both software and hardware implementations.
I

Normally, the marker layer simply transfers the bytes from the iSCSI layer
to
the TCP. Every X number of bytes, the marker layer inserts the marker into
the
byte stream.

Since your software will probably not benefit from the receipt of the
markers,
it would negotiate not to receive the markers.  It would only send markers
*IF* the remote node requested them to be sent.

-Matt Wakeley
Agilent Technologies