Subject: RE: iSCSI: No Framing

From: "Peglar, Robert" <[email-masked]>
Date: Mon, 4 Feb 2002 14:21:22 -0600

>-----Original Message-----
>From: Douglas Otis [mailto:[email-masked]]
>Sent: Monday, February 04, 2002 11:37 AM
>To: [email-masked]
>Subject: RE: iSCSI: No Framing
>
>
>Rob,
>
>Although it will be true large amounts of bandwidth will be exchanged
>between the typical SAN, such systems also use cache due to mechanical
>limitations with respect to rotation and linear access.  If it 
>were just
>storage to storage, Fibre-Channel or FC encapsulation would be 
>preferable.
>At least with that scheme, there are direct placement 
>interfaces available.

Without bogging down the list in a debate about storage subsystems, I must
say that there are examples to the contrary w.r.t. the usage of subsystem
cache. 

iSCSI is, of course, more than just storage-to-storage.  But we should not
limit iSCSI to just host-to-storage.  Remember, storage subsystems can (and
are, in some cases) both initiator and target at the same time.

The original thread began with a question (paraphrased) about '...what
applications could consume a 10G pipe for long periods of time'.  I answered
that question - disk-disk backup and subsystem replication.

FC is not sufficient.  Storage-to-storage needs all the advantages as well
as that which iSCSI has to offer the host-storage model.

>
>Direct placement is needed with low latency high bandwidth to reduce
>overhead lost during a subsequent memory copy.

This assumes a subsequent memory copy.  

>SCTP offers a means of
>implementing direct placement without kludging a portion of 
>the application
>beneath the transport.  In addition, this direct placement 
>scheme can be
>done in an general fashion to support thousands of higher 
>level protocols.
>As with most SAN, network related failures are not well 
>tolerated, so the
>added robustness of SCTP becomes highly beneficial.

Having read this list since its creation, I choose to not participate in the
SCTP/TCP religious war.  

>In establishing a large remote mirror, the highest bandwidth means of
>initialization would be physical transport of the image.  Once 
>initialized,
>only differential information is exchanged largely limited by 
>the bandwidth
>and error rate of the interconnect.

Well put.  Consider 3-10 TB of differential per day as a start.  Consider
images of 100s to 1,000s of TB, spread across 100s to 1,000s of 15KRPM (or
more, stay tuned) spindles.  Now, we're talking.

Rob

Rob Peglar
Corporate Architect
XIOtech Corporation, a Seagate Company
(314) 308-6983

>
>Doug
>
>> De-lurking...
>>
>> Right now, I'd say disk-disk backup/restore and mirroring.  
>The individual
>> disk devices of today can perform at ~70 MB/s.  One need 
>only gang 14 of
>> these together, streaming, to reach nearly 1000 MB/s, or 1 GB/s,
>> or 8 Gb/s.
>> The disk devices of tomorrow are going to perform at rates 
>approaching 100
>> MB/s per spindle.  Only ten devices are then necessary.
>>
>> This would be a single TCP stream (one initiator, one 
>target, many LUs).
>>
>> If one has to mirror (/move) 10 TB in a reasonable amount of
>> time, say, then
>> a stream of 1 GB/s is not only necessary, but paramount.  Even at
>> that rate,
>> it would still take nearly three hours to move just 10 TB.  In a
>> business-critical recovery situation, three hours is an eon.
>>
>> Don't think application (CPU/memory in a server) to storage.
>> Think storage
>> to storage.
>>
>> Rob
>>
>> Rob Peglar
>> Corporate Architect
>> XIOtech Corporation, a Seagate Company
>> (314) 308-6983
>

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 4 Feb 2002 13:23:19 -0800

Rob,

I agree with your conclusions.  10 TB/day would represent an average of
about 1 Gb/sec.  This would be possible within a MAN environment.  Indeed,
within that environment and through the use of specialized memory handling,
the need for any framing technique quickly vanishes.

Doug

> >limitations with respect to rotation and linear access.  If it were
> >just storage to storage, Fibre-Channel or FC encapsulation would be
> >preferable. At least with that scheme, there are direct placement
> >interfaces available.
>
> limit iSCSI to just host-to-storage.  Remember, storage subsystems can
> (and are, in some cases) both initiator and target at the same time.
>
> applications could consume a 10G pipe for long periods of time'.
> I answered that question - disk-disk backup and subsystem replication.
>
> > overhead loss during a subsequent memory copy.
>
> >SCTP offers a means of implementing direct placement without kludging
> >a portion of the application beneath the transport.  In addition,
> >this direct placement scheme can be done in an general fashion to
> >support thousands of higher level protocols. As with most SAN,
> >network related failures are not well tolerated, so the added
> >robustness of SCTP becomes highly beneficial.
>
> Having read this list since its creation, I choose to not
> participate in the SCTP/TCP religious war.
>
> >initialized, only differential information is exchanged largely
> >limited by the bandwidth and error rate of the interconnect.
>

---

From: Jonathan Stone <[email-masked]>
Date: Tue, 05 Feb 2002 19:09:41 -0800

In message <[email-masked]>,

>that question - disk-disk backup and subsystem replication.

Even disk-to-disk applications or backup applications really want
approximately BW*RTT worth of buffering.  Hugh Holbrook's recent
Stanford PhD thesis traces the conventional wisdom back to an email
from Van Jacobson to the e2e list in 1990.

It's reasonably well-known in the TCP community that TCP slow-start
generates spiky traffic. It leads to bursts of high buffer occupancy
(e.g., at the point where the exponential rampup switches to
congestion avoidance.)  Indeed, that was the motivation behind
TCP-Vegas, and the recent work on TCP pacing.

The whole debate over framing/marking only makes sense if one views
outboard NIC buffering of RTT*BW as very expensive (e.g., forcing a
design from onchip RAM to external SRAM). Adding framing of iSCSI PDUs
allows the NIC to continue doing direct data placement into host
buffers, accomodating the BW*RTT of TCP buffering in "cheap" host RAM
rather than "expensive" NIC RAM.  

But you can't get away from providing the buffers. Not unless you are
also willing to artificially restrict throughput.  If iSCSI doesn't
provide some form of framing, then what can a NIC on a MAN with medium
BW*RTT do, if it sees a drop? It has only a few choices:

  1. start buffering data outboard, hoping that TCP fast-retransmit will
     send the missing segment(s) before the outboard buffers  are exhausted;

  2. Give up on direct  data placment, and start delivering packets to
    host memory, any old how --at the cost of SW reassembly and alignment
    problems, and a software CRC, once the missing segment is recovered.

  3. Start dropping packets, and pay a huge performance cost.

There are some important caveats around the BW*RTT: if we can
*guarantee* that the iSCSI NICs are never the bottleneck point, or
that TCP never tries to reach the true link BW*RTT (due to undersized
windows), then one can get away with less. (See Hugh Holbrook's thesis
for more concrete details).

But the lesson to take away is that even in relatively well-behaved
LANs, TCP *by design* is always oscillating around overloading the
available buffers, causing a drop, then backing off.  See, for
example, Figure 2 of the paper by Janey Hoe which introduced "New
Reno"; or Fig. 2 and 3 of the paper by Floyd and Fall. New Reno avoids
the long-timeouts between each drop, but the drops themselves still
occur.

Moral: TCP can require significant buffering even on quite modest
networks.  It __may__ be worth keeping framing, so that host NICs can
do more of that buffering in host memory rather than outboard; and so
they can continue performing DDP rather than software reassembly and
software CRC checking. Storage devices are another issue again.

References:

Van Jacobson, modified TCP congestion avoidance algorithm.
Email to [email-masked], April 1990.

L Brakmo, , S O'Malley, L Peterson, TCP Vegas: new techniques for congestion
detection and control, SIGCOMM 94.

J Kulik, R Coulter, D Rockwell, and C Partridge, A
simulation study of paced TCP. BBN TEchnical Memorandum 1218, BBN,
August 1999.

J Hoe, Improving the Startup Behaviour of a Congestion Control Scheme
for TCP,  ACM SIGCOMM 1996, 

S Floyd and K Fall, Simulation-based comparisons of Tahoe, Reno, and SACK
TCP, Comp. COmm. Review no 6 v 3, April 1996.

H Holbrook.  A Channel Model for Multicast.  PhD Dissertation.
Department of Computer Science.  Stanford University.  August, 2001.
http://dsg.stanford.edu/~holbrook/thesis.ps{,.gz}. (See Chapter 5.)

(Holbrook cites Aggrawal, Savage, and Anderson, INFOCOMM 2000, on the
downsides of TCP pacing; but I haven't read that.  The PILC draft on
link designs touch the same issue, but the throughput equations cited
there factor out buffer size.)

>as that which iSCSI has to offer the host-storage model.

But it will still need approximately BW*RTT of buffering, even for
low-delay LANS. Or performance will fall off a cliff under
"congestion" -- e.g., each time some other iSCSI flow starts up,
begins competing for the same TCP endpoint buffers, on the same iSCSI
device, and triggering a burst of TCP loss events for the
storage-to-storage flow.