Subject: RE: iSCSI: No Framing

From: "Peglar, Robert" <[email-masked]>
Date: Mon, 4 Feb 2002 08:29:32 -0600

De-lurking...

Right now, I'd say disk-disk backup/restore and mirroring.  The individual
disk devices of today can perform at ~70 MB/s.  One need only gang 14 of
these together, streaming, to reach nearly 1000 MB/s, or 1 GB/s, or 8 Gb/s.
The disk devices of tomorrow are going to perform at rates approaching 100
MB/s per spindle.  Only ten devices are then necessary.

This would be a single TCP stream (one initiator, one target, many LUs).

If one has to mirror (/move) 10 TB in a reasonable amount of time, say, then
a stream of 1 GB/s is not only necessary, but paramount.  Even at that rate,
it would still take nearly three hours to move just 10 TB.  In a
business-critical recovery situation, three hours is an eon.

Don't think application (CPU/memory in a server) to storage.  Think storage
to storage.

Rob

Rob Peglar
Corporate Architect
XIOtech Corporation, a Seagate Company
(314) 308-6983

>-----Original Message-----
>From: Prasenjit Sarkar [mailto:[email-masked]]
>Sent: Sunday, February 03, 2002 9:23 PM
>To: THALER,PAT (A-Roseville,ex1)
>Cc: [email-masked]; WENDT,JIM (HP-Roseville,ex1); 
>[email-masked]
>Subject: RE: iSCSI: No Framing
>
>
>
>Hi Pat,
>
>I had a few questions about your 10 Gbps example. If we assume 
>that this is
>a single TCP stream, I would like to know which application in 
>the near and
>forseeable future can sustain a 10 Gbps data rate for a 
>reasonable amount
>of time? Or if they are multiple TCP streams, are you assuming 
>packet drops
>in all streams simultaneously?
>
>Thanks,
>
>
>   Prasenjit Sarkar
>   Research Staff Member
>   IBM Almaden Research
>   San Jose
>
>
>
>                                                               
>                                            
>                      "THALER,PAT                              
>                                            
>                      (A-Roseville,ex1)        To:       
>"WENDT,JIM (HP-Roseville,ex1)"                    
>                      "                         
><[email-masked]>, [email-masked]                        
>                      <pat_thaler@agile        cc:             
>                                            
>                      nt.com>                  Subject:  RE: 
>iSCSI: No Framing                             
>                      Sent by:                                 
>                                            
>                      [email-masked]                        
>                                            
>                      .edu                                     
>                                            
>                                                               
>                                            
>                                                               
>                                            
>                      02/03/2002 03:01                         
>                                            
>                      PM                                       
>                                            
>                                                               
>                                            
>                                                               
>                                            
>
>
>
>Jim,
>
>In answer to your questions:
>
>Agilent is planning to implement framing. We view both FIM and
>COWS as having about the same utility so we would implement
>whichever one went into the standard.
>
>A smoothing buffer on the chip is feasible wrt your point 2.
>There are some configurations that would use external memory.
>
>Also, one concern is that the very situation where one would
>need large window size for efficiency - high bandwidth long
>distance communication - is where out of order receipt becomes
>increasingly likely so the amount of memory for this situation
>could push up cost to an excessive extent.
>
>Reducing the amount of traffic that has to be shunted to an
>external memory affects the bandwidth that needs to be provided
>for that memory. If we can handle most PDUs internally then the
>external memory doesn't have to be as wide and as fast. For
>instance, if a drop means that the iSCSI PDU with the drop
>is put into external memory then that takes less memory bandwidth
>than if a drop means the whole data stream for that session will
>be going into the buffer until the hole is filled.
>
>This decision also effects latency after a drop and required
>backplane bandwidth.
>
>Let's say one is on a 10 Gbit/s extreme long distance WAN
>connection and that there is a drop. If the round trip delay
>for getting the whole filled is 100 ms, then without framing,
>one could have about 1 Gbit stored in the local buffer memory
>by the time the hole is filled. One will have to process all
>of this through iSCSI and across the backplane before one can
>deal with the new traffic coming in. If the backplane data rate
>is closely sized to the external data rate, an extra 100 ms latency
>has just been inserted into the path until traffic slacks off.
>(Congestion control might mitigate this to the extent that one
>is talking to just one source, but with multiple sessions, one
>will still have to keep up.) To get the buffer emptied to make
>space for further drops and to get the latency back down, one
>will have to size the backplane bandwidth wider and have the
>iSCSI engine process at a higher data rate.
>
>With FIM or COWS, one can end the incident with much less data
>in the buffer as one processes PDUs after the hole while waiting
>for it to be filled. Then, when the hole is filled one just has
>to process a small amount of data to catch up.
>
>Regards,
>Pat Thaler
>
>-----Original Message-----
>From: WENDT,JIM (HP-Roseville,ex1) [mailto:[email-masked]]
>Sent: Tuesday, January 29, 2002 10:47 PM
>To: [email-masked]
>Subject: iSCSI: No Framing
>
>
>
>So, it would be good to hear from several iSCSI
>NIC/chip implementors who:
>- have or plan to implement FIM or COWS (or some
>other framing mechanism) and take advantage of it to
>minimize demands on on-NIC buffer memory
>bandwidth/quantity.
>- believe that all-buffers-on-chip solutions are
>feasible and valid (wrt the points above, including
>#2)
>- can quantify the memory/pin/power/space cost
>savings for all-buffers-on-chip solutions
>
>Jim
>
>
>
>

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 4 Feb 2002 09:37:09 -0800

Rob,

Although it will be true large amounts of bandwidth will be exchanged
between the typical SAN, such systems also use cache due to mechanical
limitations with respect to rotation and linear access.  If it were just
storage to storage, Fibre-Channel or FC encapsulation would be preferable.
At least with that scheme, there are direct placement interfaces available.

Direct placement is needed with low latency high bandwidth to reduce
overhead lost during a subsequent memory copy.  SCTP offers a means of
implementing direct placement without kludging a portion of the application
beneath the transport.  In addition, this direct placement scheme can be
done in an general fashion to support thousands of higher level protocols.
As with most SAN, network related failures are not well tolerated, so the
added robustness of SCTP becomes highly beneficial.

In establishing a large remote mirror, the highest bandwidth means of
initialization would be physical transport of the image.  Once initialized,
only differential information is exchanged largely limited by the bandwidth
and error rate of the interconnect.

Doug