Subject: FCIP link: Severe underutilization problem?

From: Jyothi Vaidyanathan <[email-masked]>
Date: Wed, 8 Aug 2001 10:21:26 -0700 (PDT)

Satish,

Has anybody else confirmed these calculations? I hope
they are wrong, because if they are correct they
severely limit the utility of the FCIP standard.

Regards,

Jyothi

Jyothi Vaidyanathan
Elmwood Semiconductors, Inc.
(401) 941-3910

--------------Original Message------------------

TCP works with some other structures. It is a sliding
window protocol. And
yes for the links you mentioned the window has to be
sizable. And the
number-of-SCSI commans/R2Ts/Data in flow has to be
also large to fill the
pipe.

Julo

"Satish Menon" <[email-masked]> on
08-08-2001 00:55:41

Please respond to "Satish Menon"
<[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:
Subject:  your query

Here is how the  calculation was done. Also, some
folks told me that N-port
HBAs don't reserve  end-to-end credits by exchange,
they just have a common
pool of
credits  for the entire HBA (around 16), so the
calculations factor in
this
new
data point.

Assume a 1 Gbps link.
Assume a 2K FC  frame.

So it takes 16us to transmit a frame. For 200ms RTT,
that means  that
200ms/
16us ~ 10,000 credits.

Case A. For a 1Gbps FCIP link with  100 N-port HBAs,
each HBA needs to
advertise 100 credits (10,000/100) to  fully utilize
the link. The
utilization with 16 credits/ HBA will be  16%.

Case B. For a 1 Gbps FCIP link with 5 N-port HBAs,
each HBA needs  to
advertise 2,000 credits (10,000/5) to fully utilize
the link.  The
utilization with 16 credits/ HBA will be 0.8%.

Case C. For a 10  Gbps FCIP link with 100 N-port HBAs,
each HBA needs to
advertise 1,000  credits to fully utilize the link.
The utilization with 16
credits/ HBA will  be 1.6%.

Case D. For a 10 Gbps FCIP link with 5 N-port HBAs,
each HBA  needs to
advertise 20,000 credits to fully utilize the link.
The utilization  with
16
credits/ HBA will be 0.08%.
------------

> -----  Original Message -----
>
> From: "Julian Satran" <[email-masked]>
> To: <[email-masked]>
> Sent: Monday, August 06, 2001 11:01  PM
> Subject: Re:
>
>
> >
> > How did you  come to this number? What did you
assume for the exchange
> >  length?
> > Julo
> > "Satish Menon" 
<[email-masked]>@ece.cmu.edu on 06-08-2001
> > 23:44:05
>  >
> > Please respond to "Satish Menon"
<[email-masked]>
> >
> > Sent by: [email-masked]
> >
> >
> > To: <[email-masked]>
> > cc:
> > Subject:
> >
>  >
> >
> >
> >
> > Does anyone know how  many end-to-end credits a
typical HBA (N-port)
> issues
> > per  exchange? For a long distance link over FCIP,
clearly this number
> >  needs to be large. If not, then the link
utilization will be extremely
>  > poor. For e.g. a 200ms RTT link at 1 Gbps
assuming 1 exchange  through
it,
> > it will need 10,000 credits to keep the link fully
 utilized. Using a
> > different number of exchanges say 2000, the 
number of credits per
> exchange
> > drop to 5.

__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

---

From: Stephen Bailey <[email-masked]>
Date: Thu, 09 Aug 2001 04:17:48 -0400

> Has anybody else confirmed these calculations?

I don't think they are correct.

When you are within the FC network, credits are link-layer.  That
means, each LINK must have enough credits to satisfy only its
bandwidth * delay product.  You will not get a 200 MS delay on a
single FC link.  You will get a speed-of-light in a fibre (or twinax)
+ some small overhead type delay.

When you are in TCP, YES, you need enough buffering to cover the
bandwidth * delay of the entire TCP connection.  If THAT has a 200 MS
RTT and a bandwidth of 10 Gbits/s, you need 200 MB of buffering
(analogous to FC credits) in the FCIP gateways to `guarantee' it works
at line rate under ideal circumstances.  The FCP HBAs do not need 200
MB worth of BB credits.  For the HBA's first hop FC link, a number
like Tachyon's 4 BB credits is more than enough.

> And the number-of-SCSI commans/R2Ts/Data in flow has to be also
> large to fill the pipe.

Julian makes the vastly more important point here.  This applies no
matter WHAT the link technology.

Even if you have flow control buffering to saturate your links at 10
Gb/s * 200 MS, most file systems don't generate nearly enough
concurrent demand (e.g. 200 MB of total outstanding transfers AT ALL
TIMES in this case), to maintain performance at this bandwidth *
delay.

You can sing the aggregation song in this case, though.  Each client
may only generate O(10 MB) of concurrent demand, but if you have 20
clients, you could fully use your 10 Gb/s * 200 MS.

Even though present hosts may not be tuned for the next bandwidth *
delay step, that's never been a reason not to take the step.  The
single-stream performance of FCP @ > 10 km was pretty lackluster
initially on many platforms.  Make sure you appropriately calibrate
your initial expecations on FCIP.

Steph