Subject: Re: Performance of iSCSI, FCIP and iFCP

From: [email-masked]
Date: Fri, 12 Jan 2001 09:30:10 +0200

Victor,

Thanks for your excellent analysis.

We became also aware about your paper on TCP performance in face of
failures.

Using effectively links was very much on our minds when advocating multiple
connections
but so was also the need to use more than a link.

I have to admit that we did not have the model to support numerically our
contention that multiple links
will behave better in the presence of errors and performance will degrade
more gracefully even on congestion.

Regards,
Julo

---

From: Stephen Bailey <[email-masked]>
Date: Sat, 13 Jan 2001 14:15:13 -0600

Victor,

> I have to admit that we did not have the model to support
> numerically our contention that multiple links will behave better in
> the presence of errors and performance will degrade more gracefully
> even on congestion.

The last time we had this discussion (multiple links between the same
endpoints enabling higher performance) on this list, it was my opinion
that doing this is not `TCP friendly', and so should not be allowed in
the general, Internet context, which is one of the design points for
iSCSI.

Clearly, when slow starting, and recovering from congestion, n
connections can effectively open the window n times as fast as 1
connection, but doing so will potentially be at the expense of other
competing flows which don't use multiple connections.  If ALL flows
use multiple connections, you simply end up hitting the congestion
wall harder, which, in an abstract sense, makes the system more likely
to oscillate (or collapse?).

I don't know if it WILL create oscillation or collapse (I'm just a
hammer mechanic), but it seems to me that if hitting the congestion
wall harder in this way were acceptable to overall network behavior,
the single connection TCP congestion avoidance could be adjusted to
create this behavior (and capture this benefit) without requiring
multiple TCP connections.

Your refereed, published work on this seems suggests that I am holding
on to `myths' about the need to play fair with TCP.  If so, I would
greatly appreciate your debunking my myths here on this list, in the
clearest way.  Particularly, we need to know if the IETF (or IESG or
IAB, or whomever it is....), will accept this type of behavior out of
iSCSI.

While I am opposed to iSCSI's multiconnection session, I do admit its
benefits.  My objections are that the feature will not be widely used,
is difficult to implement correctly, AND the same benefits are already
available through long-standing upper layer mechanisms.  That aside, I
accept the WG consensus on multiconnection sessions, but what the
iSCSI spec still needs to do, in no uncertain terms, is indicate
whether multiple connections per endpoint pair is something which
implementations SHOULD or SHOULD NOT allow for use in a general,
Internet context, or at all.

Even if iSCSI specifies that multiple connections per endpoint pair
SHOULD NOT be used in a general context, we know that implementations
are not going to prohibit it (it's primarily a configuration
decision), and that still allows the feature to be used in
specifically engineered applications, such as those built with
dedicated storage fabrics.

Thanks,
  Steph

---

From: "Victor Firoiu" <[email-masked]>
Date: Mon, 15 Jan 2001 13:26:36 -0500

Steph,

thanks for raising this point, which is clearly central to the fairness
discussion in a more general context, not restricted to IP Storage.

There are two cases to consider: whether the storage traffic is isolated
or is multiplexed with other traffic in the IP network.

My original message on this subject of performance was assuming the
first case, where all available bandwidth (in my example 10Gb/s) was
available for storage traffic, and showed that in some scenarios, about
1/4 of bandwidth was lost unnecessarily with the one-TCP solution. 
Definitely, no fairness issue was involved.

Now let's consider the second case, where storage traffic traverses a
non-exclusive IP network, such as "the Internet", where fairness is an
issue.

Unfortunately, as far as I know, despite the TCP friendly mandate
expressed or implied in the IETF community, there is no clear definition
of what is considered to be fair use of network bandwidth (TCP friendly)
and what is unfair.  RFC 2914, citing RFC 2309:

  ".. the issue of the appropriate granularity of a "flow",
   where we define a `flow' as the level of granularity appropriate for
   the application of both fairness and congestion control.  From RFC
   2309:  "There are a few `natural' answers: 1) a TCP or UDP connection
   (source address/port, destination address/port); 2) a
   source/destination host pair; 3) a given source host or a given
   destination host.  We would guess that the source/destination host
   pair gives the most appropriate granularity in many circumstances.
   The granularity of flows for congestion management is, at least in
   part, a policy question that needs to be addressed in the wider IETF
   community."

In the context of IP storage, the IPS gateways are different than the
source and destination hosts for a TCP flow in a regular IP network.

For example, in the ECM (Endpoint Congestion Manager) WG, we had a
discussion on this subject.  The ECM proposal attempts to regulate the
sending rate for a pair of endpoints to be limited to roughly one TCP
connection between those two hosts, given the current network conditions
(loss probability, round trip delay).  I raised a few questions.  For
example, if an endpoint is a multi-user system, and N users from this
system simultaneously connect to the same web server, under ECM they get
1/N of the bandwidth they would otherwise get if using N PCs, which is
not fair.  Some of us agreed that the fair unit of bandwidth should be
per user and not per system, but was not put on paper.  

This opinion is underscored by the following from RFC 2616, also cited
by RFC 2914:
  "A single-user client SHOULD NOT maintain
   more than 2 connections with any server or proxy."

In the same spirit, I would argue that the storage traffic would be
entitled to one share of (TCP-friendly) bandwidth per storage
connection, and not per pair of gateways, since one storage connection
more closely emulates one user activity (or maybe even more that that). 
In other words, mandating one TCP connection per pair of IP storage
gateways would be like mandating one TCP per pair of IP routers.

Of course, this is my subjective opinion with a bit of common sense, in
the absence of a clear definition of fair network usage.

The paper I referred to in my other message does not contain any
consideration to fairness, only an analytical model for TCP throughput
as a function of delay and packet loss.  A side implication is that
different TCP implementations are not fair (friendly) to each other, but
that's a long known fact.

Surely, you can modify TCP congestion control algorithm to get as much
bandwidth as N other TCPs, but that creates a new, "super-TCP" protocol
that has to be designed, verified and approved. (Observe that super-TCP
would have a different way to recover from losses that N TCPs.) 
Moreover, even if super-TCP would be used, the IP storage gateways would
still need to be aware of the identity and number of storage
connections.

Your comments, as always, are highly appreciated.	

Victor

---

From: "Victor Firoiu" <[email-masked]>
Date: Tue, 16 Jan 2001 10:53:50 -0500

Somesh,

my answers between your lines.

Victor

Somesh Gupta wrote:
> 
> Victor,
> 
> First I would like to address the second case where - based on the
> need to be fair - you raise the point that multiple connections per
> session is closer to emulating multiple users on a server than a
> single connection. That being a valuable point, is probably better
> addressed at the upper layers of the host software, where multiple
> independent sessions can be created to the same target. The argument
> made more often in favor of multiple connections is that the likelyhood
> of some connections escaping RED - or other disasters - when you have
> multiple connections is good and therefore the overall performance
> will be better.
> 

Not sure what you mean by "multiple connections per session."
Maybe I was not clear, I was arguing for one TCP connection for one
storage connection, which can give multiple TCP connectons for one pair
of IP storage gateways.  The performance considerations that I presented
in my first message regard any possible packet loss, including RED.

> However, as Steph pointed out, that skirts the issue of fairness.
> 
> My point about the modifications to congestion control algorithm was
> to point out that your work makes an interesting observation(or my
> paraphrasing of it) - that it becomes less and less possible to
> acheive the maximum possible throughput (in LFNs) as the rate and
> distances get larger and larger - and that issue should be addressed
> directly. This is true even when a single "data flow" or "application"
> might have the entire TCP bandwidth at its disposal. A modification
> hopefully will not create super TCPs i.e. would be a reasonable approx
> of current equation (flow slow long links or fast short links), but
> would provide a better choice (which should propagate to all TCPs)
> for 10G and beyond at long distances.
> 

As I was saying in some other message, modifying TCP is very problematic
for many reasons, and it is definetly outside the scope of this WG.

> There is a cost of synchronizing across multiple connections - both
> on the target and the initiator. Someone has to do the work, be it
> the host or the adapter. On the initiator side, multiple
> connections/adapters (actually their completion routine) will be updating
> a single variable MaxCmdRn and the software posting the commands will
> have to ensure that MaxCmdRn is not exceeded and has to post the commands
> to the appropriate queues (how does it know which). And that is when
> there is no stall. Assuming this informations is written/read from
> multiple processors (with the pinging of the cache lines), the cost
> will escalate rapidly with the number of processors involved.
> 
> There is similar situation on the target. It probably
> will prevent different controller boards on the target from sharing
> connections in a session (whenever one of the controller decides on a
> new increased value of MaxCmdRn, it should be propagated to the other
> controller also ??) - maybe this is not so.
> 
> That is why I am very interested in any pointers you or anyone else
> could be provide on high-speed implementation of a "single data-flow"
> across multiple connections. I am sure this has been tried in the
> past so hopefully there are some pointers to results.
> 
> Somesh
> 

I think that what you are saying is somewhat orthogonal to my points.  I
was arguing for one TCP connection for one storage connection, which is
different, simpler and more natural than bundling or anonymizing all
storage connections in one big flow and then striping a "single
data-flow" across multiple connections.  

I am not sure about other literature, I'll be looking for it myself.

Victor

---

From: "Somesh Gupta" <[email-masked]>
Date: Tue, 16 Jan 2001 04:58:47 -0800

Victor,

Somesh

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Victor Firoiu
Sent: Monday, January 15, 2001 10:27 AM
To: Stephen Bailey
Cc: [email-masked]; Franco Travostino
Subject: Re: Performance of iSCSI, FCIP and iFCP

Steph,

Victor

---

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 16 Jan 2001 10:42:15 -0800

>hammer mechanic)

The result will be synchronization and oscillation, resulting in
under-utilization of the link, unless RED is enabled. With RED
turned on, the TCP connections will not see packet drops at the
same time, so synchronizatoin will not occur and overall 
utilization will be improved.

---

From: "Somesh Gupta" <[email-masked]>
Date: Tue, 16 Jan 2001 20:43:09 -0800

Victor,

I just assumed that you were familiar with the iSCSI specification
of multiple connections per session - which is essentially as you
describe in the second last paragraph - there is one big "flow"
between the initiator and the target (called an iSCSI session)
which is then striped across multiple TCP connections - and I
think people assumed that your statements were in support of
such a proposal.

Having said that, the math still holds. And also the fact that
fundamental limits to the throughput of a single TCP connection
really do require a change to TCP congestion avoidance (esp when
mistaking link error for congestion) formulae.

If the link error rate is such that errors occur faster than time
to recover (or of that order), TCP will have to accomodate that
somehow. I don't think the TCP gurus are inflexible in this area
- PAWS, window scaling, and SACK are all example of TCP accomodating
the characteristics of the underlying network.

Somesh

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Victor Firoiu
Sent: Tuesday, January 16, 2001 7:54 AM
To: somesh_gupta
Cc: Stephen Bailey; ips; Franco Travostino
Subject: Re: Performance of iSCSI, FCIP and iFCP

Somesh,

Victor

Victor

---

From: "Victor Firoiu" <[email-masked]>
Date: Thu, 18 Jan 2001 16:33:58 -0500

Somesh,

I was looking more at the comparison between the FCIP case of multiple
FC connections (all connections between two FCIP gateways) per TCP
connection and the iFCP case of one/few TCP connections per FC
connection (thus multiple TCP connections per pair of iFCP gateways).

I think that, in general, one TCP connection per storage connection is
closer to "one/two TCP connections per user" model, and thus 
easier to defend as being "fair", compared to multiple TCP connections
per storage connection.  I agree, this is quite a fuzzy quantification,
but is as precise as RFC 2914 and 2309 get in this matter.

Regarding modifing TCP congestion control as you suggest, I am again
very skeptical.

Victor

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 18 Jan 2001 13:51:31 -0800

I am more hopeful on the point of adapting the algorithm
as environments change. The TCP gurus are conservative but
not totally inflexible.

-----Original Message-----
From: Victor Firoiu [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 1:34 PM
To: somesh_gupta
Cc: IPS Reflector
Subject: Re: Performance of iSCSI, FCIP and iFCP

Somesh,

Victor