Subject: RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports

From: "John Hufferd" <[email-masked]>
Date: Fri, 19 Jan 2001 00:11:41 -0800

YP,
Not sure where you are going here, the purpose of a standard is to permit
interconnection with other manufactures equipment.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
(408) 256-0403, Tie: 276-0403
Internet address: [email-masked]

"Y P Cheng" <[email-masked]>@ece.cmu.edu on 01/18/2001 04:06:53 PM

Sent by:  [email-masked]

To:   "Matt Wakeley" <[email-masked]>, "IPS Reflector"
      <[email-masked]>
cc:
Subject:  RE: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports

> > But, if iFCP and FCIP use TCP encapsulation, then, we back to
> > square zero.)
> Please explain why/how you think iFCP and FCIP do not require the
> use of TCP?
>
> -Matt Wakeley

Matt,

Certainly, both drafts have proposed TCP.

However, when two iFCP or two FCIP devices talk to another of its own kind,
they have the freedom to use any protocol they wish, even SCTP, as long as
the protocol provides reliable delivery.  This is because the frames are
only received by another device of same implementation.  There is no
interoperability issue with any legacy implementations.  Now, does the
other
protocol obeys the fairness rule of Internet?  I assume it would.

Y.P.

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 19 Jan 2001 09:51:11 -0800

> interconnection with other manufactures equipment.

John,

I guess that I failed miserably in communicating my thinking to this WG.

Most people if not all in this WG view iSCSI, iFCP, and FCIP as protocols to
be implemented by either hardware or software or the combination of.
Because their implementations could be software that takes advantage of
existing TCP stacks in different OS's that are proven working, this WG tries
real hard to fit the new protocols into the existing implementations.

In the world where I live, iSCSI, iFCP, and FCIP will be implemented in a
box or an adapter running RTOS or microcode with fresh new implementations.
While it is essential to intemperate with the world that runs the existing
TCP implementations, nothing prohibits the box and adapter to interoperate
with each other running in "fast mode" in correct TCP packets as long as
they obey the Internet fairness rule without creating so called "Super TCP".
In my adapter, I don't have to live with any old TCP implementations.  I
asked often how do we streaming data on a 10 Gb/sec network with roundtrip
time over 100 milliseconds?  I would like to hear discussions providing
answers to the above question.  The statement "the TCP implementation
guarantees in-order delivery and retries lost packets and has the necessary
flow control and congestion avoidance" does not answer the question for me.

If everyone agrees that this group can put iSCSI, iFCP, and FCIP together by
assuming the current TCP implementations having all the solutions, please
let me know.

Y.P.

---

From: David Robinson <[email-masked]>
Date: Fri, 19 Jan 2001 14:10:31 -0800

> flow control and congestion avoidance" does not answer the question for me.

In general I have not seen people in this WG constraining the design
based
on TCP implementations, in fact some have been very abstract in their
comments
and referring to what has been proven in theory (if not limited test
implementations)
that does not reflect current widely deployed implementations.

If I read you correctly, you are asserting that there is a fundemental
problem
in the design on the TCP *protocol* which prevents it from taking
advantage
of a 10G/100ms network. If so what exactly do you see as a problem?
Since we
cannot (ips WG) cannot change TCP how should an IPS protocol work around
this
problem while still being friendly with other protocols?
 
> assuming the current TCP implementations having all the solutions, please

Conversely, if you feel that this group is designing to the TCP
implementations
instead of the protocol, please let us know.

	-David

---

From: Matt Wakeley <[email-masked]>
Date: Fri, 19 Jan 2001 16:16:24 -0800

> of a 10G/100ms network. If so what exactly do you see as a problem?

That's easy - The lack of PDU framing in TCP.

-Matt

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 19 Jan 2001 17:45:36 -0800

> problem in the design on the TCP *protocol* which prevents it from taking
> advantage of a 10G/100ms network. If so what exactly do you see as a
problem?

I have been very careful in stating the problem is in implementation.  The
protocol is fine -- as the more and more I learn about the TCP options.  I
think more new options such as framing, marking, NCK, SACK, and RDMA can be
defined and placed in login negotiation to allow the TCP protocol to run
well in 10G/100ms network.

> Since we cannot (ips WG) cannot change TCP how should an IPS protocol
> work around this problem while still being friendly with other protocols?

I do believe many people in this group do understand the problems.  It new
options that facilitate the 10G/100ms network are negotiated, we can still
be friendly with older implementation which does not support the new
options.  In such case, we just run slow.

> implementations instead of the protocol, please let us know.
>
> 	-David

I did sense that some people in this group were worry about compatibility
with older implementations and reluctant to discuss or add new TCP options.
In general, I do believe lots of people are quite up to speed, pun
unintended.  My statements made in herein previous postings were saying that
two iSCSI or FCIP adapters of same kind -- with same TCP implementations --
should be able to run the new options.

Y.P.

---

From: "Murali Rajagopal" <[email-masked]>
Date: Mon, 22 Jan 2001 20:58:08 -0800

A point of clarification on FCIP Ordering and multiple TCP connections:

The FCIP draft does not preclude multiple TCP connections.
Somesh is right in pointing that we could result in out-of-order if we
have more than one TCP connections between same two FCIP Gateways.

However, an FCIP gateway (say FCIP-A) can carry on simultaneous TCP
connections
say with FCIP-B and FCIP-C gateways without the danger of the out-of-order
issue.

Out-of-order is therotically possible even with FC Switch fabrics running a
dynamic routing protocol such as FSPF, although in practice FC switches
vendors seldom run into this condition.

In summary, multiple TCP connections is not precluded but a solution is
specified in the current FCIP draft. The authors of the FCIP draft will take
an action to clarify this in the next version.

Regards,

Murali Rajagopal
LightSand Communication

---

From: Santosh Rao <[email-masked]>
Date: Mon, 22 Jan 2001 21:46:31 -0800 (PST)

> vendors seldom run into this condition.

A FC switch that guarantees in-order delivery by ACCing an FLOGI which is
requesting sequential delivery and then causes out-of-order traffic is
violating the FC-FS and FC-FLA standards. Out-of-order issues are avoided
in FC by requesting in-order delivery in the class sp. params of FLOGI.

- Santosh

-- 
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : [email-masked]
Phone : 408-447-3751
#################################

---

From: "Venkat Rangan" <[email-masked]>
Date: Tue, 23 Jan 2001 08:29:22 -0800

> vendors seldom run into this condition.

FC switches do not run into this condition by taking special steps. One
approach is to institute a new route (in response to a link going down)
after waiting for at least R_A_TOV time. This way, there are no frames
within that time window whose route may change. It does mean less
responsiveness to link failures, but it preserves the in-order semantics
negotiated at FLOGI.

Venkat Rangan
Rhapsody Networks Inc.
http://www.rhapsodynetworks.com

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Murali Rajagopal
Sent: Monday, January 22, 2001 8:58 PM
To: David Robinson; IPS Reflector
Subject: Re: FCIP/iFCP : Guarantee In-Order delivery for FC N/NL_ports

Regards,

Murali Rajagopal
LightSand Communication

---

From: "Y P Cheng" <[email-masked]>
Date: Tue, 23 Jan 2001 09:19:48 -0800

> have more than one TCP connections between same two FCIP Gateways.

Even with just one TCP connection, when frames travel through the "IP
cloud", don't we risk out-of-order and duplicated frames at the other end?
For multiple connections, it is a sure thing.

> connections say with FCIP-B and FCIP-C gateways without the
> danger of the out-of-order issue.

FCIP-A can take responsibility for in-order delivery of outgoing frames.
But, there is no guarantee for in-order and non-duplicated reception on
either FCIP-B or FCIP-C.

> Out-of-order is therotically possible even with FC Switch fabrics
> running a dynamic routing protocol such as FSPF, although in practice
> FC switches vendors seldom run into this condition.

I am not an expert in implementing switches.  But, in talking to both
Finisar and Brocade, I was told that the switches do see this problem and
ensure in-order delivery by buffering the frames.

> specified in the current FCIP draft. The authors of the FCIP
> draft will take an action to clarify this in the next version.
>
> Murali Rajagopal
> LightSand Communication

I always thought that FCIP can count on an E-port for frame buffering and
ordered delivery.  If not, then, FCIP even using TCP must address the same
problem of moving large amount of data on a 10Gb/100ms network like that of
iFCP and iSCSI.  One just can not assume that the TCP without help of RDMA
header and frame marker will run fast on a 10Gb/100ms network.  I thought
FCIP by leverage the effort in the fibre channel switches gets away with the
out-of-order reception problem.  But, I am not a switch expert.  Just my 2
cents' worth.

Y.P. Cheng, ConnectCom Solutions.

---

From: Matt Wakeley <[email-masked]>
Date: Tue, 23 Jan 2001 11:04:08 -0800

YP,

You are correct that most people in *this* group understand the problems. 
However, *this* group cannot invent new TCP options.  That is controlled by
the end2end group, and *they* do not understand the problems (or if they do,
they are unwilling to help solve them).

-Matt

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 23 Jan 2001 11:12:02 -0800

Murali,

To allow detection of stale frames due from a disrupted IP network and to
maintaining ordering on multiple connections, a local time-stamp that
resolves to the packet rate would solve both these issues.  To ensure
ordered delivery, especially with multiple connections, both situations must
be accommodated.  The R_A_TOV fabric time-outs are meaningless unless there
is a means to police stale frames finding their way across IP space.  A
locally generated time-stamp would allow a means to both ensure ordering
over multiple connections as well as a means of rejecting stale frames held
up by some IP disruption otherwise frame ordering is not assured.

Doug

> running a
> draft will take
> > > In the world where I live, iSCSI, iFCP, and FCIP will be
> implemented in
> a
> > > flow control and congestion avoidance" does not answer the
> question for
> me.
> >

---

From: "Y P Cheng" <[email-masked]>
Date: Tue, 23 Jan 2001 13:15:08 -0800

> responsiveness to link failures, but it preserves the in-order semantics

If I read you correctly, even switches do not buffer the frames to ensure
in-order delivery. Instead it relies on R_A_TOV timeout.  So, it always
passes a sequence of in-order incoming frames quickly to an ISL.  If the ISL
fails, it waits until R_A_TOV timeout that forces all unfinished sequences
being thrown away, then, allocates a new link.  Does the switch perform any
checking for out-of-order incoming frames?  Or, is it simply
garbage-in-garbage-out?

Now, if all iFCP, iSCSI, and FCIP must rely on TCP implementation for
ordered delivery, I am the poor guy trying real hard to put TCP in the
microcode of the I/O interface chips in every host, target, and switch port.
If I solve the performance problem of TCP on a 10Gb/100Ms network without
requiring 50 megabytes of reassembly buffer for every single port, I can
corner this market?  This is becomes more interesting every minute.  Thank
God, that should a standard that allows me to write the microcode for this
chip only once.

Other switch experts care to comment?