Subject: RE: [Forces-protocol] Issue 2: Multiple channels/connections

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Fri, 12 Mar 2004 17:52:15 -0800

Jamal,

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Jamal Hadi Salim

Guys,

The goal is to have a NEW protocol. The protocol is NOT Netlink2, NOT
GRMP and NOT FACT. At least thats the approach i have been taking[1].
The goal is also to share most importantly experiences. Nothing (at
least at the IETF) is as important as an experience (the "we believe in
running code" part). This also does not rule out good ideas that havent
been implemented in my view.

[HK] Completely Agree

Hormuzd: to be fair to Weiming, you did make a very strong statement
without substantiating it. I know we have had discussions on the topic
so far, but lets recall some of the points made so we can move on.
Please repost your arguements.

[HK] Agreed, I will send out a more detailed email over the weekend.
The problem is that I have done this before and so have many others,
However Weiming keeps repeating his point of view again and again.
I don't disagree with everything that Weiming is suggesting, infact
FACT and GRMP are probably closer in this respect that netlink2...
Its just a bit tiresome sometimes...anyways I will do this again.

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Fri, 12 Mar 2004 20:47:25 -0800

Avri,

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of [email-masked]

I guess I am confused.  Hormuzd, are you saying that it is critical to 
you that use of 2 channels is a MUST even in a system that does not 
support multiple channels?

[HK] May be the terminology is confusing, lets go with connections for
now. Any OS that has a TCP/IP stack can provide multiple connections,
either TCP connections on different ports, or TCP and UDP connections,
or TCP and DCCP connections (in the near future).

As I read Weiming's suggestion, it was that one SHOULD use multiple 
channels in IP systems where they are available.  On the very close 
systems in a box, where multiple physical channels may not be 
available, it doesn't seem to make sense to require their use.

[HK] For IP systems, I believe separate D&C connections are a MUST. For
systems in a box running ForCES directly over L2, I was thinking using
something like TIPC which can provide separate connections might provide
the answer. However, this is something that we need to discuss further
and I am flexible on this.

Hope this helps clarify my previous email,
Hormuzd

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Fri, 12 Mar 2004 22:32:46 -0800

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Wang,Weiming

1. The idea to distinguish the Control packets from Data packets to have
them able to be differently processed is very useful for DoS protection
as well as for achieving different transport reliability, therefore it
MUST be included in ForCES protocol.
[HK] Agree

2. It seems no use (I suppose Jamal says little use) for us to protect
DoS by stretching out such distinguish to physical link (that is to
setup two different channels for the two kinds of packets), therefore,
it should not be described as a kind of DoS protection means in the
protocol.

[HK] I disagree on this one, having separate connections which are
congestion aware will provide robustness against DoS, not completely
protect in all situations. 

3. To setup two different channels to achieve different reliability may
be
practical, but it should not be described as a MUST item in the
protocol,
instead a "RECOMMEND" is better. This is because we must consider that
we have already required the ForCES protocol SHOULD be able to support
multiple
interconnections, and in some interconnections like some backplanes (we
should allow many kinds of backplane connections), we are actually not
very sure if a separate channel is available or how it is separated.
Actually currently, we only know that over IP network, UDP and TCP can
be used for such separation. We also don't have to explicitly list how
such separation is implemented. On the other hand, not to use lower
reliability channel for Data packet transport actually does not harm too
much. As a result, we suggest the protocol description may be look like
"If available at transport layer, ForCES protocol RECOMMENS to use two
different channels with different reliability to transport Control
packets and Data packets, with the lower reliability channel assigned to
Data packet, so as to save system resources and improve the Control
channel transport performance.

[HK] For IP, using separate C&D connections should be a MUST. However,
UDP is not the best fit for D channel, this is probably the reason why
you see no benefit of multiple connections against DoS in your
experiments. The D channel should be congestion aware. For non-IP
situation, multiple connections needs to be discussed. If we use TIPC
for non-IP situation, it might provide the answer.

---

From: [email-masked]
Date: Sat, 13 Mar 2004 14:34:06 EST

This is where application level ACKs come in handy.  We have all agreed that 
ACKs are
needed in ForCES message exchanges between CE and FEs. 

Regards,
Alex.

In a message dated 3/13/2004 11:45:09 AM Central Standard Time, [email-masked] 
writes:
Hi Avri,

On Sat, 2004-03-13 at 10:32, [email-masked] wrote:
> Hi,
> 
> I have a question.  If there is no control traffic on the data link, 
> how will you know if there is a data link failure?  Do you just wait 
> for a transport protocol time out? And is that sufficiently timely?  Or 
> will you have some other mechanism?

I think it is useful for whatever is managing the connections to know
when a link goes down. These typically (should) happen via event
notifications.
The fallback (albeit slower way) is to depend on some control data or
hearbeats.

Not sure if that answers it.

cheers,
jamal

---

From: [email-masked]
Date: Sat, 13 Mar 2004 15:13:26 EST

It is just common sense! Carry C/D on separate channels and you decouple the 
events
happening on any one of the channels from impacting on the other.  With the 
increasing
threats hackers pose to our networks today, and the increasing important role 
that the 
internet plays in our lives (especially commercially), we owe it to society 
to build as 
secure a network/network element as we can.  You should note that there is a 
movement
afoot to start making service providers liable for damage done to consumers 
as result of
security breaches in the providers' networks.  It is coming. It is only a 
matter of time.
We don't have to wait till then before we take security engineering seriously.

Regards,
Alex.

In a message dated 3/13/2004 9:08:28 AM Central Standard Time, [email-masked] 
writes:
[After reading Alexes email on how great PSTN networks are i cringed a
little. I thought i had paid for my sins already for working on SS7 
all those years, but he had to remind me;->].
The bellheads have this dogmatic view that there has to be clear
separation of control and data. Sure they will waste a gigabit pipe to
send 10kbps traffic on it. I thought these guys lost the battle 
already ;-> The design then is driven by "i am going to build this and
it will last 50 years and no replacement needed". Its anti-innovation
(although others may think its pro-robustness).
On a serious side, pick any commodity L2/3 switch blade (i know all my
companys products have this) these days and have at least one management
port thats out of band; it is not unusual to see two or even three.
Then theres the switch fabric which has many ports some of which could
be used for management.
But even given theres two classes of links (and i think people doing
CSIX may have a different view), what has this got to do with Forces
the protocol other than it being a config issue?

Let me cut the email here for readability. Go and get some coffee.

cheers,
jamal

---

From: [email-masked]
Date: Sat, 13 Mar 2004 15:49:10 EST

Hello Jon,

We manadate dual channels (one for C and the other D) to guarrantee 
interoperable
secured networking.  

Why more secure?  As soon as CE detects that FE is being DOS'd, it should be 
able
to send control signals easily to mitigate the situation. If the control 
packets are caught
behind a million D packets still in the queue, there is no chance for this to 
happen quickly.

Some have also floated the idea that in a single box environment, maybe we 
should relax
the requirement a bit.  Well,..I think as long as CE and FE are separate, the 
link/channel
between them may be a choke point if that link carries both C and D packets.  
Such C/D
packets should be conveyed over different channels (queues,..etc).

Regards,
Alex.

In a message dated 3/12/2004 5:05:36 PM Central Standard Time, 
[email-masked] writes:
A MUST would imply that we think other solutions are impossible, and 
would force
vendors to add functionality they may consider unecessary.

> But what is the implication of that on  interoperable secure 
> operation?  If different vendors
> make C and FE boxes, the chance that one would support separate C/D 
> channels while the
> other doesn't is greater.  That will force the C/FE inter-action to 
> fall back on the less secure
> single channel model. 

Less reliable, yes, but why less secure ?

> I am not sure that is  a  desireable solution given the increased 
> attention
> on security these days.

---

From: [email-masked]
Date: Sun, 14 Mar 2004 18:56:12 EST

FACT uses a heartbeat message to do CE/FE link health checks.  We can 
certainly
do the same here too.

Regards,
Alex.

In a message dated 3/14/2004 8:29:29 AM Central Standard Time, [email-masked] 
writes:
Hi,

I am not suggesting that we use LMB.  What I am suggesting is that is 
we have a separate connection that does not fate-share with the 
control, then we need to have a monitor it. I..e something that 
monitors that link.

a.

On 14 mar 2004, at 21.20, Jamal Hadi Salim wrote:

>
>> 2. Is forcing an ACK for every data packet a sufficient for 
>> determining
>> the health of a separate data channel.  I suspect it isn't and
>> 'LMP-like' mechanisms such as heartbeat may also be necessary.
>
> BTW, this topic probably should fall under the issue of Availability
> (issue #7).
> Havent looked at LMP; I may be going off (#2) topic, but here goes:
> I am familiar with other health check/availability schemes like VRRP
> which are purely dependent on hearbeats.
> The problem with VRRP is it is unidirectional. In this case i would
> think from CE to FEs; This provides knowledge to the FEs when CEs go
> down but not the other way around.
> In our case, a single message from the CE is sent on the multicast
> channel every heartbeat period. Netlink2 has ability to selectively
> request for ACKs. So on every third heartbeat message to the FEs, a
> request for an ACK is made. We arbitrarly selected number 3. However 
> the
> request for ACKs may never be triggered if the state of all FEs is 
> known
> to be healthy (example because of recent route add response from all 
> the
> FEs).
> Additionaly (we never implemented this) you could have responses from
> the FEs carry health information (such as link states).
> A smart ACK implosion scheme also controls how many FEs respond
> to reduce traffic to be processed by CE.
>

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sun, 14 Mar 2004 20:23:02 -0800

Alex,

I don't think security is the correct word to be used here. You could
probably say robust.

Hormuzd

  _____  

From: [email-masked]
[mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Saturday, March 13, 2004 12:49 PM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [Forces-protocol] Issue 2: Multiple channels/connections

Hello Jon,

secured networking.  

Why more secure?  As soon as CE detects that FE is being DOS'd, it
should be able

behind a million D packets still in the queue, there is no chance for
this to happen quickly.

Some have also floated the idea that in a single box environment, maybe
we should relax

the requirement a bit.  Well,..I think as long as CE and FE are
separate, the link/channel

between them may be a choke point if that link carries both C and D
packets.  Such C/D

 

Regards,

Alex.

	A MUST would imply that we think other solutions are impossible,
and 
	would force
	vendors to add functionality they may consider unecessary.

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sun, 14 Mar 2004 20:55:30 -0800

Here is why I think we need separate connections/channels for D&C...
(BTW, most of these reasons have been pointed out by several WG members
in the past)

1. Different reliability requirements - The C channel needs strict
reliability, CC, etc. The D channel does not need strict reliability but
CC would be needed. For example, TCP for C channel, DCCP for D channel.
(I think we will also have reliable multicast in addition to TCP for C
connections, but I will defer that discussion for issue#3)
 
2. Efficient delivery of D packets into kernel stack on CE side -
pointed out by Steve, also something which we learned from our
implementations. In one of our earlier implementations, we have also
used IP-in-IP encapsulation for this reason.

3. Performance requirements for D packets - Typically our customers have
demanded higher throughput for D channel as compared to C channel,
another reason why we need to separate the 2 transports.

4. Single reliable (TCP) connection interference in end-to-end
reliability mechanisms for D packets - This has been pointed out by
Joel, Steve and other wg members in the past.

5. Take advantage of separate queuing by OS to provide robustness
against DoS - Note, I am not saying complete protection although I
believe that using a Congestion aware D channel will provide reasonable
protection for most DoS attacks.

I believe that reasons 1-4 are strong enough for us to go with separate
C&D channels/connections using different transports. I do agree with the
point that Weiming has raised around ForCES running directly over L2,
and for this situation we may either have to just RECOMMEND using
different transports or something like TIPC...but this needs to be
discussed.

However, in case when using IP, we MUST use separate transports for C&D
channels/connections...still need to bottom out on those. Does that
sound reasonable ? (For IP, in general we should use only MUST in the
language for inter-op reasons, for L2 we can be more flexible.)

Hormuzd

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Mon, 15 Mar 2004 12:00:22 -0800

Weiming,

I would like to understand some technical reasons as to why using MUST
is not acceptable.
I haven't seen any so far. Do you not agree with the 5 reasons that I
pointed out ?
So far Jamal has been asking for good reasons and I finally got the
chance to provide those
Last night. I think they are pretty strong and no one has commented on
them so far.
I don't think this is as much about compromise, as it is about technical
reasons.
Unless folks don't agree with the reasons that I have pointed out, we
should be able 
to go with MUST for IP and RECOMMEND for non-IP...this one is still to
be discussed.

Regards
Hormuzd

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Monday, March 15, 2004 11:08 AM
To: [email-masked]
Subject: RE: [Forces-protocol] Issue 2: Multiple channels/connections

Hormuzd,

To be honest, your email has put me to a quite upset corner. From one
side, I 
just can not find new reasons that can persuade me to believe this is
vital for 
ForCES protocol so that it should use 'MUST'. From the other side, I
don't want
to let you have a feeling that I am 'pushing on something again and 
again'(though I have much to say), and moreover, it's only a few days
before 
20th (as you said, to save cycles). You should notice that what I post
for 
discussion as 'RECOMMEND' has already been considered as a possible
compromise 
among the different views of previous discussions. Actually when I check
recent 
emails and also your newly posted reasons, I have a strong feeling that
a 
word 'MAY' is the best to fit the case. You may also notice that it is
not only 
I who is objective. 

You may ask why I can either go along with 'NONE', 'MAY' or 'RECOMMEND',
it is 
just like that someone wants to give you something but you think the
thing is 
actually not vitally needed.

Since the discussion began, I have a feeling that when we meet things
that we 
can not agree on which is better, being ready to give up something is
vitally 
needed. I think I have done like this, I also appreciate Jamal has done
like 
this. In fact, the 'RECOMMEND' and even the 'MAY' has been very close to
what 
FACT supposes.

I think another solution for this issue is to leave the remained
(actually only 
a little) disagreement for further discussion, so that at that time we
may have 
more time to deal with it and based on more understanding of the problem
we may 
reach an agreement.

I'd like to hear Jamal and others' idea on this very much.

Best Regards,
Weiming

"Khosravi, Hormuzd M" <[email-masked]>:

> Weiming,
> 
> After going thru all the emails, I am still trying to understand why
you
> are opposed to having separate D&C connections using different
> transports as a MUST for IP ? Do you see some harm if we mandate this
> for IP ?
> 
> I understand your reasoning below for non-IP situation and the answer
to
> that might depend on what we decide to do for that case when ForCES is
> directly running over L2. But other than that, even you agree that
> separate C&D connections SHOULD be used for IP case. Does changing the
> SHOULD to a MUST break something ? I think using stronger language in
> the protocol is helpful for interop. Do you agree ?
> 
> The only thing that you disagree on is that using separate connections
> is useful for DoS and that is because of the results of your
> experimentation. This is very understandable to me, however you must
> recognize that a) there are other good reasons for having this b) your
> experiments were based on TCP for C channel and UDP for D channel and
> clearly I don't think UDP without some form of CC (at app level) is
very
> useful for the D channel.
> 
> I am Not trying to say that your experimental results are incorrect or
> you didn't conduct the right experiments...all I am saying is that,
> other than the DoS issue I don't think we are disagreeing on anything
> else...pls correct me if I am missing something here. If that is the
> case, then why are we spending so many cycles debating this issue ?
> 
> Regards
> Hormuzd
> 

-------------------------------------------------
This mail sent through HUCNIC Webmail system.

---

From: [email-masked]
Date: Mon, 15 Mar 2004 15:29:48 -0500

Alex,
 
We are not solving the problem by introducing one more level of indirection and layering.  As per your diagram the ForCES core will do all functional processing  and while forwarding it will ship via appropriate interface (Adaptation layer). May be it would be simpler to state what needs to be done rather than introducing addition layer...
 
As Hormuzd summarized in earlier emails, I think we should highlight all those and provide the recommendations accordingly.
 
Regards
Ramg
 
 -----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of ext Alex Audu
Sent: Monday, March 15, 2004 3:17 PM
To: Putzolu, David
Cc: [email-masked]; [email-masked]
Subject: Re: [Forces-protocol] Issue 2: Multiple channels/connections

I gave further thoughts to the dual channel proposal and how that impacts the
ForCES protocol specification.  It became clear to me that there are really
two parts to the protocol itself:  
  1) the Core protocol that is independent of the transport
   2) the Adaptation layer that resolves issues with specific trasnport mothodology.

The debate we have been having, I think has been due to the thought that the we need
to specify the core protocol to aslo adapt to one/two-channel transport links.

The is the model I see going forward:

|------------------|
|    ForCES Core |
|        (CE)           |
|------------------|
|       Adaptation   |
|         Layer         |
|------------------|
           |
           |  transport link
           |
|-----------------|
|      Adaptation   |
|        Layer         |
|-----------------|
|   ForCES Core |
|     ( FE )           |
|-----------------|         

The ForCES Core is generic and independent of the transport mode. The Adaptation
layer is a thin layer that mostly does mapping to/from trasport. This is where dual/single
channel issue is resolved.  We can also do tunneling here if we want .  This allows us to
decouple the transport issue from the core protocol.

Any comments?

Regards,
Alex.

Putzolu, David wrote:

Weiming Wang wrote> 

I think another solution for this issue is to leave the 

remained (actually only a little) disagreement for 

further discussion, so that at that time we may have 

more time to deal with it and based on more understanding of 

the problem we may reach an agreement.

I think this is the best path forward.

This seems like one of the more subtle issues and might

be better dealt with later.  I think someone else (Avri?

Jamal?) also mentioned leaving this as an open topic and

looking at some of the later issues.

Unless someone strongly objects, I suggest we take Weiming's

proposed solution and mark this as "needs further discussion"

and focus on the remaining issues, coming back to this later.

-David

[email-masked]