Subject: [Forces-protocol] FW: ForCES Protocol Analysis Design Team Administrivia

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:32:24 -0800

(Resending to add to archive) 

-----Original Message-----
From: Khosravi, Hormuzd M 
Sent: Wednesday, February 25, 2004 6:39 PM
To: Putzolu, David; [email-masked]'; [email-masked]';
[email-masked]'; [email-masked]'; 'Steven
Blake'; 'Robert Haas'; [email-masked]'; [email-masked]';
'Wang,Weiming'; 'Patrick Droz'; 'jeff pickering'; [email-masked]';
Deleganes, Ellen M; [email-masked]'; [email-masked]'
Subject: RE: ForCES Protocol Analysis Design Team Administrivia

Hi All,

I am personally fine with the proposal by Jamal to list 
any features the protocol authors cant live without. 

Jamal, are there any other ground rules you have in mind ?
Why don't you send a list of features which you would like 
to see and we can start the discussion from there ?

Thanks
Hormuzd

-----Original Message-----
From: Putzolu, David 
Sent: Friday, February 20, 2004 9:49 AM
To: [email-masked]; Khosravi, Hormuzd M; [email-masked];
[email-masked]; [email-masked]; Steven Blake;
Robert Haas; [email-masked]; [email-masked]; Wang,Weiming;
Patrick Droz; jeff pickering; [email-masked]; Deleganes, Ellen M;
[email-masked]; [email-masked]'; Putzolu, David
Subject: ForCES Protocol Analysis Design Team Administrivia

All,

This email is going out to all of the current design
team volunteers.  In order to avoid problems with
long to/cc lists, I've asked the IETF to setup a temporary
reflector for us to use, and will subscribe all of us to
it once it is setup.  With any luck it will be ready
by Monday or Tuesday.

If you know of someone who volunteered for the team, but
is not on the recipient list of this email, please have
them contact me.

This team is chartered with examining the aspects of the
currently existing protocol proposals, seeing what common
ground there is, and determining whether the common ground
is sufficient for coming up with a single merged proposal.
The design team is to report out to the ForCES list by
March 20, 2004 (one month from today).

Below is an email that some of you received.  I've included 
it for the benefit of those who did not receive it.

Given the concern with word docs I think either emails
or (when necessary) PDF are reasonable formats.  I've
converted a word doc previously sent out by Hormuzd to 
PDF format and put it on the web for those who missed it:
http://www.sstanamera.com/~forces/proto/ForCESProtoDiscussion1.pdf

Below are some ground rules proposed by Jamal.  Are 
people comfortable with them/are there any additions
that should be made?

Cheers,
David

> -----Original Message-----
> From: Jamal Hadi Salim [mailto:[email-masked]] 
> Sent: Friday, February 20, 2004 6:59 AM
> To: Khosravi, Hormuzd M
> Cc: Steven Blake; Robert Haas; Jon Maloy; 
> [email-masked]; [email-masked]; 
> [email-masked]; Wang,Weiming; Putzolu, David; Patrick 
> Droz; jeff pickering
> Subject: Re: ForCES protocol merger
> 
> 
> I have added to the list Jeff Pickering who is a new couathor for us.
> 
> To start i will say that the netlink2 people are interested 
> in being on
> this boat. For this reason we have not releasd an updated netlink2
> draft.
> 
> I dont like the brainstorming document as a starter.
> Brainstorming can be done via emails or concals. And please no word
> docs.
> Our attempt in Mineapolis was in the wrong direction because there
> were no ground rules on how to proceed. We need to agree on 
> some ground
> rules first otherwise we are going to end in some deadlock. 
> I am going to suggest one which i came up with after the last
> meeting:
> 
> As in any conflict, people should be open to giving up some things in
> order to reach a compromise. I can understand that this may not be 
> 100% technical driven; what i am hoping for is to have less than 5%
> policitical drive.
> This is the main reason i suggested that each protocol 
> authors list what
> they cant live without. Lets iron that out so we can reduce conflict.
> For each item draft authors cant live without, the how and 
> justification
> of why must be listed. 
> In the IETF typical approach of such "cant live without mechanisms"
> would be based on implementation and deployment experience. I have to
> say i learnt a few things on implementing netlink2 that i would have
> previously disagreed with - so that approach is useful.
> 
> I will try to be open hoping it is the correct thing to do to avoid
> conflict:
> Speaking from a political/bussiness side for myself: we have 
> implemented
> the full netlink2 which means we have invested time and 
> resources in it
> (and infact have deployed it in one site). However,
> we are willing to throw all that away provided there will be progress
> here which is not driven by politics rather by a technical drive.
> 
> So lets start with a list of ground rules.
> 
> cheers,
> jamal
> 
> 

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:32:47 -0800

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, February 25, 2004 7:22 PM
To: Khosravi, Hormuzd M
Cc: [email-masked]; [email-masked]; [email-masked];
Deleganes, Ellen M; [email-masked]; Steven Blake;
[email-masked]; Patrick Droz; [email-masked];
Robert Haas; [email-masked]; jeff pickering; Wang,Weiming; Putzolu,
David; [email-masked]
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

Hi,

I too think this is a good start.  I am also wondering whether there is 
any value in the non-authors writing up a list of the features they 
value from each of the 3 proposals.

a.

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:32:55 -0800

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, February 25, 2004 7:29 PM
To: Putzolu, David; Patrick Droz
Cc: Jamal Hadi Salim; Steven Blake; jeff pickering; Deleganes, Ellen M;
Weiming Wang; <[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>;
<[email-masked]>; Robert Haas; <[email-masked]>;
Khosravi, Hormuzd M; <[email-masked]>
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

On another topic:

Is it worth setting up a mailing list for this effort?  I could 
probably set it up on one of the servers I use if there is a desire for 
that to be done.

a.

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:32:36 -0800

-----Original Message-----
From: Wang,Weiming [mailto:[email-masked]] 
Sent: Wednesday, February 25, 2004 12:35 AM
To: avri; Alex Audu; Chaoping Wu; Deleganes, Ellen M; Dong, Ligang; Guo,
Yunfei; Jamal Hadi Salim; Jeff Pickering; Jon Maloy; Khosravi, Hormuzd
M; Patrick Droz; Putzolu, David; Robert Haas; Spyros Denazis; Steve
Blake; Wang Weiming; Wang, Guangming
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

Hi David,

Regarding the proposed basic principles, we (GRMP group) have some
comments for  principles item 7&8:
--------
7.      The ForCES protocol should support separate channels for Control
and Data messages. The data 
channel carries the routing/control protocol packets such as RIP, OSPF
packets, between the CEs and FEs. 
These packets are unique in most cases for a particular CE:FE pair and
hence the data channel will use unicast 
connections between the CEs and FEs. Also, the data channel does not
have to be strictly reliable but does 
need to provide flow/congestion control to protect against DoS attacks.

8.      The control channel must provide strict reliability
(non-corruption, no loss, no reordering, timeliness) at the 
transport level in accordance with the RFC 3654. In addition, ForCES
protocol level responses/acknowledgements 
are also needed from FE to CE to learn the results of
configuration/control messages.
--------

First of all, we agree that to distinguish ForCES messages into control
messages and data (redirect) messages are 
essential for DoS protection scheme. Then, we argue that to separate the
two kinds of messages into two different 
transport channels with the data channel being appointed to a lower
reliable level (such as UDP, compared with TCP) 
may not be a good enough scheme for such purpose. The reasons are:

1. The data (redirect) ForCES messages seem have no reason to be
appointed to a lower reliability, compared with 
control channel messages.  We may needn't  to argue how important some
of the redirected messages are for a 
router (an NE), we just argue that  the reliability for ForCES data
message transport should not be lower than that of the 
connection over which the routing/control protocol packets  are
transported. An example is the BGP protocol messages, 
which have already been assigned to use TCP instead of UDP for the
transports. We think the communication between CEs 
and FEs (which is inside an NE) has no reason to greatly induce extra
performance loss other than that from outside of 
an NE for such routing protocol packets. By saying this, we don't mean
all ForCES messages should not be loaded over UDP, 
we just propose that it's may be too early to distinguish the redirect
messages from other configuration messages inside ForCES 
protocol specifications.

2. Generally speaking, separated C channel and D channel are still put
over the same physical link (or else there is almost 
no need to protect DoS).  Therefore,  such a separation itself does not
provide any DoS protection, instead, the protection is 
obtained by the rate control of D channel. We argue that  such a rate
control actually can be obtained at the ForCES message 
layer instead of at the message transport layer. Some of our experiment
results (which will be presented at the IETF meeting) 
have shown this feasibility.  Moreover, a separation at transport layer
surely will complicate the CE-FE association and the 
followed message transport management. 

3.  We have also noticed a phenomenon during our experiments, which we
may need to pay some attention. I recalled when 
FACT first presented the D and C channel idea,  there was a question
worrying about the asyncronization between D and C channel.  
Our experiments have shown that,  although there is no strict
asyncronization problem between D and C channel owing to the data 
independence, there is a possibility that D channel packets may be
seriously delayed and even blocked owing to the busy traffic at 
C channel if D packets are separated from C packets. The busy traffic at
C channel may happen when CE queries FE for something 
and FE dumps quite a huge number of messages (such as routing table
items.)  Such a delay or blocking may be counted in seconds, 
resulted in very harmful effect on the users of redirected packets.  As
a result, we propose that a moderate rate control for C channel 
be also necessary. 

4. To explicitly separate D and C channel also consumes more resources,
because a resources for D channel have to be reserved. 
Instead, by separating D and C packets at the ForCES protocol layer and
uniting them again at the transport layer, we may able to 
make a scheduling discipline by which the D and C packets can share the
link resources effectively. We are trying to construct such 
a scheduling scheme, and it seems quite practical to so.

To summarize, we propose that the 7&8 basic principles only mention the
separation of D and C packets for DoS protection currently, 
leaving some space for further discussions. The discussions may be like:

1. Should such a separation be put at the ForCES message layer or at the
transport layer?  Which is better?
2. Whether united scheduling for D and C packets are necessary?
3. Should D packets be distinguished from C packets for its transport
reliability in ForCES protocol so as to assign a different transport
means?

Regarding the protocol merge proposal, we'd like to present some of our
thinking in a few words:

Firstly, as a new comer to IETF work, the GRMP group surely welcome such
a merge proposal, sincerely hoping an all merits in 
one draft can be achieved for ForCES WG, and at the same time hoping to
learn from other candidate authors, whom we have 
already been respecting to for their rich knowledge and excellent
contributions to IETF and other fields. Secondly, we believe 
that the interaction among the candidate drafts are very helpful for the
final ForCES protocol. Actually there already has been such 
interaction since the ForCES protocol evaluation process began, which
has already made individual candidate drafts ever improved.  
Thirdly, to make the process successful, we believe that a nice
atmosphere is very important. In such an atmosphere, everyone has 
a peaceable state of mind, precisely speaking, a mind ready to give up
and accept, and individuals' work can be fully respected. 
Our past experience has made us believe that it is almost impossible to
reach a compromise among group members where some 
think of their work much outstanding to others. Lastly, as Jamal has
mentioned, some ground rules is important so as to eliminate 
conflicts whenever there are. 

Regarding the merge action way, we agree candidates to propose their
"can't live without" first. Along with this, we also suggest 
candidates propose "what they think the contents in their drafts are
useful for final protocol and hope able to be put in it". This useful 
contents are a little different from the "features of the drafts",
because we suppose the contents proposed should be conservatively 
considered by presenters to be able to merge with other contents. 

Thank you.

Beset Regards,
Weiming Wang

P.S.: To avoid unexpected alignment for the email, I have also made a
pdf format file in the attachment to ease reading. Thank you.

----- Original Message ----- 
From: "Putzolu, David" <[email-masked]>
To: <[email-masked]>; "Khosravi, Hormuzd M"
<[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>; "Steven
Blake" <[email-masked]>; "Robert Haas" <[email-masked]>;
<[email-masked]>; <[email-masked]>; "Wang,Weiming"
<[email-masked]>; "Patrick Droz" <[email-masked]>; "jeff
pickering" <[email-masked]>; <[email-masked]>; "Deleganes,
Ellen M" <[email-masked]>; <[email-masked]>;
<[email-masked]>; "Putzolu, David"
<[email-masked]>
Sent: Saturday, February 21, 2004 1:48 AM
Subject: ForCES Protocol Analysis Design Team Administrivia

All,

Cheers,
David

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:33:04 -0800

-----Original Message-----
From: Steven Blake [mailto:[email-masked]] 
Sent: Thursday, February 26, 2004 6:57 AM
To: Wang,Weiming
Cc: avri; Alex Audu; Chaoping Wu; Deleganes, Ellen M; Dong, Ligang; Guo,
Yunfei; Jamal Hadi Salim; Jeff Pickering; Jon Maloy; Khosravi, Hormuzd
M; Patrick Droz; Putzolu, David; Robert Haas; Spyros Denazis; Wang,
Guangming; Zsolt Haraszti
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

> protocol specifications.

The IP network that delivers the data traffic to the NE is inherently
unreliable; therefore overall reliability cannot be improved by applying
strict reliability to the D channel.  In fact, the D channel traffic
(e.g., BGP over TCP) that requires end-to-end reliability will already
employ transport-layer reliability.  Requiring D channel reliability
might improve performance somewhat; on the other hand, the D channel
reliability mechanisms might interact badly with TCP.

[snip]

>    reliability in ForCES protocol so as to assign a different
transport means?'

I'm not sure I understand your questions, but I am sure of this:
I want the D channel traffic (IP packets and occassionally raw L2
frames) to be delivered to the bottom of my CE's networking stack with
sufficient metadata to indicate the FE and the interface it arrived on. 
I want my CE networking stack (which in my case lives in an OS kernel
and not in user space) to treat those packets as if they had arrived to
the stack on the interface indicated.  If you transmit the D channel
traffic over a complicated transport such as TCP, you will severely
complicate the ability to treat the traffic as locally delivered
packets; it will be impossible to do this with off-the-shelf kernels
without significant modification.  So I believe that the ForCES D
channel transport needs to be simple enough to implement in a driver,
and/or should use IP tunneling (almost any interesting stack can
demultiplex tunneled traffic onto other interfaces).  Forcing ForCES CEs
to run another networking stack in user space is not an option, IMO.

For the C channel, I expect my CE components to usually run in user
space; use of any IP transport protocol that can be found in an OS
networking stack (or raw frames over L2 + a user-space transport) is
acceptable for this (although there are other reasons to prefer one
transport over another).

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven L. Blake               <[email-masked]>
Ericsson IP Infrastructure                +1 919-472-9913

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:33:13 -0800

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
Sent: Thursday, February 26, 2004 8:27 AM
To: Khosravi, Hormuzd M
Cc: Putzolu, David; [email-masked]; [email-masked];
[email-masked]; Steve Blake; Robert Haas;
[email-masked]; [email-masked]; Wang,Weiming; Patrick Droz;
jeff pickering; [email-masked]; Deleganes, Ellen M; [email-masked];
[email-masked]
Subject: RE: ForCES Protocol Analysis Design Team Administrivia

> to see and we can start the discussion from there ?

I think the most important approach is to look at this as
an opportunity (for all of us) to create a very useful protocol.
Seems to me we all have good experiences that we can put to use.
There will be differences in opionions, but the hope is that there
are differences in technical (not political) opinion. Lets all be ready
to make technical compromises for the sake of a good protocol.
My thoughts are we should be doing some sort of interop by San Diego
time. So we dont have much time to waste on quibbling.

As a starter, lets share some experiences. We can then move towards
defininig what would be useful. We can then debate about different
aspects of what mechanisms fit better and why.

So i am attaching a text file which describes some of the experiences we
have gathered. If we can get something from the two other protocols 
then it would be extremely a good start. 

Thoughts?

cheers,
jamal

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:33:19 -0800

-----Original Message-----
From: Alex Audu [mailto:[email-masked]] 
Sent: Thursday, February 26, 2004 9:23 AM
To: [email-masked]
Cc: Khosravi, Hormuzd M; [email-masked]; [email-masked];

Hello Avri,

I guess this (non-authors favorite features list)  would be a good input

to the team.

Thanks,
Alex.

[email-masked] wrote:

> Hi,
>
> I too think this is a good start.  I am also wondering whether there 
> is any value in the non-authors writing up a list of the features they

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:33:25 -0800

_____  

From: Alex Audu [mailto:[email-masked]] 
Sent: Thursday, February 26, 2004 10:07 AM
To: Wang,Weiming
Cc: avri; Chaoping Wu; Deleganes, Ellen M; Dong, Ligang; Guo, Yunfei;
Jamal Hadi Salim; Jeff Pickering; Jon Maloy; Khosravi, Hormuzd M;
Patrick Droz; Putzolu, David; Robert Haas; Spyros Denazis; Steve Blake;
Wang, Guangming
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

Hello Weiming,

I'll address some of the issues you raised.

1) Seperate Data Redirect and Control channels/ relative importance of
Control vs Redirect info
    It is easy to see that the redirect data can be manipulated by an
outside source (External to 
     the NE).  In the case of such a source being malicious, it is
necessary to give priority to 
     the control data to maintain integrity of the NE.
      
      You sighted BGP experience.  Indeed, the realization that it is
safer to separate externally
       generated control data like redirect-data from internally
generated control data grew out
       of  experience gleamed from past DOS attacks on BGP.

2)  C channel and D channel are on the same physical link.   For DOS
survivability, you need 
      to seperate then somehow. If you use SCTP, you can put them on
separate streams on
       the same link. That will work.  But what will be best is a
non-reliable congestion aware
      stream/link for  D-channel and a reliable stream/link for
C-channel.
       Rate controlling is important. But if  you need to allow your C
information to flow freely
       to adaptively effect counter-meassures if the NE is being DOS'd.

3) Delay of D packets by  delay of C packets.   This sould not happen if
your channels are
     independent. And  D packets may dictate what C packets get sent to
the FE.  The
     problem might be related to the way your real-time processes are
running the D and C 
     queues.  You can play with thread priorities,..etc to resolve this.

4) C/D separation resource cost:   This is not as expensive as the cost
paid when DOS
     attacks occur.  Besides, hardware resources are getting cheaper by
the day,..and 
     processors are getting faster too.  

General comments. May be a way to go forward is to make it a SHOULD to
implement
 (on seperate D/C channels) and leave the rest up to implementation,...I
am not sure. Any comments?

Regards,
Alex.

Wang,Weiming wrote:

	Hi David,
	
	Regarding the proposed basic principles, we (GRMP group) have
some comments for  principles item 7&8:
	--------
	7.      The ForCES protocol should support separate channels for
Control and Data messages. The data 
	channel carries the routing/control protocol packets such as
RIP, OSPF packets, between the CEs and FEs. 
	These packets are unique in most cases for a particular CE:FE
pair and hence the data channel will use unicast 
	connections between the CEs and FEs. Also, the data channel does
not have to be strictly reliable but does 
	need to provide flow/congestion control to protect against DoS
attacks.
	
	8.      The control channel must provide strict reliability
(non-corruption, no loss, no reordering, timeliness) at the 
	transport level in accordance with the RFC 3654. In addition,
ForCES protocol level responses/acknowledgements 
	are also needed from FE to CE to learn the results of
configuration/control messages.
	--------
	
	First of all, we agree that to distinguish ForCES messages into
control messages and data (redirect) messages are 
	essential for DoS protection scheme. Then, we argue that to
separate the two kinds of messages into two different 
	transport channels with the data channel being appointed to a
lower reliable level (such as UDP, compared with TCP) 
	may not be a good enough scheme for such purpose. The reasons
are:
	
	1. The data (redirect) ForCES messages seem have no reason to be
appointed to a lower reliability, compared with 
	control channel messages.  We may needn't  to argue how
important some of the redirected messages are for a 
	router (an NE), we just argue that  the reliability for ForCES
data message transport should not be lower than that of the 
	connection over which the routing/control protocol packets  are
transported. An example is the BGP protocol messages, 
	which have already been assigned to use TCP instead of UDP for
the transports. We think the communication between CEs 
	and FEs (which is inside an NE) has no reason to greatly induce
extra performance loss other than that from outside of 
	an NE for such routing protocol packets. By saying this, we
don't mean all ForCES messages should not be loaded over UDP, 
	we just propose that it's may be too early to distinguish the
redirect messages from other configuration messages inside ForCES 
	protocol specifications.
	
	2. Generally speaking, separated C channel and D channel are
still put over the same physical link (or else there is almost 
	no need to protect DoS).  Therefore,  such a separation itself
does not provide any DoS protection, instead, the protection is 
	obtained by the rate control of D channel. We argue that  such a
rate control actually can be obtained at the ForCES message 
	layer instead of at the message transport layer. Some of our
experiment results (which will be presented at the IETF meeting) 
	have shown this feasibility.  Moreover, a separation at
transport layer surely will complicate the CE-FE association and the 
	followed message transport management. 
	
	3.  We have also noticed a phenomenon during our experiments,
which we may need to pay some attention. I recalled when 
	FACT first presented the D and C channel idea,  there was a
question worrying about the asyncronization between D and C channel.  
	Our experiments have shown that,  although there is no strict
asyncronization problem between D and C channel owing to the data 
	independence, there is a possibility that D channel packets may
be seriously delayed and even blocked owing to the busy traffic at 
	C channel if D packets are separated from C packets. The busy
traffic at C channel may happen when CE queries FE for something 
	and FE dumps quite a huge number of messages (such as routing
table items.)  Such a delay or blocking may be counted in seconds, 
	resulted in very harmful effect on the users of redirected
packets.  As a result, we propose that a moderate rate control for C
channel 
	be also necessary. 
	
	4. To explicitly separate D and C channel also consumes more
resources, because a resources for D channel have to be reserved. 
	Instead, by separating D and C packets at the ForCES protocol
layer and uniting them again at the transport layer, we may able to 
	make a scheduling discipline by which the D and C packets can
share the link resources effectively. We are trying to construct such 
	a scheduling scheme, and it seems quite practical to so.
	
	To summarize, we propose that the 7&8 basic principles only
mention the separation of D and C packets for DoS protection currently, 
	leaving some space for further discussions. The discussions may
be like: 
	
	1. Should such a separation be put at the ForCES message layer
or at the transport layer?  Which is better?
	2. Whether united scheduling for D and C packets are necessary?
	3. Should D packets be distinguished from C packets for its
transport reliability in ForCES protocol so as to assign a different
transport means?
	
	Regarding the protocol merge proposal, we'd like to present some
of our thinking in a few words:
	
	Firstly, as a new comer to IETF work, the GRMP group surely
welcome such a merge proposal, sincerely hoping an all merits in 
	one draft can be achieved for ForCES WG, and at the same time
hoping to learn from other candidate authors, whom we have 
	already been respecting to for their rich knowledge and
excellent contributions to IETF and other fields. Secondly, we believe 
	that the interaction among the candidate drafts are very helpful
for the final ForCES protocol. Actually there already has been such 
	interaction since the ForCES protocol evaluation process began,
which has already made individual candidate drafts ever improved.  
	Thirdly, to make the process successful, we believe that a nice
atmosphere is very important. In such an atmosphere, everyone has 
	a peaceable state of mind, precisely speaking, a mind ready to
give up and accept, and individuals' work can be fully respected. 
	Our past experience has made us believe that it is almost
impossible to reach a compromise among group members where some 
	think of their work much outstanding to others. Lastly, as Jamal
has mentioned, some ground rules is important so as to eliminate 
	conflicts whenever there are. 
	
	Regarding the merge action way, we agree candidates to propose
their "can't live without" first. Along with this, we also suggest 
	candidates propose "what they think the contents in their drafts
are useful for final protocol and hope able to be put in it". This
useful 
	contents are a little different from the "features of the
drafts", because we suppose the contents proposed should be
conservatively 
	considered by presenters to be able to merge with other
contents. 
	
	Thank you.
	
	Beset Regards,
	Weiming Wang
	
	P.S.: To avoid unexpected alignment for the email, I have also
made a pdf format file in the attachment to ease reading. Thank you.
	
	----- Original Message ----- 
	From: "Putzolu, David" <[email-masked]>
<mailto:[email-masked]> 
	To: <[email-masked]> <mailto:[email-masked]> ; "Khosravi, Hormuzd
M" <[email-masked]> <mailto:[email-masked]>
; <[email-masked]> <mailto:[email-masked]> ;
<[email-masked]> <mailto:[email-masked]> ;
<[email-masked]> <mailto:[email-masked]> ;
"Steven Blake" <[email-masked]> <mailto:[email-masked]>
; "Robert Haas" <[email-masked]> <mailto:[email-masked]> ;
<[email-masked]> <mailto:[email-masked]> ; <alex.audu@alcat
	el.com> <mailto:[email-masked]> ; "Wang,Weiming"
<[email-masked]> <mailto:[email-masked]> ; "Patrick
Droz" <[email-masked]> <mailto:[email-masked]> ; "jeff
pickering" <[email-masked]>
<mailto:[email-masked]> ; <[email-masked]>
<mailto:[email-masked]> ; "Deleganes, Ellen M"
<[email-masked]> <mailto:[email-masked]> ;
<[email-masked]> <mailto:[email-masked]> ; <[email-masked]>
<mailto:[email-masked]> ; "Putzolu, David"
<[email-masked]> <mailto:[email-masked]> 
	Sent: Saturday, February 21, 2004 1:48 AM
	Subject: ForCES Protocol Analysis Design Team Administrivia

	All,

	

	

	Cheers,
	David

		-----Original Message-----
		From: Jamal Hadi Salim [mailto:[email-masked]] 
		Sent: Friday, February 20, 2004 6:59 AM
		To: Khosravi, Hormuzd M
		Cc: Steven Blake; Robert Haas; Jon Maloy; 
		[email-masked]; [email-masked]; 
		[email-masked]; Wang,Weiming; Putzolu, David;
Patrick 
		Droz; jeff pickering

		I have added to the list Jeff Pickering who is a new
couathor for us.
		
		To start i will say that the netlink2 people are
interested 
		in being on
		this boat. For this reason we have not releasd an
updated netlink2
		draft.
		
		I dont like the brainstorming document as a starter.
		Brainstorming can be done via emails or concals. And
please no word
		docs.
		Our attempt in Mineapolis was in the wrong direction
because there
		were no ground rules on how to proceed. We need to agree
on 
		some ground
		rules first otherwise we are going to end in some
deadlock. 
		I am going to suggest one which i came up with after the
last
		meeting:
		
		As in any conflict, people should be open to giving up
some things in
		order to reach a compromise. I can understand that this
may not be 
		100% technical driven; what i am hoping for is to have
less than 5%
		policitical drive.
		This is the main reason i suggested that each protocol 
		authors list what
		they cant live without. Lets iron that out so we can
reduce conflict.
		For each item draft authors cant live without, the how
and 
		justification
		of why must be listed. 
		In the IETF typical approach of such "cant live without
mechanisms"
		would be based on implementation and deployment
experience. I have to
		say i learnt a few things on implementing netlink2 that
i would have
		previously disagreed with - so that approach is useful.
		
		I will try to be open hoping it is the correct thing to
do to avoid
		conflict:
		Speaking from a political/bussiness side for myself: we
have 
		implemented
		the full netlink2 which means we have invested time and 
		resources in it
		(and infact have deployed it in one site). However,
		we are willing to throw all that away provided there
will be progress
		here which is not driven by politics rather by a
technical drive.

		cheers,
		jamal

---

From: "Putzolu, David" <[email-masked]>
Date: Thu, 26 Feb 2004 13:33:33 -0800

_____  

From: Alex Audu [mailto:[email-masked]] 
Sent: Thursday, February 26, 2004 12:32 PM
To: [email-masked]
Cc: Khosravi, Hormuzd M; Putzolu, David; [email-masked];
[email-masked]; [email-masked]; Steve Blake;
Robert Haas; [email-masked]; Wang,Weiming; Patrick Droz; jeff
pickering; [email-masked]; Deleganes, Ellen M; [email-masked];
[email-masked]
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

Hello Jamal,

Thanks for the experience report with NetLink2.  After a quick scan, I
quickly came
to the conclusion that we may not want to go this route.  My reason is
the severely
limited scope of the test bed.: 
  1) one hop (far cry from ForCEs requirement)
  2) does not take into account the currently evolving meta-data work.
So,..what we currently 
      have out there is not fully ForCEs compliant.

As a starter, you can't burden the application layer with the
responsibility of correcting
 problems resulting from delivering application data out of order.  You
just can't mandate
that. At least, that is not what currently prevails.

What I think we should do is start with a list of features we can't live
without and then
take it from there.  And also, we should all respect the spirit of the
ForCEs requirements.
That is the only thing we have as a guide. 

Regards,
Alex.

Jamal Hadi Salim wrote:

		Hi All,

		Jamal, are there any other ground rules you have in mind
?
		Why don't you send a list of features which you would
like 
		to see and we can start the discussion from there ?

	
	I think the most important approach is to look at this as
	an opportunity (for all of us) to create a very useful protocol.
	Seems to me we all have good experiences that we can put to use.
	There will be differences in opionions, but the hope is that
there
	are differences in technical (not political) opinion. Lets all
be ready
	to make technical compromises for the sake of a good protocol.
	My thoughts are we should be doing some sort of interop by San
Diego
	time. So we dont have much time to waste on quibbling.
	
	As a starter, lets share some experiences. We can then move
towards
	defininig what would be useful. We can then debate about
different
	aspects of what mechanisms fit better and why.
	
	So i am attaching a text file which describes some of the
experiences we
	have gathered. If we can get something from the two other
protocols 
	then it would be extremely a good start. 
	
	Thoughts?
	
	cheers,
	jamal

  _____  

	1) Application level multicasting and broadcasting in addition
	to unicasting.
	
	We have found this to be extremely useful.
	
	Accomplished via PID addressing.
	
	We used source and destination 32 bit PIDs. It may be useful
	to have larger sized PIDs to make them UUIDs. Example 128 bits.
	
	PIDs were used in addressing the following:
	- LFBs on the FE, 
	- proxies for LFBs,
	- proxies for control application in the form of  CECs on CEs, 
	- the FE,
	- the CE.
	
	2) Reliability and using unicast, multicast, broadcast
transport.
	
	Note this has nothing to do with application multicasting.
	
	Although we never had the opportunity to work with a 
	broadcast transport, we do believe that the experience
	here applies to broadcast as well using some other transports
	such as TIPC.
	
	The config selection (on what transports/sockets to use) happens

	via FEM/CEM channel agreement.
	In most setups each FE had one TCP connection and one or more 
	multicast over UDP connections. The several multicast groups 
	were created on a per need basis. Example we had one group that
sent 
	config and another that waited for events (such as link events).
	What we discovered that although we found use for the one TCP
connection,
	there was clearly no such clear demarcation for how many
multicast connections
	would be needed.
	
	Caution on reliability needs:
	The focus in Netlink2 is application level reliability.
	Therefore regradless of what transport is used, when reliability
is
	desired the application level must take care of ensuring it.
	When the transport underneath provides reliability it will be
taken
	advanatge of. It should be noted, however, that a reliable
transport merely
	provides reliable transmission; in ForCES, you need to know more
than
	that - you need to know that the remote end successfully applied
the
	command issued (i.e successful transaction).
	
	More, in the case of multicast reliability whcih we successfuly
used:
	Reliability was provided on a per message basis (not possible
	to do with TCP). App also had opportunity to remove messages
	considered obsolete(send_undo).
	The design goal to provide Netlink2 reliable multicast was
	narrowed down within a single hop of an IP network. In other
words,
	no multicast state needs to be maintained within any
intermediate 
	router. Summary: Reliable multicast is only expected to work
	within a single hop.
	This reduces a lot of the complexity in the protocol and the
	implementation that is typical in building reliable multicast 
	protocols that span multi IP hops (such as the case of RMT like 
	reliability).
	While some of the existing RM protocols could have been taken
and
	adopted in their simplest form, the requirement for reliability
is very
	simple in netlink2 given the single hop need and therefore there
is
	no need to adopt any of the existing protocols.
	As an example LGMP could have been used with no need to build a 
	hierarchy of Local Groups for error recovery. It would have been

	sufficient to define a single "local group" with the 
	multicast sender(CE) as "group controller", i.e there would be
no need 
	for the LGCP protocol for dynamic building of hierarchies. But
even
	this simplification is considered fat.
	
	The unicast transport was used only for backup purposes when
	reliability of the multicast became suspect. As an example:
	when a multicast transmit happened from CE to a group of FEs and

	one or more of those FEs failed to ACK, then the retransmit for
the 
	particular failed FE(s) happens over the unicast TCP connection.
	
	Our experience indicates there was very rarely any need to
retransmit.
	We believe this had to do with the size of our standard NE which
	had six to twelve FEs connected over a fairly reliable multicast

	capable fabric.  i.e this experience does not dilute the need
for 
	reliability.
	
	3) The following  mechanisms were found to be very useful.
	a) ability to do atomic transactions via 2 phase commits. Allows
	several FEs to sync their entries in one shot.
	b) Batching for increased throughput and markers for batching
	and what happens when a failure occurs.
	In our case we implemented the "All up to the failed one are
applied"
	
	4) IP independence
	IP is very valuable as the underlying transport.
	We initially coded so that we could run directly over 
	ethernet, but eventually did not exercise that code very much.
	The reason is we ended focussing on building around IP instead.
	We do feel however that it would be valuable to have
independence
	so that the protocol can run over Ethernet if needed.
	In our investigation we found that many vendors have config
protocols 
	that run over Layer2 to avoid having to depend on IP i.e the IP
layer may
	either not be ready or provisioned (example many of the CISCO
	protocols like CPD, UDLD etc).
	We also found that there are a lot of board to board and chip
	to chip transports (dont run over IP) that would be valuable to
consider 
	if ForCES is to be the protocol of choice.

	5) Packet encoding. 
	- Our design goal was to have to have the highest throughput
	and lowest latency possible. The reason for this is because 
	this is how the market judges the capacity of a NE. 
	Our FEs had slow CPUs mostly (in the range of PPC 82xx, ARM
SA110)
	controlling some local switch chips. We expect initially this is

	the setup that will be the most common in the market.
	
	Binary encoding was used to be able to pack as much as possible
	within an allowed MTU. TLVs were used to map the various
attributes.
	TLVs within TLVs were used to encode complex attributes such as
	lists or compound structures.
	
	In retrospect based on our current experiences, the following
layout
	maybe useful:
	
	     0                   1                   2
3
	     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
0 1
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |
|
	    |                   Netlink2 message header
|
	    |
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |
|
	    |                        Attributes
|
	    |
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

	Although the per-service template(refer to netlink and netlink2
docs)
	was useful, we believe that we could be more generic and pack
more 
	attributes per message with the above layout.

	     0                   1                   2
3
	     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
0 1
	                     0               1               2
3
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |             Command         |               Length
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |                          Source PID
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |                        Destination PID
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |                        Sequence Number
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	    |             flags           |             typeid
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

	The commands are no longer specific to or used to address the
service. 
	The service addressing now solely resides in the PIDs.
	We envision that the command would be of the sort of ADD, DEL,
GET.
	The PID will address the entity being targeted.
	[A command from a CE -> FE is for querying or configuration,
whereas one
	from FE -> CE is a response or event].
	
	In our implementation we extended the popular tcpdump sniffer to
look
	at the packet and decode it for protocol debugging purposes.
	For this reason we need to identify the class of LFB etc the
target
	entity is on. Hence the new typeid field makes sense as an
addition.
	
	XML encoding:
	We believe XML encoding may be useful in capabilities definition
	(i.e slow path) but not in the configuration of data path
tables.
	This is because of its nature to require higher bandwith and
	CPU computational needs.
	
	TLVs:
	
	    0                   1                   2
3
	    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
0 1
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   | TLV Type                    | variable TLV Length
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   ~
~
	   |            Value (Data of size TLV length)
|
	   ~
~
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	
	We were able to extend the data in the TLV to contain another
	TLV to build complex structures (such as the ones described in
	XML by the model draft).
	
	Example:
	    0                   1                   2
3
	    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
0 1
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   | Outer TLV Type              |    Outer TLV Length
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   | Inner TLV1 Type             |   Inner TLV1 Length
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   | ~~~~~~~~~~~~~~           VALUE1    ~~~~~~~~~~~~~~~~~~~~~~
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   | Inner TLV2 Type             |   Inner TLV2 Length
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	   | ~~~~~~~~~~~~~~           VALUE2    ~~~~~~~~~~~~~~~~~~~~~~
|
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

	Which could be used to represent the following layout :
	
	  +-+-+-+-+-+-+-+-+-+-+-+        .-~-~-~-~-~-~-.
	  |  TLV1               | ------>|   param 1   |
	  +-+-+-+-+-+-+-+-+-+-+-+         -~-~-~-~-~-~-
	  |  TLV2               | --->--+
	  +-+-+-+-+-+-+-+-+-+-+-+       |
	                                V
	                                |
	                               /
	                  +-----------+
	                  |           
	                  V                 .-~-~-~-~-~-~-.
	                  |            +--->|  param2_1   | 
	  +-+-+-+-+-+-+-+-+-+-+-+      |     -~-~-~-~-~-~- 
	  |  TLV2_1             | --->-+       .-~-~-~-~-~-~-.
	  +-+-+-+-+-+-+-+-+-+-+-+         +--->| param2_2    |
	  |  TLV2_2             | --->----+     -~-~-~-~-~-~- 
	  +-+-+-+-+-+-+-+-+-+-+-+  

	It is thought that TLVs and nested TLVs will generally
constitute
	slightly higher requirements for compute and bandwdith than
encodings 
	based on fixed layouts (XML is many more factors demanding). 
	The added advantage with TLV usage of course is ability to add
new 
	TLVs easily without redefining the packet or the protocol.
	In addition, being able to optionally leave a TLV out during
communication
	(the message example would have been perfectly decoded if TLV1
was
	never encoded).
	
	A small challenge that needs to be overcome for TLVs is to
detect
	where failures happen (when you have a list or other compound
structure). 
	This is mostly useful/needed for batching or configuring a list
of elements. 
	If we were to use our implementation as a sample space: we had
the "all 
	upto failed one applied" scheme, in which case a NACK with the
failed 
	TLV Type is sent back.
	
	An interesting idea is to encode an 8 bit sequence number in the

	type to find precisely which TLV failed.

	6) Dynamic capability addition
	
	In our implementation we found it was very useful to be able
	to add dynamic capabilities at runtime as opposed to assume
	they are available at setup time. For this reason we introduced
an 
	event to announce capabilities of an FE to a CE.
	Although the FEM/CEM interface could have been used, we
implemented
	via the FE/CE interface.
	
	An alternative scheme may have been for the CE to poll the FE to
ask for
	its capabilities in a timer. This approach may also be
considered
	orthogonal to the previous one.
	
	7) PID distribution/Allocation
	
	In our implementation a range of PIDs were allocated by the CE
to each FE at
	the FEM/CEM interface. It was upto the FE to issue some of these

	PIDs to the LFBs or their proxies (without going into details by

	doing local OS PID mapping).
	
	We implemented such that broadcast PIDs (example: AllFEs,
AllNAT, etc) 
	were statically defined as opposed to be dynamically selected.
	This was just to reduce state in the CE.
	
	8) High Availability
	
	Our efforts to discover any challenges posed by HA are still
ongoing.
	At the moment FEs and CEs backing each other sync by joining the
	same multicast groups and listen to the same logical PIDs.
	
	We have implemented such that a logical PID may be mapped to
several
	unicast connections. We however, have not experimented yet on 
	uni-replicasting to multiple C/FEs. 
	
	We have various alternatives that maybe useful (To be ADDED
here).

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 27 Feb 2004 13:31:20 +0800

That would be a very helpful effort for the merge process.

Weiming

----- Original Message ----- 
From: <[email-masked]>
To: "Khosravi, Hormuzd M" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>; "Deleganes, Ellen M" <[email-masked]>; <[email-masked]>; "Steven Blake" <[email-masked]>; <[email-masked]>; "Patrick Droz" <[email-masked]>; <[email-masked]>; "Robert Haas" <[email-masked]>; <[email-masked]>; "jeff pickering" <[email-masked]>; "Wang,Weiming" <[email-masked]>; "Putzolu, David" <[email-masked]>; <[email-masked]>
Sent: Thursday, February 26, 2004 11:22 AM
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

> >> So lets start with a list of ground rules.
ŠÜzÊk¢Ú¢YšŠX§‚X¬´Z+që)®‹hr‰bz×è®m¶›?ÿ0Ö'~Šàþf¢–f§þX¬¶)ß£÷èÇ¬¦º-¡Ê%

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 27 Feb 2004 20:55:50 +0800

Hi Alex,

Thank you for your comments. Pls see the inline reply.

Best regards,
Weiming
  ----- Original Message ----- 
  From: Alex Audu 
  To: Wang,Weiming 
  Cc: avri ; Chaoping Wu ; Deleganes Ellen M ; Dong, Ligang ; Guo, Yunfei ; Jamal Hadi Salim ; Jeff Pickering ; Jon Maloy ; Khosravi, Hormuzd M ; Patrick Droz ; Putzolu David ; Robert Haas ; Spyros Denazis ; Steve Blake ; Wang, Guangming 
  Sent: Friday, February 27, 2004 2:06 AM
  Subject: Re: ForCES Protocol Analysis Design Team Administrivia

  Hello Weiming,

  1) Seperate Data Redirect and Control channels/ relative importance of Control vs Redirect info
      It is easy to see that the redirect data can be manipulated by an outside source (External to 
       the NE).  In the case of such a source being malicious, it is necessary to give priority to 
       the control data to maintain integrity of the NE.

        You sighted BGP experience.  Indeed, the realization that it is safer to separate externally
         generated control data like redirect-data from internally generated control data grew out
         of  experience gleamed from past DOS attacks on BGP.

  2)  C channel and D channel are on the same physical link.   For DOS survivability, you need 
        to seperate then somehow. If you use SCTP, you can put them on separate streams on
         the same link. That will work.  But what will be best is a non-reliable congestion aware
        stream/link for  D-channel and a reliable stream/link for C-channel.
         Rate controlling is important. But if  you need to allow your C information to flow freely
         to adaptively effect counter-meassures if the NE is being DOS'd.

  Re. 1&2:  I suppose a separation of D and C packets inside FE while using only one stream(channel) to accross the 
  physical link (GRMP DoS protection scheme) can also achieve all these goals (DoS protection, rate control, priority, 
  security, reliability). We just ask that if we don't need to provide the two diffenrent channels in the physical link so
   as to prevent DoS, are we still interested in giving the D packets a different transport reliability?

  3) Delay of D packets by  delay of C packets.   This sould not happen if your channels are
       independent. And  D packets may dictate what C packets get sent to the FE.  The
       problem might be related to the way your real-time processes are running the D and C 
       queues.  You can play with thread priorities,..etc to resolve this.
  Re: Yes, this is because of the congestion of queues of D packets and C packets, but the same will
  also happen to separated D/C channels if they use the same physical link. Only the difference is the latter
  may use OS kernel to schedule while the former uses ForCES protocol defined scheduler. 

  4) C/D separation resource cost:   This is not as expensive as the cost paid when DOS
       attacks occur.  Besides, hardware resources are getting cheaper by the day,..and 
       processors are getting faster too.  

  Re:  again, by separating the D packets and C packets inside an FE, we can also prevent DoS efficiently. 

  General comments. May be a way to go forward is to make it a SHOULD to implement
   (on seperate D/C channels) and leave the rest up to implementation,...I am not sure. Any comments?

  Re: I agree. What we propose is only to limit such a separation inside an FE instead of expanding it to the FE-CE link.

  Regards,
  Alex.21

  Wang,Weiming wrote:

Hi David,

Regarding the proposed basic principles, we (GRMP group) have some comments for  principles item 7&8:
--------
7.      The ForCES protocol should support separate channels for Control and Data messages. The data 
channel carries the routing/control protocol packets such as RIP, OSPF packets, between the CEs and FEs. 
These packets are unique in most cases for a particular CE:FE pair and hence the data channel will use unicast 
connections between the CEs and FEs. Also, the data channel does not have to be strictly reliable but does 
need to provide flow/congestion control to protect against DoS attacks.

8.      The control channel must provide strict reliability (non-corruption, no loss, no reordering, timeliness) at the 
transport level in accordance with the RFC 3654. In addition, ForCES protocol level responses/acknowledgements 
are also needed from FE to CE to learn the results of configuration/control messages.
--------

First of all, we agree that to distinguish ForCES messages into control messages and data (redirect) messages are 
essential for DoS protection scheme. Then, we argue that to separate the two kinds of messages into two different 
transport channels with the data channel being appointed to a lower reliable level (such as UDP, compared with TCP) 
may not be a good enough scheme for such purpose. The reasons are:

1. The data (redirect) ForCES messages seem have no reason to be appointed to a lower reliability, compared with 
control channel messages.  We may needn't  to argue how important some of the redirected messages are for a 
router (an NE), we just argue that  the reliability for ForCES data message transport should not be lower than that of the 
connection over which the routing/control protocol packets  are transported. An example is the BGP protocol messages, 
which have already been assigned to use TCP instead of UDP for the transports. We think the communication between CEs 
and FEs (which is inside an NE) has no reason to greatly induce extra performance loss other than that from outside of 
an NE for such routing protocol packets. By saying this, we don't mean all ForCES messages should not be loaded over UDP, 
we just propose that it's may be too early to distinguish the redirect messages from other configuration messages inside ForCES 
protocol specifications.

2. Generally speaking, separated C channel and D channel are still put over the same physical link (or else there is almost 
no need to protect DoS).  Therefore,  such a separation itself does not provide any DoS protection, instead, the protection is 
obtained by the rate control of D channel. We argue that  such a rate control actually can be obtained at the ForCES message 
layer instead of at the message transport layer. Some of our experiment results (which will be presented at the IETF meeting) 
have shown this feasibility.  Moreover, a separation at transport layer surely will complicate the CE-FE association and the 
followed message transport management. 

3.  We have also noticed a phenomenon during our experiments, which we may need to pay some attention. I recalled when 
FACT first presented the D and C channel idea,  there was a question worrying about the asyncronization between D and C channel.  
Our experiments have shown that,  although there is no strict asyncronization problem between D and C channel owing to the data 
independence, there is a possibility that D channel packets may be seriously delayed and even blocked owing to the busy traffic at 
C channel if D packets are separated from C packets. The busy traffic at C channel may happen when CE queries FE for something 
and FE dumps quite a huge number of messages (such as routing table items.)  Such a delay or blocking may be counted in seconds, 
resulted in very harmful effect on the users of redirected packets.  As a result, we propose that a moderate rate control for C channel 
be also necessary. 

4. To explicitly separate D and C channel also consumes more resources, because a resources for D channel have to be reserved. 
Instead, by separating D and C packets at the ForCES protocol layer and uniting them again at the transport layer, we may able to 
make a scheduling discipline by which the D and C packets can share the link resources effectively. We are trying to construct such 
a scheduling scheme, and it seems quite practical to so.

To summarize, we propose that the 7&8 basic principles only mention the separation of D and C packets for DoS protection currently, 
leaving some space for further discussions. The discussions may be like: 

1. Should such a separation be put at the ForCES message layer or at the transport layer?  Which is better?
2. Whether united scheduling for D and C packets are necessary?
3. Should D packets be distinguished from C packets for its transport reliability in ForCES protocol so as to assign a different transport means?

Regarding the protocol merge proposal, we'd like to present some of our thinking in a few words:

Firstly, as a new comer to IETF work, the GRMP group surely welcome such a merge proposal, sincerely hoping an all merits in 
one draft can be achieved for ForCES WG, and at the same time hoping to learn from other candidate authors, whom we have 
already been respecting to for their rich knowledge and excellent contributions to IETF and other fields. Secondly, we believe 
that the interaction among the candidate drafts are very helpful for the final ForCES protocol. Actually there already has been such 
interaction since the ForCES protocol evaluation process began, which has already made individual candidate drafts ever improved.  
Thirdly, to make the process successful, we believe that a nice atmosphere is very important. In such an atmosphere, everyone has 
a peaceable state of mind, precisely speaking, a mind ready to give up and accept, and individuals' work can be fully respected. 
Our past experience has made us believe that it is almost impossible to reach a compromise among group members where some 
think of their work much outstanding to others. Lastly, as Jamal has mentioned, some ground rules is important so as to eliminate 
conflicts whenever there are. 

Regarding the merge action way, we agree candidates to propose their "can't live without" first. Along with this, we also suggest 
candidates propose "what they think the contents in their drafts are useful for final protocol and hope able to be put in it". This useful 
contents are a little different from the "features of the drafts", because we suppose the contents proposed should be conservatively 
considered by presenters to be able to merge with other contents. 

Thank you.

Beset Regards,
Weiming Wang

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 27 Feb 2004 22:07:14 +0800

Hi Ram, 

Glad to see your comments and please find the reply inlines. 

Best regards,
Weiming
  ----- Original Message ----- 
  From: [email-masked] 
  To: [email-masked] ; [email-masked] 
  Sent: Friday, February 27, 2004 9:35 PM
  Subject: RE: [Forces-protocol] Fw: ForCES Protocol Analysis Design Team Administrivia

  Hello Weiming,

  Alex missed out my name in his first email. See my comments inside  the tag <Ramg>  
    -----Original Message-----
    From: [email-masked] [mailto:[email-masked]]On Behalf Of ext Wang,Weiming
    Sent: Friday, February 27, 2004 7:56 AM
    To: [email-masked]
    Subject: [Forces-protocol] Fw: ForCES Protocol Analysis Design Team Administrivia

    Hi Alex,

    Best regards,
    Weiming

      Regards,
      Alex.21

      Wang,Weiming wrote:

Hi David,

[<Ramg>]  As you pointed out exactly that BGP runs on TCP and provides its own reliability mechanism and so as other routing protocols have some form of inbuit mechanism and this is the reason
we need not address in our ForCES protocol for data channel. Some traffic we don't need retransmission and reliability in D channel as it is handled by those protocols themselves.  <RamG>
WangRe: as I have replied to Alex, I just suppose if we don't have to separate the D/C channels in the physical link (instead only separate it inside an FE), are we still interested in
appointing to D packets different transport reliabilities? 
   2. Generally speaking, separated C channel and D channel are still put over the same physical link (or else there is almost 
        no need to protect DoS). Therefore, such a separation itself does not provide any DoS protection, instead, the protection is obtained by the rate control of D channel. We argue that such a rate control actually can be obtained at the ForCES message layer instead of at the message transport layer. Some of our experiment results (which will be presented at the IETF meeting) have shown this feasibility. Moreover, a separation at transport layer surely will complicate the CE-FE association and the followed message transport management. 
        [<Ramg>] 

        I think everyone agrees this and understand this point. It is the CE which uses ForCES Message to convey the ratelimting value for C and D channel FE. It is the FE which forwards the packet to the CE and hence one FE need to know the parameter. Here the rate limiter values could be different in either direction..

        WangRe: yes, I totally agree.

         3. We have also noticed a phenomenon during our experiments, which we may need to pay some attention. I recalled when FACT first presented the D and C channel idea, there was a question worrying about the asyncronization between D and C channel. Our experiments have shown that, although there is no strict asyncronization problem between D and C channel owing to the data independence, there is a possibility that D channel packets may be seriously delayed and even blocked owing to the busy traffic at C channel if D packets are separated from C packets. The busy traffic at C channel may happen when CE queries FE for something and FE dumps quite a huge number of messages (such as routing table items.) Such a delay or blocking may be counted in seconds, resulted in very harmful effect on the users of redirected packets. As a result, we propose that a moderate rate control for C channel be also necessary. 
        [<Ramg>] 

        Please see the previous reply. The dumping of huge data confirms to the rate limiter value which was done and acks are being presented in FACT layer to ensure that message delivery and message execution. We have implemented this and it is running fine.

        <Ramg>

        WangRe: we just want to point out this D pakcets blocking facts for more attention. I'm glad to know that FACT also use some rating mechanisim for C channels as well as D chennels, because GRMP also be in this way. 

         4. To explicitly separate D and C channel also consumes more resources, because a resources for D channel have to be reserved. Instead, by separating D and C packets at the ForCES protocol layer and uniting them again at the transport layer, we may able to make a scheduling discipline by which the D and C packets can share the link resources effectively. We are trying to construct such a scheduling scheme, and it seems quite practical to so.To summarize, we propose that the 7&8 basic principles only mention the separation of D and C packets for DoS protection currently, leaving some space for further discussions. The discussions may be like: 1. Should such a separation be put at the ForCES message layer or at the transport layer? Which is better?2. Whether united scheduling for D and C packets are necessary?3. Should D packets be distinguished from C packets for its transport reliability in ForCES protocol so as to assign a different transport means?Regarding the protocol merge proposal, we'd like to present some of our thinking in a few words:Firstly, as a new comer to IETF work, the GRMP group surely welcome such a merge proposal, sincerely hoping an all merits in one draft can be achieved for ForCES WG, and at the same time hoping to learn from other candidate authors, whom we have already been respecting to for their rich knowledge and excellent contributions to IETF and other fields. Secondly, we believe that the interaction among the candidate drafts are very helpful for the final ForCES protocol. Actually there already has been such interaction since the ForCES protocol evaluation process began, which has already made individual candidate drafts ever improved. Thirdly, to make the process successful, we believe that a nice atmosphere is very important. In such an atmosphere, everyone has a peaceable state of mind, precisely speaking, a mind ready to give up and accept, and individuals' work can be fully respected. Our past experience has made us believe that it is almost impossible to reach a compromise among group members where some think of their work much outstanding to others. Lastly, as Jamal has mentioned, some ground rules is important so as to eliminate conflicts whenever there are. Regarding the merge action way, we agree candidates to propose their "can't live without" first. Along with this, we also suggest candidates propose "what they think the contents in their drafts are useful for final protocol and hope able to be put in it". This useful contents are a little different from the "features of the drafts", because we suppose the contents proposed should be conservatively considered by presenters to be able to merge with other contents. Thank you.Beset Regards,Weiming Wang
        [<Ramg>] 
        If we make separation on transport layer then we have few transport choices and many not be widely deployable. It is upto the group to decide. If we deal with the ForCES messaging layer then it has this messaging can be applied to other non-ip transport too.

        WangRe: yes, I totally agree. What we propose is exactly to limit the separation of  the C/D packets into ForCES C/D messages instead of expanding such a separation into transport layer (over physical link). I'm not sure if I'v missed your meaning or not? 

        <Ramg>
         End of comment 
        Regards
        Ramg

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 27 Feb 2004 22:15:01 +0800

Hi Ram, 

Best regards,
Weiming
  ----- Original Message ----- 
  From: [email-masked] 
  To: [email-masked] ; [email-masked] 
  Sent: Friday, February 27, 2004 9:35 PM
  Subject: RE: [Forces-protocol] Fw: ForCES Protocol Analysis Design Team Administrivia

  Hello Weiming,

     ----- Original Message ----- 
      From: Alex Audu 
      To: Wang,Weiming 
      Cc: avri ; Chaoping Wu ; Deleganes Ellen M ; Dong, Ligang ; Guo, Yunfei ; Jamal Hadi Salim ; Jeff Pickering ; Jon Maloy ; Khosravi, Hormuzd M ; Patrick Droz ; Putzolu David ; Robert Haas ; Spyros Denazis ; Steve Blake ; Wang, Guangming 
      Sent: Friday, February 27, 2004 2:06 AM
      Subject: Re: ForCES Protocol Analysis Design Team Administrivia

      Hello Weiming,

      Regards,
      Alex.21

      Wang,Weiming wrote:

Hi David,

        <Ramg>

         4. To explicitly separate D and C channel also consumes more resources, because a resources for D channel have to be reserved. Instead, by separating D and C packets at the ForCES protocol layer and uniting them again at the transport layer, we may able to make a scheduling discipline by which the D and C packets can share the link resources effectively. We are trying to construct such a scheduling scheme, and it seems quite practical to so.To summarize, we propose that the 7&8 basic principles only mention the separation of D and C packets for DoS protection currently, leaving some space for further discussions. The discussions may be like: 1. Should such a separation be put at the ForCES message layer or at the transport layer? Which is better?2. Whether united scheduling for D and C packets are necessary?3. Should D packets be distinguished from C packets for its transport reliability in ForCES protocol so as to assign a different transport means? 
        [<Ramg>] 
        If we make separation on transport layer then we have few transport choices and many not be widely deployable. It is upto the group to decide. If we deal with the ForCES messaging layer then it has this messaging can be applied to other non-ip transport too.

        <Ramg>
         End of comment 
        Regards
        Ramg

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 28 Feb 2004 13:48:10 +0800

Hi Alex, 

Pls see more reply in the lines.

Thank you.

Weiming
  ----- Original Message ----- 
  From: Alex Audu 
  To: Wang,Weiming 
  Cc: [email-masked] 
  Sent: Saturday, February 28, 2004 6:57 AM
  Subject: Re: [Forces-protocol] Re: ForCES Protocol Analysis Design Team Administrivia

  Hello Weiming,

  See my comments in-line.

  Regards,
  Alex.

  Wang,Weiming wrote:

    Hi Alex,

    Best regards,
    Weiming
      ----- Original Message ----- 
      From: Alex Audu 
      To: Wang,Weiming 
      Cc: avri ; Chaoping Wu ; Deleganes Ellen M ; Dong, Ligang ; Guo, Yunfei ; Jamal Hadi Salim ; Jeff Pickering ; Jon Maloy ; Khosravi, Hormuzd M ; Patrick Droz ; Putzolu David ; Robert Haas ; Spyros Denazis ; Steve Blake ; Wang, Guangming 
      Sent: Friday, February 27, 2004 2:06 AM
      Subject: Re: ForCES Protocol Analysis Design Team Administrivia

      Hello Weiming,

      Re. 1&2:  I suppose a separation of D and C packets inside FE while using only one stream(channel) to accross the 
      physical link (GRMP DoS protection scheme) can also achieve all these goals (DoS protection, rate control, priority, 
      security, reliability). We just ask that if we don't need to provide the two diffenrent channels in the physical link so
       as to prevent DoS, are we still interested in giving the D packets a different transport reliability?
  [Alex] If I understand your question correctly, no,..this does not mean you are giving D 
             packets a different transport reliability. But if you carry D and C data on the same
             channel, you have to be mindful not to let D data impact flow of C data.
             And frankly, I don't know what that means regarding your question.

  [WangRe]
  We are just using the model presented in GRMP (slave model), which by experiments have initially shown 
  the feasibility to protect DoS. 

      Re:  again, by separating the D packets and C packets inside an FE, we can also prevent DoS efficiently. 
  [Alex]  Say you are been DOS'd with a million D packets, and your C sends one message to
              help counter the problem.  Your packet separator will have to do a lot of work
               before getting to the C packets in the queue.  I don't think this makes for quick
              recovery from the attack.

  [WangRe] No, this is exactly the same as D packets are put on a different channel (as your notion, different stream), the only exception is the stream scheduling is done by OS kernel scheduling (usually using round robin), while in our scheme it is done by ForCES assigned scheduling whose scheduling decispline can be flexibly set by ForCES protocol, and D/C packets can be very easily and differentially scheduled.  The problem you mentioned will also happen to a separate stream scheme if the OS kernel scheduler is not good enough. 
  Remember that congestion happens when DoS attacks only because there is not enough physical link layer resources. Therefore, separation of the D packets to a specific stream itself is of no use to prevent DoS. 

      Regards,
      Alex.21

  ..[snip]..

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 28 Feb 2004 14:10:00 +0800

Hi Sandy,

Thank you for the comments. Pls see reply in the lines. 

[[Sandy]] 

I am concerned about the separation of the D and C packets at the ForCES protocol layer. I wonder if it is not too late at both ends of the communication channel; by this I mean resources are allocated to distinguish between the two but DoS might have already occurred or C packets might have already been lost due to congestion on D. 

[WangRe]

I can not find any reason it will be delayed more to schedule D and C packets at ForCES protocol layer than to schedule them at the transport layer. Actually the former happens ahead of the latter.  

[Sandy]

I believe marking the two types and doing the separation at transport layer is safer and uses existing mechanisms that do not overload the protocol.  

[WangRe]

Could you give more reasons why it is safer to schedue the D/C packets at the transport layer? My opinion is they are the same, except one is done by OS kernel scheduling (usually using round robin), the other is done by ForCES assigned scheduling whose scheduling

decispline can be flexibly set by ForCES protocol, and D/C packets can be very easily and differentialtedly scheduled. 

[Sandy]

Differentiated scheduling for D and C would be my preference. 

[WangRe]

 By saying to use united scheduling, I actually mean it may be able to exploit more from usual separate scheduling. If it is impossible (we are doing experiments to try it), there is no any limit for us to separately schedule D/C packets as usual. 

[Sandy]

And yes, there should be a differentiation between C and D for transport reliability.   

[WangRe]

Actually I agree to differentiate the D/C packets for its transport relaibility, if possible. Maybe I should have stated the question as " Is it worthy or not to define a separate transport layer channel for D packets in ForCES protocol specifications  to only achieve a little lower reliability for the D packets transmision?"  Taking the chance to reply your comments, pls allow me to point out more cons on this: 

1.  First of all, as I'v mentioned, such a D channel is only of use for assigning different transport reliability for D packets. It has no use for DoS protection. 

2. Such a little difference for reliability requiements can be distinguished via ForCES protocol built in mechanisims by deciding to or not to use something like ACK, checksum, etc.

3. To define a separate channel for D packets is also very case specific. That is, in UDP/TCP based CE-FE interconnection case, we may think of using UDP for D packets and TCP for C packets. But what then if the  CE-FE link is Ethernet based, ATM based, backplane bus based, and any future technology based?  Are we going to define it one by one? And what then if the CE-FE link can only provide a channel? 

4. On countrary, it surely consumes more resources and makes ForCES protocol management more complex. 

Best Regards, 

Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 28 Feb 2004 13:47:57 +0800

Hi Sandy,

----- Original Message ----- 

  From: Sandy Pavel 
  To: [email-masked] 
  Sent: Saturday, February 28, 2004 12:07 AM
  Subject: RE: [Forces-protocol] Fw: ForCES Protocol Analysis Design Team Administrivia

  See comments below:

          [[Sandy]] 

          [WangRe]

          [WangRe]

          [WangRe]

          [[Sandy]] 

          [WangRe]

          Best Regards, 

          Weiming

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sun, 29 Feb 2004 20:02:20 -0800

Weiming,

After going thru all the emails on this topic, seems like most protocol
authors as well as non-authors

except yourself agree that there should be separate transports for C & D
channel. 2 good reasons

have been given for this: different reliability and robustness against
DoS attacks. You agree with the

first but not with the latter. The problem with implementing scheduling
at the protocol level and sending

all the messages over a single TCP channel is that the app or protocol
scheduling has little control

over the TCP queues inside the OS. So scheduling at the protocol level
will not help during a DoS attack

once the TCP queue inside the OS if full.

This is the reason why most people on this list are suggesting using
different transports or multiple

channels for D and C channel. We can discuss this further during the
meeting if you like.

BTW, I have been receiving many duplicate emails from you, but am not
seeing all the other emails.

There might be some problems with this mailing list.

Regards

Hormuzd

  _____  

From: [email-masked]
[mailto:[email-masked]] On Behalf Of Wang,Weiming
Sent: Friday, February 27, 2004 10:10 PM
To: [email-masked]
Subject: [Forces-protocol] Re: ForCES Protocol Analysis Design Team
Administrivia

Hi Sandy,

 

[[Sandy]] 

I am concerned about the separation of the D and C packets at the ForCES
protocol layer. I wonder if it is not too late at both ends of the
communication channel; by this I mean resources are allocated to
distinguish between the two but DoS might have already occurred or C
packets might have already been lost due to congestion on D. 

[WangRe]

I can not find any reason it will be delayed more to schedule D and C
packets at ForCES protocol layer than to schedule them at the transport
layer. Actually the former happens ahead of the latter.  

[Sandy]

I believe marking the two types and doing the separation at transport
layer is safer and uses existing mechanisms that do not overload the
protocol.  

[WangRe]

Could you give more reasons why it is safer to schedue the D/C packets
at the transport layer? My opinion is they are the same, except one is
done by OS kernel scheduling (usually using round robin), the other is
done by ForCES assigned scheduling whose scheduling

decispline can be flexibly set by ForCES protocol, and D/C packets can
be very easily and differentialtedly scheduled. 

[Sandy]

[WangRe]

 By saying to use united scheduling, I actually mean it may be able to
exploit more from usual separate scheduling. If it is impossible (we are
doing experiments to try it), there is no any limit for us to separately
schedule D/C packets as usual. 

[Sandy]

And yes, there should be a differentiation between C and D for transport
reliability.   

[WangRe]

Actually I agree to differentiate the D/C packets for its transport
relaibility, if possible. Maybe I should have stated the question as "
Is it worthy or not to define a separate transport layer channel for D
packets in ForCES protocol specifications  to only achieve a little
lower reliability for the D packets transmision?"  Taking the chance to
reply your comments, pls allow me to point out more cons on this: 

1.  First of all, as I'v mentioned, such a D channel is only of use for
assigning different transport reliability for D packets. It has no use
for DoS protection. 

2. Such a little difference for reliability requiements can be
distinguished via ForCES protocol built in mechanisims by deciding to or
not to use something like ACK, checksum, etc.

3. To define a separate channel for D packets is also very case
specific. That is, in UDP/TCP based CE-FE interconnection case, we may
think of using UDP for D packets and TCP for C packets. But what then if
the  CE-FE link is Ethernet based, ATM based, backplane bus based, and
any future technology based?  Are we going to define it one by one? And
what then if the CE-FE link can only provide a channel? 

4. On countrary, it surely consumes more resources and makes ForCES
protocol management more complex. 

Best Regards, 

Weiming

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sun, 29 Feb 2004 20:19:30 -0800

Hi All

I was thinking about the issue of transport(s) for the ForCES protocol
which is probably one of the most important and might be quite
contentious.

One way to resolve this and make some progress is to separate this from
the ForCES base protocol, let us consider having 2 drafts...one which 

defines the base protocol assuming a reliable transport and the second
which defines how the base protocol will work over different reliable as
well as 

unreliable transports e.g. TCP/IP, UDP/IP, TIPC, Ethernet, PCI, etc. The
base protocol can concentrate on the basic protocol messages needed for 

control, configuration, capability exchange, etc. as well as other
transactioning semantics (2-phase commit), etc. The 2nd draft, say on
'encapsulation

for the base protocol' can discuss the details of providing reliability
over unreliable transports using some encapsulation header or TLV, etc.
and how

the base protocol will work over different transports.

We will eventually need to mandate 1 or minimum set of transports for
the protocol, but that can be decided later. In the mean time we can
continue 

making progress on both drafts. How does that sound to you guys ?

Let us know.

Regards

Hormuzd