Subject: [Forces-protocol] Re: ForCES Protocol Analysis Design Team Administrivia

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 27 Feb 2004 15:55:41 +0800

Hi Steven,

Actually the basics you'v presented to approach the ForCES redirect function is quite different from that of GRMP and FACT.
I'm not sure if it's the Netlink2 thought (because current Netlink2 leaves TBD for this). If it is, could you (or Netlink2
group) present a more detailed scheme for the packet redirection and for DoS attack protection so that we can combine these 
schemes together for more discussion? I think it will be very progressive. 

The basics of your idea different from GRMP and FACT approaches is, GRMP and FACT have already put the redirect packets 
to ForCES message layer (as your words, have already used another protocol stack). With this assupmtion ahead,we argue
that there is no need to separate back the two kind of messages again at the ForCES message transport layer, for it seems 
no help for DoS protection (the original reason to separate them is for DoS protection). 

I agree the simplest the way to redirect packets from FE to CE the better. I suppose no candidate protocols would like
to use such TCP->IP->ForCES messages->TCP->IP...IP->TCP->ForCES messages->IP->TCP way if a better approach that can have
the same function, agrees to the ForCES requirements and framework, and are not specific implementation oriented.

Best regards,

Weiming

----- Original Message ----- 
From: "Steven Blake" <[email-masked]>
To: "Wang,Weiming" <[email-masked]>
Cc: "avri" <[email-masked]>; "Alex Audu" <[email-masked]>; "Chaoping Wu" <[email-masked]>; "Deleganes Ellen M" <[email-masked]>; "Dong, Ligang" <[email-masked]>; "Guo, Yunfei" <[email-masked]>; "Jamal Hadi Salim" <[email-masked]>; "Jeff Pickering" <[email-masked]>; "Jon Maloy" <[email-masked]>; "Khosravi, Hormuzd M" <[email-masked]>; "Patrick Droz" <[email-masked]>; "Putzolu David" <[email-masked]>; "Robert Haas" <[email-masked]>; "Spyros Denazis" <[email-masked]>; "Wang, Guangming" <[email-masked]>; "Zsolt Haraszti" <[email-masked]>
Sent: Thursday, February 26, 2004 10:57 PM
Subject: Re: ForCES Protocol Analysis Design Team Administrivia

> On Wed, 2004-02-25 at 03:34, Wang,Weiming wrote:
> 
> > 1. The data (redirect) ForCES messages seem have no reason to be appointed to a lower reliability, compared with 
> > control channel messages.  We may needn't  to argue how important some of the redirected messages are for a 
> > router (an NE), we just argue that  the reliability for ForCES data message transport should not be lower than that of the 
> > connection over which the routing/control protocol packets  are transported. An example is the BGP protocol messages, 
> > which have already been assigned to use TCP instead of UDP for the transports. We think the communication between CEs 
> > and FEs (which is inside an NE) has no reason to greatly induce extra performance loss other than that from outside of 
> > an NE for such routing protocol packets. By saying this, we don't mean all ForCES messages should not be loaded over UDP, 
> > we just propose that it's may be too early to distinguish the redirect messages from other configuration messages inside ForCES 
> > protocol specifications.
> 
> The IP network that delivers the data traffic to the NE is inherently
> unreliable; therefore overall reliability cannot be improved by applying
> strict reliability to the D channel.  In fact, the D channel traffic
> (e.g., BGP over TCP) that requires end-to-end reliability will already
> employ transport-layer reliability.  Requiring D channel reliability
> might improve performance somewhat; on the other hand, the D channel
> reliability mechanisms might interact badly with TCP.
> 

> [snip]
> 
> > To summarize, we propose that the 7&8 basic principles only mention the 
> > separation of D and C packets for DoS protection currently, 
> > leaving some space for further discussions. The discussions may be like: 
> > 
> > 1. Should such a separation be put at the ForCES message layer or at the 
> >    transport layer?  Which is better?
> > 2. Whether united scheduling for D and C packets are necessary?
> > 3. Should D packets be distinguished from C packets for its transport 
> >    reliability in ForCES protocol so as to assign a different transport means?'
> 
> I'm not sure I understand your questions, but I am sure of this:
> I want the D channel traffic (IP packets and occassionally raw L2
> frames) to be delivered to the bottom of my CE's networking stack with
> sufficient metadata to indicate the FE and the interface it arrived on. 
> I want my CE networking stack (which in my case lives in an OS kernel
> and not in user space) to treat those packets as if they had arrived to
> the stack on the interface indicated.  If you transmit the D channel
> traffic over a complicated transport such as TCP, you will severely
> complicate the ability to treat the traffic as locally delivered
> packets; it will be impossible to do this with off-the-shelf kernels
> without significant modification.  So I believe that the ForCES D
> channel transport needs to be simple enough to implement in a driver,
> and/or should use IP tunneling (almost any interesting stack can
> demultiplex tunneled traffic onto other interfaces).  Forcing ForCES CEs
> to run another networking stack in user space is not an option, IMO.
> 
> For the C channel, I expect my CE components to usually run in user
> space; use of any IP transport protocol that can be found in an OS
> networking stack (or raw frames over L2 + a user-space transport) is
> acceptable for this (although there are other reasons to prefer one
> transport over another).
> 
> 
> Regards,
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Steven L. Blake               <[email-masked]>
> Ericsson IP Infrastructure                +1 919-472-9913
> 
��z�k���Y��X��X��Z+q�)��hr�bz��m��?�0�'�~���f��f��X��)ߣ��Ǭ��-��%

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 27 Feb 2004 20:48:20 +0800

Hi Alex,

Thank you for your comments. Pls see the inline reply.

Best regards,
Weiming
  ----- Original Message ----- 
  From: Alex Audu 
  To: Wang,Weiming 
  Cc: avri ; Chaoping Wu ; Deleganes Ellen M ; Dong, Ligang ; Guo, Yunfei ; Jamal Hadi Salim ; Jeff Pickering ; Jon Maloy ; Khosravi, Hormuzd M ; Patrick Droz ; Putzolu David ; Robert Haas ; Spyros Denazis ; Steve Blake ; Wang, Guangming 
  Sent: Friday, February 27, 2004 2:06 AM
  Subject: Re: ForCES Protocol Analysis Design Team Administrivia

  Hello Weiming,

  I'll address some of the issues you raised.

  1) Seperate Data Redirect and Control channels/ relative importance of Control vs Redirect info
      It is easy to see that the redirect data can be manipulated by an outside source (External to 
       the NE).  In the case of such a source being malicious, it is necessary to give priority to 
       the control data to maintain integrity of the NE.

        You sighted BGP experience.  Indeed, the realization that it is safer to separate externally
         generated control data like redirect-data from internally generated control data grew out
         of  experience gleamed from past DOS attacks on BGP.

  2)  C channel and D channel are on the same physical link.   For DOS survivability, you need 
        to seperate then somehow. If you use SCTP, you can put them on separate streams on
         the same link. That will work.  But what will be best is a non-reliable congestion aware
        stream/link for  D-channel and a reliable stream/link for C-channel.
         Rate controlling is important. But if  you need to allow your C information to flow freely
         to adaptively effect counter-meassures if the NE is being DOS'd.

  Re. 1&2:  I suppose a separation of D and C packets inside FE while using only one stream(channel) to accross the 
  physical link (GRMP DoS protection scheme) can also achieve all these goals (DoS protection, rate control, priority, 
  security, reliability). We just ask that if we don't need to provide the two diffenrent channels in the physical link so
   as to prevent DoS, are we still interested in giving the D packets a different transport reliability?

  3) Delay of D packets by  delay of C packets.   This sould not happen if your channels are
       independent. And  D packets may dictate what C packets get sent to the FE.  The
       problem might be related to the way your real-time processes are running the D and C 
       queues.  You can play with thread priorities,..etc to resolve this.
  Re: Yes, this is because of the congestion of queues of D packets and C packets, but the same will
  also happen to separated D/C channels if they use the same physical link. Only the difference is the latter
  may use OS kernel to schedule while the former uses ForCES protocol defined scheduler. 

  4) C/D separation resource cost:   This is not as expensive as the cost paid when DOS
       attacks occur.  Besides, hardware resources are getting cheaper by the day,..and 
       processors are getting faster too.  

  Re:  again, by separating the D packets and C packets inside an FE, we can also prevent DoS efficiently. 

  General comments. May be a way to go forward is to make it a SHOULD to implement
   (on seperate D/C channels) and leave the rest up to implementation,...I am not sure. Any comments?

  Re: I agree. What we propose is only to limit such a separation inside an FE instead of expanding it to the FE-CE link.

  Regards,
  Alex.21

  Wang,Weiming wrote:

Hi David,

Regarding the proposed basic principles, we (GRMP group) have some comments for  principles item 7&8:
--------
7.      The ForCES protocol should support separate channels for Control and Data messages. The data 
channel carries the routing/control protocol packets such as RIP, OSPF packets, between the CEs and FEs. 
These packets are unique in most cases for a particular CE:FE pair and hence the data channel will use unicast 
connections between the CEs and FEs. Also, the data channel does not have to be strictly reliable but does 
need to provide flow/congestion control to protect against DoS attacks.

8.      The control channel must provide strict reliability (non-corruption, no loss, no reordering, timeliness) at the 
transport level in accordance with the RFC 3654. In addition, ForCES protocol level responses/acknowledgements 
are also needed from FE to CE to learn the results of configuration/control messages.
--------

First of all, we agree that to distinguish ForCES messages into control messages and data (redirect) messages are 
essential for DoS protection scheme. Then, we argue that to separate the two kinds of messages into two different 
transport channels with the data channel being appointed to a lower reliable level (such as UDP, compared with TCP) 
may not be a good enough scheme for such purpose. The reasons are:

2. Generally speaking, separated C channel and D channel are still put over the same physical link (or else there is almost 
no need to protect DoS).  Therefore,  such a separation itself does not provide any DoS protection, instead, the protection is 
obtained by the rate control of D channel. We argue that  such a rate control actually can be obtained at the ForCES message 
layer instead of at the message transport layer. Some of our experiment results (which will be presented at the IETF meeting) 
have shown this feasibility.  Moreover, a separation at transport layer surely will complicate the CE-FE association and the 
followed message transport management. 

3.  We have also noticed a phenomenon during our experiments, which we may need to pay some attention. I recalled when 
FACT first presented the D and C channel idea,  there was a question worrying about the asyncronization between D and C channel.  
Our experiments have shown that,  although there is no strict asyncronization problem between D and C channel owing to the data 
independence, there is a possibility that D channel packets may be seriously delayed and even blocked owing to the busy traffic at 
C channel if D packets are separated from C packets. The busy traffic at C channel may happen when CE queries FE for something 
and FE dumps quite a huge number of messages (such as routing table items.)  Such a delay or blocking may be counted in seconds, 
resulted in very harmful effect on the users of redirected packets.  As a result, we propose that a moderate rate control for C channel 
be also necessary. 

4. To explicitly separate D and C channel also consumes more resources, because a resources for D channel have to be reserved. 
Instead, by separating D and C packets at the ForCES protocol layer and uniting them again at the transport layer, we may able to 
make a scheduling discipline by which the D and C packets can share the link resources effectively. We are trying to construct such 
a scheduling scheme, and it seems quite practical to so.

To summarize, we propose that the 7&8 basic principles only mention the separation of D and C packets for DoS protection currently, 
leaving some space for further discussions. The discussions may be like: 

1. Should such a separation be put at the ForCES message layer or at the transport layer?  Which is better?
2. Whether united scheduling for D and C packets are necessary?
3. Should D packets be distinguished from C packets for its transport reliability in ForCES protocol so as to assign a different transport means?

Regarding the protocol merge proposal, we'd like to present some of our thinking in a few words:

Firstly, as a new comer to IETF work, the GRMP group surely welcome such a merge proposal, sincerely hoping an all merits in 
one draft can be achieved for ForCES WG, and at the same time hoping to learn from other candidate authors, whom we have 
already been respecting to for their rich knowledge and excellent contributions to IETF and other fields. Secondly, we believe 
that the interaction among the candidate drafts are very helpful for the final ForCES protocol. Actually there already has been such 
interaction since the ForCES protocol evaluation process began, which has already made individual candidate drafts ever improved.  
Thirdly, to make the process successful, we believe that a nice atmosphere is very important. In such an atmosphere, everyone has 
a peaceable state of mind, precisely speaking, a mind ready to give up and accept, and individuals' work can be fully respected. 
Our past experience has made us believe that it is almost impossible to reach a compromise among group members where some 
think of their work much outstanding to others. Lastly, as Jamal has mentioned, some ground rules is important so as to eliminate 
conflicts whenever there are. 

Regarding the merge action way, we agree candidates to propose their "can't live without" first. Along with this, we also suggest 
candidates propose "what they think the contents in their drafts are useful for final protocol and hope able to be put in it". This useful 
contents are a little different from the "features of the drafts", because we suppose the contents proposed should be conservatively 
considered by presenters to be able to merge with other contents. 

Thank you.

Beset Regards,
Weiming Wang

P.S.: To avoid unexpected alignment for the email, I have also made a pdf format file in the attachment to ease reading. Thank you.

----- Original Message ----- 
From: "Putzolu, David" <[email-masked]>
To: <[email-masked]>; "Khosravi, Hormuzd M" <[email-masked]>; <[email-masked]>; <[email-masked]>; <[email-masked]>; "Steven Blake" <[email-masked]>; "Robert Haas" <[email-masked]>; <[email-masked]>; <alex.audu@alcat
el.com>; "Wang,Weiming" <[email-masked]>; "Patrick Droz" <[email-masked]>; "jeff pickering" <[email-masked]>; <[email-masked]>; "Deleganes, Ellen M" <[email-masked]>; <[email-masked]>; <[email-masked]>; "Putzolu, David" <[email-masked]>
Sent: Saturday, February 21, 2004 1:48 AM
Subject: ForCES Protocol Analysis Design Team Administrivia

All,

This email is going out to all of the current design
team volunteers.  In order to avoid problems with
long to/cc lists, I've asked the IETF to setup a temporary
reflector for us to use, and will subscribe all of us to
it once it is setup.  With any luck it will be ready
by Monday or Tuesday.

If you know of someone who volunteered for the team, but
is not on the recipient list of this email, please have
them contact me.

This team is chartered with examining the aspects of the
currently existing protocol proposals, seeing what common
ground there is, and determining whether the common ground
is sufficient for coming up with a single merged proposal.
The design team is to report out to the ForCES list by
March 20, 2004 (one month from today).

Below is an email that some of you received.  I've included 
it for the benefit of those who did not receive it.

Given the concern with word docs I think either emails
or (when necessary) PDF are reasonable formats.  I've
converted a word doc previously sent out by Hormuzd to 
PDF format and put it on the web for those who missed it:
http://www.sstanamera.com/~forces/proto/ForCESProtoDiscussion1.pdf

Below are some ground rules proposed by Jamal.  Are 
people comfortable with them/are there any additions
that should be made?

Cheers,
David

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
Sent: Friday, February 20, 2004 6:59 AM
To: Khosravi, Hormuzd M
Cc: Steven Blake; Robert Haas; Jon Maloy; 
[email-masked]; [email-masked]; 
[email-masked]; Wang,Weiming; Putzolu, David; Patrick 
Droz; jeff pickering
Subject: Re: ForCES protocol merger

I have added to the list Jeff Pickering who is a new couathor for us.

To start i will say that the netlink2 people are interested 
in being on
this boat. For this reason we have not releasd an updated netlink2
draft.

I dont like the brainstorming document as a starter.
Brainstorming can be done via emails or concals. And please no word
docs.
Our attempt in Mineapolis was in the wrong direction because there
were no ground rules on how to proceed. We need to agree on 
some ground
rules first otherwise we are going to end in some deadlock. 
I am going to suggest one which i came up with after the last
meeting:

As in any conflict, people should be open to giving up some things in
order to reach a compromise. I can understand that this may not be 
100% technical driven; what i am hoping for is to have less than 5%
policitical drive.
This is the main reason i suggested that each protocol 
authors list what
they cant live without. Lets iron that out so we can reduce conflict.
For each item draft authors cant live without, the how and 
justification
of why must be listed. 
In the IETF typical approach of such "cant live without mechanisms"
would be based on implementation and deployment experience. I have to
say i learnt a few things on implementing netlink2 that i would have
previously disagreed with - so that approach is useful.

I will try to be open hoping it is the correct thing to do to avoid
conflict:
Speaking from a political/bussiness side for myself: we have 
implemented
the full netlink2 which means we have invested time and 
resources in it
(and infact have deployed it in one site). However,
we are willing to throw all that away provided there will be progress
here which is not driven by politics rather by a technical drive.

So lets start with a list of ground rules.

cheers,
jamal

---

From: Alex Audu <[email-masked]>
Date: Fri, 27 Feb 2004 16:57:32 -0600

Hello Weiming,

See my comments in-line.

Regards,
Alex.

Wang,Weiming wrote:

>     From: Alex Audu <mailto:[email-masked]>
>     To: Wang,Weiming <mailto:[email-masked]>
>     Cc: avri <mailto:[email-masked]> ; Chaoping Wu
>     <mailto:[email-masked]> ; Deleganes Ellen M
>     <mailto:[email-masked]> ; Dong, Ligang
>     <mailto:[email-masked]> ; Guo, Yunfei
>     <mailto:[email-masked]> ; Jamal Hadi Salim <mailto:[email-masked]> ;
>     Jeff Pickering <mailto:[email-masked]> ; Jon Maloy
>     <mailto:[email-masked]> ; Khosravi, Hormuzd M
>     <mailto:[email-masked]> ; Patrick Droz
>     <mailto:[email-masked]> ; Putzolu David
>     <mailto:[email-masked]> ; Robert Haas
>     <mailto:[email-masked]> ; Spyros Denazis
>     <mailto:[email-masked]> ; Steve Blake
>     <mailto:[email-masked]> ; Wang, Guangming
>     <mailto:[email-masked]>
>     Re. 1&2:  I suppose a separation of D and C packets inside FE
>     while using only one stream(channel) to accross the
>     physical link (GRMP DoS protection scheme) can also achieve all
>     these goals (DoS protection, rate control, priority,
>     security, reliability). We just ask that if we don't need
>     to provide the two diffenrent channels in the physical link so
>      as to prevent DoS, are we still interested in giving the D
>     packets a different transport reliability?
>
[Alex] If I understand your question correctly, no,..this does not mean 
you are giving D
           packets a different transport reliability. But if you carry D 
and C data on the same
           channel, you have to be mindful not to let D data impact flow 
of C data.
           And frankly, I don't know what that means regarding your 
question.

>     also happen to separated D/C channels if they use the same
>     physical link. Only the difference is the latter
[Alex]  Say you are been DOS'd with a million D packets, and your C 
sends one message to
            help counter the problem.  Your packet separator will have 
to do a lot of work
             before getting to the C packets in the queue.  I don't 
think this makes for quick
            recovery from the attack.

>      (on seperate D/C channels) and leave the rest up to
>     implementation,...I am not sure. Any comments?
>      
>     Re: I agree. What we propose is only to limit such a separation
>     inside an FE instead of expanding it to the FE-CE link.
>
>     Regards,
>     Alex.21
>
..[snip]..

---

From: "Weiming Wang" <[email-masked]>
Date: Sat, 28 Feb 2004 12:08:54 +0800

Hi Alex, 

Pls see more reply in the lines.

Thank you.

Weiming
  ----- Original Message ----- 
  From: Alex Audu 
  To: Wang,Weiming 
  Cc: [email-masked] 
  Sent: Saturday, February 28, 2004 6:57 AM
  Subject: Re: [Forces-protocol] Re: ForCES Protocol Analysis Design Team Administrivia

  Hello Weiming,

  See my comments in-line.

  Regards,
  Alex.

  Wang,Weiming wrote:

    Hi Alex,

    Best regards,
    Weiming
      ----- Original Message ----- 
      From: Alex Audu 
      To: Wang,Weiming 
      Cc: avri ; Chaoping Wu ; Deleganes Ellen M ; Dong, Ligang ; Guo, Yunfei ; Jamal Hadi Salim ; Jeff Pickering ; Jon Maloy ; Khosravi, Hormuzd M ; Patrick Droz ; Putzolu David ; Robert Haas ; Spyros Denazis ; Steve Blake ; Wang, Guangming 
      Sent: Friday, February 27, 2004 2:06 AM
      Subject: Re: ForCES Protocol Analysis Design Team Administrivia

      Hello Weiming,

      Re. 1&2:  I suppose a separation of D and C packets inside FE while using only one stream(channel) to accross the 
      physical link (GRMP DoS protection scheme) can also achieve all these goals (DoS protection, rate control, priority, 
      security, reliability). We just ask that if we don't need to provide the two diffenrent channels in the physical link so
       as to prevent DoS, are we still interested in giving the D packets a different transport reliability?
  [Alex] If I understand your question correctly, no,..this does not mean you are giving D 
             packets a different transport reliability. But if you carry D and C data on the same
             channel, you have to be mindful not to let D data impact flow of C data.
             And frankly, I don't know what that means regarding your question.

  [WangRe]
  We are just using the model presented in GRMP (slave model), which by experiments have initially shown 
  the feasibility to protect DoS. 

      Re:  again, by separating the D packets and C packets inside an FE, we can also prevent DoS efficiently. 
  [Alex]  Say you are been DOS'd with a million D packets, and your C sends one message to
              help counter the problem.  Your packet separator will have to do a lot of work
               before getting to the C packets in the queue.  I don't think this makes for quick
              recovery from the attack.

  [WangRe] No, this is exactly the same as D packets are put on a different channel (as your notion, different stream), the only exception is the stream scheduling is done by OS kernel scheduling (usually using round robin), while in our scheme it is done by ForCES assigned scheduling whose scheduling decispline can be flexibly set by ForCES protocol, and D/C packets can be very easily and differentially scheduled.  The problem you mentioned will also happen to a separate stream scheme if the OS kernel scheduler is not good enough. 
  Remember that congestion happens when DoS attacks only because there is not enough physical link layer resources. Therefore, separation of the D packets to a specific stream itself is of no use to prevent DoS. 

      Regards,
      Alex.21

  ..[snip]..