Subject: [Forces-protocol] Comments from chat with ADs

From: "Putzolu, David" <[email-masked]>
Date: Tue, 2 Mar 2004 05:06:01 -0800

Patrick and I spent about 90 minutes last night
chatting with Bill and Alex about ForCES, with
maybe half of the time on the protocol.  In
the conversations with them, along with some
follow-up chats I had with Allison Mankin, the
following topics where covered:  multiple 
channels, multicast/unicast, and congestion 
control.  Below is what I recall from memory 
(Patrick please correct me/elaborate as needed :)

Multiple channels:  

I discussed the interference issue that 
was identified by the GRMP team.  Bill &
Alex agreed that many OS implementations can have
an interference property (i.e. where two different
connections will have queuing interference lower in
the networking stack in the OS).  So it is necessary
to make sure when implementing a FE (or CE probably
as well) that this is taken into account.  Methods
of doing that include:
* Manually queuing packets (i.e. feed OS few enough
  packets that it cannot get into trouble - although
  you pay a performance price for that, in that fine
  grained scheduling takes cycles)
* Indicating to the OS to treat flows differently - 
  some real-time OS will let you do this, that is they
  are implemented to internally retain QoS between
  flows.
However, it was also mentioned that intra-FE 
interference is only one kind of interference.  Good
FE implementation (e.g. by manually queuing above the
OS) helps to fix intra-FE interference, but it does
not solve inter-FE interference.  That is, if all
(C & D) packets are always on the same connection,
that makes it impossible on the wire to differentiate
them from one another.  Imagine you have a bunch of
FEs with some of them being subject to DOS attack for
D packets, and you want to query one of the FEs for
statistics.  It would be good to have some way of
ensuring the C packets (or any high priority packets -
OSPF HELLO maybe) get through.  Separate channels
(as Jamal indicated, don't even have to be C & D)
makes this possible.   Putting everything on one
channel makes it impossible to deal with inter-FE
interference in a differentiated fashion. 

Multicast & Unicast:  

The discussion on this started out with a discussion
of the IETF protocol design BKM that options can ruin
compatibility.  That is, if there are four ways to
implement something, all optional, then you can have
client 1 implement A & B, and client 2 implement C & D,
and even though both clients are 100% compliant, they
cannot talk to one another. Better to say all clients
must implement A and make B, C, and D optional, so that
way you know that all implementations will be able to 
at least talk.

Patrick & I pushed on this a bit, explaining that we
actually had potentially (at least) two different
environments - very close environments where multicast
support could be present in the interconnect,  and 
not-as-close environments (and some close
environments as well) where multicast was not available.

Bill & Alex indicated that if there really are two
quite different cases, then it is possible to have a
"dual mandatory" approach.  I.e.:

If your underlying interconnect supports multicast,
then you MUST implement the following (multicast)
method of communication.

If your underlying interconnect supports unicast, then
you MUST implement the following (unicast) method
of communication.

This will guarantee interop, although at a potential
extra cost (cost being having to potentially support
both when both are present).  While this isn't optimal,
it does at least give some flexibility.

Congestion control:

Several times over the whole conversation the topic
of congestion control came up.  Also, as part of some
TIPC discussions, I ended up spending some time talking
to Allison Mankin (transport AD) about congestion control
as well.  All the ADs I talked to where consistent on 
the following:

If you are running over IP, you MUST act in a RFC2914
compliant fashion (i.e. respond to congestion like TCP
or SCTP or RMT does).  Two reasons where put forth for
this:
1) Any time you define an IP-based protocol you cannot
guarantee that it will not go out over the Internet. As
such, the IESG will NOT standardize IP protocols with
"bad" (non-2914) behaviors.
2) Lots of implementation experience showing that other
congestion control mechanisms don't work (congestion
collapse etc).

They where very firm on the above and my impression was
that convincing them otherwise would be very difficult.

That being said, a second alternative did arise:  If
you are not implementing on top of IP (e.g. running on
Infiniband, ATM, Ethernet) then you could rely on mechanisms
for congestion control built into what you are relying on,
or even consider building your own congestion control
mechanisms (ala TIPC).  This makes it possible to define
(for example) a multicast transport for forces without
having to go all the way and implementing RMT - as long
as it is clear that it isn't over IP.  The ADs went as
far as saying that the IETF could even accept standards-
track drafts that say nothing about IP (e.g. define an
Ethernet-only transport for ForCES), if it is necessary 
for ForCES. GSMP and (some other group I forgot) was
given as an example.

Cheers,
David

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 5 Mar 2004 13:35:08 +0800

Hi David,

Some more comments on the multi-channels from your last email as below:

I agree that multiple FEs sharing the same physical link makes problems more complicated. To make things more clear, I want state GRMP's ideas
again here, for I think there might be some misunderstanding on it:

1. GRMP presents that pure separation of C/D in the same physical link is of no use to DoS protection. DoS protection can only be achieved by inside FE mechanisms except  a separate physical link is used by C/D. This also applies to multi-FEs sharing the same physical links as you mentioned above. Actually I can not see such separation has solved the DoS problem for multi-FE case from the above txt.  

2. We do not oppose to have such separation if it's proven useful for others. Actually the key GRMP has defined for DoS and congestion control is the FE attribute for DoS protection policy, which has no limitation with multi transport layer channels. If it's proven separation of C/D channels are useful, GRMP can easily adapt it. But what we need more discussion is trying to see more reasons for such separations. 

3. A combined control of C and D is useful for DoS protection as well as for congestion. 
I have to note that the scheme proposed by FACT and also mentioned in the proposed basic merge principles only assume to provide  a Data channel control mechanism ( a rate limit). We just argue that this is not enough. Actually what you mentioned above is also to ask for such a control combination for multi-FEs cases. GRMP scheme can supply this very well by using a FE attribute to specify DoS protection policy in which C packets with higher priorities can be ensured. I don't think other protocols have provided this mechanism (Note that this priority is different from that in ForCES message headers, which actually does not take effect for this purpose).  

To summarize, we just argue that a DoS protection policy like that presented by GRMP should be included in final ForCES protocol other than just by separation of C/D channels so as to effectively avoid DoS attack and also to manage the congestion. I think this idea is quite the same as what you presented in another email as some QoS hints transmitted from CE to FE for DoS protection and congestion control. 

Yours,
Weiming

----- Original Message ----- 
From: "Putzolu, David" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, March 02, 2004 9:06 PM
Subject: [Forces-protocol] Comments from chat with ADs

Multiple channels:  

[WangRe]

¢·²šè¶‡(–f¢–)à–+-ŠÜzÊk¢Ú¢Xžµú+‚m¦ÏÿÃ5‰ë_¢¸?™¨¥™©ÿ–+-Šwèýú+që)®‹hr‰

---

From: Alex Audu <[email-masked]>
Date: Fri, 05 Mar 2004 10:16:43 -0600

Hello Weiming,

I still believe that the result you got from your experiment could be 
much more
improved if you allocated two separate queues, one for C and the other 
for D
data. Then create two separate threads to process the queues (one for each).
Give both threads equal priority and you should be fine. If you want to skew
things in favor of the C, then you can adjust priority accordingly.

Regards,
Alex.

Wang,Weiming wrote:

>ŠÜzÊk¢Ú¢YšŠX§‚X¬´Z+që)®‹hr‰bz×è®m¶›?ÿ0Ö'~Šàþf¢–f§þX¬¶)ß£÷èÇ¬¦º-¡Ê%
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: 05 Mar 2004 14:55:49 -0500

I think theres a lot of implementation details being tossed around
as opposed to what the protocol needs. 
Weiming, i apologize for not having paid close scrutiny to GRMPs issue
of DOS in you draft; however looking at the slides Ligang presented it
seems to me the issue maybe implementation. I thought David did a good
job explaining the link aspect of things. I actually agree with you that
a single link may be insufficient to prevent a DOS. In particular this
issue gets exaceberated if the DOS is infact filling the pipe. 
The suggestion was if you could prioritize before the packets hit the
OS, such as via different DMA channels, things will improve (because the
DOS is limited to attacking the physical link). This is of course only
true until someone starts DOSing with high priority packets and filling
that high prio DMA channel. The problem is not solvable if all DMA
channels regardless of priority are being DOSed. This also applies to
the dual (physical) links. i.e you cant protect if all physical links
are being attacked. The solution is to have a trusted vs non trusted
link.
And if you have trusted sources, then can do a very good job addressing
that issue with the multiple DMA channels.

Once the packet hits the OS, you need not only to protect against
bandwidth, but also against CPU(thorugh thread prioritization alex is
suggesting) and RAM abuse. i.e you need to protect all those three
resources by providing QOS for them. Some OSes help you more than others
to achieve this.

So far all the above is an implementation issue. Nobody will teach you
these tricks, but if you want to have a robust implementation, you will
need to do the above. This is regardless of the protocol (OSPF, BGP
etc). i.e has nothing to do with Forces.

The only protocol/header semantic that is valuable is prioritization
bit which are specific to Forces. Is this what you are refering to?
I believe (I could be mistaken) all three proposals have this. This is
valuable. However, this should probably be optional. Its easier
to use whatever the underlying mechanism provides (802.1p, diffserv etc)
To quote from netlink2 draft:

"  Prioritization is an orthogonal mechanism to reliability.  When a
   node runs out of resources, a message sent with a higher priority
   will get preferential treatment.  For instance, if a FE has only
   enough memory to allocate one message in response to a message from
   the CE and it has to choose between one of two messages to respond
   to, then it will use that memory for the request which was sent with
   the higher priority.  This also applies to other resources such as
   computing cycles 
"

cheers,
jamal

---

From: [email-masked]
Date: Sat, 6 Mar 2004 07:41:40 +0800

Hi Alex,

Because we have several experiment cases,  could you tell more which one you 
are addressing and what is the result you think should be?

I strongly suggest  the same experiments be done by more guys here so that we 
can share to analyze the results.

Cheers,
Weiming

Quoting Alex Audu <[email-masked]>:

> >I agree that multiple FEs sharing the same physical link makes problems more
> complicated. To make things more clear, I want state GRMP's ideas
> >1. GRMP presents that pure separation of C/D in the same physical link is of
> no use to DoS protection. DoS protection can only be achieved by inside FE
> mechanisms except  a separate physical link is used by C/D. This also applies
> to multi-FEs sharing the same physical links as you mentioned above. Actually
> I can not see such separation has solved the DoS problem for multi-FE case
> from the above txt.  
> >
> >2. We do not oppose to have such separation if it's proven useful for
> others. Actually the key GRMP has defined for DoS and congestion control is
> the FE attribute for DoS protection policy, which has no limitation with
> multi transport layer channels. If it's proven separation of C/D channels are
> useful, GRMP can easily adapt it. But what we need more discussion is trying
> to see more reasons for such separations. 
> >
> >I have to note that the scheme proposed by FACT and also mentioned in the
> proposed basic merge principles only assume to provide  a Data channel
> control mechanism ( a rate limit). We just argue that this is not enough.
> Actually what you mentioned above is also to ask for such a control
> combination for multi-FEs cases. GRMP scheme can supply this very well by
> using a FE attribute to specify DoS protection policy in which C packets with
> higher priorities can be ensured. I don't think other protocols have provided
> this mechanism (Note that this priority is different from that in ForCES
> message headers, which actually does not take effect for this purpose).  
> >
> >To summarize, we just argue that a DoS protection policy like that presented
> by GRMP should be included in final ForCES protocol other than just by
> separation of C/D channels so as to effectively avoid DoS attack and also to
> manage the congestion. I think this idea is quite the same as what you
> presented in another email as some QoS hints transmitted from CE to FE for
> DoS protection and congestion control. 
> >
> >Yours,
> >Weiming
> >

-------------------------------------------------
This mail sent through HUCNIC Webmail system.

---

From: "Wang,Weiming" <[email-masked]>
Date: Sun, 7 Mar 2004 02:28:12 +0800

Hi Jamal, and Alex,  

Thank you very much for your kind suggestion for the implementation. 
But what I should clarify is that the experiment we do are all 
in a situation that it follows the ForCES protocol. This means, e.m., 
for the UDP and TCP separation channel experiments, we just assume that
ForCES use the separate channel and then we actually produce redirection 
packets from outside of the FE and the FE itself produces Control packets
to pass through it. It's true we do not go deep into the thread scheduling
or even DMA channel management instead we just use default strategy with 
equal priority for the two threads, because we think even if we need to do
this, it should be controlled by ForCES CE via setting some FE attribute (like
David said some QoS hints) to the FE. This has also proven that a kind of mechanism
like FE attribute for DoS protection policy shoud be included in ForCES protocol. 

I basically follows Jamal's issue list on the congestion control issue, but have used
this "DoS protection congestion control mechanisms" as the name, I think these two issues
have been quite tightly connected. 

To ease the discussion, I just want to summarize our thoughs towards the issue based on 
recent discussions as well as on experiments as below:

1. Agree to the idea to distinguish packets by Data packets and Control packets, where 
Data packets are actually redirected packets by FE to CE, and the Control packets are actually ForCES protocol
messages generated by FE and should be transported to CE.

2. To protect DoS and control congestion better, an FE attributes as FE DoS protection and congestion
control policy should be included in ForCES protocol. This also means some kind of scheduling strategies
should be applied to FE as well as rate limiters to prevent DoS and control congestion. 

3. Whether such control should be implemented at ForCES protocol layer or at OS layer, the individual pros and cons
should be further discussed.

4. A separation applied at transport layer to the two kinds of packets has little effect to protect DoS.
Whether to apply it should mainly depends on other reasonable reasons. If the reasons are enough, we are glad
to see it in the protocol. This item may specifically be discussed in the multichannel issue (Issue 2). 

5. A case in which the Data packets may be blocked by Control packets owing to improper congestion control
setup, and the harm of such case (delays the packtes delivery) may be addressed in the protocol. 

Other items for which we just want to raise discussion in this Issue 4 are as:

6. The DoS alert way. Actually GRMP has used an FE attribute as "FE DoS alert policy" and FE event as "FE DoS alert event"
for this purpose. We just suggest to use this useful mechanism.

7. Rate control for Control packets as defined above may can only be manipulated by ForCES protocols instead by LFBs
defined by FE models, for it seems be out of the scope of FE models because the packtes pass it are ForCES protocol 
messages and it should be launched before the FE model mechanism can work.

The prioritization and associated priority definition in ForCES protocol seems have a lot of confusions among the three
candidates and among the discussions. I would suggest to have this as another special issue for more discussion. 

Yours,
Weiming

----- Original Message ----- 
From: "Jamal Hadi Salim" <[email-masked]>
To: <[email-masked]>
Cc: "Wang,Weiming" <[email-masked]>; "Putzolu, David" <[email-masked]>; <[email-masked]>
Sent: Saturday, March 06, 2004 3:55 AM
Subject: Re: [Forces-protocol] Comments from chat with ADs

> > things in favor of the C, then you can adjust priority accordingly.
����趇(�f��)��+-��z�k���X���+�m����5��_��?�������+-�w���+q�)��hr�

---

From: [email-masked]
Date: Mon, 8 Mar 2004 20:29:35 +0900

I definitely recommend this approach. It simplifies all of the 
documents and leaves the option open foe people to standardized using 
the protocol over their underlying layer/protocol of choice.

a.

On 2 mar 2004, at 22.06, Putzolu, David wrote:

> The ADs went as

---

From: Alex Audu <[email-masked]>
Date: Mon, 08 Mar 2004 09:44:28 -0600

Hello Weiming,

I was refering to the C/D different queueing experiment (I think it is 
Expeiriment II).
The bottom line is you should be able to decouple events on C queue and 
D queue from
interfering with each other.   You can  set up simulation runs on OPNET 
or similar
environments to check this out.

Regards,
Alex.

[email-masked] wrote:

>>>3. A combined control of C and D is useful for DoS protection as well as for
>>>      
>>>
>>congestion. 
>>    
>>
>>>S(ÜzÊk¢Ú¢Ys(S(X§,X¬´Z+që)®?ÿ0Ö'~S(àþf¢-f§þX¬¶)ß£÷èÇ¬¦º-¡Ê%
>>>
>>>      
>>>

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 9 Mar 2004 13:27:30 +0800

Hi Alex,

The experiment II result actually only shows that the C and D channel will interfere with 
each other when the physical link they are on is overloaded (50%+80%>100%).
I think this is reasonable. You may never expect the two channels could still work as you expected 
when the phycisal link has been overloaded. 

Yours,
Weiming
----- Original Message ----- 
  From: Alex Audu 
  To: [email-masked] 
  Cc: [email-masked] 
  Sent: Monday, March 08, 2004 11:44 PM
  Subject: Re: [Forces-protocol] Comments from chat with ADs

  Hello Weiming,

  I was refering to the C/D different queueing experiment (I think it is Expeiriment II).
  The bottom line is you should be able to decouple events on C queue and D queue from
  interfering with each other.   You can  set up simulation runs on OPNET or similar
  environments to check this out. 

  Regards,
  Alex.