Subject: RE: [Forces-protocol] Comments from chat with ADs

From: "Deleganes, Ellen M" <[email-masked]>
Date: Fri, 5 Mar 2004 12:18:33 -0800

I think what Hormuzd is saying (Hormudz can correct me if I am wrong) is
that even if Multicast is optional, those procedures that noted in
Hormudz's email should be described in the ForCES protocol spec. That
is, if a ForCES protocol implementation supports multicast, then it MUST
implement it in the manner described in the ForCES protocol
specification. Otherwise, ForCES implementations using multicast will
probably not interoperate.

Whether or not implementing multicast for transports that support
multicast is a requirement is a separate question. 

Regards,
Ellen

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Khosravi, Hormuzd M
Sent: Thursday, March 04, 2004 6:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Forces-protocol] Comments from chat with ADs

Hi Alex,

In general, I prefer using language such as MUST so that there is no
confusion or interop issues in the protocol implementations. However,
you raise a reasonable point and let me tell you my thoughts on this. We
will definitely need Unicast between FEs and CEs at the minimum in the
protocol. In addition, if the interconnect technology supports Multicast
and there is a case where it is useful to the application (such as
downloading IPv4 routes) we must allow it as well without breaking
interop. In order to do that, we will need specific procedures in the
protocol that will allow the CE and FE to negotiate on using multicast
dynamically at run-time (say during the binding or joining phase between
the CE and FE) and also associate different multicast groups with LFBs.
We will also need to define how Reliability/CC etc. will be provided for
multicast...may be using RMT, TIPC, etc. (?) If we do this correctly,
then applications will be able to use both unicast and multicast without
any interop issues.

Let me know what you guys think about this.

Regards
Hormuzd

-----Original Message-----
From: Alex Audu [mailto:[email-masked]] 
Sent: Thursday, March 04, 2004 1:22 PM
To: Khosravi, Hormuzd M
Cc: [email-masked]
Subject: Re: [Forces-protocol] Comments from chat with ADs

Hormuzd,

Why should multicast be "mandated for certain cases where it will be 
useful" if there
is a simpler, more robust way to do the same thing?  I'll say we don't 
mandate this.
We can make it a "SHOULD" strength, not a "MUST".  We mandate UNICAST
and make MULTICAST optional.  Any issues with this?

Regards,
Alex.

Khosravi, Hormuzd M wrote:

>Hi David
>
>Your summary below of the discussion with Ads on the different issues
>seems like very reasonable approaches to me.
>
>I agree that it will be a good idea to present transport alternatives
at
>IP and non-IP level for the ForCES protocol. We can mandate one or more
>transport for IP and one or more transport for non-IP, this way the
>interoperability requirements are met and at the same time, the in-box,
>out of box scenarios are met.
>
>Also, agree with having multiple channels/connections to differentiate
C
>& D traffic between CEs and FEs. On the unicast/multicast topic, dual
>mandatory approach again seems a good idea to support interoperability.
>The one thing that I would like to point out is that for any ForCES
>protocol implementation to work, unicast will definitely be required at
>the minimum. One could also mandate using multicast for certain cases
>where it will be useful (e.g. IPv4/config table download), if it is
>supported by the interconnect as you stated. But just multicast without
>unicast will not work, atleast not efficiently.
>
>Regards
>Hormuzd
>
>-----Original Message-----
>From: [email-masked]
>[mailto:[email-masked]] On Behalf Of Putzolu, David
>Sent: Tuesday, March 02, 2004 5:06 AM
>To: [email-masked]
>Subject: [Forces-protocol] Comments from chat with ADs
>
>Patrick and I spent about 90 minutes last night
>chatting with Bill and Alex about ForCES, with
>maybe half of the time on the protocol.  In
>the conversations with them, along with some
>follow-up chats I had with Allison Mankin, the
>following topics where covered:  multiple 
>channels, multicast/unicast, and congestion 
>control.  Below is what I recall from memory 
>(Patrick please correct me/elaborate as needed :)
>
>
>
>Multiple channels:  
>
>I discussed the interference issue that 
>was identified by the GRMP team.  Bill &
>Alex agreed that many OS implementations can have
>an interference property (i.e. where two different
>connections will have queuing interference lower in
>the networking stack in the OS).  So it is necessary
>to make sure when implementing a FE (or CE probably
>as well) that this is taken into account.  Methods
>of doing that include:
>* Manually queuing packets (i.e. feed OS few enough
>  packets that it cannot get into trouble - although
>  you pay a performance price for that, in that fine
>  grained scheduling takes cycles)
>* Indicating to the OS to treat flows differently - 
>  some real-time OS will let you do this, that is they
>  are implemented to internally retain QoS between
>  flows.
>However, it was also mentioned that intra-FE 
>interference is only one kind of interference.  Good
>FE implementation (e.g. by manually queuing above the
>OS) helps to fix intra-FE interference, but it does
>not solve inter-FE interference.  That is, if all
>(C & D) packets are always on the same connection,
>that makes it impossible on the wire to differentiate
>them from one another.  Imagine you have a bunch of
>FEs with some of them being subject to DOS attack for
>D packets, and you want to query one of the FEs for
>statistics.  It would be good to have some way of
>ensuring the C packets (or any high priority packets -
>OSPF HELLO maybe) get through.  Separate channels
>(as Jamal indicated, don't even have to be C & D)
>makes this possible.   Putting everything on one
>channel makes it impossible to deal with inter-FE
>interference in a differentiated fashion. 
>
>
>Multicast & Unicast:  
>
>The discussion on this started out with a discussion
>of the IETF protocol design BKM that options can ruin
>compatibility.  That is, if there are four ways to
>implement something, all optional, then you can have
>client 1 implement A & B, and client 2 implement C & D,
>and even though both clients are 100% compliant, they
>cannot talk to one another. Better to say all clients
>must implement A and make B, C, and D optional, so that
>way you know that all implementations will be able to 
>at least talk.
>
>Patrick & I pushed on this a bit, explaining that we
>actually had potentially (at least) two different
>environments - very close environments where multicast
>support could be present in the interconnect,  and 
>not-as-close environments (and some close
>environments as well) where multicast was not available.
>
>Bill & Alex indicated that if there really are two
>quite different cases, then it is possible to have a
>"dual mandatory" approach.  I.e.:
>
>If your underlying interconnect supports multicast,
>then you MUST implement the following (multicast)
>method of communication.
>
>If your underlying interconnect supports unicast, then
>you MUST implement the following (unicast) method
>of communication.
>
>This will guarantee interop, although at a potential
>extra cost (cost being having to potentially support
>both when both are present).  While this isn't optimal,
>it does at least give some flexibility.
>
>
>
>Congestion control:
>
>Several times over the whole conversation the topic
>of congestion control came up.  Also, as part of some
>TIPC discussions, I ended up spending some time talking
>to Allison Mankin (transport AD) about congestion control
>as well.  All the ADs I talked to where consistent on 
>the following:
>
>If you are running over IP, you MUST act in a RFC2914
>compliant fashion (i.e. respond to congestion like TCP
>or SCTP or RMT does).  Two reasons where put forth for
>this:
>1) Any time you define an IP-based protocol you cannot
>guarantee that it will not go out over the Internet. As
>such, the IESG will NOT standardize IP protocols with
>"bad" (non-2914) behaviors.
>2) Lots of implementation experience showing that other
>congestion control mechanisms don't work (congestion
>collapse etc).
>
>They where very firm on the above and my impression was
>that convincing them otherwise would be very difficult.
>
>That being said, a second alternative did arise:  If
>you are not implementing on top of IP (e.g. running on
>Infiniband, ATM, Ethernet) then you could rely on mechanisms
>for congestion control built into what you are relying on,
>or even consider building your own congestion control
>mechanisms (ala TIPC).  This makes it possible to define
>(for example) a multicast transport for forces without
>having to go all the way and implementing RMT - as long
>as it is clear that it isn't over IP.  The ADs went as
>far as saying that the IETF could even accept standards-
>track drafts that say nothing about IP (e.g. define an
>Ethernet-only transport for ForCES), if it is necessary 
>for ForCES. GSMP and (some other group I forgot) was
>given as an example.
>
>Cheers,
>David
>
>_______________________________________________
>Forces-protocol mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/forces-protocol
>
>
>_______________________________________________
>Forces-protocol mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/forces-protocol
>  
>

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: [email-masked]
Date: Fri, 5 Mar 2004 17:33:45 -0500

Hello Hormuzd,

Comments are line.

> I agree that it will be a good idea to present transport 
> alternatives at

Ok.

> 
> Also, agree with having multiple channels/connections to 
> differentiate C

> the minimum. 

> One could also mandate using multicast for certain cases

Mandate one and make option depending upon the operating environment 
and availability of interconnect. But for single box solution, it will be less of multicast
and its mostly broadcast... For close proximity its responsibility of the operator to ensure 
what type of interconnect can be used.....

Regards
Ramg

---

From: [email-masked]
Date: Fri, 5 Mar 2004 17:28:38 -0500

Hello All,

I just came back after a break due to some personal work. I have to catch up with all other email threads.

I agree to the summary and below are my comments.

> [mailto:[email-masked]]On Behalf Of ext Putzolu, David
> Sent: Tuesday, March 02, 2004 8:06 AM
> was identified by the GRMP team.  

Not sure how GRMP has experienced this - may be more detail if possible.

> Bill &

I agree. We have implemented this using earlier version of FACT with multiple stack implementation on QNX (RTOS) and FreeBSD platforms.

FreeBSD implementation is available publicly at http://www.tel.fer.hr/zec/papers/zec-03.pdf and people have used this and tweaked such multiple stacks
to resist against DoS attack (logical channel separation for various C and D types of packets with different rate limiter values ) 

> statistics.  

If the query is from CE -> FE with the support of different rate control mechanism, and always allowing CE -> FE message to get processed FE as that is more important, we can achieve this. 

But there is a limit to which we  can control or defend against DoS  scenario.

> It would be good to have some way of

currently there are various solutions exists that uses same physical channel, and  tries to provide resist to DoS attacks...
- Multiple stack implementation
- Rate control (Asymmetric rate control)
- Perflow queuing 
and many other solutions..

> way you know that all implementations will be able to 

I agree, too much option and generaliazation will run into problems.
We have systems implementing unicast both for single box and also for muliple-hops. 
I don't remember exactly even systems on Point-Of-Presence uses unicast as they want reliability.  

> environments as well) where multicast was not available.

is the interconnect refers to  layer-2 or layer-3 support? 

> you MUST implement the following (unicast) method

First unicast - this look like a default case.   
I'm not aware of the case where we have multicast and not unicast?

To accommodate multicast then if the underlying mechanism supports then its more like 
a configuration options.

> it does at least give some flexibility.

Mandate unicast and mulicast when layer-2 or layer -3 is supported may little bit easier.

For example in OSPF if the routers are on shared LAN then multicasting takes place or else its unicasting for hub-and-spoke topology....

Regards
Ramg

---

From: [email-masked]
Date: Fri, 5 Mar 2004 17:50:54 -0500

Hello Weiming,

Alex as already answered to this - just to add more to his question my comments are line.

> [mailto:[email-masked]]On Behalf Of ext Wang,Weiming
> Sent: Friday, March 05, 2004 12:35 AM
> To: Putzolu, David; [email-masked]
> Hi David,
> Some more comments on the multi-channels from your last email 
> as below:
> 
> I agree that multiple FEs sharing the same physical link 
> makes problems more complicated. To make things more clear, I 
> want state GRMP's ideas
> again here, for I think there might be some misunderstanding on it:
> 
> 1. GRMP presents that pure separation of C/D in the same 
> physical link is of no use to DoS protection. DoS protection 
> can only be achieved by inside FE mechanisms except  a 
> separate physical link is used by C/D. This also applies to 
> multi-FEs sharing the same physical links as you mentioned 
> above. Actually I can not see such separation has solved the 
> DoS problem for multi-FE case from the above txt.  
> 
> 2. We do not oppose to have such separation if it's proven 
> useful for others. Actually the key GRMP has defined for DoS 
> and congestion control is the FE attribute for DoS protection 
> policy, which has no limitation with multi transport layer 
> channels. If it's proven separation of C/D channels are 
> useful, GRMP can easily adapt it. But what we need more 
> discussion is trying to see more reasons for such separations. 
> 
> 3. A combined control of C and D is useful for DoS protection 
> as well as for congestion. 
> I have to note that the scheme proposed by FACT and also 
> mentioned in the proposed basic merge principles only assume 
> to provide  a Data channel control mechanism ( a rate limit). 
> We just argue that this is not enough. Actually what you 
> mentioned above is also to ask for such a control combination 
> for multi-FEs cases. GRMP scheme can supply this very well by 
> using a FE attribute to specify DoS protection policy in 
> which C packets with higher priorities can be ensured. I 
> don't think other protocols have provided this mechanism 
> (Note that this priority is different from that in ForCES 
> message headers, which actually does not take effect for this 
> purpose).  

May be your understand is not complete on FACT.

In FACT we proposed two rate limiter value one for C and another one for D in adddition the rate from
CE -> FE is different and from  FE -> CE. FE after processing  the packet discrimiates the C and D and accordingly 
act on it.   We have mentioned in our draft that rate limiting can be done based on even various traffic (C and D) types.
FE endpoint has intelligence to perform such operation.  Hope this helps.  

 End of comments
Regards
Ramg

> 
> To summarize, we just argue that a DoS protection policy like 
> that presented by GRMP should be included in final ForCES 
> protocol other than just by separation of C/D channels so as 
> to effectively avoid DoS attack and also to manage the 
> congestion. I think this idea is quite the same as what you 
> presented in another email as some QoS hints transmitted from 
> CE to FE for DoS protection and congestion control. 
> 

> Yours,
> Weiming
> 
> ----- Original Message ----- 
> From: "Putzolu, David" <[email-masked]>
> To: <[email-masked]>
> Sent: Tuesday, March 02, 2004 9:06 PM
> [WangRe]
> SzÊ¢Ú¢YSX'X´Zqë®<r?z×®¶>ÿ> Ö'~Sþf-fþX¶)£÷Ç¦º¡Ê
>

---

From: "Putzolu, David" <[email-masked]>
Date: Fri, 5 Mar 2004 15:25:52 -0800

Nice to have you back Ram!

I agree that _defining_ the FEM/CEM interface & associated
FEM/CEM messages is outside our scope.

That being said, the protocol could assert that certain
functions will be done in the FEM/CEM interface (as opposed
to by the protocol itself).  I.e. "We do not do (foo) here -
we assert that it has already been done.  (Foo) is
expected to be done during by the FEM/CEM and the mechanisms 
for doing (Foo) in FEM/CEM are beyond the scope of this 
document."

Cheers,
David

> [mailto:[email-masked]] On Behalf Of 
> [email-masked]
> Sent: Friday, March 05, 2004 3:13 PM
> To: [email-masked]; [email-masked]
> Cc: Khosravi, Hormuzd M; Deleganes, Ellen M; [email-masked]
> Hello Jamal,
> FEM/CEM interface and message description is outside the 
> scope of the ForCES framework. I'm not sure why we to deal 
> now within our scope?
> 
> The initial configuration like FE and CE endpoint and other 
> vital information are pre-configuration and is outside the scope
> how we do it either thro' CLI or SNMP or ????. 
> 
> 
> Regards
> Ramg
> 
> 
> > [mailto:[email-masked]]On Behalf Of ext Jamal 
> > Hadi Salim
> > Sent: Friday, March 05, 2004 4:39 PM
> > Cc: Khosravi, Hormuzd M; Deleganes, Ellen M; 
> [email-masked]
> > Alex,
> > Lets just follow the discussion in order of issues, to make faster
> > progress.
> > One issue at a time. At the end if it is shown that multicast
> > has all these problems you say it does, we should probably throw it
> > out totaly. All the issues you address are listed below.
> > I sent text on issue 0 yesterday. Hormuzd is sending something
> > on at least 2. Maybe i should send on 1. 
> > These three are the easisest to resolve.
> > 
> > Here are the issues again (with two new additions at the end):
> > If you ahve any that are missing here, please make suggestions.
> > 
> > 0) ID on the header.
> > 1) Encoding
> > 2) Multiple channels/connections
> > 3) Transport: unicast/multicast/broadcast 
> > { TCP, UDP, SCTP, DCCP, Direct over L2 or proprietary 
> > backplanes, TIPC} 
> > 4) Congestion control
> > 5) Reliability
> > 6) Security
> > 7) High availability
> > 8) Capability exchange
> > 9) The FEM/CEM interface
> > 
> > cheers,
> > jamal
> > 
> > On Fri, 2004-03-05 at 16:26, Alex Audu wrote:
> > > I still maintain that MULTICAST is best left at SHOULD implement
> > > strength.  Why?
> > > A lot of issues are still unresolved concerning multicast
> > > reliabililty, security, etc., unless
> > > we want to define an application-layer method to resolve 
> > issues like 
> > > synchronization,
> > > packet-loss, ACK/NACK implosion, etc.  
> > > 
> > > Alex.
> > > 
> > 
> > 
> >

---

From: [email-masked]
Date: Fri, 5 Mar 2004 18:33:08 -0500

Hello weiming,

Just one clarification to avoid confusion on my text. (see my text below).

> [mailto:[email-masked]]On Behalf Of ext
> [email-masked]
> Sent: Friday, March 05, 2004 5:51 PM
> To: [email-masked]; [email-masked];
> [email-masked]
> In FACT we proposed two rate limiter value one for C and 
> another one for D in adddition the rate from
> CE -> FE is different and from  FE -> CE. FE after processing 
>  the packet discrimiates the C and D and accordingly 
> act on it.   We have mentioned in our draft that rate 
> limiting can be done based on even various traffic (C and D) types.

Please note that there is a "rate limiter" LFB block and it is  part of FE model and FACT will use those to acheive this.
FACT doesn't implement the rate limiter as part of the protocol.

Regards
Ramg

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sat, 6 Mar 2004 14:39:15 -0800

I understand your concern and definitely agree that we will have to
resolve the issues of reliability/CC, security (may be)

for multicast before deciding to use it. I was assuming this will be
done. How we acheive this is a separate discussion...

using RMT, TIPC, others, etc...we can have that discussion after
resolving the other topics that Jamal has raised.

Regards

Hormuzd

-----Original Message-----
From: Alex Audu [mailto:[email-masked]] 
Sent: Friday, March 05, 2004 1:26 PM
To: Khosravi, Hormuzd M
Cc: Deleganes, Ellen M; [email-masked]
Subject: Re: [Forces-protocol] Comments from chat with ADs

I still maintain that MULTICAST is best left at SHOULD implement
strength.  Why?
A lot of issues are still unresolved concerning multicast reliabililty,
security, etc., unless
we want to define an application-layer method to resolve issues like
synchronization,
packet-loss, ACK/NACK implosion, etc.  

Alex.

Yes Ellen, that was my intent.
 
Thanks for the clarification,
Hormuzd
 
-----Original Message-----
From: Deleganes, Ellen M 
Sent: Friday, March 05, 2004 12:19 PM
To: Khosravi, Hormuzd M; [email-masked]
Cc: [email-masked]
Subject: RE: [Forces-protocol] Comments from chat with ADs

 
Regards,
Ellen
 
-----Original Message-----
 
Hi Alex,

 
Regards
Hormuzd
 
-----Original Message-----
From: Alex Audu [mailto:[email-masked]] 
Sent: Thursday, March 04, 2004 1:22 PM
To: Khosravi, Hormuzd M
Cc: [email-masked]
Subject: Re: [Forces-protocol] Comments from chat with ADs

Hormuzd,

Regards,
Alex.

Hi David

at

C

Regards
Hormuzd
 
-----Original Message-----

 
Multiple channels:  

Multicast & Unicast:  

 

 

Congestion control:

 

Cheers,
David

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sat, 6 Mar 2004 15:31:16 -0800

Hi Jamal

My thinking on this is a little different. I think this should be part
of the protocol phase call it binding phase during which after the
initial unicast connection (say TCP) is set up, other multicast
connections can be negotiated and setup. Also, an unreliable D
channel/connection will be setup during this phase. This is reduce our
dependence on FEM/CEM interface and also help with interop.

This approach is similar in basics to approach used by say FTP (for
setting up data connections) or more sophisticated approach used by
iSCSI to setup multiple TCP connections for performance reasons. 

Anyways, lets discuss this issue later otherwise it will be very
confusing.
Lets get the header resolved first,

Thanks
Hormuzd

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
Sent: Friday, March 05, 2004 1:32 PM
To: Khosravi, Hormuzd M
Cc: jeff pickering; [email-masked]; [email-masked]
Subject: RE: [Forces-protocol] Comments from chat with ADs

Is this negotiation for transport going to happen at the
FEM/CEM plane? In toher words i would think this is more
of a configuration/bootstrapping issue, no? And those typically belong
to the FEM/CEM plane.
My opinion is that typically a single NE will be controlled by a single
admin; therefore selection of transport can be enforced within that
domain - such a static setup will simplify things a great deal.

BTW, we should add capability exchange as something else that needs
discussion.

cheers,
jamal

On Fri, 2004-03-05 at 16:09, Khosravi, Hormuzd M wrote:
> Hi Jeff,
> 
> Just wanted to clarify one comment below. I think some sort of
> negotiation will be necessary even if we mandate multicast for certain
> cases in order to have an easy interop. In general, the more specific
we
> are about all the details in the protocol, the stronger language we
use,
> the easier it will be for us to interop. The question that you have
> asked below is a very valid question and will need to be answered in
the
> protocol spec explicitly. My gut feeling for the answer is that there
> will be a mix traffic, if the other FEs support multicast and the CE
> would like to use it or is using it already.
>