Subject: RE: [Forces-protocol] Comments from chat with ADs

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Fri, 5 Mar 2004 12:58:18 -0800

Yes Ellen, that was my intent.

Thanks for the clarification,
Hormuzd

-----Original Message-----
From: Deleganes, Ellen M 
Sent: Friday, March 05, 2004 12:19 PM
To: Khosravi, Hormuzd M; [email-masked]
Cc: [email-masked]
Subject: RE: [Forces-protocol] Comments from chat with ADs

I think what Hormuzd is saying (Hormudz can correct me if I am wrong) is
that even if Multicast is optional, those procedures that noted in
Hormudz's email should be described in the ForCES protocol spec. That
is, if a ForCES protocol implementation supports multicast, then it MUST
implement it in the manner described in the ForCES protocol
specification. Otherwise, ForCES implementations using multicast will
probably not interoperate.

Whether or not implementing multicast for transports that support
multicast is a requirement is a separate question. 

Regards,
Ellen

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Khosravi, Hormuzd M
Sent: Thursday, March 04, 2004 6:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Forces-protocol] Comments from chat with ADs

Hi Alex,

In general, I prefer using language such as MUST so that there is no
confusion or interop issues in the protocol implementations. However,
you raise a reasonable point and let me tell you my thoughts on this. We
will definitely need Unicast between FEs and CEs at the minimum in the
protocol. In addition, if the interconnect technology supports Multicast
and there is a case where it is useful to the application (such as
downloading IPv4 routes) we must allow it as well without breaking
interop. In order to do that, we will need specific procedures in the
protocol that will allow the CE and FE to negotiate on using multicast
dynamically at run-time (say during the binding or joining phase between
the CE and FE) and also associate different multicast groups with LFBs.
We will also need to define how Reliability/CC etc. will be provided for
multicast...may be using RMT, TIPC, etc. (?) If we do this correctly,
then applications will be able to use both unicast and multicast without
any interop issues.

Let me know what you guys think about this.

Regards
Hormuzd

-----Original Message-----
From: Alex Audu [mailto:[email-masked]] 
Sent: Thursday, March 04, 2004 1:22 PM
To: Khosravi, Hormuzd M
Cc: [email-masked]
Subject: Re: [Forces-protocol] Comments from chat with ADs

Hormuzd,

Why should multicast be "mandated for certain cases where it will be 
useful" if there
is a simpler, more robust way to do the same thing?  I'll say we don't 
mandate this.
We can make it a "SHOULD" strength, not a "MUST".  We mandate UNICAST
and make MULTICAST optional.  Any issues with this?

Regards,
Alex.

Khosravi, Hormuzd M wrote:

>Hi David
>
>Your summary below of the discussion with Ads on the different issues
>seems like very reasonable approaches to me.
>
>I agree that it will be a good idea to present transport alternatives
at
>IP and non-IP level for the ForCES protocol. We can mandate one or more
>transport for IP and one or more transport for non-IP, this way the
>interoperability requirements are met and at the same time, the in-box,
>out of box scenarios are met.
>
>Also, agree with having multiple channels/connections to differentiate
C
>& D traffic between CEs and FEs. On the unicast/multicast topic, dual
>mandatory approach again seems a good idea to support interoperability.
>The one thing that I would like to point out is that for any ForCES
>protocol implementation to work, unicast will definitely be required at
>the minimum. One could also mandate using multicast for certain cases
>where it will be useful (e.g. IPv4/config table download), if it is
>supported by the interconnect as you stated. But just multicast without
>unicast will not work, atleast not efficiently.
>
>Regards
>Hormuzd
>
>-----Original Message-----
>From: [email-masked]
>[mailto:[email-masked]] On Behalf Of Putzolu, David
>Sent: Tuesday, March 02, 2004 5:06 AM
>To: [email-masked]
>Subject: [Forces-protocol] Comments from chat with ADs
>
>Patrick and I spent about 90 minutes last night
>chatting with Bill and Alex about ForCES, with
>maybe half of the time on the protocol.  In
>the conversations with them, along with some
>follow-up chats I had with Allison Mankin, the
>following topics where covered:  multiple 
>channels, multicast/unicast, and congestion 
>control.  Below is what I recall from memory 
>(Patrick please correct me/elaborate as needed :)
>
>
>
>Multiple channels:  
>
>I discussed the interference issue that 
>was identified by the GRMP team.  Bill &
>Alex agreed that many OS implementations can have
>an interference property (i.e. where two different
>connections will have queuing interference lower in
>the networking stack in the OS).  So it is necessary
>to make sure when implementing a FE (or CE probably
>as well) that this is taken into account.  Methods
>of doing that include:
>* Manually queuing packets (i.e. feed OS few enough
>  packets that it cannot get into trouble - although
>  you pay a performance price for that, in that fine
>  grained scheduling takes cycles)
>* Indicating to the OS to treat flows differently - 
>  some real-time OS will let you do this, that is they
>  are implemented to internally retain QoS between
>  flows.
>However, it was also mentioned that intra-FE 
>interference is only one kind of interference.  Good
>FE implementation (e.g. by manually queuing above the
>OS) helps to fix intra-FE interference, but it does
>not solve inter-FE interference.  That is, if all
>(C & D) packets are always on the same connection,
>that makes it impossible on the wire to differentiate
>them from one another.  Imagine you have a bunch of
>FEs with some of them being subject to DOS attack for
>D packets, and you want to query one of the FEs for
>statistics.  It would be good to have some way of
>ensuring the C packets (or any high priority packets -
>OSPF HELLO maybe) get through.  Separate channels
>(as Jamal indicated, don't even have to be C & D)
>makes this possible.   Putting everything on one
>channel makes it impossible to deal with inter-FE
>interference in a differentiated fashion. 
>
>
>Multicast & Unicast:  
>
>The discussion on this started out with a discussion
>of the IETF protocol design BKM that options can ruin
>compatibility.  That is, if there are four ways to
>implement something, all optional, then you can have
>client 1 implement A & B, and client 2 implement C & D,
>and even though both clients are 100% compliant, they
>cannot talk to one another. Better to say all clients
>must implement A and make B, C, and D optional, so that
>way you know that all implementations will be able to 
>at least talk.
>
>Patrick & I pushed on this a bit, explaining that we
>actually had potentially (at least) two different
>environments - very close environments where multicast
>support could be present in the interconnect,  and 
>not-as-close environments (and some close
>environments as well) where multicast was not available.
>
>Bill & Alex indicated that if there really are two
>quite different cases, then it is possible to have a
>"dual mandatory" approach.  I.e.:
>
>If your underlying interconnect supports multicast,
>then you MUST implement the following (multicast)
>method of communication.
>
>If your underlying interconnect supports unicast, then
>you MUST implement the following (unicast) method
>of communication.
>
>This will guarantee interop, although at a potential
>extra cost (cost being having to potentially support
>both when both are present).  While this isn't optimal,
>it does at least give some flexibility.
>
>
>
>Congestion control:
>
>Several times over the whole conversation the topic
>of congestion control came up.  Also, as part of some
>TIPC discussions, I ended up spending some time talking
>to Allison Mankin (transport AD) about congestion control
>as well.  All the ADs I talked to where consistent on 
>the following:
>
>If you are running over IP, you MUST act in a RFC2914
>compliant fashion (i.e. respond to congestion like TCP
>or SCTP or RMT does).  Two reasons where put forth for
>this:
>1) Any time you define an IP-based protocol you cannot
>guarantee that it will not go out over the Internet. As
>such, the IESG will NOT standardize IP protocols with
>"bad" (non-2914) behaviors.
>2) Lots of implementation experience showing that other
>congestion control mechanisms don't work (congestion
>collapse etc).
>
>They where very firm on the above and my impression was
>that convincing them otherwise would be very difficult.
>
>That being said, a second alternative did arise:  If
>you are not implementing on top of IP (e.g. running on
>Infiniband, ATM, Ethernet) then you could rely on mechanisms
>for congestion control built into what you are relying on,
>or even consider building your own congestion control
>mechanisms (ala TIPC).  This makes it possible to define
>(for example) a multicast transport for forces without
>having to go all the way and implementing RMT - as long
>as it is clear that it isn't over IP.  The ADs went as
>far as saying that the IETF could even accept standards-
>track drafts that say nothing about IP (e.g. define an
>Ethernet-only transport for ForCES), if it is necessary 
>for ForCES. GSMP and (some other group I forgot) was
>given as an example.
>
>Cheers,
>David
>
>_______________________________________________
>Forces-protocol mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/forces-protocol
>
>
>_______________________________________________
>Forces-protocol mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/forces-protocol
>  
>

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: Alex Audu <[email-masked]>
Date: Fri, 05 Mar 2004 15:26:06 -0600

I still maintain that MULTICAST is best left at SHOULD implement 
strength.  Why?
A lot of issues are still unresolved concerning multicast reliabililty, 
security, etc., unless
we want to define an application-layer method to resolve issues like  
synchronization,
packet-loss, ACK/NACK implosion, etc. 

Alex.

---

From: Jamal Hadi Salim <[email-masked]>
Date: 05 Mar 2004 16:38:57 -0500

Alex,

Lets just follow the discussion in order of issues, to make faster
progress.
One issue at a time. At the end if it is shown that multicast
has all these problems you say it does, we should probably throw it
out totaly. All the issues you address are listed below.
I sent text on issue 0 yesterday. Hormuzd is sending something
on at least 2. Maybe i should send on 1. 
These three are the easisest to resolve.

Here are the issues again (with two new additions at the end):
If you ahve any that are missing here, please make suggestions.

0) ID on the header.
1) Encoding
2) Multiple channels/connections
3) Transport: unicast/multicast/broadcast 
{ TCP, UDP, SCTP, DCCP, Direct over L2 or proprietary backplanes, TIPC} 
4) Congestion control
5) Reliability
6) Security
7) High availability
8) Capability exchange
9) The FEM/CEM interface

cheers,
jamal

On Fri, 2004-03-05 at 16:26, Alex Audu wrote:
> A lot of issues are still unresolved concerning multicast
> reliabililty, security, etc., unless

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 6 Mar 2004 21:58:04 +0800

Hi Jamal and all, 

I really enjoy your proposal to set individual discussing items as Issue X and to 
set a specific email to discuss it. Moreover, I have some suggestions for such play:

1. The issue number is given according to the acsend order when it is appeared as 
a specific email. Issue 0-99 are reserved for discussion related to the protocol techniques,
and Issue 100-199 are reserved for other issues, just like I boldly put this email as Issue 100. 
In the end, we just need to pick up the results from the Issue 0-99 as delivarables to submit to 
WG. The possible results may be a positive agreement, a negative agreement (like "can not reach 
agreement", "not compromised", "not compromised yet but may in the future", "not important", 
"don't care", anyway, anything. In this way, a result without any objection can surely be made. 

2. The email subject is marked as Issue<number>: <issue statement>. In this way, others are easier
to follow the discussion. We may also use the number as the way Jaml does to list them in advance, I am just
a little worry it may be hard to find the issue number when it has not be posted as an indipendant email.

3. In order to speed the process, individual candidate protocol team is recommended 
to speak in the same voice so that others can easily understand and have less confusion 
on the team attitute. Discussions inside a team are supposed to be made privately. By playing in this way, 
we just assume that how can a compromise be made if the team inside can not make a compromise first.
Of course, this does not at all mean to exclude individuals (including individuals in some team) to provide comments
on the issues. It just means to try best. 

Yours,
Weiming

(P.S. I suppose the mailing list now runs in order, individual addresses needn't be additionally added
if a reply is to public and the additionals are already in the list, to make less duplicates).

h�Ǭ��-��%����x%��E����趇(�'�~����i����
bz���j)fj��b��?~��z�k���

---

From: Jamal Hadi Salim <[email-masked]>
Date: 06 Mar 2004 10:55:33 -0500

Weiming,

> "don't care", anyway, anything. In this way, a result without any objection can surely be made. 

I dont think we have be very rigorous in the numbering of the issues. I
agree with you on the formating for the March 20th report. The list i
presented is a suggestion - i dont think it is totaly complete. Your
suggestion on how the results get tagged is fine. I dont think in the
final report we have to say exactly how the "congestion  control" is met
rather what we should say is "reached agreement". The how goes on the
protocol draft.

> a little worry it may be hard to find the issue number when it has not be posted as an indipendant email.

I agree with this (and made the same suggestion in my other email). 
Lets have issues with on separate emails. Your subject suggestion is
good.

If i may add to this suggestion: We propbably should not have more than
3-4 subjects being disscussed concurently. I am pretty sure we can nail
the first 3 very quickly.

> on the issues. It just means to try best. 

The way i look at it is we are now going towards a new protocol. 
Lets share experiences/opinions etc so we can move faster.

Weiming, why dont you the following so we could move faster:
1) add any missing items to that list or format whichever way you
prefer.
2) Start responding to the issues posted. Issue 0 (ID semantics) and
issue 1 (packet encoding). Hormuzd is planning to post on the recent
subject discussion as issue 2.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Sun, 7 Mar 2004 02:26:41 +0800

Hi Jamal,

Pls allow me just reply to mailing list to reply to you as well. I wish you may only need to reply to me
by replying to the list, for it produces duplications :)

>[Jamal Re]
> [/Jamal Re]
>[Jamal Re] 
> [/Jamal Re]
[Weiming Re]
That's fine, but I just want post the "DoS protection and congestion control mechanism" very soon, for I have 
already finished the email.

I also came up with an idea that a sub-issue may be opened when some issue discussion 
has lead to too complex state or some problem in the issue is relatively independent. 
I'm afraid this may happen when some topics are quite wide spread. The issue may be 
marked as Issue X.Y .
[/Weiming Re]

> [Jamal Re] 
>[/Jamal Re]

[Weiming Re]
I absolutely do not mean to put any limit to opinion sharing. I'm sorry to have misexpressed it.
What my original thought is just I thought a tree structure may be more efficient than a linear structure.

Yes, I'm glad to add some more items to follow the list as:

10) In-protocol mechanism for reliability
11) Support for vendor functions
12) Support for network management tools
13) Message organizing
14) Conflict check with current FE model

For the Issue 10, I'm just not sure if the reliability issue (Issue 5) you listed include this. 
It's also fine to have this as a sub-issue, but anyway should be specially addressed because it is a big issue. 

I'd like to add some more, but I'm afraid I have to go to bed now, it's already 2'clock in the morning. ~o~ 
[/Weiming Re]

Cheers,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: 06 Mar 2004 21:50:40 -0500

Weiming,

> by replying to the list, for it produces duplications :)

I have been bitten on many lists where i send a message and it never
makes it. CCing the original poster always helps for redundancy
purposes. So bear with my emails because in the worst case you will
receive two;->

> already finished the email.

I saw that email. Can we ignore it for now if you dont mind until we
address 0-2? I am not gonna respond to your other emails so we can have
some order to any chaos.

> I'm afraid this may happen when some topics are quite wide spread. The issue may be 

I agree. If it is something that can be obviously identified then by all
means go ahead and propose the sub-topics; otherwise the subtopics will
come up as we hit each topic.

> 10) In-protocol mechanism for reliability

Should be part of issue 5.

> 11) Support for vendor functions

Not sure if i see this as anything speacial right now;
but lets add it to the list.

> 12) Support for network management tools

Not sure i follow this. This can always be resolved later.

> 13) Message organizing

Is this different from issue 1?

> 14) Conflict check with current FE model

This is not a contetious issue - it can be resolved
later. I actually havent read the latest incarnation
of the model draft. what did you have in mind?

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Sun, 7 Mar 2004 11:46:11 +0800

Hi Jamal,

From: "Jamal Hadi Salim" <[email-masked]>
> address 0-2? I am not gonna respond to your other emails so we can have

[Weiming Re]
The multichannel issue is quite related to the congestion control issue, therefore, if we want to be more focus, I suggest
just to bigin with 0,1 only, leaving2,3,4 as the following group. Or else, I think it's reasonable to start with 0-4. Actually we
already had many discussions on these. It seems not very well to stop these discussions all of a sudden. Moreover, issue 1 is 
quite a big issue, I don't think it can be closed within one week. Then, we can count how much time is left. Therefore, I suggest
to work with more parallellism. 

> Not sure i follow this. This can always be resolved later.

[Wang Re]
I'm not sure the later means. I suppose we are just presenting some topics for later discussion, right?
> Is this different from issue 1?

[Weiming Re]
Issue 1 is about packet encoding format, this issue is about message organizing way, such as what kind of ForCES messages
we need and how they are organized. Different?
 
> of the model draft. what did you have in mind?

[Weiming Re]
Maybe I should not use 'conflict', instead should be a cohererent check. For instance, new FE model suppose to use 
ASCII bsed LFB names like "IPv4 LPF". I'm sure an LFB ID will appear in ForCES protocol, then how they are consistent with
each other. Others like something not sured who will define them, and at what level, etc.

Cheers,
Weiming

����趇(�f��)��+-��z�k���X���+�m����5��_��?�������+-�w���+q�)��hr�

---

From: Jamal Hadi Salim <[email-masked]>
Date: 06 Mar 2004 23:16:43 -0500

> to work with more parallellism. 

The idea is not to agree on the "how"; rather on what of the merger i.e 
that we can narrow any differences into a small tight area. Thats what
needs to be reported back by the 20th. So it should not take many emails
to address issue 1 in that scope. As far as i can see from the
discussion this should be soon closed. 
On issue 2 Hormuzd was putting some text around it. Why not wait until
he does so? I know its hot for you - but it would help to have a
starting point.
In the meantime i think it would be useful if you address issue 0 and 1.
We are going to talk about every issue that is bothering you, not to
worry; lets just do it sequentially.

> I'm not sure the later means. I suppose we are just presenting some topics for later discussion, right?

We are trying to reach a general idea that the issues are resolvable.
"Later" is when the protocol is being written.
I dont see network management tools as something either imposing
on the protocol or contentious to the merging.

> we need and how they are organized. Different?

Repsond to issue 1 with examples and maybe it could be a separate topic.
I dont see it as different but cant tell without seeing your text.

> each other. Others like something not sured who will define them, and at what level, etc.

Ok you can add this. I would recommend the model draft doesnt get
approved to final call incase we have issues with it. 

cheers,
jamal