Subject: RE: [Forces-protocol] Issue 3:Transport/Unicast/Multicast/Broadcast

From: "Deleganes, Ellen M" <[email-masked]>
Date: Tue, 16 Mar 2004 18:26:05 -0800

Ram,

Ram,

Regarding your comment below enclosed in <<< >>>

<<<We are defering one issue after another. We should try to close one
issue after another. 

For unicast its much straightforward choice. As TCP is widely deployed
we should opt for TCP.
If we strong compelling reasons for not having TCP that should be
explicitly discussed.

IMHO we should try to close each issue rather than deferring.>>>

The team is currently in the phase of determining whether or not it will
be possible to "merge" the protocols, not to close all issues.
Discussion needs to occur on all of the major issues to determine
whether or not consensus can be reached by March 20. This won't be
possible if the team attempts to close on all of the issues. Once
general consensus is reached, or it is decided that the team will never
agree, it is time to move on.

I agree that during the design phase, the team must close on every issue
that is brought up, but now is not that time.

Regards,
Ellen

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 16 Mar 2004 18:47:25 -0800

Ram,

I share your concern and am personally not too happy with deferring
important issues. However seems like this way of making progress is
kinda working so far. I would summarize this approach as "postpone and
progress" :-))

Anyways, on a more serious note, Ellen has pointed out some good reasons
for this approach, for now.

regards
Hormuzd

-----Original Message-----
From: Deleganes, Ellen M 

Regards,
Ellen

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 16 Mar 2004 23:37:13 -0800

Hi Ram,

You have made some interesting comments below on the categories. I think
what you would like to see in general is categories based around IP
rather than inside-box, outside-box scope which I have outlined.
However, seemed to me like most of the folks like the box or close
proximity scope rather than IP scope. I am fine with either of the two
approaches...what do others think about this ?

Regards 
Hormuzd

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
>  
> > Here are the basic ForCES Impl Categories...
> > 
> > Cat 1: Local (inside box) with reliable,CC multicast 
> support (Example,
> > ATM fabric backplane)
> > No need of IP in this case, ForCES can directly run over L2 or use
> > something like TIPC.
> > Unicast+ multicast connections can be used in this case for
> > communication.
> >

The statement is not clear to me... 
Are we recommending that for single box its non-ip. There are systems
that uses IPv6 and also IPv4 within single box...
they have unicast mechanism. 

May be we should have something like this.... for non-IP interconnects
(with in single box) we can use 
the proposed mechanism....

> > Cat 2: Local with No reliable,CC multicast support (Example,
> > Gig-Ethernet backplane)
> > No need of IP in this case, ForCES can directly run over L2 or use
> > something like TIPC.
> > Unicast connections can be used in this case for communication.

The above two statement make a case for non-IP interconnects. We should
also consider the case for IP interconnect with single box.

> > 
> > Cat 3: Non-local (Example, Ethernet wire)
> > ForCES runs over IP/transport protocols in this case.
> > Unicast connections such as TCP, DCCP can be used in this case.
> 
Here we should also consider non-IP physical box. for example .....
distributed GGSN (or SSGN) connected via l2 swichtes in wireless
interconnect
....

---

From: [email-masked]
Date: Thu, 18 Mar 2004 00:13:26 +0800

Hi Hormuzd, Jamal, and all,

Let me try to view this issue from a little bit different viewpoint. 

I would prefer to consider the  issue at the two different layers:

1. At the Forces protocol layer (as Alex said the Forces core layer), support 
for Unicast/Multicast/Broadcast should be all described as 'MUST'. 

2. At the transport  (adapation) layer, only a basic connection scheme for 
unicast is explicitly defined  as 'MUST' for every different interconnection 
means, leaving other possible implementation schemes for 
Unicast/Multicast/Broadcast as well as for reliability (as the Categories as 
you posted) all as  'MAY'.  Based on the 'MUST' supported unicast connection,  
a negotiation can be made between the CEs and FEs for them to choose a both 
acceptable scheme for transport layer support of U/M/B as well as reliability ( 
this idea was presented by Avri when we had a disscusion on Master-Slave 
model). This process can be done by FEM/CEM as Netlink2 proposed, or also can 
be done during the post-association phase. 

Hormuzd, 
I'm afraid what you just listed below for the approaches of three protocols are 
not also imcomplete for GRMP as well as for Netlink2. In fact, like Netlink2, 
GRMP can also support much more connection schemes for CE and FE. We can also 
allow to have many TCPs, Uni/Multicast UDPs, and many others,  if only vendors 
would like to do like this. Why? because in GRMP we in fact have not explicitly 
limited how a connection scheme should be adpoted by vendors, leaving it  to be 
flexibly implemented. This actually has shown one different design idea between 
us (I suppose also between Netlink2). Our idea is trying to make transport 
adapation more flexible to implementations, because as ForCES being a complex 
distributed system, we think it may be unwise and difficult to try to list out 
( also become a limit when being listed out)  all possible schemes for IPC for 
such a system.  Speaking of this, I'm afraid your idea on GRMP against two 
channels may also be imcomplete. We are definitely not saying we want to only 
have one TCP channel so as to object two (UDP/TCP) channels. 

Cheers,
Weiming

----- Original Message ----- 
From: "Khosravi, Hormuzd M" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, March 16, 2004 10:15 AM
Subject: RE: [Forces-protocol] Issue 3:Transport/Unicast/Multicast/Broadcast

Jamal, Thanks for the clarification.

Here is some text around this issue which I think is a reasonable
compromise,

All ForCES protocol implementations, MUST support a reliable, congestion
aware Unicast method of communication between the CEs and the FEs.
Furthermore, in case of IP this will be a TCP connection between the CE
and FEs. In addition to this, in case of 'close proximity' between the
CEs and FEs And in case the interconnect between them supports
multicast, the CEs and FEs Should/Must also establish reliable,
congestion aware Multicast connections. The establishment and semantics
of these Multicast connections is negotiated during the
binding/capability discovery phase in the ForCES protocol
(post-association). For example, the CE and subset of FEs might
negotiate on establishing a Multicast connection for communication of
IPv4 LFB attributes. These reliable, congestion aware connections will
be used to communicate the control messages outlined in RFC 3654
(section 6, #6c). (For data messages such as those outlined in RFC 3654
(section 6, #6b), the ForCES protocol Must/Should also support an
un-reliable but congestion aware Unicast connection between the CE and
FEs.- we can defer on this one till we come back to issue#2)

In this text I have tried to accommodate the different approaches of
netlink2, GRMP and FACT, at the same time I have made the text strong by
using Must/Should to have easy interop. Let me explain this further...

Netlink2-uses 1 TCP + 1 Multicast UDP connection
FACT-uses 1 TCP+ 1 unicast UDP+CC/DCCP like connection
GRMP-uses 1 TCP+ 1 UDP(in some experiments)...also implemented with only
TCP.

Pls correct me if I am wrong here. The above text tries to accommodate
all these approaches as well as the different scenarios or categories in
which they are applicable.

Cat 1: Local (inside box) with reliable,CC multicast support (Example,
ATM fabric backplane)
No need of IP in this case, ForCES can directly run over L2 or use
something like TIPC.
Unicast+ multicast connections can be used in this case for
communication.

Let me know what you guys think ?

Regards
Hormuzd

-------------------------------------------------
This mail sent through HUCNIC Webmail system.

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 17 Mar 2004 10:55:42 -0800

On Wed, 2004-03-17 at 02:26, Khosravi, Hormuzd M wrote:
> From: Jamal Hadi Salim [mailto:[email-masked]] 
>
> 
> This could be done in the CEM/FEM plane or negotiated via the CE/FE;
> we can decide on the merits later. I am not religious about which one
> should be used. My experiences are with CEM/FEM.
> 
> [HK] I think have this as part of the protocol binding phase rather
than
> CEM/FEM will be much better for interop.

Then we need to allow rebinding during the protocol run.
Example i should be able to add another multicast connection without
having to restart the application.

[HK] Agreed, infact we will definitely need the capability of adding 
Multicast connections on the fly.

> To come back to my note from above: We still differ on some specifics.

> Example we saw value on send-on-multicast-resend-on-unicast. For this 
> to be met, the CE or FE should not care which connection it received
the
> message on. As an example it may receive the hearbeat on 1,2 or 3.
> Does this sound reasonable?
> 
> [HK] As I said heartbeats are special so I am not sure what the best
> solution for this might be...we might have to send HBs on all 3
> connections...that is one possibility...or any of these 3 connections
as
> you are suggesting. 

Refer to my previous email.
This is not specific to heartbeats. I suspect we will never send unicast
on the data redirector connection but there should be nothing 
blocking that. 
This maybe related or resolvable via the transport abstraction
path.

[HK] Agree.

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Thu, 18 Mar 2004 11:20:23 -0800

Hi Jamal,

Glad to hear that we are in sync on some of the basics below.
One question, you say that Reliability is a MUST, do you also agree that
the Multicast connections should be Congestion Aware ? I think you do,
but just want to confirm this.

Also, I think its important to specify which messages are delivered on
which connections in order to have interoperable solutions. I fear that
if we have too many SHOULD/MAYs in the protocol in order to satisfy
everyone on this team, we will never have a truly interoperable IETF
protocol.

I will send more comments on some of the points you have mentioned below
in context with issue#4,5. Is that fine ?

Thanks
Hormuzd

-----Original Message-----
> 
> As I have said before, I am not opposed to multicast. However, if we
use
> it for control messages, it should be reliable, congestion aware, etc.
> to meet the Requirements. The scheme you have described below, will
work
> under normal conditions...

Messages that need to be delivered reliably MUST be reliably
delivered[1].
I did say infact that the multicast channel is capable of doing
reliable delivery. For example the discovery that a retransmit is needed
is through the multicast channel. The retransmit could be done through
the multicast channel but that would complicate the messaging if you
want to remove ambiguity. One way i described how to remove ambiguity is
to have within the message a list of IDs for all nodes that have
responded. This way a node seeing itself listed doesnt respond.

> however I do see some interesting race
> conditions in addition to the performance issue that I pointed out. 

The performance issues are against our experience and intuition for that
matter. There are no race conditions if you assume all connections
belong to the device and the differentiation of the messages based on
priorities.

But this brings a good point: this is why it shouldnt matter which
connection the message is sent or received on. Then people wanting
to implement the way you or i describe should have no issues (while
fully interoperating). Then lets have a few benchmarks.

> In any case, I thought you had agreed in some of your previous emails
> and even on the forces list that reliability, CC is a must for
> control...are we still in agreement on this ?

Absolutely. 
Although i think you may be looking too closely at the connection detail
as opposed to the mile high view. 
The blackbox view is that the message gets delivered reliably, and that 
CC is obeyed on a per link basis. Even though the multicast connection
is reliable as i said only partial capabilities of reliability are used
(example retransmit doesnt happen on it).
In the above scenario it actually doesnt matter whether the send is on
a multicast and ACK on one of the two other connections. 

I think where we differ is you are advocating a specific connection to
be used for specific messages, correct? And i am saying there is no such
restriction.

> If yes, then whether we provide reliability by building such
mechanisms
> into the ForCES protocol or whether we use the underlying transport is
a
> separate issue. The discussion around UDP seems to be getting into the
> details of that issue, which we could defer for now. Do you agree ?

Lets defer it and close this issue.

> Just one clarification related to your scheme, do you send the
protocol
> Responses or as you term ACKs from FE to CE back on the unicast
> connection or multicast it. I thought I heard Jeff saying that it is
> sent back on the unicast connection, but I just wanted to confirm.

For reliability purposes, it doesnt matter which connection the ACK
comes back on, IMO.
The main reason you want the ACK to come back on the multicast is so
that an FE tracking another gets a free ride of knowing of its
state (such as the fact it is still alive, what table entries it is
accepting, etc). But this is a detail that belongs to #7.

I think we should move on to topic #4 and #5 since they are related to
this discussion.