Subject: RE: [Forces-protocol] Issue 3:Transport/Unicast/Multicast/Broadcast

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 17 Mar 2004 23:29:34 -0800

Hi Jamal

As I have said before, I am not opposed to multicast. However, if we use
it for control messages, it should be reliable, congestion aware, etc.
to meet the Requirements. The scheme you have described below, will work
under normal conditions...however I do see some interesting race
conditions in addition to the performance issue that I pointed out. 

In any case, I thought you had agreed in some of your previous emails
and even on the forces list that reliability, CC is a must for
control...are we still in agreement on this ?
If yes, then whether we provide reliability by building such mechanisms
into the ForCES protocol or whether we use the underlying transport is a
separate issue. The discussion around UDP seems to be getting into the
details of that issue, which we could defer for now. Do you agree ?

Just one clarification related to your scheme, do you send the protocol
Responses or as you term ACKs from FE to CE back on the unicast
connection or multicast it. I thought I heard Jeff saying that it is
sent back on the unicast connection, but I just wanted to confirm.

Thanks
Hormuzd

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
Sent: Wednesday, March 17, 2004 6:50 AM
To: Khosravi, Hormuzd M
Cc: [email-masked]
Subject: RE: [Forces-protocol] Issue
3:Transport/Unicast/Multicast/Broadcast

On Wed, 2004-03-17 at 02:26, Khosravi, Hormuzd M wrote:
> -----Original Message-----
> From: Jamal Hadi Salim [mailto:[email-masked]] 

[..]

> 
> [HK] Alright, so what youre saying is that you first send messages
over
> unreliable UDP multicast and if you don't get a response then you send

The UDP multicast is still capable of being reliable. For it to discover
that a message was not ACKed it needs that (reliability) feature.
The real advantage is being able to update multiple FEs.
The problem with retransmitting on the multicast channel is that
there maybe other FEs that have responded. Retransmit gets to them
and they have to process it (even if they dont respond to it). Besides
reprocessing the mesaage, there is ambiguity: the FE is not sure why its
receiving the message a second time i.e is it because the ACK it sent
got lost? (You could remove this ambiguity by adding a list of all FEs
that responded to the message, but that adds b/width cost).
So the safest thing is to retransmit on the unicast.

> it over TCP. Have you done this for control/configuration messages or
> only for heartbeats ? 

All clases of messages that need to be delivered reliably; heartbeats
are a speacial case because you may not care about some heartbeats being
delivered reliably.

> I agree that the Requirements around heartbeats
> are not as strict as those for control messages. The problem which I
can
> see with using this approach for control is that you cannot have
> deterministic performance. For example, you cant tell your customer
that
> we can deliver x routes/sec with our protocol implementation, which is
> what a lot of our customers have asked for (another thing asked is y
> data packets/sec). 

You can only give those numbers when you are sure about congestion
levels (of bandwidth and CPU) really. Typically this is where the MLFR
(Max Loss Free Rate) is measured. Our experience shows you have much
much higher MLFR for updates/sec with this scheme.
The multicast to FEs is infact driven by desire to have a high MLFR.
As the numbers of FEs grow, so does the MLFR.

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Wang,Weiming" <[email-masked]>
Date: Thu, 18 Mar 2004 16:41:17 +0800

Hi Hormuzd,

Though you asked Jamal for this,  I would also like to make a reply.
I think a built-in mechanism adpoted  to garuantee reliability should not be
excluded even when the transport network is IP based. I don't think it is
unacceptable to mulitcast using UDP first then for receivers to respond using
unicast UDP and even for sender to retransmit using unicast UDP with the unicast
UDP supported by built-in mechanism for reliability. In this way, the multicast
is actually reliable.

I'm afraid the word 'semi-reliable' is not proper to describe multicast.

Regards,
Weiming

----- Original Message -----
From: "Khosravi, Hormuzd M" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, March 18, 2004 3:29 PM
Subject: RE: [Forces-protocol] Issue 3:Transport/Unicast/Multicast/Broadcast

Hi Jamal

Thanks
Hormuzd

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]]
Sent: Wednesday, March 17, 2004 6:50 AM
To: Khosravi, Hormuzd M
Cc: [email-masked]
Subject: RE: [Forces-protocol] Issue

[..]

over

can
that

---

From: Jamal Hadi Salim <[email-masked]>
Date: 18 Mar 2004 06:51:24 -0500

Hi Hormuzd,

On Thu, 2004-03-18 at 02:29, Khosravi, Hormuzd M wrote:
> under normal conditions...

Messages that need to be delivered reliably MUST be reliably
delivered[1].
I did say infact that the multicast channel is capable of doing
reliable delivery. For example the discovery that a retransmit is needed
is through the multicast channel. The retransmit could be done through
the multicast channel but that would complicate the messaging if you
want to remove ambiguity. One way i described how to remove ambiguity is
to have within the message a list of IDs for all nodes that have
responded. This way a node seeing itself listed doesnt respond.

> however I do see some interesting race

The performance issues are against our experience and intuition for that
matter. There are no race conditions if you assume all connections
belong to the device and the differentiation of the messages based on
priorities.

But this brings a good point: this is why it shouldnt matter which
connection the message is sent or received on. Then people wanting
to implement the way you or i describe should have no issues (while
fully interoperating). Then lets have a few benchmarks.

> control...are we still in agreement on this ?

Absolutely. 
Although i think you may be looking too closely at the connection detail
as opposed to the mile high view. 
The blackbox view is that the message gets delivered reliably, and that 
CC is obeyed on a per link basis. Even though the multicast connection
is reliable as i said only partial capabilities of reliability are used
(example retransmit doesnt happen on it).
In the above scenario it actually doesnt matter whether the send is on
a multicast and ACK on one of the two other connections. 

I think where we differ is you are advocating a specific connection to
be used for specific messages, correct? And i am saying there is no such
restriction.

> details of that issue, which we could defer for now. Do you agree ?

Lets defer it and close this issue.

> sent back on the unicast connection, but I just wanted to confirm.

For reliability purposes, it doesnt matter which connection the ACK
comes back on, IMO.
The main reason you want the ACK to come back on the multicast is so
that an FE tracking another gets a free ride of knowing of its
state (such as the fact it is still alive, what table entries it is
accepting, etc). But this is a detail that belongs to #7.

I think we should move on to topic #4 and #5 since they are related to
this discussion.

cheers,
jamal

[1] I liked some text that Weiming used to describe this in an email he
had earlier but cant seem to locate it now,

---

From: jeff pickering <[email-masked]>
Date: Thu, 18 Mar 2004 09:12:21 -0500

I think the "race condition" issue brought up when sending on two 
channels (mcast/ucast)
only arises if the retransmit (unicast) is received before the original 
(multicast). This could be
avoided by not retransmitting on unicast unless at least one FE had 
acked the multicast.

Jeff