Subject: RE: [Forces-protocol] Issue 3:Transport/Unicast/Multicast/Broadcast

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 16 Mar 2004 23:26:30 -0800

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 

> I personally dont think that a single multicast connection would
> be an efficient thing for a transport. On the same token i dont
> think a single unicast connection would be efficient.
> If you have a reliable unicast connection, i see the value
> of having a semi-reliable multicast connection - at least based on our
> experiences. Maybe people with other experiences can share.
> 
> [HK] Based on our experiences and customer requirements, we have been
> able to satisfy them with 2 unicast connections. I am not sure what
you
> mean by semi-reliable multicast...can you explain ? I think
reliability,

I was refering to our experiences in using both together.
(BTW, this is also one area we differ slightly).
When a message is to be delivered reliably, CE would send frist on
multicast; when things were fine we received all FE responses on
multicast connection. When things were not fine we didnt receive some
responses from some FEs. In this case we resent on TCP only to those FEs
that failed to respond. There are various reasons why you would resend
on TCP but i dont wannna go into them at the moment. 
Netlink2 can ask for reliability on a per message level. This helps
the underlying transport abstraction decide how to deliver that message.
An Add route for example MUST be reliably delivered whereas some
heartbeats dont have to.
I agree with realibility, congestion control etc being met - but
we could simplify a lot. 

[HK] Alright, so what youre saying is that you first send messages over
unreliable UDP multicast and if you don't get a response then you send
it over TCP. Have you done this for control/configuration messages or
only for heartbeats ? I agree that the Requirements around heartbeats
are not as strict as those for control messages. The problem which I can
see with using this approach for control is that you cannot have
deterministic performance. For example, you cant tell your customer that
we can deliver x routes/sec with our protocol implementation, which is
what a lot of our customers have asked for (another thing asked is y
data packets/sec). This is because in your implementation it will
completely depend on the interconnect characteristics as well as
load...in the best case your performance will be very good and in the
worst case it will be pretty bad, cause all your control messages have
to be sent again on the TCP connection after some timeout. Anyways,
these are my thoughts...wonder what others have to say about this ?? 

> > congestion aware Multicast connections. The establishment and
> semantics
> > of these Multicast connections is negotiated during the
> > binding/capability discovery phase in the ForCES protocol
> > (post-association). 
> 
> This part we can resolve later.
> [HK] That's fine, but this is important in order to use Multicast in
an
> efficient way.

This could be done in the CEM/FEM plane or negotiated via the CE/FE;
we can decide on the merits later. I am not religious about which one
should be used. My experiences are with CEM/FEM.

[HK] I think have this as part of the protocol binding phase rather than
CEM/FEM will be much better for interop.

> 
> So three classes of connections:
> 1) totaly reliable unicast
> 2) probably a range of unreliable, semi-reliable to reliable multicast
> 3) unreliable UDP-like unicast for data redirection.
> 
> You also seem to be stating for 2) above you could have more than
> one connection. Is this accurate?
> [HK] Perfect !

That does seem reasonable to me.
To come back to my note from above: We still differ on some specifics. 
Example we saw value on send-on-multicast-resend-on-unicast. For this 
to be met, the CE or FE should not care which connection it received the
message on. As an example it may receive the hearbeat on 1,2 or 3.
Does this sound reasonable?

[HK] As I said heartbeats are special so I am not sure what the best
solution for this might be...we might have to send HBs on all 3
connections...that is one possibility...or any of these 3 connections as
you are suggesting. 
> 
> Additionaly i think that there would be demarcations between close
> and non-close proximity where other protocols such as TIPC or L2
direct
> maybe constrained. There may be tunnels of sorts connecting
> such demarcations to create a large NE.
> [HK] Not sure if I understand this properly...can you explain a bit
more

If you say use TIPC or something TIPC-like you probably want it to run
only within the close proximity case. i.e you most likey dont want such
packets to hit a shared network. Same with L2.
You could make TIPC think things are in the same chasis by connecting
several such chasis across multiple hops interconnected maybe by TCP.
This would be like DVMRP "islands" for example in the old mbone days
interconnected using IPIP tunnels. OSPF does similar things.
If you use L2, you could use something like L2VPN. etc.

[HK] Lets defer this one, I am not sure if it will be needed. This seems
similar to what Alex is suggesting regarding Transport Abstraction...am
I off track ?

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: Jamal Hadi Salim <[email-masked]>
Date: 17 Mar 2004 09:49:46 -0500

> From: Jamal Hadi Salim [mailto:[email-masked]] 

[..]

> unreliable UDP multicast and if you don't get a response then you send

The UDP multicast is still capable of being reliable. For it to discover
that a message was not ACKed it needs that (reliability) feature.
The real advantage is being able to update multiple FEs.
The problem with retransmitting on the multicast channel is that
there maybe other FEs that have responded. Retransmit gets to them
and they have to process it (even if they dont respond to it). Besides
reprocessing the mesaage, there is ambiguity: the FE is not sure why its
receiving the message a second time i.e is it because the ACK it sent
got lost? (You could remove this ambiguity by adding a list of all FEs
that responded to the message, but that adds b/width cost).
So the safest thing is to retransmit on the unicast.

> only for heartbeats ? 

All clases of messages that need to be delivered reliably; heartbeats
are a speacial case because you may not care about some heartbeats being
delivered reliably.

> I agree that the Requirements around heartbeats
> data packets/sec). 

You can only give those numbers when you are sure about congestion
levels (of bandwidth and CPU) really. Typically this is where the MLFR
(Max Loss Free Rate) is measured. Our experience shows you have much
much higher MLFR for updates/sec with this scheme.
The multicast to FEs is infact driven by desire to have a high MLFR.
As the numbers of FEs grow, so does the MLFR.

> This is because in your implementation it will

If you have congested CPUs this applies to TCP or pigeons as transport.
i.e retransmits will happen regardless when messages are lost. With
multicast first, the cost of retransmits is reduced.

chopping email here for readability.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: 17 Mar 2004 10:01:28 -0500

> CEM/FEM will be much better for interop.

Then we need to allow rebinding during the protocol run.
Example i should be able to add another multicast connection without
having to restart the application.

> connections...that is one possibility...or any of these 3 connections as

Refer to my previous email.
This is not specific to heartbeats. I suspect we will never send unicast
on the data redirector connection but there should be nothing 
blocking that. 
This maybe related or resolvable via the transport abstraction
path.

> similar to what Alex is suggesting regarding Transport Abstraction...am

Yes. We can defer this. I noticed Ram may have other thoughts on the
subject as well - i am not gonna respond to his email so we can move on.

cheers,
jamal

---

From: Alex Audu <[email-masked]>
Date: Wed, 17 Mar 2004 14:20:59 -0600

Hello Hormuzd,

I think you've just raised a very crucial flaw about this multicast 
scheme. A scheme that
works some time and fails the rest of time  is really useless (in my 
view) in a critical system.
Dependability and consistency are attributes of a well designed protocol.

Regards,
Alex.