Subject: [Dime] Failover problems with Relays and crashed applications

From: Michael Hillier <[email-masked]>
Date: Wed, 04 Jul 2007 09:42:07 +0200

Hi all,

another question has come up in practice.  Again we are in the context 
of Credit Control Application, but the issue must be applicable to all 
such applications which define their own failover mechanisms

Given a client (C) communicating via a primary and secondary relay (Rp, 
Rs) towards a primary and secondary server (Sp, Ss).

All Diameter connections are up and running.  The client selects Rp and 
Rp selects Sp, being the primary next hops.

Now the primary Server stops answering the application requests (i.e  no 
CCAs are  returned) - for what ever reason,
however the connection Rp <=> Sp is still ticking over, i.e. DWRs are 
being answered by Sp.

Short timers, like Tx,  may expire in the client, possibly allowing 
service to be delivered, but that is a side issue.

Now the transaction timer expires causing the client to failover to the 
secondary relay (Rs), i.e. re-sending the CCR with T-bit
Rs selects the primary Server, because it does not know any reason not 
to, and thus the CCR(with T-bit) is sent to the same server which has 
failed.

Summary:  client failover to secondary next hop works nicely when 
directly connected to the Diameter servers, but does not achieve the 
desired effect when relays are used, which have no "application timer" 
awareness.

Can anybody point out where I have gone wrong?

Many thanks

Mike

Michael Hillier
Telenet AG Rhein-Main
[email-masked]

_______________________________________________
DiME mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dime

---

From: Rajith R <[email-masked]>
Date: Wed, 04 Jul 2007 14:28:37 +0530

IMO, there seems to be an implementation issue in the primary server not
responding to "application" requests (like CCR), but the connection still
ticking with DWR-DWA exchanges. Either DWR should not be replied or the
connection layer should understand that the application layer crashed (or to
that effect) and indicate this to its peers in a CER update.
AFAIK, DMT does not recommend/require a transaction time out based fail
over.
 
Regards
 
Rajith

  _____  

From: Michael Hillier [mailto:[email-masked]] 
Sent: Wednesday, July 04, 2007 1:12 PM
To: [email-masked]
Subject: [Dime] Failover problems with Relays and crashed applications

Hi all,

another question has come up in practice.  Again we are in the context of
Credit Control Application, but the issue must be applicable to all such
applications which define their own failover mechanisms

Given a client (C) communicating via a primary and secondary relay (Rp, Rs)
towards a primary and secondary server (Sp, Ss).

All Diameter connections are up and running.  The client selects Rp and Rp
selects Sp, being the primary next hops.

Now the primary Server stops answering the application requests (i.e  no
CCAs are  returned) - for what ever reason, 
however the connection Rp <=> Sp is still ticking over, i.e. DWRs are being
answered by Sp.

Short timers, like Tx,  may expire in the client, possibly allowing service
to be delivered, but that is a side issue.

Now the transaction timer expires causing the client to failover to the
secondary relay (Rs), i.e. re-sending the CCR with T-bit
Rs selects the primary Server, because it does not know any reason not to,
and thus the CCR(with T-bit) is sent to the same server which has failed.

Summary:  client failover to secondary next hop works nicely when directly
connected to the Diameter servers, but does not achieve the desired effect
when relays are used, which have no "application timer" awareness.

Many thanks

Mike

Michael Hillier
Telenet AG Rhein-Main
[email-masked]

---

From: "naveen sarma" <[email-masked]>
Date: Wed, 4 Jul 2007 17:22:51 +0530

Hi Mike,

Failover will be done only in case of the peer node is in suspect state and
not even responding for the DWR message.  In your case it might happen that
Rp is still in open state (Rp is responding for DWR from C), so the messages
are re-sent to the same Rp.  As per the 3588, base protocol need not start
any timer for each message sent to peer.  So inorder to test failover with
the existing relay setup, kill or suspend (in gdb) Rp.

Regards,
Naveen

On 7/4/07, Michael Hillier <[email-masked]> wrote:
>
> Hi all,
>
> another question has come up in practice.  Again we are in the context of Credit
> Control Application, but the issue must be applicable to all such

-- 
Yours
Naveen.

---

From: "Asveren, Tolga" <[email-masked]>
Date: Thu, 5 Jul 2007 21:14:36 -0400

I agree that this would be faulty behavior. OTOH, it is always a good idea to be protected against misbehaving nodes. By applying a local check, where neighbors, from which no Diameter application level reply has been received for a while, are no more used to send messages to could be a good idea (or disconnecting connection to such nodes).

   Thanks,
   Tolga

________________________________________
From: Rajith R [mailto:[email-masked]] 
Sent: Wednesday, July 04, 2007 4:59 AM
To: 'Michael Hillier'; [email-masked]
Subject: RE: [Dime] Failover problems with Relays and crashed applications

IMO, there seems to be an implementation issue in the primary server not responding to "application" requests (like CCR), but the connection still ticking with DWR-DWA exchanges. Either DWR should not be replied or the connection layer should understand that the application layer crashed (or to that effect) and indicate this to its peers in a CER update.
AFAIK, DMT does not recommend/require a transaction time out based fail over.
 
Regards
 
Rajith

________________________________________
From: Michael Hillier [mailto:[email-masked]] 
Sent: Wednesday, July 04, 2007 1:12 PM
To: [email-masked]
Subject: [Dime] Failover problems with Relays and crashed applications
Hi all,

another question has come up in practice.  Again we are in the context of Credit Control Application, but the issue must be applicable to all such applications which define their own failover mechanisms

Given a client (C) communicating via a primary and secondary relay (Rp, Rs) towards a primary and secondary server (Sp, Ss).

All Diameter connections are up and running.  The client selects Rp and Rp selects Sp, being the primary next hops.

Now the primary Server stops answering the application requests (i.e  no CCAs are  returned) - for what ever reason, 
however the connection Rp <=> Sp is still ticking over, i.e. DWRs are being answered by Sp.

Short timers, like Tx,  may expire in the client, possibly allowing service to be delivered, but that is a side issue.

Now the transaction timer expires causing the client to failover to the secondary relay (Rs), i.e. re-sending the CCR with T-bit
Rs selects the primary Server, because it does not know any reason not to, and thus the CCR(with T-bit) is sent to the same server which has failed.

Summary:  client failover to secondary next hop works nicely when directly connected to the Diameter servers, but does not achieve the desired effect when relays are used, which have no "application timer" awareness.

Many thanks

Mike

Michael Hillier
Telenet AG Rhein-Main
[email-masked]

---

From: "harish raghuveer" <[email-masked]>
Date: Fri, 6 Jul 2007 12:13:45 +0530

I feel that this could be achieved by including some re-negotiation facility
to the existing capability exchange mechanism of the base protocol. This
would need state machine modifications though. But definite advantage I can
see in a service provider scenario will be:
1. new applications can be added to a server in a hit-less manner
2. misbehaving applications (detected by some local implementation dependent
means) can be removed from the list of locally supported applications and
can be re-advertised. this can cause a client to re-trigger to some backup
server that can currently serve the application.

This should work in either case (direct connection as well as connection
through proxies).

thanks and regards,
Harish

On 7/6/07, Asveren, Tolga <[email-masked]> wrote:
>
> I agree that this would be faulty behavior. OTOH, it is always a good idea
> to be protected against misbehaving nodes. By applying a local check, where
> neighbors, from which no Diameter application level reply has been received
> for a while, are no more used to send messages to could be a good idea (or
> disconnecting connection to such nodes).
>
>    Thanks,
>    Tolga
>

-- 
Harish R Prabhu
Bangalore, India.

---

From: Rajith R <[email-masked]>
Date: Fri, 06 Jul 2007 13:50:54 +0530

Doesn't the section 5.6.5 of the 3588-bis-04 describe this "facility"?

  _____  

From: harish raghuveer [mailto:[email-masked]] 
Sent: Friday, July 06, 2007 12:14 PM
To: [email-masked]
Subject: Re: [Dime] Failover problems with Relays and crashed applications

thanks and regards,
Harish

   Thanks,
   Tolga

Regards

Rajith

________________________________________
From: Michael Hillier [mailto:  <mailto:[email-masked]>
[email-masked]]
Sent: Wednesday, July 04, 2007 1:12 PM
To: [email-masked]
Subject: [Dime] Failover problems with Relays and crashed applications
Hi all,

Now the primary Server stops answering the application requests (i.e no CCAs
are returned) - for what ever reason,
however the connection Rp <=> Sp is still ticking over, i.e. DWRs are being
answered by Sp. 

Many thanks

Mike

Michael Hillier
Telenet AG Rhein-Main
[email-masked]

-- 
Harish R Prabhu
Bangalore, India.

---

From: "harish raghuveer" <[email-masked]>
Date: Fri, 6 Jul 2007 14:16:46 +0530

Yes, it does - thanks for pointed it out! I am still using 3588 as reference
- I probably should upgrade.. :-)

On 7/6/07, Rajith R <[email-masked]> wrote:
>  ------------------------------
> *From:* harish raghuveer [mailto:[email-masked]]
> *Sent:* Friday, July 06, 2007 12:14 PM
> *To:* [email-masked]
> *Subject:* Re: [Dime] Failover problems with Relays and crashed
> applications
>
> I feel that this could be achieved by including some re-negotiation
> facility to the existing capability exchange mechanism of the base protocol.
> This would need state machine modifications though. But definite advantage I
> can see in a service provider scenario will be:
> 2. misbehaving applications (detected by some local implementation
> dependent means) can be removed from the list of locally supported
> applications and can be re-advertised. this can cause a client to re-trigger
> to some backup server that can currently serve the application.
>
> > I agree that this would be faulty behavior. OTOH, it is always a good
> > idea to be protected against misbehaving nodes. By applying a local check,
> > where neighbors, from which no Diameter application level reply has been
> > received for a while, are no more used to send messages to could be a good
> > idea (or disconnecting connection to such nodes).
> >
> >    Thanks,
> >    Tolga
> >
> > From: Michael Hillier [mailto: [email-masked]]
> > directly connected to the Diameter servers, but does not achieve the desired
> > effect when relays are used, which have no "application timer" awareness.
> >