Subject: [AAA-WG]: Diameter Credit Control: Failover procedures updated

From: <[email-masked]>
Date: Tue, 2 Sep 2003 15:59:28 +0300

Hi all,

After some discussion, we are proposing the following changes to the Credit Control 
Failover procedure.  I believe Avi raised this point in his review.  We'd appreciate
comments on this, we plan on incorporating this into the next version of the draft.

thanks,
John

1) Change the second and third paragraphs in section 4.1.4 Failure Procedures

FROM

If a communication failure occurs during an ongoing credit-control session the credit-control client can move the credit control message stream to an alternative server if the value of the CC-Session-Failover AVP is set to FAILOVER SUPPORTED. A secondary Credit control server name received for instance from the AAA server, can be used as an address of the backup server. If the CC-Session-Failover AVP is set to FAILOVER_NOT SUPPORTED the credit control message stream MUST NOT be moved to backup server and the credit control client will terminate or continue the service depending on the value set in the Credit-Control-Failure-Handling AVP. The Credit-Control-Failure-Handling AVP MAY be sent from the authorization server and in the Credit-Control-Answer from the credit-control server. For new credit-control sessions, failover to an alternate credit-control server SHOULD be performed, if possible.

The timer, Tx (as defined in section 10), is used in the credit-control client to supervise the communication with the credit-control server.

TO

If a failure occurs during an ongoing credit-control session the credit-control client may move the credit control message stream to an alternative server if the value of the CC-Session-Failover AVP is set to FAILOVER SUPPORTED. A secondary Credit control server name received for instance from the AAA server or locally configured, can be used as an address of the backup server. If the CC-Session-Failover AVP is set to FAILOVER_NOT SUPPORTED the credit control message stream MUST NOT be moved to backup server. 

For new credit control sessions, failover to an alternative credit-control server SHOULD be performed if possible. For instance, if an implementation of the credit control client can determine primary credit control server unavailability it can establish the new credit control sessions with a possibly available secondary credit control server.

The AAA client/agent is typically using only a single persistent transport connection to the AAA agent or server, but it may have connections to multiple AAA agents or servers and treat them as primary/secondary or balance load between them. The AAA transport profile [AAATRANS] defines the application layer watchdog algorithm that enables failover from a peer that has failed and is controlled by the timer Twinit. The recommended default value for Twinit is 30 seconds. Since the AAA infrastructure is common to several different types of AAA applications, tuning the timer Twinit to a lower value in order to satisfy the requirements of real-time applications, such as the Diameter Credit Control application, will certainly increase the probability of premature failover significantly and potentially cause congestive collapse in heavy loaded networks. For prepaid services, however, the end user expects an answer from the network in a reasonable time, thus the Diameter credit control client shall react faster than the underlaying base protocol. Therefore this specification defines the timer Tx that is used by the credit-control client (as defined in section 10) to supervise the communication with the credit-control server. When the timer Tx elapses the credit-control client takes an action to the end user according to the Credit-Control-Failure-Handling AVP. The Credit-Control-Failure-Handling AVP MAY be sent from the authorization server and in the Credit-Control-Answer from the credit-control server.  

When Tx expires, the Diameter credit control client always terminates the service if the Credit-Control-Failure-Handling (CCFH) AVP is set to the value TERMINATE. The credit control session may be moved to an alternative server only in case a protocol error DIAMETER_TOO_BUSY or DIAMETER_UNABLE_TO_DELIVER is received before Tx expires, therefore, the value TERMINATE is not appropriate if proper failover behavior is desired.

If the Credit-Control-Failure-Handling AVP is set to the value CONTINUE or RETRY_AND_TERMINATE, the service will be granted to the end user upon the timer Tx expires. An answer message with granted-units may arrive later on due to the base protocol transport failover occurred in the path to the Credit Control Server (Twinit default value is 3 times more than the Tx recommended value). The credit control client SHOULD grant the service to the end user, start monitoring the resource usage and wait for the possible late answer until the timeout of the request (e.g. 120 seconds).  If the request fails and the CC-Session-Failover AVP is set to FAILOVER_NOT SUPPORTED, the credit control client terminates or continues the service depending on the value set in the CCFH and MUST free all the reserved resources for the credit control session.If a protocol error DIAMETER_UNABLE_TO_DELIVER or DIAMETER_TOO_BUSY is received or the request timeout and the CC-Session-Failover AVP is set to FAILOVER SUPPORTED, the credit control client MAY send the request to a backup server if possible. If the credit control client receives a successful answer from the backup server, it continues the credit control session with such a server. If also the re-transmitted request fails, the credit control client terminates or continues the service depending on the value set in the CCFH and MUST free all the reserved resources for the credit control session.

2) Add the following sentence at the end of section 4.1.4

In order to support the failover between credit control servers information transfer about the credit control session and account state SHOULD take place between the primary and the secondary credit control server. Implementations supporting the credit control session failover MUST also ensure proper detection of duplicate or out of sequence messages. The communication between the servers is regarded as an implementation issue and is outside of the scope of this specification.

3) Change the first and fourth paragraphs in section 4.2.5 Failure Procedures

FROM

Failover to an alternate credit-control server is allowed for one time event since the server is not maintaining session states. There MAY exist protocol transparent Diameter relays and redirect agents or Diameter credit-control proxies between credit-control client and credit-control server. Failover may occur at any point in the path between credit-control client and credit-control server in the event that a transport failure is detected with a peer, as described in [DIAMBASE].

If the requested action is DIRECT_DEBITING and the Direct-Debiting-Failure-Handling AVP is set to TERMINATE_OR_BUFFER the credit-control client SHOULD terminate the service if it can determine from the result code or error code in the answer message that units have not been debited. Otherwise the credit-control client SHOULD grant the service to the end user and store the request in the credit-control application level non-volatile storage. The credit-control client MUST mark these request messages as possible duplicate by setting the T-flag in the command header as described in [DIAMBASE] section 3. If the Direct-Debiting-Failure-Handling AVP is set to CONTINUE the service SHOULD be granted even if credit-control messages can't be delivered. If the timer Tx expires the credit-control client MUST continue the service and eventually buffer the request according to the value of the Direct-Debiting-Failure-Handling AVP. 
 
TO

Failover to an alternative credit-control server is allowed for one time event since the server is not maintaining session states, for instance, if the credit control client receives a protocol error DIAMETER_UNABLE_TO_DELIVER or DIAMETER_TOO_BUSY it can re-send the request to an alternative server if possible. There MAY exist protocol transparent Diameter relays and redirect agents or Diameter credit-control proxies between credit-control client and credit-control server. Failover may occur at any point in the path between credit-control client and credit-control server in the event that a transport failure is detected with a peer, as described in [DIAMBASE].

If the requested action is DIRECT_DEBITING and the Direct-Debiting-Failure-Handling AVP is set to TERMINATE_OR_BUFFER, the credit-control client SHOULD terminate the service if it can determine, eventually after a possible re-transmission attempt to an alternative credit control server, from the result code or error code in the answer message that units have not been debited. Otherwise the credit-control client SHOULD grant the service to the end user and store the request in the credit-control application level non-volatile storage (Note that re-sending the request at a later time is not a guarantee that the service will be debited, since the user's account may be empty at the time when the server successfully processes the request). The credit-control client MUST mark these request messages as possible duplicate by setting the T-flag in the command header as described in [DIAMBASE] section 3. If the Direct-Debiting-Failure-Handling AVP is set to CONTINUE, the service SHOULD be granted even if credit-control messages cannot be delivered and messages are not buffered.
If the timer Tx expires the credit-control client MUST continue the service and wait for a possible late answer. If the request timeout the credit control client re-transmit the request (marked with T-flag) to a backup credit control server if possible. In the event that also the re-transmitted request timeout or a temporary error is received in answer to such a request, the credit control client eventually buffers the request according to the value of the Direct-Debiting-Failure-Handling AVP. If a failed answer is received for the re-transmitted request, the credit control client free all the resources reserved for the event message.

4) Change the 'Failure to send' definition in section 4.5

FROM

In the state table, the event 'Failure to send' means that the Diameter credit-control client is unable to communicate with the desired destination (i.e. the answer message is not received within the validity time of the request). This could be due to the peer being down, or due to a physical link failure in the path to/from the credit-control server.

TO

In the state table, the event 'Failure to send' means that the Diameter credit-control client is unable to communicate with the desired destination or with a possibly defined alternative destination in case failover procedure is supported (e.g. the request timeout and the answer message is not received). This could be due to the peer being down, or due to a physical link failure in the path to/from the credit-control server.

5) Change the 'Temporary error' definition in section 4.5

FROM

The event 'Temporary error' means that the Diameter credit-control client received a transient failure notification in the Credit-Control-Answer command (i.e. the peer sending back a transient failure or temporary protocol error notification DIAMETER_TOO_BUSY, or DIAMETER_LOOP_DETECTED in the Result-Code AVP).

TO

The event 'Temporary error' means that the Diameter credit-control client received a protocol error notification DIAMETER_TOO_BUSY, DIAMETER_UNABLE_TO_DELIVER or DIAMETER_LOOP_DETECTED in the Result-Code AVP of the Credit-Control-Answer command. The above protocol error notification may be ultimately received in answer to the re-transmitted request to a possibly defined alternative destination if failover is supported.

6) Change the 'Failed answer' definition in section 4.5

FROM

The event 'Failed answer' means that the Diameter credit-control client received non-transient failure (permanent failure) notification in the Credit-Control-Answer command. 

TO

The event 'Failed answer' means that the Diameter credit-control client received non-transient failure (permanent failure) notification in the Credit-Control-Answer command. The above permanent failure notification may be ultimately received in answer to the re-transmitted request to a possibly defined alternative destination if failover is supported.

7) Update the state machine in section 4.5 as follow

In the following state machine table the failover to a possibly secondary server upon 'Temporary error' or 'Failure to send' is not explicitely described. Moving an ongoing credit control message stream to an alternative server is, however, possible if the CC-Session-Failover AVP is set to FAILOVER_SUPPORTED as described in section 4.3.4.
Re-sending a credit control event to a an alternative server is supported
as described in section 4.2.5.

    CLIENT, SESSION BASED for the first interrogation with CCR 

   State     Event                          Action       New State 
   --------------------------------------------------------------- 

  Idle      Client or device requests      Send         PendingI 
            access/service                 CC initial
                                           req., 
                                           start Tx. 
   
  PendingI  Successful CC initial          Stop Tx      Open 
            answer received 
   
  PendingI  Failure to send, or            Grant        Idle 
            temporary error and            service to           
            credit-control fault           end user 
            handling equal to CONTINUE  
   
  PendingI  Failure to send, or            Terminate    Idle 
            temporary error and            end user's 
            credit-control fault        	 service
            handling equal to TERMINATE
				or equal to RETRY_AND_TERMINATE

  PendingI  Tx expired and credit          Terminate    Idle 
            Control fault handling         end user's
            equal to TERMINATE 				 service
 
  PendingI  Tx expired and credit-control  Grant        PendingI 
            fault handling equal to        service to   
            CONTINUE or equal to           end user
	         RETRY_AND_TERMINATE 
 
  PendingI  CC initial answer              Terminate    Idle 
            received with result code      end user's 
            SERVICE_ DENIED or 				 service
            USER_UNKNOWN 
   
  PendingI  CC initial answer              Grant        Idle 
            received with result code      service  
            equal to credit-control N/A    to end user 
 
  PendingI  Failed CC initial answer       Grant        Idle 
            received and credit-control    Service to 
            fault handling                 end user 
            equal to CONTINUE 
 
     PendingI  Failed CC initial answer       Terminate    Idle 
            received and credit-control    end user's 
            failure handling equal to 		 service
            TERMINATE or equal to
				RETRY_AND_TERMINATE 
 
  PendingI  User service terminated        Queue        PendingI 
                                           termination 
                                           event  
   
  PendingI  Change in rating condition     Queue        PendingI 
                                           changed 
                                           rating  
                                           condition   
                                           event  

   CLIENT, SESSION BASED for intermediate and final interrogations 
   State     Event                          Action       New State 
   --------------------------------------------------------------- 

  Open      Granted unit elapses           Send         PendingU 
            and no final unit              CC update 
            indication received            req., 
                                           start Tx.   
   
  Open      Granted unit elapses           Terminate    PendingT 
            and final unit indication      end user's  
            received                       service, send
														 CC termination  
                                           req., 
                                           start Tx. 
   
  Open      Change in rating condition     Send         PendingU 
            in queue                       CC update 
                                           req., 
                                           Start Tx.            
 
  Open      Service terminated in queue    Send         PendingT 
                                           CC termination 
                                           req., 
                                           start Tx 
 
  Open      Change in rating condition     Send         PendingU 
            or Validity-Time elapses       CC update 
                                           req., 
                                           Start Tx.            
 
  Open      User service terminated        Send         PendingT 
                                           CC termination 
                                           req., 
                                           start Tx  
 
  PendingU  Successful CC update           Stop Tx      Open 
            answer received 
   
  PendingU  Failure to send, or            Grant        Idle 
            temporary error and            service to 
            credit-control fault           end user 
            handling equal to CONTINUE  
   
  PendingU  Failure to send, or            Terminate    Idle 
            temporary error and            end user's 
            credit-control fault        	 service
            handling equal to TERMINATE
				or equal to RETRY_AND_TERMINATE 
    
  PendingU  Tx expired and credit-control  Terminate    Idle 
            fault handling equal to        end user's 
            TERMINATE             	   	 service
 
  PendingU  Tx expired and credit-control  Grant        PendingU 
            fault handling equal to        service to   
            CONTINUE or equal to           end user.
				RETRY_AND_TERMINATE 
   
  PendingU  CC update answer               Terminate    Idle 
            received with result code      end user's 
            SERVICE_DENIED  		   		 service
 
  PendingU  CC update answer               Grant        Idle 
            received with result code      service  
            equal to credit-control N/A    to end user 
 
  PendingU  Failed CC update               Grant        Idle 
            answer received and credit     service to 
            control fault handling equal   end user. 
            to CONTINUE 
 
  PendingU  Failed CC update               Terminate    Idle 
            answer received and credit     end user's 
            control fault handling 			 service
            equal to TERMINATE or
				equal to RETRY_AND_TERMINATE 
 
  PendingU  User service terminated        Queue        PendingU
                                           termination 
                                           event 
   
  PendingU  Change in rating               Queue        PendingU 
            condition                      changed  
                                           rating 
                                           condition  
                                           event 
   
  PendingT  Successful CC                               Idle 
            termination answer received 
   
  PendingT  Tx expired                                  PendingT 
   
  PendingT  Failure to send, or temporary               Idle 
            error or failed answer 
                             
  PendingT  Change in rating condition                  PendingT 

                            CLIENT, EVENT BASED 
  State     Event                          Action        New State 
  ---------------------------------------------------------------- 
  Idle      Client or device requests      Send          PendingE 
            a one-time service             CC event 
                                           req., 
                                           Start Tx. 
   
  Idle      Request in storage             Send          PendingB 
                                           stored  
                                           request 
    
  PendingE  Successful CC event                          Idle 
            answer received 
 
  PendingE  Failure to send, temporary     Indicate      Idle 
            error or failed CC event       service 
            answer received, or            error 
            Tx expired, requested           
            action GET_BALANCE or 
            PRICE_ENQUIRY 
   
  PendingE  CC event answer                Terminate     Idle 
            received with result code      end user's 
            		SERVICE_DENIED or					 service
               USER_UNKNOWN and Tx running		 
 
  PendingE  CC event answer                Grant         Idle 
            received with result code      service 
            credit-control N/A, requested  to end          
            action DIRECT_DEBITING         user 
 
  PendingE  Failure to send, temporary     Grant         Idle 
            error or failed CC event       service 
            answer received, requested		 to end 
            action DIRECT_DEBITING and     user
            fault handling equal to         
            CONTINUE 
 
  PendingE  Failed CC event                Terminate     Idle 
            answer received or temporary   end user's
				error, requested action        service
            DIRECT_DEBITING and       		 
            fault handling equal to          
            TERMINATE_OR_BUFFER and
				Tx running 

  PendingE  Tx expired, requested				 Grant			PendingE
				action DIRECT_DEBITING			 service
												       to end 
														 user
   
  PendingE  Failure to send, requested		 Store	      Idle 
            action DIRECT_DEBITING and     request with 
            fault handling equal to        T-flag  
            TERMINATE_OR_BUFFER                                                                                                 

  PendingE  Temporary error, requested     Store    		Idle 
            action DIRECT_DEBITING and     request 
            fault handling equal to         
            TERMINATE_OR_BUFFER and
				Tx expired

  PendingE	Failed answer or answer             			   Idle
				received with result code
				SERVICE DENIED or USER_UNKNOWN, 
				requested action 
				DIRECT_DEBITING and Tx expired  
                         
  PendingE  Failed CC event answer         Indicate      Idle 
            received, requested            service  
            action REFUND                  error and                  
                                           delete request                             
 
  PendingE  Failure to send or             Store         Idle 
            Tx expired, requested          request 
            action REFUND                  with T-flag
 
  PendingE  Temporary error                Store         Idle 
            and requested action           request 
            REFUND            
   
  PendingB  Successful CC answer           Delete        Idle 
            received                       request 
   
  PendingB  Failed CC answer               Delete        Idle 
            received                       request 
        
  PendingB  Failure to send or                           Idle                   
               temporary error

8) Change the description of the Tx timer in section 10

FROM

Tx timer 
 
When real-time credit-control is required, the credit-control client contacts the credit-control server before and during the service is provided to an end user. Due to real-time nature of application the communication delays SHOULD be minimized, e.g. to avoid too long service set up time experienced by the end user. The Tx timer is introduced to control the waiting time in the client in the PENDING state. 
     
	The recommended value is 10 seconds.

TO

Tx timer 
 
When real-time credit-control is required, the credit-control client contacts the credit-control server before and during the service is provided to an end user. Due to real-time nature of application the communication delays SHOULD be minimized, e.g. to avoid too long service set up time experienced by the end user. The Tx timer is introduced to control the waiting time in the client in the PENDING state. When the Tx timer elapses the credit-control client takes an action to the end user according to the value of the Credit-Control-Failure-Handling AVP or according to the value of the Direct-Debiting-Failure-Handling AVP.
     
	The recommended value is 10 seconds.

9) Change the section 5.10

FROM
     
   The Credit-Control-Failure-Handling AVP (AVP Code TBD) is of type 
   Enumerated. The credit-control client uses information in this AVP to 
   decide what to do if the sending of credit-control messages to the 
   credit-control server has been for instance temporarily prevented due 
   to a network problem.  
        
   TERMINATE                                                0  
    
   When the Credit-Control-Failure-Handling AVP is set to TERMINATE the 
   service MUST only be granted as long as there is a connection to the
   credit-control server. If the credit-control client does not receive 
   any Credit-Control-Answer message within the Tx timer (as defined in 
   section 10) the credit-control request is regarded failed. The moving
   of already started credit-control session to alternative server is 
   not allowed.   
            
   This is the default behavior if the AVP isn't included in the reply 
   from the authorization or credit-control server.  
     
   CONTINUE                                                 1  
    
   When the Credit-Control-Failure-Handling AVP is set to CONTINUE the 
   service SHOULD be granted even if credit-control messages can't be 
   delivered.  

TO
     
   The Credit-Control-Failure-Handling AVP (AVP Code TBD) is of type 
   Enumerated. The credit-control client uses information in this AVP to 
   decide what to do if the sending of credit-control messages to the 
   credit-control server has been for instance temporarily prevented due 
   to a network problem. Depending on the service logic, the 
   credit-control server can order the client to terminate the service
   immediately when there is a reason to believe that the service cannot
   be charged, or to try failover to an alternative server, if possible,
   and then either terminate or grant the service should also the
   alternative connection fail.
        
   TERMINATE                                                0  
    
   When the Credit-Control-Failure-Handling AVP is set to TERMINATE the 
   service MUST only be granted as long as there is a connection to the 
   credit-control server. If the credit-control client does not receive 
   any Credit-Control-Answer message within the Tx timer (as defined in 
   section 10) the credit-control request is regarded failed and the end
   user's service session is terminated.
            
   This is the default behavior if the AVP isn't included in the reply 
   from the authorization or credit-control server.  
     
   CONTINUE                                                 1  
    
   When the Credit-Control-Failure-Handling AVP is set to CONTINUE the
   credit-control client SHOULD re-send the request to an alternative
   server in case of transport or temporary failures, provided that
   failover procedure is supported in the credit-control server
   and the credit-control client, and an alternative server is available.
   Otherwise, the service SHOULD be granted even if credit-control
   messages can't be delivered.

   RETRY_AND_TERMINATE                                      2  
    
   When the Credit-Control-Failure-Handling AVP is set to
   RETRY_AND_TERMINATE the credit-control client SHOULD re-send the
   request to an alternative server in case of transport or temporary
   failures, provided that failover procedure is supported in the
   credit-control server and the credit-control client, and an
   alternative server is available. Otherwise, the service SHOULD not
   be granted when the credit-control messages can't be delivered.