Subject: [Dime] RE: The Auditing Problem

From: "Ulf Bodin" <[email-masked]>
Date: Tue, 16 Jan 2007 13:05:48 +0100

Folks, 

Following Hannes advice I've put together a first proposal for more
detailed example scenarios/use cases. 

Before going into those use cases I just want to remind of the earlier
discussion on applicability by repeating my opinion of that a good
design approach would be to for a new application first explore if
soft-state without auditing is sufficient before considering designing
for auditing (in case we manage to specify a solution for auditing that
is :-). Such an apporach is in my view beneficial since auditing adds
complexity and creates excess signaling traffic (potentially at times
when additional signaling load should be avoided). 

Anyway, considering applications for which auditing is deemed needed I
beleive the following use cases to represent typical scenarios for which
auditing provides a clear value. 

I'm starting with describing the last one (I think the hard-state use
cases more clearly need auditing). Also, I believe the requirements to
be quite similar for both client and server failover. Hence, for now I'm
only considering server failover (i.e. to not take a too big bite too
soon ... ). 

Use case 1: Hard-state without state replication

In general this type of use case can be expected to only apply to
low-volume and long holding-time applications. For such applications the
overhead of performing state rebuild through auditing may be tolerated
as it can spare resources (i.e. state replication may consume hardware
(CPU and memory) and network resources). 

Use case TBD.  

Use case 2: Hard-state with replication 

Server failover 

This use case would be basically the same as use case 4 for server
failover except that the period for which the server sB operats on
inaccurate information will last until a management function terminates
malicious states. 

Use case 3: Soft-state without replication 

Use case TBD. 

Use case 4: Soft-state with replication 

Server failover 

The application I have in mind for this use case is a resource
reservation made by a client to ETSI TISPAN RACS. I think this use case
should prefereably be agnostic to the particular application though.
Hence, the text does not refer explicitly to this application (which I
believe is a good example, but not necessarily the only example). 

Server (B) is a backup for server (A). States of server (A) is
replicated via its database (a) (e.g. using high availability middleware
for replicating data). Writing data to database (a) is asyncronous and
so is the replication between the two databases. 

+------------+   1   +------------+   2   +---------------+
| soft-state +------>+ soft-state +------>+ replicated    |
| client (cA)+       | server (sA)|       | database (DBa)|
+------------+       +------------+       +------+--------+
                                               3 |
                                                 v
+------------+       +------------+       +---------------+
| soft-state |       | soft-state |   5   | replicated    |
| client (cB)+------>+ server (sB)|<------+ database (DBb)|
+------------+   4   +------------+       +---------------+

Now, say that the client cA "1" requests a service from server sA
resulting in that a soft-state is created in this server. Server sA
replies to the request in parallel with "2" storing the state in its
database DBa, whereafter this database "3" replicates the state to
database DBb. After cA receives the reply sA however craches and
subsequent requests of this client will therefore go to sB (e.g.
automatically selected through SCTP multihoming or gratuitous ARP for IP
takeover). The change of server is transparant to the client. 

The state established by client cA may not have been written to database
DBa before server sA crached. In case it was not written the state would
not have been replicated to DBb either. Alternatively, the state may
have been written to DBa, but not replicated to DBb. This means that
when sB receives "4" a request from cB (which may has had sB as its
primary server since before or also has switched from sA because it
chrached) and uses the "5" obtained set of states from DBb, which is
inaccurate. Consequently, sB will produce the wrong answer to cB. E.g.
in case of RACS accept (or deny) a reservation request because it didn't
know that the requested resources are already booked (or released) by
cA. 

The state is indeed soft and client cA will hence eventually learn that
it's no longer valid. Client cA will however believe the state to be
valid for a period up to the timer value, which can be several hundereds
of seconds. E.g. in case of reservations for telephony the mean holding
time for calls can be in the order of 300 seconds and setting the timer
to this value make sence since that keeps the signaling traffic for
updates sufficiently low.   

An optimization of this use case using auditing is to have server sB ask
its clients for lists of their respective active sessions (i.e. lists of
active sessions IDs). Such auditing request would be triggered by the
event of that sB learns that it has taken over clients from sA. Based on
these lists sB can determine of sessions it believes being active are
really terminated and identify active but unknown sessions. For sessions
being unknown to sB it can chose to either just wait until they expires,
or explicitly ask the clients for more information on those sessions
(i.e. in order to completely or partly re-create the missing states).
The benefit of auditing is in this use case the period for which sB
operates on inaccurate information can be reduced to a few seconds
instead of potentially several hundreds of seconds. 

To summarize; the auditing mechanisms involved in this use case are (1)
request from server to client for all active sessions and (2) request
from server to client for all details known by the client on a specific
session.  

Comments?

Best regards,
Ulf 

-----Original Message-----
From: Hannes Tschofenig [mailto:[email-masked]]
Sent: den 1 januari 2007 13:06
To: Ulf Bodin; Tolga Asveren
Cc: [email-masked]
Subject: The Auditing Problem

Hi Ulf,
Hi Tolga,

I think that the discussion triggered by Tolga was quite helpful in
order to get a better understanding of the auditing work.

What I would like to see is a more detailed example scenario (possibly
with message flows). Tolga provided a scenario with a figure but it
needs to be enhanced when it comes to the "service logic functionality".

I think that this is important since there is a front-end and a back-end
part to the entire story. The TISPAN QoS scenario has been mentioned as
a potential customer of this work and it could be a nice example to see
how the failure cases happen and how state is requested and moved
around. I can imagine that there are a number of issues that need to be
discussed when we jump into the details.

I have also noticed that a number of terms are used without clearly
describing them. For example, there are a number of different failover
cases that need more description to ensure that we are on the same page.

Another example: What is the definition for "session leaking"?

Ciao
Hannes

_______________________________________________
DiME mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dime

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 16 Jan 2007 17:04:46 -0500

Hi Ulf,

Please see below for some comments/questions.

I will try to reply in multiple messages (and change the subject line), so
that it is easier to track different issues.

    Thanks,
    Tolga

> From: Ulf Bodin [mailto:[email-masked]]
> Sent: Tuesday, January 16, 2007 7:06 AM
> To: [email-masked]
> Subject: [Dime] RE: The Auditing Problem
>
>
> Folks,
>
[TOLGA]Just to make sure that I have the correct understanding of soft v.s.
hard states: I guess we have the soft state if both of the following are
present:
a) The protocol dictates periodic requests, e.g. a timer is running on the
client, where after expiry of it, client generates a request.
b) A timer is running on the server and server drops the session if the
timer expires before another requests is received for the session.

IMHO, that type of timers are always necessary (regardless of the
presence/absence of an auditing mechanism), because:
i) Client and server may not have a direct Diameter connection where one
isn't able to detect failure of the client by base protocol procedures, e.g.
watchdog mechanism.
ii) There is no gurantee that a backup node will be present to run auditing
type of procedures to clean up the session on the peer.

In such a case, client may sit indefinetly with the assumption everything is
fine (or till the entity, which actually requested the service complains
about it). This approach seems risky to me.

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 16 Jan 2007 17:28:20 -0500

Hi Ulf,

[TOLGA] This is probably a side issue for auditing but IMHO still an
important one for the whole failover concept in Diameter:
a) Unless one comes up with a SCTP implementation distributed over multiple
hosts (which probably is not practical to implement), SCTP multihoming won't
provide a seamless failover during host failures.
b) Similarly IP takeover will cause the TCP/SCTP connection to be dropped,
unless one can synchronize TCP/SCTP state variables among different hosts.
Again, loss of TCP/SCTP connection will be visible to the application.

IMHO, the proper way of handling Diameter failovers after server crashes is
that the request is retried with an alternate server identity.

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 16 Jan 2007 17:55:20 -0500

Hi Ulf,

> when additional signaling load should be avoided).
[TOLGA] Recovery after failovers is definitly an interesting problem and may
or may not require some heavier than lightweight solution. In general my
concerns for auditing type of functionality are:
a)It seems to expect too much from the peer to succeed for recovery. One
could argue that this is what protocols are for but maybe there is a
difference between necesary communication to provide services and
communication to aide for local recovery. Usually, operators don't like to
have much dependency between entities, especially if the entity they rely
for recovery is controlled by some other organization. Of course, this is
open to argument (like any statement with subjective words like "much"), but
just an observation. Maybe that level of interaction is unavoidable.

b)It is not clear to me whether such a mechanism should be generic or per
application. Here I am not talking about whether all applications have to
use it but rather the mechanics of the procedure.

Having said all of that, I am definitly for exploring this problem and
hopefully coming up with a solution.

---

From: "Ulf Bodin" <[email-masked]>
Date: Wed, 17 Jan 2007 10:34:28 +0100

Hi Tolga, 

My replies below; [Ulf]. 

Best regards, 
Ulf 

> for auditing (in case we manage to specify a solution for auditing 
> that is :-). Such an apporach is in my view beneficial since auditing 
> adds complexity and creates excess signaling traffic (potentially at 
> times when additional signaling load should be avoided).
[TOLGA]Just to make sure that I have the correct understanding of soft
v.s.
hard states: I guess we have the soft state if both of the following are
present:
a) The protocol dictates periodic requests, e.g. a timer is running on
the client, where after expiry of it, client generates a request.
b) A timer is running on the server and server drops the session if the
timer expires before another requests is received for the session.

[Ulf] Yes. I'm thinking of using the existing AVPs for soft-state as
they're defined. I.e. the Authorization-Lifetime AVP, which to my
understanding relates to the time of the client timer setting and the
Auth-Grace-Period AVP, which is the additional time used for the server
timer setting. 

IMHO, that type of timers are always necessary (regardless of the
presence/absence of an auditing mechanism), because:
i) Client and server may not have a direct Diameter connection where one
isn't able to detect failure of the client by base protocol procedures,
e.g.
watchdog mechanism.
ii) There is no gurantee that a backup node will be present to run
auditing type of procedures to clean up the session on the peer.

In such a case, client may sit indefinetly with the assumption
everything is fine (or till the entity, which actually requested the
service complains about it). This approach seems risky to me.

[Ulf] Indefinite Diameter states may indeed not be the legacy we want to
leave our children :-). I completely agree that timers are needed also
for hard-state. I'm not sure such timers should be visible in the
protocol though. This is because a would prefer having the timers
triggered at loss of connectivity instead of having them initiated by a
Diameter message. 

I picture that the death of a server (or client) may need to be
confirmed by other means than timers only. E.g. say that a server died
and that the client fails to connect to its hot backup (because it's
dead as well). However, as the fault management system detects this
event operational people may decide to boot a new device and have it
take over as a cold backup. When coming up this server will audit the
client when it connects. 

Indeed, also in this scenario a timer needs to be present in the client
in case this procedure fails in its entire. This timer may be set to a
number of minutes or even an hour or so and could be started at the
event of loss of connectivity to the primary server (i.e. if a backup
server has not been brought up within this time all states will be
removed in the affected clients). I'm not an expert on COPS, but I
believe it implements such timers (should look before saying that, but I
hope you take it for what it is: more or less a guess :-)).

---

From: "Ulf Bodin" <[email-masked]>
Date: Wed, 17 Jan 2007 11:16:56 +0100

Again, replies below [Ulf]. 
Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 16 januari 2007 23:28
To: [email-masked]
Subject: [Dime] The Auditing Problem : Selection of the backup

Hi Ulf,

> automatically selected through SCTP multihoming or gratuitous ARP for 
> IP takeover). The change of server is transparant to the client.
[TOLGA] This is probably a side issue for auditing but IMHO still an
important one for the whole failover concept in Diameter:
a) Unless one comes up with a SCTP implementation distributed over
multiple hosts (which probably is not practical to implement), SCTP
multihoming won't provide a seamless failover during host failures.
b) Similarly IP takeover will cause the TCP/SCTP connection to be
dropped, unless one can synchronize TCP/SCTP state variables among
different hosts.
Again, loss of TCP/SCTP connection will be visible to the application.

[Ulf] You're right of course. Loss of connectivity is certainly visible
to the client and the failover os thus not completely transparent. I
ment that the failover would be transparent to the client in the sense
that the application service should not be affected. A wird and
inadequate way of saing that referring to tranparency though. 

IMHO, the proper way of handling Diameter failovers after server crashes
is that the request is retried with an alternate server identity.

[Ulf] Yes, that is fine when failing over to a backup server. However,
when using SCTP multihoming for path failover the SCTP stack can
transparntly reconnect to the same server but using an IP address
different from the one used when the connection broke. That is, two
alternative paths are configured through the network between client and
server. As I see it, having the client retry with a different server
identity is mainly an alternative to IP takeover. 

In case IP takover is not to be used at all maybe failover to a backup
server could be implemented either with alternative server identities in
the application or using SCTP multihoming. To be honest, I don't know
why either one of those approaches would be better then the other one.

---

From: "Ulf Bodin" <[email-masked]>
Date: Wed, 17 Jan 2007 11:34:57 +0100

Replies below [Ulf]. 
Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 16 januari 2007 23:55
To: [email-masked]
Subject: [Dime] The Auditing Problem: General Concerns

Hi Ulf,

> times when additional signaling load should be avoided).
[TOLGA] Recovery after failovers is definitly an interesting problem and
may or may not require some heavier than lightweight solution. In
general my concerns for auditing type of functionality are:
a)It seems to expect too much from the peer to succeed for recovery. One
could argue that this is what protocols are for but maybe there is a
difference between necesary communication to provide services and
communication to aide for local recovery. Usually, operators don't like
to have much dependency between entities, especially if the entity they
rely for recovery is controlled by some other organization. Of course,
this is open to argument (like any statement with subjective words like
"much"), but just an observation. Maybe that level of interaction is
unavoidable.

[Ulf: I agree that dependency between entities is something that
introduce risk and may reduce system reliability as such. I believe that
this is important to have in mind when looking into mechanisms for
auditing. In particular, system reliability should (must) not be put at
risk by the use of auditing. Having said that I think we need to see
(and design) auditing mechanisms as something to use for optimization
rather than to ensure correct functioning of a system. Your notes on
that timers are needed also for hard-states is a good example following
that observation I think.  

b)It is not clear to me whether such a mechanism should be generic or
per application. Here I am not talking about whether all applications
have to use it but rather the mechanics of the procedure.

[Ulf: In my view support for auditing can be deveoped per application,
but I also believe a generic mechanism fitting most if not all
applications that need auditing is within reach (i.e. I think the
Calhoun draft on the topic is already quite close). IMHO, a generic
mechanism is preferable to avoid people doing different solutions and
potentially make different (or the same) misstakes :-}.

---

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 17 Jan 2007 17:20:00 -0500

Hi Ulf,

Please see for comments/questions below.

   Thanks,
   Tolga

> Auth-Grace-Period AVP, which is the additional time used for the server
[TOLGA]One dummy question about hard-state case, let's assume the following
abstract configuration:

  Requester
  for the
  Resource        Middleman      Resources

   <--Diameter------->    <--XYZ--->

   <------Resource usage------------>
          relationship

If I (as the requeser) want the resource to be available, until I explicitly
indicate that I don't want to use it anymore, what is bad if the Middleman
dies? Let's say the requester wants to release the resource, sends the
corresponding request to the middleman it used when requesting the
resources. It gets back an error answer "UNABLE_TO_DELIVER" and can try with
another middleman. The "new" middleman would release the resources. Here I
assume that the middleman is stateless or able to reconstruct the state
based on the content of the release message and some additional queries on
some other entities but I guess this is not unrealistic with careful
protocol design (as a negative example, if the release resources message
contains only the Session-Id, it won't work)

I guess the question is, why do I need to "resurrect" sessions on a "new"
middleman in advance, if I can do it whenever I need the services of the
middleman?
> hope you take it for what it is: more or less a guess :-)).
[TOLGA] I think if it is necessary to detect failure of servers from
protocol machinery point of view, a protocol timer is required. Like you
indicated, we can't assume that a certain type of configuration will be
present on the peer side. With the possible presence of intermediaries, e.g.
relay agents, there is also no straightforward way of determining crash of a
server. Currently one may use an "UNABLE_TO_DELIVER" error answer for a
subsequent request sent to a specific server as an indication that it went
down/unreachable. Actually, maybe this is what we need, a message with which
backups/cluster members can notify clients about failure of a particular
server.

I agree with you that to guard against local resource leaking, e.g. memory,
one can use a local non-protocol timer, if a protocol timer is not
available. Still, such a timer is also not without its worries if it is
completely decoupled from protocol processing. It must run longer than the
maximum expected life for a session, which could be quite a while for
certain types of applications (but again this is an implementation issue)

---

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 17 Jan 2007 17:30:33 -0500

Hi Ulf,

Please see below for comments/questions.

   Thanks,
   Tolga

> Sent: Wednesday, January 17, 2007 5:17 AM
> To: [email-masked]
> Subject: RE: [Dime] The Auditing Problem : Selection of the backup
>
>
[TOLGA]Yes, SCTP provides network/NIC redundancy. It is host failures where
it doesn't help much.
> why either one of those approaches would be better then the other one.
[TOLGA]I think IP failover is a dead end. Both TCP and SCTP are connection
oriented protocols, which makes IP takeover approach non-seamless. The
connection needs to be reestablished. For SCTP case, I guess I am missing
something in what you describe. I don't see how multihoming could be
helpfull when the host goes down. Regarding host failures, it is essentially
the same as TCP, no redundancy :-)

---

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 17 Jan 2007 18:16:28 -0500

Hi Ulf,

   Thanks,
   Tolga

> from server to client for all details known by the client on a specific
[TOLGA] I will skip the local/implementation dependent solution, afterall it
is not what we are trying to standardize but I guess also in that domain we
have quite a few different approaches which have been used in telecom
industry with more or less success in many years.

In terms of protocol aided solution, initially I can think of the following
list (they may or may not make sense for particular applications and I am
not claiming it is a full list :-)):

i- Client triggered refresh upon expiry of a timer on the client
This is essentially the soft state case, where the client can send a message
to an alternate server after expiry of a protocol timer with enough
information embedded in the message, so that the alternate server can
process it.

ii- Client triggered refresh upon receipt of an error answer
Similar to i-, except that it is triggered upon receip of an
"UNABLE_TO_DELIVER" error answer.

iii- Peer triggered state data transfer upon receipt of a notification from
a peer
This is the approach described in the draft-calhoun-diameter-res-mgmt-08.txt

iv- Peer triggered refresh
This is similar to iii- but relies on the application messages to rebuild
the state. A notification message is sent from a peer informing the client
that a server went down. The client refreshes sessions on the alternate
server. The message can be sent immediately after receiving the notification
or could be delayed till the corresponding session(s) actually require a
message to be sent to update/delete.

i and ii seem to work with existing Diameter wisdom, but they may run into
issues, if servers also generate autonomous requests to clients (actually
this would probably answer the question I asked in a previous message about
the necessity of resurrection sessions/states on backup servers immediately
after they take over the role of a failed host)

For iii and iv, a backup peer needs to know which clients it needs to send a
notification to. This probably can be synchronized between peers of a
cluster.

It could be useful to think over different alternatives considering the RACS
scenario. For RACS, which interface are we talking about, Gq', Rq , any or
both?

---

From: "Ulf Bodin" <[email-masked]>
Date: Thu, 18 Jan 2007 14:46:30 +0100

Hi Tolga, 

Replies below. 

Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 17 januari 2007 23:20
To: [email-masked]
Subject: RE: [Dime] The Auditing Problem: Soft v.s. Hard States

Hi Ulf,

   Thanks,
   Tolga

> b) A timer is running on the server and server drops the session if 
> the timer expires before another requests is received for the session.
>
> Auth-Grace-Period AVP, which is the additional time used for the 
> server timer setting.
[TOLGA]One dummy question about hard-state case, let's assume the
following abstract configuration:

  Requester
  for the

If I (as the requeser) want the resource to be available, until I
explicitly indicate that I don't want to use it anymore, what is bad if
the Middleman dies? Let's say the requester wants to release the
resource, sends the corresponding request to the middleman it used when
requesting the resources. It gets back an error answer
"UNABLE_TO_DELIVER" and can try with another middleman. The "new"
middleman would release the resources. Here I assume that the middleman
is stateless or able to reconstruct the state based on the content of
the release message and some additional queries on some other entities
but I guess this is not unrealistic with careful protocol design (as a
negative example, if the release resources message contains only the
Session-Id, it won't work)

[Ulf2] Firstly, (this doesn't matter for the question though) shouldn't
the requester detect loss of connection followed by failing to
re-connect and not get a UNABLE_TO_DELIVER? I mean, the middleman is
dead and shouldn't hence return anything (I might miss something here).
Of course, if it's instead the server that died the middleman
could/should return an UNABLE_TO_DELIVER. Anyway, a stateless middleman
performing message routing is basically a Diameter relay agent. Isn't
it? In that case I agee to that state replication and/or auditing isn't
needed. 

On the other hand, for a middleman performing application layer
processing (maybe the term middleman is inappropriate for such an
entity?) or a middleman that is more of a proxy agent keeping states for
routing the above won't work as you say. In fact the SPDF can be either
one of the above-mention types of an entity (i.e. a relay agent, a proxy
agent or a entity performing application layer processing). As I see it,
the role of the SPDF currently depends on the actual deployment and the
vendor chosen.  

I guess the question is, why do I need to "resurrect" sessions on a
"new"
middleman in advance, if I can do it whenever I need the services of the
middleman?

[Ulf2] No, you don't need to in case we're talking about a relay agent
(if I understand your question correctly). 
> i) Client and server may not have a direct Diameter connection where 
> one isn't able to detect failure of the client by base protocol 
> procedures, e.g.
> watchdog mechanism.
> [Ulf] Indefinite Diameter states may indeed not be the legacy we want 
> to leave our children :-). I completely agree that timers are needed 
> also for hard-state. I'm not sure such timers should be visible in the

> triggered at loss of connectivity instead of having them initiated by 
> a Diameter message.
>

> Indeed, also in this scenario a timer needs to be present in the 
> client in case this procedure fails in its entire. This timer may be 
> set to a number of minutes or even an hour or so and could be started 
> at the event of loss of connectivity to the primary server (i.e. if a 
> backup server has not been brought up within this time all states will

> be removed in the affected clients). I'm not an expert on COPS, but I 
> believe it implements such timers (should look before saying that, but

> I hope you take it for what it is: more or less a guess :-)).
[TOLGA] I think if it is necessary to detect failure of servers from
protocol machinery point of view, a protocol timer is required. Like you
indicated, we can't assume that a certain type of configuration will be
present on the peer side. With the possible presence of intermediaries,
e.g.
relay agents, there is also no straightforward way of determining crash
of a server. Currently one may use an "UNABLE_TO_DELIVER" error answer
for a subsequent request sent to a specific server as an indication that
it went down/unreachable. Actually, maybe this is what we need, a
message with which backups/cluster members can notify clients about
failure of a particular server.

[Ulf2] Question: how would such a result code differ from the current
definition of UNABLE_TO_DELIVER? I'm asking because the answer will
hopefully help me understand better what you have in mind :-). 

I agree with you that to guard against local resource leaking, e.g.
memory, one can use a local non-protocol timer, if a protocol timer is
not available. Still, such a timer is also not without its worries if it
is completely decoupled from protocol processing. It must run longer
than the maximum expected life for a session, which could be quite a
while for certain types of applications (but again this is an
implementation issue)

[Ulf2] Do you really think it must run longer than the maximum expected
life of a session (would be troublesome if the session lengths
distribution is heavy tailed). In my view it could just be long enough
to allow for defined actions to be excecuted (e.g. actions taken by the
operationals).

---

From: "Ulf Bodin" <[email-masked]>
Date: Thu, 18 Jan 2007 14:52:00 +0100

Hi Tolga, 

Replies below. 

Best regards, 
Ulf 

>
> IMHO, the proper way of handling Diameter failovers after server 
> crashes is that the request is retried with an alternate server
identity.

> alternative paths are configured through the network between client 
> and server. As I see it, having the client retry with a different 
> server identity is mainly an alternative to IP takeover.
[TOLGA]Yes, SCTP provides network/NIC redundancy. It is host failures
where it doesn't help much.

> server could be implemented either with alternative server identities 
> in the application or using SCTP multihoming. To be honest, I don't 
> know why either one of those approaches would be better then the other
one.
[TOLGA]I think IP failover is a dead end. Both TCP and SCTP are
connection oriented protocols, which makes IP takeover approach
non-seamless. The connection needs to be reestablished. For SCTP case, I
guess I am missing something in what you describe. I don't see how
multihoming could be helpfull when the host goes down. Regarding host
failures, it is essentially the same as TCP, no redundancy :-)

[Ulf2] Yes. I guess IP failover is only potentially useful in case
connection information can be replicated as well (far fetched to my
knowledge), or with UDP. Gq' and Rq mandates SCTP so UDP is out of reach
at least for those application though. And, no, multihoming would not
provide any additional value (bad idea, sorry).

---

From: "Tolga Asveren" <[email-masked]>
Date: Thu, 18 Jan 2007 15:07:13 -0500

Hi Ulf,

  Thanks,
  Tolga

> it? In that case I agee to that state replication and/or auditing isn't
[TOLGA]My bad not being clear on this example. What I had in mind was that
the middleman is a Diameter server, it is not a middleman from Diameter
relaying perspective, just the middleman in terms of the whole service
(physical reservation/allocation of the resource). It gets a request to
reserve some resource, checks it against policy and relays a corresponding
request for the physical owner/controller of the resource. If the Diameter
client and server are directly connected, failure of server would result a
transport error/Diameter watchdog failure but if there are relay
agents/proxies etc.. between them, what the client gets will be
UNABLE_TO_DELIVER. So, what I refer as stateless middleman was a stateless
(or better said where the state can be constructed based on the messages
received from the client) Diameter server.
> the role of the SPDF currently depends on the actual deployment and the
[TOLGA]If the application processing entity is unable to reconstruct state
based on messages received, is there any way at all to achieve what we are
looking for? We try to come up with a mechanism, where the state is
reconstructed on a backup server by processing incoming messages. If this is
not possible, local implementation dependent mechanisms are all what we can
utilize for state recovery/replication.
> (if I understand your question correctly).
[TOLGA]Again sorry for the misrepresentation of what I thought. I was
refering to an application processing entity. Basically the question (after
rewording) is, why do I need to "resurrect" sessions on a "new" Diameter
server, if I can do that whenever the client need the services of the
server? Actually I guess now I have an idea what the answer might be: If the
server needs to generate autonomous requests for sessions. AFAICS, for any
application, which doesn't have that characeristic, there is no need for
premature session resurrection on a backup server, after its active pair
fails.
> hopefully help me understand better what you have in mind :-).
[TOLGA]If it were a result code, it wouldn't differ at all :-) What I think
is a message, which can indicate failure of a particular host. Of course it
needs to be sent from another host and should include the identity of the
failed host in an AVP. Without such a message, a client can be aware of a
server failure only after it send a request with Destination-Host AVP equal
to the identity to the failed server (even then, such an assumprtion would
require that the network between client and server is well engineered so
that a problem in IP or Diameter network won't cause lack of connectivity).
I guess if we want to provide premature state reconstruction on backup
servers, there has to be some sort of notification from this server to the
clients. I guess the overall problem can be split in two chunks:
a) How to notify clients
b) How to get the data necessary for state reconstruction from the clients
> to allow for defined actions to be excecuted (e.g. actions taken by the
[TOLGA]You may be right, one probably can come up with some clever way of
dealing with this problem. Initially I thought of a very simple mechanism of
deleting session which hang around for longer than anticipated. In any case,
this type of timer is probably implementation dependent and people will come
up with some mechanism depending on their environment/the application type
running etc...

---

From: "Ulf Bodin" <[email-masked]>
Date: Fri, 19 Jan 2007 11:18:44 +0100

Hi Tolga, 

Replies below. 

Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 18 januari 2007 00:16
To: [email-masked]
Subject: [Dime] The Auditing Problem: Possible approaches

Hi Ulf,

   Thanks,
   Tolga

> An optimization of this use case using auditing is to have server sB 
> ask its clients for lists of their respective active sessions (i.e. 
> lists of active sessions IDs). Such auditing request would be 
> triggered by the event of that sB learns that it has taken over 
> clients from sA. Based on these lists sB can determine of sessions it 
> believes being active are really terminated and identify active but 
> unknown sessions. For sessions being unknown to sB it can chose to 
> either just wait until they expires, or explicitly ask the clients for

> more information on those sessions (i.e. in order to completely or
partly re-create the missing states).
> To summarize; the auditing mechanisms involved in this use case are 
> (1) request from server to client for all active sessions and (2) 
> request from server to client for all details known by the client on a

> specific session.
[TOLGA] I will skip the local/implementation dependent solution,
afterall it is not what we are trying to standardize but I guess also in
that domain we have quite a few different approaches which have been
used in telecom industry with more or less success in many years.

In terms of protocol aided solution, initially I can think of the
following list (they may or may not make sense for particular
applications and I am not claiming it is a full list :-)):

[Ulf2] Ok. My intent was to identify requirements on a auditing
mechanisms from use cases (which tend to be local/implementation
dependent I guess :-). I agree that we are not to standardize
local/implementation dependent solutions. For the list of protocol aided
solutions I'm not sure if we need to explicitly account for whether
states are replicated or not. That is, a general enough auditing
mechansim may support both cases. Hence, I think your list is a good
start. 

i- Client triggered refresh upon expiry of a timer on the client This is
essentially the soft state case, where the client can send a message to
an alternate server after expiry of a protocol timer with enough
information embedded in the message, so that the alternate server can
process it.

[Ulf2] Yes. 

ii- Client triggered refresh upon receipt of an error answer Similar to
i-, except that it is triggered upon receip of an "UNABLE_TO_DELIVER"
error answer.

[Ulf2] Question for clarificaiton; would this mean to refresh all
session states associated with the target that could not be reached, or
just the session for which the error was given? 

iii- Peer triggered state data transfer upon receipt of a notification
from a peer This is the approach described in the
draft-calhoun-diameter-res-mgmt-08.txt

[Ulf2] Ok. 

iv- Peer triggered refresh
This is similar to iii- but relies on the application messages to
rebuild the state. A notification message is sent from a peer informing
the client that a server went down. The client refreshes sessions on the
alternate server. The message can be sent immediately after receiving
the notification or could be delayed till the corresponding session(s)
actually require a message to be sent to update/delete.

[Ulf2] Ok. What come to my mind though is the amount of messages that'll
be generated (this concern applies also to ii in case all session states
are to be refreshed). In my view some recommendations on how to control
the rate of messages and/or to make it driven by a receiver of the
updates would be needed. I might be jumping ahead too quick to the
solution now though.  

i and ii seem to work with existing Diameter wisdom, but they may run
into issues, if servers also generate autonomous requests to clients
(actually this would probably answer the question I asked in a previous
message about the necessity of resurrection sessions/states on backup
servers immediately after they take over the role of a failed host)

[Ulf2] Yes, autonomous requests/unsolicited call backs is a reason for
resurrection sessions/states on backup servers. I believe an additional
reason for why states should for some applications be resurrected is
that the application are controlling access to scarce resources (e.g.
RACS). 

For iii and iv, a backup peer needs to know which clients it needs to
send a notification to. This probably can be synchronized between peers
of a cluster.

[Ulf2] Yes. 

It could be useful to think over different alternatives considering the
RACS scenario. For RACS, which interface are we talking about, Gq', Rq ,
any or both?

[Ulf2] I'm considering both Gq' and Rq as well as the corresponding
interfaces as of ITU and 3GPP. I believe all your list items to
applicable to those interfaces. Which specific approach that'll be
adopted for each individual deployment (and by different vendors)
remains to be seen. My personal belief is that most A-RACF entities (the
part of RACS performing admission control) will replicate both session
and reservation states. The A-RACF is likely to offer soft-state only to
the SPDF (the part of RACS facing application frameworks such as e.g.
SIP proxies). The SPDF will probably also replicate states and offer
soft-state to its clients. The SPDF is however likely to also offer
hard-state to some application frameworks (e.g. leased line application
level servers). 

Given the above assumptions i and ii will be used without complete
information in refresh messages to rebuld states. In addition, iii will
be used to ensure syncronisation after failover (and potentially
periodically between application and SPDF in the case of hard-state).
Since states are replicated iv would not apply (if I understand the
approach correctly it applies to non cases when data is not replicated
only). However, for a deployment without replication for e.g. the A-RACF
iv would however apply.

---

From: "Ulf Bodin" <[email-masked]>
Date: Fri, 19 Jan 2007 13:03:07 +0100

Hi Tolga, 

Replies below. 

Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 18 januari 2007 21:07
To: [email-masked]
Subject: RE: [Dime] The Auditing Problem: Soft v.s. Hard States

Hi Ulf,

  Thanks,
  Tolga

> explicitly indicate that I don't want to use it anymore, what is bad 
> if the Middleman dies? Let's say the requester wants to release the 
> resource, sends the corresponding request to the middleman it used 
> when requesting the resources. It gets back an error answer 
> middleman would release the resources. Here I assume that the 
> middleman is stateless or able to reconstruct the state based on the 
> content of the release message and some additional queries on some 
> other entities but I guess this is not unrealistic with careful 
> protocol design (as a negative example, if the release resources 
> message contains only the Session-Id, it won't work)
>
> [Ulf2] Firstly, (this doesn't matter for the question though) 
> shouldn't the requester detect loss of connection followed by failing 
> to re-connect and not get a UNABLE_TO_DELIVER? I mean, the middleman 
> is dead and shouldn't hence return anything (I might miss something
here).
> could/should return an UNABLE_TO_DELIVER. Anyway, a stateless 
> middleman performing message routing is basically a Diameter relay 
> agent. Isn't it? In that case I agee to that state replication and/or 
> auditing isn't needed.
[TOLGA]My bad not being clear on this example. What I had in mind was
that the middleman is a Diameter server, it is not a middleman from
Diameter relaying perspective, just the middleman in terms of the whole
service (physical reservation/allocation of the resource). It gets a
request to reserve some resource, checks it against policy and relays a
corresponding request for the physical owner/controller of the resource.
If the Diameter client and server are directly connected, failure of
server would result a transport error/Diameter watchdog failure but if
there are relay agents/proxies etc.. between them, what the client gets
will be UNABLE_TO_DELIVER. So, what I refer as stateless middleman was a
stateless (or better said where the state can be constructed based on
the messages received from the client) Diameter server.
> entity?) or a middleman that is more of a proxy agent keeping states 
> for routing the above won't work as you say. In fact the SPDF can be 
> either one of the above-mention types of an entity (i.e. a relay 
> agent, a proxy agent or a entity performing application layer 
> processing). As I see it, the role of the SPDF currently depends on 
> the actual deployment and the vendor chosen.
[TOLGA]If the application processing entity is unable to reconstruct
state based on messages received, is there any way at all to achieve
what we are looking for? We try to come up with a mechanism, where the
state is reconstructed on a backup server by processing incoming
messages. If this is not possible, local implementation dependent
mechanisms are all what we can utilize for state recovery/replication.

[Ulf3] You're right of course. If the client cannot provide complete
enough state information in refresh messages it cannot be expected to do
that for explicit re-establishmen of states through auditing either.
Maybe there are cases when the client can reconstruct state informaiton
on demand for such an auditing action (i.e. states it doesn't normally
keep), but such scenario is in my view too specific to design general
mechanisms for.  

> middleman in advance, if I can do it whenever I need the services of 
> the middleman?
>

[TOLGA]Again sorry for the misrepresentation of what I thought. I was
refering to an application processing entity. Basically the question
(after
rewording) is, why do I need to "resurrect" sessions on a "new" Diameter
server, if I can do that whenever the client need the services of the
server? Actually I guess now I have an idea what the answer might be: If
the server needs to generate autonomous requests for sessions. AFAICS,
for any application, which doesn't have that characeristic, there is no
need for premature session resurrection on a backup server, after its
active pair fails.

[Ulf3] I actually see an additional reason for resurrecting states being
that the server is performing admission control to resources. That is,
an active session may have reserved resources which other sessions
should not get access to (since they're booked already). 

> > [Ulf] Indefinite Diameter states may indeed not be the legacy we 
> > want to leave our children :-). I completely agree that timers are 
> > needed also for hard-state. I'm not sure such timers should be 
> > visible in the
>
> > triggered at loss of connectivity instead of having them initiated 
> > by a Diameter message.
> >

> > take over as a cold backup. When coming up this server will audit 
> > the client when it connects.
> >

> > set to a number of minutes or even an hour or so and could be 
> > started at the event of loss of connectivity to the primary server 
> > (i.e. if a backup server has not been brought up within this time 
> > all states will
>
> > be removed in the affected clients). I'm not an expert on COPS, but 
> > I believe it implements such timers (should look before saying that,

> > but
> protocol machinery point of view, a protocol timer is required. Like 
> you indicated, we can't assume that a certain type of configuration 
> will be present on the peer side. With the possible presence of 
> intermediaries, e.g.
> relay agents, there is also no straightforward way of determining 
> crash of a server. Currently one may use an "UNABLE_TO_DELIVER" error 
> answer for a subsequent request sent to a specific server as an 
> indication that it went down/unreachable. Actually, maybe this is what

> we need, a message with which backups/cluster members can notify 
> clients about failure of a particular server.
>
[TOLGA]If it were a result code, it wouldn't differ at all :-) What I
think is a message, which can indicate failure of a particular host. Of
course it needs to be sent from another host and should include the
identity of the failed host in an AVP. Without such a message, a client
can be aware of a server failure only after it send a request with
Destination-Host AVP equal to the identity to the failed server (even
then, such an assumprtion would require that the network between client
and server is well engineered so that a problem in IP or Diameter
network won't cause lack of connectivity).
I guess if we want to provide premature state reconstruction on backup
servers, there has to be some sort of notification from this server to
the clients. I guess the overall problem can be split in two chunks:
a) How to notify clients
b) How to get the data necessary for state reconstruction from the
clients

[Ulf3] Ok. Now I understand what you're saying. And, I completely agree
to a). Regarding b) I would suggest to also consider the case when data
is replicated. This case would mean to extend b) to also allow for state
synchronization (e.g. by sharing a list of active sesisons to identify
potential discrepancies). 

> not available. Still, such a timer is also not without its worries if 
> it is completely decoupled from protocol processing. It must run 
> longer than the maximum expected life for a session, which could be 
> quite a while for certain types of applications (but again this is an 
> implementation issue)
>
> [Ulf2] Do you really think it must run longer than the maximum 
> expected life of a session (would be troublesome if the session 
> lengths distribution is heavy tailed). In my view it could just be 
> long enough to allow for defined actions to be excecuted (e.g. actions

> taken by the operationals).
[TOLGA]You may be right, one probably can come up with some clever way
of dealing with this problem. Initially I thought of a very simple
mechanism of deleting session which hang around for longer than
anticipated. In any case, this type of timer is probably implementation
dependent and people will come up with some mechanism depending on their
environment/the application type running etc...

[Ulf3] Yes, I agree. And, I certainly believe recommendations and even
some strics rules for how to use a timer (protocol or non-protocol)
detecting that the connection is lost would be good to have. I think
such rules should be related to the values of the authorization-timeout
and auth-grace-period rather than expected lifetime of sesisons though.

---

From: "Tolga Asveren" <[email-masked]>
Date: Mon, 22 Jan 2007 15:53:53 -0500

Hi Ulf, Hannes and all,

I feel like now we are ready for detailed investigation/documentation of the
problems/choices we discussed.

What would be a good way to proceed, producing drafts? If so, what would be
the content? To me it makes sense to start with a draft which explains
different choices, their applicability and how they handle different
functional aspects (notification of clients, transfer of session data, load
distribution etc...) Some of the methods described may require separate
drafts defining new messages/AVPs.

Another alternative is to continue discussing with e-mail messages on the
list but I think this started to get a little bit hard to follow and we may
loose track of issues discussed.

   Thanks,
   Tolga

---

From: Hannes Tschofenig <[email-masked]>
Date: Tue, 23 Jan 2007 20:35:00 +0100

Hi Tolga,

I went through the discussion thread and noticed that it is indeed a bit 
difficult to follow the individual issues.

Your mail entitled with "[Dime] The Auditing Problem: Possible 
approaches" http://www1.ietf.org/mail-archive/web/dime/current/msg01196.html
might be a good starting point for further descriptions.

I am interested in a description that tells me:
* How and who detects the failure of the server?
* What messages need to be exchanged to establish the state at the new 
server?
* What are the assumptions made with each proposal?

Let us consider other protocols with respect to their ability to deal 
with server failure, such as:

ftp://ftp.isi.edu/in-notes/rfc4507.txt
http://www.ietf.org/internet-drafts/draft-sheffer-ike-session-resumption-00.txt
http://www.ietf.org/internet-drafts/draft-vidya-ipsec-failover-ps-00.txt

The first two documents explain a procedure to quickly establish state 
at a new server after the old server failed based on an encrypted ticket 
that is sent to the client and presented at the new server during 
session setup.

Ciao
Hannes

---

From: "Tolga Asveren" <[email-masked]>
Date: Tue, 23 Jan 2007 17:11:47 -0500

Hi Hannes,

Let me try to summarize the discussion till now (or better said my
understanding of it) and try to answer your questions (Ulf, please feel free
to add, correct etc...)

>From functionality point of view, I see three different pieces (which
probbaly will answer your first two questions and possibly the third one to
some degree):

1- Notification of client about server failure
a) Relying on an application layer timer
The protocol specification mandates periodic messaging from client to server
and use of a timer for the maximum allowed time to wait for a corresponding
answer, e.g. Diameter Credit Control Application. If no answer is received
in time, the server is assumed as down. This is the soft state approach. For
this to work, there is no need to change anything in the base Diameter
infrastructure, application protocol authors need to define the periodic
messaging behavior.

b) Relying on receipt of an error answer
Here again we assume that the client is sending periodic requests during a
session. Because mid-session requests will have a Destination-Host AVP, if
an UNABLE_TO_DELIVER answer is received, this would be interpreted as the
server being down. Here, we assume that Diaemter network is properly
engineered, i.e. transport problems won't cause a message not being
delivered successfully. This aproach is similar to a), probably will be used
if Tw granularity on the message path is acceptable for the application.
Again, what is needed is that application protocol authors defining periodci
mid-session requests mechanism.

c) A notification from the backup server
When the backup server detects failure of an active server and is ready to
take over its responsibilities, it sends a notification to the clients, with
which the active server had ongoing sessions. The identity of such clients
needs to be synchronized between active and standby servers. Currently,
there is no generic message defined in Diameter for this purpose. Such a
message could be defined by applications or we can have a generic one.

2- Transfer of session state data
a) Using application messages
Mid-session application messages can be used to transfer the state data.
Applications can define an AVP to mark that certain mid-session request is a
"refresh" or this could be done with a separate mid-session application
message. The message should carry enough information to reconstruct session
state at the server. If a special application message is used, this could
carry session state for multiple sessions.

b) Using a generic message
A generic message can be defined to carry session data to the backup server.
In this approach, although the message itself could be generic, still each
application needs to define the content of the message so that it contains
enough information to recosntruct the state.

3- Backup Server Selection / Load Distribution
a) Backup information fed by the active server
Information about backup servers could be sent to clients by the active
server, when it was up and running. This information could indicate on a per
session basis, which backup server should be used for that particulat
session if the active server fails before the session terminates.
Alternatively, a list of backup servers with priorities could be indicated,
which is applicable for all sessions.

b) Acceptable load fed by backup server
If the backup server sendsa notification to the client ( approach c) for
issue 1) ), the notification may contain number of sessions, backup server
is willing to take over.

c) Trial and error
Client sends messages to backup server candidates. If the answer result code
is not SUCCESS, it tries another one.

Another important question seems to be, when does the need arise for servers
to trigger the session reconstruction mechanism rather than waiting for the
client to send a message for that purpose. AFAICS, we came up with two
different reasons for this:
a) The server needs to send autonomous requests to the client, e.g. for some
type of periodic reporting
b) The server needs to know about existing sessions at any given time to be
able to process new sessions requests properly, e.g. so that it can know
whether accepting a new reservation would exceed the available free
resources (I didn't think abouth this throughly but I feel like this can be
done just by relying on responses from the physical owner of resources
rather than relying on Diameter session state)

I checked briefly RFC4507 you mentioned. There it seems for 1- a) and b)
(depending on whether the host or the application crashes), for 2- a), for
3- c) is used.

What makes this problem interesting is that at the end it will be the
decision of the application protocol author what counts. What we can provide
is some guidelines about different protocol design choices and if we see it
necesary some generic infrastructure messages/AVPs.

     Thanks,
     Tolga

> Sent: Tuesday, January 23, 2007 2:35 PM
> To: Tolga Asveren
> Cc: [email-masked]
> Subject: Re: [Dime] The Auditing Problem: Possible approaches
>
>
> Hi Tolga,
>
> approaches"
> http://www1.ietf.org/mail-archive/web/dime/current/msg01196.html

---

From: "Ulf Bodin" <[email-masked]>
Date: Wed, 24 Jan 2007 15:14:23 +0100

Hi Tolga and Hannes, 

I agree to the summary as of below, but would like to add a few pieces: 
1) Prior to transferring session state data I believe for the case of
state replication that either part would benefit from a way to query the
other part for a list of active sessions (i.e. to synchronize the views
of clients and their server). This is useful e.g. to not unnecessarily
send state information for states that have been sucessfully replicated.

2) I think we need to explore client failover somewhat further. This
applies to some of the bullets below: 1.a no messages from a client for
for a period means its dead, 2.b and 2.c may apply if the server sends
autonomous requests to the client. 2.a don't apply since state
information don't normally travel in the direction from server to
client, 2.b may apply since a generic message could be defined for usage
in both directions. The 3.a, 3.b and 3.c approaches could be applied to
the client side. 

Also, just a quick comment to the session reconstruction and/or
replication issue. I agree to a) and for b) I would like to add that I
see the Diameter server as the physical owner of the mentioned
resources. I might miss something here, but in my view that means to
rely on Dimater sesison state. 

>From my perspective it might be a good time to gather information in a
draft as you suggest Hannes. I'm not completely sure exactly how sujch
draft would materialize. Explaining different choices, thri
applicability and how thay handle different functional aspects sounds
good to me (starting from the list below). 

Best regards, 
Ulf 

-----Original Message-----
From: Tolga Asveren [mailto:[email-masked]] 
Sent: den 23 januari 2007 23:12
To: [email-masked]
Subject: RE: [Dime] The Auditing Problem: Possible approaches

Hi Hannes,

Let me try to summarize the discussion till now (or better said my
understanding of it) and try to answer your questions (Ulf, please feel
free to add, correct etc...)

>From functionality point of view, I see three different pieces (which
probbaly will answer your first two questions and possibly the third one
to some degree):

1- Notification of client about server failure
a) Relying on an application layer timer The protocol specification
mandates periodic messaging from client to server and use of a timer for
the maximum allowed time to wait for a corresponding answer, e.g.
Diameter Credit Control Application. If no answer is received in time,
the server is assumed as down. This is the soft state approach. For this
to work, there is no need to change anything in the base Diameter
infrastructure, application protocol authors need to define the periodic
messaging behavior.

b) Relying on receipt of an error answer Here again we assume that the
client is sending periodic requests during a session. Because
mid-session requests will have a Destination-Host AVP, if an
UNABLE_TO_DELIVER answer is received, this would be interpreted as the
server being down. Here, we assume that Diaemter network is properly
engineered, i.e. transport problems won't cause a message not being
delivered successfully. This aproach is similar to a), probably will be
used if Tw granularity on the message path is acceptable for the
application.
Again, what is needed is that application protocol authors defining
periodci mid-session requests mechanism.

c) A notification from the backup server When the backup server detects
failure of an active server and is ready to take over its
responsibilities, it sends a notification to the clients, with which the
active server had ongoing sessions. The identity of such clients needs
to be synchronized between active and standby servers. Currently, there
is no generic message defined in Diameter for this purpose. Such a
message could be defined by applications or we can have a generic one.

2- Transfer of session state data
a) Using application messages
Mid-session application messages can be used to transfer the state data.
Applications can define an AVP to mark that certain mid-session request
is a "refresh" or this could be done with a separate mid-session
application message. The message should carry enough information to
reconstruct session state at the server. If a special application
message is used, this could carry session state for multiple sessions.

b) Using a generic message
A generic message can be defined to carry session data to the backup
server.
In this approach, although the message itself could be generic, still
each application needs to define the content of the message so that it
contains enough information to recosntruct the state.

3- Backup Server Selection / Load Distribution
a) Backup information fed by the active server Information about backup
servers could be sent to clients by the active server, when it was up
and running. This information could indicate on a per session basis,
which backup server should be used for that particulat session if the
active server fails before the session terminates.
Alternatively, a list of backup servers with priorities could be
indicated, which is applicable for all sessions.

b) Acceptable load fed by backup server
If the backup server sendsa notification to the client ( approach c) for
issue 1) ), the notification may contain number of sessions, backup
server is willing to take over.

c) Trial and error
Client sends messages to backup server candidates. If the answer result
code is not SUCCESS, it tries another one.

Another important question seems to be, when does the need arise for
servers to trigger the session reconstruction mechanism rather than
waiting for the client to send a message for that purpose. AFAICS, we
came up with two different reasons for this:
a) The server needs to send autonomous requests to the client, e.g. for
some type of periodic reporting
b) The server needs to know about existing sessions at any given time to
be able to process new sessions requests properly, e.g. so that it can
know whether accepting a new reservation would exceed the available free
resources (I didn't think abouth this throughly but I feel like this can
be done just by relying on responses from the physical owner of
resources rather than relying on Diameter session state)

I checked briefly RFC4507 you mentioned. There it seems for 1- a) and b)
(depending on whether the host or the application crashes), for 2- a),
for
3- c) is used.

What makes this problem interesting is that at the end it will be the
decision of the application protocol author what counts. What we can
provide is some guidelines about different protocol design choices and
if we see it necesary some generic infrastructure messages/AVPs.

     Thanks,
     Tolga

> I went through the discussion thread and noticed that it is indeed a 
> bit difficult to follow the individual issues.
>

> at a new server after the old server failed based on an encrypted 
> ticket that is sent to the client and presented at the new server 
> during session setup.
>
> Ciao
> Hannes
>
> Tolga Asveren wrote:
> > Hi Ulf, Hannes and all,
> >
> > the content? To me it makes sense to start with a draft which 
> > explains different choices, their applicability and how they handle 
> > different functional aspects (notification of clients, transfer of
> session data, load
> > distribution etc...) Some of the methods described may require 
> > separate drafts defining new messages/AVPs.
> >
> >> essentially the soft state case, where the client can send a 
> >> message to an alternate server after expiry of a protocol timer 
> >> with enough information embedded in the message, so that the 
> >> alternate server can process it.
> >>
> >> [Ulf2] Yes.
> >>
"UNABLE_TO_DELIVER"
> >> session states associated with the target that could not be 
> >> reached, or just the session for which the error was given?
> >>
> >> iii- Peer triggered state data transfer upon receipt of a 
> >> notification from a peer This is the approach described in the 
> >> rebuild the state. A notification message is sent from a peer 
> >> informing the client that a server went down. The client refreshes
> sessions on the
> >> alternate server. The message can be sent immediately after 
> >> receiving the notification or could be delayed till the 
> >> corresponding session(s) actually require a message to be sent to
update/delete.
> >> are to be refreshed). In my view some recommendations on how to 
> >> control the rate of messages and/or to make it driven by a receiver

> >> of the updates would be needed. I might be jumping ahead too quick 
> >> to the solution now though.
> >>
> >> i and ii seem to work with existing Diameter wisdom, but they may 
> >> run into issues, if servers also generate autonomous requests to 
> >> clients (actually this would probably answer the question I asked 
> >> in a previous message about the necessity of resurrection 
> >> sessions/states on backup servers immediately after they take over 
> >> the role of a failed host)
> >>
> >> [Ulf2] Yes, autonomous requests/unsolicited call backs is a reason 
> >> for resurrection sessions/states on backup servers. I believe an 
> >> additional reason for why states should for some applications be 
> >> resurrected is that the application are controlling access to
scarce resources (e.g.
> >> RACS).
> >>
> >> For iii and iv, a backup peer needs to know which clients it needs 
> >> to send a notification to. This probably can be synchronized 
> >> between peers of a cluster.
> >>
> >> [Ulf2] Yes.
> >>
> >> It could be useful to think over different alternatives considering

> >> the RACS scenario. For RACS, which interface are we talking about,
> Gq', Rq ,
> >> any or both?
> >>

> >> part of RACS performing admission control) will replicate both 
> >> session and reservation states. The A-RACF is likely to offer
> soft-state only to
> >> SIP proxies). The SPDF will probably also replicate states and 
> >> offer soft-state to its clients. The SPDF is however likely to also

> >> offer hard-state to some application frameworks (e.g. leased line 
> >> application level servers).
> >>
> >> information in refresh messages to rebuld states. In addition, iii 
> >> will be used to ensure syncronisation after failover (and 
> >> potentially periodically between application and SPDF in the case
of hard-state).

> >> approach correctly it applies to non cases when data is not 
> >> replicated only). However, for a deployment without replication for
e.g.
> the A-RACF

---

From: Hannes Tschofenig <[email-masked]>
Date: Wed, 24 Jan 2007 19:54:31 +0100

Hi Tolga,

thanks for the nice summary. Please find my response below:

> infrastructure, application protocol authors need to define the periodic
Ok

> mid-session requests mechanism.
This approach assumes that there is an intermediate node that provides 
the error message or that the error message is offered from a lower layer.

> message could be defined by applications or we can have a generic one.
With this approach the backup server at least has to know Session IDs 
and Diameter client identifiers.

It would also be necessary to determine (on a per-application basis) 
which state the Diameter client would sent back to the Diameter server.
The state that is kept at the server and at the client might be different.

I could also imagine that there are security issues: The Diameter client 
might not know the backup server and hence any arbitrary server could 
just retrieve state from the client.

> Mid-session application messages can be used to transfer the state data
>   

I am not sure why you mean that it would be a mid-session message?
Who would sent the message?

> enough information to recosntruct the state.
I think that there are three possible modes:

(*) State is sent from the Diameter server to the Diameter client and in 
case of a failure from the Diameter client to the Diameter backup server.
This is similar to what is done with RFC 4507. This approach does not 
require state to be synchronized between the backup and the original 
Diameter server.

(**)  The backup server asks the Diameter client for a state transfer 
from the client to the backup server. This obviously requires state 
synchronization between the two Diameter servers to let the backup 
server to know the identity of the Diameter client.

(***) The Diameter client knows the backup server and contacts it when 
the main server is not reachable anymore.

> which is applicable for all sessions.
Agree.

> issue 1) ), the notification may contain number of sessions, backup server
True.

> (depending on whether the host or the application crashes), for 2- a), for
It would probably be something like 1a or 1b but the details are not 
mentioned.

Not sure about (2) since I have some questions.

Ciao
Hannes

>>> I feel like now we are ready for detailed
>>>       
>> investigation/documentation of the
>>     
>>> What would be a good way to proceed, producing drafts? If so,
>>>       
>> what would be
>>     
>>> functional aspects (notification of clients, transfer of
>>>       
>> session data, load
>>     
>>> Another alternative is to continue discussing with e-mail
>>>       
>> messages on the
>>     
>>> list but I think this started to get a little bit hard to
>>>       
>> follow and we may
>>     
>>>> i- Client triggered refresh upon expiry of a timer on the
>>>>         
>> client This is
>>     
>>>> the client that a server went down. The client refreshes
>>>>         
>> sessions on the
>>     
>>>> [Ulf2] Ok. What come to my mind though is the amount of
>>>>         
>> messages that'll
>>     
>>>> be generated (this concern applies also to ii in case all
>>>>         
>> session states
>>     
>>>> RACS scenario. For RACS, which interface are we talking about,
>>>>         
>> Gq', Rq ,
>>     
>>>> any or both?
>>>>
>>>> remains to be seen. My personal belief is that most A-RACF
>>>>         
>> entities (the
>>     
>>>> and reservation states. The A-RACF is likely to offer
>>>>         
>> soft-state only to
>>     
>>>> only). However, for a deployment without replication for e.g.
>>>>         
>> the A-RACF
>>     
>>>> iv would however apply.
>>>>
>>>>
>>>>

---

From: Hannes Tschofenig <[email-masked]>
Date: Wed, 24 Jan 2007 20:00:18 +0100

Hi Ulf,

> send state information for states that have been sucessfully replicated.
Wouldn't be the right way to ensure that you are only authorized to 
retrieve the state that you previously created.

> rely on Dimater sesison state. 
I agree that we also have to consider client failover. At the moment we 
are still focusing on the server failover case.
The two might be a bit different.

> good to me (starting from the list below). 

I have one question for you: Do you consider a stateless solution at the 
backup server, i.e., where the backup server does not ever store Session 
Ids or the identities of the Diameter clients?

Ciao
Hannes

>>>> the SPDF (the part of RACS facing application frameworks such as
>>>>         
> e.g.
>

---

From: "Tolga Asveren" <[email-masked]>
Date: Wed, 24 Jan 2007 17:39:52 -0500

Hi Hannes,

Please see below for questions/comments.

    Thanks,
    Tolga

> Sent: Wednesday, January 24, 2007 1:55 PM
> To: Tolga Asveren
> Cc: [email-masked]
[TOLGA]I guess the need for the knowledge of Session-Id may be avoided.
Diameter client would probably be able to determine sessions to be
transferred to the backup server based on the identity of the failed active
server.
> The state that is kept at the server and at the client might be different.
[TOLGA]This is one of the key issues. What comprises the state should be
defined by the protocol specification of the corresponding application and
servers should be able to reconstruct a working context based on that.
> just retrieve state from the client.
[TOLGA]Maybe that active server provides a list of backup candidates could
help.
> Who would sent the message?
[TOLGA]I was referring to the case that a mid-session message is sent from
client to server. Let's asume the client is sending periodic requests to the
server during the session. If an answer is not received for such a request
in a predetermined amount of time, the request could be resent to an
alternate server. Alternatively, the client could start a new session to a
new server for the saem user service context.
> message to
> Similar to
> reached, or
> notification
> replicated

---

From: "Ulf Bodin" <[email-masked]>
Date: Thu, 25 Jan 2007 08:56:43 +0100

-----Original Message-----
From: Hannes Tschofenig [mailto:[email-masked]] 
Sent: den 24 januari 2007 20:00
To: Ulf Bodin
Cc: [email-masked]
Subject: Re: [Dime] The Auditing Problem: Possible approaches

Hi Ulf,

Ulf Bodin wrote:
> state replication that either part would benefit from a way to query 
> the other part for a list of active sessions (i.e. to synchronize the 
> views of clients and their server). This is useful e.g. to not 
> unnecessarily send state information for states that have been
sucessfully replicated.

[Ulf] Good point. A query from a client should not result in that the
reserver returns a list of all its active sessions. It should only
return those sessions installed by the querying client.  

> applies to some of the bullets below: 1.a no messages from a client 
> for for a period means its dead, 2.b and 2.c may apply if the server 
> sends autonomous requests to the client. 2.a don't apply since state 
> client, 2.b may apply since a generic message could be defined for 
> usage in both directions. The 3.a, 3.b and 3.c approaches could be 
> applied to the client side.
>

[Ulf] Ok. I try to be a litte more patient :-). 

>> >From my perspective it might be a good time to gather information in

>> >a

[Ulf] Yes, I see that as an option. In my view list item 2 in Tolga's
list can apply to both the case of when the backup server is completely
stateless and when it has access to replicated session data. In both
cases I agree to that we need a method to differentiate between
mid-session updates and initial messages creating a new session. In both
Gq (3GPP) and Gq' (TISPAN) mid-session messages are separated from
initial message by known and unknown session ids only. A flag making
this separation or a separate messages is clearly needed (generic or
application specific). 

Ciao
Hannes

> understanding of it) and try to answer your questions (Ulf, please 
> feel free to add, correct etc...)
>
>   
> probbaly will answer your first two questions and possibly the third 
> one to some degree):
>
> mandates periodic messaging from client to server and use of a timer 
> for the maximum allowed time to wait for a corresponding answer, e.g.

> the server is assumed as down. This is the soft state approach. For 

> infrastructure, application protocol authors need to define the 
> periodic messaging behavior.
>

> delivered successfully. This aproach is similar to a), probably will 
> be used if Tw granularity on the message path is acceptable for the 
> application.
> c) A notification from the backup server When the backup server 
> detects failure of an active server and is ready to take over its 
> responsibilities, it sends a notification to the clients, with which 
> the active server had ongoing sessions. The identity of such clients 
> needs to be synchronized between active and standby servers. 
> Currently, there is no generic message defined in Diameter for this 
> purpose. Such a message could be defined by applications or we can
have a generic one.
> Applications can define an AVP to mark that certain mid-session 
> request is a "refresh" or this could be done with a separate 
> mid-session application message. The message should carry enough 
> information to reconstruct session state at the server. If a special 
> application message is used, this could carry session state for
multiple sessions.

> a) Backup information fed by the active server Information about 
> backup servers could be sent to clients by the active server, when it 
> was up and running. This information could indicate on a per session 
> basis, which backup server should be used for that particulat session 
> if the active server fails before the session terminates.
> b) Acceptable load fed by backup server If the backup server sendsa 
> notification to the client ( approach c) for issue 1) ), the 
> notification may contain number of sessions, backup server is willing 
> to take over.
>
> c) Trial and error
> Client sends messages to backup server candidates. If the answer 
> result code is not SUCCESS, it tries another one.
>
>
> a) The server needs to send autonomous requests to the client, e.g. 
> for some type of periodic reporting
> b) The server needs to know about existing sessions at any given time 
> to be able to process new sessions requests properly, e.g. so that it 
> can know whether accepting a new reservation would exceed the 
> available free resources (I didn't think abouth this throughly but I 
> feel like this can be done just by relying on responses from the 
> physical owner of resources rather than relying on Diameter session 
> state)
>
> I checked briefly RFC4507 you mentioned. There it seems for 1- a) and 
> b) (depending on whether the host or the application crashes), for 2- 
> a), for
> 3- c) is used.
>

>> http://www.ietf.org/internet-drafts/draft-vidya-ipsec-failover-ps-00.
>> t
>> xt
>>
>>