Subject: RE: [AAA-WG]: issue: real-time duplicate detection and proposedchange.

From: <[email-masked]>
Date: Tue, 15 Oct 2002 21:17:46 +0300

Hi all,

> optional != useful. I'd argue mandatory on clients, but optionally
> supported on servers.
> 
> PatC
> On Tue, 2002-10-15 at 16:57, Bernard Aboba wrote:
> > > So anything that is retransmitted due to a lack of an application level
> > > ack/nak gets the 'D' bit set.... If that's all it is ..seems pretty
> > > reasonable to me.
> > > 
> > > Could make it optional but require the 'D' bit always set 
> > > for servers not supporting the hint.
> > 
> > Yes, that's a good point. Do all Diameter implementations 
> > need to support this? I think that's the proposal in Issue 372.

This is a good compromise.  Could we like with this?

John

---

From: Yoshihiro Ohba <[email-masked]>
Date: Tue, 15 Oct 2002 15:08:10 -0400

Making it mandatory for client only might be better than the original 
proposal.

See my additional comments.

The End-to-End Identifier in the header (plus Origin-Host AVP can
be also used for duplicate detection, except for the case in which
a duplicate accounting record is sent from a rebooted Diameter
client with a new End-to-End Identifier.  This means that there
are basically two cases for duplications: (a) duplication due to a
change in diameter message routing, and (b) duplication due to
rebooting at a Diameter client.  The original D-bit proposal seems 
to deal with those two cases equally without distinction.  

Making it mandatory for client only is actually considering 
separation of the two cases, where case (b) is can be dealt 
with D-bit set by clients and case (a) can be dealt with E-E ID and 
Origin-Host AVP.  I think this is a better direction to go.

On the other hand, we can still argue for defining a new AVP (e.g.,
Duplication-Indicatation AVP) instead of defining D-bit in the
Diameter header, because case (b) is not common to all applications
and I think Diameter header should contain information only common to
all applications.  The new AVP is inserted only by a client which has
experianced rebooting and have stored unACK'ed accounting records in a
non-volatile memory.

Yoshihiro Ohba

---

From: Pat Calhoun <[email-masked]>
Date: 15 Oct 2002 12:22:27 +0000

A new AVP may be a better approach, but now requires that the
application be involved in the message retransmission engine... not a
great idea. However, your point on end-to-end made me think that if an
intermediate agent retransmits the packet, it must set the 'D' bit, and
this could break e2e packet hash/signature.

PatC

---

From: Yoshihiro Ohba <[email-masked]>
Date: Tue, 15 Oct 2002 15:49:55 -0400

On Tue, Oct 15, 2002 at 12:22:27PM +0000, Pat Calhoun wrote:
> great idea. 

I think retransmission after rebooting based on stored 
unacknowlegded accounting records is a task of the application 
(the diameter engine should process them as new requests for which 
new End-to-End Identifiers are to be assigned), while retransmission
after _normal_ failover/failback is a task of the retransmission
engine.  Better to make a distinction between those two cases.

> However, your point on end-to-end made me think that if an

By dealing the two duplication cases separately, intermediate agent
does not have to set 'D'-bit or insert a new AVP.

Yoshihiro Ohba

---

From: Pat Calhoun <[email-masked]>
Date: 15 Oct 2002 12:58:48 +0000

> engine.  Better to make a distinction between those two cases.
fair enough

> does not have to set 'D'-bit or insert a new AVP.

Not sure I completely understand. Since each intermediate hop is
responsible for ack and retransmission, how could it get away with not
setting the 'D' bit (or the new AVP for that matter)?

PatC

---

From: Yoshihiro Ohba <[email-masked]>
Date: Tue, 15 Oct 2002 16:27:53 -0400

> > engine.  Better to make a distinction between those two cases.

OK.

> setting the 'D' bit (or the new AVP for that matter)?

That part (i.e., duplicated requests based on normal
failover/failback) can be detected based solely on the combination of
End-to-End Identifier and Origin-Host AVP (perhaps with using either
time-based window or hash table at Diameter servers, but without 
D-bit in the header).

Yoshihiro Ohba

---

From: Pat Calhoun <[email-masked]>
Date: 15 Oct 2002 13:50:54 +0000

> time-based window or hash table at Diameter servers, but without 

But the comment made earlier is that this scheme doesn't scale when
dealing with thousands of messages per minute. I believe you are still
arguing the fact that the 'D' bit is really unnecessary, correct?

PatC

---

From: Jari Arkko <[email-masked]>
Date: Wed, 16 Oct 2002 00:15:29 +0300

> time-based window or hash table at Diameter servers, but without 

*All* duplication can be detected based on server implementations.
Some known methods:
- full search every time through a db index
- hash table based methods
- keeping the last session ids for a user in the user "account" object at the
  server
- heuristic methods, such as tracking correct order and checking
  duplicates only if we appear to get out-of-order records
The current debate is about the speed and efficiency of various
ways of doing this, and whether the D-bit helps the server in making
the duplicate-or-not decision.

There's two kinds of duplicate detection, based on end-to-end id and
session id, and based on the session-id and accounting-record-number.
Same techniques can be applied to both, however.

I believe that if we are to adopt D-bit, then it should be informative,
and its setting be made mandatory for clients and agents, and use optional
for servers. Question: for all apps, or just for the one that needs
this?

All duplicate detection should also cover the following cases:
- Regular failover duplicate from a client.
- Failover from an agent to the server. And here we need
  to worry about invalidating any signatures; cmssec isn't
  done yet but I think we need to sign not just the AVPs
  but also include the command code and the bits.
- Reboot and then resend acct records from nvm. This needs
  care with the d-bit, since the rebooted server must not
  send d=0 records as they may have been already sent.

In general, methods based on protocol information (such as the d-bit)
provide duplicate detection that relies on the client and agents
to perform it correctly. Implementation and configuration problems,
for instance, can't be fixed on the server side in this case. It is
debatable, however, whether this is a requirement. (Server-only approaches
to duplicate detection do not rely on the other nodes.)

Jari

---

From: Yoshihiro Ohba <[email-masked]>
Date: Tue, 15 Oct 2002 17:22:56 -0400

> arguing the fact that the 'D' bit is really unnecessary, correct?

Might be.  As far as I understand, D-bit scheme still needs time-based
window.  If there is an alternative that will(can) also use time-based
window but does not require a new flag in the Diameter header, the
alternative should be considered _unless_ the alternative requires
much more time window than the other.

Introducing a new flag in the common header should be the last resort.

Yoshihiro Ohba