Subject: RE: [AAA-WG]: issue: real-time duplicate detection

From: <[email-masked]>
Date: Mon, 14 Oct 2002 10:20:42 +0300

Hi Bernard;

In general, when discussing with you, I can say that you are
very good in turning up side down the discussions that I initiate.

Busy people receiving hundred of e-mail per day, as high-tech people are, 
usually don't follow the full mail thread.
Instead they read the last received mail, especially if the last mail  is an 
answer of the WG chair. If I say  'You are stupid if you think that Saddam doesn't 
have nuclear army' and you comment only  the statement 'You are stupid' , the 
readers will simply conclude that I say 'You are stupid' and not only they'll disagree
with me but also they'll get very bad feeling about me.
This is a good way to mess up discussions and also unfair in the respect of 
the individual. So , I invite you to comment full stories rather than some 
incomplete statement picked up at your convenience.

You wrote

>You may need to understand better how this work.

BAb: "Since you noted yourself that the proposal does not work at all
when Diameter agents are present, I don't think we're in disagreement."

MSt: I noted my self that Re-transmission AVP approach doesn't work
when Diameter agents are present. Unfortunately the D-bit approach it
does work....and my sentence was referring to that.
Also it seems that you don't object anymore that the same technique is
used in several operational GPRS networks with millions of subscribers and 
it works.

>I guess these kind of failure are handled by the transport protocol
>(i.e. SCTP/TCP). Reliable transport protocols such as SCTP/TCP insure
>that messages sent by application are received at the other end of the
>wire (i.e. the associated peer).

BAb: "Actually, they *don't* guarantee this -- that's the problem. If there is a
network partition, TCP/SCTP or any other transport protocol can't get
through. As Marshall Rose once said "you don't need reliable transport,
you need more voltage!"

All a TCP acknowledgment tells you is that the segment was received by the
TCP on the destination -- it doesn't tell you that the application at the
destination received it.

So in practice to know that the application received the data, and that
it was processed correctly, you need an application layer ACK."

MSt: again you should have answered to the whole paragraph. What I say is 
that transport layer (SCTP, TCP) never deliver the message to the 
application layer before all the segments or data chunks have not been 
succesfully received. And here is my full story again.

"I guess these kind of failure are handled by the transport protocol
(i.e. SCTP/TCP). Reliable transport protocols such as SCTP/TCP insure
that messages sent by application are received at the other end of the
wire (i.e. the associated peer). Only re-transmission of data chunks or
message's bytes occur, this is transparent to the application layer that
just get the message when the transport layer has successfully received
all the bytes of the message. No message duplicate occur in this
case."

No application layer message duplicate occur in this case. And that I need 
an application ACK to know that the other Diameter peer received the 
message, well I was not waiting for your lesson to learn that you know...

I'm very surprised that you accept a broad statement as the one in AnnexC
"Secondly, hashing techniques or other schemes may be used to eliminate the 
need to do a full search even in this set except for rare cases."
What does it mean hashing techniques in the context of  real-time duplicate 
detection? does it mean that the full disk database for record storage is built up 
using hashing techniques ? or does it mean that a hashing table to keep track of the received 
Session-Ids within e.g one day time window is maintained in RAM memory ?
In the latter case is it so that you perform Key== hash(Session-id), you set e.g . 
'True' in the bucket identified by the key and you store the record in the database 
and  If the value stored in the bucket was already 'True' you perform disk database search 
for the possible duplicate ?
Well a broad statement that refer to 'hashing technique' does not suffice in 
a technical specification .  You should also recommend the hashing function to be used 
to minimize collision cases, with the relevant arguments, the proper hashing algorithm 
as well as the proper criteria to select the table size etc. and don't say that you can 
find the proper hashing function to be applied to Session-id from the literature,  this 
would sound like a political answer to a technical problem (i.e. not a solution).

I don't know what it could be the hashing function but let's try to analyze what size 
the hashing table might be considering the multimedia messaging use case. 
Currently in GSM networks people send in average 1 SMS per hour.
If your server need to handle 4 millions of prepaid subscribers it means that in 24
hours it will get 4*24=96 millions requests. If we consider a Key of 32-bits(4 bytes)
the hashing table should be 96*4= around 370MB. Now if we want to limit the risk of collision
we may double the size to reduce to load factor 370*2=740MB. But I just considered one 
service, if we consider other services as well we can safely assume 2 or even 3 requests
per hour per subscriber.
Just with 2 requests per hour the size would double to 740*2= aroun 1.5GB in addition to the
disk database.
Well, to me it sounds quite huge amount of memory.

Moreover, the hashing approach to address real-time duplicate detection 
hasn't been tried in practise and hasn't been evaluated in full  so I don't 
see any good technical reason why the broad statement referring to hashing techniques 
is in and a proven to work and widely deployed technique for duplicate detection  that 
falls within 'other schemes' category is out a priori.
Since the Diameter protocol is gonna be used as accounting protocol to deploy real-time prepaid 
services ,I just would insure that the protocol offers the required tool to properly 
perform simple real-time duplicate detection without placing huge memory amount requirement to
implementation. 
In conclusion my proposal is then to include the Issue 372 proposed changes to the base protocol 
specification (base 14), which in light of the above adding D flag approach in addition to
hushing techniques is just question of common sense nothing more nothing less.

With my best regards
Marco

> -----Original Message-----
> From: ext Bernard Aboba [mailto:[email-masked]]
> Sent: 11. October 2002 19:04
> To: Stura Marco (NET/Helsinki)
> Cc: [email-masked]
> Subject: RE: [AAA-WG]: issue: real-time duplicate detection
> 
> 
> > You may need to understand better how this work.
> 
> Since you noted yourself that the proposal does not work at all
> when Diameter agents are present, I don't think we're in disagreement.
> 
> > I guess these kind of failure are handled by the transport protocol
> > (i.e. SCTP/TCP). Reliable transport protocols such as 
> SCTP/TCP insure
> > that messages sent by application are received at the other 
> end of the
> > wire (i.e. the associated peer).
> 
> Actually, they *don't* guarantee this -- that's the problem. 
> If there is a
> network partition, TCP/SCTP or any other transport protocol can't get
> through. As Marshall Rose once said "you don't need reliable 
> transport,
> you need more voltage!"
> 
> All a TCP acknowledgment tells you is that the segment was 
> received by the
> TCP on the destination -- it doesn't tell you that the 
> application at the
> destination received it.
> 
> So in practice to know that the application received the 
> data, and that
> it was processed correctly, you need an application layer ACK.
> 
> > Considering the D-bit approach this simply means that D=1 may arrive
> > before D=0. The solution for that is in the proposal I'm gonna send.
> 
> Looking forward to it.
> 
>