Subject: [Dime] Proposed Example Text for draft-docdt-dime-ovli-01

From: Ben Campbell <[email-masked]>
Date: Mon, 4 Nov 2013 16:36:41 -0800

Hi,

draft-docdt-dime-ovli-01 has a TBD item in appendix C, section 3. Namely,  an example showing a mix of Destination-Realm and Destination-Host routed requests. Here's a straw man proposal for that example. I don't expect this to be the "one true way" to approach the scenario; rather, it's a possible way to do it, and there are likely other ways as well.

Comments solicited.

Thanks!

Ben.

----------------------------

C.3.  Mix of Destination-Realm routed requests and Destination-Host
      reouted requests

   Diameter allows a client to optionally select the destination server
   of a request, even if there are agents between the client and the
   server.  The client does this using the Destination-Host AVP.  In
   cases where the client does not care if a specific server receives
   the request, it can omit Destination-Host and route the request using
   the Destination-Realm and ApplicationId AVPs, effectively letting an
   agent select the server.

   Clients commonly send mixtures of Destination-Host and Destination-
   Realm routed requests.  For example, in an application that uses user
   sessions, a client typically won't care which server handles a
   session-initiating requests.  But once the session is initiated, the
   client will send all subsequent requests in that session to the same
   server.  Therefore it would send the initial request with no
   Destination-Host AVP.  If it receives a successful answer, the client
   would copy the Origin-Host value from the answer message into a
   Destination-Host AVP in each subsequent request in the session.

   An agent has very limited options in applying overload abatement to
   requests that contain Destination-Host AVPs.  It typically cannot
   route the request to a different server than the one identified in
   Destination-Host.  It's only remaining options are to throttle such
   requests locally, or to send an overload report back towards the
   client so the client can throttle the requests.  The second choice is
   usually more efficient, since it prevents any throttled requests from
   being sent in the first place, and removes the agent's need to send
   errors back to the client for each dropped request.

   On the other hand, an agent has much more leeway to apply overload
   abatement for requests that do not contain Destination-Host AVPs.  If
   the agent has multiple servers in its peer table for the given realm
   and application, it can route such requests to other, less overloaded
   servers.

   If the overload severity increases, the agent may reach a point where
   there is not sufficient capacity across all servers to handle even

Korhonen, et al.          Expires May 03, 2014                 [Page 35]
Internet-Draft                    DOIC                      October 2013

   realm-routed requests.  In this case, the realm itself can be
   considered overloaded.  The agent may need the client to throttle
   realm-routed requests in addition to Destination-Host routed
   requests.  The overload severity may be different for each server,
   and the severity for the realm at is likely to be different than for
   any specific server.  Therefore, an agent may need to forward, or
   originate, multiple overload reports with differing ReportType and
   Reduction-Percentage values.

   Figure 8 illustrates such a mixed-routing scenario.  In this example,
   the servers S1, S2, and S3 handle requests for the realm "realm".
   Any of the three can handle requests that are not part of a user
   session (i.e. routed by Destination-Realm).  But once a session is
   established, all requests in that session must go to the same server.

                  Client     Agent      S1        S2        S3
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |(1) Request (DR:realm)       |         |
                     |-------->|         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |Agent selects S1   |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |(2) Request (DR:realm)       |
                     |         |-------->|         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |S1 overloaded, returns OLR
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |(3) Answer (OR:realm,OH:S1,OLR:RT=DH)
                     |         |<--------|         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |sees OLR,routes DR traffic to S2&S3
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |(4) Answer (OR:realm,OH:S1, OLR:RT=DH) |
                     |<--------|         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |Client throttles requests with DH:S1   |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |(5) Request (DR:realm)       |         |
                     |-------->|         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |Agent selects S2   |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |(6) Request (DR:realm)       |
                     |         |------------------>|         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |S2 is overloaded...
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |(7) Answer (OH:S2, OLR:RT=DH)|
                     |         |<------------------|         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |Agent sees OLR, realm now overloaded
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |(8) Answer (OR:realm,OH:S2, OLR:RT=DH, OLR: RT=R)
                     |<--------|         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |Client throttles DH:S1, DH:S2, and DR:realm
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |
                     |         |         |         |         |

      Figure 8: Mix of Destination-Host and Destination-Realm Routed
                                 Requests

   1: The client sends a request with no Destination-Host AVP (that is,
      a Destination-Realm routed request.)

   2: The agent follows local policy to select a server from its peer
      table.  In this case, the agent selects S2 and forwards the
      request.

   3: S1 is overloaded.  It sends a answer indicating success, but also
      includes an overload report.  Since the overload report only
      applies to S1, the ReportType is "Destination-Host".

   4: The agent sees the overload report, and records that S1 is
      overloaded by the value in the Reduction-Percentage AVP.  It
      begins diverting the indicated percentage of realm-routed traffic
      from S1 to S2 and S3.  Since it can't divert Destination-Host
      routed traffic, it forwards the overload report to the client.
      This effectively delegates the throttling of traffic with
      Destination-Host:S1 to the client.

   5: The client sends another Destination-Realm routed request.

   6: The agent selects S2, and forwards the request.

   7: It turns out that S2 is also overloaded, perhaps due to all that
      traffic it took over for S1.  S2 returns an successful answer
      containing an overload report.  Since this report only applies to
      S2, the ReportType is "Destination-Host".

   8: The agent sees that S2 is also overloaded by the value in
      Reduction-Percentage.  This value is probably different than the
      value from S1's report.  The agent diverts the remaining traffic
      to S3 as best as it can, but it calculates that the remaining
      capacity across all three servers is no longer sufficient to
      handle all of the realm-routed traffic.  This means the realm
      itself is overloaded.  The realm's overload percentage is most
      likely different than that for either S1 or S2.  The agent
      forward's S2's report back to the client in the Diameter answer.
      Additionally, the agent generates a new report for the realm of
      "realm", and inserts that report into the answer.  The client
      throttles requests with Destination-Host:S1 at one rate, requests
      with Destination-Host:S2 at another rate, and requests with no
      Destination-Host AVP at yet a third rate.  (Since S3 has not
      indicated overload, the client does not throttle requests with
      Destination-Host:S3.)

---

From: Ben Campbell <[email-masked]>
Date: Tue, 26 Nov 2013 14:20:47 -0600

[I've received a number of in-person comments on this, but non on the list. Given that I will never remember everything people said to me when traveling, I'd appreciate it if people would repeat their comments on list. Failing to do so risks me describing your argument incorrectly; resulting in an unintentional strawman. ]

That being said, if I understand the comments correctly, there are a few major open questions here:

1) Is this a real problem?

Several people argued to me that, if the agent load intelligently load balances traffic across all the servers, then the servers should become overloaded roughly at the same time. OTOH, certain sessions may have much more traffic than other sessions. If these "big" sessions are distributed proportionately across the servers, load should still be fairly balanced.

IMHO, we cannot assume smart load balancing, as there is no standard for doing it. Our solution should still work for agents that use simple (and perhaps naive) load distributions strategies. But even if we do have super-smart load balancing, things like hardware failures can throw things out of balance. Also, the argument that "big" sessions should still be proportionately distributed only works if you have a large enough sample of "big" sessions to distribute, compared to the number of servers. That may be true most of the time, but I don't think we want a solution that fails if it's not true.

2)  Lionel argued that OLRs are best used to describe overload for the realm as a whole. Overload for specific nodes would be better handled by sending Diameter error codes, either by fixing one or more existing error codes to have the correct semantics, or introducing new ones. If we did this, we would not need different report types.

My opinion is that I would like a consistent way of reporting overload, that is, I don't like using OLRs for one kind of overload, and error result codes for another. In particular, I would like to be able to report overload before reaching the point that I need to fail a transaction, i.e. I would like to be able to report any kind of overload in a Diameter answer message with a result code of DIAMETER_SUCCESS.

3) Are we allowed to put more than one OLRs in a single answer, as my example shows in step 8?

It might be possible to construct an example where you never see more than one OLR in a single answer. But I don't see what purpose would be served by such a limitation, as long as multiple OLRs do not contradict each other. Since the reports in the example have different report types, there is no conflict. On disadvantage of _not_ allowing multiple reports in one answer is that, if the servers choose to send reports in every answer, life gets complicated for the agent when trying to find a place to put the "realm" report.  It either needs to strip the server reports (which is hard given that the server overload conditions are best handled by the clients.) Or it needs to originate its own answers, which means forcing the failure of at least some transactions.

On Nov 4, 2013, at 6:36 PM, Ben Campbell <[email-masked]> wrote:

> _______________________________________________
> DiME mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dime

---

From: Jouni Korhonen <[email-masked]>
Date: Wed, 27 Nov 2013 13:59:17 +0200

> My opinion is that I would like a consistent way of reporting overload, that is, I don't like using OLRs for one kind of overload, and error result codes for another. In particular, I would like to…

I am towards Ben's conclusion here.

> It might be possible to construct an example where you never see more than one OLR in a single answer. But I don't see what purpose would be served by such a limitation, as long as multiple OLRs do…

My recollection was that we want to get rid off the ReportType in the baseline. That would also imply a single OLR in an answer. I might remember wrong, though.

Anyway, just to be on the safe side the current draft-ietf-*-01 still has the ReportType.

Could we conclude this point asap? Removing the ReportType has implications in multiple places.

- Jouni

---

From: Ben Campbell <[email-masked]>
Date: Wed, 27 Nov 2013 09:22:46 -0600

> Could we conclude this point asap? Removing the ReportType has implications in multiple places.

I think we need to keep report type.

---

From: "Wiehe, Ulrich (NSN - DE/Munich)" <[email-masked]>
Date: Wed, 27 Nov 2013 15:33:10 +0000

Jouni,

with regard to 2) I agree with Ben and you.

For 3) I find it beneficial to distinguish between a) an Overload Report that requests a traffic reduction for traffic destined to a specific Host and b) an Overload Report that requests a traffic reduction for traffic to (an unspecified Host within) a specific Realm. 
It may however be possible to implicitly derive the ReportType (Host or Realm) from the presence/absence of a Destination Host in the corresponding request message. That means: only one OLR in an anwer, no explicit ReportType in the OLR. I think this proposal is more inline with the end-point concept.

Ulrich

-----Original Message-----
From: DiME [mailto:[email-masked]] On Behalf Of ext Jouni Korhonen
Sent: Wednesday, November 27, 2013 12:59 PM
To: Ben Campbell; [email-masked]
Subject: Re: [Dime] Proposed Example Text for draft-docdt-dime-ovli-01

- Jouni

---

From: Ben Campbell <[email-masked]>
Date: Wed, 27 Nov 2013 14:44:06 -0600

> It may however be possible to implicitly derive the ReportType (Host or Realm) from the presence/absence of a Destination Host in the corresponding request message. That means: only one OLR in an a…

Why is that beneficial? It means that the responding node has to refer back to the original request to determine the meaning of an OLR in an answer. That creates a lot more work and implementation complexity than you would have if we put the ReportType in the OLR itself. it also puts unnecessary constraints on when a reporting node can send the ORL in the first place.

---

From: Jouni Korhonen <[email-masked]>
Date: Wed, 27 Nov 2013 22:50:08 +0200

> Why is that beneficial? It means that the responding node has to refer back to the original request to determine the meaning of an OLR in an answer. That creates a lot more work and implementation…

Good point.

- Jouni

---

From: <[email-masked]>
Date: Thu, 28 Nov 2013 08:22:34 +0000

I could discuss the need for the report type for a longtime...
But I can live with the following approach:

- Keep the report type AVP
- Keep the Report type as optional in the OC-OLR
- Clarify that the OC-OLR applies to the Origin-Host of the Answer when the report type is absent
- Multiple OLRs can be found in the answer only if the OLRs have different Report types e.g. response to an initial request (with only dest-Realm AVP) may contain overload status of the node and of the realm
- Remove "aggregated"

Is it OK for everyone?

Lionel

-----Message d'origine-----
De : DiME [mailto:[email-masked]] De la part de Jouni Korhonen
Envoyé : mercredi 27 novembre 2013 12:59
À : Ben Campbell; [email-masked]
Objet : Re: [Dime] Proposed Example Text for draft-docdt-dime-ovli-01

- Jouni

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Wiehe, Ulrich (NSN - DE/Munich)" <[email-masked]>
Date: Thu, 28 Nov 2013 08:30:40 +0000

Ben,

would you agree that your proposal is an optimization that could be introduced by extension?

Ulrich

---

From: Jouni Korhonen <[email-masked]>
Date: Thu, 28 Nov 2013 10:35:08 +0200

> - Keep the Report type as optional in the OC-OLR

Ok.

> - Clarify that the OC-OLR applies to the Origin-Host of the Answer when the report type is absent

Ok.

> - Multiple OLRs can be found in the answer only if the OLRs have different Report types e.g. response to an initial request (with only dest-Realm AVP) may contain overload status of the node and of…

Ok.

> - Remove "aggregated"

Ok.

> Is it OK for everyone?

I hope so ;)

- Jouni

---

From: "Nirav Salot (nsalot)" <[email-masked]>
Date: Thu, 28 Nov 2013 09:12:04 +0000

I am fine with all the principles mentioned below except for "Multiple OLRs can be found in the answer only if the OLRs have different Report types".
I am not 100% sure if the above restriction is needed yet. I will initiate a separate e-mail thread for the related discussion.

Regards,
Nirav.

-----Original Message-----
From: DiME [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, November 28, 2013 1:53 PM
To: Jouni Korhonen; Ben Campbell; [email-masked]
Subject: Re: [Dime] Proposed Example Text for draft-docdt-dime-ovli-01

Is it OK for everyone?

Lionel

-----Message d'origine-----
De : DiME [mailto:[email-masked]] De la part de Jouni Korhonen Envoyé : mercredi 27 novembre 2013 12:59 À : Ben Campbell; [email-masked] Objet : Re: [Dime] Proposed Example Text for draft-docdt-dime-ovli-01

- Jouni

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Jouni Korhonen <[email-masked]>
Date: Thu, 28 Nov 2013 11:21:46 +0200

> I am not 100% sure if the above restriction is needed yet. I will initiate a separate e-mail thread for the related discussion.

AFAIR we agreed that an OLR origin is implicitly learnt from the underlying Diameter command's Origin-Host/Realm, thus we cannot really distinguish between multiple OLRs of the same ReportType.

- Jouni

---

From: "TROTTIN, JEAN-JACQUES (JEAN-JACQUES)" <[email-masked]>
Date: Fri, 29 Nov 2013 17:09:22 +0000

I am fine with Lionel's proposals

Best regards

JJacques 

-----Message d'origine-----
De : DiME [mailto:[email-masked]] De la part de Jouni Korhonen
Envoyé : jeudi 28 novembre 2013 10:22
À : Nirav Salot
Cc : Ben Campbell; [email-masked]
Objet : Re: [Dime] Proposed Example Text for draft-docdt-dime-ovli-01

- Jouni