Subject: [Dime] draft-ietf-dime-overload-reqs-03: general comments

From: Andrew Booth <[email-masked]>
Date: Mon, 4 Feb 2013 23:50:35 -0500

First I'd like to say that the draft is quite good, and I think it serves a useful function.

I have a few questions and comments, and some more specific comments/questions that I'll include in a follow up email.  Sorry for leaving them so late.

General question/comment: I have a concern about the complexity that could be added processing possibly large or complex overload updates, especially if they require encyption or other end-to-end security mechanisms.

A related question is whether a client should get to keep control of its routing tables in the presence of overload updates.  Specifically, I assume the proposed mechanism must not require that a node process and respect all overload updates, regardless of the size and complexity of the update, the load on the client, etc.  Is this assumption correct?  Does it deserve explicit mention?

The following case may demonstrate the complexity concern, and might be worth including.  Consider a DRA in front of two PCRFs and doing (IP-CAN) session binding.  One PCRF becomes overloaded with 3 million (or 30 million, ... ) active sessions.  Updating overload state to the DRA is easy (PCRF->DRA: I'm overloaded).  Updating overload to other (remote) nodes in a timely fashion is trickier unless some kind of grouping was communicated when the sessions were first established.

What happens if no solutions can meet all requirements?  Do we revisit the requirements document?  Or are some of the requirements more "objectives" than "requirements"?

Is it clear what a proposed mechanism must to do demonstrate compliance to the requirements?  Specifically, consider REQ 3 and 17, is it clear how to decide if a proposed mechanism is compliant?
Do they require explicit experiments rather than just thought experiments (it is not clear to me what set of experiments will be sufficient)?

BTW, do requirements 3 and 17 pretty much force a proposed mechanism to include procedures (a default algorithm), and not just candidate messages, to demonstrate compliance?

2.2.  Agent Scenarios

Is it worth mentioning a scenario with more than one relay?  For instance, consider end nodes A and B with a meshed quad of "STPs" (Diameter Agents) in between.  This might be a useful configuration to keep in mind for instance if the proposed mechanism recommends (directly or indirectly) some kind of retransmission logic e.g. if a node sheds work by returning DIAMETER_UNABLE_TO_DELIVER, then Agents on the request path could retransmit on another path, potentially multiplying the number of messages that must be processed.

This might be an unusual configuration in Diameter, but it is simple may demonstrate issues that could exist when more and larger networks are hooked together.

4.1.  Problems with Implicit Mechanism

"Diameter treats the failure to receive an answer as a transport failure."

I'm not clear on this... do you mean if you fail to receive a single answer?  Or if a node is not providing any answers?  Perhaps a spec reference would make this clearer?

6.  Extensibility and Application Independence

"a single algorithm for handling overload may not be sufficient."

If a single algorithm is not sufficient, does that make it difficult for agents to detect and react to overload of adjacent servers?

---

From: Eric McMurry <[email-masked]>
Date: Tue, 5 Feb 2013 10:04:23 -0600

Hi Andrew,

thanks for your comments.  responses are inline.

Eric

> A related question is whether a client should get to keep control of its routing tables in the presence of overload updates.  Specifically, I assume the proposed mechanism must not require that a n…

There are certainly complexity concerns depending on how flexible the resulting mechanism is.  I think the assumption which may bear more clarification is that overload information sent should be relevant to the recipient.  That is, overload control information should not be sent to every client without regard to wether the scope of the information applies to that client.  Besides the extra processing on the client this would require, it would also have security implications.

It is important though that recipients take the specified action on the information they receive (the specified action is defined by the mechanism).  If they could arbitrarily ignore overload control information, that would make for a somewhat unpredictable and potentially ineffective mechanism.  This is up to the specification of the mechanism though.

> The following case may demonstrate the complexity concern, and might be worth including.  Consider a DRA in front of two PCRFs and doing (IP-CAN) session binding.  One PCRF becomes overloaded with…

Depending on how much topology that DRA is hiding, it could indeed be tricky.  One of the proposed mechanisms has a grouping mechanism as you describe for that situation.  The final mechanism will hopefully have some flexibility where the DRA implementors could come up with reasonable approaches for various overload scenarios.  Part of the exercise for development of a mechanism is to go through use cases such as this and evaluate what would be needed for overload control to mitigate them.  That has been done already for several and feedback from those went back into the requirements.  That is not to say at all though that everything has been thought of or covered.  For this particular case, I think the extensibility of algorithms and scopes should cover it even if something in the resulting base mechanism is not well suited.

> What happens if no solutions can meet all requirements?  Do we revisit the requirements document?  Or are some of the requirements more "objectives" than "requirements"?

Well, requirements are written at different levels (e.g. MUST vs. SHOULD) to cover this somewhat.  If there turn out to be glaring errors in the requirements later, then the best corse will have to be decided then.

> 
> Is it clear what a proposed mechanism must to do demonstrate compliance to the requirements?  Specifically, consider REQ 3 and 17, is it clear how to decide if a proposed mechanism is compliant?   Do they require explicit experiments rather than just thought experiments (it is not clear to me what set of experiments will be sufficient)?

Part of the process of developing the resulting mechanism will be to make calls on that.  If contributors to mechanism development think that experimentation is needed, there is nothing to stop them from doing that.  I think thought can go a long way towards satisfying reqs 3 and 17, but the real proof would be deployment.  I'm sure there would be multiple opinions on this.

The inclusion of a default algorithm is certainly implied, but we have an action to make this explicit so it is more clear.

> This might be an unusual configuration in Diameter, but it is simple may demonstrate issues that could exist when more and larger networks are hooked together.

Multiple agents have certainly been in the thinking.  The thinking on the single agent scenarios show in the draft was that the issues they brought up would cover more complicated topologies in general, as I recall.  That said, there are some very complex topologies possible that make overload control rather tricky, as you have pointed out.  If there are missing requirements for these scenarios, then lets go over them and get them covered. 

>    I'm not clear on this... do you mean if you fail to receive a single answer?  Or if a node is not providing any answers?  Perhaps a spec reference would make this clearer?

generally, it's lack of an answer to a WDR after timer Tw.  Implementations may be more aggressive than this though.  Ben, do you recall if we were talking about something other than that here?

> If a single algorithm is not sufficient, does that make it difficult for agents to detect and react to overload of adjacent servers?

I think the context of that statement had to do with variability of Diameter applications.  The thinking is that different applications may need different algorithms for handling overload control.  Extensions would also have to be negotiated, so any agent would only be expected to use algorithms that it supports.

---

From: [email-masked]
Date: Tue, 5 Feb 2013 13:59:05 -0500 (EST)

Hi Eric,

Your answers sound good, so I've removed most of them to shorten this
email.  Some more comments/questions inline.

>> I have a few questions and comments, and some more specific
>> comments/questions that I'll include in a follow up email.  Sorry for
>> leaving them so late.
>>
>> General question/comment: I have a concern about the complexity that
>> could be added processing possibly large or complex overload updates,
>> especially if they require encyption or other end-to-end security
>> mechanisms.
>> A related question is whether a client should get to keep control of its
>> routing tables in the presence of overload updates.  Specifically, I
>> assume the proposed mechanism must not require that a node process and
>> respect all overload updates, regardless of the size and complexity of
>> the update, the load on the client, etc.  Is this assumption correct?
>> Does it deserve explicit mention?
>
> There are certainly complexity concerns depending on how flexible the
> resulting mechanism is.  I think the assumption which may bear more
> clarification is that overload information sent should be relevant to the
> recipient.  That is, overload control information should not be sent to
> every client without regard to wether the scope of the information applies
> to that client.  Besides the extra processing on the client this would
> require, it would also have security implications.

That makes sense.  Is that something that should be expanded on in this
document?

>
> It is important though that recipients take the specified action on the
> information they receive (the specified action is defined by the
> mechanism).  If they could arbitrarily ignore overload control
> information, that would make for a somewhat unpredictable and potentially
> ineffective mechanism.  This is up to the specification of the mechanism
> though.

Agreed, it is certainly undesirable if a client/agent ignores an update.

On the other hand, for your consideration:

  REQ XX: A node should not get an unfair advantage by advertising
          support for the overload procedures and then ignoring
          overload indications.
          This could be the result of a malicious or compromised node,
          or of simple misconfiguration (sending load information of
          a type that a node does not support, mismatching
          authentication information, etc).

Practically speaking, any such weakness should probably be caught by a
review of the proposed mechanism.

[snip]
>>
>> 2.2.  Agent Scenarios
>>
>> Is it worth mentioning a scenario with more than one relay?  For
>> instance, consider end nodes A and B with a meshed quad of "STPs"
>> (Diameter Agents) in between.  This might be a useful configuration to
>> keep in mind for instance if the proposed mechanism recommends (directly
>> or indirectly) some kind of retransmission logic e.g. if a node sheds
>> work by returning DIAMETER_UNABLE_TO_DELIVER, then Agents on the request
>> path could retransmit on another path, potentially multiplying the
>> number of messages that must be processed.
>>
>> This might be an unusual configuration in Diameter, but it is simple may
>> demonstrate issues that could exist when more and larger networks are
>> hooked together.
>
> Multiple agents have certainly been in the thinking.  The thinking on the
> single agent scenarios show in the draft was that the issues they brought
> up would cover more complicated topologies in general, as I recall.  That
> said, there are some very complex topologies possible that make overload
> control rather tricky, as you have pointed out.  If there are missing
> requirements for these scenarios, then lets go over them and get them
> covered.

Ok.  Is it worth including something like the following as an
informational comment?  Possibly towards the end of 4.2 or as a 4.2.1? 
It's a little longer than I'd like, but it's a quick read.

In some network configurations, the retry mechanisms built in to the base
Diameter protocol can cause message amplification when a node fails, with
the possibility of propagating an overload condition to other nodes.
Consider for example the scenario in [RFC6733] section 7, Figure 7,
reproduced here as Figure X.

                          1. Request        +---------+ Link Broken
                +-------------------------->|Diameter |----///----+
                |     +---------------------|         |           v
         +------+--+  | 2. answer + 'E' set | Relay 2 |     +--------+
         |Diameter |<-+ (Unable to Forward) +---------+     |Diameter|
         |         |                                        |  Home  |
         | Relay 1 |--+                     +---------+     | Server |
         +---------+  |   3. Request        |Diameter |     +--------+
                      +-------------------->|         |           ^
                                            | Relay 3 |-----------+
                                            +---------+

        Figure X: Example of Protocol Error Causing Message Amplification

In the success path via Relay 3, Relay 1 generates 1 request and receives
on answer.
If the request is instead sent via Relay 2, Relay 1 generates 2 requests
and processes 1 error and 1 answer.
If the Diameter Home Server is unavailable via either path, Relay 1
generates 2 requests and processes 2 errors.
If there were N intermediate relays connected to Relay 1 and the Diameter
Home Server were unavailable, then Relay 1 could generate N requests and
process N errors.

Resolving this issue is out of scope of the present work, though the
situation may be improved as a side effect of the overload mechanism.

A proposed overload handling mechanism should consider possible message
amplification if it adds additional retransmissions, or if it uses
Diameter error codes that prompt automatic retransmissions.

Thoughts?
Andrew

---

From: Eric McMurry <[email-masked]>
Date: Sat, 16 Feb 2013 15:43:03 -0600

Hi Andrew,

please see inline.

Thanks,

Eric

> That makes sense.  Is that something that should be expanded on in this

Not sending inappropriate information is covered in the security considerations.  I think this is probably worth more elaboration, but the mechanism specification may be a better place to do this, since it is placing constraints on the implementors of the mechanism.

> review of the proposed mechanism.

I think a mechanism would have a hard time efficiently preventing this.  A node implementing a mechanism though could detect this and take other actions if enforcement was deemed worthwhile.

>         +---------+  |   3. Request        |Diameter |     +--------+

I think this would happen only once (or a relatively small number of times) for each failure.  That is, Relay 1, when it receives the error response from Relay 2 could decide not to send similar requests down that path again.  So, the next time Relay 1 receives a request that would route to the Diameter Home Server, it could send it directly to Relay 3.