Subject: Re: [Dime] Fw: Re: draft-ietf-dime-overload-reqs-03: some REQ comments and questions

From: Andrew Booth <[email-masked]>
Date: Wed, 20 Feb 2013 14:29:53 -0500

Hi Eric,

Sorry for the delay, comments inline.

On Wed, Feb 20, 2013 at 11:12 AM, Andrew Booth <[email-masked]> wrote:

>
>
> -----Forwarded by Andrew Booth/PTI on 02/20/2013 11:10AM -----
> To: Andrew Booth <[email-masked]>
> From: Eric McMurry <[email-masked]>
> Date: 02/16/2013 12:55PM
> Cc: [email-masked], [email-masked]
> Subject: Re: draft-ietf-dime-overload-reqs-03: some REQ comments and
> questions
>
> Hi Andrew,
>
> Sorry for the delay.  Please see inline.
>
> Thanks,
>
> Eric
>
>
> On Feb 4, 2013, at 23:13 , Andrew Booth < [email-masked]> wrote:
>
>  Some minor questions and comments on various requirements.
>
> REQ 7: I'm not sure that "the system remains stable" adds much on its own,
> removing it doesn't seem to lose much:
>       ALT REQ 7: The overload control mechanism and any associated
>                default algorithm(s) MUST ensure that when the
>                offered load drops from above the overall
>                capacity of the network to below the overall
>                capacity, the throughput MUST stabilize and become
>                equal to the offered load.  Note that this also
>                requires that the mechanism MUST allow nodes to
>                shed load without introducing oscillations.
>
>
> The text after the first sentence was intended to elaborate on it.  It
> would probably read about the same either way, but I don't think it
> detracts having it there.
>

Ok.

>
> REQ 7: query: Does this requirement imply that the proposed mechanism must
> define one or more options for shedding load (as opposed to simply
> requesting the clients to slow down)?  Is it accurate to say that
> processing TLS to get at the Diameter message may be a substantial part of
> the message processing time, so a node may have limited ability to
> distinguish between Diameter messages when shedding load?  This query
> applies to REQ 22 also.
>
>
> I don't think this requirement implies that there be multiple algorithms,
> but that is called out elsewhere (at least the ability to add them).
>

I think my question was poorly phrased, but perhaps it is not relevant
anyway.

To me there is a difference between shedding work locally (returning
errors, throttling inbound messages, etc) vs sending load information to a
peer who can reduce the amount they send.  I wasn't clear which version of
"shedding work" was referred to by REQ 7.

Should the mechanism provide a way for an overloaded server to shed work
locally?  Or is that up to the mechanism?  Both the existing proposals (as
far as I can tell) only specify the behavior of a node that is sending a
request; one of the proposals additionally specifies an error code that
could potentially be used by a server, but no such procedures are specified
(again, unless I missed something).

> Impact of TLS is rather implementation dependent.  I'm not sure what we
> could say about that here (or that we should)
>

Ok.

>
>
>
> REQ 9: Is this simpler/clearer?  Or did I miss some cross references?
>       ALT REQ 9:   The mechanism MUST function across fully loaded as well
> as
>                quiescent transport connections.  This is partially derived
>                from REQ 7 above.
>
>
> makes sense.  good catch.
>
>
> REQ 13: consider adding "The mechanism SHOULD NOT introduce substantial
> additional work for nodes that are not in overload; in particular it should
> not cause nodes to become overloaded".
>
>
> not causing other nodes to be come overloaded is covered in req 12 and 23.
>  No introducing substantial work when not overloaded is probably a subset
> of what is there.  I think that was in an earlier version of the draft and
> was considered redundant.  Perhaps others remember that better.
>

Ok.

>
>
> REQ 20: does "actual overload" deserve more clarification?  For example,
> would any or all of the following would represent "actual overload": CPU
> usage, a deep send queue, messages incoming at an unexpectedly high rate?
>  Or is this something that would be defined by the proposed mechanism?  Or
> have I misunderstood completely?
>
>
> the word actual may not actually add anything here.  All of the things you
> mentioned could be causes of overload.  I think the intent though is that
> existing Diameter errors not be overloaded to be used for overload.  Any
> overload control mechanism needs to be clear about what is being
> communicated.
>

Ok.  I'm fine with leaving it as it is, or removing "actual".

Aside: I'm left wondering though whether the wording of this requirement
prevents overload signaling from being used by an agent to signal
unavailability of a server via that agent.  Allowing node unavailability to
be signaled as overload could have some nice benefits.

>
>
> REQ 21: Does this imply that a node must be able to deduce overload of a
> peer or indirectly accessible node based on local metrics only (response
> time, send queue depth, etc)?  Since "The mechanism MUST properly function
> in these cases", does that require that explicit messaging is unnecessary?
>
>
> perhaps "properly function" should be stated differently.  The intent here
> is that the mechanism not break things in this case and that the overload
> is handled at least as well as it would have been were the overload control
> signaling intact.  The language in req 17 is close to this point I think.
>  How about we barrow that language?
>
> REQ 21:  In cases where a network node fails, is so overloaded that
>             it cannot process messages, or cannot communicate due to a
>             network failure, it may not be able to provide explicit
>             indications of the nature of the failure or its levels of
>             congestion.  The mechanism MUST result
>             in at least as much useful throughput as would have resulted
>             if the overload control signaling were present.
>
>
> This may imply the actions you mention, as so other requirements around
> functioning in environments with nodes that do, and do not support the
> mechanism, and fairness in that case.  Overload can be mitigated somewhat
> without overload control signaling, but that is not sufficient to handle
> issues that are occurring on networks now.  A good bit of the front matter
> in the draft discusses this point.
>

It seems to me that if overload signaling is worthwhile, it must be because
it increases useful throughput (at least in some cases).
So I would expect that useful throughput without overload procedures <=
useful throughput with only local overload procedures <= useful throughput
with explicit signaling.  Hence I would expect something more like this:

   ALT REQ 21:  In cases where a network node fails, is so overloaded that
            it cannot process messages, or cannot communicate due to a
            network failure, it may not be able to provide explicit
            indications of the nature of the failure or its levels of
            congestion.  In this case, actions based on local metrics at
the affected
            or adjacent nodes may still provide some benefit.
            The mechanism MUST result in at least as much useful throughput
as
            if the affected nodes did not implement the mechanism.
            The mechanism SHOULD result in more useful throughput than if
none
            of the nodes implemented the mechanism.

Does that make sense?  Or have I misunderstood completely?

>
> REQ 29: query: why do we need to know who sent the overload info (note
> that the security concerns are addressed in other requirements)?  Do we
> lose anything by removing the first sentence?
>       ALT REQ 29:  The mechanism MUST allow a node to distinguish between
>                overload at a next-hop peer from overload at a node upstream
>                of the peer.  For example, in Figure 5, the client must not
>                mistake overload at server 1 for overload at the agent,
>                whether or not the agent supports the mechanism.
> REQ 29: nit: why the cross reference to REQ 4?
>
>
> I agree with this and would go further.  Upon reading this again, I don't
> think req 29 is adding anything.  Perhaps we should just add this example
> to 33 and remove 29.
>

That works for me.

>
>
> REQ 33: "The mechanism MUST allow Diameter nodes to indicate overload with
> sufficient granularity to allow clients to take action based on the
> overloaded resources without forcing available capacity to go unused.":
> this wording seems to imply that the mechanism must support overload
> specification of arbitrary granularity, with a corresponding increase in
> the complexity required on implementations.  Even supporting "Diameter
> session" could force a node to maintain session state when it is otherwise
> not required (more an issue for Agents than end nodes), at a corresponding
> increase in the cost and complexity of that node.  Is there anything to be
> concerned about here?  Or is REQ 13 sufficient to limit complexity?
>
>
> I think req 13 probably covers this.  I understand your concern though,
> and share it.  There is potential for significant complexity.  There have
> been a lot of comments on this point and I think there will be a fair
> amount of thinking going into appropriate scoping of overload control
> information in ways that balance work imposed on nodes with capacity left
> on the table. Perhaps adding the word "unreasonably" in between "without"
> and "forcing" would make folks feel more comfortable that the necessary
> wiggle room is here for the mechanism specifiers.
>

That works.

>
> REQ 35: query: explicitly mention that end-to-end security is required
> here?  Or is it not?
>
> REQ 35: query: does this pretty much rule out strictly a hop-by-hop
> approach?
>
>
> Jouni answered these last two and that looked reasonable to me.
>

Ok.

Thanks,
Andrew

>  Thanks,
> Andrew

---

From: Eric McMurry <[email-masked]>
Date: Wed, 20 Feb 2013 15:45:58 -0600

Hi Andrew,

comments inline.

Thanks,

Eric

On Feb 20, 2013, at 13:29 , Andrew Booth <[email-masked]> wrote:

> Subject: Re: draft-ietf-dime-overload-reqs-03: some REQ comments and questions
> 
> Hi Andrew,
> 

[...]

>  
> 
>> REQ 7: query: Does this requirement imply that the proposed mechanism must define one or more options for shedding load (as opposed to simply requesting the clients to slow down)?  Is it accurate to say that processing TLS to get at the Diameter message may be a substantial part of the message processing time, so a node may have limited ability to distinguish between Diameter messages when shedding load?  This query applies to REQ 22 also.
> 
> I don't think this requirement implies that there be multiple algorithms, but that is called out elsewhere (at least the ability to add them).
> 
> I think my question was poorly phrased, but perhaps it is not relevant anyway.
> 
> To me there is a difference between shedding work locally (returning errors, throttling inbound messages, etc) vs sending load information to a peer who can reduce the amount they send.  I wasn't clear which version of "shedding work" was referred to by REQ 7.
> 
> Should the mechanism provide a way for an overloaded server to shed work locally?  Or is that up to the mechanism?  Both the existing proposals (as far as I can tell) only specify the behavior of a node that is sending a request; one of the proposals additionally specifies an error code that could potentially be used by a server, but no such procedures are specified (again, unless I missed something).

I don't think the requirements are specific on this point.  A mechanism could use local actions as well as remote requests for overload mitigation.  The mechanism has to support the remote requests at a minimum though.

[...]

> 
>  
> 
>> 
>> REQ 20: does "actual overload" deserve more clarification?  For example, would any or all of the following would represent "actual overload": CPU usage, a deep send queue, messages incoming at an unexpectedly high rate?  Or is this something that would be defined by the proposed mechanism?  Or have I misunderstood completely?
> 
> the word actual may not actually add anything here.  All of the things you mentioned could be causes of overload.  I think the intent though is that existing Diameter errors not be overloaded to be used for overload.  Any overload control mechanism needs to be clear about what is being communicated. 
> 
> Aside: I'm left wondering though whether the wording of this requirement prevents overload signaling from being used by an agent to signal unavailability of a server via that agent.  Allowing node unavailability to be signaled as overload could have some nice benefits.

That is certainly not the intent.  I can see where your comment comes from.  Do others think we should change the wording here?

> 
>  
> 
>> 
>> REQ 21: Does this imply that a node must be able to deduce overload of a peer or indirectly accessible node based on local metrics only (response time, send queue depth, etc)?  Since "The mechanism MUST properly function in these cases", does that require that explicit messaging is unnecessary?
> 
> perhaps "properly function" should be stated differently.  The intent here is that the mechanism not break things in this case and that the overload is handled at least as well as it would have been were the overload control signaling intact.  The language in req 17 is close to this point I think.  How about we barrow that language?
> 
> This may imply the actions you mention, as so other requirements around functioning in environments with nodes that do, and do not support the mechanism, and fairness in that case.  Overload can be mitigated somewhat without overload control signaling, but that is not sufficient to handle issues that are occurring on networks now.  A good bit of the front matter in the draft discusses this point.
> 
> It seems to me that if overload signaling is worthwhile, it must be because it increases useful throughput (at least in some cases).
> So I would expect that useful throughput without overload procedures <= useful throughput with only local overload procedures <= useful throughput with explicit signaling.  Hence I would expect something more like this:
> 
>             congestion.  In this case, actions based on local metrics at the affected
>             The mechanism MUST result in at least as much useful throughput as
>             The mechanism SHOULD result in more useful throughput than if none

I think this shouldn't presuppose actions based on local metrics.  It is possible for the mechanism to do other things such as using other paths to communicate information.  That said, if there truly is no path to communicate overload information, the overload mechanism can't really do more than could be done now, in its absence.  So I'm not sure the SHOULD on the end would add anything.

[...]

> 
> 
>  
> 
>> 
>> REQ 33: "The mechanism MUST allow Diameter nodes to indicate overload with sufficient granularity to allow clients to take action based on the overloaded resources without forcing available capacity to go unused.": this wording seems to imply that the mechanism must support overload specification of arbitrary granularity, with a corresponding increase in the complexity required on implementations.  Even supporting "Diameter session" could force a node to maintain session state when it is otherwise not required (more an issue for Agents than end nodes), at a corresponding increase in the cost and complexity of that node.  Is there anything to be concerned about here?  Or is REQ 13 sufficient to limit complexity?
> 
> I think req 13 probably covers this.  I understand your concern though, and share it.  There is potential for significant complexity.  There have been a lot of comments on this point and I think there will be a fair amount of thinking going into appropriate scoping of overload control information in ways that balance work imposed on nodes with capacity left on the table. Perhaps adding the word "unreasonably" in between "without" and "forcing" would make folks feel more comfortable that the necessary wiggle room is here for the mechanism specifiers.
> 
> That works.

okay, I think we can add that to the todo list.
[...]

---

From: Andrew Booth <[email-masked]>
Date: Thu, 21 Feb 2013 14:14:47 -0500

Hi Eric,

Comments inline.

Thanks,
Andrew

On Wed, Feb 20, 2013 at 4:45 PM, Eric McMurry <[email-masked]> wrote:
>> is that the mechanism not break things in this case and that the
overload is
>> handled at least as well as it would have been were the overload control
the
> affected
> I think this shouldn't presuppose actions based on local metrics.  It is
> possible for the mechanism to do other things such as using other paths to
> communicate information.  That said, if there truly is no path to
> communicate overload information, the overload mechanism can't really do
> more than could be done now, in its absence.  So I'm not sure the SHOULD
on
> the end would add anything.
>

I'd be ok with simply removing the mention of local metrics and the
SHOULD.  However, that is somewhat weaker than what you suggested in the
REQ 21 above.  On the other hand, REQ 21 above confused me a bit on
transitional issues (the time between when the node fails and others find
out about it, especially if the adjacent nodes do not support the mechanism
and can't signal others on behalf of the failed node).
Does this come closer to capturing the intent of REQ 21, while satisfying
the transitional issues?  Or have I misunderstood the intent?

ALT REQ 21: In cases where a network node X fails, is so overloaded that
                    it cannot process messages, or cannot communicate due
to a
                    network failure, it may not be able to provide explicit
                    indications of the nature of the failure or its levels
of
                    congestion.
                    In this case, the mechanism MUST result in at least as
much
                    useful throughput as if none of the nodes implemented
the mechanism.
                    Also, if all relevant nodes support the mechanism, and
once sufficient nodes
                    have discovered that X has become unavailable, the
mechanism MUST
                    result in at least as much useful throughput as would
have resulted if X
                    had successfully signaled complete overload.

---

From: Eric McMurry <[email-masked]>
Date: Thu, 21 Feb 2013 13:35:31 -0600

Hi Andrew,

I'm a bit lost on what you are wanting to add here.  The gist of this requirement is that the mechanism shouldn't make things worse in this situation, and no more than that.  I don't think we can strengthen that to the point that the mechanism will work the same in cases where signaling is not possible as in cases where it is, and requiring that it find ways to route overload signaling information around the issues is probably saying too much about the implementation.

I'm also not understanding your comment on transitional issues and how it relates here.

Sorry, I'm a bit slow today.  you'll have to bear with me :-)

Thanks,

Eric

On Feb 21, 2013, at 13:14 , Andrew Booth <[email-masked]> wrote:

> >> is that the mechanism not break things in this case and that the overload is
> > more than could be done now, in its absence.  So I'm not sure the SHOULD on
> I'd be ok with simply removing the mention of local metrics and the SHOULD.  However, that is somewhat weaker than what you suggested in the REQ 21 above.  On the other hand, REQ 21 above confused me a bit on transitional issues (the time between when the node fails and others find out about it, especially if the adjacent nodes do not support the mechanism and can't signal others on behalf of the failed node).
> Does this come closer to capturing the intent of REQ 21, while satisfying the transitional issues?  Or have I misunderstood the intent?
> 
>                     In this case, the mechanism MUST result in at least as much
>                     useful throughput as if none of the nodes implemented the mechanism.
>                     Also, if all relevant nodes support the mechanism, and once sufficient nodes
>                     have discovered that X has become unavailable, the mechanism MUST
>                     result in at least as much useful throughput as would have resulted if X

---

From: Andrew Booth <[email-masked]>
Date: Thu, 21 Feb 2013 15:15:32 -0500

When I re-read the thread, I was worried that I'd neutered REQ 21 away from
what you originally intended.
Sounds like I'm just over-complicating things.  Let's take a step back.

What is your current favorite candidate for REQ 21?

Andrew

On Thu, Feb 21, 2013 at 2:35 PM, Eric McMurry <[email-masked]> wrote:

> Hi Andrew,
>
> I'm a bit lost on what you are wanting to add here.  The gist of this
> requirement is that the mechanism shouldn't make things worse in this
> situation, and no more than that.
>

I don't think we can strengthen that to the point that the mechanism will
> work the same in cases where signaling is not possible as in cases where it
> is, and requiring that it find ways to route overload signaling information
> around the issues is probably saying too much about the implementation.
>

---

From: Eric McMurry <[email-masked]>
Date: Thu, 21 Feb 2013 14:32:28 -0600

okay.  Here's what I have in my working copy of the 04 reqs:

REQ 21:  In cases where a network node fails, is so overloaded that
            it cannot process messages, or cannot communicate due to a
            network failure, it may not be able to provide explicit
            indications of the nature of the failure or its levels of
            congestion.  The mechanism MUST result in at least as much
            useful throughput as would have resulted if the overload
            control signaling were present.

This wasn't too far from the original, but I think the wording is more clear.  I think this was from the first or second round of our conversation.

thanks,

Eric

On Feb 21, 2013, at 14:15 , Andrew Booth <[email-masked]> wrote:

> When I re-read the thread, I was worried that I'd neutered REQ 21 away from what you originally intended.
> I'm a bit lost on what you are wanting to add here.  The gist of this requirement is that the mechanism shouldn't make things worse in this situation, and no more than that.
>  
> I don't think we can strengthen that to the point that the mechanism will work the same in cases where signaling is not possible as in cases where it is, and requiring that it find ways to route overload signaling information around the issues is probably saying too much about the implementation.
>

---

From: Eric McMurry <[email-masked]>
Date: Thu, 21 Feb 2013 14:42:18 -0600

Ben caught me in a thinko on this one.  Here is the corrected proposal:

            control mechanism was not in place.

On Feb 21, 2013, at 14:32 , Eric McMurry <[email-masked]> wrote:

> _______________________________________________
> DiME mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dime

---

From: Andrew Booth <[email-masked]>
Date: Thu, 21 Feb 2013 20:23:17 -0500

That works for me.

Thanks!
Andrew

On Thu, Feb 21, 2013 at 3:42 PM, Eric McMurry <[email-masked]> wrote:

> This wasn't too far from the original, but I think the wording is more
> clear.  I think this was from the first or second round of our conversation.
>
> thanks,
>
> Eric
>
>
>
> On Thu, Feb 21, 2013 at 2:35 PM, Eric McMurry <[email-masked]>wrote:
>
>> Hi Andrew,
>>
>> >             congestion.  In this case, actions based on local metrics
>> at the
>> > affected

---

From: Eric McMurry <[email-masked]>
Date: Sat, 23 Feb 2013 20:33:17 -0600

I think the discussion on req 20 was lost in the flurry of other stuff, so I am reviving it:

> That is certainly not the intent.  I can see where your comment comes from.  Do others think we should change the wording here?

I'll comment on my own comment.  This is difficult to interpret.  How about:

Any explicit overload indication MUST be clearly distinguishable from other errors reported via Diameter.

---

From: Ben Campbell <[email-masked]>
Date: Sun, 24 Feb 2013 09:26:16 -0600

> Any explicit overload indication MUST be clearly distinguishable from other errors reported via Diameter.

WFM

---

From: "DOLLY, MARTIN C" <[email-masked]>
Date: Sun, 24 Feb 2013 17:31:36 +0000

Ben,

What is the need for this, as one does not have to with the other?

If you think so , please explain.

Thanks,

Martin 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ben Campbell
Sent: Sunday, February 24, 2013 10:26 AM
To: Eric McMurry
Cc: [email-masked]
Subject: Re: [Dime] draft-ietf-dime-overload-reqs-03: some REQ comments and questions

WFM

---

From: Eric McMurry <[email-masked]>
Date: Sun, 24 Feb 2013 12:01:40 -0600

Hi Martin,

Many times when functionality is added to protocols existing machinery is reused for the new purpose.  We wanted to ensure that if this is done, it is done in a way that doesn't cause confusion.  For example, using DIAMETER_TOO_BUSY would not work well as it has other semantics and is not suited for overload control (and it has multiple interpretations already).

Thanks,

Eric

---

From: Ben Campbell <[email-masked]>
Date: Sun, 24 Feb 2013 13:22:34 -0600

If I recall correctly, the original intent for this was to make sure that, if a mechanism was chosen that used Diameter error codes to indicate overload, the codes would be sufficiently granular to make sure that overload was not confused with other error types. As an analogy, some SIP implementations would use 500 to indicate certain overload conditions, but also used 500 for other things. That made it hard for the UAC to tell if the problem was an error with the specific request, or an overload condition that should apply to other requests.

OTOH, since the proposed mechanisms don't work that way, it may not matter. People clearly read too much into the original wording. Eric's proposal reduces the potential for misunderstanding. But I would also be okay with just removing it.

---

From: Andrew Booth <[email-masked]>
Date: Mon, 25 Feb 2013 10:24:01 -0500

I like the new wording better than the old wording.

My preference would be to use the new wording rather than removing the
requirement  (as per your example, it wouldn't be the first time that
re-used error codes cause complications), but it is only a weak
preference.  I won't object if the end decision is simply to remove
the requirement.

Thanks,
Andrew

---

From: Eric McMurry <[email-masked]>
Date: Mon, 25 Feb 2013 14:12:29 -0600

I did a quick spin to get this change for req 20 in before the IETF 86 deadline.  It is the only change.

Thanks,

Eric